[PATCH 0/7] Create ext* filesystems using debugfs

[PATCH 0/7] Create ext* filesystems using debugfs

[Robert Yang]

===Contents===
  * Summary
  * Brief design
  * Test info
  * The rootfs generation time changes
  * The rootfs size changes

* Summary:
  The ext3 and ext4 rootfs generation relied on genext2fs and flipping
  some bits to "convert" the ext2 filesystem to ext3 and ext4 in the past,
  now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
  to copy the files to the image.

* Brief design:
  - Upgrade the e2fsprogs to the update to date version (the git
    repo) which contains the "symlink" command in debugfs shell that we
    need.
  - Replace the current genext2fs command with mkdebugfs.sh in
    image_types.bbclass.
  - The new steps to generate the root filesystem are:
    -> use "dd" command to make rootfs.ext* file
    -> run "mkfs.ext* rootfs.ext*" to create the filesystem
    -> run the mkdebugfs.sh to generate the root filesystem

* Test info: (MACHINE = qemuarm, IMAGE_FSTYPES = "ext2 ext3 ext4 tar.bz2")
  $ bitbake core-image-minimal core-image-sato meta-toolchain-sdk core-image-sato-sdk
  $ runqemu runqemu qemuarm core-image-sato ext3
  $ runqemu runqemu qemuarm core-image-minimal ext3

  All of them are OK

* The rootfs generation time changes
  - For a core-image-minimal generation (IMAGE_FSTYPES="tar.bz2 ext3"):
  
    Before:
    $ time bitbake core-image-minimal
    
    real    1m10.823s
    user    0m37.108s
    sys     0m15.894s
    
    After:
    $ time bitbake core-image-minimal
    
    real    1m17.501s
    user    0m29.304s
    sys     0m20.731s
    
    # 7 seconds lost.
  
  - For a core-image-sato generation (IMAGE_FSTYPES="tar.bz2 ext3"):
  
    Before:
    $ time bitbake core-image-sato
    
    real    11m10.645s
    user    2m43.503s
    sys     1m1.589s
    
    After:
    $ time bitbake core-image-sato
    
    real    11m53.131s
    user    3m18.988s
    sys     2m8.350s
    
    # 43 seconds lost.
  
* The rootfs size changes
  - The image size are the same by "ls -h", but different by "du -sh":
    (core-image-sato)
  
    $ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}'
    357M BEFORE.rootfs.ext3
    357M AFTER.rootfs.ext3
    
    $ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
    238M    BEFORE.rootfs.ext3
    357M    AFTER.rootfs.ext3
    
    It seems that the genext2fs has optimized the rootfs generation.

// Robert

The following changes since commit 2cc4fe4a0874c42421b1bf3fa100160a9e60a9da:

  upstream_tracking.inc: Coonectivity and multimedia packages updates (2013-02-25 05:58:20 -0800)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/e2fsprogs
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/e2fsprogs

Robert Yang (7):
  e2fsprogs: upgrade to the git version (rename only)
  e2fsprogs: upgrade to the git version
  e2fsprogs: add the original mkdebugfs.sh
  mkdebugfs.sh: convert the tab to 4 spaces
  mkdebugfs.sh: several fixes
  e2fsprogs: ship mkfsdebug.sh
  image_types.bbclass: replace genext2fs with mkdebugfs.sh

 meta/classes/image_types.bbclass                   | 45 +++++-------
 meta/recipes-devtools/e2fsprogs/e2fsprogs.inc      |  4 +-
 .../{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4   |  0
 .../fallocate.patch                                |  0
 .../e2fsprogs/e2fsprogs/mkdebugfs.sh               | 82 ++++++++++++++++++++++
 .../{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch    |  0
 .../remove.ldconfig.call.patch                     |  0
 .../{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb}      |  6 +-
 8 files changed, 106 insertions(+), 31 deletions(-)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4 (100%)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/fallocate.patch (100%)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch (100%)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/remove.ldconfig.call.patch (100%)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb} (92%)

-- 
1.7.11.2

[PATCH 1/7] e2fsprogs: upgrade to the git version (rename only)

[Robert Yang]

We will replace the "genext2fs" command in the bbclass with the debugfs
command from e2fsprogs, and we need the "symlink" subcommand in the
debugfs which only exists in the up-to-date git repo currently, so
change the source to use the git repo.

renamed:    e2fsprogs-1.42.6/acinclude.m4 -> e2fsprogs/acinclude.m4
renamed:    e2fsprogs-1.42.6/fallocate.patch -> e2fsprogs/fallocate.patch
renamed:    e2fsprogs-1.42.6/mkdir.patch -> e2fsprogs/mkdir.patch
renamed:    e2fsprogs-1.42.6/remove.ldconfig.call.patch -> e2fsprogs/remove.ldconfig.call.patch
renamed:    e2fsprogs_1.42.6.bb -> e2fsprogs_git.bb

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4                | 0
 .../e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/fallocate.patch             | 0
 .../e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch                 | 0
 .../e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/remove.ldconfig.call.patch  | 0
 meta/recipes-devtools/e2fsprogs/{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb} | 0
 5 files changed, 0 insertions(+), 0 deletions(-)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4 (100%)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/fallocate.patch (100%)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch (100%)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/remove.ldconfig.call.patch (100%)
 rename meta/recipes-devtools/e2fsprogs/{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb} (100%)

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/acinclude.m4 b/meta/recipes-devtools/e2fsprogs/e2fsprogs/acinclude.m4
similarity index 100%
rename from meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/acinclude.m4
rename to meta/recipes-devtools/e2fsprogs/e2fsprogs/acinclude.m4
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/fallocate.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs/fallocate.patch
similarity index 100%
rename from meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/fallocate.patch
rename to meta/recipes-devtools/e2fsprogs/e2fsprogs/fallocate.patch
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/mkdir.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdir.patch
similarity index 100%
rename from meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/mkdir.patch
rename to meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdir.patch
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/remove.ldconfig.call.patch b/meta/recipes-devtools/e2fsprogs/e2fsprogs/remove.ldconfig.call.patch
similarity index 100%
rename from meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.6/remove.ldconfig.call.patch
rename to meta/recipes-devtools/e2fsprogs/e2fsprogs/remove.ldconfig.call.patch
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.6.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
similarity index 100%
rename from meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.6.bb
rename to meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
-- 
1.7.11.2

[PATCH 2/7] e2fsprogs: upgrade to the git version

[Robert Yang]

Change the source from tarball to git repo, and upgrade it to the
up-to-date git version.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/recipes-devtools/e2fsprogs/e2fsprogs.inc    | 4 +++-
 meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb | 4 ++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs.inc b/meta/recipes-devtools/e2fsprogs/e2fsprogs.inc
index aea74f7..d8ae037 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs.inc
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs.inc
@@ -19,8 +19,10 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=b48f21d765b875bd10400975d12c1ca2 \
 SECTION = "base"
 DEPENDS = "util-linux"
 
-SRC_URI = "${SOURCEFORGE_MIRROR}/e2fsprogs/e2fsprogs-${PV}.tar.gz \
+SRC_URI = "git://git.kernel.org/pub/scm/fs/ext2/${BPN}.git;protocol=git \
            file://mkdir.patch"
 
+S = "${WORKDIR}/git"
+
 inherit autotools gettext
 
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
index e6d90a1..70255ce 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
@@ -6,8 +6,8 @@ SRC_URI += "file://acinclude.m4 \
             file://remove.ldconfig.call.patch \
 "
 
-SRC_URI[md5sum] = "9e444c240c1001b3292d108fbad0f49c"
-SRC_URI[sha256sum] = "726b5919edeaee5b077d7a5b602284f1453ea31cf499efa433af26c679b667a5"
+SRCREV = "fca8b1b2416c384b91b4fe00d6d37cfccb8fb3d5"
+PV = "1.43-WIP+git${SRCPV}"
 
 EXTRA_OECONF += "--libdir=${base_libdir} --sbindir=${base_sbindir} --enable-elf-shlibs --disable-libuuid --disable-uuidd"
 EXTRA_OECONF_darwin = "--libdir=${base_libdir} --sbindir=${base_sbindir} --enable-bsd-shlibs"
-- 
1.7.11.2

[PATCH 3/7] e2fsprogs: add the original mkdebugfs.sh

[Robert Yang]

This script is from Darren Hart, it will be used for creating the extX
filesystem from a given directory, which will replace the genext2fs in
image_types.bbclass.

It isn't shipp in e2fsprogs package at the moment, just add it intact,
and will do more work on it in the following commits.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs/e2fsprogs/mkdebugfs.sh               | 49 ++++++++++++++++++++++
 1 file changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
new file mode 100644
index 0000000..5dfa380
--- /dev/null
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
@@ -0,0 +1,49 @@
+#!/bin/sh
+SRCDIR=$1
+DEVICE=$2
+DEBUGFS="debugfs/debugfs"
+
+yes | mkfs.ext4 test.img
+
+{
+	CWD="/"
+	find $SRCDIR | while read FILE; do
+		#TGT=${FILE#$SRCDIR}
+		TGT=$(basename "${FILE#$SRCDIR}")
+		DIR=$(dirname "${FILE#$SRCDIR}")
+
+		# Skip the root dir
+		if [ -z "$TGT" ]; then
+			continue
+		fi
+
+		if [ "$DIR" != "$CWD" ]; then
+			echo "cd $DIR"
+			CWD="$DIR"
+		fi
+
+		case $(stat -c "%F" $FILE) in
+		"directory")
+			echo "mkdir $TGT"
+			;;
+		"regular file")
+			echo "write $FILE $TGT"
+			;;
+		"symbolic link")
+			LINK_TGT=$(ls -l $FILE | sed -e 's/.*-> //')
+			echo "symlink $TGT $LINK_TGT"
+			;;
+		"block special file")
+			DEVNO=$(stat -c "%t %T" $FILE)
+			echo "mknod $TGT b $DEVNO"
+			;;
+		"character special file")
+			DEVNO=$(stat -c "%t %T" $FILE)
+			echo "mknod $TGT c $DEVNO"
+			;;
+		*)
+			echo "Unknown file $FILE" 1>&2
+			;;
+		esac
+	done
+} | $DEBUGFS -w -f /dev/stdin $DEVICE
-- 
1.7.11.2

[PATCH 4/7] mkdebugfs.sh: convert the tab to 4 spaces

[Robert Yang]

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs/e2fsprogs/mkdebugfs.sh               | 74 +++++++++++-----------
 1 file changed, 37 insertions(+), 37 deletions(-)

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
index 5dfa380..41a13cb 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
@@ -6,44 +6,44 @@ DEBUGFS="debugfs/debugfs"
 yes | mkfs.ext4 test.img
 
 {
-	CWD="/"
-	find $SRCDIR | while read FILE; do
-		#TGT=${FILE#$SRCDIR}
-		TGT=$(basename "${FILE#$SRCDIR}")
-		DIR=$(dirname "${FILE#$SRCDIR}")
+    CWD="/"
+    find $SRCDIR | while read FILE; do
+        #TGT=${FILE#$SRCDIR}
+        TGT=$(basename "${FILE#$SRCDIR}")
+        DIR=$(dirname "${FILE#$SRCDIR}")
 
-		# Skip the root dir
-		if [ -z "$TGT" ]; then
-			continue
-		fi
+        # Skip the root dir
+        if [ -z "$TGT" ]; then
+            continue
+        fi
 
-		if [ "$DIR" != "$CWD" ]; then
-			echo "cd $DIR"
-			CWD="$DIR"
-		fi
+        if [ "$DIR" != "$CWD" ]; then
+            echo "cd $DIR"
+            CWD="$DIR"
+        fi
 
-		case $(stat -c "%F" $FILE) in
-		"directory")
-			echo "mkdir $TGT"
-			;;
-		"regular file")
-			echo "write $FILE $TGT"
-			;;
-		"symbolic link")
-			LINK_TGT=$(ls -l $FILE | sed -e 's/.*-> //')
-			echo "symlink $TGT $LINK_TGT"
-			;;
-		"block special file")
-			DEVNO=$(stat -c "%t %T" $FILE)
-			echo "mknod $TGT b $DEVNO"
-			;;
-		"character special file")
-			DEVNO=$(stat -c "%t %T" $FILE)
-			echo "mknod $TGT c $DEVNO"
-			;;
-		*)
-			echo "Unknown file $FILE" 1>&2
-			;;
-		esac
-	done
+        case $(stat -c "%F" $FILE) in
+        "directory")
+            echo "mkdir $TGT"
+            ;;
+        "regular file")
+            echo "write $FILE $TGT"
+            ;;
+        "symbolic link")
+            LINK_TGT=$(ls -l $FILE | sed -e 's/.*-> //')
+            echo "symlink $TGT $LINK_TGT"
+            ;;
+        "block special file")
+            DEVNO=$(stat -c "%t %T" $FILE)
+            echo "mknod $TGT b $DEVNO"
+            ;;
+        "character special file")
+            DEVNO=$(stat -c "%t %T" $FILE)
+            echo "mknod $TGT c $DEVNO"
+            ;;
+        *)
+            echo "Unknown file $FILE" 1>&2
+            ;;
+        esac
+    done
 } | $DEBUGFS -w -f /dev/stdin $DEVICE
-- 
1.7.11.2

[PATCH 5/7] mkdebugfs.sh: several fixes

[Robert Yang]

Fix mkdebugfs.sh:

* Add a simple usage
* Add checking for the number of the parameters
* Add the "regular empty file" and "fifo" file type
* Set mode, uid and gid for the file
* Save the command lines to a file and batch run them

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../e2fsprogs/e2fsprogs/mkdebugfs.sh               | 43 +++++++++++++++++++---
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
index 41a13cb..88abc35 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
@@ -1,10 +1,25 @@
 #!/bin/sh
+
+do_usage () {
+    cat << _EOF
+Usage: mkdebugfs.sh <source> <device>
+Create an ext2/ext3/ext4 filesystem from a directory or file
+
+  source: The source directory or file
+  device: The target device
+
+_EOF
+    exit 1
+}
+
+[ $# -ne 2 ] && do_usage
+
 SRCDIR=$1
 DEVICE=$2
-DEBUGFS="debugfs/debugfs"
-
-yes | mkfs.ext4 test.img
+DEBUGFS="debugfs"
+CMD_FILE=`mktemp`
 
+# Save the debugfs command lines to a file, then run them
 {
     CWD="/"
     find $SRCDIR | while read FILE; do
@@ -26,7 +41,7 @@ yes | mkfs.ext4 test.img
         "directory")
             echo "mkdir $TGT"
             ;;
-        "regular file")
+        "regular file" | "regular empty file")
             echo "write $FILE $TGT"
             ;;
         "symbolic link")
@@ -41,9 +56,27 @@ yes | mkfs.ext4 test.img
             DEVNO=$(stat -c "%t %T" $FILE)
             echo "mknod $TGT c $DEVNO"
             ;;
+        "fifo")
+            echo "mknod $TGT p"
+            ;;
         *)
             echo "Unknown file $FILE" 1>&2
             ;;
         esac
+
+        # Set the file mode
+        MY_MODE=$(stat -c "%f" $FILE)
+        echo "sif $TGT mode 0x$MY_MODE"
+
+        # Set uid and gid
+        MY_UID=$(stat -c "%u" $FILE)
+        echo "sif $TGT uid $MY_UID"
+        MY_GID=$(stat -c "%g" $FILE)
+        echo "sif $TGT gid $MY_GID"
     done
-} | $DEBUGFS -w -f /dev/stdin $DEVICE
+} > $CMD_FILE
+
+# Run the command lines
+$DEBUGFS -w -f $CMD_FILE $DEVICE
+
+rm -f $CMD_FILE
-- 
1.7.11.2

[PATCH 6/7] e2fsprogs: ship mkfsdebug.sh

[Robert Yang]

Ship mkfsdebug.sh, it will be installed to /usr/bin/mkfsdebug.sh and
used for replacing the genext2fs in image_types.bbclass.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
index 70255ce..1609d42 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
@@ -4,6 +4,7 @@ PR = "r0"
 
 SRC_URI += "file://acinclude.m4 \
             file://remove.ldconfig.call.patch \
+            file://mkdebugfs.sh \
 "
 
 SRCREV = "fca8b1b2416c384b91b4fe00d6d37cfccb8fb3d5"
@@ -41,6 +42,7 @@ do_install_append () {
 		mv ${D}${base_libdir}/e2initrd_helper ${D}${libdir}
 		mv ${D}${base_libdir}/pkgconfig ${D}${libdir}
 	fi
+	install -m 0755 ${WORKDIR}/mkdebugfs.sh ${D}${bindir}
 }
 
 RDEPENDS_e2fsprogs = "e2fsprogs-badblocks"
-- 
1.7.11.2

[PATCH 7/7] image_types.bbclass: replace genext2fs with mkdebugfs.sh

[Robert Yang]

The ext3 and ext4 rootfs generation relied on genext2fs and flipping
some bits to "convert" the ext2 filesystem to ext3 and ext4 in the past,
now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
to copy the files to the image.

[YOCTO #3848]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/image_types.bbclass | 45 +++++++++++++++-------------------------
 1 file changed, 17 insertions(+), 28 deletions(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 6bb113d..92f85ed 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -135,34 +135,23 @@ IMAGE_CMD_sum.jffs2 = "${IMAGE_CMD_jffs2} && sumtool -i ${DEPLOY_DIR_IMAGE}/${IM
 
 IMAGE_CMD_cramfs = "mkcramfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cramfs ${EXTRA_IMAGECMD}"
 
-IMAGE_CMD_ext2 () {
-	rm -rf ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN} && mkdir ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}/${IMAGE_NAME}.rootfs.ext2
-	mv ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}/${IMAGE_NAME}.rootfs.ext2 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext2
-	rmdir ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}
-}
+oe_mkext234fs () {
+	fstype=$1
+	extra_imagecmd=""
 
-IMAGE_CMD_ext3 () {
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3
-	tune2fs -j ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3
-}
+	if [ $# -gt 1 ]; then
+		shift
+		extra_imagecmd=$@
+	fi
 
-oe_mkext4fs () {
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} $1
-	tune2fs -O extents,uninit_bg,dir_index,has_journal $1
-	e2fsck -yfDC0 $1 || chk=$?
-	case $chk in
-	0|1|2)
-	    ;;
-	*)
-	    return $chk
-	    ;;
-	esac
+	dd if=/dev/zero of=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype count=$ROOTFS_SIZE bs=1k
+	yes | mkfs.$fstype $extra_imagecmd ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype
+	mkdebugfs.sh ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype
 }
 
-IMAGE_CMD_ext4 () {
-	oe_mkext4fs ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext4
-}
+IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
+IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
+IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
 
 IMAGE_CMD_btrfs () {
 	mkfs.btrfs -b `expr ${ROOTFS_SIZE} \* 1024` ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.btrfs
@@ -212,7 +201,7 @@ JFFS2_ENDIANNESS ?= "${@base_conditional('SITEINFO_ENDIANNESS', 'le', '--little-
 JFFS2_ERASEBLOCK ?= "0x40000"
 EXTRA_IMAGECMD_jffs2 ?= "--pad ${JFFS2_ENDIANNESS} --eraseblock=${JFFS2_ERASEBLOCK} --no-cleanmarkers"
 
-# Change these if you want default genext2fs behavior (i.e. create minimal inode number)
+# Change these if you want default mkfs behavior (i.e. create minimal inode number)
 EXTRA_IMAGECMD_ext2 ?= "-i 8192"
 EXTRA_IMAGECMD_ext3 ?= "-i 8192"
 EXTRA_IMAGECMD_ext4 ?= "-i 8192"
@@ -223,9 +212,9 @@ IMAGE_DEPENDS = ""
 IMAGE_DEPENDS_jffs2 = "mtd-utils-native"
 IMAGE_DEPENDS_sum.jffs2 = "mtd-utils-native"
 IMAGE_DEPENDS_cramfs = "cramfs-native"
-IMAGE_DEPENDS_ext2 = "genext2fs-native"
-IMAGE_DEPENDS_ext3 = "genext2fs-native e2fsprogs-native"
-IMAGE_DEPENDS_ext4 = "genext2fs-native e2fsprogs-native"
+IMAGE_DEPENDS_ext2 = "e2fsprogs-native"
+IMAGE_DEPENDS_ext3 = "e2fsprogs-native"
+IMAGE_DEPENDS_ext4 = "e2fsprogs-native"
 IMAGE_DEPENDS_btrfs = "btrfs-tools-native"
 IMAGE_DEPENDS_squashfs = "squashfs-tools-native"
 IMAGE_DEPENDS_squashfs-lzma = "squashfs-tools-native"
-- 
1.7.11.2

Re: [PATCH 5/7] mkdebugfs.sh: several fixes

[Trevor Woerner]

On Tue, Feb 26, 2013 at 4:24 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
> Fix mkdebugfs.sh:
>
>          *)
>              echo "Unknown file $FILE" 1>&2
>              ;;
>          esac

I know this doesn't come from your work, but since you're making
changes, I think it would make for a better error message if the above
said something along the lines of:

    echo "Unknown/unhandled file type '$(stat -c "%F" $FILE)' file:$FILE"

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Trevor Woerner]

On Tue, Feb 26, 2013 at 4:24 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
> * Summary:
>   [...]now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
>   to copy the files to the image.

For a while this had me confused because I somehow thought you were
proposing using the Linux kernel's debugfs to create filesystems (i.e.
mount -t debugfs none /sys/kernel/debug). Maybe I'm still confused,
but if not I think, perhaps, it would be nice to come up with a better
name for this script since the name clash will probably trip others up
too.

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Saul Wold]

On 02/26/2013 01:24 AM, Robert Yang wrote:
> ===Contents===
>    * Summary
>    * Brief design
>    * Test info
>    * The rootfs generation time changes
>    * The rootfs size changes
>
> * Summary:
>    The ext3 and ext4 rootfs generation relied on genext2fs and flipping
>    some bits to "convert" the ext2 filesystem to ext3 and ext4 in the past,
>    now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
>    to copy the files to the image.
>
> * Brief design:
>    - Upgrade the e2fsprogs to the update to date version (the git
>      repo) which contains the "symlink" command in debugfs shell that we
>      need.
This should really be an RFC

This work was intended to be a step toward incorporating it into 
genext2fs directly.  At this point we do not want to switch to the GIT 
version of e2fsprogs, there is an update to e2fsprogs pending on the 
patch queue that addes the symlink support (crossed with your request)

>    - Replace the current genext2fs command with mkdebugfs.sh in
>      image_types.bbclass.

The goal is to change genext2fs to support a rootfs option an not 
replace it with mkdebugfs.

>    - The new steps to generate the root filesystem are:
>      -> use "dd" command to make rootfs.ext* file
Why the dd?

>      -> run "mkfs.ext* rootfs.ext*" to create the filesystem
>      -> run the mkdebugfs.sh to generate the root filesystem
>



> * Test info: (MACHINE = qemuarm, IMAGE_FSTYPES = "ext2 ext3 ext4 tar.bz2")
>    $ bitbake core-image-minimal core-image-sato meta-toolchain-sdk core-image-sato-sdk
>    $ runqemu runqemu qemuarm core-image-sato ext3
>    $ runqemu runqemu qemuarm core-image-minimal ext3
>
>    All of them are OK

What about booting an ext4 image, part of the goal of this was to enable 
ext4.

>
> * The rootfs generation time changes
>    - For a core-image-minimal generation (IMAGE_FSTYPES="tar.bz2 ext3"):
>
>      Before:
>      $ time bitbake core-image-minimal
>
>      real    1m10.823s
>      user    0m37.108s
>      sys     0m15.894s
>
>      After:
>      $ time bitbake core-image-minimal
>
>      real    1m17.501s
>      user    0m29.304s
>      sys     0m20.731s
>
>      # 7 seconds lost.
>
>    - For a core-image-sato generation (IMAGE_FSTYPES="tar.bz2 ext3"):
>
>      Before:
>      $ time bitbake core-image-sato
>
>      real    11m10.645s
>      user    2m43.503s
>      sys     1m1.589s
>
>      After:
>      $ time bitbake core-image-sato
>
>      real    11m53.131s
>      user    3m18.988s
>      sys     2m8.350s
>
>      # 43 seconds lost.
>
> * The rootfs size changes
>    - The image size are the same by "ls -h", but different by "du -sh":
>      (core-image-sato)
>
>      $ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}'
>      357M BEFORE.rootfs.ext3
>      357M AFTER.rootfs.ext3
>
>      $ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
>      238M    BEFORE.rootfs.ext3
>      357M    AFTER.rootfs.ext3
>
>      It seems that the genext2fs has optimized the rootfs generation.
>
> // Robert
>
> The following changes since commit 2cc4fe4a0874c42421b1bf3fa100160a9e60a9da:
>
>    upstream_tracking.inc: Coonectivity and multimedia packages updates (2013-02-25 05:58:20 -0800)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib robert/e2fsprogs
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/e2fsprogs
>
> Robert Yang (7):
>    e2fsprogs: upgrade to the git version (rename only)
>    e2fsprogs: upgrade to the git version
>    e2fsprogs: add the original mkdebugfs.sh
>    mkdebugfs.sh: convert the tab to 4 spaces
>    mkdebugfs.sh: several fixes
>    e2fsprogs: ship mkfsdebug.sh
>    image_types.bbclass: replace genext2fs with mkdebugfs.sh
>
>   meta/classes/image_types.bbclass                   | 45 +++++-------
>   meta/recipes-devtools/e2fsprogs/e2fsprogs.inc      |  4 +-
>   .../{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4   |  0
>   .../fallocate.patch                                |  0
>   .../e2fsprogs/e2fsprogs/mkdebugfs.sh               | 82 ++++++++++++++++++++++
>   .../{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch    |  0
>   .../remove.ldconfig.call.patch                     |  0
>   .../{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb}      |  6 +-
>   8 files changed, 106 insertions(+), 31 deletions(-)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4 (100%)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/fallocate.patch (100%)
>   create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch (100%)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/remove.ldconfig.call.patch (100%)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb} (92%)
>

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Phil Blundell]

On Tue, 2013-02-26 at 17:24 +0800, Robert Yang wrote:
>     Before:
>     $ time bitbake core-image-minimal
>     
>     real    1m10.823s
>     user    0m37.108s
>     sys     0m15.894s
>     
>     After:
>     $ time bitbake core-image-minimal
>     
>     real    1m17.501s
>     user    0m29.304s
>     sys     0m20.731s
>     
>     # 7 seconds lost.
>   
>   - For a core-image-sato generation (IMAGE_FSTYPES="tar.bz2 ext3"):
>   
>     Before:
>     $ time bitbake core-image-sato
>     
>     real    11m10.645s
>     user    2m43.503s
>     sys     1m1.589s
>     
>     After:
>     $ time bitbake core-image-sato
>     
>     real    11m53.131s
>     user    3m18.988s
>     sys     2m8.350s
>     
>     # 43 seconds lost.
>   
> * The rootfs size changes
>   - The image size are the same by "ls -h", but different by "du -sh":
>     (core-image-sato)
>   
>     $ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}'
>     357M BEFORE.rootfs.ext3
>     357M AFTER.rootfs.ext3
>     
>     $ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
>     238M    BEFORE.rootfs.ext3
>     357M    AFTER.rootfs.ext3

So, am I understanding correctly that your patch causes the rootfs
generation to take longer, and the resulting files to be bigger?  It
doesn't seem totally obvious to me that this is a good thing, and your
cover letter doesn't provide any other explanation for why the changes
are desirable.  Can you expand on the rationale for doing this?

Also, when sending patches for review, there is no point in sending a
series which adds a new file and then makes changes to it as separate
patches (in particular when the changes are just whitespace).  Please
squash those into a single patch before posting them on the list.

thanks

p.

Re: [PATCH 6/7] e2fsprogs: ship mkfsdebug.sh

[Darren Hart]

This can be merged with 3/7 right?

On 02/26/2013 01:24 AM, Robert Yang wrote:
> Ship mkfsdebug.sh, it will be installed to /usr/bin/mkfsdebug.sh and
> used for replacing the genext2fs in image_types.bbclass.
> 
> [YOCTO #3848]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
> index 70255ce..1609d42 100644
> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_git.bb
> @@ -4,6 +4,7 @@ PR = "r0"
>  
>  SRC_URI += "file://acinclude.m4 \
>              file://remove.ldconfig.call.patch \
> +            file://mkdebugfs.sh \
>  "
>  
>  SRCREV = "fca8b1b2416c384b91b4fe00d6d37cfccb8fb3d5"
> @@ -41,6 +42,7 @@ do_install_append () {
>  		mv ${D}${base_libdir}/e2initrd_helper ${D}${libdir}
>  		mv ${D}${base_libdir}/pkgconfig ${D}${libdir}
>  	fi
> +	install -m 0755 ${WORKDIR}/mkdebugfs.sh ${D}${bindir}
>  }
>  
>  RDEPENDS_e2fsprogs = "e2fsprogs-badblocks"
> 

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel

Re: [PATCH 5/7] mkdebugfs.sh: several fixes

[Darren Hart]

On 02/26/2013 09:43 AM, Trevor Woerner wrote:
> On Tue, Feb 26, 2013 at 4:24 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>> Fix mkdebugfs.sh:
>>
>>          *)
>>              echo "Unknown file $FILE" 1>&2
>>              ;;
>>          esac
> 
> I know this doesn't come from your work, but since you're making
> changes, I think it would make for a better error message if the above
> said something along the lines of:
> 
>     echo "Unknown/unhandled file type '$(stat -c "%F" $FILE)' file:$FILE"
> 

That's fine. Also, there is no need to send my script first and then
patch it. It was a 5 minute hack, so just modify and include as a single
patch.


-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Darren Hart]

On 02/26/2013 09:53 AM, Trevor Woerner wrote:
> On Tue, Feb 26, 2013 at 4:24 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>> * Summary:
>>   [...]now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
>>   to copy the files to the image.
> 
> For a while this had me confused because I somehow thought you were
> proposing using the Linux kernel's debugfs to create filesystems (i.e.
> mount -t debugfs none /sys/kernel/debug). Maybe I'm still confused,
> but if not I think, perhaps, it would be nice to come up with a better
> name for this script since the name clash will probably trip others up
> too.

It is confusing, unfortunately, debugfs is provided by e2fsprogs, not
something of our making. If you would prefer another name for
mkdebugfs.sh, that's fine. What would you propose?

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Darren Hart]

On 02/26/2013 09:55 AM, Saul Wold wrote:
> On 02/26/2013 01:24 AM, Robert Yang wrote:
>> ===Contents===
>>    * Summary
>>    * Brief design
>>    * Test info
>>    * The rootfs generation time changes
>>    * The rootfs size changes
>>
>> * Summary:
>>    The ext3 and ext4 rootfs generation relied on genext2fs and flipping
>>    some bits to "convert" the ext2 filesystem to ext3 and ext4 in the past,
>>    now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
>>    to copy the files to the image.
>>
>> * Brief design:
>>    - Upgrade the e2fsprogs to the update to date version (the git
>>      repo) which contains the "symlink" command in debugfs shell that we
>>      need.
> This should really be an RFC
> 
> This work was intended to be a step toward incorporating it into 
> genext2fs directly.


I think that was a thinko, but I'll correct for everyone else's benefit.
This will eventually be incorporated into libext2fs and mke2fs directly
and genext2fs will be retired as obsolete.


> At this point we do not want to switch to the GIT
> version of e2fsprogs, there is an update to e2fsprogs pending on the 
> patch queue that addes the symlink support (crossed with your request)
> 
>>    - Replace the current genext2fs command with mkdebugfs.sh in
>>      image_types.bbclass.
> 
> The goal is to change genext2fs to support a rootfs option an not 
> replace it with mkdebugfs.


Again, mke2fs. We can go about this in two ways. One is to just
prototype this in a branch and use it to validate the functionality and
not make any changes to oe-core image generation until mke2fs has
initial directory support. The other is to merge this and get broader
testing of the concept and later move to the full mke2fs implementation
once it becomes available. I understand the resistance to the latter,
but long term I think it will result in a more robust solution as we
will have caught more of the corner cases and have been able to do a
better job integrating into mke2fs the first time.


> 
>>    - The new steps to generate the root filesystem are:
>>      -> use "dd" command to make rootfs.ext* file
> Why the dd?


mke2fs requires an image file to format. We could create it in other
ways, but this seems reasonable. We could probably reduce the time to
create it by using a larger block size. Say 1M?


> 
>>      -> run "mkfs.ext* rootfs.ext*" to create the filesystem
>>      -> run the mkdebugfs.sh to generate the root filesystem
>>
> 
> 
> 
>> * Test info: (MACHINE = qemuarm, IMAGE_FSTYPES = "ext2 ext3 ext4 tar.bz2")
>>    $ bitbake core-image-minimal core-image-sato meta-toolchain-sdk core-image-sato-sdk
>>    $ runqemu runqemu qemuarm core-image-sato ext3
>>    $ runqemu runqemu qemuarm core-image-minimal ext3
>>
>>    All of them are OK
> 
> What about booting an ext4 image, part of the goal of this was to enable 
> ext4.


Seconded.



-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Darren Hart]

On 02/26/2013 10:15 AM, Phil Blundell wrote:
> On Tue, 2013-02-26 at 17:24 +0800, Robert Yang wrote:
>>     Before:
>>     $ time bitbake core-image-minimal
>>     
>>     real    1m10.823s
>>     user    0m37.108s
>>     sys     0m15.894s
>>     
>>     After:
>>     $ time bitbake core-image-minimal
>>     
>>     real    1m17.501s
>>     user    0m29.304s
>>     sys     0m20.731s
>>     
>>     # 7 seconds lost.
>>   
>>   - For a core-image-sato generation (IMAGE_FSTYPES="tar.bz2 ext3"):
>>   
>>     Before:
>>     $ time bitbake core-image-sato
>>     
>>     real    11m10.645s
>>     user    2m43.503s
>>     sys     1m1.589s
>>     
>>     After:
>>     $ time bitbake core-image-sato
>>     
>>     real    11m53.131s
>>     user    3m18.988s
>>     sys     2m8.350s
>>     
>>     # 43 seconds lost.
>>   
>> * The rootfs size changes
>>   - The image size are the same by "ls -h", but different by "du -sh":
>>     (core-image-sato)
>>   
>>     $ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}'
>>     357M BEFORE.rootfs.ext3
>>     357M AFTER.rootfs.ext3
>>     
>>     $ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
>>     238M    BEFORE.rootfs.ext3
>>     357M    AFTER.rootfs.ext3
> 
> So, am I understanding correctly that your patch causes the rootfs
> generation to take longer, and the resulting files to be bigger?  It
> doesn't seem totally obvious to me that this is a good thing, and your
> cover letter doesn't provide any other explanation for why the changes
> are desirable.  Can you expand on the rationale for doing this?


A better motivation description is indeed required.

Basically, genext2fs doesn't support creating ext4 filesystems. It
creates, as I understand it, an ext2 filesystem, then adds a journal,
and sets some bits. It can't support the newer features like extents. So
what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated
solution for generating prepopulated filesystem images as many other
mkfs* tools do. One thing missing was symlink support in libext2fs. I
added that support and demonstrated a script which uses the e2fsprogs
debugfs tool that can populate the newly formatted filesystem from a
directory and without root privileges.

Robert's patches integrate this stage of development into OE-Core. As I
replied to Saul's 0/7 response:

"We can go about this in two ways. One is to just
prototype this in a branch and use it to validate the functionality and
not make any changes to oe-core image generation until mke2fs has
initial directory support. The other is to merge this and get broader
testing of the concept and later move to the full mke2fs implementation
once it becomes available. I understand the resistance to the latter,
but long term I think it will result in a more robust solution as we
will have caught more of the corner cases and have been able to do a
better job integrating into mke2fs the first time."


> Also, when sending patches for review, there is no point in sending a
> series which adds a new file and then makes changes to it as separate
> patches (in particular when the changes are just whitespace).  Please
> squash those into a single patch before posting them on the list.


I think Robert was trying to maintain some history by keeping my
original script intact. I agree, it isn't necessary and those three
patches can be merged into one.

Thank you for reviewing!

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Trevor Woerner]

On Tue, Feb 26, 2013 at 4:52 PM, Darren Hart <dvhart@linux.intel.com> wrote:
> On 02/26/2013 09:53 AM, Trevor Woerner wrote:
>> On Tue, Feb 26, 2013 at 4:24 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>>> * Summary:
>>>   [...]now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
>>>   to copy the files to the image.
>>
>> For a while this had me confused because I somehow thought you were
>> proposing using the Linux kernel's debugfs to create filesystems (i.e.
>> mount -t debugfs none /sys/kernel/debug).
>
> It is confusing, unfortunately, debugfs is provided by e2fsprogs, not
> something of our making. If you would prefer another name for
> mkdebugfs.sh, that's fine. What would you propose?

mkextXfs.sh ?

Re: [PATCH 4/7] mkdebugfs.sh: convert the tab to 4 spaces

[Darren Hart]

And NACK on this one :-)

Shell scripts should use tabs.

--
Darren

On 02/26/2013 01:24 AM, Robert Yang wrote:
> [YOCTO #3848]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  .../e2fsprogs/e2fsprogs/mkdebugfs.sh               | 74 +++++++++++-----------
>  1 file changed, 37 insertions(+), 37 deletions(-)
> 
> diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
> index 5dfa380..41a13cb 100644
> --- a/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
> +++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
> @@ -6,44 +6,44 @@ DEBUGFS="debugfs/debugfs"
>  yes | mkfs.ext4 test.img
>  
>  {
> -	CWD="/"
> -	find $SRCDIR | while read FILE; do
> -		#TGT=${FILE#$SRCDIR}
> -		TGT=$(basename "${FILE#$SRCDIR}")
> -		DIR=$(dirname "${FILE#$SRCDIR}")
> +    CWD="/"
> +    find $SRCDIR | while read FILE; do
> +        #TGT=${FILE#$SRCDIR}
> +        TGT=$(basename "${FILE#$SRCDIR}")
> +        DIR=$(dirname "${FILE#$SRCDIR}")
>  
> -		# Skip the root dir
> -		if [ -z "$TGT" ]; then
> -			continue
> -		fi
> +        # Skip the root dir
> +        if [ -z "$TGT" ]; then
> +            continue
> +        fi
>  
> -		if [ "$DIR" != "$CWD" ]; then
> -			echo "cd $DIR"
> -			CWD="$DIR"
> -		fi
> +        if [ "$DIR" != "$CWD" ]; then
> +            echo "cd $DIR"
> +            CWD="$DIR"
> +        fi
>  
> -		case $(stat -c "%F" $FILE) in
> -		"directory")
> -			echo "mkdir $TGT"
> -			;;
> -		"regular file")
> -			echo "write $FILE $TGT"
> -			;;
> -		"symbolic link")
> -			LINK_TGT=$(ls -l $FILE | sed -e 's/.*-> //')
> -			echo "symlink $TGT $LINK_TGT"
> -			;;
> -		"block special file")
> -			DEVNO=$(stat -c "%t %T" $FILE)
> -			echo "mknod $TGT b $DEVNO"
> -			;;
> -		"character special file")
> -			DEVNO=$(stat -c "%t %T" $FILE)
> -			echo "mknod $TGT c $DEVNO"
> -			;;
> -		*)
> -			echo "Unknown file $FILE" 1>&2
> -			;;
> -		esac
> -	done
> +        case $(stat -c "%F" $FILE) in
> +        "directory")
> +            echo "mkdir $TGT"
> +            ;;
> +        "regular file")
> +            echo "write $FILE $TGT"
> +            ;;
> +        "symbolic link")
> +            LINK_TGT=$(ls -l $FILE | sed -e 's/.*-> //')
> +            echo "symlink $TGT $LINK_TGT"
> +            ;;
> +        "block special file")
> +            DEVNO=$(stat -c "%t %T" $FILE)
> +            echo "mknod $TGT b $DEVNO"
> +            ;;
> +        "character special file")
> +            DEVNO=$(stat -c "%t %T" $FILE)
> +            echo "mknod $TGT c $DEVNO"
> +            ;;
> +        *)
> +            echo "Unknown file $FILE" 1>&2
> +            ;;
> +        esac
> +    done
>  } | $DEBUGFS -w -f /dev/stdin $DEVICE
> 

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Darren Hart]

On 02/26/2013 02:37 PM, Trevor Woerner wrote:
> On Tue, Feb 26, 2013 at 4:52 PM, Darren Hart <dvhart@linux.intel.com> wrote:
>> On 02/26/2013 09:53 AM, Trevor Woerner wrote:
>>> On Tue, Feb 26, 2013 at 4:24 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>>>> * Summary:
>>>>   [...]now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
>>>>   to copy the files to the image.
>>>
>>> For a while this had me confused because I somehow thought you were
>>> proposing using the Linux kernel's debugfs to create filesystems (i.e.
>>> mount -t debugfs none /sys/kernel/debug).
>>
>> It is confusing, unfortunately, debugfs is provided by e2fsprogs, not
>> something of our making. If you would prefer another name for
>> mkdebugfs.sh, that's fine. What would you propose?
> 
> mkextXfs.sh ?

Likely to be confused with mke2fs itself. Perhaps:

populate-extfs.sh ?


-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Trevor Woerner]

On Tue, Feb 26, 2013 at 6:00 PM, Darren Hart <dvhart@linux.intel.com> wrote:
>> mkextXfs.sh ?
>
> Likely to be confused with mke2fs itself. Perhaps:
>
> populate-extfs.sh ?

Sounds great.

Re: [PATCH 0/7] Create ext* filesystems using debugfs

[Robert Yang]

Hi all,

Thank you very much for all your reviewing, I will pull your comments in
and send a RFC sooner.

// Robert

On 02/26/2013 05:24 PM, Robert Yang wrote:
> ===Contents===
>    * Summary
>    * Brief design
>    * Test info
>    * The rootfs generation time changes
>    * The rootfs size changes
>
> * Summary:
>    The ext3 and ext4 rootfs generation relied on genext2fs and flipping
>    some bits to "convert" the ext2 filesystem to ext3 and ext4 in the past,
>    now we use the mkfs.ext3/ext4 to create the image, and use mkdebugfs.sh
>    to copy the files to the image.
>
> * Brief design:
>    - Upgrade the e2fsprogs to the update to date version (the git
>      repo) which contains the "symlink" command in debugfs shell that we
>      need.
>    - Replace the current genext2fs command with mkdebugfs.sh in
>      image_types.bbclass.
>    - The new steps to generate the root filesystem are:
>      -> use "dd" command to make rootfs.ext* file
>      -> run "mkfs.ext* rootfs.ext*" to create the filesystem
>      -> run the mkdebugfs.sh to generate the root filesystem
>
> * Test info: (MACHINE = qemuarm, IMAGE_FSTYPES = "ext2 ext3 ext4 tar.bz2")
>    $ bitbake core-image-minimal core-image-sato meta-toolchain-sdk core-image-sato-sdk
>    $ runqemu runqemu qemuarm core-image-sato ext3
>    $ runqemu runqemu qemuarm core-image-minimal ext3
>
>    All of them are OK
>
> * The rootfs generation time changes
>    - For a core-image-minimal generation (IMAGE_FSTYPES="tar.bz2 ext3"):
>
>      Before:
>      $ time bitbake core-image-minimal
>
>      real    1m10.823s
>      user    0m37.108s
>      sys     0m15.894s
>
>      After:
>      $ time bitbake core-image-minimal
>
>      real    1m17.501s
>      user    0m29.304s
>      sys     0m20.731s
>
>      # 7 seconds lost.
>
>    - For a core-image-sato generation (IMAGE_FSTYPES="tar.bz2 ext3"):
>
>      Before:
>      $ time bitbake core-image-sato
>
>      real    11m10.645s
>      user    2m43.503s
>      sys     1m1.589s
>
>      After:
>      $ time bitbake core-image-sato
>
>      real    11m53.131s
>      user    3m18.988s
>      sys     2m8.350s
>
>      # 43 seconds lost.
>
> * The rootfs size changes
>    - The image size are the same by "ls -h", but different by "du -sh":
>      (core-image-sato)
>
>      $ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}'
>      357M BEFORE.rootfs.ext3
>      357M AFTER.rootfs.ext3
>
>      $ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
>      238M    BEFORE.rootfs.ext3
>      357M    AFTER.rootfs.ext3
>
>      It seems that the genext2fs has optimized the rootfs generation.
>
> // Robert
>
> The following changes since commit 2cc4fe4a0874c42421b1bf3fa100160a9e60a9da:
>
>    upstream_tracking.inc: Coonectivity and multimedia packages updates (2013-02-25 05:58:20 -0800)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib robert/e2fsprogs
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/e2fsprogs
>
> Robert Yang (7):
>    e2fsprogs: upgrade to the git version (rename only)
>    e2fsprogs: upgrade to the git version
>    e2fsprogs: add the original mkdebugfs.sh
>    mkdebugfs.sh: convert the tab to 4 spaces
>    mkdebugfs.sh: several fixes
>    e2fsprogs: ship mkfsdebug.sh
>    image_types.bbclass: replace genext2fs with mkdebugfs.sh
>
>   meta/classes/image_types.bbclass                   | 45 +++++-------
>   meta/recipes-devtools/e2fsprogs/e2fsprogs.inc      |  4 +-
>   .../{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4   |  0
>   .../fallocate.patch                                |  0
>   .../e2fsprogs/e2fsprogs/mkdebugfs.sh               | 82 ++++++++++++++++++++++
>   .../{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch    |  0
>   .../remove.ldconfig.call.patch                     |  0
>   .../{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb}      |  6 +-
>   8 files changed, 106 insertions(+), 31 deletions(-)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/acinclude.m4 (100%)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/fallocate.patch (100%)
>   create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs/mkdebugfs.sh
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/mkdir.patch (100%)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs-1.42.6 => e2fsprogs}/remove.ldconfig.call.patch (100%)
>   rename meta/recipes-devtools/e2fsprogs/{e2fsprogs_1.42.6.bb => e2fsprogs_git.bb} (92%)
>