[U-Boot] [RESEND PATCH v5 0/7] Add support for symlink creation in EXT4

[U-Boot] [RESEND PATCH v5 0/7] Add support for symlink creation in EXT4

[Jean-Jacques Hiblot]

This series adds support for the creation of symbolic links on ext4
file-systems.
The motivation behind this work is to have the ability to "do" the job
of update-alternatives in u-boot.
Firmware on TI's platform are usually managed with update-alternatives and
are thus targeted by a symbolic link. In some situations we need the
ability to select an alternate firmware before the linux kernel is started
so that when a early driver needing the firmware comes up, it can be fed
the firmware of our choice.

Tested on a am57xx_evm, using a EXT4 partition on external SDcard.
The filesystem can be checked later with: fsck.ext4 -f <dev>

usage example:
=> ln mmc 0:2 zImage /boot/the_linux_kernel

Changes in v5:
- Added filesystem integrity checks
- Fix bug in ext4fs_delete_file(). The type must be read from the
  inode.

Changes in v4:
- replaced u64 and u32 with uint64_t and uint32_t

Changes in v3:
- reworded commit log
- removed FS integrity test (fsck) in python tests

Changes in v2:
- Prevent write access if metadata checksum is enabled
- Fix issue in ext4fs_delete_file() when target in not stored in an
  allocated block
- Added python tests for symlinks under sandbox

Jean-Jacques Hiblot (7):
  fs: ext4: do not allow writes if metadata checksum is active
  test: fs: disable the metadata checksums on ext4 filesystems
  test: fs: Add filesystem integrity checks
  fs: ext4: constify the buffer passed to write functions
  fs: ext4: Add support for the creation of symbolic links
  fs: Add a new command to create symbolic links
  test: fs: Added tests for symlinks

 cmd/fs.c                                |  14 +++
 fs/ext4/ext4_common.c                   |   4 +-
 fs/ext4/ext4_common.h                   |   2 +-
 fs/ext4/ext4_write.c                    |  72 ++++++++++---
 fs/fs.c                                 |  44 ++++++++
 include/ext4fs.h                        |   6 +-
 include/fs.h                            |   2 +
 test/py/tests/test_fs/conftest.py       |  77 ++++++++++++++
 test/py/tests/test_fs/fstest_defs.py    |   3 +
 test/py/tests/test_fs/fstest_helpers.py |  15 +++
 test/py/tests/test_fs/test_basic.py     |   4 +
 test/py/tests/test_fs/test_ext.py       |  10 ++
 test/py/tests/test_fs/test_mkdir.py     |   8 ++
 test/py/tests/test_fs/test_symlink.py   | 130 ++++++++++++++++++++++++
 test/py/tests/test_fs/test_unlink.py    |  14 ++-
 15 files changed, 382 insertions(+), 23 deletions(-)
 create mode 100644 test/py/tests/test_fs/fstest_helpers.py
 create mode 100644 test/py/tests/test_fs/test_symlink.py

-- 
2.17.1

[U-Boot] [RESEND PATCH v5 1/7] fs: ext4: do not allow writes if metadata checksum is active

[Jean-Jacques Hiblot]

u-boot does not supports updating the metadata chacksums

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

---

Changes in v5: None
Changes in v4: None
Changes in v3: None
Changes in v2:
- Prevent write access if metadata checksum is enabled

 fs/ext4/ext4_write.c | 12 ++++++++++--
 include/ext4fs.h     |  1 +
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/fs/ext4/ext4_write.c b/fs/ext4/ext4_write.c
index a7f543f7df..1de29236f0 100644
--- a/fs/ext4/ext4_write.c
+++ b/fs/ext4/ext4_write.c
@@ -858,12 +858,19 @@ int ext4fs_write(const char *fname, unsigned char *buffer,
 
 	g_parent_inode = zalloc(fs->inodesz);
 	if (!g_parent_inode)
-		goto fail;
+		goto fail_ext4fs_init;
 
 	if (ext4fs_init() != 0) {
 		printf("error in File System init\n");
-		return -1;
+		goto fail_ext4fs_init;
+	}
+
+	if (le32_to_cpu(fs->sb->feature_ro_compat) &
+		EXT4_FEATURE_RO_COMPAT_METADATA_CSUM) {
+		printf("u-boot doesn't support updating the metadata checksums yet\n");
+		goto fail;
 	}
+
 	inodes_per_block = fs->blksz / fs->inodesz;
 	parent_inodeno = ext4fs_get_parent_inode_num(fname, filename, F_FILE);
 	if (parent_inodeno == -1)
@@ -990,6 +997,7 @@ int ext4fs_write(const char *fname, unsigned char *buffer,
 	return 0;
 fail:
 	ext4fs_deinit();
+fail_ext4fs_init:
 	free(inode_buffer);
 	free(g_parent_inode);
 	free(temp_ptr);
diff --git a/include/ext4fs.h b/include/ext4fs.h
index bb55639107..bcf440364e 100644
--- a/include/ext4fs.h
+++ b/include/ext4fs.h
@@ -32,6 +32,7 @@
 #define EXT4_EXTENTS_FL		0x00080000 /* Inode uses extents */
 #define EXT4_EXT_MAGIC			0xf30a
 #define EXT4_FEATURE_RO_COMPAT_GDT_CSUM	0x0010
+#define EXT4_FEATURE_RO_COMPAT_METADATA_CSUM	0x0400
 #define EXT4_FEATURE_INCOMPAT_EXTENTS	0x0040
 #define EXT4_FEATURE_INCOMPAT_64BIT	0x0080
 #define EXT4_INDIRECT_BLOCKS		12
-- 
2.17.1

[U-Boot] [RESEND PATCH v5 2/7] test: fs: disable the metadata checksums on ext4 filesystems

[Jean-Jacques Hiblot]

If the metadata checksums are enabled, all write operations will fail.

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
---

Changes in v5: None
Changes in v4: None
Changes in v3: None
Changes in v2: None

 test/py/tests/test_fs/conftest.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 43eeb4be0b..745ed0ed38 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -143,6 +143,8 @@ def mk_fs(config, fs_type, size, id):
         mkfs_opt = '-F 16'
     elif fs_type == 'fat32':
         mkfs_opt = '-F 32'
+    elif fs_type == 'ext4':
+        mkfs_opt = '-O ^metadata_csum'
     else:
         mkfs_opt = ''
 
-- 
2.17.1

[U-Boot] [RESEND PATCH v5 3/7] test: fs: Add filesystem integrity checks

[Jean-Jacques Hiblot]

We need to make sure that file writes,file creation, etc. are properly
performed and do not corrupt the filesystem.
To help with this, introduce the assert_fs_integrity() function that
executes the appropriate fsck tool. It should be called at the end of any
test that modify the content/organization of the filesystem.
Currently only supports FATs and EXT4.

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>

---

Changes in v5:
- Added filesystem integrity checks

Changes in v4: None
Changes in v3: None
Changes in v2: None

 test/py/tests/test_fs/fstest_helpers.py | 15 +++++++++++++++
 test/py/tests/test_fs/test_basic.py     |  4 ++++
 test/py/tests/test_fs/test_ext.py       | 10 ++++++++++
 test/py/tests/test_fs/test_mkdir.py     |  8 ++++++++
 test/py/tests/test_fs/test_unlink.py    | 14 +++++++++++---
 5 files changed, 48 insertions(+), 3 deletions(-)
 create mode 100644 test/py/tests/test_fs/fstest_helpers.py

diff --git a/test/py/tests/test_fs/fstest_helpers.py b/test/py/tests/test_fs/fstest_helpers.py
new file mode 100644
index 0000000000..637bc30a91
--- /dev/null
+++ b/test/py/tests/test_fs/fstest_helpers.py
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier:      GPL-2.0+
+# Copyright (c) 2019, Texas Instrument
+# Author: JJ Hiblot <jjhiblot@ti.com>
+#
+
+from subprocess import check_call, CalledProcessError
+
+def assert_fs_integrity(fs_type, fs_img):
+    try:
+        if fs_type == 'ext4':
+            check_call('fsck.ext4 -n -f %s' % fs_img, shell=True)
+        if fs_type == 'fat':
+            check_call('fsck.fat -n %s' % fs_img, shell=True)
+    except CalledProcessError:
+        raise
diff --git a/test/py/tests/test_fs/test_basic.py b/test/py/tests/test_fs/test_basic.py
index 140ca29ac7..71f3e86fb1 100644
--- a/test/py/tests/test_fs/test_basic.py
+++ b/test/py/tests/test_fs/test_basic.py
@@ -11,6 +11,7 @@ This test verifies basic read/write operation on file system.
 import pytest
 import re
 from fstest_defs import *
+from fstest_helpers import assert_fs_integrity
 
 @pytest.mark.boardspec('sandbox')
 @pytest.mark.slow
@@ -237,6 +238,7 @@ class TestFsBasic(object):
                 'md5sum %x $filesize' % ADDR,
                 'setenv filesize'])
             assert(md5val[0] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs12(self, u_boot_console, fs_obj_basic):
         """
@@ -252,6 +254,7 @@ class TestFsBasic(object):
                 'host bind 0 %s' % fs_img,
                 '%swrite host 0:0 %x /. 0x10' % (fs_type, ADDR)])
             assert('Unable to write' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs13(self, u_boot_console, fs_obj_basic):
         """
@@ -286,3 +289,4 @@ class TestFsBasic(object):
                 'md5sum %x $filesize' % ADDR,
                 'setenv filesize'])
             assert(md5val[0] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
diff --git a/test/py/tests/test_fs/test_ext.py b/test/py/tests/test_fs/test_ext.py
index 06cad5516d..2c47738b8d 100644
--- a/test/py/tests/test_fs/test_ext.py
+++ b/test/py/tests/test_fs/test_ext.py
@@ -11,6 +11,7 @@ This test verifies extended write operation on file system.
 import pytest
 import re
 from fstest_defs import *
+from fstest_helpers import assert_fs_integrity
 
 @pytest.mark.boardspec('sandbox')
 @pytest.mark.slow
@@ -36,6 +37,7 @@ class TestFsExt(object):
                 'md5sum %x $filesize' % ADDR,
                 'setenv filesize'])
             assert(md5val[0] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext2(self, u_boot_console, fs_obj_ext):
         """
@@ -58,6 +60,7 @@ class TestFsExt(object):
                 'md5sum %x $filesize' % ADDR,
                 'setenv filesize'])
             assert(md5val[0] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext3(self, u_boot_console, fs_obj_ext):
         """
@@ -72,6 +75,7 @@ class TestFsExt(object):
                 '%swrite host 0:0 %x /dir1/none/%s.w3 $filesize'
                     % (fs_type, ADDR, MIN_FILE)])
             assert('Unable to write "/dir1/none/' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext4(self, u_boot_console, fs_obj_ext):
         """
@@ -104,6 +108,7 @@ class TestFsExt(object):
                 'md5sum %x $filesize' % ADDR,
                 'setenv filesize'])
             assert(md5val[1] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext5(self, u_boot_console, fs_obj_ext):
         """
@@ -136,6 +141,7 @@ class TestFsExt(object):
                 'md5sum %x $filesize' % ADDR,
                 'setenv filesize'])
             assert(md5val[2] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext6(self, u_boot_console, fs_obj_ext):
         """
@@ -160,6 +166,7 @@ class TestFsExt(object):
                 'printenv filesize',
                 'setenv filesize'])
             assert('filesize=0' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext7(self, u_boot_console, fs_obj_ext):
         """
@@ -192,6 +199,7 @@ class TestFsExt(object):
                 'md5sum %x $filesize' % ADDR,
                 'setenv filesize'])
             assert(md5val[3] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext8(self, u_boot_console, fs_obj_ext):
         """
@@ -209,6 +217,7 @@ class TestFsExt(object):
                 '%swrite host 0:0 %x /dir1/%s.w8 0x1400 %x'
                     % (fs_type, ADDR, MIN_FILE, 0x100000 + 0x1400))
             assert('Unable to write "/dir1' in output)
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_fs_ext9(self, u_boot_console, fs_obj_ext):
         """
@@ -223,3 +232,4 @@ class TestFsExt(object):
                 '%swrite host 0:0 %x /dir1/%s.w9 0x1400 0x1400'
                     % (fs_type, ADDR, MIN_FILE)])
             assert('Unable to write "/dir1' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
diff --git a/test/py/tests/test_fs/test_mkdir.py b/test/py/tests/test_fs/test_mkdir.py
index b3fe11cf3b..fa9561ec35 100644
--- a/test/py/tests/test_fs/test_mkdir.py
+++ b/test/py/tests/test_fs/test_mkdir.py
@@ -9,6 +9,7 @@ This test verifies mkdir operation on file system.
 """
 
 import pytest
+from fstest_helpers import assert_fs_integrity
 
 @pytest.mark.boardspec('sandbox')
 @pytest.mark.slow
@@ -29,6 +30,8 @@ class TestMkdir(object):
                 '%sls host 0:0 dir1' % fs_type)
             assert('./'   in output)
             assert('../'  in output)
+            assert_fs_integrity(fs_type, fs_img)
+
 
     def test_mkdir2(self, u_boot_console, fs_obj_mkdir):
         """
@@ -46,6 +49,7 @@ class TestMkdir(object):
                 '%sls host 0:0 dir1/dir2' % fs_type)
             assert('./'   in output)
             assert('../'  in output)
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_mkdir3(self, u_boot_console, fs_obj_mkdir):
         """
@@ -58,6 +62,7 @@ class TestMkdir(object):
                 'host bind 0 %s' % fs_img,
                 '%smkdir host 0:0 none/dir3' % fs_type])
             assert('Unable to create a directory' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_mkdir4(self, u_boot_console, fs_obj_mkdir):
         """
@@ -69,6 +74,7 @@ class TestMkdir(object):
                 'host bind 0 %s' % fs_img,
                 '%smkdir host 0:0 .' % fs_type])
             assert('Unable to create a directory' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_mkdir5(self, u_boot_console, fs_obj_mkdir):
         """
@@ -80,6 +86,7 @@ class TestMkdir(object):
                 'host bind 0 %s' % fs_img,
                 '%smkdir host 0:0 ..' % fs_type])
             assert('Unable to create a directory' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_mkdir6(self, u_boot_console, fs_obj_mkdir):
         """
@@ -111,3 +118,4 @@ class TestMkdir(object):
                 '%sls host 0:0 dir6/0123456789abcdef13/..' % fs_type)
             assert('0123456789abcdef00/'  in output)
             assert('0123456789abcdef13/'  in output)
+            assert_fs_integrity(fs_type, fs_img)
diff --git a/test/py/tests/test_fs/test_unlink.py b/test/py/tests/test_fs/test_unlink.py
index 2b817468ed..97aafc63bb 100644
--- a/test/py/tests/test_fs/test_unlink.py
+++ b/test/py/tests/test_fs/test_unlink.py
@@ -10,6 +10,7 @@ on file system.
 """
 
 import pytest
+from fstest_helpers import assert_fs_integrity
 
 @pytest.mark.boardspec('sandbox')
 @pytest.mark.slow
@@ -30,6 +31,7 @@ class TestUnlink(object):
                 '%sls host 0:0 dir1/' % fs_type)
             assert(not 'file1' in output)
             assert('file2' in output)
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_unlink2(self, u_boot_console, fs_obj_unlink):
         """
@@ -48,6 +50,7 @@ class TestUnlink(object):
             output = u_boot_console.run_command(
                 '%sls host 0:0 dir2' % fs_type)
             assert('0 file(s), 2 dir(s)' in output)
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_unlink3(self, u_boot_console, fs_obj_unlink):
         """
@@ -59,6 +62,7 @@ class TestUnlink(object):
                 'host bind 0 %s' % fs_img,
                 '%srm host 0:0 dir1/nofile' % fs_type])
             assert('nofile: doesn\'t exist' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_unlink4(self, u_boot_console, fs_obj_unlink):
         """
@@ -71,9 +75,10 @@ class TestUnlink(object):
                 '%srm host 0:0 dir4' % fs_type])
             assert('' == ''.join(output))
 
-        output = u_boot_console.run_command(
-            '%sls host 0:0 /' % fs_type)
-        assert(not 'dir4' in output)
+            output = u_boot_console.run_command(
+                '%sls host 0:0 /' % fs_type)
+            assert(not 'dir4' in output)
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_unlink5(self, u_boot_console, fs_obj_unlink):
         """
@@ -86,6 +91,7 @@ class TestUnlink(object):
                 'host bind 0 %s' % fs_img,
                 '%srm host 0:0 dir5' % fs_type])
             assert('directory is not empty' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_unlink6(self, u_boot_console, fs_obj_unlink):
         """
@@ -97,6 +103,7 @@ class TestUnlink(object):
                 'host bind 0 %s' % fs_img,
                 '%srm host 0:0 dir5/.' % fs_type])
             assert('directory is not empty' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
 
     def test_unlink7(self, u_boot_console, fs_obj_unlink):
         """
@@ -108,3 +115,4 @@ class TestUnlink(object):
                 'host bind 0 %s' % fs_img,
                 '%srm host 0:0 dir5/..' % fs_type])
             assert('directory is not empty' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
-- 
2.17.1

[U-Boot] [RESEND PATCH v5 4/7] fs: ext4: constify the buffer passed to write functions

[Jean-Jacques Hiblot]

There is no need to modify the buffer passed to ext4fs_write_file().
The memset() call is not required here and was likely copied from the
equivalent part of the ext4fs_read_file() function where we do need it.

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

---

Changes in v5: None
Changes in v4:
- replaced u64 and u32 with uint64_t and uint32_t

Changes in v3:
- reworded commit log

Changes in v2: None

 fs/ext4/ext4_common.c |  2 +-
 fs/ext4/ext4_common.h |  2 +-
 fs/ext4/ext4_write.c  | 11 +++++------
 include/ext4fs.h      |  2 +-
 4 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/fs/ext4/ext4_common.c b/fs/ext4/ext4_common.c
index 67e2471bd3..1ff2cd28a6 100644
--- a/fs/ext4/ext4_common.c
+++ b/fs/ext4/ext4_common.c
@@ -190,7 +190,7 @@ uint32_t ext4fs_div_roundup(uint32_t size, uint32_t n)
 	return res;
 }
 
-void put_ext4(uint64_t off, void *buf, uint32_t size)
+void put_ext4(uint64_t off, const void *buf, uint32_t size)
 {
 	uint64_t startblock;
 	uint64_t remainder;
diff --git a/fs/ext4/ext4_common.h b/fs/ext4/ext4_common.h
index 1ee81ab7ce..4dff1914d9 100644
--- a/fs/ext4/ext4_common.h
+++ b/fs/ext4/ext4_common.h
@@ -72,7 +72,7 @@ int ext4fs_iget(int inode_no, struct ext2_inode *inode);
 void ext4fs_allocate_blocks(struct ext2_inode *file_inode,
 				unsigned int total_remaining_blocks,
 				unsigned int *total_no_of_block);
-void put_ext4(uint64_t off, void *buf, uint32_t size);
+void put_ext4(uint64_t off, const void *buf, uint32_t size);
 struct ext2_block_group *ext4fs_get_group_descriptor
 	(const struct ext_filesystem *fs, uint32_t bg_idx);
 uint64_t ext4fs_bg_get_block_id(const struct ext2_block_group *bg,
diff --git a/fs/ext4/ext4_write.c b/fs/ext4/ext4_write.c
index 1de29236f0..51ca1c0916 100644
--- a/fs/ext4/ext4_write.c
+++ b/fs/ext4/ext4_write.c
@@ -752,7 +752,7 @@ void ext4fs_deinit(void)
  * contigous sectors as ext4fs_read_file
  */
 static int ext4fs_write_file(struct ext2_inode *file_inode,
-			     int pos, unsigned int len, char *buf)
+			     int pos, unsigned int len, const char *buf)
 {
 	int i;
 	int blockcnt;
@@ -764,7 +764,7 @@ static int ext4fs_write_file(struct ext2_inode *file_inode,
 	int delayed_start = 0;
 	int delayed_extent = 0;
 	int delayed_next = 0;
-	char *delayed_buf = NULL;
+	const char *delayed_buf = NULL;
 
 	/* Adjust len so it we can't read past the end of the file. */
 	if (len > filesize)
@@ -816,7 +816,6 @@ static int ext4fs_write_file(struct ext2_inode *file_inode,
 					 (uint32_t) delayed_extent);
 				previous_block_number = -1;
 			}
-			memset(buf, 0, fs->blksz - skipfirst);
 		}
 		buf += fs->blksz - skipfirst;
 	}
@@ -830,8 +829,8 @@ static int ext4fs_write_file(struct ext2_inode *file_inode,
 	return len;
 }
 
-int ext4fs_write(const char *fname, unsigned char *buffer,
-					unsigned long sizebytes)
+int ext4fs_write(const char *fname, const char *buffer,
+		 unsigned long sizebytes)
 {
 	int ret = 0;
 	struct ext2_inode *file_inode = NULL;
@@ -950,7 +949,7 @@ int ext4fs_write(const char *fname, unsigned char *buffer,
 	if (ext4fs_put_metadata(temp_ptr, itable_blkno))
 		goto fail;
 	/* copy the file content into data blocks */
-	if (ext4fs_write_file(file_inode, 0, sizebytes, (char *)buffer) == -1) {
+	if (ext4fs_write_file(file_inode, 0, sizebytes, buffer) == -1) {
 		printf("Error in copying content\n");
 		/* FIXME: Deallocate data blocks */
 		goto fail;
diff --git a/include/ext4fs.h b/include/ext4fs.h
index bcf440364e..d4dcefaa96 100644
--- a/include/ext4fs.h
+++ b/include/ext4fs.h
@@ -128,7 +128,7 @@ extern int gindex;
 int ext4fs_init(void);
 void ext4fs_deinit(void);
 int ext4fs_filename_unlink(char *filename);
-int ext4fs_write(const char *fname, unsigned char *buffer,
+int ext4fs_write(const char *fname, const char *buffer,
 		 unsigned long sizebytes);
 int ext4_write_file(const char *filename, void *buf, loff_t offset, loff_t len,
 		    loff_t *actwrite);
-- 
2.17.1

[U-Boot] [RESEND PATCH v5 5/7] fs: ext4: Add support for the creation of symbolic links

[Jean-Jacques Hiblot]

Re-use the functions used to write/create a file, to support creation of a
symbolic link.
The difference with a regular file are small:
- The inode mode is flagged with S_IFLNK instead of S_IFREG
- The ext2_dirent's filetype is FILETYPE_SYMLINK instead of FILETYPE_REG
- Instead of storing the content of a file in allocated blocks, the path
to the target is stored. And if the target's path is short enough, no block
is allocated and the target's path is stored in ext2_inode.b.symlink

As with regulars files, if a file/symlink with the same name exits, it is
unlinked first and then re-created.

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

---

Changes in v5:
- Fix bug in ext4fs_delete_file(). The type must be read from the
  inode.

Changes in v4: None
Changes in v3: None
Changes in v2:
- Fix issue in ext4fs_delete_file() when target in not stored in an
  allocated block

 fs/ext4/ext4_common.c |  2 +-
 fs/ext4/ext4_write.c  | 51 ++++++++++++++++++++++++++++++++++++-------
 include/ext4fs.h      |  3 ++-
 3 files changed, 46 insertions(+), 10 deletions(-)

diff --git a/fs/ext4/ext4_common.c b/fs/ext4/ext4_common.c
index 1ff2cd28a6..4bc23ab92e 100644
--- a/fs/ext4/ext4_common.c
+++ b/fs/ext4/ext4_common.c
@@ -607,7 +607,7 @@ restart_read:
 		dir->direntlen = cpu_to_le16(fs->blksz - totalbytes);
 
 	dir->namelen = strlen(filename);
-	dir->filetype = FILETYPE_REG;	/* regular file */
+	dir->filetype = file_type;
 	temp_dir = (char *)dir;
 	temp_dir = temp_dir + sizeof(struct ext2_dirent);
 	memcpy(temp_dir, filename, strlen(filename));
diff --git a/fs/ext4/ext4_write.c b/fs/ext4/ext4_write.c
index 51ca1c0916..cf2ac7ed41 100644
--- a/fs/ext4/ext4_write.c
+++ b/fs/ext4/ext4_write.c
@@ -465,6 +465,15 @@ static int ext4fs_delete_file(int inodeno)
 	if (le32_to_cpu(inode.size) % fs->blksz)
 		no_blocks++;
 
+	/*
+	 * special case for symlinks whose target are small enough that
+	 *it fits in struct ext2_inode.b.symlink: no block had been allocated
+	 */
+	if ((le16_to_cpu(inode.mode) & S_IFLNK) &&
+	    le32_to_cpu(inode.size) <= sizeof(inode.b.symlink)) {
+		no_blocks = 0;
+	}
+
 	if (le32_to_cpu(inode.flags) & EXT4_EXTENTS_FL) {
 		/* FIXME delete extent index blocks, i.e. eh_depth >= 1 */
 		struct ext4_extent_header *eh =
@@ -830,7 +839,7 @@ static int ext4fs_write_file(struct ext2_inode *file_inode,
 }
 
 int ext4fs_write(const char *fname, const char *buffer,
-		 unsigned long sizebytes)
+		 unsigned long sizebytes, int type)
 {
 	int ret = 0;
 	struct ext2_inode *file_inode = NULL;
@@ -853,8 +862,12 @@ int ext4fs_write(const char *fname, const char *buffer,
 	struct ext2_block_group *bgd = NULL;
 	struct ext_filesystem *fs = get_fs();
 	ALLOC_CACHE_ALIGN_BUFFER(char, filename, 256);
+	bool store_link_in_inode = false;
 	memset(filename, 0x00, 256);
 
+	if (type != FILETYPE_REG && type != FILETYPE_SYMLINK)
+		return -1;
+
 	g_parent_inode = zalloc(fs->inodesz);
 	if (!g_parent_inode)
 		goto fail_ext4fs_init;
@@ -893,8 +906,16 @@ int ext4fs_write(const char *fname, const char *buffer,
 		if (ret)
 			goto fail;
 	}
-	/* calucalate how many blocks required */
-	bytes_reqd_for_file = sizebytes;
+
+	/* calculate how many blocks required */
+	if (type == FILETYPE_SYMLINK &&
+	    sizebytes <= sizeof(file_inode->b.symlink)) {
+		store_link_in_inode = true;
+		bytes_reqd_for_file = 0;
+	} else {
+		bytes_reqd_for_file = sizebytes;
+	}
+
 	blks_reqd_for_file = lldiv(bytes_reqd_for_file, fs->blksz);
 	if (do_div(bytes_reqd_for_file, fs->blksz) != 0) {
 		blks_reqd_for_file++;
@@ -907,7 +928,7 @@ int ext4fs_write(const char *fname, const char *buffer,
 		goto fail;
 	}
 
-	inodeno = ext4fs_update_parent_dentry(filename, FILETYPE_REG);
+	inodeno = ext4fs_update_parent_dentry(filename, type);
 	if (inodeno == -1)
 		goto fail;
 	/* prepare file inode */
@@ -915,14 +936,23 @@ int ext4fs_write(const char *fname, const char *buffer,
 	if (!inode_buffer)
 		goto fail;
 	file_inode = (struct ext2_inode *)inode_buffer;
-	file_inode->mode = cpu_to_le16(S_IFREG | S_IRWXU |
-	    S_IRGRP | S_IROTH | S_IXGRP | S_IXOTH);
+	file_inode->size = cpu_to_le32(sizebytes);
+	if (type == FILETYPE_SYMLINK) {
+		file_inode->mode = cpu_to_le16(S_IFLNK | S_IRWXU | S_IRWXG |
+					       S_IRWXO);
+		if (store_link_in_inode) {
+			strncpy(file_inode->b.symlink, buffer, sizebytes);
+			sizebytes = 0;
+		}
+	} else {
+		file_inode->mode = cpu_to_le16(S_IFREG | S_IRWXU | S_IRGRP |
+					       S_IROTH | S_IXGRP | S_IXOTH);
+	}
 	/* ToDo: Update correct time */
 	file_inode->mtime = cpu_to_le32(timestamp);
 	file_inode->atime = cpu_to_le32(timestamp);
 	file_inode->ctime = cpu_to_le32(timestamp);
 	file_inode->nlinks = cpu_to_le16(1);
-	file_inode->size = cpu_to_le32(sizebytes);
 
 	/* Allocate data blocks */
 	ext4fs_allocate_blocks(file_inode, blocks_remaining,
@@ -1015,7 +1045,7 @@ int ext4_write_file(const char *filename, void *buf, loff_t offset,
 		return -1;
 	}
 
-	ret = ext4fs_write(filename, buf, len);
+	ret = ext4fs_write(filename, buf, len, FILETYPE_REG);
 	if (ret) {
 		printf("** Error ext4fs_write() **\n");
 		goto fail;
@@ -1030,3 +1060,8 @@ fail:
 
 	return -1;
 }
+
+int ext4fs_create_link(const char *target, const char *fname)
+{
+	return ext4fs_write(fname, target, strlen(target), FILETYPE_SYMLINK);
+}
diff --git a/include/ext4fs.h b/include/ext4fs.h
index d4dcefaa96..946dc6a302 100644
--- a/include/ext4fs.h
+++ b/include/ext4fs.h
@@ -129,9 +129,10 @@ int ext4fs_init(void);
 void ext4fs_deinit(void);
 int ext4fs_filename_unlink(char *filename);
 int ext4fs_write(const char *fname, const char *buffer,
-		 unsigned long sizebytes);
+				 unsigned long sizebytes, int type);
 int ext4_write_file(const char *filename, void *buf, loff_t offset, loff_t len,
 		    loff_t *actwrite);
+int ext4fs_create_link(const char *target, const char *fname);
 #endif
 
 struct ext_filesystem *get_fs(void);
-- 
2.17.1

[U-Boot] [RESEND PATCH v5 6/7] fs: Add a new command to create symbolic links

[Jean-Jacques Hiblot]

The command line is:
ln <interface> <dev[:part]> target linkname

Currently symbolic links are supported only in ext4 and only if the option
CMD_EXT4_WRITE is enabled.

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
---

Changes in v5: None
Changes in v4: None
Changes in v3: None
Changes in v2: None

 cmd/fs.c     | 14 ++++++++++++++
 fs/fs.c      | 44 ++++++++++++++++++++++++++++++++++++++++++++
 include/fs.h |  2 ++
 3 files changed, 60 insertions(+)

diff --git a/cmd/fs.c b/cmd/fs.c
index 8064a1c84d..ba0e08cd12 100644
--- a/cmd/fs.c
+++ b/cmd/fs.c
@@ -74,6 +74,20 @@ U_BOOT_CMD(
 	"      device type 'interface' instance 'dev'."
 )
 
+static int do_ln_wrapper(cmd_tbl_t *cmdtp, int flag, int argc,
+			 char * const argv[])
+{
+	return do_ln(cmdtp, flag, argc, argv, FS_TYPE_ANY);
+}
+
+U_BOOT_CMD(
+	ln,	5,	1,	do_ln_wrapper,
+	"Create a symbolic link",
+	"<interface> <dev[:part]> target linkname\n"
+	"    - create a symbolic link to 'target' with the name 'linkname' on\n"
+	"      device type 'interface' instance 'dev'."
+)
+
 static int do_fstype_wrapper(cmd_tbl_t *cmdtp, int flag, int argc,
 				char * const argv[])
 {
diff --git a/fs/fs.c b/fs/fs.c
index c5c35ebf5f..736ebef4a9 100644
--- a/fs/fs.c
+++ b/fs/fs.c
@@ -90,6 +90,11 @@ static inline int fs_write_unsupported(const char *filename, void *buf,
 	return -1;
 }
 
+static inline int fs_ln_unsupported(const char *filename, const char *target)
+{
+	return -1;
+}
+
 static inline void fs_close_unsupported(void)
 {
 }
@@ -154,6 +159,7 @@ struct fstype_info {
 	void (*closedir)(struct fs_dir_stream *dirs);
 	int (*unlink)(const char *filename);
 	int (*mkdir)(const char *dirname);
+	int (*ln)(const char *filename, const char *target);
 };
 
 static struct fstype_info fstypes[] = {
@@ -181,6 +187,7 @@ static struct fstype_info fstypes[] = {
 		.opendir = fat_opendir,
 		.readdir = fat_readdir,
 		.closedir = fat_closedir,
+		.ln = fs_ln_unsupported,
 	},
 #endif
 
@@ -197,8 +204,10 @@ static struct fstype_info fstypes[] = {
 		.read = ext4_read_file,
 #ifdef CONFIG_CMD_EXT4_WRITE
 		.write = ext4_write_file,
+		.ln = ext4fs_create_link,
 #else
 		.write = fs_write_unsupported,
+		.ln = fs_ln_unsupported,
 #endif
 		.uuid = ext4fs_uuid,
 		.opendir = fs_opendir_unsupported,
@@ -222,6 +231,7 @@ static struct fstype_info fstypes[] = {
 		.opendir = fs_opendir_unsupported,
 		.unlink = fs_unlink_unsupported,
 		.mkdir = fs_mkdir_unsupported,
+		.ln = fs_ln_unsupported,
 	},
 #endif
 #ifdef CONFIG_CMD_UBIFS
@@ -240,6 +250,7 @@ static struct fstype_info fstypes[] = {
 		.opendir = fs_opendir_unsupported,
 		.unlink = fs_unlink_unsupported,
 		.mkdir = fs_mkdir_unsupported,
+		.ln = fs_ln_unsupported,
 	},
 #endif
 #ifdef CONFIG_FS_BTRFS
@@ -258,6 +269,7 @@ static struct fstype_info fstypes[] = {
 		.opendir = fs_opendir_unsupported,
 		.unlink = fs_unlink_unsupported,
 		.mkdir = fs_mkdir_unsupported,
+		.ln = fs_ln_unsupported,
 	},
 #endif
 	{
@@ -275,6 +287,7 @@ static struct fstype_info fstypes[] = {
 		.opendir = fs_opendir_unsupported,
 		.unlink = fs_unlink_unsupported,
 		.mkdir = fs_mkdir_unsupported,
+		.ln = fs_ln_unsupported,
 	},
 };
 
@@ -602,6 +615,22 @@ int fs_mkdir(const char *dirname)
 	return ret;
 }
 
+int fs_ln(const char *fname, const char *target)
+{
+	struct fstype_info *info = fs_get_info(fs_type);
+	int ret;
+
+	ret = info->ln(fname, target);
+
+	if (ret < 0) {
+		printf("** Unable to create link %s -> %s **\n", fname, target);
+		ret = -1;
+	}
+	fs_close();
+
+	return ret;
+}
+
 int do_size(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[],
 		int fstype)
 {
@@ -840,3 +869,18 @@ int do_mkdir(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[],
 
 	return 0;
 }
+
+int do_ln(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[],
+	  int fstype)
+{
+	if (argc != 5)
+		return CMD_RET_USAGE;
+
+	if (fs_set_blk_dev(argv[1], argv[2], fstype))
+		return 1;
+
+	if (fs_ln(argv[3], argv[4]))
+		return 1;
+
+	return 0;
+}
diff --git a/include/fs.h b/include/fs.h
index aa3604db8d..6854597700 100644
--- a/include/fs.h
+++ b/include/fs.h
@@ -191,6 +191,8 @@ int do_rm(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[],
 		int fstype);
 int do_mkdir(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[],
 		int fstype);
+int do_ln(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[],
+	  int fstype);
 
 /*
  * Determine the UUID of the specified filesystem and print it. Optionally it is
-- 
2.17.1

[U-Boot] [RESEND PATCH v5 7/7] test: fs: Added tests for symlinks

[Jean-Jacques Hiblot]

Test cases are:
1) basic link creation, verify it can be followed
2) chained links, verify it can be followed
3) replace exiting file a with a link, and a link with a link. verify it
   can be followed
4) create a broken link, verify it can't be followed

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

---

Changes in v5:
- Added FS integrity test

Changes in v4: None
Changes in v3:
- removed FS integrity test (fsck) in python tests

Changes in v2:
- Added python tests for symlinks under sandbox

 test/py/tests/test_fs/conftest.py     |  75 +++++++++++++++
 test/py/tests/test_fs/fstest_defs.py  |   3 +
 test/py/tests/test_fs/test_symlink.py | 130 ++++++++++++++++++++++++++
 3 files changed, 208 insertions(+)
 create mode 100644 test/py/tests/test_fs/test_symlink.py

diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
index 745ed0ed38..9324657d21 100644
--- a/test/py/tests/test_fs/conftest.py
+++ b/test/py/tests/test_fs/conftest.py
@@ -13,6 +13,7 @@ supported_fs_basic = ['fat16', 'fat32', 'ext4']
 supported_fs_ext = ['fat16', 'fat32']
 supported_fs_mkdir = ['fat16', 'fat32']
 supported_fs_unlink = ['fat16', 'fat32']
+supported_fs_symlink = ['ext4']
 
 #
 # Filesystem test specific setup
@@ -48,6 +49,7 @@ def pytest_configure(config):
     global supported_fs_ext
     global supported_fs_mkdir
     global supported_fs_unlink
+    global supported_fs_symlink
 
     def intersect(listA, listB):
         return  [x for x in listA if x in listB]
@@ -59,6 +61,7 @@ def pytest_configure(config):
         supported_fs_ext =  intersect(supported_fs, supported_fs_ext)
         supported_fs_mkdir =  intersect(supported_fs, supported_fs_mkdir)
         supported_fs_unlink =  intersect(supported_fs, supported_fs_unlink)
+        supported_fs_symlink =  intersect(supported_fs, supported_fs_symlink)
 
 def pytest_generate_tests(metafunc):
     """Parametrize fixtures, fs_obj_xxx
@@ -84,6 +87,9 @@ def pytest_generate_tests(metafunc):
     if 'fs_obj_unlink' in metafunc.fixturenames:
         metafunc.parametrize('fs_obj_unlink', supported_fs_unlink,
             indirect=True, scope='module')
+    if 'fs_obj_symlink' in metafunc.fixturenames:
+        metafunc.parametrize('fs_obj_symlink', supported_fs_symlink,
+            indirect=True, scope='module')
 
 #
 # Helper functions
@@ -525,3 +531,72 @@ def fs_obj_unlink(request, u_boot_config):
         call('rmdir %s' % mount_dir, shell=True)
         if fs_img:
             call('rm -f %s' % fs_img, shell=True)
+
+#
+# Fixture for symlink fs test
+#
+# NOTE: yield_fixture was deprecated since pytest-3.0
+ at pytest.yield_fixture()
+def fs_obj_symlink(request, u_boot_config):
+    """Set up a file system to be used in symlink fs test.
+
+    Args:
+        request: Pytest request object.
+        u_boot_config: U-boot configuration.
+
+    Return:
+        A fixture for basic fs test, i.e. a triplet of file system type,
+        volume file name and  a list of MD5 hashes.
+    """
+    fs_type = request.param
+    fs_img = ''
+
+    fs_ubtype = fstype_to_ubname(fs_type)
+    check_ubconfig(u_boot_config, fs_ubtype)
+
+    mount_dir = u_boot_config.persistent_data_dir + '/mnt'
+
+    small_file = mount_dir + '/' + SMALL_FILE
+    medium_file = mount_dir + '/' + MEDIUM_FILE
+
+    try:
+
+        # 3GiB volume
+        fs_img = mk_fs(u_boot_config, fs_type, 0x40000000, '1GB')
+
+        # Mount the image so we can populate it.
+        check_call('mkdir -p %s' % mount_dir, shell=True)
+        mount_fs(fs_type, fs_img, mount_dir)
+
+        # Create a subdirectory.
+        check_call('mkdir %s/SUBDIR' % mount_dir, shell=True)
+
+        # Create a small file in this image.
+        check_call('dd if=/dev/urandom of=%s bs=1M count=1'
+                   % small_file, shell=True)
+
+        # Create a medium file in this image.
+        check_call('dd if=/dev/urandom of=%s bs=10M count=1'
+                   % medium_file, shell=True)
+
+        # Generate the md5sums of reads that we will test against small file
+        out = check_output(
+            'dd if=%s bs=1M skip=0 count=1 2> /dev/null | md5sum'
+            % small_file, shell=True)
+        md5val = [out.split()[0]]
+        out = check_output(
+            'dd if=%s bs=10M skip=0 count=1 2> /dev/null | md5sum'
+            % medium_file, shell=True)
+        md5val.extend([out.split()[0]])
+
+        umount_fs(mount_dir)
+    except CalledProcessError:
+        pytest.skip('Setup failed for filesystem: ' + fs_type)
+        return
+    else:
+        yield [fs_ubtype, fs_img, md5val]
+    finally:
+        umount_fs(mount_dir)
+        call('rmdir %s' % mount_dir, shell=True)
+        if fs_img:
+            call('rm -f %s' % fs_img, shell=True)
diff --git a/test/py/tests/test_fs/fstest_defs.py b/test/py/tests/test_fs/fstest_defs.py
index 5f107562d9..35b2bb6518 100644
--- a/test/py/tests/test_fs/fstest_defs.py
+++ b/test/py/tests/test_fs/fstest_defs.py
@@ -6,6 +6,9 @@ MIN_FILE='testfile'
 # $SMALL_FILE is the name of the 1MB file in the file system image
 SMALL_FILE='1MB.file'
 
+# $MEDIUM_FILE is the name of the 10MB file in the file system image
+MEDIUM_FILE='10MB.file'
+
 # $BIG_FILE is the name of the 2.5GB file in the file system image
 BIG_FILE='2.5GB.file'
 
diff --git a/test/py/tests/test_fs/test_symlink.py b/test/py/tests/test_fs/test_symlink.py
new file mode 100644
index 0000000000..9ced101a29
--- /dev/null
+++ b/test/py/tests/test_fs/test_symlink.py
@@ -0,0 +1,130 @@
+# SPDX-License-Identifier:      GPL-2.0+
+# Copyright (c) 2019, Texas Instrument
+# Author: Jean-Jacques Hiblot <jjhiblot@ti.com>
+#
+# U-Boot File System:symlink Test
+
+"""
+This test verifies unlink operation (deleting a file or a directory)
+on file system.
+"""
+
+import pytest
+import re
+from fstest_defs import *
+from fstest_helpers import assert_fs_integrity
+
+
+ at pytest.mark.boardspec('sandbox')
+ at pytest.mark.slow
+class TestSymlink(object):
+    def test_symlink1(self, u_boot_console, fs_obj_symlink):
+        """
+        Test Case 1 - create a link. and follow it when reading
+        """
+        fs_type, fs_img, md5val = fs_obj_symlink
+        with u_boot_console.log.section('Test Case 1 - create link and read'):
+            output = u_boot_console.run_command_list([
+                'host bind 0 %s' % fs_img,
+                'setenv filesize',
+                'ln host 0:0 %s /%s.link ' % (SMALL_FILE, SMALL_FILE),
+            ])
+            assert('' in ''.join(output))
+
+            output = u_boot_console.run_command_list([
+                '%sload host 0:0 %x /%s.link' % (fs_type, ADDR, SMALL_FILE),
+                'printenv filesize'])
+            assert('filesize=100000' in ''.join(output))
+
+            # Test Case 4b - Read full 1MB of small file
+            output = u_boot_console.run_command_list([
+                'md5sum %x $filesize' % ADDR,
+                'setenv filesize'])
+            assert(md5val[0] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
+
+    def test_symlink2(self, u_boot_console, fs_obj_symlink):
+        """
+        Test Case 2 - create chained links
+        """
+        fs_type, fs_img, md5val = fs_obj_symlink
+        with u_boot_console.log.section('Test Case 2 - create chained links'):
+            output = u_boot_console.run_command_list([
+                'host bind 0 %s' % fs_img,
+                'setenv filesize',
+                'ln host 0:0 %s /%s.link1 ' % (SMALL_FILE, SMALL_FILE),
+                'ln host 0:0 /%s.link1 /SUBDIR/%s.link2' % (
+                    SMALL_FILE, SMALL_FILE),
+                'ln host 0:0 SUBDIR/%s.link2 /%s.link3' % (
+                    SMALL_FILE, SMALL_FILE),
+            ])
+            assert('' in ''.join(output))
+
+            output = u_boot_console.run_command_list([
+                '%sload host 0:0 %x /%s.link3' % (fs_type, ADDR, SMALL_FILE),
+                'printenv filesize'])
+            assert('filesize=100000' in ''.join(output))
+
+            # Test Case 4b - Read full 1MB of small file
+            output = u_boot_console.run_command_list([
+                'md5sum %x $filesize' % ADDR,
+                'setenv filesize'])
+            assert(md5val[0] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
+
+    def test_symlink3(self, u_boot_console, fs_obj_symlink):
+        """
+        Test Case 3 - replace file/link with link
+        """
+        fs_type, fs_img, md5val = fs_obj_symlink
+        with u_boot_console.log.section('Test Case 1 - create link and read'):
+            output = u_boot_console.run_command_list([
+                'host bind 0 %s' % fs_img,
+                'setenv filesize',
+                'ln host 0:0 %s /%s ' % (MEDIUM_FILE, SMALL_FILE),
+                'ln host 0:0 %s /%s.link ' % (MEDIUM_FILE, MEDIUM_FILE),
+            ])
+            assert('' in ''.join(output))
+
+            output = u_boot_console.run_command_list([
+                '%sload host 0:0 %x /%s' % (fs_type, ADDR, SMALL_FILE),
+                'printenv filesize'])
+            assert('filesize=a00000' in ''.join(output))
+
+            output = u_boot_console.run_command_list([
+                'md5sum %x $filesize' % ADDR,
+                'setenv filesize'])
+            assert(md5val[1] in ''.join(output))
+
+            output = u_boot_console.run_command_list([
+                'ln host 0:0 %s.link /%s ' % (MEDIUM_FILE, SMALL_FILE),
+                '%sload host 0:0 %x /%s' % (fs_type, ADDR, SMALL_FILE),
+                'printenv filesize'])
+            assert('filesize=a00000' in ''.join(output))
+
+            output = u_boot_console.run_command_list([
+                'md5sum %x $filesize' % ADDR,
+                'setenv filesize'])
+            assert(md5val[1] in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
+
+    def test_symlink4(self, u_boot_console, fs_obj_symlink):
+        """
+        Test Case 4 - create a broken link
+        """
+        fs_type, fs_img, md5val = fs_obj_symlink
+        with u_boot_console.log.section('Test Case 1 - create link and read'):
+
+            output = u_boot_console.run_command_list([
+                'setenv filesize',
+                'ln host 0:0 nowhere /link ',
+            ])
+            assert('' in ''.join(output))
+
+            output = u_boot_console.run_command(
+                '%sload host 0:0 %x /link' %
+                (fs_type, ADDR))
+            with u_boot_console.disable_check('error_notification'):
+                output = u_boot_console.run_command('printenv filesize')
+            assert('"filesize" not defined' in ''.join(output))
+            assert_fs_integrity(fs_type, fs_img)
-- 
2.17.1

[U-Boot] [RESEND PATCH v5 3/7] test: fs: Add filesystem integrity checks

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:

> We need to make sure that file writes,file creation, etc. are properly
> performed and do not corrupt the filesystem.
> To help with this, introduce the assert_fs_integrity() function that
> executes the appropriate fsck tool. It should be called at the end of any
> test that modify the content/organization of the filesystem.
> Currently only supports FATs and EXT4.
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> 

Reviewed-by: Tom Rini <trini@konsulko.com>

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190213/c7d03a5a/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 1/7] fs: ext4: do not allow writes if metadata checksum is active

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:21PM +0100, Jean-Jacques Hiblot wrote:

> u-boot does not supports updating the metadata chacksums
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

Deferred as I picked up
commit 2e7365518acdb8fb6e9be332c8a6c57b457188d9
Author: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
Date:   Fri Mar 22 09:33:52 2019 +0100

    fs: ext4: do not write on filesystem with metadata_csum feature
    
as it slipped my mind you had posted this patch in this series, sorry!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190409/e6f203cc/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:

> We need to make sure that file writes,file creation, etc. are properly
> performed and do not corrupt the filesystem.
> To help with this, introduce the assert_fs_integrity() function that
> executes the appropriate fsck tool. It should be called at the end of any
> test that modify the content/organization of the filesystem.
> Currently only supports FATs and EXT4.
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

OK, I'm adding in a bunch of people to CC here.  The issue with this
patch is that by adding fsck to our tests we see 34 FAT16/FAT32
failures:
TestFsBasic.test_fs13[fat16]
TestFsBasic.test_fs11[fat32]
TestFsBasic.test_fs12[fat32]
TestFsBasic.test_fs13[fat32]
TestFsExt.test_fs_ext1[fat32]
TestFsExt.test_fs_ext2[fat32]
TestFsExt.test_fs_ext3[fat32]
TestFsExt.test_fs_ext4[fat32]
TestFsExt.test_fs_ext5[fat32]
TestFsExt.test_fs_ext6[fat32]
TestFsExt.test_fs_ext7[fat32]
TestFsExt.test_fs_ext8[fat32]
TestFsExt.test_fs_ext9[fat32]
TestMkdir.test_mkdir6[fat16]
TestMkdir.test_mkdir1[fat32]
TestMkdir.test_mkdir2[fat32]
TestMkdir.test_mkdir3[fat32]
TestMkdir.test_mkdir4[fat32]
TestMkdir.test_mkdir5[fat32]
TestMkdir.test_mkdir6[fat32]
TestUnlink.test_unlink1[fat16]
TestUnlink.test_unlink2[fat16]
TestUnlink.test_unlink3[fat16]
TestUnlink.test_unlink4[fat16]
TestUnlink.test_unlink5[fat16]
TestUnlink.test_unlink6[fat16]
TestUnlink.test_unlink7[fat16]
TestUnlink.test_unlink1[fat32]
TestUnlink.test_unlink2[fat32]
TestUnlink.test_unlink3[fat32]
TestUnlink.test_unlink4[fat32]
TestUnlink.test_unlink5[fat32]
TestUnlink.test_unlink6[fat32]
TestUnlink.test_unlink7[fat32]

So... I'm inclined to say that to start with, I bring this patch in and
then disable FAT fsck (as I cannot see how to mark these as xfail with
a comment that we need to fix them, only for FAT).  But we should get
these FAT problems fixed.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190409/23986eae/attachment.sig>

[U-Boot] [PATCH] test.py: Disable fsck for FAT tests for now

[Tom Rini]

Currently enabling fsck on FAT16/FAT32 exposes that we have problems
with:
TestFsBasic.test_fs13[fat16]
TestFsBasic.test_fs11[fat32]
TestFsBasic.test_fs12[fat32]
TestFsBasic.test_fs13[fat32]
TestFsExt.test_fs_ext1[fat32]
TestFsExt.test_fs_ext2[fat32]
TestFsExt.test_fs_ext3[fat32]
TestFsExt.test_fs_ext4[fat32]
TestFsExt.test_fs_ext5[fat32]
TestFsExt.test_fs_ext6[fat32]
TestFsExt.test_fs_ext7[fat32]
TestFsExt.test_fs_ext8[fat32]
TestFsExt.test_fs_ext9[fat32]
TestMkdir.test_mkdir6[fat16]
TestMkdir.test_mkdir1[fat32]
TestMkdir.test_mkdir2[fat32]
TestMkdir.test_mkdir3[fat32]
TestMkdir.test_mkdir4[fat32]
TestMkdir.test_mkdir5[fat32]
TestMkdir.test_mkdir6[fat32]
TestUnlink.test_unlink1[fat16]
TestUnlink.test_unlink2[fat16]
TestUnlink.test_unlink3[fat16]
TestUnlink.test_unlink4[fat16]
TestUnlink.test_unlink5[fat16]
TestUnlink.test_unlink6[fat16]
TestUnlink.test_unlink7[fat16]
TestUnlink.test_unlink1[fat32]
TestUnlink.test_unlink2[fat32]
TestUnlink.test_unlink3[fat32]
TestUnlink.test_unlink4[fat32]
TestUnlink.test_unlink5[fat32]
TestUnlink.test_unlink6[fat32]
TestUnlink.test_unlink7[fat32]

While we need to resolve these problems we also should enable fsck for
ext4 to ensure that things remain in good shape there.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 test/py/tests/test_fs/fstest_helpers.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/test/py/tests/test_fs/fstest_helpers.py b/test/py/tests/test_fs/fstest_helpers.py
index 637bc30a9141..faec2982489d 100644
--- a/test/py/tests/test_fs/fstest_helpers.py
+++ b/test/py/tests/test_fs/fstest_helpers.py
@@ -9,7 +9,5 @@ def assert_fs_integrity(fs_type, fs_img):
     try:
         if fs_type == 'ext4':
             check_call('fsck.ext4 -n -f %s' % fs_img, shell=True)
-        if fs_type == 'fat':
-            check_call('fsck.fat -n %s' % fs_img, shell=True)
     except CalledProcessError:
         raise
-- 
2.7.4

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Heinrich Schuchardt]

On 4/9/19 10:03 PM, Tom Rini wrote:
> On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:
>
>> We need to make sure that file writes,file creation, etc. are properly
>> performed and do not corrupt the filesystem.
>> To help with this, introduce the assert_fs_integrity() function that
>> executes the appropriate fsck tool. It should be called at the end of any
>> test that modify the content/organization of the filesystem.
>> Currently only supports FATs and EXT4.
>>
>> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
>> Reviewed-by: Tom Rini <trini@konsulko.com>
>
> OK, I'm adding in a bunch of people to CC here.  The issue with this
> patch is that by adding fsck to our tests we see 34 FAT16/FAT32
> failures:
> TestFsBasic.test_fs13[fat16]
> TestFsBasic.test_fs11[fat32]
> TestFsBasic.test_fs12[fat32]
> TestFsBasic.test_fs13[fat32]
> TestFsExt.test_fs_ext1[fat32]
> TestFsExt.test_fs_ext2[fat32]
> TestFsExt.test_fs_ext3[fat32]
> TestFsExt.test_fs_ext4[fat32]
> TestFsExt.test_fs_ext5[fat32]
> TestFsExt.test_fs_ext6[fat32]
> TestFsExt.test_fs_ext7[fat32]
> TestFsExt.test_fs_ext8[fat32]
> TestFsExt.test_fs_ext9[fat32]
> TestMkdir.test_mkdir6[fat16]
> TestMkdir.test_mkdir1[fat32]
> TestMkdir.test_mkdir2[fat32]
> TestMkdir.test_mkdir3[fat32]
> TestMkdir.test_mkdir4[fat32]
> TestMkdir.test_mkdir5[fat32]
> TestMkdir.test_mkdir6[fat32]
> TestUnlink.test_unlink1[fat16]
> TestUnlink.test_unlink2[fat16]
> TestUnlink.test_unlink3[fat16]
> TestUnlink.test_unlink4[fat16]
> TestUnlink.test_unlink5[fat16]
> TestUnlink.test_unlink6[fat16]
> TestUnlink.test_unlink7[fat16]
> TestUnlink.test_unlink1[fat32]
> TestUnlink.test_unlink2[fat32]
> TestUnlink.test_unlink3[fat32]
> TestUnlink.test_unlink4[fat32]
> TestUnlink.test_unlink5[fat32]
> TestUnlink.test_unlink6[fat32]
> TestUnlink.test_unlink7[fat32]

I appreciate that we get tests for file system functions.

Unfortunately the test output is rudimentary. Can we have something more
expressive than unlink1 - unlink7?

CCing Takahiro as he was contributing recently to FAT.

Best regards

Heinrich

>
> So... I'm inclined to say that to start with, I bring this patch in and
> then disable FAT fsck (as I cannot see how to mark these as xfail with
> a comment that we need to fix them, only for FAT).  But we should get
> these FAT problems fixed.
>

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Tom Rini]

On Wed, Apr 10, 2019 at 02:10:12AM +0200, Heinrich Schuchardt wrote:
> On 4/9/19 10:03 PM, Tom Rini wrote:
> > On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:
> >
> >> We need to make sure that file writes,file creation, etc. are properly
> >> performed and do not corrupt the filesystem.
> >> To help with this, introduce the assert_fs_integrity() function that
> >> executes the appropriate fsck tool. It should be called at the end of any
> >> test that modify the content/organization of the filesystem.
> >> Currently only supports FATs and EXT4.
> >>
> >> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> >> Reviewed-by: Tom Rini <trini@konsulko.com>
> >
> > OK, I'm adding in a bunch of people to CC here.  The issue with this
> > patch is that by adding fsck to our tests we see 34 FAT16/FAT32
> > failures:
> > TestFsBasic.test_fs13[fat16]
> > TestFsBasic.test_fs11[fat32]
> > TestFsBasic.test_fs12[fat32]
> > TestFsBasic.test_fs13[fat32]
> > TestFsExt.test_fs_ext1[fat32]
> > TestFsExt.test_fs_ext2[fat32]
> > TestFsExt.test_fs_ext3[fat32]
> > TestFsExt.test_fs_ext4[fat32]
> > TestFsExt.test_fs_ext5[fat32]
> > TestFsExt.test_fs_ext6[fat32]
> > TestFsExt.test_fs_ext7[fat32]
> > TestFsExt.test_fs_ext8[fat32]
> > TestFsExt.test_fs_ext9[fat32]
> > TestMkdir.test_mkdir6[fat16]
> > TestMkdir.test_mkdir1[fat32]
> > TestMkdir.test_mkdir2[fat32]
> > TestMkdir.test_mkdir3[fat32]
> > TestMkdir.test_mkdir4[fat32]
> > TestMkdir.test_mkdir5[fat32]
> > TestMkdir.test_mkdir6[fat32]
> > TestUnlink.test_unlink1[fat16]
> > TestUnlink.test_unlink2[fat16]
> > TestUnlink.test_unlink3[fat16]
> > TestUnlink.test_unlink4[fat16]
> > TestUnlink.test_unlink5[fat16]
> > TestUnlink.test_unlink6[fat16]
> > TestUnlink.test_unlink7[fat16]
> > TestUnlink.test_unlink1[fat32]
> > TestUnlink.test_unlink2[fat32]
> > TestUnlink.test_unlink3[fat32]
> > TestUnlink.test_unlink4[fat32]
> > TestUnlink.test_unlink5[fat32]
> > TestUnlink.test_unlink6[fat32]
> > TestUnlink.test_unlink7[fat32]
> 
> I appreciate that we get tests for file system functions.
> 
> Unfortunately the test output is rudimentary. Can we have something more
> expressive than unlink1 - unlink7?
> 
> CCing Takahiro as he was contributing recently to FAT.

Sorry, yes, kind of?  I pasted that in for brevity, but it's basically
that for example all of test/py/tests/test_fs/test_unlink.py fails if
you fsck the image in question after each test.  If you apply
https://patchwork.ozlabs.org/patch/1041186/ (to avoid spurious ext4
failures) and then https://patchwork.ozlabs.org/patch/1041181/ and run
'make tests' you'll see the full output.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190409/5ebf4b86/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Takahiro Akashi]

On Tue, Apr 09, 2019 at 08:19:40PM -0400, Tom Rini wrote:
> On Wed, Apr 10, 2019 at 02:10:12AM +0200, Heinrich Schuchardt wrote:
> > On 4/9/19 10:03 PM, Tom Rini wrote:
> > > On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:
> > >
> > >> We need to make sure that file writes,file creation, etc. are properly
> > >> performed and do not corrupt the filesystem.
> > >> To help with this, introduce the assert_fs_integrity() function that
> > >> executes the appropriate fsck tool. It should be called at the end of any
> > >> test that modify the content/organization of the filesystem.
> > >> Currently only supports FATs and EXT4.
> > >>
> > >> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> > >> Reviewed-by: Tom Rini <trini@konsulko.com>
> > >
> > > OK, I'm adding in a bunch of people to CC here.  The issue with this
> > > patch is that by adding fsck to our tests we see 34 FAT16/FAT32
> > > failures:
> > > TestFsBasic.test_fs13[fat16]
> > > TestFsBasic.test_fs11[fat32]
> > > TestFsBasic.test_fs12[fat32]
> > > TestFsBasic.test_fs13[fat32]
> > > TestFsExt.test_fs_ext1[fat32]
> > > TestFsExt.test_fs_ext2[fat32]
> > > TestFsExt.test_fs_ext3[fat32]
> > > TestFsExt.test_fs_ext4[fat32]
> > > TestFsExt.test_fs_ext5[fat32]
> > > TestFsExt.test_fs_ext6[fat32]
> > > TestFsExt.test_fs_ext7[fat32]
> > > TestFsExt.test_fs_ext8[fat32]
> > > TestFsExt.test_fs_ext9[fat32]
> > > TestMkdir.test_mkdir6[fat16]
> > > TestMkdir.test_mkdir1[fat32]
> > > TestMkdir.test_mkdir2[fat32]
> > > TestMkdir.test_mkdir3[fat32]
> > > TestMkdir.test_mkdir4[fat32]
> > > TestMkdir.test_mkdir5[fat32]
> > > TestMkdir.test_mkdir6[fat32]
> > > TestUnlink.test_unlink1[fat16]
> > > TestUnlink.test_unlink2[fat16]
> > > TestUnlink.test_unlink3[fat16]
> > > TestUnlink.test_unlink4[fat16]
> > > TestUnlink.test_unlink5[fat16]
> > > TestUnlink.test_unlink6[fat16]
> > > TestUnlink.test_unlink7[fat16]
> > > TestUnlink.test_unlink1[fat32]
> > > TestUnlink.test_unlink2[fat32]
> > > TestUnlink.test_unlink3[fat32]
> > > TestUnlink.test_unlink4[fat32]
> > > TestUnlink.test_unlink5[fat32]
> > > TestUnlink.test_unlink6[fat32]
> > > TestUnlink.test_unlink7[fat32]
> > 
> > I appreciate that we get tests for file system functions.
> > 
> > Unfortunately the test output is rudimentary. Can we have something more
> > expressive than unlink1 - unlink7?
> > 
> > CCing Takahiro as he was contributing recently to FAT.
> 
> Sorry, yes, kind of?  I pasted that in for brevity, but it's basically
> that for example all of test/py/tests/test_fs/test_unlink.py fails if
> you fsck the image in question after each test.  If you apply
> https://patchwork.ozlabs.org/patch/1041186/ (to avoid spurious ext4
> failures) and then https://patchwork.ozlabs.org/patch/1041181/ and run
> 'make tests' you'll see the full output.

I have no time to dig into this issue right now,
but if you give me a log from fsck, particularly
why fsck failed here, it would help me to understand
the problem.

# like the case of ext4, we might have to turn off
# some option at fsck?

Thanks,
-Takahiro Akashi

> -- 
> Tom

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Tom Rini]

On Wed, Apr 10, 2019 at 10:37:42AM +0900, Takahiro Akashi wrote:
> On Tue, Apr 09, 2019 at 08:19:40PM -0400, Tom Rini wrote:
> > On Wed, Apr 10, 2019 at 02:10:12AM +0200, Heinrich Schuchardt wrote:
> > > On 4/9/19 10:03 PM, Tom Rini wrote:
> > > > On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:
> > > >
> > > >> We need to make sure that file writes,file creation, etc. are properly
> > > >> performed and do not corrupt the filesystem.
> > > >> To help with this, introduce the assert_fs_integrity() function that
> > > >> executes the appropriate fsck tool. It should be called at the end of any
> > > >> test that modify the content/organization of the filesystem.
> > > >> Currently only supports FATs and EXT4.
> > > >>
> > > >> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> > > >> Reviewed-by: Tom Rini <trini@konsulko.com>
> > > >
> > > > OK, I'm adding in a bunch of people to CC here.  The issue with this
> > > > patch is that by adding fsck to our tests we see 34 FAT16/FAT32
> > > > failures:
> > > > TestFsBasic.test_fs13[fat16]
> > > > TestFsBasic.test_fs11[fat32]
> > > > TestFsBasic.test_fs12[fat32]
> > > > TestFsBasic.test_fs13[fat32]
> > > > TestFsExt.test_fs_ext1[fat32]
> > > > TestFsExt.test_fs_ext2[fat32]
> > > > TestFsExt.test_fs_ext3[fat32]
> > > > TestFsExt.test_fs_ext4[fat32]
> > > > TestFsExt.test_fs_ext5[fat32]
> > > > TestFsExt.test_fs_ext6[fat32]
> > > > TestFsExt.test_fs_ext7[fat32]
> > > > TestFsExt.test_fs_ext8[fat32]
> > > > TestFsExt.test_fs_ext9[fat32]
> > > > TestMkdir.test_mkdir6[fat16]
> > > > TestMkdir.test_mkdir1[fat32]
> > > > TestMkdir.test_mkdir2[fat32]
> > > > TestMkdir.test_mkdir3[fat32]
> > > > TestMkdir.test_mkdir4[fat32]
> > > > TestMkdir.test_mkdir5[fat32]
> > > > TestMkdir.test_mkdir6[fat32]
> > > > TestUnlink.test_unlink1[fat16]
> > > > TestUnlink.test_unlink2[fat16]
> > > > TestUnlink.test_unlink3[fat16]
> > > > TestUnlink.test_unlink4[fat16]
> > > > TestUnlink.test_unlink5[fat16]
> > > > TestUnlink.test_unlink6[fat16]
> > > > TestUnlink.test_unlink7[fat16]
> > > > TestUnlink.test_unlink1[fat32]
> > > > TestUnlink.test_unlink2[fat32]
> > > > TestUnlink.test_unlink3[fat32]
> > > > TestUnlink.test_unlink4[fat32]
> > > > TestUnlink.test_unlink5[fat32]
> > > > TestUnlink.test_unlink6[fat32]
> > > > TestUnlink.test_unlink7[fat32]
> > > 
> > > I appreciate that we get tests for file system functions.
> > > 
> > > Unfortunately the test output is rudimentary. Can we have something more
> > > expressive than unlink1 - unlink7?
> > > 
> > > CCing Takahiro as he was contributing recently to FAT.
> > 
> > Sorry, yes, kind of?  I pasted that in for brevity, but it's basically
> > that for example all of test/py/tests/test_fs/test_unlink.py fails if
> > you fsck the image in question after each test.  If you apply
> > https://patchwork.ozlabs.org/patch/1041186/ (to avoid spurious ext4
> > failures) and then https://patchwork.ozlabs.org/patch/1041181/ and run
> > 'make tests' you'll see the full output.
> 
> I have no time to dig into this issue right now,
> but if you give me a log from fsck, particularly
> why fsck failed here, it would help me to understand
> the problem.

The raw log can be seen here:
https://gist.github.com/trini/b68799ed9880a31a3289e9bea033831d

> # like the case of ext4, we might have to turn off
> # some option at fsck?

Note that for ext4 we're turning off the metadata csum feature of the
filesystem as we do not support it.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190409/2bf69d69/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Takahiro Akashi]

On Tue, Apr 09, 2019 at 10:25:14PM -0400, Tom Rini wrote:
> On Wed, Apr 10, 2019 at 10:37:42AM +0900, Takahiro Akashi wrote:
> > On Tue, Apr 09, 2019 at 08:19:40PM -0400, Tom Rini wrote:
> > > On Wed, Apr 10, 2019 at 02:10:12AM +0200, Heinrich Schuchardt wrote:
> > > > On 4/9/19 10:03 PM, Tom Rini wrote:
> > > > > On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:
> > > > >
> > > > >> We need to make sure that file writes,file creation, etc. are properly
> > > > >> performed and do not corrupt the filesystem.
> > > > >> To help with this, introduce the assert_fs_integrity() function that
> > > > >> executes the appropriate fsck tool. It should be called at the end of any
> > > > >> test that modify the content/organization of the filesystem.
> > > > >> Currently only supports FATs and EXT4.
> > > > >>
> > > > >> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> > > > >> Reviewed-by: Tom Rini <trini@konsulko.com>
> > > > >
> > > > > OK, I'm adding in a bunch of people to CC here.  The issue with this
> > > > > patch is that by adding fsck to our tests we see 34 FAT16/FAT32
> > > > > failures:
> > > > > TestFsBasic.test_fs13[fat16]
> > > > > TestFsBasic.test_fs11[fat32]
> > > > > TestFsBasic.test_fs12[fat32]
> > > > > TestFsBasic.test_fs13[fat32]
> > > > > TestFsExt.test_fs_ext1[fat32]
> > > > > TestFsExt.test_fs_ext2[fat32]
> > > > > TestFsExt.test_fs_ext3[fat32]
> > > > > TestFsExt.test_fs_ext4[fat32]
> > > > > TestFsExt.test_fs_ext5[fat32]
> > > > > TestFsExt.test_fs_ext6[fat32]
> > > > > TestFsExt.test_fs_ext7[fat32]
> > > > > TestFsExt.test_fs_ext8[fat32]
> > > > > TestFsExt.test_fs_ext9[fat32]
> > > > > TestMkdir.test_mkdir6[fat16]
> > > > > TestMkdir.test_mkdir1[fat32]
> > > > > TestMkdir.test_mkdir2[fat32]
> > > > > TestMkdir.test_mkdir3[fat32]
> > > > > TestMkdir.test_mkdir4[fat32]
> > > > > TestMkdir.test_mkdir5[fat32]
> > > > > TestMkdir.test_mkdir6[fat32]
> > > > > TestUnlink.test_unlink1[fat16]
> > > > > TestUnlink.test_unlink2[fat16]
> > > > > TestUnlink.test_unlink3[fat16]
> > > > > TestUnlink.test_unlink4[fat16]
> > > > > TestUnlink.test_unlink5[fat16]
> > > > > TestUnlink.test_unlink6[fat16]
> > > > > TestUnlink.test_unlink7[fat16]
> > > > > TestUnlink.test_unlink1[fat32]
> > > > > TestUnlink.test_unlink2[fat32]
> > > > > TestUnlink.test_unlink3[fat32]
> > > > > TestUnlink.test_unlink4[fat32]
> > > > > TestUnlink.test_unlink5[fat32]
> > > > > TestUnlink.test_unlink6[fat32]
> > > > > TestUnlink.test_unlink7[fat32]
> > > > 
> > > > I appreciate that we get tests for file system functions.
> > > > 
> > > > Unfortunately the test output is rudimentary. Can we have something more
> > > > expressive than unlink1 - unlink7?
> > > > 
> > > > CCing Takahiro as he was contributing recently to FAT.
> > > 
> > > Sorry, yes, kind of?  I pasted that in for brevity, but it's basically
> > > that for example all of test/py/tests/test_fs/test_unlink.py fails if
> > > you fsck the image in question after each test.  If you apply
> > > https://patchwork.ozlabs.org/patch/1041186/ (to avoid spurious ext4
> > > failures) and then https://patchwork.ozlabs.org/patch/1041181/ and run
> > > 'make tests' you'll see the full output.
> > 
> > I have no time to dig into this issue right now,
> > but if you give me a log from fsck, particularly
> > why fsck failed here, it would help me to understand
> > the problem.
> 
> The raw log can be seen here:
> https://gist.github.com/trini/b68799ed9880a31a3289e9bea033831d

Thanks. I can find error messages like:
Free cluster summary wrong (144636 vs. really 144380)

So there seems to be a leak in reclaiming freed clusters.

> > # like the case of ext4, we might have to turn off
> > # some option at fsck?
> 
> Note that for ext4 we're turning off the metadata csum feature of the
> filesystem as we do not support it.

I'm afraid that this is not the case.

-Takahiro Akashi

> -- 
> Tom

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Takahiro Akashi]

On Wed, Apr 10, 2019 at 11:51:20AM +0900, Takahiro Akashi wrote:
> On Tue, Apr 09, 2019 at 10:25:14PM -0400, Tom Rini wrote:
> > On Wed, Apr 10, 2019 at 10:37:42AM +0900, Takahiro Akashi wrote:
> > > On Tue, Apr 09, 2019 at 08:19:40PM -0400, Tom Rini wrote:
> > > > On Wed, Apr 10, 2019 at 02:10:12AM +0200, Heinrich Schuchardt wrote:
> > > > > On 4/9/19 10:03 PM, Tom Rini wrote:
> > > > > > On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:
> > > > > >
> > > > > >> We need to make sure that file writes,file creation, etc. are properly
> > > > > >> performed and do not corrupt the filesystem.
> > > > > >> To help with this, introduce the assert_fs_integrity() function that
> > > > > >> executes the appropriate fsck tool. It should be called at the end of any
> > > > > >> test that modify the content/organization of the filesystem.
> > > > > >> Currently only supports FATs and EXT4.
> > > > > >>
> > > > > >> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> > > > > >> Reviewed-by: Tom Rini <trini@konsulko.com>
> > > > > >
> > > > > > OK, I'm adding in a bunch of people to CC here.  The issue with this
> > > > > > patch is that by adding fsck to our tests we see 34 FAT16/FAT32
> > > > > > failures:
> > > > > > TestFsBasic.test_fs13[fat16]
> > > > > > TestFsBasic.test_fs11[fat32]
> > > > > > TestFsBasic.test_fs12[fat32]
> > > > > > TestFsBasic.test_fs13[fat32]
> > > > > > TestFsExt.test_fs_ext1[fat32]
> > > > > > TestFsExt.test_fs_ext2[fat32]
> > > > > > TestFsExt.test_fs_ext3[fat32]
> > > > > > TestFsExt.test_fs_ext4[fat32]
> > > > > > TestFsExt.test_fs_ext5[fat32]
> > > > > > TestFsExt.test_fs_ext6[fat32]
> > > > > > TestFsExt.test_fs_ext7[fat32]
> > > > > > TestFsExt.test_fs_ext8[fat32]
> > > > > > TestFsExt.test_fs_ext9[fat32]
> > > > > > TestMkdir.test_mkdir6[fat16]
> > > > > > TestMkdir.test_mkdir1[fat32]
> > > > > > TestMkdir.test_mkdir2[fat32]
> > > > > > TestMkdir.test_mkdir3[fat32]
> > > > > > TestMkdir.test_mkdir4[fat32]
> > > > > > TestMkdir.test_mkdir5[fat32]
> > > > > > TestMkdir.test_mkdir6[fat32]
> > > > > > TestUnlink.test_unlink1[fat16]
> > > > > > TestUnlink.test_unlink2[fat16]
> > > > > > TestUnlink.test_unlink3[fat16]
> > > > > > TestUnlink.test_unlink4[fat16]
> > > > > > TestUnlink.test_unlink5[fat16]
> > > > > > TestUnlink.test_unlink6[fat16]
> > > > > > TestUnlink.test_unlink7[fat16]
> > > > > > TestUnlink.test_unlink1[fat32]
> > > > > > TestUnlink.test_unlink2[fat32]
> > > > > > TestUnlink.test_unlink3[fat32]
> > > > > > TestUnlink.test_unlink4[fat32]
> > > > > > TestUnlink.test_unlink5[fat32]
> > > > > > TestUnlink.test_unlink6[fat32]
> > > > > > TestUnlink.test_unlink7[fat32]
> > > > > 
> > > > > I appreciate that we get tests for file system functions.
> > > > > 
> > > > > Unfortunately the test output is rudimentary. Can we have something more
> > > > > expressive than unlink1 - unlink7?
> > > > > 
> > > > > CCing Takahiro as he was contributing recently to FAT.
> > > > 
> > > > Sorry, yes, kind of?  I pasted that in for brevity, but it's basically
> > > > that for example all of test/py/tests/test_fs/test_unlink.py fails if
> > > > you fsck the image in question after each test.  If you apply
> > > > https://patchwork.ozlabs.org/patch/1041186/ (to avoid spurious ext4
> > > > failures) and then https://patchwork.ozlabs.org/patch/1041181/ and run
> > > > 'make tests' you'll see the full output.
> > > 
> > > I have no time to dig into this issue right now,
> > > but if you give me a log from fsck, particularly
> > > why fsck failed here, it would help me to understand
> > > the problem.
> > 
> > The raw log can be seen here:
> > https://gist.github.com/trini/b68799ed9880a31a3289e9bea033831d
> 
> Thanks. I can find error messages like:
> Free cluster summary wrong (144636 vs. really 144380)
> 
> So there seems to be a leak in reclaiming freed clusters.

A count of free clusters, along with other info, is kept in an "info sector"
on a file system (only for fat32), but in U-Boot fat, none of information
in that sector is currently maintained. So this error would be inevitable.
I don't know any fsck option to suppress this kind of check.

-Takahiro Akashi

> > > # like the case of ext4, we might have to turn off
> > > # some option at fsck?
> > 
> > Note that for ext4 we're turning off the metadata csum feature of the
> > filesystem as we do not support it.
> 
> I'm afraid that this is not the case.
> 
> -Takahiro Akashi
> 
> > -- 
> > Tom
> 
> 

[U-Boot] [U-Boot, RESEND, v5, 1/7] fs: ext4: do not allow writes if metadata checksum is active

[Jean-Jacques Hiblot]

On 09/04/2019 21:34, Tom Rini wrote:
> On Wed, Feb 13, 2019 at 12:15:21PM +0100, Jean-Jacques Hiblot wrote:
>
>> u-boot does not supports updating the metadata chacksums
>>
>> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
>> Reviewed-by: Tom Rini <trini@konsulko.com>
> Deferred as I picked up
> commit 2e7365518acdb8fb6e9be332c8a6c57b457188d9
> Author: Sébastien Szymanski <sebastien.szymanski@armadeus.com>
> Date:   Fri Mar 22 09:33:52 2019 +0100
>
>      fs: ext4: do not write on filesystem with metadata_csum feature
>      
> as it slipped my mind you had posted this patch in this series, sorry!

That is totally ok


>

[U-Boot] [U-Boot, RESEND, v5, 2/7] test: fs: disable the metadata checksums on ext4 filesystems

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:22PM +0100, Jean-Jacques Hiblot wrote:

> If the metadata checksums are enabled, all write operations will fail.
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190410/89ee09c9/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 3/7] test: fs: Add filesystem integrity checks

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:23PM +0100, Jean-Jacques Hiblot wrote:

> We need to make sure that file writes,file creation, etc. are properly
> performed and do not corrupt the filesystem.
> To help with this, introduce the assert_fs_integrity() function that
> executes the appropriate fsck tool. It should be called at the end of any
> test that modify the content/organization of the filesystem.
> Currently only supports FATs and EXT4.
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190410/87e752cd/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 4/7] fs: ext4: constify the buffer passed to write functions

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:24PM +0100, Jean-Jacques Hiblot wrote:

> There is no need to modify the buffer passed to ext4fs_write_file().
> The memset() call is not required here and was likely copied from the
> equivalent part of the ext4fs_read_file() function where we do need it.
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190410/2a02caa7/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 5/7] fs: ext4: Add support for the creation of symbolic links

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:25PM +0100, Jean-Jacques Hiblot wrote:

> Re-use the functions used to write/create a file, to support creation of a
> symbolic link.
> The difference with a regular file are small:
> - The inode mode is flagged with S_IFLNK instead of S_IFREG
> - The ext2_dirent's filetype is FILETYPE_SYMLINK instead of FILETYPE_REG
> - Instead of storing the content of a file in allocated blocks, the path
> to the target is stored. And if the target's path is short enough, no block
> is allocated and the target's path is stored in ext2_inode.b.symlink
> 
> As with regulars files, if a file/symlink with the same name exits, it is
> unlinked first and then re-created.
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190410/6782f67b/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 6/7] fs: Add a new command to create symbolic links

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:26PM +0100, Jean-Jacques Hiblot wrote:

> The command line is:
> ln <interface> <dev[:part]> target linkname
> 
> Currently symbolic links are supported only in ext4 and only if the option
> CMD_EXT4_WRITE is enabled.
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190410/ad59eb94/attachment.sig>

[U-Boot] [U-Boot, RESEND, v5, 7/7] test: fs: Added tests for symlinks

[Tom Rini]

On Wed, Feb 13, 2019 at 12:15:27PM +0100, Jean-Jacques Hiblot wrote:

> Test cases are:
> 1) basic link creation, verify it can be followed
> 2) chained links, verify it can be followed
> 3) replace exiting file a with a link, and a link with a link. verify it
>    can be followed
> 4) create a broken link, verify it can't be followed
> 
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190410/56ba827b/attachment.sig>