[PATCH 0/5] btrfs-progs: check: simple errors repair in lowmem

[PATCH 0/5] btrfs-progs: check: simple errors repair in lowmem

[Su Yue]

The series include following contents:

1) Repair wrong nbytes of file inode item.
   After traversal of extents in one file, we should get the actual nbytes
   of the file. If nbytes in the file inode differs from the actual value,
   set the value to actual one.
   
   The wrong nbytes of file inode case corresponds to fsck-test/016.
   
2) Repair wrong iszie of directory inode item.
   After traversal of dir_index and dir_item in one dir, we should get
   the actual isize of the dirctory. If size in the inode item differs
   from the actual value, set the value to actual one.

   New test case 'fsck-test/026' is added for the case wrong isize of
   directory inode.
   
3) Allow fsck check test to repair in lowmem mode for certain test cases
   if TEST_ENABLE_OVERRIDE=true.

Qu Wenruo (1):
  btrfs-progs: fsck-check: Allow fsck check test to repair in lowmem
    mode for certain test cases

Su Yue (4):
  btrfs-progs: check: inode nbytes fix in lowmem
  btrfs-progs: check: dir isize fix in lowmem
  btrfs-progs: check: enable lowmem repair
  btrfs-progs: fsck-check: test cases for nbytes and dir isize

 cmds-check.c                                       | 149 +++++++++++++++++++--
 tests/common.local                                 |  14 +-
 .../016-wrong-inode-nbytes/.lowmem_repairable      |   0
 .../026-wrong-dir-inode-isize/.lowmem_repairable   |   0
 .../026-wrong-dir-inode-isize/default_case.img     | Bin 0 -> 4096 bytes
 5 files changed, 153 insertions(+), 10 deletions(-)
 create mode 100644 tests/fsck-tests/016-wrong-inode-nbytes/.lowmem_repairable
 create mode 100644 tests/fsck-tests/026-wrong-dir-inode-isize/.lowmem_repairable
 create mode 100644 tests/fsck-tests/026-wrong-dir-inode-isize/default_case.img

-- 
2.12.0

[PATCH 1/5] btrfs-progs: check: inode nbytes fix in lowmem

[Su Yue]

After checking one inode item, we should get the actual nbytes of the
inode item.
Introduce function 'repair_inode_nbytes_lowmem' to set nbytes in struct
btrfs_inode_item to the actual nbytes. After call of the function, the
wrong nbytes should have been corrected.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 cmds-check.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 73 insertions(+), 3 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 17b7efbf..2bdf6a20 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -1907,6 +1907,9 @@ static int process_one_leaf_v2(struct btrfs_root *root, struct btrfs_path *path,
 again:
 	err |= check_inode_item(root, path, ext_ref);
 
+	/* modified cur since check_inode_item may change path */
+	cur = path->nodes[0];
+
 	if (err & LAST_ITEM)
 		goto out;
 
@@ -2255,6 +2258,7 @@ static int walk_down_tree_v2(struct btrfs_root *root, struct btrfs_path *path,
 			}
 			ret = process_one_leaf_v2(root, path, nrefs,
 						  level, ext_ref);
+			cur = path->nodes[*level];
 			break;
 		} else {
 			ret = btrfs_check_node(root, NULL, cur);
@@ -4815,10 +4819,69 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_key *fkey,
 }
 
 /*
+ * Set inode item nbytes to @nbytes
+ *
+ * Returns <0  means on error
+ * Returns  0  means successful repair
+ */
+static int repair_inode_nbytes_lowmem(struct btrfs_root *root,
+				      struct btrfs_path *path,
+				      u64 ino, u64 nbytes)
+{
+	struct btrfs_trans_handle *trans;
+	struct btrfs_inode_item *ii;
+	struct btrfs_key key;
+	struct btrfs_key research_key;
+	int ret;
+	int ret2;
+
+	key.objectid = ino;
+	key.type = BTRFS_INODE_ITEM_KEY;
+	key.offset = 0;
+	btrfs_item_key_to_cpu(path->nodes[0], &research_key, path->slots[0]);
+	btrfs_release_path(path);
+
+	trans = btrfs_start_transaction(root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		goto out;
+	}
+
+	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
+	if (ret < 0)
+		goto out;
+	if (ret > 0) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	ii = btrfs_item_ptr(path->nodes[0], path->slots[0],
+			    struct btrfs_inode_item);
+	btrfs_set_inode_nbytes(path->nodes[0], ii, nbytes);
+	btrfs_mark_buffer_dirty(path->nodes[0]);
+
+	printf("reset nbytes for inode %llu root %llu\n", ino,
+	       root->root_key.objectid);
+
+	btrfs_commit_transaction(trans, root);
+out:
+	if (ret < 0)
+		error("failed to reset nbytes for inode %llu root %llu due to %s",
+		      ino, root->root_key.objectid, strerror(-ret));
+
+	/* research path */
+	btrfs_release_path(path);
+	ret2 = btrfs_search_slot(NULL, root, &research_key, path, 0, 0);
+	return ret2 < 0 ? ret2 : ret;
+}
+
+/*
  * Check INODE_ITEM and related ITEMs (the same inode number)
  * 1. check link count
  * 2. check inode ref/extref
  * 3. check dir item/index
+ * Be Careful, if repair is enable, @path may be changed.
+ * Remember to reassign any context about @path in repair mode.
  *
  * @ext_ref:	the EXTENDED_IREF feature
  *
@@ -4968,9 +5031,16 @@ out:
 		}
 
 		if (nbytes != extent_size) {
-			err |= NBYTES_ERROR;
-			error("root %llu INODE[%llu] nbytes(%llu) not equal to extent_size(%llu)",
-			      root->objectid, inode_id, nbytes, extent_size);
+			if (repair) {
+				ret = repair_inode_nbytes_lowmem(root, path,
+							inode_id, extent_size);
+			}
+			if (!repair || ret) {
+				err |= NBYTES_ERROR;
+				error("root %llu INODE[%llu] nbytes(%llu) not equal to extent_size(%llu)",
+				      root->objectid, inode_id, nbytes,
+				      extent_size);
+			}
 		}
 	}
 
-- 
2.12.0

[PATCH 2/5] btrfs-progs: check: dir isize fix in lowmem

[Su Yue]

After traversal of whole directory, we should get the actual isize.

Introduce function 'repair_dir_isize_lowmem' to set isize in the directory
inode item to actual size. After call of the function, the wrong dir isize
should have been corrected.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 cmds-check.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 65 insertions(+), 3 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 2bdf6a20..6ab18b26 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -4876,6 +4876,62 @@ out:
 }
 
 /*
+ * Set dir isize to @isize
+ *
+ * Returns <0  means on error
+ * Returns  0  means successful repair
+ */
+static int repair_dir_isize_lowmem(struct btrfs_root *root,
+				   struct btrfs_path *path,
+				   u64 ino, u64 isize)
+{
+	struct btrfs_trans_handle *trans;
+	struct btrfs_inode_item *ii;
+	struct btrfs_key key;
+	struct btrfs_key research_key;
+	int ret;
+	int ret2;
+
+	key.objectid = ino;
+	key.type = BTRFS_INODE_ITEM_KEY;
+	key.offset = 0;
+
+	btrfs_item_key_to_cpu(path->nodes[0], &research_key, path->slots[0]);
+	btrfs_release_path(path);
+
+	trans = btrfs_start_transaction(root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		goto out;
+	}
+
+	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
+	if (ret < 0)
+		goto out;
+	if (ret > 0) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	ii = btrfs_item_ptr(path->nodes[0], path->slots[0],
+			    struct btrfs_inode_item);
+	btrfs_set_inode_size(path->nodes[0], ii, isize);
+	btrfs_mark_buffer_dirty(path->nodes[0]);
+
+	printf("reset isize for inode %llu root %llu\n", ino,
+	       root->root_key.objectid);
+
+	btrfs_commit_transaction(trans, root);
+out:
+	if (ret < 0)
+		error("failed to reset isize for inode %llu root %llu due to %s",
+		      ino, root->root_key.objectid, strerror(-ret));
+	btrfs_release_path(path);
+	ret2 = btrfs_search_slot(NULL, root, &research_key, path, 0, 0);
+	return ret2 < 0 ? ret2 : ret;
+}
+
+/*
  * Check INODE_ITEM and related ITEMs (the same inode number)
  * 1. check link count
  * 2. check inode ref/extref
@@ -5011,9 +5067,15 @@ out:
 		}
 
 		if (isize != size) {
-			err |= ISIZE_ERROR;
-			error("root %llu DIR INODE [%llu] size(%llu) not equal to %llu",
-			      root->objectid, inode_id, isize, size);
+			if (repair)
+				ret = repair_dir_isize_lowmem(root, path,
+							      inode_id, size);
+
+			if (!repair || ret) {
+				err |= ISIZE_ERROR;
+				error("root %llu DIR INODE [%llu] size(%llu) not equal to %llu",
+				      root->objectid, inode_id, isize, size);
+			}
 		}
 	} else {
 		if (nlink != refs) {
-- 
2.12.0

[PATCH 3/5] btrfs-progs: check: enable lowmem repair

[Su Yue]

Enable btrfsck option '--repair' with option '--mode=lowmem'.
Now lowmem mode only repairs wrong nbytes, dir isize.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 cmds-check.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 6ab18b26..da5fe75f 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -12925,11 +12925,10 @@ int cmd_check(int argc, char **argv)
 	}
 
 	/*
-	 * Not supported yet
+	 * Support partially
 	 */
 	if (repair && check_mode == CHECK_MODE_LOWMEM) {
-		error("low memory mode doesn't support repair yet");
-		exit(1);
+		warning("low memory mode support repair partially");
 	}
 
 	radix_tree_init();
-- 
2.12.0

[PATCH 4/5] btrfs-progs: fsck-check: Allow fsck check test to repair in lowmem mode for certain test cases

[Su Yue]

From: Qu Wenruo <quwenruo@cn.fujitsu.com>

Since lowmem mode can repair certain corruptions (mostly in fs tree),
insert a beacon into each fsck test cases to allow some of them be
tested for lowmem mode.

With this patch, fsck option override will check the beacon file
".lowmem_repairbale" in the same directory of the test image, and if the
beacon exists, then it will also run lowmem mode repair to repair the
image.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/common.local | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/tests/common.local b/tests/common.local
index 4f56bb08..af372f16 100644
--- a/tests/common.local
+++ b/tests/common.local
@@ -15,11 +15,23 @@ TEST_ARGS_CHECK=--mode=lowmem
 # gets arguments of a current command and can decide if the argument insertion
 # should happen, eg. if some option combination does not make sense or would
 # break tests
+#
+# Return 0 if we need to skip option override
+# Return 1 if we don't need to skip option override
 _skip_spec()
 {
+	beacon=.lowmem_repairable
+
+	# For loemem repair, only support fs tree repair yet
+	# So we place lowmem repair beacon in the same dir of the
+	# test case
 	if echo "$TEST_ARGS_CHECK" | grep -q 'mode=lowmem' &&
 	   echo "$@" | grep -q -- '--repair'; then
-		return 0
+		dir="$(dirname ${@: -1})"
+		if [ -f ${dir}/${beacon} ]; then
+			return 1;
+		fi
+		return 0;
 	fi
 	return 1
 }
-- 
2.12.0

[PATCH 5/5] btrfs-progs: fsck-check: test cases for nbytes and dir isize

[Su Yue]

Create test case '026-wrong-dir-inode-isize'.
Create becon files '.lowmem_repairable' under tests/fsck-test/016 and 026.

Now 'make test-fsck' will test lowmem repairable test cases if
TEST_ENABLE_OVERRIDE=true.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 .../fsck-tests/016-wrong-inode-nbytes/.lowmem_repairable |   0
 .../026-wrong-dir-inode-isize/.lowmem_repairable         |   0
 .../026-wrong-dir-inode-isize/default_case.img           | Bin 0 -> 4096 bytes
 3 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 tests/fsck-tests/016-wrong-inode-nbytes/.lowmem_repairable
 create mode 100644 tests/fsck-tests/026-wrong-dir-inode-isize/.lowmem_repairable
 create mode 100644 tests/fsck-tests/026-wrong-dir-inode-isize/default_case.img

diff --git a/tests/fsck-tests/016-wrong-inode-nbytes/.lowmem_repairable b/tests/fsck-tests/016-wrong-inode-nbytes/.lowmem_repairable
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/fsck-tests/026-wrong-dir-inode-isize/.lowmem_repairable b/tests/fsck-tests/026-wrong-dir-inode-isize/.lowmem_repairable
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/fsck-tests/026-wrong-dir-inode-isize/default_case.img b/tests/fsck-tests/026-wrong-dir-inode-isize/default_case.img
new file mode 100644
index 0000000000000000000000000000000000000000..a060cca86e945f2659595d812b3017c66c570d5e
GIT binary patch
literal 4096
zcmeHHX*d*I8)g_@X;2Lby@Ra9kg`N}hBQhtuf2#!WM`~nnW8LZ8=4OjNm<8M%Gk5C
zVH*27BfA*OSf(N7lkff2*Prj-cU^DiI@kT&&vUMG&U4?_xlX(b#&<%}uAb%Z!MPV%
z_van~?=cn~ogc^|zK>1&$ix1l-n`ETe=J76-RI<e-m=dh{@{wOKaj;{zp_t%!*2(E
zJMjPGK+te#-m^?%B&Tti8UFhFsg{Z)QYq5ZR_Q1k+X*WXgGv^kx>O@G8}B*=2wcB~
z^*(@;6D9)F0?a!pI<WZ-=0tn90YvM~4mxWC%(K)Eniy2GHU|bqf95guW^)u&Ft(#5
zFa$C$f5Cqm=xXRCJ15L<B!hxTbB8!{E-Z5CwX(vMatj^wE`2iVxSwKBA+09C?SOs3
zS`0Li34@gb;W7d`>v^`3N!pPJoFh0P`ikNKShI{EaE@j8<#d!~Ws;2m`Fwy*s;k=F
zk6t>HkjRxG@3V==J_OD=!qNhTVl+roTs2j!_L`W{V3r$D4V&V7z%K3Rle__{=j%j7
zmz*U9ad2?V?B0<rU=|i`AM>X{!_<*_6E<DGrd6zgUawT?vy0|!-FiUnbawG5(XBs7
zA1^gut0-S%Es<~#yM=kHC$|JN2TIt)C%KlB7RMx!vms)J1SapMXP~Pq2flInHpAK+
z4EcC88-9vHYVYjXSWOYk%k$glA)v?&BWk0RQ2S()T_5-0(*>V6mhGl0559Hqvz(Gi
zVH|u1LK^T)a5<qz8o7uZrbhQbl-Gurj_|9T=Q{H1SEOj^vi?5d^)8W!8Qewkf7ei#
zt&*f}mn|CM12(%O7C@G@?#W@Dn4>k|fx7PX;D2cmd#q;)op9}4n%I2G+g{Yn6IA<<
zSDga<uevA3Tl9i~=o-rpTJtH#Q<-Z_7P-VbihB-?UA$$75NulY^OD9sz`~1VY);*9
zD&c*}ZnE`RBHL3=?@%0q;2xhS>T*SS0_Gp5f{zomAG!x(_aGUbG(MD{{XBrA$4!MD
z3Xp6QH(ZGVHivjEGcGOdbd5U0r4Gk3z5E~)Mwez;$I_G>ghE4O+lun&jmn2SPnoh`
z{`0%@cZO|3A?ND0C>N)2`)1f1pY0yoMmx7H69{jhrN_0{`oK(ox5AAwN2iC+$Z69v
z8DG9RN|jHJaW3N$yr2eAWdCuxs<c+TmsT9W-8^b6cc+~dQxS><Emj_j^6V?jeTLWz
zt1@O6fH29)L>Wasdwu&pB>Dm(7ymU3{HF8Q_TOS1(zjD`I#KqJI=nq`>gF~KZ}}WD
zzdg~l1DWd(4T!iCf8Eb0RbFkom8dbf&=rqvb6L`mtkCXBC(xC4<U$L$RcmDtL5Pi6
zq>pdJ?s#WDJyW09K0pd+#Cyz;N5bs`5g544k?A}1z_AB}W3Bl%h)3sRT&Tl0i;8L`
ztagx}PI&k(E>%tX&aZ4}(9#VE-)s8DVqJ@gBh2e}JeiFzk->yYg8w$PQ|T~*tJjlU
zHwE|r&^+nxu`}cRqMqHZy*`B$(eGl=!&j>=cdSn|)yHkn#Ui!ov>Vw1Y?a-M1J%6%
z1h{=$z@NmdW;yp-IOqWSdWM>)P<<duIHk#7#d`?Y<;?a@^4A{afZltq{~3xI8@+0X
z$SL6lTSW2L=Un8MnFB<Sjg49B^TJM=DF*>ZuQ5Ubi%}H22gi$aGN9wTXkBI7N>$g<
zDMpXZBaf|$#-s=Hy8&N6+MPkSSEZw5rr@Ri^Vo1pVrTW?x*g`LnN6lEOiyd`Vei&t
z(rM{vY=2FY6Z(P<!79Cras=pUS}L7!niiCjYxE)P)rZF?3cvUv!W(3*U(@O)Y3qgP
z)y`a*6&-*1uU=t|VhmUYPl@#t_Vr%uYSh#u&BX#R!kNM}S&6lu_QYs`>%-LuA8ANR
zP;mAR$A1eSjv+%QsJ6Gw@T9`L`1~AUhs!Nz&rv%TGVplQna+T#o5>i98XSi`?-aX5
z#Q6hQ3GJFRjrXs5Pd5}VDrXtJgAPx+meo;)HX^b`mha9CM+=l%AWO<uM?OB7xXOtN
z@uw~GOS7TeZHnJzR-fU<R^wC{g+9WvULA`DFXN;}HE3^f12AEnGu8T8aKmnZZ_0)T
zUH)3^t3|b!f3>;mgo`bdD-m7NX7A4)2R97Hwc~5@cS9#YIhq-vxHT)!R7z^T)*UkC
zB4UR@<z~ifH|Ebk6ubm?|MOo)UlC*(_iH$Cwl?Zk?@(V1pRI3lPA)f}S94q8^XlqN
zUvAoA_W1)sLc?S7s^}a8i*MpvV4pDD{LZ;q%@!<0cQ!yoNkmhijBw$^K%(pG`pkY*
zf#y4F&90G(Zcj4gde*>pY;Ibpy6hr!$rLq{lB89iwS@qE5ks7_;(+v71Ag+{Htnu&
z@NA<C6)ALeb<TfHRhzN|wQO^1S*B-{1V?N#%c|&1x&WQl0QqTWzX}jHu4K8di>KKE
z+`ihQbaoIfF2#|TsC&!U%0L{iV+W{}I*e{rHdf`z?ECz9>|Sv>DMWbdkk}0xQyv%8
z6&$<t;s!XKriZ$kH55WDf7NySwcv|Z_1jC#;rjlf9uh%i4#|5?_Si^>dfsNr*MT6^
z;?rv9Zm_KOwd7Q7QUgE6!7)TkSYAXSOPB;wSkAjQIUU>)vlQb9C;94M;s2@Uzta`Y
z!>E-<WqlRZ1#BxT*$n#HbCdM%F63lh3e#Kb%~4zgz4kgQG)X^xPu-`+1G+wf9OIM<
zHHIIn(u7c!?!26wIaF8Rr~^{!W>VktDp_cA&S<JqGP(~GgxlKs8s6xI^Y32KU;At`
zlYb@42;J$XTzq#?sGc@wZ#35C{{-IN;&<ZZq*Iv<^E*bn_D!uOxsnfRNT9}=asSlv
z3bB7aoSrTHqTqPJUN;>TAhaSM(9E_MjE$6+iSmG%sJvGMUs6J-OL&++1}4#P{*_Bl
zmd%k18QXM5SznBP-1eDiXpBWP<7izGq^I^8Bh4Z4+#ipmi0!#0Sr7YFf4cCWm5u{c
R>`I?{{<B2?R{Os>@NZR9TeJWG

literal 0
HcmV?d00001

-- 
2.12.0

Re: [PATCH 0/5] btrfs-progs: check: simple errors repair in lowmem

[Su Yue]

Hi
    This patchset was sent a month ago.
    Since it's a start of lowmem repair patches, I hope to get some
    opinions about it.

Thanks
Su Yue

On 04/11/2017 11:26 AM, Su Yue wrote:
> The series include following contents:
> 
> 1) Repair wrong nbytes of file inode item.
>     After traversal of extents in one file, we should get the actual nbytes
>     of the file. If nbytes in the file inode differs from the actual value,
>     set the value to actual one.
>     
>     The wrong nbytes of file inode case corresponds to fsck-test/016.
>     
> 2) Repair wrong iszie of directory inode item.
>     After traversal of dir_index and dir_item in one dir, we should get
>     the actual isize of the dirctory. If size in the inode item differs
>     from the actual value, set the value to actual one.
> 
>     New test case 'fsck-test/026' is added for the case wrong isize of
>     directory inode.
>     
> 3) Allow fsck check test to repair in lowmem mode for certain test cases
>     if TEST_ENABLE_OVERRIDE=true.
> 
> Qu Wenruo (1):
>    btrfs-progs: fsck-check: Allow fsck check test to repair in lowmem
>      mode for certain test cases
> 
> Su Yue (4):
>    btrfs-progs: check: inode nbytes fix in lowmem
>    btrfs-progs: check: dir isize fix in lowmem
>    btrfs-progs: check: enable lowmem repair
>    btrfs-progs: fsck-check: test cases for nbytes and dir isize
> 
>   cmds-check.c                                       | 149 +++++++++++++++++++--
>   tests/common.local                                 |  14 +-
>   .../016-wrong-inode-nbytes/.lowmem_repairable      |   0
>   .../026-wrong-dir-inode-isize/.lowmem_repairable   |   0
>   .../026-wrong-dir-inode-isize/default_case.img     | Bin 0 -> 4096 bytes
>   5 files changed, 153 insertions(+), 10 deletions(-)
>   create mode 100644 tests/fsck-tests/016-wrong-inode-nbytes/.lowmem_repairable
>   create mode 100644 tests/fsck-tests/026-wrong-dir-inode-isize/.lowmem_repairable
>   create mode 100644 tests/fsck-tests/026-wrong-dir-inode-isize/default_case.img
> 

Re: [PATCH 0/5] btrfs-progs: check: simple errors repair in lowmem

[David Sterba]

On Wed, May 17, 2017 at 04:25:03PM +0800, Su Yue wrote:
>     This patchset was sent a month ago.
>     Since it's a start of lowmem repair patches, I hope to get some
>     opinions about it.

This is unfortunate, it may take more time before I get back to this.