[PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[Naohiro Aota]

This series address an issue of btrfsck to restore infinite number of
same file into `lost+found' directory. The issue occur on a file which
is linked from two different directory A and B. If links from dir A is
corrupted and links from dir B is kept valid, btrfsck won't stop
creating a file in lost+found like this:

-----
Moving file 'file.del.51' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 1876
Trying to rebuild inode:1877
Moving file 'del' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 1877
Can't get file name for inode 1876, using '1876' as fallback
Moving file '1876' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 1876
Can't get file name for inode 1876, using '1876' as fallback
Moving file '1876.1876' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 1876
(snip)
Moving file '1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 1876
Can't get file name for inode 1876, using '1876' as fallback
Can't get file name for inode 1876, using '1876' as fallback
Can't get file name for inode 1876, using '1876' as fallback
-----

The problem is early release of inode backrefs. The release prevents
`reset_nlink()' to add back valid backrefs to an inode. In the result,
the following results occur:

0. btrfsck scan a FS tree
1. It finds valid links and invalid links (some links are lost)
2. All valid links are released
3. btrfsck detects found_links != nlink
4. reset_nlink() reset nlink to 0
5. No valid links are restored (thus still nlink = 0)
6. The file is restored to lost+found since nlink == 0 (now, nlink = 1)
7. btrfsck rescan the FS tree
8. It finds `found_links' = #valid_links+1 (in lost+found) and nlink = 1
9. again all valid links are lost, and restore to lost+found

The first patch add clean up code to the test. It umount test
directory on failure path. The second patch fix the above problem. And
the last patch extend the test to check a case of multiple-linked file
corruption.

[PATCH 1/3] btrfs-progs: test: umount if confirmation failed

[Naohiro Aota]

When a check in check_inode() failed, the test should umount test target
file system. This commit add clean up umount line in failure path.

Signed-off-by: Naohiro Aota <naota@elisp.net>
---
 tests/fsck-tests/012-leaf-corruption/test.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh b/tests/fsck-tests/012-leaf-corruption/test.sh
index 6e23145..bfdd0ea 100755
--- a/tests/fsck-tests/012-leaf-corruption/test.sh
+++ b/tests/fsck-tests/012-leaf-corruption/test.sh
@@ -57,6 +57,7 @@ check_inode()
 	# Check whether the inode exists
 	exists=$($SUDO_HELPER find $path -inum $ino)
 	if [ -z "$exists" ]; then
+		$SUDO_HELPER umount $TEST_MNT
 		_fail "inode $ino not recovered correctly"
 	fi
 
@@ -64,17 +65,20 @@ check_inode()
 	found_mode=$(printf "%o" 0x$($SUDO_HELPER stat $exists -c %f))
 	if [ $found_mode -ne $mode ]; then
 		echo "$found_mode"
+		$SUDO_HELPER umount $TEST_MNT
 		_fail "inode $ino modes not recovered"
 	fi
 
 	# Check inode size
 	found_size=$($SUDO_HELPER stat $exists -c %s)
 	if [ $mode -ne 41700 -a $found_size -ne $size ]; then
+		$SUDO_HELPER umount $TEST_MNT
 		_fail "inode $ino size not recovered correctly"
 	fi
 
 	# Check inode name
 	if [ "$(basename $exists)" != "$name" ]; then
+		$SUDO_HELPER umount $TEST_MNT
 		_fail "inode $ino name not recovered correctly"
 	else
 		return 0
-- 
2.6.3

Re: [PATCH 1/3] btrfs-progs: test: umount if confirmation failed

[David Sterba]

On Fri, Dec 04, 2015 at 02:37:25PM +0900, Naohiro Aota wrote:
> When a check in check_inode() failed, the test should umount test target
> file system. This commit add clean up umount line in failure path.

The lack of cleanup after failed tests is intentional (exceptions
possible). The tests are supposed to do cleanup if they pass but if they
fail the test environment could be examined. In case of more complex
setups (loop devices, dm devices) it's some work to set it up again.

We could add an optional per-test script cleanup.sh that does the
cleanup on demand (and is also run from tests/clean-tests.sh).

Re: [PATCH 1/3] btrfs-progs: test: umount if confirmation failed

[Naohiro Aota]

On Tue, Dec 8, 2015 at 12:33 AM, David Sterba <dsterba@suse.cz> wrote:
> On Fri, Dec 04, 2015 at 02:37:25PM +0900, Naohiro Aota wrote:
>> When a check in check_inode() failed, the test should umount test target
>> file system. This commit add clean up umount line in failure path.
>
> The lack of cleanup after failed tests is intentional (exceptions
> possible). The tests are supposed to do cleanup if they pass but if they
> fail the test environment could be examined. In case of more complex
> setups (loop devices, dm devices) it's some work to set it up again.
>
> We could add an optional per-test script cleanup.sh that does the
> cleanup on demand (and is also run from tests/clean-tests.sh).

I see.
Well, I overlooked the README.md stating about clean-tests.sh...

[PATCH 2/3] btrfs-progs: properly reset nlink of multi-linked file

[Naohiro Aota]

If a file is linked from more than one directory and only one
of the links is corrupted, btrfs check dose not reset the nlink
properly. Actually it can go into infinite loop to link the broken file
into lost+found.

This patch fix two part of the code. The first one delay the freeing
valid (no error, found inode ref, directory index, and directory
item) backrefs. Freeing valid backrefs earier prevent reset_nlink() to
add back all valid links.

The second fix is obvious: passing `ref_type' to btrfs_add_link() is just
wrong. It should be `filetype' instead. The current code can break all valid
file links.

Signed-off-by: Naohiro Aota <naota@elisp.net>
---
 cmds-check.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 6a0b50a..11ff3fe 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -810,7 +810,8 @@ static void maybe_free_inode_rec(struct cache_tree *inode_cache,
 		if (backref->found_dir_item && backref->found_dir_index) {
 			if (backref->filetype != filetype)
 				backref->errors |= REF_ERR_FILETYPE_UNMATCH;
-			if (!backref->errors && backref->found_inode_ref) {
+			if (!backref->errors && backref->found_inode_ref &&
+			    rec->nlink == rec->found_link) {
 				list_del(&backref->list);
 				free(backref);
 			}
@@ -2392,7 +2393,7 @@ static int reset_nlink(struct btrfs_trans_handle *trans,
 	list_for_each_entry(backref, &rec->backrefs, list) {
 		ret = btrfs_add_link(trans, root, rec->ino, backref->dir,
 				     backref->name, backref->namelen,
-				     backref->ref_type, &backref->index, 1);
+				     backref->filetype, &backref->index, 1);
 		if (ret < 0)
 			goto out;
 	}
-- 
2.6.3

Re: [PATCH 2/3] btrfs-progs: properly reset nlink of multi-linked file

[Qu Wenruo]

On 12/04/2015 01:37 PM, Naohiro Aota wrote:
> If a file is linked from more than one directory and only one
> of the links is corrupted, btrfs check dose not reset the nlink
> properly. Actually it can go into infinite loop to link the broken file
> into lost+found.
>
> This patch fix two part of the code. The first one delay the freeing
> valid (no error, found inode ref, directory index, and directory
> item) backrefs. Freeing valid backrefs earier prevent reset_nlink() to
> add back all valid links.
>
> The second fix is obvious: passing `ref_type' to btrfs_add_link() is just
> wrong. It should be `filetype' instead. The current code can break all valid
> file links.
>
> Signed-off-by: Naohiro Aota <naota@elisp.net>

Thanks for the fix, that's truly my fault.
The fix looks good.

Reviewed-by: Qu Wenruo <quwenruo@cn.fujitsu.com>

Thanks,
Qu

> ---
>   cmds-check.c | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/cmds-check.c b/cmds-check.c
> index 6a0b50a..11ff3fe 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -810,7 +810,8 @@ static void maybe_free_inode_rec(struct cache_tree *inode_cache,
>   		if (backref->found_dir_item && backref->found_dir_index) {
>   			if (backref->filetype != filetype)
>   				backref->errors |= REF_ERR_FILETYPE_UNMATCH;
> -			if (!backref->errors && backref->found_inode_ref) {
> +			if (!backref->errors && backref->found_inode_ref &&
> +			    rec->nlink == rec->found_link) {
>   				list_del(&backref->list);
>   				free(backref);
>   			}
> @@ -2392,7 +2393,7 @@ static int reset_nlink(struct btrfs_trans_handle *trans,
>   	list_for_each_entry(backref, &rec->backrefs, list) {
>   		ret = btrfs_add_link(trans, root, rec->ino, backref->dir,
>   				     backref->name, backref->namelen,
> -				     backref->ref_type, &backref->index, 1);
> +				     backref->filetype, &backref->index, 1);
>   		if (ret < 0)
>   			goto out;
>   	}
>

Re: [PATCH 2/3] btrfs-progs: properly reset nlink of multi-linked file

[David Sterba]

On Fri, Dec 04, 2015 at 02:37:26PM +0900, Naohiro Aota wrote:
> If a file is linked from more than one directory and only one
> of the links is corrupted, btrfs check dose not reset the nlink
> properly. Actually it can go into infinite loop to link the broken file
> into lost+found.
> 
> This patch fix two part of the code. The first one delay the freeing
> valid (no error, found inode ref, directory index, and directory
> item) backrefs. Freeing valid backrefs earier prevent reset_nlink() to
> add back all valid links.
> 
> The second fix is obvious: passing `ref_type' to btrfs_add_link() is just
> wrong. It should be `filetype' instead. The current code can break all valid
> file links.
> 
> Signed-off-by: Naohiro Aota <naota@elisp.net>

Applied, thanks.

Re: [PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[Qu Wenruo]

On 12/04/2015 01:37 PM, Naohiro Aota wrote:
> This series address an issue of btrfsck to restore infinite number of
> same file into `lost+found' directory. The issue occur on a file which
> is linked from two different directory A and B. If links from dir A is
> corrupted and links from dir B is kept valid, btrfsck won't stop
> creating a file in lost+found like this:
>
> -----
> Moving file 'file.del.51' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 1876
> Trying to rebuild inode:1877
> Moving file 'del' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 1877
> Can't get file name for inode 1876, using '1876' as fallback
> Moving file '1876' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 1876
> Can't get file name for inode 1876, using '1876' as fallback
> Moving file '1876.1876' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 1876
> (snip)
> Moving file '1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876' to 'lost+found' dir since it has no valid backref
> Fixed the nlink of inode 1876
> Can't get file name for inode 1876, using '1876' as fallback
> Can't get file name for inode 1876, using '1876' as fallback
> Can't get file name for inode 1876, using '1876' as fallback
> -----
>
> The problem is early release of inode backrefs. The release prevents
> `reset_nlink()' to add back valid backrefs to an inode. In the result,
> the following results occur:
>
> 0. btrfsck scan a FS tree
> 1. It finds valid links and invalid links (some links are lost)
> 2. All valid links are released
> 3. btrfsck detects found_links != nlink
> 4. reset_nlink() reset nlink to 0
> 5. No valid links are restored (thus still nlink = 0)
> 6. The file is restored to lost+found since nlink == 0 (now, nlink = 1)
> 7. btrfsck rescan the FS tree
> 8. It finds `found_links' = #valid_links+1 (in lost+found) and nlink = 1
> 9. again all valid links are lost, and restore to lost+found

Right, that's one case I missed in the repair code.

Thanks for the fix.

>
> The first patch add clean up code to the test. It umount test
> directory on failure path. The second patch fix the above problem. And
> the last patch extend the test to check a case of multiple-linked file
> corruption.

But I only see the first 2 patches in maillist...
The last test case seems missing?

Thanks,
Qu

>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[Naohiro Aota]

On Sat, Dec 5, 2015 at 10:35 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 12/04/2015 01:37 PM, Naohiro Aota wrote:
>>
>> This series address an issue of btrfsck to restore infinite number of
>> same file into `lost+found' directory. The issue occur on a file which
>> is linked from two different directory A and B. If links from dir A is
>> corrupted and links from dir B is kept valid, btrfsck won't stop
>> creating a file in lost+found like this:
>>
>> -----
>> Moving file 'file.del.51' to 'lost+found' dir since it has no valid
>> backref
>> Fixed the nlink of inode 1876
>> Trying to rebuild inode:1877
>> Moving file 'del' to 'lost+found' dir since it has no valid backref
>> Fixed the nlink of inode 1877
>> Can't get file name for inode 1876, using '1876' as fallback
>> Moving file '1876' to 'lost+found' dir since it has no valid backref
>> Fixed the nlink of inode 1876
>> Can't get file name for inode 1876, using '1876' as fallback
>> Moving file '1876.1876' to 'lost+found' dir since it has no valid backref
>> Fixed the nlink of inode 1876
>> (snip)
>> Moving file
>> '1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876'
>> to 'lost+found' dir since it has no valid backref
>> Fixed the nlink of inode 1876
>> Can't get file name for inode 1876, using '1876' as fallback
>> Can't get file name for inode 1876, using '1876' as fallback
>> Can't get file name for inode 1876, using '1876' as fallback
>> -----
>>
>> The problem is early release of inode backrefs. The release prevents
>> `reset_nlink()' to add back valid backrefs to an inode. In the result,
>> the following results occur:
>>
>> 0. btrfsck scan a FS tree
>> 1. It finds valid links and invalid links (some links are lost)
>> 2. All valid links are released
>> 3. btrfsck detects found_links != nlink
>> 4. reset_nlink() reset nlink to 0
>> 5. No valid links are restored (thus still nlink = 0)
>> 6. The file is restored to lost+found since nlink == 0 (now, nlink = 1)
>> 7. btrfsck rescan the FS tree
>> 8. It finds `found_links' = #valid_links+1 (in lost+found) and nlink = 1
>> 9. again all valid links are lost, and restore to lost+found
>
>
> Right, that's one case I missed in the repair code.
>
> Thanks for the fix.

Thanks for the review.

>
>>
>> The first patch add clean up code to the test. It umount test
>> directory on failure path. The second patch fix the above problem. And
>> the last patch extend the test to check a case of multiple-linked file
>> corruption.
>
>
> But I only see the first 2 patches in maillist...
> The last test case seems missing?

Maybe, the last patch is too large to post to the list? Even it get
smaller, 130260 bytes seems to be a bit large.

How should I handle this? Put my repo somewhere and wait a maintainer
to pull it?

>
> Thanks,
> Qu
>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: [PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[Qu Wenruo]

Naohiro Aota wrote on 2015/12/07 11:59 +0900:
> On Sat, Dec 5, 2015 at 10:35 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>> On 12/04/2015 01:37 PM, Naohiro Aota wrote:
>>>
>>> This series address an issue of btrfsck to restore infinite number of
>>> same file into `lost+found' directory. The issue occur on a file which
>>> is linked from two different directory A and B. If links from dir A is
>>> corrupted and links from dir B is kept valid, btrfsck won't stop
>>> creating a file in lost+found like this:
>>>
>>> -----
>>> Moving file 'file.del.51' to 'lost+found' dir since it has no valid
>>> backref
>>> Fixed the nlink of inode 1876
>>> Trying to rebuild inode:1877
>>> Moving file 'del' to 'lost+found' dir since it has no valid backref
>>> Fixed the nlink of inode 1877
>>> Can't get file name for inode 1876, using '1876' as fallback
>>> Moving file '1876' to 'lost+found' dir since it has no valid backref
>>> Fixed the nlink of inode 1876
>>> Can't get file name for inode 1876, using '1876' as fallback
>>> Moving file '1876.1876' to 'lost+found' dir since it has no valid backref
>>> Fixed the nlink of inode 1876
>>> (snip)
>>> Moving file
>>> '1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876.1876'
>>> to 'lost+found' dir since it has no valid backref
>>> Fixed the nlink of inode 1876
>>> Can't get file name for inode 1876, using '1876' as fallback
>>> Can't get file name for inode 1876, using '1876' as fallback
>>> Can't get file name for inode 1876, using '1876' as fallback
>>> -----
>>>
>>> The problem is early release of inode backrefs. The release prevents
>>> `reset_nlink()' to add back valid backrefs to an inode. In the result,
>>> the following results occur:
>>>
>>> 0. btrfsck scan a FS tree
>>> 1. It finds valid links and invalid links (some links are lost)
>>> 2. All valid links are released
>>> 3. btrfsck detects found_links != nlink
>>> 4. reset_nlink() reset nlink to 0
>>> 5. No valid links are restored (thus still nlink = 0)
>>> 6. The file is restored to lost+found since nlink == 0 (now, nlink = 1)
>>> 7. btrfsck rescan the FS tree
>>> 8. It finds `found_links' = #valid_links+1 (in lost+found) and nlink = 1
>>> 9. again all valid links are lost, and restore to lost+found
>>
>>
>> Right, that's one case I missed in the repair code.
>>
>> Thanks for the fix.
>
> Thanks for the review.
>
>>
>>>
>>> The first patch add clean up code to the test. It umount test
>>> directory on failure path. The second patch fix the above problem. And
>>> the last patch extend the test to check a case of multiple-linked file
>>> corruption.
>>
>>
>> But I only see the first 2 patches in maillist...
>> The last test case seems missing?
>
> Maybe, the last patch is too large to post to the list? Even it get
> smaller, 130260 bytes seems to be a bit large.
>
> How should I handle this? Put my repo somewhere and wait a maintainer
> to pull it?

One idea is to use btrfs-image to dump the image (if it is still able to 
trigger the bug).
Btrfs-image dump is much more efficient than raw dump.

Or as you mentioned, put the repo somewhere like github, and send pull 
request to David, mentioning the patch is already sent to maillist.

Thanks,
Qu

>
>>
>> Thanks,
>> Qu
>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[David Sterba]

On Mon, Dec 07, 2015 at 11:59:19AM +0900, Naohiro Aota wrote:
> > But I only see the first 2 patches in maillist...
> > The last test case seems missing?
> 
> Maybe, the last patch is too large to post to the list? Even it get
> smaller, 130260 bytes seems to be a bit large.
> 
> How should I handle this? Put my repo somewhere and wait a maintainer
> to pull it?

Please send it to me directly. The image will be available in
btrfs-progs git and we don't necessarily need the copy in the
mailinglist.

Re: [PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[Naohiro Aota]

On Tue, Dec 8, 2015 at 12:35 AM, David Sterba <dsterba@suse.cz> wrote:
> On Mon, Dec 07, 2015 at 11:59:19AM +0900, Naohiro Aota wrote:
>> > But I only see the first 2 patches in maillist...
>> > The last test case seems missing?
>>
>> Maybe, the last patch is too large to post to the list? Even it get
>> smaller, 130260 bytes seems to be a bit large.
>>
>> How should I handle this? Put my repo somewhere and wait a maintainer
>> to pull it?
>
> Please send it to me directly. The image will be available in
> btrfs-progs git and we don't necessarily need the copy in the
> mailinglist.

Sure. I'll send it to you.

Re: [PATCH 0/3] btrfs-progs: fix file restore to lost+found bug

[David Sterba]

On Tue, Dec 08, 2015 at 11:06:28AM +0900, Naohiro Aota wrote:
> On Tue, Dec 8, 2015 at 12:35 AM, David Sterba <dsterba@suse.cz> wrote:
> > On Mon, Dec 07, 2015 at 11:59:19AM +0900, Naohiro Aota wrote:
> >> > But I only see the first 2 patches in maillist...
> >> > The last test case seems missing?
> >>
> >> Maybe, the last patch is too large to post to the list? Even it get
> >> smaller, 130260 bytes seems to be a bit large.
> >>
> >> How should I handle this? Put my repo somewhere and wait a maintainer
> >> to pull it?
> >
> > Please send it to me directly. The image will be available in
> > btrfs-progs git and we don't necessarily need the copy in the
> > mailinglist.
> 
> Sure. I'll send it to you.

Applied, thanks.