* [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation @ 2019-09-24 8:11 Qu Wenruo 2019-09-24 8:11 ` [PATCH 1/3] btrfs-progs: check/lowmem: " Qu Wenruo ` (3 more replies) 0 siblings, 4 replies; 22+ messages in thread From: Qu Wenruo @ 2019-09-24 8:11 UTC (permalink / raw) To: linux-btrfs We have at least two user reports about bad inode generation makes kernel reject the fs. According to the creation time, the inode is created by some 2014 kernel. And the generation member of INODE_ITEM is not updated (unlike the transid member) so the error persists until latest tree-checker detects. Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending one, it still should be done by btrfs-progs. This patchset adds such check and repair ability to btrfs-check, with a simple test image. Qu Wenruo (3): btrfs-progs: check/lowmem: Add check and repair for invalid inode generation btrfs-progs: check/original: Add check and repair for invalid inode generation btrfs-progs: fsck-tests: Add test image for invalid inode generation repair check/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-) create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz -- 2.23.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation 2019-09-24 8:11 [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Qu Wenruo @ 2019-09-24 8:11 ` Qu Wenruo 2019-09-30 11:36 ` Nikolay Borisov 2019-09-24 8:11 ` [PATCH 2/3] btrfs-progs: check/original: " Qu Wenruo ` (2 subsequent siblings) 3 siblings, 1 reply; 22+ messages in thread From: Qu Wenruo @ 2019-09-24 8:11 UTC (permalink / raw) To: linux-btrfs; +Cc: Charles Wright There are at least two bug reports of kernel tree-checker complaining about invalid inode generation. All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow. So add such check and repair ability to lowmem mode check first. This involves: - Calculate the inode generation upper limit If it's an inode from log tree, then the upper limit is super_generation + 1, otherwise it's super_generation. - Check if the inode generation is larger than the upper limit - Repair by resetting inode generation to current transaction generation Reported-by: Charles Wright <charles.v.wright@gmail.com> Signed-off-by: Qu Wenruo <wqu@suse.com> --- check/mode-lowmem.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c index 5f7f101d..7af29ba9 100644 --- a/check/mode-lowmem.c +++ b/check/mode-lowmem.c @@ -2472,6 +2472,59 @@ static bool has_orphan_item(struct btrfs_root *root, u64 ino) return false; } +static int repair_inode_gen_lowmem(struct btrfs_root *root, + struct btrfs_path *path) +{ + struct btrfs_trans_handle *trans; + struct btrfs_inode_item *ii; + struct btrfs_key key; + u64 transid; + int ret; + + trans = btrfs_start_transaction(root, 1); + if (IS_ERR(trans)) { + ret = PTR_ERR(trans); + errno = -ret; + error("failed to start transaction for inode gen repair: %m"); + return ret; + } + transid = trans->transid; + btrfs_item_key_to_cpu(path->nodes[0], &key, path->slots[0]); + ASSERT(key.type == BTRFS_INODE_ITEM_KEY); + + btrfs_release_path(path); + + ret = btrfs_search_slot(trans, root, &key, path, 0, 1); + if (ret > 0) { + ret = -ENOENT; + error("no inode item found for ino %llu", key.objectid); + goto error; + } + if (ret < 0) { + errno = -ret; + error("failed to find inode item for ino %llu: %m", + key.objectid); + goto error; + } + ii = btrfs_item_ptr(path->nodes[0], path->slots[0], + struct btrfs_inode_item); + btrfs_set_inode_generation(path->nodes[0], ii, trans->transid); + btrfs_mark_buffer_dirty(path->nodes[0]); + ret = btrfs_commit_transaction(trans, root); + if (ret < 0) { + errno = -ret; + error("failed to commit transaction: %m"); + goto error; + } + printf("reseting inode generation to %llu for ino %llu\n", + transid, key.objectid); + return ret; + +error: + btrfs_abort_transaction(trans, ret); + return ret; +} + /* * Check INODE_ITEM and related ITEMs (the same inode number) * 1. check link count @@ -2487,6 +2540,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) struct btrfs_inode_item *ii; struct btrfs_key key; struct btrfs_key last_key; + struct btrfs_super_block *super = root->fs_info->super_copy; u64 inode_id; u32 mode; u64 flags; @@ -2497,6 +2551,8 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) u64 refs = 0; u64 extent_end = 0; u64 extent_size = 0; + u64 generation; + u64 gen_uplimit; unsigned int dir; unsigned int nodatasum; bool is_orphan = false; @@ -2527,6 +2583,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) flags = btrfs_inode_flags(node, ii); dir = imode_to_type(mode) == BTRFS_FT_DIR; nlink = btrfs_inode_nlink(node, ii); + generation = btrfs_inode_generation(node, ii); nodatasum = btrfs_inode_flags(node, ii) & BTRFS_INODE_NODATASUM; if (!is_valid_imode(mode)) { @@ -2540,6 +2597,25 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) } } + if (btrfs_super_log_root(super) != 0 && + root->objectid == BTRFS_TREE_LOG_OBJECTID) + gen_uplimit = btrfs_super_generation(super) + 1; + else + gen_uplimit = btrfs_super_generation(super); + + if (generation > gen_uplimit) { + error( + "invalid inode generation for ino %llu, have %llu expect [0, %llu)", + inode_id, generation, gen_uplimit); + if (repair) { + ret = repair_inode_gen_lowmem(root, path); + if (ret < 0) + err |= INVALID_GENERATION; + } else { + err |= INVALID_GENERATION; + } + + } if (S_ISLNK(mode) && flags & (BTRFS_INODE_IMMUTABLE | BTRFS_INODE_APPEND)) { err |= INODE_FLAGS_ERROR; -- 2.23.0 ^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation 2019-09-24 8:11 ` [PATCH 1/3] btrfs-progs: check/lowmem: " Qu Wenruo @ 2019-09-30 11:36 ` Nikolay Borisov 2019-09-30 12:24 ` Qu Wenruo 0 siblings, 1 reply; 22+ messages in thread From: Nikolay Borisov @ 2019-09-30 11:36 UTC (permalink / raw) To: Qu Wenruo, linux-btrfs; +Cc: Charles Wright On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: > There are at least two bug reports of kernel tree-checker complaining > about invalid inode generation. > > All offending inodes seem to be caused by old kernel around 2014, with > inode generation overflow. > > So add such check and repair ability to lowmem mode check first. > > This involves: > - Calculate the inode generation upper limit > If it's an inode from log tree, then the upper limit is > super_generation + 1, otherwise it's super_generation. > > - Check if the inode generation is larger than the upper limit > > - Repair by resetting inode generation to current transaction > generation > > Reported-by: Charles Wright <charles.v.wright@gmail.com> > Signed-off-by: Qu Wenruo <wqu@suse.com> Tested-by: Nikolay Borisov <nborisov@suse.com> There is one small nit with the assert once rectified you can add: Reviewed-by: Nikolay Borisov <nborisov@suse.com> > --- > check/mode-lowmem.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 76 insertions(+) > > diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c > index 5f7f101d..7af29ba9 100644 > --- a/check/mode-lowmem.c > +++ b/check/mode-lowmem.c > @@ -2472,6 +2472,59 @@ static bool has_orphan_item(struct btrfs_root *root, u64 ino) > return false; > } > > +static int repair_inode_gen_lowmem(struct btrfs_root *root, > + struct btrfs_path *path) > +{ > + struct btrfs_trans_handle *trans; > + struct btrfs_inode_item *ii; > + struct btrfs_key key; > + u64 transid; > + int ret; > + > + trans = btrfs_start_transaction(root, 1); > + if (IS_ERR(trans)) { > + ret = PTR_ERR(trans); > + errno = -ret; > + error("failed to start transaction for inode gen repair: %m"); > + return ret; > + } > + transid = trans->transid; > + btrfs_item_key_to_cpu(path->nodes[0], &key, path->slots[0]); > + ASSERT(key.type == BTRFS_INODE_ITEM_KEY); nit: This function's sole caller, check_inode_item, is guaranteed to be called with a path pointing to BTRFS_INODE_ITEM_KEY thanks to the logic in the 'for' loop in process_one_leaf. This renders the assert redundant. At the very least I think it should be moved to check_inode_item. > + > + btrfs_release_path(path); > + > + ret = btrfs_search_slot(trans, root, &key, path, 0, 1); > + if (ret > 0) { > + ret = -ENOENT; > + error("no inode item found for ino %llu", key.objectid); > + goto error; > + } > + if (ret < 0) { > + errno = -ret; > + error("failed to find inode item for ino %llu: %m", > + key.objectid); > + goto error; > + } > + ii = btrfs_item_ptr(path->nodes[0], path->slots[0], > + struct btrfs_inode_item); > + btrfs_set_inode_generation(path->nodes[0], ii, trans->transid); > + btrfs_mark_buffer_dirty(path->nodes[0]); > + ret = btrfs_commit_transaction(trans, root); > + if (ret < 0) { > + errno = -ret; > + error("failed to commit transaction: %m"); > + goto error; > + } > + printf("reseting inode generation to %llu for ino %llu\n", > + transid, key.objectid); > + return ret; > + > +error: > + btrfs_abort_transaction(trans, ret); > + return ret; > +} > + > /* > * Check INODE_ITEM and related ITEMs (the same inode number) > * 1. check link count > @@ -2487,6 +2540,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) > struct btrfs_inode_item *ii; > struct btrfs_key key; > struct btrfs_key last_key; > + struct btrfs_super_block *super = root->fs_info->super_copy; > u64 inode_id; > u32 mode; > u64 flags; > @@ -2497,6 +2551,8 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) > u64 refs = 0; > u64 extent_end = 0; > u64 extent_size = 0; > + u64 generation; > + u64 gen_uplimit; > unsigned int dir; > unsigned int nodatasum; > bool is_orphan = false; > @@ -2527,6 +2583,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) > flags = btrfs_inode_flags(node, ii); > dir = imode_to_type(mode) == BTRFS_FT_DIR; > nlink = btrfs_inode_nlink(node, ii); > + generation = btrfs_inode_generation(node, ii); > nodatasum = btrfs_inode_flags(node, ii) & BTRFS_INODE_NODATASUM; > > if (!is_valid_imode(mode)) { > @@ -2540,6 +2597,25 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) > } > } > > + if (btrfs_super_log_root(super) != 0 && > + root->objectid == BTRFS_TREE_LOG_OBJECTID) > + gen_uplimit = btrfs_super_generation(super) + 1; > + else > + gen_uplimit = btrfs_super_generation(super); > + > + if (generation > gen_uplimit) { > + error( > + "invalid inode generation for ino %llu, have %llu expect [0, %llu)", > + inode_id, generation, gen_uplimit); > + if (repair) { > + ret = repair_inode_gen_lowmem(root, path); > + if (ret < 0) > + err |= INVALID_GENERATION; > + } else { > + err |= INVALID_GENERATION; > + } > + > + } > if (S_ISLNK(mode) && > flags & (BTRFS_INODE_IMMUTABLE | BTRFS_INODE_APPEND)) { > err |= INODE_FLAGS_ERROR; > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation 2019-09-30 11:36 ` Nikolay Borisov @ 2019-09-30 12:24 ` Qu Wenruo 2019-09-30 13:34 ` Nikolay Borisov 0 siblings, 1 reply; 22+ messages in thread From: Qu Wenruo @ 2019-09-30 12:24 UTC (permalink / raw) To: Nikolay Borisov, Qu Wenruo, linux-btrfs; +Cc: Charles Wright On 2019/9/30 下午7:36, Nikolay Borisov wrote: > > > On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: >> There are at least two bug reports of kernel tree-checker complaining >> about invalid inode generation. >> >> All offending inodes seem to be caused by old kernel around 2014, with >> inode generation overflow. >> >> So add such check and repair ability to lowmem mode check first. >> >> This involves: >> - Calculate the inode generation upper limit >> If it's an inode from log tree, then the upper limit is >> super_generation + 1, otherwise it's super_generation. >> >> - Check if the inode generation is larger than the upper limit >> >> - Repair by resetting inode generation to current transaction >> generation >> >> Reported-by: Charles Wright <charles.v.wright@gmail.com> >> Signed-off-by: Qu Wenruo <wqu@suse.com> > > Tested-by: Nikolay Borisov <nborisov@suse.com> > > There is one small nit with the assert once rectified you can add: > > Reviewed-by: Nikolay Borisov <nborisov@suse.com> > >> --- >> check/mode-lowmem.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 76 insertions(+) >> >> diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c >> index 5f7f101d..7af29ba9 100644 >> --- a/check/mode-lowmem.c >> +++ b/check/mode-lowmem.c >> @@ -2472,6 +2472,59 @@ static bool has_orphan_item(struct btrfs_root *root, u64 ino) >> return false; >> } >> >> +static int repair_inode_gen_lowmem(struct btrfs_root *root, >> + struct btrfs_path *path) >> +{ >> + struct btrfs_trans_handle *trans; >> + struct btrfs_inode_item *ii; >> + struct btrfs_key key; >> + u64 transid; >> + int ret; >> + >> + trans = btrfs_start_transaction(root, 1); >> + if (IS_ERR(trans)) { >> + ret = PTR_ERR(trans); >> + errno = -ret; >> + error("failed to start transaction for inode gen repair: %m"); >> + return ret; >> + } >> + transid = trans->transid; > >> + btrfs_item_key_to_cpu(path->nodes[0], &key, path->slots[0]); >> + ASSERT(key.type == BTRFS_INODE_ITEM_KEY); > > nit: This function's sole caller, check_inode_item, is guaranteed to be > called with a path pointing to BTRFS_INODE_ITEM_KEY thanks to the logic > in the 'for' loop in process_one_leaf. This renders the assert > redundant. At the very least I think it should be moved to > check_inode_item. Yes, the ASSERT() doesn't make much sense by itself. However I still believe it won't be a problem. It's compiler's job to remove such dead ASSERT(), but for human reader, I still believe this ASSERT() could still make sense, especially when the caller or callee can get more and more complex. Thanks, Qu > >> + >> + btrfs_release_path(path); >> + >> + ret = btrfs_search_slot(trans, root, &key, path, 0, 1); >> + if (ret > 0) { >> + ret = -ENOENT; >> + error("no inode item found for ino %llu", key.objectid); >> + goto error; >> + } >> + if (ret < 0) { >> + errno = -ret; >> + error("failed to find inode item for ino %llu: %m", >> + key.objectid); >> + goto error; >> + } >> + ii = btrfs_item_ptr(path->nodes[0], path->slots[0], >> + struct btrfs_inode_item); >> + btrfs_set_inode_generation(path->nodes[0], ii, trans->transid); >> + btrfs_mark_buffer_dirty(path->nodes[0]); >> + ret = btrfs_commit_transaction(trans, root); >> + if (ret < 0) { >> + errno = -ret; >> + error("failed to commit transaction: %m"); >> + goto error; >> + } >> + printf("reseting inode generation to %llu for ino %llu\n", >> + transid, key.objectid); >> + return ret; >> + >> +error: >> + btrfs_abort_transaction(trans, ret); >> + return ret; >> +} >> + >> /* >> * Check INODE_ITEM and related ITEMs (the same inode number) >> * 1. check link count >> @@ -2487,6 +2540,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) >> struct btrfs_inode_item *ii; >> struct btrfs_key key; >> struct btrfs_key last_key; >> + struct btrfs_super_block *super = root->fs_info->super_copy; >> u64 inode_id; >> u32 mode; >> u64 flags; >> @@ -2497,6 +2551,8 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) >> u64 refs = 0; >> u64 extent_end = 0; >> u64 extent_size = 0; >> + u64 generation; >> + u64 gen_uplimit; >> unsigned int dir; >> unsigned int nodatasum; >> bool is_orphan = false; >> @@ -2527,6 +2583,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) >> flags = btrfs_inode_flags(node, ii); >> dir = imode_to_type(mode) == BTRFS_FT_DIR; >> nlink = btrfs_inode_nlink(node, ii); >> + generation = btrfs_inode_generation(node, ii); >> nodatasum = btrfs_inode_flags(node, ii) & BTRFS_INODE_NODATASUM; >> >> if (!is_valid_imode(mode)) { >> @@ -2540,6 +2597,25 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) >> } >> } >> >> + if (btrfs_super_log_root(super) != 0 && >> + root->objectid == BTRFS_TREE_LOG_OBJECTID) >> + gen_uplimit = btrfs_super_generation(super) + 1; >> + else >> + gen_uplimit = btrfs_super_generation(super); >> + >> + if (generation > gen_uplimit) { >> + error( >> + "invalid inode generation for ino %llu, have %llu expect [0, %llu)", >> + inode_id, generation, gen_uplimit); >> + if (repair) { >> + ret = repair_inode_gen_lowmem(root, path); >> + if (ret < 0) >> + err |= INVALID_GENERATION; >> + } else { >> + err |= INVALID_GENERATION; >> + } >> + >> + } >> if (S_ISLNK(mode) && >> flags & (BTRFS_INODE_IMMUTABLE | BTRFS_INODE_APPEND)) { >> err |= INODE_FLAGS_ERROR; >> ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation 2019-09-30 12:24 ` Qu Wenruo @ 2019-09-30 13:34 ` Nikolay Borisov 2019-09-30 14:05 ` Qu Wenruo 0 siblings, 1 reply; 22+ messages in thread From: Nikolay Borisov @ 2019-09-30 13:34 UTC (permalink / raw) To: Qu Wenruo, WenRuo Qu, linux-btrfs; +Cc: Charles Wright On 30.09.19 г. 15:24 ч., Qu Wenruo wrote: > Yes, the ASSERT() doesn't make much sense by itself. > > However I still believe it won't be a problem. It won't be a problem but it feels wrong to have this assert this deep into the call chain. IMO It should be put where it can trigger at the earliest which seems to be in check_inode_item. That function assumes it's working with an inode item and goes to dereference inode members so if the type is wrong we'd crash there instead of in repair_inode_gen_lowmem. > > It's compiler's job to remove such dead ASSERT(), but for human reader, > I still believe this ASSERT() could still make sense, especially when > the caller or callee can get more and more complex. > > Thanks, ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation 2019-09-30 13:34 ` Nikolay Borisov @ 2019-09-30 14:05 ` Qu Wenruo 0 siblings, 0 replies; 22+ messages in thread From: Qu Wenruo @ 2019-09-30 14:05 UTC (permalink / raw) To: Nikolay Borisov, WenRuo Qu, linux-btrfs; +Cc: Charles Wright On 2019/9/30 下午9:34, Nikolay Borisov wrote: > > > On 30.09.19 г. 15:24 ч., Qu Wenruo wrote: >> Yes, the ASSERT() doesn't make much sense by itself. >> >> However I still believe it won't be a problem. > > It won't be a problem but it feels wrong to have this assert this deep > into the call chain. IMO It should be put where it can trigger at the > earliest which seems to be in check_inode_item. That function assumes > it's working with an inode item and goes to dereference inode members so > if the type is wrong we'd crash there instead of in repair_inode_gen_lowmem. This is going to be a preference thing. To my taste, if possible, I would put ASSERT() in every function with more than 24 lines, as an alternative to comment, to show the prerequisite or the assumption. Thanks, Qu > >> >> It's compiler's job to remove such dead ASSERT(), but for human reader, >> I still believe this ASSERT() could still make sense, especially when >> the caller or callee can get more and more complex. >> >> Thanks, ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation 2019-09-24 8:11 [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Qu Wenruo 2019-09-24 8:11 ` [PATCH 1/3] btrfs-progs: check/lowmem: " Qu Wenruo @ 2019-09-24 8:11 ` Qu Wenruo 2019-09-30 8:41 ` Nikolay Borisov 2019-09-24 8:11 ` [PATCH 3/3] btrfs-progs: fsck-tests: Add test image for invalid inode generation repair Qu Wenruo 2019-10-18 20:32 ` [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Ferry Toth 3 siblings, 1 reply; 22+ messages in thread From: Qu Wenruo @ 2019-09-24 8:11 UTC (permalink / raw) To: linux-btrfs; +Cc: Charles Wright There are at least two bug reports of kernel tree-checker complaining about invalid inode generation. All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow. So add such check and repair ability to lowmem mode check first. This involves: - Calculate the inode generation upper limit Unlike the original mode context, we don't have anyway to determine if this inode belongs to log tree. So we use super_generation + 1 as upper limit, just like what we did in kernel tree checker. - Check if the inode generation is larger than the upper limit - Repair by resetting inode generation to current transaction generation The difference is, in original mode, we have a common trans handle for all repair and reset path for each repair. Reported-by: Charles Wright <charles.v.wright@gmail.com> Signed-off-by: Qu Wenruo <wqu@suse.com> --- check/main.c | 50 ++++++++++++++++++++++++++++++++++++++++++- check/mode-original.h | 1 + 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/check/main.c b/check/main.c index c2d0f394..018707c8 100644 --- a/check/main.c +++ b/check/main.c @@ -604,6 +604,8 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec) if (errors & I_ERR_INVALID_IMODE) fprintf(stderr, ", invalid inode mode bit 0%o", rec->imode & ~07777); + if (errors & I_ERR_INVALID_GEN) + fprintf(stderr, ", invalid inode generation"); fprintf(stderr, "\n"); /* Print the holes if needed */ @@ -857,6 +859,7 @@ static int process_inode_item(struct extent_buffer *eb, { struct inode_record *rec; struct btrfs_inode_item *item; + u64 gen_uplimit; u64 flags; rec = active_node->current; @@ -879,6 +882,13 @@ static int process_inode_item(struct extent_buffer *eb, if (S_ISLNK(rec->imode) && flags & (BTRFS_INODE_IMMUTABLE | BTRFS_INODE_APPEND)) rec->errors |= I_ERR_ODD_INODE_FLAGS; + /* + * We don't have accurate root info to determine the correct + * inode generation uplimit, use super_generation + 1 anyway + */ + gen_uplimit = btrfs_super_generation(global_info->super_copy) + 1; + if (btrfs_inode_generation(eb, item) > gen_uplimit) + rec->errors |= I_ERR_INVALID_GEN; maybe_free_inode_rec(&active_node->inode_cache, rec); return 0; } @@ -2774,6 +2784,41 @@ static int repair_imode_original(struct btrfs_trans_handle *trans, return ret; } +static int repair_inode_gen_original(struct btrfs_trans_handle *trans, + struct btrfs_root *root, + struct btrfs_path *path, + struct inode_record *rec) +{ + struct btrfs_inode_item *ii; + struct btrfs_key key; + int ret; + + key.objectid = rec->ino; + key.offset = 0; + key.type = BTRFS_INODE_ITEM_KEY; + + ret = btrfs_search_slot(trans, root, &key, path, 0, 1); + if (ret > 0) { + ret = -ENOENT; + error("no inode item found for ino %llu", rec->ino); + return ret; + } + if (ret < 0) { + errno = -ret; + error("failed to search inode item for ino %llu: %m", rec->ino); + return ret; + } + ii = btrfs_item_ptr(path->nodes[0], path->slots[0], + struct btrfs_inode_item); + btrfs_set_inode_generation(path->nodes[0], ii, trans->transid); + btrfs_mark_buffer_dirty(path->nodes[0]); + btrfs_release_path(path); + printf("resetting inode generation to %llu for ino %llu\n", + trans->transid, rec->ino); + rec->errors &= ~I_ERR_INVALID_GEN; + return 0; +} + static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) { struct btrfs_trans_handle *trans; @@ -2794,7 +2839,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) I_ERR_FILE_NBYTES_WRONG | I_ERR_INLINE_RAM_BYTES_WRONG | I_ERR_MISMATCH_DIR_HASH | - I_ERR_UNALIGNED_EXTENT_REC))) + I_ERR_UNALIGNED_EXTENT_REC | + I_ERR_INVALID_GEN))) return rec->errors; /* @@ -2829,6 +2875,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) ret = repair_unaligned_extent_recs(trans, root, &path, rec); if (!ret && rec->errors & I_ERR_INVALID_IMODE) ret = repair_imode_original(trans, root, &path, rec); + if (!ret && rec->errors & I_ERR_INVALID_GEN) + ret = repair_inode_gen_original(trans, root, &path, rec); btrfs_commit_transaction(trans, root); btrfs_release_path(&path); return ret; diff --git a/check/mode-original.h b/check/mode-original.h index 78d6bb30..b075a95c 100644 --- a/check/mode-original.h +++ b/check/mode-original.h @@ -185,6 +185,7 @@ struct unaligned_extent_rec_t { #define I_ERR_INLINE_RAM_BYTES_WRONG (1 << 17) #define I_ERR_MISMATCH_DIR_HASH (1 << 18) #define I_ERR_INVALID_IMODE (1 << 19) +#define I_ERR_INVALID_GEN (1 << 20) struct inode_record { struct list_head backrefs; -- 2.23.0 ^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation 2019-09-24 8:11 ` [PATCH 2/3] btrfs-progs: check/original: " Qu Wenruo @ 2019-09-30 8:41 ` Nikolay Borisov 2019-09-30 9:00 ` Qu Wenruo 0 siblings, 1 reply; 22+ messages in thread From: Nikolay Borisov @ 2019-09-30 8:41 UTC (permalink / raw) To: Qu Wenruo, linux-btrfs On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: > There are at least two bug reports of kernel tree-checker complaining > about invalid inode generation. > > All offending inodes seem to be caused by old kernel around 2014, with > inode generation overflow. > > So add such check and repair ability to lowmem mode check first. > > This involves: > - Calculate the inode generation upper limit > Unlike the original mode context, we don't have anyway to determine if Perhaps you meant "lowmem mode context" since in lowmem you deduce whether it's a log-tree inode or not. > this inode belongs to log tree. > So we use super_generation + 1 as upper limit, just like what we did > in kernel tree checker. > > - Check if the inode generation is larger than the upper limit > > - Repair by resetting inode generation to current transaction > generation > The difference is, in original mode, we have a common trans handle for > all repair and reset path for each repair. > > Reported-by: Charles Wright <charles.v.wright@gmail.com> > Signed-off-by: Qu Wenruo <wqu@suse.com> I tested that code with the image provided and it does work as expected as well as it's fairly obvious what's happening. So: Reviewed-by: Nikolay Borisov <nborisov@suse.com> Tested-by: Nikolay Borisov <nborisov@suse.com> > --- > check/main.c | 50 ++++++++++++++++++++++++++++++++++++++++++- > check/mode-original.h | 1 + > 2 files changed, 50 insertions(+), 1 deletion(-) > > diff --git a/check/main.c b/check/main.c > index c2d0f394..018707c8 100644 > --- a/check/main.c > +++ b/check/main.c > @@ -604,6 +604,8 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec) > if (errors & I_ERR_INVALID_IMODE) > fprintf(stderr, ", invalid inode mode bit 0%o", > rec->imode & ~07777); > + if (errors & I_ERR_INVALID_GEN) > + fprintf(stderr, ", invalid inode generation"); > fprintf(stderr, "\n"); > > /* Print the holes if needed */ > @@ -857,6 +859,7 @@ static int process_inode_item(struct extent_buffer *eb, > { > struct inode_record *rec; > struct btrfs_inode_item *item; > + u64 gen_uplimit; > u64 flags; > > rec = active_node->current; > @@ -879,6 +882,13 @@ static int process_inode_item(struct extent_buffer *eb, > if (S_ISLNK(rec->imode) && > flags & (BTRFS_INODE_IMMUTABLE | BTRFS_INODE_APPEND)) > rec->errors |= I_ERR_ODD_INODE_FLAGS; > + /* > + * We don't have accurate root info to determine the correct > + * inode generation uplimit, use super_generation + 1 anyway > + */ > + gen_uplimit = btrfs_super_generation(global_info->super_copy) + 1; > + if (btrfs_inode_generation(eb, item) > gen_uplimit) > + rec->errors |= I_ERR_INVALID_GEN; > maybe_free_inode_rec(&active_node->inode_cache, rec); > return 0; > } > @@ -2774,6 +2784,41 @@ static int repair_imode_original(struct btrfs_trans_handle *trans, > return ret; > } > > +static int repair_inode_gen_original(struct btrfs_trans_handle *trans, > + struct btrfs_root *root, > + struct btrfs_path *path, > + struct inode_record *rec) > +{ > + struct btrfs_inode_item *ii; > + struct btrfs_key key; > + int ret; > + > + key.objectid = rec->ino; > + key.offset = 0; > + key.type = BTRFS_INODE_ITEM_KEY; > + > + ret = btrfs_search_slot(trans, root, &key, path, 0, 1); > + if (ret > 0) { > + ret = -ENOENT; > + error("no inode item found for ino %llu", rec->ino); > + return ret; > + } > + if (ret < 0) { > + errno = -ret; > + error("failed to search inode item for ino %llu: %m", rec->ino); > + return ret; > + } > + ii = btrfs_item_ptr(path->nodes[0], path->slots[0], > + struct btrfs_inode_item); > + btrfs_set_inode_generation(path->nodes[0], ii, trans->transid); > + btrfs_mark_buffer_dirty(path->nodes[0]); > + btrfs_release_path(path); > + printf("resetting inode generation to %llu for ino %llu\n", > + trans->transid, rec->ino); > + rec->errors &= ~I_ERR_INVALID_GEN; > + return 0; > +} > + > static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) > { > struct btrfs_trans_handle *trans; > @@ -2794,7 +2839,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) > I_ERR_FILE_NBYTES_WRONG | > I_ERR_INLINE_RAM_BYTES_WRONG | > I_ERR_MISMATCH_DIR_HASH | > - I_ERR_UNALIGNED_EXTENT_REC))) > + I_ERR_UNALIGNED_EXTENT_REC | > + I_ERR_INVALID_GEN))) > return rec->errors; > > /* > @@ -2829,6 +2875,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) > ret = repair_unaligned_extent_recs(trans, root, &path, rec); > if (!ret && rec->errors & I_ERR_INVALID_IMODE) > ret = repair_imode_original(trans, root, &path, rec); > + if (!ret && rec->errors & I_ERR_INVALID_GEN) > + ret = repair_inode_gen_original(trans, root, &path, rec); > btrfs_commit_transaction(trans, root); > btrfs_release_path(&path); > return ret; > diff --git a/check/mode-original.h b/check/mode-original.h > index 78d6bb30..b075a95c 100644 > --- a/check/mode-original.h > +++ b/check/mode-original.h > @@ -185,6 +185,7 @@ struct unaligned_extent_rec_t { > #define I_ERR_INLINE_RAM_BYTES_WRONG (1 << 17) > #define I_ERR_MISMATCH_DIR_HASH (1 << 18) > #define I_ERR_INVALID_IMODE (1 << 19) > +#define I_ERR_INVALID_GEN (1 << 20) > > struct inode_record { > struct list_head backrefs; > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation 2019-09-30 8:41 ` Nikolay Borisov @ 2019-09-30 9:00 ` Qu Wenruo 0 siblings, 0 replies; 22+ messages in thread From: Qu Wenruo @ 2019-09-30 9:00 UTC (permalink / raw) To: Nikolay Borisov, Qu Wenruo, linux-btrfs, David Sterba On 2019/9/30 下午4:41, Nikolay Borisov wrote: > > > On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: >> There are at least two bug reports of kernel tree-checker complaining >> about invalid inode generation. >> >> All offending inodes seem to be caused by old kernel around 2014, with >> inode generation overflow. >> >> So add such check and repair ability to lowmem mode check first. >> >> This involves: >> - Calculate the inode generation upper limit >> Unlike the original mode context, we don't have anyway to determine if > > Perhaps you meant "lowmem mode context" since in lowmem you deduce > whether it's a log-tree inode or not. Oh, right. Would you mind to fix this in commit time, David? > >> this inode belongs to log tree. >> So we use super_generation + 1 as upper limit, just like what we did >> in kernel tree checker. >> >> - Check if the inode generation is larger than the upper limit >> >> - Repair by resetting inode generation to current transaction >> generation >> The difference is, in original mode, we have a common trans handle for >> all repair and reset path for each repair. >> >> Reported-by: Charles Wright <charles.v.wright@gmail.com> >> Signed-off-by: Qu Wenruo <wqu@suse.com> > > I tested that code with the image provided and it does work as expected > as well as it's fairly obvious what's happening. So: > > Reviewed-by: Nikolay Borisov <nborisov@suse.com> > Tested-by: Nikolay Borisov <nborisov@suse.com> Thanks for the review and test. Thanks, Qu > >> --- >> check/main.c | 50 ++++++++++++++++++++++++++++++++++++++++++- >> check/mode-original.h | 1 + >> 2 files changed, 50 insertions(+), 1 deletion(-) >> >> diff --git a/check/main.c b/check/main.c >> index c2d0f394..018707c8 100644 >> --- a/check/main.c >> +++ b/check/main.c >> @@ -604,6 +604,8 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec) >> if (errors & I_ERR_INVALID_IMODE) >> fprintf(stderr, ", invalid inode mode bit 0%o", >> rec->imode & ~07777); >> + if (errors & I_ERR_INVALID_GEN) >> + fprintf(stderr, ", invalid inode generation"); >> fprintf(stderr, "\n"); >> >> /* Print the holes if needed */ >> @@ -857,6 +859,7 @@ static int process_inode_item(struct extent_buffer *eb, >> { >> struct inode_record *rec; >> struct btrfs_inode_item *item; >> + u64 gen_uplimit; >> u64 flags; >> >> rec = active_node->current; >> @@ -879,6 +882,13 @@ static int process_inode_item(struct extent_buffer *eb, >> if (S_ISLNK(rec->imode) && >> flags & (BTRFS_INODE_IMMUTABLE | BTRFS_INODE_APPEND)) >> rec->errors |= I_ERR_ODD_INODE_FLAGS; >> + /* >> + * We don't have accurate root info to determine the correct >> + * inode generation uplimit, use super_generation + 1 anyway >> + */ >> + gen_uplimit = btrfs_super_generation(global_info->super_copy) + 1; >> + if (btrfs_inode_generation(eb, item) > gen_uplimit) >> + rec->errors |= I_ERR_INVALID_GEN; >> maybe_free_inode_rec(&active_node->inode_cache, rec); >> return 0; >> } >> @@ -2774,6 +2784,41 @@ static int repair_imode_original(struct btrfs_trans_handle *trans, >> return ret; >> } >> >> +static int repair_inode_gen_original(struct btrfs_trans_handle *trans, >> + struct btrfs_root *root, >> + struct btrfs_path *path, >> + struct inode_record *rec) >> +{ >> + struct btrfs_inode_item *ii; >> + struct btrfs_key key; >> + int ret; >> + >> + key.objectid = rec->ino; >> + key.offset = 0; >> + key.type = BTRFS_INODE_ITEM_KEY; >> + >> + ret = btrfs_search_slot(trans, root, &key, path, 0, 1); >> + if (ret > 0) { >> + ret = -ENOENT; >> + error("no inode item found for ino %llu", rec->ino); >> + return ret; >> + } >> + if (ret < 0) { >> + errno = -ret; >> + error("failed to search inode item for ino %llu: %m", rec->ino); >> + return ret; >> + } >> + ii = btrfs_item_ptr(path->nodes[0], path->slots[0], >> + struct btrfs_inode_item); >> + btrfs_set_inode_generation(path->nodes[0], ii, trans->transid); >> + btrfs_mark_buffer_dirty(path->nodes[0]); >> + btrfs_release_path(path); >> + printf("resetting inode generation to %llu for ino %llu\n", >> + trans->transid, rec->ino); >> + rec->errors &= ~I_ERR_INVALID_GEN; >> + return 0; >> +} >> + >> static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) >> { >> struct btrfs_trans_handle *trans; >> @@ -2794,7 +2839,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) >> I_ERR_FILE_NBYTES_WRONG | >> I_ERR_INLINE_RAM_BYTES_WRONG | >> I_ERR_MISMATCH_DIR_HASH | >> - I_ERR_UNALIGNED_EXTENT_REC))) >> + I_ERR_UNALIGNED_EXTENT_REC | >> + I_ERR_INVALID_GEN))) >> return rec->errors; >> >> /* >> @@ -2829,6 +2875,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) >> ret = repair_unaligned_extent_recs(trans, root, &path, rec); >> if (!ret && rec->errors & I_ERR_INVALID_IMODE) >> ret = repair_imode_original(trans, root, &path, rec); >> + if (!ret && rec->errors & I_ERR_INVALID_GEN) >> + ret = repair_inode_gen_original(trans, root, &path, rec); >> btrfs_commit_transaction(trans, root); >> btrfs_release_path(&path); >> return ret; >> diff --git a/check/mode-original.h b/check/mode-original.h >> index 78d6bb30..b075a95c 100644 >> --- a/check/mode-original.h >> +++ b/check/mode-original.h >> @@ -185,6 +185,7 @@ struct unaligned_extent_rec_t { >> #define I_ERR_INLINE_RAM_BYTES_WRONG (1 << 17) >> #define I_ERR_MISMATCH_DIR_HASH (1 << 18) >> #define I_ERR_INVALID_IMODE (1 << 19) >> +#define I_ERR_INVALID_GEN (1 << 20) >> >> struct inode_record { >> struct list_head backrefs; >> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 3/3] btrfs-progs: fsck-tests: Add test image for invalid inode generation repair 2019-09-24 8:11 [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Qu Wenruo 2019-09-24 8:11 ` [PATCH 1/3] btrfs-progs: check/lowmem: " Qu Wenruo 2019-09-24 8:11 ` [PATCH 2/3] btrfs-progs: check/original: " Qu Wenruo @ 2019-09-24 8:11 ` Qu Wenruo 2019-10-18 20:32 ` [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Ferry Toth 3 siblings, 0 replies; 22+ messages in thread From: Qu Wenruo @ 2019-09-24 8:11 UTC (permalink / raw) To: linux-btrfs The image contains one inode item with invalid generation. The image can be crafted by "btrfs-corrupt-block -i 257 -f generation". It should emulate the bad inode generation caused by older kernel around 2014. The image is repairable for both original and lowmem mode. Signed-off-by: Qu Wenruo <wqu@suse.com> --- .../043-bad-inode-generation/.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz diff --git a/tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable b/tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable new file mode 100644 index 00000000..e69de29b diff --git a/tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz b/tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz new file mode 100644 index 0000000000000000000000000000000000000000..d7302fcbabc653885ae1fac438183d7a9908a2f1 GIT binary patch literal 2012 zcmV<22P63XH+ooF000E$*0e?f03iV!0000G&sfah3;ze3T>wRyj;C3^v%$$4d1r37 zhA1?^g@qU6X{4JoZYUD&m)J~cEzT$F8&JddXW->df79?XOzszXC^i=3jv5zdJ#gxs z9sQ15EP30<h-K;{hVQDH?$vha-HC@MKe-p3;nw!aTGSk1iK3$jhkg^mmfi38LdMcl zI<0I=(4xt|g|iQ|g);e0j!pQZ26;x3+R`;xzoVds&J=hBAw4B8L-P|Ve!$)LX@XLr zpzF|j3*&CLczbqE9v1ss1PkPi%^0v(Dlm5pYr2bQv{LZKvdNN5!i)OgFjV9&wH^qU z$RwlSTsQ2*=LfG2wb84ro>7fR>-UpptOL6$V1qyGH7H=dX29pcbb>H?Kn*GkWCd+> z4gz!Y>ckaFgrgJeuG6M51wqhmn{kbaDVJg3Ld?lgrBdfj3zsD?PK#AyrUSv9Tph@s zlWPn@3zTdm{~HMt@#SSDZQo9pza_F;Uft;~oZgvZAnors__Ju}&Qp3w_XQ)tsS2T| zq$FIL*2A$z^pWDZ*1|!M(O+5AX(EZJ$>dBH`%1hzGX?o=*9kma$y@8m;CS@I{;nWI z*j}3J7}<P7H+Q_(*~Z?Dn4(YdJ7u&w3CIuVU<B=LH^@^ueAZn4p^hrwCEExS1qvnu z4q4*GjEao!lqtg??)^V&FT@3f935LlH<Zm7um2i^^>iP+f7~@z9j`a%V<Q{xAs~eT zLANvc3&|D0;`ED|^@hYz5<#z>r;Y*M<h-VEzVXiL7Mh6+H6<T-L&2+)TQIS?YO`=; z1+t^Sp;BfP9zw9oIlv>{DE&w$@uL;WPHEno3DACkV>>P=M{Ff3#!`V7d;rR>!$!8m zzr2L&o3Z*+c<OlucJg#(xtoceTmW4|=pQRoTY`fh<wo$+#w#exT&A73=&D}upU1>u zeHG)3b0=ql#0j_4)Gl@YHB@!`y9GEG1RvAk6KQ^YKcMM<dkd74f3yf=8L+9fzadT9 z^R}0|n8E?#M8L$TjA--=%23~K=Z?jLDgWEB%=D2u@^?NDFx_kfG5$FBzvx@Ex`6E; z$Bb%+a#%_FLpS!cB15&Hl(BJ1YJsrNI!=5S`G1G8>rE&8HfO=EEN1E(wLnK-l_6VQ zUEzt^?TcZ$wp0ul`k`yBMI3y4xfj}%wEPc%>_>?=JzO{|(p?yTYi(E~Ri(3d4PLj| z-q_V+l1HR|29?y>SZ#C;mEzg;X?sBSb`OI<^BrkUUO!z7;*qFF<o2;dkg|U<S6r~4 zT3tckU+9<pgSwBl2%ZTP%Aj*H9c^EFr68d1;A(bh?%cR!GRbY2!(teY{(>JEH3r(k z!AXLNe~a4|dsq;NVVL)uHr^WPWU_rXL-F}@@>i&G@F%st)hh9}CE+-jOR(h79xHk6 z^dF)hH#&Z+_Zm95ii!A=V@&q*g58Rx(*t2p*X(59-0l|_l8}Tc_=7*sX$P+QbM+cU zD9khoX3gK0U$`(inhSdoZ$DF>nS7fUp98+}-M;HFDb*iCz-L!tXG+#5T6AIEtKcQv zhvAhr@`FA8s`s@OM{J||gVAGFT&>`tX@G+Cg=Ei-+0E(NrO;(56yJ<mn87((;{&<$ z!vP>zdmtUKI8O1Kf)!vjac5Mm>@t~X+|<K2k7pmf2h^whU`9F^M@g6Vf`aSTg#7QK z2(a&Zf%OC!HmVf#sH&uNw_mGH@-yi@660h)RhD+9fo&z{++a?!tQp^6O`9n0do-`c zi59G~u!yh#9U1zupeXJzuNfvTSa_>wv7i~&1yym3r0)M?og*@PZg_dS41n#{io?=L zciazGdOvd@Mjk}WI|g@>=qLkIwL0;W+Q`Jz%`|psoEMb<`|X2P4|$@YN5>%uq&X2^ zkyrwTpaXx3X0D2tvu9=WKrQiTKb>Q=rA#!_$5|X8@4P;=6i;P0gxgU}Ui371E42dz zn>c#uoIz?0HY94QQQ>8fu27(WWks^}FEXr`TwbDak}wd!J&V%r4bv4`30pmS+T9WT z=CP(|%4;_3elq$P8k7GWSdXE|_%te~?dW~>3V334JW;b6sM=7DecM0c4J54{58)H; z-uMbZ9#jCD;4(DYlCY|hSzb~klkcDdPB;a34yOjt&_BAjLZAMSiB5n_KF&COtxbes zEcszYE;#FVX-vDZ#vs(#R~@^8*^}Dym5@sYubU#`M6B)U4M!{M*{y(555${5Rbm4a z6JvM!4sOI~3}h^vXC`V4yqE^R&0-o@!R}HlW&R;T32p{en5w!+OZCEHJe^Fia)@}X zFP1X0u%e_f=pquM0wgP!<`;?+KaY6k(S!tV^G%<uo0IZ`#j)Q18sGk;4M^IaGB4N% z4q&QW$Md|Hjs2CLsf-H?jQ6x^Rq~%ENLB+>Jl`BSY`lb3-&5!g^eQ6<>R-=RsPB-E z<vAbCA-Wrm(Pc9YM7>13E=;JDLif@t_INz&YKa!;Zw#THD=8~YwVLhT_7j2&I>Eq6 zaQA0+X>(drz2NQzo;VMNU>DoE_n5rZXC*Rc-adER=MD9Y1w21(wlG${buZ6hhdtQK utt-Hd+V!p60001@#^<1LEIQ)=0k;o;7ytk=B&NBs#Ao{g000001X)^iOycnX literal 0 HcmV?d00001 -- 2.23.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-09-24 8:11 [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Qu Wenruo ` (2 preceding siblings ...) 2019-09-24 8:11 ` [PATCH 3/3] btrfs-progs: fsck-tests: Add test image for invalid inode generation repair Qu Wenruo @ 2019-10-18 20:32 ` Ferry Toth 2019-10-18 23:50 ` Qu WenRuo 3 siblings, 1 reply; 22+ messages in thread From: Ferry Toth @ 2019-10-18 20:32 UTC (permalink / raw) To: Qu Wenruo, linux-btrfs Op 24-09-2019 om 10:11 schreef Qu Wenruo: > We have at least two user reports about bad inode generation makes > kernel reject the fs. May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too). Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@ In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation open_ctree failed Booting old kernel works just as before, no errors. > According to the creation time, the inode is created by some 2014 > kernel. How do I get the creation time? > And the generation member of INODE_ITEM is not updated (unlike the > transid member) so the error persists until latest tree-checker detects. > > Even the situation can be fixed by reverting back to older kernel and > copying the offending dir/file to another inode and delete the offending > one, it still should be done by btrfs-progs. > How to find the offending dir/file from the command line manually? > This patchset adds such check and repair ability to btrfs-check, with a > simple test image. > > Qu Wenruo (3): > btrfs-progs: check/lowmem: Add check and repair for invalid inode > generation > btrfs-progs: check/original: Add check and repair for invalid inode > generation > btrfs-progs: fsck-tests: Add test image for invalid inode generation > repair > > check/main.c | 50 +++++++++++- > check/mode-lowmem.c | 76 ++++++++++++++++++ > check/mode-original.h | 1 + > .../.lowmem_repairable | 0 > .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes > 5 files changed, 126 insertions(+), 1 deletion(-) > create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable > create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-18 20:32 ` [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Ferry Toth @ 2019-10-18 23:50 ` Qu WenRuo 2019-10-19 16:24 ` Ferry Toth 0 siblings, 1 reply; 22+ messages in thread From: Qu WenRuo @ 2019-10-18 23:50 UTC (permalink / raw) To: Ferry Toth, linux-btrfs On 2019/10/19 上午4:32, Ferry Toth wrote: > Op 24-09-2019 om 10:11 schreef Qu Wenruo: >> We have at least two user reports about bad inode generation makes >> kernel reject the fs. > > May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so > kernel went from 5.0 -> 5.3 (but I was using 4.15 too). > > Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@ > > In initramfs I can try to mount but get something like > btrfs critical corrupt leaf invalid inode generation open_ctree failed > > Booting old kernel works just as before, no errors. > >> According to the creation time, the inode is created by some 2014 >> kernel. > > How do I get the creation time? # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> > >> And the generation member of INODE_ITEM is not updated (unlike the >> transid member) so the error persists until latest tree-checker detects. >> >> Even the situation can be fixed by reverting back to older kernel and >> copying the offending dir/file to another inode and delete the offending >> one, it still should be done by btrfs-progs. >> > How to find the offending dir/file from the command line manually? # find <mount point> -inum <inode number> Thanks, Qu > >> This patchset adds such check and repair ability to btrfs-check, with a >> simple test image. >> >> Qu Wenruo (3): >> btrfs-progs: check/lowmem: Add check and repair for invalid inode >> generation >> btrfs-progs: check/original: Add check and repair for invalid inode >> generation >> btrfs-progs: fsck-tests: Add test image for invalid inode generation >> repair >> >> check/main.c | 50 +++++++++++- >> check/mode-lowmem.c | 76 ++++++++++++++++++ >> check/mode-original.h | 1 + >> .../.lowmem_repairable | 0 >> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes >> 5 files changed, 126 insertions(+), 1 deletion(-) >> create mode 100644 >> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >> create mode 100644 >> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >> > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-18 23:50 ` Qu WenRuo @ 2019-10-19 16:24 ` Ferry Toth 2019-10-20 0:26 ` Qu Wenruo 0 siblings, 1 reply; 22+ messages in thread From: Ferry Toth @ 2019-10-19 16:24 UTC (permalink / raw) To: Qu WenRuo, linux-btrfs Hi, Op 19-10-2019 om 01:50 schreef Qu WenRuo: > > > On 2019/10/19 上午4:32, Ferry Toth wrote: >> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>> We have at least two user reports about bad inode generation makes >>> kernel reject the fs. >> >> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >> >> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@ >> >> In initramfs I can try to mount but get something like >> btrfs critical corrupt leaf invalid inode generation open_ctree failed >> >> Booting old kernel works just as before, no errors. >> >>> According to the creation time, the inode is created by some 2014 >>> kernel. >> >> How do I get the creation time? > > # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014. >> >>> And the generation member of INODE_ITEM is not updated (unlike the >>> transid member) so the error persists until latest tree-checker detects. >>> >>> Even the situation can be fixed by reverting back to older kernel and >>> copying the offending dir/file to another inode and delete the offending >>> one, it still should be done by btrfs-progs. >>> >> How to find the offending dir/file from the command line manually? > > # find <mount point> -inum <inode number> This works, thanks. But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up. > Thanks, > Qu > >> >>> This patchset adds such check and repair ability to btrfs-check, with a >>> simple test image. >>> >>> Qu Wenruo (3): >>> btrfs-progs: check/lowmem: Add check and repair for invalid inode >>> generation >>> btrfs-progs: check/original: Add check and repair for invalid inode >>> generation >>> btrfs-progs: fsck-tests: Add test image for invalid inode generation >>> repair >>> >>> check/main.c | 50 +++++++++++- >>> check/mode-lowmem.c | 76 ++++++++++++++++++ >>> check/mode-original.h | 1 + >>> .../.lowmem_repairable | 0 >>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes >>> 5 files changed, 126 insertions(+), 1 deletion(-) >>> create mode 100644 >>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>> create mode 100644 >>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>> >> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors). I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs. If you like I can share the log. Let me know. This issue can potentially cause a lot of grief. Our company server runs Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)? Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released? ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-19 16:24 ` Ferry Toth @ 2019-10-20 0:26 ` Qu Wenruo 2019-10-20 0:51 ` Qu Wenruo 2019-10-20 11:50 ` Ferry Toth 0 siblings, 2 replies; 22+ messages in thread From: Qu Wenruo @ 2019-10-20 0:26 UTC (permalink / raw) To: Ferry Toth, Qu WenRuo, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 4057 bytes --] On 2019/10/20 上午12:24, Ferry Toth wrote: > Hi, > > Op 19-10-2019 om 01:50 schreef Qu WenRuo: >> >> >> On 2019/10/19 上午4:32, Ferry Toth wrote: >>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>> We have at least two user reports about bad inode generation makes >>>> kernel reject the fs. >>> >>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>> >>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@ >>> >>> In initramfs I can try to mount but get something like >>> btrfs critical corrupt leaf invalid inode generation open_ctree failed >>> >>> Booting old kernel works just as before, no errors. >>> >>>> According to the creation time, the inode is created by some 2014 >>>> kernel. >>> >>> How do I get the creation time? >> >> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> > > I just went back to the office to reboot to 5.3 and check the creation > times and found they were 2013 - 2014. > >>> >>>> And the generation member of INODE_ITEM is not updated (unlike the >>>> transid member) so the error persists until latest tree-checker >>>> detects. >>>> >>>> Even the situation can be fixed by reverting back to older kernel and >>>> copying the offending dir/file to another inode and delete the >>>> offending >>>> one, it still should be done by btrfs-progs. >>>> >>> How to find the offending dir/file from the command line manually? >> >> # find <mount point> -inum <inode number> > > This works, thanks. > > But appears unpractical. After fix 2 files and reboot, I found 4 more, > then 16, then I gave up. > >> Thanks, >> Qu >> >>> >>>> This patchset adds such check and repair ability to btrfs-check, with a >>>> simple test image. >>>> >>>> Qu Wenruo (3): >>>> btrfs-progs: check/lowmem: Add check and repair for invalid inode >>>> generation >>>> btrfs-progs: check/original: Add check and repair for invalid inode >>>> generation >>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>> generation >>>> repair >>>> >>>> check/main.c | 50 +++++++++++- >>>> check/mode-lowmem.c | 76 >>>> ++++++++++++++++++ >>>> check/mode-original.h | 1 + >>>> .../.lowmem_repairable | 0 >>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes >>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>> create mode 100644 >>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>> create mode 100644 >>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>> >>> > > I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my > mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 > errors). It's really not recommended to run btrfs check, especially repair on the mounted fs, unless it's RO. A new transaction from kernel can easily screw up the repaired fs. > > I'm not sure if using the v5.0 kernel and/or checking mounted distorts > the results? Else I'm going to need a live usb with a v5.3 kernel and > v5.3 btrfs-progs. > > If you like I can share the log. Let me know. > > This issue can potentially cause a lot of grief. Our company server runs > Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with > ~100 snapshots. I guess the problematic inodes need to be fixed on each > snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)? Yes. > > Do I understand correctly that this FTB is caused by more strict > checking of the fs by the kernel, while the tools to fix the detected > corruptions are not yet released? Yes. Thanks, Qu [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 0:26 ` Qu Wenruo @ 2019-10-20 0:51 ` Qu Wenruo 2019-10-20 13:04 ` Ferry Toth 2019-10-20 11:50 ` Ferry Toth 1 sibling, 1 reply; 22+ messages in thread From: Qu Wenruo @ 2019-10-20 0:51 UTC (permalink / raw) To: Ferry Toth, Qu WenRuo, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 4461 bytes --] On 2019/10/20 上午8:26, Qu Wenruo wrote: > > > On 2019/10/20 上午12:24, Ferry Toth wrote: >> Hi, >> >> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>> >>> >>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>> We have at least two user reports about bad inode generation makes >>>>> kernel reject the fs. >>>> >>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>> >>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@ >>>> >>>> In initramfs I can try to mount but get something like >>>> btrfs critical corrupt leaf invalid inode generation open_ctree failed >>>> >>>> Booting old kernel works just as before, no errors. >>>> >>>>> According to the creation time, the inode is created by some 2014 >>>>> kernel. >>>> >>>> How do I get the creation time? >>> >>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> >> >> I just went back to the office to reboot to 5.3 and check the creation >> times and found they were 2013 - 2014. >> >>>> >>>>> And the generation member of INODE_ITEM is not updated (unlike the >>>>> transid member) so the error persists until latest tree-checker >>>>> detects. >>>>> >>>>> Even the situation can be fixed by reverting back to older kernel and >>>>> copying the offending dir/file to another inode and delete the >>>>> offending >>>>> one, it still should be done by btrfs-progs. >>>>> >>>> How to find the offending dir/file from the command line manually? >>> >>> # find <mount point> -inum <inode number> >> >> This works, thanks. >> >> But appears unpractical. After fix 2 files and reboot, I found 4 more, >> then 16, then I gave up. Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with the new file. It would be much safer than btrfs check --repair. Thanks, Qu >> >>> Thanks, >>> Qu >>> >>>> >>>>> This patchset adds such check and repair ability to btrfs-check, with a >>>>> simple test image. >>>>> >>>>> Qu Wenruo (3): >>>>> btrfs-progs: check/lowmem: Add check and repair for invalid inode >>>>> generation >>>>> btrfs-progs: check/original: Add check and repair for invalid inode >>>>> generation >>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>> generation >>>>> repair >>>>> >>>>> check/main.c | 50 +++++++++++- >>>>> check/mode-lowmem.c | 76 >>>>> ++++++++++++++++++ >>>>> check/mode-original.h | 1 + >>>>> .../.lowmem_repairable | 0 >>>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes >>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>> create mode 100644 >>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>> create mode 100644 >>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>> >>>> >> >> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >> errors). > > It's really not recommended to run btrfs check, especially repair on the > mounted fs, unless it's RO. > > A new transaction from kernel can easily screw up the repaired fs. >> >> I'm not sure if using the v5.0 kernel and/or checking mounted distorts >> the results? Else I'm going to need a live usb with a v5.3 kernel and >> v5.3 btrfs-progs. >> >> If you like I can share the log. Let me know. >> >> This issue can potentially cause a lot of grief. Our company server runs >> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with >> ~100 snapshots. I guess the problematic inodes need to be fixed on each >> snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)? > > Yes. > >> >> Do I understand correctly that this FTB is caused by more strict >> checking of the fs by the kernel, while the tools to fix the detected >> corruptions are not yet released? > > Yes. > > Thanks, > Qu > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 0:51 ` Qu Wenruo @ 2019-10-20 13:04 ` Ferry Toth 2019-10-20 13:15 ` Qu WenRuo 0 siblings, 1 reply; 22+ messages in thread From: Ferry Toth @ 2019-10-20 13:04 UTC (permalink / raw) To: Qu Wenruo, Qu WenRuo, linux-btrfs Op 20-10-2019 om 02:51 schreef Qu Wenruo: > > > On 2019/10/20 上午8:26, Qu Wenruo wrote: >> >> >> On 2019/10/20 上午12:24, Ferry Toth wrote: >>> Hi, >>> >>> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>>> >>>> >>>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>>> We have at least two user reports about bad inode generation makes >>>>>> kernel reject the fs. >>>>> >>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>>> >>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@ >>>>> >>>>> In initramfs I can try to mount but get something like >>>>> btrfs critical corrupt leaf invalid inode generation open_ctree failed >>>>> >>>>> Booting old kernel works just as before, no errors. >>>>> >>>>>> According to the creation time, the inode is created by some 2014 >>>>>> kernel. >>>>> >>>>> How do I get the creation time? >>>> >>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> >>> >>> I just went back to the office to reboot to 5.3 and check the creation >>> times and found they were 2013 - 2014. >>> >>>>> >>>>>> And the generation member of INODE_ITEM is not updated (unlike the >>>>>> transid member) so the error persists until latest tree-checker >>>>>> detects. >>>>>> >>>>>> Even the situation can be fixed by reverting back to older kernel and >>>>>> copying the offending dir/file to another inode and delete the >>>>>> offending >>>>>> one, it still should be done by btrfs-progs. >>>>>> >>>>> How to find the offending dir/file from the command line manually? >>>> >>>> # find <mount point> -inum <inode number> >>> >>> This works, thanks. >>> >>> But appears unpractical. After fix 2 files and reboot, I found 4 more, >>> then 16, then I gave up. > > Another solution is use "find" to locate the files with creation time > before 2015, and copy them to a new file, then replace the old file with > the new file. Hmm. But how do I "find" by creation time (otime)? Do you have a suggestion for this? > It would be much safer than btrfs check --repair. > > Thanks, > Qu > > >>> >>>> Thanks, >>>> Qu >>>> >>>>> >>>>>> This patchset adds such check and repair ability to btrfs-check, with a >>>>>> simple test image. >>>>>> >>>>>> Qu Wenruo (3): >>>>>> btrfs-progs: check/lowmem: Add check and repair for invalid inode >>>>>> generation >>>>>> btrfs-progs: check/original: Add check and repair for invalid inode >>>>>> generation >>>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>>> generation >>>>>> repair >>>>>> >>>>>> check/main.c | 50 +++++++++++- >>>>>> check/mode-lowmem.c | 76 >>>>>> ++++++++++++++++++ >>>>>> check/mode-original.h | 1 + >>>>>> .../.lowmem_repairable | 0 >>>>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes >>>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>>> create mode 100644 >>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>>> create mode 100644 >>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>>> >>>>> >>> >>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >>> errors). >> >> It's really not recommended to run btrfs check, especially repair on the >> mounted fs, unless it's RO. >> >> A new transaction from kernel can easily screw up the repaired fs. >>> >>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts >>> the results? Else I'm going to need a live usb with a v5.3 kernel and >>> v5.3 btrfs-progs. >>> >>> If you like I can share the log. Let me know. >>> >>> This issue can potentially cause a lot of grief. Our company server runs >>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with >>> ~100 snapshots. I guess the problematic inodes need to be fixed on each >>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)? >> >> Yes. >> >>> >>> Do I understand correctly that this FTB is caused by more strict >>> checking of the fs by the kernel, while the tools to fix the detected >>> corruptions are not yet released? >> >> Yes. >> >> Thanks, >> Qu >> > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 13:04 ` Ferry Toth @ 2019-10-20 13:15 ` Qu WenRuo 2019-10-20 13:29 ` Ferry Toth 0 siblings, 1 reply; 22+ messages in thread From: Qu WenRuo @ 2019-10-20 13:15 UTC (permalink / raw) To: Ferry Toth, Qu Wenruo, linux-btrfs On 2019/10/20 下午9:04, Ferry Toth wrote: > Op 20-10-2019 om 02:51 schreef Qu Wenruo: >> >> >> On 2019/10/20 上午8:26, Qu Wenruo wrote: >>> >>> >>> On 2019/10/20 上午12:24, Ferry Toth wrote: >>>> Hi, >>>> >>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>>>> >>>>> >>>>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>>>> We have at least two user reports about bad inode generation makes >>>>>>> kernel reject the fs. >>>>>> >>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>>>> >>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / >>>>>> on /@ >>>>>> >>>>>> In initramfs I can try to mount but get something like >>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree >>>>>> failed >>>>>> >>>>>> Booting old kernel works just as before, no errors. >>>>>> >>>>>>> According to the creation time, the inode is created by some 2014 >>>>>>> kernel. >>>>>> >>>>>> How do I get the creation time? >>>>> >>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> >>>> >>>> I just went back to the office to reboot to 5.3 and check the creation >>>> times and found they were 2013 - 2014. >>>> >>>>>> >>>>>>> And the generation member of INODE_ITEM is not updated (unlike the >>>>>>> transid member) so the error persists until latest tree-checker >>>>>>> detects. >>>>>>> >>>>>>> Even the situation can be fixed by reverting back to older kernel >>>>>>> and >>>>>>> copying the offending dir/file to another inode and delete the >>>>>>> offending >>>>>>> one, it still should be done by btrfs-progs. >>>>>>> >>>>>> How to find the offending dir/file from the command line manually? >>>>> >>>>> # find <mount point> -inum <inode number> >>>> >>>> This works, thanks. >>>> >>>> But appears unpractical. After fix 2 files and reboot, I found 4 more, >>>> then 16, then I gave up. >> >> Another solution is use "find" to locate the files with creation time >> before 2015, and copy them to a new file, then replace the old file with >> the new file. > > Hmm. But how do I "find" by creation time (otime)? Do you have a > suggestion for this? $ touch -t 201501010000 /tmp/sample $ find <mnt> -not -cnewer /tmp/sample If you want, you can add -exec to that find, but I'd only add that after confirming the execution load is verified. Thanks, Qu > >> It would be much safer than btrfs check --repair. >> >> Thanks, >> Qu >> >> >>>> >>>>> Thanks, >>>>> Qu >>>>> >>>>>> >>>>>>> This patchset adds such check and repair ability to btrfs-check, >>>>>>> with a >>>>>>> simple test image. >>>>>>> >>>>>>> Qu Wenruo (3): >>>>>>> btrfs-progs: check/lowmem: Add check and repair for invalid >>>>>>> inode >>>>>>> generation >>>>>>> btrfs-progs: check/original: Add check and repair for >>>>>>> invalid inode >>>>>>> generation >>>>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>>>> generation >>>>>>> repair >>>>>>> >>>>>>> check/main.c | 50 +++++++++++- >>>>>>> check/mode-lowmem.c | 76 >>>>>>> ++++++++++++++++++ >>>>>>> check/mode-original.h | 1 + >>>>>>> .../.lowmem_repairable | 0 >>>>>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 >>>>>>> bytes >>>>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>>>> create mode 100644 >>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>>>> create mode 100644 >>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>>>> >>>>>>> >>>>>> >>>> >>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >>>> errors). >>> >>> It's really not recommended to run btrfs check, especially repair on the >>> mounted fs, unless it's RO. >>> >>> A new transaction from kernel can easily screw up the repaired fs. >>>> >>>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts >>>> the results? Else I'm going to need a live usb with a v5.3 kernel and >>>> v5.3 btrfs-progs. >>>> >>>> If you like I can share the log. Let me know. >>>> >>>> This issue can potentially cause a lot of grief. Our company server >>>> runs >>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with >>>> ~100 snapshots. I guess the problematic inodes need to be fixed on each >>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel >>>> ~5.6)? >>> >>> Yes. >>> >>>> >>>> Do I understand correctly that this FTB is caused by more strict >>>> checking of the fs by the kernel, while the tools to fix the detected >>>> corruptions are not yet released? >>> >>> Yes. >>> >>> Thanks, >>> Qu >>> >> > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 13:15 ` Qu WenRuo @ 2019-10-20 13:29 ` Ferry Toth 2019-10-20 14:11 ` Qu Wenruo 0 siblings, 1 reply; 22+ messages in thread From: Ferry Toth @ 2019-10-20 13:29 UTC (permalink / raw) To: Qu WenRuo, Qu Wenruo, linux-btrfs Op 20-10-2019 om 15:15 schreef Qu WenRuo: > > > On 2019/10/20 下午9:04, Ferry Toth wrote: >> Op 20-10-2019 om 02:51 schreef Qu Wenruo: >>> >>> >>> On 2019/10/20 上午8:26, Qu Wenruo wrote: >>>> >>>> >>>> On 2019/10/20 上午12:24, Ferry Toth wrote: >>>>> Hi, >>>>> >>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>>>>> >>>>>> >>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>>>>> We have at least two user reports about bad inode generation makes >>>>>>>> kernel reject the fs. >>>>>>> >>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>>>>> >>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / >>>>>>> on /@ >>>>>>> >>>>>>> In initramfs I can try to mount but get something like >>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree >>>>>>> failed >>>>>>> >>>>>>> Booting old kernel works just as before, no errors. >>>>>>> >>>>>>>> According to the creation time, the inode is created by some 2014 >>>>>>>> kernel. >>>>>>> >>>>>>> How do I get the creation time? >>>>>> >>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> >>>>> >>>>> I just went back to the office to reboot to 5.3 and check the creation >>>>> times and found they were 2013 - 2014. >>>>> >>>>>>> >>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the >>>>>>>> transid member) so the error persists until latest tree-checker >>>>>>>> detects. >>>>>>>> >>>>>>>> Even the situation can be fixed by reverting back to older kernel >>>>>>>> and >>>>>>>> copying the offending dir/file to another inode and delete the >>>>>>>> offending >>>>>>>> one, it still should be done by btrfs-progs. >>>>>>>> >>>>>>> How to find the offending dir/file from the command line manually? >>>>>> >>>>>> # find <mount point> -inum <inode number> >>>>> >>>>> This works, thanks. >>>>> >>>>> But appears unpractical. After fix 2 files and reboot, I found 4 more, >>>>> then 16, then I gave up. >>> >>> Another solution is use "find" to locate the files with creation time >>> before 2015, and copy them to a new file, then replace the old file with >>> the new file. >> >> Hmm. But how do I "find" by creation time (otime)? Do you have a >> suggestion for this? > > $ touch -t 201501010000 /tmp/sample > $ find <mnt> -not -cnewer /tmp/sample AFAIK this compares file modified date with status changed date. So, no search for creation date. And stat /tmp/sample (sorry dutch lang output): ferry@ferry-quad:~$ stat /tmp/sample Bestand: /tmp/sample Grootte: 0 Blokken: 0 IO-blok: 4096 leeg normaal bestand Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ ferry) Toegang: 2015-01-01 00:00:00.000000000 +0100 Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 Veranderd: 2019-10-20 15:20:50.366163766 +0200 Ontstaan: - > If you want, you can add -exec to that find, but I'd only add that after > confirming the execution load is verified. > > Thanks, > Qu > >> >>> It would be much safer than btrfs check --repair. >>> >>> Thanks, >>> Qu >>> >>> >>>>> >>>>>> Thanks, >>>>>> Qu >>>>>> >>>>>>> >>>>>>>> This patchset adds such check and repair ability to btrfs-check, >>>>>>>> with a >>>>>>>> simple test image. >>>>>>>> >>>>>>>> Qu Wenruo (3): >>>>>>>> btrfs-progs: check/lowmem: Add check and repair for invalid >>>>>>>> inode >>>>>>>> generation >>>>>>>> btrfs-progs: check/original: Add check and repair for >>>>>>>> invalid inode >>>>>>>> generation >>>>>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>>>>> generation >>>>>>>> repair >>>>>>>> >>>>>>>> check/main.c | 50 +++++++++++- >>>>>>>> check/mode-lowmem.c | 76 >>>>>>>> ++++++++++++++++++ >>>>>>>> check/mode-original.h | 1 + >>>>>>>> .../.lowmem_repairable | 0 >>>>>>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 >>>>>>>> bytes >>>>>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>>>>> create mode 100644 >>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>>>>> create mode 100644 >>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >>>>> errors). >>>> >>>> It's really not recommended to run btrfs check, especially repair on the >>>> mounted fs, unless it's RO. >>>> >>>> A new transaction from kernel can easily screw up the repaired fs. >>>>> >>>>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts >>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and >>>>> v5.3 btrfs-progs. >>>>> >>>>> If you like I can share the log. Let me know. >>>>> >>>>> This issue can potentially cause a lot of grief. Our company server >>>>> runs >>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with >>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on each >>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel >>>>> ~5.6)? >>>> >>>> Yes. >>>> >>>>> >>>>> Do I understand correctly that this FTB is caused by more strict >>>>> checking of the fs by the kernel, while the tools to fix the detected >>>>> corruptions are not yet released? >>>> >>>> Yes. >>>> >>>> Thanks, >>>> Qu >>>> >>> >> ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 13:29 ` Ferry Toth @ 2019-10-20 14:11 ` Qu Wenruo 2019-10-20 14:24 ` Ferry Toth 0 siblings, 1 reply; 22+ messages in thread From: Qu Wenruo @ 2019-10-20 14:11 UTC (permalink / raw) To: Ferry Toth, Qu WenRuo, linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 6844 bytes --] On 2019/10/20 下午9:29, Ferry Toth wrote: > Op 20-10-2019 om 15:15 schreef Qu WenRuo: >> >> >> On 2019/10/20 下午9:04, Ferry Toth wrote: >>> Op 20-10-2019 om 02:51 schreef Qu Wenruo: >>>> >>>> >>>> On 2019/10/20 上午8:26, Qu Wenruo wrote: >>>>> >>>>> >>>>> On 2019/10/20 上午12:24, Ferry Toth wrote: >>>>>> Hi, >>>>>> >>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>>>>>> >>>>>>> >>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>>>>>> We have at least two user reports about bad inode generation makes >>>>>>>>> kernel reject the fs. >>>>>>>> >>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>>>>>> >>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / >>>>>>>> on /@ >>>>>>>> >>>>>>>> In initramfs I can try to mount but get something like >>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree >>>>>>>> failed >>>>>>>> >>>>>>>> Booting old kernel works just as before, no errors. >>>>>>>> >>>>>>>>> According to the creation time, the inode is created by some 2014 >>>>>>>>> kernel. >>>>>>>> >>>>>>>> How do I get the creation time? >>>>>>> >>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your >>>>>>> device> >>>>>> >>>>>> I just went back to the office to reboot to 5.3 and check the >>>>>> creation >>>>>> times and found they were 2013 - 2014. >>>>>> >>>>>>>> >>>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the >>>>>>>>> transid member) so the error persists until latest tree-checker >>>>>>>>> detects. >>>>>>>>> >>>>>>>>> Even the situation can be fixed by reverting back to older kernel >>>>>>>>> and >>>>>>>>> copying the offending dir/file to another inode and delete the >>>>>>>>> offending >>>>>>>>> one, it still should be done by btrfs-progs. >>>>>>>>> >>>>>>>> How to find the offending dir/file from the command line manually? >>>>>>> >>>>>>> # find <mount point> -inum <inode number> >>>>>> >>>>>> This works, thanks. >>>>>> >>>>>> But appears unpractical. After fix 2 files and reboot, I found 4 >>>>>> more, >>>>>> then 16, then I gave up. >>>> >>>> Another solution is use "find" to locate the files with creation time >>>> before 2015, and copy them to a new file, then replace the old file >>>> with >>>> the new file. >>> >>> Hmm. But how do I "find" by creation time (otime)? Do you have a >>> suggestion for this? >> >> $ touch -t 201501010000 /tmp/sample >> $ find <mnt> -not -cnewer /tmp/sample > > AFAIK this compares file modified date with status changed date. So, no > search for creation date. > > And stat /tmp/sample (sorry dutch lang output): > > ferry@ferry-quad:~$ stat /tmp/sample > Bestand: /tmp/sample > Grootte: 0 Blokken: 0 IO-blok: 4096 leeg > normaal bestand > Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 > Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ ferry) > Toegang: 2015-01-01 00:00:00.000000000 +0100 > Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 > Veranderd: 2019-10-20 15:20:50.366163766 +0200 > Ontstaan: - My bad, always got confused by o/a/c/mtime, as c really looks like *c* reation, so I always got confused between ctime and otime. Then considering not all fs supports otime, find doesn't support that. I guess it's only possible by other tools.... BTW, did you find any patterns in those existing offending inodes? I guess it would be faster than finding a tool supporting otime search. Thanks, Qu > > >> If you want, you can add -exec to that find, but I'd only add that after >> confirming the execution load is verified. >> >> Thanks, >> Qu >> >>> >>>> It would be much safer than btrfs check --repair. >>>> >>>> Thanks, >>>> Qu >>>> >>>> >>>>>> >>>>>>> Thanks, >>>>>>> Qu >>>>>>> >>>>>>>> >>>>>>>>> This patchset adds such check and repair ability to btrfs-check, >>>>>>>>> with a >>>>>>>>> simple test image. >>>>>>>>> >>>>>>>>> Qu Wenruo (3): >>>>>>>>> btrfs-progs: check/lowmem: Add check and repair for invalid >>>>>>>>> inode >>>>>>>>> generation >>>>>>>>> btrfs-progs: check/original: Add check and repair for >>>>>>>>> invalid inode >>>>>>>>> generation >>>>>>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>>>>>> generation >>>>>>>>> repair >>>>>>>>> >>>>>>>>> check/main.c | 50 >>>>>>>>> +++++++++++- >>>>>>>>> check/mode-lowmem.c | 76 >>>>>>>>> ++++++++++++++++++ >>>>>>>>> check/mode-original.h | 1 + >>>>>>>>> .../.lowmem_repairable | 0 >>>>>>>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 >>>>>>>>> bytes >>>>>>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>>>>>> create mode 100644 >>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>>>>>> create mode 100644 >>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >>>>>> errors). >>>>> >>>>> It's really not recommended to run btrfs check, especially repair >>>>> on the >>>>> mounted fs, unless it's RO. >>>>> >>>>> A new transaction from kernel can easily screw up the repaired fs. >>>>>> >>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted >>>>>> distorts >>>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and >>>>>> v5.3 btrfs-progs. >>>>>> >>>>>> If you like I can share the log. Let me know. >>>>>> >>>>>> This issue can potentially cause a lot of grief. Our company server >>>>>> runs >>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with >>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on >>>>>> each >>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel >>>>>> ~5.6)? >>>>> >>>>> Yes. >>>>> >>>>>> >>>>>> Do I understand correctly that this FTB is caused by more strict >>>>>> checking of the fs by the kernel, while the tools to fix the detected >>>>>> corruptions are not yet released? >>>>> >>>>> Yes. >>>>> >>>>> Thanks, >>>>> Qu >>>>> >>>> >>> > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 14:11 ` Qu Wenruo @ 2019-10-20 14:24 ` Ferry Toth 2019-10-21 16:01 ` Ferry Toth 0 siblings, 1 reply; 22+ messages in thread From: Ferry Toth @ 2019-10-20 14:24 UTC (permalink / raw) To: Qu Wenruo, Qu WenRuo, linux-btrfs Op 20-10-2019 om 16:11 schreef Qu Wenruo: > > > On 2019/10/20 下午9:29, Ferry Toth wrote: >> Op 20-10-2019 om 15:15 schreef Qu WenRuo: >>> >>> >>> On 2019/10/20 下午9:04, Ferry Toth wrote: >>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo: >>>>> >>>>> >>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote: >>>>>> >>>>>> >>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>>>>>>> >>>>>>>> >>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>>>>>>> We have at least two user reports about bad inode generation makes >>>>>>>>>> kernel reject the fs. >>>>>>>>> >>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>>>>>>> >>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / >>>>>>>>> on /@ >>>>>>>>> >>>>>>>>> In initramfs I can try to mount but get something like >>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree >>>>>>>>> failed >>>>>>>>> >>>>>>>>> Booting old kernel works just as before, no errors. >>>>>>>>> >>>>>>>>>> According to the creation time, the inode is created by some 2014 >>>>>>>>>> kernel. >>>>>>>>> >>>>>>>>> How do I get the creation time? >>>>>>>> >>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your >>>>>>>> device> >>>>>>> >>>>>>> I just went back to the office to reboot to 5.3 and check the >>>>>>> creation >>>>>>> times and found they were 2013 - 2014. >>>>>>> >>>>>>>>> >>>>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the >>>>>>>>>> transid member) so the error persists until latest tree-checker >>>>>>>>>> detects. >>>>>>>>>> >>>>>>>>>> Even the situation can be fixed by reverting back to older kernel >>>>>>>>>> and >>>>>>>>>> copying the offending dir/file to another inode and delete the >>>>>>>>>> offending >>>>>>>>>> one, it still should be done by btrfs-progs. >>>>>>>>>> >>>>>>>>> How to find the offending dir/file from the command line manually? >>>>>>>> >>>>>>>> # find <mount point> -inum <inode number> >>>>>>> >>>>>>> This works, thanks. >>>>>>> >>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4 >>>>>>> more, >>>>>>> then 16, then I gave up. >>>>> >>>>> Another solution is use "find" to locate the files with creation time >>>>> before 2015, and copy them to a new file, then replace the old file >>>>> with >>>>> the new file. >>>> >>>> Hmm. But how do I "find" by creation time (otime)? Do you have a >>>> suggestion for this? >>> >>> $ touch -t 201501010000 /tmp/sample >>> $ find <mnt> -not -cnewer /tmp/sample >> >> AFAIK this compares file modified date with status changed date. So, no >> search for creation date. >> >> And stat /tmp/sample (sorry dutch lang output): >> >> ferry@ferry-quad:~$ stat /tmp/sample >> Bestand: /tmp/sample >> Grootte: 0 Blokken: 0 IO-blok: 4096 leeg >> normaal bestand >> Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 >> Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ ferry) >> Toegang: 2015-01-01 00:00:00.000000000 +0100 >> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 >> Veranderd: 2019-10-20 15:20:50.366163766 +0200 >> Ontstaan: - > > My bad, always got confused by o/a/c/mtime, as c really looks like *c* > reation, so I always got confused between ctime and otime. > > Then considering not all fs supports otime, find doesn't support that. > I guess it's only possible by other tools.... New stat will support it, but not in Ubuntu 19.10. We did find this: https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.c and are trying to work out a script that will gzip all files with creation < 2015. Then we can unzip again and overwrite. > BTW, did you find any patterns in those existing offending inodes? > I guess it would be faster than finding a tool supporting otime search. I didn't see any logic. A mix of logs, cached files, journal files etc. + a .kde directory in the / > Thanks, > Qu > >> >> >>> If you want, you can add -exec to that find, but I'd only add that after >>> confirming the execution load is verified. >>> >>> Thanks, >>> Qu >>> >>>> >>>>> It would be much safer than btrfs check --repair. >>>>> >>>>> Thanks, >>>>> Qu >>>>> >>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> Qu >>>>>>>> >>>>>>>>> >>>>>>>>>> This patchset adds such check and repair ability to btrfs-check, >>>>>>>>>> with a >>>>>>>>>> simple test image. >>>>>>>>>> >>>>>>>>>> Qu Wenruo (3): >>>>>>>>>> btrfs-progs: check/lowmem: Add check and repair for invalid >>>>>>>>>> inode >>>>>>>>>> generation >>>>>>>>>> btrfs-progs: check/original: Add check and repair for >>>>>>>>>> invalid inode >>>>>>>>>> generation >>>>>>>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>>>>>>> generation >>>>>>>>>> repair >>>>>>>>>> >>>>>>>>>> check/main.c | 50 >>>>>>>>>> +++++++++++- >>>>>>>>>> check/mode-lowmem.c | 76 >>>>>>>>>> ++++++++++++++++++ >>>>>>>>>> check/mode-original.h | 1 + >>>>>>>>>> .../.lowmem_repairable | 0 >>>>>>>>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 >>>>>>>>>> bytes >>>>>>>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>>>>>>> create mode 100644 >>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>>>>>>> create mode 100644 >>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >>>>>>> errors). >>>>>> >>>>>> It's really not recommended to run btrfs check, especially repair >>>>>> on the >>>>>> mounted fs, unless it's RO. >>>>>> >>>>>> A new transaction from kernel can easily screw up the repaired fs. >>>>>>> >>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted >>>>>>> distorts >>>>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and >>>>>>> v5.3 btrfs-progs. >>>>>>> >>>>>>> If you like I can share the log. Let me know. >>>>>>> >>>>>>> This issue can potentially cause a lot of grief. Our company server >>>>>>> runs >>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with >>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on >>>>>>> each >>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel >>>>>>> ~5.6)? >>>>>> >>>>>> Yes. >>>>>> >>>>>>> >>>>>>> Do I understand correctly that this FTB is caused by more strict >>>>>>> checking of the fs by the kernel, while the tools to fix the detected >>>>>>> corruptions are not yet released? >>>>>> >>>>>> Yes. >>>>>> >>>>>> Thanks, >>>>>> Qu >>>>>> >>>>> >>>> >> > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 14:24 ` Ferry Toth @ 2019-10-21 16:01 ` Ferry Toth 0 siblings, 0 replies; 22+ messages in thread From: Ferry Toth @ 2019-10-21 16:01 UTC (permalink / raw) To: Qu Wenruo, Qu WenRuo, linux-btrfs Op 20-10-2019 om 16:24 schreef Ferry Toth: > Op 20-10-2019 om 16:11 schreef Qu Wenruo: >> >> >> On 2019/10/20 下午9:29, Ferry Toth wrote: >>> Op 20-10-2019 om 15:15 schreef Qu WenRuo: >>>> >>>> >>>> On 2019/10/20 下午9:04, Ferry Toth wrote: >>>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo: >>>>>> >>>>>> >>>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote: >>>>>>> >>>>>>> >>>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>>>>>>>> We have at least two user reports about bad inode generation >>>>>>>>>>> makes >>>>>>>>>>> kernel reject the fs. >>>>>>>>>> >>>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> >>>>>>>>>> 19.10 so >>>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>>>>>>>> >>>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / >>>>>>>>>> on /@ >>>>>>>>>> >>>>>>>>>> In initramfs I can try to mount but get something like >>>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree >>>>>>>>>> failed >>>>>>>>>> >>>>>>>>>> Booting old kernel works just as before, no errors. >>>>>>>>>> >>>>>>>>>>> According to the creation time, the inode is created by some >>>>>>>>>>> 2014 >>>>>>>>>>> kernel. >>>>>>>>>> >>>>>>>>>> How do I get the creation time? >>>>>>>>> >>>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your >>>>>>>>> device> >>>>>>>> >>>>>>>> I just went back to the office to reboot to 5.3 and check the >>>>>>>> creation >>>>>>>> times and found they were 2013 - 2014. >>>>>>>> >>>>>>>>>> >>>>>>>>>>> And the generation member of INODE_ITEM is not updated >>>>>>>>>>> (unlike the >>>>>>>>>>> transid member) so the error persists until latest tree-checker >>>>>>>>>>> detects. >>>>>>>>>>> >>>>>>>>>>> Even the situation can be fixed by reverting back to older >>>>>>>>>>> kernel >>>>>>>>>>> and >>>>>>>>>>> copying the offending dir/file to another inode and delete the >>>>>>>>>>> offending >>>>>>>>>>> one, it still should be done by btrfs-progs. >>>>>>>>>>> >>>>>>>>>> How to find the offending dir/file from the command line >>>>>>>>>> manually? >>>>>>>>> >>>>>>>>> # find <mount point> -inum <inode number> >>>>>>>> >>>>>>>> This works, thanks. >>>>>>>> >>>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4 >>>>>>>> more, >>>>>>>> then 16, then I gave up. >>>>>> >>>>>> Another solution is use "find" to locate the files with creation time >>>>>> before 2015, and copy them to a new file, then replace the old file >>>>>> with >>>>>> the new file. >>>>> >>>>> Hmm. But how do I "find" by creation time (otime)? Do you have a >>>>> suggestion for this? >>>> >>>> $ touch -t 201501010000 /tmp/sample >>>> $ find <mnt> -not -cnewer /tmp/sample >>> >>> AFAIK this compares file modified date with status changed date. So, no >>> search for creation date. >>> >>> And stat /tmp/sample (sorry dutch lang output): >>> >>> ferry@ferry-quad:~$ stat /tmp/sample >>> Bestand: /tmp/sample >>> Grootte: 0 Blokken: 0 IO-blok: 4096 leeg >>> normaal bestand >>> Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 >>> Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ >>> ferry) >>> Toegang: 2015-01-01 00:00:00.000000000 +0100 >>> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 >>> Veranderd: 2019-10-20 15:20:50.366163766 +0200 >>> Ontstaan: - >> >> My bad, always got confused by o/a/c/mtime, as c really looks like *c* >> reation, so I always got confused between ctime and otime. >> >> Then considering not all fs supports otime, find doesn't support that. >> I guess it's only possible by other tools.... > > New stat will support it, but not in Ubuntu 19.10. We did find this: > https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.c > > and are trying to work out a script that will gzip all files with > creation < 2015. Then we can unzip again and overwrite. So we combined find + test-statx + awk to produce a list of files with creation date 1970 that are then fed to tar. Finally we untar that to overwrite the original files. I found the following: - the problematic files are both ordinary and symlinks - they all seem to have the same inode generation - all had no creation date (so were set to jan 1970) - not all files without creation date were problematic (only ~10%) - modified dates were 2013/2014 afaikt The procedure keeps the modified date, but sets creation date to today, creates a new inode and fixes the error. >> BTW, did you find any patterns in those existing offending inodes? >> I guess it would be faster than finding a tool supporting otime search. > > I didn't see any logic. A mix of logs, cached files, journal files etc. > + a .kde directory in the / > >> Thanks, >> Qu >> >>> >>> >>>> If you want, you can add -exec to that find, but I'd only add that >>>> after >>>> confirming the execution load is verified. >>>> >>>> Thanks, >>>> Qu >>>> >>>>> >>>>>> It would be much safer than btrfs check --repair. >>>>>> >>>>>> Thanks, >>>>>> Qu >>>>>> >>>>>> >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Qu >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> This patchset adds such check and repair ability to btrfs-check, >>>>>>>>>>> with a >>>>>>>>>>> simple test image. >>>>>>>>>>> >>>>>>>>>>> Qu Wenruo (3): >>>>>>>>>>> btrfs-progs: check/lowmem: Add check and repair for >>>>>>>>>>> invalid >>>>>>>>>>> inode >>>>>>>>>>> generation >>>>>>>>>>> btrfs-progs: check/original: Add check and repair for >>>>>>>>>>> invalid inode >>>>>>>>>>> generation >>>>>>>>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>>>>>>>> generation >>>>>>>>>>> repair >>>>>>>>>>> >>>>>>>>>>> check/main.c | 50 >>>>>>>>>>> +++++++++++- >>>>>>>>>>> check/mode-lowmem.c | 76 >>>>>>>>>>> ++++++++++++++++++ >>>>>>>>>>> check/mode-original.h | 1 + >>>>>>>>>>> .../.lowmem_repairable | 0 >>>>>>>>>>> .../bad_inode_geneartion.img.xz | Bin 0 >>>>>>>>>>> -> 2012 >>>>>>>>>>> bytes >>>>>>>>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>>>>>>>> create mode 100644 >>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>>>>>>>> create mode 100644 >>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >>>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >>>>>>>> errors). >>>>>>> >>>>>>> It's really not recommended to run btrfs check, especially repair >>>>>>> on the >>>>>>> mounted fs, unless it's RO. >>>>>>> >>>>>>> A new transaction from kernel can easily screw up the repaired fs. >>>>>>>> >>>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted >>>>>>>> distorts >>>>>>>> the results? Else I'm going to need a live usb with a v5.3 >>>>>>>> kernel and >>>>>>>> v5.3 btrfs-progs. >>>>>>>> >>>>>>>> If you like I can share the log. Let me know. >>>>>>>> >>>>>>>> This issue can potentially cause a lot of grief. Our company server >>>>>>>> runs >>>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs >>>>>>>> with >>>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on >>>>>>>> each >>>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel >>>>>>>> ~5.6)? >>>>>>> >>>>>>> Yes. >>>>>>> >>>>>>>> >>>>>>>> Do I understand correctly that this FTB is caused by more strict >>>>>>>> checking of the fs by the kernel, while the tools to fix the >>>>>>>> detected >>>>>>>> corruptions are not yet released? >>>>>>> >>>>>>> Yes. >>>>>>> >>>>>>> Thanks, >>>>>>> Qu >>>>>>> >>>>>> >>>>> >>> >> > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation 2019-10-20 0:26 ` Qu Wenruo 2019-10-20 0:51 ` Qu Wenruo @ 2019-10-20 11:50 ` Ferry Toth 1 sibling, 0 replies; 22+ messages in thread From: Ferry Toth @ 2019-10-20 11:50 UTC (permalink / raw) To: Qu Wenruo, Qu WenRuo, linux-btrfs Op 20-10-2019 om 02:26 schreef Qu Wenruo: > > > On 2019/10/20 上午12:24, Ferry Toth wrote: >> Hi, >> >> Op 19-10-2019 om 01:50 schreef Qu WenRuo: >>> >>> >>> On 2019/10/19 上午4:32, Ferry Toth wrote: >>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo: >>>>> We have at least two user reports about bad inode generation makes >>>>> kernel reject the fs. >>>> >>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so >>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too). >>>> >>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@ >>>> >>>> In initramfs I can try to mount but get something like >>>> btrfs critical corrupt leaf invalid inode generation open_ctree failed >>>> >>>> Booting old kernel works just as before, no errors. >>>> >>>>> According to the creation time, the inode is created by some 2014 >>>>> kernel. >>>> >>>> How do I get the creation time? >>> >>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device> >> >> I just went back to the office to reboot to 5.3 and check the creation >> times and found they were 2013 - 2014. >> >>>> >>>>> And the generation member of INODE_ITEM is not updated (unlike the >>>>> transid member) so the error persists until latest tree-checker >>>>> detects. >>>>> >>>>> Even the situation can be fixed by reverting back to older kernel and >>>>> copying the offending dir/file to another inode and delete the >>>>> offending >>>>> one, it still should be done by btrfs-progs. >>>>> >>>> How to find the offending dir/file from the command line manually? >>> >>> # find <mount point> -inum <inode number> >> >> This works, thanks. >> >> But appears unpractical. After fix 2 files and reboot, I found 4 more, >> then 16, then I gave up. >> >>> Thanks, >>> Qu >>> >>>> >>>>> This patchset adds such check and repair ability to btrfs-check, with a >>>>> simple test image. >>>>> >>>>> Qu Wenruo (3): >>>>> btrfs-progs: check/lowmem: Add check and repair for invalid inode >>>>> generation >>>>> btrfs-progs: check/original: Add check and repair for invalid inode >>>>> generation >>>>> btrfs-progs: fsck-tests: Add test image for invalid inode >>>>> generation >>>>> repair >>>>> >>>>> check/main.c | 50 +++++++++++- >>>>> check/mode-lowmem.c | 76 >>>>> ++++++++++++++++++ >>>>> check/mode-original.h | 1 + >>>>> .../.lowmem_repairable | 0 >>>>> .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes >>>>> 5 files changed, 126 insertions(+), 1 deletion(-) >>>>> create mode 100644 >>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable >>>>> create mode 100644 >>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz >>>>> >>>> >> >> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my >> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 >> errors). > > It's really not recommended to run btrfs check, especially repair on the > mounted fs, unless it's RO. Yes, I know. As the fs is mounted btrfs refuses to repair it. So I only forced to skip the mount check, but no repair has been done, only the check. > A new transaction from kernel can easily screw up the repaired fs. >> >> I'm not sure if using the v5.0 kernel and/or checking mounted distorts >> the results? Else I'm going to need a live usb with a v5.3 kernel and >> v5.3 btrfs-progs. >> >> If you like I can share the log. Let me know. >> >> This issue can potentially cause a lot of grief. Our company server runs >> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with >> ~100 snapshots. I guess the problematic inodes need to be fixed on each >> snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)? > > Yes. > >> >> Do I understand correctly that this FTB is caused by more strict >> checking of the fs by the kernel, while the tools to fix the detected >> corruptions are not yet released? > > Yes. > > Thanks, > Qu > ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2019-10-21 16:08 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-24 8:11 [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Qu Wenruo 2019-09-24 8:11 ` [PATCH 1/3] btrfs-progs: check/lowmem: " Qu Wenruo 2019-09-30 11:36 ` Nikolay Borisov 2019-09-30 12:24 ` Qu Wenruo 2019-09-30 13:34 ` Nikolay Borisov 2019-09-30 14:05 ` Qu Wenruo 2019-09-24 8:11 ` [PATCH 2/3] btrfs-progs: check/original: " Qu Wenruo 2019-09-30 8:41 ` Nikolay Borisov 2019-09-30 9:00 ` Qu Wenruo 2019-09-24 8:11 ` [PATCH 3/3] btrfs-progs: fsck-tests: Add test image for invalid inode generation repair Qu Wenruo 2019-10-18 20:32 ` [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation Ferry Toth 2019-10-18 23:50 ` Qu WenRuo 2019-10-19 16:24 ` Ferry Toth 2019-10-20 0:26 ` Qu Wenruo 2019-10-20 0:51 ` Qu Wenruo 2019-10-20 13:04 ` Ferry Toth 2019-10-20 13:15 ` Qu WenRuo 2019-10-20 13:29 ` Ferry Toth 2019-10-20 14:11 ` Qu Wenruo 2019-10-20 14:24 ` Ferry Toth 2019-10-21 16:01 ` Ferry Toth 2019-10-20 11:50 ` Ferry Toth
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.