All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] btrfs-progs: Skip device tree when we failed to read it
@ 2019-12-06  3:44 Qu Wenruo
  2019-12-06  6:12 ` Anand Jain
  0 siblings, 1 reply; 53+ messages in thread
From: Qu Wenruo @ 2019-12-06  3:44 UTC (permalink / raw)
  To: linux-btrfs; +Cc: Christian Wimmer

Device tree is one of the least important tree, it only contains:
- Device status
  Like various error count
- Device extents
  Only makes sense for chunk allocation and physical->logical map, and
  even for that only purpose, we can rebuild it easily from chunk tree.

So device tree even makes less sense compared to extent tree, while we
still can't skip it at btrfs-progs.

This makes restore less useful. So this patch will make device tree to
follow the same requirement for OPEN_CTREE_PARTIAL.

Reported-by: Christian Wimmer <telefonchris@icloud.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 disk-io.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/disk-io.c b/disk-io.c
index 659f8b93..22aa2a61 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -947,12 +947,10 @@ int btrfs_setup_all_roots(struct btrfs_fs_info *fs_info, u64 root_tree_bytenr,
 		return ret;
 	fs_info->extent_root->track_dirty = 1;
 
-	ret = find_and_setup_root(root, fs_info, BTRFS_DEV_TREE_OBJECTID,
-				  fs_info->dev_root);
-	if (ret) {
-		printk("Couldn't setup device tree\n");
-		return -EIO;
-	}
+	ret = setup_root_or_create_block(fs_info, flags, fs_info->dev_root,
+					 BTRFS_DEV_TREE_OBJECTID, "device");
+	if (ret)
+		return ret;
 	fs_info->dev_root->track_dirty = 1;
 
 	ret = setup_root_or_create_block(fs_info, flags, fs_info->csum_root,
-- 
2.24.0


^ permalink raw reply related	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-06  3:44 [PATCH] btrfs-progs: Skip device tree when we failed to read it Qu Wenruo
@ 2019-12-06  6:12 ` Anand Jain
  2019-12-06 15:50   ` Christian Wimmer
                     ` (2 more replies)
  0 siblings, 3 replies; 53+ messages in thread
From: Anand Jain @ 2019-12-06  6:12 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs; +Cc: Christian Wimmer

On 6/12/19 11:44 AM, Qu Wenruo wrote:
> Device tree is one of the least important tree, it only contains:
> - Device status
>    Like various error count
> - Device extents
>    Only makes sense for chunk allocation and physical->logical map, and
>    even for that only purpose, we can rebuild it easily from chunk tree.
> 
> So device tree even makes less sense compared to extent tree, while we
> still can't skip it at btrfs-progs.
> 
> This makes restore less useful. So this patch will make device tree to
> follow the same requirement for OPEN_CTREE_PARTIAL.
> 
> Reported-by: Christian Wimmer <telefonchris@icloud.com>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>   disk-io.c | 10 ++++------
>   1 file changed, 4 insertions(+), 6 deletions(-)
> 
> diff --git a/disk-io.c b/disk-io.c
> index 659f8b93..22aa2a61 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -947,12 +947,10 @@ int btrfs_setup_all_roots(struct btrfs_fs_info *fs_info, u64 root_tree_bytenr,
>   		return ret;
>   	fs_info->extent_root->track_dirty = 1;
>   
> -	ret = find_and_setup_root(root, fs_info, BTRFS_DEV_TREE_OBJECTID,
> -				  fs_info->dev_root);
> -	if (ret) {
> -		printk("Couldn't setup device tree\n");
> -		return -EIO;
> -	}
> +	ret = setup_root_or_create_block(fs_info, flags, fs_info->dev_root,
> +					 BTRFS_DEV_TREE_OBJECTID, "device");
> +	if (ret)
> +		return ret;
>   	fs_info->dev_root->track_dirty = 1;
>   
>   	ret = setup_root_or_create_block(fs_info, flags, fs_info->csum_root,
> 

Cool.
Reviewed-by: Anand Jain <anand.jain@oracle.com>

Thanks Anand


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-06  6:12 ` Anand Jain
@ 2019-12-06 15:50   ` Christian Wimmer
  2019-12-06 16:34   ` Christian Wimmer
       [not found]   ` <762365A0-8BDF-454B-ABA9-AB2F0C958106@icloud.com>
  2 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2019-12-06 15:50 UTC (permalink / raw)
  To: Anand Jain; +Cc: Qu Wenruo, linux-btrfs

Hi folks,

I could not succeed in patching the btrfs yet. 
Could you give me a hint of how to do?

a) how and where to download the source code?
b) in case that it is not self-explaining, how to compile?

BTW, I installed the latest Manjaro distribution (5.3.12-1-Manjaro) because I need copy/past functionality and so on.
btrfs version is v5.3.1
This is good enough?
Will your patch work against this version?

Thanks,

Chris



^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-06  6:12 ` Anand Jain
  2019-12-06 15:50   ` Christian Wimmer
@ 2019-12-06 16:34   ` Christian Wimmer
       [not found]   ` <762365A0-8BDF-454B-ABA9-AB2F0C958106@icloud.com>
  2 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2019-12-06 16:34 UTC (permalink / raw)
  To: Anand Jain; +Cc: Qu Wenruo, linux-btrfs

Hi Anand and Qu,

seems that all mails to you were not delivered because of the attachment or the extensive body.
How can I pass the information to you for analysis?

Regards,
Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
       [not found]   ` <762365A0-8BDF-454B-ABA9-AB2F0C958106@icloud.com>
@ 2019-12-07  1:16     ` Qu WenRuo
  2019-12-07  3:47       ` Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Qu WenRuo @ 2019-12-07  1:16 UTC (permalink / raw)
  To: Christian Wimmer, Anand Jain; +Cc: linux-btrfs



On 2019/12/7 上午12:33, Christian Wimmer wrote:
> Hi Anand and Qu,
> 
> sorry for the many emails (in case you get them, because all of them
> come back with complains about being non-conform).
> Here the “reduced” contents of the outputs. In case of doubts ask me for
> other lines.
> 
> I succeeded on downloading the latest btrfs-progs (v5.4) and applied
> your patch.
> Then I called again the commands that you suggested and I got a lot of
> output.
> 
> 
> t1: # btrfs ins dump-tree -t root /dev/sde1 2>&1
> t2: # btrfs ins dump-tree -t extent /dev/sde1 2>&1 >/dev/null
> t3: # btrfs ins dump-tree -t chunk /dev/sde1
> t4: # btrfs ins dump-tree -t 5 /dev/sde1 2>&1 >/dev/null

Chunk tree is good, so is root tree and extent tree.

You can go btrfs restore without problem. (Of course, need to use
patched version)

Thanks,
Qu

> 
> I hope you can read some useful information from it.
> 
> 
> Thanks,
> 
> Chris
> 
> Here comes the contents of the files:
> 
> t1:
> 
> btrfs-progs v5.4 
> root tree
> node 658046976 level 1 items 112 free 381 generation 6407 owner ROOT_TREE
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> key (EXTENT_TREE ROOT_ITEM 0) block 658096128 gen 6407
> key (276 EXTENT_DATA 0) block 645955584 gen 6406
> key (316 EXTENT_DATA 0) block 3542131507200 gen 3968
> key (367 INODE_ITEM 0) block 653115392 gen 6395
> key (428 INODE_ITEM 0) block 658243584 gen 6407
> key (489 EXTENT_DATA 0) block 658571264 gen 6407
> ...
> 
> key (7503 INODE_ITEM 0) block 5349895684096 gen 6316
> key (7562 INODE_ITEM 0) block 110690304 gen 6316
> key (7623 INODE_ITEM 0) block 115949568 gen 6316
> key (7683 INODE_ITEM 0) block 110837760 gen 6316
> key (7744 INODE_ITEM 0) block 5349742592000 gen 6315
> key (7803 EXTENT_DATA 0) block 2478564245504 gen 6233
> key (7865 INODE_ITEM 0) block 5349894946816 gen 6316
> key (7950 INODE_ITEM 0) block 106397696 gen 6316
> key (8009 EXTENT_DATA 0) block 513032192 gen 6343
> key (8060 ROOT_ITEM 6283) block 658391040 gen 6407
> key (FREE_SPACE UNTYPED 9694085120) block 658554880 gen 6407
> key (FREE_SPACE UNTYPED 273834573824) block 655966208 gen 6399
> key (FREE_SPACE UNTYPED 405904818176) block 543784960 gen 6328
> key (FREE_SPACE UNTYPED 538511933440) block 655032320 gen 6397
> key (FREE_SPACE UNTYPED 802652422144) block 165003264 gen 6316
> key (FREE_SPACE UNTYPED 1000220917760) block 648265728 gen 6391
> key (FREE_SPACE UNTYPED 1263824535552) block 510803968 gen 6327
> key (FREE_SPACE UNTYPED 1527428153344) block 514916352 gen 6334
> key (FREE_SPACE UNTYPED 1791568642048) block 152944640 gen 6316
> key (FREE_SPACE UNTYPED 2055172259840) block 543801344 gen 6328
> key (FREE_SPACE UNTYPED 2319312748544) block 648282112 gen 6391
> key (FREE_SPACE UNTYPED 2582916366336) block 510722048 gen 6327
> key (FREE_SPACE UNTYPED 2852962435072) block 654966784 gen 6395
> key (FREE_SPACE UNTYPED 3645383901184) block 648249344 gen 6391
> key (FREE_SPACE UNTYPED 4508135456768) block 654983168 gen 6395
> key (FREE_SPACE UNTYPED 5821321707520) block 164937728 gen 6316
> key (FREE_SPACE UNTYPED 6085462196224) block 164986880 gen 6316
> key (FREE_SPACE UNTYPED 6390404874240) block 164970496 gen 6316
> key (FREE_SPACE UNTYPED 6799500509184) block 546701312 gen 6329
> key (FREE_SPACE UNTYPED 7069009707008) block 5349895487488 gen 6316
> key (FREE_SPACE UNTYPED 7333150195712) block 164954112 gen 6316
> key (FREE_SPACE UNTYPED 7620913004544) block 5349894979584 gen 6316
> leaf 658096128 items 52 free space 8088 generation 6407 owner ROOT_TREE
> leaf 658096128 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439
> generation 6407 root_dirid 0 bytenr 641515520 level 2 refs 1
> lastsnap 0 byte_limit 0 bytes_used 42352640 flags 0x0(none)
> uuid 00000000-0000-0000-0000-000000000000
> drop key (0 UNKNOWN.0 0) level 0
> item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439
> generation 6316 root_dirid 0 bytenr 5349895454720 level 1 refs 1
> lastsnap 0 byte_limit 0 bytes_used 376832 flags 0x0(none)
> uuid 00000000-0000-0000-0000-000000000000
> drop key (0 UNKNOWN.0 0) level 0
> item 50 key (275 EXTENT_DATA 0) itemoff 9548 itemsize 53
> generation 3485 type 1 (regular)
> extent data disk byte 225507221504 nr 262144
> extent data offset 0 nr 262144 ram 262144
> extent compression 0 (none)
> item 51 key (276 INODE_ITEM 0) itemoff 9388 itemsize 160
> generation 28 transid 28 size 262144 nbytes 786432
> block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
> sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
> atime 0.0 (1969-12-31 21:00:00)
> ctime 1573845267.989069139 (2019-11-15 16:14:27)
> mtime 0.0 (1969-12-31 21:00:00)
> otime 0.0 (1969-12-31 21:00:00)
> leaf 645955584 items 105 free space 4234 generation 6406 owner ROOT_TREE
> leaf 645955584 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (276 EXTENT_DATA 0) itemoff 16230 itemsize 53
> generation 28 type 1 (regular)
> extent data disk byte 95582855168 nr 262144
> extent data offset 0 nr 262144 ram 262144
> extent compression 0 (none)
> item 1 key (277 INODE_ITEM 0) itemoff 16070 itemsize 160
> generation 3485 transid 3485 size 262144 nbytes 6553600
> block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
> sequence 25 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
> atime 0.0 (1969-12-31 21:00:00)
> ctime 1575072859.622521074 (2019-11-29 21:14:19)
> mtime 0.0 (1969-12-31 21:00:00)
> otime 0.0 (1969-12-31 21:00:00)
> item 2 key (277 EXTENT_DATA 0) itemoff 16017 itemsize 53
> generation 3485 type 1 (regular)
> extent data disk byte 225507745792 nr 262144
> extent data offset 0 nr 262144 ram 262144
> extent compression 0 (none)
> …
> 
> item 75 key (314 INODE_ITEM 0) itemoff 8189 itemsize 160
> generation 296 transid 296 size 262144 nbytes 4718592
> block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
> sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
> atime 0.0 (1969-12-31 21:00:00)
> ctime 1574111004.400410206 (2019-11-18 18:03:24)
> mtime 0.0 (1969-12-31 21:00:00)
> otime 0.0 (1969-12-31 21:00:00)
> item 76 key (314 EXTENT_DATA 0) itemoff 8136 itemsize 53
> generation 296 type 1 (regular)
> extent data disk byte 596492996608 nr 262144
> extent data offset 0 nr 262144 ram 262144
> extent compression 0 (none)
> item 77 key (315 ROOT_ITEM 0) itemoff 7697 itemsize 439
> generation 6406 root_dirid 256 bytenr 637386752 level 1 refs 1
> lastsnap 0 byte_limit 0 bytes_used 49152 flags 0x0(none)
> uuid 38795581-63fa-0d42-9dd0-94d31a02f202
> ctransid 6406 otransid 20 stransid 0 rtransid 0
> ctime 1575496848.689538973 (2019-12-04 19:00:48)
> otime 1573845050.57041096 (2019-11-15 16:10:50)
> drop key (0 UNKNOWN.0 0) level 0
> item 78 key (315 ROOT_BACKREF 5) itemoff 7669 itemsize 28
> root backref key dirid 256 sequence 4 name .snapshots
> item 79 key (315 ROOT_REF 6893) itemoff 7643 itemsize 26
> root ref key dirid 833 sequence 2 name snapshot
> item 80 key (315 ROOT_REF 6900) itemoff 7617 itemsize 26
> root ref key dirid 839 sequence 2 name snapshot
> item 81 key (315 ROOT_REF 7012) itemoff 7591 itemsize 26
> root ref key dirid 887 sequence 2 name snapshot
> item 82 key (315 ROOT_REF 7288) itemoff 7565 itemsize 26
> root ref key dirid 935 sequence 2 name snapshot
> item 83 key (315 ROOT_REF 7897) itemoff 7539 itemsize 26
> root ref key dirid 983 sequence 2 name snapshot
> item 84 key (315 ROOT_REF 8044) itemoff 7513 itemsize 26
> root ref key dirid 1031 sequence 2 name snapshot
> item 85 key (315 ROOT_REF 8046) itemoff 7487 itemsize 26
> root ref key dirid 1033 sequence 2 name snapshot
> item 86 key (315 ROOT_REF 8048) itemoff 7461 itemsize 26
> root ref key dirid 1035 sequence 2 name snapshot
> item 87 key (315 ROOT_REF 8050) itemoff 7435 itemsize 26
> root ref key dirid 1037 sequence 2 name snapshot
> item 88 key (315 ROOT_REF 8052) itemoff 7409 itemsize 26
> root ref key dirid 1039 sequence 2 name snapshot
> item 89 key (315 ROOT_REF 8054) itemoff 7383 itemsize 26
> root ref key dirid 1041 sequence 2 name snapshot
> item 90 key (315 ROOT_REF 8056) itemoff 7357 itemsize 26
> root ref key dirid 1043 sequence 2 name snapshot
> item 91 key (315 ROOT_REF 8058) itemoff 7331 itemsize 26
> root ref key dirid 1045 sequence 2 name snapshot
> item 92 key (315 ROOT_REF 8060) itemoff 7305 itemsize 26
> root ref key dirid 1047 sequence 2 name snapshot
> item 93 key (315 ROOT_REF 8062) itemoff 7279 itemsize 26
> root ref key dirid 1049 sequence 2 name snapshot
> item 94 key (315 ROOT_REF 8066) itemoff 7253 itemsize 26
> root ref key dirid 1051 sequence 2 name snapshot
> item 95 key (315 ROOT_REF 8069) itemoff 7227 itemsize 26
> root ref key dirid 1053 sequence 2 name snapshot
> item 96 key (315 ROOT_REF 8071) itemoff 7201 itemsize 26
> root ref key dirid 1055 sequence 2 name snapshot
> item 97 key (315 ROOT_REF 8073) itemoff 7175 itemsize 26
> root ref key dirid 1057 sequence 2 name snapshot
> item 98 key (315 ROOT_REF 8075) itemoff 7149 itemsize 26
> root ref key dirid 1059 sequence 2 name snapshot
> item 99 key (315 ROOT_REF 8077) itemoff 7123 itemsize 26
> root ref key dirid 1061 sequence 2 name snapshot
> item 100 key (315 ROOT_REF 8079) itemoff 7097 itemsize 26
> root ref key dirid 1063 sequence 2 name snapshot
> item 101 key (315 ROOT_REF 8081) itemoff 7071 itemsize 26
> root ref key dirid 1065 sequence 2 name snapshot
> item 102 key (315 ROOT_REF 8083) itemoff 7045 itemsize 26
> root ref key dirid 1067 sequence 2 name snapshot
> item 103 key (315 ROOT_REF 8085) itemoff 7019 itemsize 26
> root ref key dirid 1069 sequence 2 name snapshot
> item 104 key (316 INODE_ITEM 0) itemoff 6859 itemsize 160
> generation 23 transid 23 size 262144 nbytes 786432
> block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
> sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
> atime 0.0 (1969-12-31 21:00:00)
> ctime 1573845114.382223306 (2019-11-15 16:11:54)
> mtime 0.0 (1969-12-31 21:00:00)
> otime 0.0 (1969-12-31 21:00:00)
> 
> t2:
> 
> btrfs-progs v5.4 
> extent tree key (EXTENT_TREE ROOT_ITEM 0) 
> node 641515520 level 2 items 9 free 484 generation 6407 owner EXTENT_TREE
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> key (13631488 EXTENT_ITEM 53248) block 641695744 gen 6407
> key (1033535488 METADATA_ITEM 0) block 653606912 gen 6407
> key (518073761792 METADATA_ITEM 0) block 653672448 gen 6407
> key (1395894779904 EXTENT_ITEM 134217728) block 654147584 gen 6404
> key (1988204576768 METADATA_ITEM 0) block 654688256 gen 6407
> key (2478674034688 METADATA_ITEM 0) block 655851520 gen 6407
> key (2815430119424 EXTENT_ITEM 24576) block 656883712 gen 6407
> key (3542281109504 METADATA_ITEM 0) block 657539072 gen 6407
> key (4845643841536 METADATA_ITEM 0) block 657752064 gen 6407
> node 641695744 level 1 items 237 free 256 generation 6407 owner EXTENT_TREE
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> key (13631488 EXTENT_ITEM 53248) block 658227200 gen 6407
> key (32735232 METADATA_ITEM 0) block 542949376 gen 6328
> key (34930688 METADATA_ITEM 0) block 46350336 gen 6316
> key (37322752 METADATA_ITEM 0) block 196444160 gen 6318
> key (40992768 METADATA_ITEM 0) block 42123264 gen 6316
> key (44335104 METADATA_ITEM 0) block 640090112 gen 6391
> key (47235072 METADATA_ITEM 0) block 509968384 gen 6334
> ...
> key (997294080 METADATA_ITEM 0) block 655409152 gen 6405
> key (1001963520 METADATA_ITEM 0) block 646987776 gen 6393
> key (1004994560 METADATA_ITEM 0) block 647036928 gen 6393
> key (1009188864 METADATA_ITEM 0) block 4845625622528 gen 6314
> key (1013661696 METADATA_ITEM 0) block 5349887180800 gen 6316
> key (1018249216 METADATA_ITEM 0) block 5349897142272 gen 6316
> key (1021984768 METADATA_ITEM 0) block 544047104 gen 6357
> key (1032503296 METADATA_ITEM 0) block 5349576409088 gen 6315
> node 653606912 level 1 items 308 free 185 generation 6407 owner EXTENT_TREE
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> key (1033535488 METADATA_ITEM 0) block 647069696 gen 6393
> key (1037975552 METADATA_ITEM 0) block 5349576916992 gen 6315
> key (1041924096 METADATA_ITEM 0) block 5349576933376 gen 6315
> key (1046249472 METADATA_ITEM 0) block 647151616 gen 6393
> key (1050476544 METADATA_ITEM 0) block 647184384 gen 6393
> key (1054949376 METADATA_ITEM 0) block 1063862272 gen 5979
> key (1057112064 METADATA_ITEM 0) block 5349817122816 gen 6316
> key (1061715968 METADATA_ITEM 0) block 5349787877376 gen 6316
> key (1066090496 METADATA_ITEM 0) block 5349572984832 gen 6315
> key (1068990464 METADATA_ITEM 0) block 647299072 gen 6393
> key (1073545216 METADATA_ITEM 0) block 5349899354112 gen 6316
> key (1327049871360 EXTENT_DATA_REF 511140681634713929) block 653410304
> gen 6405
> key (1331346198528 EXTENT_ITEM 72044544) block 5349601558528 gen 6315
> key (1343281430528 EXTENT_ITEM 134217728) block 5349681512448 gen 6315
> key (1356300550144 EXTENT_ITEM 134217728) block 5349678972928 gen 6315
> key (1364622049280 EXTENT_ITEM 134217728) block 4845670170624 gen 6314
> key (1381667700736 EXTENT_ITEM 134217728) block 544456704 gen 6359
> node 654147584 level 1 items 260 free 233 generation 6404 owner EXTENT_TREE
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> key (1395894779904 EXTENT_ITEM 134217728) block 544636928 gen 6359
> key (1400592400384 EXTENT_ITEM 134217728) block 544735232 gen 6359
> key (1412135124992 EXTENT_ITEM 134217728) block 5349476745216 gen 6314
> key (1421933019136 EXTENT_ITEM 134217728) block 5349627396096 gen 6315
> key (1432383651840 EXTENT_ITEM 3014656) block 122847232 gen 6316
> key (1443139125248 EXTENT_ITEM 126976) block 111656960 gen 6316
> key (1446320312320 EXTENT_ITEM 39538688) block 540934144 gen 6328
> key (1455524798464 EXTENT_ITEM 39743488) block 5349813846016 gen 6316
> key (1465003573248 EXTENT_ITEM 39743488) block 122961920 gen 6316
> key (1483002085376 EXTENT_ITEM 24743936) block 5349926928384 gen 6316
> key (1492690505728 EXTENT_ITEM 134217728) block 112820224 gen 6316
> key (1501123149824 METADATA_ITEM 0) block 5349565579264 gen 6315
> key (1501126279168 METADATA_ITEM 0) block 5349651349504 gen 6315
> key (1501130489856 METADATA_ITEM 0) block 5349597364224 gen 6315
> key (1501134602240 METADATA_ITEM 0) block 5349481496576 gen 6314
> key (1501136322560 METADATA_ITEM 0) block 4845792608256 gen 6314
> key (1501140664320 METADATA_ITEM 0) block 642154496 gen 6391
> key (1501144612864 METADATA_ITEM 0) block 5349676105728 gen 6315
> key (1501149134848 METADATA_ITEM 0) block 648953856 gen 6393
> key (1501153607680 METADATA_ITEM 0) block 648970240 gen 6393
> key (1501158195200 METADATA_ITEM 0) block 648986624 gen 6393
> key (1501161848832 METADATA_ITEM 0) block 649003008 gen 6393
> key (1501165568000 METADATA_ITEM 0) block 649035776 gen 6393
> key (1501169238016 METADATA_ITEM 0) block 649052160 gen 6393
> key (7447752724480 EXTENT_ITEM 68128768) block 5349912002560 gen 6316
> key (7458971426816 EXTENT_ITEM 70864896) block 5349589204992 gen 6315
> key (7469765107712 EXTENT_ITEM 46231552) block 5349914918912 gen 6316
> key (7478524293120 EXTENT_ITEM 16805888) block 5349914951680 gen 6316
> key (7484347707392 EXTENT_ITEM 67518464) block 5349884985344 gen 6316
> key (7490480381952 EXTENT_ITEM 56569856) block 5349885132800 gen 6316
> key (7496477503488 EXTENT_ITEM 118550528) block 5349912100864 gen 6316
> key (7512185139200 EXTENT_ITEM 68734976) block 5349604687872 gen 6315
> key (7526497325056 EXTENT_ITEM 66912256) block 5349604737024 gen 6315
> key (7534305189888 EXTENT_ITEM 73601024) block 5349700141056 gen 6315
> key (7545428533248 EXTENT_ITEM 60219392) block 5349588762624 gen 6315
> key (7553610645504 EXTENT_ITEM 56569856) block 5349693652992 gen 6315
> key (7562050322432 EXTENT_ITEM 74514432) block 5349604917248 gen 6315
> key (7568299536384 EXTENT_ITEM 4096) block 5349604933632 gen 6315
> key (7576889589760 BLOCK_GROUP_ITEM 1073741824) block 5349588205568 gen 6315
> key (7586620178432 EXTENT_ITEM 128954368) block 5349588221952 gen 6315
> key (7592870580224 EXTENT_ITEM 67518464) block 5349589008384 gen 6315
> key (7596959027200 EXTENT_ITEM 67518464) block 5349589532672 gen 6315
> key (7604780417024 EXTENT_ITEM 21254144) block 5349912363008 gen 6316
> key (7616275439616 EXTENT_ITEM 71778304) block 5349912182784 gen 6316
> key (7622987653120 EXTENT_ITEM 59305984) block 5349912379392 gen 6316
> key (7632724164608 BLOCK_GROUP_ITEM 1073741824) block 2478555742208 gen 6233
> key (7643048140800 EXTENT_ITEM 4096) block 3305523232768 gen 6289
> key (7649904033792 BLOCK_GROUP_ITEM 1073741824) block 3305523249152 gen 6289
> key (7661401780224 EXTENT_ITEM 134217728) block 2737802133504 gen 6244
> key (7671378870272 EXTENT_ITEM 129560576) block 3305523265536 gen 6289
> key (7680984322048 EXTENT_ITEM 4096) block 3305523281920 gen 6289
> key (7693060820992 EXTENT_ITEM 70864896) block 2737827086336 gen 6245
> key (7704639602688 EXTENT_ITEM 217088) block 651837440 gen 6395
> leaf 658227200 items 149 free space 7122 generation 6407 owner EXTENT_TREE
> leaf 658227200 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (13631488 EXTENT_ITEM 53248) itemoff 16230 itemsize 53
> refs 1 gen 8 flags DATA
> extent data backref root 3301 objectid 22080 offset 0 count 1
> item 1 key (13631488 BLOCK_GROUP_ITEM 8388608) itemoff 16206 itemsize 24
> block group used 8388608 chunk_objectid 256 flags DATA
> item 2 key (13684736 EXTENT_ITEM 40960) itemoff 16153 itemsize 53
> refs 1 gen 8 flags DATA
> extent data backref root 3301 objectid 22213 offset 0 count 1
> item 3 key (13725696 EXTENT_ITEM 40960) itemoff 16100 itemsize 53
> refs 1 gen 8 flags DATA
> extent data backref root 3301 objectid 22214 offset 0 count 1
> item 4 key (13766656 EXTENT_ITEM 16384) itemoff 16047 itemsize 53
> refs 1 gen 8 flags DATA
> extent data backref root 3301 objectid 22221 offset 0 count 1
> item 5 key (13783040 EXTENT_ITEM 32768) itemoff 15994 itemsize 53
> refs 1 gen 8 flags DATA
> extent data backref root 3301 objectid 22222 offset 0 count 1
> item 6 key (13815808 EXTENT_ITEM 32768) itemoff 15941 itemsize 53
> refs 1 gen 8 flags DATA
> extent data backref root 3301 objectid 22223 offset 0 count 1
> item 7 key (13848576 EXTENT_ITEM 45056) itemoff 15888 itemsize 53
> refs 1 gen 8 flags DATA
> extent data backref root 3301 objectid 22247 offset 0 count 1
> item 8 key (13893632 EXTENT_ITEM 12288) itemoff 15835 itemsize 53
> refs 1 gen 9 flags DATA
> extent data backref root 3301 objectid 22321 offset 0 count 1
> item 9 key (13905920 EXTENT_ITEM 40960) itemoff 15782 itemsize 53
> refs 1 gen 9 flags DATA
> ...
> item 148 key (32702464 METADATA_ITEM 0) itemoff 10847 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 542949376 items 131 free space 8685 generation 6328 owner EXTENT_TREE
> leaf 542949376 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (32735232 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (32751616 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (32784384 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> ...
> item 130 key (34914304 METADATA_ITEM 0) itemoff 11960 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 46350336 items 128 free space 8859 generation 6316 owner EXTENT_TREE
> leaf 46350336 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (34930688 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (34947072 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (34963456 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (34979840 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (34996224 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (35012608 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 127 key (37306368 METADATA_ITEM 0) itemoff 12059 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 196444160 items 204 free space 4451 generation 6318 owner EXTENT_TREE
> leaf 196444160 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (37322752 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (37339136 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (37355520 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (37371904 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (37388288 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (37404672 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 9 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 203 key (40976384 METADATA_ITEM 0) itemoff 9551 itemsize 33
> refs 1 gen 6316 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 42123264 items 190 free space 5263 generation 6316 owner EXTENT_TREE
> leaf 42123264 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (40992768 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (41009152 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (41025536 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (41041920 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (41058304 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (41074688 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 6 key (41091072 METADATA_ITEM 0) itemoff 16052 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 7 key (41107456 METADATA_ITEM 0) itemoff 16019 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 8 key (41123840 METADATA_ITEM 0) itemoff 15986 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 189 key (44318720 METADATA_ITEM 0) itemoff 10013 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 640090112 items 160 free space 7003 generation 6391 owner EXTENT_TREE
> leaf 640090112 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (44335104 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (44351488 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (44367872 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 10 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 159 key (47218688 METADATA_ITEM 0) itemoff 11003 itemsize 33
> refs 1 gen 5364 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 509968384 items 221 free space 3465 generation 6334 owner EXTENT_TREE
> leaf 509968384 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (47235072 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (47251456 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (47267840 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (47284224 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 220 key (50921472 METADATA_ITEM 1) itemoff 8990 itemsize 33
> refs 1 gen 6316 flags TREE_BLOCK
> tree block skinny level 1
> tree block backref root CSUM_TREE
> leaf 495222784 items 203 free space 4509 generation 6330 owner EXTENT_TREE
> leaf 495222784 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (50937856 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (50954240 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (50970624 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (50987008 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (51003392 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (51019776 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 6 key (51036160 METADATA_ITEM 0) itemoff 16052 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 7 key (51052544 METADATA_ITEM 0) itemoff 16019 itemsize 33
> refs 1 gen 11 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 202 key (54820864 METADATA_ITEM 0) itemoff 9584 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 58785792 items 176 free space 6075 generation 6316 owner EXTENT_TREE
> leaf 58785792 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (54837248 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (54853632 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> ...
> item 175 key (57769984 METADATA_ITEM 0) itemoff 10475 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 511049728 items 165 free space 6713 generation 6333 owner EXTENT_TREE
> leaf 511049728 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (57819136 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (57835520 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> ...
> item 164 key (60637184 METADATA_ITEM 0) itemoff 10838 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 79200256 items 150 free space 7583 generation 6316 owner EXTENT_TREE
> leaf 79200256 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (60653568 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (60669952 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (60686336 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (60702720 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (60719104 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (60735488 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 70008832 items 169 free space 6481 generation 6316 owner EXTENT_TREE
> leaf 70008832 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (63176704 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (63193088 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (63209472 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (63225856 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (63242240 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (63258624 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 6 key (63275008 METADATA_ITEM 0) itemoff 16052 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 7 key (63291392 METADATA_ITEM 0) itemoff 16019 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 8 key (63307776 METADATA_ITEM 0) itemoff 15986 itemsize 33
> refs 1 gen 13 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 168 key (66256896 METADATA_ITEM 0) itemoff 10706 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 79560704 items 261 free space 1145 generation 6316 owner EXTENT_TREE
> leaf 79560704 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (66273280 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (66289664 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (66306048 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (66322432 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (66338816 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (66355200 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 6 key (66371584 METADATA_ITEM 0) itemoff 16052 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 7 key (66404352 METADATA_ITEM 0) itemoff 16019 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 8 key (66420736 METADATA_ITEM 0) itemoff 15986 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 9 key (66437120 METADATA_ITEM 0) itemoff 15953 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 10 key (66453504 METADATA_ITEM 0) itemoff 15920 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 11 key (66469888 METADATA_ITEM 0) itemoff 15887 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 12 key (66486272 METADATA_ITEM 0) itemoff 15854 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 13 key (66502656 METADATA_ITEM 0) itemoff 15821 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 14 key (66519040 METADATA_ITEM 0) itemoff 15788 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 260 key (70631424 METADATA_ITEM 0) itemoff 7670 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 271269888 items 240 free space 2363 generation 6320 owner EXTENT_TREE
> leaf 271269888 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (70647808 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (70664192 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (70680576 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (70696960 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 14 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> ...
> item 239 key (74956800 METADATA_ITEM 0) itemoff 8363 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 233046016 items 148 free space 7699 generation 6320 owner EXTENT_TREE
> leaf 233046016 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (74973184 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (74989568 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 2 key (75005952 METADATA_ITEM 0) itemoff 16184 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 3 key (75022336 METADATA_ITEM 0) itemoff 16151 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 4 key (75038720 METADATA_ITEM 0) itemoff 16118 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 5 key (75055104 METADATA_ITEM 0) itemoff 16085 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 6 key (75071488 METADATA_ITEM 0) itemoff 16052 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 7 key (75087872 METADATA_ITEM 0) itemoff 16019 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 8 key (75104256 METADATA_ITEM 0) itemoff 15986 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 9 key (75153408 METADATA_ITEM 0) itemoff 15953 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> item 146 key (77611008 METADATA_ITEM 0) itemoff 11432 itemsize 33
> refs 1 gen 6316 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 147 key (77627392 METADATA_ITEM 0) itemoff 11399 itemsize 33
> refs 1 gen 6316 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> leaf 495271936 items 169 free space 6481 generation 6330 owner EXTENT_TREE
> leaf 495271936 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (77643776 METADATA_ITEM 0) itemoff 16250 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 1 key (77660160 METADATA_ITEM 0) itemoff 16217 itemsize 33
> refs 1 gen 15 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> …
> 
> refs 1 gen 6316 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 167 key (80707584 METADATA_ITEM 0) itemoff 10739 itemsize 33
> refs 1 gen 6316 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> item 168 key (80723968 METADATA_ITEM 0) itemoff 10706 itemsize 33
> refs 1 gen 6316 flags TREE_BLOCK
> tree block skinny level 0
> tree block backref root CSUM_TREE
> total bytes 12884899790848
> bytes used 4681564540928
> uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> 
> 
> t3:
> 
> btrfs-progs v5.4 
> chunk tree
> node 22036480 level 1 items 34 free 459 generation 6316 owner CHUNK_TREE
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> key (DEV_ITEMS DEV_ITEM 1) block 22052864 gen 6316
> key (FIRST_CHUNK_TREE CHUNK_ITEM 162165424128) block 22183936 gen 84
> key (FIRST_CHUNK_TREE CHUNK_ITEM 328595406848) block 22495232 gen 3373
> key (FIRST_CHUNK_TREE CHUNK_ITEM 495025389568) block 22593536 gen 3373
> key (FIRST_CHUNK_TREE CHUNK_ITEM 659844759552) block 22609920 gen 3373
> key (FIRST_CHUNK_TREE CHUNK_ITEM 826274742272) block 22200320 gen 298
> key (FIRST_CHUNK_TREE CHUNK_ITEM 992704724992) block 22216704 gen 303
> key (FIRST_CHUNK_TREE CHUNK_ITEM 1157524094976) block 22069248 gen 346
> key (FIRST_CHUNK_TREE CHUNK_ITEM 1323954077696) block 22265856 gen 3378
> key (FIRST_CHUNK_TREE CHUNK_ITEM 1490384060416) block 22282240 gen 3378
> key (FIRST_CHUNK_TREE CHUNK_ITEM 1655203430400) block 22413312 gen 3378
> key (FIRST_CHUNK_TREE CHUNK_ITEM 1821633413120) block 22020096 gen 858
> key (FIRST_CHUNK_TREE CHUNK_ITEM 1988063395840) block 22315008 gen 865
> key (FIRST_CHUNK_TREE CHUNK_ITEM 2152882765824) block 22331392 gen 876
> key (FIRST_CHUNK_TREE CHUNK_ITEM 2319312748544) block 22347776 gen 887
> key (FIRST_CHUNK_TREE CHUNK_ITEM 2484132118528) block 22364160 gen 995
> key (FIRST_CHUNK_TREE CHUNK_ITEM 2650562101248) block 22380544 gen 1192
> key (FIRST_CHUNK_TREE CHUNK_ITEM 2821823922176) block 22151168 gen 3381
> key (FIRST_CHUNK_TREE CHUNK_ITEM 3469827112960) block 22396928 gen 3380
> key (FIRST_CHUNK_TREE CHUNK_ITEM 3487006982144) block 22462464 gen 3380
> key (FIRST_CHUNK_TREE CHUNK_ITEM 3651826352128) block 22167552 gen 3381
> key (FIRST_CHUNK_TREE CHUNK_ITEM 3819330076672) block 22528000 gen 3380
> key (FIRST_CHUNK_TREE CHUNK_ITEM 5597983408128) block 22511616 gen 2256
> key (FIRST_CHUNK_TREE CHUNK_ITEM 5797699387392) block 22249472 gen 3379
> key (FIRST_CHUNK_TREE CHUNK_ITEM 5964129370112) block 22626304 gen 3379
> key (FIRST_CHUNK_TREE CHUNK_ITEM 6130559352832) block 22544384 gen 4085
> key (FIRST_CHUNK_TREE CHUNK_ITEM 6313095462912) block 22478848 gen 4085
> key (FIRST_CHUNK_TREE CHUNK_ITEM 6550392406016) block 22233088 gen 4085
> key (FIRST_CHUNK_TREE CHUNK_ITEM 6716822388736) block 22429696 gen 4821
> key (FIRST_CHUNK_TREE CHUNK_ITEM 6944455655424) block 22560768 gen 6315
> key (FIRST_CHUNK_TREE CHUNK_ITEM 7115180605440) block 22446080 gen 6316
> key (FIRST_CHUNK_TREE CHUNK_ITEM 7281610588160) block 22577152 gen 6207
> key (FIRST_CHUNK_TREE CHUNK_ITEM 7469515407360) block 22134784 gen 6217
> key (FIRST_CHUNK_TREE CHUNK_ITEM 7555414753280) block 22102016 gen 6308
> leaf 22052864 items 154 free space 31 generation 6316 owner CHUNK_TREE
> leaf 22052864 flags 0x1(WRITTEN) backref revision 1
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
> devid 1 total_bytes 12884899790848 bytes_used 4981113487360
> io_align 4096 io_width 4096 sector_size 4096 type 0
> generation 0 start_offset 0 dev_group 0
> seek_speed 0 bandwidth 0
> uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> fsid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 13631488) itemoff 16105 itemsize 80
> length 8388608 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 0
> stripe 0 devid 1 offset 13631488
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15993 itemsize 112
> length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 0
> stripe 0 devid 1 offset 22020096
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> stripe 1 devid 1 offset 30408704
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15881 itemsize 112
> length 1073741824 owner 2 stripe_len 65536 type METADATA|DUP
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 0
> stripe 0 devid 1 offset 38797312
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> stripe 1 devid 1 offset 1112539136
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 1104150528) itemoff 15801
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 2186280960
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 2177892352) itemoff 15721
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 3260022784
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 3251634176) itemoff 15641
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 4333764608
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 4325376000) itemoff 15561
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 5407506432
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 5399117824) itemoff 15481
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 6481248256
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 6472859648) itemoff 15401
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 7554990080
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 10 key (FIRST_CHUNK_TREE CHUNK_ITEM 7546601472) itemoff 15321
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 8628731904
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 11 key (FIRST_CHUNK_TREE CHUNK_ITEM 8620343296) itemoff 15241
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 9702473728
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> item 12 key (FIRST_CHUNK_TREE CHUNK_ITEM 9694085120) itemoff 15161
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 10776215552
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> 
> …
> 
> item 154 key (FIRST_CHUNK_TREE CHUNK_ITEM 7721844736000) itemoff 3883
> itemsize 80
> length 1073741824 owner 2 stripe_len 65536 type DATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 4677258182656
> dev_uuid bf8b6461-d289-4b6d-a05a-7400a74fcc87
> 
> 
> t4:
> 
> btrfs-progs v5.4 
> fs tree key (FS_TREE ROOT_ITEM 0) 
> node 653901824 level 1 items 21 free 472 generation 6404 owner FS_TREE
> fs uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> chunk uuid 37a4bae8-ccc0-4f1e-965e-0994d8ae38d9
> key (256 INODE_ITEM 0) block 3541835317248 gen 6293
> key (65689 DIR_ITEM 1123567632) block 1501444014080 gen 1204
> key (65715 INODE_ITEM 0) block 1501444030464 gen 1204
> key (65715 DIR_ITEM 3537358747) block 1501444046848 gen 1204
> key (65717 DIR_ITEM 925474960) block 517652987904 gen 341
> key (65734 INODE_ITEM 0) block 517650530304 gen 341
> key (65749 EXTENT_DATA 0) block 517653004288 gen 341
> key (65783 EXTENT_DATA 0) block 517652955136 gen 341
> key (65820 INODE_ITEM 0) block 518052806656 gen 342
> key (65840 INODE_ITEM 0) block 1501444063232 gen 1204
> key (65877 DIR_ITEM 2138916007) block 1501444079616 gen 1204
> key (65897 DIR_ITEM 964150343) block 518052560896 gen 342
> key (65897 DIR_ITEM 4142044485) block 1501444096000 gen 1204
> key (65906 DIR_ITEM 1733445785) block 518052790272 gen 342
> key (65922 INODE_REF 65906) block 518054084608 gen 342
> key (65956 INODE_ITEM 0) block 518054100992 gen 342
> key (65974 EXTENT_DATA 0) block 517736955904 gen 343
> key (65990 INODE_REF 65897) block 517734727680 gen 343
> key (66040 INODE_ITEM 0) block 1501444112384 gen 1204
> key (66083 DIR_ITEM 1754417305) block 183844864 gen 3381
> key (66117 EXTENT_DATA 0) block 183812096 gen 3381
> total bytes 12884899790848
> bytes used 4681564540928
> uuid cca3d8e7-f4d2-4fe2-a80d-11663e3565fa
> 
> 
> 

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-07  1:16     ` Qu WenRuo
@ 2019-12-07  3:47       ` Christian Wimmer
  2019-12-07  4:31         ` Qu Wenruo
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2019-12-07  3:47 UTC (permalink / raw)
  To: Qu WenRuo; +Cc: Anand Jain, linux-btrfs

Hi Qu,


> Chunk tree is good, so is root tree and extent tree.
> 
> You can go btrfs restore without problem. (Of course, need to use
> patched version)
> 
> Thanks,
> Qu


Unfortunately I can not restore the contents:

# ./btrfs restore /dev/sdb1 test/
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
bad tree block 5349895454720, bytenr mismatch, want=5349895454720, have=14275350892879035392
WARNING: could not setup device tree, skipping it
checksum verify failed on 3541835317248 found 00000044 wanted 00000000
checksum verify failed on 3541835317248 found 00000061 wanted 0000001C
checksum verify failed on 3541835317248 found 00000061 wanted 0000001C
bad tree block 3541835317248, bytenr mismatch, want=3541835317248, have=18445340955138505899
Error searching -5
# 

What else can I try?

Thanks,

Chris



^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-07  3:47       ` Christian Wimmer
@ 2019-12-07  4:31         ` Qu Wenruo
  2019-12-07 13:03           ` Christian Wimmer
       [not found]           ` <9FB359ED-EAD4-41DD-B846-1422F2DC4242@icloud.com>
  0 siblings, 2 replies; 53+ messages in thread
From: Qu Wenruo @ 2019-12-07  4:31 UTC (permalink / raw)
  To: Christian Wimmer, Qu WenRuo; +Cc: Anand Jain, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 1657 bytes --]



On 2019/12/7 上午11:47, Christian Wimmer wrote:
> Hi Qu,
> 
> 
>> Chunk tree is good, so is root tree and extent tree.
>>
>> You can go btrfs restore without problem. (Of course, need to use
>> patched version)
>>
>> Thanks,
>> Qu
> 
> 
> Unfortunately I can not restore the contents:
> 
> # ./btrfs restore /dev/sdb1 test/
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> bad tree block 5349895454720, bytenr mismatch, want=5349895454720, have=14275350892879035392
> WARNING: could not setup device tree, skipping it
> checksum verify failed on 3541835317248 found 00000044 wanted 00000000
> checksum verify failed on 3541835317248 found 00000061 wanted 0000001C
> checksum verify failed on 3541835317248 found 00000061 wanted 0000001C
> bad tree block 3541835317248, bytenr mismatch, want=3541835317248, have=18445340955138505899

This is another tree block corrupted, not the one in device tree.

And unfortunately, it's the fs tree (from your previews reply), and
that's not a good news.

This proved one bad news, there are more corruptions than we thought.

BTW, do you have any subvolumes/snapshots?
Since the corruption is in fs tree, we may have chances for other trees.

You can try "btrfs restore -l <dev>" grab the numbers and pass it to
"btrfs restore -r <number> <dev>" to try to recover other
subvolumes/snapshot.

Thanks,
Qu

> Error searching -5
> # 
> 
> What else can I try?
> 
> Thanks,
> 
> Chris
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-07  4:31         ` Qu Wenruo
@ 2019-12-07 13:03           ` Christian Wimmer
  2019-12-07 14:10             ` Qu Wenruo
       [not found]           ` <9FB359ED-EAD4-41DD-B846-1422F2DC4242@icloud.com>
  1 sibling, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2019-12-07 13:03 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

Sorry for the repeated post, I am trying to pass the mail server...

Hi Qui, 

I tried what you said and I got at least some very small files out of the device!

Yes, I have sub volumes/snapshots.

There should be a subvolume called “projects” which I am interested in.
Inside this subvolume should be lots of snapshots.

Any chance to recover one of them?

Here the output of the restore command that you suggested:

# ./btrfs restore -l /dev/sdb1 
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
bad tree block 5349895454720, bytenr mismatch, want=5349895454720, have=14275350892879035392
WARNING: could not setup device tree, skipping it
 tree key (EXTENT_TREE ROOT_ITEM 0) 641515520 level 2
 tree key (DEV_TREE ROOT_ITEM 0) 5349895454720 level 1
 tree key (FS_TREE ROOT_ITEM 0) 653901824 level 1
 tree key (CSUM_TREE ROOT_ITEM 0) 658161664 level 3
 tree key (UUID_TREE ROOT_ITEM 0) 657014784 level 0
 tree key (315 ROOT_ITEM 0) 637386752 level 1
checksum verify failed on 3542131507200 found 0000008C wanted 00000000
checksum verify failed on 3542131507200 found 000000F5 wanted 00000000
checksum verify failed on 3542131507200 found 0000008C wanted 00000000
bad tree block 3542131507200, bytenr mismatch, want=3542131507200, have=14275350892879035392

And I tried:

# ./btrfs restore -r 315 -v -D /dev/sdb1 test/
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
bad tree block 5349895454720, bytenr mismatch, want=5349895454720, have=14275350892879035392
WARNING: could not setup device tree, skipping it
This is a dry-run, no files are going to be restored
Restoring test/1
Skipping snapshot snapshot
Skipping existing file test/1/info.xml
If you wish to overwrite use -o
Done searching /1
Restoring test/4
Skipping snapshot snapshot
Skipping existing file test/4/info.xml
Done searching /4
Restoring test/28
Skipping snapshot snapshot
Skipping existing file test/28/info.xml
Done searching /28
Restoring test/52
checksum verify failed on 3305202188288 found 0000009E wanted FFFFFFA9
checksum verify failed on 3305202188288 found 000000FA wanted 00000000
checksum verify failed on 3305202188288 found 0000009E wanted FFFFFFA9
bad tree block 3305202188288, bytenr mismatch, want=3305202188288, have=18446556327804403584
Error reading subvolume test/52/snapshot: 18446744073709551611
Error searching test/52/snapshot


Well, I got the very small files info.xml.
How can I get the rest? Any ideas?

Thanks a lot for your help,



Chris




^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-07 13:03           ` Christian Wimmer
@ 2019-12-07 14:10             ` Qu Wenruo
  2019-12-07 14:25               ` Christian Wimmer
  2019-12-07 16:44               ` Christian Wimmer
  0 siblings, 2 replies; 53+ messages in thread
From: Qu Wenruo @ 2019-12-07 14:10 UTC (permalink / raw)
  To: Christian Wimmer; +Cc: Qu WenRuo, Anand Jain, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 3367 bytes --]



On 2019/12/7 下午9:03, Christian Wimmer wrote:
> Sorry for the repeated post, I am trying to pass the mail server...
> 
> Hi Qui, 
> 
> I tried what you said and I got at least some very small files out of the device!
> 
> Yes, I have sub volumes/snapshots.
> 
> There should be a subvolume called “projects” which I am interested in.
> Inside this subvolume should be lots of snapshots.
> 
> Any chance to recover one of them?
> 
> Here the output of the restore command that you suggested:
> 
> # ./btrfs restore -l /dev/sdb1 
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> bad tree block 5349895454720, bytenr mismatch, want=5349895454720, have=14275350892879035392
> WARNING: could not setup device tree, skipping it
>  tree key (EXTENT_TREE ROOT_ITEM 0) 641515520 level 2
>  tree key (DEV_TREE ROOT_ITEM 0) 5349895454720 level 1
>  tree key (FS_TREE ROOT_ITEM 0) 653901824 level 1
>  tree key (CSUM_TREE ROOT_ITEM 0) 658161664 level 3
>  tree key (UUID_TREE ROOT_ITEM 0) 657014784 level 0
>  tree key (315 ROOT_ITEM 0) 637386752 level 1
> checksum verify failed on 3542131507200 found 0000008C wanted 00000000
> checksum verify failed on 3542131507200 found 000000F5 wanted 00000000
> checksum verify failed on 3542131507200 found 0000008C wanted 00000000
> bad tree block 3542131507200, bytenr mismatch, want=3542131507200, have=14275350892879035392
> 
> And I tried:
> 
> # ./btrfs restore -r 315 -v -D /dev/sdb1 test/
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> checksum verify failed on 5349895454720 found 000000A8 wanted 00000000
> bad tree block 5349895454720, bytenr mismatch, want=5349895454720, have=14275350892879035392
> WARNING: could not setup device tree, skipping it
> This is a dry-run, no files are going to be restored
> Restoring test/1
> Skipping snapshot snapshot
> Skipping existing file test/1/info.xml
> If you wish to overwrite use -o
> Done searching /1
> Restoring test/4
> Skipping snapshot snapshot
> Skipping existing file test/4/info.xml
> Done searching /4
> Restoring test/28
> Skipping snapshot snapshot
> Skipping existing file test/28/info.xml
> Done searching /28
> Restoring test/52
> checksum verify failed on 3305202188288 found 0000009E wanted FFFFFFA9
> checksum verify failed on 3305202188288 found 000000FA wanted 00000000
> checksum verify failed on 3305202188288 found 0000009E wanted FFFFFFA9
> bad tree block 3305202188288, bytenr mismatch, want=3305202188288, have=18446556327804403584
> Error reading subvolume test/52/snapshot: 18446744073709551611
> Error searching test/52/snapshot
> 
> 
> Well, I got the very small files info.xml.
> How can I get the rest? Any ideas?

I'm afraid there are too many corruptions in your filesystem.

From root tree to fs tree, all had corruptions.
Thus really hard to get recovered.

Just curious, how this happened? Btrfs itself shouldn't cause so many
random corruptions. Looks more like a hardware problem or random wipe.

Thanks,
Qu

> 
> Thanks a lot for your help,
> 
> 
> 
> Chris
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-07 14:10             ` Qu Wenruo
@ 2019-12-07 14:25               ` Christian Wimmer
  2019-12-07 16:44               ` Christian Wimmer
  1 sibling, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2019-12-07 14:25 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

ok, backup device did not work  as well.

Any ideas?

The organisation was like this:

mount /dev/sde1 /home/promise

inside /home/promise was a subvolume “projects” like:

/home/promise/projects

and inside /home/promise/projects was .snapshots with the snapshots of it.

Anything that I could recover would be fine (but this you know already).

Thanks,

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-07 14:10             ` Qu Wenruo
  2019-12-07 14:25               ` Christian Wimmer
@ 2019-12-07 16:44               ` Christian Wimmer
  2019-12-08  1:21                 ` Qu WenRuo
  1 sibling, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2019-12-07 16:44 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

Hi Qu,

I was reading about chunk-recover. Do you think this could be worth a try?

Is there any other command that can search for files that make sense to recover?

Regards,

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-07 16:44               ` Christian Wimmer
@ 2019-12-08  1:21                 ` Qu WenRuo
  2019-12-10 21:25                   ` Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Qu WenRuo @ 2019-12-08  1:21 UTC (permalink / raw)
  To: Christian Wimmer, Qu Wenruo; +Cc: Anand Jain, linux-btrfs



On 2019/12/8 上午12:44, Christian Wimmer wrote:
> Hi Qu,
> 
> I was reading about chunk-recover. Do you think this could be worth a try?

Nope, your chunk tree is good, so that makes no sense.

> 
> Is there any other command that can search for files that make sense to recover?

The only sane behavior here is to search the whole disk and grab
anything looks like a tree block, and then extract data from it.

This is not something supported by btrfs-progs yet, so really not much
more can be done here.

Thanks,
Qu

> 
> Regards,
> 
> Chris
> 

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-08  1:21                 ` Qu WenRuo
@ 2019-12-10 21:25                   ` Christian Wimmer
  2019-12-11  0:36                     ` Qu Wenruo
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2019-12-10 21:25 UTC (permalink / raw)
  To: Qu WenRuo; +Cc: Qu Wenruo, Anand Jain, linux-btrfs

Hi Qu and all others,

thanks a lot for your help and patience!

Unfortunately I could not get out any file from the arrive yet but I can survive.

I would like just one more answer from you. Do you think with the newest version of btrfs it would not have happened?
Should I update to the newest version?

I have many partitions with btrfs and I like them a lot. Very nice file system indeed but am I safe with the version that I have (4.19.1)?

BTW, you are welcome to suggest any command or try anything with my broken file system that I still have backed up in case that you want to experiment with it.

Thanks 

Chris


> On 7. Dec 2019, at 22:21, Qu WenRuo <wqu@suse.com> wrote:
> 
> 
> 
> On 2019/12/8 上午12:44, Christian Wimmer wrote:
>> Hi Qu,
>> 
>> I was reading about chunk-recover. Do you think this could be worth a try?
> 
> Nope, your chunk tree is good, so that makes no sense.
> 
>> 
>> Is there any other command that can search for files that make sense to recover?
> 
> The only sane behavior here is to search the whole disk and grab
> anything looks like a tree block, and then extract data from it.
> 
> This is not something supported by btrfs-progs yet, so really not much
> more can be done here.
> 
> Thanks,
> Qu
> 
>> 
>> Regards,
>> 
>> Chris
>> 


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-10 21:25                   ` Christian Wimmer
@ 2019-12-11  0:36                     ` Qu Wenruo
  2019-12-11 15:57                       ` Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Qu Wenruo @ 2019-12-11  0:36 UTC (permalink / raw)
  To: Christian Wimmer, Qu WenRuo; +Cc: Anand Jain, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 2280 bytes --]



On 2019/12/11 上午5:25, Christian Wimmer wrote:
> Hi Qu and all others,
> 
> thanks a lot for your help and patience!
> 
> Unfortunately I could not get out any file from the arrive yet but I can survive.
> 
> I would like just one more answer from you. Do you think with the newest version of btrfs it would not have happened?

From the result, it looks like either btrfs is doing wrong trim, or the
storage stack below (including the Parallels, the apple fs, and the
apple drivers) is blowing up data.

In the latter case, it doesn't matter whatever kernel version you're
using, if it happens, it will take your data along with it.

But for the former case, newer kernel has improved trim check to prevent
bad trim, so at least newer kernel is a little more safer.

> Should I update to the newest version?

Not always the newest, although we're trying our best to prevent bugs,
but sometimes we still have some bugs sneaking into latest kernel.

> 
> I have many partitions with btrfs and I like them a lot. Very nice file system indeed but am I safe with the version that I have (4.19.1)?

Can't say it's unsafe, since SUSE has all necessary backports and quite
some customers are using (testing) it.
As long as you're using the latest SUSE updates, it should be safe and
all found bugs should have fixes backported.

Thanks,
Qu
> 
> BTW, you are welcome to suggest any command or try anything with my broken file system that I still have backed up in case that you want to experiment with it.
> 
> Thanks 
> 
> Chris
> 
> 
>> On 7. Dec 2019, at 22:21, Qu WenRuo <wqu@suse.com> wrote:
>>
>>
>>
>> On 2019/12/8 上午12:44, Christian Wimmer wrote:
>>> Hi Qu,
>>>
>>> I was reading about chunk-recover. Do you think this could be worth a try?
>>
>> Nope, your chunk tree is good, so that makes no sense.
>>
>>>
>>> Is there any other command that can search for files that make sense to recover?
>>
>> The only sane behavior here is to search the whole disk and grab
>> anything looks like a tree block, and then extract data from it.
>>
>> This is not something supported by btrfs-progs yet, so really not much
>> more can be done here.
>>
>> Thanks,
>> Qu
>>
>>>
>>> Regards,
>>>
>>> Chris
>>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: [PATCH] btrfs-progs: Skip device tree when we failed to read it
  2019-12-11  0:36                     ` Qu Wenruo
@ 2019-12-11 15:57                       ` Christian Wimmer
  0 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2019-12-11 15:57 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

Perfect!

Thanks a lot!

Chris


> On 10. Dec 2019, at 21:36, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> 
> 
> On 2019/12/11 上午5:25, Christian Wimmer wrote:
>> Hi Qu and all others,
>> 
>> thanks a lot for your help and patience!
>> 
>> Unfortunately I could not get out any file from the arrive yet but I can survive.
>> 
>> I would like just one more answer from you. Do you think with the newest version of btrfs it would not have happened?
> 
> From the result, it looks like either btrfs is doing wrong trim, or the
> storage stack below (including the Parallels, the apple fs, and the
> apple drivers) is blowing up data.
> 
> In the latter case, it doesn't matter whatever kernel version you're
> using, if it happens, it will take your data along with it.
> 
> But for the former case, newer kernel has improved trim check to prevent
> bad trim, so at least newer kernel is a little more safer.
> 
>> Should I update to the newest version?
> 
> Not always the newest, although we're trying our best to prevent bugs,
> but sometimes we still have some bugs sneaking into latest kernel.
> 
>> 
>> I have many partitions with btrfs and I like them a lot. Very nice file system indeed but am I safe with the version that I have (4.19.1)?
> 
> Can't say it's unsafe, since SUSE has all necessary backports and quite
> some customers are using (testing) it.
> As long as you're using the latest SUSE updates, it should be safe and
> all found bugs should have fixes backported.
> 
> Thanks,
> Qu
>> 
>> BTW, you are welcome to suggest any command or try anything with my broken file system that I still have backed up in case that you want to experiment with it.
>> 
>> Thanks 
>> 
>> Chris
>> 
>> 
>>> On 7. Dec 2019, at 22:21, Qu WenRuo <wqu@suse.com> wrote:
>>> 
>>> 
>>> 
>>> On 2019/12/8 上午12:44, Christian Wimmer wrote:
>>>> Hi Qu,
>>>> 
>>>> I was reading about chunk-recover. Do you think this could be worth a try?
>>> 
>>> Nope, your chunk tree is good, so that makes no sense.
>>> 
>>>> 
>>>> Is there any other command that can search for files that make sense to recover?
>>> 
>>> The only sane behavior here is to search the whole disk and grab
>>> anything looks like a tree block, and then extract data from it.
>>> 
>>> This is not something supported by btrfs-progs yet, so really not much
>>> more can be done here.
>>> 
>>> Thanks,
>>> Qu
>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> Chris
>>>> 
>> 
> 


^ permalink raw reply	[flat|nested] 53+ messages in thread

* 12 TB btrfs file system on virtual machine broke again
       [not found]           ` <9FB359ED-EAD4-41DD-B846-1422F2DC4242@icloud.com>
@ 2020-01-04 17:07             ` Christian Wimmer
  2020-01-05  4:03               ` Chris Murphy
  2020-01-05  4:25               ` Qu Wenruo
  0 siblings, 2 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-04 17:07 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

Hi guys, 

I run again in a problem with my btrfs files system.
I start wondering if this filesystem type is right for my needs.
Could you please help me in recovering my 12TB partition?

What happened? 
-> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.

I immediately powered off again and started my Arch Linux where I have btrfs-progs version 5.4 installed.
I tried one of the commands that you gave me in the past (restore) and I got following messages:


btrfs-progs-5.4]# ./btrfs restore -l /dev/sdb1
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
checksum verify failed on 3181912915968 found 00000071 wanted 00000066
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
Could not open root, trying backup super
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
checksum verify failed on 3181912915968 found 00000071 wanted 00000066
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
Could not open root, trying backup super
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
checksum verify failed on 3181912915968 found 00000071 wanted 00000066
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
Could not open root, trying backup super
btrfs-progs-5.4]# 



What can I do now to recover files?

Please help me.

Thanks,

Chris



^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-04 17:07             ` 12 TB btrfs file system on virtual machine broke again Christian Wimmer
@ 2020-01-05  4:03               ` Chris Murphy
  2020-01-05 13:40                 ` Christian Wimmer
  2020-01-05  4:25               ` Qu Wenruo
  1 sibling, 1 reply; 53+ messages in thread
From: Chris Murphy @ 2020-01-05  4:03 UTC (permalink / raw)
  To: Christian Wimmer; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sat, Jan 4, 2020 at 10:07 AM Christian Wimmer
<telefonchris@icloud.com> wrote:
>
> Hi guys,
>
> I run again in a problem with my btrfs files system.
> I start wondering if this filesystem type is right for my needs.
> Could you please help me in recovering my 12TB partition?

If you're having recurring problems, there's a decent chance it's
hardware related and not Btrfs, because Btrfs is pretty stable on
stable hardware. Btrfs is actually fussier than other file systems
because everything is checksummed.


> What happened?
> -> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.

A complete dmesg leading up to the problem would be useful. Kernel
messages at the last moments it was working, and also messages for the
first mount attempt that failed. And kernel versions for both (sounds
like 5.4 series).

And also `btrfs check` output, no repair.

> btrfs-progs-5.4]# ./btrfs restore -l /dev/sdb1
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
> Could not open root, trying backup super
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
> Could not open root, trying backup super
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
> Could not open root, trying backup super
> btrfs-progs-5.4]#


I'm not sure, the checksums found vs wanted look a little suspicious.

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-04 17:07             ` 12 TB btrfs file system on virtual machine broke again Christian Wimmer
  2020-01-05  4:03               ` Chris Murphy
@ 2020-01-05  4:25               ` Qu Wenruo
  2020-01-05 14:17                 ` Christian Wimmer
  1 sibling, 1 reply; 53+ messages in thread
From: Qu Wenruo @ 2020-01-05  4:25 UTC (permalink / raw)
  To: Christian Wimmer; +Cc: Qu WenRuo, Anand Jain, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 3299 bytes --]



On 2020/1/5 上午1:07, Christian Wimmer wrote:
> Hi guys, 
> 
> I run again in a problem with my btrfs files system.
> I start wondering if this filesystem type is right for my needs.
> Could you please help me in recovering my 12TB partition?
> 
> What happened? 
> -> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.
> 
> I immediately powered off again and started my Arch Linux where I have btrfs-progs version 5.4 installed.
> I tried one of the commands that you gave me in the past (restore) and I got following messages:
> 
> 
> btrfs-progs-5.4]# ./btrfs restore -l /dev/sdb1
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935

All these tree blocks are garbage. This doesn't look good at all.

The weird found csum pattern make no sense at all.

Are you using fstrim or discard mount option? If so, there could be some
old bug causing the problem.

SUSE Kernel version please.

Thanks,
Qu

> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
> Could not open root, trying backup super
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
> Could not open root, trying backup super
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> checksum verify failed on 2688520683520 found 000000EE wanted FFFFFFEB
> checksum verify failed on 2688520683520 found 000000D5 wanted FFFFFFA9
> bad tree block 2688520683520, bytenr mismatch, want=2688520683520, have=10123237912294
> Could not open root, trying backup super
> btrfs-progs-5.4]# 
> 
> 
> 
> What can I do now to recover files?
> 
> Please help me.
> 
> Thanks,
> 
> Chris
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05  4:03               ` Chris Murphy
@ 2020-01-05 13:40                 ` Christian Wimmer
  2020-01-05 14:07                   ` Martin Raiber
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 13:40 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

Hi Chris,



> On 5. Jan 2020, at 01:03, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Sat, Jan 4, 2020 at 10:07 AM Christian Wimmer
> <telefonchris@icloud.com> wrote:
>> 
>> Hi guys,
>> 
>> I run again in a problem with my btrfs files system.
>> I start wondering if this filesystem type is right for my needs.
>> Could you please help me in recovering my 12TB partition?
> 
> If you're having recurring problems, there's a decent chance it's
> hardware related and not Btrfs, because Btrfs is pretty stable on
> stable hardware. Btrfs is actually fussier than other file systems
> because everything is checksummed.
> 

I think I can exclude hardware problems. Everything is brand new and well tested.
The biggest chance for being the source of errors is the Parallels Virtual machine where Linux (Suse 15.1) is running in.
In this Virtual Machine I specify a “growing hard disc" that is actually a file on my 32 TB Promise Pegasus Storage.
I just can not understand why it runs fine for almost 1 month (and actually more for other growing hard discs of smaller size) and then shows this behaviour.
As soon as you “are sure" that btrfs is not the problem I will switch to different kind of hard discs.


> 
>> What happened?
>> -> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.
> 
> A complete dmesg leading up to the problem would be useful. Kernel
> messages at the last moments it was working, and also messages for the
> first mount attempt that failed. And kernel versions for both (sounds
> like 5.4 series).

I have one System with Suse 15.1 where the error happened. The btrfs-progs is of version 4.19.1.
The other system is 5.3.12-1-MANJARO with btrfs-progs version 5.4. Here I try to repair the broken file system.

> 
> And also `btrfs check` output, no repair.

Here the output:

btrfs-progs-5.4]# ./btrfs check /dev/sdb1
Opening filesystem to check...
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
checksum verify failed on 3181912915968 found 00000071 wanted 00000066
checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
checksum verify failed on 2688623902720 found 00000029 wanted 00000026
checksum verify failed on 2688623902720 found 000000E4 wanted 0000006F
checksum verify failed on 2688623902720 found 00000029 wanted 00000026
bad tree block 2688623902720, bytenr mismatch, want=2688623902720, have=256465382155451
ERROR: failed to read block groups: Input/output error
ERROR: cannot open file system
btrfs-progs-5.4]# 


Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 13:40                 ` Christian Wimmer
@ 2020-01-05 14:07                   ` Martin Raiber
  2020-01-05 14:14                     ` Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Martin Raiber @ 2020-01-05 14:07 UTC (permalink / raw)
  To: Christian Wimmer, Chris Murphy
  Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On 05.01.2020 14:40 Christian Wimmer wrote:
>> On 5. Jan 2020, at 01:03, Chris Murphy <lists@colorremedies.com> wrote:
>>
>> On Sat, Jan 4, 2020 at 10:07 AM Christian Wimmer
>> <telefonchris@icloud.com> wrote:
>>> Hi guys,
>>>
>>> I run again in a problem with my btrfs files system.
>>> I start wondering if this filesystem type is right for my needs.
>>> Could you please help me in recovering my 12TB partition?
>> If you're having recurring problems, there's a decent chance it's
>> hardware related and not Btrfs, because Btrfs is pretty stable on
>> stable hardware. Btrfs is actually fussier than other file systems
>> because everything is checksummed.
>>
> I think I can exclude hardware problems. Everything is brand new and well tested.
> The biggest chance for being the source of errors is the Parallels Virtual machine where Linux (Suse 15.1) is running in.
> In this Virtual Machine I specify a “growing hard disc" that is actually a file on my 32 TB Promise Pegasus Storage.
> I just can not understand why it runs fine for almost 1 month (and actually more for other growing hard discs of smaller size) and then shows this behaviour.
>
Does the host machine use ECC-RAM?

That Promise Pegasus Storage looks like a simple harware RAID without
integrity protection (like a btrfs RAID1/RAID6 or something like ceph
would give you).

Both of those can cause the errors you posted.



^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 14:07                   ` Martin Raiber
@ 2020-01-05 14:14                     ` Christian Wimmer
  2020-01-05 14:23                       ` Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 14:14 UTC (permalink / raw)
  To: Martin Raiber; +Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 11:07, Martin Raiber <martin@urbackup.org> wrote:
> 
> On 05.01.2020 14:40 Christian Wimmer wrote:
>>> On 5. Jan 2020, at 01:03, Chris Murphy <lists@colorremedies.com> wrote:
>>> 
>>> On Sat, Jan 4, 2020 at 10:07 AM Christian Wimmer
>>> <telefonchris@icloud.com> wrote:
>>>> Hi guys,
>>>> 
>>>> I run again in a problem with my btrfs files system.
>>>> I start wondering if this filesystem type is right for my needs.
>>>> Could you please help me in recovering my 12TB partition?
>>> If you're having recurring problems, there's a decent chance it's
>>> hardware related and not Btrfs, because Btrfs is pretty stable on
>>> stable hardware. Btrfs is actually fussier than other file systems
>>> because everything is checksummed.
>>> 
>> I think I can exclude hardware problems. Everything is brand new and well tested.
>> The biggest chance for being the source of errors is the Parallels Virtual machine where Linux (Suse 15.1) is running in.
>> In this Virtual Machine I specify a “growing hard disc" that is actually a file on my 32 TB Promise Pegasus Storage.
>> I just can not understand why it runs fine for almost 1 month (and actually more for other growing hard discs of smaller size) and then shows this behaviour.
>> 
> Does the host machine use ECC-RAM?

The host is a Mac Mini 2018 where I took of the 8GB memory from Apple and put:

Samsung 2x 32GB = 64GB KIT DDR4 RAM PC4-21300 2666MHz SO-DIMM Herstellerartikelnummer: M471A4G43MB1-CTD

I bought at: Mac Speichershop André Estel Steinbecker Dorfstr. 10 19399 Goldberg


> 
> That Promise Pegasus Storage looks like a simple harware RAID without
> integrity protection (like a btrfs RAID1/RAID6 or something like ceph
> would give you).

It is a Pegasus Promise3 R8 running in RAID 5. 




^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05  4:25               ` Qu Wenruo
@ 2020-01-05 14:17                 ` Christian Wimmer
  2020-01-05 18:50                   ` Chris Murphy
  2020-01-05 23:50                   ` 12 TB btrfs file system on virtual machine broke again Qu Wenruo
  0 siblings, 2 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 14:17 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

Hi Qu,


> On 5. Jan 2020, at 01:25, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> 
> 
> On 2020/1/5 上午1:07, Christian Wimmer wrote:
>> Hi guys, 
>> 
>> I run again in a problem with my btrfs files system.
>> I start wondering if this filesystem type is right for my needs.
>> Could you please help me in recovering my 12TB partition?
>> 
>> What happened? 
>> -> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.
>> 
>> I immediately powered off again and started my Arch Linux where I have btrfs-progs version 5.4 installed.
>> I tried one of the commands that you gave me in the past (restore) and I got following messages:
>> 
>> 
>> btrfs-progs-5.4]# ./btrfs restore -l /dev/sdb1
>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
> 
> All these tree blocks are garbage. This doesn't look good at all.
> 
> The weird found csum pattern make no sense at all.
> 
> Are you using fstrim or discard mount option? If so, there could be some
> old bug causing the problem.


Seems that I am using fstrim (I did not know this, what is it?):

BTW, sda2 is here my root partition which is practically the same configuration (just smaller) than the 12TB hard disc

2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400
2020-01-03T11:30:47.479538-03:00 linux-ze6w kernel: [1297857.324658] BTRFS warning (device sda2): failed to trim 1 device(s), last error -5
2020-01-03T11:30:48.376543-03:00 linux-ze6w fstrim[27910]: fstrim: /opt: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:48.378998-03:00 linux-ze6w kernel: [1297858.223675] attempt to access beyond end of device
2020-01-03T11:30:48.379012-03:00 linux-ze6w kernel: [1297858.223677] sda2: rw=3, want=421570540, limit=419430400
2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] attempt to access beyond end of device
2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] sda2: rw=3, want=429959147, limit=419430400
2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] attempt to access beyond end of device
2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] sda2: rw=3, want=438347754, limit=419430400
2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223680] attempt to access beyond end of device

Could this be the problem?


Suse Kernel version is 4.12.14-lp151.28.13-default #1 SMP



^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 14:14                     ` Christian Wimmer
@ 2020-01-05 14:23                       ` Christian Wimmer
  0 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 14:23 UTC (permalink / raw)
  To: Martin Raiber; +Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

In the warn file (var/log/messages/warn) If found following lines:

2019-12-29T13:10:33.097033-03:00 linux-ze6w kernel: [1297807.325928] btrfs_dev_stat_print_on_error: 25 callbacks suppressed
2019-12-29T13:10:33.097048-03:00 linux-ze6w kernel: [1297807.325935] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097051-03:00 linux-ze6w kernel: [1297807.326276] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097052-03:00 linux-ze6w kernel: [1297807.326584] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097052-03:00 linux-ze6w kernel: [1297807.326937] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097057-03:00 linux-ze6w kernel: [1297807.327051] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097058-03:00 linux-ze6w kernel: [1297807.327263] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097058-03:00 linux-ze6w kernel: [1297807.327264] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097058-03:00 linux-ze6w kernel: [1297807.327272] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097059-03:00 linux-ze6w kernel: [1297807.327273] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097059-03:00 linux-ze6w kernel: [1297807.327280] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
2019-12-29T13:10:33.097068-03:00 linux-ze6w kernel: [1297807.327416] BTRFS warning (device sdb1): Skipping commit of aborted transaction.
2019-12-29T13:10:33.097069-03:00 linux-ze6w kernel: [1297807.327433] BTRFS error (device sdb1): commit super ret -5
2019-12-29T13:10:33.097069-03:00 linux-ze6w kernel: [1297807.327531] BTRFS error (device sdb1): cleaner transaction attach returned -30
2019-12-29T13:10:33.097059-03:00 linux-ze6w kernel: [1297807.327414] BTRFS: error (device sdb1) in btrfs_commit_transaction:2261: errno=-5 IO failure (Error while writing out transaction)
2019-12-29T13:10:33.097069-03:00 linux-ze6w kernel: [1297807.327418] BTRFS: error (device sdb1) in cleanup_transaction:1881: errno=-5 IO failure




^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 14:17                 ` Christian Wimmer
@ 2020-01-05 18:50                   ` Chris Murphy
  2020-01-05 19:18                     ` Christian Wimmer
  2020-01-05 23:50                   ` 12 TB btrfs file system on virtual machine broke again Qu Wenruo
  1 sibling, 1 reply; 53+ messages in thread
From: Chris Murphy @ 2020-01-05 18:50 UTC (permalink / raw)
  To: Christian Wimmer; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sun, Jan 5, 2020 at 7:17 AM Christian Wimmer <telefonchris@icloud.com> wrote:
>
> Seems that I am using fstrim (I did not know this, what is it?):

Frees unused blocks from underlying storage: in the case of sparse
files it punches holds, for thin provisioning it frees logical extents
back to the pool, and for real physical SSDs it informs the firmware
those blocks are no longer used and can be garbage collected.

Most bugs in this area have either been fixed by firmware updates by
manufacturers for the SSD, or they've been blacklisted in the kernel
so that FITRIM is a no op.


>
> BTW, sda2 is here my root partition which is practically the same configuration (just smaller) than the 12TB hard disc
>
> 2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400
> 2020-01-03T11:30:47.479538-03:00 linux-ze6w kernel: [1297857.324658] BTRFS warning (device sda2): failed to trim 1 device(s), last error -5
> 2020-01-03T11:30:48.376543-03:00 linux-ze6w fstrim[27910]: fstrim: /opt: FITRIM ioctl failed: Input/output error
> 2020-01-03T11:30:48.378998-03:00 linux-ze6w kernel: [1297858.223675] attempt to access beyond end of device
> 2020-01-03T11:30:48.379012-03:00 linux-ze6w kernel: [1297858.223677] sda2: rw=3, want=421570540, limit=419430400

Yeah that's a problem. That may not be *the* problem, but there is
confusion here. What is /dev/sda? This is a virtual drive inside the
guest VM? And is backed by a file on the Promise storage? What about
/dev/sdb? Same thing? You're only having a problem with /dev/sdb,
which contains a Btrfs file system. But if its storage stack is the
same as /dev/sda, then the FITRIM problem might be related, we just
don't have logs proving it.

What do you get for 'sudo systemctl status fstrim.timer' ?

While it might be simple to just disable the fstrim.timer if it's
enabled; I don't know that this tells you whether there might be other
problems with the storage stack somewhere. The fact that this
particular FITRIM command is trying to access beyond the end of the
device, means there is confusion about /dev/sda size. Is that
transient and related to the resize maybe not yet having completed? Or
is it a persistent problem that could affect the mapping in otherwise
valid regions inside this virtual block device?

What should be true is FITRIM inside a guest VM either completely
fails (unmap or trim are not advertised and therefore FITRIM fails
safely) or it completely succeeds. You shouldn't get corruptions and
you shouldn't get the message you're seeing. So my advice is trying to
figure out why that's happening. And also if logs show whether it has
ever happened for /dev/sdb




-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 18:50                   ` Chris Murphy
@ 2020-01-05 19:18                     ` Christian Wimmer
  2020-01-05 19:36                       ` Chris Murphy
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 19:18 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 15:50, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Sun, Jan 5, 2020 at 7:17 AM Christian Wimmer <telefonchris@icloud.com> wrote:
>> 
>> Seems that I am using fstrim (I did not know this, what is it?):
> 
> Frees unused blocks from underlying storage: in the case of sparse
> files it punches holds, for thin provisioning it frees logical extents
> back to the pool, and for real physical SSDs it informs the firmware
> those blocks are no longer used and can be garbage collected.
> 
> Most bugs in this area have either been fixed by firmware updates by
> manufacturers for the SSD, or they've been blacklisted in the kernel
> so that FITRIM is a no op.
> 
> 
>> 
>> BTW, sda2 is here my root partition which is practically the same configuration (just smaller) than the 12TB hard disc
>> 
>> 2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400
>> 2020-01-03T11:30:47.479538-03:00 linux-ze6w kernel: [1297857.324658] BTRFS warning (device sda2): failed to trim 1 device(s), last error -5
>> 2020-01-03T11:30:48.376543-03:00 linux-ze6w fstrim[27910]: fstrim: /opt: FITRIM ioctl failed: Input/output error
>> 2020-01-03T11:30:48.378998-03:00 linux-ze6w kernel: [1297858.223675] attempt to access beyond end of device
>> 2020-01-03T11:30:48.379012-03:00 linux-ze6w kernel: [1297858.223677] sda2: rw=3, want=421570540, limit=419430400
> 
> Yeah that's a problem. That may not be *the* problem, but there is
> confusion here. What is /dev/sda?

/dev/sda is the hard disc file that holds the \x10Linux:

#fdisk -l
Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
Disk model: Suse 15.1-0 SSD 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215

Device         Start       End   Sectors  Size Type
/dev/sda1       2048     18431     16384    8M BIOS boot
/dev/sda2      18432 419448831 419430400  200G Linux filesystem
/dev/sda3  532674560 536870878   4196319    2G Linux swap


This file is located on the SSD of my MAC Mini. /dev/sda2 is formatted with btrfs.

> This is a virtual drive inside the
> guest VM? And is backed by a file on the Promise storage? What about
> /dev/sdb? Same thing? You're only having a problem with /dev/sdb,
> which contains a Btrfs file system.

Actually I have only a problem with the /dev/sdb which is a hard disc file on my Promise storage. The sda2 complains but boots normally.

Regarding any logs. Which log files I should look at and how to display them?
I looked at the /var/log/messages but did not find any related information.

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 19:18                     ` Christian Wimmer
@ 2020-01-05 19:36                       ` Chris Murphy
  2020-01-05 19:49                         ` Christian Wimmer
                                           ` (2 more replies)
  0 siblings, 3 replies; 53+ messages in thread
From: Chris Murphy @ 2020-01-05 19:36 UTC (permalink / raw)
  To: Christian Wimmer
  Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sun, Jan 5, 2020 at 12:18 PM Christian Wimmer
<telefonchris@icloud.com> wrote:
>
>
>
> > On 5. Jan 2020, at 15:50, Chris Murphy <lists@colorremedies.com> wrote:
> >
> > On Sun, Jan 5, 2020 at 7:17 AM Christian Wimmer <telefonchris@icloud.com> wrote:
> >>

> >> 2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400

> /dev/sda is the hard disc file that holds the  Linux:
>
> #fdisk -l
> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
> Disk model: Suse 15.1-0 SSD
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disklabel type: gpt
> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
>
> Device         Start       End   Sectors  Size Type
> /dev/sda1       2048     18431     16384    8M BIOS boot
> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> /dev/sda3  532674560 536870878   4196319    2G Linux swap


Why does the kernel want=532656128 but knows the limit=419430400? The
limit matches the GPT partition map.

What do you get for

btrfs insp dump-s /dev/sda2


> > This is a virtual drive inside the
> > guest VM? And is backed by a file on the Promise storage? What about
> > /dev/sdb? Same thing? You're only having a problem with /dev/sdb,
> > which contains a Btrfs file system.
>
> Actually I have only a problem with the /dev/sdb which is a hard disc file on my Promise storage. The sda2 complains but boots normally.

sda2 complains? You mean just the previously mentioned FITRIM I/O
failures? Or there's more?


>
> Regarding any logs. Which log files I should look at and how to display them?
> I looked at the /var/log/messages but did not find any related information.

Start with

systemctl status fstrim.timer
systemctl status fstrim.service

Find the location of the fstrim.service file and cat it, and post that
too. I want to know exactly what fstrim options it's using. Older
versions try to trim all file systems.

journalctl --since=-8d | grep fstrim

You don't have to post that output but you should see if fstrim has
been called on /dev/sdb any time in the past 8 days. By default
fstrim.timer if enabled, runs once per week.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 19:36                       ` Chris Murphy
@ 2020-01-05 19:49                         ` Christian Wimmer
  2020-01-05 19:52                         ` Christian Wimmer
       [not found]                         ` <3F43DDB8-0372-4CDE-B143-D2727D3447BC@icloud.com>
  2 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 19:49 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 16:36, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Sun, Jan 5, 2020 at 12:18 PM Christian Wimmer
> <telefonchris@icloud.com> wrote:
>> 
>> 
>> 
>>> On 5. Jan 2020, at 15:50, Chris Murphy <lists@colorremedies.com> wrote:
>>> 
>>> On Sun, Jan 5, 2020 at 7:17 AM Christian Wimmer <telefonchris@icloud.com> wrote:
>>>> 
> 
>>>> 2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400
> 
>> /dev/sda is the hard disc file that holds the  Linux:
>> 
>> #fdisk -l
>> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
>> Disk model: Suse 15.1-0 SSD
>> Units: sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>> Disklabel type: gpt
>> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
>> 
>> Device         Start       End   Sectors  Size Type
>> /dev/sda1       2048     18431     16384    8M BIOS boot
>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
> 
> 
> Why does the kernel want=532656128 but knows the limit=419430400? The
> limit matches the GPT partition map.
> 
> What do you get for
> 
> btrfs insp dump-s /dev/sda2

Here I have only btrfs-progs version 4.19.1:

linux-ze6w:~ # btrfs version
btrfs-progs v4.19.1 
linux-ze6w:~ # btrfs insp dump-s /dev/sda2
superblock: bytenr=65536, device=/dev/sda2
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0x6d9388e2 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
label
generation              799183
root                    724205568
sys_array_size          97
chunk_root_generation   797617
root_level              1
chunk_root              158835163136
chunk_root_level        0
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             272719937536
bytes_used              106188886016
sectorsize              4096
nodesize                16384
leafsize (deprecated)           16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x163
                        ( MIXED_BACKREF |
                          DEFAULT_SUBVOL |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
cache_generation        799183
uuid_tree_generation    557352
dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
dev_item.type           0
dev_item.total_bytes    272719937536
dev_item.bytes_used     129973092352
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0



> 
> 
>>> This is a virtual drive inside the
>>> guest VM? And is backed by a file on the Promise storage? What about
>>> /dev/sdb? Same thing? You're only having a problem with /dev/sdb,
>>> which contains a Btrfs file system.
>> 
>> Actually I have only a problem with the /dev/sdb which is a hard disc file on my Promise storage. The sda2 complains but boots normally.
> 
> sda2 complains? You mean just the previously mentioned FITRIM I/O
> failures? Or there's more?

Only what I found in the previously mentioned messages. Nothing else.

> 
> 
>> 
>> Regarding any logs. Which log files I should look at and how to display them?
>> I looked at the /var/log/messages but did not find any related information.
> 
> Start with
> 
> systemctl status fstrim.timer
> systemctl status fstrim.service

linux-ze6w:~ # systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
  Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
     Docs: man:fstrim

Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.

linux-ze6w:~ # systemctl status fstrim.service
● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
   Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:fstrim(8)
linux-ze6w:~ # 

> 
> Find the location of the fstrim.service file and cat it, and post that
> too. I want to know exactly what fstrim options it's using. Older
> versions try to trim all file systems.

linux-ze6w:~ # cat /usr/lib/systemd/system/fstrim.service
[Unit]
Description=Discard unused blocks on filesystems from /etc/fstab
Documentation=man:fstrim(8)

[Service]
Type=oneshot
ExecStart=/usr/sbin/fstrim -Av
linux-ze6w:~ # 



> 
> journalctl --since=-8d | grep fstrim


journalctl --since=-8d

this command shows only the messages from today and there is no fstrim inside

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 19:36                       ` Chris Murphy
  2020-01-05 19:49                         ` Christian Wimmer
@ 2020-01-05 19:52                         ` Christian Wimmer
  2020-01-05 20:34                           ` Chris Murphy
       [not found]                         ` <3F43DDB8-0372-4CDE-B143-D2727D3447BC@icloud.com>
  2 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 19:52 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

BTW, I found this messages in the messages-20200105 file:



bash$ grep fstrim messages-20200105
2019-12-23T00:00:03.533050-03:00 linux-ze6w fstrim[32008]: fstrim: /boot/grub2/i386-pc: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:04.354989-03:00 linux-ze6w fstrim[32008]: fstrim: /boot/grub2/x86_64-efi: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:05.149687-03:00 linux-ze6w fstrim[32008]: fstrim: /home: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:05.941978-03:00 linux-ze6w fstrim[32008]: fstrim: /opt: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:06.740810-03:00 linux-ze6w fstrim[32008]: fstrim: /root: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:07.523365-03:00 linux-ze6w fstrim[32008]: fstrim: /srv: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:08.361831-03:00 linux-ze6w fstrim[32008]: fstrim: /tmp: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:09.188937-03:00 linux-ze6w fstrim[32008]: fstrim: /usr/local: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:09.974086-03:00 linux-ze6w fstrim[32008]: fstrim: /var: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:10.761933-03:00 linux-ze6w fstrim[32008]: fstrim: /: FITRIM ioctl failed: Input/output error
2019-12-23T00:00:10.762050-03:00 linux-ze6w fstrim[32008]: /mnt/so_logic: 27.1 GiB (29089808384 bytes) trimmed on /dev/sdd1
2019-12-23T00:00:10.762121-03:00 linux-ze6w fstrim[32008]: /home/chris2: 265.4 GiB (284938117120 bytes) trimmed on /dev/sdc1
2019-12-23T00:00:10.762198-03:00 linux-ze6w systemd[1]: fstrim.service: Main process exited, code=exited, status=64/n/a
2019-12-23T00:00:10.762449-03:00 linux-ze6w systemd[1]: fstrim.service: Unit entered failed state.
2019-12-23T00:00:10.762538-03:00 linux-ze6w systemd[1]: fstrim.service: Failed with result 'exit-code'.
2020-01-03T11:30:45.742369-03:00 linux-ze6w fstrim[27910]: fstrim: /boot/grub2/i386-pc: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:46.592336-03:00 linux-ze6w fstrim[27910]: fstrim: /boot/grub2/x86_64-efi: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:47.476629-03:00 linux-ze6w fstrim[27910]: fstrim: /home: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:48.376543-03:00 linux-ze6w fstrim[27910]: fstrim: /opt: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:49.295675-03:00 linux-ze6w fstrim[27910]: fstrim: /root: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:50.180612-03:00 linux-ze6w fstrim[27910]: fstrim: /srv: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:51.049373-03:00 linux-ze6w fstrim[27910]: fstrim: /tmp: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:51.919945-03:00 linux-ze6w fstrim[27910]: fstrim: /usr/local: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:52.777132-03:00 linux-ze6w fstrim[27910]: fstrim: /var: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:53.623997-03:00 linux-ze6w fstrim[27910]: fstrim: /: FITRIM ioctl failed: Input/output error
2020-01-03T11:30:53.624172-03:00 linux-ze6w fstrim[27910]: /mnt/so_logic: 27.1 GiB (29089808384 bytes) trimmed on /dev/sdd1
2020-01-03T11:30:53.624247-03:00 linux-ze6w fstrim[27910]: /home/chris2: 262.5 GiB (281862184960 bytes) trimmed on /dev/sdc1
2020-01-03T11:30:53.624365-03:00 linux-ze6w systemd[1]: fstrim.service: Main process exited, code=exited, status=64/n/a
2020-01-03T11:30:53.624883-03:00 linux-ze6w systemd[1]: fstrim.service: Unit entered failed state.
2020-01-03T11:30:53.624991-03:00 linux-ze6w systemd[1]: fstrim.service: Failed with result 'exit-code'.


Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
       [not found]                         ` <3F43DDB8-0372-4CDE-B143-D2727D3447BC@icloud.com>
@ 2020-01-05 20:30                           ` Chris Murphy
  2020-01-05 20:36                             ` Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Chris Murphy @ 2020-01-05 20:30 UTC (permalink / raw)
  To: Christian Wimmer
  Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sun, Jan 5, 2020 at 12:48 PM Christian Wimmer
<telefonchris@icloud.com> wrote:
>
>
> #fdisk -l
> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
> Disk model: Suse 15.1-0 SSD
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disklabel type: gpt
> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
>
> Device         Start       End   Sectors  Size Type
> /dev/sda1       2048     18431     16384    8M BIOS boot
> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> /dev/sda3  532674560 536870878   4196319    2G Linux swap



> btrfs insp dump-s /dev/sda2
>
>
> Here I have only btrfs-progs version 4.19.1:
>
> linux-ze6w:~ # btrfs version
> btrfs-progs v4.19.1
> linux-ze6w:~ # btrfs insp dump-s /dev/sda2
> superblock: bytenr=65536, device=/dev/sda2
> ---------------------------------------------------------
> csum_type               0 (crc32c)
> csum_size               4
> csum                    0x6d9388e2 [match]
> bytenr                  65536
> flags                   0x1
>                         ( WRITTEN )
> magic                   _BHRfS_M [match]
> fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
> metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
> label
> generation              799183
> root                    724205568
> sys_array_size          97
> chunk_root_generation   797617
> root_level              1
> chunk_root              158835163136
> chunk_root_level        0
> log_root                0
> log_root_transid        0
> log_root_level          0
> total_bytes             272719937536
> bytes_used              106188886016
> sectorsize              4096
> nodesize                16384
> leafsize (deprecated)           16384
> stripesize              4096
> root_dir                6
> num_devices             1
> compat_flags            0x0
> compat_ro_flags         0x0
> incompat_flags          0x163
>                         ( MIXED_BACKREF |
>                           DEFAULT_SUBVOL |
>                           BIG_METADATA |
>                           EXTENDED_IREF |
>                           SKINNY_METADATA )
> cache_generation        799183
> uuid_tree_generation    557352
> dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
> dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
> dev_item.type           0
> dev_item.total_bytes    272719937536
> dev_item.bytes_used     129973092352
> dev_item.io_align       4096
> dev_item.io_width       4096
> dev_item.sector_size    4096
> dev_item.devid          1
> dev_item.dev_group      0
> dev_item.seek_speed     0
> dev_item.bandwidth      0
> dev_item.generation     0

Partition map says
> /dev/sda2      18432 419448831 419430400  200G Linux filesystem

Btrfs super says
> total_bytes             272719937536

272719937536*512=532656128

Kernel FITRIM want is want=532656128

OK so the problem is the Btrfs super isn't set to the size of the
partition. The usual way this happens is user error: partition is
resized (shrink) without resizing the file system first. This file
system is still at risk of having problems even if you disable
fstrim.timer. You need to shrink the file system is the same size as
the partition.



> linux-ze6w:~ # systemctl status fstrim.timer
> ● fstrim.timer - Discard unused blocks once a week
>    Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
>    Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
>   Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
>      Docs: man:fstrim
>
> Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.
>
> linux-ze6w:~ # systemctl status fstrim.service
> ● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
>    Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
>    Active: inactive (dead)
>      Docs: man:fstrim(8)
> linux-ze6w:~ #

OK so it's not set to run. Why do you have FITRIM being called?

What are the mount options for this file system?

> this command shows only the messages from today and there is no fstrim inside

Something else is calling FITRIM.

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 19:52                         ` Christian Wimmer
@ 2020-01-05 20:34                           ` Chris Murphy
  2020-01-05 20:36                             ` Chris Murphy
  0 siblings, 1 reply; 53+ messages in thread
From: Chris Murphy @ 2020-01-05 20:34 UTC (permalink / raw)
  To: Christian Wimmer
  Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sun, Jan 5, 2020 at 12:52 PM Christian Wimmer
<telefonchris@icloud.com> wrote:
>
> BTW, I found this messages in the messages-20200105 file:
>
>
>
> bash$ grep fstrim messages-20200105
> 2019-12-23T00:00:03.533050-03:00 linux-ze6w fstrim[32008]: fstrim: /boot/grub2/i386-pc: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:04.354989-03:00 linux-ze6w fstrim[32008]: fstrim: /boot/grub2/x86_64-efi: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:05.149687-03:00 linux-ze6w fstrim[32008]: fstrim: /home: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:05.941978-03:00 linux-ze6w fstrim[32008]: fstrim: /opt: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:06.740810-03:00 linux-ze6w fstrim[32008]: fstrim: /root: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:07.523365-03:00 linux-ze6w fstrim[32008]: fstrim: /srv: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:08.361831-03:00 linux-ze6w fstrim[32008]: fstrim: /tmp: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:09.188937-03:00 linux-ze6w fstrim[32008]: fstrim: /usr/local: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:09.974086-03:00 linux-ze6w fstrim[32008]: fstrim: /var: FITRIM ioctl failed: Input/output error
> 2019-12-23T00:00:10.761933-03:00 linux-ze6w fstrim[32008]: fstrim: /: FITRIM ioctl failed: Input/output error

Bet these are all on /dev/sda2 file system.

> 2019-12-23T00:00:10.762050-03:00 linux-ze6w fstrim[32008]: /mnt/so_logic: 27.1 GiB (29089808384 bytes) trimmed on /dev/sdd1
> 2019-12-23T00:00:10.762121-03:00 linux-ze6w fstrim[32008]: /home/chris2: 265.4 GiB (284938117120 bytes) trimmed on /dev/sdc1

I can't tell what file system or device these are on, but it appears
to succeed without error. Therefore the VM is advertising discard
support to the guest. But what is calling fstrim? It appears to not be
fstrim.timer. We still don't know whether it might be the cause of
problems elsewhere in the storage stack that might explain why the
Btrfs file system you care about is being corrupted.



> 2019-12-23T00:00:10.762538-03:00 linux-ze6w systemd[1]: fstrim.service: Failed with result 'exit-code'.
> 2020-01-03T11:30:45.742369-03:00 linux-ze6w fstrim[27910]: fstrim: /boot/grub2/i386-pc: FITRIM ioctl failed: Input/output error

10 days.

"weekly" for systemd timers means "monday at 00:00" local time. Which
is what you're seeing in the first line, but not the second.




-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 20:34                           ` Chris Murphy
@ 2020-01-05 20:36                             ` Chris Murphy
  0 siblings, 0 replies; 53+ messages in thread
From: Chris Murphy @ 2020-01-05 20:36 UTC (permalink / raw)
  To: Chris Murphy
  Cc: Christian Wimmer, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sun, Jan 5, 2020 at 1:34 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> > 2019-12-23T00:00:10.762538-03:00 linux-ze6w systemd[1]: fstrim.service: Failed with result 'exit-code'.
> > 2020-01-03T11:30:45.742369-03:00 linux-ze6w fstrim[27910]: fstrim: /boot/grub2/i386-pc: FITRIM ioctl failed: Input/output error
>
> 10 days.

11 days, but doesn't really matter. Still need to know why
fstrim.service is being called. Maybe a cron or anacon job or
something else is calling this unit to run. Or maybe even more than
one thing is doing it.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 20:30                           ` Chris Murphy
@ 2020-01-05 20:36                             ` Christian Wimmer
  2020-01-05 21:13                               ` Chris Murphy
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 20:36 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 17:30, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Sun, Jan 5, 2020 at 12:48 PM Christian Wimmer
> <telefonchris@icloud.com> wrote:
>> 
>> 
>> #fdisk -l
>> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
>> Disk model: Suse 15.1-0 SSD
>> Units: sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>> Disklabel type: gpt
>> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
>> 
>> Device         Start       End   Sectors  Size Type
>> /dev/sda1       2048     18431     16384    8M BIOS boot
>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
> 
> 
> 
>> btrfs insp dump-s /dev/sda2
>> 
>> 
>> Here I have only btrfs-progs version 4.19.1:
>> 
>> linux-ze6w:~ # btrfs version
>> btrfs-progs v4.19.1
>> linux-ze6w:~ # btrfs insp dump-s /dev/sda2
>> superblock: bytenr=65536, device=/dev/sda2
>> ---------------------------------------------------------
>> csum_type               0 (crc32c)
>> csum_size               4
>> csum                    0x6d9388e2 [match]
>> bytenr                  65536
>> flags                   0x1
>>                        ( WRITTEN )
>> magic                   _BHRfS_M [match]
>> fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
>> metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
>> label
>> generation              799183
>> root                    724205568
>> sys_array_size          97
>> chunk_root_generation   797617
>> root_level              1
>> chunk_root              158835163136
>> chunk_root_level        0
>> log_root                0
>> log_root_transid        0
>> log_root_level          0
>> total_bytes             272719937536
>> bytes_used              106188886016
>> sectorsize              4096
>> nodesize                16384
>> leafsize (deprecated)           16384
>> stripesize              4096
>> root_dir                6
>> num_devices             1
>> compat_flags            0x0
>> compat_ro_flags         0x0
>> incompat_flags          0x163
>>                        ( MIXED_BACKREF |
>>                          DEFAULT_SUBVOL |
>>                          BIG_METADATA |
>>                          EXTENDED_IREF |
>>                          SKINNY_METADATA )
>> cache_generation        799183
>> uuid_tree_generation    557352
>> dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
>> dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
>> dev_item.type           0
>> dev_item.total_bytes    272719937536
>> dev_item.bytes_used     129973092352
>> dev_item.io_align       4096
>> dev_item.io_width       4096
>> dev_item.sector_size    4096
>> dev_item.devid          1
>> dev_item.dev_group      0
>> dev_item.seek_speed     0
>> dev_item.bandwidth      0
>> dev_item.generation     0
> 
> Partition map says
>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> 
> Btrfs super says
>> total_bytes             272719937536
> 
> 272719937536*512=532656128
> 
> Kernel FITRIM want is want=532656128
> 
> OK so the problem is the Btrfs super isn't set to the size of the
> partition. The usual way this happens is user error: partition is
> resized (shrink) without resizing the file system first. This file
> system is still at risk of having problems even if you disable
> fstrim.timer. You need to shrink the file system is the same size as
> the partition.
> 

Could this be a problem of Parallels Virtual machine that maybe sometimes try to get more space on the hosting file system?
One solution would be to have a fixed size of the disc file instead of a growing one.

> 
> 
>> linux-ze6w:~ # systemctl status fstrim.timer
>> ● fstrim.timer - Discard unused blocks once a week
>>   Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
>>   Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
>>  Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
>>     Docs: man:fstrim
>> 
>> Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.
>> 
>> linux-ze6w:~ # systemctl status fstrim.service
>> ● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
>>   Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
>>   Active: inactive (dead)
>>     Docs: man:fstrim(8)
>> linux-ze6w:~ #
> 
> OK so it's not set to run. Why do you have FITRIM being called?

No idea.

> 
> What are the mount options for this file system?

should have been mounted like:

UUID=bb34b1db-ee47-4367-b207-1de1671087e7  /home/promise2          btrfs  defaults                      0  0

/dev/sdc1 on /home/promise2 type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)


Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 20:36                             ` Christian Wimmer
@ 2020-01-05 21:13                               ` Chris Murphy
  2020-01-05 21:58                                 ` Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Chris Murphy @ 2020-01-05 21:13 UTC (permalink / raw)
  To: Christian Wimmer
  Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sun, Jan 5, 2020 at 1:36 PM Christian Wimmer <telefonchris@icloud.com> wrote:
>
>
>
> > On 5. Jan 2020, at 17:30, Chris Murphy <lists@colorremedies.com> wrote:
> >
> > On Sun, Jan 5, 2020 at 12:48 PM Christian Wimmer
> > <telefonchris@icloud.com> wrote:
> >>
> >>
> >> #fdisk -l
> >> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
> >> Disk model: Suse 15.1-0 SSD
> >> Units: sectors of 1 * 512 = 512 bytes
> >> Sector size (logical/physical): 512 bytes / 4096 bytes
> >> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> >> Disklabel type: gpt
> >> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
> >>
> >> Device         Start       End   Sectors  Size Type
> >> /dev/sda1       2048     18431     16384    8M BIOS boot
> >> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> >> /dev/sda3  532674560 536870878   4196319    2G Linux swap
> >
> >
> >
> >> btrfs insp dump-s /dev/sda2
> >>
> >>
> >> Here I have only btrfs-progs version 4.19.1:
> >>
> >> linux-ze6w:~ # btrfs version
> >> btrfs-progs v4.19.1
> >> linux-ze6w:~ # btrfs insp dump-s /dev/sda2
> >> superblock: bytenr=65536, device=/dev/sda2
> >> ---------------------------------------------------------
> >> csum_type               0 (crc32c)
> >> csum_size               4
> >> csum                    0x6d9388e2 [match]
> >> bytenr                  65536
> >> flags                   0x1
> >>                        ( WRITTEN )
> >> magic                   _BHRfS_M [match]
> >> fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
> >> metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
> >> label
> >> generation              799183
> >> root                    724205568
> >> sys_array_size          97
> >> chunk_root_generation   797617
> >> root_level              1
> >> chunk_root              158835163136
> >> chunk_root_level        0
> >> log_root                0
> >> log_root_transid        0
> >> log_root_level          0
> >> total_bytes             272719937536
> >> bytes_used              106188886016
> >> sectorsize              4096
> >> nodesize                16384
> >> leafsize (deprecated)           16384
> >> stripesize              4096
> >> root_dir                6
> >> num_devices             1
> >> compat_flags            0x0
> >> compat_ro_flags         0x0
> >> incompat_flags          0x163
> >>                        ( MIXED_BACKREF |
> >>                          DEFAULT_SUBVOL |
> >>                          BIG_METADATA |
> >>                          EXTENDED_IREF |
> >>                          SKINNY_METADATA )
> >> cache_generation        799183
> >> uuid_tree_generation    557352
> >> dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
> >> dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
> >> dev_item.type           0
> >> dev_item.total_bytes    272719937536
> >> dev_item.bytes_used     129973092352
> >> dev_item.io_align       4096
> >> dev_item.io_width       4096
> >> dev_item.sector_size    4096
> >> dev_item.devid          1
> >> dev_item.dev_group      0
> >> dev_item.seek_speed     0
> >> dev_item.bandwidth      0
> >> dev_item.generation     0
> >
> > Partition map says
> >> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> >
> > Btrfs super says
> >> total_bytes             272719937536
> >
> > 272719937536*512=532656128
> >
> > Kernel FITRIM want is want=532656128
> >
> > OK so the problem is the Btrfs super isn't set to the size of the
> > partition. The usual way this happens is user error: partition is
> > resized (shrink) without resizing the file system first. This file
> > system is still at risk of having problems even if you disable
> > fstrim.timer. You need to shrink the file system is the same size as
> > the partition.
> >
>
> Could this be a problem of Parallels Virtual machine that maybe sometimes try to get more space on the hosting file system?
> One solution would be to have a fixed size of the disc file instead of a growing one.

I don't see how it's related. Parallels has no ability I'm aware of to
change the GPT partition map or the Btrfs super block - as in, rewrite
it out with a modification correctly including all checksums being
valid. This /dev/sda has somehow been mangled on purpose.

Again, from the GPT
> >> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> >> /dev/sda3  532674560 536870878   4196319    2G Linux swap

The end LBA for sda2 is 419448831, but the start LBA for sda3 is
532674560. There's a ~54G gap in there as if something was removed.
I'm not sure why a software installer would produce this kind of
layout on purpose, because it has no purpose.





>
> >
> >
> >> linux-ze6w:~ # systemctl status fstrim.timer
> >> ● fstrim.timer - Discard unused blocks once a week
> >>   Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
> >>   Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
> >>  Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
> >>     Docs: man:fstrim
> >>
> >> Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.
> >>
> >> linux-ze6w:~ # systemctl status fstrim.service
> >> ● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
> >>   Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
> >>   Active: inactive (dead)
> >>     Docs: man:fstrim(8)
> >> linux-ze6w:~ #
> >
> > OK so it's not set to run. Why do you have FITRIM being called?
>
> No idea.

Well you're going to have to find it. I can't do that for you.



>
> >
> > What are the mount options for this file system?
>
> should have been mounted like:
>
> UUID=bb34b1db-ee47-4367-b207-1de1671087e7  /home/promise2          btrfs  defaults                      0  0
>
> /dev/sdc1 on /home/promise2 type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

No discard option. So you need to find out what's calling fstrim. And
then stop using it and see if the problem goes away.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 21:13                               ` Chris Murphy
@ 2020-01-05 21:58                                 ` Christian Wimmer
  2020-01-05 22:28                                   ` Chris Murphy
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-05 21:58 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 18:13, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Sun, Jan 5, 2020 at 1:36 PM Christian Wimmer <telefonchris@icloud.com> wrote:
>> 
>> 
>> 
>>> On 5. Jan 2020, at 17:30, Chris Murphy <lists@colorremedies.com> wrote:
>>> 
>>> On Sun, Jan 5, 2020 at 12:48 PM Christian Wimmer
>>> <telefonchris@icloud.com> wrote:
>>>> 
>>>> 
>>>> #fdisk -l
>>>> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
>>>> Disk model: Suse 15.1-0 SSD
>>>> Units: sectors of 1 * 512 = 512 bytes
>>>> Sector size (logical/physical): 512 bytes / 4096 bytes
>>>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>>>> Disklabel type: gpt
>>>> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
>>>> 
>>>> Device         Start       End   Sectors  Size Type
>>>> /dev/sda1       2048     18431     16384    8M BIOS boot
>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
>>> 
>>> 
>>> 
>>>> btrfs insp dump-s /dev/sda2
>>>> 
>>>> 
>>>> Here I have only btrfs-progs version 4.19.1:
>>>> 
>>>> linux-ze6w:~ # btrfs version
>>>> btrfs-progs v4.19.1
>>>> linux-ze6w:~ # btrfs insp dump-s /dev/sda2
>>>> superblock: bytenr=65536, device=/dev/sda2
>>>> ---------------------------------------------------------
>>>> csum_type               0 (crc32c)
>>>> csum_size               4
>>>> csum                    0x6d9388e2 [match]
>>>> bytenr                  65536
>>>> flags                   0x1
>>>>                       ( WRITTEN )
>>>> magic                   _BHRfS_M [match]
>>>> fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
>>>> metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
>>>> label
>>>> generation              799183
>>>> root                    724205568
>>>> sys_array_size          97
>>>> chunk_root_generation   797617
>>>> root_level              1
>>>> chunk_root              158835163136
>>>> chunk_root_level        0
>>>> log_root                0
>>>> log_root_transid        0
>>>> log_root_level          0
>>>> total_bytes             272719937536
>>>> bytes_used              106188886016
>>>> sectorsize              4096
>>>> nodesize                16384
>>>> leafsize (deprecated)           16384
>>>> stripesize              4096
>>>> root_dir                6
>>>> num_devices             1
>>>> compat_flags            0x0
>>>> compat_ro_flags         0x0
>>>> incompat_flags          0x163
>>>>                       ( MIXED_BACKREF |
>>>>                         DEFAULT_SUBVOL |
>>>>                         BIG_METADATA |
>>>>                         EXTENDED_IREF |
>>>>                         SKINNY_METADATA )
>>>> cache_generation        799183
>>>> uuid_tree_generation    557352
>>>> dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
>>>> dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
>>>> dev_item.type           0
>>>> dev_item.total_bytes    272719937536
>>>> dev_item.bytes_used     129973092352
>>>> dev_item.io_align       4096
>>>> dev_item.io_width       4096
>>>> dev_item.sector_size    4096
>>>> dev_item.devid          1
>>>> dev_item.dev_group      0
>>>> dev_item.seek_speed     0
>>>> dev_item.bandwidth      0
>>>> dev_item.generation     0
>>> 
>>> Partition map says
>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>> 
>>> Btrfs super says
>>>> total_bytes             272719937536
>>> 
>>> 272719937536*512=532656128
>>> 
>>> Kernel FITRIM want is want=532656128
>>> 
>>> OK so the problem is the Btrfs super isn't set to the size of the
>>> partition. The usual way this happens is user error: partition is
>>> resized (shrink) without resizing the file system first. This file
>>> system is still at risk of having problems even if you disable
>>> fstrim.timer. You need to shrink the file system is the same size as
>>> the partition.
>>> 
>> 
>> Could this be a problem of Parallels Virtual machine that maybe sometimes try to get more space on the hosting file system?
>> One solution would be to have a fixed size of the disc file instead of a growing one.
> 
> I don't see how it's related. Parallels has no ability I'm aware of to
> change the GPT partition map or the Btrfs super block - as in, rewrite
> it out with a modification correctly including all checksums being
> valid. This /dev/sda has somehow been mangled on purpose.
> 
> Again, from the GPT
>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
> 
> The end LBA for sda2 is 419448831, but the start LBA for sda3 is
> 532674560. There's a ~54G gap in there as if something was removed.
> I'm not sure why a software installer would produce this kind of
> layout on purpose, because it has no purpose.
> 
> 

Ok, understand. Very strange. Maybe we should forget about this particular problem.
Should I repair it somehow? And if yes, how?

> 
> 
> 
>> 
>>> 
>>> 
>>>> linux-ze6w:~ # systemctl status fstrim.timer
>>>> ● fstrim.timer - Discard unused blocks once a week
>>>>  Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
>>>>  Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
>>>> Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
>>>>    Docs: man:fstrim
>>>> 
>>>> Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.
>>>> 
>>>> linux-ze6w:~ # systemctl status fstrim.service
>>>> ● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
>>>>  Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
>>>>  Active: inactive (dead)
>>>>    Docs: man:fstrim(8)
>>>> linux-ze6w:~ #
>>> 
>>> OK so it's not set to run. Why do you have FITRIM being called?
>> 
>> No idea.
> 
> Well you're going to have to find it. I can't do that for you.


Ok, I will have a look. Can I simply deactivate the service?


Chris

> 
> 
> 
>> 
>>> 
>>> What are the mount options for this file system?
>> 
>> should have been mounted like:
>> 
>> UUID=bb34b1db-ee47-4367-b207-1de1671087e7  /home/promise2          btrfs  defaults                      0  0
>> 
>> /dev/sdc1 on /home/promise2 type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
> 
> No discard option. So you need to find out what's calling fstrim. And
> then stop using it and see if the problem goes away.
> 
> 
> -- 
> Chris Murphy


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 21:58                                 ` Christian Wimmer
@ 2020-01-05 22:28                                   ` Chris Murphy
  2020-01-06  1:31                                     ` Christian Wimmer
                                                       ` (3 more replies)
  0 siblings, 4 replies; 53+ messages in thread
From: Chris Murphy @ 2020-01-05 22:28 UTC (permalink / raw)
  To: Christian Wimmer
  Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sun, Jan 5, 2020 at 2:58 PM Christian Wimmer <telefonchris@icloud.com> wrote:
> > On 5. Jan 2020, at 18:13, Chris Murphy <lists@colorremedies.com> wrote:
> >
> > On Sun, Jan 5, 2020 at 1:36 PM Christian Wimmer <telefonchris@icloud.com> wrote:
> >>
> >>
> >>
> >>> On 5. Jan 2020, at 17:30, Chris Murphy <lists@colorremedies.com> wrote:
> >>>
> >>> On Sun, Jan 5, 2020 at 12:48 PM Christian Wimmer
> >>> <telefonchris@icloud.com> wrote:
> >>>>
> >>>>
> >>>> #fdisk -l
> >>>> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
> >>>> Disk model: Suse 15.1-0 SSD
> >>>> Units: sectors of 1 * 512 = 512 bytes
> >>>> Sector size (logical/physical): 512 bytes / 4096 bytes
> >>>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> >>>> Disklabel type: gpt
> >>>> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
> >>>>
> >>>> Device         Start       End   Sectors  Size Type
> >>>> /dev/sda1       2048     18431     16384    8M BIOS boot
> >>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> >>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
> >>>
> >>>
> >>>
> >>>> btrfs insp dump-s /dev/sda2
> >>>>
> >>>>
> >>>> Here I have only btrfs-progs version 4.19.1:
> >>>>
> >>>> linux-ze6w:~ # btrfs version
> >>>> btrfs-progs v4.19.1
> >>>> linux-ze6w:~ # btrfs insp dump-s /dev/sda2
> >>>> superblock: bytenr=65536, device=/dev/sda2
> >>>> ---------------------------------------------------------
> >>>> csum_type               0 (crc32c)
> >>>> csum_size               4
> >>>> csum                    0x6d9388e2 [match]
> >>>> bytenr                  65536
> >>>> flags                   0x1
> >>>>                       ( WRITTEN )
> >>>> magic                   _BHRfS_M [match]
> >>>> fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
> >>>> metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
> >>>> label
> >>>> generation              799183
> >>>> root                    724205568
> >>>> sys_array_size          97
> >>>> chunk_root_generation   797617
> >>>> root_level              1
> >>>> chunk_root              158835163136
> >>>> chunk_root_level        0
> >>>> log_root                0
> >>>> log_root_transid        0
> >>>> log_root_level          0
> >>>> total_bytes             272719937536
> >>>> bytes_used              106188886016
> >>>> sectorsize              4096
> >>>> nodesize                16384
> >>>> leafsize (deprecated)           16384
> >>>> stripesize              4096
> >>>> root_dir                6
> >>>> num_devices             1
> >>>> compat_flags            0x0
> >>>> compat_ro_flags         0x0
> >>>> incompat_flags          0x163
> >>>>                       ( MIXED_BACKREF |
> >>>>                         DEFAULT_SUBVOL |
> >>>>                         BIG_METADATA |
> >>>>                         EXTENDED_IREF |
> >>>>                         SKINNY_METADATA )
> >>>> cache_generation        799183
> >>>> uuid_tree_generation    557352
> >>>> dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
> >>>> dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
> >>>> dev_item.type           0
> >>>> dev_item.total_bytes    272719937536
> >>>> dev_item.bytes_used     129973092352
> >>>> dev_item.io_align       4096
> >>>> dev_item.io_width       4096
> >>>> dev_item.sector_size    4096
> >>>> dev_item.devid          1
> >>>> dev_item.dev_group      0
> >>>> dev_item.seek_speed     0
> >>>> dev_item.bandwidth      0
> >>>> dev_item.generation     0
> >>>
> >>> Partition map says
> >>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> >>>
> >>> Btrfs super says
> >>>> total_bytes             272719937536
> >>>
> >>> 272719937536*512=532656128
> >>>
> >>> Kernel FITRIM want is want=532656128
> >>>
> >>> OK so the problem is the Btrfs super isn't set to the size of the
> >>> partition. The usual way this happens is user error: partition is
> >>> resized (shrink) without resizing the file system first. This file
> >>> system is still at risk of having problems even if you disable
> >>> fstrim.timer. You need to shrink the file system is the same size as
> >>> the partition.
> >>>
> >>
> >> Could this be a problem of Parallels Virtual machine that maybe sometimes try to get more space on the hosting file system?
> >> One solution would be to have a fixed size of the disc file instead of a growing one.
> >
> > I don't see how it's related. Parallels has no ability I'm aware of to
> > change the GPT partition map or the Btrfs super block - as in, rewrite
> > it out with a modification correctly including all checksums being
> > valid. This /dev/sda has somehow been mangled on purpose.
> >
> > Again, from the GPT
> >>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> >>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
> >
> > The end LBA for sda2 is 419448831, but the start LBA for sda3 is
> > 532674560. There's a ~54G gap in there as if something was removed.
> > I'm not sure why a software installer would produce this kind of
> > layout on purpose, because it has no purpose.
> >
> >
>
> Ok, understand. Very strange. Maybe we should forget about this particular problem.
> Should I repair it somehow? And if yes, how?


> >>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem

delete this partition, recreate a new one with the same start LBA,
18432 and a new end LBA that matches the actual fs size:

18432+(272719937536/512)=532674560

write it and reboot the VM. You could instead resize Btrfs to match
the partition but that might piss off the kernel if Btrfs thinks it
needs to move block groups from a location outside the partition. So I
would just resize the partition. And then you need to do a scrub and a
btrfs check on this volume to see if it's damaged.

I don't know but I suspect it could be possible that this malformed
root might have resulted in a significant instability of the system at
some point, and in it's last states of confusion as it face planted,
wrote out very spurious data causing your broken Btrfs file system. I
can't prove that.






>
> >
> >
> >
> >>
> >>>
> >>>
> >>>> linux-ze6w:~ # systemctl status fstrim.timer
> >>>> ● fstrim.timer - Discard unused blocks once a week
> >>>>  Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
> >>>>  Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
> >>>> Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
> >>>>    Docs: man:fstrim
> >>>>
> >>>> Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.
> >>>>
> >>>> linux-ze6w:~ # systemctl status fstrim.service
> >>>> ● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
> >>>>  Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
> >>>>  Active: inactive (dead)
> >>>>    Docs: man:fstrim(8)
> >>>> linux-ze6w:~ #
> >>>
> >>> OK so it's not set to run. Why do you have FITRIM being called?
> >>
> >> No idea.
> >
> > Well you're going to have to find it. I can't do that for you.
>
>
> Ok, I will have a look. Can I simply deactivate the service?

fstrim.service is a one shot. The usual method of it being activated
once per week is via fstrim.timer - but your status check of
fstrim.timer says it's disabled. So something else is running fstrim.
I have no idea what it is, you have to find it in order to deactivate
it.

This 12T file system is a single "device" backed by a 12T file on the
Promise drive? And it's a Parallel's formatted VM file? I guess I
would have used raw instead of a Parallels format. That way you can
inspect things from outside the VM. But that's perhaps a minor point.
Check that this 12T (virtual) physical block device inside the guest
has the exact correct size you expect. The partition start and end are
correct, and that this partition's size matches that of the Btrfs
super total_bytes or dev_item.total_bytes. Those two should be the
same if it's a single device Btrfs file system.

Something still doesn't pass the smell test so it's not at all clear
this is an fstrim bug and not some other file system vs device resize
problem.

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 14:17                 ` Christian Wimmer
  2020-01-05 18:50                   ` Chris Murphy
@ 2020-01-05 23:50                   ` Qu Wenruo
  2020-01-06  1:32                     ` Christian Wimmer
                                       ` (2 more replies)
  1 sibling, 3 replies; 53+ messages in thread
From: Qu Wenruo @ 2020-01-05 23:50 UTC (permalink / raw)
  To: Christian Wimmer; +Cc: Qu WenRuo, Anand Jain, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 3508 bytes --]



On 2020/1/5 下午10:17, Christian Wimmer wrote:
> Hi Qu,
> 
> 
>> On 5. Jan 2020, at 01:25, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/1/5 上午1:07, Christian Wimmer wrote:
>>> Hi guys, 
>>>
>>> I run again in a problem with my btrfs files system.
>>> I start wondering if this filesystem type is right for my needs.
>>> Could you please help me in recovering my 12TB partition?
>>>
>>> What happened? 
>>> -> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.
>>>
>>> I immediately powered off again and started my Arch Linux where I have btrfs-progs version 5.4 installed.
>>> I tried one of the commands that you gave me in the past (restore) and I got following messages:
>>>
>>>
>>> btrfs-progs-5.4]# ./btrfs restore -l /dev/sdb1
>>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>>> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
>>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>>> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
>>
>> All these tree blocks are garbage. This doesn't look good at all.
>>
>> The weird found csum pattern make no sense at all.
>>
>> Are you using fstrim or discard mount option? If so, there could be some
>> old bug causing the problem.
> 
> 
> Seems that I am using fstrim (I did not know this, what is it?):
> 
> BTW, sda2 is here my root partition which is practically the same configuration (just smaller) than the 12TB hard disc
> 
> 2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400
> 2020-01-03T11:30:47.479538-03:00 linux-ze6w kernel: [1297857.324658] BTRFS warning (device sda2): failed to trim 1 device(s), last error -5
> 2020-01-03T11:30:48.376543-03:00 linux-ze6w fstrim[27910]: fstrim: /opt: FITRIM ioctl failed: Input/output error

That's the cause. The older kernel had a bug where btrfs can trim
unrelated data, causing data loss.

And I'm afraid that bug trimmed some of your tree blocks, screwing up
the whole fs.


> 2020-01-03T11:30:48.378998-03:00 linux-ze6w kernel: [1297858.223675] attempt to access beyond end of device
> 2020-01-03T11:30:48.379012-03:00 linux-ze6w kernel: [1297858.223677] sda2: rw=3, want=421570540, limit=419430400
> 2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] attempt to access beyond end of device
> 2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] sda2: rw=3, want=429959147, limit=419430400
> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] attempt to access beyond end of device
> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] sda2: rw=3, want=438347754, limit=419430400
> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223680] attempt to access beyond end of device
> 
> Could this be the problem?
> 
> 
> Suse Kernel version is 4.12.14-lp151.28.13-default #1 SMP
I can't find any source tag matching your version. So I can't be 100%
sure about the bug, but that error message still shows the same symptom.

I recommend to check updates about your distro.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 22:28                                   ` Chris Murphy
@ 2020-01-06  1:31                                     ` Christian Wimmer
  2020-01-06  1:33                                     ` Christian Wimmer
                                                       ` (2 subsequent siblings)
  3 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-06  1:31 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 19:28, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Sun, Jan 5, 2020 at 2:58 PM Christian Wimmer <telefonchris@icloud.com> wrote:
>>> On 5. Jan 2020, at 18:13, Chris Murphy <lists@colorremedies.com> wrote:
>>> 
>>> On Sun, Jan 5, 2020 at 1:36 PM Christian Wimmer <telefonchris@icloud.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 5. Jan 2020, at 17:30, Chris Murphy <lists@colorremedies.com> wrote:
>>>>> 
>>>>> On Sun, Jan 5, 2020 at 12:48 PM Christian Wimmer
>>>>> <telefonchris@icloud.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> #fdisk -l
>>>>>> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
>>>>>> Disk model: Suse 15.1-0 SSD
>>>>>> Units: sectors of 1 * 512 = 512 bytes
>>>>>> Sector size (logical/physical): 512 bytes / 4096 bytes
>>>>>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>>>>>> Disklabel type: gpt
>>>>>> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
>>>>>> 
>>>>>> Device         Start       End   Sectors  Size Type
>>>>>> /dev/sda1       2048     18431     16384    8M BIOS boot
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
>>>>> 
>>>>> 
>>>>> 
>>>>>> btrfs insp dump-s /dev/sda2
>>>>>> 
>>>>>> 
>>>>>> Here I have only btrfs-progs version 4.19.1:
>>>>>> 
>>>>>> linux-ze6w:~ # btrfs version
>>>>>> btrfs-progs v4.19.1
>>>>>> linux-ze6w:~ # btrfs insp dump-s /dev/sda2
>>>>>> superblock: bytenr=65536, device=/dev/sda2
>>>>>> ---------------------------------------------------------
>>>>>> csum_type               0 (crc32c)
>>>>>> csum_size               4
>>>>>> csum                    0x6d9388e2 [match]
>>>>>> bytenr                  65536
>>>>>> flags                   0x1
>>>>>>                      ( WRITTEN )
>>>>>> magic                   _BHRfS_M [match]
>>>>>> fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
>>>>>> metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
>>>>>> label
>>>>>> generation              799183
>>>>>> root                    724205568
>>>>>> sys_array_size          97
>>>>>> chunk_root_generation   797617
>>>>>> root_level              1
>>>>>> chunk_root              158835163136
>>>>>> chunk_root_level        0
>>>>>> log_root                0
>>>>>> log_root_transid        0
>>>>>> log_root_level          0
>>>>>> total_bytes             272719937536
>>>>>> bytes_used              106188886016
>>>>>> sectorsize              4096
>>>>>> nodesize                16384
>>>>>> leafsize (deprecated)           16384
>>>>>> stripesize              4096
>>>>>> root_dir                6
>>>>>> num_devices             1
>>>>>> compat_flags            0x0
>>>>>> compat_ro_flags         0x0
>>>>>> incompat_flags          0x163
>>>>>>                      ( MIXED_BACKREF |
>>>>>>                        DEFAULT_SUBVOL |
>>>>>>                        BIG_METADATA |
>>>>>>                        EXTENDED_IREF |
>>>>>>                        SKINNY_METADATA )
>>>>>> cache_generation        799183
>>>>>> uuid_tree_generation    557352
>>>>>> dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
>>>>>> dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
>>>>>> dev_item.type           0
>>>>>> dev_item.total_bytes    272719937536
>>>>>> dev_item.bytes_used     129973092352
>>>>>> dev_item.io_align       4096
>>>>>> dev_item.io_width       4096
>>>>>> dev_item.sector_size    4096
>>>>>> dev_item.devid          1
>>>>>> dev_item.dev_group      0
>>>>>> dev_item.seek_speed     0
>>>>>> dev_item.bandwidth      0
>>>>>> dev_item.generation     0
>>>>> 
>>>>> Partition map says
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>>> 
>>>>> Btrfs super says
>>>>>> total_bytes             272719937536
>>>>> 
>>>>> 272719937536*512=532656128
>>>>> 
>>>>> Kernel FITRIM want is want=532656128
>>>>> 
>>>>> OK so the problem is the Btrfs super isn't set to the size of the
>>>>> partition. The usual way this happens is user error: partition is
>>>>> resized (shrink) without resizing the file system first. This file
>>>>> system is still at risk of having problems even if you disable
>>>>> fstrim.timer. You need to shrink the file system is the same size as
>>>>> the partition.
>>>>> 
>>>> 
>>>> Could this be a problem of Parallels Virtual machine that maybe sometimes try to get more space on the hosting file system?
>>>> One solution would be to have a fixed size of the disc file instead of a growing one.
>>> 
>>> I don't see how it's related. Parallels has no ability I'm aware of to
>>> change the GPT partition map or the Btrfs super block - as in, rewrite
>>> it out with a modification correctly including all checksums being
>>> valid. This /dev/sda has somehow been mangled on purpose.
>>> 
>>> Again, from the GPT
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
>>> 
>>> The end LBA for sda2 is 419448831, but the start LBA for sda3 is
>>> 532674560. There's a ~54G gap in there as if something was removed.
>>> I'm not sure why a software installer would produce this kind of
>>> layout on purpose, because it has no purpose.
>>> 
>>> 
>> 
>> Ok, understand. Very strange. Maybe we should forget about this particular problem.
>> Should I repair it somehow? And if yes, how?
> 
> 
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> 
> delete this partition, recreate a new one with the same start LBA,
> 18432 and a new end LBA that matches the actual fs size:
> 
> 18432+(272719937536/512)=532674560
> 
> write it and reboot the VM. You could instead resize Btrfs to match
> the partition but that might piss off the kernel if Btrfs thinks it
> needs to move block groups from a location outside the partition. So I
> would just resize the partition. And then you need to do a scrub and a
> btrfs check on this volume to see if it's damaged.
> 
> I don't know but I suspect it could be possible that this malformed
> root might have resulted in a significant instability of the system at
> some point, and in it's last states of confusion as it face planted,
> wrote out very spurious data causing your broken Btrfs file system. I
> can't prove that.
> 
> 

Ok, I will try.

> 
> 
> 
> 
>> 
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>> linux-ze6w:~ # systemctl status fstrim.timer
>>>>>> ● fstrim.timer - Discard unused blocks once a week
>>>>>> Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
>>>>>> Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
>>>>>> Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
>>>>>>   Docs: man:fstrim
>>>>>> 
>>>>>> Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.
>>>>>> 
>>>>>> linux-ze6w:~ # systemctl status fstrim.service
>>>>>> ● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
>>>>>> Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
>>>>>> Active: inactive (dead)
>>>>>>   Docs: man:fstrim(8)
>>>>>> linux-ze6w:~ #
>>>>> 
>>>>> OK so it's not set to run. Why do you have FITRIM being called?
>>>> 
>>>> No idea.
>>> 
>>> Well you're going to have to find it. I can't do that for you.
>> 
>> 
>> Ok, I will have a look. Can I simply deactivate the service?
> 
> fstrim.service is a one shot. The usual method of it being activated
> once per week is via fstrim.timer - but your status check of
> fstrim.timer says it's disabled. So something else is running fstrim.
> I have no idea what it is, you have to find it in order to deactivate
> it.


Ok, got it.

> 
> This 12T file system is a single "device" backed by a 12T file on the
> Promise drive? And it's a Parallel's formatted VM file? I guess I
> would have used raw instead of a Parallels format. That way you can
> inspect things from outside the VM. But that's perhaps a minor point.

I would like to do so! I will investigate on how to do so.

I am using this way that I am doing because of the speed.
I have a 2TB Samsung_X5 where I have a 1.8TB disc file and writing a 10GB file takes only 4.4 seconds:

bash$ dd if=/dev/zero of=erasemenow6 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 4.43557 s, 2.4 GB/s
bash$ 

Isn’t that fantastic?

A little worse results I get with the Pegasus Promise3:

bash$ dd if=/dev/zero of=erasemenow6 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 5.49031 s, 1.9 GB/s
bash$ 

This speeds I did not achieve with any other combination of files systems.
And all this I get together with the fantastic btrfs file system that allows me to copy files of 10GB size in just a fraction of a second.
It is just that I am a little afraid now because of the two mega crashes that damaged all my data.

The Parallels Virtual machine can not access the Samsung_X5 or Pegasus directly in order to partition it, so I have to format them in Mac OS with ‘Mac Os Extended (Journaled). Then the Parallels formatted VM files are created on it. 
BTW, this files are created as “Expanding disc”, so they occupy only some MB in the beginning and grow by time. Should this be a problem?

Actually I liked a lot your idea of creating the file in raw format and thus being able to inspect things outside the VM. 
How can I do this? Do you have any idea?

Thanks a lot guys!

Chris




^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 23:50                   ` 12 TB btrfs file system on virtual machine broke again Qu Wenruo
@ 2020-01-06  1:32                     ` Christian Wimmer
  2020-01-11  7:25                     ` Andrei Borzenkov
  2021-10-15 21:01                     ` need help in a broken 2TB BTRFS partition Christian Wimmer
  2 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-06  1:32 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 20:50, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> 
> 
> On 2020/1/5 下午10:17, Christian Wimmer wrote:
>> Hi Qu,
>> 
>> 
>>> On 5. Jan 2020, at 01:25, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>> 
>>> 
>>> 
>>> On 2020/1/5 上午1:07, Christian Wimmer wrote:
>>>> Hi guys, 
>>>> 
>>>> I run again in a problem with my btrfs files system.
>>>> I start wondering if this filesystem type is right for my needs.
>>>> Could you please help me in recovering my 12TB partition?
>>>> 
>>>> What happened? 
>>>> -> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.
>>>> 
>>>> I immediately powered off again and started my Arch Linux where I have btrfs-progs version 5.4 installed.
>>>> I tried one of the commands that you gave me in the past (restore) and I got following messages:
>>>> 
>>>> 
>>>> btrfs-progs-5.4]# ./btrfs restore -l /dev/sdb1
>>>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>>>> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
>>>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>>>> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
>>> 
>>> All these tree blocks are garbage. This doesn't look good at all.
>>> 
>>> The weird found csum pattern make no sense at all.
>>> 
>>> Are you using fstrim or discard mount option? If so, there could be some
>>> old bug causing the problem.
>> 
>> 
>> Seems that I am using fstrim (I did not know this, what is it?):
>> 
>> BTW, sda2 is here my root partition which is practically the same configuration (just smaller) than the 12TB hard disc
>> 
>> 2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400
>> 2020-01-03T11:30:47.479538-03:00 linux-ze6w kernel: [1297857.324658] BTRFS warning (device sda2): failed to trim 1 device(s), last error -5
>> 2020-01-03T11:30:48.376543-03:00 linux-ze6w fstrim[27910]: fstrim: /opt: FITRIM ioctl failed: Input/output error
> 
> That's the cause. The older kernel had a bug where btrfs can trim
> unrelated data, causing data loss.
> 
> And I'm afraid that bug trimmed some of your tree blocks, screwing up
> the whole fs.
> 
> 
>> 2020-01-03T11:30:48.378998-03:00 linux-ze6w kernel: [1297858.223675] attempt to access beyond end of device
>> 2020-01-03T11:30:48.379012-03:00 linux-ze6w kernel: [1297858.223677] sda2: rw=3, want=421570540, limit=419430400
>> 2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] attempt to access beyond end of device
>> 2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] sda2: rw=3, want=429959147, limit=419430400
>> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] attempt to access beyond end of device
>> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] sda2: rw=3, want=438347754, limit=419430400
>> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223680] attempt to access beyond end of device
>> 
>> Could this be the problem?
>> 
>> 
>> Suse Kernel version is 4.12.14-lp151.28.13-default #1 SMP
> I can't find any source tag matching your version. So I can't be 100%
> sure about the bug, but that error message still shows the same symptom.


As far as I remember I just downloaded Suse 15.1. Maybe I run 'zypper up’ after this once.

> 
> I recommend to check updates about your distro.

I will check if I can update anyhow.

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 22:28                                   ` Chris Murphy
  2020-01-06  1:31                                     ` Christian Wimmer
@ 2020-01-06  1:33                                     ` Christian Wimmer
  2020-01-11 17:04                                     ` 12 TB btrfs file system on virtual machine broke again (third time) Christian Wimmer
  2020-01-11 17:23                                     ` Christian Wimmer
  3 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-06  1:33 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs



> On 5. Jan 2020, at 19:28, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Sun, Jan 5, 2020 at 2:58 PM Christian Wimmer <telefonchris@icloud.com> wrote:
>>> On 5. Jan 2020, at 18:13, Chris Murphy <lists@colorremedies.com> wrote:
>>> 
>>> On Sun, Jan 5, 2020 at 1:36 PM Christian Wimmer <telefonchris@icloud.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 5. Jan 2020, at 17:30, Chris Murphy <lists@colorremedies.com> wrote:
>>>>> 
>>>>> On Sun, Jan 5, 2020 at 12:48 PM Christian Wimmer
>>>>> <telefonchris@icloud.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> #fdisk -l
>>>>>> Disk /dev/sda: 256 GiB, 274877906944 bytes, 536870912 sectors
>>>>>> Disk model: Suse 15.1-0 SSD
>>>>>> Units: sectors of 1 * 512 = 512 bytes
>>>>>> Sector size (logical/physical): 512 bytes / 4096 bytes
>>>>>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>>>>>> Disklabel type: gpt
>>>>>> Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215
>>>>>> 
>>>>>> Device         Start       End   Sectors  Size Type
>>>>>> /dev/sda1       2048     18431     16384    8M BIOS boot
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
>>>>> 
>>>>> 
>>>>> 
>>>>>> btrfs insp dump-s /dev/sda2
>>>>>> 
>>>>>> 
>>>>>> Here I have only btrfs-progs version 4.19.1:
>>>>>> 
>>>>>> linux-ze6w:~ # btrfs version
>>>>>> btrfs-progs v4.19.1
>>>>>> linux-ze6w:~ # btrfs insp dump-s /dev/sda2
>>>>>> superblock: bytenr=65536, device=/dev/sda2
>>>>>> ---------------------------------------------------------
>>>>>> csum_type               0 (crc32c)
>>>>>> csum_size               4
>>>>>> csum                    0x6d9388e2 [match]
>>>>>> bytenr                  65536
>>>>>> flags                   0x1
>>>>>>                     ( WRITTEN )
>>>>>> magic                   _BHRfS_M [match]
>>>>>> fsid                    affdbdfa-7b54-4888-b6e9-951da79540a3
>>>>>> metadata_uuid           affdbdfa-7b54-4888-b6e9-951da79540a3
>>>>>> label
>>>>>> generation              799183
>>>>>> root                    724205568
>>>>>> sys_array_size          97
>>>>>> chunk_root_generation   797617
>>>>>> root_level              1
>>>>>> chunk_root              158835163136
>>>>>> chunk_root_level        0
>>>>>> log_root                0
>>>>>> log_root_transid        0
>>>>>> log_root_level          0
>>>>>> total_bytes             272719937536
>>>>>> bytes_used              106188886016
>>>>>> sectorsize              4096
>>>>>> nodesize                16384
>>>>>> leafsize (deprecated)           16384
>>>>>> stripesize              4096
>>>>>> root_dir                6
>>>>>> num_devices             1
>>>>>> compat_flags            0x0
>>>>>> compat_ro_flags         0x0
>>>>>> incompat_flags          0x163
>>>>>>                     ( MIXED_BACKREF |
>>>>>>                       DEFAULT_SUBVOL |
>>>>>>                       BIG_METADATA |
>>>>>>                       EXTENDED_IREF |
>>>>>>                       SKINNY_METADATA )
>>>>>> cache_generation        799183
>>>>>> uuid_tree_generation    557352
>>>>>> dev_item.uuid           8968cd08-0c45-4aff-ab64-65f979b21694
>>>>>> dev_item.fsid           affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
>>>>>> dev_item.type           0
>>>>>> dev_item.total_bytes    272719937536
>>>>>> dev_item.bytes_used     129973092352
>>>>>> dev_item.io_align       4096
>>>>>> dev_item.io_width       4096
>>>>>> dev_item.sector_size    4096
>>>>>> dev_item.devid          1
>>>>>> dev_item.dev_group      0
>>>>>> dev_item.seek_speed     0
>>>>>> dev_item.bandwidth      0
>>>>>> dev_item.generation     0
>>>>> 
>>>>> Partition map says
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>>> 
>>>>> Btrfs super says
>>>>>> total_bytes             272719937536
>>>>> 
>>>>> 272719937536*512=532656128
>>>>> 
>>>>> Kernel FITRIM want is want=532656128
>>>>> 
>>>>> OK so the problem is the Btrfs super isn't set to the size of the
>>>>> partition. The usual way this happens is user error: partition is
>>>>> resized (shrink) without resizing the file system first. This file
>>>>> system is still at risk of having problems even if you disable
>>>>> fstrim.timer. You need to shrink the file system is the same size as
>>>>> the partition.
>>>>> 
>>>> 
>>>> Could this be a problem of Parallels Virtual machine that maybe sometimes try to get more space on the hosting file system?
>>>> One solution would be to have a fixed size of the disc file instead of a growing one.
>>> 
>>> I don't see how it's related. Parallels has no ability I'm aware of to
>>> change the GPT partition map or the Btrfs super block - as in, rewrite
>>> it out with a modification correctly including all checksums being
>>> valid. This /dev/sda has somehow been mangled on purpose.
>>> 
>>> Again, from the GPT
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
>>>>>> /dev/sda3  532674560 536870878   4196319    2G Linux swap
>>> 
>>> The end LBA for sda2 is 419448831, but the start LBA for sda3 is
>>> 532674560. There's a ~54G gap in there as if something was removed.
>>> I'm not sure why a software installer would produce this kind of
>>> layout on purpose, because it has no purpose.
>>> 
>>> 
>> 
>> Ok, understand. Very strange. Maybe we should forget about this particular problem.
>> Should I repair it somehow? And if yes, how?
> 
> 
>>>>>> /dev/sda2      18432 419448831 419430400  200G Linux filesystem
> 
> delete this partition, recreate a new one with the same start LBA,
> 18432 and a new end LBA that matches the actual fs size:
> 
> 18432+(272719937536/512)=532674560
> 
> write it and reboot the VM. You could instead resize Btrfs to match
> the partition but that might piss off the kernel if Btrfs thinks it
> needs to move block groups from a location outside the partition. So I
> would just resize the partition. And then you need to do a scrub and a
> btrfs check on this volume to see if it's damaged.
> 
> I don't know but I suspect it could be possible that this malformed
> root might have resulted in a significant instability of the system at
> some point, and in it's last states of confusion as it face planted,
> wrote out very spurious data causing your broken Btrfs file system. I
> can't prove that.
> 
> 

Ok, I will try.

> 
> 
> 
> 
>> 
>>> 
>>> 
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>> linux-ze6w:~ # systemctl status fstrim.timer
>>>>>> ● fstrim.timer - Discard unused blocks once a week
>>>>>> Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
>>>>>> Active: active (waiting) since Sun 2020-01-05 15:24:59 -03; 1h 19min ago
>>>>>> Trigger: Mon 2020-01-06 00:00:00 -03; 7h left
>>>>>>  Docs: man:fstrim
>>>>>> 
>>>>>> Jan 05 15:24:59 linux-ze6w systemd[1]: Started Discard unused blocks once a week.
>>>>>> 
>>>>>> linux-ze6w:~ # systemctl status fstrim.service
>>>>>> ● fstrim.service - Discard unused blocks on filesystems from /etc/fstab
>>>>>> Loaded: loaded (/usr/lib/systemd/system/fstrim.service; static; vendor preset: disabled)
>>>>>> Active: inactive (dead)
>>>>>>  Docs: man:fstrim(8)
>>>>>> linux-ze6w:~ #
>>>>> 
>>>>> OK so it's not set to run. Why do you have FITRIM being called?
>>>> 
>>>> No idea.
>>> 
>>> Well you're going to have to find it. I can't do that for you.
>> 
>> 
>> Ok, I will have a look. Can I simply deactivate the service?
> 
> fstrim.service is a one shot. The usual method of it being activated
> once per week is via fstrim.timer - but your status check of
> fstrim.timer says it's disabled. So something else is running fstrim.
> I have no idea what it is, you have to find it in order to deactivate
> it.


Ok, got it.

> 
> This 12T file system is a single "device" backed by a 12T file on the
> Promise drive? And it's a Parallel's formatted VM file? I guess I
> would have used raw instead of a Parallels format. That way you can
> inspect things from outside the VM. But that's perhaps a minor point.

I would like to do so! I will investigate on how to do so.

I am using this way that I am doing because of the speed.
I have a 2TB Samsung_X5 where I have a 1.8TB disc file and writing a 10GB file takes only 4.4 seconds:

bash$ dd if=/dev/zero of=erasemenow6 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 4.43557 s, 2.4 GB/s
bash$ 

Isn’t that fantastic?

A little worse results I get with the Pegasus Promise3:

bash$ dd if=/dev/zero of=erasemenow6 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 5.49031 s, 1.9 GB/s
bash$ 

This speeds I did not achieve with any other combination of files systems.
And all this I get together with the fantastic btrfs file system that allows me to copy files of 10GB size in just a fraction of a second.
It is just that I am a little afraid now because of the two mega crashes that damaged all my data.

The Parallels Virtual machine can not access the Samsung_X5 or Pegasus directly in order to partition it, so I have to format them in Mac OS with ‘Mac Os Extended (Journaled). Then the Parallels formatted VM files are created on it. 
BTW, this files are created as “Expanding disc”, so they occupy only some MB in the beginning and grow by time. Should this be a problem?

Actually I liked a lot your idea of creating the file in raw format and thus being able to inspect things outside the VM. 
How can I do this? Do you have any idea?

Thanks a lot guys!

Chris




^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again
  2020-01-05 23:50                   ` 12 TB btrfs file system on virtual machine broke again Qu Wenruo
  2020-01-06  1:32                     ` Christian Wimmer
@ 2020-01-11  7:25                     ` Andrei Borzenkov
  2021-10-15 21:01                     ` need help in a broken 2TB BTRFS partition Christian Wimmer
  2 siblings, 0 replies; 53+ messages in thread
From: Andrei Borzenkov @ 2020-01-11  7:25 UTC (permalink / raw)
  To: Qu Wenruo, Christian Wimmer; +Cc: Qu WenRuo, Anand Jain, linux-btrfs


[-- Attachment #1.1: Type: text/plain, Size: 3719 bytes --]

06.01.2020 02:50, Qu Wenruo пишет:
> 
> 
> On 2020/1/5 下午10:17, Christian Wimmer wrote:
>> Hi Qu,
>>
>>
>>> On 5. Jan 2020, at 01:25, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2020/1/5 上午1:07, Christian Wimmer wrote:
>>>> Hi guys, 
>>>>
>>>> I run again in a problem with my btrfs files system.
>>>> I start wondering if this filesystem type is right for my needs.
>>>> Could you please help me in recovering my 12TB partition?
>>>>
>>>> What happened? 
>>>> -> This time I was just rebooting normally my virtual machine. I discovered during the past days that the system hangs for some seconds so I thought it would be a good idea to reboot my SUSE Linux after 14 days of working. The machine powered off normally but when starting it run into messages like the pasted ones.
>>>>
>>>> I immediately powered off again and started my Arch Linux where I have btrfs-progs version 5.4 installed.
>>>> I tried one of the commands that you gave me in the past (restore) and I got following messages:
>>>>
>>>>
>>>> btrfs-progs-5.4]# ./btrfs restore -l /dev/sdb1
>>>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>>>> checksum verify failed on 3181912915968 found 00000071 wanted 00000066
>>>> checksum verify failed on 3181912915968 found 000000A9 wanted 00000064
>>>> bad tree block 3181912915968, bytenr mismatch, want=3181912915968, have=4908658797358025935
>>>
>>> All these tree blocks are garbage. This doesn't look good at all.
>>>
>>> The weird found csum pattern make no sense at all.
>>>
>>> Are you using fstrim or discard mount option? If so, there could be some
>>> old bug causing the problem.
>>
>>
>> Seems that I am using fstrim (I did not know this, what is it?):
>>
>> BTW, sda2 is here my root partition which is practically the same configuration (just smaller) than the 12TB hard disc
>>
>> 2020-01-03T11:30:47.479028-03:00 linux-ze6w kernel: [1297857.324177] sda2: rw=2051, want=532656128, limit=419430400
>> 2020-01-03T11:30:47.479538-03:00 linux-ze6w kernel: [1297857.324658] BTRFS warning (device sda2): failed to trim 1 device(s), last error -5
>> 2020-01-03T11:30:48.376543-03:00 linux-ze6w fstrim[27910]: fstrim: /opt: FITRIM ioctl failed: Input/output error
> 
> That's the cause. The older kernel had a bug where btrfs can trim
> unrelated data, causing data loss.
> 
> And I'm afraid that bug trimmed some of your tree blocks, screwing up
> the whole fs.
> 
> 
>> 2020-01-03T11:30:48.378998-03:00 linux-ze6w kernel: [1297858.223675] attempt to access beyond end of device
>> 2020-01-03T11:30:48.379012-03:00 linux-ze6w kernel: [1297858.223677] sda2: rw=3, want=421570540, limit=419430400
>> 2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] attempt to access beyond end of device
>> 2020-01-03T11:30:48.379013-03:00 linux-ze6w kernel: [1297858.223678] sda2: rw=3, want=429959147, limit=419430400
>> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] attempt to access beyond end of device
>> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223679] sda2: rw=3, want=438347754, limit=419430400
>> 2020-01-03T11:30:48.379014-03:00 linux-ze6w kernel: [1297858.223680] attempt to access beyond end of device
>>
>> Could this be the problem?
>>
>>
>> Suse Kernel version is 4.12.14-lp151.28.13-default #1 SMP
> I can't find any source tag matching your version. 

This is commit 3e458e04fab3ee8b0d234acf09db1d9279f356d9 of kernel-source.

> So I can't be 100%
> sure about the bug, but that error message still shows the same symptom.
> 
> I recommend to check updates about your distro.
> 
> Thanks,
> Qu
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

^ permalink raw reply	[flat|nested] 53+ messages in thread

* 12 TB btrfs file system on virtual machine broke again (third time)
  2020-01-05 22:28                                   ` Chris Murphy
  2020-01-06  1:31                                     ` Christian Wimmer
  2020-01-06  1:33                                     ` Christian Wimmer
@ 2020-01-11 17:04                                     ` Christian Wimmer
  2020-01-11 17:23                                     ` Christian Wimmer
  3 siblings, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-11 17:04 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

Hi guys,

and again my 12TB virtual machine btrfs file system broke into pieces.
I checked today in the morning and there was not any error message regarding btrfs.
10 minutes ago I needed to suspend the virtual machine and after I resumed, all folders on my (still mounted) file system have gone.
So in my opinion, the Parallels Virtual machine is not cleanly handling this.

Is there anything I can do now to save my data?
This are the error log messages:



# tail -f /var/log/messages
2020-01-11T13:57:55.739361-03:00 linux-ze6w kernel: [512260.892881] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:57:57.986773-03:00 linux-ze6w chronyd[1943]: Selected source 185.184.223.224
2020-01-11T13:58:00.766543-03:00 linux-ze6w kernel: [512265.894004] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:00.766559-03:00 linux-ze6w kernel: [512265.894089] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:02.891790-03:00 linux-ze6w sudo:    chris : TTY=pts/100 ; PWD=/home/chris ; USER=root ; COMMAND=/usr/bin/su
2020-01-11T13:58:02.893238-03:00 linux-ze6w sudo: pam_unix(sudo:session): session opened for user root by chris(uid=0)
2020-01-11T13:58:02.897765-03:00 linux-ze6w su: (to root) chris on pts/100
2020-01-11T13:58:02.898820-03:00 linux-ze6w su: pam_unix(su:session): session opened for user root by chris(uid=0)
2020-01-11T13:58:05.798179-03:00 linux-ze6w kernel: [512270.925046] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:05.798193-03:00 linux-ze6w kernel: [512270.925102] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:10.822192-03:00 linux-ze6w kernel: [512275.951137] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:10.822204-03:00 linux-ze6w kernel: [512275.951174] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:15.850385-03:00 linux-ze6w kernel: [512280.977909] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:15.850400-03:00 linux-ze6w kernel: [512280.977975] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:20.882236-03:00 linux-ze6w kernel: [512286.010503] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:20.882247-03:00 linux-ze6w kernel: [512286.010577] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:22.234199-03:00 linux-ze6w kernel: [512287.361025] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562110521344
2020-01-11T13:58:22.234213-03:00 linux-ze6w kernel: [512287.364054] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562110472192
2020-01-11T13:58:22.234214-03:00 linux-ze6w kernel: [512287.364060] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562110455808
2020-01-11T13:58:22.250399-03:00 linux-ze6w kernel: [512287.377085] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562110488576
2020-01-11T13:58:22.270208-03:00 linux-ze6w kernel: [512287.397958] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562110455808
2020-01-11T13:58:22.270221-03:00 linux-ze6w kernel: [512287.398038] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562110455808
2020-01-11T13:58:22.270221-03:00 linux-ze6w kernel: [512287.398067] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562110455808
2020-01-11T13:58:23.098451-03:00 linux-ze6w kernel: [512288.225913] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562059993088
2020-01-11T13:58:25.910246-03:00 linux-ze6w kernel: [512291.036604] btree_readpage_end_io_hook: 17 callbacks suppressed
2020-01-11T13:58:25.910266-03:00 linux-ze6w kernel: [512291.036607] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:25.910267-03:00 linux-ze6w kernel: [512291.036708] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:26.338226-03:00 linux-ze6w kernel: [512291.467459] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:26.338237-03:00 linux-ze6w kernel: [512291.467517] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:26.338238-03:00 linux-ze6w kernel: [512291.467585] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:26.338238-03:00 linux-ze6w kernel: [512291.467624] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:28.882182-03:00 linux-ze6w kernel: [512294.011515] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:28.914197-03:00 linux-ze6w kernel: [512294.041798] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:28.914214-03:00 linux-ze6w kernel: [512294.041883] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:28.914214-03:00 linux-ze6w kernel: [512294.041936] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:30.938342-03:00 linux-ze6w kernel: [512296.064421] btree_readpage_end_io_hook: 4 callbacks suppressed
2020-01-11T13:58:30.938354-03:00 linux-ze6w kernel: [512296.064425] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:30.938355-03:00 linux-ze6w kernel: [512296.064503] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:32.078216-03:00 linux-ze6w kernel: [512297.204873] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562285453312
2020-01-11T13:58:32.078230-03:00 linux-ze6w kernel: [512297.205110] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562285453312
2020-01-11T13:58:32.078230-03:00 linux-ze6w kernel: [512297.205174] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562285453312
2020-01-11T13:58:32.078232-03:00 linux-ze6w kernel: [512297.205212] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562285453312
2020-01-11T13:58:35.962296-03:00 linux-ze6w kernel: [512301.090582] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:35.962308-03:00 linux-ze6w kernel: [512301.090653] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:40.482196-03:00 linux-ze6w kernel: [512305.611514] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:40.482211-03:00 linux-ze6w kernel: [512305.611572] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:40.482212-03:00 linux-ze6w kernel: [512305.611624] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:40.482213-03:00 linux-ze6w kernel: [512305.611664] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562258780160
2020-01-11T13:58:40.990210-03:00 linux-ze6w kernel: [512306.116117] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:40.990226-03:00 linux-ze6w kernel: [512306.116218] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:42.530181-03:00 linux-ze6w kernel: [512307.659684] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:42.530194-03:00 linux-ze6w kernel: [512307.659724] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:42.530194-03:00 linux-ze6w kernel: [512307.659762] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:42.530195-03:00 linux-ze6w kernel: [512307.659800] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345484288
2020-01-11T13:58:43.462391-03:00 linux-ze6w kernel: [512308.589145] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:43.462410-03:00 linux-ze6w kernel: [512308.590058] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:43.462410-03:00 linux-ze6w kernel: [512308.590113] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:43.462412-03:00 linux-ze6w kernel: [512308.590152] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:46.014183-03:00 linux-ze6w kernel: [512311.142992] btree_readpage_end_io_hook: 20 callbacks suppressed
2020-01-11T13:58:46.014200-03:00 linux-ze6w kernel: [512311.142994] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:46.014200-03:00 linux-ze6w kernel: [512311.143077] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:46.290228-03:00 linux-ze6w kernel: [512311.418594] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:46.290246-03:00 linux-ze6w kernel: [512311.418638] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:51.042208-03:00 linux-ze6w kernel: [512316.170407] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:51.042229-03:00 linux-ze6w kernel: [512316.170495] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:53.786220-03:00 linux-ze6w kernel: [512318.911878] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562059993088
2020-01-11T13:58:53.786240-03:00 linux-ze6w kernel: [512318.911963] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562059993088
2020-01-11T13:58:53.786241-03:00 linux-ze6w kernel: [512318.913482] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:53.786241-03:00 linux-ze6w kernel: [512318.913583] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562345451520
2020-01-11T13:58:53.786242-03:00 linux-ze6w kernel: [512318.913588] BTRFS: error (device sdc1) in __btrfs_run_delayed_items:1148: errno=-5 IO failure
2020-01-11T13:58:53.786242-03:00 linux-ze6w kernel: [512318.913590] BTRFS info (device sdc1): forced readonly
2020-01-11T13:58:53.786242-03:00 linux-ze6w kernel: [512318.913591] BTRFS warning (device sdc1): Skipping commit of aborted transaction.
2020-01-11T13:58:53.786243-03:00 linux-ze6w kernel: [512318.913592] BTRFS: error (device sdc1) in cleanup_transaction:1881: errno=-5 IO failure
2020-01-11T13:58:56.078317-03:00 linux-ze6w kernel: [512321.203683] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472
2020-01-11T13:58:56.078334-03:00 linux-ze6w kernel: [512321.203751] BTRFS error (device sdc1): bad tree block start 14275350892879035392 2562508521472

Thanks,

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* 12 TB btrfs file system on virtual machine broke again (third time)
  2020-01-05 22:28                                   ` Chris Murphy
                                                       ` (2 preceding siblings ...)
  2020-01-11 17:04                                     ` 12 TB btrfs file system on virtual machine broke again (third time) Christian Wimmer
@ 2020-01-11 17:23                                     ` Christian Wimmer
  2020-01-11 19:46                                       ` Chris Murphy
  3 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-11 17:23 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

Hi guys,

here the last lines before the suspend and after it:


66939.439945] Buffer I/O error on dev loop1, logical block 0, async page read
[66939.621686] print_req_error: I/O error, dev loop1, sector 83885952
[66939.621738] print_req_error: I/O error, dev loop1, sector 83885952
[66939.621740] Buffer I/O error on dev loop1, logical block 10485744, async page read
[509652.385601] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[509652.385623] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current] 
[509652.385628] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
[509652.385631] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 16 b2 c0 00 00 00 20 00 00
[509652.385634] print_req_error: critical target error, dev sdb, sector 1487552
[509652.385642] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
[509652.386117] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
[509652.386120] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b65150
[509652.386132] sd 6:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[509652.386135] sd 6:0:0:0: [sdb] tag#1 Sense Key : Illegal Request [current] 
[509652.386137] sd 6:0:0:0: [sdb] tag#1 Add. Sense: Invalid command operation code
[509652.386139] sd 6:0:0:0: [sdb] tag#1 CDB: Write(16) 8a 00 00 00 00 00 01 1d 9a 18 00 00 02 00 00 00
[509652.386141] print_req_error: critical target error, dev sdb, sector 18717208
[509652.386158] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0
[509682.560535] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
[509682.560537] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b65190
[509682.560566] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[509682.560567] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current] 
[509682.560569] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
[509682.560571] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 36 b2 c0 00 00 00 20 00 00
[509682.560572] print_req_error: critical target error, dev sdb, sector 3584704
[509682.560579] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 2, flush 0, corrupt 0, gen 0
[509682.560667] BTRFS: error (device sdb1) in btrfs_start_dirty_block_groups:3716: errno=-5 IO failure
[509682.560669] BTRFS info (device sdb1): forced readonly
[509712.672325] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
[509712.672330] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b651d0
[509712.672373] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[509712.672377] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current] 
[509712.672379] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
[509712.672382] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 16 b2 c0 00 00 00 20 00 00
[509712.672384] print_req_error: critical target error, dev sdb, sector 1487552
[509712.672388] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 3, flush 0, corrupt 0, gen 0
[509742.783978] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
[509742.783982] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b65210
[509742.784016] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[509742.784019] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current] 
[509742.784021] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
[509742.784024] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 36 b2 c0 00 00 00 20 00 00
[509742.784026] print_req_error: critical target error, dev sdb, sector 3584704
[509742.784031] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 4, flush 0, corrupt 0, gen 0
[509742.784079] BTRFS: error (device sdb1) in btrfs_start_dirty_block_groups:3716: errno=-5 IO failure
[509742.784082] BTRFS warning (device sdb1): Skipping commit of aborted transaction.
[509742.784084] BTRFS: error (device sdb1) in cleanup_transaction:1881: errno=-5 IO failure
[509742.784084] BTRFS info (device sdb1): delayed_refs has NO entry
[512066.481724] usb 4-1: USB disconnect, device number 2
[512066.484604] sd 6:0:0:0: [sdb] Synchronizing SCSI cache
[512066.661516] usb 2-1: USB disconnect, device number 2
[512066.661725] usblp0: removed
[512066.716156] usb 1-1: USB disconnect, device number 2
[512066.751428] sd 6:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
[512066.811207] usb 2-2: USB disconnect, device number 3
[512066.811350] usblp1: removed
[512066.963131] usb 2-3: USB disconnect, device number 4
[512066.963296] usblp2: removed
[512067.091114] usb 4-1: new SuperSpeed USB device number 3 using xhci_hcd
[512067.107446] usb 1-1: new full-speed USB device number 3 using uhci_hcd
[512067.113881] usb 2-4: USB disconnect, device number 5
[512067.114143] usblp3: removed
[512067.126193] usb 4-1: New USB device found, idVendor=2109, idProduct=0711
[512067.126195] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[512067.126196] usb 4-1: Product: 20231
[512067.126196] usb 4-1: Manufacturer: Ugreen
[512067.126197] usb 4-1: SerialNumber: 000000128DA4
[512067.140176] scsi host9: uas
[512067.141472] scsi 9:0:0:0: Direct-Access     ST6000DM 004-2EH11C       DN02 PQ: 0 ANSI: 6
[512067.142615] sd 9:0:0:0: 


Thanks,


Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again (third time)
  2020-01-11 17:23                                     ` Christian Wimmer
@ 2020-01-11 19:46                                       ` Chris Murphy
  2020-01-13 19:41                                         ` 12 TB btrfs file system on virtual machine broke again (fourth time) Christian Wimmer
  0 siblings, 1 reply; 53+ messages in thread
From: Chris Murphy @ 2020-01-11 19:46 UTC (permalink / raw)
  To: Christian Wimmer
  Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Sat, Jan 11, 2020 at 10:23 AM Christian Wimmer
<telefonchris@icloud.com> wrote:
>
> Hi guys,
>
> here the last lines before the suspend and after it:
>
>
> 66939.439945] Buffer I/O error on dev loop1, logical block 0, async page read
> [66939.621686] print_req_error: I/O error, dev loop1, sector 83885952
> [66939.621738] print_req_error: I/O error, dev loop1, sector 83885952
> [66939.621740] Buffer I/O error on dev loop1, logical block 10485744, async page read

I don't know what /dev/loop1 is or the origin of the i/o error, but
I'm gonna guess it's a problem with the underlying device, or a bug
related to either the VM or the host or guest kernels or some
interaction of the three.

> [509652.385601] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [509652.385623] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current]
> [509652.385628] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
> [509652.385631] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 16 b2 c0 00 00 00 20 00 00
> [509652.385634] print_req_error: critical target error, dev sdb, sector 1487552
> [509652.385642] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
> [509652.386117] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
> [509652.386120] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b65150
> [509652.386132] sd 6:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [509652.386135] sd 6:0:0:0: [sdb] tag#1 Sense Key : Illegal Request [current]
> [509652.386137] sd 6:0:0:0: [sdb] tag#1 Add. Sense: Invalid command operation code
> [509652.386139] sd 6:0:0:0: [sdb] tag#1 CDB: Write(16) 8a 00 00 00 00 00 01 1d 9a 18 00 00 02 00 00 00
> [509652.386141] print_req_error: critical target error, dev sdb, sector 18717208
> [509652.386158] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0
> [509682.560535] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
> [509682.560537] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b65190
> [509682.560566] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [509682.560567] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current]
> [509682.560569] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
> [509682.560571] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 36 b2 c0 00 00 00 20 00 00
> [509682.560572] print_req_error: critical target error, dev sdb, sector 3584704
> [509682.560579] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 2, flush 0, corrupt 0, gen 0
> [509682.560667] BTRFS: error (device sdb1) in btrfs_start_dirty_block_groups:3716: errno=-5 IO failure
> [509682.560669] BTRFS info (device sdb1): forced readonly
> [509712.672325] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
> [509712.672330] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b651d0
> [509712.672373] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [509712.672377] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current]
> [509712.672379] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
> [509712.672382] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 16 b2 c0 00 00 00 20 00 00
> [509712.672384] print_req_error: critical target error, dev sdb, sector 1487552
> [509712.672388] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 3, flush 0, corrupt 0, gen 0
> [509742.783978] xhci_hcd 0000:00:1d.6: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
> [509742.783982] xhci_hcd 0000:00:1d.6: ep deq seg = ffff880bd3fbab40, deq ptr = ffff880bd3b65210
> [509742.784016] sd 6:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [509742.784019] sd 6:0:0:0: [sdb] tag#0 Sense Key : Illegal Request [current]
> [509742.784021] sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation code
> [509742.784024] sd 6:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 36 b2 c0 00 00 00 20 00 00
> [509742.784026] print_req_error: critical target error, dev sdb, sector 3584704
> [509742.784031] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 4, flush 0, corrupt 0, gen 0
> [509742.784079] BTRFS: error (device sdb1) in btrfs_start_dirty_block_groups:3716: errno=-5 IO failure
> [509742.784082] BTRFS warning (device sdb1): Skipping commit of aborted transaction.
> [509742.784084] BTRFS: error (device sdb1) in cleanup_transaction:1881: errno=-5 IO failure
> [509742.784084] BTRFS info (device sdb1): delayed_refs has NO entry
> [512066.481724] usb 4-1: USB disconnect, device number 2
> [512066.484604] sd 6:0:0:0: [sdb] Synchronizing SCSI cache
> [512066.661516] usb 2-1: USB disconnect, device number 2
> [512066.661725] usblp0: removed

There are multiple hardware related errors here. Timing wise, it looks
like some kind of USB bus related error "xhci_hcd 0000:00:1d.6:
Mismatch between..." and maybe in the course of that error is what
causes the drive, /dev/sdb, to think a command it received is invalid,
hence "sd 6:0:0:0: [sdb] tag#0 Add. Sense: Invalid command operation
code" and the subsequent sector error, and the subsequent Btrfs error.

So Btrfs is getting confused because lower layers it has no control
over are doing the wrong thing; resulting in either dropped reads or
writes. Dropped writes can cause permanent corruption, depending on
what the writes are.

And then it looks like a USB device is disconnected.

Is /dev/sdb a USB stick or  drive in a USB enclosure? That's not
working out well, whatever it is.


> [512066.716156] usb 1-1: USB disconnect, device number 2
> [512066.751428] sd 6:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
> [512066.811207] usb 2-2: USB disconnect, device number 3
> [512066.811350] usblp1: removed
> [512066.963131] usb 2-3: USB disconnect, device number 4
> [512066.963296] usblp2: removed
> [512067.091114] usb 4-1: new SuperSpeed USB device number 3 using xhci_hcd
> [512067.107446] usb 1-1: new full-speed USB device number 3 using uhci_hcd
> [512067.113881] usb 2-4: USB disconnect, device number 5
> [512067.114143] usblp3: removed
> [512067.126193] usb 4-1: New USB device found, idVendor=2109, idProduct=0711
> [512067.126195] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> [512067.126196] usb 4-1: Product: 20231
> [512067.126196] usb 4-1: Manufacturer: Ugreen
> [512067.126197] usb 4-1: SerialNumber: 000000128DA4
> [512067.140176] scsi host9: uas
> [512067.141472] scsi 9:0:0:0: Direct-Access     ST6000DM 004-2EH11C       DN02 PQ: 0 ANSI: 6
> [512067.142615] sd 9:0:0:0:


The problem with snippets is that they're over too small of a time
period, and after problems already started, to determine the sequence
of what went wrong, and what else is involved in the confusion. The
problem with getting the entire dmesg is that it's tedious to read
through to try and find the problem.

The gist though is the setup is flawed and now you've had three
failures it's telling you something is wrong.

Any hardware errors inside a VM guest, can be either hardware or
software errors in the host. It could be the hypervisor. It could be
the host kernel. Have you checked the host kernel messages while these
errors are happening inside the VM? What about the hypervisor logs?





-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again (fourth time)
  2020-01-11 19:46                                       ` Chris Murphy
@ 2020-01-13 19:41                                         ` Christian Wimmer
  2020-01-13 20:03                                           ` Chris Murphy
  0 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2020-01-13 19:41 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

Hi guys,

just to update you. 

I tried a 4th time with a brand new 12 TB archive and right after setting up everything and putting some data on it I performed a controlled suspend and restore and so on and yes, it broke again with that terrible messages that it can not mount any more the file system.

Then I decided to create only 2TB archives (hard disks) as the slider in the Parallels VM suggests to do.
I created 4 x 2TB files, one with persistent size and three expandable.
I filled up all hard discs with data and rebooted 5 times (even when writing data to it) and everything runs very stable.

So to resume, the Parallels Virtual machine has a problem when the virtual disk is being selected bigger than 2TB. There is (and was) never a problem with the btrfs files system I think.

Sorry for bothering you all the time with my bad setup.

Thanks a lot for all your help!

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: 12 TB btrfs file system on virtual machine broke again (fourth time)
  2020-01-13 19:41                                         ` 12 TB btrfs file system on virtual machine broke again (fourth time) Christian Wimmer
@ 2020-01-13 20:03                                           ` Chris Murphy
  2020-01-31 16:35                                             ` btrfs not booting any more Christian Wimmer
  2020-05-08 12:20                                             ` btrfs reports bad key ordering after out of memory situation Christian Wimmer
  0 siblings, 2 replies; 53+ messages in thread
From: Chris Murphy @ 2020-01-13 20:03 UTC (permalink / raw)
  To: Christian Wimmer
  Cc: Chris Murphy, Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

On Mon, Jan 13, 2020 at 12:41 PM Christian Wimmer
<telefonchris@icloud.com> wrote:
>
> Hi guys,
>
> just to update you.
>
> I tried a 4th time with a brand new 12 TB archive and right after setting up everything and putting some data on it I performed a controlled suspend and restore and so on and yes, it broke again with that terrible messages that it can not mount any more the file system.
>
> Then I decided to create only 2TB archives (hard disks) as the slider in the Parallels VM suggests to do.
> I created 4 x 2TB files, one with persistent size and three expandable.
> I filled up all hard discs with data and rebooted 5 times (even when writing data to it) and everything runs very stable.
>
> So to resume, the Parallels Virtual machine has a problem when the virtual disk is being selected bigger than 2TB. There is (and was) never a problem with the btrfs files system I think.
>
> Sorry for bothering you all the time with my bad setup.
>
> Thanks a lot for all your help!

Yeah it's no problem. I mean, there's no way to know in advance that
the setup is bad, it has to be proven. And with any complicated setup,
it's really tedious to find out where the problem is happening. And
even still it's not certain if this is some bug flushing to Parallels,
or if it's getting the flush and not doing it completely for very
large backing files, or if the host OS file system or block device
drive is dropping something.

There are definitely bugs in Btrfs too so you can't absolutely exclude
the possibility, but those look different. They tend to be pernicious.
Whereas in your case these problems appeared quickly, back to back.
But also it's super complicated, multiple layers have to all work
exactly right or there will be problems. I would be very skeptical of
VM guest suspend with any file system. It should be true there is fs
sync that happens when the guest kernel is told to suspend, but then
what happens to that flush once it's in the VM or hypervisor? *shrug*
How long does it take to actually get from VM all the way to stable
media?

Because necessarily you have to consider worst case scenario, like
doing file writes and a complete loss of power. Since Btrfs and kernel
block layers aren't directly responsible for writing to the disks,
it's ambiguous exactly when and in what order, the writes do get to
stable media.

Ok so now what? It's entirely possible you've totally eliminated the
problem. Or it might be possible you've only reduced the chance it'll
happen - meaning something like it will happen at some point.

Is it superfluous extra work for no benefit, to unmount this Btrfs
file system, or use fsfreeze, prior to suspending the VM? Or never
suspend the VM? Or maybe the non-default mount option flushoncommit is
useful in this case? It's a huge hassle to have to rebuild a 12TB
volume, it's a high penalty.

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 53+ messages in thread

* btrfs not booting any more
  2020-01-13 20:03                                           ` Chris Murphy
@ 2020-01-31 16:35                                             ` Christian Wimmer
  2020-05-08 12:20                                             ` btrfs reports bad key ordering after out of memory situation Christian Wimmer
  1 sibling, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-01-31 16:35 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

Hi guys,

I would need one more help from you before I can do that things by myself.

I have a different machine that runs windows and inside Vmware Fusion 15.
Inside runs as well Suse 15.1.

The linux  operating system is on a 1TB virtual hard disk with following setup:


Disk /dev/sdb: 1000 GiB, 1073741824000 bytes, 2097152000 sectors
Disk model: VMware Virtual S
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 186C0CD6-F3B8-471C-B2AF-AE3D325EC215

Device          Start        End    Sectors   Size Type
/dev/sdb1        2048      18431      16384     8M BIOS boot
/dev/sdb2       18432 1920520191 1920501760 915.8G Linux filesystem
/dev/sdb3  2082992128 2097149951   14157824   6.8G Linux filesystem



The OS is on /dev/sdb2 but it does not boot any more (stays at grub_rescue).

What can I do now to repair it?




root@ubuntu:~# btrfs ins dump-tree -t root  /dev/sdb2
btrfs-progs v5.2.1 
parent transid verify failed on 72980660224 wanted 652165 found 652163
parent transid verify failed on 72980660224 wanted 652165 found 652163
Ignoring transid failure
leaf parent key incorrect 72980660224
Couldn't setup extent tree
parent transid verify failed on 72980660224 wanted 652165 found 652163
Ignoring transid failure
leaf parent key incorrect 72980660224
Couldn't setup device tree
ERROR: unable to open /dev/sdb2
root@ubuntu:~# btrfs ins dump-s /dev/sdb2
superblock: bytenr=65536, device=/dev/sdb2
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0xd35229e5 [match]
bytenr			65536
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			affdbdfa-7b54-4888-b6e9-951da79540a3
metadata_uuid		affdbdfa-7b54-4888-b6e9-951da79540a3
label			
generation		652165
root			72978268160
sys_array_size		97
chunk_root_generation	649961
root_level		1
chunk_root		1265374330880
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		983296901120
bytes_used		504341774336
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
root_dir		6
num_devices		1
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x163
			( MIXED_BACKREF |
			  DEFAULT_SUBVOL |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
cache_generation	652165
uuid_tree_generation	557352
dev_item.uuid		8968cd08-0c45-4aff-ab64-65f979b21694
dev_item.fsid		affdbdfa-7b54-4888-b6e9-951da79540a3 [match]
dev_item.type		0
dev_item.total_bytes	983296901120
dev_item.bytes_used	507877785600
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0




Where can I find default repair methods?

Thanks,

Chris



^ permalink raw reply	[flat|nested] 53+ messages in thread

* btrfs reports bad key ordering after out of memory situation
  2020-01-13 20:03                                           ` Chris Murphy
  2020-01-31 16:35                                             ` btrfs not booting any more Christian Wimmer
@ 2020-05-08 12:20                                             ` Christian Wimmer
  1 sibling, 0 replies; 53+ messages in thread
From: Christian Wimmer @ 2020-05-08 12:20 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Qu Wenruo, Qu WenRuo, Anand Jain, linux-btrfs

Hi Chris,

after reducing all partition sizes to 2TB I run my system now since 3 months without any problems. Many reboots and still very stable. Yesterday I run into a situation where I run out of memory on my 48Gbyte Virtual machine and suddenly the home partition was mounted read only. After a reboot the system did not come back.

I did a btrfs check —readonly /dev/sda2 and it reported 

btrfs restore -l  /dev/sdb2 test
 tree key (EXTENT_TREE ROOT_ITEM 0) 278911795200 level 2
 tree key (DEV_TREE ROOT_ITEM 0) 279226892288 level 0
 tree key (FS_TREE ROOT_ITEM 0) 22118400 level 0
 tree key (CSUM_TREE ROOT_ITEM 0) 437370880 level 2
 tree key (QUOTA_TREE ROOT_ITEM 0) 279164583936 level 0
 tree key (UUID_TREE ROOT_ITEM 0) 460750848 level 0
 tree key (257 ROOT_ITEM 0) 22642688 level 0
 tree key (258 ROOT_ITEM 0) 279163060224 level 1
 tree key (259 ROOT_ITEM 0) 278998171648 level 2
 tree key (260 ROOT_ITEM 0) 279162814464 level 2
 tree key (261 ROOT_ITEM 0) 594477056 level 0
 tree key (262 ROOT_ITEM 0) 278959783936 level 1
 tree key (263 ROOT_ITEM 0) 279005970432 level 2
 tree key (264 ROOT_ITEM 0) 279162961920 level 2
 tree key (265 ROOT_ITEM 0) 655015936 level 0
 tree key (266 ROOT_ITEM 0) 654376960 level 1
 tree key (267 ROOT_ITEM 0) 461029376 level 0
 tree key (268 ROOT_ITEM 0) 279156670464 level 2
bad key ordering 192 193
bad key ordering 192 193
 tree key (DATA_RELOC_TREE ROOT_ITEM 0) 43673436160 level 0


Could you please help me to repair this system?

Which command I can use ?

For the btrfs restore command I used the newest btrfs version.
The btrfs version the system was running on is 4.19 

Thanks a lot,

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* need help in a broken 2TB BTRFS partition
  2020-01-05 23:50                   ` 12 TB btrfs file system on virtual machine broke again Qu Wenruo
  2020-01-06  1:32                     ` Christian Wimmer
  2020-01-11  7:25                     ` Andrei Borzenkov
@ 2021-10-15 21:01                     ` Christian Wimmer
  2021-10-16 10:08                       ` Qu Wenruo
  2 siblings, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2021-10-15 21:01 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

Hi Qu,

I hope I find you well.

Almost two years that my system runs without any failure.
Since this is very boring I tried make my life somehow harder and tested again the snapshot feature of my Parallels Desktop installation yesterday:-)
When I erased the old snapshots I could feel (and actually hear) already that the system is writing too much to the partitions.
What I want to say is that it took too long (for any reason) to erase the old snapshots and to shut the system down.

Well, after booting I saw that one of the discs is not coming back and I got the following error message:

Suse_Tumbleweed:/home/proc # btrfs check /dev/sdd1
Opening filesystem to check...
parent transid verify failed on 324239360 wanted 208553 found 184371
parent transid verify failed on 324239360 wanted 208553 found 184371
parent transid verify failed on 324239360 wanted 208553 found 184371
Ignoring transid failure
leaf parent key incorrect 324239360
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system


Could you help me to debug and repair this please?

I already run the command btrfs restore /dev/sdd1 . and could restore 90% of the data but not the important last 10%.

My system is:

Suse Tumbleweed inside Parallels Desktop on a Mac Mini

Mac Min: Big Sur
Parallels Desktop: 17.1.0
Suse: Linux Suse_Tumbleweed 5.13.2-1-default #1 SMP Thu Jul 15 03:36:02 UTC 2021 (89416ca) x86_64 x86_64 x86_64 GNU/Linux

Suse_Tumbleweed:~ # btrfs --version
btrfs-progs v5.13 

The disk /dev/sdd1 is one of several 2TB partitions that reside on a NAS attached to the Mac Mini like 

Disk /dev/sde: 2 TiB, 2197949513728 bytes, 4292870144 sectors
Disk model: Linux_raid5_2tb_
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 942781EC-8969-408B-BE8D-67F6A8AD6355

Device     Start        End    Sectors Size Type
/dev/sde1   2048 4292868095 4292866048   2T Linux filesystem


What would be the next steps to repair this disk?

Thank you all in advance for your help,

Chris


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: need help in a broken 2TB BTRFS partition
  2021-10-15 21:01                     ` need help in a broken 2TB BTRFS partition Christian Wimmer
@ 2021-10-16 10:08                       ` Qu Wenruo
  2021-10-16 17:29                         ` Christian Wimmer
  2021-10-16 17:35                         ` Christian Wimmer
  0 siblings, 2 replies; 53+ messages in thread
From: Qu Wenruo @ 2021-10-16 10:08 UTC (permalink / raw)
  To: Christian Wimmer; +Cc: Qu WenRuo, Anand Jain, linux-btrfs



On 2021/10/16 05:01, Christian Wimmer wrote:
> Hi Qu,
>
> I hope I find you well.
>
> Almost two years that my system runs without any failure.
> Since this is very boring I tried make my life somehow harder and tested again the snapshot feature of my Parallels Desktop installation yesterday:-)
> When I erased the old snapshots I could feel (and actually hear) already that the system is writing too much to the partitions.
> What I want to say is that it took too long (for any reason) to erase the old snapshots and to shut the system down.

The slow down seems to be caused by qgroup.

We already have an idea how to solve the problem and have some patches
for that.

Although it would add a new sysfs interface and may need user space
tools support.

>
> Well, after booting I saw that one of the discs is not coming back and I got the following error message:
>
> Suse_Tumbleweed:/home/proc # btrfs check /dev/sdd1
> Opening filesystem to check...
> parent transid verify failed on 324239360 wanted 208553 found 184371
> parent transid verify failed on 324239360 wanted 208553 found 184371
> parent transid verify failed on 324239360 wanted 208553 found 184371

This is the typical transid mismatch, caused by missing writes.

Normally if it's a physical machine, the first thing we suspect would be
the disk.

But since you're using an VM in MacOS, it has a whole storage stack to
go through.

And any of the stack is not handling flush/fua correctly, then it can
definitely go wrong like this.


> Ignoring transid failure
> leaf parent key incorrect 324239360
> ERROR: failed to read block groups: Operation not permitted
> ERROR: cannot open file system
>
>
> Could you help me to debug and repair this please?

Repair is not really possible.

>
> I already run the command btrfs restore /dev/sdd1 . and could restore 90% of the data but not the important last 10%.

Using newer kernel like v5.14, you can using "-o ro,rescue=all" mount
option, which would act mostly like btrfs-restore, and you may have a
chance to recover the lost 10%.

>
> My system is:
>
> Suse Tumbleweed inside Parallels Desktop on a Mac Mini
>
> Mac Min: Big Sur
> Parallels Desktop: 17.1.0
> Suse: Linux Suse_Tumbleweed 5.13.2-1-default #1 SMP Thu Jul 15 03:36:02 UTC 2021 (89416ca) x86_64 x86_64 x86_64 GNU/Linux
>
> Suse_Tumbleweed:~ # btrfs --version
> btrfs-progs v5.13
>
> The disk /dev/sdd1 is one of several 2TB partitions that reside on a NAS attached to the Mac Mini like

/dev/sdd1 is directly mapped into the VM or something else?

Or a file in remote filesystem (like NFS) then mapped into the VM?

Thanks,
Qu

>
> Disk /dev/sde: 2 TiB, 2197949513728 bytes, 4292870144 sectors
> Disk model: Linux_raid5_2tb_
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disklabel type: gpt
> Disk identifier: 942781EC-8969-408B-BE8D-67F6A8AD6355
>
> Device     Start        End    Sectors Size Type
> /dev/sde1   2048 4292868095 4292866048   2T Linux filesystem
>
>
> What would be the next steps to repair this disk?
>
> Thank you all in advance for your help,
>
> Chris
>

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: need help in a broken 2TB BTRFS partition
  2021-10-16 10:08                       ` Qu Wenruo
@ 2021-10-16 17:29                         ` Christian Wimmer
  2021-10-16 22:55                           ` Qu Wenruo
  2021-10-16 17:35                         ` Christian Wimmer
  1 sibling, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2021-10-16 17:29 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

>> 
>> I already run the command btrfs restore /dev/sdd1 . and could restore 90% of the data but not the important last 10%.
> 
> Using newer kernel like v5.14, you can using "-o ro,rescue=all" mount
> option, which would act mostly like btrfs-restore, and you may have a
> chance to recover the lost 10%.

Very nice! I updated to 5.14 and mounted with "-o ro,rescue=all” and yes, I can see all data now.
I guess this is just for data recovery, not a permanent mount option, right?
I should rescue the data and format the disc again, right?

> 
>> 
>> My system is:
>> 
>> Suse Tumbleweed inside Parallels Desktop on a Mac Mini
>> 
>> Mac Min: Big Sur
>> Parallels Desktop: 17.1.0
>> Suse: Linux Suse_Tumbleweed 5.13.2-1-default #1 SMP Thu Jul 15 03:36:02 UTC 2021 (89416ca) x86_64 x86_64 x86_64 GNU/Linux
>> 
>> Suse_Tumbleweed:~ # btrfs --version
>> btrfs-progs v5.13
>> 
>> The disk /dev/sdd1 is one of several 2TB partitions that reside on a NAS attached to the Mac Mini like
> 
> /dev/sdd1 is directly mapped into the VM or something else?
> 
> Or a file in remote filesystem (like NFS) then mapped into the VM?

It is a file on a SAS External Physical Volume that is formatted in Mac OS Extended and attached to the Parallels as an additional disc, then formatted inside linux with btrfs


BTW, I think now I know exactly what I did wrong to get to this stage.
I suspended my Virtual Machine with all discs still mounted. Then I started a new Virtual machine with the same discs attached and  this confused the discs.
Should avoid this.


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: need help in a broken 2TB BTRFS partition
  2021-10-16 10:08                       ` Qu Wenruo
  2021-10-16 17:29                         ` Christian Wimmer
@ 2021-10-16 17:35                         ` Christian Wimmer
  2021-10-16 23:27                           ` Qu Wenruo
  1 sibling, 1 reply; 53+ messages in thread
From: Christian Wimmer @ 2021-10-16 17:35 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: Qu WenRuo, Anand Jain, linux-btrfs

BTW, due to some unsuccessful boot attempts this disc /dev/sdd1 does not work any more with “-o ro,rescue=all”
so I decided to try some nasty commands like the following:


Suse_Tumbleweed:/home/proc # btrfs rescue chunk-recover /dev/sdd1
Scanning: 4914069504 in dev0cmds/rescue-chunk-recover.c:130: process_extent_buffer: BUG_ON `exist->nmirrors >= BTRFS_MAX_MIRRORS` triggered, value 1
btrfs(+0x1a121)[0x55830a51d121]
btrfs(+0x508dc)[0x55830a5538dc]
/lib64/libc.so.6(+0x8db37)[0x7fa8984cbb37]
/lib64/libc.so.6(+0x112640)[0x7fa898550640]
Aborted (core dumped)

Unfortunately the program crashes. Is this expected?

What else can I try if the mount command reports:

Suse_Tumbleweed:/home/proc # mount -o ro,rescue=all /dev/sdd1 /home/promise2/disk3
mount: /home/promise2/disk3: wrong fs type, bad option, bad superblock on /dev/sdd1, missing codepage or helper program, or other error.





> On 16. Oct 2021, at 07:08, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> 
> 
> On 2021/10/16 05:01, Christian Wimmer wrote:
>> Hi Qu,
>> 
>> I hope I find you well.
>> 
>> Almost two years that my system runs without any failure.
>> Since this is very boring I tried make my life somehow harder and tested again the snapshot feature of my Parallels Desktop installation yesterday:-)
>> When I erased the old snapshots I could feel (and actually hear) already that the system is writing too much to the partitions.
>> What I want to say is that it took too long (for any reason) to erase the old snapshots and to shut the system down.
> 
> The slow down seems to be caused by qgroup.
> 
> We already have an idea how to solve the problem and have some patches
> for that.
> 
> Although it would add a new sysfs interface and may need user space
> tools support.
> 
>> 
>> Well, after booting I saw that one of the discs is not coming back and I got the following error message:
>> 
>> Suse_Tumbleweed:/home/proc # btrfs check /dev/sdd1
>> Opening filesystem to check...
>> parent transid verify failed on 324239360 wanted 208553 found 184371
>> parent transid verify failed on 324239360 wanted 208553 found 184371
>> parent transid verify failed on 324239360 wanted 208553 found 184371
> 
> This is the typical transid mismatch, caused by missing writes.
> 
> Normally if it's a physical machine, the first thing we suspect would be
> the disk.
> 
> But since you're using an VM in MacOS, it has a whole storage stack to
> go through.
> 
> And any of the stack is not handling flush/fua correctly, then it can
> definitely go wrong like this.
> 
> 
>> Ignoring transid failure
>> leaf parent key incorrect 324239360
>> ERROR: failed to read block groups: Operation not permitted
>> ERROR: cannot open file system
>> 
>> 
>> Could you help me to debug and repair this please?
> 
> Repair is not really possible.
> 
>> 
>> I already run the command btrfs restore /dev/sdd1 . and could restore 90% of the data but not the important last 10%.
> 
> Using newer kernel like v5.14, you can using "-o ro,rescue=all" mount
> option, which would act mostly like btrfs-restore, and you may have a
> chance to recover the lost 10%.
> 
>> 
>> My system is:
>> 
>> Suse Tumbleweed inside Parallels Desktop on a Mac Mini
>> 
>> Mac Min: Big Sur
>> Parallels Desktop: 17.1.0
>> Suse: Linux Suse_Tumbleweed 5.13.2-1-default #1 SMP Thu Jul 15 03:36:02 UTC 2021 (89416ca) x86_64 x86_64 x86_64 GNU/Linux
>> 
>> Suse_Tumbleweed:~ # btrfs --version
>> btrfs-progs v5.13
>> 
>> The disk /dev/sdd1 is one of several 2TB partitions that reside on a NAS attached to the Mac Mini like
> 
> /dev/sdd1 is directly mapped into the VM or something else?
> 
> Or a file in remote filesystem (like NFS) then mapped into the VM?
> 
> Thanks,
> Qu
> 
>> 
>> Disk /dev/sde: 2 TiB, 2197949513728 bytes, 4292870144 sectors
>> Disk model: Linux_raid5_2tb_
>> Units: sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>> Disklabel type: gpt
>> Disk identifier: 942781EC-8969-408B-BE8D-67F6A8AD6355
>> 
>> Device     Start        End    Sectors Size Type
>> /dev/sde1   2048 4292868095 4292866048   2T Linux filesystem
>> 
>> 
>> What would be the next steps to repair this disk?
>> 
>> Thank you all in advance for your help,
>> 
>> Chris
>> 


^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: need help in a broken 2TB BTRFS partition
  2021-10-16 17:29                         ` Christian Wimmer
@ 2021-10-16 22:55                           ` Qu Wenruo
  0 siblings, 0 replies; 53+ messages in thread
From: Qu Wenruo @ 2021-10-16 22:55 UTC (permalink / raw)
  To: Christian Wimmer; +Cc: Qu WenRuo, Anand Jain, linux-btrfs



On 2021/10/17 01:29, Christian Wimmer wrote:
>>>
>>> I already run the command btrfs restore /dev/sdd1 . and could restore 90% of the data but not the important last 10%.
>>
>> Using newer kernel like v5.14, you can using "-o ro,rescue=all" mount
>> option, which would act mostly like btrfs-restore, and you may have a
>> chance to recover the lost 10%.
>
> Very nice! I updated to 5.14 and mounted with "-o ro,rescue=all” and yes, I can see all data now.
> I guess this is just for data recovery, not a permanent mount option, right?
> I should rescue the data and format the disc again, right?

Yes, that's just to rescue.

Thanks,
Qu
>
>>
>>>
>>> My system is:
>>>
>>> Suse Tumbleweed inside Parallels Desktop on a Mac Mini
>>>
>>> Mac Min: Big Sur
>>> Parallels Desktop: 17.1.0
>>> Suse: Linux Suse_Tumbleweed 5.13.2-1-default #1 SMP Thu Jul 15 03:36:02 UTC 2021 (89416ca) x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> Suse_Tumbleweed:~ # btrfs --version
>>> btrfs-progs v5.13
>>>
>>> The disk /dev/sdd1 is one of several 2TB partitions that reside on a NAS attached to the Mac Mini like
>>
>> /dev/sdd1 is directly mapped into the VM or something else?
>>
>> Or a file in remote filesystem (like NFS) then mapped into the VM?
>
> It is a file on a SAS External Physical Volume that is formatted in Mac OS Extended and attached to the Parallels as an additional disc, then formatted inside linux with btrfs
>
>
> BTW, I think now I know exactly what I did wrong to get to this stage.
> I suspended my Virtual Machine with all discs still mounted. Then I started a new Virtual machine with the same discs attached and  this confused the discs.
> Should avoid this.
>

^ permalink raw reply	[flat|nested] 53+ messages in thread

* Re: need help in a broken 2TB BTRFS partition
  2021-10-16 17:35                         ` Christian Wimmer
@ 2021-10-16 23:27                           ` Qu Wenruo
  0 siblings, 0 replies; 53+ messages in thread
From: Qu Wenruo @ 2021-10-16 23:27 UTC (permalink / raw)
  To: Christian Wimmer, Qu Wenruo; +Cc: Anand Jain, linux-btrfs



On 2021/10/17 01:35, Christian Wimmer wrote:
> BTW, due to some unsuccessful boot attempts this disc /dev/sdd1 does not work any more with “-o ro,rescue=all”
> so I decided to try some nasty commands like the following:

Well, any attempt to do any write into that fs would only further 
degrade the fs.

> 
> 
> Suse_Tumbleweed:/home/proc # btrfs rescue chunk-recover /dev/sdd1
> Scanning: 4914069504 in dev0cmds/rescue-chunk-recover.c:130: process_extent_buffer: BUG_ON `exist->nmirrors >= BTRFS_MAX_MIRRORS` triggered, value 1
> btrfs(+0x1a121)[0x55830a51d121]
> btrfs(+0x508dc)[0x55830a5538dc]
> /lib64/libc.so.6(+0x8db37)[0x7fa8984cbb37]
> /lib64/libc.so.6(+0x112640)[0x7fa898550640]
> Aborted (core dumped)
> 
> Unfortunately the program crashes. Is this expected?

No, but it doesn't matter anymore.

The problem would not be chunk tree related afaik.

> 
> What else can I try if the mount command reports:
> 
> Suse_Tumbleweed:/home/proc # mount -o ro,rescue=all /dev/sdd1 /home/promise2/disk3
> mount: /home/promise2/disk3: wrong fs type, bad option, bad superblock on /dev/sdd1, missing codepage or helper program, or other error.

Dmesg would provide the reason why it fails to mount.

Thanks,
Qu

> 
> 
> 
> 
> 
>> On 16. Oct 2021, at 07:08, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2021/10/16 05:01, Christian Wimmer wrote:
>>> Hi Qu,
>>>
>>> I hope I find you well.
>>>
>>> Almost two years that my system runs without any failure.
>>> Since this is very boring I tried make my life somehow harder and tested again the snapshot feature of my Parallels Desktop installation yesterday:-)
>>> When I erased the old snapshots I could feel (and actually hear) already that the system is writing too much to the partitions.
>>> What I want to say is that it took too long (for any reason) to erase the old snapshots and to shut the system down.
>>
>> The slow down seems to be caused by qgroup.
>>
>> We already have an idea how to solve the problem and have some patches
>> for that.
>>
>> Although it would add a new sysfs interface and may need user space
>> tools support.
>>
>>>
>>> Well, after booting I saw that one of the discs is not coming back and I got the following error message:
>>>
>>> Suse_Tumbleweed:/home/proc # btrfs check /dev/sdd1
>>> Opening filesystem to check...
>>> parent transid verify failed on 324239360 wanted 208553 found 184371
>>> parent transid verify failed on 324239360 wanted 208553 found 184371
>>> parent transid verify failed on 324239360 wanted 208553 found 184371
>>
>> This is the typical transid mismatch, caused by missing writes.
>>
>> Normally if it's a physical machine, the first thing we suspect would be
>> the disk.
>>
>> But since you're using an VM in MacOS, it has a whole storage stack to
>> go through.
>>
>> And any of the stack is not handling flush/fua correctly, then it can
>> definitely go wrong like this.
>>
>>
>>> Ignoring transid failure
>>> leaf parent key incorrect 324239360
>>> ERROR: failed to read block groups: Operation not permitted
>>> ERROR: cannot open file system
>>>
>>>
>>> Could you help me to debug and repair this please?
>>
>> Repair is not really possible.
>>
>>>
>>> I already run the command btrfs restore /dev/sdd1 . and could restore 90% of the data but not the important last 10%.
>>
>> Using newer kernel like v5.14, you can using "-o ro,rescue=all" mount
>> option, which would act mostly like btrfs-restore, and you may have a
>> chance to recover the lost 10%.
>>
>>>
>>> My system is:
>>>
>>> Suse Tumbleweed inside Parallels Desktop on a Mac Mini
>>>
>>> Mac Min: Big Sur
>>> Parallels Desktop: 17.1.0
>>> Suse: Linux Suse_Tumbleweed 5.13.2-1-default #1 SMP Thu Jul 15 03:36:02 UTC 2021 (89416ca) x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> Suse_Tumbleweed:~ # btrfs --version
>>> btrfs-progs v5.13
>>>
>>> The disk /dev/sdd1 is one of several 2TB partitions that reside on a NAS attached to the Mac Mini like
>>
>> /dev/sdd1 is directly mapped into the VM or something else?
>>
>> Or a file in remote filesystem (like NFS) then mapped into the VM?
>>
>> Thanks,
>> Qu
>>
>>>
>>> Disk /dev/sde: 2 TiB, 2197949513728 bytes, 4292870144 sectors
>>> Disk model: Linux_raid5_2tb_
>>> Units: sectors of 1 * 512 = 512 bytes
>>> Sector size (logical/physical): 512 bytes / 4096 bytes
>>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>>> Disklabel type: gpt
>>> Disk identifier: 942781EC-8969-408B-BE8D-67F6A8AD6355
>>>
>>> Device     Start        End    Sectors Size Type
>>> /dev/sde1   2048 4292868095 4292866048   2T Linux filesystem
>>>
>>>
>>> What would be the next steps to repair this disk?
>>>
>>> Thank you all in advance for your help,
>>>
>>> Chris
>>>
> 


^ permalink raw reply	[flat|nested] 53+ messages in thread

end of thread, other threads:[~2021-10-16 23:27 UTC | newest]

Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-06  3:44 [PATCH] btrfs-progs: Skip device tree when we failed to read it Qu Wenruo
2019-12-06  6:12 ` Anand Jain
2019-12-06 15:50   ` Christian Wimmer
2019-12-06 16:34   ` Christian Wimmer
     [not found]   ` <762365A0-8BDF-454B-ABA9-AB2F0C958106@icloud.com>
2019-12-07  1:16     ` Qu WenRuo
2019-12-07  3:47       ` Christian Wimmer
2019-12-07  4:31         ` Qu Wenruo
2019-12-07 13:03           ` Christian Wimmer
2019-12-07 14:10             ` Qu Wenruo
2019-12-07 14:25               ` Christian Wimmer
2019-12-07 16:44               ` Christian Wimmer
2019-12-08  1:21                 ` Qu WenRuo
2019-12-10 21:25                   ` Christian Wimmer
2019-12-11  0:36                     ` Qu Wenruo
2019-12-11 15:57                       ` Christian Wimmer
     [not found]           ` <9FB359ED-EAD4-41DD-B846-1422F2DC4242@icloud.com>
2020-01-04 17:07             ` 12 TB btrfs file system on virtual machine broke again Christian Wimmer
2020-01-05  4:03               ` Chris Murphy
2020-01-05 13:40                 ` Christian Wimmer
2020-01-05 14:07                   ` Martin Raiber
2020-01-05 14:14                     ` Christian Wimmer
2020-01-05 14:23                       ` Christian Wimmer
2020-01-05  4:25               ` Qu Wenruo
2020-01-05 14:17                 ` Christian Wimmer
2020-01-05 18:50                   ` Chris Murphy
2020-01-05 19:18                     ` Christian Wimmer
2020-01-05 19:36                       ` Chris Murphy
2020-01-05 19:49                         ` Christian Wimmer
2020-01-05 19:52                         ` Christian Wimmer
2020-01-05 20:34                           ` Chris Murphy
2020-01-05 20:36                             ` Chris Murphy
     [not found]                         ` <3F43DDB8-0372-4CDE-B143-D2727D3447BC@icloud.com>
2020-01-05 20:30                           ` Chris Murphy
2020-01-05 20:36                             ` Christian Wimmer
2020-01-05 21:13                               ` Chris Murphy
2020-01-05 21:58                                 ` Christian Wimmer
2020-01-05 22:28                                   ` Chris Murphy
2020-01-06  1:31                                     ` Christian Wimmer
2020-01-06  1:33                                     ` Christian Wimmer
2020-01-11 17:04                                     ` 12 TB btrfs file system on virtual machine broke again (third time) Christian Wimmer
2020-01-11 17:23                                     ` Christian Wimmer
2020-01-11 19:46                                       ` Chris Murphy
2020-01-13 19:41                                         ` 12 TB btrfs file system on virtual machine broke again (fourth time) Christian Wimmer
2020-01-13 20:03                                           ` Chris Murphy
2020-01-31 16:35                                             ` btrfs not booting any more Christian Wimmer
2020-05-08 12:20                                             ` btrfs reports bad key ordering after out of memory situation Christian Wimmer
2020-01-05 23:50                   ` 12 TB btrfs file system on virtual machine broke again Qu Wenruo
2020-01-06  1:32                     ` Christian Wimmer
2020-01-11  7:25                     ` Andrei Borzenkov
2021-10-15 21:01                     ` need help in a broken 2TB BTRFS partition Christian Wimmer
2021-10-16 10:08                       ` Qu Wenruo
2021-10-16 17:29                         ` Christian Wimmer
2021-10-16 22:55                           ` Qu Wenruo
2021-10-16 17:35                         ` Christian Wimmer
2021-10-16 23:27                           ` Qu Wenruo

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.