[REQ] Help to gain access to btrfsraid6 pool that won't mount

[REQ] Help to gain access to btrfsraid6 pool that won't mount

[TheJay]

$ uname -a
Linux thejay-srv03 4.0.0-040000-generic #201504121935 SMP Sun Apr 12 
23:36:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ btrfs --version
btrfs-progs v3.19.1

$ sudo btrfs fi show
Label: 'roms03'  uuid: b0025417-4045-4c63-a432-0625b7cc41ea
	Total devices 4 FS bytes used 2.67TiB
	devid    1 size 1.36TiB used 1.36TiB path /dev/sdz
	devid    3 size 1.36TiB used 1.36TiB path /dev/sdaa
	devid    4 size 1.36TiB used 1.36TiB path /dev/sdab
	*** Some devices missing

This pool is a btrfsraid6 pool

when trying to mount I get:

$ sudo mount -t btrfs -o recovery,ro /dev/sdaa /mnt/roms03
mount: wrong fs type, bad option, bad superblock on /dev/sdaa,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

$ dmesg -T|tail
[Mon May  4 08:35:08 2015] BTRFS info (device sdz): enabling auto recovery
[Mon May  4 08:35:08 2015] BTRFS info (device sdz): disk space caching is 
enabled
[Mon May  4 08:35:08 2015] BTRFS: has skinny extents
[Mon May  4 08:35:08 2015] BTRFS: failed to read chunk tree on sdz
[Mon May  4 08:35:08 2015] BTRFS: open_ctree failed

What could I try to see if I can mount this pool and gain access to the 
data?

While I have a backup of the data, next time I may not, and I am trying 
to learn how to go about. 

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[Goffredo Baroncelli]

On 2015-05-04 17:44, TheJay wrote:
> $ uname -a
> Linux thejay-srv03 4.0.0-040000-generic #201504121935 SMP Sun Apr 12 
> 23:36:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> 
> $ btrfs --version
> btrfs-progs v3.19.1
> 
> $ sudo btrfs fi show
> Label: 'roms03'  uuid: b0025417-4045-4c63-a432-0625b7cc41ea
> 	Total devices 4 FS bytes used 2.67TiB
> 	devid    1 size 1.36TiB used 1.36TiB path /dev/sdz
> 	devid    3 size 1.36TiB used 1.36TiB path /dev/sdaa
> 	devid    4 size 1.36TiB used 1.36TiB path /dev/sdab
> 	*** Some devices missing
> 
> This pool is a btrfsraid6 pool
> 
> when trying to mount I get:
> 
> $ sudo mount -t btrfs -o recovery,ro /dev/sdaa /mnt/roms03
> mount: wrong fs type, bad option, bad superblock on /dev/sdaa,
>        missing codepage or helper program, or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail  or so
> 
> $ dmesg -T|tail
> [Mon May  4 08:35:08 2015] BTRFS info (device sdz): enabling auto recovery
> [Mon May  4 08:35:08 2015] BTRFS info (device sdz): disk space caching is 
> enabled
> [Mon May  4 08:35:08 2015] BTRFS: has skinny extents
> [Mon May  4 08:35:08 2015] BTRFS: failed to read chunk tree on sdz
> [Mon May  4 08:35:08 2015] BTRFS: open_ctree failed
> 
> What could I try to see if I can mount this pool and gain access to the 
> data?

A disk is missing, in this case you should use the "degraded" mount option:

mount -o degraded /dev/sdaa /mnt/roms03



> 
> While I have a backup of the data, next time I may not, and I am trying 
> to learn how to go about. 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[TheJay]

On Mon, 04 May 2015 23:25:19 +0200, Goffredo Baroncelli wrote:


> A disk is missing, in this case you should use the "degraded" mount option:
> 
> mount -o degraded /dev/sdaa /mnt/roms03


Goffredo - Thanks for responding - I tried that, but to no avail:

$ sudo mount -o degraded /dev/sdaa /mnt/roms03
mount: wrong fs type, bad option, bad superblock on /dev/sdaa,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so


$ dmesg -T|tail
[Mon May  4 14:33:17 2015] BTRFS info (device sdaa): allowing degraded mounts
[Mon May  4 14:33:17 2015] BTRFS info (device sdaa): disk space caching is enabled
[Mon May  4 14:33:17 2015] BTRFS: has skinny extents
[Mon May  4 14:33:17 2015] BTRFS: bdev (null) errs: wr 3, rd 28773, flush 2, corrupt 0, gen 0
[Mon May  4 14:33:22 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Mon May  4 14:33:22 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Mon May  4 14:33:22 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Mon May  4 14:33:22 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Mon May  4 14:33:22 2015] BTRFS: Failed to read block groups: -5
[Mon May  4 14:33:22 2015] BTRFS: open_ctree failed

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[Chris Murphy]

On Mon, May 4, 2015 at 9:44 AM, TheJay <obiwantje@gmail.com> wrote:
> $ sudo btrfs fi show
> Label: 'roms03'  uuid: b0025417-4045-4c63-a432-0625b7cc41ea
>         Total devices 4 FS bytes used 2.67TiB
>         devid    1 size 1.36TiB used 1.36TiB path /dev/sdz
>         devid    3 size 1.36TiB used 1.36TiB path /dev/sdaa
>         devid    4 size 1.36TiB used 1.36TiB path /dev/sdab
>         *** Some devices missing

How many devices are missing?

> [Mon May  4 08:35:08 2015] BTRFS: failed to read chunk tree on sdz

That's not good. What do you get for 'btrfs check" (without --repair)?

-- 
Chris Murphy

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[Chris Murphy]

On Mon, May 4, 2015 at 3:36 PM, TheJay <obiwantje@gmail.com> wrote:

> [Mon May  4 14:33:17 2015] BTRFS: bdev (null) errs: wr 3, rd 28773, flush 2, corrupt 0, gen 0

So there might be more than one thing going wrong and they're
colliding to make a worse problem. You've got three write and 2 flush
errors, and many thousands of read errors (probably due to missing
devices).

It might be worth trying -o degraded,recovery because recovery can't
work with devices missing, and degraded alone won't run the recovery
code.


-- 
Chris Murphy

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[TheJay]

On Tue, 05 May 2015 00:52:16 -0600, Chris Murphy wrote:

> On Mon, May 4, 2015 at 3:36 PM, TheJay <obiwantje@gmail.com> wrote:
> 
>> [Mon May  4 14:33:17 2015] BTRFS: bdev (null) errs: wr 3, rd 28773, flush 2, corrupt 0, gen 0
> 
> So there might be more than one thing going wrong and they're
> colliding to make a worse problem. You've got three write and 2 flush
> errors, and many thousands of read errors (probably due to missing
> devices).
> 
> It might be worth trying -o degraded,recovery because recovery can't
> work with devices missing, and degraded alone won't run the recovery
> code.
> 
> 
> -- 
> Chris Murphy

$ mount  -o degraded,recovery /dev/sdz /mnt/roms03
mount: wrong fs type, bad option, bad superblock on /dev/sdy,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

$ dmesg -T|tail
[Tue May  5 00:26:01 2015] BTRFS info (device sdaa): enabling auto recovery
[Tue May  5 00:26:01 2015] BTRFS info (device sdaa): disk space caching is enabled
[Tue May  5 00:26:01 2015] BTRFS: has skinny extents
[Tue May  5 00:26:01 2015] BTRFS: bdev (null) errs: wr 3, rd 28773, flush 2, corrupt 0, gen 0
[Tue May  5 00:26:05 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Tue May  5 00:26:05 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Tue May  5 00:26:05 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Tue May  5 00:26:05 2015] BTRFS (device sdaa): bad tree block start 8714499618262386624 1585233920
[Tue May  5 00:26:05 2015] BTRFS: Failed to read block groups: -5
[Tue May  5 00:26:06 2015] BTRFS: open_ctree failed

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[TheJay]

On Tue, 05 May 2015 00:48:35 -0600, Chris Murphy wrote:

> On Mon, May 4, 2015 at 9:44 AM, TheJay <obiwantje@gmail.com> wrote:
>> $ sudo btrfs fi show
>> Label: 'roms03'  uuid: b0025417-4045-4c63-a432-0625b7cc41ea
>>         Total devices 4 FS bytes used 2.67TiB
>>         devid    1 size 1.36TiB used 1.36TiB path /dev/sdz
>>         devid    3 size 1.36TiB used 1.36TiB path /dev/sdaa
>>         devid    4 size 1.36TiB used 1.36TiB path /dev/sdab
>>         *** Some devices missing
> 
> How many devices are missing?
> 
>> [Mon May  4 08:35:08 2015] BTRFS: failed to read chunk tree on sdz
> 
> That's not good. What do you get for 'btrfs check" (without --repair)?
> 
> -- 
> Chris Murphy

Appriciate the response Chris

> How many devices are missing?
One (1) of the 4 devices is missing - "device 2".

> What do you get for 'btrfs check" (without --repair)?
$ btrfs check /dev/sdaa
warning, device 2 is missing
warning devid 2 not found already
checksum verify failed on 21200896 found 44715E95 wanted A092E323
checksum verify failed on 21200896 found 91AC994F wanted 987F5E0F
checksum verify failed on 21200896 found 44715E95 wanted A092E323
bytenr mismatch, want=21200896, have=65536
Couldn't read chunk tree
Couldn't open file system

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[Duncan]

TheJay posted on Mon, 04 May 2015 15:44:35 +0000 as excerpted:

> What could I try to see if I can mount this pool and gain access to the
> data?
> 
> While I have a backup of the data, next time I may not, and I am trying
> to learn how to go about.

If you can't get it to mount based on the other subthreads, and you have 
sufficient space on an existing/working filesystem to recover the files 
to, you can try btrfs restore.

I wrote up a longer reply covering restore in another recent thread, 
"Help on broken filesystem", OP by Ermanno Baschiera, which should be in 
the archives... yes...

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/44765

That in turn refers to the (outdated) restore writeup on the wiki.

Since then, I had to use it again myself, updating my year-old personal 
experience.  Among other things, btrfs-progs v4.0 has a new metadata 
restore option, which should restore ownership/perms/dates as well as the 
file data.  There's a patch to restore symlinks too, but it apparently 
didn't make it into 4.0, so you'll have to try the integration branch or 
apply it manually after finding it on the list.  The looping too many 
times point apparently doesn't apply any more, at least on a full run, 
tho it still cuts short if you use the dry-run option.  Additionally, the 
dry-run option gave me some metadata restoration errors that a full write-
files run didn't give me, so dry-run ends up being rather more 
pessimistic about what can be restored than what it actually restores.

If you're lucky, restore will "just work".  If not, you can try feeding 
it roots you found using btrfs-find-root, as detailed in the above post.  
On my recent experience I was lucky and it worked without that, so I 
didn't get to update my personal experience knowledge in that regard.

You did say you have a backup (good admin! =:^) this time, but wanted to 
practice in case you weren't so lucky next time.  Practice is good, and 
practicing restore when you don't /have/ to depend on it as you have a 
current backup is even better, but do keep in mind the admin's backup 
rule I mention in the other post.

As it happens, I had a backup I could have used in my recent restore 
experience case as well, but it wasn't as current as I'd have liked.  I 
had knowingly taken that risk and would have settled for the old backup 
if I had to, but restore gave me the very pleasant (under the 
circumstances) choice of either the more current copies from btrfs 
restore but having to redo the symlinks manually, or using the older 
backup.  It's actually very nice having that choice as it takes all the 
pressure away of worrying about fat-fingering the one chance you'd 
otherwise have left! =:^)

Plus, while restore "just worked" for me this time, if it hadn't and I 
had to do the find-root thing, I would have had the luxury of simply 
saying OK, too much hassle, I'll just use the older backups and go from 
there.  I'd have probably still tried the find-root thing, because they 
/were/ more current and as you note having the experience is good, but 
again, with the backups if I had to use 'em, the pressure would have been 
off, and that in itself is valuable. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[TheJay]

On Wed, 06 May 2015 11:55:51 +0000, Duncan wrote:

> Since then, I had to use it again myself, updating my year-old personal 
> experience.  Among other things, btrfs-progs v4.0 has a new metadata 
> restore option, which should restore ownership/perms/dates as well as the 
> file data.  There's a patch to restore symlinks too, but it apparently 
> didn't make it into 4.0, so you'll have to try the integration branch or 
> apply it manually after finding it on the list.  The looping too many 
> times point apparently doesn't apply any more, at least on a full run, 
> tho it still cuts short if you use the dry-run option.  Additionally, the 
> dry-run option gave me some metadata restoration errors that a full write-
> files run didn't give me, so dry-run ends up being rather more 
> pessimistic about what can be restored than what it actually restores.
> 
> If you're lucky, restore will "just work".  If not, you can try feeding 
> it roots you found using btrfs-find-root, as detailed in the above post.  
> On my recent experience I was lucky and it worked without that, so I 
> didn't get to update my personal experience knowledge in that regard.

Duncan,

Many thanks for taking the time to look at my case, and to make the effort of writing such an elaborate response - much appreciated. 
I understand the direction I should be taking, and by just reading your response have learned quite a bit already.

Unfortunately I got stranded right at the get-go, with the btrfs-show super command not working for me. Let me share a couple of outputs, and hopefully you or others can point me where to go from here.
As a side-note, I did update to btrfs-progs 4.0 from 3.19.2 as suggested.

$ btrfs --version
btrfs-progs v4.0

$ sudo uname -a
Linux thejay-srv03 4.0.0-040000-generic #201504121935 SMP Sun Apr 12 23:36:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Just for clarity - this is a btrfsraid6 pool.

$ sudo btrfs fi show /dev/sdy
warning, device 2 is missing
warning devid 2 not found already
checksum verify failed on 21200896 found 44715E95 wanted A092E323
checksum verify failed on 21200896 found 91AC994F wanted 987F5E0F
checksum verify failed on 21200896 found 44715E95 wanted A092E323
bytenr mismatch, want=21200896, have=65536
Couldn't read chunk tree
Label: 'roms03'  uuid: b0025417-4045-4c63-a432-0625b7cc41ea
	Total devices 4 FS bytes used 2.67TiB
	devid    1 size 1.36TiB used 1.36TiB path /dev/sdy
	devid    3 size 1.36TiB used 1.36TiB path /dev/sdz
	devid    4 size 1.36TiB used 1.36TiB path /dev/sdaa
	*** Some devices missing

btrfs-progs v4.0

$ sudo btrfs-find-root /dev/sdy
warning, device 2 is missing
warning devid 2 not found already
Couldn't read chunk tree
Open ctree failed


(above output is the same for the other 2 drives sdz and sdaa)

$ sudo btrfs-show-super /dev/sdy
superblock: bytenr=65536, device=/dev/sdy
---------------------------------------------------------
csum			0x8c55bcfe [match]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
fsid			b0025417-4045-4c63-a432-0625b7cc41ea
label			roms03
generation		17310
root			1622245376
sys_array_size		290
chunk_root_generation	16372
root_level		1
chunk_root		20971520
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		6001207640064
bytes_used		2937720725504
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		4
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x1e1
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  RAID56 |
			  SKINNY_METADATA )
csum_type		0
csum_size		4
cache_generation	17310
uuid_tree_generation	17310
dev_item.uuid		f39d7af1-1eec-4f99-8353-4c70371e1942
dev_item.fsid		b0025417-4045-4c63-a432-0625b7cc41ea [match]
dev_item.type		0
dev_item.total_bytes	1500301910016
dev_item.bytes_used	1500301885440
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0

----

$ sudo btrfs-show-super /dev/sdz
superblock: bytenr=65536, device=/dev/sdz
---------------------------------------------------------
csum			0xb48f16eb [match]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
fsid			b0025417-4045-4c63-a432-0625b7cc41ea
label			roms03
generation		17310
root			1622245376
sys_array_size		290
chunk_root_generation	16372
root_level		1
chunk_root		20971520
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		6001207640064
bytes_used		2937720725504
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		4
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x1e1
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  RAID56 |
			  SKINNY_METADATA )
csum_type		0
csum_size		4
cache_generation	17310
uuid_tree_generation	17310
dev_item.uuid		6560226f-a966-4699-8a9e-e7377d436cc6
dev_item.fsid		b0025417-4045-4c63-a432-0625b7cc41ea [match]
dev_item.type		0
dev_item.total_bytes	1500301910016
dev_item.bytes_used	1500300836864
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		3
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0

----

$ sudo btrfs-show-super /dev/sdaa
superblock: bytenr=65536, device=/dev/sdaa
---------------------------------------------------------
csum			0xf3cbbf1f [match]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
fsid			b0025417-4045-4c63-a432-0625b7cc41ea
label			roms03
generation		17310
root			1622245376
sys_array_size		290
chunk_root_generation	16372
root_level		1
chunk_root		20971520
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		6001207640064
bytes_used		2937720725504
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		4
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x1e1
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  RAID56 |
			  SKINNY_METADATA )
csum_type		0
csum_size		4
cache_generation	17310
uuid_tree_generation	17310
dev_item.uuid		3c026bb0-dcf0-45da-8c90-cb668323180d
dev_item.fsid		b0025417-4045-4c63-a432-0625b7cc41ea [match]
dev_item.type		0
dev_item.total_bytes	1500301910016
dev_item.bytes_used	1500300836864
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		4
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0

Re: [REQ] Help to gain access to btrfsraid6 pool that won't mount

[TheJay]

On Wed, 06 May 2015 11:55:51 +0000, Duncan wrote:

> 
> Since then, I had to use it again myself, updating my year-old personal 
> experience.  Among other things, btrfs-progs v4.0 has a new metadata 
> restore option, which should restore ownership/perms/dates as well as the 
> file data.  There's a patch to restore symlinks too, but it apparently 
> didn't make it into 4.0, so you'll have to try the integration branch or 
> apply it manually after finding it on the list.  The looping too many 
> times point apparently doesn't apply any more, at least on a full run, 
> tho it still cuts short if you use the dry-run option.  Additionally, the 
> dry-run option gave me some metadata restoration errors that a full write-
> files run didn't give me, so dry-run ends up being rather more 
> pessimistic about what can be restored than what it actually restores.

Please ignore the last post - had 2 typos in it that could confuse my problem as to where I now got stuck (btrfs-show-super instead of btrfs-find-root, btrfs-progs version)


Duncan,

Many thanks for taking the time to look at my case, and to make the effort of writing such an elaborate response - much appreciated. 
I understand the direction I should be taking, and by just reading your response have learned quite a bit already.

Unfortunately I got stranded right at the get-go, with the btrfs-find-root command not working for me. Let me share a couple of outputs, and hopefully you or others can point me where to go from here.
As a side-note, I did update to btrfs-progs 4.0 from 3.19.1 as suggested.

$ btrfs --version
btrfs-progs v4.0

$ sudo uname -a
Linux thejay-srv03 4.0.0-040000-generic #201504121935 SMP Sun Apr 12 23:36:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Just for clarity - this is a btrfsraid6 pool.

$ sudo btrfs fi show /dev/sdy
warning, device 2 is missing
warning devid 2 not found already
checksum verify failed on 21200896 found 44715E95 wanted A092E323
checksum verify failed on 21200896 found 91AC994F wanted 987F5E0F
checksum verify failed on 21200896 found 44715E95 wanted A092E323
bytenr mismatch, want=21200896, have=65536
Couldn't read chunk tree
Label: 'roms03'  uuid: b0025417-4045-4c63-a432-0625b7cc41ea
	Total devices 4 FS bytes used 2.67TiB
	devid    1 size 1.36TiB used 1.36TiB path /dev/sdy
	devid    3 size 1.36TiB used 1.36TiB path /dev/sdz
	devid    4 size 1.36TiB used 1.36TiB path /dev/sdaa
	*** Some devices missing

btrfs-progs v4.0

$ sudo btrfs-find-root /dev/sdy
warning, device 2 is missing
warning devid 2 not found already
Couldn't read chunk tree
Open ctree failed


(above output is the same for the other 2 drives sdz and sdaa)

$ sudo btrfs-show-super /dev/sdy
superblock: bytenr=65536, device=/dev/sdy
---------------------------------------------------------
csum			0x8c55bcfe [match]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
fsid			b0025417-4045-4c63-a432-0625b7cc41ea
label			roms03
generation		17310
root			1622245376
sys_array_size		290
chunk_root_generation	16372
root_level		1
chunk_root		20971520
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		6001207640064
bytes_used		2937720725504
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		4
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x1e1
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  RAID56 |
			  SKINNY_METADATA )
csum_type		0
csum_size		4
cache_generation	17310
uuid_tree_generation	17310
dev_item.uuid		f39d7af1-1eec-4f99-8353-4c70371e1942
dev_item.fsid		b0025417-4045-4c63-a432-0625b7cc41ea [match]
dev_item.type		0
dev_item.total_bytes	1500301910016
dev_item.bytes_used	1500301885440
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0

----

$ sudo btrfs-show-super /dev/sdz
superblock: bytenr=65536, device=/dev/sdz
---------------------------------------------------------
csum			0xb48f16eb [match]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
fsid			b0025417-4045-4c63-a432-0625b7cc41ea
label			roms03
generation		17310
root			1622245376
sys_array_size		290
chunk_root_generation	16372
root_level		1
chunk_root		20971520
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		6001207640064
bytes_used		2937720725504
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		4
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x1e1
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  RAID56 |
			  SKINNY_METADATA )
csum_type		0
csum_size		4
cache_generation	17310
uuid_tree_generation	17310
dev_item.uuid		6560226f-a966-4699-8a9e-e7377d436cc6
dev_item.fsid		b0025417-4045-4c63-a432-0625b7cc41ea [match]
dev_item.type		0
dev_item.total_bytes	1500301910016
dev_item.bytes_used	1500300836864
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		3
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0

----

$ sudo btrfs-show-super /dev/sdaa
superblock: bytenr=65536, device=/dev/sdaa
---------------------------------------------------------
csum			0xf3cbbf1f [match]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
fsid			b0025417-4045-4c63-a432-0625b7cc41ea
label			roms03
generation		17310
root			1622245376
sys_array_size		290
chunk_root_generation	16372
root_level		1
chunk_root		20971520
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		6001207640064
bytes_used		2937720725504
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		4
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x1e1
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  RAID56 |
			  SKINNY_METADATA )
csum_type		0
csum_size		4
cache_generation	17310
uuid_tree_generation	17310
dev_item.uuid		3c026bb0-dcf0-45da-8c90-cb668323180d
dev_item.fsid		b0025417-4045-4c63-a432-0625b7cc41ea [match]
dev_item.type		0
dev_item.total_bytes	1500301910016
dev_item.bytes_used	1500300836864
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		4
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0