All of lore.kernel.org
 help / color / mirror / Atom feed
* help request for an unmountable raid1 filesystem
@ 2019-03-09 21:35 Glenn Trigg
  2019-03-23  0:45 ` Glenn Trigg
  2019-03-29  2:27 ` Chris Murphy
  0 siblings, 2 replies; 12+ messages in thread
From: Glenn Trigg @ 2019-03-09 21:35 UTC (permalink / raw)
  To: linux-btrfs

Hello,

I had some random machine freezing events which I suspected was due to
issues with a raid1 filesystem and kernel module crashes. I attempted
to use the information available to get the filesystem into a good
state where "btrfs check" and "btrfs scrub" would not have any errors,
however I fear things have become worse.

The current state of things is that the filesystem won't mount at all now.

% mount -r /dev/sda1 /data
mount: /data: can't read superblock on /dev/sda1.

and dmesg says:

[15944.017629] BTRFS info (device sda1): disk space caching is enabled
[15944.017632] BTRFS info (device sda1): has skinny extents
[15944.024480] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd
0, flush 0, corrupt 1, gen 0
[15944.024487] BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd
0, flush 0, corrupt 4, gen 0
[15944.029292] BTRFS error (device sda1): parent transid verify failed
on 628168376320 wanted 37601 found 37700
[15944.029466] BTRFS error (device sda1): parent transid verify failed
on 628168376320 wanted 37601 found 37700

Other system information is:
% uname -a
Linux izen 4.18.0-16-generic #17-Ubuntu SMP Fri Feb 8 00:06:57 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux

% btrfs --version
btrfs-progs v4.16.1

% btrfs fi show
Label: 'root'  uuid: 65fd7f11-4f60-435f-928b-6d15f12bc417
Total devices 1 FS bytes used 101.75GiB
devid    1 size 232.88GiB used 232.85GiB path /dev/nvme0n1p1

Label: 'data'  uuid: d5e50511-3e31-4de6-ba37-c5841895be9f
Total devices 2 FS bytes used 830.44GiB
devid    1 size 1.82TiB used 669.03GiB path /dev/sda1
devid    2 size 1.82TiB used 817.06GiB path /dev/sdb1

% btrfs check /dev/sda1
Checking filesystem on /dev/sda1
UUID: d5e50511-3e31-4de6-ba37-c5841895be9f
checking extents
parent transid verify failed on 628168343552 wanted 28163 found 37700
parent transid verify failed on 628168343552 wanted 28163 found 37700
parent transid verify failed on 628168343552 wanted 28163 found 37700
parent transid verify failed on 628168343552 wanted 28163 found 37700
Ignoring transid failure
bad block 628168343552
ERROR: errors found in extent allocation tree or chunk allocation
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
root 5 root dir 256 not found
ERROR: errors found in fs roots
found 528138240 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 1785856
total fs tree bytes: 1064960
total extent tree bytes: 81920
btree space waste bytes: 606983
file data blocks allocated: 215220224
 referenced 215220224

% btrfs rescue super-recover /dev/sda1
All supers are valid, no need to recover

Where do I go from here?

Regards,
Glenn

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-03-09 21:35 help request for an unmountable raid1 filesystem Glenn Trigg
@ 2019-03-23  0:45 ` Glenn Trigg
  2019-03-29  0:53   ` Glenn Trigg
  2019-03-29  2:27 ` Chris Murphy
  1 sibling, 1 reply; 12+ messages in thread
From: Glenn Trigg @ 2019-03-23  0:45 UTC (permalink / raw)
  To: linux-btrfs

Hi,

Since mailing this I have tried using the more recent utils - version
btrfs-progs v4.20.2.

I still have not had any success in getting the filesystem to a
mountable state and I have now also tried recovering files with btrfs
restore, also with no success. The restore output is:

% ./btrfs restore -D /dev/sdc1 /data2/
This is a dry-run, no files are going to be restored
parent transid verify failed on 628168376320 wanted 37601 found 37712
parent transid verify failed on 628168376320 wanted 37601 found 37712
parent transid verify failed on 628168376320 wanted 37601 found 37712
parent transid verify failed on 628168376320 wanted 37601 found 37712
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=628624064512 item=0 parent
level=1 child level=1
Error searching -5

Is there something else I could do to recover either the filesystem or
at least the files?

Regards,
Glenn

On Sun, 10 Mar 2019 at 08:35, Glenn Trigg <ggtrigg@gmail.com> wrote:
>
> Hello,
>
> I had some random machine freezing events which I suspected was due to
> issues with a raid1 filesystem and kernel module crashes. I attempted
> to use the information available to get the filesystem into a good
> state where "btrfs check" and "btrfs scrub" would not have any errors,
> however I fear things have become worse.
>
> The current state of things is that the filesystem won't mount at all now.
>
> % mount -r /dev/sda1 /data
> mount: /data: can't read superblock on /dev/sda1.
>
> and dmesg says:
>
> [15944.017629] BTRFS info (device sda1): disk space caching is enabled
> [15944.017632] BTRFS info (device sda1): has skinny extents
> [15944.024480] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd
> 0, flush 0, corrupt 1, gen 0
> [15944.024487] BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd
> 0, flush 0, corrupt 4, gen 0
> [15944.029292] BTRFS error (device sda1): parent transid verify failed
> on 628168376320 wanted 37601 found 37700
> [15944.029466] BTRFS error (device sda1): parent transid verify failed
> on 628168376320 wanted 37601 found 37700
>
> Other system information is:
> % uname -a
> Linux izen 4.18.0-16-generic #17-Ubuntu SMP Fri Feb 8 00:06:57 UTC
> 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> % btrfs --version
> btrfs-progs v4.16.1
>
> % btrfs fi show
> Label: 'root'  uuid: 65fd7f11-4f60-435f-928b-6d15f12bc417
> Total devices 1 FS bytes used 101.75GiB
> devid    1 size 232.88GiB used 232.85GiB path /dev/nvme0n1p1
>
> Label: 'data'  uuid: d5e50511-3e31-4de6-ba37-c5841895be9f
> Total devices 2 FS bytes used 830.44GiB
> devid    1 size 1.82TiB used 669.03GiB path /dev/sda1
> devid    2 size 1.82TiB used 817.06GiB path /dev/sdb1
>
> % btrfs check /dev/sda1
> Checking filesystem on /dev/sda1
> UUID: d5e50511-3e31-4de6-ba37-c5841895be9f
> checking extents
> parent transid verify failed on 628168343552 wanted 28163 found 37700
> parent transid verify failed on 628168343552 wanted 28163 found 37700
> parent transid verify failed on 628168343552 wanted 28163 found 37700
> parent transid verify failed on 628168343552 wanted 28163 found 37700
> Ignoring transid failure
> bad block 628168343552
> ERROR: errors found in extent allocation tree or chunk allocation
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> root 5 root dir 256 not found
> ERROR: errors found in fs roots
> found 528138240 bytes used, error(s) found
> total csum bytes: 0
> total tree bytes: 1785856
> total fs tree bytes: 1064960
> total extent tree bytes: 81920
> btree space waste bytes: 606983
> file data blocks allocated: 215220224
>  referenced 215220224
>
> % btrfs rescue super-recover /dev/sda1
> All supers are valid, no need to recover
>
> Where do I go from here?
>
> Regards,
> Glenn

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-03-23  0:45 ` Glenn Trigg
@ 2019-03-29  0:53   ` Glenn Trigg
  0 siblings, 0 replies; 12+ messages in thread
From: Glenn Trigg @ 2019-03-29  0:53 UTC (permalink / raw)
  To: linux-btrfs

I wonder why this is not getting any replies?

On Sat, 23 Mar 2019 at 11:45, Glenn Trigg <ggtrigg@gmail.com> wrote:
>
> Hi,
>
> Since mailing this I have tried using the more recent utils - version
> btrfs-progs v4.20.2.
>
> I still have not had any success in getting the filesystem to a
> mountable state and I have now also tried recovering files with btrfs
> restore, also with no success. The restore output is:
>
> % ./btrfs restore -D /dev/sdc1 /data2/
> This is a dry-run, no files are going to be restored
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=628624064512 item=0 parent
> level=1 child level=1
> Error searching -5
>
> Is there something else I could do to recover either the filesystem or
> at least the files?
>
> Regards,
> Glenn
>
> On Sun, 10 Mar 2019 at 08:35, Glenn Trigg <ggtrigg@gmail.com> wrote:
> >
> > Hello,
> >
> > I had some random machine freezing events which I suspected was due to
> > issues with a raid1 filesystem and kernel module crashes. I attempted
> > to use the information available to get the filesystem into a good
> > state where "btrfs check" and "btrfs scrub" would not have any errors,
> > however I fear things have become worse.
> >
> > The current state of things is that the filesystem won't mount at all now.
> >
> > % mount -r /dev/sda1 /data
> > mount: /data: can't read superblock on /dev/sda1.
> >
> > and dmesg says:
> >
> > [15944.017629] BTRFS info (device sda1): disk space caching is enabled
> > [15944.017632] BTRFS info (device sda1): has skinny extents
> > [15944.024480] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd
> > 0, flush 0, corrupt 1, gen 0
> > [15944.024487] BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd
> > 0, flush 0, corrupt 4, gen 0
> > [15944.029292] BTRFS error (device sda1): parent transid verify failed
> > on 628168376320 wanted 37601 found 37700
> > [15944.029466] BTRFS error (device sda1): parent transid verify failed
> > on 628168376320 wanted 37601 found 37700
> >
> > Other system information is:
> > % uname -a
> > Linux izen 4.18.0-16-generic #17-Ubuntu SMP Fri Feb 8 00:06:57 UTC
> > 2019 x86_64 x86_64 x86_64 GNU/Linux
> >
> > % btrfs --version
> > btrfs-progs v4.16.1
> >
> > % btrfs fi show
> > Label: 'root'  uuid: 65fd7f11-4f60-435f-928b-6d15f12bc417
> > Total devices 1 FS bytes used 101.75GiB
> > devid    1 size 232.88GiB used 232.85GiB path /dev/nvme0n1p1
> >
> > Label: 'data'  uuid: d5e50511-3e31-4de6-ba37-c5841895be9f
> > Total devices 2 FS bytes used 830.44GiB
> > devid    1 size 1.82TiB used 669.03GiB path /dev/sda1
> > devid    2 size 1.82TiB used 817.06GiB path /dev/sdb1
> >
> > % btrfs check /dev/sda1
> > Checking filesystem on /dev/sda1
> > UUID: d5e50511-3e31-4de6-ba37-c5841895be9f
> > checking extents
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
> > Ignoring transid failure
> > bad block 628168343552
> > ERROR: errors found in extent allocation tree or chunk allocation
> > checking free space cache
> > cache and super generation don't match, space cache will be invalidated
> > checking fs roots
> > root 5 root dir 256 not found
> > ERROR: errors found in fs roots
> > found 528138240 bytes used, error(s) found
> > total csum bytes: 0
> > total tree bytes: 1785856
> > total fs tree bytes: 1064960
> > total extent tree bytes: 81920
> > btree space waste bytes: 606983
> > file data blocks allocated: 215220224
> >  referenced 215220224
> >
> > % btrfs rescue super-recover /dev/sda1
> > All supers are valid, no need to recover
> >
> > Where do I go from here?
> >
> > Regards,
> > Glenn

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-03-09 21:35 help request for an unmountable raid1 filesystem Glenn Trigg
  2019-03-23  0:45 ` Glenn Trigg
@ 2019-03-29  2:27 ` Chris Murphy
  2019-03-29  3:21   ` Chris Murphy
  2019-03-30 23:43   ` Glenn Trigg
  1 sibling, 2 replies; 12+ messages in thread
From: Chris Murphy @ 2019-03-29  2:27 UTC (permalink / raw)
  To: Glenn Trigg; +Cc: Btrfs BTRFS

On Sat, Mar 9, 2019 at 2:36 PM Glenn Trigg <ggtrigg@gmail.com> wrote:

> I had some random machine freezing events which I suspected was due to
> issues with a raid1 filesystem and kernel module crashes.

Hard to say with available information. It's more likely hardware
related, and then there's on-disk corruption.


This:

> % mount -r /dev/sda1 /data
> mount: /data: can't read superblock on /dev/sda1.

and this:

> % btrfs rescue super-recover /dev/sda1
> All supers are valid, no need to recover

Seem in conflict. I don't really understand how the kernel complains
about a bad super and yet user space tools say they're all OK.  What
happens if you try:

# mount -o ro,nologreplay,usebackuproot

If that doesn't work, including kernel messages again, and also
include output from:

# btrfs insp dump-s -fa /dev/sda1
# btrfs insp dump-s -fa /dev/sdb1



>
> and dmesg says:
>
> [15944.017629] BTRFS info (device sda1): disk space caching is enabled
> [15944.017632] BTRFS info (device sda1): has skinny extents
> [15944.024480] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd
> 0, flush 0, corrupt 1, gen 0
> [15944.024487] BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd
> 0, flush 0, corrupt 4, gen 0
> [15944.029292] BTRFS error (device sda1): parent transid verify failed
> on 628168376320 wanted 37601 found 37700
> [15944.029466] BTRFS error (device sda1): parent transid verify failed
> on 628168376320 wanted 37601 found 37700

That's usually bad.


> Other system information is:
> % uname -a
> Linux izen 4.18.0-16-generic #17-Ubuntu SMP Fri Feb 8 00:06:57 UTC
> 2019 x86_64 x86_64 x86_64 GNU/Linux

It looks like extent tree corruption so I don't think it'll help to
use a newer kernel; but I'd try it anyway in the meantime until a
developer gets around to responding. Distro specific kernels tend to
be supported by that distribution where upstream lists tend to support
mainline. So I suggest 5.0.4, or 4.19.32, or you can be brave and
download this, image it to a USB stick (dd if=file of=/dev/ bs=1M
oflag=direct) which of course will erase everything on the stick.

https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190327.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20190327.n.0.iso

That might have 5.1rc2 on it, or something in between rc1 and rc2.
You're still going to try and mount it read-only per above command, so
even if it blows up it's not going to make this worse.


> % btrfs check /dev/sda1
> Checking filesystem on /dev/sda1
> UUID: d5e50511-3e31-4de6-ba37-c5841895be9f
> checking extents
> parent transid verify failed on 628168343552 wanted 28163 found 37700
> parent transid verify failed on 628168343552 wanted 28163 found 37700
> parent transid verify failed on 628168343552 wanted 28163 found 37700
> parent transid verify failed on 628168343552 wanted 28163 found 37700

The transid's are really far apart, definitely something went really
wrong. It could be hardware or both hardware and btrfs bug. That it
affected *both* copies is a little weird unless it's memory corruption
related, and then a lot of things can go wrong.


>
> Where do I go from here?

If it can't be mounted, then the only chance is `btrfs-find-tree` and
`btrfs restore` to try and scrape out whatever data you need that
isn't already backed up. The priority before trying to repair it, is
to get anything important off because trying to repair it has a good
chance of permanent data loss. Definitely the latest tools are
recommended for repair, kernel doesn't matter so much.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-03-29  2:27 ` Chris Murphy
@ 2019-03-29  3:21   ` Chris Murphy
  2019-03-30 23:43   ` Glenn Trigg
  1 sibling, 0 replies; 12+ messages in thread
From: Chris Murphy @ 2019-03-29  3:21 UTC (permalink / raw)
  To: Btrfs BTRFS; +Cc: Glenn Trigg

On Thu, Mar 28, 2019 at 8:27 PM Chris Murphy <lists@colorremedies.com> wrote:
>>So I suggest 5.0.4, or 4.19.32, or you can be brave and
> download this, image it to a USB stick (dd if=file of=/dev/ bs=1M
> oflag=direct) which of course will erase everything on the stick.
>
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190327.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20190327.n.0.iso
>
> That might have 5.1rc2 on it, or something in between rc1 and rc2.

Oh yeah, in case you go this route, at the bootloader menu, choose the
troubleshooting submenu and then choose to rescue a system. When you
get to the text installer there will be an option to get dropped to a
shell, at which point the installer quits. ~600MB


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-03-29  2:27 ` Chris Murphy
  2019-03-29  3:21   ` Chris Murphy
@ 2019-03-30 23:43   ` Glenn Trigg
  2019-03-31 21:34     ` Chris Murphy
  1 sibling, 1 reply; 12+ messages in thread
From: Glenn Trigg @ 2019-03-30 23:43 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

[-- Attachment #1: Type: text/plain, Size: 4487 bytes --]

Hi Chris,

Thanks for replying.

On Fri, 29 Mar 2019 at 13:27, Chris Murphy <lists@colorremedies.com> wrote:
...
> Seem in conflict. I don't really understand how the kernel complains
> about a bad super and yet user space tools say they're all OK.  What
> happens if you try:
>
> # mount -o ro,nologreplay,usebackuproot

I got essentially the same results...

% mount -o ro,nologreplay,usebackuproot /dev/sda1
mount: /data: can't read superblock on /dev/sda1.

and from dmesg...

[73194.642197] BTRFS info (device sda1): use lzo compression, level 0
[73194.642206] BTRFS info (device sda1): disabling log replay at mount time
[73194.642210] BTRFS info (device sda1): trying to use backup root at mount time
[73194.642211] BTRFS info (device sda1): disk space caching is enabled
[73194.642213] BTRFS info (device sda1): has skinny extents
[73194.726290] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd
0, flush 0, corrupt 1, gen 0
[73194.726297] BTRFS info (device sda1): bdev /dev/sdc1 errs: wr 0, rd
0, flush 0, corrupt 4, gen 0
[73194.753315] BTRFS error (device sda1): parent transid verify failed
on 628168376320 wanted 37601 found 37712
[73194.779836] BTRFS error (device sda1): parent transid verify failed
on 628168376320 wanted 37601 found 37712

> If that doesn't work, including kernel messages again, and also
> include output from:
>
> # btrfs insp dump-s -fa /dev/sda1
> # btrfs insp dump-s -fa /dev/sdb1

I've attached the output from these commands (note that since I added
an extra disk, what was sdb1 has now become sdc1).

I'll have a go at the newer kernel approach you suggest and see what
happens there.

Regards,
Glenn

> >
> > and dmesg says:
> >
> > [15944.017629] BTRFS info (device sda1): disk space caching is enabled
> > [15944.017632] BTRFS info (device sda1): has skinny extents
> > [15944.024480] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd
> > 0, flush 0, corrupt 1, gen 0
> > [15944.024487] BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 0, rd
> > 0, flush 0, corrupt 4, gen 0
> > [15944.029292] BTRFS error (device sda1): parent transid verify failed
> > on 628168376320 wanted 37601 found 37700
> > [15944.029466] BTRFS error (device sda1): parent transid verify failed
> > on 628168376320 wanted 37601 found 37700
>
> That's usually bad.
>
>
> > Other system information is:
> > % uname -a
> > Linux izen 4.18.0-16-generic #17-Ubuntu SMP Fri Feb 8 00:06:57 UTC
> > 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> It looks like extent tree corruption so I don't think it'll help to
> use a newer kernel; but I'd try it anyway in the meantime until a
> developer gets around to responding. Distro specific kernels tend to
> be supported by that distribution where upstream lists tend to support
> mainline. So I suggest 5.0.4, or 4.19.32, or you can be brave and
> download this, image it to a USB stick (dd if=file of=/dev/ bs=1M
> oflag=direct) which of course will erase everything on the stick.
>
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190327.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20190327.n.0.iso
>
> That might have 5.1rc2 on it, or something in between rc1 and rc2.
> You're still going to try and mount it read-only per above command, so
> even if it blows up it's not going to make this worse.
>
>
> > % btrfs check /dev/sda1
> > Checking filesystem on /dev/sda1
> > UUID: d5e50511-3e31-4de6-ba37-c5841895be9f
> > checking extents
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
> > parent transid verify failed on 628168343552 wanted 28163 found 37700
>
> The transid's are really far apart, definitely something went really
> wrong. It could be hardware or both hardware and btrfs bug. That it
> affected *both* copies is a little weird unless it's memory corruption
> related, and then a lot of things can go wrong.
>
>
> >
> > Where do I go from here?
>
> If it can't be mounted, then the only chance is `btrfs-find-tree` and
> `btrfs restore` to try and scrape out whatever data you need that
> isn't already backed up. The priority before trying to repair it, is
> to get anything important off because trying to repair it has a good
> chance of permanent data loss. Definitely the latest tools are
> recommended for repair, kernel doesn't matter so much.
>
>
> --
> Chris Murphy

[-- Attachment #2: btrfs_insp_sda1.txt --]
[-- Type: text/plain, Size: 11042 bytes --]

superblock: bytenr=65536, device=/dev/sda1
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0x68d72f78 [match]
bytenr			65536
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			d5e50511-3e31-4de6-ba37-c5841895be9f
metadata_uuid		d5e50511-3e31-4de6-ba37-c5841895be9f
label			data
generation		37713
root			628168458240
sys_array_size		226
chunk_root_generation	37696
root_level		1
chunk_root		1085582344192
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		4000795660288
bytes_used		891682791424
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
root_dir		6
num_devices		2
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x169
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
cache_generation	37697
uuid_tree_generation	37697
dev_item.uuid		a28b0f34-f14d-492b-995b-2dd8a78ec9bb
dev_item.fsid		d5e50511-3e31-4de6-ba37-c5841895be9f [match]
dev_item.type		0
dev_item.total_bytes	2000397795328
dev_item.bytes_used	718366834688
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1085582344192)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 543314411520
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
			stripe 1 devid 1 offset 2185232384
			dev_uuid a28b0f34-f14d-492b-995b-2dd8a78ec9bb
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 1470082580480)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 2 offset 817219239936
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
backup_roots[4]:
	backup 0:
		backup_tree_root:	628642381824	gen: 37697	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628642529280	gen: 37697	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628645314560	gen: 37697	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 1:
		backup_tree_root:	628628897792	gen: 37694	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628624818176	gen: 37694	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628629946368	gen: 37694	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 2:
		backup_tree_root:	628633518080	gen: 37695	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628633534464	gen: 37695	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628634140672	gen: 37695	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 3:
		backup_tree_root:	628632535040	gen: 37696	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628624818176	gen: 37696	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628632649728	gen: 37696	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2


superblock: bytenr=67108864, device=/dev/sda1
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0xc8b607b6 [match]
bytenr			67108864
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			d5e50511-3e31-4de6-ba37-c5841895be9f
metadata_uuid		d5e50511-3e31-4de6-ba37-c5841895be9f
label			data
generation		37713
root			628168458240
sys_array_size		226
chunk_root_generation	37696
root_level		1
chunk_root		1085582344192
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		4000795660288
bytes_used		891682791424
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
root_dir		6
num_devices		2
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x169
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
cache_generation	37697
uuid_tree_generation	37697
dev_item.uuid		a28b0f34-f14d-492b-995b-2dd8a78ec9bb
dev_item.fsid		d5e50511-3e31-4de6-ba37-c5841895be9f [match]
dev_item.type		0
dev_item.total_bytes	2000397795328
dev_item.bytes_used	718366834688
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1085582344192)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 543314411520
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
			stripe 1 devid 1 offset 2185232384
			dev_uuid a28b0f34-f14d-492b-995b-2dd8a78ec9bb
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 1470082580480)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 2 offset 817219239936
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
backup_roots[4]:
	backup 0:
		backup_tree_root:	628642381824	gen: 37697	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628642529280	gen: 37697	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628645314560	gen: 37697	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 1:
		backup_tree_root:	628628897792	gen: 37694	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628624818176	gen: 37694	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628629946368	gen: 37694	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 2:
		backup_tree_root:	628633518080	gen: 37695	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628633534464	gen: 37695	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628634140672	gen: 37695	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 3:
		backup_tree_root:	628632535040	gen: 37696	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628624818176	gen: 37696	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628632649728	gen: 37696	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2


superblock: bytenr=274877906944, device=/dev/sda1
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0x35315187 [match]
bytenr			274877906944
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			d5e50511-3e31-4de6-ba37-c5841895be9f
metadata_uuid		d5e50511-3e31-4de6-ba37-c5841895be9f
label			data
generation		37713
root			628168458240
sys_array_size		226
chunk_root_generation	37696
root_level		1
chunk_root		1085582344192
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		4000795660288
bytes_used		891682791424
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
root_dir		6
num_devices		2
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x169
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
cache_generation	37697
uuid_tree_generation	37697
dev_item.uuid		a28b0f34-f14d-492b-995b-2dd8a78ec9bb
dev_item.fsid		d5e50511-3e31-4de6-ba37-c5841895be9f [match]
dev_item.type		0
dev_item.total_bytes	2000397795328
dev_item.bytes_used	718366834688
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1085582344192)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 543314411520
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
			stripe 1 devid 1 offset 2185232384
			dev_uuid a28b0f34-f14d-492b-995b-2dd8a78ec9bb
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 1470082580480)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 2 offset 817219239936
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
backup_roots[4]:
	backup 0:
		backup_tree_root:	628642381824	gen: 37697	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628642529280	gen: 37697	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628645314560	gen: 37697	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 1:
		backup_tree_root:	628628897792	gen: 37694	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628624818176	gen: 37694	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628629946368	gen: 37694	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 2:
		backup_tree_root:	628633518080	gen: 37695	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628633534464	gen: 37695	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628634140672	gen: 37695	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 3:
		backup_tree_root:	628632535040	gen: 37696	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628624818176	gen: 37696	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628632649728	gen: 37696	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2



[-- Attachment #3: btrfs_insp_sdc1.txt --]
[-- Type: text/plain, Size: 11042 bytes --]

superblock: bytenr=65536, device=/dev/sdc1
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0xbc7e4798 [match]
bytenr			65536
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			d5e50511-3e31-4de6-ba37-c5841895be9f
metadata_uuid		d5e50511-3e31-4de6-ba37-c5841895be9f
label			data
generation		37713
root			628168458240
sys_array_size		226
chunk_root_generation	37696
root_level		1
chunk_root		1085582344192
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		4000795660288
bytes_used		891682791424
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
root_dir		6
num_devices		2
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x169
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
cache_generation	37697
uuid_tree_generation	37697
dev_item.uuid		d9c56d7a-21a1-4197-a701-5493392e1ae1
dev_item.fsid		d5e50511-3e31-4de6-ba37-c5841895be9f [match]
dev_item.type		0
dev_item.total_bytes	2000397864960
dev_item.bytes_used	877314179072
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		2
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1085582344192)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 543314411520
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
			stripe 1 devid 1 offset 2185232384
			dev_uuid a28b0f34-f14d-492b-995b-2dd8a78ec9bb
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 1470082580480)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 2 offset 817219239936
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
backup_roots[4]:
	backup 0:
		backup_tree_root:	628642381824	gen: 37697	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628642529280	gen: 37697	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628645314560	gen: 37697	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 1:
		backup_tree_root:	628628897792	gen: 37694	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628624818176	gen: 37694	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628629946368	gen: 37694	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 2:
		backup_tree_root:	628633518080	gen: 37695	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628633534464	gen: 37695	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628634140672	gen: 37695	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 3:
		backup_tree_root:	628632535040	gen: 37696	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628624818176	gen: 37696	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628632649728	gen: 37696	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2


superblock: bytenr=67108864, device=/dev/sdc1
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0x1c1f6f56 [match]
bytenr			67108864
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			d5e50511-3e31-4de6-ba37-c5841895be9f
metadata_uuid		d5e50511-3e31-4de6-ba37-c5841895be9f
label			data
generation		37713
root			628168458240
sys_array_size		226
chunk_root_generation	37696
root_level		1
chunk_root		1085582344192
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		4000795660288
bytes_used		891682791424
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
root_dir		6
num_devices		2
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x169
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
cache_generation	37697
uuid_tree_generation	37697
dev_item.uuid		d9c56d7a-21a1-4197-a701-5493392e1ae1
dev_item.fsid		d5e50511-3e31-4de6-ba37-c5841895be9f [match]
dev_item.type		0
dev_item.total_bytes	2000397864960
dev_item.bytes_used	877314179072
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		2
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1085582344192)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 543314411520
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
			stripe 1 devid 1 offset 2185232384
			dev_uuid a28b0f34-f14d-492b-995b-2dd8a78ec9bb
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 1470082580480)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 2 offset 817219239936
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
backup_roots[4]:
	backup 0:
		backup_tree_root:	628642381824	gen: 37697	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628642529280	gen: 37697	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628645314560	gen: 37697	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 1:
		backup_tree_root:	628628897792	gen: 37694	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628624818176	gen: 37694	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628629946368	gen: 37694	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 2:
		backup_tree_root:	628633518080	gen: 37695	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628633534464	gen: 37695	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628634140672	gen: 37695	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 3:
		backup_tree_root:	628632535040	gen: 37696	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628624818176	gen: 37696	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628632649728	gen: 37696	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2


superblock: bytenr=274877906944, device=/dev/sdc1
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0xe1983967 [match]
bytenr			274877906944
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			d5e50511-3e31-4de6-ba37-c5841895be9f
metadata_uuid		d5e50511-3e31-4de6-ba37-c5841895be9f
label			data
generation		37713
root			628168458240
sys_array_size		226
chunk_root_generation	37696
root_level		1
chunk_root		1085582344192
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		4000795660288
bytes_used		891682791424
sectorsize		4096
nodesize		16384
leafsize (deprecated)	16384
stripesize		4096
root_dir		6
num_devices		2
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x169
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
cache_generation	37697
uuid_tree_generation	37697
dev_item.uuid		d9c56d7a-21a1-4197-a701-5493392e1ae1
dev_item.fsid		d5e50511-3e31-4de6-ba37-c5841895be9f [match]
dev_item.type		0
dev_item.total_bytes	2000397864960
dev_item.bytes_used	877314179072
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		2
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 1085582344192)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 543314411520
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
			stripe 1 devid 1 offset 2185232384
			dev_uuid a28b0f34-f14d-492b-995b-2dd8a78ec9bb
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 1470082580480)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 2 offset 817219239936
			dev_uuid d9c56d7a-21a1-4197-a701-5493392e1ae1
backup_roots[4]:
	backup 0:
		backup_tree_root:	628642381824	gen: 37697	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628642529280	gen: 37697	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628645314560	gen: 37697	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 1:
		backup_tree_root:	628628897792	gen: 37694	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628624818176	gen: 37694	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628629946368	gen: 37694	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 2:
		backup_tree_root:	628633518080	gen: 37695	level: 1
		backup_chunk_root:	1085582409728	gen: 37616	level: 1
		backup_extent_root:	628633534464	gen: 37695	level: 2
		backup_fs_root:		628623933440	gen: 37694	level: 2
		backup_dev_root:	628452327424	gen: 37616	level: 1
		backup_csum_root:	628634140672	gen: 37695	level: 2
		backup_total_bytes:	4000795663872
		backup_bytes_used:	891681054720
		backup_num_devices:	2

	backup 3:
		backup_tree_root:	628632535040	gen: 37696	level: 1
		backup_chunk_root:	1085582344192	gen: 37696	level: 1
		backup_extent_root:	628624818176	gen: 37696	level: 2
		backup_fs_root:		628635549696	gen: 37696	level: 2
		backup_dev_root:	628642152448	gen: 37696	level: 1
		backup_csum_root:	628632649728	gen: 37696	level: 2
		backup_total_bytes:	4000795660288
		backup_bytes_used:	891681054720
		backup_num_devices:	2



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-03-30 23:43   ` Glenn Trigg
@ 2019-03-31 21:34     ` Chris Murphy
  2019-04-01  5:48       ` Glenn Trigg
  0 siblings, 1 reply; 12+ messages in thread
From: Chris Murphy @ 2019-03-31 21:34 UTC (permalink / raw)
  To: Glenn Trigg; +Cc: Chris Murphy, Btrfs BTRFS

On Sat, Mar 30, 2019 at 5:43 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
>
> Hi Chris,
>
> Thanks for replying.
>
> On Fri, 29 Mar 2019 at 13:27, Chris Murphy <lists@colorremedies.com> wrote:
> ...
> > Seem in conflict. I don't really understand how the kernel complains
> > about a bad super and yet user space tools say they're all OK.  What
> > happens if you try:
> >
> > # mount -o ro,nologreplay,usebackuproot
>
> I got essentially the same results...
>
> % mount -o ro,nologreplay,usebackuproot /dev/sda1
> mount: /data: can't read superblock on /dev/sda1.

I'm confused because "can't read superblock" isn't found in fs/btrfs.
I'm only finding it in fs/gfs2/ops_fstype.c

From what you provided, /dev/sda1 definitely has a valid btrfs
superblock. I wonder if there's some other stale something or other on
this partition?

What do you get for

$ sudo blkid
$ sudo wipefs -an /dev/sda1
$ sudo mount -v -o ro,nologreplay,usebackuproot

-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-03-31 21:34     ` Chris Murphy
@ 2019-04-01  5:48       ` Glenn Trigg
  2019-04-01 17:14         ` Chris Murphy
  0 siblings, 1 reply; 12+ messages in thread
From: Glenn Trigg @ 2019-04-01  5:48 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

Hi Chris,

After booting the fedora usb stick (running rc2), I got the same results.

On Mon, 1 Apr 2019 at 08:35, Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Mar 30, 2019 at 5:43 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
...
>
> I'm confused because "can't read superblock" isn't found in fs/btrfs.
> I'm only finding it in fs/gfs2/ops_fstype.c
>
> From what you provided, /dev/sda1 definitely has a valid btrfs
> superblock. I wonder if there's some other stale something or other on
> this partition?
>
> What do you get for
>
> $ sudo blkid

/dev/sda1: LABEL="data" UUID="d5e50511-3e31-4de6-ba37-c5841895be9f"
UUID_SUB="a28b0f34-f14d-492b-995b-2dd8a78ec9bb" TYPE="btrfs"
PARTUUID="efa240ec-01"
/dev/sdc1: LABEL="data" UUID="d5e50511-3e31-4de6-ba37-c5841895be9f"
UUID_SUB="d9c56d7a-21a1-4197-a701-5493392e1ae1" TYPE="btrfs"
PARTUUID="1aae1443-c03e-4509-ad06-124a64c4df4f"

> $ sudo wipefs -an /dev/sda1 /dev/sdc1

/dev/sda1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
52 66 53 5f 4d
/dev/sdc1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
52 66 53 5f 4d

> $ sudo mount -v -o ro,nologreplay,usebackuproot

mount: /data: can't read superblock on /dev/sda1.

and in dmesg...

[ 1045.029981] BTRFS info (device sda1): use lzo compression, level 0
[ 1045.029987] BTRFS info (device sda1): disabling log replay at mount time
[ 1045.029991] BTRFS info (device sda1): trying to use backup root at mount time
[ 1045.029992] BTRFS info (device sda1): disk space caching is enabled
[ 1045.029994] BTRFS info (device sda1): has skinny extents
[ 1045.127375] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd
0, flush 0, corrupt 1, gen 0
[ 1045.127381] BTRFS info (device sda1): bdev /dev/sdc1 errs: wr 0, rd
0, flush 0, corrupt 4, gen 0
[ 1045.154198] BTRFS error (device sda1): parent transid verify failed
on 628168376320 wanted 37601 found 37712
[ 1045.178310] BTRFS error (device sda1): parent transid verify failed
on 628168376320 wanted 37601 found 37712

I.e. the same as before. It's puzzling!

Regards,
Glenn

>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-04-01  5:48       ` Glenn Trigg
@ 2019-04-01 17:14         ` Chris Murphy
  2019-04-05  5:51           ` Glenn Trigg
  0 siblings, 1 reply; 12+ messages in thread
From: Chris Murphy @ 2019-04-01 17:14 UTC (permalink / raw)
  To: Glenn Trigg, Btrfs BTRFS

On Sun, Mar 31, 2019 at 11:48 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
>
> Hi Chris,
>
> After booting the fedora usb stick (running rc2), I got the same results.
>
> On Mon, 1 Apr 2019 at 08:35, Chris Murphy <lists@colorremedies.com> wrote:
> >
> > On Sat, Mar 30, 2019 at 5:43 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
> ...
> >
> > I'm confused because "can't read superblock" isn't found in fs/btrfs.
> > I'm only finding it in fs/gfs2/ops_fstype.c
> >
> > From what you provided, /dev/sda1 definitely has a valid btrfs
> > superblock. I wonder if there's some other stale something or other on
> > this partition?
> >
> > What do you get for
> >
> > $ sudo blkid
>
> /dev/sda1: LABEL="data" UUID="d5e50511-3e31-4de6-ba37-c5841895be9f"
> UUID_SUB="a28b0f34-f14d-492b-995b-2dd8a78ec9bb" TYPE="btrfs"
> PARTUUID="efa240ec-01"
> /dev/sdc1: LABEL="data" UUID="d5e50511-3e31-4de6-ba37-c5841895be9f"
> UUID_SUB="d9c56d7a-21a1-4197-a701-5493392e1ae1" TYPE="btrfs"
> PARTUUID="1aae1443-c03e-4509-ad06-124a64c4df4f"
>
> > $ sudo wipefs -an /dev/sda1 /dev/sdc1
>
> /dev/sda1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
> 52 66 53 5f 4d
> /dev/sdc1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
> 52 66 53 5f 4d
>
> > $ sudo mount -v -o ro,nologreplay,usebackuproot
>
> mount: /data: can't read superblock on /dev/sda1.
>

That is the sort of error I'd expect if the device were in fstab to be
mounted at /data. But you're running this mount command while booted
from the Fedora USB stick? I'm pretty lost at this point.

Anyway, your best bet is to scrape with `btrfs restore` and then move
on, i.e. start over with a new file system. Right now I have no
confidence in the repair tools, but maybe a developer will come along
and offer some different advice on additional steps. For sure any of
the user space repair options are a last resort. There should be a
btrfs-progs 5.0 soon that is supposed to be safer, and might be worth
a shot. But I think it's a coin toss.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-04-01 17:14         ` Chris Murphy
@ 2019-04-05  5:51           ` Glenn Trigg
  2019-04-05  6:44             ` Chris Murphy
  0 siblings, 1 reply; 12+ messages in thread
From: Glenn Trigg @ 2019-04-05  5:51 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

Hi Chris,

Thanks for spending the time and energy to help me look into this.

btrfs restore isn't very happy either, so I guess I'll wait until
btrfs-progs v5.0 comes out and see if that helps.

btrfs restore says...
% ./btrfs restore -v -D /dev/sda1 /data2
This is a dry-run, no files are going to be restored
parent transid verify failed on 628168376320 wanted 37601 found 37712
parent transid verify failed on 628168376320 wanted 37601 found 37712
parent transid verify failed on 628168376320 wanted 37601 found 37712
parent transid verify failed on 628168376320 wanted 37601 found 37712
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=628624064512 item=0 parent
level=1 child level=1
Error searching -5

and...

% ./btrfs restore -l /dev/sda1
 tree key (EXTENT_TREE ROOT_ITEM 0) 628168359936 level 1
 tree key (DEV_TREE ROOT_ITEM 0) 628642152448 level 1
 tree key (FS_TREE ROOT_ITEM 0) 628635549696 level 2
 tree key (CSUM_TREE ROOT_ITEM 0) 628168343552 level 0
 tree key (UUID_TREE ROOT_ITEM 0) 628810366976 level 0
 tree key (DATA_RELOC_TREE ROOT_ITEM 0) 628168491008 level 0

Regards,
Glenn

On Tue, 2 Apr 2019 at 04:14, Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sun, Mar 31, 2019 at 11:48 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
> >
> > Hi Chris,
> >
> > After booting the fedora usb stick (running rc2), I got the same results.
> >
> > On Mon, 1 Apr 2019 at 08:35, Chris Murphy <lists@colorremedies.com> wrote:
> > >
> > > On Sat, Mar 30, 2019 at 5:43 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
> > ...
> > >
> > > I'm confused because "can't read superblock" isn't found in fs/btrfs.
> > > I'm only finding it in fs/gfs2/ops_fstype.c
> > >
> > > From what you provided, /dev/sda1 definitely has a valid btrfs
> > > superblock. I wonder if there's some other stale something or other on
> > > this partition?
> > >
> > > What do you get for
> > >
> > > $ sudo blkid
> >
> > /dev/sda1: LABEL="data" UUID="d5e50511-3e31-4de6-ba37-c5841895be9f"
> > UUID_SUB="a28b0f34-f14d-492b-995b-2dd8a78ec9bb" TYPE="btrfs"
> > PARTUUID="efa240ec-01"
> > /dev/sdc1: LABEL="data" UUID="d5e50511-3e31-4de6-ba37-c5841895be9f"
> > UUID_SUB="d9c56d7a-21a1-4197-a701-5493392e1ae1" TYPE="btrfs"
> > PARTUUID="1aae1443-c03e-4509-ad06-124a64c4df4f"
> >
> > > $ sudo wipefs -an /dev/sda1 /dev/sdc1
> >
> > /dev/sda1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
> > 52 66 53 5f 4d
> > /dev/sdc1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
> > 52 66 53 5f 4d
> >
> > > $ sudo mount -v -o ro,nologreplay,usebackuproot
> >
> > mount: /data: can't read superblock on /dev/sda1.
> >
>
> That is the sort of error I'd expect if the device were in fstab to be
> mounted at /data. But you're running this mount command while booted
> from the Fedora USB stick? I'm pretty lost at this point.
>
> Anyway, your best bet is to scrape with `btrfs restore` and then move
> on, i.e. start over with a new file system. Right now I have no
> confidence in the repair tools, but maybe a developer will come along
> and offer some different advice on additional steps. For sure any of
> the user space repair options are a last resort. There should be a
> btrfs-progs 5.0 soon that is supposed to be safer, and might be worth
> a shot. But I think it's a coin toss.
>
>
> --
> Chris Murphy

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-04-05  5:51           ` Glenn Trigg
@ 2019-04-05  6:44             ` Chris Murphy
  2019-04-06  0:37               ` Glenn Trigg
  0 siblings, 1 reply; 12+ messages in thread
From: Chris Murphy @ 2019-04-05  6:44 UTC (permalink / raw)
  To: Glenn Trigg; +Cc: Chris Murphy, Btrfs BTRFS

On Thu, Apr 4, 2019 at 11:51 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
>
> Hi Chris,
>
> Thanks for spending the time and energy to help me look into this.
>
> btrfs restore isn't very happy either, so I guess I'll wait until
> btrfs-progs v5.0 comes out and see if that helps.
>
> btrfs restore says...
> % ./btrfs restore -v -D /dev/sda1 /data2
> This is a dry-run, no files are going to be restored
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> parent transid verify failed on 628168376320 wanted 37601 found 37712
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=628624064512 item=0 parent
> level=1 child level=1
> Error searching -5

I'm gonna suggest that's a bug, just because btrfs restore needs to be
expected to do better than this. I suggest taking a btrfs-image while
waiting for btrfs-progs 5.0.

# btrfs-image -c9 -t4 -ss -w /dev/ /pathtoimage/


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: help request for an unmountable raid1 filesystem
  2019-04-05  6:44             ` Chris Murphy
@ 2019-04-06  0:37               ` Glenn Trigg
  0 siblings, 0 replies; 12+ messages in thread
From: Glenn Trigg @ 2019-04-06  0:37 UTC (permalink / raw)
  To: Chris Murphy; +Cc: Btrfs BTRFS

Hi Chris,

On Fri, 5 Apr 2019 at 17:45, Chris Murphy <lists@colorremedies.com> wrote:
>
> On Thu, Apr 4, 2019 at 11:51 PM Glenn Trigg <ggtrigg@gmail.com> wrote:
> >
> > Hi Chris,
> >
> > Thanks for spending the time and energy to help me look into this.
> >
> > btrfs restore isn't very happy either, so I guess I'll wait until
> > btrfs-progs v5.0 comes out and see if that helps.
> >
> > btrfs restore says...
> > % ./btrfs restore -v -D /dev/sda1 /data2
> > This is a dry-run, no files are going to be restored
> > parent transid verify failed on 628168376320 wanted 37601 found 37712
> > parent transid verify failed on 628168376320 wanted 37601 found 37712
> > parent transid verify failed on 628168376320 wanted 37601 found 37712
> > parent transid verify failed on 628168376320 wanted 37601 found 37712
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=628624064512 item=0 parent
> > level=1 child level=1
> > Error searching -5
>
> I'm gonna suggest that's a bug, just because btrfs restore needs to be
> expected to do better than this. I suggest taking a btrfs-image while
> waiting for btrfs-progs 5.0.
>
> # btrfs-image -c9 -t4 -ss -w /dev/ /pathtoimage/

I ran this command (using version 4.20.2) and after a whole bunch of
WARNING: cannot find a hash collision for ...
type messages it then produced this output:

Couldn't map the block 1350591496192
Invalid mapping for 1350591496192-1350591512576, got 1469008838656-1470082580480
Couldn't map the block 1350591496192
bad tree block 1350591496192, bytenr mismatch, want=1350591496192, have=0
ERROR: unable to read log root block
ERROR: create failed: No such file or directory

and no output file was created. This doesn't look very good!

Regards,
Glenn

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2019-04-06  0:40 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-09 21:35 help request for an unmountable raid1 filesystem Glenn Trigg
2019-03-23  0:45 ` Glenn Trigg
2019-03-29  0:53   ` Glenn Trigg
2019-03-29  2:27 ` Chris Murphy
2019-03-29  3:21   ` Chris Murphy
2019-03-30 23:43   ` Glenn Trigg
2019-03-31 21:34     ` Chris Murphy
2019-04-01  5:48       ` Glenn Trigg
2019-04-01 17:14         ` Chris Murphy
2019-04-05  5:51           ` Glenn Trigg
2019-04-05  6:44             ` Chris Murphy
2019-04-06  0:37               ` Glenn Trigg

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.