linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Need help with potential ~45TB dataloss
@ 2018-11-30 13:53 Patrick Dijkgraaf
  2018-11-30 23:57 ` Qu Wenruo
  0 siblings, 1 reply; 14+ messages in thread
From: Patrick Dijkgraaf @ 2018-11-30 13:53 UTC (permalink / raw)
  To: linux-btrfs

Hi all,

I have been a happy BTRFS user for quite some time. But now I'm facing
a potential ~45TB dataloss... :-(
I hope someone can help!

I have Server A and Server B. Both having a 20-devices BTRFS RAID6
filesystem. Because of known RAID5/6 risks, Server B was a backup of
Server A.
After applying updates to server B and reboot, the FS would not mount
anymore. Because it was "just" a backup. I decided to recreate the FS
and perform a new backup. Later, I discovered that the FS was not
broken, but I faced this issue: 
https://patchwork.kernel.org/patch/10694997/

Anyway, the FS was already recreated, so I needed to do a new backup.
During the backup (using rsync -vah), Server A (the source) encountered
an I/O error and my rsync failed. In an attempt to "quick fix" the
issue, I rebooted Server A after which the FS would not mount anymore.

I documented what I have tried, below. I have not yet tried anything
except what is shown, because I am afraid of causing more harm to
the FS. I hope somebody here can give me advice on how to (hopefully)
retrieve my data...

Thanks in advance!

==========================================

[root@cornelis ~]# btrfs fi show
Label: 'cornelis-btrfs'  uuid: ac643516-670e-40f3-aa4c-f329fc3795fd
	Total devices 1 FS bytes used 463.92GiB
	devid    1 size 800.00GiB used 493.02GiB path
/dev/mapper/cornelis-cornelis--btrfs

Label: 'data'  uuid: 4c66fa8b-8fc6-4bba-9d83-02a2a1d69ad5
	Total devices 20 FS bytes used 44.85TiB
	devid    1 size 3.64TiB used 3.64TiB path /dev/sdn2
	devid    2 size 3.64TiB used 3.64TiB path /dev/sdp2
	devid    3 size 3.64TiB used 3.64TiB path /dev/sdu2
	devid    4 size 3.64TiB used 3.64TiB path /dev/sdx2
	devid    5 size 3.64TiB used 3.64TiB path /dev/sdh2
	devid    6 size 3.64TiB used 3.64TiB path /dev/sdg2
	devid    7 size 3.64TiB used 3.64TiB path /dev/sdm2
	devid    8 size 3.64TiB used 3.64TiB path /dev/sdw2
	devid    9 size 3.64TiB used 3.64TiB path /dev/sdj2
	devid   10 size 3.64TiB used 3.64TiB path /dev/sdt2
	devid   11 size 3.64TiB used 3.64TiB path /dev/sdk2
	devid   12 size 3.64TiB used 3.64TiB path /dev/sdq2
	devid   13 size 3.64TiB used 3.64TiB path /dev/sds2
	devid   14 size 3.64TiB used 3.64TiB path /dev/sdf2
	devid   15 size 7.28TiB used 588.80GiB path /dev/sdr2
	devid   16 size 7.28TiB used 588.80GiB path /dev/sdo2
	devid   17 size 7.28TiB used 588.80GiB path /dev/sdv2
	devid   18 size 7.28TiB used 588.80GiB path /dev/sdi2
	devid   19 size 7.28TiB used 588.80GiB path /dev/sdl2
	devid   20 size 7.28TiB used 588.80GiB path /dev/sde2

[root@cornelis ~]# mount /dev/sdn2 /mnt/data
mount: /mnt/data: wrong fs type, bad option, bad superblock on
/dev/sdn2, missing codepage or helper program, or other error.

[root@cornelis ~]# btrfs check /dev/sdn2
Opening filesystem to check...
parent transid verify failed on 46451963543552 wanted 114401 found
114173
parent transid verify failed on 46451963543552 wanted 114401 found
114173
checksum verify failed on 46451963543552 found A8F2A769 wanted 4C111ADF
checksum verify failed on 46451963543552 found 32153BE8 wanted 8B07ABE4
checksum verify failed on 46451963543552 found 32153BE8 wanted 8B07ABE4
bad tree block 46451963543552, bytenr mismatch, want=46451963543552,
have=75208089814272
Couldn't read tree root
ERROR: cannot open file system

[root@cornelis ~]# btrfs restore /dev/sdn2 /mnt/data/
parent transid verify failed on 46451963543552 wanted 114401 found
114173
parent transid verify failed on 46451963543552 wanted 114401 found
114173
checksum verify failed on 46451963543552 found A8F2A769 wanted 4C111ADF
checksum verify failed on 46451963543552 found 32153BE8 wanted 8B07ABE4
checksum verify failed on 46451963543552 found 32153BE8 wanted 8B07ABE4
bad tree block 46451963543552, bytenr mismatch, want=46451963543552,
have=75208089814272
Couldn't read tree root
Could not open root, trying backup super
warning, device 14 is missing
warning, device 13 is missing
warning, device 12 is missing
warning, device 11 is missing
warning, device 10 is missing
warning, device 9 is missing
warning, device 8 is missing
warning, device 7 is missing
warning, device 6 is missing
warning, device 5 is missing
warning, device 4 is missing
warning, device 3 is missing
warning, device 2 is missing
checksum verify failed on 22085632 found 5630EA32 wanted 1AA6FFF0
checksum verify failed on 22085632 found 5630EA32 wanted 1AA6FFF0
bad tree block 22085632, bytenr mismatch, want=22085632,
have=1147797504
ERROR: cannot read chunk root
Could not open root, trying backup super
warning, device 14 is missing
warning, device 13 is missing
warning, device 12 is missing
warning, device 11 is missing
warning, device 10 is missing
warning, device 9 is missing
warning, device 8 is missing
warning, device 7 is missing
warning, device 6 is missing
warning, device 5 is missing
warning, device 4 is missing
warning, device 3 is missing
warning, device 2 is missing
checksum verify failed on 22085632 found 5630EA32 wanted 1AA6FFF0
checksum verify failed on 22085632 found 5630EA32 wanted 1AA6FFF0
bad tree block 22085632, bytenr mismatch, want=22085632,
have=1147797504
ERROR: cannot read chunk root
Could not open root, trying backup super

[root@cornelis ~]# uname -r
4.18.16-arch1-1-ARCH

[root@cornelis ~]# btrfs --version
btrfs-progs v4.19

-- 
Cheers,
Patrick




^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2018-12-09  9:32 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-30 13:53 Need help with potential ~45TB dataloss Patrick Dijkgraaf
2018-11-30 23:57 ` Qu Wenruo
2018-12-02  9:03   ` Patrick Dijkgraaf
2018-12-02 20:14     ` Patrick Dijkgraaf
2018-12-02 20:30       ` Andrei Borzenkov
2018-12-03  5:58         ` Qu Wenruo
2018-12-04  3:16           ` Chris Murphy
2018-12-04 10:09             ` Patrick Dijkgraaf
2018-12-04 19:38               ` Chris Murphy
2018-12-09  9:28                 ` Patrick Dijkgraaf
2018-12-03  0:35     ` Qu Wenruo
2018-12-03  0:45       ` Qu Wenruo
2018-12-04  9:58       ` Patrick Dijkgraaf
2018-12-09  9:32         ` Patrick Dijkgraaf

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).