btrfs crash in raid1 setup

btrfs crash in raid1 setup

[pierre-alexis]

Hi,

I'm having a massive crash of my BTRFS volume.

The server is a HP ProLiant DL380G6 running CentOS on an xfs volume. It 
is attached to a HP StorageWorks JBOD device which contains 8 drives, 
each formatted as single-volume raid0 devices visible by the OS as 
`/dev/sd{b-i}`. `/dev/sdf` is not formatted for BTRFS, otherwise all 7 
other devices are formatted as full-drive BTRFS (no partitions). The 
BTRFS volume is a raid1 volume, keeping my data and metadata on at least 
2 different drives. At 4:30am on Sep 7, a backup from a remote system 
came in, finished at 04:34:40, and at 04:35:03 the local system crashed 
with the following log. After a lot of operations described below, I can 
mount the filesystem but it does not show any folders anymore except a 
'lost+found' folder of 1GB. The BTRFS fi show still shows the (correct) 
use of 4+ TB.

kernel error log:
https://pastebin.com/7T58UwGn

The local system is used to backup a remote system (~20GB) but also to 
store a lot (4TB) of data (photo and video, projects, ...). This lot of 
data is not backed up anywhere else, it was the plan but the crash 
happened before i could buy the hardware for a second fallback setup. It 
has run nicely for a year, with a few fixing scrubs and a replaced 
hardware faulty drive (`/dev/sdf` -> `/dev/sdi`). I noticed the crash 
yesterday (2019/09/17) and issued the following commands, I truncated 
the output of thousands of similar lines.

actions taken:
https://pastebin.com/u7H5R9fS

Any help how to recover as much data as possible would be welcome.
Thank you.
Pierre-Alexis Ciavaldini