* All Three Superblocks Damaged After Kernel Panic
@ 2021-10-11 14:13 Kyle James Chapman
2021-10-11 18:23 ` Matt Corallo
2021-10-11 23:22 ` Qu Wenruo
0 siblings, 2 replies; 7+ messages in thread
From: Kyle James Chapman @ 2021-10-11 14:13 UTC (permalink / raw)
To: linux-btrfs
Hello,
I lost access to my home BTRFS filesystem on a 4TB SATA drive without partitioning today.
I am a graduate student of translation studies at a foreign university. Programming is not my specialty but have ran Linux for over a decade because I support open source endeavors. I say this for background on my technical capabilities, I am also a radio amateur extra-class in the United States. I have some technical competence and run Arch Linux for general duty.
The system boots and reads the kernel into memory from an optical ROM drive, reads keyfiles from a USB stick, decrypts the home and swap partitions on different devices, asks for a password for the root partition, and finishes booting. I was rebooting my system from userspace and observed a kernel panic after finishing some work repartitioning a small USB stick using gparted. After rebooting, my home BTRFS system was not mountable. Some information seems present in some of the superblocks, but I do not understand everything. I was careful in my use of the disk tool, and it was clear that only the USB stick was being accessed, because of the simple work I was doing, resizing a partition on the obvious drive. I would like to recover the data. I finished writing my master's thesis and have already submitted it, so no real work was lost, but I would like to understand why this is happening and how the data can be recovered. A memory fault seems plausible, but the fact that I was accessin
g the partitions (through gparted) seems to indicate other possibilities. Please do not get sidetracked into thinking I wrote data on the drive through carelessness. I have made many many uses of disk tools throughout my life and the simple nature of the operations which did not involve the filesystem in question are completely understandable and I remember them clearly.
Please advise.
Relevant /dev/fstab entry which has worked for several months until I commented it out today:
#/dev/mapper/home /home btrfs rw,compress,autodefrag,inode_cache 0 1
cat /proc/version:
Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4 (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000
sudo smartctl -a /dev/sdd excerpt:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 2.5 5400
Device Model: ST4000LM024-2AN17V
Serial Number: WCK03H8Y
LU WWN Device Id: 5 000c50 09e55827b
Firmware Version: 0001
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5526 rpm
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Oct 11 16:07:49 2021 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
btrfs --version:
btrfs-progs v5.14.1
sudo btrfs inspect-internal dump-super -s 0 /dev/mapper/home :
superblock: bytenr=65536, device=/dev/mapper/home
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xc59083ff [DON'T MATCH]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid bd5872ad-8fee-4d90-b048-b81120ce3254
metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254
label
generation 161054
root 4028020703232
sys_array_size 129
chunk_root_generation 154912
root_level 1
chunk_root 22020096
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 4000787030016
bytes_used 3969357348864
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 161054
uuid_tree_generation 161054
dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
dev_item.type 0
dev_item.total_bytes 4000787030016
dev_item.bytes_used 3997564731392
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sudo btrfs inspect-internal dump-super -s 1 /dev/mapper/home :
superblock: bytenr=67108864, device=/dev/mapper/home
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x65f1ab31 [DON'T MATCH]
bytenr 14039944498490823899
flags 0xcdc898509422934d
( WRITTEN |
unknown flag: 0xcdc898509422934c )
magic _BHRfS_M [match]
fsid 4468b2e4-d249-639c-dc54-792caf170cc9
metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
label
generation 161054
root 4028020703232
sys_array_size 129
chunk_root_generation 154912
root_level 1
chunk_root 22020096
chunk_root_level 1
log_root 11922660382425005768
log_root_transid 14582268926853107316
log_root_level 0
total_bytes 10532150587539085458
bytes_used 8124524553913841719
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 161054
uuid_tree_generation 161054
dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH]
dev_item.type 0
dev_item.total_bytes 4000787030016
dev_item.bytes_used 3997564731392
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sudo btrfs inspect-internal dump-super -s 2 /dev/mapper/home:
superblock: bytenr=274877906944, device=/dev/mapper/home
---------------------------------------------------------
csum_type 62267 (INVALID)
csum_size 32
csum 0x9876fd0000000000000000000000000000000000000000000000000000000000 [UNKNOWN CSUM TYPE OR SIZE]
bytenr 274877906944
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid bd5872ad-8fee-4d90-b048-b81120ce3254
metadata_uuid 07eb556b-df56-afbd-12c1-32f496c9ae5e
label ...^k...cA..[....ws.......!...&+..@...U.^..Fi.....^s....%.....G.....\..z.0..8N......l
generation 161054
root 4028020703232
sys_array_size 1346087890
chunk_root_generation 14973196025592430218
root_level 113
chunk_root 22020096
chunk_root_level 54
log_root 0
log_root_transid 0
log_root_level 77
total_bytes 4000787030016
bytes_used 3969357348864
sectorsize 544999736
nodesize 1626255393
leafsize (deprecated) 365786189
stripesize 1625894637
root_dir 4621774357935484814
num_devices 6281988177397337675
compat_flags 0x9c8dbbf37bf3e638
compat_ro_flags 0xe392e94b904707de
( FREE_SPACE_TREE_VALID |
unknown flag: 0xe392e94b904707dc )
incompat_flags 0xe7c3113fe08815af
( MIXED_BACKREF |
DEFAULT_SUBVOL |
MIXED_GROUPS |
COMPRESS_LZO |
BIG_METADATA |
RAID56 |
SKINNY_METADATA |
METADATA_UUID |
ZONED |
unknown flag: 0xe7c3113fe0880000 )
cache_generation 2740636164699663627
uuid_tree_generation 143387309824099247
dev_item.uuid 8ca1383a-3aaf-986e-30f3-1081cc4423b9
dev_item.fsid d3b80203-160e-b664-1fa0-0121596e5fbf [DON'T MATCH]
dev_item.type 2243501307224589742
dev_item.total_bytes 17886909296177165879
dev_item.bytes_used 843442598421986990
dev_item.io_align 3037217492
dev_item.io_width 3952451380
dev_item.sector_size 2707671125
dev_item.devid 4105250638360812501
dev_item.dev_group 3766056873
dev_item.seek_speed 182
dev_item.bandwidth 229
dev_item.generation 9452573969509059034
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: All Three Superblocks Damaged After Kernel Panic
2021-10-11 14:13 All Three Superblocks Damaged After Kernel Panic Kyle James Chapman
@ 2021-10-11 18:23 ` Matt Corallo
2021-10-11 23:22 ` Qu Wenruo
1 sibling, 0 replies; 7+ messages in thread
From: Matt Corallo @ 2021-10-11 18:23 UTC (permalink / raw)
To: Kyle James Chapman; +Cc: linux-btrfs
List subscribers,
Kyle’s email, below, may not have made it to your inbox. VGER’s SMTP RFC violation here should have resulted in your mail server refusing to accept this.
Kyle,
Please note that your email client, your upstream mail server, and vger.kernel.org are all miscofigured, resulting in your email violating the SMTP RFCs upon delivery. Because individual lines are not broken and longer than 1000 characters, many folks likely didn’t receive your email (eg default exim configurations will refuse your mail). You should reach out to the administrators for sogo01.urz.uni-heidelberg.de and relay.uni-heidelberg.de and have them fix their configuration, as well as fix your local MUA.
Matt
> On Oct 11, 2021, at 11:09, Kyle James Chapman <kyle.chapman@stud.uni-heidelberg.de> wrote:
>
> Hello,
>
> I lost access to my home BTRFS filesystem on a 4TB SATA drive without partitioning today.
>
> I am a graduate student of translation studies at a foreign university. Programming is not my specialty but have ran Linux for over a decade because I support open source endeavors. I say this for background on my technical capabilities, I am also a radio amateur extra-class in the United States. I have some technical competence and run Arch Linux for general duty.
>
> The system boots and reads the kernel into memory from an optical ROM drive, reads keyfiles from a USB stick, decrypts the home and swap partitions on different devices, asks for a password for the root partition, and finishes booting. I was rebooting my system from userspace and observed a kernel panic after finishing some work repartitioning a small USB stick using gparted. After rebooting, my home BTRFS system was not mountable. Some information seems present in some of the superblocks, but I do not understand everything. I was careful in my use of the disk tool, and it was clear that only the USB stick was being accessed, because of the simple work I was doing, resizing a partition on the obvious drive. I would like to recover the data. I finished writing my master's thesis and have already submitted it, so no real work was lost, but I would like to understand why this is happening and how the data can be recovered. A memory fault seems plausible, but the fact that I was accessing the partitions (through gparted) seems to indicate other possibilities. Please do not get sidetracked into thinking I wrote data on the drive through carelessness. I have made many many uses of disk tools throughout my life and the simple nature of the operations which did not involve the filesystem in question are completely understandable and I remember them clearly.
>
> Please advise.
>
> Relevant /dev/fstab entry which has worked for several months until I commented it out today:
>
> #/dev/mapper/home /home btrfs rw,compress,autodefrag,inode_cache 0 1
>
> cat /proc/version:
>
> Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4 (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000
>
>
> sudo smartctl -a /dev/sdd excerpt:
>
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda 2.5 5400
> Device Model: ST4000LM024-2AN17V
> Serial Number: WCK03H8Y
> LU WWN Device Id: 5 000c50 09e55827b
> Firmware Version: 0001
> User Capacity: 4,000,787,030,016 bytes [4.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5526 rpm
> Form Factor: 2.5 inches
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ACS-3 T13/2161-D revision 5
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Oct 11 16:07:49 2021 CEST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> btrfs --version:
>
> btrfs-progs v5.14.1
>
>
> sudo btrfs inspect-internal dump-super -s 0 /dev/mapper/home :
>
> superblock: bytenr=65536, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0xc59083ff [DON'T MATCH]
> bytenr 65536
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
> metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254
> label
> generation 161054
> root 4028020703232
> sys_array_size 129
> chunk_root_generation 154912
> root_level 1
> chunk_root 22020096
> chunk_root_level 1
> log_root 0
> log_root_transid 0
> log_root_level 0
> total_bytes 4000787030016
> bytes_used 3969357348864
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 1
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation 161054
> uuid_tree_generation 161054
> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
> dev_item.type 0
> dev_item.total_bytes 4000787030016
> dev_item.bytes_used 3997564731392
> dev_item.io_align 4096
> dev_item.io_width 4096
> dev_item.sector_size 4096
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
>
> sudo btrfs inspect-internal dump-super -s 1 /dev/mapper/home :
>
> superblock: bytenr=67108864, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0x65f1ab31 [DON'T MATCH]
> bytenr 14039944498490823899
> flags 0xcdc898509422934d
> ( WRITTEN |
> unknown flag: 0xcdc898509422934c )
> magic _BHRfS_M [match]
> fsid 4468b2e4-d249-639c-dc54-792caf170cc9
> metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
> label
> generation 161054
> root 4028020703232
> sys_array_size 129
> chunk_root_generation 154912
> root_level 1
> chunk_root 22020096
> chunk_root_level 1
> log_root 11922660382425005768
> log_root_transid 14582268926853107316
> log_root_level 0
> total_bytes 10532150587539085458
> bytes_used 8124524553913841719
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 1
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation 161054
> uuid_tree_generation 161054
> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH]
> dev_item.type 0
> dev_item.total_bytes 4000787030016
> dev_item.bytes_used 3997564731392
> dev_item.io_align 4096
> dev_item.io_width 4096
> dev_item.sector_size 4096
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
>
> sudo btrfs inspect-internal dump-super -s 2 /dev/mapper/home:
>
> superblock: bytenr=274877906944, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 62267 (INVALID)
> csum_size 32
> csum 0x9876fd0000000000000000000000000000000000000000000000000000000000 [UNKNOWN CSUM TYPE OR SIZE]
> bytenr 274877906944
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
> metadata_uuid 07eb556b-df56-afbd-12c1-32f496c9ae5e
> label ...^k...cA..[....ws.......!...&+..@...U.^..Fi.....^s....%.....G.....\..z.0..8N......l
> generation 161054
> root 4028020703232
> sys_array_size 1346087890
> chunk_root_generation 14973196025592430218
> root_level 113
> chunk_root 22020096
> chunk_root_level 54
> log_root 0
> log_root_transid 0
> log_root_level 77
> total_bytes 4000787030016
> bytes_used 3969357348864
> sectorsize 544999736
> nodesize 1626255393
> leafsize (deprecated) 365786189
> stripesize 1625894637
> root_dir 4621774357935484814
> num_devices 6281988177397337675
> compat_flags 0x9c8dbbf37bf3e638
> compat_ro_flags 0xe392e94b904707de
> ( FREE_SPACE_TREE_VALID |
> unknown flag: 0xe392e94b904707dc )
> incompat_flags 0xe7c3113fe08815af
> ( MIXED_BACKREF |
> DEFAULT_SUBVOL |
> MIXED_GROUPS |
> COMPRESS_LZO |
> BIG_METADATA |
> RAID56 |
> SKINNY_METADATA |
> METADATA_UUID |
> ZONED |
> unknown flag: 0xe7c3113fe0880000 )
> cache_generation 2740636164699663627
> uuid_tree_generation 143387309824099247
> dev_item.uuid 8ca1383a-3aaf-986e-30f3-1081cc4423b9
> dev_item.fsid d3b80203-160e-b664-1fa0-0121596e5fbf [DON'T MATCH]
> dev_item.type 2243501307224589742
> dev_item.total_bytes 17886909296177165879
> dev_item.bytes_used 843442598421986990
> dev_item.io_align 3037217492
> dev_item.io_width 3952451380
> dev_item.sector_size 2707671125
> dev_item.devid 4105250638360812501
> dev_item.dev_group 3766056873
> dev_item.seek_speed 182
> dev_item.bandwidth 229
> dev_item.generation 9452573969509059034
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: All Three Superblocks Damaged After Kernel Panic
2021-10-11 14:13 All Three Superblocks Damaged After Kernel Panic Kyle James Chapman
2021-10-11 18:23 ` Matt Corallo
@ 2021-10-11 23:22 ` Qu Wenruo
2021-10-12 7:41 ` K Chapman
1 sibling, 1 reply; 7+ messages in thread
From: Qu Wenruo @ 2021-10-11 23:22 UTC (permalink / raw)
To: Kyle James Chapman, linux-btrfs
On 2021/10/11 22:13, Kyle James Chapman wrote:
> Hello,
>
> I lost access to my home BTRFS filesystem on a 4TB SATA drive without partitioning today.
>
> I am a graduate student of translation studies at a foreign university. Programming is not my specialty but have ran Linux for over a decade because I support open source endeavors. I say this for background on my technical capabilities, I am also a radio amateur extra-class in the United States. I have some technical competence and run Arch Linux for general duty.
>
> The system boots and reads the kernel into memory from an optical ROM drive, reads keyfiles from a USB stick, decrypts the home and swap partitions on different devices, asks for a password for the root partition, and finishes booting. I was rebooting my system from userspace and observed a kernel panic after finishing some work repartitioning a small USB stick using gparted.
Any idea on what part is causing the panic, or any log/screenshot of the
panic message?
> After rebooting, my home BTRFS system was not mountable. Some information seems present in some of the superblocks, but I do not understand everything. I was careful in my use of the disk tool, and it was clear that only the USB stick was being accessed, because of the simple work I was doing, resizing a partition on the obvious drive. I would like to recover the data. I finished writing my master's thesis and have already submitted it, so no real work was lost, but I would like to understand why this is happening and how the data can be recovered. A memory fault seems plausible, but the fact that I was accessing the partitions (through gparted) seems to indicate other possibilities. Please do not get sidetracked into thinking I wrote data on the drive through carelessness. I have made many many uses of disk tools throughout my life and the simple nature of the operations which did not involve the filesystem in question are completely understandable and I remember them clearly.
>
> Please advise.
>
> Relevant /dev/fstab entry which has worked for several months until I commented it out today:
>
> #/dev/mapper/home /home btrfs rw,compress,autodefrag,inode_cache 0 1
Inode cache is already deprecated, but it won't do anything to the
kernel, thus it shouldn't cause any problem.
>
> cat /proc/version:
>
> Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4 (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000
>
>
> sudo smartctl -a /dev/sdd excerpt:
>
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda 2.5 5400
> Device Model: ST4000LM024-2AN17V
> Serial Number: WCK03H8Y
> LU WWN Device Id: 5 000c50 09e55827b
> Firmware Version: 0001
> User Capacity: 4,000,787,030,016 bytes [4.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5526 rpm
> Form Factor: 2.5 inches
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ACS-3 T13/2161-D revision 5
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Oct 11 16:07:49 2021 CEST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> btrfs --version:
>
> btrfs-progs v5.14.1
>
>
> sudo btrfs inspect-internal dump-super -s 0 /dev/mapper/home :
>
> superblock: bytenr=65536, device=/dev/mapper/home
The first copy in fact looks pretty good.
If there is something wrong, then it would be in the backup roots or sys
chunk map.
Mind to dump the full super block with "btrfs ins dump-super -Ffa"?
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0xc59083ff [DON'T MATCH]
> bytenr 65536
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
> metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254
> label
> generation 161054
> root 4028020703232
> sys_array_size 129
> chunk_root_generation 154912
> root_level 1
> chunk_root 22020096
> chunk_root_level 1
> log_root 0
> log_root_transid 0
> log_root_level 0
> total_bytes 4000787030016
> bytes_used 3969357348864
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 1
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation 161054
> uuid_tree_generation 161054
> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
> dev_item.type 0
> dev_item.total_bytes 4000787030016
> dev_item.bytes_used 3997564731392
> dev_item.io_align 4096
> dev_item.io_width 4096
> dev_item.sector_size 4096
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
>
> sudo btrfs inspect-internal dump-super -s 1 /dev/mapper/home :
>
> superblock: bytenr=67108864, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0x65f1ab31 [DON'T MATCH]
> bytenr 14039944498490823899
> flags 0xcdc898509422934d
> ( WRITTEN |
> unknown flag: 0xcdc898509422934c )
The 2nd one begins to have some garbage, but not yet full of garbage.
This means there are some writes into the 2nd super block, and mostly
the writes are just into part of the super block.
> magic _BHRfS_M [match]
> fsid 4468b2e4-d249-639c-dc54-792caf170cc9
> metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
The UUID is damaged.
> label
> generation 161054
> root 4028020703232 > sys_array_size 129
> chunk_root_generation 154912
> root_level 1
> chunk_root 22020096
> chunk_root_level 1
But then some good data.
> log_root 11922660382425005768
> log_root_transid 14582268926853107316
> log_root_level 0
> total_bytes 10532150587539085458
> bytes_used 8124524553913841719
Some garbage again.
I don't think it can be even caused by simple memory corruption, too
many random part gets corrupted.
It's really hard to say if btrfs itself is involved, until we see the
calltrace.
For the recovery, the superblock needs to be manually fixed to allow any
further data salvage attempts.
And that can only be done with the extra "btrfs ins dump-super -Ffa" output.
Thanks,
Qu
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 1
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation 161054
> uuid_tree_generation 161054
> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH]
> dev_item.type 0
> dev_item.total_bytes 4000787030016
> dev_item.bytes_used 3997564731392
> dev_item.io_align 4096
> dev_item.io_width 4096
> dev_item.sector_size 4096
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
>
> sudo btrfs inspect-internal dump-super -s 2 /dev/mapper/home:
>
> superblock: bytenr=274877906944, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 62267 (INVALID)
> csum_size 32
> csum 0x9876fd0000000000000000000000000000000000000000000000000000000000 [UNKNOWN CSUM TYPE OR SIZE]
> bytenr 274877906944
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
> metadata_uuid 07eb556b-df56-afbd-12c1-32f496c9ae5e
> label ...^k...cA..[....ws.......!...&+..@...U.^..Fi.....^s....%.....G.....\..z.0..8N......l
> generation 161054
> root 4028020703232
> sys_array_size 1346087890
> chunk_root_generation 14973196025592430218
> root_level 113
> chunk_root 22020096
> chunk_root_level 54
> log_root 0
> log_root_transid 0
> log_root_level 77
> total_bytes 4000787030016
> bytes_used 3969357348864
> sectorsize 544999736
> nodesize 1626255393
> leafsize (deprecated) 365786189
> stripesize 1625894637
> root_dir 4621774357935484814
> num_devices 6281988177397337675
> compat_flags 0x9c8dbbf37bf3e638
> compat_ro_flags 0xe392e94b904707de
> ( FREE_SPACE_TREE_VALID |
> unknown flag: 0xe392e94b904707dc )
> incompat_flags 0xe7c3113fe08815af
> ( MIXED_BACKREF |
> DEFAULT_SUBVOL |
> MIXED_GROUPS |
> COMPRESS_LZO |
> BIG_METADATA |
> RAID56 |
> SKINNY_METADATA |
> METADATA_UUID |
> ZONED |
> unknown flag: 0xe7c3113fe0880000 )
> cache_generation 2740636164699663627
> uuid_tree_generation 143387309824099247
> dev_item.uuid 8ca1383a-3aaf-986e-30f3-1081cc4423b9
> dev_item.fsid d3b80203-160e-b664-1fa0-0121596e5fbf [DON'T MATCH]
> dev_item.type 2243501307224589742
> dev_item.total_bytes 17886909296177165879
> dev_item.bytes_used 843442598421986990
> dev_item.io_align 3037217492
> dev_item.io_width 3952451380
> dev_item.sector_size 2707671125
> dev_item.devid 4105250638360812501
> dev_item.dev_group 3766056873
> dev_item.seek_speed 182
> dev_item.bandwidth 229
> dev_item.generation 9452573969509059034
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: All Three Superblocks Damaged After Kernel Panic
2021-10-11 23:22 ` Qu Wenruo
@ 2021-10-12 7:41 ` K Chapman
2021-10-12 10:29 ` Qu Wenruo
0 siblings, 1 reply; 7+ messages in thread
From: K Chapman @ 2021-10-12 7:41 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
Thank you so much for the help! I have images of my projects and dog
which passed away among other things I would like to save. Make a backup
is fine to say, but I am a student and have not had much luck earning to
buy a new hdd in this foreign country.
Re:
Any idea on what part is causing the panic, or any log/screenshot of the
panic message?
This remains unclear. I did not photograph the kernel panic and I have
no log of the error. The two actions which I performed at the time was
the execution of the reboot command and unplugging the usb stick. I then
looked at the screen and saw a panic message and then hit the power
switch on the power supply and reset the machine.
Re: Mind to dump the full super block with "btrfs ins dump-super -Ffa"?
sudo btrfs ins dump-super -Ffa /dev/mapper/home :
superblock: bytenr=65536, device=/dev/mapper/home
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xc59083ff [DON'T MATCH]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid bd5872ad-8fee-4d90-b048-b81120ce3254
metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254
label
generation 161054
root 4028020703232
sys_array_size 129
chunk_root_generation 154912
root_level 1
chunk_root 22020096
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 4000787030016
bytes_used 3969357348864
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 161054
uuid_tree_generation 161054
dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
dev_item.type 0
dev_item.total_bytes 4000787030016
dev_item.bytes_used 3997564731392
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 22020096
dev_uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
stripe 1 devid 1 offset 30408704
dev_uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
backup_roots[4]:
backup 0:
backup_tree_root: 4028018622464 gen: 161053 level: 1
backup_chunk_root: 22020096 gen: 154912 level: 1
backup_extent_root: 4028018229248 gen: 161053 level: 2
backup_fs_root: 4028019884032 gen: 161054 level: 3
backup_dev_root: 4027637743616 gen: 161049 level: 1
backup_csum_root: 4028020047872 gen: 161054 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969357348864
backup_num_devices: 1
backup 1:
backup_tree_root: 4028020703232 gen: 161054 level: 1
backup_chunk_root: 22020096 gen: 154912 level: 1
backup_extent_root: 4028020277248 gen: 161054 level: 2
backup_fs_root: 4028021620736 gen: 161054 level: 3
backup_dev_root: 4027637743616 gen: 161049 level: 1
backup_csum_root: 4028020326400 gen: 161054 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969357348864
backup_num_devices: 1
backup 2:
backup_tree_root: 4027794194432 gen: 161051 level: 1
backup_chunk_root: 22020096 gen: 154912 level: 1
backup_extent_root: 4027793915904 gen: 161051 level: 2
backup_fs_root: 4027794866176 gen: 161051 level: 3
backup_dev_root: 4027637743616 gen: 161049 level: 1
backup_csum_root: 4027793981440 gen: 161051 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969416499200
backup_num_devices: 1
backup 3:
backup_tree_root: 4028014985216 gen: 161052 level: 1
backup_chunk_root: 22020096 gen: 154912 level: 1
backup_extent_root: 4027953004544 gen: 161052 level: 2
backup_fs_root: 4027876769792 gen: 161052 level: 3
backup_dev_root: 4027637743616 gen: 161049 level: 1
backup_csum_root: 4027969748992 gen: 161052 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969357332480
backup_num_devices: 1
superblock: bytenr=67108864, device=/dev/mapper/home
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x65f1ab31 [DON'T MATCH]
bytenr 14039944498490823899
flags 0xcdc898509422934d
( WRITTEN |
unknown flag: 0xcdc898509422934c )
magic _BHRfS_M [match]
fsid 4468b2e4-d249-639c-dc54-792caf170cc9
metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
label
generation 161054
root 4028020703232
sys_array_size 129
chunk_root_generation 154912
root_level 1
chunk_root 22020096
chunk_root_level 1
log_root 11922660382425005768
log_root_transid 14582268926853107316
log_root_level 0
total_bytes 10532150587539085458
bytes_used 8124524553913841719
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 161054
uuid_tree_generation 161054
dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH]
dev_item.type 0
dev_item.total_bytes 4000787030016
dev_item.bytes_used 3997564731392
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (288472139764826295 UNKNOWN.227 16728631262381099805)
ERROR: unexpected item type 227 in sys_array at offset 17
backup_roots[4]:
backup 0:
backup_tree_root: 2937111841462972043 gen:
6902395341942098838 level: 1
backup_chunk_root: 17657898596786216606 gen:
1348021672883110445 level: 1
backup_extent_root: 11170562329367461854 gen:
11756089894903112279 level: 2
backup_fs_root: 10659646040436297474 gen:
2903212909590398647 level: 3
backup_dev_root: 14579526832388012464 gen:
4376531062597251514 level: 1
backup_csum_root: 4242763241112 gen: 161054 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969357348864
backup_num_devices: 1
backup 1:
backup_tree_root: 4028020703232 gen: 161054 level: 198
backup_chunk_root: 22020096 gen: 154912 level: 243
backup_extent_root: 4028020277248 gen:
16089197629711480094 level: 206
backup_fs_root: 5604276798285345924 gen:
10182153286217385848 level: 58
backup_dev_root: 12659493148880815901 gen:
9869374175363434348 level: 78
backup_csum_root: 8949053250522901049 gen:
5400723786290947185 level: 164
backup_total_bytes: 15031814655203781796
backup_bytes_used: 1855277020722950832
backup_num_devices: 13548686937124434823
backup 2:
backup_tree_root: 4043263646859 gen: 161051 level: 1
backup_chunk_root: 22020096 gen: 154912 level: 1
backup_extent_root: 4027793915904 gen: 161051 level: 2
backup_fs_root: 4027794866176 gen: 161051 level: 3
backup_dev_root: 4027637743616 gen: 161049 level: 1
backup_csum_root: 4027793981440 gen: 161051 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969416499200
backup_num_devices: 1
backup 3:
backup_tree_root: 4028014985216 gen: 161052 level: 78
backup_chunk_root: 22020096 gen: 154912 level: 70
backup_extent_root: 4027953004544 gen: 161052 level: 16
backup_fs_root: 4027876769792 gen: 161052 level: 11
backup_dev_root: 4027637743616 gen: 161049 level: 125
backup_csum_root: 4027969748992 gen:
15626176790580589852 level: 222
backup_total_bytes: 18338658601355635447
backup_bytes_used: 12906905608640099773
backup_num_devices: 11951688830094275180
superblock: bytenr=274877906944, device=/dev/mapper/home
---------------------------------------------------------
csum_type 62267 (INVALID)
csum_size 32
csum 0x9876fd0000000000000000000000000000000000000000000000000000000000
[UNKNOWN CSUM TYPE OR SIZE]
bytenr 274877906944
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid bd5872ad-8fee-4d90-b048-b81120ce3254
metadata_uuid 07eb556b-df56-afbd-12c1-32f496c9ae5e
label
...^k...cA..[....ws.......!...&+..@...U.^..Fi.....^s....%.....G.....\..z.0..8N......l
generation 161054
root 4028020703232
sys_array_size 1346087890
chunk_root_generation 14973196025592430218
root_level 113
chunk_root 22020096
chunk_root_level 54
log_root 0
log_root_transid 0
log_root_level 77
total_bytes 4000787030016
bytes_used 3969357348864
sectorsize 544999736
nodesize 1626255393
leafsize (deprecated) 365786189
stripesize 1625894637
root_dir 4621774357935484814
num_devices 6281988177397337675
compat_flags 0x9c8dbbf37bf3e638
compat_ro_flags 0xe392e94b904707de
( FREE_SPACE_TREE_VALID |
unknown flag: 0xe392e94b904707dc )
incompat_flags 0xe7c3113fe08815af
( MIXED_BACKREF |
DEFAULT_SUBVOL |
MIXED_GROUPS |
COMPRESS_LZO |
BIG_METADATA |
RAID56 |
SKINNY_METADATA |
METADATA_UUID |
ZONED |
unknown flag: 0xe7c3113fe0880000 )
cache_generation 2740636164699663627
uuid_tree_generation 143387309824099247
dev_item.uuid 8ca1383a-3aaf-986e-30f3-1081cc4423b9
dev_item.fsid d3b80203-160e-b664-1fa0-0121596e5fbf [DON'T MATCH]
dev_item.type 2243501307224589742
dev_item.total_bytes 17886909296177165879
dev_item.bytes_used 843442598421986990
dev_item.io_align 3037217492
dev_item.io_width 3952451380
dev_item.sector_size 2707671125
dev_item.devid 4105250638360812501
dev_item.dev_group 3766056873
dev_item.seek_speed 182
dev_item.bandwidth 229
dev_item.generation 9452573969509059034
sys_chunk_array[2048]:
ERROR: sys_array_size 1346087890 shouldn't exceed 2048 bytes
backup_roots[4]:
backup 0:
backup_tree_root: 4028018622464 gen: 161053 level: 243
backup_chunk_root: 22020096 gen: 154912 level: 189
backup_extent_root: 4028018229248 gen: 161053 level: 184
backup_fs_root: 4028019884032 gen: 161054 level: 247
backup_dev_root: 4027637743616 gen: 161049 level: 241
backup_csum_root: 11051166811494301696 gen:
11918979514504630695 level: 43
backup_total_bytes: 6128247333213658072
backup_bytes_used: 13316083672891396332
backup_num_devices: 12986852272682621048
backup 1:
backup_tree_root: 11912830949670804138 gen:
4162465539222756018 level: 1
backup_chunk_root: 10938026073880195285 gen:
8406240668729036675 level: 1
backup_extent_root: 17210210806549355879 gen:
455024150221 level: 2
backup_fs_root: 4028021620736 gen: 161054 level: 3
backup_dev_root: 4027637743616 gen: 161049 level: 1
backup_csum_root: 4028020326400 gen: 161054 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969357348864
backup_num_devices: 1
backup 2:
backup_tree_root: 12811350786919235584 gen:
3323034891246254263 level: 217
backup_chunk_root: 17845957508331272086 gen:
12071669715000553588 level: 235
backup_extent_root: 2368736995830619850 gen:
14589654675806411696 level: 163
backup_fs_root: 7496041380651946087 gen:
16394240941193014637 level: 29
backup_dev_root: 11890518936409562630 gen:
11935615530850632810 level: 174
backup_csum_root: 4804084557218082292 gen:
3037656397741716549 level: 57
backup_total_bytes: 12628714618049392498
backup_bytes_used: 4517632456508041139
backup_num_devices: 11559351643267246189
backup 3:
backup_tree_root: 9502548224584054781 gen:
3258916775715815651 level: 1
backup_chunk_root: 2996837717780471142 gen:
9778119635099568422 level: 1
backup_extent_root: 2696351959151677890 gen:
5061715638962835499 level: 2
backup_fs_root: 16900363948952743119 gen:
2413589847950453178 level: 3
backup_dev_root: 18344731769370366813 gen:
11720350780674912754 level: 1
backup_csum_root: 6130366103217558429 gen:
598304231798 level: 3
backup_total_bytes: 4000787030016
backup_bytes_used: 3969357332480
backup_num_devices: 1
On 12.10.21 01:22, Qu Wenruo wrote:
>
>
> On 2021/10/11 22:13, Kyle James Chapman wrote:
>> Hello,
>>
>> I lost access to my home BTRFS filesystem on a 4TB SATA drive without
>> partitioning today.
>>
>> I am a graduate student of translation studies at a foreign
>> university. Programming is not my specialty but have ran Linux for
>> over a decade because I support open source endeavors. I say this for
>> background on my technical capabilities, I am also a radio amateur
>> extra-class in the United States. I have some technical competence
>> and run Arch Linux for general duty.
>>
>> The system boots and reads the kernel into memory from an optical ROM
>> drive, reads keyfiles from a USB stick, decrypts the home and swap
>> partitions on different devices, asks for a password for the root
>> partition, and finishes booting. I was rebooting my system from
>> userspace and observed a kernel panic after finishing some work
>> repartitioning a small USB stick using gparted.
>
> Any idea on what part is causing the panic, or any log/screenshot of the
> panic message?
>
>> After rebooting, my home BTRFS system was not mountable. Some
>> information seems present in some of the superblocks, but I do not
>> understand everything. I was careful in my use of the disk tool, and
>> it was clear that only the USB stick was being accessed, because of
>> the simple work I was doing, resizing a partition on the obvious
>> drive. I would like to recover the data. I finished writing my
>> master's thesis and have already submitted it, so no real work was
>> lost, but I would like to understand why this is happening and how
>> the data can be recovered. A memory fault seems plausible, but the
>> fact that I was accessing the partitions (through gparted) seems to
>> indicate other possibilities. Please do not get sidetracked into
>> thinking I wrote data on the drive through carelessness. I have made
>> many many uses of disk tools throughout my life and the simple nature
>> of the operations which did not involve the filesystem in question
>> are completely understandable and I remember them clearly.
>>
>> Please advise.
>>
>> Relevant /dev/fstab entry which has worked for several months until I
>> commented it out today:
>>
>> #/dev/mapper/home /home btrfs
>> rw,compress,autodefrag,inode_cache 0 1
>
> Inode cache is already deprecated, but it won't do anything to the
> kernel, thus it shouldn't cause any problem.
>
>>
>> cat /proc/version:
>>
>> Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4
>> (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils)
>> 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000
>>
>>
>> sudo smartctl -a /dev/sdd excerpt:
>>
>> === START OF INFORMATION SECTION ===
>> Model Family: Seagate Barracuda 2.5 5400
>> Device Model: ST4000LM024-2AN17V
>> Serial Number: WCK03H8Y
>> LU WWN Device Id: 5 000c50 09e55827b
>> Firmware Version: 0001
>> User Capacity: 4,000,787,030,016 bytes [4.00 TB]
>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>> Rotation Rate: 5526 rpm
>> Form Factor: 2.5 inches
>> Device is: In smartctl database [for details use: -P show]
>> ATA Version is: ACS-3 T13/2161-D revision 5
>> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
>> Local Time is: Mon Oct 11 16:07:49 2021 CEST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART overall-health self-assessment test result: PASSED
>>
>> btrfs --version:
>>
>> btrfs-progs v5.14.1
>>
>>
>> sudo btrfs inspect-internal dump-super -s 0 /dev/mapper/home :
>>
>> superblock: bytenr=65536, device=/dev/mapper/home
>
> The first copy in fact looks pretty good.
>
> If there is something wrong, then it would be in the backup roots or sys
> chunk map.
>
> Mind to dump the full super block with "btrfs ins dump-super -Ffa"?
>> ---------------------------------------------------------
>> csum_type 0 (crc32c)
>> csum_size 4
>> csum 0xc59083ff [DON'T MATCH]
>> bytenr 65536
>> flags 0x1
>> ( WRITTEN )
>> magic _BHRfS_M [match]
>> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
>> metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254
>> label
>> generation 161054
>> root 4028020703232
>> sys_array_size 129
>> chunk_root_generation 154912
>> root_level 1
>> chunk_root 22020096
>> chunk_root_level 1
>> log_root 0
>> log_root_transid 0
>> log_root_level 0
>> total_bytes 4000787030016
>> bytes_used 3969357348864
>> sectorsize 4096
>> nodesize 16384
>> leafsize (deprecated) 16384
>> stripesize 4096
>> root_dir 6
>> num_devices 1
>> compat_flags 0x0
>> compat_ro_flags 0x0
>> incompat_flags 0x161
>> ( MIXED_BACKREF |
>> BIG_METADATA |
>> EXTENDED_IREF |
>> SKINNY_METADATA )
>> cache_generation 161054
>> uuid_tree_generation 161054
>> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
>> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
>> dev_item.type 0
>> dev_item.total_bytes 4000787030016
>> dev_item.bytes_used 3997564731392
>> dev_item.io_align 4096
>> dev_item.io_width 4096
>> dev_item.sector_size 4096
>> dev_item.devid 1
>> dev_item.dev_group 0
>> dev_item.seek_speed 0
>> dev_item.bandwidth 0
>> dev_item.generation 0
>>
>> sudo btrfs inspect-internal dump-super -s 1 /dev/mapper/home :
>>
>> superblock: bytenr=67108864, device=/dev/mapper/home
>> ---------------------------------------------------------
>> csum_type 0 (crc32c)
>> csum_size 4
>> csum 0x65f1ab31 [DON'T MATCH]
>> bytenr 14039944498490823899
>> flags 0xcdc898509422934d
>> ( WRITTEN |
>> unknown flag: 0xcdc898509422934c )
>
> The 2nd one begins to have some garbage, but not yet full of garbage.
>
> This means there are some writes into the 2nd super block, and mostly
> the writes are just into part of the super block.
>
>> magic _BHRfS_M [match]
>> fsid 4468b2e4-d249-639c-dc54-792caf170cc9
>> metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
>
> The UUID is damaged.
>
>> label
>> generation 161054
>> root 4028020703232 > sys_array_size 129
>> chunk_root_generation 154912
>> root_level 1
>> chunk_root 22020096
>> chunk_root_level 1
>
> But then some good data.
>
>> log_root 11922660382425005768
>> log_root_transid 14582268926853107316
>> log_root_level 0
>> total_bytes 10532150587539085458
>> bytes_used 8124524553913841719
>
> Some garbage again.
>
> I don't think it can be even caused by simple memory corruption, too
> many random part gets corrupted.
>
> It's really hard to say if btrfs itself is involved, until we see the
> calltrace.
>
>
> For the recovery, the superblock needs to be manually fixed to allow any
> further data salvage attempts.
>
> And that can only be done with the extra "btrfs ins dump-super -Ffa"
> output.
>
> Thanks,
> Qu
>
>> sectorsize 4096
>> nodesize 16384
>> leafsize (deprecated) 16384
>> stripesize 4096
>> root_dir 6
>> num_devices 1
>> compat_flags 0x0
>> compat_ro_flags 0x0
>> incompat_flags 0x161
>> ( MIXED_BACKREF |
>> BIG_METADATA |
>> EXTENDED_IREF |
>> SKINNY_METADATA )
>> cache_generation 161054
>> uuid_tree_generation 161054
>> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
>> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH]
>> dev_item.type 0
>> dev_item.total_bytes 4000787030016
>> dev_item.bytes_used 3997564731392
>> dev_item.io_align 4096
>> dev_item.io_width 4096
>> dev_item.sector_size 4096
>> dev_item.devid 1
>> dev_item.dev_group 0
>> dev_item.seek_speed 0
>> dev_item.bandwidth 0
>> dev_item.generation 0
>>
>> sudo btrfs inspect-internal dump-super -s 2 /dev/mapper/home:
>>
>> superblock: bytenr=274877906944, device=/dev/mapper/home
>> ---------------------------------------------------------
>> csum_type 62267 (INVALID)
>> csum_size 32
>> csum
>> 0x9876fd0000000000000000000000000000000000000000000000000000000000
>> [UNKNOWN CSUM TYPE OR SIZE]
>> bytenr 274877906944
>> flags 0x1
>> ( WRITTEN )
>> magic _BHRfS_M [match]
>> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
>> metadata_uuid 07eb556b-df56-afbd-12c1-32f496c9ae5e
>> label
>> ...^k...cA..[....ws.......!...&+..@...U.^..Fi.....^s....%.....G.....\..z.0..8N......l
>> generation 161054
>> root 4028020703232
>> sys_array_size 1346087890
>> chunk_root_generation 14973196025592430218
>> root_level 113
>> chunk_root 22020096
>> chunk_root_level 54
>> log_root 0
>> log_root_transid 0
>> log_root_level 77
>> total_bytes 4000787030016
>> bytes_used 3969357348864
>> sectorsize 544999736
>> nodesize 1626255393
>> leafsize (deprecated) 365786189
>> stripesize 1625894637
>> root_dir 4621774357935484814
>> num_devices 6281988177397337675
>> compat_flags 0x9c8dbbf37bf3e638
>> compat_ro_flags 0xe392e94b904707de
>> ( FREE_SPACE_TREE_VALID |
>> unknown flag: 0xe392e94b904707dc )
>> incompat_flags 0xe7c3113fe08815af
>> ( MIXED_BACKREF |
>> DEFAULT_SUBVOL |
>> MIXED_GROUPS |
>> COMPRESS_LZO |
>> BIG_METADATA |
>> RAID56 |
>> SKINNY_METADATA |
>> METADATA_UUID |
>> ZONED |
>> unknown flag: 0xe7c3113fe0880000 )
>> cache_generation 2740636164699663627
>> uuid_tree_generation 143387309824099247
>> dev_item.uuid 8ca1383a-3aaf-986e-30f3-1081cc4423b9
>> dev_item.fsid d3b80203-160e-b664-1fa0-0121596e5fbf [DON'T MATCH]
>> dev_item.type 2243501307224589742
>> dev_item.total_bytes 17886909296177165879
>> dev_item.bytes_used 843442598421986990
>> dev_item.io_align 3037217492
>> dev_item.io_width 3952451380
>> dev_item.sector_size 2707671125
>> dev_item.devid 4105250638360812501
>> dev_item.dev_group 3766056873
>> dev_item.seek_speed 182
>> dev_item.bandwidth 229
>> dev_item.generation 9452573969509059034
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: All Three Superblocks Damaged After Kernel Panic
2021-10-12 7:41 ` K Chapman
@ 2021-10-12 10:29 ` Qu Wenruo
2021-10-12 10:35 ` K Chapman
0 siblings, 1 reply; 7+ messages in thread
From: Qu Wenruo @ 2021-10-12 10:29 UTC (permalink / raw)
To: K Chapman, linux-btrfs
On 2021/10/12 15:41, K Chapman wrote:
> Thank you so much for the help! I have images of my projects and dog
> which passed away among other things I would like to save. Make a backup
> is fine to say, but I am a student and have not had much luck earning to
> buy a new hdd in this foreign country.
>
> Re:
> Any idea on what part is causing the panic, or any log/screenshot of the
> panic message?
>
> This remains unclear. I did not photograph the kernel panic and I have
> no log of the error. The two actions which I performed at the time was
> the execution of the reboot command and unplugging the usb stick. I then
> looked at the screen and saw a panic message and then hit the power
> switch on the power supply and reset the machine.
Any clue would be helpful.
Like if it's a warning or BUG_ON() or whatever elese.
Another thing is, when you hit the power switch to force shutdown, does
the system already hang there for a while, or you see the panic message
and immediately try to force shutdown?
If the power is lost at exactly certain unlucky point, I guess the SSD
may have a chance to corrupt its data (which shouldn't be).
For now, what we really have is just whatever left on the disk, without
any clue how it's corrupted.
>
>
> Re: Mind to dump the full super block with "btrfs ins dump-super -Ffa"?
>
> sudo btrfs ins dump-super -Ffa /dev/mapper/home :
>
> superblock: bytenr=65536, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0xc59083ff [DON'T MATCH]
> bytenr 65536
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
> metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254
> label
> generation 161054
> root 4028020703232
> sys_array_size 129
> chunk_root_generation 154912
> root_level 1
> chunk_root 22020096
> chunk_root_level 1
> log_root 0
> log_root_transid 0
> log_root_level 0
> total_bytes 4000787030016
> bytes_used 3969357348864
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 1
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation 161054
> uuid_tree_generation 161054
> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
> dev_item.type 0
> dev_item.total_bytes 4000787030016
> dev_item.bytes_used 3997564731392
> dev_item.io_align 4096
> dev_item.io_width 4096
> dev_item.sector_size 4096
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
> sys_chunk_array[2048]:
> item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
> length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 1
> stripe 0 devid 1 offset 22020096
> dev_uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> stripe 1 devid 1 offset 30408704
> dev_uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> backup_roots[4]:
> backup 0:
> backup_tree_root: 4028018622464 gen: 161053 level: 1
> backup_chunk_root: 22020096 gen: 154912 level: 1
> backup_extent_root: 4028018229248 gen: 161053 level: 2
> backup_fs_root: 4028019884032 gen: 161054 level: 3
> backup_dev_root: 4027637743616 gen: 161049 level: 1
> backup_csum_root: 4028020047872 gen: 161054 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969357348864
> backup_num_devices: 1
>
> backup 1:
> backup_tree_root: 4028020703232 gen: 161054 level: 1
> backup_chunk_root: 22020096 gen: 154912 level: 1
> backup_extent_root: 4028020277248 gen: 161054 level: 2
> backup_fs_root: 4028021620736 gen: 161054 level: 3
> backup_dev_root: 4027637743616 gen: 161049 level: 1
> backup_csum_root: 4028020326400 gen: 161054 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969357348864
> backup_num_devices: 1
>
> backup 2:
> backup_tree_root: 4027794194432 gen: 161051 level: 1
> backup_chunk_root: 22020096 gen: 154912 level: 1
> backup_extent_root: 4027793915904 gen: 161051 level: 2
> backup_fs_root: 4027794866176 gen: 161051 level: 3
> backup_dev_root: 4027637743616 gen: 161049 level: 1
> backup_csum_root: 4027793981440 gen: 161051 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969416499200
> backup_num_devices: 1
>
> backup 3:
> backup_tree_root: 4028014985216 gen: 161052 level: 1
> backup_chunk_root: 22020096 gen: 154912 level: 1
> backup_extent_root: 4027953004544 gen: 161052 level: 2
> backup_fs_root: 4027876769792 gen: 161052 level: 3
> backup_dev_root: 4027637743616 gen: 161049 level: 1
> backup_csum_root: 4027969748992 gen: 161052 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969357332480
> backup_num_devices: 1
So far, the first super block looks completely fine.
I guess the corrupted part is in the unused part of the sys chunk array.
So you can re-calculte the checksum of the first super block, and try
mount/btrfs-rescue.
If you don't have the tool to do it by your own, please send the 4K
super block (dd if=/dev/mapper/home bs=1 skip=65536 count=4096
of=/tmp/sb.dump) to me so I could do that for you.
But considering how corrupted other super blocks are, I doubt the chance
to mount.
Thanks,
Qu
>
>
> superblock: bytenr=67108864, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 0 (crc32c)
> csum_size 4
> csum 0x65f1ab31 [DON'T MATCH]
> bytenr 14039944498490823899
> flags 0xcdc898509422934d
> ( WRITTEN |
> unknown flag: 0xcdc898509422934c )
> magic _BHRfS_M [match]
> fsid 4468b2e4-d249-639c-dc54-792caf170cc9
> metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
> label
> generation 161054
> root 4028020703232
> sys_array_size 129
> chunk_root_generation 154912
> root_level 1
> chunk_root 22020096
> chunk_root_level 1
> log_root 11922660382425005768
> log_root_transid 14582268926853107316
> log_root_level 0
> total_bytes 10532150587539085458
> bytes_used 8124524553913841719
> sectorsize 4096
> nodesize 16384
> leafsize (deprecated) 16384
> stripesize 4096
> root_dir 6
> num_devices 1
> compat_flags 0x0
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation 161054
> uuid_tree_generation 161054
> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH]
> dev_item.type 0
> dev_item.total_bytes 4000787030016
> dev_item.bytes_used 3997564731392
> dev_item.io_align 4096
> dev_item.io_width 4096
> dev_item.sector_size 4096
> dev_item.devid 1
> dev_item.dev_group 0
> dev_item.seek_speed 0
> dev_item.bandwidth 0
> dev_item.generation 0
> sys_chunk_array[2048]:
> item 0 key (288472139764826295 UNKNOWN.227 16728631262381099805)
> ERROR: unexpected item type 227 in sys_array at offset 17
> backup_roots[4]:
> backup 0:
> backup_tree_root: 2937111841462972043 gen:
> 6902395341942098838 level: 1
> backup_chunk_root: 17657898596786216606 gen:
> 1348021672883110445 level: 1
> backup_extent_root: 11170562329367461854 gen:
> 11756089894903112279 level: 2
> backup_fs_root: 10659646040436297474 gen:
> 2903212909590398647 level: 3
> backup_dev_root: 14579526832388012464 gen:
> 4376531062597251514 level: 1
> backup_csum_root: 4242763241112 gen: 161054 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969357348864
> backup_num_devices: 1
>
> backup 1:
> backup_tree_root: 4028020703232 gen: 161054 level: 198
> backup_chunk_root: 22020096 gen: 154912 level: 243
> backup_extent_root: 4028020277248 gen:
> 16089197629711480094 level: 206
> backup_fs_root: 5604276798285345924 gen:
> 10182153286217385848 level: 58
> backup_dev_root: 12659493148880815901 gen:
> 9869374175363434348 level: 78
> backup_csum_root: 8949053250522901049 gen:
> 5400723786290947185 level: 164
> backup_total_bytes: 15031814655203781796
> backup_bytes_used: 1855277020722950832
> backup_num_devices: 13548686937124434823
>
> backup 2:
> backup_tree_root: 4043263646859 gen: 161051 level: 1
> backup_chunk_root: 22020096 gen: 154912 level: 1
> backup_extent_root: 4027793915904 gen: 161051 level: 2
> backup_fs_root: 4027794866176 gen: 161051 level: 3
> backup_dev_root: 4027637743616 gen: 161049 level: 1
> backup_csum_root: 4027793981440 gen: 161051 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969416499200
> backup_num_devices: 1
>
> backup 3:
> backup_tree_root: 4028014985216 gen: 161052 level: 78
> backup_chunk_root: 22020096 gen: 154912 level: 70
> backup_extent_root: 4027953004544 gen: 161052 level: 16
> backup_fs_root: 4027876769792 gen: 161052 level: 11
> backup_dev_root: 4027637743616 gen: 161049 level: 125
> backup_csum_root: 4027969748992 gen:
> 15626176790580589852 level: 222
> backup_total_bytes: 18338658601355635447
> backup_bytes_used: 12906905608640099773
> backup_num_devices: 11951688830094275180
>
>
> superblock: bytenr=274877906944, device=/dev/mapper/home
> ---------------------------------------------------------
> csum_type 62267 (INVALID)
> csum_size 32
> csum 0x9876fd0000000000000000000000000000000000000000000000000000000000
> [UNKNOWN CSUM TYPE OR SIZE]
> bytenr 274877906944
> flags 0x1
> ( WRITTEN )
> magic _BHRfS_M [match]
> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
> metadata_uuid 07eb556b-df56-afbd-12c1-32f496c9ae5e
> label
> ...^k...cA..[....ws.......!...&+..@...U.^..Fi.....^s....%.....G.....\..z.0..8N......l
>
> generation 161054
> root 4028020703232
> sys_array_size 1346087890
> chunk_root_generation 14973196025592430218
> root_level 113
> chunk_root 22020096
> chunk_root_level 54
> log_root 0
> log_root_transid 0
> log_root_level 77
> total_bytes 4000787030016
> bytes_used 3969357348864
> sectorsize 544999736
> nodesize 1626255393
> leafsize (deprecated) 365786189
> stripesize 1625894637
> root_dir 4621774357935484814
> num_devices 6281988177397337675
> compat_flags 0x9c8dbbf37bf3e638
> compat_ro_flags 0xe392e94b904707de
> ( FREE_SPACE_TREE_VALID |
> unknown flag: 0xe392e94b904707dc )
> incompat_flags 0xe7c3113fe08815af
> ( MIXED_BACKREF |
> DEFAULT_SUBVOL |
> MIXED_GROUPS |
> COMPRESS_LZO |
> BIG_METADATA |
> RAID56 |
> SKINNY_METADATA |
> METADATA_UUID |
> ZONED |
> unknown flag: 0xe7c3113fe0880000 )
> cache_generation 2740636164699663627
> uuid_tree_generation 143387309824099247
> dev_item.uuid 8ca1383a-3aaf-986e-30f3-1081cc4423b9
> dev_item.fsid d3b80203-160e-b664-1fa0-0121596e5fbf [DON'T MATCH]
> dev_item.type 2243501307224589742
> dev_item.total_bytes 17886909296177165879
> dev_item.bytes_used 843442598421986990
> dev_item.io_align 3037217492
> dev_item.io_width 3952451380
> dev_item.sector_size 2707671125
> dev_item.devid 4105250638360812501
> dev_item.dev_group 3766056873
> dev_item.seek_speed 182
> dev_item.bandwidth 229
> dev_item.generation 9452573969509059034
> sys_chunk_array[2048]:
> ERROR: sys_array_size 1346087890 shouldn't exceed 2048 bytes
> backup_roots[4]:
> backup 0:
> backup_tree_root: 4028018622464 gen: 161053 level: 243
> backup_chunk_root: 22020096 gen: 154912 level: 189
> backup_extent_root: 4028018229248 gen: 161053 level: 184
> backup_fs_root: 4028019884032 gen: 161054 level: 247
> backup_dev_root: 4027637743616 gen: 161049 level: 241
> backup_csum_root: 11051166811494301696 gen:
> 11918979514504630695 level: 43
> backup_total_bytes: 6128247333213658072
> backup_bytes_used: 13316083672891396332
> backup_num_devices: 12986852272682621048
>
> backup 1:
> backup_tree_root: 11912830949670804138 gen:
> 4162465539222756018 level: 1
> backup_chunk_root: 10938026073880195285 gen:
> 8406240668729036675 level: 1
> backup_extent_root: 17210210806549355879 gen:
> 455024150221 level: 2
> backup_fs_root: 4028021620736 gen: 161054 level: 3
> backup_dev_root: 4027637743616 gen: 161049 level: 1
> backup_csum_root: 4028020326400 gen: 161054 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969357348864
> backup_num_devices: 1
>
> backup 2:
> backup_tree_root: 12811350786919235584 gen:
> 3323034891246254263 level: 217
> backup_chunk_root: 17845957508331272086 gen:
> 12071669715000553588 level: 235
> backup_extent_root: 2368736995830619850 gen:
> 14589654675806411696 level: 163
> backup_fs_root: 7496041380651946087 gen:
> 16394240941193014637 level: 29
> backup_dev_root: 11890518936409562630 gen:
> 11935615530850632810 level: 174
> backup_csum_root: 4804084557218082292 gen:
> 3037656397741716549 level: 57
> backup_total_bytes: 12628714618049392498
> backup_bytes_used: 4517632456508041139
> backup_num_devices: 11559351643267246189
>
> backup 3:
> backup_tree_root: 9502548224584054781 gen:
> 3258916775715815651 level: 1
> backup_chunk_root: 2996837717780471142 gen:
> 9778119635099568422 level: 1
> backup_extent_root: 2696351959151677890 gen:
> 5061715638962835499 level: 2
> backup_fs_root: 16900363948952743119 gen:
> 2413589847950453178 level: 3
> backup_dev_root: 18344731769370366813 gen:
> 11720350780674912754 level: 1
> backup_csum_root: 6130366103217558429 gen:
> 598304231798 level: 3
> backup_total_bytes: 4000787030016
> backup_bytes_used: 3969357332480
> backup_num_devices: 1
>
>
> On 12.10.21 01:22, Qu Wenruo wrote:
>>
>>
>> On 2021/10/11 22:13, Kyle James Chapman wrote:
>>> Hello,
>>>
>>> I lost access to my home BTRFS filesystem on a 4TB SATA drive without
>>> partitioning today.
>>>
>>> I am a graduate student of translation studies at a foreign
>>> university. Programming is not my specialty but have ran Linux for
>>> over a decade because I support open source endeavors. I say this for
>>> background on my technical capabilities, I am also a radio amateur
>>> extra-class in the United States. I have some technical competence
>>> and run Arch Linux for general duty.
>>>
>>> The system boots and reads the kernel into memory from an optical ROM
>>> drive, reads keyfiles from a USB stick, decrypts the home and swap
>>> partitions on different devices, asks for a password for the root
>>> partition, and finishes booting. I was rebooting my system from
>>> userspace and observed a kernel panic after finishing some work
>>> repartitioning a small USB stick using gparted.
>>
>> Any idea on what part is causing the panic, or any log/screenshot of the
>> panic message?
>>
>>> After rebooting, my home BTRFS system was not mountable. Some
>>> information seems present in some of the superblocks, but I do not
>>> understand everything. I was careful in my use of the disk tool, and
>>> it was clear that only the USB stick was being accessed, because of
>>> the simple work I was doing, resizing a partition on the obvious
>>> drive. I would like to recover the data. I finished writing my
>>> master's thesis and have already submitted it, so no real work was
>>> lost, but I would like to understand why this is happening and how
>>> the data can be recovered. A memory fault seems plausible, but the
>>> fact that I was accessing the partitions (through gparted) seems to
>>> indicate other possibilities. Please do not get sidetracked into
>>> thinking I wrote data on the drive through carelessness. I have made
>>> many many uses of disk tools throughout my life and the simple nature
>>> of the operations which did not involve the filesystem in question
>>> are completely understandable and I remember them clearly.
>>>
>>> Please advise.
>>>
>>> Relevant /dev/fstab entry which has worked for several months until I
>>> commented it out today:
>>>
>>> #/dev/mapper/home /home btrfs
>>> rw,compress,autodefrag,inode_cache 0 1
>>
>> Inode cache is already deprecated, but it won't do anything to the
>> kernel, thus it shouldn't cause any problem.
>>
>>>
>>> cat /proc/version:
>>>
>>> Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4
>>> (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils)
>>> 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000
>>>
>>>
>>> sudo smartctl -a /dev/sdd excerpt:
>>>
>>> === START OF INFORMATION SECTION ===
>>> Model Family: Seagate Barracuda 2.5 5400
>>> Device Model: ST4000LM024-2AN17V
>>> Serial Number: WCK03H8Y
>>> LU WWN Device Id: 5 000c50 09e55827b
>>> Firmware Version: 0001
>>> User Capacity: 4,000,787,030,016 bytes [4.00 TB]
>>> Sector Sizes: 512 bytes logical, 4096 bytes physical
>>> Rotation Rate: 5526 rpm
>>> Form Factor: 2.5 inches
>>> Device is: In smartctl database [for details use: -P show]
>>> ATA Version is: ACS-3 T13/2161-D revision 5
>>> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
>>> Local Time is: Mon Oct 11 16:07:49 2021 CEST
>>> SMART support is: Available - device has SMART capability.
>>> SMART support is: Enabled
>>>
>>> === START OF READ SMART DATA SECTION ===
>>> SMART overall-health self-assessment test result: PASSED
>>>
>>> btrfs --version:
>>>
>>> btrfs-progs v5.14.1
>>>
>>>
>>> sudo btrfs inspect-internal dump-super -s 0 /dev/mapper/home :
>>>
>>> superblock: bytenr=65536, device=/dev/mapper/home
>>
>> The first copy in fact looks pretty good.
>>
>> If there is something wrong, then it would be in the backup roots or sys
>> chunk map.
>>
>> Mind to dump the full super block with "btrfs ins dump-super -Ffa"?
>>> ---------------------------------------------------------
>>> csum_type 0 (crc32c)
>>> csum_size 4
>>> csum 0xc59083ff [DON'T MATCH]
>>> bytenr 65536
>>> flags 0x1
>>> ( WRITTEN )
>>> magic _BHRfS_M [match]
>>> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
>>> metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254
>>> label
>>> generation 161054
>>> root 4028020703232
>>> sys_array_size 129
>>> chunk_root_generation 154912
>>> root_level 1
>>> chunk_root 22020096
>>> chunk_root_level 1
>>> log_root 0
>>> log_root_transid 0
>>> log_root_level 0
>>> total_bytes 4000787030016
>>> bytes_used 3969357348864
>>> sectorsize 4096
>>> nodesize 16384
>>> leafsize (deprecated) 16384
>>> stripesize 4096
>>> root_dir 6
>>> num_devices 1
>>> compat_flags 0x0
>>> compat_ro_flags 0x0
>>> incompat_flags 0x161
>>> ( MIXED_BACKREF |
>>> BIG_METADATA |
>>> EXTENDED_IREF |
>>> SKINNY_METADATA )
>>> cache_generation 161054
>>> uuid_tree_generation 161054
>>> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
>>> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
>>> dev_item.type 0
>>> dev_item.total_bytes 4000787030016
>>> dev_item.bytes_used 3997564731392
>>> dev_item.io_align 4096
>>> dev_item.io_width 4096
>>> dev_item.sector_size 4096
>>> dev_item.devid 1
>>> dev_item.dev_group 0
>>> dev_item.seek_speed 0
>>> dev_item.bandwidth 0
>>> dev_item.generation 0
>>>
>>> sudo btrfs inspect-internal dump-super -s 1 /dev/mapper/home :
>>>
>>> superblock: bytenr=67108864, device=/dev/mapper/home
>>> ---------------------------------------------------------
>>> csum_type 0 (crc32c)
>>> csum_size 4
>>> csum 0x65f1ab31 [DON'T MATCH]
>>> bytenr 14039944498490823899
>>> flags 0xcdc898509422934d
>>> ( WRITTEN |
>>> unknown flag: 0xcdc898509422934c )
>>
>> The 2nd one begins to have some garbage, but not yet full of garbage.
>>
>> This means there are some writes into the 2nd super block, and mostly
>> the writes are just into part of the super block.
>>
>>> magic _BHRfS_M [match]
>>> fsid 4468b2e4-d249-639c-dc54-792caf170cc9
>>> metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
>>
>> The UUID is damaged.
>>
>>> label
>>> generation 161054
>>> root 4028020703232 > sys_array_size 129
>>> chunk_root_generation 154912
>>> root_level 1
>>> chunk_root 22020096
>>> chunk_root_level 1
>>
>> But then some good data.
>>
>>> log_root 11922660382425005768
>>> log_root_transid 14582268926853107316
>>> log_root_level 0
>>> total_bytes 10532150587539085458
>>> bytes_used 8124524553913841719
>>
>> Some garbage again.
>>
>> I don't think it can be even caused by simple memory corruption, too
>> many random part gets corrupted.
>>
>> It's really hard to say if btrfs itself is involved, until we see the
>> calltrace.
>>
>>
>> For the recovery, the superblock needs to be manually fixed to allow any
>> further data salvage attempts.
>>
>> And that can only be done with the extra "btrfs ins dump-super -Ffa"
>> output.
>>
>> Thanks,
>> Qu
>>
>>> sectorsize 4096
>>> nodesize 16384
>>> leafsize (deprecated) 16384
>>> stripesize 4096
>>> root_dir 6
>>> num_devices 1
>>> compat_flags 0x0
>>> compat_ro_flags 0x0
>>> incompat_flags 0x161
>>> ( MIXED_BACKREF |
>>> BIG_METADATA |
>>> EXTENDED_IREF |
>>> SKINNY_METADATA )
>>> cache_generation 161054
>>> uuid_tree_generation 161054
>>> dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51
>>> dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH]
>>> dev_item.type 0
>>> dev_item.total_bytes 4000787030016
>>> dev_item.bytes_used 3997564731392
>>> dev_item.io_align 4096
>>> dev_item.io_width 4096
>>> dev_item.sector_size 4096
>>> dev_item.devid 1
>>> dev_item.dev_group 0
>>> dev_item.seek_speed 0
>>> dev_item.bandwidth 0
>>> dev_item.generation 0
>>>
>>> sudo btrfs inspect-internal dump-super -s 2 /dev/mapper/home:
>>>
>>> superblock: bytenr=274877906944, device=/dev/mapper/home
>>> ---------------------------------------------------------
>>> csum_type 62267 (INVALID)
>>> csum_size 32
>>> csum
>>> 0x9876fd0000000000000000000000000000000000000000000000000000000000
>>> [UNKNOWN CSUM TYPE OR SIZE]
>>> bytenr 274877906944
>>> flags 0x1
>>> ( WRITTEN )
>>> magic _BHRfS_M [match]
>>> fsid bd5872ad-8fee-4d90-b048-b81120ce3254
>>> metadata_uuid 07eb556b-df56-afbd-12c1-32f496c9ae5e
>>> label
>>> ...^k...cA..[....ws.......!...&+..@...U.^..Fi.....^s....%.....G.....\..z.0..8N......l
>>>
>>> generation 161054
>>> root 4028020703232
>>> sys_array_size 1346087890
>>> chunk_root_generation 14973196025592430218
>>> root_level 113
>>> chunk_root 22020096
>>> chunk_root_level 54
>>> log_root 0
>>> log_root_transid 0
>>> log_root_level 77
>>> total_bytes 4000787030016
>>> bytes_used 3969357348864
>>> sectorsize 544999736
>>> nodesize 1626255393
>>> leafsize (deprecated) 365786189
>>> stripesize 1625894637
>>> root_dir 4621774357935484814
>>> num_devices 6281988177397337675
>>> compat_flags 0x9c8dbbf37bf3e638
>>> compat_ro_flags 0xe392e94b904707de
>>> ( FREE_SPACE_TREE_VALID |
>>> unknown flag: 0xe392e94b904707dc )
>>> incompat_flags 0xe7c3113fe08815af
>>> ( MIXED_BACKREF |
>>> DEFAULT_SUBVOL |
>>> MIXED_GROUPS |
>>> COMPRESS_LZO |
>>> BIG_METADATA |
>>> RAID56 |
>>> SKINNY_METADATA |
>>> METADATA_UUID |
>>> ZONED |
>>> unknown flag: 0xe7c3113fe0880000 )
>>> cache_generation 2740636164699663627
>>> uuid_tree_generation 143387309824099247
>>> dev_item.uuid 8ca1383a-3aaf-986e-30f3-1081cc4423b9
>>> dev_item.fsid d3b80203-160e-b664-1fa0-0121596e5fbf [DON'T MATCH]
>>> dev_item.type 2243501307224589742
>>> dev_item.total_bytes 17886909296177165879
>>> dev_item.bytes_used 843442598421986990
>>> dev_item.io_align 3037217492
>>> dev_item.io_width 3952451380
>>> dev_item.sector_size 2707671125
>>> dev_item.devid 4105250638360812501
>>> dev_item.dev_group 3766056873
>>> dev_item.seek_speed 182
>>> dev_item.bandwidth 229
>>> dev_item.generation 9452573969509059034
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: All Three Superblocks Damaged After Kernel Panic
2021-10-12 10:29 ` Qu Wenruo
@ 2021-10-12 10:35 ` K Chapman
2021-10-12 12:59 ` Qu Wenruo
0 siblings, 1 reply; 7+ messages in thread
From: K Chapman @ 2021-10-12 10:35 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 607 bytes --]
Qu,
RE: If you don't have the tool to do it by your own, please send the 4K
super block (dd if=/dev/mapper/home bs=1 skip=65536 count=4096
of=/tmp/sb.dump) to me so I could do that for you.
See attached.
RE: Power loss and data writes.
I believe the possibility is present but remote. I have not had a panic
on x86 in many years during general work, this is more of an indication
of some sort of problem than simply removing power at a bad time. It
seems more likely, that for whatever reason, the kernel wrote spurious
data to certain sectors, it will be interesting to see.
Thank you!
Kyle C.
[-- Attachment #2: sb.dump --]
[-- Type: application/octet-stream, Size: 4096 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: All Three Superblocks Damaged After Kernel Panic
2021-10-12 10:35 ` K Chapman
@ 2021-10-12 12:59 ` Qu Wenruo
0 siblings, 0 replies; 7+ messages in thread
From: Qu Wenruo @ 2021-10-12 12:59 UTC (permalink / raw)
To: K Chapman, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]
On 2021/10/12 18:35, K Chapman wrote:
> Qu,
>
> RE: If you don't have the tool to do it by your own, please send the 4K
> super block (dd if=/dev/mapper/home bs=1 skip=65536 count=4096
> of=/tmp/sb.dump) to me so I could do that for you.
>
> See attached.
Re-csumed super block is attached.
It passes my local test using btrfs-ins-dump-super.
Thus the csum should be correct.
Wish you good luck salvaging your data.
I would try mount with "-o ro,rescue=all" first, if no luck then
btrfs-restore --dry-run to see what can be salvaged.
But I doubt how many can be salvaged, consider how corrupted the backup
super blocks are.
Thanks,
Qu
>
> RE: Power loss and data writes.
>
> I believe the possibility is present but remote. I have not had a panic
> on x86 in many years during general work, this is more of an indication
> of some sort of problem than simply removing power at a bad time. It
> seems more likely, that for whatever reason, the kernel wrote spurious
> data to certain sectors, it will be interesting to see.
>
> Thank you!
>
> Kyle C.
[-- Attachment #2: sb.dump --]
[-- Type: application/octet-stream, Size: 4096 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-10-12 12:59 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-11 14:13 All Three Superblocks Damaged After Kernel Panic Kyle James Chapman
2021-10-11 18:23 ` Matt Corallo
2021-10-11 23:22 ` Qu Wenruo
2021-10-12 7:41 ` K Chapman
2021-10-12 10:29 ` Qu Wenruo
2021-10-12 10:35 ` K Chapman
2021-10-12 12:59 ` Qu Wenruo
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.