Tried to replace a drive in a raid 1 and all hell broke loose

Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

Hello, this is my first time interacting with a mailing list and the 
first time actually using btrfs.

THE CURRENT SITUATION:
I think I have two working drives with dm-crypt in what i think is raid 
1
when booted on a either fedora live disk or debian testing i can mount 
the filesystem but check fails and scrub aborts.
    -= FEDORA =-
The first time i scrub i get this on the kernel log: ( attachment 
fedora_scrub_1 )
i unmounted, mounted and scrubbed again and got a different error: ( 
attachment fedora_scrub_2 )
check says this: ( attachment fedora_check )
This is the distro info: ( attachment fedora_info ))

  -= DEBIAN TESTING =-
The first time i scrub i get this on the kernel log: ( attachment 
debian_scrub1 )
I unmounted and got a call trace and register dump on the kernel log: ( 
attachment debian_unmount )
i mounted and scrubbed again and got a similar error to fedora's second 
scrub: ( attachment debian_scrub2 )
check says the same thing
This is the distro info: ( attachment debian_info )

THE BACKSTORY:
here is a timeline of the events that lead up to this
(by the way, all of this was done with clean unmounts and reboots)
  [ Using debian stable btrfs-progs 5.10.1-2 ]
- Btrfs raid 1 works great srubs ok with 2 drives
- One drive starts dying, i get one to replace it
- Add the new one to the array (with all three plugged in) (i now know i 
should've use replace instead)
- run btrfs balance (with all three again)
- tried to delete the drive but failed (around this time i started 
running into weird errors)
- updated debian to testing, updated kernel, rebooted ( btrgs-progs 
5.10.1-2 -> 5.17-1 )
- managed to remove the failing drive (not sure if the update helped)
- Have to mount degraded and check shows a lot of errors
- bite the bullet and run check --repair, it says it is fixing errors 
and at the end it says no errors
- run just check again and get the same errors as before roughly
- run check --repair again, gets stuck in what seems to be an infinite 
loop. Says its fixing the same things again and again
- joined the irc channel to ask for help
- send sigterm to btrfs check --repair ( no one had replied by then)
- it actually mounts without needing -o degraded now
- get told to run 'btrfs fi us -T /mnt' and realize that i have both 
raid1 and some small single chunks
- run btrfs balance start -mconvert=raid1,soft -dconvert=raid1,soft /mnt
- btrfs check shows much less errors and all single chunks are gone ( 
the btrfs check output at that point is attached as old_completing_check 
on the email )
- try to run scrubs, they abort
- check and there are some single chunks yet again
- try fedora, run check and it gives the error i have attached and 
doesn't go through with actually checking everything
- try debian again and now checking doesn't complete with the same error 
as in fedora
I also tried running 'btrfs balance start -mconvert=raid1,soft 
-dconvert=raid1,soft /mnt' again now and the first time I run it, it 
says
ERROR: error during balancing '/mnt/sd': No such file or directory
and it the second time
ERROR: error during balancing '/mnt/sd': Read-only file system

Thanks a lot for reading my email and sorry if it winds up being 
something not related to btrfs

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

Oh right thanks

On 2022-05-23 20:45, Forza wrote:
> Hi, the attachments were not included in the email.
> 
> Forza
> 

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

[-- Attachment #1: Type: text/plain, Size: 75 bytes --]

On 2022-05-23 21:00, efkf@firemail.cc wrote:
Sorry they should be sent now

[-- Attachment #2: debian_info --]
[-- Type: text/plain, Size: 925 bytes --]

#uname -a
Linux debian 5.17.0-1-amd64 #1 SMP PREEMPT Debian 5.17.3-1 (2022-04-18) x86_64 GNU/Linux
#btrfs --version
btrfs-progs v5.17
#btrfs fi show
Label: none  uuid: 8039cd28-5e4c-46a7-8a5f-afb57ea68c08
	Total devices 2 FS bytes used 772.76GiB
	devid    2 size 1.82TiB used 334.00GiB path /dev/mapper/ST2000DL003-###############
	devid    3 size 1.82TiB used 661.00GiB path /dev/mapper/ST3000VN007-###############

#btrfs fi df /mnt/sd/
Data, RAID1: total=772.00GiB, used=771.22GiB
Data, single: total=1.00GiB, used=2.25MiB
System, RAID1: total=32.00MiB, used=96.00KiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=3.00GiB, used=1.54GiB
Metadata, single: total=1.00GiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
WARNING:   Data: single, raid1
WARNING:   Metadata: single, raid1
WARNING:   System: single, raid1

[-- Attachment #3: debian_scrub1 --]
[-- Type: text/plain, Size: 15564 bytes --]

[    5.797266] EXT4-fs (sdc3): mounted filesystem with ordered data mode. Quota mode: none.
[    6.057243] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[    7.628455] systemd[1]: Inserted module 'autofs4'
[    7.863975] systemd[1]: systemd 247.3-7 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
[    7.888069] systemd[1]: Detected architecture x86-64.
[    7.906132] systemd[1]: Set hostname to <debian>.
[    9.357810] systemd[1]: Queued start job for default target Graphical Interface.
[    9.369891] systemd[1]: Created slice system-getty.slice.
[    9.370814] systemd[1]: Created slice system-modprobe.slice.
[    9.371511] systemd[1]: Created slice User and Session Slice.
[    9.371733] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    9.371937] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    9.372367] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    9.372472] systemd[1]: Reached target Local Encrypted Volumes.
[    9.372542] systemd[1]: Reached target Paths.
[    9.372582] systemd[1]: Reached target Remote File Systems.
[    9.372612] systemd[1]: Reached target Slices.
[    9.372968] systemd[1]: Listening on Syslog Socket.
[    9.373303] systemd[1]: Listening on fsck to fsckd communication Socket.
[    9.373530] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    9.374213] systemd[1]: Listening on Journal Audit Socket.
[    9.374540] systemd[1]: Listening on Journal Socket (/dev/log).
[    9.374923] systemd[1]: Listening on Journal Socket.
[    9.382652] systemd[1]: Listening on udev Control Socket.
[    9.382968] systemd[1]: Listening on udev Kernel Socket.
[    9.385128] systemd[1]: Mounting Huge Pages File System...
[    9.387744] systemd[1]: Mounting POSIX Message Queue File System...
[    9.390230] systemd[1]: Mounting Kernel Debug File System...
[    9.392806] systemd[1]: Mounting Kernel Trace File System...
[    9.402681] systemd[1]: Starting Set the console keyboard layout...
[    9.405354] systemd[1]: Starting Create list of static device nodes for the current kernel...
[    9.408279] systemd[1]: Starting Load Kernel Module configfs...
[    9.411098] systemd[1]: Starting Load Kernel Module drm...
[    9.417826] systemd[1]: Starting Load Kernel Module fuse...
[    9.494252] systemd[1]: Starting Set Up Additional Binary Formats...
[    9.494406] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[    9.499144] systemd[1]: Starting Journal Service...
[    9.635774] systemd[1]: Starting Load Kernel Modules...
[    9.638484] systemd[1]: Starting Remount Root and Kernel File Systems...
[    9.641761] systemd[1]: Starting Coldplug All udev Devices...
[    9.646766] systemd[1]: Finished Create list of static device nodes for the current kernel.
[    9.647708] systemd[1]: modprobe@configfs.service: Succeeded.
[    9.648335] systemd[1]: Finished Load Kernel Module configfs.
[    9.648746] systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 263 (systemd-binfmt)
[    9.651075] systemd[1]: Mounting Arbitrary Executable File Formats File System...
[    9.655456] systemd[1]: Mounting Kernel Configuration File System...
[    9.703484] systemd[1]: Finished Load Kernel Modules.
[    9.708727] systemd[1]: Starting Apply Kernel Variables...
[    9.717126] fuse: init (API version 7.36)
[    9.719111] systemd[1]: modprobe@fuse.service: Succeeded.
[    9.719781] systemd[1]: Finished Load Kernel Module fuse.
[    9.722478] systemd[1]: Mounting FUSE Control File System...
[    9.961524] systemd[1]: Mounted Huge Pages File System.
[    9.961971] systemd[1]: Mounted POSIX Message Queue File System.
[    9.962402] systemd[1]: Mounted Kernel Debug File System.
[    9.962845] systemd[1]: Mounted Kernel Trace File System.
[    9.963986] systemd[1]: Finished Set the console keyboard layout.
[    9.964562] systemd[1]: Mounted Kernel Configuration File System.
[    9.965022] systemd[1]: Mounted FUSE Control File System.
[    9.965246] EXT4-fs (sdc3): re-mounted. Quota mode: none.
[    9.968206] systemd[1]: Finished Remount Root and Kernel File Systems.
[    9.968532] systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
[    9.970921] systemd[1]: Starting Load/Save Random Seed...
[    9.973459] systemd[1]: Starting Create System Users...
[   10.046058] systemd[1]: Mounted Arbitrary Executable File Formats File System.
[   10.132218] systemd[1]: Finished Apply Kernel Variables.
[   10.232882] systemd[1]: Started Journal Service.
[   10.295238] systemd-journald[264]: Received client request to flush runtime journal.
[   10.338324] ACPI: bus type drm_connector registered
[   13.023835] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input6
[   13.023973] ACPI: button: Power Button [PWRB]
[   13.024130] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input7
[   13.027130] audit: type=1400 audit(1653318123.330:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=296 comm="apparmor_parser"
[   13.027148] audit: type=1400 audit(1653318123.330:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=296 comm="apparmor_parser"
[   13.039994] ACPI: button: Power Button [PWRF]
[   13.045575] audit: type=1400 audit(1653318123.350:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=295 comm="apparmor_parser"
[   13.391302] audit: type=1400 audit(1653318123.694:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=318 comm="apparmor_parser"
[   13.391319] audit: type=1400 audit(1653318123.694:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=318 comm="apparmor_parser"
[   13.391325] audit: type=1400 audit(1653318123.694:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=318 comm="apparmor_parser"
[   13.664356] input: PC Speaker as /devices/platform/pcspkr/input/input8
[   13.742775] intel_rng: FWH not detected
[   13.805466] coretemp coretemp.0: Using relative temperature scale!
[   13.805894] coretemp coretemp.0: Using relative temperature scale!
[   13.936288] leds_ss4200: no LED devices found
[   13.984875] intel_powerclamp: No package C-state available
[   14.020425] intel_powerclamp: No package C-state available
[   14.165867] parport_pc 00:04: reported by Plug and Play ACPI
[   14.165966] parport0: PC-style at 0x378, irq 7 [PCSPP]
[   14.243611] sd 2:0:0:0: Attached scsi generic sg0 type 0
[   14.244524] sd 2:0:1:0: Attached scsi generic sg1 type 0
[   14.245141] sd 4:0:0:0: Attached scsi generic sg2 type 0
[   14.398315] audit: type=1400 audit(1653318124.702:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="system_tor" pid=317 comm="apparmor_parser"
[   14.501816] ppdev: user-space parallel port driver
[   14.731709] iTCO_vendor_support: vendor-support=0
[   15.068605] iTCO_wdt iTCO_wdt.1.auto: Found a ICH7 or ICH7R TCO device (Version=2, TCOBASE=0x0860)
[   15.069204] iTCO_wdt iTCO_wdt.1.auto: initialized. heartbeat=30 sec (nowayout=0)
[   15.771837] sd 4:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[   15.771864] sd 4:0:0:0: [sdc] tag#0 Sense Key : Hardware Error [current] 
[   15.771875] sd 4:0:0:0: [sdc] tag#0 Add. Sense: No additional sense information
[   15.771917] sd 4:0:0:0: [sdc] tag#0 CDB: opcode=0xdf (vendor) df 10 00 00 00 05 fe 00 00 00 b0 ef
[   16.713069] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC888: line_outs=4 (0x14/0x15/0x16/0x17/0x0) type:line
[   16.713087] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[   16.713092] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[   16.713097] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[   16.713099] snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x1e/0x0
[   16.713103] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[   16.713106] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[   16.713109] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[   16.713113] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[   16.732818] input: HDA Intel Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[   16.735950] Adding 999420k swap on /dev/sdc4.  Priority:-2 extents:1 across:999420k FS
[   16.736391] input: HDA Intel Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[   16.740739] input: HDA Intel Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input11
[   16.740883] input: HDA Intel Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[   16.741016] input: HDA Intel Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[   16.741155] input: HDA Intel Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card0/input14
[   16.741812] input: HDA Intel Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card0/input15
[   16.742366] input: HDA Intel Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input16
[   18.132325] pci 0000:00:00.0: Intel G33 Chipset
[   18.132353] pci 0000:00:00.0: detected gtt size: 524288K total, 262144K mappable
[   18.132428] pci 0000:00:00.0: detected 8192K stolen memory
[   18.132478] i915 0000:00:02.0: vgaarb: deactivate vga console
[   18.133120] Console: switching to colour dummy device 80x25
[   18.135931] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[   18.151203] i915 0000:00:02.0: [drm] Initialized overlay support.
[   18.152366] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[   18.186776] fbcon: i915drmfb (fb0) is primary device
[   18.237350] Console: switching to colour frame buffer device 160x64
[   18.257554] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[   81.347551] r8169 0000:02:00.0: firmware: failed to load rtl_nic/rtl8168d-1.fw (-2)
[   81.347563] firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware
[   81.347567] r8169 0000:02:00.0: Direct firmware load for rtl_nic/rtl8168d-1.fw failed with error -2
[   81.347573] r8169 0000:02:00.0: Unable to load firmware rtl_nic/rtl8168d-1.fw (-2)
[   81.349448] RTL8211B Gigabit Ethernet r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC)
[   81.417792] r8169 0000:02:00.0 enp2s0: Link is Down
[   83.755004] r8169 0000:02:00.0 enp2s0: Link is Up - 1Gbps/Full - flow control rx/tx
[   83.755028] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
[  150.304422] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[  150.304509] device-mapper: uevent: version 1.0.3
[  150.304702] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
[  154.035886] cryptd: max_cpu_qlen set to 1000
[  154.319396] BTRFS: device fsid 8039cd28-5e4c-46a7-8a5f-afb57ea68c08 devid 2 transid 31152 /dev/dm-0 scanned by systemd-udevd (771)
[  158.237670] BTRFS: device fsid 8039cd28-5e4c-46a7-8a5f-afb57ea68c08 devid 3 transid 31152 /dev/dm-1 scanned by systemd-udevd (828)
[  177.927066] BTRFS info (device dm-0): flagging fs with big metadata feature
[  177.927079] BTRFS info (device dm-0): disk space caching is enabled
[  177.927084] BTRFS info (device dm-0): has skinny extents
[  269.369547] BTRFS info (device dm-0): scrub: started on devid 3
[  269.411010] BTRFS info (device dm-0): scrub: started on devid 2
[  281.002989] BTRFS error (device dm-0): tree first key mismatch detected, bytenr=1166186053632 parent_transid=31136 key expected=(1995819548672,169,0) has=(1749212258304,168,262144)
[  281.003011] BTRFS error (device dm-0): tree first key mismatch detected, bytenr=1166186053632 parent_transid=31136 key expected=(1995819548672,169,0) has=(1749212258304,168,262144)
[  281.005115] BTRFS info (device dm-0): scrub: not finished on devid 2 with status: -117
[  305.197836] BTRFS error (device dm-0): tree block 1166181154816 owner 1 already locked by pid=859, extent tree corruption detected
[  305.197880] BTRFS warning (device dm-0): Skipping commit of aborted transaction.
[  305.197885] ------------[ cut here ]------------
[  305.197886] BTRFS: Transaction aborted (error -117)
[  305.197971] WARNING: CPU: 1 PID: 859 at fs/btrfs/transaction.c:1974 btrfs_commit_transaction.cold+0x22f/0x2f9 [btrfs]
[  305.198126] Modules linked in: crypto_simd cryptd xts ecb dm_crypt dm_mod i915 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio video snd_hda_intel snd_intel_dspcfg ttm snd_intel_sdw_acpi drm_kms_helper snd_hda_codec snd_hda_core iTCO_wdt intel_pmc_bxt snd_hwdep cec iTCO_vendor_support watchdog rc_core snd_pcm ppdev i2c_algo_bit snd_timer sg parport_pc snd parport soundcore evdev coretemp rng_core pcspkr serio_raw button acpi_cpufreq drm binfmt_misc fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 uas usb_storage btrfs blake2b_generic xor raid6_pq zstd_compress libcrc32c crc32c_generic hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_common ata_generic ata_piix uhci_hcd libata ehci_pci ehci_hcd usbcore scsi_mod r8169 psmouse i2c_i801 realtek mdio_devres i2c_smbus scsi_common lpc_ich libphy usb_common
[  305.198224] CPU: 1 PID: 859 Comm: btrfs-transacti Not tainted 5.17.0-1-amd64 #1  Debian 5.17.3-1
[  305.198229] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD MS-7529/G31TM-P35 (MS-7529), BIOS V4.4 11/25/2009
[  305.198232] RIP: 0010:btrfs_commit_transaction.cold+0x22f/0x2f9 [btrfs]
[  305.198359] Code: 4b 99 e7 e9 1d 55 f5 ff 49 8b 55 28 49 8b 45 30 48 89 42 08 48 89 10 e9 fb fe ff ff 89 de 48 c7 c7 90 09 75 c0 e8 39 99 fc e7 <0f> 0b e9 31 fe ff ff 48 8b 7d 50 89 da 48 c7 c6 c0 09 75 c0 e8 01
[  305.198363] RSP: 0018:ffffb1cec08dfe58 EFLAGS: 00010282
[  305.198367] RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000000
[  305.198370] RDX: 0000000000000001 RSI: ffffffffa8f52a0e RDI: 00000000ffffffff
[  305.198372] RBP: ffff8eca83bc8a28 R08: 0000000000000000 R09: ffffb1cec08dfc88
[  305.198374] R10: ffffb1cec08dfc80 R11: ffffffffa96d1568 R12: ffff8eca83bc8978
[  305.198376] R13: ffff8eca8474ba00 R14: ffff8eca89ec0000 R15: ffff8eca83bc8978
[  305.198379] FS:  0000000000000000(0000) GS:ffff8ecaabc80000(0000) knlGS:0000000000000000
[  305.198381] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  305.198384] CR2: 00007fce1f0cfb70 CR3: 0000000104a4c000 CR4: 00000000000006e0
[  305.198386] Call Trace:
[  305.198395]  <TASK>
[  305.198401]  ? start_transaction+0xc7/0x610 [btrfs]
[  305.198496]  transaction_kthread+0x139/0x1b0 [btrfs]
[  305.198582]  ? btrfs_cleanup_transaction.isra.0+0x580/0x580 [btrfs]
[  305.198661]  kthread+0xe8/0x110
[  305.198672]  ? kthread_complete_and_exit+0x20/0x20
[  305.198676]  ret_from_fork+0x22/0x30
[  305.198684]  </TASK>
[  305.198687] ---[ end trace 0000000000000000 ]---
[  305.198697] BTRFS: error (device dm-0) in cleanup_transaction:1974: errno=-117 Filesystem corrupted
[  305.198709] BTRFS info (device dm-0): forced readonly
[  306.330192] BTRFS info (device dm-0): scrub: not finished on devid 3 with status: -125

[-- Attachment #4: debian_scrub2 --]
[-- Type: text/plain, Size: 16309 bytes --]

[  499.002892] BTRFS info (device dm-0): flagging fs with big metadata feature
[  499.002906] BTRFS info (device dm-0): disk space caching is enabled
[  499.002910] BTRFS info (device dm-0): has skinny extents
[  518.892031] BTRFS info (device dm-0): scrub: started on devid 3
[  518.918485] BTRFS info (device dm-0): scrub: started on devid 2
[  535.479811] BTRFS error (device dm-0): tree first key mismatch detected, bytenr=1166185938944 parent_transid=31136 key expected=(1166224539648,169,0) has=(1995819548672,169,0)
[  535.479832] BTRFS error (device dm-0): tree first key mismatch detected, bytenr=1166185938944 parent_transid=31136 key expected=(1166224539648,169,0) has=(1995819548672,169,0)
[  535.493265] BTRFS info (device dm-0): scrub: not finished on devid 3 with status: -117
[  543.833178] perf: interrupt took too long (2529 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
[  552.823060] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  552.823076] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  552.824526] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  552.824542] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  552.824546] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  552.826281] BTRFS error (device dm-0): tree first key mismatch detected, bytenr=1166185938944 parent_transid=31136 key expected=(1166224539648,169,0) has=(1995819548672,169,0)
[  552.826296] BTRFS error (device dm-0): tree first key mismatch detected, bytenr=1166185938944 parent_transid=31136 key expected=(1166224539648,169,0) has=(1995819548672,169,0)
[  552.826302] BTRFS error (device dm-0): tree first key mismatch detected, bytenr=1166185938944 parent_transid=31136 key expected=(1166224539648,169,0) has=(1995819548672,169,0)
[  574.429895] perf: interrupt took too long (3170 > 3161), lowering kernel.perf_event_max_sample_rate to 63000
[  591.481255] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  591.481272] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  591.481277] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  591.481345] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  591.481353] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  591.481357] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.620904] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.620921] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.620926] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.620994] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.621002] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.621006] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.621074] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.621082] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  628.621086] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  630.721933] perf: interrupt took too long (3997 > 3962), lowering kernel.perf_event_max_sample_rate to 50000
[  664.267052] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267069] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267074] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267138] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267147] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267151] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267214] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267221] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  664.267225] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  699.968367] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  699.968384] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  699.968389] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  699.968458] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  699.968466] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  699.968470] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475258] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475276] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475280] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475514] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475524] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475529] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475741] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475754] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.475759] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[  735.574047] BTRFS error (device dm-0): parent transid verify failed on 1166187577344 wanted 31136 found 31158
[  735.590944] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.590969] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591162] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591176] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591259] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591269] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591277] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591343] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591353] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.591359] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595330] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595350] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595358] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595422] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595432] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595439] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595492] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595502] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595509] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595554] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595563] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595570] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595618] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595627] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595634] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595678] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595687] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.595694] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606269] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606291] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606299] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606368] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606379] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606386] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606454] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606466] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606474] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606543] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606553] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.606560] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.626099] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.626121] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.626129] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635719] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635738] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635746] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635812] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635822] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635829] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635890] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635900] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635907] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.635928] BTRFS error (device dm-0): parent transid verify failed on 1166187560960 wanted 31136 found 31158
[  735.656523] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.656545] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.656553] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.656699] BTRFS error (device dm-0): parent transid verify failed on 1166187577344 wanted 31158 found 31136
[  735.656965] BTRFS error (device dm-0): parent transid verify failed on 1166187560960 wanted 31136 found 30938
[  735.670736] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.670758] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.670766] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.670974] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.670989] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.670996] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.671066] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.671076] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.671083] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.671133] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.671142] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.671141] BTRFS error (device dm-0): parent transid verify failed on 1166187577344 wanted 31158 found 31136
[  735.671151] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.671158] BTRFS: error (device dm-0) in __btrfs_free_extent:3079: errno=-5 IO failure
[  735.671167] BTRFS info (device dm-0): forced readonly
[  735.671174] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2159: errno=-5 IO failure
[  735.671195] BTRFS warning (device dm-0): Skipping commit of aborted transaction.
[  735.671200] BTRFS: error (device dm-0) in cleanup_transaction:1974: errno=-5 IO failure
[  735.673614] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.673634] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.673643] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.673918] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.673935] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  735.673942] BTRFS error (device dm-0): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[  737.306591] BTRFS info (device dm-0): scrub: not finished on devid 2 with status: -125

[-- Attachment #5: debian_unmount --]
[-- Type: text/plain, Size: 8081 bytes --]

[  456.465127] BTRFS error (device dm-0): tree block 1166181154816 owner 1 already locked by pid=1300, extent tree corruption detected
[  456.465163] BTRFS warning (device dm-0): Skipping commit of aborted transaction.
[  456.465168] ------------[ cut here ]------------
[  456.465171] BTRFS: Transaction aborted (error -117)
[  456.465260] WARNING: CPU: 0 PID: 1300 at fs/btrfs/transaction.c:1974 btrfs_commit_transaction.cold+0x22f/0x2f9 [btrfs]
[  456.465455] Modules linked in: crypto_simd cryptd xts ecb dm_crypt dm_mod i915 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio video snd_hda_intel snd_intel_dspcfg ttm snd_intel_sdw_acpi drm_kms_helper snd_hda_codec snd_hda_core iTCO_wdt intel_pmc_bxt snd_hwdep cec iTCO_vendor_support watchdog rc_core snd_pcm ppdev i2c_algo_bit snd_timer sg parport_pc snd parport soundcore evdev coretemp rng_core pcspkr serio_raw button acpi_cpufreq drm binfmt_misc fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 uas usb_storage btrfs blake2b_generic xor raid6_pq zstd_compress libcrc32c crc32c_generic hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_common ata_generic ata_piix uhci_hcd libata ehci_pci ehci_hcd usbcore scsi_mod r8169 psmouse i2c_i801 realtek mdio_devres i2c_smbus scsi_common lpc_ich libphy usb_common
[  456.465617] CPU: 0 PID: 1300 Comm: umount Tainted: G        W         5.17.0-1-amd64 #1  Debian 5.17.3-1
[  456.465625] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD MS-7529/G31TM-P35 (MS-7529), BIOS V4.4 11/25/2009
[  456.465628] RIP: 0010:btrfs_commit_transaction.cold+0x22f/0x2f9 [btrfs]
[  456.465786] Code: 4b 99 e7 e9 1d 55 f5 ff 49 8b 55 28 49 8b 45 30 48 89 42 08 48 89 10 e9 fb fe ff ff 89 de 48 c7 c7 90 09 75 c0 e8 39 99 fc e7 <0f> 0b e9 31 fe ff ff 48 8b 7d 50 89 da 48 c7 c6 c0 09 75 c0 e8 01
[  456.465792] RSP: 0018:ffffb1cec0ccfd80 EFLAGS: 00010282
[  456.465798] RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000000
[  456.465802] RDX: 0000000000000001 RSI: ffffffffa8f52a0e RDI: 00000000ffffffff
[  456.465806] RBP: ffff8eca83c6e000 R08: 0000000000000000 R09: ffffb1cec0ccfbb0
[  456.465810] R10: ffffb1cec0ccfba8 R11: ffffffffa96d1568 R12: ffff8eca83c6df50
[  456.465814] R13: ffff8eca8474d400 R14: ffff8eca81a3f000 R15: ffff8eca83c6df50
[  456.465818] FS:  00007fa2225b5800(0000) GS:ffff8ecaabc00000(0000) knlGS:0000000000000000
[  456.465823] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  456.465828] CR2: 00007f952eb07c10 CR3: 00000001088d8000 CR4: 00000000000006f0
[  456.465832] Call Trace:
[  456.465839]  <TASK>
[  456.465847]  close_ctree+0x146/0x421 [btrfs]
[  456.465968]  ? evict_inodes+0x166/0x1b0
[  456.465980]  generic_shutdown_super+0x74/0x120
[  456.465988]  kill_anon_super+0x14/0x30
[  456.465994]  btrfs_kill_super+0x12/0x20 [btrfs]
[  456.466092]  deactivate_locked_super+0x31/0xa0
[  456.466099]  cleanup_mnt+0x131/0x190
[  456.466106]  task_work_run+0x5c/0x90
[  456.466113]  exit_to_user_mode_prepare+0x1fe/0x210
[  456.466120]  syscall_exit_to_user_mode+0x18/0x40
[  456.466126]  do_syscall_64+0x48/0xc0
[  456.466133]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  456.466139] RIP: 0033:0x7fa2227fb7a7
[  456.466146] Code: 36 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 99 36 0d 00 f7 d8 64 89 02 b8
[  456.466151] RSP: 002b:00007fffc698fe38 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[  456.466157] RAX: 0000000000000000 RBX: 00007fa22293e264 RCX: 00007fa2227fb7a7
[  456.466161] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000562ea7552d20
[  456.466165] RBP: 0000562ea7552af0 R08: 0000000000000000 R09: 00007fffc698ebe0
[  456.466169] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[  456.466172] R13: 0000562ea7552d20 R14: 0000562ea7552c08 R15: 0000000000000000
[  456.466179]  </TASK>
[  456.466181] ---[ end trace 0000000000000000 ]---
[  456.466187] BTRFS: error (device dm-0) in cleanup_transaction:1974: errno=-117 Filesystem corrupted
[  456.466198] BTRFS info (device dm-0): forced readonly
[  456.466211] BTRFS error (device dm-0): commit super ret -117
[  456.568108] ------------[ cut here ]------------
[  456.568114] WARNING: CPU: 0 PID: 1300 at fs/btrfs/space-info.h:110 btrfs_space_info_update_bytes_may_use+0xf1/0x100 [btrfs]
[  456.568207] Modules linked in: crypto_simd cryptd xts ecb dm_crypt dm_mod i915 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio video snd_hda_intel snd_intel_dspcfg ttm snd_intel_sdw_acpi drm_kms_helper snd_hda_codec snd_hda_core iTCO_wdt intel_pmc_bxt snd_hwdep cec iTCO_vendor_support watchdog rc_core snd_pcm ppdev i2c_algo_bit snd_timer sg parport_pc snd parport soundcore evdev coretemp rng_core pcspkr serio_raw button acpi_cpufreq drm binfmt_misc fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 uas usb_storage btrfs blake2b_generic xor raid6_pq zstd_compress libcrc32c crc32c_generic hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_common ata_generic ata_piix uhci_hcd libata ehci_pci ehci_hcd usbcore scsi_mod r8169 psmouse i2c_i801 realtek mdio_devres i2c_smbus scsi_common lpc_ich libphy usb_common
[  456.568295] CPU: 0 PID: 1300 Comm: umount Tainted: G        W         5.17.0-1-amd64 #1  Debian 5.17.3-1
[  456.568299] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD MS-7529/G31TM-P35 (MS-7529), BIOS V4.4 11/25/2009
[  456.568301] RIP: 0010:btrfs_space_info_update_bytes_may_use+0xf1/0x100 [btrfs]
[  456.568372] Code: c5 d2 f3 ff 65 ff 0d ee f4 8f 3f 0f 85 37 ff ff ff 0f 1f 44 00 00 e9 2d ff ff ff 48 89 da 48 f7 da 48 39 c2 0f 86 2d ff ff ff <0f> 0b 31 db 48 89 5d 28 5b 5d 41 5c c3 66 90 0f 1f 44 00 00 41 56
[  456.568375] RSP: 0018:ffffb1cec0ccfd50 EFLAGS: 00010206
[  456.568378] RAX: 000000001fffc000 RBX: ffffffffe0000000 RCX: 0000000000000000
[  456.568381] RDX: 0000000020000000 RSI: ffff8eca84a44200 RDI: ffff8eca81a3f000
[  456.568383] RBP: ffff8eca84a44200 R08: 0000000020000000 R09: 0000000080100004
[  456.568385] R10: ffff8eca84618c00 R11: 0000000000036800 R12: ffff8eca81a3f000
[  456.568387] R13: 0000000000000000 R14: ffff8eca84a44200 R15: 0000000000000000
[  456.568390] FS:  00007fa2225b5800(0000) GS:ffff8ecaabc00000(0000) knlGS:0000000000000000
[  456.568392] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  456.568395] CR2: 00007f952eb07c10 CR3: 00000001088d8000 CR4: 00000000000006f0
[  456.568398] Call Trace:
[  456.568402]  <TASK>
[  456.568407]  btrfs_block_rsv_release+0x1ac/0x1d0 [btrfs]
[  456.568477]  btrfs_release_global_block_rsv+0x1e/0xc0 [btrfs]
[  456.568545]  btrfs_free_block_groups+0x2d2/0x390 [btrfs]
[  456.568613]  close_ctree+0x2bf/0x421 [btrfs]
[  456.568679]  ? evict_inodes+0x166/0x1b0
[  456.568686]  generic_shutdown_super+0x74/0x120
[  456.568691]  kill_anon_super+0x14/0x30
[  456.568694]  btrfs_kill_super+0x12/0x20 [btrfs]
[  456.568748]  deactivate_locked_super+0x31/0xa0
[  456.568752]  cleanup_mnt+0x131/0x190
[  456.568755]  task_work_run+0x5c/0x90
[  456.568760]  exit_to_user_mode_prepare+0x1fe/0x210
[  456.568764]  syscall_exit_to_user_mode+0x18/0x40
[  456.568768]  do_syscall_64+0x48/0xc0
[  456.568772]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  456.568776] RIP: 0033:0x7fa2227fb7a7
[  456.568780] Code: 36 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 99 36 0d 00 f7 d8 64 89 02 b8
[  456.568783] RSP: 002b:00007fffc698fe38 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[  456.568786] RAX: 0000000000000000 RBX: 00007fa22293e264 RCX: 00007fa2227fb7a7
[  456.568789] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000562ea7552d20
[  456.568791] RBP: 0000562ea7552af0 R08: 0000000000000000 R09: 00007fffc698ebe0
[  456.568793] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[  456.568795] R13: 0000562ea7552d20 R14: 0000562ea7552c08 R15: 0000000000000000
[  456.568798]  </TASK>
[  456.568799] ---[ end trace 0000000000000000 ]---

[-- Attachment #6: fedora_check --]
[-- Type: text/plain, Size: 195 bytes --]

# btrfs check /dev/mapper/ST3000VN007-############### 
Opening filesystem to check...
ERROR: root [7 0] level 1 does not match 2

ERROR: could not setup csum tree
ERROR: cannot open file system


[-- Attachment #7: fedora_info --]
[-- Type: text/plain, Size: 956 bytes --]

#uname -a
Linux localhost-live 5.17.5-300.fc36.x86_64 #1 SMP PREEMPT Thu Apr 28 15:51:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
#btrfs --version
btrfs-progs v5.16.2
#btrfs fi show
Label: none  uuid: 8039cd28-5e4c-46a7-8a5f-afb57ea68c08
	Total devices 2 FS bytes used 772.76GiB
	devid    2 size 1.82TiB used 334.00GiB path /dev/mapper/ST2000DL003-###############
	devid    3 size 1.82TiB used 661.00GiB path /dev/mapper/ST3000VN007-###############
#btrfs fi df /mnt/sd/
Data, RAID1: total=772.00GiB, used=771.22GiB
Data, single: total=1.00GiB, used=2.25MiB
System, RAID1: total=32.00MiB, used=96.00KiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=3.00GiB, used=1.54GiB
Metadata, single: total=1.00GiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
WARNING:   Data: single, raid1
WARNING:   Metadata: single, raid1
WARNING:   System: single, raid1

[-- Attachment #8: fedora_scrub_1 --]
[-- Type: text/plain, Size: 3979 bytes --]

[  756.226116] BTRFS error (device dm-2): tree first key mismatch detected, bytenr=1166186053632 parent_transid=31136 key expected=(1995819548672,169,0) has=(1749212258304,168,262144)
[  756.226136] BTRFS error (device dm-2): tree first key mismatch detected, bytenr=1166186053632 parent_transid=31136 key expected=(1995819548672,169,0) has=(1749212258304,168,262144)
[  756.229237] BTRFS info (device dm-2): scrub: not finished on devid 2 with status: -117
[  777.876662] ------------[ cut here ]------------
[  777.876670] BTRFS: Transaction aborted (error -17)
[  777.876714] WARNING: CPU: 1 PID: 3469 at fs/btrfs/extent-tree.c:2159 btrfs_run_delayed_refs+0x196/0x1e0
[  777.876729] Modules linked in: libfc scsi_transport_fc iscsi_ibft sha256_ssse3 dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 uinput snd_seq_dummy snd_hrtimer nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt intel_pmc_bxt ppdev gpio_ich iTCO_vendor_support i915 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep coretemp snd_seq snd_seq_device snd_pcm pcspkr snd_timer video parport_pc snd ttm parport soundcore lpc_ich i2c_i801 i2c_smbus acpi_cpufreq zram isofs squashfs serio_raw ata_generic pata_acpi r8169 uas usb_storage sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3
[  777.876968]  mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop ip6_tables ip_tables ipmi_devintf ipmi_msghandler fuse
[  777.877004] CPU: 1 PID: 3469 Comm: btrfs-transacti Not tainted 5.17.5-300.fc36.x86_64 #1
[  777.877013] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD MS-7529/G31TM-P35 (MS-7529), BIOS V4.4 11/25/2009
[  777.877017] RIP: 0010:btrfs_run_delayed_refs+0x196/0x1e0
[  777.877030] Code: 48 8d 91 48 0a 00 00 f0 48 0f ba 2a 03 72 20 83 f8 fb 74 39 83 f8 e2 74 34 89 c6 48 c7 c7 60 11 65 86 89 04 24 e8 e3 0f 7d 00 <0f> 0b 8b 04 24 89 c1 ba 6f 08 00 00 48 89 df 89 04 24 48 c7 c6 80
[  777.877036] RSP: 0018:ffffa5c68258bdb0 EFLAGS: 00010282
[  777.877042] RAX: 0000000000000026 RBX: ffff92aa82302750 RCX: 0000000000000000
[  777.877046] RDX: 0000000000000001 RSI: ffffffff86665ad5 RDI: 00000000ffffffff
[  777.877050] RBP: ffff92aa4ae62b78 R08: 0000000000000000 R09: ffffa5c68258bbf0
[  777.877054] R10: ffffa5c68258bbe8 R11: 0000000000000003 R12: ffff92aac3a57060
[  777.877057] R13: ffff92aac3a57010 R14: ffff92aac3a57018 R15: ffff92aa4ae62a00
[  777.877062] FS:  0000000000000000(0000) GS:ffff92aaebc80000(0000) knlGS:0000000000000000
[  777.877067] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  777.877071] CR2: 00007fb85e21b000 CR3: 0000000101f42000 CR4: 00000000000006e0
[  777.877076] Call Trace:
[  777.877081]  <TASK>
[  777.877091]  commit_cowonly_roots+0x1e4/0x240
[  777.877102]  btrfs_commit_transaction+0x4e9/0xb00
[  777.877110]  ? start_transaction+0xc3/0x5e0
[  777.877118]  transaction_kthread+0x13a/0x1a0
[  777.877125]  ? btrfs_cleanup_transaction.isra.0+0x540/0x540
[  777.877132]  kthread+0xe9/0x110
[  777.877139]  ? kthread_complete_and_exit+0x20/0x20
[  777.877145]  ret_from_fork+0x22/0x30
[  777.877157]  </TASK>
[  777.877159] ---[ end trace 0000000000000000 ]---
[  777.877167] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2159: errno=-17 Object already exists
[  777.877179] BTRFS info (device dm-2): forced readonly
[  777.877206] BTRFS warning (device dm-2): Skipping commit of aborted transaction.
[  777.877213] BTRFS: error (device dm-2) in cleanup_transaction:1974: errno=-17 Object already exists
[  777.877379] BTRFS info (device dm-2): scrub: finished on devid 3 with status: 0

[-- Attachment #9: fedora_scrub_2 --]
[-- Type: text/plain, Size: 16052 bytes --]

[ 1351.928362] EXT4-fs (sdc2): warning: mounting fs with errors, running e2fsck is recommended
[ 1351.937403] EXT4-fs (sdc2): mounted filesystem with ordered data mode. Quota mode: none.
[ 1628.913429] BTRFS info (device dm-2): flagging fs with big metadata feature
[ 1628.913442] BTRFS info (device dm-2): disk space caching is enabled
[ 1628.913446] BTRFS info (device dm-2): has skinny extents
[ 1628.944147] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186610688 level expected=0 has=1
[ 1628.944160] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186610688 level expected=0 has=1
[ 1628.945575] BTRFS error (device dm-2): open_ctree failed
[ 1643.725577] BTRFS info (device dm-2): flagging fs with big metadata feature
[ 1643.725590] BTRFS info (device dm-2): disk space caching is enabled
[ 1643.725593] BTRFS info (device dm-2): has skinny extents
[ 1676.608110] EXT4-fs (sdc2): error count since last fsck: 2
[ 1676.608141] EXT4-fs (sdc2): initial error at time 1653286599: ext4_update_backup_sb:165
[ 1676.608155] EXT4-fs (sdc2): last error at time 1653286599: ext4_update_backup_sb:165
[ 1684.470383] BTRFS info (device dm-2): scrub: started on devid 2
[ 1684.504477] BTRFS info (device dm-2): scrub: started on devid 3
[ 1698.207209] BTRFS error (device dm-2): tree first key mismatch detected, bytenr=1166186053632 parent_transid=31136 key expected=(1995819548672,169,0) has=(1749212258304,168,262144)
[ 1698.207241] BTRFS error (device dm-2): tree first key mismatch detected, bytenr=1166186053632 parent_transid=31136 key expected=(1995819548672,169,0) has=(1749212258304,168,262144)
[ 1698.230073] BTRFS info (device dm-2): scrub: not finished on devid 2 with status: -117
[ 1717.859075] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1717.859091] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1717.861272] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1717.861287] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1717.861292] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1749.107561] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1749.107577] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1749.107580] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1749.107646] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1749.107653] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1749.107656] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1784.954360] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1784.954374] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1784.954378] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1784.954447] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1784.954454] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1784.954457] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.079873] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.079887] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.079890] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.079955] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.079961] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.079964] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.080032] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.080038] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1819.080041] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.460985] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.460999] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.461002] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.461064] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.461070] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.461073] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.461134] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.461140] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1849.461143] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1884.119360] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1884.119375] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1884.119378] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1884.119596] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1884.119607] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1884.119610] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166187315200 level expected=0 has=1
[ 1884.225293] BTRFS error (device dm-2): parent transid verify failed on 1166187577344 wanted 31136 found 31152
[ 1884.248369] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.248388] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.248611] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.248615] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.248622] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.248623] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.251502] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.251517] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.251522] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.261065] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.261082] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.261088] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262279] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262296] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262302] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262562] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262575] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262580] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262641] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262650] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262655] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262703] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262711] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262716] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262904] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262951] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.262959] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264142] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264194] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264306] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264318] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264471] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264483] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264489] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264574] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264581] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.264587] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.293254] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.293270] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.293277] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302107] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302124] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302130] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302568] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302582] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302588] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302658] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302667] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302673] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302693] BTRFS error (device dm-2): parent transid verify failed on 1166187560960 wanted 31136 found 31152
[ 1884.302803] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302819] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302825] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.302871] BTRFS error (device dm-2): parent transid verify failed on 1166187577344 wanted 31152 found 31136
[ 1884.319643] BTRFS error (device dm-2): tree first key mismatch detected, bytenr=1166186512384 parent_transid=31136 key expected=(1752896905216,168,57344) has=(1761020604416,168,4096)
[ 1884.319664] BTRFS error (device dm-2): tree first key mismatch detected, bytenr=1166186512384 parent_transid=31136 key expected=(1752896905216,168,57344) has=(1761020604416,168,4096)
[ 1884.344899] BTRFS error (device dm-2): parent transid verify failed on 1166187560960 wanted 31136 found 30938
[ 1884.345484] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345502] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345508] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345594] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345605] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345610] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345682] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345691] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345697] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345749] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345757] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345762] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345851] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345862] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.345867] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.346074] BTRFS error (device dm-2): parent transid verify failed on 1166187577344 wanted 31152 found 31136
[ 1884.346097] BTRFS: error (device dm-2) in __btrfs_free_extent:3079: errno=-5 IO failure
[ 1884.346108] BTRFS info (device dm-2): forced readonly
[ 1884.346115] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2159: errno=-5 IO failure
[ 1884.346146] BTRFS warning (device dm-2): Skipping commit of aborted transaction.
[ 1884.346151] BTRFS: error (device dm-2) in cleanup_transaction:1974: errno=-5 IO failure
[ 1884.366267] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.366289] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.366296] BTRFS error (device dm-2): tree level mismatch detected, bytenr=1166186938368 level expected=0 has=1
[ 1884.998536] BTRFS info (device dm-2): scrub: not finished on devid 3 with status: -125

[-- Attachment #10: old_completing_check --]
[-- Type: text/plain, Size: 1657 bytes --]

Opening filesystem to check...
Checking filesystem on /dev/mapper/ST2000DL003-###############
UUID: 8039cd28-5e4c-46a7-8a5f-############
[1/7] checking root items
[2/7] checking extents
Chunk[256, 228, 2000512090112] stripe[2, 1048576] is not found in dev extent
Chunk[256, 228, 2001585831936] stripe[2, 313542049792] is not found in dev extent
Chunk[256, 228, 2001619386368] stripe[2, 832159350784] is not found in dev extent
Device extent[3, 831110774784, 1073741824] didn't find the relative chunk.
Device extent[3, 833258258432, 33554432] didn't find the relative chunk.
Device extent[3, 833291812864, 1073741824] didn't find the relative chunk.
Dev extent's total-byte(832183468032) is not equal to byte-used(358629769216) in dev[1, 216, 2]
Dev extent's total-byte(834364506112) is not equal to byte-used(709743345664) in dev[1, 216, 3]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
block group 1166181138432 has wrong amount of free space, free space cache has 17334272 block group has 17350656
failed to load free space cache for block group 1166181138432
there is no free space entry for 1999438381056-1999438479360
cache appears valid but isn't 1999438348288
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 829745819648 bytes used, error(s) found
total csum bytes: 808489372
total tree bytes: 1653473280
total fs tree bytes: 736575488
total extent tree bytes: 64471040
btree space waste bytes: 192745626
file data blocks allocated: 835171110912
 referenced 827757674496

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Update: I started copying out the data, for a lot of files cp was 
reporting I/O errors but I don't see any actual I/O errors to the drives 
on the kernel log. After this was done nothing could be read from the 
filesystem, all directories I tried were reporting i/o errors (they 
where working before the copy) and now I cannot mount it at all anymore.

I'm probably going to wait a few days if anyone wants me to run some 
command on whatever is left of the filesystem, count my losses and 
format it.

At time 19846.218470 on the kernel log I mount the filesystem I was 
going to copy the data to and after that the transfer happens.

By the way from some checks on the old kernel logs from when the first 
balance command was run it seems that all the weird I/O errors that I've 
been having since started right after it. Maybe that caused some 
corruption.

[-- Attachment #2: transfer_dmesg.xz --]
[-- Type: application/x-xz, Size: 30744 bytes --]

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Chris Murphy]

I suggest mounting with "mount -o ro,rescue=all" and copying
everything you can out and check the most important files for
corruption. At least this will get the data, such as it is, out.
Hopefully. The rescue=all option includes ignoring data checksums so
it *will* permit the copying of corrupt data. So you'll want to keep
the data embargoed. This is a bit painful and tedious but it's a good
early attempt to have available in case subsequent attempts don't work
at all or are even worse.

Do you have a complete dmesg that shows boot, mount, and the kernel
errors while copying? This would be useful to see which device has all
this corruption and if fixups are even being attempted.

From one of your attached files:

>Total devices 2 FS bytes used 772.76GiB
>devid    2 size 1.82TiB used 334.00GiB path /dev/mapper/ST2000DL003-###############
>devid    3 size 1.82TiB used 661.00GiB path /dev/mapper/ST3000VN007-###############

This doesn't list a 3rd device so it suggests it's a 2x device raid1. However:

>#btrfs fi df /mnt/sd/
>Data, RAID1: total=772.00GiB, used=771.22GiB
>Data, single: total=1.00GiB, used=2.25MiB
>System, RAID1: total=32.00MiB, used=96.00KiB
>System, single: total=32.00MiB, used=48.00KiB
>Metadata, RAID1: total=3.00GiB, used=1.54GiB
>Metadata, single: total=1.00GiB, used=0.00B

This is not good. Some of the data and some of the metadata
(specifically system profile which is the chunk tree) is only
available on one drive and I can't tell from this if it's on a drive
that is missing or is spewing errors. Anything that has a single copy
that's also damaged, cannot be recovered. Unfortunately this file
system is not completely raid1 and that's likely one source of the
problem. The chunk tree is really critical so if any part of it is bad
and not redundant (no good copy) the file system is not likely
repairable. Get the data out as best you can. If rescue=all mount
option doesn't work, the next opportunity is btrfs restore, but it too
depends on the chunk tree being intact. There is a 'btrfs restore
chunk-tree' option that will scan all the drives looking for plausible
fragments of the chunk tree to try and recover it but it takes a long
time (hours).

48KiB of chunk tree, if it's corrupt, is quite a lot and might prevent
quite a lot of recovery. Some older kernels would create single
profile chunks when a raid1 file system was mounted in degraded,rw
mode with a missing device. This happens silently. And then when the
raid1 is back to full strength again, there's no automatic conversion
or even a warning by the kernel that this critical metadata isn't
redundant still. The burden right now is unfortunately on the user to
identify this reduction in redundancy and make sure to do a filtered
balance to convert the single chunks into raid1 chunks.


--
Chris Murphy

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

thanks a lot for reading into this

On 2022-05-24 20:11, Chris Murphy wrote:
> I suggest mounting with "mount -o ro,rescue=all"
Thanks a lot, with this command I was able to mount the filesystem again 
and retrieve a lot more data!
Only a very small percentage of the files (that i could check) were 
corrupt.

> Do you have a complete dmesg that shows boot, mount, and the kernel
> errors while copying?
If you mean the copying of the data mentioned on my update email then 
yes, its the attached file on this one.


> From one of your attached files:
> 
>> Total devices 2 FS bytes used 772.76GiB
>> devid    2 size 1.82TiB used 334.00GiB path 
>> /dev/mapper/ST2000DL003-###############
>> devid    3 size 1.82TiB used 661.00GiB path 
>> /dev/mapper/ST3000VN007-###############
> 
> This doesn't list a 3rd device so it suggests it's a 2x device raid1. 
> However:

I had (i think) successfully removed the failing drive with devid 1

> 
>> #btrfs fi df /mnt/sd/
>> Data, RAID1: total=772.00GiB, used=771.22GiB
>> Data, single: total=1.00GiB, used=2.25MiB
>> System, RAID1: total=32.00MiB, used=96.00KiB
>> System, single: total=32.00MiB, used=48.00KiB
>> Metadata, RAID1: total=3.00GiB, used=1.54GiB
>> Metadata, single: total=1.00GiB, used=0.00B
> 
> This is not good. Some of the data and some of the metadata
> (specifically system profile which is the chunk tree) is only
> available on one drive
I had that issue with single chunks, run the command to make it all 
raid1, run a scrub some checks messed around a tiny bit, most likely 
mounted with -o degraded in the process and they appeared again ( on 
linux 5.17.0 ).


> Some older kernels would create single
> profile chunks when a raid1 file system was mounted in degraded,rw
> mode with a missing device. This happens silently.

I had mounted the fs with -o degraded and one drive a couple of times 
just as a sanity check to make sure the data really is in both drives, i 
assume this would mount it rw and fall into the category you described. 
The first chunks were created before having updated to debian testing 
under kernel 5.10.0-11 but the same thing happened after updating to 
testing
I think i had tried to mount ro,degraded and it failed but i'm not sure.

Once again thank you so much for suggesting -o rescue=all, i had 
previously managed to recover around 300G and now i think i got the full 
~800G with a very small amount of corrupted files !!

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

Seems like attachments still didn't get attached, sorry

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Forza]

On 2022-05-27 17:13, efkf@firemail.cc wrote:
...

>> Some older kernels would create single
>> profile chunks when a raid1 file system was mounted in degraded,rw
>> mode with a missing device. This happens silently.
> 
> I had mounted the fs with -o degraded and one drive a couple of times 
> just as a sanity check to make sure the data really is in both drives, i 
> assume this would mount it rw and fall into the category you described. 

With Btrfs, this is not good practice. If you mount RAID mirrors 
independently as degraded, the metadata could be updated differently and 
when you combine the drives again, the data would not be the same on 
each device - which would lead to corruption. This is true even for ro 
mounts. ro is a Linux VFS thing, Btrfs will still write to the disk for 
its internal things.


> The first chunks were created before having updated to debian testing 
> under kernel 5.10.0-11 but the same thing happened after updating to 
> testing
> I think i had tried to mount ro,degraded and it failed but i'm not sure.
> 
> Once again thank you so much for suggesting -o rescue=all, i had 
> previously managed to recover around 300G and now i think i got the full 
> ~800G with a very small amount of corrupted files !!

Good to hear that you got most of your data back. :)

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

[-- Attachment #1: Type: text/plain, Size: 1195 bytes --]

On 5/27/22 16:25, Forza wrote
> On 2022-05-27 17:13, efkf@firemail.cc wrote:
>> I had mounted the fs with -o degraded and one drive a couple of times 
>> just as a sanity check to make sure the data really is in both drives, 
>> i assume this would mount it rw and fall into the category you described. 
> 
> With Btrfs, this is not good practice. If you mount RAID mirrors 
> independently as degraded, the metadata could be updated differently and 
> when you combine the drives again, the data would not be the same on 
> each device - which would lead to corruption. This is true even for ro 
> mounts. ro is a Linux VFS thing, Btrfs will still write to the disk for 
> its internal things.

I don't know much about advanced filesystems but I can imagine scenarious where it would be beneficial to reintroduce an old drive into an array that maybe even has changed without whiping it, maybe it's the only one storing the intact copy of an old file.
Anyway, is there a way to check the data is really redundant without trusting the filesystem telling me it's so?

> Good to hear that you got most of your data back. :)
:)

I tried another client now, hopefully the attachment has gone through

[-- Attachment #2: transfer_dmesg_full.xz --]
[-- Type: application/x-xz, Size: 44660 bytes --]

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Forza]

---- From: efkf <efkf@firemail.cc> -- Sent: 2022-05-27 - 18:28 ----

> On 5/27/22 16:25, Forza wrote
>> On 2022-05-27 17:13, efkf@firemail.cc wrote:
>>> I had mounted the fs with -o degraded and one drive a couple of times 
>>> just as a sanity check to make sure the data really is in both drives, 
>>> i assume this would mount it rw and fall into the category you described. 
>> 
>> With Btrfs, this is not good practice. If you mount RAID mirrors 
>> independently as degraded, the metadata could be updated differently and 
>> when you combine the drives again, the data would not be the same on 
>> each device - which would lead to corruption. This is true even for ro 
>> mounts. ro is a Linux VFS thing, Btrfs will still write to the disk for 
>> its internal things.
> 
> I don't know much about advanced filesystems but I can imagine scenarious where it would be beneficial to reintroduce an old drive into an array that maybe even has changed without whiping it, 

The correct way to re-introduce an old device to Btrfs is to first wipe it with 'wipefs' first. 

I do agree that this is perhaps counter-intuitive as some other raid systems allows to re-add old disks. Btrfs could handle this better, and I know there was a discussion on #btrfs about this a while back. 

>maybe it's the only one storing the intact copy of an old file. 

You should mount this drive on another system to recover this file. It is likely to cause serious corruption to try to introduce a drive after changes has been made. (perhaps there is some protection against this, but I've not heard of it). 

> Anyway, is there a way to check the data is really redundant without trusting the filesystem telling me it's so?

Yes, you use 'btrfs scrub' to read all data and metadata blocks from all devices and compare the checksums. If there are problems, scrub will tell you. 

https://btrfs.readthedocs.io/en/latest/btrfs-scrub.html
https://wiki.tnonline.net/w/Btrfs/Scrub

> 
>> Good to hear that you got most of your data back. :)
> :)
> 
> I tried another client now, hopefully the attachment has gone through

It did. Lots of mismatching metadata. Not sure that kind of problem can be fixed. 

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 2349 bytes --]

Hi Chris, Efkf, Duncan, and anyone else reading this,

Chris Murphy <lists@colorremedies.com> writes:

>>#btrfs fi df /mnt/sd/
>>Data, RAID1: total=772.00GiB, used=771.22GiB
>>Data, single: total=1.00GiB, used=2.25MiB
>>System, RAID1: total=32.00MiB, used=96.00KiB
>>System, single: total=32.00MiB, used=48.00KiB
>>Metadata, RAID1: total=3.00GiB, used=1.54GiB
>>Metadata, single: total=1.00GiB, used=0.00B
>
> This is not good. Some of the data and some of the metadata
> (specifically system profile which is the chunk tree) is only
> available on one drive and I can't tell from this if it's on a drive
> that is missing or is spewing errors. Anything that has a single copy
> that's also damaged, cannot be recovered. Unfortunately this file
> system is not completely raid1 and that's likely one source of the
> problem. The chunk tree is really critical so if any part of it is bad
> and not redundant (no good copy) the file system is not likely
> repairable. Get the data out as best you can. If rescue=all mount
> option doesn't work, the next opportunity is btrfs restore, but it too
> depends on the chunk tree being intact. There is a 'btrfs restore
> chunk-tree' option that will scan all the drives looking for plausible
> fragments of the chunk tree to try and recover it but it takes a long
> time (hours).
>
> 48KiB of chunk tree, if it's corrupt, is quite a lot and might prevent
> quite a lot of recovery. Some older kernels would create single
> profile chunks when a raid1 file system was mounted in degraded,rw
> mode with a missing device. This happens silently. And then when the
> raid1 is back to full strength again, there's no automatic conversion
> or even a warning by the kernel that this critical metadata isn't
> redundant still. The burden right now is unfortunately on the user to
> identify this reduction in redundancy and make sure to do a filtered
> balance to convert the single chunks into raid1 chunks.
>

I reported this issue "Wed, 02 Mar 2016 20:25:46 -0500" with Subject
"incomplete conversion to RAID1?", and it now looks that there's
evidence that this bug isn't harmless after all.

Efkf, would you please confirm if the filesystem was created with Linux
and btrfs-progs 5.10.x? (please keep me in CC)

If anyone knows if this issue was fixed for 5.15, please share the good
news!


Regards,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Forza]

---- From: Nicholas D Steeves <nsteeves@gmail.com> -- Sent: 2022-05-28 - 22:20 ----

> Hi Chris, Efkf, Duncan, and anyone else reading this,
> 
> Chris Murphy <lists@colorremedies.com> writes:
> 
>>>#btrfs fi df /mnt/sd/
>>>Data, RAID1: total=772.00GiB, used=771.22GiB
>>>Data, single: total=1.00GiB, used=2.25MiB
>>>System, RAID1: total=32.00MiB, used=96.00KiB
>>>System, single: total=32.00MiB, used=48.00KiB
>>>Metadata, RAID1: total=3.00GiB, used=1.54GiB
>>>Metadata, single: total=1.00GiB, used=0.00B
>>
>> This is not good. Some of the data and some of the metadata
>> (specifically system profile which is the chunk tree) is only
>> available on one drive and I can't tell from this if it's on a drive
>> that is missing or is spewing errors. Anything that has a single copy
>> that's also damaged, cannot be recovered. Unfortunately this file
>> system is not completely raid1 and that's likely one source of the
>> problem. The chunk tree is really critical so if any part of it is bad
>> and not redundant (no good copy) the file system is not likely
>> repairable. Get the data out as best you can. If rescue=all mount
>> option doesn't work, the next opportunity is btrfs restore, but it too
>> depends on the chunk tree being intact. There is a 'btrfs restore
>> chunk-tree' option that will scan all the drives looking for plausible
>> fragments of the chunk tree to try and recover it but it takes a long
>> time (hours).
>>
>> 48KiB of chunk tree, if it's corrupt, is quite a lot and might prevent
>> quite a lot of recovery. Some older kernels would create single
>> profile chunks when a raid1 file system was mounted in degraded,rw
>> mode with a missing device. This happens silently. And then when the
>> raid1 is back to full strength again, there's no automatic conversion
>> or even a warning by the kernel that this critical metadata isn't
>> redundant still. The burden right now is unfortunately on the user to
>> identify this reduction in redundancy and make sure to do a filtered
>> balance to convert the single chunks into raid1 chunks.
>>
> 
> I reported this issue "Wed, 02 Mar 2016 20:25:46 -0500" with Subject
> "incomplete conversion to RAID1?", and it now looks that there's
> evidence that this bug isn't harmless after all.
> 
> Efkf, would you please confirm if the filesystem was created with Linux
> and btrfs-progs 5.10.x? (please keep me in CC)
> 
> If anyone knows if this issue was fixed for 5.15, please share the good
> news!
> 
> 
> Regards,
> Nicholas

I believe this is a problem of having degraded mounts. If you go below two disks with a RAID1 profile with degraded mount, any data that needs to be written to the filesystem will be created in single chunks. If you then re-introduce a second device, a balance conversion is needed. (*) 

Personally, l haven't seen the problem you mentioned in 2016 in a long time. Just did some quick testing using btrfs-progs 5.18 and could not see any issues. 

* https://wiki.tnonline.net/w/Btrfs/Replacing_a_disk#Restoring_redundancy_after_a_replaced_disk

Thanks
Forza

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf]

On 5/28/22 21:20, Nicholas D Steeves wrote:
> Efkf, would you please confirm if the filesystem was created with Linux
> and btrfs-progs 5.10.x? (please keep me in CC)
It was created under linux and I'm 99% sure kernel 5.10.0 and btrfs-progs 5.10.1
It was surely that configuration when I started messing with it.
Now that i think about it i had mounted degraded when i had initially created the filesystem so maybe single metadata got created and has been bitrotting away since.
If that's the case though it didn't cause any problems before running the first balance command after which everything went downhill.


On 5/27/22 22:37, Forza wrote:
>> Anyway, is there a way to check the data is really redundant without trusting the filesystem telling me it's so?
> 
> Yes, you use 'btrfs scrub' to read all data and metadata blocks from all devices and compare the checksums. If there are problems, scrub will tell you.
> 
> https://btrfs.readthedocs.io/en/latest/btrfs-scrub.html
> https://wiki.tnonline.net/w/Btrfs/Scrub
> 

Yeah but that relies on me having actually set up RAID1.
The point I'm trying to make is that as a beginner who learns as they go you don't know what you don't know so maybe there is some detail you don't know about that's making your data unsafe . (in this case scrubbing without checking if the whole filesystem is raid1, I assumed it was set in stone from the fs's creation)
I should have read more about it but i think there will be more new users that will try what i did to sanity check their setup so in my opinion it would be important to make it so that if you don't write to the FS, especially if you mount it read only it should be safe to mount degraded and not put any data in jeopardy.

On 5/28/22 22:04, Forza wrote:
> I believe this is a problem of having degraded mounts.
So you think the single chunks from the degraded mount got corrupted due to something unrelated to btrfs and that caused the problem i had?


Either way does anyone want me to run something on the filesystem to provide any help for any possible debugging or can i wipe it and move on? (i kind of need the storage)

Thanks a lot again by the way to everyone who looked into it and especially for all the great help!

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Forza]

On 2022-05-29 22:48, efkf wrote:
> On 5/28/22 21:20, Nicholas D Steeves wrote:
>> Efkf, would you please confirm if the filesystem was created with Linux
>> and btrfs-progs 5.10.x? (please keep me in CC)
> It was created under linux and I'm 99% sure kernel 5.10.0 and 
> btrfs-progs 5.10.1
> It was surely that configuration when I started messing with it.
> Now that i think about it i had mounted degraded when i had initially 
> created the filesystem so maybe single metadata got created and has been 
> bitrotting away since.
> If that's the case though it didn't cause any problems before running 
> the first balance command after which everything went downhill.
> 
> 
> On 5/27/22 22:37, Forza wrote:
>>> Anyway, is there a way to check the data is really redundant without 
>>> trusting the filesystem telling me it's so?
>>
>> Yes, you use 'btrfs scrub' to read all data and metadata blocks from 
>> all devices and compare the checksums. If there are problems, scrub 
>> will tell you.
>>
>> https://btrfs.readthedocs.io/en/latest/btrfs-scrub.html
>> https://wiki.tnonline.net/w/Btrfs/Scrub
>>
> 
> Yeah but that relies on me having actually set up RAID1.
> The point I'm trying to make is that as a beginner who learns as they go 
> you don't know what you don't know so maybe there is some detail you 
> don't know about that's making your data unsafe . (in this case 
> scrubbing without checking if the whole filesystem is raid1, I assumed 
> it was set in stone from the fs's creation)

Indeed. Btrfs supports multiple profiles, and a combination of profiles 
as you discovered. Some Btrfs tools do show a warning on multiple 
profiles detected.

> I should have read more about it but i think there will be more new 
> users that will try what i did to sanity check their setup so in my 
> opinion it would be important to make it so that if you don't write to 
> the FS, especially if you mount it read only it should be safe to mount 
> degraded and not put any data in jeopardy.
> 

I had a discussion with some Windows users, and they did exactly the 
same thing - yanked the mirror out and then inserted it again. 4 times 
out of 5 it "worked" and they got upset when it didn't work the last time.

So, with that said, there is room to improve documentation, man pages 
and guides to help users find the information they need to check their 
system correctly.

For now, mounting each mirror independently and then combine them again 
is not good for Btrfs. This use-case seems to be unhandled.

> On 5/28/22 22:04, Forza wrote:
>> I believe this is a problem of having degraded mounts.
> So you think the single chunks from the degraded mount got corrupted due 
> to something unrelated to btrfs and that caused the problem i had?
> 

It is possible the errors are older, but not surfacing until you tried 
to do that full balance after adding the third drive. This could have 
caused balance to fail, leading up to all the subsequent errors.

> 
> Either way does anyone want me to run something on the filesystem to 
> provide any help for any possible debugging or can i wipe it and move 
> on? (i kind of need the storage >
> Thanks a lot again by the way to everyone who looked into it and 
> especially for all the great help!

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Graham Cobb]

On 30/05/2022 21:47, Forza wrote:
> 
> 
> On 2022-05-29 22:48, efkf wrote:
>> On 5/28/22 21:20, Nicholas D Steeves wrote:
>>> Efkf, would you please confirm if the filesystem was created with Linux
>>> and btrfs-progs 5.10.x? (please keep me in CC)
>> It was created under linux and I'm 99% sure kernel 5.10.0 and
>> btrfs-progs 5.10.1
>> It was surely that configuration when I started messing with it.
>> Now that i think about it i had mounted degraded when i had initially
>> created the filesystem so maybe single metadata got created and has
>> been bitrotting away since.
>> If that's the case though it didn't cause any problems before running
>> the first balance command after which everything went downhill.
>>
>>
>> On 5/27/22 22:37, Forza wrote:
>>>> Anyway, is there a way to check the data is really redundant without
>>>> trusting the filesystem telling me it's so?
>>>
>>> Yes, you use 'btrfs scrub' to read all data and metadata blocks from
>>> all devices and compare the checksums. If there are problems, scrub
>>> will tell you.
>>>
>>> https://btrfs.readthedocs.io/en/latest/btrfs-scrub.html
>>> https://wiki.tnonline.net/w/Btrfs/Scrub
>>>
>>
>> Yeah but that relies on me having actually set up RAID1.
>> The point I'm trying to make is that as a beginner who learns as they
>> go you don't know what you don't know so maybe there is some detail
>> you don't know about that's making your data unsafe . (in this case
>> scrubbing without checking if the whole filesystem is raid1, I assumed
>> it was set in stone from the fs's creation)
> 
> Indeed. Btrfs supports multiple profiles, and a combination of profiles
> as you discovered. Some Btrfs tools do show a warning on multiple
> profiles detected.
> 
>> I should have read more about it but i think there will be more new
>> users that will try what i did to sanity check their setup so in my
>> opinion it would be important to make it so that if you don't write to
>> the FS, especially if you mount it read only it should be safe to
>> mount degraded and not put any data in jeopardy.
>>
> 
> I had a discussion with some Windows users, and they did exactly the
> same thing - yanked the mirror out and then inserted it again. 4 times
> out of 5 it "worked" and they got upset when it didn't work the last time.
> 
> So, with that said, there is room to improve documentation, man pages
> and guides to help users find the information they need to check their
> system correctly.
> 
> For now, mounting each mirror independently and then combine them again
> is not good for Btrfs. This use-case seems to be unhandled.

Sounds like btrfs should do something like assign the filesystem a
completely new UUID (updated onto all the superblocks present at the
time) if you mount degraded. To prevent any disks not present at that
time from being reintroduced later.

A bit drastic but that is what is really happening with a degraded
mount: you are creating a new filesystem, with some of the contents
inherited from an old one, and some missing.

Graham

Re: Tried to replace a drive in a raid 1 and all hell broke loose

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 2639 bytes --]

Graham Cobb <g.btrfs@cobb.me.uk> writes:

> On 30/05/2022 21:47, Forza wrote:
>> 
>> 
>> I had a discussion with some Windows users, and they did exactly the
>> same thing - yanked the mirror out and then inserted it again. 4 times
>> out of 5 it "worked" and they got upset when it didn't work the last time.
>> 
>> So, with that said, there is room to improve documentation, man pages
>> and guides to help users find the information they need to check their
>> system correctly.
>> 
>> For now, mounting each mirror independently and then combine them again
>> is not good for Btrfs. This use-case seems to be unhandled.
>
> Sounds like btrfs should do something like assign the filesystem a
> completely new UUID (updated onto all the superblocks present at the
> time) if you mount degraded. To prevent any disks not present at that
> time from being reintroduced later.
>
> A bit drastic but that is what is really happening with a degraded
> mount: you are creating a new filesystem, with some of the contents
> inherited from an old one, and some missing.
>

Yes, I agree something should be done, but I'm not sure this is it.
Rather than this, I wonder why a multidisk profile of btrfs doesn't
do something like the following:

1. Maintain a list of devices that are part of the filesystem, using
/dev/disk/by-id or by-uuid identifiers.  At fs creation, these are
added to the "good list"

2. If ever the filesystem is mounted degraded, the IDs of missing
device[s] should be moved to a "bad list", and permanently blocked from
use.

3. If ever those IDs reappear (ie: they match an element of the "bad
list"), a warning should be emitted in the kernel log, and btrfs-progs
tools should warn that a "wipefs" of those devices is required before
readding them.

4. It also seems like it would be user-friendly to emit a warning if
ever single block groups are found on a on what should be a 100%
profile=raid{1,10,c3,c4,5,6} filesystem, because this is a dangerous
situation to be in.  This would signal that an urgent rebalance is
required after step #3.


Qu, is this possible with the current on-disk format?  If not, then
could something like this (specifically the "bad device list" at #2)
please be included in the design of the next on-disk format?  Ideally it
would be nice if reattached disks could replay all transactions since
they were detached, so maybe the future on-disk format could also
reserve a field for this?  The silent creation of profile=single blocks
makes using Btrfs profile=raid{1,c3,c4} risky when compared to ZFS
mirrors.

Regards,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]