I have attached the full journalctl from the boot where this first happened. Note that this happened again after a scrub and a reboot during a different write operation. I'm currently doing a backup (not overwriting any of my other backups), so I will do a memory test to see if I have bad RAM. I don't have ECC memory so I can't easily check. On Sat, 16 Oct 2021 at 02:52, Qu Wenruo wrote: > > > > On 2021/10/16 08:14, James Harvey wrote: > > My server consists of a single 16TB external drive (I have backups, > > and I was planning to make a proper server at some point) and I used > > BTRFS for the drive's filesystem. Recently, the file system would go > > into read only and put a load of errors into the system logs. Running > > a BTRFS scrub returned no errors, a readonly BTRFS check returned no > > errors, and a SMART check showed no issues/bad sectors. > > This is very strange, as normally if there is really on-disk corruption, > especially in metadata, btrfs check should detect it. > > > Has BTRFS > > broke itself or is this a drive issue: > > > > Here are the errors: > > Could you please provide the full dmesg? > > We want the context to see get a whole picture of the problem, not only > just error messages from btrfs. > > If the problem only happens at write time, maybe you want to do a memory > test to verify it's not some bitflip in your memory in the mean time. > > Thanks, > Qu > > > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105460736 csum 0x75ab540e expected csum > > 0xaeb99694 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105464832 csum 0xe83b4c2a expected csum > > 0xb9a65172 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105468928 csum 0x4769b37a expected csum > > 0x3598cf9e mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105473024 csum 0x7c39a990 expected csum > > 0x9c523a6c mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105477120 csum 0xfedc09f1 expected csum > > 0x68386e9a mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105481216 csum 0xf9f25835 expected csum > > 0x96d2dea3 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105485312 csum 0x37643155 expected csum > > 0x6139f8a1 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105489408 csum 0x13893c06 expected csum > > 0xb28c00a8 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105493504 csum 0x2a89fcff expected csum > > 0x4c5758ed mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 97395 off 14105497600 csum 0x7484b77c expected csum > > 0x0a9f3138 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bdev > > /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343812173824 have 9856732008096476660 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343806013440 have 757116834938933 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343812173824 have 9856732008096476660 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9622003011584, 9622003015680) > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343806013440 have 757116834938933 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343812173824 have 9856732008096476660 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343812173824 have 9856732008096476660 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9622003015680, 9622003019776) > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343947784192 have 17536680014548819927 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343812173824 have 9856732008096476660 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343947784192 have 17536680014548819927 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9644356001792, 9644356005888) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS error (device sdb1): bad > > tree block start, want 9343812173824 have 9856732008096476660 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9622003019776, 9622003023872) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9644356005888, 9644356009984) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9622003023872, 9622003027968) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9633973551104, 9633973555200) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9644356009984, 9644356014080) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9622003027968, 9622003032064) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > hole found for disk bytenr range [9633973555200, 9633973559296) > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:37 James-Server kernel: BTRFS warning (device sdb1): csum > > failed root 5 ino 173568 off 3875945435136 csum 0x23ed6941 expected > > csum 0xc096fec5 mirror 1 > > Oct 14 21:50:41 James-Server kernel: BTRFS: error (device sdb1) in > > btrfs_finish_ordered_io:3064: errno=-5 IO failure > > Oct 14 21:50:41 James-Server kernel: BTRFS info (device sdb1): forced readonly > > > > uname -a: Linux James-Server 5.14.11-arch1-1 #1 SMP PREEMPT Sun, 10 > > Oct 2021 00:48:26 +0000 x86_64 GNU/Linux > > > > btrfs --version: btrfs-progs v5.14.2 > > > > btrfs fi show: > > > > Label: 'Seagate 16TB 1' uuid: e183a876-95e0-4d15-a641-69f4a8e8e7e7 > > Total devices 1 FS bytes used 9.61TiB > > devid 1 size 14.55TiB used 9.62TiB path /dev/sdb1 > > > > btrfs fi df: > > > > Data, single: total=9.60TiB, used=9.60TiB > > System, DUP: total=8.00MiB, used=1.09MiB > > Metadata, DUP: total=11.00GiB, used=10.74GiB > > GlobalReserve, single: total=512.00MiB, used=0.00B > > > > Mount options: rw,noatime,compress=zstd:3,space_cache=v2,autodefrag,subvolid=5,subvol=/ > >