Hi list! I have btrfs pool consisting of 5x 2,72 TiB LUKS (dm-crypt) partitions in RAID1, mounted on Linux 4.4 with btrfs-progs 4.4. I never had any crashes or power loss here, but recently about every 60 - 120 minutes (while in use) btrfs detects corruptions, aborts the transaction and drops to read-only mode. btrfs still mounts normally without any special options (it does take about 60 seconds, which I guess is normal for this kind of size). All LUKS partitions have at least 400GiB of free space. I don't see any HW problems here; I doubt there is a corruption coming from the LUKS partition. I did test the RAM but it seems fine in multiple memtest86+ amd memtest86 runs. Are there any known bugs in 4.4? Any suggestions would be greatly appreciated! I have to admit I did not regularly scrub. Thanks, Lukas --- ~# uname -a Linux srv1-dom0 4.4.0-63-generic #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux ~# btrfs --version btrfs-progs v4.4 ~# btrfs fi show Label: 'dom0-os' uuid: e475636c-21e0-4563-87d6-91f03c519a62 Total devices 5 FS bytes used 3.52GiB devid 1 size 10.00GiB used 3.53GiB path /dev/sda2 devid 2 size 10.00GiB used 4.25GiB path /dev/sdb2 devid 3 size 10.00GiB used 3.28GiB path /dev/sdc2 devid 4 size 10.00GiB used 4.00GiB path /dev/sdd2 devid 5 size 10.00GiB used 4.00GiB path /dev/sde2 Label: 'storage_pool' uuid: f50f980e-7640-49c7-bf8d-20d55cfe6005 Total devices 5 FS bytes used 5.77TiB devid 1 size 2.72TiB used 2.31TiB path /dev/mapper/sda3_crypt devid 2 size 2.72TiB used 2.31TiB path /dev/mapper/sdb3_crypt devid 3 size 2.72TiB used 2.31TiB path /dev/mapper/sdc3_crypt devid 4 size 2.72TiB used 2.31TiB path /dev/mapper/sdd3_crypt devid 5 size 2.72TiB used 2.31TiB path /dev/mapper/sde3_crypt ~# btrfs fi df /storage/users/ Data, RAID1: total=5.77TiB, used=5.76TiB System, RAID1: total=32.00MiB, used=832.00KiB Metadata, RAID1: total=8.00GiB, used=6.96GiB GlobalReserve, single: total=512.00MiB, used=0.00B ~# ~# partial dmesg: [ 1509.033492] BTRFS: device label storage_pool devid 1 transid 238135 /dev/dm-5 [ 1510.498804] BTRFS: device label storage_pool devid 2 transid 238135 /dev/dm-6 [ 1511.980968] BTRFS: device label storage_pool devid 3 transid 238135 /dev/dm-7 [ 1513.461799] BTRFS: device label storage_pool devid 4 transid 238135 /dev/dm-8 [ 1514.838757] BTRFS: device label storage_pool devid 5 transid 238135 /dev/dm-9 [ 1517.726471] BTRFS info (device dm-9): btrfs: use no compression [ 1517.726477] BTRFS info (device dm-9): disk space caching is enabled [ 1517.726479] BTRFS: has skinny extents [ 1569.598633] BTRFS: checking UUID tree [ 3540.825747] BTRFS critical (device dm-9): corrupt leaf, slot offset bad: block=5242107641856,root=1, slot=39 [ 3540.836168] BTRFS critical (device dm-9): corrupt leaf, slot offset bad: block=5242107641856,root=1, slot=39 [ 3540.846413] ------------[ cut here ]------------ [ 3540.846432] WARNING: CPU: 2 PID: 2757 at /build/linux-mPTI9s/linux-4.4.0/fs/btrfs/extent-tree.c:2930 btrfs_run_delayed_refs+0x26b/0x2a0 [btrfs]() [ 3540.846433] BTRFS: Transaction aborted (error -5) [ 3540.846434] Modules linked in: algif_skcipher af_alg xen_gntdev xen_evtchn xenfs xen_privcmd drbg ansi_cprng dm_crypt nls_iso8859_1 bridge stp llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel serio_raw joydev input_leds nuvoton_cir 8250_fintek ie31200_edac mac_hid rc_core lpc_ich edac_core shpchp mei_me mei ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid mxm_wmi i915 i2c_algo_bit drm_kms_helper aesni_intel aes_x86_64 glue_helper syscopyarea sysfillrect firewire_ohci sysimgblt firewire_core fb_sys_fops lrw psmouse [ 3540.846466] tg3 gf128mul ablk_helper cryptd crc_itu_t ptp ahci drm pps_core libahci fjes wmi video [ 3540.846473] CPU: 2 PID: 2757 Comm: btrfs-transacti Not tainted 4.4.0-63-generic #84-Ubuntu [ 3540.846475] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Extreme6, BIOS P2.80 07/01/2013 [ 3540.846476] 0000000000000200 0000000002709bc3 ffff88007615fc90 ffffffff813f8083 [ 3540.846478] ffff88007615fcd8 ffffffffc048d498 ffff88007615fcc8 ffffffff810812d2 [ 3540.846479] ffff8802adf562f8 ffff8802a9c71800 ffff8800056caef0 ffffffffffffffff [ 3540.846481] Call Trace: [ 3540.846486] [] dump_stack+0x63/0x90 [ 3540.846489] [] warn_slowpath_common+0x82/0xc0 [ 3540.846491] [] warn_slowpath_fmt+0x5c/0x80 [ 3540.846500] [] ? __btrfs_run_delayed_refs+0xcdd/0x1220 [btrfs] [ 3540.846509] [] btrfs_run_delayed_refs+0x26b/0x2a0 [btrfs] [ 3540.846520] [] commit_cowonly_roots+0x22b/0x2c2 [btrfs] [ 3540.846530] [] btrfs_commit_transaction+0x576/0xa90 [btrfs] [ 3540.846533] [] ? wake_atomic_t_function+0x60/0x60 [ 3540.846542] [] transaction_kthread+0x229/0x240 [btrfs] [ 3540.846558] [] ? btrfs_cleanup_transaction+0x570/0x570 [btrfs] [ 3540.846560] [] kthread+0xd8/0xf0 [ 3540.846562] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 3540.846564] [] ret_from_fork+0x3f/0x70 [ 3540.846566] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 3540.846567] ---[ end trace 70830ce6f0e320dd ]--- [ 3540.846587] BTRFS: error (device dm-9) in btrfs_run_delayed_refs:2930: errno=-5 IO failure [ 3540.855086] BTRFS info (device dm-9): forced readonly [ 3540.855088] BTRFS warning (device dm-9): Skipping commit of aborted transaction. [ 3540.855090] BTRFS: error (device dm-9) in cleanup_transaction:1746: errno=-5 IO failure full dmesg attached and online at: http://pastebin.com/raw/K8FNNEnS