Re: Another btrfs corruption (unable to find ref byte, kernel 5.0-rc4)

[Nikolay Borisov <nborisov@suse.com>]

On 7.02.19 г. 6:07 ч., Nazar Mokrynskyi wrote:
> NOTE: I don't need asistance with data recovery, everything was restored shortly from fully automated backups, I only hope this information is useful for developers in some way.
> 
> So my primary BTRFS filesystem corrupted itself again.
> 
<SNIP>

>> [   82.910327] BTRFS error (device dm-0): unable to find ref byte nr 65470324736 parent 62522359808 root 267  owner 819644 offset 1552384

Funnily enough, this looks like the problem that Josef sent patches for
several hours ago - 'Fix missing reference aborts when resuming snapshot
delete'

>> [   82.910329] ------------[ cut here ]------------
>> [   82.910329] BTRFS: Transaction aborted (error -2)
>> [   82.910339] WARNING: CPU: 14 PID: 6082 at fs/btrfs/extent-tree.c:6950 __btrfs_free_extent.isra.72+0x7ad/0xac0 [btrfs]
>> [   82.910339] Modules linked in: btrfs zstd_compress libcrc32c xor raid6_pq dm_crypt algif_skcipher af_alg intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_codec_hdmi snd_hda_intel kvm snd_usb_audio snd_hda_codec snd_usbmidi_lib snd_hda_core snd_hwdep snd_pcm irqbypass crct10dif_pclmul snd_seq_midi snd_seq_midi_event crc32_pclmul snd_rawmidi ghash_clmulni_intel uvcvideo pcbc snd_seq videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_seq_device snd_timer aesni_intel videodev snd aes_x86_64 crypto_simd cdc_acm soundcore media input_leds joydev cryptd mei_me glue_helper intel_wmi_thunderbolt intel_cstate mei intel_rapl_perf acpi_pad mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 overlay nls_iso8859_1 dm_mirror dm_region_hash dm_log
>> [   82.910354]  hid_generic usbhid hid uas usb_storage amdkfd amd_iommu_v2 amdgpu nouveau chash gpu_sched ttm mxm_wmi drm_kms_helper syscopyarea sysfillrect sysimgblt nvme fb_sys_fops igb e1000e atlantic drm ahci dca i2c_algo_bit nvme_core libahci wmi video
>> [   82.910361] CPU: 14 PID: 6082 Comm: btrfs-cleaner Tainted: G        W         4.18.0-10-generic #11-Ubuntu
>> [   82.910361] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z370 Professional Gaming i7, BIOS P3.40 11/08/2018
>> [   82.910367] RIP: 0010:__btrfs_free_extent.isra.72+0x7ad/0xac0 [btrfs]
>> [   82.910367] Code: 48 8b 40 50 f0 48 0f ba a8 10 ce 00 00 02 0f 92 c0 41 59 84 c0 0f 85 1d c1 09 00 44 89 ee 48 c7 c7 70 e2 ee c0 e8 75 4a 45 f8 <0f> 0b e9 07 c1 09 00 4c 89 e7 e8 14 07 ff ff 48 8b 7d 88 4d 89 f9
>> [   82.910380] RSP: 0018:ffffa71bc6833bf8 EFLAGS: 00010282
>> [   82.910381] RAX: 0000000000000000 RBX: 0000000f3e55e000 RCX: 0000000000000006
>> [   82.910381] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff89d09e5964b0
>> [   82.910382] RBP: ffffa71bc6833ca0 R08: 0000000000000001 R09: 0000000000000805
>> [   82.910382] R10: 0000000000000004 R11: 0000000000000000 R12: ffff89cfd8338af0
>> [   82.910382] R13: 00000000fffffffe R14: 0000000e8e9f8000 R15: 000000000000010b
>> [   82.910383] FS:  0000000000000000(0000) GS:ffff89d09e580000(0000) knlGS:0000000000000000
>> [   82.910383] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [   82.910384] CR2: 0000564398e70760 CR3: 000000076ce0a002 CR4: 00000000003606e0
>> [   82.910384] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [   82.910385] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [   82.910385] Call Trace:
>> [   82.910391]  __btrfs_run_delayed_refs+0x20e/0x1010 [btrfs]
>> [   82.910397]  ? add_pinned_bytes+0x67/0x70 [btrfs]
>> [   82.910402]  ? btrfs_free_tree_block+0x167/0x2d0 [btrfs]
>> [   82.910408]  btrfs_run_delayed_refs+0x80/0x190 [btrfs]
>> [   82.910415]  btrfs_should_end_transaction+0x47/0x60 [btrfs]
>> [   82.910421]  btrfs_drop_snapshot+0x3d1/0x800 [btrfs]
>> [   82.910428]  btrfs_clean_one_deleted_snapshot+0xbb/0xf0 [btrfs]
>> [   82.910434]  cleaner_kthread+0x136/0x160 [btrfs]
>> [   82.910435]  kthread+0x120/0x140
>> [   82.910442]  ? btree_submit_bio_start+0x20/0x20 [btrfs]
>> [   82.910443]  ? kthread_bind+0x40/0x40
>> [   82.910443]  ret_from_fork+0x35/0x40
>> [   82.910444] ---[ end trace 32964933c87d1d28 ]---
>> [   82.910445] BTRFS: error (device dm-0) in __btrfs_free_extent:6950: errno=-2 No such entry
>> [   82.910446] BTRFS info (device dm-0): forced readonly
>> [   82.910447] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:3057: errno=-2 No such entry
>> [   82.910763] BTRFS error (device dm-0): pending csums is 221184
> It became clear that filesystem is ready to be thown to /dev/null, so I've made an image for future investigation and recreated everything from ~10 minutes old backup.
> 
> If there is anything suspicios there and I can run useful tests on filesystem image - let me know, I'll spin up qcow2 CoW image and can do anything with it in VMs.
> 
> Unfortunately, it was just over a month since filesystem creation. Other BTRFS filesystem containing backups is perfectly fine (was created late April 2018), but it is only used for sending snapshots to it and not much other activity.
>