BTRFS partition over a SSD got totally broken on Ubuntu 18.10 -> errno=-17 Object already exists (Failed to recover log tree)

BTRFS partition over a SSD got totally broken on Ubuntu 18.10 -> errno=-17 Object already exists (Failed to recover log tree)

[Luis Panadero Guardeño]

I have a BTRFS partititon on a Toshiba TR200 SSD . Two days ago, it
got to readonly mode. Inspecting the dmesg I got this :
>
> [ 3490.935497] BTRFS info (device sdc2): disk space caching is enabled
> [ 3490.935500] BTRFS info (device sdc2): has skinny extents
> [ 3490.968534] BTRFS info (device sdc2): enabling ssd optimizations
> [ 3490.975507] ------------[ cut here ]------------
> [ 3490.975509] BTRFS: Transaction aborted (error -17)
> [ 3490.975614] WARNING: CPU: 6 PID: 22107 at /home/kernel/COD/linux/fs/btrfs/extent-tree.c:3083 btrfs_run_delayed_refs+0x170/0x1c0 [btrfs]
> [ 3490.975615] Modules linked in: xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key l2tp_ppp l2tp_netlink l2tp_core ip6_udp_tunnel udp_tunnel xfrm_algo pppox snd_hrtimer zram binfmt_misc nls_iso8859_1 snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core edac_mce_amd snd_hwdep snd_pcm kvm_amd ccp kvm snd_seq_midi snd_seq_midi_event ir_rc6_decoder snd_rawmidi irqbypass snd_seq rc_rc6_mce crct10dif_pclmul crc32_pclmul ghash_clmulni_intel mceusb joydev input_leds rc_core snd_seq_device pcbc snd_timer aesni_intel aes_x86_64 snd soundcore crypto_simd glue_helper mac_hid cryptd parport_serial fam15h_power shpchp k10temp sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq hid_generic usbhid hid
> [ 3490.975666]  amdkfd amd_iommu_v2 amdgpu chash gpu_sched i2c_algo_bit ttm mxm_wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops r8169 drm ahci mii i2c_piix4 libahci wmi
> [ 3490.975684] CPU: 6 PID: 22107 Comm: mount Not tainted 4.16.2-041602-generic #201804121130
> [ 3490.975685] Hardware name: MSI MS-7974/970A-G43 PLUS (MS-7974), BIOS V1.1 07/04/2016
> [ 3490.975713] RIP: 0010:btrfs_run_delayed_refs+0x170/0x1c0 [btrfs]
> [ 3490.975714] RSP: 0018:ffffb5a486c0f8c0 EFLAGS: 00010282
> [ 3490.975716] RAX: 0000000000000000 RBX: ffff9f4cab66df08 RCX: 0000000000000006
> [ 3490.975718] RDX: 0000000000000007 RSI: 0000000000000096 RDI: ffff9f4cbed964b0
> [ 3490.975719] RBP: ffffb5a486c0f910 R08: 000000000000043e R09: 0000000000000004
> [ 3490.975721] R10: ffff9f4ab44cfc40 R11: 0000000000000001 R12: ffff9f4ab44cfc40
> [ 3490.975722] R13: ffff9f4a9bb94800 R14: ffff9f4cab66df08 R15: 0000000000000038
> [ 3490.975725] FS:  00007f2f0355d080(0000) GS:ffff9f4cbed80000(0000) knlGS:0000000000000000
> [ 3490.975726] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 3490.975728] CR2: 000055ffe06115c0 CR3: 000000023aab0000 CR4: 00000000000406e0
> [ 3490.975729] Call Trace:
> [ 3490.975762]  btrfs_commit_transaction+0x45/0x8e0 [btrfs]
> [ 3490.975795]  btrfs_recover_log_trees+0x416/0x4a0 [btrfs]
> [ 3490.975828]  ? replay_one_extent+0x7d0/0x7d0 [btrfs]
> [ 3490.975856]  open_ctree+0x1feb/0x2440 [btrfs]
> [ 3490.975882]  btrfs_mount_root+0x6d1/0x7a0 [btrfs]
> [ 3490.975906]  ? btrfs_mount_root+0x6d1/0x7a0 [btrfs]
> [ 3490.975912]  mount_fs+0x37/0x150
> [ 3490.975915]  ? mount_fs+0x37/0x150
> [ 3490.975918]  ? alloc_vfsmnt+0x1b3/0x230
> [ 3490.975921]  vfs_kern_mount.part.23+0x5d/0x110
> [ 3490.975924]  vfs_kern_mount+0x13/0x20
> [ 3490.975951]  btrfs_mount+0x182/0x950 [btrfs]
> [ 3490.975955]  ? cpumask_next+0x1b/0x20
> [ 3490.975958]  ? pcpu_alloc+0x31b/0x650
> [ 3490.975962]  mount_fs+0x37/0x150
> [ 3490.975964]  ? mount_fs+0x37/0x150
> [ 3490.975966]  ? alloc_vfsmnt+0x1b3/0x230
> [ 3490.975969]  vfs_kern_mount.part.23+0x5d/0x110
> [ 3490.975972]  do_mount+0x5ed/0xce0
> [ 3490.975976]  ? memdup_user+0x4f/0x80
> [ 3490.975979]  SyS_mount+0x98/0xe0
> [ 3490.975982]  do_syscall_64+0x73/0x130
> [ 3490.975986]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> [ 3490.975988] RIP: 0033:0x7f2f02e423ca
> [ 3490.975990] RSP: 002b:00007ffc9434edd8 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
> [ 3490.975992] RAX: ffffffffffffffda RBX: 00005588bc39da40 RCX: 00007f2f02e423ca
> [ 3490.975994] RDX: 00005588bc3ab850 RSI: 00005588bc39dc40 RDI: 00005588bc39dc20
> [ 3490.975995] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f2f02e8e1b0
> [ 3490.975996] R10: 00000000c0ed0000 R11: 0000000000000206 R12: 00005588bc39dc20
> [ 3490.975998] R13: 00005588bc3ab850 R14: 0000000000000000 R15: 00007f2f0336a8a4
> [ 3490.975999] Code: 1f 40 00 eb c4 48 8b 53 50 f0 48 0f ba aa d8 cd 00 00 02 72 1b 83 f8 fb 74 34 89 c6 48 c7 c7 c0 a5 7c c0 89 45 d0 e8 40 a4 b6 e7 <0f> 0b 8b 45 d0 89 c1 ba 0b 0c 00 00 48 c7 c6 00 39 7c c0 48 89
> [ 3490.976045] ---[ end trace 633ae76f1f0eee99 ]---
> [ 3490.976048] BTRFS: error (device sdc2) in btrfs_run_delayed_refs:3083: errno=-17 Object already exists
> [ 3490.976062] BTRFS: error (device sdc2) in btrfs_replay_log:2308: errno=-17 Object already exists (Failed to recover log tree)
> [ 3490.976172] BTRFS error (device sdc2): cleaner transaction attach returned -30
> [ 3491.002005] BTRFS error (device sdc2): open_ctree failed


I try to reboot to load a older kernel, thinking that was a issue of
the last kernel updater. I can't mount the partition any more.
Trying to mount the BTRFS partition from a Live dvd, gets the same error.

I try to execute btrfs check without any sucess :
>
> sudo btrfs check -p --readonly /dev/sdc2
> Checking filesystem on /dev/sdc2
> UUID: 595cf1ca-7df2-46ff-8472-a2789f0bab77
> check/main.c:3654: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1
> btrfs(+0x4b7d0)[0x559daa67e7d0]
> btrfs(+0x12df8)[0x559daa645df8]
> btrfs(+0x4bd21)[0x559daa67ed21]
> btrfs(+0x4efdb)[0x559daa681fdb]
> btrfs(+0x4fbdf)[0x559daa682bdf]
> btrfs(+0x505f1)[0x559daa6835f1]
> btrfs(cmd_check+0x133f)[0x559daa686926]
> btrfs(main+0x143)[0x559daa649c87]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f5ac3a59b97]
> btrfs(_start+0x2a)[0x559daa649cca]
> 'sudo btrfs check -p --readonly…' terminated by signal SIGABRT (Abort)
>

I think that the partition can't be recovered. However, I would like
to know why has happened this or how avoid this problem as I would
like to keep using BTRFS.

Smart values shows that the SSD it's ok but before doing a reinstall I
would try to stress it to see any signal of problems.

Info about my system and versions :

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

uname -a
Linux blackbox-linux 4.16.2-041602-generic #201804121130 SMP Thu Apr
12 11:32:17 UTC 2018 x86_64 x86_64 x86_64 GNU/L

btrfs --version
btrfs-progs v4.15.1

mount options on fstab : defaults,discard,compress=lzo

Re: BTRFS partition over a SSD got totally broken on Ubuntu 18.10 -> errno=-17 Object already exists (Failed to recover log tree)

[Chris Murphy]

On Sat, Mar 16, 2019 at 3:35 AM Luis Panadero Guardeño
<luis.panadero@gmail.com> wrote:
>
> > [ 3490.976048] BTRFS: error (device sdc2) in btrfs_run_delayed_refs:3083: errno=-17 Object already exists
> > [ 3490.976062] BTRFS: error (device sdc2) in btrfs_replay_log:2308: errno=-17 Object already exists (Failed to recover log tree)

It's a problem with log replay. That you're stuck here means it's a
bug, but I can't tell you where. It might have started days or weeks
earlier and cumulatively ended up here where the kernel can't deal
with it; or maybe it's an SSD firmware bug; or maybe some of both.
Also, when Btrfs goes read only it's trying to avoid causing more
corruption, but it might have already been too late. Anyway, you can
see if it will mount with a newer kernel, try the newest you can from
4.19 or 4.20 series.

Another possibility that will come with some data loss is `btrfs
rescue zero-log /dev/` which will zero the log tree, and whatever
writes were happening at the time the log tree was being written to
will be lost - up to 30 seconds worth from before going read-only. So
if you can live with that you can try it.

The big problem with 4.16 series is it's not long term so it's EOL and
not getting any bug fixes. Ideally run the newest stable kernel.

Also, the crash with btrfs check is a bug in btrfs-progs 4.15. I'm
pretty sure it's fixed, but you can update btrfs-progs to 4.20.2 to
find out. I don't know any distro doing backports, so it's reasonable
to just use the latest upstream version once it's a couple weeks old.
For the purpose of zeroing the log, I don't think you need to update
btrfs-progs.


-- 
Chris Murphy