* [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
@ 2016-08-28 13:29 Kai Krakow
2017-02-02 12:01 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Kai Krakow @ 2016-08-28 13:29 UTC (permalink / raw)
To: linux-btrfs
Hello list!
It happened again. While using VirtualBox the following crash happened,
btrfs check found a lot of errors which it couldn't repair. Earlier
that day my system crashed which may already introduced errors into my
filesystem. Apparently, I couldn't create an image (not enough space
available), I only can give this trace from dmesg:
[44819.903435] ------------[ cut here ]------------
[44819.903443] WARNING: CPU: 3 PID: 2787 at fs/btrfs/extent-tree.c:2963 btrfs_run_delayed_refs+0x26c/0x290
[44819.903444] BTRFS: Transaction aborted (error -17)
[44819.903445] Modules linked in: nls_iso8859_15 nls_cp437 vfat fat fuse rfcomm veth af_packet ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc w83627ehf bnep hwmon_vid cachefiles btusb btintel bluetooth snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec rfkill snd_hwdep snd_hda_core snd_pcm snd_timer coretemp hwmon snd r8169 mii kvm_intel kvm iTCO_wdt iTCO_vendor_support rtc_cmos irqbypass soundcore ip_tables uas usb_storage nvidia_drm(PO) vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) nvidia_modeset(PO) nvidia(PO) efivarfs unix ipv6
[44819.903484] CPU: 3 PID: 2787 Comm: BrowserBlocking Tainted: P O 4.7.2-gentoo #2
[44819.903485] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Pro3, BIOS L2.16A 02/22/2013
[44819.903487] 0000000000000000 ffffffff8130af2d ffff8800b7d03d20 0000000000000000
[44819.903489] ffffffff810865fa ffff880409374428 ffff8800b7d03d70 ffff8803bf299760
[44819.903491] 0000000000000000 00000000ffffffef ffff8803f677f000 ffffffff8108666a
[44819.903493] Call Trace:
[44819.903496] [<ffffffff8130af2d>] ? dump_stack+0x46/0x59
[44819.903500] [<ffffffff810865fa>] ? __warn+0xba/0xe0
[44819.903502] [<ffffffff8108666a>] ? warn_slowpath_fmt+0x4a/0x50
[44819.903504] [<ffffffff8121351c>] ? btrfs_run_delayed_refs+0x26c/0x290
[44819.903507] [<ffffffff811feb1e>] ? btrfs_release_path+0xe/0x80
[44819.903509] [<ffffffff81216afa>] ? btrfs_start_dirty_block_groups+0x2da/0x420
[44819.903511] [<ffffffff812279f3>] ? btrfs_commit_transaction+0x143/0x990
[44819.903514] [<ffffffff8116a2c5>] ? kmem_cache_free+0x165/0x180
[44819.903516] [<ffffffff8124396c>] ? btrfs_wait_ordered_range+0x7c/0x110
[44819.903518] [<ffffffff8123ecf6>] ? btrfs_sync_file+0x286/0x360
[44819.903522] [<ffffffff811ae343>] ? do_fsync+0x33/0x60
[44819.903524] [<ffffffff811ae57a>] ? SyS_fdatasync+0xa/0x10
[44819.903528] [<ffffffff8162299b>] ? entry_SYSCALL_64_fastpath+0x13/0x8f
[44819.903529] ---[ end trace 6944811e170a0e57 ]---
[44819.903531] BTRFS: error (device bcache2) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
[44819.903533] BTRFS info (device bcache2): forced readonly
Since I had to get back up and running fast, I restored from backup. I
now bought some extra 3TB backup space and created a rescue system
including all tools on a USB stick, so next time it happens I may be
able to create an image of the broken filesystem.
Btrfs --repair refused to repair the filesystem telling me something
about compressed extents and an unsupported case, wanting me to take an
image and send it to the devs. *sigh*
System is kernel 4.7.2, Gentoo Linux, latest VirtualBox stable.
VirtualBox was using VDI image format without nocow. I now reverted
back to using nocow on VDI files and hope it doesn't strike again too
soon. I didn't try again yet, first I need to refresh my backup which
takes a while.
The filesystem runs on 3x SATA 1TB mraid1 draid0 through bcache in
writeback mode, backed by a 500GB 850 Evo - if that matters.
The problem occurred during high IO on 4.7.2. I previously ran 4.6.6
which didn't show this problem. Part of the culprit may be that I was
using bfq patches - I removed them for now and went back to deadline io
scheduler. The bfq patches froze my system a few times when I booted
4.7.2 which may already have broken my btrfs (although it shouldn't,
right? btrfs is transactional). Last time this happened (on an earlier
kernel), bfq may have been part of the problem, too. So I think bfq
does something to btrfs which may break the fs, or at least interferes
badly with the transaction as otherwise it shouldn't break. You may
want to run your test suites with bfq also (or different io schedulers
in general).
My home partition is mounted as a subvolume:
/dev/bcache0 on /home type btrfs (rw,noatime,compress=lzo,nossd,space_cache,autodefrag,subvolid=261,subvol=/home)
--
Regards,
Kai
Replies to list-only preferred.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2016-08-28 13:29 [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists Kai Krakow
@ 2017-02-02 12:01 ` Marc Joliet
2017-02-03 22:44 ` Kai Krakow
2017-03-02 0:43 ` Qu Wenruo
0 siblings, 2 replies; 23+ messages in thread
From: Marc Joliet @ 2017-02-02 12:01 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 21084 bytes --]
On Sunday 28 August 2016 15:29:08 Kai Krakow wrote:
> Hello list!
Hi list
> It happened again. While using VirtualBox the following crash happened,
> btrfs check found a lot of errors which it couldn't repair. Earlier
> that day my system crashed which may already introduced errors into my
> filesystem. Apparently, I couldn't create an image (not enough space
> available), I only can give this trace from dmesg:
>
> [44819.903435] ------------[ cut here ]------------
> [44819.903443] WARNING: CPU: 3 PID: 2787 at fs/btrfs/extent-tree.c:2963
> btrfs_run_delayed_refs+0x26c/0x290 [44819.903444] BTRFS: Transaction
> aborted (error -17)
> [44819.903445] Modules linked in: nls_iso8859_15 nls_cp437 vfat fat fuse
> rfcomm veth af_packet ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp
> llc w83627ehf bnep hwmon_vid cachefiles btusb btintel bluetooth
> snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic
> snd_hda_intel snd_hda_codec rfkill snd_hwdep snd_hda_core snd_pcm snd_timer
> coretemp hwmon snd r8169 mii kvm_intel kvm iTCO_wdt iTCO_vendor_support
> rtc_cmos irqbypass soundcore ip_tables uas usb_storage nvidia_drm(PO)
> vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) nvidia_modeset(PO)
> nvidia(PO) efivarfs unix ipv6 [44819.903484] CPU: 3 PID: 2787 Comm:
> BrowserBlocking Tainted: P O 4.7.2-gentoo #2 [44819.903485]
> Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Pro3, BIOS
> L2.16A 02/22/2013 [44819.903487] 0000000000000000 ffffffff8130af2d
> ffff8800b7d03d20 0000000000000000 [44819.903489] ffffffff810865fa
> ffff880409374428 ffff8800b7d03d70 ffff8803bf299760 [44819.903491]
> 0000000000000000 00000000ffffffef ffff8803f677f000 ffffffff8108666a
> [44819.903493] Call Trace:
> [44819.903496] [<ffffffff8130af2d>] ? dump_stack+0x46/0x59
> [44819.903500] [<ffffffff810865fa>] ? __warn+0xba/0xe0
> [44819.903502] [<ffffffff8108666a>] ? warn_slowpath_fmt+0x4a/0x50
> [44819.903504] [<ffffffff8121351c>] ? btrfs_run_delayed_refs+0x26c/0x290
> [44819.903507] [<ffffffff811feb1e>] ? btrfs_release_path+0xe/0x80
> [44819.903509] [<ffffffff81216afa>] ?
> btrfs_start_dirty_block_groups+0x2da/0x420 [44819.903511]
> [<ffffffff812279f3>] ? btrfs_commit_transaction+0x143/0x990 [44819.903514]
> [<ffffffff8116a2c5>] ? kmem_cache_free+0x165/0x180 [44819.903516]
> [<ffffffff8124396c>] ? btrfs_wait_ordered_range+0x7c/0x110 [44819.903518]
> [<ffffffff8123ecf6>] ? btrfs_sync_file+0x286/0x360 [44819.903522]
> [<ffffffff811ae343>] ? do_fsync+0x33/0x60
> [44819.903524] [<ffffffff811ae57a>] ? SyS_fdatasync+0xa/0x10
> [44819.903528] [<ffffffff8162299b>] ? entry_SYSCALL_64_fastpath+0x13/0x8f
> [44819.903529] ---[ end trace 6944811e170a0e57 ]---
> [44819.903531] BTRFS: error (device bcache2) in btrfs_run_delayed_refs:2963:
> errno=-17 Object already exists [44819.903533] BTRFS info (device bcache2):
> forced readonly
I got the same error myself, with this stack trace:
-- Logs begin at Fr 2016-04-01 17:07:28 CEST, end at Mi 2017-02-01 22:03:57
CET. --
Feb 01 01:46:26 diefledermaus kernel: ------------[ cut here ]------------
Feb 01 01:46:26 diefledermaus kernel: WARNING: CPU: 1 PID: 16727 at
fs/btrfs/extent-tree.c:2967 btrfs_run_delayed_refs+0x278/0x2b0
Feb 01 01:46:26 diefledermaus kernel: BTRFS: Transaction aborted (error -17)
Feb 01 01:46:26 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 01 01:46:27 diefledermaus kernel: BTRFS info (device sdb2): forced
readonly
Feb 01 01:46:27 diefledermaus kernel: Modules linked in: msr ctr ccm tun arc4
snd_hda_codec_idt applesmc snd_hda_codec_generic input_polldev hwmon
snd_hda_intel ath5k snd_hda_codec mac80211 snd_hda_core ath snd_pcm cfg80211
snd_timer video acpi_cpufreq snd backlight sky2 rfkill processor button
soundcore sg usb_storage sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 01 01:46:27 diefledermaus kernel: CPU: 1 PID: 16727 Comm: kworker/u4:0 Not
tainted 4.9.6-gentoo #1
Feb 01 01:46:27 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 01 01:46:27 diefledermaus kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper
Feb 01 01:46:27 diefledermaus kernel: 0000000000000000 ffffffff812cf739
ffffc9000285fd60 0000000000000000
Feb 01 01:46:27 diefledermaus kernel: ffffffff8104908a ffff8800428df1e0
ffffc9000285fdb0 0000000000000020
Feb 01 01:46:27 diefledermaus kernel: ffff880003c1b1b8 ffff8800bb73e900
0000000000000000 ffffffff810490fa
Feb 01 01:46:27 diefledermaus kernel: Call Trace:
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff812cf739>] ?
dump_stack+0x46/0x5d
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff810490fa>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81201fa8>] ?
btrfs_run_delayed_refs+0x278/0x2b0
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81202064>] ?
delayed_ref_async_start+0x84/0xa0
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105ce76>] ?
process_one_work+0x126/0x310
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105a893>] ?
pwq_activate_delayed_work+0x33/0x60
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105d0a5>] ?
worker_thread+0x45/0x450
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105d060>] ?
process_one_work+0x310/0x310
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105d060>] ?
process_one_work+0x310/0x310
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8104c7e7>] ?
do_group_exit+0x37/0xa0
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81061c99>] ?
kthread+0xb9/0xd0
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81061be0>] ?
kthread_park+0x50/0x50
Feb 01 01:46:27 diefledermaus kernel: [<ffffffff814f6252>] ?
ret_from_fork+0x22/0x30
Feb 01 01:46:27 diefledermaus kernel: ---[ end trace 5d090b8f212318e0 ]---
Feb 01 01:46:27 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 01 07:33:52 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:52 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:53 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:53 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:53 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:53 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:54 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:54 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:54 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 07:33:54 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:33 diefledermaus kernel: verify_parent_transid: 14 callbacks
suppressed
Feb 01 09:03:34 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:34 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:34 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:34 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:35 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:35 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:35 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:35 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:36 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:36 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: verify_parent_transid: 10 callbacks
suppressed
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:10 diefledermaus kernel: verify_parent_transid: 14 callbacks
suppressed
Feb 01 09:04:10 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:10 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:10 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:10 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:11 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:11 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:11 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:11 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:11 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:11 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:19 diefledermaus kernel: verify_parent_transid: 14 callbacks
suppressed
Feb 01 09:04:19 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:04:19 diefledermaus kernel: BTRFS error (device sdb2): parent
transid verify failed on 4722283290624 wanted 283904 found 194340
Feb 01 09:10:20 diefledermaus kernel: BTRFS error (device sdb2): cleaner
transaction attach returned -30
Feb 01 09:11:41 diefledermaus kernel: usb 1-6: USB disconnect, device number 2
Feb 01 09:11:41 diefledermaus kernel: sd 6:0:0:0: [sdb] Synchronizing SCSI
cache
Feb 01 09:11:41 diefledermaus kernel: sd 6:0:0:0: [sdb] Synchronize Cache(10)
failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
(This is the entire dmesg output for that day up to that time. The non-dmesg
log before that is just getmail running (on a different FS) with the previous
entries a few minutes beforehand, and afterwards it's just a bunch of SSH
related messages from btrbk trying to continue the backups, followed by
unrelated log entries.)
> Since I had to get back up and running fast, I restored from backup. I
> now bought some extra 3TB backup space and created a rescue system
> including all tools on a USB stick, so next time it happens I may be
> able to create an image of the broken filesystem.
I wanted to create an image, but btrfs-image segfaults after a few seconds.
> Btrfs --repair refused to repair the filesystem telling me something
> about compressed extents and an unsupported case, wanting me to take an
> image and send it to the devs. *sigh*
I haven't tried a repair yet; it's a big file system, and btrfs-check is still
running:
# btrfs check -p /dev/sdd2
Checking filesystem on /dev/sdd2
UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
parent transid verify failed on 3829276291072 wanted 224274 found 283858
parent transid verify failed on 3829276291072 wanted 224274 found 283858
parent transid verify failed on 3829276291072 wanted 224274 found 283858
parent transid verify failed on 3829276291072 wanted 224274 found 283858
Ignoring transid failure
leaf parent key incorrect 3829276291072
bad block 3829276291072
ERROR: errors found in extent allocation tree or chunk allocation
block group 4722282987520 has wrong amount of free space
failed to load free space cache for block group 4722282987520
checking free space cache [O]
root 32018 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32089 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32091 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32092 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32107 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32189 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32190 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32191 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32265 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32266 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32409 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32410 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32411 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32412 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32413 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32631 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32632 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32633 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32634 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32635 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32636 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32718 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32732 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
checking fs roots [o]
I know that the "file extend discount" errors are fixable from my previous
email to this ML, but what about the rest? From looking through the ML
archives it seems that --repair won't be able to fix the transid failures. It
seems that one person had success with the "usebackuproot" mount option, which
I haven't tried yet.
> System is kernel 4.7.2, Gentoo Linux, latest VirtualBox stable.
> VirtualBox was using VDI image format without nocow. I now reverted
> back to using nocow on VDI files and hope it doesn't strike again too
> soon. I didn't try again yet, first I need to refresh my backup which
> takes a while.
>
> The filesystem runs on 3x SATA 1TB mraid1 draid0 through bcache in
> writeback mode, backed by a 500GB 850 Evo - if that matters.
>
> The problem occurred during high IO on 4.7.2. I previously ran 4.6.6
> which didn't show this problem. Part of the culprit may be that I was
> using bfq patches - I removed them for now and went back to deadline io
> scheduler. The bfq patches froze my system a few times when I booted
> 4.7.2 which may already have broken my btrfs (although it shouldn't,
> right? btrfs is transactional). Last time this happened (on an earlier
> kernel), bfq may have been part of the problem, too. So I think bfq
> does something to btrfs which may break the fs, or at least interferes
> badly with the transaction as otherwise it shouldn't break. You may
> want to run your test suites with bfq also (or different io schedulers
> in general).
>
> My home partition is mounted as a subvolume:
> /dev/bcache0 on /home type btrfs
> (rw,noatime,compress=lzo,nossd,space_cache,autodefrag,subvolid=261,subvol=/
> home)
The system the drive runs on is:
% uname -a
Linux diefledermaus 4.9.7-gentoo #1 SMP Wed Feb 1 23:52:56 CET 2017 x86_64
Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz GenuineIntel GNU/Linux
However during the crash it was running 4.9.6-gentoo. The system uses the
standard CFQ scheduler, so perhaps BFQ is not at fault in Kai's case.
The system I am running btrfs-check on is:
% uname -a
Linux thetick 4.9.6-gentoo #1 SMP PREEMPT Fri Jan 27 00:50:02 CET 2017 x86_64
AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux
Both have btrfs-progs 4.9:
% /sbin/btrfs --version
btrfs-progs v4.9
And the file system in question:
% sudo /sbin/btrfs fi show /dev/sdd2
Label: 'MARCEC_BACKUP' uuid: f97b3cda-15e8-418b-bb9b-235391ef2a38
Total devices 1 FS bytes used 842.50GiB
devid 1 size 976.56GiB used 877.31GiB path /dev/sdd2
The file system is mounted with "noatime,compress,comment=systemd.automount".
In my case the crash also happened during high I/O load (three btrfs-
send/receive backups running at the same time). If "usebackuproot" (now
called "recovery"?) fails, then I'll just wipe the FS and start the backups
from scratch.
Since I would like to have that done by Saturday: is there any information I
can provide that might help fix whatever bug(s) caused this? Should I file a
bug if one doesn't exist yet (I haven't checked yet, sorry)?
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-02 12:01 ` Marc Joliet
@ 2017-02-03 22:44 ` Kai Krakow
2017-02-10 22:15 ` Marc Joliet
2017-03-02 0:43 ` Qu Wenruo
1 sibling, 1 reply; 23+ messages in thread
From: Kai Krakow @ 2017-02-03 22:44 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 21975 bytes --]
Am Thu, 02 Feb 2017 13:01:03 +0100
schrieb Marc Joliet <marcec@gmx.de>:
> On Sunday 28 August 2016 15:29:08 Kai Krakow wrote:
> > Hello list!
>
> Hi list
>
> > It happened again. While using VirtualBox the following crash
> > happened, btrfs check found a lot of errors which it couldn't
> > repair. Earlier that day my system crashed which may already
> > introduced errors into my filesystem. Apparently, I couldn't create
> > an image (not enough space available), I only can give this trace
> > from dmesg:
> >
> > [44819.903435] ------------[ cut here ]------------
> > [44819.903443] WARNING: CPU: 3 PID: 2787 at
> > fs/btrfs/extent-tree.c:2963 btrfs_run_delayed_refs+0x26c/0x290
> > [44819.903444] BTRFS: Transaction aborted (error -17)
> > [44819.903445] Modules linked in: nls_iso8859_15 nls_cp437 vfat fat
> > fuse rfcomm veth af_packet ipt_MASQUERADE nf_nat_masquerade_ipv4
> > iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> > nf_conntrack bridge stp llc w83627ehf bnep hwmon_vid cachefiles
> > btusb btintel bluetooth snd_hda_codec_hdmi snd_hda_codec_realtek
> > snd_hda_codec_generic snd_hda_intel snd_hda_codec rfkill snd_hwdep
> > snd_hda_core snd_pcm snd_timer coretemp hwmon snd r8169 mii
> > kvm_intel kvm iTCO_wdt iTCO_vendor_support rtc_cmos irqbypass
> > soundcore ip_tables uas usb_storage nvidia_drm(PO) vboxpci(O)
> > vboxnetadp(O) vboxnetflt(O) vboxdrv(O) nvidia_modeset(PO)
> > nvidia(PO) efivarfs unix ipv6 [44819.903484] CPU: 3 PID: 2787 Comm:
> > BrowserBlocking Tainted: P O 4.7.2-gentoo #2
> > [44819.903485] Hardware name: To Be Filled By O.E.M. To Be Filled
> > By O.E.M./Z68 Pro3, BIOS L2.16A 02/22/2013 [44819.903487]
> > 0000000000000000 ffffffff8130af2d ffff8800b7d03d20 0000000000000000
> > [44819.903489] ffffffff810865fa ffff880409374428 ffff8800b7d03d70
> > ffff8803bf299760 [44819.903491] 0000000000000000 00000000ffffffef
> > ffff8803f677f000 ffffffff8108666a [44819.903493] Call Trace:
> > [44819.903496] [<ffffffff8130af2d>] ? dump_stack+0x46/0x59
> > [44819.903500] [<ffffffff810865fa>] ? __warn+0xba/0xe0
> > [44819.903502] [<ffffffff8108666a>] ? warn_slowpath_fmt+0x4a/0x50
> > [44819.903504] [<ffffffff8121351c>] ?
> > btrfs_run_delayed_refs+0x26c/0x290 [44819.903507]
> > [<ffffffff811feb1e>] ? btrfs_release_path+0xe/0x80 [44819.903509]
> > [<ffffffff81216afa>] ? btrfs_start_dirty_block_groups+0x2da/0x420
> > [44819.903511] [<ffffffff812279f3>] ?
> > btrfs_commit_transaction+0x143/0x990 [44819.903514]
> > [<ffffffff8116a2c5>] ? kmem_cache_free+0x165/0x180 [44819.903516]
> > [<ffffffff8124396c>] ? btrfs_wait_ordered_range+0x7c/0x110
> > [44819.903518] [<ffffffff8123ecf6>] ? btrfs_sync_file+0x286/0x360
> > [44819.903522] [<ffffffff811ae343>] ? do_fsync+0x33/0x60
> > [44819.903524] [<ffffffff811ae57a>] ? SyS_fdatasync+0xa/0x10
> > [44819.903528] [<ffffffff8162299b>] ?
> > entry_SYSCALL_64_fastpath+0x13/0x8f [44819.903529] ---[ end trace
> > 6944811e170a0e57 ]--- [44819.903531] BTRFS: error (device bcache2)
> > in btrfs_run_delayed_refs:2963: errno=-17 Object already exists
> > [44819.903533] BTRFS info (device bcache2): forced readonly
>
> I got the same error myself, with this stack trace:
>
> -- Logs begin at Fr 2016-04-01 17:07:28 CEST, end at Mi 2017-02-01
> 22:03:57 CET. --
> Feb 01 01:46:26 diefledermaus kernel: ------------[ cut here
> ]------------ Feb 01 01:46:26 diefledermaus kernel: WARNING: CPU: 1
> PID: 16727 at fs/btrfs/extent-tree.c:2967
> btrfs_run_delayed_refs+0x278/0x2b0 Feb 01 01:46:26 diefledermaus
> kernel: BTRFS: Transaction aborted (error -17) Feb 01 01:46:26
> diefledermaus kernel: BTRFS: error (device sdb2) in
> btrfs_run_delayed_refs:2967: errno=-17 Object already exists Feb 01
> 01:46:27 diefledermaus kernel: BTRFS info (device sdb2): forced
> readonly Feb 01 01:46:27 diefledermaus kernel: Modules linked in: msr
> ctr ccm tun arc4 snd_hda_codec_idt applesmc snd_hda_codec_generic
> input_polldev hwmon snd_hda_intel ath5k snd_hda_codec mac80211
> snd_hda_core ath snd_pcm cfg80211 snd_timer video acpi_cpufreq snd
> backlight sky2 rfkill processor button soundcore sg usb_storage
> sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci libahci ata_piix
> libata ehci_pci ehci_hcd Feb 01 01:46:27 diefledermaus kernel: CPU: 1
> PID: 16727 Comm: kworker/u4:0 Not tainted 4.9.6-gentoo #1
> Feb 01 01:46:27 diefledermaus kernel: Hardware name: Apple Inc.
> Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359
> 06/28/07 Feb 01 01:46:27 diefledermaus kernel: Workqueue:
> btrfs-extent-refs btrfs_extent_refs_helper
> Feb 01 01:46:27 diefledermaus kernel: 0000000000000000
> ffffffff812cf739 ffffc9000285fd60 0000000000000000
> Feb 01 01:46:27 diefledermaus kernel: ffffffff8104908a
> ffff8800428df1e0 ffffc9000285fdb0 0000000000000020
> Feb 01 01:46:27 diefledermaus kernel: ffff880003c1b1b8
> ffff8800bb73e900 0000000000000000 ffffffff810490fa
> Feb 01 01:46:27 diefledermaus kernel: Call Trace:
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff812cf739>] ?
> dump_stack+0x46/0x5d
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8104908a>] ?
> __warn+0xba/0xe0 Feb 01 01:46:27 diefledermaus kernel:
> [<ffffffff810490fa>] ? warn_slowpath_fmt+0x4a/0x50
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81201fa8>] ?
> btrfs_run_delayed_refs+0x278/0x2b0
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81202064>] ?
> delayed_ref_async_start+0x84/0xa0
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105ce76>] ?
> process_one_work+0x126/0x310
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105a893>] ?
> pwq_activate_delayed_work+0x33/0x60
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105d0a5>] ?
> worker_thread+0x45/0x450
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105d060>] ?
> process_one_work+0x310/0x310
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8105d060>] ?
> process_one_work+0x310/0x310
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff8104c7e7>] ?
> do_group_exit+0x37/0xa0
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81061c99>] ?
> kthread+0xb9/0xd0
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff81061be0>] ?
> kthread_park+0x50/0x50
> Feb 01 01:46:27 diefledermaus kernel: [<ffffffff814f6252>] ?
> ret_from_fork+0x22/0x30
> Feb 01 01:46:27 diefledermaus kernel: ---[ end trace 5d090b8f212318e0
> ]--- Feb 01 01:46:27 diefledermaus kernel: BTRFS: error (device sdb2)
> in btrfs_run_delayed_refs:2967: errno=-17 Object already exists
> Feb 01 07:33:52 diefledermaus kernel: BTRFS error (device sdb2):
> parent transid verify failed on 4722283290624 wanted 283904 found
> 194340 Feb 01 07:33:52 diefledermaus kernel: BTRFS error (device
> sdb2): parent transid verify failed on 4722283290624 wanted 283904
> found 194340 Feb 01 07:33:53 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 07:33:53 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 07:33:53 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 07:33:53 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 07:33:54 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 07:33:54 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 07:33:54 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 07:33:54 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:33 diefledermaus kernel:
> verify_parent_transid: 14 callbacks suppressed
> Feb 01 09:03:34 diefledermaus kernel: BTRFS error (device sdb2):
> parent transid verify failed on 4722283290624 wanted 283904 found
> 194340 Feb 01 09:03:34 diefledermaus kernel: BTRFS error (device
> sdb2): parent transid verify failed on 4722283290624 wanted 283904
> found 194340 Feb 01 09:03:34 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:34 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:35 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:35 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:35 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:35 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:36 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:36 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel:
> verify_parent_transid: 10 callbacks suppressed
> Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device sdb2):
> parent transid verify failed on 4722283290624 wanted 283904 found
> 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error (device
> sdb2): parent transid verify failed on 4722283290624 wanted 283904
> found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:03:39 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:10 diefledermaus kernel:
> verify_parent_transid: 14 callbacks suppressed
> Feb 01 09:04:10 diefledermaus kernel: BTRFS error (device sdb2):
> parent transid verify failed on 4722283290624 wanted 283904 found
> 194340 Feb 01 09:04:10 diefledermaus kernel: BTRFS error (device
> sdb2): parent transid verify failed on 4722283290624 wanted 283904
> found 194340 Feb 01 09:04:10 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:10 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:11 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:11 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:11 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:11 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:11 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:11 diefledermaus kernel: BTRFS error
> (device sdb2): parent transid verify failed on 4722283290624 wanted
> 283904 found 194340 Feb 01 09:04:19 diefledermaus kernel:
> verify_parent_transid: 14 callbacks suppressed
> Feb 01 09:04:19 diefledermaus kernel: BTRFS error (device sdb2):
> parent transid verify failed on 4722283290624 wanted 283904 found
> 194340 Feb 01 09:04:19 diefledermaus kernel: BTRFS error (device
> sdb2): parent transid verify failed on 4722283290624 wanted 283904
> found 194340 Feb 01 09:10:20 diefledermaus kernel: BTRFS error
> (device sdb2): cleaner transaction attach returned -30
> Feb 01 09:11:41 diefledermaus kernel: usb 1-6: USB disconnect, device
> number 2 Feb 01 09:11:41 diefledermaus kernel: sd 6:0:0:0: [sdb]
> Synchronizing SCSI cache
> Feb 01 09:11:41 diefledermaus kernel: sd 6:0:0:0: [sdb] Synchronize
> Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
>
> (This is the entire dmesg output for that day up to that time. The
> non-dmesg log before that is just getmail running (on a different FS)
> with the previous entries a few minutes beforehand, and afterwards
> it's just a bunch of SSH related messages from btrbk trying to
> continue the backups, followed by unrelated log entries.)
>
> > Since I had to get back up and running fast, I restored from
> > backup. I now bought some extra 3TB backup space and created a
> > rescue system including all tools on a USB stick, so next time it
> > happens I may be able to create an image of the broken filesystem.
>
> I wanted to create an image, but btrfs-image segfaults after a few
> seconds.
>
> > Btrfs --repair refused to repair the filesystem telling me something
> > about compressed extents and an unsupported case, wanting me to
> > take an image and send it to the devs. *sigh*
>
> I haven't tried a repair yet; it's a big file system, and btrfs-check
> is still running:
>
> # btrfs check -p /dev/sdd2
> Checking filesystem on /dev/sdd2
> UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
> parent transid verify failed on 3829276291072 wanted 224274 found
> 283858 parent transid verify failed on 3829276291072 wanted 224274
> found 283858 parent transid verify failed on 3829276291072 wanted
> 224274 found 283858 parent transid verify failed on 3829276291072
> wanted 224274 found 283858 Ignoring transid failure
> leaf parent key incorrect 3829276291072
> bad block 3829276291072
>
> ERROR: errors found in extent allocation tree or chunk allocation
> block group 4722282987520 has wrong amount of free space
> failed to load free space cache for block group 4722282987520
> checking free space cache [O]
> root 32018 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32089 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32091 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32092 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32107 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32189 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32190 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32191 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32265 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32266 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32409 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32410 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32411 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32412 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32413 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32631 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32632 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32633 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32634 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32635 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32636 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32718 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32732 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> checking fs roots [o]
>
> I know that the "file extend discount" errors are fixable from my
> previous email to this ML, but what about the rest? From looking
> through the ML archives it seems that --repair won't be able to fix
> the transid failures. It seems that one person had success with the
> "usebackuproot" mount option, which I haven't tried yet.
>
> > System is kernel 4.7.2, Gentoo Linux, latest VirtualBox stable.
> > VirtualBox was using VDI image format without nocow. I now reverted
> > back to using nocow on VDI files and hope it doesn't strike again
> > too soon. I didn't try again yet, first I need to refresh my backup
> > which takes a while.
> >
> > The filesystem runs on 3x SATA 1TB mraid1 draid0 through bcache in
> > writeback mode, backed by a 500GB 850 Evo - if that matters.
> >
> > The problem occurred during high IO on 4.7.2. I previously ran 4.6.6
> > which didn't show this problem. Part of the culprit may be that I
> > was using bfq patches - I removed them for now and went back to
> > deadline io scheduler. The bfq patches froze my system a few times
> > when I booted 4.7.2 which may already have broken my btrfs
> > (although it shouldn't, right? btrfs is transactional). Last time
> > this happened (on an earlier kernel), bfq may have been part of the
> > problem, too. So I think bfq does something to btrfs which may
> > break the fs, or at least interferes badly with the transaction as
> > otherwise it shouldn't break. You may want to run your test suites
> > with bfq also (or different io schedulers in general).
> >
> > My home partition is mounted as a subvolume:
> > /dev/bcache0 on /home type btrfs
> > (rw,noatime,compress=lzo,nossd,space_cache,autodefrag,subvolid=261,subvol=/
> > home)
>
> The system the drive runs on is:
>
> % uname -a
> Linux diefledermaus 4.9.7-gentoo #1 SMP Wed Feb 1 23:52:56 CET 2017
> x86_64 Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz GenuineIntel GNU/Linux
>
> However during the crash it was running 4.9.6-gentoo. The system
> uses the standard CFQ scheduler, so perhaps BFQ is not at fault in
> Kai's case.
>
> The system I am running btrfs-check on is:
>
> % uname -a
> Linux thetick 4.9.6-gentoo #1 SMP PREEMPT Fri Jan 27 00:50:02 CET
> 2017 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
> AuthenticAMD GNU/Linux
>
> Both have btrfs-progs 4.9:
>
> % /sbin/btrfs --version
> btrfs-progs v4.9
>
> And the file system in question:
>
> % sudo /sbin/btrfs fi show /dev/sdd2
> Label: 'MARCEC_BACKUP' uuid:
> f97b3cda-15e8-418b-bb9b-235391ef2a38 Total devices 1 FS bytes used
> 842.50GiB devid 1 size 976.56GiB used 877.31GiB path /dev/sdd2
>
> The file system is mounted with
> "noatime,compress,comment=systemd.automount".
So our common denominator is only "compress". Well, and systemd and
Gentoo - but that probably doesn't count.
It looks like you're running without bcache and on a single disk.
Do you use files that should maybe better be marked nocow? (chattr
+C)... in terms like VirtualBox images or database files?
I'm running 4.9.6-gentoo also and this problem didn't happen again yet.
And I switched from deadline back to cfq (but I will probably revert
this).
I'm backing up with borgbackup to an internal drive running XFS, this
runs flawless so far and without major performance impacts.
> In my case the crash also happened during high I/O load (three btrfs-
> send/receive backups running at the same time). If
> "usebackuproot" (now called "recovery"?) fails, then I'll just wipe
> the FS and start the backups from scratch.
>
> Since I would like to have that done by Saturday: is there any
> information I can provide that might help fix whatever bug(s) caused
> this? Should I file a bug if one doesn't exist yet (I haven't
> checked yet, sorry)?
>
> Greetings
--
Regards,
Kai
Replies to list-only preferred.
[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 163 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-03 22:44 ` Kai Krakow
@ 2017-02-10 22:15 ` Marc Joliet
2017-02-11 2:01 ` Kai Krakow
0 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-02-10 22:15 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 20371 bytes --]
Sorry for the late reply, see below for why :) .
On Friday 03 February 2017 23:44:10 Kai Krakow wrote:
> Am Thu, 02 Feb 2017 13:01:03 +0100
>
> schrieb Marc Joliet <marcec@gmx.de>:
[...]
> > > Btrfs --repair refused to repair the filesystem telling me something
> > > about compressed extents and an unsupported case, wanting me to
> > > take an image and send it to the devs. *sigh*
> >
> > I haven't tried a repair yet; it's a big file system, and btrfs-check
> > is still running:
> >
> > # btrfs check -p /dev/sdd2
> > Checking filesystem on /dev/sdd2
> > UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
> > parent transid verify failed on 3829276291072 wanted 224274 found
> > 283858 parent transid verify failed on 3829276291072 wanted 224274
> > found 283858 parent transid verify failed on 3829276291072 wanted
> > 224274 found 283858 parent transid verify failed on 3829276291072
> > wanted 224274 found 283858 Ignoring transid failure
> > leaf parent key incorrect 3829276291072
> > bad block 3829276291072
> >
> > ERROR: errors found in extent allocation tree or chunk allocation
> > block group 4722282987520 has wrong amount of free space
> > failed to load free space cache for block group 4722282987520
> > checking free space cache [O]
> > root 32018 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32089 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32091 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32092 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32107 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32189 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32190 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32191 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32265 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32266 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32409 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32410 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32411 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32412 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32413 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32631 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32632 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32633 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32634 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32635 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32636 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32718 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32732 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > checking fs roots [o]
> >
> > I know that the "file extend discount" errors are fixable from my
> > previous email to this ML, but what about the rest? From looking
> > through the ML archives it seems that --repair won't be able to fix
> > the transid failures. It seems that one person had success with the
> > "usebackuproot" mount option, which I haven't tried yet.
Well, it turns out that the problem fixed itself. After check finished, I
tried mounting the FS normally, which succeeded. Shortly after, btrfs-cleaner
started removing deleted snapshots. About 40 minutes later I got the error
again:
Feb 02 22:49:14 thetick kernel: BTRFS: device label MARCEC_BACKUP devid 1
transid 283903 /dev/sdd2
Feb 02 22:49:19 thetick kernel: EXT4-fs (sdd1): mounted filesystem with
ordered data mode. Opts: (null)
Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): use zlib compression
Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): disk space caching
is enabled
Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): has skinny extents
Feb 03 00:20:09 thetick kernel: BTRFS info (device sdd2): The free space cache
file (3967375376384) is invalid. skip it
Feb 03 01:05:58 thetick kernel: ------------[ cut here ]------------
Feb 03 01:05:58 thetick kernel: WARNING: CPU: 1 PID: 26544 at fs/btrfs/extent-
tree.c:2967 btrfs_run_delayed_refs+0x26c/0x290
Feb 03 01:05:58 thetick kernel: BTRFS: Transaction aborted (error -17)
Feb 03 01:05:58 thetick kernel: Modules linked in: dummy iptable_filter
ip_tables x_tables joydev hid_logitech_hidpp hid_logitech_dj
snd_hda_codec_hdmi snd_hda_codec_analog snd_hda_codec_generic uvcvideo
videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core sn
Feb 03 01:05:58 thetick kernel: ehci_pci ohci_hcd ehci_hcd
Feb 03 01:05:58 thetick kernel: CPU: 1 PID: 26544 Comm: btrfs-cleaner Not
tainted 4.9.7-gentoo #1
Feb 03 01:05:58 thetick kernel: Hardware name: System manufacturer System
Product Name/M2N-E, BIOS ASUS M2N-E ACPI BIOS Revision 1701 10/30/2008
Feb 03 01:05:58 thetick kernel: 0000000000000000 ffffffff8137af83
ffffc90000a4fd30 0000000000000000
Feb 03 01:05:58 thetick kernel: ffffffff810537ad ffff880138b2ec80
ffffc90000a4fd80 ffff880138b4e850
Feb 03 01:05:58 thetick kernel: 00000000ffffffef 00000000000000f2
ffff880138b2ec80 ffffffff8105381a
Feb 03 01:05:58 thetick kernel: Call Trace:
Feb 03 01:05:58 thetick kernel: [<ffffffff8137af83>] ? dump_stack+0x46/0x63
Feb 03 01:05:58 thetick kernel: [<ffffffff810537ad>] ? __warn+0xed/0x110
Feb 03 01:05:58 thetick kernel: [<ffffffff8105381a>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 03 01:05:58 thetick kernel: [<ffffffff81286f7a>] ?
__btrfs_run_delayed_refs+0x11ba/0x1390
Feb 03 01:05:58 thetick kernel: [<ffffffff8128a2ec>] ?
btrfs_run_delayed_refs+0x26c/0x290
Feb 03 01:05:58 thetick kernel: [<ffffffff81287fbe>] ?
walk_up_tree+0xde/0x1d0
Feb 03 01:05:58 thetick kernel: [<ffffffff8129f6d5>] ?
btrfs_should_end_transaction+0x45/0x80
Feb 03 01:05:58 thetick kernel: [<ffffffff81288851>] ?
btrfs_drop_snapshot+0x401/0x7a0
Feb 03 01:05:58 thetick kernel: [<ffffffff812a09f6>] ?
btrfs_clean_one_deleted_snapshot+0xb6/0xf0
Feb 03 01:05:58 thetick kernel: [<ffffffff8129757f>] ?
cleaner_kthread+0x13f/0x1b0
Feb 03 01:05:58 thetick kernel: [<ffffffff81297440>] ?
btrfs_destroy_pinned_extent+0xc0/0xc0
Feb 03 01:05:58 thetick kernel: [<ffffffff81070ee9>] ? kthread+0xb9/0xd0
Feb 03 01:05:58 thetick kernel: [<ffffffff81070e30>] ? kthread_park+0x70/0x70
Feb 03 01:05:58 thetick kernel: [<ffffffff8160dfd2>] ?
ret_from_fork+0x22/0x30
Feb 03 01:05:58 thetick kernel: ---[ end trace 9b83fb0ab757cd99 ]---
Feb 03 01:05:58 thetick kernel: BTRFS: error (device sdd2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 03 01:05:58 thetick kernel: BTRFS info (device sdd2): forced readonly
Feb 03 01:26:39 thetick kernel: BTRFS error (device sdd2): cleaner transaction
attach returned -30
Feb 03 01:26:59 thetick kernel: BTRFS info (device sdd2): use zlib compression
Feb 03 01:26:59 thetick kernel: BTRFS info (device sdd2): disk space caching
is enabled
Feb 03 01:26:59 thetick kernel: BTRFS info (device sdd2): has skinny extents
Feb 03 01:27:44 thetick kernel: BTRFS info (device sdd2): The free space cache
file (4690842484736) is invalid. skip it
I then mounted it again, after which btrfs-cleaner resumed its task:
Feb 03 09:54:05 thetick kernel: BTRFS info (device sdd2): use zlib compression
Feb 03 09:54:05 thetick kernel: BTRFS info (device sdd2): disk space caching
is enabled
Feb 03 09:54:05 thetick kernel: BTRFS info (device sdd2): has skinny extents
This time it finished successfully, after that there are no more messages
pertaining to that file system in dmesg.
I then ran btrfs-check again, which didn't find the transid errors any more:
# btrfs check -p /dev/sdd2
Checking filesystem on /dev/sdd2
UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
checking extents [o]
block group 4722282987520 has wrong amount of free space
failed to load free space cache for block group 4722282987520
checking free space cache [o]
root 32018 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32089 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32091 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32092 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32107 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32189 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32190 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32191 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32265 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32266 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32409 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32410 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32411 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32412 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32413 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32631 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32632 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32633 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32634 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32635 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32636 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32718 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32732 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32830 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32832 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32833 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32834 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32903 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 32961 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
root 33002 inode 95066 errors 100, file extent discount
Found file extent holes:
start: 413696, len: 4096
checking fs roots [O]
found 901576467411 bytes used err is 1
total csum bytes: 871524676
total tree bytes: 11151159296
total fs tree bytes: 8709533696
total extent tree bytes: 1307844608
btree space waste bytes: 2428036988
file data blocks allocated: 17604037283840
referenced 1003776770048
So, since the error was now limited to something btrfs-check could repair, I
ran check with --repair:
# btrfs check -p --repair /dev/sdd2
enabling repair mode
Checking filesystem on /dev/sdd2
UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
checking extents [O]o]
Fixed 0 roots.
cache and super generation don't match, space cache will be invalidated
Fixed discount file extents for inode: 95066 in root: 32018
Fixed discount file extents for inode: 95066 in root: 32089
Fixed discount file extents for inode: 95066 in root: 32091
Fixed discount file extents for inode: 95066 in root: 32092
Fixed discount file extents for inode: 95066 in root: 32107
Fixed discount file extents for inode: 95066 in root: 32189
Fixed discount file extents for inode: 95066 in root: 32190
Fixed discount file extents for inode: 95066 in root: 32191
Fixed discount file extents for inode: 95066 in root: 32265
Fixed discount file extents for inode: 95066 in root: 32266
Fixed discount file extents for inode: 95066 in root: 32409
Fixed discount file extents for inode: 95066 in root: 32410
Fixed discount file extents for inode: 95066 in root: 32411
Fixed discount file extents for inode: 95066 in root: 32412
Fixed discount file extents for inode: 95066 in root: 32413
Fixed discount file extents for inode: 95066 in root: 32631
Fixed discount file extents for inode: 95066 in root: 32632
Fixed discount file extents for inode: 95066 in root: 32633
Fixed discount file extents for inode: 95066 in root: 32634
Fixed discount file extents for inode: 95066 in root: 32635
Fixed discount file extents for inode: 95066 in root: 32636
Fixed discount file extents for inode: 95066 in root: 32718
Fixed discount file extents for inode: 95066 in root: 32732
Fixed discount file extents for inode: 95066 in root: 32830ts [O]
Fixed discount file extents for inode: 95066 in root: 32832
Fixed discount file extents for inode: 95066 in root: 32833
btrfs unable to find ref byte nr 4607605415936 parent 0 root 32834 owner 0
offset 1
Fixed discount file extents for inode: 95066 in root: 32834
Fixed discount file extents for inode: 95066 in root: 32903
Fixed discount file extents for inode: 95066 in root: 32961
Fixed discount file extents for inode: 95066 in root: 33002
warning line 4112 [.]
checking csums
checking root refs
found 901576467411 bytes used err is 0
total csum bytes: 871524676
total tree bytes: 11151159296
total fs tree bytes: 8709533696
total extent tree bytes: 1307844608
btree space waste bytes: 2428036988
file data blocks allocated: 17604037283840
referenced 1003776770048
This, BTW, *just* completed a few hours ago, i.e., it took almost a full week
(it started on Friday, February 3rd, at about 20:53:35, and finished today at
20:42:48). Since it required between about 75% and 85% of the measly 4GB of
RAM my desktop has, I decided not to use it for the duration of the --repair
run, which is why I haven't replied until now (since that's the machine I use
for private email). Running check without --repair, on the other hand, took
only 9-10 hours.
Right now I have a scrub running, the status of which is, at the moment of
writing:
# btrfs scrub status /media/MARCEC_BACKUP
scrub status for f97b3cda-15e8-418b-bb9b-235391ef2a38
scrub started at Fri Feb 10 21:00:36 2017, running for 02:14:05
total bytes scrubbed: 631.51GiB with 0 errors
(which is about 2x slower than expected so far)
Otherwise, since I have the FS mounted now, I can now show this:
# btrfs filesystem show /media/MARCEC_BACKUP/
Label: 'MARCEC_BACKUP' uuid: f97b3cda-15e8-418b-bb9b-235391ef2a38
Total devices 1 FS bytes used 841.75GiB
devid 1 size 976.56GiB used 878.25GiB path /dev/sdd2
# btrfs filesystem usage /media/MARCEC_BACKUP/
Overall:
Device size: 976.56GiB
Device allocated: 878.25GiB
Device unallocated: 98.31GiB
Device missing: 0.00B
Used: 852.13GiB
Free (estimated): 117.95GiB (min: 68.80GiB)
Data ratio: 1.00
Metadata ratio: 1.92
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:851.00GiB, Used:831.36GiB
/dev/sdd2 851.00GiB
Metadata,single: Size:1.12GiB, Used:0.00B
/dev/sdd2 1.12GiB
Metadata,DUP: Size:13.00GiB, Used:10.38GiB
/dev/sdd2 26.00GiB
System,DUP: Size:64.00MiB, Used:120.00KiB
/dev/sdd2 128.00MiB
Unallocated:
/dev/sdd2 98.31GiB
# btrfs filesystem df /media/MARCEC_BACKUP/
Data, single: total=851.00GiB, used=831.36GiB
System, DUP: total=64.00MiB, used=120.00KiB
Metadata, DUP: total=13.00GiB, used=10.38GiB
Metadata, single: total=1.12GiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B
Hmm, I take it that the single metadata is a leftover from running --repair?
[...]
> > The system the drive runs on is:
> >
> > % uname -a
> > Linux diefledermaus 4.9.7-gentoo #1 SMP Wed Feb 1 23:52:56 CET 2017
> > x86_64 Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz GenuineIntel GNU/Linux
> >
> > However during the crash it was running 4.9.6-gentoo. The system
> > uses the standard CFQ scheduler, so perhaps BFQ is not at fault in
> > Kai's case.
> >
> > The system I am running btrfs-check on is:
> >
> > % uname -a
> > Linux thetick 4.9.6-gentoo #1 SMP PREEMPT Fri Jan 27 00:50:02 CET
> > 2017 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
> > AuthenticAMD GNU/Linux
> >
> > Both have btrfs-progs 4.9:
> >
> > % /sbin/btrfs --version
> > btrfs-progs v4.9
> >
> > And the file system in question:
> >
> > % sudo /sbin/btrfs fi show /dev/sdd2
> > Label: 'MARCEC_BACKUP' uuid:
> > f97b3cda-15e8-418b-bb9b-235391ef2a38 Total devices 1 FS bytes used
> > 842.50GiB devid 1 size 976.56GiB used 877.31GiB path /dev/sdd2
> >
> > The file system is mounted with
> > "noatime,compress,comment=systemd.automount".
>
> So our common denominator is only "compress". Well, and systemd and
> Gentoo - but that probably doesn't count.
Yeah, probably not :) .
> It looks like you're running without bcache and on a single disk.
Exactly.
> Do you use files that should maybe better be marked nocow? (chattr
> +C)... in terms like VirtualBox images or database files?
Nope. It's used exclusively as a target for btrfs-receive and rsync (OK, my
music backups are also used by MPD on the home server it's normally attached
to, as an alternative to leaving my desktop running just to hear music, but
that's it).
[...]
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-10 22:15 ` Marc Joliet
@ 2017-02-11 2:01 ` Kai Krakow
2017-02-11 10:18 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Kai Krakow @ 2017-02-11 2:01 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2429 bytes --]
Am Fri, 10 Feb 2017 23:15:03 +0100
schrieb Marc Joliet <marcec@gmx.de>:
> # btrfs filesystem df /media/MARCEC_BACKUP/
> Data, single: total=851.00GiB, used=831.36GiB
> System, DUP: total=64.00MiB, used=120.00KiB
> Metadata, DUP: total=13.00GiB, used=10.38GiB
> Metadata, single: total=1.12GiB, used=0.00B
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> Hmm, I take it that the single metadata is a leftover from running
> --repair?
It's more probably a remnant of an incomplete balance operation or an
older mkfs version. I'd simply rebalance metadata to fix this.
I don't think that btrfs-repair would migrate missing metadata
duplicates back to single profile, it would more likely trigger
recreating the missing duplicates. But I'm not sure.
If it is a result of the repair operation, that could be an
interesting clue. Could it explain "error -17" from your logs? But that
would mean the duplicates were already missing before the repair
operation and triggered that problem. So the question is, why are those
duplicates missing in the first place as a result of normal operation?
From your logs:
---8<--- snip
Feb 02 22:49:14 thetick kernel: BTRFS: device label MARCEC_BACKUP devid
1 transid 283903 /dev/sdd2
Feb 02 22:49:19 thetick kernel: EXT4-fs (sdd1): mounted filesystem with
ordered data mode. Opts: (null)
Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): use zlib
compression Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2):
disk space caching is enabled
Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): has skinny
extents Feb 03 00:20:09 thetick kernel: BTRFS info (device sdd2): The
free space cache file (3967375376384) is invalid. skip it
Feb 03 01:05:58 thetick kernel: ------------[ cut here ]------------
Feb 03 01:05:58 thetick kernel: WARNING: CPU: 1 PID: 26544 at
fs/btrfs/extent- tree.c:2967 btrfs_run_delayed_refs+0x26c/0x290
Feb 03 01:05:58 thetick kernel: BTRFS: Transaction aborted (error -17)
--->8--- snap
"error -17" being "object already exists". My only theory would be this
has a direct connection to you finding the single metadata profile.
Like in "the kernel thinks the objects already exists when it really
didn't, and as a result the object is there only once now" aka "single
metadata".
But I'm no dev and no expert on the internals.
--
Regards,
Kai
Replies to list-only preferred.
[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 163 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-11 2:01 ` Kai Krakow
@ 2017-02-11 10:18 ` Marc Joliet
2017-02-14 12:52 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-02-11 10:18 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 55579 bytes --]
On Saturday 11 February 2017 03:01:39 Kai Krakow wrote:
> Am Fri, 10 Feb 2017 23:15:03 +0100
>
> schrieb Marc Joliet <marcec@gmx.de>:
> > # btrfs filesystem df /media/MARCEC_BACKUP/
> > Data, single: total=851.00GiB, used=831.36GiB
> > System, DUP: total=64.00MiB, used=120.00KiB
> > Metadata, DUP: total=13.00GiB, used=10.38GiB
> > Metadata, single: total=1.12GiB, used=0.00B
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> >
> > Hmm, I take it that the single metadata is a leftover from running
> > --repair?
>
> It's more probably a remnant of an incomplete balance operation or an
> older mkfs version. I'd simply rebalance metadata to fix this.
>
> I don't think that btrfs-repair would migrate missing metadata
> duplicates back to single profile, it would more likely trigger
> recreating the missing duplicates. But I'm not sure.
I'm fairly certain it's from the repair operation, the device didn't have any
single metadata before (and it still didn't really, notice that used=0.00B).
So my best guess is that it was allocated during the --repair.
> If it is a result of the repair operation, that could be an
> interesting clue. Could it explain "error -17" from your logs? But that
> would mean the duplicates were already missing before the repair
> operation and triggered that problem. So the question is, why are those
> duplicates missing in the first place as a result of normal operation?
> From your logs:
I certainly didn't create them myself, the device has always had metadata=dup
(you can check previous threads of mine, which also contain "btrfs fi df"
output). So I don't think it has anything to do with the failures.
> ---8<--- snip
> Feb 02 22:49:14 thetick kernel: BTRFS: device label MARCEC_BACKUP devid
> 1 transid 283903 /dev/sdd2
> Feb 02 22:49:19 thetick kernel: EXT4-fs (sdd1): mounted filesystem with
> ordered data mode. Opts: (null)
> Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): use zlib
> compression Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2):
> disk space caching is enabled
> Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): has skinny
> extents Feb 03 00:20:09 thetick kernel: BTRFS info (device sdd2): The
> free space cache file (3967375376384) is invalid. skip it
> Feb 03 01:05:58 thetick kernel: ------------[ cut here ]------------
> Feb 03 01:05:58 thetick kernel: WARNING: CPU: 1 PID: 26544 at
> fs/btrfs/extent- tree.c:2967 btrfs_run_delayed_refs+0x26c/0x290
> Feb 03 01:05:58 thetick kernel: BTRFS: Transaction aborted (error -17)
> --->8--- snap
>
> "error -17" being "object already exists". My only theory would be this
> has a direct connection to you finding the single metadata profile.
> Like in "the kernel thinks the objects already exists when it really
> didn't, and as a result the object is there only once now" aka "single
> metadata".
>
> But I'm no dev and no expert on the internals.
Again, I don't think it has anything to do with the single metadata. A "btrfs
balance start -mprofile=single" got rid of the (empty) single metadata blocks.
What I *can* say is that the error seems to be transient. For example, I ran
an rsync last night that failed with the same error as before. After
unmounting and mounting the FS again, I could run it to completion (and
verified some files).
Anyway, here's the whole log since this morning. I can't correlate individual
stack traces to specific actions any more, but the rough order of actions on
my part (not counting unmount/mount cycles, those are easy enough to find):
- backups from the home server run, then
- btrfs-cleaner drops old snapshots, after which
- I run rsync (which fails), then
- I run rsync again (which succeeds), then
- I run "balance start -mprofile=single", which succeeds (I might have run
this before rsync, I'm not sure), after which
- I start backups on my laptop (which fails after I finally went to bed, at
about 3:58).
After that I just try "btrbk -r run" a few times, which managed to transfer
the first two snapshots before failing.
Feb 11 00:00:06 diefledermaus kernel: usb 1-1: new high-speed USB device
number 4 using ehci-pci
Feb 11 00:00:06 diefledermaus kernel: usb 1-1: New USB device found,
idVendor=0480, idProduct=d010
Feb 11 00:00:06 diefledermaus kernel: usb 1-1: New USB device strings: Mfr=1,
Product=2, SerialNumber=3
Feb 11 00:00:06 diefledermaus kernel: usb 1-1: Product: External USB 3.0
Feb 11 00:00:06 diefledermaus kernel: usb 1-1: Manufacturer: Toshiba
Feb 11 00:00:06 diefledermaus kernel: usb 1-1: SerialNumber: 20130421020612
Feb 11 00:00:07 diefledermaus kernel: usb-storage 1-1:1.0: USB Mass Storage
device detected
Feb 11 00:00:07 diefledermaus kernel: usb-storage 1-1:1.0: Quirks match for
vid 0480 pid d010: 20000000
Feb 11 00:00:07 diefledermaus kernel: scsi host6: usb-storage 1-1:1.0
Feb 11 00:00:07 diefledermaus kernel: usbcore: registered new interface driver
usb-storage
Feb 11 00:00:16 diefledermaus kernel: scsi 6:0:0:0: Direct-Access Toshiba
External USB 3.0 0201 PQ: 0 ANSI: 6
Feb 11 00:00:16 diefledermaus kernel: sd 6:0:0:0: Attached scsi generic sg2
type 0
Feb 11 00:00:16 diefledermaus kernel: sd 6:0:0:0: [sdb] Very big device.
Trying to use READ CAPACITY(16).
Feb 11 00:00:16 diefledermaus kernel: sd 6:0:0:0: [sdb] 732566645 4096-byte
logical blocks: (3.00 TB/2.73 TiB)
Feb 11 00:00:16 diefledermaus kernel: sd 6:0:0:0: [sdb] Assuming Write Enabled
Feb 11 00:00:16 diefledermaus kernel: sd 6:0:0:0: [sdb] Assuming drive cache:
write back
Feb 11 00:00:16 diefledermaus kernel: sdb: sdb1 sdb2
Feb 11 00:00:16 diefledermaus kernel: sd 6:0:0:0: [sdb] Attached SCSI disk
Feb 11 00:00:16 diefledermaus kernel: BTRFS: device label MARCEC_BACKUP devid
1 transid 284246 /dev/sdb2
Feb 11 00:00:16 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 00:00:16 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 00:00:16 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 00:00:17 diefledermaus kernel: EXT4-fs (sdb1): mounted filesystem with
ordered data mode. Opts: (null)
Feb 11 00:01:21 diefledermaus kernel: BTRFS warning (device sdb2): block group
3831010164736 has wrong amount of free space
Feb 11 00:01:21 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3831010164736, rebuilding it now
Feb 11 00:01:21 diefledermaus kernel: BTRFS warning (device sdb2): block group
3831547035648 has wrong amount of free space
Feb 11 00:01:21 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3831547035648, rebuilding it now
Feb 11 00:01:21 diefledermaus kernel: BTRFS warning (device sdb2): block group
3833694519296 has wrong amount of free space
Feb 11 00:01:21 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3833694519296, rebuilding it now
Feb 11 00:01:22 diefledermaus kernel: BTRFS warning (device sdb2): block group
4649738305536 has wrong amount of free space
Feb 11 00:01:22 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 4649738305536, rebuilding it now
Feb 11 00:01:27 diefledermaus kernel: BTRFS warning (device sdb2): block group
4607325503488 has wrong amount of free space
Feb 11 00:01:27 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 4607325503488, rebuilding it now
Feb 11 00:01:37 diefledermaus kernel: BTRFS warning (device sdb2): block group
3832083906560 has wrong amount of free space
Feb 11 00:01:37 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3832083906560, rebuilding it now
Feb 11 00:01:37 diefledermaus kernel: BTRFS warning (device sdb2): block group
3832620777472 has wrong amount of free space
Feb 11 00:01:37 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3832620777472, rebuilding it now
Feb 11 00:01:38 diefledermaus kernel: BTRFS warning (device sdb2): block group
3833157648384 has wrong amount of free space
Feb 11 00:01:38 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3833157648384, rebuilding it now
Feb 11 00:03:06 diefledermaus kernel: BTRFS warning (device sdb2): block group
3829936422912 has wrong amount of free space
Feb 11 00:03:06 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3829936422912, rebuilding it now
Feb 11 00:06:26 diefledermaus kernel: BTRFS warning (device sdb2): block group
4735906086912 has wrong amount of free space
Feb 11 00:06:26 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 4735906086912, rebuilding it now
Feb 11 00:06:53 diefledermaus kernel: BTRFS warning (device sdb2): block group
4736442957824 has wrong amount of free space
Feb 11 00:06:53 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 4736442957824, rebuilding it now
Feb 11 00:06:55 diefledermaus kernel: BTRFS warning (device sdb2): block group
3830473293824 has wrong amount of free space
Feb 11 00:06:55 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 3830473293824, rebuilding it now
Feb 11 00:07:11 diefledermaus kernel: BTRFS warning (device sdb2): block group
4745066446848 has wrong amount of free space
Feb 11 00:07:11 diefledermaus kernel: BTRFS warning (device sdb2): failed to
load free space cache for block group 4745066446848, rebuilding it now
Feb 11 00:47:39 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 00:47:39 diefledermaus kernel: WARNING: CPU: 1 PID: 8963 at
fs/btrfs/extent-tree.c:2967 btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 00:47:39 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 00:47:39 diefledermaus kernel: BTRFS info (device sdb2): forced
readonly
Feb 11 00:47:39 diefledermaus kernel: BTRFS: Transaction aborted (error -17)
Feb 11 00:47:39 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 00:47:39 diefledermaus kernel: CPU: 1 PID: 8963 Comm: kworker/u4:4 Not
tainted 4.9.7-gentoo #1
Feb 11 00:47:39 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 00:47:39 diefledermaus kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper
Feb 11 00:47:39 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
ffffc900051a7d60 0000000000000000
Feb 11 00:47:39 diefledermaus kernel: ffffffff8104908a ffff88007edc63c0
ffffc900051a7db0 0000000000001732
Feb 11 00:47:39 diefledermaus kernel: ffff88008f47a1f8 ffff8800beac3700
0000000000000000 ffffffff810490fa
Feb 11 00:47:39 diefledermaus kernel: Call Trace:
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff810490fa>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff81202188>] ?
btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff81202244>] ?
delayed_ref_async_start+0x84/0xa0
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff8105ce76>] ?
process_one_work+0x126/0x310
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff8105d0a5>] ?
worker_thread+0x45/0x450
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff8105d060>] ?
process_one_work+0x310/0x310
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff81061c99>] ?
kthread+0xb9/0xd0
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff81061be0>] ?
kthread_park+0x50/0x50
Feb 11 00:47:39 diefledermaus kernel: [<ffffffff814f6412>] ?
ret_from_fork+0x22/0x30
Feb 11 00:47:39 diefledermaus kernel: ---[ end trace ba59f5981cc5a675 ]---
Feb 11 00:47:39 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 01:07:43 diefledermaus kernel: BTRFS error (device sdb2): cleaner
transaction attach returned -30
Feb 11 01:07:43 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 01:07:43 diefledermaus kernel: WARNING: CPU: 0 PID: 9450 at
fs/btrfs/extent-tree.c:134 btrfs_put_block_group+0x55/0x60
Feb 11 01:07:43 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 01:07:43 diefledermaus kernel: CPU: 0 PID: 9450 Comm: umount Tainted: G
W 4.9.7-gentoo #1
Feb 11 01:07:43 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 01:07:43 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
0000000000000000 0000000000000000
Feb 11 01:07:43 diefledermaus kernel: ffffffff8104908a ffff88004203b200
ffff88004203b200 ffff8800174160a0
Feb 11 01:07:43 diefledermaus kernel: ffff880017416000 ffff880017416090
ffff8800371193f8 ffffffff811f9e05
Feb 11 01:07:43 diefledermaus kernel: Call Trace:
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff811f9e05>] ?
btrfs_put_block_group+0x55/0x60
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff81201295>] ?
btrfs_free_block_groups+0xf5/0x420
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8121231b>] ?
close_ctree+0xfb/0x320
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8111105f>] ?
generic_shutdown_super+0x5f/0xe0
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff811112e9>] ?
kill_anon_super+0x9/0x20
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff811e70ed>] ?
btrfs_kill_super+0xd/0x90
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff811114bf>] ?
deactivate_locked_super+0x2f/0x60
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8112b006>] ?
cleanup_mnt+0x36/0x80
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff810605cd>] ?
task_work_run+0x6d/0x90
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff81000fe6>] ?
exit_to_usermode_loop+0x66/0x70
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff810012ee>] ?
syscall_return_slowpath+0x4e/0x50
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff814f625f>] ?
entry_SYSCALL_64_fastpath+0x92/0x94
Feb 11 01:07:43 diefledermaus kernel: ---[ end trace ba59f5981cc5a676 ]---
Feb 11 01:07:43 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 01:07:43 diefledermaus kernel: WARNING: CPU: 0 PID: 9450 at
fs/btrfs/extent-tree.c:10045 btrfs_free_block_groups+0x28d/0x420
Feb 11 01:07:43 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 01:07:43 diefledermaus kernel: CPU: 0 PID: 9450 Comm: umount Tainted: G
W 4.9.7-gentoo #1
Feb 11 01:07:43 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 01:07:43 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
0000000000000000 0000000000000000
Feb 11 01:07:43 diefledermaus kernel: ffffffff8104908a ffff8800bbcc6800
0000000000000000 ffff8800bbcc6888
Feb 11 01:07:43 diefledermaus kernel: ffff880017416000 ffff8800bb6569f8
ffff880017416a70 ffffffff8120142d
Feb 11 01:07:43 diefledermaus kernel: Call Trace:
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8120142d>] ?
btrfs_free_block_groups+0x28d/0x420
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8121231b>] ?
close_ctree+0xfb/0x320
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8111105f>] ?
generic_shutdown_super+0x5f/0xe0
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff811112e9>] ?
kill_anon_super+0x9/0x20
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff811e70ed>] ?
btrfs_kill_super+0xd/0x90
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff811114bf>] ?
deactivate_locked_super+0x2f/0x60
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff8112b006>] ?
cleanup_mnt+0x36/0x80
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff810605cd>] ?
task_work_run+0x6d/0x90
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff81000fe6>] ?
exit_to_usermode_loop+0x66/0x70
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff810012ee>] ?
syscall_return_slowpath+0x4e/0x50
Feb 11 01:07:43 diefledermaus kernel: [<ffffffff814f625f>] ?
entry_SYSCALL_64_fastpath+0x92/0x94
Feb 11 01:07:43 diefledermaus kernel: ---[ end trace ba59f5981cc5a677 ]---
Feb 11 01:07:43 diefledermaus kernel: BTRFS info (device sdb2): space_info 4
has 2744418304 free, is not full
Feb 11 01:07:43 diefledermaus kernel: BTRFS info (device sdb2): space_info
total=15166603264, used=11214192640, pinned=0, reserved=32768, may_use=0,
readonly=1207959552
Feb 11 01:07:57 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 01:07:57 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 01:07:57 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 02:02:58 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757540306944 flags 4
Feb 11 02:03:06 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757531918336 flags 4
Feb 11 02:03:07 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757523529728 flags 4
Feb 11 02:03:08 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757515141120 flags 4
Feb 11 02:03:08 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757506752512 flags 4
Feb 11 02:03:09 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757498363904 flags 4
Feb 11 02:03:10 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757489975296 flags 4
Feb 11 02:03:11 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757481586688 flags 4
Feb 11 02:03:12 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757473198080 flags 4
Feb 11 02:03:13 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757464809472 flags 4
Feb 11 02:03:14 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757456420864 flags 4
Feb 11 02:03:15 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757448032256 flags 4
Feb 11 02:03:16 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757439643648 flags 4
Feb 11 02:03:18 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757431255040 flags 4
Feb 11 02:03:19 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757422866432 flags 4
Feb 11 02:03:23 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757414477824 flags 4
Feb 11 02:03:25 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757406089216 flags 4
Feb 11 02:03:28 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757397700608 flags 4
Feb 11 02:03:30 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757389312000 flags 4
Feb 11 02:03:33 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757380923392 flags 4
Feb 11 02:03:35 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757372534784 flags 4
Feb 11 02:03:38 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757364146176 flags 4
Feb 11 02:03:40 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757355757568 flags 4
Feb 11 02:03:43 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757347368960 flags 4
Feb 11 02:03:45 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757338980352 flags 4
Feb 11 02:03:47 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757330591744 flags 4
Feb 11 02:03:49 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757322203136 flags 4
Feb 11 02:03:51 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757313814528 flags 4
Feb 11 02:03:53 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757305425920 flags 4
Feb 11 02:03:55 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757297037312 flags 4
Feb 11 02:03:57 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757288648704 flags 4
Feb 11 02:03:59 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757280260096 flags 4
Feb 11 02:04:01 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757271871488 flags 4
Feb 11 02:04:03 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757263482880 flags 4
Feb 11 02:04:06 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757255094272 flags 4
Feb 11 02:04:07 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757246705664 flags 4
Feb 11 02:04:09 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757238317056 flags 4
Feb 11 02:04:11 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757229928448 flags 4
Feb 11 02:04:12 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757221539840 flags 4
Feb 11 02:04:13 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757213151232 flags 4
Feb 11 02:04:15 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757204762624 flags 4
Feb 11 02:04:16 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757196374016 flags 4
Feb 11 02:04:17 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757187985408 flags 4
Feb 11 02:04:19 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757179596800 flags 4
Feb 11 02:04:20 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757171208192 flags 4
Feb 11 02:04:21 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757162819584 flags 4
Feb 11 02:04:22 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757154430976 flags 4
Feb 11 02:04:24 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757146042368 flags 4
Feb 11 02:04:25 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757137653760 flags 4
Feb 11 02:04:26 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757129265152 flags 4
Feb 11 02:04:27 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757120876544 flags 4
Feb 11 02:04:28 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757112487936 flags 4
Feb 11 02:04:29 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757104099328 flags 4
Feb 11 02:04:30 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757095710720 flags 4
Feb 11 02:04:31 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757087322112 flags 4
Feb 11 02:04:37 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757078933504 flags 4
Feb 11 02:04:38 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757070544896 flags 4
Feb 11 02:04:39 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757062156288 flags 4
Feb 11 02:04:40 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757053767680 flags 4
Feb 11 02:04:41 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757045379072 flags 4
Feb 11 02:04:42 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757036990464 flags 4
Feb 11 02:04:43 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757028601856 flags 4
Feb 11 02:04:44 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757020213248 flags 4
Feb 11 02:04:45 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757011824640 flags 4
Feb 11 02:04:47 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4757003436032 flags 4
Feb 11 02:04:47 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756995047424 flags 4
Feb 11 02:04:48 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756986658816 flags 4
Feb 11 02:04:49 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756978270208 flags 4
Feb 11 02:04:50 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756969881600 flags 4
Feb 11 02:04:50 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756961492992 flags 4
Feb 11 02:04:51 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756953104384 flags 4
Feb 11 02:04:52 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756944715776 flags 4
Feb 11 02:04:53 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756936327168 flags 4
Feb 11 02:04:53 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756927938560 flags 4
Feb 11 02:04:54 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756919549952 flags 4
Feb 11 02:04:55 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756911161344 flags 4
Feb 11 02:04:57 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756902772736 flags 4
Feb 11 02:04:58 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756894384128 flags 4
Feb 11 02:04:58 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756885995520 flags 4
Feb 11 02:05:00 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756877606912 flags 4
Feb 11 02:05:01 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756869218304 flags 4
Feb 11 02:05:03 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756860829696 flags 4
Feb 11 02:05:04 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756852441088 flags 4
Feb 11 02:05:04 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756844052480 flags 4
Feb 11 02:05:05 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756835663872 flags 4
Feb 11 02:05:06 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756827275264 flags 4
Feb 11 02:05:06 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756818886656 flags 4
Feb 11 02:05:07 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756810498048 flags 4
Feb 11 02:05:08 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756802109440 flags 4
Feb 11 02:05:09 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756793720832 flags 4
Feb 11 02:05:09 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756785332224 flags 4
Feb 11 02:05:10 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756776943616 flags 4
Feb 11 02:05:11 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756768555008 flags 4
Feb 11 02:05:13 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756760166400 flags 4
Feb 11 02:05:13 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756751777792 flags 4
Feb 11 02:05:14 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756743389184 flags 4
Feb 11 02:05:15 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756735000576 flags 4
Feb 11 02:05:15 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756726611968 flags 4
Feb 11 02:05:16 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756718223360 flags 4
Feb 11 02:05:17 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756709834752 flags 4
Feb 11 02:05:20 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756701446144 flags 4
Feb 11 02:05:20 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756693057536 flags 4
Feb 11 02:05:22 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756684668928 flags 4
Feb 11 02:05:23 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756676280320 flags 4
Feb 11 02:05:24 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756667891712 flags 4
Feb 11 02:05:26 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756659503104 flags 4
Feb 11 02:05:27 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756651114496 flags 4
Feb 11 02:05:45 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756642725888 flags 4
Feb 11 02:05:46 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756634337280 flags 4
Feb 11 02:05:48 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756625948672 flags 4
Feb 11 02:05:49 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756617560064 flags 4
Feb 11 02:05:51 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756609171456 flags 4
Feb 11 02:05:52 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756600782848 flags 4
Feb 11 02:05:54 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756592394240 flags 4
Feb 11 02:05:56 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756584005632 flags 4
Feb 11 02:05:58 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756575617024 flags 4
Feb 11 02:05:59 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756567228416 flags 4
Feb 11 02:06:01 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756558839808 flags 4
Feb 11 02:06:02 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756550451200 flags 4
Feb 11 02:06:04 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4756542062592 flags 4
Feb 11 02:06:05 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751164964864 flags 4
Feb 11 02:06:07 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751156576256 flags 4
Feb 11 02:06:09 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751148187648 flags 4
Feb 11 02:06:10 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751139799040 flags 4
Feb 11 02:06:12 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751131410432 flags 4
Feb 11 02:06:13 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751123021824 flags 4
Feb 11 02:06:15 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751114633216 flags 4
Feb 11 02:06:17 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751106244608 flags 4
Feb 11 02:06:18 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751097856000 flags 4
Feb 11 02:06:20 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751089467392 flags 4
Feb 11 02:06:21 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751081078784 flags 4
Feb 11 02:06:22 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751072690176 flags 4
Feb 11 02:06:24 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751064301568 flags 4
Feb 11 02:06:25 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751055912960 flags 4
Feb 11 02:06:26 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751047524352 flags 4
Feb 11 02:06:27 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751039135744 flags 4
Feb 11 02:06:28 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751030747136 flags 4
Feb 11 02:06:29 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751022358528 flags 4
Feb 11 02:06:30 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751013969920 flags 4
Feb 11 02:06:32 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4751005581312 flags 4
Feb 11 02:06:33 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4750997192704 flags 4
Feb 11 02:06:34 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4750988804096 flags 4
Feb 11 02:06:35 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4750980415488 flags 4
Feb 11 02:06:36 diefledermaus kernel: BTRFS info (device sdb2): relocating
block group 4750972026880 flags 4
Feb 11 02:27:34 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 02:27:35 diefledermaus kernel: WARNING: CPU: 0 PID: 10268 at
fs/btrfs/extent-tree.c:2967 btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 02:27:35 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 02:27:35 diefledermaus kernel: BTRFS info (device sdb2): forced
readonly
Feb 11 02:27:35 diefledermaus kernel: BTRFS: Transaction aborted (error -17)
Feb 11 02:27:35 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 02:27:35 diefledermaus kernel: CPU: 0 PID: 10268 Comm: kworker/u4:4
Tainted: G W 4.9.7-gentoo #1
Feb 11 02:27:35 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 02:27:35 diefledermaus kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper
Feb 11 02:27:35 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
ffffc90005fe3d60 0000000000000000
Feb 11 02:27:35 diefledermaus kernel: ffffffff8104908a ffff880052a738c0
ffffc90005fe3db0 00000000000001c2
Feb 11 02:27:35 diefledermaus kernel: ffff880019a0ea50 ffff8800bc583c00
0000000000000000 ffffffff810490fa
Feb 11 02:27:35 diefledermaus kernel: Call Trace:
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff810490fa>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff81202188>] ?
btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff81202244>] ?
delayed_ref_async_start+0x84/0xa0
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff8105ce76>] ?
process_one_work+0x126/0x310
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff8105d0a5>] ?
worker_thread+0x45/0x450
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff8105d060>] ?
process_one_work+0x310/0x310
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff8105d060>] ?
process_one_work+0x310/0x310
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff81061c99>] ?
kthread+0xb9/0xd0
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff81061be0>] ?
kthread_park+0x50/0x50
Feb 11 02:27:35 diefledermaus kernel: [<ffffffff814f6412>] ?
ret_from_fork+0x22/0x30
Feb 11 02:27:35 diefledermaus kernel: ---[ end trace ba59f5981cc5a678 ]---
Feb 11 02:27:35 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 02:29:09 diefledermaus kernel: BTRFS error (device sdb2): cleaner
transaction attach returned -30
Feb 11 02:29:11 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 02:29:11 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 02:29:11 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 02:54:20 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 02:54:20 diefledermaus kernel: WARNING: CPU: 0 PID: 10382 at
fs/btrfs/extent-tree.c:2967 btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 02:54:20 diefledermaus kernel: BTRFS: Transaction aborted (error -17)
Feb 11 02:54:20 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 02:54:20 diefledermaus kernel: CPU: 0 PID: 10382 Comm: btrfs-cleaner
Tainted: G W 4.9.7-gentoo #1
Feb 11 02:54:20 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 02:54:20 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
ffffc9000601bd28 0000000000000000
Feb 11 02:54:20 diefledermaus kernel: ffffffff8104908a ffff8800037f3780
ffffc9000601bd78 00000000000003a0
Feb 11 02:54:20 diefledermaus kernel: ffff88005b83d3b0 ffff8800225f19df
ffff8800037f3780 ffffffff810490fa
Feb 11 02:54:20 diefledermaus kernel: Call Trace:
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff810490fa>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff81202188>] ?
btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff81215dda>] ?
btrfs_should_end_transaction+0x3a/0x60
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff81200787>] ?
btrfs_drop_snapshot+0x417/0x7b0
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff81217012>] ?
btrfs_clean_one_deleted_snapshot+0xb2/0xe0
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff8120e04f>] ?
cleaner_kthread+0x13f/0x1b0
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff8120df10>] ?
btrfs_destroy_pinned_extent+0xc0/0xc0
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff81061c99>] ?
kthread+0xb9/0xd0
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff8104b678>] ?
do_exit+0x698/0xa70
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff81061be0>] ?
kthread_park+0x50/0x50
Feb 11 02:54:20 diefledermaus kernel: [<ffffffff814f6412>] ?
ret_from_fork+0x22/0x30
Feb 11 02:54:20 diefledermaus kernel: ---[ end trace ba59f5981cc5a679 ]---
Feb 11 02:54:20 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 02:54:20 diefledermaus kernel: BTRFS info (device sdb2): forced
readonly
Feb 11 02:55:18 diefledermaus kernel: BTRFS error (device sdb2): cleaner
transaction attach returned -30
Feb 11 02:55:21 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 02:55:21 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 02:55:21 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 03:04:21 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 03:04:21 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 03:04:21 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 03:30:30 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 03:30:30 diefledermaus kernel: WARNING: CPU: 0 PID: 10620 at
fs/btrfs/extent-tree.c:2967 btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 03:30:30 diefledermaus kernel: BTRFS: Transaction aborted (error -17)
Feb 11 03:30:30 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 03:30:30 diefledermaus kernel: CPU: 0 PID: 10620 Comm: kworker/u4:4
Tainted: G W 4.9.7-gentoo #1
Feb 11 03:30:30 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 03:30:30 diefledermaus kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper
Feb 11 03:30:30 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
ffffc9000645bd60 0000000000000000
Feb 11 03:30:31 diefledermaus kernel: ffffffff8104908a ffff8800b7bd5a00
ffffc9000645bdb0 0000000000000020
Feb 11 03:30:31 diefledermaus kernel: ffff880003394898 ffff8800bc583500
0000000000000000 ffffffff810490fa
Feb 11 03:30:31 diefledermaus kernel: Call Trace:
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff810490fa>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff81202188>] ?
btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff81202244>] ?
delayed_ref_async_start+0x84/0xa0
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff8105ce76>] ?
process_one_work+0x126/0x310
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff8105a893>] ?
pwq_activate_delayed_work+0x33/0x60
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff8105d0a5>] ?
worker_thread+0x45/0x450
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff8105d060>] ?
process_one_work+0x310/0x310
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff81061c99>] ?
kthread+0xb9/0xd0
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff81061be0>] ?
kthread_park+0x50/0x50
Feb 11 03:30:31 diefledermaus kernel: [<ffffffff814f6412>] ?
ret_from_fork+0x22/0x30
Feb 11 03:30:31 diefledermaus kernel: ---[ end trace ba59f5981cc5a67a ]---
Feb 11 03:30:31 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 03:30:31 diefledermaus kernel: BTRFS info (device sdb2): forced
readonly
Feb 11 03:31:39 diefledermaus kernel: BTRFS error (device sdb2): cleaner
transaction attach returned -30
Feb 11 03:31:41 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 03:31:41 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 03:31:41 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 03:58:38 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 03:58:38 diefledermaus kernel: WARNING: CPU: 0 PID: 11174 at
fs/btrfs/extent-tree.c:2967 btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 03:58:38 diefledermaus kernel: BTRFS: Transaction aborted (error -17)
Feb 11 03:58:38 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 03:58:38 diefledermaus kernel: CPU: 0 PID: 11174 Comm: btrfs Tainted: G
W 4.9.7-gentoo #1
Feb 11 03:58:38 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 03:58:38 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
ffffc90006bfbb88 0000000000000000
Feb 11 03:58:38 diefledermaus kernel: ffffffff8104908a ffff880047adb1e0
ffffc90006bfbbd8 0000000000000000
Feb 11 03:58:38 diefledermaus kernel: ffff88000a6f8898 0000000000000a34
ffff880023d59240 ffffffff810490fa
Feb 11 03:58:38 diefledermaus kernel: Call Trace:
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff810490fa>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff81202188>] ?
btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff812170eb>] ?
btrfs_commit_transaction.part.25+0x5b/0x970
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8124c04d>] ?
btrfs_mksubvol+0x45d/0x470
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff810796b0>] ?
wake_up_bit+0x20/0x20
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8124c1f5>] ?
btrfs_ioctl_snap_create_transid+0x195/0x1a0
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8110bfcd>] ?
__check_object_size+0x11d/0x1d2
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8124c38a>] ?
btrfs_ioctl_snap_create_v2+0xfa/0x150
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8124f386>] ?
btrfs_ioctl+0x656/0x22b0
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8111d7ad>] ?
do_filp_open+0x8d/0xd0
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff810ecf92>] ?
handle_mm_fault+0xd42/0xdb0
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8110bfcd>] ?
__check_object_size+0x11d/0x1d2
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8111f756>] ?
do_vfs_ioctl+0x86/0x5c0
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff8111fcc6>] ?
SyS_ioctl+0x36/0x70
Feb 11 03:58:38 diefledermaus kernel: [<ffffffff814f61e0>] ?
entry_SYSCALL_64_fastpath+0x13/0x94
Feb 11 03:58:38 diefledermaus kernel: ---[ end trace ba59f5981cc5a67b ]---
Feb 11 03:58:38 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 03:58:39 diefledermaus kernel: BTRFS info (device sdb2): forced
readonly
Feb 11 09:47:57 diefledermaus kernel: BTRFS error (device sdb2): cleaner
transaction attach returned -30
Feb 11 09:48:10 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 09:48:10 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 09:48:10 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 10:06:46 diefledermaus kernel: ------------[ cut here ]------------
Feb 11 10:06:46 diefledermaus kernel: WARNING: CPU: 1 PID: 12677 at
fs/btrfs/extent-tree.c:2967 btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 10:06:46 diefledermaus kernel: BTRFS: Transaction aborted (error -17)
Feb 11 10:06:46 diefledermaus kernel: Modules linked in: usb_storage msr ctr
ccm tun arc4 snd_hda_codec_idt snd_hda_codec_generic ath5k applesmc
snd_hda_intel input_polldev snd_hda_codec mac80211 hwmon ath snd_hda_core
snd_pcm cfg80211 sky2 rfkill snd_timer acpi_cpufreq video snd backlight
soundcore processor button sg sr_mod cdrom ata_generic pata_acpi uhci_hcd ahci
libahci ata_piix libata ehci_pci ehci_hcd
Feb 11 10:06:46 diefledermaus kernel: CPU: 1 PID: 12677 Comm: kworker/u4:6
Tainted: G W 4.9.7-gentoo #1
Feb 11 10:06:46 diefledermaus kernel: Hardware name: Apple Inc.
Macmini2,1/Mac-F4208EAA, BIOS MM21.88Z.009A.B00.0706281359 06/28/07
Feb 11 10:06:46 diefledermaus kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper
Feb 11 10:06:46 diefledermaus kernel: 0000000000000000 ffffffff812cf8f9
ffffc90001c73d60 0000000000000000
Feb 11 10:06:46 diefledermaus kernel: ffffffff8104908a ffff8800258696e0
ffffc90001c73db0 0000000000000020
Feb 11 10:06:46 diefledermaus kernel: ffff8800214131f8 ffff880087de7f00
0000000000000000 ffffffff810490fa
Feb 11 10:06:46 diefledermaus kernel: Call Trace:
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff812cf8f9>] ?
dump_stack+0x46/0x5d
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff8104908a>] ? __warn+0xba/0xe0
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff810490fa>] ?
warn_slowpath_fmt+0x4a/0x50
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff81202188>] ?
btrfs_run_delayed_refs+0x278/0x2b0
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff81202244>] ?
delayed_ref_async_start+0x84/0xa0
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff8105ce76>] ?
process_one_work+0x126/0x310
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff8105d0a5>] ?
worker_thread+0x45/0x450
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff8105d060>] ?
process_one_work+0x310/0x310
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff81061c99>] ?
kthread+0xb9/0xd0
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff81061be0>] ?
kthread_park+0x50/0x50
Feb 11 10:06:46 diefledermaus kernel: [<ffffffff814f6412>] ?
ret_from_fork+0x22/0x30
Feb 11 10:06:46 diefledermaus kernel: ---[ end trace ba59f5981cc5a67c ]---
Feb 11 10:06:46 diefledermaus kernel: BTRFS: error (device sdb2) in
btrfs_run_delayed_refs:2967: errno=-17 Object already exists
Feb 11 10:06:46 diefledermaus kernel: BTRFS info (device sdb2): forced
readonly
Feb 11 10:18:06 diefledermaus kernel: BTRFS error (device sdb2): cleaner
transaction attach returned -30
Feb 11 10:18:19 diefledermaus kernel: BTRFS info (device sdb2): use zlib
compression
Feb 11 10:18:19 diefledermaus kernel: BTRFS info (device sdb2): disk space
caching is enabled
Feb 11 10:18:19 diefledermaus kernel: BTRFS info (device sdb2): has skinny
extents
Feb 11 10:24:09 diefledermaus kernel: BTRFS info (device sdb2): The free space
cache file (3833157648384) is invalid. skip it
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-11 10:18 ` Marc Joliet
@ 2017-02-14 12:52 ` Marc Joliet
2017-02-17 8:19 ` Kai Krakow
0 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-02-14 12:52 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 778 bytes --]
Hi again,
so, it seems that I've solved the problem: After having to umount/mount the
FS several times to get btrfs-cleaner to finish, I thought of the "failed to
load free space [...], rebuilding it now" type errors and decided to try the
clear_cache mount option. Since then, my home server has been running its
backups regularly again. Furthermore, I was able back up my desktop again via
send/recv (the rsync based backup is still running, but I expect it to
succeed). The kernel log has also stayed clean.
Kai, I'd be curious whether clear_cache will help in your case, too, if you
haven't tried it already.
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-14 12:52 ` Marc Joliet
@ 2017-02-17 8:19 ` Kai Krakow
2017-02-28 22:14 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Kai Krakow @ 2017-02-17 8:19 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]
Am Tue, 14 Feb 2017 13:52:48 +0100
schrieb Marc Joliet <marcec@gmx.de>:
> Hi again,
>
> so, it seems that I've solved the problem: After having to
> umount/mount the FS several times to get btrfs-cleaner to finish, I
> thought of the "failed to load free space [...], rebuilding it now"
> type errors and decided to try the clear_cache mount option. Since
> then, my home server has been running its backups regularly again.
> Furthermore, I was able back up my desktop again via send/recv (the
> rsync based backup is still running, but I expect it to succeed).
> The kernel log has also stayed clean.
>
> Kai, I'd be curious whether clear_cache will help in your case, too,
> if you haven't tried it already.
The FS in question has been rebuilt from scratch. AFAIR it wasn't even
mountable back then, or at it froze the system short after mounting.
I needed the system back in usable state, so I had to recreate and
restore the system.
Next time, if it happens again (finger crossed it does not), I'll give
it a try.
--
Regards,
Kai
Replies to list-only preferred.
[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 163 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-17 8:19 ` Kai Krakow
@ 2017-02-28 22:14 ` Marc Joliet
2017-03-01 8:23 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-02-28 22:14 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 32389 bytes --]
=2D-nextPart2283738.6rFPRxsZn9
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="utf-8"
(Resend with the log compressed this time.)
On Friday 17 February 2017 09:19:15 Kai Krakow wrote:
> Am Tue, 14 Feb 2017 13:52:48 +0100
>
> schrieb Marc Joliet <marcec@gmx.de>:
> > Hi again,
> >
> > so, it seems that I've solved the problem: After having to
> > umount/mount the FS several times to get btrfs-cleaner to finish, I
> > thought of the "failed to load free space [...], rebuilding it now"
> > type errors and decided to try the clear_cache mount option. Since
> > then, my home server has been running its backups regularly again.
> > Furthermore, I was able back up my desktop again via send/recv (the
> > rsync based backup is still running, but I expect it to succeed).
> > The kernel log has also stayed clean.
*sigh*
It was working fine... until the 26th (so almost 2 weeks without incident).
Now the error keeps occurring, and the clear_cache mount option stopped
helping. Just out of curiosity, I used "clear_cache,space_cache=v2", which
also didn't help. I've attached the full log, in case it's in any way useful.
This is all with:
% uname -a
Linux diefledermaus 4.9.12-gentoo #1 SMP Fri Feb 24 20:09:28 CET 2017 x86_64
Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz GenuineIntel GNU/Linux
% /sbin/btrfs --version
btrfs-progs v4.9
> > Kai, I'd be curious whether clear_cache will help in your case, too,
> > if you haven't tried it already.
>
> The FS in question has been rebuilt from scratch. AFAIR it wasn't even
> mountable back then, or at it froze the system short after mounting.
>
> I needed the system back in usable state, so I had to recreate and
> restore the system.
I think I'm at that point now myself, unless anybody has any other ideas.
What I *am* going to do before giving up and reformatting is create an image.
I found out that using the -w option to btrfs-image keeps it from segfaulting,
so I guess the underlying problem has to do with the extent tree? However,
the image is... pretty damn big (8.6 GB by now, *after* compressing with "xz
=2D9"), and it's not even finished yet. So, under the assumption that the image
could be useful to somebody, I would welcome any suggestions for where to
upload such an image once it's done.
> Next time, if it happens again (finger crossed it does not), I'll give
> it a try.
Yeah, I hope it doesn't come back, either, for either of us.
Greetings
=2D-
Marc Joliet
=2D-
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
=2D-nextPart2283738.6rFPRxsZn9
Content-Disposition: attachment; filename="dmesg_since_20170226.log.zip"
Content-Transfer-Encoding: base64
Content-Type: application/zip; name="dmesg_since_20170226.log.zip"
UEsDBBQAAAAIAMW5XErwNMkOxlQAAAC9BAAYAAAAZG1lc2dfc2luY2VfMjAxNzAyMjYubG9n7Z1Z
kxtHkqDf91ekzT6MZCuScR+w1axRR8/SpkX1iOyemW1rg+FIsGCsAmoAlNjsX78RmahiHpGRcSUS
BaIkUazL3dMzItw9ji/+kM8zxDIAJhhOCM+W63x1my/z3d3sYZ99zHeb/HaSvXn7h18n2WG2/5jN
D7vVfoIoxTKb324XH/NlttrusrvtLs8ON7NNBhHI9vliu1nuX/6PPzjJLz/ebg9Kx3pzUCLJS/kS
ohcf8s1hu83+J3QU9E/54mabgexfslf3u+3i1f7z/lX5vVc3D5sPU/0I08P6Lt8+HKbKyP0/KUH7
2fw23yvj1/vsLt/vZx9yV8MLb2RfPn7Sf4CsdE/xPyXi7+D44eoN0PjIVupDCABmhCBKKl+AKyJX
tPKF+WIBKRLOqh5/D3OqPkgpaCGVUsg4XiJRfKH4KUhWKGcg46vaR+6rqmUzgYAt0bwledVyhKOq
H2e3t9n73WyRT1yN++v/bj7nv/wt+z/ZdLpf3OTLh9v8f4G/QwFegb8rJ4UJxYgsC6EVkWipJLq/
rapAhFHOaSGwaIXT/WG2O0y3O/Vb+XKa//2g+o5SQWiwCsAlk6UjPs0+5tOH++l8rWUi7QgUaDZG
bF4x+9Ns/cXq3WzzQftlPlcKIAzTgJCY8YqGw+5hs5gdtFzMteGzQLnqDS6qDs8Ps8Nhp/2BFkou
DvMIRIKTvJC72R7Wq8/Txc3RDYjkWm4eJhcscsQKuctt1QmUKaEyVOZKyELm73XPQiT1K2OBYpcA
gkdT1aBdMxcEm0tWjBx7suoMu8/Td//17sfXf/zjlJHparY/3M8ON9p2rDWQqgY8oWDIYGiR7xcM
LYIGCYYWfeHB0OaNxMHQpipxMHRRlSgYWlR1BEObccHB0F2oYzB0FRgRDF1V+ARDd7NDg6GzBs9g
6C7XLxi6yvUNhs5yPYKhh0yfYOgu1i8Yune60GBIJxQPGQwt8v2CoUXQIMHQoi88GNq8kTgY2lQl
DoYuqhIFQ4uqjmBoMy44GLoLdQyGrgIjgqGrCp9g6G52aDB01uAZDN3l+gVDV7m+wdBZrkcw9JDp
EwzdxfoFQ/dOFxoM+YSyIYOhRb5fMLQIGiQYWvSFB0ObNxIHQ5uqxMHQRVWiYGhR1REMbcYFB0N3
oY7B0FVgRDB0VeETDN3NDg2Gzho8g6G7XL9g6CrXNxg6y/UIhh4yfYKhu1i/YOje6UKDoZxQOWQw
tMj3C4YWQYMEQ4u+8GBo80biYGhTlTgYuqhKFAwtqjqCoc244GDoLtQxGLoKjAiGrip8gqG72aHB
0FmDZzB0l+sXDF3l+gZDZ7kewdBDpk8wdBfrFwzdO11gMCRoAtCAwdAm3ysY2gQNEQxt+oKDodUb
aYOhVVXaYOikKk0wtKkyB0OrcaHB0EOoWzB0FhgeDJ1VeARDD7MDg6G7Br9g6CHXKxg6y/UMhu5y
3YOhj0yPYOgh1isYenS60GBIJoAMGQwt8v2CoUXQIMHQoi88GNq8kTgY2lQlDoYuqhIFQ4uqjmCo
foMmD4buQh2DoavAiGDoqsInGLqbHRoMnTV4BkN3uX7B0FWubzB0lusRDD1k+gRDd7F+wdC904UE
Q4gnSNWeXaPLi8rHX7PFwyG7UW03+1v16w1RXcb+x+vf3r55+69qxPrTnycqbvzpzU+TjAqWzQ7Z
av+qaGavyp784rDL85eLCZMUqWGpbIEr9bVjT3+53u9mLzlVj8OkHlWWX3pPjxW/bPVotM9u1xsd
zdebSXa332WLg/pvcZcdHjbZbLcg2X6znN4sZ9PFdpkvpuvlofEVFazz3Xrx9FUdyW/Vo9zQj/Wf
zGb390rh3ULpuleB+H57e7vMf8/uZgsBEISVn1aOVQKym093203x5Xtl0WL14cvP6Wi+y35fL/Nt
Nlvcr6eL+wfll//O5rPFx9v1h5vCzEynAyq2q0xlt/q4VvFh//EzyuYPh4MWvH3YLAtl+w/Zw36u
xtLtTqUB2X43vdsus8Vyt71ThsyenvFef6LVZbObxbr41v16/Xflw3nxhQf1x/RmsSy+cJhluf78
Xn0jP37D8d00G8aP27u7Y9rxYnGbz5Q1TnlTj5r/O9stP83U829md/kke63fT/Zms3iZ/TJb3K03
a/QdfKX++uIPBAHx8+vX32U/vPn1XZHz/PILgi+F+H8vAZCvX/4AwEvAAUMCYiozwF4h8QpwRzvM
qc9x/F2spMqqWj/RlST0qarEOSIBmz0Jekxsyi9gPl+pdAVW8zAEEIGSeakqsrOlGpJNNj/9hFws
lNtm1eeGq5mEzu2lnfr0GVeLcYWPy4H44e5e5xSLjzqJ0CGDuhpRTyIK5x6zKdXGNjq8z5S43Nl9
tVBReOMoLuUo2NCSc1bXogLSdJnfzj7rNCVf7XVkWmkVKnsN0oFUJrwClXzCoEEWidY8TD6kOZpV
85Wb7cOtygz1oKkSrf1scVhv9evA+nWEPgTgy2qypUZK1Ww2s3ulrUg/oU65eOgj6JGkIr0Y8qbb
Ta4dlasBr6pqjoJbFQI5kKtC0XFUnX483Ozy2bJID1b6NQc+AliuWPUtL/P9Ybf9rMLFZlPN0xf6
VS+CdAAGF7zstV+snut8bBkmj8wZFY/pWP73IudnUo3jf5/xUAvnC1C1cHo/2318zO5okFCV3TFZ
to5dflDDwPZuutoWUpFuCthVapHIqW6RHfTAmc2XsxyqCpBCuVIFtShSO0dRP7z/7Q/vVFqz2mbf
qJRmrcTtl3P07SRTzWqVYYGBkIJKzCXKDtvD7Da7P+z2GVGFthrGsv29tgAKgV09cvxYH/I7lSh8
zD9n3ygtEEiJJNdaIFNB89viB7aqpsaSw+KT/fofeYaaWW+fnsePst1mepzKIMhUwpEhChCj2ep2
9mGvPhEhTwCNT8BZhpEEuOspnJtP40Pn1OUsVpEtqocJsRlZbGbwPG3GnTZjld6cp82k22Yo+Xna
TLttRoCcp83MZjM7T5u5xWZ6pu1ZdNuMz7UPym6b2bmOz9AcFgVS4RhxRCEUmMHkDXt/owrqZQLz
6zERA5UdiHZUx+KLydg3e3j8qEX1IqhDwhgnj0E92XvMdtvtIXOeFqj5ox5v1R9AgLY/AP1a/IEb
/iC6Pzb9ITgaxB9ICUZ4IH+owC5cp7DqPiENn+iO3vYJll9LG6nnIRgxwkjbH4AN5Q89d3lO/qjn
OCreUtYeQzjHX0v74A1/qDfWHkM4AV+LP+q5GSZIkHbM5YAPM6YSQlSGElpIP36YEpDsXn1NadMT
EZRKKSBlYYW6bHiISdIeUfRrHaTFqCCnLBjaQxgygjkFvpMk5bRAPe3ElELc7lOMwGHakIQMyYH6
lK+80h+NPJap0qzdpxis9Kk0s1M0RZMpHwEZH6GY6lEZRtdjjDw9hbuNVtXQmRpNOo3GkJMzNZra
jKZnajSzGc3O1GhuMVp0jicjGy1sRuMzNVrajD7TjoiBzegzHaexOT4ejZZnanR3RFRFxJkGF9wd
ETE+146ILRGRoHP1tCUiqhLzTI22RESKzzS44GZE5AS16zHKK7OiJLQsqGbX6FjBC0SGrccIx4xi
KhFgJOWUfKXeU46jTAeIoPoGN8I7hxS155QIrczDQpSk4HuciVWF6lCzbLq7pmvgTzKpTC8TpbcT
iwFkYt+ZqX6ZSKa3E5H0dkLvHuYi03elwEVmen8C4dvmy9GlkYdzyWF7RUN998vowkI7V3VwKSdP
EOKUwejx/dw97SCTs9QykUxvJ5IkaAMZaRROQo187Tk6TIdZBzi/OUvSqMkEl6CdVWE40DrA+fmj
Xu4RAKBhhyESw8z6I8GpHGqdSEU6GDTPT3DDJwQZ9mcgOsw8//mtnRHS8IcaRNp9Rm+/GSJS6a3G
MD5S2SsRQgECKgRDHBoN7PJVjYM514eIQ9+oVT6TykVAlTpikEpKVQqcCMG5qgkHqtQQEhDobVtB
yzikPitBICKGHUSQDLM7BAnJERxsFMNh1SthDZ8wathFBOEwO0TOdGSvz6moypIZdhEBMcwukbPx
SXmmId/ttrvmoYaHjWYnZIdttlpvltr6bP75kGebXVbfbXnsuuAxPxWYZNn2kz6+CDLlxn1+cJ5A
G/0Arug5esaB9Dp6Vjh4oo/tPR7Kymbz7U6f5/ymdPsL9K2jrOth3ud9mPd9eZB3kv3rYxf+j+P/
L/9or/6kZI8QuprjIY/21lWJkY72auuWM9fWc+lHe0tvHHEju810f7v9pAkN09VdcZJ0FnxSUB8a
PmIr0o7c10PDYQ9xPTTcI/l6aPh5HxqW3oeGJ6YMW+VUphGrSEKLX9hsv3+BsrfbbP+wuCkhNw2l
XTzx7pPKyh2LXCfzs+V2c+slr/MhzIOVEsi4/Tm4Bv6of1WC6F2dHIuOYtxaL7PfVbK3+pytZmsl
JVOZot7iyxllDKnMR8WcgqCCJKKqOFnpLFJ/8rSeeLXkasnVkqsll2sJ6rKklD8ttU6PWicZyxaq
JtGl+V6N2/f3O1WT50tHmefzdKe3pOvCq6/ZkoDZx6slV0uulihLunLyjnGbuozb9rrhlE/XdXH8
1ZKrJVVLxNWSqyX9luAujn4KSxgG5HlZAieAT+A55IHnZsk5xPerJVdLnqMl55A9npslXRGwI0sn
9izdLvN8nu5qyTlY0nWR3tWSqyUVSxC4WlK3RATNGz/u5qpsmchmh8NscaNXnR/U7y+zF7ihqPPG
ke6l2od9nv3jdj3PFtu7IkAoRdFCi/Xf4hF2682HbLtSEvYfVTRa3OTRwgtRJeVaC9QK1vssL/aR
LqOl38xUrPy43mw+HzfU7isiIeq+Msp9T2mvqPqeUlhu90OS0aS3uvSacd0IOtpG0N5302oZ5U7Q
j5+2O/Ujrx7IhHlvBO1VeqKNoP39Qz3kfz/kD/nj1tdjVyi2vpc9oPxKsUVkepPf3uc7R9np7o/p
V9W/yfR4f8xsuVyyvLoVFCxncyQWXqr07/HlihD7JlMoFitIu+6P6W+ZzU2m/cZ5bzL1E9m7ydRP
XOj9Mb5aQraCejqGLVYAl9sPb/LFx6lKAPK7+8NU39X3cK/3IAJ9sRkKewT/naa+8hGSpGwqX6RO
Z/vPm0V5l6FSIIjePhjmHrrIaXn93TFKFFtAP5V7/SDSLRKHuQbQJRDl1YvliD192kBZ3LZIaLBY
ArosVrZGWEwWfMEfN2d+2G0f7h+3aBZXIga6uHcLqbc8lw2ffkJdN3y6pIiWDZ+zyobPXlG9t8Qg
RjGiknMGarfEsNotMRgwV49Uj7U9nfFHTFU7nKiO2D6fa7wlxlmP8VQbOB7QpfoMfeOArt8TQOMT
ON4S4/0QXYAmP5uRxebeW2JGshl32uxwS8xINpNum/tviRnJZtptc/8tMSPZzGw2912mMZLN3GJz
7y0xI9ksum3uvyVmJJtlt839t8SMZDM0h0XPW2K8je++JcbT/GZMJNhAn5OwYrJ72dH4qEZ1coRr
Sy5aV78leJNPfKhHWGESB1dJGKpkBlhfdz+AfCywhiWp94FBaApll48Yh4QiymRQKgibWYkgsE19
ENxwt1BcLli2Gk4RwNGtpu8NEMwlQ08sEk8PNXIgypiBiFW7KURluAlcJI9XQUCOWldBJOxYWIY2
TBvaMLQz2dCGrvOc7jLDBxULeC3Yzh7kkB6rBOGQDdFNNFxVtTTBsUQpI1kFaYQo4EzDoIYYyFXv
FFJwLr9c1uHZzRtlAwPcAHozXoCTYiDEAPLwmtjNQypbBIQDJrxDXemhRpHCiABlglEtXhmrYL14
6INUPcQeoUn6BrMaDCfhSECR6+qFj8x0CW0LwptS5iOEN6XMAZ79CcLr2W4bhSoTQhqu4UEG2FeC
nq13F6jKYph26yuv9EejCOb6Epg2KFwaQF8p/MGZJGDgkQ4xDIGqSBAJm11slNxClX4GlDozYL9S
eEgIStlgIx2Svslp6ZNGSS8oMiDzKB4mPg48+geOLKgxYyAkNiDziDTAYZP4BGDRAl2mzGSJr7Gl
VxoTERKTIzSvmifo6yefZUvxze5Ln9TLbAEgM1y/iVmlzPau500+ESfJnUT6qpSyIfKx9BUkEekr
XTJAlU8GeHb/vvD40TulhVSdi0hY9P5y3d2xr1FuuNpVNYdhKjk1sH1Z3Uzsc98+UfqDNPwhheFq
V1UcfiX5L6rXsQJivfeyhetmA9UDglOMh8vuYNDsB6rXSAIpOe34hPBANUGaOc7e+S0OBQQgcFTh
DQ+pP9ujip7N/opaTb1O0vflGi64hXygObPzG2llwx+6SGr7Aw9TD5xoMQVzShChMGjsxfWKSRAV
6tu9CIjKOENDk4yqh/BjHgw5HKwXETREvRHaQCwyvVc9yndXr+sEJdJwARGgw4yAg/Z2KsM20uFG
WafkGNYGQTW1ujbnc2nOjTJBTxmS5jQFkqwSvIJXC9v3JQ5ckp/R9DtulB96xG8N+apKGGiS7BlM
N+NGQaJK3/YChWowz3TCLKzVNAoSrq/CafuEGe5U+lpSqUZBIjBtL1KoOuVZ79yJ81CjPFHRsj09
jzioTAQ93cca56KkY89pF5ifbhBNuc2Gpx9Xnu52TCpzCDvTT7yC4H0VDvdaplxASv+OkKShe+uS
L3ThRrkvqWjfDqUnsweK2wxIOljchmE+IY0CX0rJWhWRypeGmVyFRMAES6L2HcOIqxbDVbEXFpVI
vYyWgID2UiBiYKCpVko5bd3Tnjofljol5pSF5X4ENTwkYHsBB1E+0FSrRAgONdUa1mLqlaqECLUX
cBDFcJBK9QSZXrG5FTKOhf/WRLcWqTNJBqh8uk/T8w2QxhtguL1kpKo/OcgbUNWPysuGG+lF+qV2
KIIWFEi9EpYq9W8vQ6kvDjM2FjfHxp9hsUcPCiBW+QaQztEj9PbF6ikmj9sXe+0Zn5Tjef2im4fd
rl/slXWl7lypOzWlV+pO/WpHnOfzuaoJhqPu1FXNxTjUnerVjv0t89KpOz5XO/pJDr3a0VfLlefT
K//K87nyfC6G5zP35vkkvsDxSakd6u18gaObvEEucFR6SfdlB2/e/uHXSXaY7T8eM5MnaOoEQ0rL
Sko9jnqq7E7nlYebmarWEMj2+WK7We5fOioqP8IyPIvYf8oXN1tVR/1L9kp381f7z/tX5fde3Txs
Pkz1kxWZ9Val58rk/T8pQXtdue3Vo6z32Z0aGVSC7PoYDSdlP2nbQZYVzio+UAb+bsqrbL7pzG30
odcFqdyQLZZkyWnlJ+aLBdTHgVxVffk9soSUgC8ZG4EC63u/q70dqT5pzRZdVLVsnq3EiuWqEKsI
RbXnVtZxTOtLLRZVHSmczbj6qKaf85hg7FX812WkjmuiGCmJ8xPXhWJEyqylIhLpJMX9bdXTILZa
sWMatz5Mi655DD8FHljlLNsPmkOsEy7WCkSuegCXTIKjno/5VImfF8ENFQlKkEykctBVmV7p2ZLq
aKyjJtGp1WIRKJqqWqOaWuX7fPd7RTzPdWK4CpOOABa8lnuqzFA1t+1iulrf5lM1Enwo2grQoR8H
agFoDssMvXyT+5l6gH1+KDJPNNMZOg5rhso9y9Wieqe4zgany/Xu8LnahHSjgatFO+FyVgQ5zKuX
ly9Ugb8+VC8uf6kSkMNLRAt36WeSPFAVQqR85RXp1fu/58Urz0OlA1nN1QueuOoF9SvYSXH9Nwlr
tK7Jnoc892TPfQTzSfbYBHclN2mTDIuimCTDInaQJMOiLzzJsPkmcZJhU5U4yXBRlSjJsKjqSDJs
xgUnGe5CHZMMV4GxSYarHp8kw1VmQJLhLjokyXCWHpVkuD+Dd5Lh4Z64JMNZUXyS4a4qJMnwkB6c
ZDj3Mcckw0Oee5LhPoL5JBligruudUqbZFgUxSQZFrGDJBkWfeFJhs03iZMMm6rESYaLqkRJhkVV
R5JhMy44yXAX6phkuAqMTTJc9fgkGa4yA5IMd9EhSYaz9Kgkw/0ZvJMMD/fEJRnOiuKTDHdVIUmG
h/TgJMO5jzkmGR7y3JMM9xHMI8lAtPv+O8/9VzZRxv1XRfBNflOZzYzrnqlx90zZ3k2rZZR7ppqZ
W1AqatN7ym1TNjsSXyhmVVUZgXouFMvlQgXH5oViKz9VblubNOkPyVom27pQzNqAjNmk1biwrU3O
It22NjmLi7pQzENL8AYkdx35Imekmt2oQFXs0ipSm/XmQ5FeKg2B8lcqNeM1+Wpcv9GbwYpNYIti
54pzIEyxu8nHNyvOa4mxSmT2eeEfHcN5sxTxMJ3my5lbUomWs1ZS6a4oOqn0UBWQVPpID00q3UcJ
t6TSR55zUuksNGVSadkLtWjuhbKJcrjbjBBVpEKJWfVuM717tnK3Gawd9LR65PhRHCGpnD8kBHFM
mek+C1mhetRva3BR9PhhvK2BEm48QuIj2JW1mUgmc06GnGWSAZ69zrBMI7POsEwiE0np3JedZYJg
f9qPI2Gk+qJQvZENIV/fOcC47oSUkJDeDBu9WXIDckrQLnZMaGeuQPdMJ6zTNL06dM/ZJajuEvX2
DMgpgbpgMXEugUQgHD++9eF0GNHeqd8i4+wg3HAQl8BwkQeU1whwjQDXCHDuEYDUezPVK1vtyztE
14ngyx/uaMNBDJhuN+mkMkaGSImGC5G+8gp3sLo7GICGy00Y6kLSXmLGwBsu0bCw9s0dYpgk6jl0
IdFwkMCm617oMCnV+XUhWXeHxhoabnpBXei9y28vsDHJwBk13fsiutB7kQ1Gs4fMtPSUHtKMc0Ek
8pVfeqhRuAnA2mgslSJ23cFwaV0KNqo2QXgbi6WJu19vn2qUbUII0w05ogscFushRgAfqMU06NLu
LmnkvrKgEbcvMqkMMzS0ljHSpQU3XUOWqjyU6WWi5GVXg1/r/u4aabnkwHjpylczANbzcgEgNF26
0slKvDh/8IY/qMqyDBeudNHHQ/1xJIoRFX/YgJNUCPqOeI8fPZMJkFIMCMAibDpZNLwuseGaG9h5
YV9kkAEMcjNfLGEY5lQKzJl0X6Kveaie/AuoHNSOObDz8r7LT1QQaHhIlfyGS2/wUOURE9J050OK
fus7CVz6o57qC9X1seFaEDlMMfQsWgxqeIhyw7VAgFf6FPId3oweAo9NRmUfwS4qHwEbH0HlhxoY
jrseI106FmQ0sRjN4JkaTTuN1hfJnKnRrNtoFU3P1GjebTQC5EyNFjaj2ZkaLS1G0zNt0xh0G43P
tSPiZjCuGM3OdZzG5vgoUJHOIgqhwAwmb9ummB9kfjM2Sr37ohXeyTAJ4fmVtrgRdjGWyHDHH/h6
E0LciPFY3wfevhKLVRa1r8V/dPGPG0kKQbC9mQxJPNBKuV6KNl9NkbL41zcmASa830DpoUZGRBhq
7y1T+f5Ai+fnN5I1ki0KcHsrGeKyeoHUdSvZdSvZdSvZWW4lw40yhOqTb+3uzL6WNQjSqHCoIIZ7
Jzkaasn1/BM10iinGKKGWygZq3jIe9rd5CFykhXYYFO/2lVd0qhUGWPtrXSI4We+G1Vf3AdoWI9p
FMMccMOtpAwMk2Q/Cw81ymNORHt7HaJ8mCT7WXioUR5zIQ33tlL8zDesxnioUcqqzMhw8yQFX8uO
VdIoXAWHhlsnCf9apuBIo3CVEBnumCR4oA2q5+ePRuYvKTbcKEkueTdW+C1yX6bkfW+R67VniBsz
tFIYpDTuhotevWbn3+eb5XrzIVvsH+722Vq1GaovjvkSFtgEgAkMkVy+rQKJsF5mv+e79epztpqt
lZRsuynfLddHRfV1op9mmt/z1FY1j6j4BMKrJVdLrpZcLblcSyCYUDqBXRcSlPKnpdbpUeskY9li
dnurg/FeRYT7e80KzZeOMs/n6a6WXC25WnK15NlaQrDfuK0PnvSP3J1Sz+f5rpZcLblacrXkmVrC
1b/+lhSkx3xXpUpms8NhtrjRBMQH9fvL7MUX7KFWJHrubTWyCh/2efaP2/U8W2zvigChFEULLe5l
LR5hp+d8tislYf+xpNRHCy9ElfBELVArWO+zvJjcW0ZLv5mpWPlxvdl8Ps5w7p9EwglQ/3ZN9Png
yntEGXHlVOPf0uLKe8y44spHxJX3vJtWyyhx5R/La71fPZAJCGCV9yg9Gau8r3+oh/zvh/whf+Sz
H7tCsRZR9oDyK8XE9vQmv73Pd46yU3LQ+1Q5ctAJyedsKesc9JyJ6gU+Lqr07yFJ5tjKQadzlucc
dHPQ+1pmm4PeZ1wAB91HpAMH3UdcOAfdT0sYB93LMWyxAuX9OiowLz7qS2ryu/vDVN/gU1wiA4G+
YweFPUIIqtxPPlLNuWwqX6ROZ/vPm0XJFlcKBKldaeTlHrrIaXl10jFKTLebfPqp5E5DpFskDnMN
oEsgaHlbUjFiT5/Q24TWwee+YgnosljZGmGxAybcU54bJtxHqDsmvD+Zs2DClzVMeI8oB0w4hkJS
ojelVjDhhNYw4RhiV49UdwQct76qmoVBBFVlIg2H+GWVmvjlBI6jHtOGgOMJHMH0fp3GhgBPsd2b
G2GwqX2nS/QWGilUDhPiclh3uSoVDcfk9FExwx6MCJc/7cEgsgV4SuRyX3nlodiGOwQ2nJITZipj
vDsoEKKFkRjTHbjuDoyIidtvZjBeYOsgdXcQlcmYKN/GDW0X6A7acAfhJqa3GbiYwB0qQODBhmtK
g4ZT1nCJrlHaLjETFhO4BEgphnNJWFDndZeoMdWAQuZmyOKFthLRcAmTJhyymap4geOIrLuDQWBA
ITNqPL9xge6AjTRYFQkG1C9DxtMal+iPRo6qj1O3RxAqB8pRBaCitU841QiiEkwa5JNGoiooM4BZ
qfkOlgQ+kUwM1Uawer1h7aSRreoDKiZE9kDZKsUC4PYJlYTFnn4wBFVckPQLbMHLQ40EVqrc3YDI
lgMlsM/BQ/WclgEGDSxKQp9hThvWp+oJLSsO7RoQ2UMltGcXi+rZLEOIGqiBWA6UzT6HHiQaHmLM
MKOk19EG8RATgsjBPQSLCTckwqbc6vku08cEDQRx9LXkuwg0/EGEYZYJyYHy3WfQYhBseEifHWx7
iA2UAT8HD9XzYUYKlFnLQ2igfPg5eAg3PMShYbZOQwmfPERdt1rYPPR0bwE1gbZGnIlxkOk94dUv
E8ugeg/VM3dGITJMLKrs7GuJGY08nVJkmFWE8KtZt2rk6aq0M0wpAj7Q+Hd+/mjk6ay4+KnlD3Qd
7c5ytGvUEIxTA69GDpURnqzKkgwCFFRloUYNodEabXaCHCojPLv+jhs1BKfcQKeRX3H+hxs1BJfC
QKcR8mtZy8eNikFgaaDT6Dv9htlsNOSqQhX2O/ZKxSOI+8nPAhioNwI+3/kgJlWeSRDBHIa1xEZu
Lxlqz8ojbr5NOd5DHGBVXQzpoRowPshD9WyfA4Db95Qhbr5OOcFq15ltDCx9who+IaR9VxniEA40
fp2ZT5zbYcxmScwbPte0Wjt59OrzWJ/XawGuoWntTLeD7pliSyaj8ZWcPbNDKvxBzDgJqwWwbHiI
8fYaXRfdM95DhOiBd+D4ITQfTtVK7seVqh4i9eqAq6bYvg2ti+75HOpJrJ6HSAqA5DJsZytseEjf
Me1K90yQgwhChm9DiCKVXFEelIMQ1PCQbo6udM/n0Iai5yQeKcOPHlIhpb2u2cX7/Do8RBoe4shw
xUwHAfQ5eChkHApjYNaLJS8GZo894x+NFz1HNTmQXkc1j4DN91U4w3y70zSIb0q/v0DfOsq6HrO/
HrOvKb0esy+O2etPFlJ9i83gfM7EgMfs66rmYpxj9tq65cy1ZV76MfvSG+URaSVsur/dflLd/ma6
utPnusks+JSwPsAvpMMBft+ocD3A7yv/eoD/eoB/hAP8ufcB/uQE96NSO12yk62lM/fZcru59ZI3
BBFe6YVkgkSa1N8mqp76gzIpY5Cx9FQsmxnXdH3cdN32bloto0zXy+z1iaQXlrHb9J4yY7fZkRhe
ZVXVn1WX8CoK5isIZi14Vc68VLll1YwQglSAtMGrrA3ImFVbjQvLqp1FumXVzuKi4FUeWoJzX3cd
+SJnpJKc7lXCUBQIxTzaevNBZ0w6eQyUv1LpKa/JV+P6ja5DivpjwXVeioNkBybWPr5Z8artu/w2
n+3zwj86l9K2i0DTaX4sy45u0Xn6dLneHT6Xnp9+2G0f7vUjoOWslQq7K4Ic5lVFCzWWr49863LO
7aVKOA8vES3qHK1K8kBVCJG8UFWRPv2SJsN5rltqWJdT0oGsvuwC7fpwX32SokHpF67+DBol3JJ7
H3nOyb2zUK/kvieptCT3q2ZybxPlQudSuZO+71nU6FyoRudCtZ1cVo9UVy+eFlEJEUQSjKjhGL4k
lQUw6pqS2BYvjtduc1TZMFdbvPAR3L39QQXs1DKRBDhQZsh2KucXCY0vUv2nLGaw62U696Je94TY
jDptxlDy87QZd9uMADlPm4nNZnaeNlOLzRSdp82s22YMztRm3rCZsfYOcixh1/p+aAB4XL0miIjo
ANC/k1cAod6Fr6cLB4m6gwjg7S3lWODKRiwe+hxVB7HH7YGQmuhfaeIjHSDm1o8cpZFZh8ekkYmk
b8LULxOSgXIDwRgAqh0D3/dVNGHZaMJEGLhCXKDESd7TuTmV1gzWhAmV6Zube2HqLhM7T6FVXx1s
JOgUAwPdh5OukxaXPz7DRuZLOWyfI8CsukMt5QBNGSJDljBUDCCT+bbFfpkEhQ4ZPfuoIaAIE5VM
hfbJvqFVo685R95ll6N8Drhq2SreBA3dsFEjMYjaJ0Iwldf0Y8j0Y4BU4XJTmrLdNupkRrGBuUY7
zzJHxi2JKDCdxEzhE195pT8aNbgaE9qnFTBFXXvxL84fjfpeQG5grpFOcs01Jx0xJ23McwgqDDQ4
0smtOeO2HOaPxhyKJMDAOqvxA69t+VzacmN6RwpooLBhAq/v7vzeXX1egwKETHy4KkP6+u7O5N0h
0Hh3DBvIdaiTdR0bQyDH5JzyIVSfxaAQEAMLDoFrPnSGbRk13l1xkVaLW1e9IOT67s7l3eHGuxPU
wNiDnTeZxM6vUoTMJIJ0M2iqNULM1JBXYz+7e6heuSqDmYm6V13RvM5AXWeLki+AScgExTxocwyi
jTbMeJs9iMFgq9zPoJfXa3qKgehjD15j2LnEsPr8AyUYGqh33ZzES5uPQaLhD45MdLrr3OI5tuVG
Ta+Cafu+H019ub67s3t3uFHTU0pMzMPOe1TOeBwKqulxo6an6t+ytoBVf3TemhI9x0GR6d6hsTLn
0ieNWplhZiIcAnHt3+fXvxu1MuPcRErkXfdGRbZnASEHg2fRgkCVMPCwOgM3amWusvF29KKSX2vl
51UrD1PXFrwrVdwSAuQwu4+wvsFZEIHDqkLcqJtVrDJQKCm5ZtJnOFo3KnoBgImPWWUbklAnm7i2
g767c/Jzo/qWCJsIiVWOddI+MuiYf7nzjuW7a8wUSEZM7EY01C61c1uFxPXqmwHC2ivqCIuBZpK4
INRMrU+a4WHGimNOIR4ioOEhwdvr1gjTLmLsV+ChetXLVIhprxchJCpVAgsNxFUP0ZPkwNfdxSON
26FE2epRTF+ibK89QzC5tFIZpDSeoWXVa3b+fb5ZrjcfssX+4W6frfdZAe+tehHjicpFvQUXjAr1
Zg5Vwu3hMFOm7/LDg/r9ZfYCNxSpwcemyERZeNjn2T9u1/Nssb273+X7vVIULbRApBWPsNPO2a6U
hP3HbKGsz6OFF6JK7IMWqBUov+dFL1hGS7+Z7bP9x/Vm8/k4vO6rIpmS2iHSE7RmE2VkLDMCUHrQ
ms2MK2htXNCa7d20WkaLi0zDKGs2paekrFn7RyQX2SY7McHNqsqR4Ib5PF9IfQS6QnATTNUhwkuV
/j0MVjC3EtwKLjJcWAlu1pZpJLhZjQsjuDmLdCO4OYuLIrh5aAkmuLn7WkOVjvRiHasfe818rQFr
CDeRUO6Cw/BqHvKDuMXuLcabW+wh2odb7CXWj1t8FqLdqGkePX3OqCjbxVa15qIdMyl0Q+ChFjpz
2JyFenHYenLObg7bEjQ5bDZR/Rw2jgiG5fRuhcOmbxKrctjcPVKdNTlOK2GBAVcFq+CwvdCCVWnV
Ma3kosc0aVJOK+mXjLhxWslHcN9SFNeHGvR8WYiDYN1BUo3dBk4R6tpLEe0gwM237CR0EAEEI4lZ
bS3Q2UGo4SCKDQdqhey6Ui/eQcw8MZnOQYhLQCkR9cuSnR2EGw6SxHBqVWWVgzgIYWX3sC2IMMQp
oYgJhIK6GKk6iALAmeEYq/rmMA5CAkAzSCVhFwNcswgYC3MQrTsISmk4s8Zl1+pIZBdDasQz7yJP
OUgjNf5wREmQg1jdQfpqx/YhNs66Fkcu30G84SABDCfFOO46hxDZxYig2ExzStnFJJRE6OPuIQ4S
dQepF2o4jqUS2mfuICGIcJ8mqTpINhzEkOE0FmNde/Auf5CGoO4hArDhrI+mrwwyCBEGVdRsrz/6
iHU5DeHuD9jwByHtkz2YyoEy52cwKEPU8JAwQbApGyp1ppIN7qHiTgpJUVhtAXHdQ/oAnYFehIfJ
nZ9DcQEbyTNlJkysSqqfeeAKj+ywkT0zYKKQEjZM9nwiD3EGOAQwcBxqpM9M7y9qewgPlD4r19Pz
ilyNbJlJE/uUgK83W4aNdFlVDgb2KWZfb7oMG/ky5yZ+JsbD5MsniVzKQ4xjQtWrD5o3bOTLAppI
jaia/dT264V66LhfD6uYy6I9ZLmuQ7ou7rvL5NJ3fHd5k4QjwJDeglxn0SZsKQghNYzyOkY8YQ7F
oNRnPj12ANRaYqNSESqfNzCQYNfZ39CWiFO2xOsbtr3hRqUlJDUsc0Ex2DLXCUZjRAAnQJXZQXXE
I4/o0UMSmzixsJP3fvnVOqpXWrp8bp+3wXoS5uvIklG9rtKXRBiWbYAYaF3rGWTJiDU8JEycSECH
qc2fRZ+qV1raZMOpFlmdI7xG3ucVeUXjDTMDX1I5aKBR8zn0gXqlCFUPaK/NITnQHOdzqKUxaHiI
GFiJSA40x/kc2hCGDQ8JAyuxTp7yXuQyegiUjQhIxMIbUQ3Q0niE8iJJgLsew7dBPX5YL7FzNxpb
jK7efnlWRpNOo/XFpWdqNO02unZn51kZzbqNrl3aeVZGc5vR7EyNFhaj6bm26WbkrxiNz7UjkmYw
rhjNznWcJub4KFD9/uPUxptifpD5jdhIkAEji0TnxSfPJUUML7VJIxATZsLICvj1poikEfUpkO3N
JojzgZbBhy+1Y0rh0kONFINS0N5sgjgZZhn8WXiokc9QaQL+8k6c9VfgoUbyxLAJAcz4MAvjz2Ic
amRqjJtAu6plffGQ8J2fNnmIP+LBVPscaBq9SICSy6Q8vUwSvJJuk5l8ER2rRCe5TBzcnCwyA/tC
owDg0AQupnSg0UKFNyGH35rGNb1R0KANoLRRbXBK2xuNVIE30LZzCTEBA40WrvLCAET1SscXQGSz
Z3zciug5/s+B9Dr+f6Qbva8Cf+bb3UG16m9Kv79A3zrKuqJbruiWmtIruqVAt+hPFlJ9i+VoPldV
w3DolrqquRgH3aKtW85cW+alo1tKb5TwDSVsur/dflLd/ma6utOkCDILRjpoKIyQDlAY36hwhcK4
y79CYa5QGFOnv0JhvkBhoDcUZgjKp1ZKrJVGJ1ZSFxiz5XZz6yVvIGooBRNA01QoNlHmCgVznh4I
aTPjWlWMW1XY3k2rZZRVRZlkP0FkwwoLm95TFhY2OxJzG62q+pP/R27jakUBbHEbV36q3JJ/KJYr
PFtZuY3WBmRM/q3GhSX/ziLdkn9ncVHcRg8twSm6u458kTNSyaT3KmEo6phigm+9+aBzMJ3pBspf
qVya1+Srcf1Gl0tFmbTgOnfCQbIDqwAf36x41fZdrqqYfV74R+dS2nYRaDrNj9Xj0S26qJgu17vD
59Lz0w+77cO9fgS0nLXydndFkMO8qmihxvL1YVphgb9UCefhJdLNFQKtSvJAVQiRvFBVkT79knjD
ea5baliXU9KBrL7sgmr+cF99kqJB6Reu/gwaJdzKBY9Rx7Fc8LHQuVxwFupVLvSkqZZyATXLBZso
J4YkgVDoIzoVhiQRNYakENjVI9VFmi9rNBBIiSTnBnaS5JUV3drmYxc9xjWa4+ZjChAzs4GcnwAa
n6C997j+FM7Np3dFKMRmZLG5uvX4nGzGnTbXdx6fk82k2+baxuNzspl221zbd3xONjObzew8beYW
m+mZtmfRbTM+1z4ou21m5zo+Q3NY7N5znMb4nj3H7ubXYyJWZa4B+Cdx196U4Kj+eNyZcdMVdXHv
sX3c2d0f9Xir/gAGnJ0EXTtRLs4fuOEPAg3wOsG7kApx/kBKsPGS8RT+UIFduM7C1X1CGj7RHd1A
Mu6CKFxcG6nnIXovrAG9JkDX4d9of1BhQiaM5496jqPiLTWA1jjvOsdxce2DN/yh3pgBS0y6Tm1c
nD/quRkmSBigYRx0ndGIHFMJISpDCS2kHz/69oNSKqWAlIUV6rLhISYN0DD9WgdpMSrIQfOO2ZQe
wpARzCnwnSQppwXqaSem1HSjByNdZzQi25CEDMmB+pSvvNIfjTyWYdP9FAxW+lSa2SmaosmUj4CM
j1BM9agMo+sxRp6ewt1Gq2roTI0mnUZjyMmZGk1tRtMzNZrZjGZnajS3GC06x5ORjRY2o/GZGi1t
Rp9pR8TAZvSZjtPYHB+PRsszNbo7IupDY2dqdHdExPhcOyK2RESCztXTloioSswzNdoSESk+0+CC
mxGRmy43o7wyK0pCy4Jqdo2OFbxAZNh6jHDMKNaoT0ZSTslXz1MDSZkOEEH1DW6Edw6pAUtKaGUe
FqIkBd/jTKwqVIeaZdPdNV0Df5JJZXqZKL2d2H23lLtM7Dsz1S8TyfR2IpLeTujdw1xk+q4UuMhM
708gfNt8Obo08nAuOTRc9FCF/LLQzmWA9CNOO64ouiRPO8jkLLVMJNPbiSQJ2kBGGoWTUCOf4bKM
TlDypc1ZkkZNJrjpwk8MB1oHOD9/1Ms9AoDpdkYkhpn1R4JTOdQ6kYp0MGiev0I8K31CTBcyIjrM
PP/5rZ0R0vCHMF2/WLviIWGk0luNYXykslcihAIEVAiGODQa9KLKMeeU1hl8yeQzqVwEVKkjBqmk
VKXAiRCcq5pwoEoNaYKg3rYVtIxD6rMSBCLThZjdV2zEjmKSIxP5KlFdEVa9EtbwCTNdgdl9qcZF
juz1ORVVWZouvey+WONCfBLMParstvTlHvXaM8SRbK2UBSmNP0Jt1Wt2/n2+Wa43H7LF/uFun61V
I+GIy1ZICpBcnFFSr+ZQBTEdDjNl+y4/PKjfX2YvWsdruFWR6UzMwz7P/nG7nmeL7d39Lt/vlaJo
ocv1/uPxHM1CWaw9pHyTFy21eQjTX/rNbJ/tP643m8/HLrv3Eml091a9yemi6Kqr2TqFlVfH9jgW
TSibQPuI5+3YUKFujg2VbnFsIRJ1iXzYzzP4Ak6yP7/7Qdu42G42+eLwXXaUv3m4m6tBAjmK2y8z
NgHFP9lflW1/y9593ixudtvN+h/6kd/9+O5N8fx5sMDD7MP/BNkfXr/5488/Zb/l+4fbg3r+7f6g
g9L3P735afr21+mPv759+/OP77Plbv17viu/89ubv/z82/TXf4vU/ONPP0yU3tnyG6gyBKTZWhlY
ZmKR4aXec1ocXs8AzCqn3+2a5rcfpw/3y9khVxHlvx/yvXqgN69+LZt78Sb0a/4u26sXo9o/FCop
plRUrrC2y7d0nrkW/kr98Up/Qf/MfpJ9UvH6u2ynnuY7lZk87G/0p4vtbvdwf9B/1YmL67MF60aD
6i5HCp3T6HOqx+SpPNU8yV7QBK09z37UzbxoJKW2yZCtNXyARF1j7tPQsMk/ZTfrDzcv9ve5clox
UtRGB6p+VvduzfJ4cb9Y+yp4qxRUpK40feS7bL38i0p+trvvgarc9Kd/2m2XD4vD90sAm03AV8X+
sFMGq3f9y2r3vWrmj6JVq3uX79az27fFk32PffUcBU2yn9UwvNvMbgut+KW3wb/MNg8rlZY97PLd
JHu/3d+s5zNfIdVnmWQI6OuuEASqxoHNEd0i68Uj/UXLhC9BGSx+me332bvjd45uXeYHNUb5tLC2
7H9/WO8+7rO72UFloqttgbLJdBPI7tVf9LvXT9Igmdj17Bf7ddHpmFFnXQqGNilP/f2n9U496ovX
C03RKcqw4zvKWq8+Uw6H2Z/+faLKoNdv372ZZMxVZWWAeV3k5qr/FXY8sYU+oOzw+T5vDopOIssx
6y/57nM2X384vsSX2fvdZ92b1eCoM6Hffn79U/bj6z+9/vHN+//6BrJvXwZr4hhRxhihGQGSvSiq
yNvth/VC+aoYglWH/EY5DGTvf3iFXnKcvV//8G2wutd7VTTpJ/mPnards59NuVaIuGKYLrMYHbC0
bF13OwrWw7IWP4fFAB1uzmN7KHIqnV46iopJfAOEeiS+AdL7El+MOkT+/J/vyYvVPvtGvwkl6E4F
Hc0gXqkIuf+811NDn9aHm2y7U7+pvr7U1Kq77VL1j1/vD7qhbh5ub780TjwBYEK66nAvSJpdlBmS
JlVnSQxJs5txhaSNCUmzv5tWy2ihl0UIIc2u9HSEtJ7+EYVetstOSl/rURVBX4MriCEkXqoKsLL6
1R76GiZ5TpYW+lpPyzTQ13qMC6GveYh0oa95iIugr3lpCaSv+TiGcSnmJSD5Jl98nKqMIL+7P0wX
D7ud9vm8ThjzeY0hcDQv+QGIZB/XeCKSvUS7I5I9xfpwjM9ENFnwBX9klBXTM4+kMszD354LSc1P
niP3zEOoB/esN/O0cM9wnXtmF9XPPWNCYCiBvi/RjXvW45HqQm0U98xNj3Gdtod75vEEsdwzv4ew
HDnwsDmWezaGzbjTZjfu2Rg2x3LPxrA5lns2hs2x3LMxbI7lno1hcyz3bAybY7lnY9icjHvmZ7yV
e+ZjfhT3LCKq2/bVxr7H5r5aH39Ecc8u0B+44Q8v7lmsPzq4Z2n8Ueee+fgkint2gW0kinuWwB8m
7tmY/ojinl1g+4jinl2gP6K4Z9Fjahf3LD4BsXPPfDwUxT2LbjFdEKu0Hmpzz3ymBaK4Z9FtyHxm
Lk2f8pVX+iOKexbhjx7umc8jxHLPRpmewt1GO3HPRjE6lns2itGx3LNRjI7lno1idCz3bBSjY7ln
oxgdyz0bZXY7lns2itGx3LNRjI7lno1idHdEdOOejWJ0LPdsFKNjuWejGB3LPRvF6CjuWXh23cM9
S1ePmblnSes9A/fM5w3Ecc8iChwr9yzVTCxx3yLlLrPCPUsmE6W3E7vv4XKXiX1npvplVrlnyWSS
9HZC7x7mItN3pcBFZnp/VmlcPqNLFPcsfHDp4Z5dlqcdZFa4Z4lk1rhnyWSSoA1kcdyzy5uzjOOe
XaA/orhnCdaaTSSYZJEOBs3zx3HPLm/tLI57FhupOrlnKSsRE/cspXwT9yyhfCP3LKV8E/csaaVm
4J75tNAo7ln8KGbkniWrK8KqVxLFPbvQkT2Ke3YxPjk998xuz/gHhUXPwTUOpNfBtSNU7X0VMzbf
7vRZ629Kv79Arseor4eOr4eOa0qvh46LQ8f6k4VU34KAzOdMDHjouK5qLsY5dKytW85cW+alHzou
vVEeGlXCpvvb7SfV7W+mqzt9VpLMgo8h6uPMQjocZ/aNCtfjzD7yr8eZr8eZuxv29Tjz43Fm4n2c
OT1iuFQqrPWG6Qy1csciXxY8wu3m1kveIMhipZdOsP05koCF7Yp8CyKLKGNBxAFFloIIAkBoVmkK
Rb1Zdl49riOhg1JjXdZiw7WCGbmCsbybVrMoK5iHAg0WWLtY1J20drHYkRpqZFPVqi8ec/zZSrI5
qiT9xRcWzS8gIXxU1eqL+hdk4wszXntuAAl3bjfm+sJmXGB94SrSsb5wtrDwRiXBTTES1jVAhNkx
O7/d7vOSV6ukrnRajsOEFgsBq0LocSiZ7m8eDsvtJ1UiPagCvCgqgr2iPlB+TLzUIDedbbZf5Or8
K9DqnENcdXYh/FGu9rEMNZfMV8cqQofl3zVwWr9DlRY8ikfaHSxMvqpuADu+QpUBqOT2blPktqxV
WTm3Y5XZzUpfHGb7j0UWrtMZzc8K94SezshLS3X6PT1spw979QsqXCp/bO+1dG0zD5SumkVeSN9/
3i/U2DAt056nslz37byVSrvKV6k0WZbvsUjYpu/+692Pr//4xykj09VsfzhqkDqtlsQ9y7Kk1bSV
VvdmhqYMt+D/TIvvwALciQnCesWIYlYQgr7TZNDNViVjD7e3KdUVMKLvJaQSQM6p4Og7DTtdfi8E
ZExwfXoNfJfdrzcqOf1eA8hz1SB+L/9+N/us28f3UBDCOCGAYw4kJZyi4ifLfP17qKrC+vIUnRDg
bbWVwRoqtKgtim6psdfZdpUVVNY6iD9UuBvgNVS6BfCqRLIJIYkKB4uoeuEAHjNEAQdArlrMuNYO
I9cOlnfTahll7VAuBjxWxGElhEXrSUsIix2pSwibqlYJ0VqiKD4zclEFyXkjmexTVfwew3OjzV9+
guLVfEHsXFRb8zGXEDbjAksIV5GOJYSruDguqruW8IUEdx3qjdNmMXR4qoh00aKnthEOEh+6mOAs
H9IczSry9zfbh9vlNNdD9JcZO523z1q1gPtDAL7kFSVqXFatcjO7V9qKJSmoJ7x56CPogaoivRhf
i2l65aj8oOuZL6rmKLjRIpADufpS0uS76Zcpdoh1rQQDHwEsV6z6lpf5/rDbfp6WKeixWygtC/2q
F0E6XBcGPOS5Lwy4CvVbGLCne5YKhrUqGIsoB84pUVWEZHrHlDPn1OaR6sasWM6pgx7jvqx+zqnr
EyTgnHo8hP2IoavNCTinJ7cZd9rszDk9uc0JOKcntzkB5/TkNifgnJ7c5gSc05PbnIBzenKbE3BO
T25zSs6ph/F9nFNn82M5p6FRveccTdR7NJyjcfZHLOf00vyBG/7w5ZxG+aObc5rAHy3OqbNPYjmn
l9ZGYjmnsf7o4JyO5o9YzumltY9Yzuml+SOWcxo3plo4p5EJSC/n1NlDsZzTuBZjgVYm9JCRc+o8
LRDLOY1rQ51n5BP0KV95pT9iOaeh/ujnnDo/QgLO6emnp3C30a6c09MbnYBzenqjE3BOT290As7p
6Y1OwDk9vdEJOKenNzoB5/T0s9sJOKenNzoB5/T0RifgnJ7e6O6I6Mw5Pb3RCTinpzc6Aef09EYn
4Jye3uhYzmlgdt3POU1Uj3VyTtPVe2bOqfMbiOachhY4fZzTJDOxxH07j7vMOuc0jUz3PXrOMhuc
0zQyse/MVL/MBuc0jUyS3k7o3cNcZPquFLjITO/PBn3TeXSJ5ZwGDi79nNML8rSDzDrnNIXMJuc0
jUwStIEsmnN6YXOW0ZzTS/NHLOc0dq25g/yWJtLBoHn+aM7pha2dkVjOaVSksnFOk1UiHZzTZPI7
OKep5HdxTpPJ7+CcpqvUzJxT5xYayzmNHMW6OKdp6oqw6jWac3qJI3ss5/QyfDIK59Riz/inc4M4
p30e9uGcWmRdT/peT/rWtT6jk776k5IQSharOR7ypG9dlRjppG8bRmprPpd+0tcfRuoqOQ5G6q7l
eoa4T/71DLHHY1zPEF/GGWLufYZ4ELioUkqtSb4nXLRP3kBwUUgn0E4DSgQXhWyCcIcilVhm8AWc
ZH9+94P6kf1iq3rO4vBddlS6ebibK2XUUdx+mbEJKP7J/qoM/lv27vNmcbPbbtb/0Fikdz++e9OC
LoUKzFUqoSR9A1Udu5qtb3W2+Vu+f7g9TLKb7f6ga7jvf3rz0/Ttr9Mff3379ucf32fL3fr3fFd+
57c3f/n5t+mv/9YwpYuzVboKT7JN/im7UUn8i/19rpxdeK7mLaZ+Vj+tzqtfqAS7oUD2KXirFFSk
rnQl8F22Xv5F9czt7ntABNCf/mmniqTF4fslgM3X7atCDZvK4L2qu1a77+F32aNo9F32TtUWs9u3
xZN9j331HAVNsp9VH99tZreFVvzS2+BfZpuHlWruD7t8p2rM7f5GFTG+QqrPMslUkoABQRAgNSgj
d1kvHisxLRO+BGXn+WW232fvjt85unWpwujiUGOAecv+94f17uNe1aEH1cPVKKbLykw3gexe/UW/
e/0krbrCpme/2K+LHsKMOutScNfYWEh56pw/rXfqUV+8XuiKtii6ju8oa736TDkcZn/6d11Evn77
7s0kY64qK6PB62LMU/2vsOOpzv+AssPn+/o+JleR5QDzl3z3OZuvPxxf4ktVq3zWvfmwLdB0v/38
+qfsx9d/ev3jm/f/9Q1k375saOqibbc1qfqMMsZU9CFAshfFnNPt9sN6oXxVZMGqQ36jHAay9z+8
Qi85zt6vf/g2WN3r/f7hTj/Jf+zWStPPLT5doLhiTC1H9Un2qZCt5zkcBeu4psXPYRHhws15bA9F
jNFIvqooPkH2C5ACSIRhQh1JhGHCXUmEYdKtJEIoJhimmeu0iTIjzBEg6UmENjOu85Pjzk/a3k2r
ZSSbn7RpPeX8pM2OxCRCq6r++clOEqEqLZd85qdK/x5CMzazzk/S5YrmcGElEVqbj3F+0mpc2Pyk
s0i3+UlncVEkQg8twbOI7r4Om+Zzlx8xzefxEAHTfB6PEDfN5/EYntN8HpKDp/ncu5jbNJ+PPOdp
PmehXtN8PfmYZZpPNKf5bKKcUIGSCagR566oQKtHqnsbIlGBLnqMWxt6UYHOTxCPCvR5COspHWeb
41GBp7cZd9rsigo8vc3xqMDT2xyPCjy9zfGowNPbHI8KPL3N8ajA09scjwo8vc0JUYE+xvegAt3N
j0QFBkd1+1b0uPfY3oru7o9IVODF+QM3/OGJCozzRycqMIU/mqhAd59EogIvro1EogKj/WFGBY7n
j0hU4MW1j0hU4MX5IxIVGDmmdqMCYxOQPlSgu4ciUYGRLaab+5bSQyZUoPu0QCQqMLINdR0zTdGn
fOWV/ohEBQb7oxcV6P4I8ajAEaancLfRjqjAEYyORwWOYHQ8KnAEo+NRgSMYHY8KHMHoeFTgCEbH
owJHmN2ORwWOYHQ8KnAEo+NRgSMY3R0RXVGBIxgdjwocweh4VOAIRsejAkcwOhIVGJpd96ICU9Vj
XajAhPWeERXo/gZiUYHBBU4PKjDNTCxx387jLrOGCkwkE6W3s44KTCQT+85M9cusowITySTp7YTe
PcxFpu9KgYvM9P6sA+zcR5dIVGDo4NKLCrwkTzvIrKECk8hsoAITySRBG8hiUYGXNmcZiwq8OH9E
ogKj15rN8KREkQ4GzfPHogIvbe0sFhUYF6ksqMB0lYgZFZhOvhkVmEx+ByownXwzKjBhpWZEBbq3
0EhUYOwo1oEKTFRXhFWvsajAixzZI1GBF+KTMVCBNnvGPz4bggrs9bAHKtAm63oU93oUt671GR3F
1Z+U/D62xD2oQKuq/qO4dVVipKO4LVSgtflc+lFcb1Sgs+QoVKCHlushX8eHuB7y7ZF8PeR7ykO+
0vuQ7xAsP60UWbNwP5Zfr7yhWH5yQrq4cUlZfghMgL1sCQAZhQl1BBmFCS+QRVps5ZC4LkcSuGII
qVYykhIJ7VR6k8iFat4Hu61UToC91ZkEq8qzkAtVaZqtlJIHlSnrojuZZGSXDP39q5rU4qMW/ec/
v/mp7gaiOz3kHSKf0H92viVzFBfAtwwXmJxvWZrSS0ns5VvyLr6lo4IYvmWYCn++paOefr6loyA7
39JRiBPfsldWBN8yQHYQ39Kux5VvWUhBXUPzIHzLHpUhfEt3kbF8S19NkXxLX3U9fMtgcX18S7vg
Lr6ltzndfEui809sRzz3hkDuKC4sBAYKHCIEqlS9l4bdGwKFJQTKbij6k4LIEBigIigEuuhxCoEu
gnpDoIsQ1xBol1UJGdA/BPrKDg2BFj3dIRA2Q6Cc0K6SeagQaFMZGAIdRSYIgV6a4kOgl7r+EBgm
ziEEWgRbQqCfOT0hkHaN64EzI+FCbXV7qFTLHINd5M//+Z68WO2zb/QbUILuVLDRK60rFU/3n/d6
4f/T+nCTbXfqN9XXl3rF7267VP3i1/uDbqCbh9vbaqNEoHu6y2ux2i7KfK+d3v2VmPVsN+O6wDzm
ArP93bRaRqIFZrvW0y0w2+1IynruUdW/wNzJeoaSEj4nXqr0783gIs+tC8xyjvL5nFlYzz3Nx7DA
3GNcyAKzh0iXBWYPcRGsZy8tgcvAPr4OWQb2kR+8DOz1EN7LwF6PELMM7PUYXsvAXpIDl4F9upjL
MrCfPMdlYA+hHsvAvfmYZRl4Vl8GtotyYD1jQhCmAAhH1nOPR6qbU6NYz256jHtTe1jPHk8Qy3r2
ewjLMWsPm2NZz2PYjDttdmM9j2FzLOt5DJtjWc9j2BzLeh7D5ljW8xg2x7Kex7A5lvU8hs3JWM9+
xltZzz7mR7GeI6K67Sxh7HtsniX08UcU6/kC/YEb/vBiPcf6o4P1nMYfddazj0+iWM8X2EaiWM8J
/GFiPY/pjyjW8wW2jyjW8wX6I4r1HD2mdrGe4xMQO+vZx0NRrOfoFtMF7k3roTbr2WdaIIr1HN2G
zJyQNH3KV17pjyjWc4Q/eljPPo8Qy3oeZXoKdxvtxHoexehY1vMoRseynkcxOpb1PIrRsaznUYyO
ZT2PYnQs63mU2e1Y1vMoRseynkcxOpb1PIrR3RHRjfU8itGxrOdRjI5lPY9idCzreRSjo1jP4dl1
D+s5XT1mZj0nrfcMrGefNxDHeo4ocKys51QzscR9O4+7zArrOZlMlN7OKus5mUzsOzPVL7PKek4m
k6S3E3r3MBeZvisFLjLT+7NKIPYZXaJYz+GDSw/r+bI87SCTs9Qya6znZDJJ0AayONbz5c1ZxrGe
L9AfUaznBGvNJvplskgHg+b541jPl7d2Fsd6jo1UnaznlJWIifWcUr6J9ZxQvpH1nFK+ifWctFIz
sJ59WmgU6zl+FDOynpPVFWHVaxzr+UJH9ijW88X45PSsZ7s94x+f9Wc9O3jYmfVsl3U9ins9ilvX
+oyO4upPSgDzEs2trOceVf1HceuqxEhHcRus557mc+lHcT1Zzx6SI1jPXlquh3wdH+J6yLdH8vWQ
7ykP+c69D/mmZz2XSruIjf6sZwd5g7CelV48oQHVhC/rWSkiE5CcaNQvVAcnzS9qyP3nXb7Y/p7v
Pv9ztt5ny1zJXszUaPNdofGf1R86i3241zWQ+pHN/qBeWfTTHGpgrlJDWWapGqYAGWU6q07gNTuy
KUyqFdmEaDdp2rcKtIgy3/jD1S+mhyhZzLhWbiNXbpZ302oZ6So3i9aTVm4WO1JDlGyq+iu3TogS
mC0wpDMvVfr3yGq2ZNbKbTabSzRb2SFKtuZjrtxsxgVWbq4iHSs3V3FxECV3LeH1lbOvA+srZ/kx
9ZX7Q4TUV+6PEFlfuT+Gb33lLjm8vnLuYo71lYc89/rKVahffWXPxyz11aJVX1lEOUCUkGrAhAjJ
3CFKNo9UV31iIUoOeoyLPv0QJdcnSABR8ngI+/5lV5sTQJRObjPutNkZonRymxNAlE5ucwKI0slt
TgBROrnNCSBKJ7c5AUTp5DYngCid3OaUECUP4/sgSs7mx0KUQqN6zya9qPdo2KTn7I9YiNKl+QM3
/OELUYryRzdEKYE/WhAlZ5/EQpQurY3EQpRi/dEBURrNH7EQpUtrH7EQpUvzRyxEKW5MtUCUIhOQ
XoiSs4diIUpxLcZCxEnoISNEyXlaIBaiFNeGOg/gJOhTvvJKf8RClEL90Q9Rcn6EBBCl009P4W6j
XSFKpzc6AUTp9EYngCid3ugEEKXTG50AonR6oxNAlE5vdAKI0ulntxNAlE5vdAKI0umNTgBROr3R
3RHRGaJ0eqMTQJROb3QCiNLpjU4AUTq90bEQpcDsuh+ilKge64Qopav3zBAl5zcQDVEKLXD6IEpJ
ZmKJ+3Yed5l1iFIamSi9nQ2IUhqZ2Hdmql9mA6KURiZJbyf07mEuMn1XClxkpvdnA+3jPLrEQpQC
B5d+iNIFedpBZh2ilEJmE6KURiYJ2kAWDVG6sDnLaIjSpfkjFqIUu9bcgZVIE+lg0Dx/NETpwtbO
oiFKUZHKBlFKVol0QJSSye+AKKWS3wVRSia/A6KUrlIzQ5ScW2gsRClyFOuCKKWpK8Kq12iI0iWO
7LEQpcvwySgQJYs94x+fDYIo9XnYB6JkkXU9ins9ilvX+oyO4upPSrJRDnEfRMmmqv8obl2VGOko
bhuiZGs+l34U1x+i5Co5DqLkruV6yNfxIa6HfHskXw/5nvKQ79L7kO8gEKXeLNwTohT8EJEQJTYB
xL+aCIAo8QnoojWVWdX6TmV+f+WciL9Nsn3+YTV7uC3wQqrAnS1UcF7fVwI2IXpXdLYvv8ZVC+SC
0aUKB0eLyZNzjpJJ8Xv/S+ULAPytapmYqFrZ91X24Z0ChfbQj4Kk2ulHcqKK5zTlm0WUmYGrqokB
6EcWM64l18gll+XdtFpGupLLovWkJZfFjtT0I5uq/pKr+KyTfjTzU6V/j+dysbSWXJRBIRdzO/3I
1nzMJZfNuMCSy1WkY8nlKi6OfuSuJbwwcteh3jitpMzFc+hRf1pMgCoFK6rrIhwkPrTucpYfU3e5
P0RI3eX+CJF1l/tj+NZd7pLD6y7nHuxYd3nIc6+7XIX61V32dM9Sd+WtussiygWuBCWlTDDiDley
eaS6GhQLV3LQY1wM6ocruT5BAriSx0PY9zW72pwArnRym3Gnzc5wpZPbnACudHKbE8CVTm5zArjS
yW1OAFc6uc0J4EontzkBXOnkNqeEK3kY3wdXcjY/Fq4UGtV7Nu9FvUfD5j1nf8TClS7NH7jhD1+4
UpQ/uuFKCfzRgis5+yQWrnRpbSQWrhTrjw640mj+iIUrXVr7iIUrPR9/ON1NGQ1XclDU7Q8bXMlD
cBBcydlDsXClKA/ZSDkJPWSEK7l6KBquFNeGOg/meIh1Opjj7I9YuFKoP/rhSs6PkACuFOX+IKNx
t9GucKXTG50ArnR6oxPAlU5vdAK40umNTgBXOr3RCeBKpzc6AVzp5EangCud3ugEcKXTG50ArnR6
o7sjojNc6fRGJ4Arnd7oBHCl0xudAK50eqNj4UqB2XU/XClRPdYJV0pX75nhSs5vIBquFFrg9MGV
ElR8LbhSGpl1uFIamSi9nQ24UhqZ2Hdmql9mA66URiZJbyf07mEuMpsrBSlkpvdnA/njPLrEwpUC
B5d+uNIFedpBZh2ulEJmE66URibxjSzlFptYuNKFzVlGw5UuzR+xcKU4f3TiJtJEOhg0zx8NV4pb
CbGtncX6JGjtLBquFBWpbHClZJVIB1wpmfwOuFIq+V1wpWTyO+BK6So1M1zJuYXGwpUiR7EuuFKa
uiKseo2GK13iyB4LV7oMn4wCV7LYE3A6t0tU4OncILhSn4d94EoWWdeTvuOf9O16N8Oe9O3SeuqT
vp1jU/9JX/1JSTxakV64kk1V/0nfuqomXKl90teqKvikbxuuZGs+nSd9O40LP+nrJNL9pK+zOD+4
kqvkOLiSu5aoM8SOOkLPELs1kvAzxG7yI88QOz5E4Blix0eIP0Ps+BgBZ4gdJUedIXbrzO5niF3l
eZ0hdhLqfYbYkpRazhCvTGeIe7LBAdhNvUm+J7sp+CHi2E0YTmAACdaX3SQmAEwA9naYlZAULtRG
SAqVaiEkKZFwQrpEetVgdlHGGkxCLlITkuxmXOumMesm+7tptYxq3fTUnUMqJ7ve01VOdjuSMpJ6
VPVXTp2MJDFfzTD1U1VUTjM0y62VE6eSUbK0MJJ6GpChcuoxLqRy8hDpUjl5iItgJHlpCaxvfHwd
UoD4yIcc53k1e1djyfpQLT5eqrzy8BJpd2GQK1WSB6kCXDIJjuXsx3z6cD+dr3VWjQr7A81HiJTm
Vyyu1gVzbTEMa0ZKOpCgWdoou+ulGSnKArII84pLXeAnz7Eu8BDqURf0JkrddUEO6nWBXZQDWwij
AhCKkSNbqMcj1cWQKLaQmx7jWkgPW8jjCWLZQn4PYdnW62FzLFtoDJtxp81ubKExbI5lC41hcyxb
aAybY9lCY9gcyxYaw+ZYttAYNseyhcawORlbyM94K1vIx/wotlBEVLftXYt9j829az7+iGILXaA/
cMMfXmyhWH90sIXS+KPOFvLxSRRb6ALbSBRbKIE/TGyhMf0RxRa6wPYRxRa6QH9EsYWix9QutlB8
AmJnC/l4KIotFN1iukAxaT3UZgv5TAtEsYWi25D5XEqaPuUrr/RHFFsowh89bCGfR4hlC40yPYW7
jXZiC41idCxbaBSjY9lCoxgdyxYaxehYttAoRseyhUYxOpYtNMrsdixbaBSjY9lCoxgdyxYaxeju
iOjGFhrF6Fi20ChGx7KFRjE6li00itFRbKHw7LqHLZSuHjOzhZLWewa2kM8biGMLRRQ4VrZQqplY
4r5VyF1mhS2UTKb7nh1nmVW2UDKZ2Hdmql9mlS2UTCZJbyf07mEuMn1XClxkpvdnlXjjM7pEsYXC
B5cettBledpBZoUtlEhmjS2UTCYJ2kAWxxa6vDnLOLbQBfojii2UYK3ZRFtIFulg0Dw/wQ2feLGF
Lm/tLI4tFBupOtlCKSsRE1sopXwTWyihfCNbKKV8E1soaaVmYAv5tNAotlD8KGZkCyWrK8Kq1zi2
0IWO7FFsoYvxyenZQnZ7xj/X6s8WcvCwM1vILut6RvZ6Rrap9xmdkdWfFMgfDGfzuUqkhzsjW1e1
FOOckW3QhXoa0KWfkfWkC3lIjqALeWm5nr71ed3X07cmr1xP3z6dvoXep2/TU3lKpcSaHvtQeRzk
DULlKfVS63PoUVdTbBqP8u7j+v5ef70cC1TK/pShVocFH1Wdj2joURMoKHB4PownEPiXMZ7UIQQm
UE6wHW/kSx2KEGqhDgVL7aYOKZEIToC9GQU8fahQ+9OHSTU8/f8HUEsBAhQDFAAAAAgAxblcSvA0
yQ7GVAAAAL0EABgAAAAAAAAAAAAAAKSBAAAAAGRtZXNnX3NpbmNlXzIwMTcwMjI2LmxvZ1BLBQYA
AAAAAQABAEYAAAD8VAAAAAA=
=2D-nextPart2283738.6rFPRxsZn9--
This is a multi-part message in MIME format.
[-- Attachment #1.2: Type: text/plain, Size: 2475 bytes --]
(Resend with the log compressed this time.)
On Friday 17 February 2017 09:19:15 Kai Krakow wrote:
> Am Tue, 14 Feb 2017 13:52:48 +0100
>
> schrieb Marc Joliet <marcec@gmx.de>:
> > Hi again,
> >
> > so, it seems that I've solved the problem: After having to
> > umount/mount the FS several times to get btrfs-cleaner to finish, I
> > thought of the "failed to load free space [...], rebuilding it now"
> > type errors and decided to try the clear_cache mount option. Since
> > then, my home server has been running its backups regularly again.
> > Furthermore, I was able back up my desktop again via send/recv (the
> > rsync based backup is still running, but I expect it to succeed).
> > The kernel log has also stayed clean.
*sigh*
It was working fine... until the 26th (so almost 2 weeks without incident).
Now the error keeps occurring, and the clear_cache mount option stopped
helping. Just out of curiosity, I used "clear_cache,space_cache=v2", which
also didn't help. I've attached the full log, in case it's in any way useful.
This is all with:
% uname -a
Linux diefledermaus 4.9.12-gentoo #1 SMP Fri Feb 24 20:09:28 CET 2017 x86_64
Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz GenuineIntel GNU/Linux
% /sbin/btrfs --version
btrfs-progs v4.9
> > Kai, I'd be curious whether clear_cache will help in your case, too,
> > if you haven't tried it already.
>
> The FS in question has been rebuilt from scratch. AFAIR it wasn't even
> mountable back then, or at it froze the system short after mounting.
>
> I needed the system back in usable state, so I had to recreate and
> restore the system.
I think I'm at that point now myself, unless anybody has any other ideas.
What I *am* going to do before giving up and reformatting is create an image.
I found out that using the -w option to btrfs-image keeps it from segfaulting,
so I guess the underlying problem has to do with the extent tree? However,
the image is... pretty damn big (8.6 GB by now, *after* compressing with "xz
-9"), and it's not even finished yet. So, under the assumption that the image
could be useful to somebody, I would welcome any suggestions for where to
upload such an image once it's done.
> Next time, if it happens again (finger crossed it does not), I'll give
> it a try.
Yeah, I hope it doesn't come back, either, for either of us.
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #1.3: dmesg_since_20170226.log.zip --]
[-- Type: application/zip, Size: 21848 bytes --]
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-28 22:14 ` Marc Joliet
@ 2017-03-01 8:23 ` Marc Joliet
2017-03-01 9:32 ` Qu Wenruo
0 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-03-01 8:23 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 497 bytes --]
On Tuesday 28 February 2017 23:14:54 Marc Joliet wrote:
> I think I'm at that point now myself, unless anybody has any other ideas.
For example, could the --init-extent-tree option to btrfs-check help, given
that I needed to pass -w to btrfs-image?
Also, the image is complete, so I only need to find somewhere where I can
upload a 9.4 GB file.
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-01 8:23 ` Marc Joliet
@ 2017-03-01 9:32 ` Qu Wenruo
2017-03-01 18:14 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Qu Wenruo @ 2017-03-01 9:32 UTC (permalink / raw)
To: Marc Joliet, linux-btrfs
At 03/01/2017 04:23 PM, Marc Joliet wrote:
> On Tuesday 28 February 2017 23:14:54 Marc Joliet wrote:
>> I think I'm at that point now myself, unless anybody has any other ideas.
>
> For example, could the --init-extent-tree option to btrfs-check help, given
> that I needed to pass -w to btrfs-image?
--init-extent-tree should be avoided normally, unless you're 100% sure
about only extent tree is corrupted, and you have enough faith that
--init-extent-tree can finish.
Or it will mostly make things worse.
Before trying any RW btrfs check operations, did you run btrfs check on
the image?
And btrfs check --mode=lowmem is also recommended as in some rare case,
low mem mode can detect bug which original mode doesn't.
(And it also helps us to enhance lowmem mode)
>
> Also, the image is complete, so I only need to find somewhere where I can
> upload a 9.4 GB file.
Is it a compressed dump? Dumped with btrfs-image -c9?
If so, such large one won't help much unless there is some developer
really interested in inspecting the image.
Thanks,
Qu
>
> Greetings
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-01 9:32 ` Qu Wenruo
@ 2017-03-01 18:14 ` Marc Joliet
2017-03-01 18:27 ` Marc Joliet
` (3 more replies)
0 siblings, 4 replies; 23+ messages in thread
From: Marc Joliet @ 2017-03-01 18:14 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2503 bytes --]
On Wednesday 01 March 2017 17:32:35 Qu Wenruo wrote:
> At 03/01/2017 04:23 PM, Marc Joliet wrote:
> > On Tuesday 28 February 2017 23:14:54 Marc Joliet wrote:
> >> I think I'm at that point now myself, unless anybody has any other ideas.
> >
> > For example, could the --init-extent-tree option to btrfs-check help,
> > given
> > that I needed to pass -w to btrfs-image?
(First off, thanks for taking the time to respond!)
> --init-extent-tree should be avoided normally, unless you're 100% sure
> about only extent tree is corrupted, and you have enough faith that
> --init-extent-tree can finish.
>
> Or it will mostly make things worse.
OK, that's why I asked :) .
> Before trying any RW btrfs check operations, did you run btrfs check on
> the image?
If by image you mean the device in question, not since the last time (see my
previous messages). Or can I really run btrfs check on the image? In any
case, I started btrfs-check on the device itself.
> And btrfs check --mode=lowmem is also recommended as in some rare case,
> low mem mode can detect bug which original mode doesn't.
I did see differences in output the last time around (again, see my previous
messages in this thread), so I'll run with lowmem. It won't be done until
tomorrow, though.
> (And it also helps us to enhance lowmem mode)
OK
> > Also, the image is complete, so I only need to find somewhere where I can
> > upload a 9.4 GB file.
>
> Is it a compressed dump? Dumped with btrfs-image -c9?
It was created with:
btrfs-image -s -w /dev/sdb2 - | xz -9 --stdout > ./btrfs_backup_drive_2.img.xz
(Mainly because I felt more comfortable using a separate compression utility,
not for any rational reason. Although if you really meant "image" above, I
have the feeling I'll regret this decision.)
> If so, such large one won't help much unless there is some developer
> really interested in inspecting the image.
When I started without compression, it was something like 46 GB, after only
1-2 hours, so I expect the uncompressed size to be... very large :-/ .
> Thanks,
> Qu
>
> > Greetings
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-01 18:14 ` Marc Joliet
@ 2017-03-01 18:27 ` Marc Joliet
2017-03-01 18:43 ` Marc Joliet
` (2 subsequent siblings)
3 siblings, 0 replies; 23+ messages in thread
From: Marc Joliet @ 2017-03-01 18:27 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 698 bytes --]
On Wednesday 01 March 2017 19:14:07 Marc Joliet wrote:
> > And btrfs check --mode=lowmem is also recommended as in some rare case,
> > low mem mode can detect bug which original mode doesn't.
>
> I did see differences in output the last time around (again, see my
> previous messages in this thread), so I'll run with lowmem.
OK, just to prevent confusion: it turns out I never posted the output of
btrfs-check with --mode=lowmem from the first time I was seeing these errors,
but I remember the output being (slightly) different from the original mode.
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-01 18:14 ` Marc Joliet
2017-03-01 18:27 ` Marc Joliet
@ 2017-03-01 18:43 ` Marc Joliet
2017-03-02 9:44 ` Marc Joliet
2017-03-05 23:53 ` Marc Joliet
3 siblings, 0 replies; 23+ messages in thread
From: Marc Joliet @ 2017-03-01 18:43 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]
On Wednesday 01 March 2017 19:14:07 Marc Joliet wrote:
> > > Also, the image is complete, so I only need to find somewhere where I
> > > can
> > > upload a 9.4 GB file.
> >
> >
> >
> > Is it a compressed dump? Dumped with btrfs-image -c9?
>
> It was created with:
>
> btrfs-image -s -w /dev/sdb2 - | xz -9 --stdout >
> ./btrfs_backup_drive_2.img.xz
>
> (Mainly because I felt more comfortable using a separate compression
> utility, not for any rational reason. Although if you really meant
> "image" above, I have the feeling I'll regret this decision.)
Ah, never mind, it's not as big as I was worried it would be:
% xz -l btrfs_backup_drive.img.xz
Str. Blöcke Kompr. Unkompr. Verh. Check Dateiname
1 1 9.589,6 MiB 81,4 GiB 0,115 CRC64
btrfs_backup_drive.img.xz
So 81.4 GB uncompressed. I have the space to uncompress it if need be.
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-02-02 12:01 ` Marc Joliet
2017-02-03 22:44 ` Kai Krakow
@ 2017-03-02 0:43 ` Qu Wenruo
2017-03-02 9:43 ` Marc Joliet
1 sibling, 1 reply; 23+ messages in thread
From: Qu Wenruo @ 2017-03-02 0:43 UTC (permalink / raw)
To: Marc Joliet, linux-btrfs
At 02/02/2017 08:01 PM, Marc Joliet wrote:
> On Sunday 28 August 2016 15:29:08 Kai Krakow wrote:
>> Hello list!
>
> Hi list
[kernel message snipped]
>
>> Btrfs --repair refused to repair the filesystem telling me something
>> about compressed extents and an unsupported case, wanting me to take an
>> image and send it to the devs. *sigh*
>
> I haven't tried a repair yet; it's a big file system, and btrfs-check is still
> running:
>
> # btrfs check -p /dev/sdd2
> Checking filesystem on /dev/sdd2
> UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
> parent transid verify failed on 3829276291072 wanted 224274 found 283858
> parent transid verify failed on 3829276291072 wanted 224274 found 283858
> parent transid verify failed on 3829276291072 wanted 224274 found 283858
> parent transid verify failed on 3829276291072 wanted 224274 found 283858
Normal transid error, can't say much about if it's harmless, but at
least some thing went wrong.
> Ignoring transid failure
> leaf parent key incorrect 3829276291072
> bad block 3829276291072
That's some what a big problem for that tree block.
If this tree block is extent tree block, no wonder why kernel output
kernel warning and abort transaction.
You could try "btrfs-debug-tree -b 3829276291072 <device>" to show the
content of the tree block.
If it's an extent tree block, then I'm afraid that's the problem.
Not sure if repair can repair such problem, but at least from what I see
in btrfs-progs fsck self testcases, it doesn't handle extent tree error
well.
>
> ERROR: errors found in extent allocation tree or chunk allocation
> block group 4722282987520 has wrong amount of free space
> failed to load free space cache for block group 4722282987520
> checking free space cache [O]
> root 32018 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32089 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32091 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32092 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32107 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32189 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32190 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32191 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32265 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32266 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32409 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32410 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32411 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32412 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32413 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32631 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32632 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32633 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32634 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32635 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32636 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32718 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
> root 32732 inode 95066 errors 100, file extent discount
> Found file extent holes:
> start: 413696, len: 4096
File extent holes are completely fine, one of the few problems we can
fix safely in btrfs-check.
But previous extent tree one is not.
If lowmem mode also reports the same problems only (file extent discount
+ extent tree error), then there is a chance that --init-extent-tree may
help.
But it will be super time consuming though.
Thanks,
Qu
> checking fs roots [o]
>
> I know that the "file extend discount" errors are fixable from my previous
> email to this ML, but what about the rest? From looking through the ML
> archives it seems that --repair won't be able to fix the transid failures. It
> seems that one person had success with the "usebackuproot" mount option, which
> I haven't tried yet.
>
>> System is kernel 4.7.2, Gentoo Linux, latest VirtualBox stable.
>> VirtualBox was using VDI image format without nocow. I now reverted
>> back to using nocow on VDI files and hope it doesn't strike again too
>> soon. I didn't try again yet, first I need to refresh my backup which
>> takes a while.
>>
>> The filesystem runs on 3x SATA 1TB mraid1 draid0 through bcache in
>> writeback mode, backed by a 500GB 850 Evo - if that matters.
>>
>> The problem occurred during high IO on 4.7.2. I previously ran 4.6.6
>> which didn't show this problem. Part of the culprit may be that I was
>> using bfq patches - I removed them for now and went back to deadline io
>> scheduler. The bfq patches froze my system a few times when I booted
>> 4.7.2 which may already have broken my btrfs (although it shouldn't,
>> right? btrfs is transactional). Last time this happened (on an earlier
>> kernel), bfq may have been part of the problem, too. So I think bfq
>> does something to btrfs which may break the fs, or at least interferes
>> badly with the transaction as otherwise it shouldn't break. You may
>> want to run your test suites with bfq also (or different io schedulers
>> in general).
>>
>> My home partition is mounted as a subvolume:
>> /dev/bcache0 on /home type btrfs
>> (rw,noatime,compress=lzo,nossd,space_cache,autodefrag,subvolid=261,subvol=/
>> home)
>
> The system the drive runs on is:
>
> % uname -a
> Linux diefledermaus 4.9.7-gentoo #1 SMP Wed Feb 1 23:52:56 CET 2017 x86_64
> Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz GenuineIntel GNU/Linux
>
> However during the crash it was running 4.9.6-gentoo. The system uses the
> standard CFQ scheduler, so perhaps BFQ is not at fault in Kai's case.
>
> The system I am running btrfs-check on is:
>
> % uname -a
> Linux thetick 4.9.6-gentoo #1 SMP PREEMPT Fri Jan 27 00:50:02 CET 2017 x86_64
> AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ AuthenticAMD GNU/Linux
>
> Both have btrfs-progs 4.9:
>
> % /sbin/btrfs --version
> btrfs-progs v4.9
>
> And the file system in question:
>
> % sudo /sbin/btrfs fi show /dev/sdd2
> Label: 'MARCEC_BACKUP' uuid: f97b3cda-15e8-418b-bb9b-235391ef2a38
> Total devices 1 FS bytes used 842.50GiB
> devid 1 size 976.56GiB used 877.31GiB path /dev/sdd2
>
> The file system is mounted with "noatime,compress,comment=systemd.automount".
>
> In my case the crash also happened during high I/O load (three btrfs-
> send/receive backups running at the same time). If "usebackuproot" (now
> called "recovery"?) fails, then I'll just wipe the FS and start the backups
> from scratch.
>
> Since I would like to have that done by Saturday: is there any information I
> can provide that might help fix whatever bug(s) caused this? Should I file a
> bug if one doesn't exist yet (I haven't checked yet, sorry)?
>
> Greetings
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-02 0:43 ` Qu Wenruo
@ 2017-03-02 9:43 ` Marc Joliet
2017-03-03 1:00 ` Qu Wenruo
0 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-03-02 9:43 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 13045 bytes --]
On Thursday 02 March 2017 08:43:53 Qu Wenruo wrote:
> At 02/02/2017 08:01 PM, Marc Joliet wrote:
> > On Sunday 28 August 2016 15:29:08 Kai Krakow wrote:
> >> Hello list!
> >
> > Hi list
>
> [kernel message snipped]
>
> >> Btrfs --repair refused to repair the filesystem telling me something
> >> about compressed extents and an unsupported case, wanting me to take an
> >> image and send it to the devs. *sigh*
> >
> > I haven't tried a repair yet; it's a big file system, and btrfs-check is
> > still running:
> >
> > # btrfs check -p /dev/sdd2
> > Checking filesystem on /dev/sdd2
> > UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
> > parent transid verify failed on 3829276291072 wanted 224274 found 283858
> > parent transid verify failed on 3829276291072 wanted 224274 found 283858
> > parent transid verify failed on 3829276291072 wanted 224274 found 283858
> > parent transid verify failed on 3829276291072 wanted 224274 found 283858
>
> Normal transid error, can't say much about if it's harmless, but at
> least some thing went wrong.
>
> > Ignoring transid failure
> > leaf parent key incorrect 3829276291072
> > bad block 3829276291072
>
> That's some what a big problem for that tree block.
>
> If this tree block is extent tree block, no wonder why kernel output
> kernel warning and abort transaction.
>
> You could try "btrfs-debug-tree -b 3829276291072 <device>" to show the
> content of the tree block.
# btrfs-debug-tree -b 3829276291072 /dev/sdb2
btrfs-progs v4.9
node 3829276291072 level 1 items 70 free 51 generation 292525 owner 2
fs uuid f97b3cda-15e8-418b-bb9b-235391ef2a38
chunk uuid 1cee580c-3442-4717-9300-8514dd8ff297
key (3828594696192 METADATA_ITEM 0) block 3828933423104 (934798199)
gen 292523
key (3828594925568 METADATA_ITEM 0) block 3829427818496 (934918901)
gen 292525
key (3828595109888 METADATA_ITEM 0) block 3828895723520 (934788995)
gen 292523
key (3828595232768 METADATA_ITEM 0) block 3829202751488 (934863953)
gen 292524
key (3828595412992 METADATA_ITEM 0) block 3829097209856 (934838186)
gen 292523
key (3828595572736 TREE_BLOCK_REF 33178) block 3829235073024
(934871844) gen 292524
key (3828595744768 METADATA_ITEM 0) block 3829128351744 (934845789)
gen 292524
key (3828595982336 METADATA_ITEM 0) block 3829146484736 (934850216)
gen 292524
key (3828596187136 METADATA_ITEM 1) block 3829097234432 (934838192)
gen 292523
key (3828596387840 TREE_BLOCK_REF 33527) block 3829301653504
(934888099) gen 292525
key (3828596617216 METADATA_ITEM 0) block 3828885737472 (934786557)
gen 292523
key (3828596838400 METADATA_ITEM 0) block 3828885741568 (934786558)
gen 292523
key (3828597047296 METADATA_ITEM 0) block 3829320552448 (934892713)
gen 292525
key (3828597231616 METADATA_ITEM 0) block 3828945653760 (934801185)
gen 292523
key (3828597383168 METADATA_ITEM 0) block 3829276299264 (934881909)
gen 292525
key (3828597641216 METADATA_ITEM 1) block 3829349351424 (934899744)
gen 292525
key (3828597866496 METADATA_ITEM 0) block 3829364776960 (934903510)
gen 292525
key (3828598067200 METADATA_ITEM 0) block 3828598321152 (934716387)
gen 292522
key (3828598259712 METADATA_ITEM 0) block 3829422968832 (934917717)
gen 292525
key (3828598415360 TREE_BLOCK_REF 33252) block 3828885803008
(934786573) gen 292523
key (3828598665216 METADATA_ITEM 0) block 3828937863168 (934799283)
gen 292523
key (3828598829056 METADATA_ITEM 0) block 3828885811200 (934786575)
gen 292523
key (3828599054336 METADATA_ITEM 0) block 3829363744768 (934903258)
gen 292525
key (3828599246848 METADATA_ITEM 0) block 3828915838976 (934793906)
gen 292523
key (3828599504896 METADATA_ITEM 0) block 3829436194816 (934920946)
gen 292525
key (3828599672832 METADATA_ITEM 0) block 3828905140224 (934791294)
gen 292523
key (3828599771136 METADATA_ITEM 0) block 3829333323776 (934895831)
gen 292525
key (3828599988224 METADATA_ITEM 0) block 3829087199232 (934835742)
gen 292523
key (3828600135680 METADATA_ITEM 0) block 3828885827584 (934786579)
gen 292523
key (3828600389632 METADATA_ITEM 0) block 3829436284928 (934920968)
gen 292525
key (3828600528896 METADATA_ITEM 0) block 3829316214784 (934891654)
gen 292525
key (3828600729600 METADATA_ITEM 0) block 3828885905408 (934786598)
gen 292523
key (3828600934400 METADATA_ITEM 0) block 3829384486912 (934908322)
gen 292525
key (3828601143296 METADATA_ITEM 0) block 3829423611904 (934917874)
gen 292525
key (3828601356288 METADATA_ITEM 0) block 3829113688064 (934842209)
gen 292524
key (3828601556992 METADATA_ITEM 0) block 3829134540800 (934847300)
gen 292524
key (3828601696256 METADATA_ITEM 0) block 3829181837312 (934858847)
gen 292524
key (3828601823232 METADATA_ITEM 0) block 3829157421056 (934852886)
gen 292524
key (3828602015744 TREE_BLOCK_REF 32943) block 3829316218880
(934891655) gen 292525
key (3828602245120 METADATA_ITEM 0) block 3829207601152 (934865137)
gen 292524
key (3828602384384 METADATA_ITEM 0) block 3828933246976 (934798156)
gen 292523
key (3828602531840 METADATA_ITEM 0) block 3829346013184 (934898929)
gen 292525
key (3828602740736 METADATA_ITEM 0) block 3828916629504 (934794099)
gen 292523
key (3828602990592 METADATA_ITEM 0) block 3829160525824 (934853644)
gen 292524
key (3828603097088 METADATA_ITEM 0) block 3828933267456 (934798161)
gen 292523
key (3828603195392 METADATA_ITEM 0) block 4632523673600 (1130987225)
gen 292474
key (3828603293696 TREE_BLOCK_REF 32827) block 3829160529920
(934853645) gen 292524
key (3828603453440 METADATA_ITEM 0) block 3828886011904 (934786624)
gen 292523
key (3828603711488 METADATA_ITEM 0) block 3829299806208 (934887648)
gen 292525
key (3828603940864 METADATA_ITEM 0) block 3829325082624 (934893819)
gen 292525
key (3828604145664 METADATA_ITEM 0) block 3828886040576 (934786631)
gen 292523
key (3828604321792 METADATA_ITEM 0) block 3829207818240 (934865190)
gen 292524
key (3828604538880 METADATA_ITEM 0) block 3829347282944 (934899239)
gen 292525
key (3828604796928 METADATA_ITEM 1) block 3829304320000 (934888750)
gen 292525
key (3828605042688 METADATA_ITEM 0) block 3829159145472 (934853307)
gen 292524
key (3828605239296 METADATA_ITEM 0) block 3829170442240 (934856065)
gen 292524
key (3828605505536 METADATA_ITEM 0) block 3829360103424 (934902369)
gen 292525
key (3828605698048 METADATA_ITEM 0) block 3829362032640 (934902840)
gen 292525
key (3828605915136 METADATA_ITEM 0) block 3829336616960 (934896635)
gen 292525
key (3828606103552 METADATA_ITEM 0) block 3829360107520 (934902370)
gen 292525
key (3828606291968 METADATA_ITEM 0) block 3829210537984 (934865854)
gen 292524
key (3828606504960 METADATA_ITEM 0) block 3829406662656 (934913736)
gen 292525
key (3828606754816 METADATA_ITEM 0) block 3829345562624 (934898819)
gen 292525
key (3828606943232 METADATA_ITEM 0) block 3829342674944 (934898114)
gen 292525
key (3828607156224 METADATA_ITEM 0) block 3829444628480 (934923005)
gen 292525
key (3828607324160 METADATA_ITEM 0) block 3829087207424 (934835744)
gen 292523
key (3828607561728 METADATA_ITEM 0) block 3829313220608 (934890923)
gen 292525
key (3828607680512 METADATA_ITEM 0) block 3829370867712 (934904997)
gen 292525
key (3828607930368 METADATA_ITEM 0) block 3829411094528 (934914818)
gen 292525
key (3828608180224 METADATA_ITEM 0) block 3829299822592 (934887652)
gen 292525
I wonder, though, whether that is still the same block or whether it has been
replaced during the past two weeks of FS usage.
> If it's an extent tree block, then I'm afraid that's the problem.
> Not sure if repair can repair such problem, but at least from what I see
> in btrfs-progs fsck self testcases, it doesn't handle extent tree error
> well.
OK
> > ERROR: errors found in extent allocation tree or chunk allocation
> > block group 4722282987520 has wrong amount of free space
> > failed to load free space cache for block group 4722282987520
> > checking free space cache [O]
> > root 32018 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32089 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32091 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32092 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32107 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32189 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32190 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32191 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32265 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32266 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32409 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32410 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32411 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32412 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32413 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32631 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32632 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32633 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32634 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32635 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32636 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32718 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
> >
> > root 32732 inode 95066 errors 100, file extent discount
> >
> > Found file extent holes:
> > start: 413696, len: 4096
>
> File extent holes are completely fine, one of the few problems we can
> fix safely in btrfs-check.
Yeah, I found that out myself by searching my btrfs ML archive.
> But previous extent tree one is not.
OK
> If lowmem mode also reports the same problems only (file extent discount
> + extent tree error), then there is a chance that --init-extent-tree may
> help.
>
> But it will be super time consuming though.
Alright, in that case, I'll just reformat the partition and start my backups
over from scratch once btrfs-check is done running (I'll post the current
output in another Email).
FWIW, as per my later messages, after mounting with clear_cache and letting
btrfs-cleaner finish, btrfs-check did *not* print out those errors after
running again. It's now about two weeks later that the file system is showing
problems again.
Oh, and just in case it's relevant, the file system was created with btrfs-
convert (a long time, maybe 1.5 years ago, though; it was originally ext4).
[...]
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-01 18:14 ` Marc Joliet
2017-03-01 18:27 ` Marc Joliet
2017-03-01 18:43 ` Marc Joliet
@ 2017-03-02 9:44 ` Marc Joliet
2017-03-03 1:09 ` Qu Wenruo
2017-03-05 23:53 ` Marc Joliet
3 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-03-02 9:44 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 5809 bytes --]
On Wednesday 01 March 2017 19:14:07 Marc Joliet wrote:
> In any
> case, I started btrfs-check on the device itself.
OK, it's still running, but the output so far is:
# btrfs check --mode=lowmem --progress /dev/sdb2
Checking filesystem on /dev/sdb2
UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
ERROR: shared extent[3826242740224 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3826442825728 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3826744471552 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827106349056 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827141001216 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827150958592 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827251724288 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827433795584 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827536166912 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827536183296 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3827621646336 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3828179406848 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3828267970560 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3828284530688 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3828714246144 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3828794187776 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3829161340928 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3829373693952 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3830252130304 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3830421159936 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3830439141376 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3830441398272 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3830785138688 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831099297792 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831128768512 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831371513856 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831535570944 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831591952384 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831799398400 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831829250048 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3831829512192 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832011440128 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832011767808 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832023920640 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832024678400 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832027316224 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832028762112 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832030236672 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832030330880 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832161079296 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832164904960 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832164945920 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3832613765120 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3833727565824 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3833914073088 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3833929310208 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: shared extent[3833930141696 4096] lost its parent (parent:
3827251183616, level: 0)
ERROR: extent[3837768077312, 24576] referencer count mismatch (root: 33174,
owner: 1277577, offset: 4767744) wanted: 1, have: 0
[snip many more referencer count mismatches]
ERROR: extent[3878247383040, 8192] referencer count mismatch (root: 33495,
owner: 2688918, offset: 3874816) wanted: 2, have: 3
ERROR: block group[3879328546816 1073741824] used 1072840704 but extent items
used 1129164800
ERROR: extent[3879516581888, 40960] referencer count mismatch (root: 33530,
owner: 1277577, offset: 23461888) wanted: 1, have: 0
[snip many more referencer count mismatches]
ERROR: extent[3951758860288, 16384] referencer count mismatch (root: 33189,
owner: 1277577, offset: 519880704) wanted: 1, have: 0
This is 56 lines of output, the unabridged output contains enough "referencer
count mismatches" to fill 1329 lines of output. Could that be because there
are deleted snapshots that btrfs-cleaner hasn't been able to clean up?
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-02 9:43 ` Marc Joliet
@ 2017-03-03 1:00 ` Qu Wenruo
2017-03-03 11:54 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Qu Wenruo @ 2017-03-03 1:00 UTC (permalink / raw)
To: Marc Joliet, linux-btrfs
At 03/02/2017 05:43 PM, Marc Joliet wrote:
> On Thursday 02 March 2017 08:43:53 Qu Wenruo wrote:
>> At 02/02/2017 08:01 PM, Marc Joliet wrote:
>>> On Sunday 28 August 2016 15:29:08 Kai Krakow wrote:
>>>> Hello list!
>>>
>>> Hi list
>>
>> [kernel message snipped]
>>
>>>> Btrfs --repair refused to repair the filesystem telling me something
>>>> about compressed extents and an unsupported case, wanting me to take an
>>>> image and send it to the devs. *sigh*
>>>
>>> I haven't tried a repair yet; it's a big file system, and btrfs-check is
>>> still running:
>>>
>>> # btrfs check -p /dev/sdd2
>>> Checking filesystem on /dev/sdd2
>>> UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
>>> parent transid verify failed on 3829276291072 wanted 224274 found 283858
>>> parent transid verify failed on 3829276291072 wanted 224274 found 283858
>>> parent transid verify failed on 3829276291072 wanted 224274 found 283858
>>> parent transid verify failed on 3829276291072 wanted 224274 found 283858
>>
>> Normal transid error, can't say much about if it's harmless, but at
>> least some thing went wrong.
>>
>>> Ignoring transid failure
>>> leaf parent key incorrect 3829276291072
>>> bad block 3829276291072
>>
>> That's some what a big problem for that tree block.
>>
>> If this tree block is extent tree block, no wonder why kernel output
>> kernel warning and abort transaction.
>>
>> You could try "btrfs-debug-tree -b 3829276291072 <device>" to show the
>> content of the tree block.
>
> # btrfs-debug-tree -b 3829276291072 /dev/sdb2
> btrfs-progs v4.9
> node 3829276291072 level 1 items 70 free 51 generation 292525 owner 2
> fs uuid f97b3cda-15e8-418b-bb9b-235391ef2a38
> chunk uuid 1cee580c-3442-4717-9300-8514dd8ff297
> key (3828594696192 METADATA_ITEM 0) block 3828933423104 (934798199)
> gen 292523
> key (3828594925568 METADATA_ITEM 0) block 3829427818496 (934918901)
> gen 292525
> key (3828595109888 METADATA_ITEM 0) block 3828895723520 (934788995)
> gen 292523
> key (3828595232768 METADATA_ITEM 0) block 3829202751488 (934863953)
> gen 292524
> key (3828595412992 METADATA_ITEM 0) block 3829097209856 (934838186)
> gen 292523
> key (3828595572736 TREE_BLOCK_REF 33178) block 3829235073024
> (934871844) gen 292524
> key (3828595744768 METADATA_ITEM 0) block 3829128351744 (934845789)
> gen 292524
> key (3828595982336 METADATA_ITEM 0) block 3829146484736 (934850216)
> gen 292524
> key (3828596187136 METADATA_ITEM 1) block 3829097234432 (934838192)
> gen 292523
> key (3828596387840 TREE_BLOCK_REF 33527) block 3829301653504
> (934888099) gen 292525
> key (3828596617216 METADATA_ITEM 0) block 3828885737472 (934786557)
> gen 292523
> key (3828596838400 METADATA_ITEM 0) block 3828885741568 (934786558)
> gen 292523
> key (3828597047296 METADATA_ITEM 0) block 3829320552448 (934892713)
> gen 292525
> key (3828597231616 METADATA_ITEM 0) block 3828945653760 (934801185)
> gen 292523
> key (3828597383168 METADATA_ITEM 0) block 3829276299264 (934881909)
> gen 292525
> key (3828597641216 METADATA_ITEM 1) block 3829349351424 (934899744)
> gen 292525
> key (3828597866496 METADATA_ITEM 0) block 3829364776960 (934903510)
> gen 292525
> key (3828598067200 METADATA_ITEM 0) block 3828598321152 (934716387)
> gen 292522
> key (3828598259712 METADATA_ITEM 0) block 3829422968832 (934917717)
> gen 292525
> key (3828598415360 TREE_BLOCK_REF 33252) block 3828885803008
> (934786573) gen 292523
> key (3828598665216 METADATA_ITEM 0) block 3828937863168 (934799283)
> gen 292523
> key (3828598829056 METADATA_ITEM 0) block 3828885811200 (934786575)
> gen 292523
> key (3828599054336 METADATA_ITEM 0) block 3829363744768 (934903258)
> gen 292525
> key (3828599246848 METADATA_ITEM 0) block 3828915838976 (934793906)
> gen 292523
> key (3828599504896 METADATA_ITEM 0) block 3829436194816 (934920946)
> gen 292525
> key (3828599672832 METADATA_ITEM 0) block 3828905140224 (934791294)
> gen 292523
> key (3828599771136 METADATA_ITEM 0) block 3829333323776 (934895831)
> gen 292525
> key (3828599988224 METADATA_ITEM 0) block 3829087199232 (934835742)
> gen 292523
> key (3828600135680 METADATA_ITEM 0) block 3828885827584 (934786579)
> gen 292523
> key (3828600389632 METADATA_ITEM 0) block 3829436284928 (934920968)
> gen 292525
> key (3828600528896 METADATA_ITEM 0) block 3829316214784 (934891654)
> gen 292525
> key (3828600729600 METADATA_ITEM 0) block 3828885905408 (934786598)
> gen 292523
> key (3828600934400 METADATA_ITEM 0) block 3829384486912 (934908322)
> gen 292525
> key (3828601143296 METADATA_ITEM 0) block 3829423611904 (934917874)
> gen 292525
> key (3828601356288 METADATA_ITEM 0) block 3829113688064 (934842209)
> gen 292524
> key (3828601556992 METADATA_ITEM 0) block 3829134540800 (934847300)
> gen 292524
> key (3828601696256 METADATA_ITEM 0) block 3829181837312 (934858847)
> gen 292524
> key (3828601823232 METADATA_ITEM 0) block 3829157421056 (934852886)
> gen 292524
> key (3828602015744 TREE_BLOCK_REF 32943) block 3829316218880
> (934891655) gen 292525
> key (3828602245120 METADATA_ITEM 0) block 3829207601152 (934865137)
> gen 292524
> key (3828602384384 METADATA_ITEM 0) block 3828933246976 (934798156)
> gen 292523
> key (3828602531840 METADATA_ITEM 0) block 3829346013184 (934898929)
> gen 292525
> key (3828602740736 METADATA_ITEM 0) block 3828916629504 (934794099)
> gen 292523
> key (3828602990592 METADATA_ITEM 0) block 3829160525824 (934853644)
> gen 292524
> key (3828603097088 METADATA_ITEM 0) block 3828933267456 (934798161)
> gen 292523
> key (3828603195392 METADATA_ITEM 0) block 4632523673600 (1130987225)
> gen 292474
> key (3828603293696 TREE_BLOCK_REF 32827) block 3829160529920
> (934853645) gen 292524
> key (3828603453440 METADATA_ITEM 0) block 3828886011904 (934786624)
> gen 292523
> key (3828603711488 METADATA_ITEM 0) block 3829299806208 (934887648)
> gen 292525
> key (3828603940864 METADATA_ITEM 0) block 3829325082624 (934893819)
> gen 292525
> key (3828604145664 METADATA_ITEM 0) block 3828886040576 (934786631)
> gen 292523
> key (3828604321792 METADATA_ITEM 0) block 3829207818240 (934865190)
> gen 292524
> key (3828604538880 METADATA_ITEM 0) block 3829347282944 (934899239)
> gen 292525
> key (3828604796928 METADATA_ITEM 1) block 3829304320000 (934888750)
> gen 292525
> key (3828605042688 METADATA_ITEM 0) block 3829159145472 (934853307)
> gen 292524
> key (3828605239296 METADATA_ITEM 0) block 3829170442240 (934856065)
> gen 292524
> key (3828605505536 METADATA_ITEM 0) block 3829360103424 (934902369)
> gen 292525
> key (3828605698048 METADATA_ITEM 0) block 3829362032640 (934902840)
> gen 292525
> key (3828605915136 METADATA_ITEM 0) block 3829336616960 (934896635)
> gen 292525
> key (3828606103552 METADATA_ITEM 0) block 3829360107520 (934902370)
> gen 292525
> key (3828606291968 METADATA_ITEM 0) block 3829210537984 (934865854)
> gen 292524
> key (3828606504960 METADATA_ITEM 0) block 3829406662656 (934913736)
> gen 292525
> key (3828606754816 METADATA_ITEM 0) block 3829345562624 (934898819)
> gen 292525
> key (3828606943232 METADATA_ITEM 0) block 3829342674944 (934898114)
> gen 292525
> key (3828607156224 METADATA_ITEM 0) block 3829444628480 (934923005)
> gen 292525
> key (3828607324160 METADATA_ITEM 0) block 3829087207424 (934835744)
> gen 292523
> key (3828607561728 METADATA_ITEM 0) block 3829313220608 (934890923)
> gen 292525
> key (3828607680512 METADATA_ITEM 0) block 3829370867712 (934904997)
> gen 292525
> key (3828607930368 METADATA_ITEM 0) block 3829411094528 (934914818)
> gen 292525
> key (3828608180224 METADATA_ITEM 0) block 3829299822592 (934887652)
> gen 292525
>
> I wonder, though, whether that is still the same block or whether it has been
> replaced during the past two weeks of FS usage.
It's a tree block of extent tree.
So the problem is all about extent tree now.
>
>> If it's an extent tree block, then I'm afraid that's the problem.
>> Not sure if repair can repair such problem, but at least from what I see
>> in btrfs-progs fsck self testcases, it doesn't handle extent tree error
>> well.
>
> OK
>
>>> ERROR: errors found in extent allocation tree or chunk allocation
>>> block group 4722282987520 has wrong amount of free space
>>> failed to load free space cache for block group 4722282987520
>>> checking free space cache [O]
>>> root 32018 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32089 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32091 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32092 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32107 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32189 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32190 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32191 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32265 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32266 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32409 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32410 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32411 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32412 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32413 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32631 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32632 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32633 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32634 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32635 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32636 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32718 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>>
>>> root 32732 inode 95066 errors 100, file extent discount
>>>
>>> Found file extent holes:
>>> start: 413696, len: 4096
>>
>> File extent holes are completely fine, one of the few problems we can
>> fix safely in btrfs-check.
>
> Yeah, I found that out myself by searching my btrfs ML archive.
>
>> But previous extent tree one is not.
>
> OK
>
>> If lowmem mode also reports the same problems only (file extent discount
>> + extent tree error), then there is a chance that --init-extent-tree may
>> help.
>>
>> But it will be super time consuming though.
>
> Alright, in that case, I'll just reformat the partition and start my backups
> over from scratch once btrfs-check is done running (I'll post the current
> output in another Email).
>
> FWIW, as per my later messages, after mounting with clear_cache and letting
> btrfs-cleaner finish, btrfs-check did *not* print out those errors after
> running again. It's now about two weeks later that the file system is showing
> problems again.
If btrfs-check didn't print out *any* error, then it should be mostly
fine. (Unless there is some case we don't expose yet)
The problem should be caused by kernel AFAIK.
>
> Oh, and just in case it's relevant, the file system was created with btrfs-
> convert (a long time, maybe 1.5 years ago, though; it was originally ext4).
Not sure if it's related.
But at least for that old convert, it's chunk layout is somewhat rare
and sometimes even bug-prone.
Did you balance the btrfs after convert? If so, it should be more like a
traditional btrfs then.
Personally speaking I don't think it is relative for your bug, but much
like a normal extent tree corruption seen in mail list.
Thanks,
Qu
>
> [...]
>
> Greetings
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-02 9:44 ` Marc Joliet
@ 2017-03-03 1:09 ` Qu Wenruo
2017-03-03 11:26 ` Marc Joliet
0 siblings, 1 reply; 23+ messages in thread
From: Qu Wenruo @ 2017-03-03 1:09 UTC (permalink / raw)
To: Marc Joliet, linux-btrfs
At 03/02/2017 05:44 PM, Marc Joliet wrote:
> On Wednesday 01 March 2017 19:14:07 Marc Joliet wrote:
>> In any
>> case, I started btrfs-check on the device itself.
>
> OK, it's still running, but the output so far is:
>
> # btrfs check --mode=lowmem --progress /dev/sdb2
> Checking filesystem on /dev/sdb2
> UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
> ERROR: shared extent[3826242740224 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3826442825728 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3826744471552 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827106349056 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827141001216 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827150958592 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827251724288 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827433795584 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827536166912 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827536183296 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3827621646336 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3828179406848 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3828267970560 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3828284530688 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3828714246144 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3828794187776 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3829161340928 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3829373693952 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3830252130304 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3830421159936 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3830439141376 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3830441398272 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3830785138688 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831099297792 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831128768512 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831371513856 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831535570944 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831591952384 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831799398400 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831829250048 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3831829512192 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832011440128 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832011767808 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832023920640 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832024678400 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832027316224 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832028762112 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832030236672 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832030330880 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832161079296 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832164904960 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832164945920 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3832613765120 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3833727565824 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3833914073088 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3833929310208 4096] lost its parent (parent:
> 3827251183616, level: 0)
> ERROR: shared extent[3833930141696 4096] lost its parent (parent:
> 3827251183616, level: 0)
The "shared extent lost its parent" is all about the same extent,
3827251183616.
It would be nice if you could paste the output of btrfs-debug-tree -b
3827251183616 to check what tree it belongs to.
> ERROR: extent[3837768077312, 24576] referencer count mismatch (root: 33174,
> owner: 1277577, offset: 4767744) wanted: 1, have: 0
> [snip many more referencer count mismatches]
> ERROR: extent[3878247383040, 8192] referencer count mismatch (root: 33495,
> owner: 2688918, offset: 3874816) wanted: 2, have: 3
That's a little dangerous now.
Some *DATA* extent even lost its referencer now.
For example, data extent at 3837768077312, length 24K exists on disk,
but no one is referring to it.
At least the good news is, these just mean there is some extra space
taken on disk, but not vice verse.
> ERROR: block group[3879328546816 1073741824] used 1072840704 but extent items
> used 1129164800
> ERROR: extent[3879516581888, 40960] referencer count mismatch (root: 33530,
> owner: 1277577, offset: 23461888) wanted: 1, have: 0
> [snip many more referencer count mismatches]
> ERROR: extent[3951758860288, 16384] referencer count mismatch (root: 33189,
> owner: 1277577, offset: 519880704) wanted: 1, have: 0
>
> This is 56 lines of output, the unabridged output contains enough "referencer
> count mismatches" to fill 1329 lines of output. Could that be because there
> are deleted snapshots that btrfs-cleaner hasn't been able to clean up?
Ah, this makes sense.
But normally cleaner should cleanup subvolume tree block and modify
extent tree.
How did you stop btrfs-cleaner?
Thanks,
Qu
>
> Greetings
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-03 1:09 ` Qu Wenruo
@ 2017-03-03 11:26 ` Marc Joliet
0 siblings, 0 replies; 23+ messages in thread
From: Marc Joliet @ 2017-03-03 11:26 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 14339 bytes --]
On Friday 03 March 2017 09:09:57 Qu Wenruo wrote:
> At 03/02/2017 05:44 PM, Marc Joliet wrote:
> > On Wednesday 01 March 2017 19:14:07 Marc Joliet wrote:
> >> In any
> >> case, I started btrfs-check on the device itself.
> >
> > OK, it's still running, but the output so far is:
> >
> > # btrfs check --mode=lowmem --progress /dev/sdb2
> > Checking filesystem on /dev/sdb2
> > UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
> > ERROR: shared extent[3826242740224 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3826442825728 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3826744471552 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827106349056 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827141001216 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827150958592 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827251724288 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827433795584 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827536166912 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827536183296 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3827621646336 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3828179406848 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3828267970560 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3828284530688 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3828714246144 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3828794187776 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3829161340928 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3829373693952 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3830252130304 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3830421159936 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3830439141376 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3830441398272 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3830785138688 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831099297792 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831128768512 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831371513856 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831535570944 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831591952384 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831799398400 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831829250048 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3831829512192 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832011440128 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832011767808 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832023920640 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832024678400 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832027316224 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832028762112 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832030236672 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832030330880 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832161079296 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832164904960 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832164945920 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3832613765120 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3833727565824 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3833914073088 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3833929310208 4096] lost its parent (parent:
> > 3827251183616, level: 0)
> > ERROR: shared extent[3833930141696 4096] lost its parent (parent:
> > 3827251183616, level: 0)
>
> The "shared extent lost its parent" is all about the same extent,
> 3827251183616.
>
> It would be nice if you could paste the output of btrfs-debug-tree -b
> 3827251183616 to check what tree it belongs to.
# btrfs-debug-tree -b 3827251183616 /dev/sdb2
btrfs-progs v4.9
leaf 3827251183616 items 33 free space 1712 generation 291223 owner 2
fs uuid f97b3cda-15e8-418b-bb9b-235391ef2a38
chunk uuid 1cee580c-3442-4717-9300-8514dd8ff297
item 0 key (3988449177600 EXTENT_ITEM 8192) itemoff 3942 itemsize 53
extent refs 1 gen 271366 flags DATA
extent data backref root 32126 objectid 1921863 offset
302157824 count 1
item 1 key (3988449185792 EXTENT_ITEM 4096) itemoff 3889 itemsize 53
extent refs 1 gen 271366 flags DATA
extent data backref root 32126 objectid 1921863 offset
302534656 count 1
item 2 key (3988449189888 EXTENT_ITEM 4096) itemoff 3852 itemsize 37
extent refs 1 gen 244831 flags DATA
shared data backref parent 4691081252864 count 1
item 3 key (3988449193984 EXTENT_ITEM 8192) itemoff 3815 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644000256 count 1
item 4 key (3988449202176 EXTENT_ITEM 4096) itemoff 3778 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644000256 count 1
item 5 key (3988449206272 EXTENT_ITEM 4096) itemoff 3741 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644000256 count 1
item 6 key (3988449210368 EXTENT_ITEM 4096) itemoff 3688 itemsize 53
extent refs 1 gen 288553 flags DATA
extent data backref root 33295 objectid 1921863 offset
233136128 count 1
item 7 key (3988449214464 EXTENT_ITEM 8192) itemoff 3635 itemsize 53
extent refs 1 gen 283033 flags DATA
extent data backref root 32994 objectid 1921863 offset
694497280 count 1
item 8 key (3988449222656 EXTENT_ITEM 4096) itemoff 3598 itemsize 37
extent refs 1 gen 279261 flags DATA
shared data backref parent 3829567111168 count 1
item 9 key (3988449226752 EXTENT_ITEM 12288) itemoff 3545 itemsize 53
extent refs 1 gen 283033 flags DATA
extent data backref root 32994 objectid 1921863 offset
694648832 count 1
item 10 key (3988449239040 EXTENT_ITEM 4096) itemoff 3508 itemsize 37
extent refs 1 gen 244831 flags DATA
shared data backref parent 4691081252864 count 1
item 11 key (3988449243136 EXTENT_ITEM 16384) itemoff 3426 itemsize 82
extent refs 3 gen 287487 flags DATA
extent data backref root 33204 objectid 214635 offset 389120
count 1
extent data backref root 33205 objectid 214635 offset 389120
count 2
item 12 key (3988449259520 EXTENT_ITEM 4096) itemoff 3389 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644000256 count 1
item 13 key (3988449263616 EXTENT_ITEM 4096) itemoff 3352 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644016640 count 1
item 14 key (3988449267712 EXTENT_ITEM 4096) itemoff 3315 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644016640 count 1
item 15 key (3988449271808 EXTENT_ITEM 4096) itemoff 3278 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644045312 count 1
item 16 key (3988449275904 EXTENT_ITEM 8192) itemoff 3225 itemsize 53
extent refs 1 gen 291191 flags DATA
extent data backref root 33530 objectid 1277577 offset
392007680 count 1
item 17 key (3988449284096 EXTENT_ITEM 8192) itemoff 3188 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644016640 count 1
item 18 key (3988449292288 EXTENT_ITEM 8192) itemoff 3135 itemsize 53
extent refs 1 gen 287570 flags DATA
extent data backref root 33214 objectid 211018 offset 131072
count 1
item 19 key (3988449300480 EXTENT_ITEM 8192) itemoff 3098 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644016640 count 1
item 20 key (3988449308672 EXTENT_ITEM 4096) itemoff 3061 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644045312 count 1
item 21 key (3988449312768 EXTENT_ITEM 4096) itemoff 3024 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644045312 count 1
item 22 key (3988449316864 EXTENT_ITEM 8192) itemoff 2987 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644053504 count 1
item 23 key (3988449325056 EXTENT_ITEM 8192) itemoff 2934 itemsize 53
extent refs 1 gen 287571 flags DATA
extent data backref root 33214 objectid 212344 offset 131072
count 1
item 24 key (3988449333248 EXTENT_ITEM 4096) itemoff 2897 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644053504 count 1
item 25 key (3988449337344 EXTENT_ITEM 4096) itemoff 2860 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644053504 count 1
item 26 key (3988449341440 EXTENT_ITEM 20480) itemoff 2807 itemsize 53
extent refs 1 gen 291191 flags DATA
extent data backref root 33530 objectid 1277577 offset
391614464 count 1
item 27 key (3988449361920 EXTENT_ITEM 4096) itemoff 2754 itemsize 53
extent refs 1 gen 291191 flags DATA
extent data backref root 33530 objectid 1277577 offset
392032256 count 1
item 28 key (3988449366016 EXTENT_ITEM 8192) itemoff 2701 itemsize 53
extent refs 1 gen 288553 flags DATA
extent data backref root 33295 objectid 1921863 offset
233062400 count 1
item 29 key (3988449374208 EXTENT_ITEM 4096) itemoff 2664 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644053504 count 1
item 30 key (3988449378304 EXTENT_ITEM 4096) itemoff 2627 itemsize 37
extent refs 1 gen 154664 flags DATA
shared data backref parent 4607644053504 count 1
item 31 key (3988449382400 EXTENT_ITEM 8192) itemoff 2590 itemsize 37
extent refs 1 gen 154663 flags DATA
shared data backref parent 4607644053504 count 1
item 32 key (3988449390592 EXTENT_ITEM 8192) itemoff 2537 itemsize 53
extent refs 1 gen 271366 flags DATA
extent data backref root 32126 objectid 1921863 offset
302276608 count 1
> > ERROR: extent[3837768077312, 24576] referencer count mismatch (root:
> > 33174,
> > owner: 1277577, offset: 4767744) wanted: 1, have: 0
> > [snip many more referencer count mismatches]
> > ERROR: extent[3878247383040, 8192] referencer count mismatch (root: 33495,
> > owner: 2688918, offset: 3874816) wanted: 2, have: 3
>
> That's a little dangerous now.
>
> Some *DATA* extent even lost its referencer now.
>
> For example, data extent at 3837768077312, length 24K exists on disk,
> but no one is referring to it.
>
> At least the good news is, these just mean there is some extra space
> taken on disk, but not vice verse.
>
> > ERROR: block group[3879328546816 1073741824] used 1072840704 but extent
> > items used 1129164800
> > ERROR: extent[3879516581888, 40960] referencer count mismatch (root:
> > 33530,
> > owner: 1277577, offset: 23461888) wanted: 1, have: 0
> > [snip many more referencer count mismatches]
> > ERROR: extent[3951758860288, 16384] referencer count mismatch (root:
> > 33189,
> > owner: 1277577, offset: 519880704) wanted: 1, have: 0
> >
> > This is 56 lines of output, the unabridged output contains enough
> > "referencer count mismatches" to fill 1329 lines of output. Could that
> > be because there are deleted snapshots that btrfs-cleaner hasn't been
> > able to clean up?
> Ah, this makes sense.
>
> But normally cleaner should cleanup subvolume tree block and modify
> extent tree.
> How did you stop btrfs-cleaner?
See the log I posted on 28th February. The file system gets forced RO due to
"Transaction aborted (error -2)" errors.
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-03 1:00 ` Qu Wenruo
@ 2017-03-03 11:54 ` Marc Joliet
0 siblings, 0 replies; 23+ messages in thread
From: Marc Joliet @ 2017-03-03 11:54 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]
On Friday 03 March 2017 09:00:10 Qu Wenruo wrote:
> > FWIW, as per my later messages, after mounting with clear_cache and
> > letting
> > btrfs-cleaner finish, btrfs-check did *not* print out those errors after
> > running again. It's now about two weeks later that the file system is
> > showing problems again.
>
> If btrfs-check didn't print out *any* error, then it should be mostly
> fine. (Unless there is some case we don't expose yet)
>
> The problem should be caused by kernel AFAIK.
So you think it could be a regression in 4.9? Should I try 4.10? Or is it
more likely just an undiscovered bug?
> > Oh, and just in case it's relevant, the file system was created with
> > btrfs-
> > convert (a long time, maybe 1.5 years ago, though; it was originally
> > ext4).
>
> Not sure if it's related.
> But at least for that old convert, it's chunk layout is somewhat rare
> and sometimes even bug-prone.
>
> Did you balance the btrfs after convert? If so, it should be more like a
> traditional btrfs then.
Yes, I'm fairly certain I did that, as that is what the btrfs wiki recommends.
> Personally speaking I don't think it is relative for your bug, but much
> like a normal extent tree corruption seen in mail list.
OK, so is there anything else I can do?
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-01 18:14 ` Marc Joliet
` (2 preceding siblings ...)
2017-03-02 9:44 ` Marc Joliet
@ 2017-03-05 23:53 ` Marc Joliet
2017-03-06 11:18 ` Marc Joliet
3 siblings, 1 reply; 23+ messages in thread
From: Marc Joliet @ 2017-03-05 23:53 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2015 bytes --]
On Mittwoch, 1. März 2017 19:14:07 CET you wrote:
> In any
> case, I started btrfs-check on the device itself.
*Sigh*, I had to restart it, because I forgot to redirect to a file and quite
frankly wasn't expecting this flood of output, but here's a summary of the
output after about 2 days:
% wc -l btrfs_check_output_20170303.log
1360252 btrfs_check_output_20170303.log
% grep -v "backref lost" btrfs_check_output_20170303.log | grep -v "check \
(leaf\|node\) failed" | grep -v "lost its parent" | grep -v "referencer count"
checking extents
ERROR: block group[3879328546816 1073741824] used 1072840704 but extent items
used 1129164800
ERROR: block group[4163870130176 1073741824] used 1072259072 but extent items
used 0
ERROR: block group[4223999672320 1073741824] used 1073664000 but extent items
used 1074188288
ERROR: block group[4278760505344 1073741824] used 1073377280 but extent items
used 1073901568
ERROR: block group[4406535782400 1073741824] used 1073627136 but extent items
used 0
ERROR: extent [3830028140544 4096] referencer bytenr mismatch, wanted:
3830028140544, have: 3826183847936
% tail btrfs_check_output_20170303.log
ERROR: data extent[3924538445824 4096] backref lost
ERROR: data extent[3933464903680 4096] backref lost
ERROR: data extent[3924538531840 4096] backref lost
ERROR: data extent[3839131480064 4096] backref lost
ERROR: data extent[3834701750272 4096] backref lost
ERROR: data extent[3873087918080 4096] backref lost
ERROR: data extent[3873072283648 4096] backref lost
ERROR: data extent[3873088090112 8192] backref lost
ERROR: data extent[3873072287744 4096] backref lost
ERROR: data extent[3856294449152 4096] backref lost
That's right, slowly approaching 1.5 million lines of btrfs-check output!
That's *way* more than I ran it the last time this error happened a few weeks
ago.
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
2017-03-05 23:53 ` Marc Joliet
@ 2017-03-06 11:18 ` Marc Joliet
0 siblings, 0 replies; 23+ messages in thread
From: Marc Joliet @ 2017-03-06 11:18 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2981 bytes --]
On Montag, 6. März 2017 00:53:40 CET Marc Joliet wrote:
> On Mittwoch, 1. März 2017 19:14:07 CET you wrote:
> > In any
> >
> > case, I started btrfs-check on the device itself.
>
> *Sigh*, I had to restart it, because I forgot to redirect to a file and
> quite frankly wasn't expecting this flood of output, but here's a summary
> of the output after about 2 days:
>
[snip old output]
OK, it finished last night. Here's the summary again:
% wc -l btrfs_check_output_20170303.log
3028222 btrfs_check_output_20170303.log
% grep -v "backref lost" btrfs_check_output_20170303.log | grep -v "check \
(leaf\|node\) failed" | grep -v "lost its parent" | grep -v "referencer count"
checking extents
ERROR: block group[3879328546816 1073741824] used 1072840704 but extent items
used 1129164800
ERROR: block group[4163870130176 1073741824] used 1072259072 but extent items
used 0
ERROR: block group[4223999672320 1073741824] used 1073664000 but extent items
used 1074188288
ERROR: block group[4278760505344 1073741824] used 1073377280 but extent items
used 1073901568
ERROR: block group[4406535782400 1073741824] used 1073627136 but extent items
used 0
ERROR: extent [3830028140544 4096] referencer bytenr mismatch, wanted:
3830028140544, have: 3826183847936
ERROR: errors found in extent allocation tree or chunk allocation
checking free space cache
checking fs roots
Checking filesystem on /dev/sdb2
UUID: f97b3cda-15e8-418b-bb9b-235391ef2a38
found 892572778496 bytes used err is -5
total csum bytes: 860790216
total tree bytes: 36906336256
total fs tree bytes: 34551476224
total extent tree bytes: 1230610432
btree space waste bytes: 7446885892
file data blocks allocated: 16359581663232
referenced 2358137831424
> That's right, slowly approaching 1.5 million lines of btrfs-check output!
> That's *way* more than I ran it the last time this error happened a few
> weeks ago.
As can be seen above, that ballooned to over 3 million lines. Since the
output is 4.2 MB, even after XZ compression, I put it up on my Dropbox, just
in case it's interesting to anybody:
https://www.dropbox.com/s/h6ftqpygfr4vsks/btrfs_check_output_20170303.log.xz?
dl=0
Since this is my backup drive, and the second time within a month that it had
problems like this, *and* I've got both the btrfs-image dump and btrfs-check
output, I'm going to go ahead and reformat, so that my three computers are
finally backed up again.
Oh, and for what it's worth, I did test against a 4.8 kernel, and pretty much
immediately got the "forced RO" error, just like with 4.9. I didn't try
anything older (or newer).
As a last step, I'll probably collect all information I have and post it to
bugzilla when I have a chance, since others might hit it, too (Kai did before
me, after all).
Greetings
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2017-03-06 11:20 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-28 13:29 [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists Kai Krakow
2017-02-02 12:01 ` Marc Joliet
2017-02-03 22:44 ` Kai Krakow
2017-02-10 22:15 ` Marc Joliet
2017-02-11 2:01 ` Kai Krakow
2017-02-11 10:18 ` Marc Joliet
2017-02-14 12:52 ` Marc Joliet
2017-02-17 8:19 ` Kai Krakow
2017-02-28 22:14 ` Marc Joliet
2017-03-01 8:23 ` Marc Joliet
2017-03-01 9:32 ` Qu Wenruo
2017-03-01 18:14 ` Marc Joliet
2017-03-01 18:27 ` Marc Joliet
2017-03-01 18:43 ` Marc Joliet
2017-03-02 9:44 ` Marc Joliet
2017-03-03 1:09 ` Qu Wenruo
2017-03-03 11:26 ` Marc Joliet
2017-03-05 23:53 ` Marc Joliet
2017-03-06 11:18 ` Marc Joliet
2017-03-02 0:43 ` Qu Wenruo
2017-03-02 9:43 ` Marc Joliet
2017-03-03 1:00 ` Qu Wenruo
2017-03-03 11:54 ` Marc Joliet
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.