* [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
@ 2016-11-21 12:09 bepi
2016-11-26 14:56 ` Giuseppe Della Bianca
2016-12-19 4:53 ` Qu Wenruo
0 siblings, 2 replies; 37+ messages in thread
From: bepi @ 2016-11-21 12:09 UTC (permalink / raw)
To: linux-btrfs
Hi.
My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64 btrfs-progs-4.4.1-1.fc23.x86_64
Testing the remote differential receive (via ssh and in local network) of 24
sequential snapshots, and simultaneously deleting the snapshot, (in the same
file system, but in a different subvolume), there has been an file access error,
and the file system has corrupt.
Both scrub, both recovery and clear_cache mount options, both btrfsck, have
failed, the file system is left in a state unusable.
After reformatting the filesystem, remote receive of 24 snapshots worked properly.
The file system is used exclusively for receive the snapshot, it is composed of
a single device.
The initial snapshot is a linux installation of 50Gb.
I think that there was a race condition between the receive and deletion of
snapshots (that were performed on two different subvolume).
Best regards.
gdb
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-11-21 12:09 [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive bepi
@ 2016-11-26 14:56 ` Giuseppe Della Bianca
2016-11-26 18:56 ` Chris Murphy
2016-12-19 4:53 ` Qu Wenruo
1 sibling, 1 reply; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-11-26 14:56 UTC (permalink / raw)
To: linux-btrfs
from /var/log/messages
Nov 20 20:04:29 exnetold kernel: Modules linked in: fuse bridge ebtable_filter
ebtables tun 8021q bnx2fc cnic uio garp mrp fcoe stp llc libfcoe libfc
scsi_transport_fc nvidia_modeset(POE) nf_log_ipv4 nf_log_common xt_LOG
xt_limit xt_multiport ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6
nf_defrag_ipv6 xt_CHECKSUM xt_conntrack iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6_tables hwmon_vid btrfs
snd_hda_codec_analog xor snd_hda_codec_generic raid6_pq snd_hda_codec_hdmi
powernow_k8 snd_hda_intel kvm_amd snd_hda_codec nvidia(POE) kvm snd_hda_core
snd_hwdep ppdev irqbypass snd_seq snd_seq_device snd_pcm amd64_edac_mod
edac_mce_amd edac_core snd_timer snd drm parport_pc k8temp soundcore shpchp
parport asus_atk0110 i2c_nforce2
Nov 20 20:04:29 exnetold kernel: acpi_cpufreq tpm_tis tpm vboxnetadp(OE)
vboxnetflt(OE) vboxdrv(OE) binfmt_misc ata_generic pata_acpi serio_raw sata_nv
forcedeth fjes analog gameport joydev i2c_dev
Nov 20 20:04:29 exnetold kernel: CPU: 1 PID: 16330 Comm: kworker/u4:0 Tainted:
P OE 4.4.6-301.fc23.x86_64 #1
Nov 20 20:04:29 exnetold kernel: Hardware name: System manufacturer System
Product Name/M2N, BIOS 0902 02/16/2009
Nov 20 20:04:29 exnetold kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper [btrfs]
Nov 20 20:04:29 exnetold kernel: 0000000000000286 0000000061a6ce66
ffff88008c797ae0 ffffffff813b542e
Nov 20 20:04:29 exnetold kernel: 0000000000000000 ffffffffa0c944d2 ffff88008c797b18
ffffffff810a40f2
Nov 20 20:04:29 exnetold kernel: 00000006e7a80000 00000000fffffffe
0000000000000000 ffff88004edc0000
Nov 20 20:04:29 exnetold kernel: Call Trace:
Nov 20 20:04:29 exnetold kernel: [<ffffffff813b542e>] dump_stack+0x63/0x85
Nov 20 20:04:29 exnetold kernel: [<ffffffff810a40f2>]
warn_slowpath_common+0x82/0xc0
Nov 20 20:04:29 exnetold kernel: [<ffffffff810a423a>] warn_slowpath_null+0x1a/0x20
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bf8148>]
__btrfs_free_extent.isra.69+0x898/0xd30 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0be99e6>] ? btrfs_free_path+0x26/0x30
[btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bf6601>] ?
__btrfs_inc_extent_ref.isra.53+0x11/0x270 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bfc0fc>]
__btrfs_run_delayed_refs+0xa7c/0x11a0 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bff4f2>]
btrfs_run_delayed_refs+0x82/0x2a0 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bff747>]
delayed_ref_async_start+0x37/0x90 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0c45f42>]
btrfs_scrubparity_helper+0xc2/0x2e0 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bae3d>] ?
pwq_activate_delayed_work+0x3d/0x90
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0c4619e>]
btrfs_extent_refs_helper+0xe/0x10 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bc596>] process_one_work+0x156/0x430
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bc8be>] worker_thread+0x4e/0x450
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bc870>] ?
process_one_work+0x430/0x430
Nov 20 20:04:29 exnetold kernel: [<ffffffff810c2648>] kthread+0xd8/0xf0
Nov 20 20:04:29 exnetold kernel: [<ffffffff810c2570>] ?
kthread_worker_fn+0x160/0x160
Nov 20 20:04:29 exnetold kernel: [<ffffffff817a090f>] ret_from_fork+0x3f/0x70
Nov 20 20:04:29 exnetold kernel: [<ffffffff810c2570>] ?
kthread_worker_fn+0x160/0x160
Nov 20 20:04:29 exnetold kernel: ---[ end trace 4a87ca727985e15e ]---
Nov 20 20:04:29 exnetold kernel: BTRFS info (device sda1): leaf 95421874176
total ptrs 209 free space 3459
Nov 20 20:04:29 exnetold kernel: #011item 0 key (29654663168 169 0) itemoff
16250 itemsize 33
Nov 20 20:04:29 exnetold kernel: #011#011extent refs 1 gen 400 flags 258
8717 itemsize 33
...... long text ......
Nov 20 20:04:30 exnetold kernel: #011#011extent refs 1 gen 473 flags 258
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent
29657972736
Nov 20 20:04:30 exnetold kernel: #011item 208 key (29658071040 169 0) itemoff
8684 itemsize 33
Nov 20 20:04:30 exnetold kernel: #011#011extent refs 1 gen 473 flags 258
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent
29657972736
Nov 20 20:04:30 exnetold kernel: BTRFS error (device sda1): unable to find ref
byte nr 29656350720 parent 0 root 448 owner 1 offset 0
Nov 20 20:04:30 exnetold kernel: ------------[ cut here ]------------
Nov 20 20:04:30 exnetold kernel: WARNING: CPU: 1 PID: 16330 at
fs/btrfs/extent-tree.c:6549 __btrfs_free_extent.isra.69+0x8ff/0xd30 [btrfs]()
Nov 20 20:04:30 exnetold kernel: Modules linked in: fuse bridge ebtable_filter
ebtables tun 8021q bnx2fc cnic uio garp mrp fcoe stp llc libfcoe libfc
scsi_transport_fc nvidia_modeset(POE) nf_log_ipv4 nf_log_common xt_LOG
xt_limit xt_multiport ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6
nf_defrag_ipv6 xt_CHECKSUM xt_conntrack iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6_tables hwmon_vid btrfs
snd_hda_codec_analog xor snd_hda_codec_generic raid6_pq snd_hda_codec_hdmi
powernow_k8 snd_hda_intel kvm_amd snd_hda_codec nvidia(POE) kvm snd_hda_core
snd_hwdep ppdev irqbypass snd_seq snd_seq_device snd_pcm amd64_edac_mod
edac_mce_amd edac_core snd_timer snd drm parport_pc k8temp soundcore shpchp
parport asus_atk0110 i2c_nforce2
Nov 20 20:04:30 exnetold kernel: acpi_cpufreq tpm_tis tpm vboxnetadp(OE)
vboxnetflt(OE) vboxdrv(OE) binfmt_misc ata_generic pata_acpi serio_raw sata_nv
forcedeth fjes analog gameport joydev i2c_dev
Nov 20 20:04:30 exnetold kernel: CPU: 1 PID: 16330 Comm: kworker/u4:0 Tainted:
P W OE 4.4.6-301.fc23.x86_64 #1
Nov 20 20:04:30 exnetold kernel: Hardware name: System manufacturer System
Product Name/M2N, BIOS 0902 02/16/2009
Nov 20 20:04:30 exnetold kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper [btrfs]
Nov 20 20:04:30 exnetold kernel: 0000000000000286 0000000061a6ce66
ffff88008c797a88 ffffffff813b542e
Nov 20 20:04:30 exnetold kernel: ffff88008c797ad0 ffffffffa0c944d2 ffff88008c797ac0
ffffffff810a40f2
Nov 20 20:04:30 exnetold kernel: 00000006e7a80000 00000000fffffffe
0000000000000000 ffff88004edc0000
Nov 20 20:04:30 exnetold kernel: Call Trace:
Nov 20 20:04:30 exnetold kernel: [<ffffffff813b542e>] dump_stack+0x63/0x85
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a40f2>]
warn_slowpath_common+0x82/0xc0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a418c>] warn_slowpath_fmt+0x5c/0x80
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bf81af>]
__btrfs_free_extent.isra.69+0x8ff/0xd30 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0be99e6>] ? btrfs_free_path+0x26/0x30
[btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bf6601>] ?
__btrfs_inc_extent_ref.isra.53+0x11/0x270 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bfc0fc>]
__btrfs_run_delayed_refs+0xa7c/0x11a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff4f2>]
btrfs_run_delayed_refs+0x82/0x2a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff747>]
delayed_ref_async_start+0x37/0x90 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c45f42>]
btrfs_scrubparity_helper+0xc2/0x2e0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bae3d>] ?
pwq_activate_delayed_work+0x3d/0x90
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c4619e>]
btrfs_extent_refs_helper+0xe/0x10 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc596>] process_one_work+0x156/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc8be>] worker_thread+0x4e/0x450
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc870>] ?
process_one_work+0x430/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2648>] kthread+0xd8/0xf0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ?
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: [<ffffffff817a090f>] ret_from_fork+0x3f/0x70
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ?
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: ---[ end trace 4a87ca727985e15f ]---
Nov 20 20:04:30 exnetold kernel: BTRFS warning (device sda1):
__btrfs_free_extent:6549: Aborting unused transaction(No such entry).
Nov 20 20:04:30 exnetold kernel: BTRFS warning (device sda1):
btrfs_run_delayed_refs:2927: Aborting unused transaction(No such entry).
Nov 20 20:04:30 exnetold kernel: ------------[ cut here ]------------
Nov 20 20:04:30 exnetold kernel: WARNING: CPU: 1 PID: 16330 at
fs/btrfs/extent-tree.c:6543 __btrfs_free_extent.isra.69+0x898/0xd30 [btrfs]()
Nov 20 20:04:30 exnetold kernel: Modules linked in: fuse bridge ebtable_filter
ebtables tun 8021q bnx2fc cnic uio garp mrp fcoe stp llc libfcoe libfc
scsi_transport_fc nvidia_modeset(POE) nf_log_ipv4 nf_log_common xt_LOG
xt_limit xt_multiport ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6
nf_defrag_ipv6 xt_CHECKSUM xt_conntrack iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6_tables hwmon_vid btrfs
snd_hda_codec_analog xor snd_hda_codec_generic raid6_pq snd_hda_codec_hdmi
powernow_k8 snd_hda_intel kvm_amd snd_hda_codec nvidia(POE) kvm snd_hda_core
snd_hwdep ppdev irqbypass snd_seq snd_seq_device snd_pcm amd64_edac_mod
edac_mce_amd edac_core snd_timer snd drm parport_pc k8temp soundcore shpchp
parport asus_atk0110 i2c_nforce2
Nov 20 20:04:30 exnetold kernel: acpi_cpufreq tpm_tis tpm vboxnetadp(OE)
vboxnetflt(OE) vboxdrv(OE) binfmt_misc ata_generic pata_acpi serio_raw sata_nv
forcedeth fjes analog gameport joydev i2c_dev
Nov 20 20:04:30 exnetold kernel: CPU: 1 PID: 16330 Comm: kworker/u4:0 Tainted:
P W OE 4.4.6-301.fc23.x86_64 #1
Nov 20 20:04:30 exnetold kernel: Hardware name: System manufacturer System
Product Name/M2N, BIOS 0902 02/16/2009
Nov 20 20:04:30 exnetold kernel: Workqueue: btrfs-extent-refs
btrfs_extent_refs_helper [btrfs]
Nov 20 20:04:30 exnetold kernel: 0000000000000286 0000000061a6ce66
ffff88008c797ae0 ffffffff813b542e
Nov 20 20:04:30 exnetold kernel: 0000000000000000 ffffffffa0c944d2 ffff88008c797b18
ffffffff810a40f2
Nov 20 20:04:30 exnetold kernel: 00000006e813c000 00000000fffffffe
0000000000000000 ffff88004edc0000
Nov 20 20:04:30 exnetold kernel: Call Trace:
Nov 20 20:04:30 exnetold kernel: [<ffffffff813b542e>] dump_stack+0x63/0x85
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a40f2>]
warn_slowpath_common+0x82/0xc0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a423a>] warn_slowpath_null+0x1a/0x20
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bf8148>]
__btrfs_free_extent.isra.69+0x898/0xd30 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810fa10a>] ? console_unlock+0x20a/0x540
Nov 20 20:04:30 exnetold kernel: [<ffffffff810dabc2>] ? enqueue_entity+0x3a2/0xc90
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bfc0fc>]
__btrfs_run_delayed_refs+0xa7c/0x11a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff8104f57d>] ?
native_smp_send_reschedule+0x4d/0x70
Nov 20 20:04:30 exnetold kernel: [<ffffffff810cc1c5>] ? resched_curr+0x65/0xc0
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff4f2>]
btrfs_run_delayed_refs+0x82/0x2a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff747>]
delayed_ref_async_start+0x37/0x90 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c45f42>]
btrfs_scrubparity_helper+0xc2/0x2e0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bae3d>] ?
pwq_activate_delayed_work+0x3d/0x90
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c4619e>]
btrfs_extent_refs_helper+0xe/0x10 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc596>] process_one_work+0x156/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc8be>] worker_thread+0x4e/0x450
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc870>] ?
process_one_work+0x430/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2648>] kthread+0xd8/0xf0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ?
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: [<ffffffff817a090f>] ret_from_fork+0x3f/0x70
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ?
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: ---[ end trace 4a87ca727985e160 ]---
Nov 20 20:04:30 exnetold kernel: BTRFS info (device sda1): leaf 95549276160
total ptrs 230 free space 270
Nov 20 20:04:30 exnetold kernel: #011item 0 key (29662576640 169 0) itemoff
16250 itemsize 33
Nov 20 20:04:30 exnetold kernel: #011#011extent refs 1 gen 473 flags 258
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent
29657972736
...... long text ......
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent
29744676864
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent
29637902336
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent
21741715456
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent
21556609024
Nov 20 20:04:30 exnetold kernel: BTRFS error (device sda1): unable to find ref
byte nr 29663412224 parent 0 root 448 owner 1 offset 0
Nov 20 20:04:30 exnetold kernel: BTRFS: error (device sda1) in
__btrfs_free_extent:6549: errno=-2 No such entry
Nov 20 20:04:31 exnetold kernel: BTRFS info (device sda1): forced readonly
Nov 20 20:04:31 exnetold kernel: BTRFS: error (device sda1) in
btrfs_run_delayed_refs:2927: errno=-2 No such entry
................
Nov 20 20:04:44 exnetold kernel: BTRFS error (device sda1): cleaner
transaction attach returned -30
In data lunedì 21 novembre 2016 13:09:15, bepi@adria.it ha scritto:
> Hi.
>
> My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64
> btrfs-progs-4.4.1-1.fc23.x86_64
>
> Testing the remote differential receive (via ssh and in local network) of 24
> sequential snapshots, and simultaneously deleting the snapshot, (in the
> same file system, but in a different subvolume), there has been an file
> access error, and the file system has corrupt.
>
> Both scrub, both recovery and clear_cache mount options, both btrfsck, have
> failed, the file system is left in a state unusable.
>
> After reformatting the filesystem, remote receive of 24 snapshots worked
> properly.
>
> The file system is used exclusively for receive the snapshot, it is composed
> of a single device.
> The initial snapshot is a linux installation of 50Gb.
>
>
> I think that there was a race condition between the receive and deletion of
> snapshots (that were performed on two different subvolume).
>
>
> Best regards.
>
> gdb
>
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-11-26 14:56 ` Giuseppe Della Bianca
@ 2016-11-26 18:56 ` Chris Murphy
2016-11-27 18:18 ` Giuseppe Della Bianca
2016-12-04 18:11 ` Giuseppe Della Bianca
0 siblings, 2 replies; 37+ messages in thread
From: Chris Murphy @ 2016-11-26 18:56 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: Btrfs BTRFS
I haven't seen this with 4.7.10. I suggest running 'btrfs check'
(without repair) using a recent btrfs-progs. You can find 4.8.3 in
koji, just download the appropriate rpm, and 'dnf update *rpm'
As for kernel, Fedora 23 has 4.8.8 in updates (stable) and 4.8.10 is
in updates-testing, I suggest moving to one of those.
Chris Murphy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-11-26 18:56 ` Chris Murphy
@ 2016-11-27 18:18 ` Giuseppe Della Bianca
2016-12-04 18:11 ` Giuseppe Della Bianca
1 sibling, 0 replies; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-11-27 18:18 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
I have reinstalled from scratch my system (for remote receiving of the
snapshot).
Now it uses the 4.8.8-100.fc23.x86_64 kernel, and now I have a separate
partition for '''possible dangerous''' test.
I do not remember the exact sequence that caused the corruption of the file
system, but now it worked without problem.
If I have problems again, I will update the btrfs-tools and I will try to fix.
P.S The silent and unrecoverable corruption are not entirely friendly ( :) ).
Thank you for answer.
Gdb
> I haven't seen this with 4.7.10. I suggest running 'btrfs check'
> (without repair) using a recent btrfs-progs. You can find 4.8.3 in
> koji, just download the appropriate rpm, and 'dnf update *rpm'
>
> As for kernel, Fedora 23 has 4.8.8 in updates (stable) and 4.8.10 is
> in updates-testing, I suggest moving to one of those.
>
>
> Chris Murphy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-11-26 18:56 ` Chris Murphy
2016-11-27 18:18 ` Giuseppe Della Bianca
@ 2016-12-04 18:11 ` Giuseppe Della Bianca
2016-12-18 19:59 ` Giuseppe Della Bianca
1 sibling, 1 reply; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-04 18:11 UTC (permalink / raw)
To: linux-btrfs
> I haven't seen this with 4.7.10. I suggest running 'btrfs check'
> (without repair) using a recent btrfs-progs. You can find 4.8.3 in
> koji, just download the appropriate rpm, and 'dnf update *rpm'
>
> As for kernel, Fedora 23 has 4.8.8 in updates (stable) and 4.8.10 is
> in updates-testing, I suggest moving to one of those.
>
>
> Chris Murphy
Same problem, this time on a local subvolume.
kernel-4.8.8-100.fc23.x86_64
btrfs-progs v4.8.5
P.S. Do you want to do some debugging?
Otherwise I proceed with the reset of the partition.
My feeling is that in the btrfs receive function there is a random bug,
which makes the file system unusable.
btrfs check /dev/sda2
Checking filesystem on /dev/sda2
UUID: d49d8f95-84ee-4452-9252-ad030a083739
checking extents
ref mismatch on [312324046848 16384] extent item 0, found 1
Backref 312324046848 parent 1309 root 1309 not found in extent tree
backpointer mismatch on [312324046848 16384]
owner ref check failed [312324046848 16384]
ref mismatch on [322595897344 16384] extent item 0, found 1
Backref 322595897344 parent 1309 root 1309 not found in extent tree
backpointer mismatch on [322595897344 16384]
owner ref check failed [322595897344 16384]
checking free space cache
checking fs roots
warning line 3822
checking csums
checking root refs
found 162417897472 bytes used err is 0
total csum bytes: 152292600
total tree bytes: 6461636608
total fs tree bytes: 6010830848
total extent tree bytes: 281214976
btree space waste bytes: 1142051927
file data blocks allocated: 6628422606848
referenced 961225981952
dmesg
[11265.556898] Call Trace:
[11265.556908] [<ffffffff923e493e>] dump_stack+0x63/0x85
[11265.556916] [<ffffffff920a0ecb>] __warn+0xcb/0xf0
[11265.556919] [<ffffffff920a0ffd>] warn_slowpath_null+0x1d/0x20
[11265.556948] [<ffffffffc0351504>] __btrfs_free_extent.isra.69+0x864/0xcc0
[btrfs]
[11265.556978] [<ffffffffc0355cce>] __btrfs_run_delayed_refs+0xace/0x1220 [btrfs]
[11265.557006] [<ffffffffc03535dc>] ? btrfs_set_disk_extent_flags+0x7c/0xb0
[btrfs]
[11265.557034] [<ffffffffc03592f3>] btrfs_run_delayed_refs+0x93/0x2b0 [btrfs]
[11265.557077] [<ffffffffc0395e5b>] ? free_extent_buffer+0x4b/0xa0 [btrfs]
[11265.557112] [<ffffffffc036e87a>] btrfs_should_end_transaction+0x5a/0x60
[btrfs]
[11265.557139] [<ffffffffc0357a69>] btrfs_drop_snapshot+0x429/0x800 [btrfs]
[11265.557172] [<ffffffffc036fba9>] btrfs_clean_one_deleted_snapshot+0xa9/0xf0
[btrfs]
[11265.557202] [<ffffffffc036650d>] cleaner_kthread+0x15d/0x1d0 [btrfs]
[11265.557229] [<ffffffffc03663b0>] ? btrfs_destroy_pinned_extent+0xf0/0xf0
[btrfs]
[11265.557235] [<ffffffff920c0bf8>] kthread+0xd8/0xf0
[11265.557242] [<ffffffff927ffcbf>] ret_from_fork+0x1f/0x40
[11265.557247] [<ffffffff920c0b20>] ? kthread_worker_fn+0x170/0x170
[11265.557282] ---[ end trace 577009d8a389f458 ]---
[11265.557288] BTRFS info (device sda2): leaf 327108411392 total ptrs 190 free
space 88
[11265.557292] item 0 key (312323358720 169 0) itemoff 16142 itemsize 141
[11265.557294] extent refs 13 gen 2089 flags 258
[11265.557295] tree block backref root 1326
[11265.557297] tree block backref root 1324
..... long text ....
[11265.558885] tree block backref root 1314
[11265.558886] shared block backref parent 322595897344
[11265.558887] shared block backref parent 322309390336
[11265.558889] shared block backref parent 322171060224
[11265.558890] shared block backref parent 319298207744
[11265.558891] shared block backref parent 312326356992
[11265.558895] BTRFS error (device sda2): unable to find ref byte nr
312324046848 parent 0 root 1309 owner 1 offset 0
[11265.558901] ------------[ cut here ]------------
[11265.558931] WARNING: CPU: 3 PID: 6915 at fs/btrfs/extent-tree.c:6951
__btrfs_free_extent.isra.69+0x8c8/0xcc0 [btrfs]
[11265.558932] BTRFS: Transaction aborted (error -2)
..... ....
[11265.559095] Call Trace:
[11265.559102] [<ffffffff923e493e>] dump_stack+0x63/0x85
[11265.559108] [<ffffffff920a0ecb>] __warn+0xcb/0xf0
[11265.559114] [<ffffffff920a0f4f>] warn_slowpath_fmt+0x5f/0x80
[11265.559141] [<ffffffffc0351568>] __btrfs_free_extent.isra.69+0x8c8/0xcc0
[btrfs]
[11265.559170] [<ffffffffc0355cce>] __btrfs_run_delayed_refs+0xace/0x1220 [btrfs]
[11265.559197] [<ffffffffc03535dc>] ? btrfs_set_disk_extent_flags+0x7c/0xb0
[btrfs]
[11265.559225] [<ffffffffc03592f3>] btrfs_run_delayed_refs+0x93/0x2b0 [btrfs]
[11265.559265] [<ffffffffc0395e5b>] ? free_extent_buffer+0x4b/0xa0 [btrfs]
[11265.559298] [<ffffffffc036e87a>] btrfs_should_end_transaction+0x5a/0x60
[btrfs]
[11265.559324] [<ffffffffc0357a69>] btrfs_drop_snapshot+0x429/0x800 [btrfs]
[11265.559355] [<ffffffffc036fba9>] btrfs_clean_one_deleted_snapshot+0xa9/0xf0
[btrfs]
[11265.559384] [<ffffffffc036650d>] cleaner_kthread+0x15d/0x1d0 [btrfs]
[11265.559411] [<ffffffffc03663b0>] ? btrfs_destroy_pinned_extent+0xf0/0xf0
[btrfs]
[11265.559416] [<ffffffff920c0bf8>] kthread+0xd8/0xf0
[11265.559423] [<ffffffff927ffcbf>] ret_from_fork+0x1f/0x40
[11265.559428] [<ffffffff920c0b20>] ? kthread_worker_fn+0x170/0x170
[11265.559431] ---[ end trace 577009d8a389f459 ]---
[11265.559434] BTRFS: error (device sda2) in __btrfs_free_extent:6951:
errno=-2 No such entry
[11265.559439] BTRFS info (device sda2): forced readonly
[11265.559444] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2960:
errno=-2 No such entry
[11265.805418] BTRFS error (device sda2): cleaner transaction attach returned
-30
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-04 18:11 ` Giuseppe Della Bianca
@ 2016-12-18 19:59 ` Giuseppe Della Bianca
2016-12-18 20:12 ` Chris Murphy
2016-12-18 21:36 ` Xin Zhou
0 siblings, 2 replies; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-18 19:59 UTC (permalink / raw)
To: linux-btrfs
> Same problem, this time on a local subvolume.
>
> kernel-4.8.8-100.fc23.x86_64
>
> btrfs-progs v4.8.5
]zac[
I had three filesystem corruption.
The point at which the problem it appeared, is similar in all three cases.
Subvolume structure and operations sequence:
btrfsreceive/
btrfsreceive/root/
btrfsreceive/root/.part/
1) Sending XYZ differential snapshot in to ' btrfsreceive/root/.part/ '.
2) Create snapshot from ' btrfsreceive/root/.part/XYZ ' to ' btrfsreceive/root
/XYZ '.
3) Delete snapshot ' btrfsreceive/root/.part/XYZ '.
Always in step 2) I had two files unreadable error (view previous posts), and
one already existing object error (see below).
All three times I had to re-create from scratch the various partitions (on
disks and systems different).
I can help you, in some way, to find the problem?
Or is useless to continue report it?
dic 18 18:29:58 exnetold.gdb.it kernel: ------------[ cut here ]------------
dic 18 18:29:58 exnetold.gdb.it kernel: WARNING: CPU: 1 PID: 4325 at
fs/btrfs/extent-tree.c:2960 btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: Transaction aborted (error -17)
dic 18 18:29:58 exnetold.gdb.it kernel: Modules linked in: fuse xt_CHECKSUM
ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
nf_conntrack_br
dic 18 18:29:58 exnetold.gdb.it kernel: soundcore acpi_cpufreq tpm_tis
tpm_tis_core tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic
nouveau vide
dic 18 18:29:58 exnetold.gdb.it kernel: CPU: 1 PID: 4325 Comm: umount Tainted:
G W 4.8.8-100.fc23.x86_64 #1
dic 18 18:29:58 exnetold.gdb.it kernel: Hardware name: System manufacturer
System Product Name/M2N, BIOS 0902 02/16/2009
dic 18 18:29:58 exnetold.gdb.it kernel: 0000000000000286 00000000dd260fac
ffff8ffa0d25bb60 ffffffffbc3e493e
dic 18 18:29:58 exnetold.gdb.it kernel: ffff8ffa0d25bbb0 0000000000000000
ffff8ffa0d25bba0 ffffffffbc0a0ecb
dic 18 18:29:58 exnetold.gdb.it kernel: 00000b9000000049 ffff8ff9e61b40a0
ffff8ffa2da77800 ffffffffffffffff
dic 18 18:29:58 exnetold.gdb.it kernel: Call Trace:
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc3e493e>] dump_stack+0x63/0x85
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0ecb>] __warn+0xcb/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0f4f>]
warn_slowpath_fmt+0x5f/0x80
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb4e3>]
btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07d62ec>] ?
btrfs_cow_block+0x10c/0x1e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ff62e>]
commit_cowonly_roots+0xae/0x2e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb466>] ?
btrfs_run_delayed_refs+0x206/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc08706b4>] ?
btrfs_qgroup_account_extents+0x84/0x180 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc0802187>]
btrfs_commit_transaction+0x547/0xa40 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07faa9f>]
btrfs_commit_super+0x8f/0xa0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07fcbcb>]
close_ctree+0x2db/0x380 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc26d3da>] ?
evict_inodes+0x15a/0x180
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ccf29>]
btrfs_put_super+0x19/0x20 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2520bf>]
generic_shutdown_super+0x6f/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2523b2>]
kill_anon_super+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07cdd98>]
btrfs_kill_super+0x18/0x110 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc252763>]
deactivate_locked_super+0x43/0x70
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2527ec>]
deactivate_super+0x5c/0x60
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2711bf>]
cleanup_mnt+0x3f/0x90
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc271252>]
__cleanup_mnt+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0bf0ce>]
task_work_run+0x7e/0xa0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0032d2>]
exit_to_usermode_loop+0xc2/0xd0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc003bf1>]
syscall_return_slowpath+0xa1/0xb0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc7ffb3a>]
entry_SYSCALL_64_fastpath+0xa2/0xa4
dic 18 18:29:58 exnetold.gdb.it kernel: ---[ end trace f7eb2e818f727168 ]---
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
btrfs_run_delayed_refs:2960: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS info (device sda3): forced
readonly
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS warning (device sda3): Skipping
commit of aborted transaction.
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
cleanup_transaction:1854: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): commit
super ret -17
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): cleaner
transaction attach returned -30
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-18 19:59 ` Giuseppe Della Bianca
@ 2016-12-18 20:12 ` Chris Murphy
2016-12-18 21:36 ` Xin Zhou
1 sibling, 0 replies; 37+ messages in thread
From: Chris Murphy @ 2016-12-18 20:12 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: Btrfs BTRFS
I'd say backup the volume conventionally to get important data
secured. Then use btrfs check --repair to fix the problems with the
file system, and then try the btrfs send receive again.
It most certainly is a bug, but it's not clear to me what the cause
is; other than the kernel isn't handling the problem on-disk
gracefully. Maybe repair will fix it. Another possibility is to move
to kernel 4.9.0 and see if it still reproduces. Per usual, there's a
ton of bug fixes in each kernel release.
Otherwise I'm out of ideas.
Chris Murphy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-18 19:59 ` Giuseppe Della Bianca
2016-12-18 20:12 ` Chris Murphy
@ 2016-12-18 21:36 ` Xin Zhou
2016-12-19 12:46 ` bepi
` (2 more replies)
1 sibling, 3 replies; 37+ messages in thread
From: Xin Zhou @ 2016-12-18 21:36 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: linux-btrfs
Hi Giuseppe,
Would you like to tell some details about:
1. the XYZ snapshot was taken from which subvolume
2. where the base (initial) snapshot is stored
3. The 3 partitions receives the same snapshot, are they in the same btrfs configuration and subvol structure?
Also, would you send the link reports "two files unreadable error" post mentioned in step 2?
Hope can see the message and figure out if the issue first comes from sender or receiver side.
Thanks,
Xin
Sent: Sunday, December 18, 2016 at 11:59 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
> Same problem, this time on a local subvolume.
>
> kernel-4.8.8-100.fc23.x86_64
>
> btrfs-progs v4.8.5
]zac[
I had three filesystem corruption.
The point at which the problem it appeared, is similar in all three cases.
Subvolume structure and operations sequence:
btrfsreceive/
btrfsreceive/root/
btrfsreceive/root/.part/
1) Sending XYZ differential snapshot in to ' btrfsreceive/root/.part/ '.
2) Create snapshot from ' btrfsreceive/root/.part/XYZ ' to ' btrfsreceive/root
/XYZ '.
3) Delete snapshot ' btrfsreceive/root/.part/XYZ '.
Always in step 2) I had two files unreadable error (view previous posts), and
one already existing object error (see below).
All three times I had to re-create from scratch the various partitions (on
disks and systems different).
I can help you, in some way, to find the problem?
Or is useless to continue report it?
dic 18 18:29:58 exnetold.gdb.it kernel: ------------[ cut here ]------------
dic 18 18:29:58 exnetold.gdb.it kernel: WARNING: CPU: 1 PID: 4325 at
fs/btrfs/extent-tree.c:2960 btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: Transaction aborted (error -17)
dic 18 18:29:58 exnetold.gdb.it kernel: Modules linked in: fuse xt_CHECKSUM
ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
nf_conntrack_br
dic 18 18:29:58 exnetold.gdb.it kernel: soundcore acpi_cpufreq tpm_tis
tpm_tis_core tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic
nouveau vide
dic 18 18:29:58 exnetold.gdb.it kernel: CPU: 1 PID: 4325 Comm: umount Tainted:
G W 4.8.8-100.fc23.x86_64 #1
dic 18 18:29:58 exnetold.gdb.it kernel: Hardware name: System manufacturer
System Product Name/M2N, BIOS 0902 02/16/2009
dic 18 18:29:58 exnetold.gdb.it kernel: 0000000000000286 00000000dd260fac
ffff8ffa0d25bb60 ffffffffbc3e493e
dic 18 18:29:58 exnetold.gdb.it kernel: ffff8ffa0d25bbb0 0000000000000000
ffff8ffa0d25bba0 ffffffffbc0a0ecb
dic 18 18:29:58 exnetold.gdb.it kernel: 00000b9000000049 ffff8ff9e61b40a0
ffff8ffa2da77800 ffffffffffffffff
dic 18 18:29:58 exnetold.gdb.it kernel: Call Trace:
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc3e493e>] dump_stack+0x63/0x85
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0ecb>] __warn+0xcb/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0f4f>]
warn_slowpath_fmt+0x5f/0x80
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb4e3>]
btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07d62ec>] ?
btrfs_cow_block+0x10c/0x1e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ff62e>]
commit_cowonly_roots+0xae/0x2e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb466>] ?
btrfs_run_delayed_refs+0x206/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc08706b4>] ?
btrfs_qgroup_account_extents+0x84/0x180 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc0802187>]
btrfs_commit_transaction+0x547/0xa40 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07faa9f>]
btrfs_commit_super+0x8f/0xa0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07fcbcb>]
close_ctree+0x2db/0x380 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc26d3da>] ?
evict_inodes+0x15a/0x180
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ccf29>]
btrfs_put_super+0x19/0x20 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2520bf>]
generic_shutdown_super+0x6f/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2523b2>]
kill_anon_super+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07cdd98>]
btrfs_kill_super+0x18/0x110 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc252763>]
deactivate_locked_super+0x43/0x70
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2527ec>]
deactivate_super+0x5c/0x60
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2711bf>]
cleanup_mnt+0x3f/0x90
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc271252>]
__cleanup_mnt+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0bf0ce>]
task_work_run+0x7e/0xa0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0032d2>]
exit_to_usermode_loop+0xc2/0xd0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc003bf1>]
syscall_return_slowpath+0xa1/0xb0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc7ffb3a>]
entry_SYSCALL_64_fastpath+0xa2/0xa4
dic 18 18:29:58 exnetold.gdb.it kernel: ---[ end trace f7eb2e818f727168 ]---
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
btrfs_run_delayed_refs:2960: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS info (device sda3): forced
readonly
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS warning (device sda3): Skipping
commit of aborted transaction.
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
cleanup_transaction:1854: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): commit
super ret -17
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): cleaner
transaction attach returned -30
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-11-21 12:09 [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive bepi
2016-11-26 14:56 ` Giuseppe Della Bianca
@ 2016-12-19 4:53 ` Qu Wenruo
2016-12-19 12:54 ` bepi
1 sibling, 1 reply; 37+ messages in thread
From: Qu Wenruo @ 2016-12-19 4:53 UTC (permalink / raw)
To: bepi, linux-btrfs
At 11/21/2016 08:09 PM, bepi@adria.it wrote:
> Hi.
>
> My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64 btrfs-progs-4.4.1-1.fc23.x86_64
>
> Testing the remote differential receive (via ssh and in local network) of 24
> sequential snapshots, and simultaneously deleting the snapshot, (in the same
> file system, but in a different subvolume), there has been an file access error,
> and the file system has corrupt.
Are you using qgroup?
IIRC, Filipe fixed a problem which could cause backref corruption which
only happens if quota is enabled.
Thanks,
Qu
>
> Both scrub, both recovery and clear_cache mount options, both btrfsck, have
> failed, the file system is left in a state unusable.
>
> After reformatting the filesystem, remote receive of 24 snapshots worked properly.
>
> The file system is used exclusively for receive the snapshot, it is composed of
> a single device.
> The initial snapshot is a linux installation of 50Gb.
>
>
> I think that there was a race condition between the receive and deletion of
> snapshots (that were performed on two different subvolume).
>
>
> Best regards.
>
> gdb
>
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-18 21:36 ` Xin Zhou
@ 2016-12-19 12:46 ` bepi
2016-12-19 13:04 ` bepi
2016-12-19 18:55 ` Giuseppe Della Bianca
2 siblings, 0 replies; 37+ messages in thread
From: bepi @ 2016-12-19 12:46 UTC (permalink / raw)
To: Xin Zhou; +Cc: linux-btrfs
Hi.
It is a bit complex.
Primary system
subvolume on SSD devices on PCIe slot
/root/ (fedora 23, 50GB usati)
/btrfssnapshot/
/btrfssnapshot/root/ (for /root/ snapshot)
/btrfssnapshot/root/root.1
/btrfssnapshot/root/root.2
/btrfssnapshot/root/root.XYZ
subvolume on device HDD "1" sata
/data_storage/ (data, 100GB usati)
/data_backup/ (backup tar files, programs, downloads, etc., used 250GB)
/btrfssnapshot/
/btrfssnapshot/data_storage/ (for /data_storage/ snapshot)
/btrfssnapshot/data_backup/ (for /data_backup/ snapshot)
/btrfssnapshot/data_storage/data_storage.1
/btrfssnapshot/data_storage/data_storage.2
/btrfssnapshot/data_storage/data_storage.XYZ
/btrfssnapshot/data_backup/data_backup.1
/btrfssnapshot/data_backup/data_backup.2
/btrfssnapshot/data_backup/data_backup.XYZ
subvolume on HDD device "2" sata
partition 1
/btrfsreceive/root/ (for receive /btrfssnapshot/root/ snapshot)
/btrfsreceive/root/.part/
partition 2
/btrfsreceive/data_storage/ (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/ (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/
Secondary system for receiving snapshot
subvolume on HDD device "3" sata
partition 1
/btrfsreceive/root/
/btrfsreceive/root/.part/
partition 2
/btrfsreceive/data_storage/ (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/ (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/
My bash script create snapshot of /root/, /data_storage/ and /data_backup/ on
/btrfssnapshot/ .
Snapshot is created with .part extension, when the creation is finished
properly, is renamed to .1, .2, .XYZ .
My bash script send snapshot from /btrfssnapshot/root/,
/btrfssnapshot/data_storage/ and /btrfssnapshot/data_backup/ (sending
differential, n - 1 -> n) to the subvolume /btrfsreceive/root/,
/btrfsreceive/data_storage/ and /btrfsreceive/data_backup/.
My bash script sends the same snapshot also to the secondary system for
receiving snapshots (using ssh).
P.S. If I recreate the receiving partition from scratch, the receive working
properly.
The previous problems information are in this thread.
Please, you can read the thread?
Thanks to you.
Gdb
Scrive Xin Zhou <xin.zhou@gmx.com>:
>
> Hi Giuseppe,
>
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
>
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2?
> Hope can see the message and figure out if the issue first comes from sender
> or receiver side.
>
> Thanks,
> Xin
> Â
>
> Sent:Â Sunday, December 18, 2016 at 11:59 AM
> From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> To:Â linux-btrfs@vger.kernel.org
> Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> > Same problem, this time on a local subvolume.
> >
> > kernel-4.8.8-100.fc23.x86_64
> >
> > btrfs-progs v4.8.5
> ]zac[
>
> I had three filesystem corruption.
>
> The point at which the problem it appeared, is similar in all three cases.
>
> Subvolume structure and operations sequence:
>
> btrfsreceive/
> btrfsreceive/root/
> btrfsreceive/root/.part/
>
> 1) Sending XYZ differential snapshot in to ' btrfsreceive/root/.part/ '.
> 2) Create snapshot from ' btrfsreceive/root/.part/XYZ ' to '
> btrfsreceive/root
> /XYZ '.
> 3) Delete snapshot ' btrfsreceive/root/.part/XYZ '.
>
> Always in step 2) I had two files unreadable error (view previous posts),
> and
> one already existing object error (see below).
>
> All three times I had to re-create from scratch the various partitions (on
> disks and systems different).
>
> I can help you, in some way, to find the problem?
>
> Or is useless to continue report it?
>
>
>
> dic 18 18:29:58 exnetold.gdb.it kernel: ------------[ cut here ]------------
> dic 18 18:29:58 exnetold.gdb.it kernel: WARNING: CPU: 1 PID: 4325 at
> fs/btrfs/extent-tree.c:2960 btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: Transaction aborted (error
> -17)
> dic 18 18:29:58 exnetold.gdb.it kernel: Modules linked in: fuse xt_CHECKSUM
> ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
> nf_conntrack_br
> dic 18 18:29:58 exnetold.gdb.it kernel: soundcore acpi_cpufreq tpm_tis
> tpm_tis_core tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic
> nouveau vide
> dic 18 18:29:58 exnetold.gdb.it kernel: CPU: 1 PID: 4325 Comm: umount
> Tainted:
> G W 4.8.8-100.fc23.x86_64 #1
> dic 18 18:29:58 exnetold.gdb.it kernel: Hardware name: System manufacturer
> System Product Name/M2N, BIOS 0902 02/16/2009
> dic 18 18:29:58 exnetold.gdb.it kernel: 0000000000000286 00000000dd260fac
> ffff8ffa0d25bb60 ffffffffbc3e493e
> dic 18 18:29:58 exnetold.gdb.it kernel: ffff8ffa0d25bbb0 0000000000000000
> ffff8ffa0d25bba0 ffffffffbc0a0ecb
> dic 18 18:29:58 exnetold.gdb.it kernel: 00000b9000000049 ffff8ff9e61b40a0
> ffff8ffa2da77800 ffffffffffffffff
> dic 18 18:29:58 exnetold.gdb.it kernel: Call Trace:
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc3e493e>]
> dump_stack+0x63/0x85
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0ecb>]
> __warn+0xcb/0xf0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0f4f>]
> warn_slowpath_fmt+0x5f/0x80
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb4e3>]
> btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07d62ec>] ?
> btrfs_cow_block+0x10c/0x1e0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ff62e>]
> commit_cowonly_roots+0xae/0x2e0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb466>] ?
> btrfs_run_delayed_refs+0x206/0x2b0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc08706b4>] ?
> btrfs_qgroup_account_extents+0x84/0x180 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc0802187>]
> btrfs_commit_transaction+0x547/0xa40 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07faa9f>]
> btrfs_commit_super+0x8f/0xa0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07fcbcb>]
> close_ctree+0x2db/0x380 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc26d3da>] ?
> evict_inodes+0x15a/0x180
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ccf29>]
> btrfs_put_super+0x19/0x20 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2520bf>]
> generic_shutdown_super+0x6f/0xf0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2523b2>]
> kill_anon_super+0x12/0x20
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07cdd98>]
> btrfs_kill_super+0x18/0x110 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc252763>]
> deactivate_locked_super+0x43/0x70
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2527ec>]
> deactivate_super+0x5c/0x60
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2711bf>]
> cleanup_mnt+0x3f/0x90
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc271252>]
> __cleanup_mnt+0x12/0x20
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0bf0ce>]
> task_work_run+0x7e/0xa0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0032d2>]
> exit_to_usermode_loop+0xc2/0xd0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc003bf1>]
> syscall_return_slowpath+0xa1/0xb0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc7ffb3a>]
> entry_SYSCALL_64_fastpath+0xa2/0xa4
> dic 18 18:29:58 exnetold.gdb.it kernel: ---[ end trace f7eb2e818f727168 ]---
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
> btrfs_run_delayed_refs:2960: errno=-17 Object already exists
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS info (device sda3): forced
> readonly
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS warning (device sda3):
> Skipping
> commit of aborted transaction.
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
> cleanup_transaction:1854: errno=-17 Object already exists
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): commit
> super ret -17
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): cleaner
> transaction attach returned -30
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-19 4:53 ` Qu Wenruo
@ 2016-12-19 12:54 ` bepi
0 siblings, 0 replies; 37+ messages in thread
From: bepi @ 2016-12-19 12:54 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs
>
>
> At 11/21/2016 08:09 PM, bepi@adria.it wrote:
> > Hi.
> >
> > My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64
> btrfs-progs-4.4.1-1.fc23.x86_64
> >
> > Testing the remote differential receive (via ssh and in local network) of
> 24
> > sequential snapshots, and simultaneously deleting the snapshot, (in the
> same
> > file system, but in a different subvolume), there has been an file access
> error,
> > and the file system has corrupt.
>
> Are you using qgroup?
>
> IIRC, Filipe fixed a problem which could cause backref corruption which
> only happens if quota is enabled.
>
> Thanks,
> Qu
Unfortunately not.
I read that thread (and others), but no one it seemed like my case.
To you (Thanks).
Gdb
>
> >
> > Both scrub, both recovery and clear_cache mount options, both btrfsck,
> have
> > failed, the file system is left in a state unusable.
> >
> > After reformatting the filesystem, remote receive of 24 snapshots worked
> properly.
> >
> > The file system is used exclusively for receive the snapshot, it is
> composed of
> > a single device.
> > The initial snapshot is a linux installation of 50Gb.
> >
> >
> > I think that there was a race condition between the receive and deletion
> of
> > snapshots (that were performed on two different subvolume).
> >
> >
> > Best regards.
> >
> > gdb
> >
> > ----------------------------------------------------
> > This mail has been sent using Alpikom webmail system
> > http://www.alpikom.it
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
>
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-18 21:36 ` Xin Zhou
2016-12-19 12:46 ` bepi
@ 2016-12-19 13:04 ` bepi
2016-12-19 18:55 ` Giuseppe Della Bianca
2 siblings, 0 replies; 37+ messages in thread
From: bepi @ 2016-12-19 13:04 UTC (permalink / raw)
To: linux-btrfs
(Resend)
Hi.
It is a bit complex.
Primary system
subvolume on SSD devices on PCIe slot
/root/ (fedora 23, 50GB usati)
/btrfssnapshot/
/btrfssnapshot/root/ (for /root/ snapshot)
/btrfssnapshot/root/root.1
/btrfssnapshot/root/root.2
/btrfssnapshot/root/root.XYZ
subvolume on device HDD "1" sata
/data_storage/ (data, 100GB usati)
/data_backup/ (backup tar files, programs, downloads, etc., used 250GB)
/btrfssnapshot/
/btrfssnapshot/data_storage/ (for /data_storage/ snapshot)
/btrfssnapshot/data_backup/ (for /data_backup/ snapshot)
/btrfssnapshot/data_storage/data_storage.1
/btrfssnapshot/data_storage/data_storage.2
/btrfssnapshot/data_storage/data_storage.XYZ
/btrfssnapshot/data_backup/data_backup.1
/btrfssnapshot/data_backup/data_backup.2
/btrfssnapshot/data_backup/data_backup.XYZ
subvolume on HDD device "2" sata
partition 1
/btrfsreceive/root/ (for receive /btrfssnapshot/root/ snapshot)
/btrfsreceive/root/.part/
partition 2
/btrfsreceive/data_storage/ (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/ (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/
Secondary system for receiving snapshot
subvolume on HDD device "3" sata
partition 1
/btrfsreceive/root/
/btrfsreceive/root/.part/
partition 2
/btrfsreceive/data_storage/ (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/ (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/
My bash script create snapshot of /root/, /data_storage/ and /data_backup/ on
/btrfssnapshot/ .
Snapshot is created with .part extension, when the creation is finished
properly, is renamed to .1, .2, .XYZ .
My bash script send snapshot from /btrfssnapshot/root/,
/btrfssnapshot/data_storage/ and /btrfssnapshot/data_backup/ (sending
differential, n - 1 -> n) to the subvolume /btrfsreceive/root/,
/btrfsreceive/data_storage/ and /btrfsreceive/data_backup/.
My bash script sends the same snapshot also to the secondary system for
receiving snapshots (using ssh).
P.S. If I recreate the receiving partition from scratch, the receive working
properly.
The previous problems information are in this thread.
Please, you can read the thread?
Thanks to you.
Gdb
Scrive Xin Zhou <xin.zhou@gmx.com>:
>
> Hi Giuseppe,
>
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
>
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2?
> Hope can see the message and figure out if the issue first comes from sender
> or receiver side.
>
> Thanks,
> Xin
> Â
>
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-18 21:36 ` Xin Zhou
2016-12-19 12:46 ` bepi
2016-12-19 13:04 ` bepi
@ 2016-12-19 18:55 ` Giuseppe Della Bianca
2016-12-20 17:43 ` Xin Zhou
2 siblings, 1 reply; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-19 18:55 UTC (permalink / raw)
To: Xin Zhou; +Cc: linux-btrfs
a concrete example
SNAPSHOT
/dev/nvme0n1p2 on /tmp/tmp.X3vU6dLLVI type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
btrfsManage SNAPSHOT /
(2016-12-19 19:44:00) Start btrfsManage
. . . Start managing SNAPSHOT ' / ' filesystem ' root ' snapshot
In ' btrfssnapshot ' latest source snapshot ' root-2016-12-18_15:10:01.40 '
. . . date ' 2016-12-18_15:10:01 ' number ' 40 '
Creation ' root-2016-12-19_19:44:00.part ' snapshot from ' root ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.X3vU6dLLVI/root' in '/tmp/tmp.X3vU6dLLVI/btrfssnapshot/root/root-2016-12-19_19:44:00.part'
Renaming ' root-2016-12-19_19:44:00.part ' into ' root-2016-12-19_19:44:00.41 ' snapshot
Source snapshot list of ' root ' subvolume
. . . btrfssnapshot/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfssnapshot/root/root-2016-12-19_19:44:00.41
(2016-12-19 19:44:05) End btrfsManage
. . . End managing SNAPSHOT ' / ' filesystem ' root ' snapshot
CORRECTLY
SEND e RECEIVE
/dev/nvme0n1p2 on /tmp/tmp.o78czE0Bo6 type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sda2 on /tmp/tmp.XcwqQCKq09 type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
btrfsManage SEND / /dev/sda2
(2016-12-19 19:47:24) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
Sending ' root-2016-12-19_19:44:00.41 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41
. . . At snapshot root-2016-12-19_19:44:00.41
Creation ' root-2016-12-19_19:44:00.41 ' snapshot from ' .part/root-2016-12-19_19:44:00.41 ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41' in '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/root-2016-12-19_19:44:00.41'
. . . Delete subvolume (commit): '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41'
Snapshot list in ' /dev/sda2 ' device
. . . btrfsreceive/data_backup/data_backup-2016-12-17_12:07:00.1
. . . btrfsreceive/data_storage/data_storage-2016-12-10_17:05:51.1
. . . btrfsreceive/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfsreceive/root/root-2016-12-19_19:44:00.41
(2016-12-19 19:48:37) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
CORRECTLY
> Hi Giuseppe,
>
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
>
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2? Hope can see the message and figure out if the issue
> first comes from sender or receiver side.
>
> Thanks,
> Xin
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-19 18:55 ` Giuseppe Della Bianca
@ 2016-12-20 17:43 ` Xin Zhou
2016-12-21 12:27 ` bepi
2016-12-26 11:24 ` Giuseppe Della Bianca
0 siblings, 2 replies; 37+ messages in thread
From: Xin Zhou @ 2016-12-20 17:43 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: linux-btrfs
Hi,
The system seems running some customized scripts continuously backup data from a NVME drive to HDDs.
If the 3 HDDs backup storage are same in btrfs config, and the there is a bug in btrfs code,
they all suppose to fail after the same operation sequence.
Otherwise, probably one of the HDDs might have issue, or there is a bug in layer below btrfs.
For the customize script, it might be helpful to check the file system consistency after each transfer.
That might be useful to figure out which step generates a corruption, and if there is error propagations.
Regards,
Xin
Sent: Monday, December 19, 2016 at 10:55 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Xin Zhou" <xin.zhou@gmx.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
a concrete example
SNAPSHOT
/dev/nvme0n1p2 on /tmp/tmp.X3vU6dLLVI type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
btrfsManage SNAPSHOT /
(2016-12-19 19:44:00) Start btrfsManage
. . . Start managing SNAPSHOT ' / ' filesystem ' root ' snapshot
In ' btrfssnapshot ' latest source snapshot ' root-2016-12-18_15:10:01.40 '
. . . date ' 2016-12-18_15:10:01 ' number ' 40 '
Creation ' root-2016-12-19_19:44:00.part ' snapshot from ' root ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.X3vU6dLLVI/root' in '/tmp/tmp.X3vU6dLLVI/btrfssnapshot/root/root-2016-12-19_19:44:00.part'
Renaming ' root-2016-12-19_19:44:00.part ' into ' root-2016-12-19_19:44:00.41 ' snapshot
Source snapshot list of ' root ' subvolume
. . . btrfssnapshot/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfssnapshot/root/root-2016-12-19_19:44:00.41
(2016-12-19 19:44:05) End btrfsManage
. . . End managing SNAPSHOT ' / ' filesystem ' root ' snapshot
CORRECTLY
SEND e RECEIVE
/dev/nvme0n1p2 on /tmp/tmp.o78czE0Bo6 type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sda2 on /tmp/tmp.XcwqQCKq09 type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
btrfsManage SEND / /dev/sda2
(2016-12-19 19:47:24) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
Sending ' root-2016-12-19_19:44:00.41 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41
. . . At snapshot root-2016-12-19_19:44:00.41
Creation ' root-2016-12-19_19:44:00.41 ' snapshot from ' .part/root-2016-12-19_19:44:00.41 ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41' in '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/root-2016-12-19_19:44:00.41'
. . . Delete subvolume (commit): '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41'
Snapshot list in ' /dev/sda2 ' device
. . . btrfsreceive/data_backup/data_backup-2016-12-17_12:07:00.1
. . . btrfsreceive/data_storage/data_storage-2016-12-10_17:05:51.1
. . . btrfsreceive/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfsreceive/root/root-2016-12-19_19:44:00.41
(2016-12-19 19:48:37) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
CORRECTLY
> Hi Giuseppe,
>
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
>
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2? Hope can see the message and figure out if the issue
> first comes from sender or receiver side.
>
> Thanks,
> Xin
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-20 17:43 ` Xin Zhou
@ 2016-12-21 12:27 ` bepi
2016-12-21 21:09 ` Chris Murphy
2016-12-26 11:24 ` Giuseppe Della Bianca
1 sibling, 1 reply; 37+ messages in thread
From: bepi @ 2016-12-21 12:27 UTC (permalink / raw)
To: Xin Zhou; +Cc: linux-btrfs
Hi.
I will insert ' btrfs check ' after each ' receive ' in my script.
I will test again my hardware.
But is not very likely that 2 computers, 3 HDD, 3 partitions, all have issue.
I think that the problem is a concomitance of operations, a race condition, a
random conditions.
I'll try to create a test case.
P.S. For find the problem may need to insert tools as ' coredumper ' and '
sanitize ' in ' btrfs ', detect in realtime the ' extent ' corruption, and log
detection.
Thank you.
Gdb
Xin Zhou <xin.zhou@gmx.com>:
> Hi,
>
> The system seems running some customized scripts continuously backup data
> from a NVME drive to HDDs.
> If the 3 HDDs backup storage are same in btrfs config, and the there is a bug
> in btrfs code,
> they all suppose to fail after the same operation sequence.
>
> Otherwise, probably one of the HDDs might have issue, or there is a bug in
> layer below btrfs.
>
> For the customize script, it might be helpful to check the file system
> consistency after each transfer.
> That might be useful to figure out which step generates a corruption, and if
> there is error propagations.
>
> Regards,
> Xin
> Â
> Â
>
> Sent:Â Monday, December 19, 2016 at 10:55 AM
> From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> To:Â "Xin Zhou" <xin.zhou@gmx.com>
> Cc:Â linux-btrfs@vger.kernel.org
> Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> a concrete example
>
>
> SNAPSHOT
>
> /dev/nvme0n1p2 on /tmp/tmp.X3vU6dLLVI type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
>
> btrfsManage SNAPSHOT /
>
> (2016-12-19 19:44:00) Start btrfsManage
> . . . Start managing SNAPSHOT ' / ' filesystem ' root ' snapshot
>
> In ' btrfssnapshot ' latest source snapshot ' root-2016-12-18_15:10:01.40 '
> . . . date ' 2016-12-18_15:10:01 ' number ' 40 '
>
> Creation ' root-2016-12-19_19:44:00.part ' snapshot from ' root ' subvolume
> . . . Create a readonly snapshot of '/tmp/tmp.X3vU6dLLVI/root' in
> '/tmp/tmp.X3vU6dLLVI/btrfssnapshot/root/root-2016-12-19_19:44:00.part'
>
> Renaming ' root-2016-12-19_19:44:00.part ' into ' root-2016-12-19_19:44:00.41
> ' snapshot
>
> Source snapshot list of ' root ' subvolume
> . . . btrfssnapshot/root/root-2016-08-28-12-35-01.1
> ]zac[
> . . . btrfssnapshot/root/root-2016-12-19_19:44:00.41
>
> (2016-12-19 19:44:05) End btrfsManage
> . . . End managing SNAPSHOT ' / ' filesystem ' root ' snapshot
> CORRECTLY
>
>
>
> SEND e RECEIVE
>
> /dev/nvme0n1p2 on /tmp/tmp.o78czE0Bo6 type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> /dev/sda2 on /tmp/tmp.XcwqQCKq09 type btrfs
> (rw,relatime,space_cache,subvolid=5,subvol=/)
>
> btrfsManage SEND / /dev/sda2
>
> (2016-12-19 19:47:24) Start btrfsManage
> . . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2
> '
>
> Sending ' root-2016-12-19_19:44:00.41 ' source snapshot to ' btrfsreceive '
> subvolume
> . . . btrfs send -p
> /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-18_15:10:01.40
> /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs
> receive /tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/
> . . . At subvol
> /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41
> . . . At snapshot root-2016-12-19_19:44:00.41
>
> Creation ' root-2016-12-19_19:44:00.41 ' snapshot from '
> .part/root-2016-12-19_19:44:00.41 ' subvolume
> . . . Create a readonly snapshot of
> '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41' in
> '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/root-2016-12-19_19:44:00.41'
> . . . Delete subvolume (commit):
> '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41'
>
> Snapshot list in ' /dev/sda2 ' device
> . . . btrfsreceive/data_backup/data_backup-2016-12-17_12:07:00.1
> . . . btrfsreceive/data_storage/data_storage-2016-12-10_17:05:51.1
> . . . btrfsreceive/root/root-2016-08-28-12-35-01.1
> ]zac[
> . . . btrfsreceive/root/root-2016-12-19_19:44:00.41
>
> (2016-12-19 19:48:37) End btrfsManage
> . . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
> CORRECTLY
>
>
>
> > Hi Giuseppe,
> >
> > Would you like to tell some details about:
> > 1. the XYZ snapshot was taken from which subvolume
> > 2. where the base (initial) snapshot is stored
> > 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> > configuration and subvol structure?
> >
> > Also, would you send the link reports "two files unreadable error" post
> > mentioned in step 2? Hope can see the message and figure out if the issue
> > first comes from sender or receiver side.
> >
> > Thanks,
> > Xin
> >
> >
> Â
>
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-21 12:27 ` bepi
@ 2016-12-21 21:09 ` Chris Murphy
2016-12-21 21:11 ` Chris Murphy
0 siblings, 1 reply; 37+ messages in thread
From: Chris Murphy @ 2016-12-21 21:09 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: Xin Zhou, Btrfs BTRFS
What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
mount option?
Chris Murphy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-21 21:09 ` Chris Murphy
@ 2016-12-21 21:11 ` Chris Murphy
2016-12-21 22:14 ` Xin Zhou
` (2 more replies)
0 siblings, 3 replies; 37+ messages in thread
From: Chris Murphy @ 2016-12-21 21:11 UTC (permalink / raw)
Cc: Giuseppe Della Bianca, Xin Zhou, Btrfs BTRFS
On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com> wrote:
> What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> mount option?
This slows things down, and in that case it might avoid the problem if
it's the result of a race condition.
--
Chris Murphy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-21 21:11 ` Chris Murphy
@ 2016-12-21 22:14 ` Xin Zhou
2016-12-23 7:28 ` Giuseppe Della Bianca
2016-12-23 7:16 ` Giuseppe Della Bianca
2016-12-27 9:29 ` Giuseppe Della Bianca
2 siblings, 1 reply; 37+ messages in thread
From: Xin Zhou @ 2016-12-21 22:14 UTC (permalink / raw)
To: Chris Murphy; +Cc: Giuseppe Della Bianca, Btrfs BTRFS
Hi,
Racing condition can happen, if running multiple transfers to the same destination.
Would you like to tell how many transfers are the scripts running at a time to a specific hdd?
Thanks,
Xin
Sent: Wednesday, December 21, 2016 at 1:11 PM
From: "Chris Murphy" <lists@colorremedies.com>
To: No recipient address
Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou" <xin.zhou@gmx.com>, "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com> wrote:
> What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> mount option?
This slows things down, and in that case it might avoid the problem if
it's the result of a race condition.
--
Chris Murphy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-21 21:11 ` Chris Murphy
2016-12-21 22:14 ` Xin Zhou
@ 2016-12-23 7:16 ` Giuseppe Della Bianca
2016-12-27 9:29 ` Giuseppe Della Bianca
2 siblings, 0 replies; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-23 7:16 UTC (permalink / raw)
To: Btrfs BTRFS
(synthetic resend)
I'll try to compile the kernel and mount with the config and option enabled.
I keep in mind the side effects of that option.
Thanks for all.
P.S. Sorry for my bad English.
Chris Murphy ha scritto:
> On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
wrote:
> > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > mount option?
>
> This slows things down, and in that case it might avoid the problem if
> it's the result of a race condition.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-21 22:14 ` Xin Zhou
@ 2016-12-23 7:28 ` Giuseppe Della Bianca
2016-12-23 16:53 ` Xin Zhou
0 siblings, 1 reply; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-23 7:28 UTC (permalink / raw)
To: Btrfs BTRFS
(synthetic resend)
Hi.
Is possible that there are transfers, cancellations and other, at the same
time, but not in the same subvolume.
My script checks that there are no transfers in progress on the same
subvolume.
Is possible that the same subvolume is mounted several times (temporary mount
at the beginning, and unmount at the end, in my script).
Thanks for all.
P.S. Sorry for my bad English.
Gdb
In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> Hi,
> Racing condition can happen, if running multiple transfers to the same
> destination. Would you like to tell how many transfers are the scripts
> running at a time to a specific hdd?
>
> Thanks,
> Xin
>
>
> Sent: Wednesday, December 21, 2016 at 1:11 PM
> From: "Chris Murphy" <lists@colorremedies.com>
> To: No recipient address
> Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou" <xin.zhou@gmx.com>,
> "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> receive
> On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
wrote:
> > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > mount option?
>
> This slows things down, and in that case it might avoid the problem if
> it's the result of a race condition.
>
> --
> Chris Murphy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-23 7:28 ` Giuseppe Della Bianca
@ 2016-12-23 16:53 ` Xin Zhou
2016-12-23 17:48 ` bepi
0 siblings, 1 reply; 37+ messages in thread
From: Xin Zhou @ 2016-12-23 16:53 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: Btrfs BTRFS
Hi,
Does the script check the transfer status, and is there a transfer returns an error code?
Thanks,
Xin
Sent: Thursday, December 22, 2016 at 11:28 PM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
(synthetic resend)
Hi.
Is possible that there are transfers, cancellations and other, at the same
time, but not in the same subvolume.
My script checks that there are no transfers in progress on the same
subvolume.
Is possible that the same subvolume is mounted several times (temporary mount
at the beginning, and unmount at the end, in my script).
Thanks for all.
P.S. Sorry for my bad English.
Gdb
In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> Hi,
> Racing condition can happen, if running multiple transfers to the same
> destination. Would you like to tell how many transfers are the scripts
> running at a time to a specific hdd?
>
> Thanks,
> Xin
>
>
> Sent: Wednesday, December 21, 2016 at 1:11 PM
> From: "Chris Murphy" <lists@colorremedies.com>
> To: No recipient address
> Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou" <xin.zhou@gmx.com>,
> "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> receive
> On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
wrote:
> > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > mount option?
>
> This slows things down, and in that case it might avoid the problem if
> it's the result of a race condition.
>
> --
> Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-23 16:53 ` Xin Zhou
@ 2016-12-23 17:48 ` bepi
2016-12-23 18:35 ` Xin Zhou
0 siblings, 1 reply; 37+ messages in thread
From: bepi @ 2016-12-23 17:48 UTC (permalink / raw)
To: Xin Zhou; +Cc: Btrfs BTRFS
Yes.
Is through to the btrfs-tools error message that the script has printed, that I
realized the filesystem corruption.
P.S. Various messages that you see in the working examples of the script, are
emitted directly by the btrfs-tools.
Gdb
Xin Zhou <xin.zhou@gmx.com>:
> Hi,
>
> Does the script check the transfer status, and is there a transfer returns an
> error code?
> Thanks,
> Xin
> Â
> Â
>
> Sent:Â Thursday, December 22, 2016 at 11:28 PM
> From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> To:Â "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> (synthetic resend)
>
> Hi.
>
> Is possible that there are transfers, cancellations and other, at the same
> time, but not in the same subvolume.
>
> My script checks that there are no transfers in progress on the same
> subvolume.
>
> Is possible that the same subvolume is mounted several times (temporary
> mount
> at the beginning, and unmount at the end, in my script).
>
>
> Thanks for all.
>
>
> P.S. Sorry for my bad English.
>
>
> Gdb
>
>
> In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > Hi,
> > Racing condition can happen, if running multiple transfers to the same
> > destination. Would you like to tell how many transfers are the scripts
> > running at a time to a specific hdd?
> >
> > Thanks,
> > Xin
> >
> >
> > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > From: "Chris Murphy" <lists@colorremedies.com>
> > To: No recipient address
> > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> <xin.zhou@gmx.com>,
> > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > receive
> > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> wrote:
> > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > mount option?
> >
> > This slows things down, and in that case it might avoid the problem if
> > it's the result of a race condition.
> >
> > --
> > Chris Murphy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-23 17:48 ` bepi
@ 2016-12-23 18:35 ` Xin Zhou
2016-12-24 12:16 ` Giuseppe Della Bianca
2016-12-24 12:47 ` Giuseppe Della Bianca
0 siblings, 2 replies; 37+ messages in thread
From: Xin Zhou @ 2016-12-23 18:35 UTC (permalink / raw)
To: bepi; +Cc: Btrfs BTRFS
Hi,
Would you like to show the "btrfs send/receive" command the script are using, including all the parameters,
and how the script waits for a completion of a transfer.
>From the beginning of the thread, it seems the transfer tests are going through different network environment.
Thanks,
Xin
Sent: Friday, December 23, 2016 at 9:48 AM
From: bepi@adria.it
To: "Xin Zhou" <xin.zhou@gmx.com>
Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Yes.
Is through to the btrfs-tools error message that the script has printed, that I
realized the filesystem corruption.
P.S. Various messages that you see in the working examples of the script, are
emitted directly by the btrfs-tools.
Gdb
Xin Zhou <xin.zhou@gmx.com>:
> Hi,
>
> Does the script check the transfer status, and is there a transfer returns an
> error code?
> Thanks,
> Xin
> Â
> Â
>
> Sent:Â Thursday, December 22, 2016 at 11:28 PM
> From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> To:Â "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> (synthetic resend)
>
> Hi.
>
> Is possible that there are transfers, cancellations and other, at the same
> time, but not in the same subvolume.
>
> My script checks that there are no transfers in progress on the same
> subvolume.
>
> Is possible that the same subvolume is mounted several times (temporary
> mount
> at the beginning, and unmount at the end, in my script).
>
>
> Thanks for all.
>
>
> P.S. Sorry for my bad English.
>
>
> Gdb
>
>
> In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > Hi,
> > Racing condition can happen, if running multiple transfers to the same
> > destination. Would you like to tell how many transfers are the scripts
> > running at a time to a specific hdd?
> >
> > Thanks,
> > Xin
> >
> >
> > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > From: "Chris Murphy" <lists@colorremedies.com>
> > To: No recipient address
> > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> <xin.zhou@gmx.com>,
> > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > receive
> > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> wrote:
> > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > mount option?
> >
> > This slows things down, and in that case it might avoid the problem if
> > it's the result of a race condition.
> >
> > --
> > Chris Murphy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it[http://www.alpikom.it]
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-23 18:35 ` Xin Zhou
@ 2016-12-24 12:16 ` Giuseppe Della Bianca
2016-12-24 20:15 ` Xin Zhou
2016-12-24 12:47 ` Giuseppe Della Bianca
1 sibling, 1 reply; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-24 12:16 UTC (permalink / raw)
To: Xin Zhou, linux-btrfs
Hi.
LOCAL "btrfs send/receive"
btrfs send -p /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.NRYgYVC3p6/btrfsreceive/root/.part/
REMOTE "btrfs send/receive"
btrfs send -p /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | ssh root@exnetold.gdb.it btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41
Remote btrfsManage script (btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41) executes
cat - | btrfs receive /tmp/tmp.Wd5togIiaL/btrfsreceive/root/.part/
The waiting for the end of a command, in shell script, is implicit/automatic.
Regards.
Gdb
> Hi,
>
> Would you like to show the "btrfs send/receive" command the script are
> using, including all the parameters, and how the script waits for a
> completion of a transfer.
>
> From the beginning of the thread, it seems the transfer tests are going
> through different network environment.
> Thanks,
> Xin
>
> Sent: Friday, December 23, 2016 at 9:48 AM
> From: bepi@adria.it
> To: "Xin Zhou" <xin.zhou@gmx.com>
> Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive Yes.
>
> Is through to the btrfs-tools error message that the script has printed,
> that I realized the filesystem corruption.
>
>
> P.S. Various messages that you see in the working examples of the script,
> are emitted directly by the btrfs-tools.
>
>
> Gdb
>
> Xin Zhou <xin.zhou@gmx.com>:
> > Hi,
> >
> > Does the script check the transfer status, and is there a transfer returns
> > an error code?
> > Thanks,
> > Xin
> > Â
> > Â
> >
> > Sent:Â Thursday, December 22, 2016 at 11:28 PM
> > From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> > To:Â "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> > Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file
> > system during the snapshot receive
> > (synthetic resend)
> >
> > Hi.
> >
> > Is possible that there are transfers, cancellations and other, at the same
> > time, but not in the same subvolume.
> >
> > My script checks that there are no transfers in progress on the same
> > subvolume.
> >
> > Is possible that the same subvolume is mounted several times (temporary
> > mount
> > at the beginning, and unmount at the end, in my script).
> >
> >
> > Thanks for all.
> >
> >
> > P.S. Sorry for my bad English.
> >
> >
> > Gdb
> >
> > In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > > Hi,
> > > Racing condition can happen, if running multiple transfers to the same
> > > destination. Would you like to tell how many transfers are the scripts
> > > running at a time to a specific hdd?
> > >
> > > Thanks,
> > > Xin
> > >
> > >
> > > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > > From: "Chris Murphy" <lists@colorremedies.com>
> > > To: No recipient address
> > > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> >
> > <xin.zhou@gmx.com>,
> >
> > > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > > receive
> > > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> >
> > wrote:
> > > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > > mount option?
> > >
> > > This slows things down, and in that case it might avoid the problem if
> > > it's the result of a race condition.
> > >
> > > --
> > > Chris Murphy
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it[http://www.alpikom.it]
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-23 18:35 ` Xin Zhou
2016-12-24 12:16 ` Giuseppe Della Bianca
@ 2016-12-24 12:47 ` Giuseppe Della Bianca
2017-08-19 14:56 ` Giuseppe Della Bianca
1 sibling, 1 reply; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-24 12:47 UTC (permalink / raw)
To: linux-btrfs
Hi.
LOCAL "btrfs send/receive"
btrfs send -p /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.NRYgYVC3p6/btrfsreceive/root/.part/
REMOTE "btrfs send/receive"
btrfs send -p /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | ssh root@exnetold.gdb.it btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41
Remote btrfsManage script (btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41) executes
cat - | btrfs receive /tmp/tmp.Wd5togIiaL/btrfsreceive/root/.part/
The waiting for the end of a command, in shell script, is implicit/automatic.
Regards.
Gdb
> Hi,
>
> Would you like to show the "btrfs send/receive" command the script are
> using, including all the parameters, and how the script waits for a
> completion of a transfer.
>
> From the beginning of the thread, it seems the transfer tests are going
> through different network environment.
> Thanks,
> Xin
>
> Sent: Friday, December 23, 2016 at 9:48 AM
> From: bepi@adria.it
> To: "Xin Zhou" <xin.zhou@gmx.com>
> Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive Yes.
>
> Is through to the btrfs-tools error message that the script has printed,
> that I realized the filesystem corruption.
>
>
> P.S. Various messages that you see in the working examples of the script,
> are emitted directly by the btrfs-tools.
>
>
> Gdb
>
> Xin Zhou <xin.zhou@gmx.com>:
> > Hi,
> >
> > Does the script check the transfer status, and is there a transfer returns
> > an error code?
> > Thanks,
> > Xin
> > Â
> > Â
> >
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-24 12:16 ` Giuseppe Della Bianca
@ 2016-12-24 20:15 ` Xin Zhou
2016-12-25 22:57 ` Duncan
2016-12-26 11:04 ` Giuseppe Della Bianca
0 siblings, 2 replies; 37+ messages in thread
From: Xin Zhou @ 2016-12-24 20:15 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: linux-btrfs
Hi,
Probably can try to use "-v" to enable more output print.
A quick look at the send / receive code, it seems a little bit risky.
It seems lack of specific error handlings, and in most cases, return the same error code.
I think it might be helpful, when a transfer succeed, the command prints the transfer id, source / dest,
and a specific "success" string.
Such output could help the script to figure out if a transfer really succeed.
The code is relatively new to me, I did not see retry logic in stream handling, please correct me if I am wrong about this.
So, I am not quite sure about the transfer behavior, if the system subject to network issues in heavy workload,
in which packets missing or connect issues are not rare.
Since the test mentioned at the begining deletes the snapshots after a transfer, while most users keep the middle snapshot even in cascading transfer, probably the current btrfs and cmds still works for regular users.
Thanks,
Xin
Sent: Saturday, December 24, 2016 at 4:16 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Xin Zhou" <xin.zhou@gmx.com>, linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Hi.
LOCAL "btrfs send/receive"
btrfs send -p /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.NRYgYVC3p6/btrfsreceive/root/.part/
REMOTE "btrfs send/receive"
btrfs send -p /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | ssh root@exnetold.gdb.it btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41
Remote btrfsManage script (btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41) executes
cat - | btrfs receive /tmp/tmp.Wd5togIiaL/btrfsreceive/root/.part/
The waiting for the end of a command, in shell script, is implicit/automatic.
Regards.
Gdb
> Hi,
>
> Would you like to show the "btrfs send/receive" command the script are
> using, including all the parameters, and how the script waits for a
> completion of a transfer.
>
> From the beginning of the thread, it seems the transfer tests are going
> through different network environment.
> Thanks,
> Xin
>
> Sent: Friday, December 23, 2016 at 9:48 AM
> From: bepi@adria.it
> To: "Xin Zhou" <xin.zhou@gmx.com>
> Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive Yes.
>
> Is through to the btrfs-tools error message that the script has printed,
> that I realized the filesystem corruption.
>
>
> P.S. Various messages that you see in the working examples of the script,
> are emitted directly by the btrfs-tools.
>
>
> Gdb
>
> Xin Zhou <xin.zhou@gmx.com>:
> > Hi,
> >
> > Does the script check the transfer status, and is there a transfer returns
> > an error code?
> > Thanks,
> > Xin
> > Â
> > Â
> >
> > Sent:Â Thursday, December 22, 2016 at 11:28 PM
> > From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> > To:Â "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> > Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file
> > system during the snapshot receive
> > (synthetic resend)
> >
> > Hi.
> >
> > Is possible that there are transfers, cancellations and other, at the same
> > time, but not in the same subvolume.
> >
> > My script checks that there are no transfers in progress on the same
> > subvolume.
> >
> > Is possible that the same subvolume is mounted several times (temporary
> > mount
> > at the beginning, and unmount at the end, in my script).
> >
> >
> > Thanks for all.
> >
> >
> > P.S. Sorry for my bad English.
> >
> >
> > Gdb
> >
> > In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > > Hi,
> > > Racing condition can happen, if running multiple transfers to the same
> > > destination. Would you like to tell how many transfers are the scripts
> > > running at a time to a specific hdd?
> > >
> > > Thanks,
> > > Xin
> > >
> > >
> > > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > > From: "Chris Murphy" <lists@colorremedies.com>
> > > To: No recipient address
> > > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> >
> > <xin.zhou@gmx.com>,
> >
> > > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > > receive
> > > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> >
> > wrote:
> > > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > > mount option?
> > >
> > > This slows things down, and in that case it might avoid the problem if
> > > it's the result of a race condition.
> > >
> > > --
> > > Chris Murphy
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it[http://www.alpikom.it][http://www.alpikom.it[http://www.alpikom.it]]
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html[http://vger.kernel.org/majordomo-info.html]
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-24 20:15 ` Xin Zhou
@ 2016-12-25 22:57 ` Duncan
2016-12-26 2:36 ` Xin Zhou
2016-12-26 11:04 ` Giuseppe Della Bianca
1 sibling, 1 reply; 37+ messages in thread
From: Duncan @ 2016-12-25 22:57 UTC (permalink / raw)
To: linux-btrfs
Xin Zhou posted on Sat, 24 Dec 2016 21:15:40 +0100 as excerpted:
> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this.
> So, I am not quite sure about the transfer behavior, if the system
> subject to network issues in heavy workload,
> in which packets missing or connect issues are not rare.
As you likely know I'm neither a dev, just a list regular and btrfs user
myself, and I'm not particularly familiar with send/receive as I don't
use it myself, but...
AFAIK, the send and receive sides are specifically designed to be
separate and to work with STDOUT/STDIN, so it's possible with STDOUT
redirection to "send" to a local file instead of directly to receive, and
then to replay that file on the other end by cat-ing it to receive.
As such, transfer behavior isn't really a factor at the btrfs layer,
since handling problems in the transfer layer is the responsibility of
whatever method the user is using to do that transfer, and the user is
presumed to use a transfer method with whatever reliability guarantees
they deem necessary. So network behavior isn't really a factor at the
btrfs level as that's the transfer layer and btrfs isn't worrying about
that, simply assuming it to have the necessary reliability.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-25 22:57 ` Duncan
@ 2016-12-26 2:36 ` Xin Zhou
2016-12-26 3:52 ` Duncan
0 siblings, 1 reply; 37+ messages in thread
From: Xin Zhou @ 2016-12-26 2:36 UTC (permalink / raw)
To: Duncan; +Cc: linux-btrfs
Hi,
For free software with open source code, that is quite good.
Most commercial product has a very robust error handling in transport, to guarantee no corruption due to transfer issues.
One interesting thing to investigate might be the btrfs send / receive result, under a disruptive network environment.
If the connection breaks in the middle of transfer (at different phase, maybe), see what could be the file system status.
Thanks,
Xin
Sent: Sunday, December 25, 2016 at 2:57 PM
From: Duncan <1i5t5.duncan@cox.net>
To: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Xin Zhou posted on Sat, 24 Dec 2016 21:15:40 +0100 as excerpted:
> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this.
> So, I am not quite sure about the transfer behavior, if the system
> subject to network issues in heavy workload,
> in which packets missing or connect issues are not rare.
As you likely know I'm neither a dev, just a list regular and btrfs user
myself, and I'm not particularly familiar with send/receive as I don't
use it myself, but...
AFAIK, the send and receive sides are specifically designed to be
separate and to work with STDOUT/STDIN, so it's possible with STDOUT
redirection to "send" to a local file instead of directly to receive, and
then to replay that file on the other end by cat-ing it to receive.
As such, transfer behavior isn't really a factor at the btrfs layer,
since handling problems in the transfer layer is the responsibility of
whatever method the user is using to do that transfer, and the user is
presumed to use a transfer method with whatever reliability guarantees
they deem necessary. So network behavior isn't really a factor at the
btrfs level as that's the transfer layer and btrfs isn't worrying about
that, simply assuming it to have the necessary reliability.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-26 2:36 ` Xin Zhou
@ 2016-12-26 3:52 ` Duncan
2016-12-27 3:20 ` Xin Zhou
0 siblings, 1 reply; 37+ messages in thread
From: Duncan @ 2016-12-26 3:52 UTC (permalink / raw)
To: linux-btrfs
Xin Zhou posted on Mon, 26 Dec 2016 03:36:09 +0100 as excerpted:
> One interesting thing to investigate might be the btrfs send / receive
> result, under a disruptive network environment. If the connection breaks
> in the middle of transfer (at different phase, maybe), see what could be
> the file system status.
Btrfs send, sends from a read-only snapshot, so the sending filesystem
shouldn't be harmed no matter what happens to send.
Btrfs receive does all its work in a new subvolume (basically a snapshot
in an incremental send, tho I'm not sure it's a full snapshot in the
technical sense), modifying the files therein using standard calls used
in other contexts as well, so absent bugs that should appear in those
other contexts too if they exist, the worst damage that a receive should
be able to do is an unfaithful replay of the send stream, such that an
appropriate copy of the sent snapshot doesn't appear on the receiver.
Which means even in the case of error, cleanup is as simple as deleting
the aborted/incompletely-received subvolume. Unless there are bugs that
would show up in other situations as well (or an out-of-space condition
is triggered that would likewise show up in other situations with a
similar amount of data/metadata written), there should be no effects
outside that received subvolume.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-24 20:15 ` Xin Zhou
2016-12-25 22:57 ` Duncan
@ 2016-12-26 11:04 ` Giuseppe Della Bianca
2016-12-26 17:41 ` Xin Zhou
1 sibling, 1 reply; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-26 11:04 UTC (permalink / raw)
To: Xin Zhou, Btrfs BTRFS
Hi.
I agree with Duncan, and I add:
- For remote transfer is used ssh.
ssh is designed to ensure integrity of data.
- Remote transfer uses a Gigabit Ethernet, it is never congested.
- I had the same problems with a local btrfs receive.
- The script currently has 907 lines of code, many of which are to ensure the
detection and display of btrfs tools errors.
- The script stops executing when btrs tools return an error code.
- Is not possible that the script does not display error messages or ignore
error code of btrfs tools.
An example of today:
(2016-12-26 10:53:51) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
Sending ' root-2016-12-04_18:13:57.35 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-03_18:07:09.34 /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35 | btrfs receive /tmp/tmp.pWWKP4vfAy/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35
. . . ERROR: truncate usr/share/locale/it/LC_MESSAGES/kio4.mo failed: Read-only file system
. . . At snapshot root-2016-12-04_18:13:57.35
. . . _EC_ERR_ 1
. . . _EC_ERR_ 141
(2016-12-26 10:54:28) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
WITH ERRORS
Checking filesystem on /dev/sda2
UUID: 44f1de7e-a65b-41ce-8ff4-20f7ed83e106
checking extents
ref mismatch on [62408097792 16384] extent item 0, found 1
Backref 62408097792 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [62408097792 16384]
owner ref check failed [62408097792 16384]
ref mismatch on [77565509632 16384] extent item 0, found 1
Backref 77565509632 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77565509632 16384]
]zac[
Backref 77826916352 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77826916352 16384]
owner ref check failed [77826916352 16384]
ref mismatch on [77853933568 16384] extent item 0, found 1
Backref 77853933568 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77853933568 16384]
owner ref check failed [77853933568 16384]
checking free space cache
checking fs roots
warning line 3822
checking csums
checking root refs
found 135128678400 bytes used err is 0
total csum bytes: 126946572
total tree bytes: 5132206080
total fs tree bytes: 4744757248
total extent tree bytes: 240795648
btree space waste bytes: 914832832
file data blocks allocated: 3311786532864
referenced 703616266240
Is likely that mine is a special case.
But a special case, with a code change in other points, can become a problem for many.
It's not nice to say, but it seems I have to hope that my problem becomes a problem of many.
Meanwhile, I'll find my own workaround of a probable serious btrfs bug.
Thank you.
Gdb
> Hi,
>
> Probably can try to use "-v" to enable more output print.
> A quick look at the send / receive code, it seems a little bit risky.
> It seems lack of specific error handlings, and in most cases, return the
> same error code. I think it might be helpful, when a transfer succeed, the
> command prints the transfer id, source / dest, and a specific "success"
> string.
> Such output could help the script to figure out if a transfer really
> succeed.
>
> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this. So, I am not quite
> sure about the transfer behavior, if the system subject to network issues
> in heavy workload, in which packets missing or connect issues are not rare.
>
> Since the test mentioned at the begining deletes the snapshots after a
> transfer, while most users keep the middle snapshot even in cascading
> transfer, probably the current btrfs and cmds still works for regular
> users.
>
> Thanks,
> Xin
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-20 17:43 ` Xin Zhou
2016-12-21 12:27 ` bepi
@ 2016-12-26 11:24 ` Giuseppe Della Bianca
1 sibling, 0 replies; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-26 11:24 UTC (permalink / raw)
To: xin.zhou, linux-btrfs
Hi.
I have executed the test that you have requested.
At this point I'm confused and demoralized
While 40 btrfs receive, btrfs check has reported many filesystem problems.
But at the end of all the btrfs receive, btrfs check did not find filesystem problem.
My tests have confirmed that the filesystem had no problems.
I think btrfs check detect false positives (this can change a repair in a destruction).
Regards.
Gdb
Xin Zhou Tue, 20 Dec 2016 09:47:01 -0800
>>Hi,
>>The system seems running some customized scripts continuously backup data from
>>a NVME drive to HDDs.
>>If the 3 HDDs backup storage are same in btrfs config, and the there is a bug
>>in btrfs code,
>>they all suppose to fail after the same operation sequence.
>>Otherwise, probably one of the HDDs might have issue, or there is a bug in
>>layer below btrfs.
>>For the customize script, it might be helpful to check the file system
>>consistency after each transfer.
>>That might be useful to figure out which step generates a corruption, and if
>>there is error propagations.
>>Regards,
>>Xin
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-26 11:04 ` Giuseppe Della Bianca
@ 2016-12-26 17:41 ` Xin Zhou
0 siblings, 0 replies; 37+ messages in thread
From: Xin Zhou @ 2016-12-26 17:41 UTC (permalink / raw)
To: Giuseppe Della Bianca; +Cc: Btrfs BTRFS
That is one way to diagnose the issue in data path.
If ssh can guarantee data transfer and retry, then those data protection company does not need to have a whole team handle the send / receive for remote data backup.
In your case, if the conection is very light, then the issue could be in other place.
Xin
Sent: Monday, December 26, 2016 at 3:04 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Xin Zhou" <xin.zhou@gmx.com>, "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Hi.
I agree with Duncan, and I add:
- For remote transfer is used ssh.
ssh is designed to ensure integrity of data.
- Remote transfer uses a Gigabit Ethernet, it is never congested.
- I had the same problems with a local btrfs receive.
- The script currently has 907 lines of code, many of which are to ensure the
detection and display of btrfs tools errors.
- The script stops executing when btrs tools return an error code.
- Is not possible that the script does not display error messages or ignore
error code of btrfs tools.
An example of today:
(2016-12-26 10:53:51) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
Sending ' root-2016-12-04_18:13:57.35 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-03_18:07:09.34 /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35 | btrfs receive /tmp/tmp.pWWKP4vfAy/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35
. . . ERROR: truncate usr/share/locale/it/LC_MESSAGES/kio4.mo failed: Read-only file system
. . . At snapshot root-2016-12-04_18:13:57.35
. . . _EC_ERR_ 1
. . . _EC_ERR_ 141
(2016-12-26 10:54:28) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
WITH ERRORS
Checking filesystem on /dev/sda2
UUID: 44f1de7e-a65b-41ce-8ff4-20f7ed83e106
checking extents
ref mismatch on [62408097792 16384] extent item 0, found 1
Backref 62408097792 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [62408097792 16384]
owner ref check failed [62408097792 16384]
ref mismatch on [77565509632 16384] extent item 0, found 1
Backref 77565509632 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77565509632 16384]
]zac[
Backref 77826916352 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77826916352 16384]
owner ref check failed [77826916352 16384]
ref mismatch on [77853933568 16384] extent item 0, found 1
Backref 77853933568 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77853933568 16384]
owner ref check failed [77853933568 16384]
checking free space cache
checking fs roots
warning line 3822
checking csums
checking root refs
found 135128678400 bytes used err is 0
total csum bytes: 126946572
total tree bytes: 5132206080
total fs tree bytes: 4744757248
total extent tree bytes: 240795648
btree space waste bytes: 914832832
file data blocks allocated: 3311786532864
referenced 703616266240
Is likely that mine is a special case.
But a special case, with a code change in other points, can become a problem for many.
It's not nice to say, but it seems I have to hope that my problem becomes a problem of many.
Meanwhile, I'll find my own workaround of a probable serious btrfs bug.
Thank you.
Gdb
> Hi,
>
> Probably can try to use "-v" to enable more output print.
> A quick look at the send / receive code, it seems a little bit risky.
> It seems lack of specific error handlings, and in most cases, return the
> same error code. I think it might be helpful, when a transfer succeed, the
> command prints the transfer id, source / dest, and a specific "success"
> string.
> Such output could help the script to figure out if a transfer really
> succeed.
>
> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this. So, I am not quite
> sure about the transfer behavior, if the system subject to network issues
> in heavy workload, in which packets missing or connect issues are not rare.
>
> Since the test mentioned at the begining deletes the snapshots after a
> transfer, while most users keep the middle snapshot even in cascading
> transfer, probably the current btrfs and cmds still works for regular
> users.
>
> Thanks,
> Xin
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-26 3:52 ` Duncan
@ 2016-12-27 3:20 ` Xin Zhou
0 siblings, 0 replies; 37+ messages in thread
From: Xin Zhou @ 2016-12-27 3:20 UTC (permalink / raw)
To: Duncan; +Cc: linux-btrfs
>Unless there are bugs that
>would show up in other situations as well (or an out-of-space condition
>is triggered that would likewise show up in other situations with a
>similar amount of data/metadata written),
That is exactly some bugs come from.
For simple cases, it is ok to assume the send/receive always succeed.
And if it errors out, assumes the delete always succeed, and the file system is in consistent status,
and good luck with the data.
Xin
Sent: Sunday, December 25, 2016 at 7:52 PM
From: Duncan <1i5t5.duncan@cox.net>
To: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Xin Zhou posted on Mon, 26 Dec 2016 03:36:09 +0100 as excerpted:
> One interesting thing to investigate might be the btrfs send / receive
> result, under a disruptive network environment. If the connection breaks
> in the middle of transfer (at different phase, maybe), see what could be
> the file system status.
Btrfs send, sends from a read-only snapshot, so the sending filesystem
shouldn't be harmed no matter what happens to send.
Btrfs receive does all its work in a new subvolume (basically a snapshot
in an incremental send, tho I'm not sure it's a full snapshot in the
technical sense), modifying the files therein using standard calls used
in other contexts as well, so absent bugs that should appear in those
other contexts too if they exist, the worst damage that a receive should
be able to do is an unfaithful replay of the send stream, such that an
appropriate copy of the sent snapshot doesn't appear on the receiver.
Which means even in the case of error, cleanup is as simple as deleting
the aborted/incompletely-received subvolume. Unless there are bugs that
would show up in other situations as well (or an out-of-space condition
is triggered that would likewise show up in other situations with a
similar amount of data/metadata written), there should be no effects
outside that received subvolume.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-21 21:11 ` Chris Murphy
2016-12-21 22:14 ` Xin Zhou
2016-12-23 7:16 ` Giuseppe Della Bianca
@ 2016-12-27 9:29 ` Giuseppe Della Bianca
2 siblings, 0 replies; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-27 9:29 UTC (permalink / raw)
To: linux-btrfs
Hi.
Only for information.
I tried ' check_int ' mount option, but with a few tens of GB used, the
kernel freezes completely (no ops, no messages, no panic, none).
Thanks for all.
Gdb
P.S. Thanks and happy work to all.
>>On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <li...@colorremedies.com> wrote:
>> What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
>> mount option?
>This slows things down, and in that case it might avoid the problem if
>it's the result of a race condition.
>--
>Chris Murphy
>--
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
2016-12-24 12:47 ` Giuseppe Della Bianca
@ 2017-08-19 14:56 ` Giuseppe Della Bianca
0 siblings, 0 replies; 37+ messages in thread
From: Giuseppe Della Bianca @ 2017-08-19 14:56 UTC (permalink / raw)
To: linux-btrfs
Hi.
Two news (one is OT and advertising to my btrfsManage script)
I had to recreate from scratch two corrupted btrfs filesystems probably due to
out of space during a snapshot receive.
Almost it's a habit rebuild the filesystems after a few months.
OT
I have released the latest version of my backup and btrfs manage scritp.
https://sourceforge.net/projects/btrfsmanage/
Bash script for managing btrfs filesystem (local and remote):
- Perform scrubs
- Creating snapshots
- Send snapshots
- List and delete snapshot
- Mount snapshot
- Simple to use and verify proper operation.
- Designed to be run from the shell or from the crontab.
- No setup, just copy btrfsManage.
- The subvolume on which to perform the snapshot is detected by the subvolume
currently mounted.
- The subvolume necessary to btrfsManage are created and mounted
automatically.
For example
/usr/local/bin/btrfsManage OPTION:usage SEND / /dev/sda3
(2017-08-19 15:45:01) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda3 '
Sending ' root-2017-08-19_15:20:01.129 ' source snapshot to ' btrfsreceive '
subvolume
. . . btrfs send -p /tmp/tmp.p8JVMMpzZI/btrfssnapshot/root/
root-2017-08-17_10:43:01.128 /tmp/tmp.p8JVMMpzZI/btrfssnapshot/root/
root-2017-08-19_15:20:01.129 | btrfs receive /tmp/tmp.UexL3E9E2F/btrfsreceive/
root/.part/
. . . At subvol /tmp/tmp.p8JVMMpzZI/btrfssnapshot/root/
root-2017-08-19_15:20:01.129
. . . At snapshot root-2017-08-19_15:20:01.129
Creation ' root-2017-08-19_15:20:01.129 ' snapshot from ' .part/
root-2017-08-19_15:20:01.129 ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.UexL3E9E2F/btrfsreceive/
root/.part/root-2017-08-19_15:20:01.129' in '/tmp/tmp.UexL3E9E2F/btrfsreceive/
root/root-2017-08-19_15:20:01.129'
. . . Delete subvolume (commit): '/tmp/tmp.UexL3E9E2F/btrfsreceive/root/.part/
root-2017-08-19_15:20:01.129'
btrfs filesystem usage -T /tmp/tmp.UexL3E9E2F
Overall:
Device size: 830.44GiB
Device allocated: 513.02GiB
Device unallocated: 317.42GiB
Device missing: 0.00B
Used: 509.83GiB
Free (estimated): 318.41GiB (min: 159.70GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data Metadata System
Id Path single DUP DUP Unallocated
-- --------- --------- -------- -------- -----------
1 /dev/sda3 505.01GiB 8.00GiB 16.00MiB 317.42GiB
-- --------- --------- -------- -------- -----------
Total 505.01GiB 4.00GiB 8.00MiB 317.42GiB
Used 504.01GiB 2.91GiB 80.00KiB
Snapshot list in ' /dev/sda3 ' device
. . . btrfsreceive/root/root-2017-07-23_19:46:01.117
. . . btrfsreceive/root/root-2017-07-29_15:20:01.118
. . . btrfsreceive/root/root-2017-07-29_16:54:01.119
. . . btrfsreceive/root/root-2017-08-05_15:20:01.120
. . . btrfsreceive/root/root-2017-08-06_17:59:01.121
. . . btrfsreceive/root/root-2017-08-12_13:00:01.122
. . . btrfsreceive/root/root-2017-08-12_15:20:01.123
. . . btrfsreceive/root/root-2017-08-14_17:59:01.124
. . . btrfsreceive/root/root-2017-08-15_15:20:01.125
. . . btrfsreceive/root/root-2017-08-15_15:56:01.126
. . . btrfsreceive/root/root-2017-08-16_18:40:01.127
. . . btrfsreceive/root/root-2017-08-17_10:43:01.128
. . . btrfsreceive/root/root-2017-08-19_15:20:01.129
(2017-08-19 15:46:09) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda3 '
CORRECTLY
Regards.
Gdb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
[not found] <CAJCQCtRmmMc3QwhFAkGqFPLc1_C4VpZCei_cevncUJduTJmg@mail.gmail.com>
@ 2016-12-18 20:39 ` Giuseppe Della Bianca
0 siblings, 0 replies; 37+ messages in thread
From: Giuseppe Della Bianca @ 2016-12-18 20:39 UTC (permalink / raw)
To: linux-btrfs
> Chris Murphy Sun, 18 Dec 2016 12:12:42 -0800
> I'd say backup the volume conventionally to get important data
> secured. Then use btrfs check --repair to fix the problems with the
> file system, and then try the btrfs send receive again.
> It most certainly is a bug, but it's not clear to me what the cause
> is; other than the kernel isn't handling the problem on-disk
> gracefully. Maybe repair will fix it. Another possibility is to move
> to kernel 4.9.0 and see if it still reproduces. Per usual, there's a
> ton of bug fixes in each kernel release.
> Otherwise I'm out of ideas.
> Chris Murphy
Thanks for the reply.
I have three partitions that receive the same snapshots (and a separate backup
for important data).
The damaged filesystems do not lose data, but only (!) waste time.
The repair attempts have only worsened the problem.
I understand the software development problems.
I would just like to be able to help you to find the problem.
Have a nice day
gdb
^ permalink raw reply [flat|nested] 37+ messages in thread
* [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
@ 2016-11-22 13:19 bepi
0 siblings, 0 replies; 37+ messages in thread
From: bepi @ 2016-11-22 13:19 UTC (permalink / raw)
To: linux-btrfs
Hi.
My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64 btrfs-progs-4.4.1-1.fc23.x86_64
Testing the remote differential receive (via ssh and in local network) of 24
sequential snapshots, and simultaneously deleting the snapshot, (in the same
file system, but in a different subvolume), there has been an file access error,
and the file system has corrupt.
Both scrub, both recovery and clear_cache mount options, both btrfsck, have
failed, the file system is left in a state unusable.
After reformatting the filesystem, remote receive of 24 snapshots worked properly.
The file system is used exclusively for receive the snapshot, it is composed of
a single device.
The initial snapshot is a linux installation of 50Gb.
I think that there was a race condition between the receive and deletion of
snapshots (that were performed on two different subvolume).
Best regards.
gdb
----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it
^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2017-08-19 14:57 UTC | newest]
Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-21 12:09 [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive bepi
2016-11-26 14:56 ` Giuseppe Della Bianca
2016-11-26 18:56 ` Chris Murphy
2016-11-27 18:18 ` Giuseppe Della Bianca
2016-12-04 18:11 ` Giuseppe Della Bianca
2016-12-18 19:59 ` Giuseppe Della Bianca
2016-12-18 20:12 ` Chris Murphy
2016-12-18 21:36 ` Xin Zhou
2016-12-19 12:46 ` bepi
2016-12-19 13:04 ` bepi
2016-12-19 18:55 ` Giuseppe Della Bianca
2016-12-20 17:43 ` Xin Zhou
2016-12-21 12:27 ` bepi
2016-12-21 21:09 ` Chris Murphy
2016-12-21 21:11 ` Chris Murphy
2016-12-21 22:14 ` Xin Zhou
2016-12-23 7:28 ` Giuseppe Della Bianca
2016-12-23 16:53 ` Xin Zhou
2016-12-23 17:48 ` bepi
2016-12-23 18:35 ` Xin Zhou
2016-12-24 12:16 ` Giuseppe Della Bianca
2016-12-24 20:15 ` Xin Zhou
2016-12-25 22:57 ` Duncan
2016-12-26 2:36 ` Xin Zhou
2016-12-26 3:52 ` Duncan
2016-12-27 3:20 ` Xin Zhou
2016-12-26 11:04 ` Giuseppe Della Bianca
2016-12-26 17:41 ` Xin Zhou
2016-12-24 12:47 ` Giuseppe Della Bianca
2017-08-19 14:56 ` Giuseppe Della Bianca
2016-12-23 7:16 ` Giuseppe Della Bianca
2016-12-27 9:29 ` Giuseppe Della Bianca
2016-12-26 11:24 ` Giuseppe Della Bianca
2016-12-19 4:53 ` Qu Wenruo
2016-12-19 12:54 ` bepi
2016-11-22 13:19 bepi
[not found] <CAJCQCtRmmMc3QwhFAkGqFPLc1_C4VpZCei_cevncUJduTJmg@mail.gmail.com>
2016-12-18 20:39 ` Giuseppe Della Bianca
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.