[CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[bepi]

Hi.

My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64 btrfs-progs-4.4.1-1.fc23.x86_64

Testing the remote differential receive (via ssh and in local network) of 24
sequential snapshots, and simultaneously deleting the snapshot, (in the same
file system, but in a different subvolume), there has been an file access error,
and the file system has corrupt.

Both scrub, both recovery and clear_cache mount options, both btrfsck, have
failed, the file system is left in a state unusable.

After reformatting the filesystem, remote receive of 24 snapshots worked properly.

The file system is used exclusively for receive the snapshot, it is composed of
a single device.
The initial snapshot is a linux installation of 50Gb.


I think that there was a race condition between the receive and deletion of
snapshots (that were performed on two different subvolume).


Best regards.

gdb

----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

from /var/log/messages

Nov 20 20:04:29 exnetold kernel: Modules linked in: fuse bridge ebtable_filter 
ebtables tun 8021q bnx2fc cnic uio garp mrp fcoe stp llc libfcoe libfc 
scsi_transport_fc nvidia_modeset(POE) nf_log_ipv4 nf_log_common xt_LOG 
xt_limit xt_multiport ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 
nf_defrag_ipv6 xt_CHECKSUM xt_conntrack iptable_mangle ipt_MASQUERADE 
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6_tables hwmon_vid btrfs 
snd_hda_codec_analog xor snd_hda_codec_generic raid6_pq snd_hda_codec_hdmi 
powernow_k8 snd_hda_intel kvm_amd snd_hda_codec nvidia(POE) kvm snd_hda_core 
snd_hwdep ppdev irqbypass snd_seq snd_seq_device snd_pcm amd64_edac_mod 
edac_mce_amd edac_core snd_timer snd drm parport_pc k8temp soundcore shpchp 
parport asus_atk0110 i2c_nforce2
Nov 20 20:04:29 exnetold kernel: acpi_cpufreq tpm_tis tpm vboxnetadp(OE) 
vboxnetflt(OE) vboxdrv(OE) binfmt_misc ata_generic pata_acpi serio_raw sata_nv 
forcedeth fjes analog gameport joydev i2c_dev
Nov 20 20:04:29 exnetold kernel: CPU: 1 PID: 16330 Comm: kworker/u4:0 Tainted: 
P           OE   4.4.6-301.fc23.x86_64 #1
Nov 20 20:04:29 exnetold kernel: Hardware name: System manufacturer System 
Product Name/M2N, BIOS 0902    02/16/2009
Nov 20 20:04:29 exnetold kernel: Workqueue: btrfs-extent-refs 
btrfs_extent_refs_helper [btrfs]
Nov 20 20:04:29 exnetold kernel: 0000000000000286 0000000061a6ce66 
ffff88008c797ae0 ffffffff813b542e
Nov 20 20:04:29 exnetold kernel: 0000000000000000 ffffffffa0c944d2 ffff88008c797b18 
ffffffff810a40f2
Nov 20 20:04:29 exnetold kernel: 00000006e7a80000 00000000fffffffe 
0000000000000000 ffff88004edc0000
Nov 20 20:04:29 exnetold kernel: Call Trace:
Nov 20 20:04:29 exnetold kernel: [<ffffffff813b542e>] dump_stack+0x63/0x85
Nov 20 20:04:29 exnetold kernel: [<ffffffff810a40f2>] 
warn_slowpath_common+0x82/0xc0
Nov 20 20:04:29 exnetold kernel: [<ffffffff810a423a>] warn_slowpath_null+0x1a/0x20
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bf8148>] 
__btrfs_free_extent.isra.69+0x898/0xd30 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0be99e6>] ? btrfs_free_path+0x26/0x30 
[btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bf6601>] ? 
__btrfs_inc_extent_ref.isra.53+0x11/0x270 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bfc0fc>] 
__btrfs_run_delayed_refs+0xa7c/0x11a0 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bff4f2>] 
btrfs_run_delayed_refs+0x82/0x2a0 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0bff747>] 
delayed_ref_async_start+0x37/0x90 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0c45f42>] 
btrfs_scrubparity_helper+0xc2/0x2e0 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bae3d>] ? 
pwq_activate_delayed_work+0x3d/0x90
Nov 20 20:04:29 exnetold kernel: [<ffffffffa0c4619e>] 
btrfs_extent_refs_helper+0xe/0x10 [btrfs]
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bc596>] process_one_work+0x156/0x430
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bc8be>] worker_thread+0x4e/0x450
Nov 20 20:04:29 exnetold kernel: [<ffffffff810bc870>] ? 
process_one_work+0x430/0x430
Nov 20 20:04:29 exnetold kernel: [<ffffffff810c2648>] kthread+0xd8/0xf0
Nov 20 20:04:29 exnetold kernel: [<ffffffff810c2570>] ? 
kthread_worker_fn+0x160/0x160
Nov 20 20:04:29 exnetold kernel: [<ffffffff817a090f>] ret_from_fork+0x3f/0x70
Nov 20 20:04:29 exnetold kernel: [<ffffffff810c2570>] ? 
kthread_worker_fn+0x160/0x160
Nov 20 20:04:29 exnetold kernel: ---[ end trace 4a87ca727985e15e ]---
Nov 20 20:04:29 exnetold kernel: BTRFS info (device sda1): leaf 95421874176 
total ptrs 209 free space 3459
Nov 20 20:04:29 exnetold kernel: #011item 0 key (29654663168 169 0) itemoff 
16250 itemsize 33
Nov 20 20:04:29 exnetold kernel: #011#011extent refs 1 gen 400 flags 258
8717 itemsize 33
...... long text ......
Nov 20 20:04:30 exnetold kernel: #011#011extent refs 1 gen 473 flags 258
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent 
29657972736
Nov 20 20:04:30 exnetold kernel: #011item 208 key (29658071040 169 0) itemoff 
8684 itemsize 33
Nov 20 20:04:30 exnetold kernel: #011#011extent refs 1 gen 473 flags 258
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent 
29657972736
Nov 20 20:04:30 exnetold kernel: BTRFS error (device sda1): unable to find ref 
byte nr 29656350720 parent 0 root 448  owner 1 offset 0
Nov 20 20:04:30 exnetold kernel: ------------[ cut here ]------------
Nov 20 20:04:30 exnetold kernel: WARNING: CPU: 1 PID: 16330 at 
fs/btrfs/extent-tree.c:6549 __btrfs_free_extent.isra.69+0x8ff/0xd30 [btrfs]()
Nov 20 20:04:30 exnetold kernel: Modules linked in: fuse bridge ebtable_filter 
ebtables tun 8021q bnx2fc cnic uio garp mrp fcoe stp llc libfcoe libfc 
scsi_transport_fc nvidia_modeset(POE) nf_log_ipv4 nf_log_common xt_LOG 
xt_limit xt_multiport ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 
nf_defrag_ipv6 xt_CHECKSUM xt_conntrack iptable_mangle ipt_MASQUERADE 
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6_tables hwmon_vid btrfs 
snd_hda_codec_analog xor snd_hda_codec_generic raid6_pq snd_hda_codec_hdmi 
powernow_k8 snd_hda_intel kvm_amd snd_hda_codec nvidia(POE) kvm snd_hda_core 
snd_hwdep ppdev irqbypass snd_seq snd_seq_device snd_pcm amd64_edac_mod 
edac_mce_amd edac_core snd_timer snd drm parport_pc k8temp soundcore shpchp 
parport asus_atk0110 i2c_nforce2
Nov 20 20:04:30 exnetold kernel: acpi_cpufreq tpm_tis tpm vboxnetadp(OE) 
vboxnetflt(OE) vboxdrv(OE) binfmt_misc ata_generic pata_acpi serio_raw sata_nv 
forcedeth fjes analog gameport joydev i2c_dev
Nov 20 20:04:30 exnetold kernel: CPU: 1 PID: 16330 Comm: kworker/u4:0 Tainted: 
P        W  OE   4.4.6-301.fc23.x86_64 #1
Nov 20 20:04:30 exnetold kernel: Hardware name: System manufacturer System 
Product Name/M2N, BIOS 0902    02/16/2009
Nov 20 20:04:30 exnetold kernel: Workqueue: btrfs-extent-refs 
btrfs_extent_refs_helper [btrfs]
Nov 20 20:04:30 exnetold kernel: 0000000000000286 0000000061a6ce66 
ffff88008c797a88 ffffffff813b542e
Nov 20 20:04:30 exnetold kernel: ffff88008c797ad0 ffffffffa0c944d2 ffff88008c797ac0 
ffffffff810a40f2
Nov 20 20:04:30 exnetold kernel: 00000006e7a80000 00000000fffffffe 
0000000000000000 ffff88004edc0000
Nov 20 20:04:30 exnetold kernel: Call Trace:
Nov 20 20:04:30 exnetold kernel: [<ffffffff813b542e>] dump_stack+0x63/0x85
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a40f2>] 
warn_slowpath_common+0x82/0xc0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a418c>] warn_slowpath_fmt+0x5c/0x80
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bf81af>] 
__btrfs_free_extent.isra.69+0x8ff/0xd30 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0be99e6>] ? btrfs_free_path+0x26/0x30 
[btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bf6601>] ? 
__btrfs_inc_extent_ref.isra.53+0x11/0x270 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bfc0fc>] 
__btrfs_run_delayed_refs+0xa7c/0x11a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff4f2>] 
btrfs_run_delayed_refs+0x82/0x2a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff747>] 
delayed_ref_async_start+0x37/0x90 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c45f42>] 
btrfs_scrubparity_helper+0xc2/0x2e0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bae3d>] ? 
pwq_activate_delayed_work+0x3d/0x90
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c4619e>] 
btrfs_extent_refs_helper+0xe/0x10 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc596>] process_one_work+0x156/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc8be>] worker_thread+0x4e/0x450
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc870>] ? 
process_one_work+0x430/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2648>] kthread+0xd8/0xf0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ? 
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: [<ffffffff817a090f>] ret_from_fork+0x3f/0x70
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ? 
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: ---[ end trace 4a87ca727985e15f ]---
Nov 20 20:04:30 exnetold kernel: BTRFS warning (device sda1): 
__btrfs_free_extent:6549: Aborting unused transaction(No such entry).
Nov 20 20:04:30 exnetold kernel: BTRFS warning (device sda1): 
btrfs_run_delayed_refs:2927: Aborting unused transaction(No such entry).
Nov 20 20:04:30 exnetold kernel: ------------[ cut here ]------------
Nov 20 20:04:30 exnetold kernel: WARNING: CPU: 1 PID: 16330 at 
fs/btrfs/extent-tree.c:6543 __btrfs_free_extent.isra.69+0x898/0xd30 [btrfs]()
Nov 20 20:04:30 exnetold kernel: Modules linked in: fuse bridge ebtable_filter 
ebtables tun 8021q bnx2fc cnic uio garp mrp fcoe stp llc libfcoe libfc 
scsi_transport_fc nvidia_modeset(POE) nf_log_ipv4 nf_log_common xt_LOG 
xt_limit xt_multiport ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 
nf_defrag_ipv6 xt_CHECKSUM xt_conntrack iptable_mangle ipt_MASQUERADE 
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6_tables hwmon_vid btrfs 
snd_hda_codec_analog xor snd_hda_codec_generic raid6_pq snd_hda_codec_hdmi 
powernow_k8 snd_hda_intel kvm_amd snd_hda_codec nvidia(POE) kvm snd_hda_core 
snd_hwdep ppdev irqbypass snd_seq snd_seq_device snd_pcm amd64_edac_mod 
edac_mce_amd edac_core snd_timer snd drm parport_pc k8temp soundcore shpchp 
parport asus_atk0110 i2c_nforce2
Nov 20 20:04:30 exnetold kernel: acpi_cpufreq tpm_tis tpm vboxnetadp(OE) 
vboxnetflt(OE) vboxdrv(OE) binfmt_misc ata_generic pata_acpi serio_raw sata_nv 
forcedeth fjes analog gameport joydev i2c_dev
Nov 20 20:04:30 exnetold kernel: CPU: 1 PID: 16330 Comm: kworker/u4:0 Tainted: 
P        W  OE   4.4.6-301.fc23.x86_64 #1
Nov 20 20:04:30 exnetold kernel: Hardware name: System manufacturer System 
Product Name/M2N, BIOS 0902    02/16/2009
Nov 20 20:04:30 exnetold kernel: Workqueue: btrfs-extent-refs 
btrfs_extent_refs_helper [btrfs]
Nov 20 20:04:30 exnetold kernel: 0000000000000286 0000000061a6ce66 
ffff88008c797ae0 ffffffff813b542e
Nov 20 20:04:30 exnetold kernel: 0000000000000000 ffffffffa0c944d2 ffff88008c797b18 
ffffffff810a40f2
Nov 20 20:04:30 exnetold kernel: 00000006e813c000 00000000fffffffe 
0000000000000000 ffff88004edc0000
Nov 20 20:04:30 exnetold kernel: Call Trace:
Nov 20 20:04:30 exnetold kernel: [<ffffffff813b542e>] dump_stack+0x63/0x85
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a40f2>] 
warn_slowpath_common+0x82/0xc0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810a423a>] warn_slowpath_null+0x1a/0x20
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bf8148>] 
__btrfs_free_extent.isra.69+0x898/0xd30 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810fa10a>] ? console_unlock+0x20a/0x540
Nov 20 20:04:30 exnetold kernel: [<ffffffff810dabc2>] ? enqueue_entity+0x3a2/0xc90
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bfc0fc>] 
__btrfs_run_delayed_refs+0xa7c/0x11a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff8104f57d>] ? 
native_smp_send_reschedule+0x4d/0x70
Nov 20 20:04:30 exnetold kernel: [<ffffffff810cc1c5>] ? resched_curr+0x65/0xc0
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff4f2>] 
btrfs_run_delayed_refs+0x82/0x2a0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0bff747>] 
delayed_ref_async_start+0x37/0x90 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c45f42>] 
btrfs_scrubparity_helper+0xc2/0x2e0 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bae3d>] ? 
pwq_activate_delayed_work+0x3d/0x90
Nov 20 20:04:30 exnetold kernel: [<ffffffffa0c4619e>] 
btrfs_extent_refs_helper+0xe/0x10 [btrfs]
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc596>] process_one_work+0x156/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc8be>] worker_thread+0x4e/0x450
Nov 20 20:04:30 exnetold kernel: [<ffffffff810bc870>] ? 
process_one_work+0x430/0x430
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2648>] kthread+0xd8/0xf0
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ? 
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: [<ffffffff817a090f>] ret_from_fork+0x3f/0x70
Nov 20 20:04:30 exnetold kernel: [<ffffffff810c2570>] ? 
kthread_worker_fn+0x160/0x160
Nov 20 20:04:30 exnetold kernel: ---[ end trace 4a87ca727985e160 ]---
Nov 20 20:04:30 exnetold kernel: BTRFS info (device sda1): leaf 95549276160 
total ptrs 230 free space 270
Nov 20 20:04:30 exnetold kernel: #011item 0 key (29662576640 169 0) itemoff 
16250 itemsize 33
Nov 20 20:04:30 exnetold kernel: #011#011extent refs 1 gen 473 flags 258
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent 
29657972736
...... long text ......
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent 
29744676864
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent 
29637902336
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent 
21741715456
Nov 20 20:04:30 exnetold kernel: #011#011shared block backref parent 
21556609024
Nov 20 20:04:30 exnetold kernel: BTRFS error (device sda1): unable to find ref 
byte nr 29663412224 parent 0 root 448  owner 1 offset 0
Nov 20 20:04:30 exnetold kernel: BTRFS: error (device sda1) in 
__btrfs_free_extent:6549: errno=-2 No such entry
Nov 20 20:04:31 exnetold kernel: BTRFS info (device sda1): forced readonly
Nov 20 20:04:31 exnetold kernel: BTRFS: error (device sda1) in 
btrfs_run_delayed_refs:2927: errno=-2 No such entry
................
Nov 20 20:04:44 exnetold kernel: BTRFS error (device sda1): cleaner 
transaction attach returned -30



In data lunedì 21 novembre 2016 13:09:15, bepi@adria.it ha scritto:
> Hi.
> 
> My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64
> btrfs-progs-4.4.1-1.fc23.x86_64
> 
> Testing the remote differential receive (via ssh and in local network) of 24
> sequential snapshots, and simultaneously deleting the snapshot, (in the
> same file system, but in a different subvolume), there has been an file
> access error, and the file system has corrupt.
> 
> Both scrub, both recovery and clear_cache mount options, both btrfsck, have
> failed, the file system is left in a state unusable.
> 
> After reformatting the filesystem, remote receive of 24 snapshots worked
> properly.
> 
> The file system is used exclusively for receive the snapshot, it is composed
> of a single device.
> The initial snapshot is a linux installation of 50Gb.
> 
> 
> I think that there was a race condition between the receive and deletion of
> snapshots (that were performed on two different subvolume).
> 
> 
> Best regards.
> 
> gdb
> 
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Chris Murphy]

I haven't seen this with 4.7.10. I suggest running 'btrfs check'
(without repair) using a recent btrfs-progs. You can find 4.8.3 in
koji, just download the appropriate rpm, and 'dnf update *rpm'

As for kernel, Fedora 23 has 4.8.8 in updates (stable) and 4.8.10 is
in updates-testing, I suggest moving to one of those.


Chris Murphy

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

I have reinstalled from scratch my system (for remote receiving of the 
snapshot).
Now it uses the 4.8.8-100.fc23.x86_64 kernel, and now I have a separate 
partition for '''possible dangerous''' test.

I do not remember the exact sequence that caused the corruption of the file 
system, but now it worked without problem.

If I have problems again, I will update the btrfs-tools and I will try to fix.

P.S The silent and unrecoverable corruption are not entirely friendly ( :) ).

Thank you for answer.

Gdb


> I haven't seen this with 4.7.10. I suggest running 'btrfs check'
> (without repair) using a recent btrfs-progs. You can find 4.8.3 in
> koji, just download the appropriate rpm, and 'dnf update *rpm'
> 
> As for kernel, Fedora 23 has 4.8.8 in updates (stable) and 4.8.10 is
> in updates-testing, I suggest moving to one of those.
> 
> 
> Chris Murphy

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

> I haven't seen this with 4.7.10. I suggest running 'btrfs check'
> (without repair) using a recent btrfs-progs. You can find 4.8.3 in
> koji, just download the appropriate rpm, and 'dnf update *rpm'
> 
> As for kernel, Fedora 23 has 4.8.8 in updates (stable) and 4.8.10 is
> in updates-testing, I suggest moving to one of those.
> 
> 
> Chris Murphy

Same problem, this time on a local subvolume.
 
kernel-4.8.8-100.fc23.x86_64

btrfs-progs v4.8.5
 
P.S. Do you want to do some debugging?
     Otherwise I proceed with the reset of the partition.
	 My feeling is that in the btrfs receive function there is a random bug,
     which makes the file system unusable.
 
 
btrfs check /dev/sda2 
Checking filesystem on /dev/sda2 
UUID: d49d8f95-84ee-4452-9252-ad030a083739 
checking extents 
ref mismatch on [312324046848 16384] extent item 0, found 1 
Backref 312324046848 parent 1309 root 1309 not found in extent tree 
backpointer mismatch on [312324046848 16384] 
owner ref check failed [312324046848 16384] 
ref mismatch on [322595897344 16384] extent item 0, found 1 
Backref 322595897344 parent 1309 root 1309 not found in extent tree 
backpointer mismatch on [322595897344 16384] 
owner ref check failed [322595897344 16384] 
checking free space cache 
checking fs roots 
warning line 3822 
checking csums 
checking root refs 
found 162417897472 bytes used err is 0 
total csum bytes: 152292600 
total tree bytes: 6461636608 
total fs tree bytes: 6010830848 
total extent tree bytes: 281214976 
btree space waste bytes: 1142051927 
file data blocks allocated: 6628422606848 
referenced 961225981952

dmesg
 
[11265.556898] Call Trace: 
[11265.556908]  [<ffffffff923e493e>] dump_stack+0x63/0x85 
[11265.556916]  [<ffffffff920a0ecb>] __warn+0xcb/0xf0 
[11265.556919]  [<ffffffff920a0ffd>] warn_slowpath_null+0x1d/0x20 
[11265.556948]  [<ffffffffc0351504>] __btrfs_free_extent.isra.69+0x864/0xcc0 
[btrfs] 
[11265.556978]  [<ffffffffc0355cce>] __btrfs_run_delayed_refs+0xace/0x1220 [btrfs] 
[11265.557006]  [<ffffffffc03535dc>] ? btrfs_set_disk_extent_flags+0x7c/0xb0 
[btrfs] 
[11265.557034]  [<ffffffffc03592f3>] btrfs_run_delayed_refs+0x93/0x2b0 [btrfs] 
[11265.557077]  [<ffffffffc0395e5b>] ? free_extent_buffer+0x4b/0xa0 [btrfs] 
[11265.557112]  [<ffffffffc036e87a>] btrfs_should_end_transaction+0x5a/0x60 
[btrfs] 
[11265.557139]  [<ffffffffc0357a69>] btrfs_drop_snapshot+0x429/0x800 [btrfs] 
[11265.557172]  [<ffffffffc036fba9>] btrfs_clean_one_deleted_snapshot+0xa9/0xf0 
[btrfs] 
[11265.557202]  [<ffffffffc036650d>] cleaner_kthread+0x15d/0x1d0 [btrfs] 
[11265.557229]  [<ffffffffc03663b0>] ? btrfs_destroy_pinned_extent+0xf0/0xf0 
[btrfs] 
[11265.557235]  [<ffffffff920c0bf8>] kthread+0xd8/0xf0 
[11265.557242]  [<ffffffff927ffcbf>] ret_from_fork+0x1f/0x40 
[11265.557247]  [<ffffffff920c0b20>] ? kthread_worker_fn+0x170/0x170 
[11265.557282] ---[ end trace 577009d8a389f458 ]--- 
[11265.557288] BTRFS info (device sda2): leaf 327108411392 total ptrs 190 free 
space 88 
[11265.557292]  item 0 key (312323358720 169 0) itemoff 16142 itemsize 141 
[11265.557294]          extent refs 13 gen 2089 flags 258 
[11265.557295]          tree block backref root 1326 
[11265.557297]          tree block backref root 1324
..... long text ....
[11265.558885]          tree block backref root 1314 
[11265.558886]          shared block backref parent 322595897344 
[11265.558887]          shared block backref parent 322309390336 
[11265.558889]          shared block backref parent 322171060224 
[11265.558890]          shared block backref parent 319298207744 
[11265.558891]          shared block backref parent 312326356992 
[11265.558895] BTRFS error (device sda2): unable to find ref byte nr 
312324046848 parent 0 root 1309  owner 1 offset 0 
[11265.558901] ------------[ cut here ]------------ 
[11265.558931] WARNING: CPU: 3 PID: 6915 at fs/btrfs/extent-tree.c:6951 
__btrfs_free_extent.isra.69+0x8c8/0xcc0 [btrfs] 
[11265.558932] BTRFS: Transaction aborted (error -2)
.....  ....
[11265.559095] Call Trace: 
[11265.559102]  [<ffffffff923e493e>] dump_stack+0x63/0x85 
[11265.559108]  [<ffffffff920a0ecb>] __warn+0xcb/0xf0 
[11265.559114]  [<ffffffff920a0f4f>] warn_slowpath_fmt+0x5f/0x80 
[11265.559141]  [<ffffffffc0351568>] __btrfs_free_extent.isra.69+0x8c8/0xcc0 
[btrfs] 
[11265.559170]  [<ffffffffc0355cce>] __btrfs_run_delayed_refs+0xace/0x1220 [btrfs] 
[11265.559197]  [<ffffffffc03535dc>] ? btrfs_set_disk_extent_flags+0x7c/0xb0 
[btrfs] 
[11265.559225]  [<ffffffffc03592f3>] btrfs_run_delayed_refs+0x93/0x2b0 [btrfs] 
[11265.559265]  [<ffffffffc0395e5b>] ? free_extent_buffer+0x4b/0xa0 [btrfs] 
[11265.559298]  [<ffffffffc036e87a>] btrfs_should_end_transaction+0x5a/0x60 
[btrfs] 
[11265.559324]  [<ffffffffc0357a69>] btrfs_drop_snapshot+0x429/0x800 [btrfs] 
[11265.559355]  [<ffffffffc036fba9>] btrfs_clean_one_deleted_snapshot+0xa9/0xf0 
[btrfs] 
[11265.559384]  [<ffffffffc036650d>] cleaner_kthread+0x15d/0x1d0 [btrfs] 
[11265.559411]  [<ffffffffc03663b0>] ? btrfs_destroy_pinned_extent+0xf0/0xf0 
[btrfs] 
[11265.559416]  [<ffffffff920c0bf8>] kthread+0xd8/0xf0 
[11265.559423]  [<ffffffff927ffcbf>] ret_from_fork+0x1f/0x40 
[11265.559428]  [<ffffffff920c0b20>] ? kthread_worker_fn+0x170/0x170 
[11265.559431] ---[ end trace 577009d8a389f459 ]--- 
[11265.559434] BTRFS: error (device sda2) in __btrfs_free_extent:6951: 
errno=-2 No such entry 
[11265.559439] BTRFS info (device sda2): forced readonly 
[11265.559444] BTRFS: error (device sda2) in btrfs_run_delayed_refs:2960: 
errno=-2 No such entry 
[11265.805418] BTRFS error (device sda2): cleaner transaction attach returned 
-30

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

> Same problem, this time on a local subvolume.
> 
> kernel-4.8.8-100.fc23.x86_64
> 
> btrfs-progs v4.8.5
]zac[

I had three filesystem corruption.

The point at which the problem it appeared, is similar in all three cases.

Subvolume structure and operations sequence:

btrfsreceive/
btrfsreceive/root/
btrfsreceive/root/.part/

1) Sending XYZ differential snapshot in to ' btrfsreceive/root/.part/ '.
2) Create snapshot from ' btrfsreceive/root/.part/XYZ ' to ' btrfsreceive/root 
/XYZ '.
3) Delete snapshot ' btrfsreceive/root/.part/XYZ '.

Always in step 2) I had two files unreadable error (view previous posts), and 
one already existing object error (see below).

All three times I had to re-create from scratch the various partitions (on 
disks and systems different).

I can help you, in some way, to find the problem?

Or is useless to continue report it?



dic 18 18:29:58 exnetold.gdb.it kernel: ------------[ cut here ]------------
dic 18 18:29:58 exnetold.gdb.it kernel: WARNING: CPU: 1 PID: 4325 at 
fs/btrfs/extent-tree.c:2960 btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: Transaction aborted (error -17)
dic 18 18:29:58 exnetold.gdb.it kernel: Modules linked in: fuse xt_CHECKSUM 
ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns 
nf_conntrack_br
dic 18 18:29:58 exnetold.gdb.it kernel:  soundcore acpi_cpufreq tpm_tis 
tpm_tis_core tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic 
nouveau vide
dic 18 18:29:58 exnetold.gdb.it kernel: CPU: 1 PID: 4325 Comm: umount Tainted: 
G        W       4.8.8-100.fc23.x86_64 #1
dic 18 18:29:58 exnetold.gdb.it kernel: Hardware name: System manufacturer 
System Product Name/M2N, BIOS 0902    02/16/2009
dic 18 18:29:58 exnetold.gdb.it kernel:  0000000000000286 00000000dd260fac 
ffff8ffa0d25bb60 ffffffffbc3e493e
dic 18 18:29:58 exnetold.gdb.it kernel:  ffff8ffa0d25bbb0 0000000000000000 
ffff8ffa0d25bba0 ffffffffbc0a0ecb
dic 18 18:29:58 exnetold.gdb.it kernel:  00000b9000000049 ffff8ff9e61b40a0 
ffff8ffa2da77800 ffffffffffffffff
dic 18 18:29:58 exnetold.gdb.it kernel: Call Trace:
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc3e493e>] dump_stack+0x63/0x85
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc0a0ecb>] __warn+0xcb/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc0a0f4f>] 
warn_slowpath_fmt+0x5f/0x80
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07eb4e3>] 
btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07d62ec>] ? 
btrfs_cow_block+0x10c/0x1e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07ff62e>] 
commit_cowonly_roots+0xae/0x2e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07eb466>] ? 
btrfs_run_delayed_refs+0x206/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc08706b4>] ? 
btrfs_qgroup_account_extents+0x84/0x180 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc0802187>] 
btrfs_commit_transaction+0x547/0xa40 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07faa9f>] 
btrfs_commit_super+0x8f/0xa0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07fcbcb>] 
close_ctree+0x2db/0x380 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc26d3da>] ? 
evict_inodes+0x15a/0x180
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07ccf29>] 
btrfs_put_super+0x19/0x20 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc2520bf>] 
generic_shutdown_super+0x6f/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc2523b2>] 
kill_anon_super+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffc07cdd98>] 
btrfs_kill_super+0x18/0x110 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc252763>] 
deactivate_locked_super+0x43/0x70
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc2527ec>] 
deactivate_super+0x5c/0x60
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc2711bf>] 
cleanup_mnt+0x3f/0x90
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc271252>] 
__cleanup_mnt+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc0bf0ce>] 
task_work_run+0x7e/0xa0
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc0032d2>] 
exit_to_usermode_loop+0xc2/0xd0
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc003bf1>] 
syscall_return_slowpath+0xa1/0xb0
dic 18 18:29:58 exnetold.gdb.it kernel:  [<ffffffffbc7ffb3a>] 
entry_SYSCALL_64_fastpath+0xa2/0xa4
dic 18 18:29:58 exnetold.gdb.it kernel: ---[ end trace f7eb2e818f727168 ]---
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in 
btrfs_run_delayed_refs:2960: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS info (device sda3): forced 
readonly
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS warning (device sda3): Skipping 
commit of aborted transaction.
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in 
cleanup_transaction:1854: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): commit 
super ret -17
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): cleaner 
transaction attach returned -30

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Chris Murphy]

I'd say backup the volume conventionally to get important data
secured. Then use btrfs check --repair to fix the problems with the
file system, and then try the btrfs send receive again.

It most certainly is a bug, but it's not clear to me what the cause
is; other than the kernel isn't handling the problem on-disk
gracefully. Maybe repair will fix it. Another possibility is to move
to kernel 4.9.0 and see if it still reproduces. Per usual, there's a
ton of bug fixes in each kernel release.

Otherwise I'm out of ideas.

Chris Murphy

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

Hi Giuseppe,

Would you like to tell some details about:
1. the XYZ snapshot was taken from which subvolume
2. where the base (initial) snapshot is stored
3. The 3 partitions receives the same snapshot, are they in the same btrfs configuration and subvol structure?

Also, would you send the link reports "two files unreadable error" post mentioned in step 2? 
Hope can see the message and figure out if the issue first comes from sender or receiver side. 

Thanks,
Xin
 

Sent: Sunday, December 18, 2016 at 11:59 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
> Same problem, this time on a local subvolume.
>
> kernel-4.8.8-100.fc23.x86_64
>
> btrfs-progs v4.8.5
]zac[

I had three filesystem corruption.

The point at which the problem it appeared, is similar in all three cases.

Subvolume structure and operations sequence:

btrfsreceive/
btrfsreceive/root/
btrfsreceive/root/.part/

1) Sending XYZ differential snapshot in to ' btrfsreceive/root/.part/ '.
2) Create snapshot from ' btrfsreceive/root/.part/XYZ ' to ' btrfsreceive/root
/XYZ '.
3) Delete snapshot ' btrfsreceive/root/.part/XYZ '.

Always in step 2) I had two files unreadable error (view previous posts), and
one already existing object error (see below).

All three times I had to re-create from scratch the various partitions (on
disks and systems different).

I can help you, in some way, to find the problem?

Or is useless to continue report it?



dic 18 18:29:58 exnetold.gdb.it kernel: ------------[ cut here ]------------
dic 18 18:29:58 exnetold.gdb.it kernel: WARNING: CPU: 1 PID: 4325 at
fs/btrfs/extent-tree.c:2960 btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: Transaction aborted (error -17)
dic 18 18:29:58 exnetold.gdb.it kernel: Modules linked in: fuse xt_CHECKSUM
ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
nf_conntrack_br
dic 18 18:29:58 exnetold.gdb.it kernel: soundcore acpi_cpufreq tpm_tis
tpm_tis_core tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic
nouveau vide
dic 18 18:29:58 exnetold.gdb.it kernel: CPU: 1 PID: 4325 Comm: umount Tainted:
G W 4.8.8-100.fc23.x86_64 #1
dic 18 18:29:58 exnetold.gdb.it kernel: Hardware name: System manufacturer
System Product Name/M2N, BIOS 0902 02/16/2009
dic 18 18:29:58 exnetold.gdb.it kernel: 0000000000000286 00000000dd260fac
ffff8ffa0d25bb60 ffffffffbc3e493e
dic 18 18:29:58 exnetold.gdb.it kernel: ffff8ffa0d25bbb0 0000000000000000
ffff8ffa0d25bba0 ffffffffbc0a0ecb
dic 18 18:29:58 exnetold.gdb.it kernel: 00000b9000000049 ffff8ff9e61b40a0
ffff8ffa2da77800 ffffffffffffffff
dic 18 18:29:58 exnetold.gdb.it kernel: Call Trace:
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc3e493e>] dump_stack+0x63/0x85
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0ecb>] __warn+0xcb/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0f4f>]
warn_slowpath_fmt+0x5f/0x80
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb4e3>]
btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07d62ec>] ?
btrfs_cow_block+0x10c/0x1e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ff62e>]
commit_cowonly_roots+0xae/0x2e0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb466>] ?
btrfs_run_delayed_refs+0x206/0x2b0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc08706b4>] ?
btrfs_qgroup_account_extents+0x84/0x180 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc0802187>]
btrfs_commit_transaction+0x547/0xa40 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07faa9f>]
btrfs_commit_super+0x8f/0xa0 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07fcbcb>]
close_ctree+0x2db/0x380 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc26d3da>] ?
evict_inodes+0x15a/0x180
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ccf29>]
btrfs_put_super+0x19/0x20 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2520bf>]
generic_shutdown_super+0x6f/0xf0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2523b2>]
kill_anon_super+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07cdd98>]
btrfs_kill_super+0x18/0x110 [btrfs]
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc252763>]
deactivate_locked_super+0x43/0x70
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2527ec>]
deactivate_super+0x5c/0x60
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2711bf>]
cleanup_mnt+0x3f/0x90
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc271252>]
__cleanup_mnt+0x12/0x20
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0bf0ce>]
task_work_run+0x7e/0xa0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0032d2>]
exit_to_usermode_loop+0xc2/0xd0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc003bf1>]
syscall_return_slowpath+0xa1/0xb0
dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc7ffb3a>]
entry_SYSCALL_64_fastpath+0xa2/0xa4
dic 18 18:29:58 exnetold.gdb.it kernel: ---[ end trace f7eb2e818f727168 ]---
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
btrfs_run_delayed_refs:2960: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS info (device sda3): forced
readonly
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS warning (device sda3): Skipping
commit of aborted transaction.
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
cleanup_transaction:1854: errno=-17 Object already exists
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): commit
super ret -17
dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): cleaner
transaction attach returned -30

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[bepi]

Hi.

It is a bit complex.


Primary system

subvolume on SSD devices on PCIe slot

/root/ (fedora 23, 50GB usati)

/btrfssnapshot/
/btrfssnapshot/root/  (for /root/ snapshot)
/btrfssnapshot/root/root.1
/btrfssnapshot/root/root.2
/btrfssnapshot/root/root.XYZ


subvolume on device HDD "1" sata

/data_storage/  (data, 100GB usati)
/data_backup/   (backup tar files, programs, downloads, etc., used 250GB)

/btrfssnapshot/
/btrfssnapshot/data_storage/  (for /data_storage/ snapshot)
/btrfssnapshot/data_backup/   (for /data_backup/ snapshot)
/btrfssnapshot/data_storage/data_storage.1
/btrfssnapshot/data_storage/data_storage.2
/btrfssnapshot/data_storage/data_storage.XYZ
/btrfssnapshot/data_backup/data_backup.1
/btrfssnapshot/data_backup/data_backup.2
/btrfssnapshot/data_backup/data_backup.XYZ


subvolume on HDD device "2" sata

partition 1

/btrfsreceive/root/  (for receive /btrfssnapshot/root/ snapshot)
/btrfsreceive/root/.part/

partition 2

/btrfsreceive/data_storage/  (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/  (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/



Secondary system for receiving snapshot

subvolume on HDD device "3" sata

partition 1

/btrfsreceive/root/
/btrfsreceive/root/.part/

partition 2

/btrfsreceive/data_storage/  (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/  (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/


My bash script create snapshot of /root/, /data_storage/ and /data_backup/ on
/btrfssnapshot/ .
Snapshot is created with .part extension, when the creation is finished
properly, is renamed to .1, .2, .XYZ .


My bash script send snapshot from /btrfssnapshot/root/,
/btrfssnapshot/data_storage/ and /btrfssnapshot/data_backup/ (sending
differential, n - 1 -> n) to the subvolume /btrfsreceive/root/,
/btrfsreceive/data_storage/ and /btrfsreceive/data_backup/.


My bash script sends the same snapshot also to the secondary system for
receiving snapshots (using ssh).


P.S. If I recreate the receiving partition from scratch, the receive working
properly.


The previous problems information are in this thread.
Please, you can read the thread?


Thanks to you.


Gdb



Scrive Xin Zhou <xin.zhou@gmx.com>:

> 
> Hi Giuseppe,
> 
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
> 
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2? 
> Hope can see the message and figure out if the issue first comes from sender
> or receiver side. 
> 
> Thanks,
> Xin
>  
> 
> Sent: Sunday, December 18, 2016 at 11:59 AM
> From: "Giuseppe Della Bianca" <bepi@adria.it>
> To: linux-btrfs@vger.kernel.org
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> > Same problem, this time on a local subvolume.
> >
> > kernel-4.8.8-100.fc23.x86_64
> >
> > btrfs-progs v4.8.5
> ]zac[
> 
> I had three filesystem corruption.
> 
> The point at which the problem it appeared, is similar in all three cases.
> 
> Subvolume structure and operations sequence:
> 
> btrfsreceive/
> btrfsreceive/root/
> btrfsreceive/root/.part/
> 
> 1) Sending XYZ differential snapshot in to ' btrfsreceive/root/.part/ '.
> 2) Create snapshot from ' btrfsreceive/root/.part/XYZ ' to '
> btrfsreceive/root
> /XYZ '.
> 3) Delete snapshot ' btrfsreceive/root/.part/XYZ '.
> 
> Always in step 2) I had two files unreadable error (view previous posts),
> and
> one already existing object error (see below).
> 
> All three times I had to re-create from scratch the various partitions (on
> disks and systems different).
> 
> I can help you, in some way, to find the problem?
> 
> Or is useless to continue report it?
> 
> 
> 
> dic 18 18:29:58 exnetold.gdb.it kernel: ------------[ cut here ]------------
> dic 18 18:29:58 exnetold.gdb.it kernel: WARNING: CPU: 1 PID: 4325 at
> fs/btrfs/extent-tree.c:2960 btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: Transaction aborted (error
> -17)
> dic 18 18:29:58 exnetold.gdb.it kernel: Modules linked in: fuse xt_CHECKSUM
> ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
> nf_conntrack_br
> dic 18 18:29:58 exnetold.gdb.it kernel: soundcore acpi_cpufreq tpm_tis
> tpm_tis_core tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic
> nouveau vide
> dic 18 18:29:58 exnetold.gdb.it kernel: CPU: 1 PID: 4325 Comm: umount
> Tainted:
> G W 4.8.8-100.fc23.x86_64 #1
> dic 18 18:29:58 exnetold.gdb.it kernel: Hardware name: System manufacturer
> System Product Name/M2N, BIOS 0902 02/16/2009
> dic 18 18:29:58 exnetold.gdb.it kernel: 0000000000000286 00000000dd260fac
> ffff8ffa0d25bb60 ffffffffbc3e493e
> dic 18 18:29:58 exnetold.gdb.it kernel: ffff8ffa0d25bbb0 0000000000000000
> ffff8ffa0d25bba0 ffffffffbc0a0ecb
> dic 18 18:29:58 exnetold.gdb.it kernel: 00000b9000000049 ffff8ff9e61b40a0
> ffff8ffa2da77800 ffffffffffffffff
> dic 18 18:29:58 exnetold.gdb.it kernel: Call Trace:
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc3e493e>]
> dump_stack+0x63/0x85
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0ecb>]
> __warn+0xcb/0xf0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0a0f4f>]
> warn_slowpath_fmt+0x5f/0x80
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb4e3>]
> btrfs_run_delayed_refs+0x283/0x2b0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07d62ec>] ?
> btrfs_cow_block+0x10c/0x1e0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ff62e>]
> commit_cowonly_roots+0xae/0x2e0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07eb466>] ?
> btrfs_run_delayed_refs+0x206/0x2b0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc08706b4>] ?
> btrfs_qgroup_account_extents+0x84/0x180 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc0802187>]
> btrfs_commit_transaction+0x547/0xa40 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07faa9f>]
> btrfs_commit_super+0x8f/0xa0 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07fcbcb>]
> close_ctree+0x2db/0x380 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc26d3da>] ?
> evict_inodes+0x15a/0x180
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07ccf29>]
> btrfs_put_super+0x19/0x20 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2520bf>]
> generic_shutdown_super+0x6f/0xf0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2523b2>]
> kill_anon_super+0x12/0x20
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffc07cdd98>]
> btrfs_kill_super+0x18/0x110 [btrfs]
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc252763>]
> deactivate_locked_super+0x43/0x70
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2527ec>]
> deactivate_super+0x5c/0x60
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc2711bf>]
> cleanup_mnt+0x3f/0x90
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc271252>]
> __cleanup_mnt+0x12/0x20
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0bf0ce>]
> task_work_run+0x7e/0xa0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc0032d2>]
> exit_to_usermode_loop+0xc2/0xd0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc003bf1>]
> syscall_return_slowpath+0xa1/0xb0
> dic 18 18:29:58 exnetold.gdb.it kernel: [<ffffffffbc7ffb3a>]
> entry_SYSCALL_64_fastpath+0xa2/0xa4
> dic 18 18:29:58 exnetold.gdb.it kernel: ---[ end trace f7eb2e818f727168 ]---
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
> btrfs_run_delayed_refs:2960: errno=-17 Object already exists
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS info (device sda3): forced
> readonly
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS warning (device sda3):
> Skipping
> commit of aborted transaction.
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS: error (device sda3) in
> cleanup_transaction:1854: errno=-17 Object already exists
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): commit
> super ret -17
> dic 18 18:29:58 exnetold.gdb.it kernel: BTRFS error (device sda3): cleaner
> transaction attach returned -30
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> 




----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[bepi]

(Resend)

Hi.

It is a bit complex.


Primary system

subvolume on SSD devices on PCIe slot

/root/ (fedora 23, 50GB usati)

/btrfssnapshot/
/btrfssnapshot/root/  (for /root/ snapshot)
/btrfssnapshot/root/root.1
/btrfssnapshot/root/root.2
/btrfssnapshot/root/root.XYZ


subvolume on device HDD "1" sata

/data_storage/  (data, 100GB usati)
/data_backup/   (backup tar files, programs, downloads, etc., used 250GB)

/btrfssnapshot/
/btrfssnapshot/data_storage/  (for /data_storage/ snapshot)
/btrfssnapshot/data_backup/   (for /data_backup/ snapshot)
/btrfssnapshot/data_storage/data_storage.1
/btrfssnapshot/data_storage/data_storage.2
/btrfssnapshot/data_storage/data_storage.XYZ
/btrfssnapshot/data_backup/data_backup.1
/btrfssnapshot/data_backup/data_backup.2
/btrfssnapshot/data_backup/data_backup.XYZ


subvolume on HDD device "2" sata

partition 1

/btrfsreceive/root/  (for receive /btrfssnapshot/root/ snapshot)
/btrfsreceive/root/.part/

partition 2

/btrfsreceive/data_storage/  (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/  (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/



Secondary system for receiving snapshot

subvolume on HDD device "3" sata

partition 1

/btrfsreceive/root/
/btrfsreceive/root/.part/

partition 2

/btrfsreceive/data_storage/  (for receive /btrfssnapshot/data_storage/ snapshot)
/btrfsreceive/data_storage/.part/
/btrfsreceive/data_backup/  (for receive /btrfssnapshot/data_backup/ snapshot)
/btrfsreceive/data_backup/.part/


My bash script create snapshot of /root/, /data_storage/ and /data_backup/ on
/btrfssnapshot/ .
Snapshot is created with .part extension, when the creation is finished
properly, is renamed to .1, .2, .XYZ .


My bash script send snapshot from /btrfssnapshot/root/,
/btrfssnapshot/data_storage/ and /btrfssnapshot/data_backup/ (sending
differential, n - 1 -> n) to the subvolume /btrfsreceive/root/,
/btrfsreceive/data_storage/ and /btrfsreceive/data_backup/.


My bash script sends the same snapshot also to the secondary system for
receiving snapshots (using ssh).


P.S. If I recreate the receiving partition from scratch, the receive working
properly.


The previous problems information are in this thread.
Please, you can read the thread?


Thanks to you.


Gdb

Scrive Xin Zhou <xin.zhou@gmx.com>:

> 
> Hi Giuseppe,
> 
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
> 
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2? 
> Hope can see the message and figure out if the issue first comes from sender
> or receiver side. 
> 
> Thanks,
> Xin
>  
> 


----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

a concrete example


SNAPSHOT

/dev/nvme0n1p2 on /tmp/tmp.X3vU6dLLVI type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)

btrfsManage SNAPSHOT /

(2016-12-19 19:44:00) Start btrfsManage
. . . Start managing SNAPSHOT ' / ' filesystem ' root ' snapshot

In ' btrfssnapshot ' latest source snapshot ' root-2016-12-18_15:10:01.40 '
. . . date ' 2016-12-18_15:10:01 ' number ' 40 '

Creation ' root-2016-12-19_19:44:00.part ' snapshot from ' root ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.X3vU6dLLVI/root' in '/tmp/tmp.X3vU6dLLVI/btrfssnapshot/root/root-2016-12-19_19:44:00.part'

Renaming ' root-2016-12-19_19:44:00.part ' into ' root-2016-12-19_19:44:00.41 ' snapshot

Source snapshot list of ' root ' subvolume
. . . btrfssnapshot/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfssnapshot/root/root-2016-12-19_19:44:00.41

(2016-12-19 19:44:05) End btrfsManage
. . . End managing SNAPSHOT ' / ' filesystem ' root ' snapshot
CORRECTLY



SEND e RECEIVE

/dev/nvme0n1p2 on /tmp/tmp.o78czE0Bo6 type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sda2 on /tmp/tmp.XcwqQCKq09 type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

btrfsManage SEND / /dev/sda2

(2016-12-19 19:47:24) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '

Sending ' root-2016-12-19_19:44:00.41 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41
. . . At snapshot root-2016-12-19_19:44:00.41

Creation ' root-2016-12-19_19:44:00.41 ' snapshot from ' .part/root-2016-12-19_19:44:00.41 ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41' in '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/root-2016-12-19_19:44:00.41'
. . . Delete subvolume (commit): '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41'

Snapshot list in ' /dev/sda2 ' device
. . . btrfsreceive/data_backup/data_backup-2016-12-17_12:07:00.1
. . . btrfsreceive/data_storage/data_storage-2016-12-10_17:05:51.1
. . . btrfsreceive/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfsreceive/root/root-2016-12-19_19:44:00.41

(2016-12-19 19:48:37) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
CORRECTLY



> Hi Giuseppe,
> 
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
> 
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2? Hope can see the message and figure out if the issue
> first comes from sender or receiver side.
> 
> Thanks,
> Xin
>  
> 

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

Hi,

The system seems running some customized scripts continuously backup data from a NVME drive to HDDs.
If the 3 HDDs backup storage are same in btrfs config, and the there is a bug in btrfs code,
they all suppose to fail after the same operation sequence.

Otherwise, probably one of the HDDs might have issue, or there is a bug in layer below btrfs.

For the customize script, it might be helpful to check the file system consistency after each transfer.
That might be useful to figure out which step generates a corruption, and if there is error propagations.

Regards,
Xin
 
 

Sent: Monday, December 19, 2016 at 10:55 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Xin Zhou" <xin.zhou@gmx.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
a concrete example


SNAPSHOT

/dev/nvme0n1p2 on /tmp/tmp.X3vU6dLLVI type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)

btrfsManage SNAPSHOT /

(2016-12-19 19:44:00) Start btrfsManage
. . . Start managing SNAPSHOT ' / ' filesystem ' root ' snapshot

In ' btrfssnapshot ' latest source snapshot ' root-2016-12-18_15:10:01.40 '
. . . date ' 2016-12-18_15:10:01 ' number ' 40 '

Creation ' root-2016-12-19_19:44:00.part ' snapshot from ' root ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.X3vU6dLLVI/root' in '/tmp/tmp.X3vU6dLLVI/btrfssnapshot/root/root-2016-12-19_19:44:00.part'

Renaming ' root-2016-12-19_19:44:00.part ' into ' root-2016-12-19_19:44:00.41 ' snapshot

Source snapshot list of ' root ' subvolume
. . . btrfssnapshot/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfssnapshot/root/root-2016-12-19_19:44:00.41

(2016-12-19 19:44:05) End btrfsManage
. . . End managing SNAPSHOT ' / ' filesystem ' root ' snapshot
CORRECTLY



SEND e RECEIVE

/dev/nvme0n1p2 on /tmp/tmp.o78czE0Bo6 type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sda2 on /tmp/tmp.XcwqQCKq09 type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

btrfsManage SEND / /dev/sda2

(2016-12-19 19:47:24) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '

Sending ' root-2016-12-19_19:44:00.41 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41
. . . At snapshot root-2016-12-19_19:44:00.41

Creation ' root-2016-12-19_19:44:00.41 ' snapshot from ' .part/root-2016-12-19_19:44:00.41 ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41' in '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/root-2016-12-19_19:44:00.41'
. . . Delete subvolume (commit): '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41'

Snapshot list in ' /dev/sda2 ' device
. . . btrfsreceive/data_backup/data_backup-2016-12-17_12:07:00.1
. . . btrfsreceive/data_storage/data_storage-2016-12-10_17:05:51.1
. . . btrfsreceive/root/root-2016-08-28-12-35-01.1
]zac[
. . . btrfsreceive/root/root-2016-12-19_19:44:00.41

(2016-12-19 19:48:37) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
CORRECTLY



> Hi Giuseppe,
>
> Would you like to tell some details about:
> 1. the XYZ snapshot was taken from which subvolume
> 2. where the base (initial) snapshot is stored
> 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> configuration and subvol structure?
>
> Also, would you send the link reports "two files unreadable error" post
> mentioned in step 2? Hope can see the message and figure out if the issue
> first comes from sender or receiver side.
>
> Thanks,
> Xin
>
>
 

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[bepi]

Hi.

I will insert ' btrfs check ' after each ' receive ' in my script.

I will test again my hardware.
But is not very likely that 2 computers, 3 HDD, 3 partitions, all have issue.

I think that the problem is a concomitance of operations, a race condition, a
random conditions.
I'll try to create a test case.


P.S. For find the problem may need to insert tools as ' coredumper ' and  '
sanitize ' in ' btrfs ', detect in realtime the ' extent ' corruption, and log
detection.


Thank you.

Gdb



Xin Zhou <xin.zhou@gmx.com>:

> Hi,
> 
> The system seems running some customized scripts continuously backup data
> from a NVME drive to HDDs.
> If the 3 HDDs backup storage are same in btrfs config, and the there is a bug
> in btrfs code,
> they all suppose to fail after the same operation sequence.
> 
> Otherwise, probably one of the HDDs might have issue, or there is a bug in
> layer below btrfs.
> 
> For the customize script, it might be helpful to check the file system
> consistency after each transfer.
> That might be useful to figure out which step generates a corruption, and if
> there is error propagations.
> 
> Regards,
> Xin
>  
>  
> 
> Sent: Monday, December 19, 2016 at 10:55 AM
> From: "Giuseppe Della Bianca" <bepi@adria.it>
> To: "Xin Zhou" <xin.zhou@gmx.com>
> Cc: linux-btrfs@vger.kernel.org
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> a concrete example
> 
> 
> SNAPSHOT
> 
> /dev/nvme0n1p2 on /tmp/tmp.X3vU6dLLVI type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> 
> btrfsManage SNAPSHOT /
> 
> (2016-12-19 19:44:00) Start btrfsManage
> . . . Start managing SNAPSHOT ' / ' filesystem ' root ' snapshot
> 
> In ' btrfssnapshot ' latest source snapshot ' root-2016-12-18_15:10:01.40 '
> . . . date ' 2016-12-18_15:10:01 ' number ' 40 '
> 
> Creation ' root-2016-12-19_19:44:00.part ' snapshot from ' root ' subvolume
> . . . Create a readonly snapshot of '/tmp/tmp.X3vU6dLLVI/root' in
> '/tmp/tmp.X3vU6dLLVI/btrfssnapshot/root/root-2016-12-19_19:44:00.part'
> 
> Renaming ' root-2016-12-19_19:44:00.part ' into ' root-2016-12-19_19:44:00.41
> ' snapshot
> 
> Source snapshot list of ' root ' subvolume
> . . . btrfssnapshot/root/root-2016-08-28-12-35-01.1
> ]zac[
> . . . btrfssnapshot/root/root-2016-12-19_19:44:00.41
> 
> (2016-12-19 19:44:05) End btrfsManage
> . . . End managing SNAPSHOT ' / ' filesystem ' root ' snapshot
> CORRECTLY
> 
> 
> 
> SEND e RECEIVE
> 
> /dev/nvme0n1p2 on /tmp/tmp.o78czE0Bo6 type btrfs
> (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> /dev/sda2 on /tmp/tmp.XcwqQCKq09 type btrfs
> (rw,relatime,space_cache,subvolid=5,subvol=/)
> 
> btrfsManage SEND / /dev/sda2
> 
> (2016-12-19 19:47:24) Start btrfsManage
> . . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2
> '
> 
> Sending ' root-2016-12-19_19:44:00.41 ' source snapshot to ' btrfsreceive '
> subvolume
> . . . btrfs send -p
> /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-18_15:10:01.40
> /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs
> receive /tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/
> . . . At subvol
> /tmp/tmp.o78czE0Bo6/btrfssnapshot/root/root-2016-12-19_19:44:00.41
> . . . At snapshot root-2016-12-19_19:44:00.41
> 
> Creation ' root-2016-12-19_19:44:00.41 ' snapshot from '
> .part/root-2016-12-19_19:44:00.41 ' subvolume
> . . . Create a readonly snapshot of
> '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41' in
> '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/root-2016-12-19_19:44:00.41'
> . . . Delete subvolume (commit):
> '/tmp/tmp.XcwqQCKq09/btrfsreceive/root/.part/root-2016-12-19_19:44:00.41'
> 
> Snapshot list in ' /dev/sda2 ' device
> . . . btrfsreceive/data_backup/data_backup-2016-12-17_12:07:00.1
> . . . btrfsreceive/data_storage/data_storage-2016-12-10_17:05:51.1
> . . . btrfsreceive/root/root-2016-08-28-12-35-01.1
> ]zac[
> . . . btrfsreceive/root/root-2016-12-19_19:44:00.41
> 
> (2016-12-19 19:48:37) End btrfsManage
> . . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
> CORRECTLY
> 
> 
> 
> > Hi Giuseppe,
> >
> > Would you like to tell some details about:
> > 1. the XYZ snapshot was taken from which subvolume
> > 2. where the base (initial) snapshot is stored
> > 3. The 3 partitions receives the same snapshot, are they in the same btrfs
> > configuration and subvol structure?
> >
> > Also, would you send the link reports "two files unreadable error" post
> > mentioned in step 2? Hope can see the message and figure out if the issue
> > first comes from sender or receiver side.
> >
> > Thanks,
> > Xin
> >
> >
>  
> 




----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Chris Murphy]

What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
mount option?


Chris Murphy

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Chris Murphy]

On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com> wrote:
> What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> mount option?

This slows things down, and in that case it might avoid the problem if
it's the result of a race condition.

-- 
Chris Murphy

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

Hi,
Racing condition can happen, if running multiple transfers to the same destination.
Would you like to tell how many transfers are the scripts running at a time to a specific hdd?

Thanks,
Xin
 

Sent: Wednesday, December 21, 2016 at 1:11 PM
From: "Chris Murphy" <lists@colorremedies.com>
To: No recipient address
Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou" <xin.zhou@gmx.com>, "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com> wrote:
> What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> mount option?

This slows things down, and in that case it might avoid the problem if
it's the result of a race condition.

--
Chris Murphy

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

(synthetic resend)

Hi.

Is possible that there are transfers, cancellations and other, at the same 
time, but not in the same subvolume.

My script checks that there are no transfers in progress on the same 
subvolume.

Is possible that the same subvolume is mounted several times (temporary mount 
at the beginning, and unmount at the end, in my script).


Thanks for all.


P.S. Sorry for my bad English.


Gdb


In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> Hi,
> Racing condition can happen, if running multiple transfers to the same
> destination. Would you like to tell how many transfers are the scripts
> running at a time to a specific hdd?
> 
> Thanks,
> Xin
>  
> 
> Sent: Wednesday, December 21, 2016 at 1:11 PM
> From: "Chris Murphy" <lists@colorremedies.com>
> To: No recipient address
> Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou" <xin.zhou@gmx.com>,
> "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> receive
> On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com> 
wrote:
> > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > mount option?
> 
> This slows things down, and in that case it might avoid the problem if
> it's the result of a race condition.
> 
> --
> Chris Murphy

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

Hi,

Does the script check the transfer status, and is there a transfer returns an error code?
Thanks,
Xin
 
 

Sent: Thursday, December 22, 2016 at 11:28 PM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
(synthetic resend)

Hi.

Is possible that there are transfers, cancellations and other, at the same
time, but not in the same subvolume.

My script checks that there are no transfers in progress on the same
subvolume.

Is possible that the same subvolume is mounted several times (temporary mount
at the beginning, and unmount at the end, in my script).


Thanks for all.


P.S. Sorry for my bad English.


Gdb


In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> Hi,
> Racing condition can happen, if running multiple transfers to the same
> destination. Would you like to tell how many transfers are the scripts
> running at a time to a specific hdd?
>
> Thanks,
> Xin
>
>
> Sent: Wednesday, December 21, 2016 at 1:11 PM
> From: "Chris Murphy" <lists@colorremedies.com>
> To: No recipient address
> Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou" <xin.zhou@gmx.com>,
> "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> receive
> On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
wrote:
> > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > mount option?
>
> This slows things down, and in that case it might avoid the problem if
> it's the result of a race condition.
>
> --
> Chris Murphy

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[bepi]

Yes.

Is through to the btrfs-tools error message that the script has printed, that I
realized the filesystem corruption.


P.S. Various messages that you see in the working examples of the script, are
emitted directly by the btrfs-tools.


Gdb

Xin Zhou <xin.zhou@gmx.com>:

> Hi,
> 
> Does the script check the transfer status, and is there a transfer returns an
> error code?
> Thanks,
> Xin
>  
>  
> 
> Sent: Thursday, December 22, 2016 at 11:28 PM
> From: "Giuseppe Della Bianca" <bepi@adria.it>
> To: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> (synthetic resend)
> 
> Hi.
> 
> Is possible that there are transfers, cancellations and other, at the same
> time, but not in the same subvolume.
> 
> My script checks that there are no transfers in progress on the same
> subvolume.
> 
> Is possible that the same subvolume is mounted several times (temporary
> mount
> at the beginning, and unmount at the end, in my script).
> 
> 
> Thanks for all.
> 
> 
> P.S. Sorry for my bad English.
> 
> 
> Gdb
> 
> 
> In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > Hi,
> > Racing condition can happen, if running multiple transfers to the same
> > destination. Would you like to tell how many transfers are the scripts
> > running at a time to a specific hdd?
> >
> > Thanks,
> > Xin
> >
> >
> > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > From: "Chris Murphy" <lists@colorremedies.com>
> > To: No recipient address
> > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> <xin.zhou@gmx.com>,
> > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > receive
> > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> wrote:
> > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > mount option?
> >
> > This slows things down, and in that case it might avoid the problem if
> > it's the result of a race condition.
> >
> > --
> > Chris Murphy
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> 




----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

Hi,

Would you like to show the "btrfs send/receive" command the script are using, including all the parameters,
and how the script waits for a completion of a transfer.

>From the beginning of the thread, it seems the transfer tests are going through different network environment.
 
Thanks,
Xin

Sent: Friday, December 23, 2016 at 9:48 AM
From: bepi@adria.it
To: "Xin Zhou" <xin.zhou@gmx.com>
Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Yes.

Is through to the btrfs-tools error message that the script has printed, that I
realized the filesystem corruption.


P.S. Various messages that you see in the working examples of the script, are
emitted directly by the btrfs-tools.


Gdb

Xin Zhou <xin.zhou@gmx.com>:

> Hi,
>
> Does the script check the transfer status, and is there a transfer returns an
> error code?
> Thanks,
> Xin
>  
>  
>
> Sent: Thursday, December 22, 2016 at 11:28 PM
> From: "Giuseppe Della Bianca" <bepi@adria.it>
> To: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive
> (synthetic resend)
>
> Hi.
>
> Is possible that there are transfers, cancellations and other, at the same
> time, but not in the same subvolume.
>
> My script checks that there are no transfers in progress on the same
> subvolume.
>
> Is possible that the same subvolume is mounted several times (temporary
> mount
> at the beginning, and unmount at the end, in my script).
>
>
> Thanks for all.
>
>
> P.S. Sorry for my bad English.
>
>
> Gdb
>
>
> In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > Hi,
> > Racing condition can happen, if running multiple transfers to the same
> > destination. Would you like to tell how many transfers are the scripts
> > running at a time to a specific hdd?
> >
> > Thanks,
> > Xin
> >
> >
> > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > From: "Chris Murphy" <lists@colorremedies.com>
> > To: No recipient address
> > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> <xin.zhou@gmx.com>,
> > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > receive
> > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> wrote:
> > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > mount option?
> >
> > This slows things down, and in that case it might avoid the problem if
> > it's the result of a race condition.
> >
> > --
> > Chris Murphy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>




----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it[http://www.alpikom.it]
 

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

Hi.

LOCAL "btrfs send/receive"

btrfs send -p /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.NRYgYVC3p6/btrfsreceive/root/.part/


REMOTE "btrfs send/receive"

btrfs send -p /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | ssh root@exnetold.gdb.it btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41

Remote btrfsManage script (btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41) executes 

cat - | btrfs receive /tmp/tmp.Wd5togIiaL/btrfsreceive/root/.part/


The waiting for the end of a command, in shell script, is implicit/automatic.


Regards.


Gdb

> Hi,
> 
> Would you like to show the "btrfs send/receive" command the script are
> using, including all the parameters, and how the script waits for a
> completion of a transfer.
> 
> From the beginning of the thread, it seems the transfer tests are going
> through different network environment. 
> Thanks,
> Xin
> 
> Sent: Friday, December 23, 2016 at 9:48 AM
> From: bepi@adria.it
> To: "Xin Zhou" <xin.zhou@gmx.com>
> Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive Yes.
> 
> Is through to the btrfs-tools error message that the script has printed,
> that I realized the filesystem corruption.
> 
> 
> P.S. Various messages that you see in the working examples of the script,
> are emitted directly by the btrfs-tools.
> 
> 
> Gdb
> 
> Xin Zhou <xin.zhou@gmx.com>:
> > Hi,
> > 
> > Does the script check the transfer status, and is there a transfer returns
> > an error code?
> > Thanks,
> > Xin
> > Â 
> > Â 
> > 
> > Sent:Â Thursday, December 22, 2016 at 11:28 PM
> > From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> > To:Â "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> > Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file
> > system during the snapshot receive
> > (synthetic resend)
> > 
> > Hi.
> > 
> > Is possible that there are transfers, cancellations and other, at the same
> > time, but not in the same subvolume.
> > 
> > My script checks that there are no transfers in progress on the same
> > subvolume.
> > 
> > Is possible that the same subvolume is mounted several times (temporary
> > mount
> > at the beginning, and unmount at the end, in my script).
> > 
> > 
> > Thanks for all.
> > 
> > 
> > P.S. Sorry for my bad English.
> > 
> > 
> > Gdb
> > 
> > In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > > Hi,
> > > Racing condition can happen, if running multiple transfers to the same
> > > destination. Would you like to tell how many transfers are the scripts
> > > running at a time to a specific hdd?
> > > 
> > > Thanks,
> > > Xin
> > > 
> > > 
> > > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > > From: "Chris Murphy" <lists@colorremedies.com>
> > > To: No recipient address
> > > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> > 
> > <xin.zhou@gmx.com>,
> > 
> > > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > > receive
> > > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> > 
> > wrote:
> > > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > > mount option?
> > > 
> > > This slows things down, and in that case it might avoid the problem if
> > > it's the result of a race condition.
> > > 
> > > --
> > > Chris Murphy
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it[http://www.alpikom.it]
>  

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

Hi,

Probably can try to use "-v" to enable more output print.
A quick look at the send / receive code, it seems a little bit risky.
It seems lack of specific error handlings, and in most cases, return the same error code.
I think it might be helpful, when a transfer succeed, the command prints the transfer id, source / dest,
and a specific "success" string. 
Such output could help the script to figure out if a transfer really succeed.

The code is relatively new to me, I did not see retry logic in stream handling, please correct me if I am wrong about this.
So, I am not quite sure about the transfer behavior, if the system subject to network issues in heavy workload,
in which packets missing or connect issues are not rare.

Since the test mentioned at the begining deletes the snapshots after a transfer, while most users keep the middle snapshot even in cascading transfer, probably the current btrfs and cmds still works for regular users.

Thanks,
Xin
 
 

Sent: Saturday, December 24, 2016 at 4:16 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Xin Zhou" <xin.zhou@gmx.com>, linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Hi.

LOCAL "btrfs send/receive"

btrfs send -p /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.NRYgYVC3p6/btrfsreceive/root/.part/


REMOTE "btrfs send/receive"

btrfs send -p /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | ssh root@exnetold.gdb.it btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41

Remote btrfsManage script (btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41) executes

cat - | btrfs receive /tmp/tmp.Wd5togIiaL/btrfsreceive/root/.part/


The waiting for the end of a command, in shell script, is implicit/automatic.


Regards.


Gdb

> Hi,
>
> Would you like to show the "btrfs send/receive" command the script are
> using, including all the parameters, and how the script waits for a
> completion of a transfer.
>
> From the beginning of the thread, it seems the transfer tests are going
> through different network environment.
> Thanks,
> Xin
>
> Sent: Friday, December 23, 2016 at 9:48 AM
> From: bepi@adria.it
> To: "Xin Zhou" <xin.zhou@gmx.com>
> Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive Yes.
>
> Is through to the btrfs-tools error message that the script has printed,
> that I realized the filesystem corruption.
>
>
> P.S. Various messages that you see in the working examples of the script,
> are emitted directly by the btrfs-tools.
>
>
> Gdb
>
> Xin Zhou <xin.zhou@gmx.com>:
> > Hi,
> >
> > Does the script check the transfer status, and is there a transfer returns
> > an error code?
> > Thanks,
> > Xin
> > Â
> > Â
> >
> > Sent:Â Thursday, December 22, 2016 at 11:28 PM
> > From:Â "Giuseppe Della Bianca" <bepi@adria.it>
> > To:Â "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> > Subject:Â Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file
> > system during the snapshot receive
> > (synthetic resend)
> >
> > Hi.
> >
> > Is possible that there are transfers, cancellations and other, at the same
> > time, but not in the same subvolume.
> >
> > My script checks that there are no transfers in progress on the same
> > subvolume.
> >
> > Is possible that the same subvolume is mounted several times (temporary
> > mount
> > at the beginning, and unmount at the end, in my script).
> >
> >
> > Thanks for all.
> >
> >
> > P.S. Sorry for my bad English.
> >
> >
> > Gdb
> >
> > In data mercoledì 21 dicembre 2016 23:14:44, Xin Zhou ha scritto:
> > > Hi,
> > > Racing condition can happen, if running multiple transfers to the same
> > > destination. Would you like to tell how many transfers are the scripts
> > > running at a time to a specific hdd?
> > >
> > > Thanks,
> > > Xin
> > >
> > >
> > > Sent: Wednesday, December 21, 2016 at 1:11 PM
> > > From: "Chris Murphy" <lists@colorremedies.com>
> > > To: No recipient address
> > > Cc: "Giuseppe Della Bianca" <bepi@adria.it>, "Xin Zhou"
> >
> > <xin.zhou@gmx.com>,
> >
> > > "Btrfs BTRFS" <linux-btrfs@vger.kernel.org> Subject: Re: [CORRUPTION
> > > FILESYSTEM] Corrupted and unrecoverable file system during the snapshot
> > > receive
> > > On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> >
> > wrote:
> > > > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > > > mount option?
> > >
> > > This slows things down, and in that case it might avoid the problem if
> > > it's the result of a race condition.
> > >
> > > --
> > > Chris Murphy
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it[http://www.alpikom.it][http://www.alpikom.it[http://www.alpikom.it]]
>

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html[http://vger.kernel.org/majordomo-info.html]

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Duncan]

Xin Zhou posted on Sat, 24 Dec 2016 21:15:40 +0100 as excerpted:

> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this.
> So, I am not quite sure about the transfer behavior, if the system
> subject to network issues in heavy workload,
> in which packets missing or connect issues are not rare.

As you likely know I'm neither a dev, just a list regular and btrfs user 
myself, and I'm not particularly familiar with send/receive as I don't 
use it myself, but...

AFAIK, the send and receive sides are specifically designed to be 
separate and to work with STDOUT/STDIN, so it's possible with STDOUT 
redirection to "send" to a local file instead of directly to receive, and 
then to replay that file on the other end by cat-ing it to receive.

As such, transfer behavior isn't really a factor at the btrfs layer, 
since handling problems in the transfer layer is the responsibility of 
whatever method the user is using to do that transfer, and the user is 
presumed to use a transfer method with whatever reliability guarantees 
they deem necessary.  So network behavior isn't really a factor at the 
btrfs level as that's the transfer layer and btrfs isn't worrying about 
that, simply assuming it to have the necessary reliability.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

Hi,
For free software with open source code, that is quite good.
Most commercial product has a very robust error handling in transport, to guarantee no corruption due to transfer issues.
One interesting thing to investigate might be the btrfs send / receive result, under a disruptive network environment.
If the connection breaks in the middle of transfer (at different phase, maybe), see what could be the file system status.

Thanks,
Xin

 
 

Sent: Sunday, December 25, 2016 at 2:57 PM
From: Duncan <1i5t5.duncan@cox.net>
To: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Xin Zhou posted on Sat, 24 Dec 2016 21:15:40 +0100 as excerpted:

> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this.
> So, I am not quite sure about the transfer behavior, if the system
> subject to network issues in heavy workload,
> in which packets missing or connect issues are not rare.

As you likely know I'm neither a dev, just a list regular and btrfs user
myself, and I'm not particularly familiar with send/receive as I don't
use it myself, but...

AFAIK, the send and receive sides are specifically designed to be
separate and to work with STDOUT/STDIN, so it's possible with STDOUT
redirection to "send" to a local file instead of directly to receive, and
then to replay that file on the other end by cat-ing it to receive.

As such, transfer behavior isn't really a factor at the btrfs layer,
since handling problems in the transfer layer is the responsibility of
whatever method the user is using to do that transfer, and the user is
presumed to use a transfer method with whatever reliability guarantees
they deem necessary. So network behavior isn't really a factor at the
btrfs level as that's the transfer layer and btrfs isn't worrying about
that, simply assuming it to have the necessary reliability.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Duncan]

Xin Zhou posted on Mon, 26 Dec 2016 03:36:09 +0100 as excerpted:

> One interesting thing to investigate might be the btrfs send / receive
> result, under a disruptive network environment. If the connection breaks
> in the middle of transfer (at different phase, maybe), see what could be
> the file system status.

Btrfs send, sends from a read-only snapshot, so the sending filesystem 
shouldn't be harmed no matter what happens to send.

Btrfs receive does all its work in a new subvolume (basically a snapshot 
in an incremental send, tho I'm not sure it's a full snapshot in the 
technical sense), modifying the files therein using standard calls used 
in other contexts as well, so absent bugs that should appear in those 
other contexts too if they exist, the worst damage that a receive should 
be able to do is an unfaithful replay of the send stream, such that an 
appropriate copy of the sent snapshot doesn't appear on the receiver.

Which means even in the case of error, cleanup is as simple as deleting 
the aborted/incompletely-received subvolume.  Unless there are bugs that 
would show up in other situations as well (or an out-of-space condition 
is triggered that would likewise show up in other situations with a 
similar amount of data/metadata written), there should be no effects 
outside that received subvolume.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

>Unless there are bugs that
>would show up in other situations as well (or an out-of-space condition
>is triggered that would likewise show up in other situations with a
>similar amount of data/metadata written),
 
That is exactly some bugs come from.
For simple cases, it is ok to assume the send/receive always succeed.
And if it errors out, assumes the delete always succeed, and the file system is in consistent status,
and good luck with the data.



Xin




Sent: Sunday, December 25, 2016 at 7:52 PM
From: Duncan <1i5t5.duncan@cox.net>
To: linux-btrfs@vger.kernel.org
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Xin Zhou posted on Mon, 26 Dec 2016 03:36:09 +0100 as excerpted:

> One interesting thing to investigate might be the btrfs send / receive
> result, under a disruptive network environment. If the connection breaks
> in the middle of transfer (at different phase, maybe), see what could be
> the file system status.

Btrfs send, sends from a read-only snapshot, so the sending filesystem
shouldn't be harmed no matter what happens to send.

Btrfs receive does all its work in a new subvolume (basically a snapshot
in an incremental send, tho I'm not sure it's a full snapshot in the
technical sense), modifying the files therein using standard calls used
in other contexts as well, so absent bugs that should appear in those
other contexts too if they exist, the worst damage that a receive should
be able to do is an unfaithful replay of the send stream, such that an
appropriate copy of the sent snapshot doesn't appear on the receiver.

Which means even in the case of error, cleanup is as simple as deleting
the aborted/incompletely-received subvolume. Unless there are bugs that
would show up in other situations as well (or an out-of-space condition
is triggered that would likewise show up in other situations with a
similar amount of data/metadata written), there should be no effects
outside that received subvolume.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

Hi.

I agree with Duncan, and I add:

- For remote transfer is used ssh.
  ssh is designed to ensure integrity of data.
- Remote transfer uses a Gigabit Ethernet, it is never congested.
- I had the same problems with a local btrfs receive.
- The script currently has 907 lines of code, many of which are to ensure the 
detection and display of btrfs tools errors.
- The script stops executing when btrs tools return an error code.
- Is not possible that the script does not display error messages or ignore 
error code of btrfs tools.

An example of today:

(2016-12-26 10:53:51) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '

Sending ' root-2016-12-04_18:13:57.35 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-03_18:07:09.34 /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35 | btrfs receive /tmp/tmp.pWWKP4vfAy/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35
. . . ERROR: truncate usr/share/locale/it/LC_MESSAGES/kio4.mo failed: Read-only file system
. . . At snapshot root-2016-12-04_18:13:57.35
. . . _EC_ERR_ 1
. . . _EC_ERR_ 141

(2016-12-26 10:54:28) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
WITH ERRORS


Checking filesystem on /dev/sda2
UUID: 44f1de7e-a65b-41ce-8ff4-20f7ed83e106
checking extents
ref mismatch on [62408097792 16384] extent item 0, found 1
Backref 62408097792 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [62408097792 16384]
owner ref check failed [62408097792 16384]
ref mismatch on [77565509632 16384] extent item 0, found 1
Backref 77565509632 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77565509632 16384]
]zac[
Backref 77826916352 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77826916352 16384]
owner ref check failed [77826916352 16384]
ref mismatch on [77853933568 16384] extent item 0, found 1
Backref 77853933568 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77853933568 16384]
owner ref check failed [77853933568 16384]
checking free space cache
checking fs roots
warning line 3822
checking csums
checking root refs
found 135128678400 bytes used err is 0
total csum bytes: 126946572
total tree bytes: 5132206080
total fs tree bytes: 4744757248
total extent tree bytes: 240795648
btree space waste bytes: 914832832
file data blocks allocated: 3311786532864
 referenced 703616266240



Is likely that mine is a special case.

But a special case, with a code change in other points, can become a problem for many.

It's not nice to say, but it seems I have to hope that my problem becomes a problem of many.

Meanwhile, I'll find my own workaround of a probable serious btrfs bug.


Thank you.

Gdb


> Hi,
> 
> Probably can try to use "-v" to enable more output print.
> A quick look at the send / receive code, it seems a little bit risky.
> It seems lack of specific error handlings, and in most cases, return the
> same error code. I think it might be helpful, when a transfer succeed, the
> command prints the transfer id, source / dest, and a specific "success"
> string.
> Such output could help the script to figure out if a transfer really
> succeed.
> 
> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this. So, I am not quite
> sure about the transfer behavior, if the system subject to network issues
> in heavy workload, in which packets missing or connect issues are not rare.
> 
> Since the test mentioned at the begining deletes the snapshots after a
> transfer, while most users keep the middle snapshot even in cascading
> transfer, probably the current btrfs and cmds still works for regular
> users.
> 
> Thanks,
> Xin
>  
>  

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Xin Zhou]

That is one way to diagnose the issue in data path.
If ssh can guarantee data transfer and retry, then those data protection company does not need to have a whole team handle the send / receive for remote data backup.

In your case, if the conection is very light, then the issue could be in other place.

Xin 
 

Sent: Monday, December 26, 2016 at 3:04 AM
From: "Giuseppe Della Bianca" <bepi@adria.it>
To: "Xin Zhou" <xin.zhou@gmx.com>, "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive
Hi.

I agree with Duncan, and I add:

- For remote transfer is used ssh.
ssh is designed to ensure integrity of data.
- Remote transfer uses a Gigabit Ethernet, it is never congested.
- I had the same problems with a local btrfs receive.
- The script currently has 907 lines of code, many of which are to ensure the
detection and display of btrfs tools errors.
- The script stops executing when btrs tools return an error code.
- Is not possible that the script does not display error messages or ignore
error code of btrfs tools.

An example of today:

(2016-12-26 10:53:51) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '

Sending ' root-2016-12-04_18:13:57.35 ' source snapshot to ' btrfsreceive ' subvolume
. . . btrfs send -p /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-03_18:07:09.34 /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35 | btrfs receive /tmp/tmp.pWWKP4vfAy/btrfsreceive/root/.part/
. . . At subvol /tmp/tmp.xJWkEN1U23/btrfssnapshot/root/root-2016-12-04_18:13:57.35
. . . ERROR: truncate usr/share/locale/it/LC_MESSAGES/kio4.mo failed: Read-only file system
. . . At snapshot root-2016-12-04_18:13:57.35
. . . _EC_ERR_ 1
. . . _EC_ERR_ 141

(2016-12-26 10:54:28) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda2 '
WITH ERRORS


Checking filesystem on /dev/sda2
UUID: 44f1de7e-a65b-41ce-8ff4-20f7ed83e106
checking extents
ref mismatch on [62408097792 16384] extent item 0, found 1
Backref 62408097792 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [62408097792 16384]
owner ref check failed [62408097792 16384]
ref mismatch on [77565509632 16384] extent item 0, found 1
Backref 77565509632 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77565509632 16384]
]zac[
Backref 77826916352 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77826916352 16384]
owner ref check failed [77826916352 16384]
ref mismatch on [77853933568 16384] extent item 0, found 1
Backref 77853933568 parent 1060 root 1060 not found in extent tree
backpointer mismatch on [77853933568 16384]
owner ref check failed [77853933568 16384]
checking free space cache
checking fs roots
warning line 3822
checking csums
checking root refs
found 135128678400 bytes used err is 0
total csum bytes: 126946572
total tree bytes: 5132206080
total fs tree bytes: 4744757248
total extent tree bytes: 240795648
btree space waste bytes: 914832832
file data blocks allocated: 3311786532864
referenced 703616266240



Is likely that mine is a special case.

But a special case, with a code change in other points, can become a problem for many.

It's not nice to say, but it seems I have to hope that my problem becomes a problem of many.

Meanwhile, I'll find my own workaround of a probable serious btrfs bug.


Thank you.

Gdb


> Hi,
>
> Probably can try to use "-v" to enable more output print.
> A quick look at the send / receive code, it seems a little bit risky.
> It seems lack of specific error handlings, and in most cases, return the
> same error code. I think it might be helpful, when a transfer succeed, the
> command prints the transfer id, source / dest, and a specific "success"
> string.
> Such output could help the script to figure out if a transfer really
> succeed.
>
> The code is relatively new to me, I did not see retry logic in stream
> handling, please correct me if I am wrong about this. So, I am not quite
> sure about the transfer behavior, if the system subject to network issues
> in heavy workload, in which packets missing or connect issues are not rare.
>
> Since the test mentioned at the begining deletes the snapshots after a
> transfer, while most users keep the middle snapshot even in cascading
> transfer, probably the current btrfs and cmds still works for regular
> users.
>
> Thanks,
> Xin
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

Hi.

LOCAL "btrfs send/receive"

btrfs send -p /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.hkAa8XaliZ/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | btrfs receive /tmp/tmp.NRYgYVC3p6/btrfsreceive/root/.part/


REMOTE "btrfs send/receive"

btrfs send -p /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-18_15:10:01.40 /tmp/tmp.BTACBk7xuK/btrfssnapshot/root/root-2016-12-19_19:44:00.41 | ssh root@exnetold.gdb.it btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41

Remote btrfsManage script (btrfsManage REMOTE_RECEIVE /dev/sda3 root root-2016-12-19_19:44:00.41) executes 

cat - | btrfs receive /tmp/tmp.Wd5togIiaL/btrfsreceive/root/.part/


The waiting for the end of a command, in shell script, is implicit/automatic.


Regards.


Gdb

> Hi,
> 
> Would you like to show the "btrfs send/receive" command the script are
> using, including all the parameters, and how the script waits for a
> completion of a transfer.
> 
> From the beginning of the thread, it seems the transfer tests are going
> through different network environment. 
> Thanks,
> Xin
> 
> Sent: Friday, December 23, 2016 at 9:48 AM
> From: bepi@adria.it
> To: "Xin Zhou" <xin.zhou@gmx.com>
> Cc: "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
> Subject: Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system
> during the snapshot receive Yes.
> 
> Is through to the btrfs-tools error message that the script has printed,
> that I realized the filesystem corruption.
> 
> 
> P.S. Various messages that you see in the working examples of the script,
> are emitted directly by the btrfs-tools.
> 
> 
> Gdb
> 
> Xin Zhou <xin.zhou@gmx.com>:
> > Hi,
> > 
> > Does the script check the transfer status, and is there a transfer returns
> > an error code?
> > Thanks,
> > Xin
> > Â 
> > Â 
> > 

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

Hi.

Two news (one is OT and advertising to my btrfsManage script)

I had to recreate from scratch two corrupted btrfs filesystems probably due to 
out of space during a snapshot receive.

Almost it's a habit rebuild the filesystems after a few months.



OT

I have released the latest version of my backup and btrfs manage scritp.

https://sourceforge.net/projects/btrfsmanage/

Bash script for managing btrfs filesystem (local and remote):
- Perform scrubs
- Creating snapshots
- Send snapshots
- List and delete snapshot
- Mount snapshot
- Simple to use and verify proper operation.
- Designed to be run from the shell or from the crontab.
- No setup, just copy btrfsManage.
- The subvolume on which to perform the snapshot is detected by the subvolume 
currently mounted.
- The subvolume necessary to btrfsManage are created and mounted 
automatically.


For example

/usr/local/bin/btrfsManage OPTION:usage SEND / /dev/sda3


(2017-08-19 15:45:01) Start btrfsManage
. . . Start managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda3 '

Sending ' root-2017-08-19_15:20:01.129 ' source snapshot to ' btrfsreceive ' 
subvolume
. . . btrfs send -p /tmp/tmp.p8JVMMpzZI/btrfssnapshot/root/
root-2017-08-17_10:43:01.128 /tmp/tmp.p8JVMMpzZI/btrfssnapshot/root/
root-2017-08-19_15:20:01.129 | btrfs receive /tmp/tmp.UexL3E9E2F/btrfsreceive/
root/.part/
. . . At subvol /tmp/tmp.p8JVMMpzZI/btrfssnapshot/root/
root-2017-08-19_15:20:01.129
. . . At snapshot root-2017-08-19_15:20:01.129

Creation ' root-2017-08-19_15:20:01.129 ' snapshot from ' .part/
root-2017-08-19_15:20:01.129 ' subvolume
. . . Create a readonly snapshot of '/tmp/tmp.UexL3E9E2F/btrfsreceive/
root/.part/root-2017-08-19_15:20:01.129' in '/tmp/tmp.UexL3E9E2F/btrfsreceive/
root/root-2017-08-19_15:20:01.129'
. . . Delete subvolume (commit): '/tmp/tmp.UexL3E9E2F/btrfsreceive/root/.part/
root-2017-08-19_15:20:01.129'

btrfs filesystem usage -T /tmp/tmp.UexL3E9E2F
Overall:
 Device size: 830.44GiB
 Device allocated: 513.02GiB
 Device unallocated: 317.42GiB
 Device missing: 0.00B
 Used: 509.83GiB
 Free (estimated): 318.41GiB (min: 159.70GiB)
 Data ratio: 1.00
 Metadata ratio: 2.00
 Global reserve: 512.00MiB (used: 0.00B)

 Data Metadata System 
Id Path single DUP DUP Unallocated
-- --------- --------- -------- -------- -----------
 1 /dev/sda3 505.01GiB 8.00GiB 16.00MiB 317.42GiB
-- --------- --------- -------- -------- -----------
 Total 505.01GiB 4.00GiB 8.00MiB 317.42GiB
 Used 504.01GiB 2.91GiB 80.00KiB 

Snapshot list in ' /dev/sda3 ' device
. . . btrfsreceive/root/root-2017-07-23_19:46:01.117
. . . btrfsreceive/root/root-2017-07-29_15:20:01.118
. . . btrfsreceive/root/root-2017-07-29_16:54:01.119
. . . btrfsreceive/root/root-2017-08-05_15:20:01.120
. . . btrfsreceive/root/root-2017-08-06_17:59:01.121
. . . btrfsreceive/root/root-2017-08-12_13:00:01.122
. . . btrfsreceive/root/root-2017-08-12_15:20:01.123
. . . btrfsreceive/root/root-2017-08-14_17:59:01.124
. . . btrfsreceive/root/root-2017-08-15_15:20:01.125
. . . btrfsreceive/root/root-2017-08-15_15:56:01.126
. . . btrfsreceive/root/root-2017-08-16_18:40:01.127
. . . btrfsreceive/root/root-2017-08-17_10:43:01.128
. . . btrfsreceive/root/root-2017-08-19_15:20:01.129

(2017-08-19 15:46:09) End btrfsManage
. . . End managing SEND ' / ' filesystem ' root ' snapshot in ' /dev/sda3 '
CORRECTLY


 
Regards.
 
 
Gdb

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

(synthetic resend)

I'll try to compile the kernel and mount with the config and option enabled.

I keep in mind the side effects of that option.


Thanks for all.


P.S. Sorry for my bad English.



Chris Murphy ha scritto:
> On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <lists@colorremedies.com> 
wrote:
> > What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
> > mount option?
> 
> This slows things down, and in that case it might avoid the problem if
> it's the result of a race condition.

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

Hi.

Only for information.

I tried ' check_int ' mount option, but with a few tens of GB used, the
kernel freezes completely (no ops, no messages, no panic, none).


Thanks for all.

Gdb

P.S. Thanks and happy work to all.

>>On Wed, Dec 21, 2016 at 2:09 PM, Chris Murphy <li...@colorremedies.com> wrote:
>> What about CONFIG_BTRFS_FS_CHECK_INTEGRITY? And then using check_int
>> mount option?
>This slows things down, and in that case it might avoid the problem if
>it's the result of a race condition.

>-- 
>Chris Murphy
>--

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

Hi.

I have executed the test that you have requested.

At this point I'm confused and demoralized

While 40 btrfs receive, btrfs check has reported many filesystem problems.
But at the end of all the btrfs receive, btrfs check did not find filesystem problem.
My tests have confirmed that the filesystem had no problems.

I think btrfs check detect false positives (this can change a repair in a destruction).


Regards.

Gdb

Xin Zhou Tue, 20 Dec 2016 09:47:01 -0800

>>Hi,

>>The system seems running some customized scripts continuously backup data from 
>>a NVME drive to HDDs.
>>If the 3 HDDs backup storage are same in btrfs config, and the there is a bug 
>>in btrfs code,
>>they all suppose to fail after the same operation sequence.


>>Otherwise, probably one of the HDDs might have issue, or there is a bug in 
>>layer below btrfs.

>>For the customize script, it might be helpful to check the file system 
>>consistency after each transfer.
>>That might be useful to figure out which step generates a corruption, and if 
>>there is error propagations.

>>Regards,
>>Xin

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Qu Wenruo]

At 11/21/2016 08:09 PM, bepi@adria.it wrote:
> Hi.
>
> My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64 btrfs-progs-4.4.1-1.fc23.x86_64
>
> Testing the remote differential receive (via ssh and in local network) of 24
> sequential snapshots, and simultaneously deleting the snapshot, (in the same
> file system, but in a different subvolume), there has been an file access error,
> and the file system has corrupt.

Are you using qgroup?

IIRC, Filipe fixed a problem which could cause backref corruption which 
only happens if quota is enabled.

Thanks,
Qu

>
> Both scrub, both recovery and clear_cache mount options, both btrfsck, have
> failed, the file system is left in a state unusable.
>
> After reformatting the filesystem, remote receive of 24 snapshots worked properly.
>
> The file system is used exclusively for receive the snapshot, it is composed of
> a single device.
> The initial snapshot is a linux installation of 50Gb.
>
>
> I think that there was a race condition between the receive and deletion of
> snapshots (that were performed on two different subvolume).
>
>
> Best regards.
>
> gdb
>
> ----------------------------------------------------
> This mail has been sent using Alpikom webmail system
> http://www.alpikom.it
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[bepi]

> 
> 
> At 11/21/2016 08:09 PM, bepi@adria.it wrote:
> > Hi.
> >
> > My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64
> btrfs-progs-4.4.1-1.fc23.x86_64
> >
> > Testing the remote differential receive (via ssh and in local network) of
> 24
> > sequential snapshots, and simultaneously deleting the snapshot, (in the
> same
> > file system, but in a different subvolume), there has been an file access
> error,
> > and the file system has corrupt.
> 
> Are you using qgroup?
> 
> IIRC, Filipe fixed a problem which could cause backref corruption which 
> only happens if quota is enabled.
> 
> Thanks,
> Qu


Unfortunately not.

I read that thread (and others), but no one it seemed like my case.


To you (Thanks).


Gdb


> 
> >
> > Both scrub, both recovery and clear_cache mount options, both btrfsck,
> have
> > failed, the file system is left in a state unusable.
> >
> > After reformatting the filesystem, remote receive of 24 snapshots worked
> properly.
> >
> > The file system is used exclusively for receive the snapshot, it is
> composed of
> > a single device.
> > The initial snapshot is a linux installation of 50Gb.
> >
> >
> > I think that there was a race condition between the receive and deletion
> of
> > snapshots (that were performed on two different subvolume).
> >
> >
> > Best regards.
> >
> > gdb
> >
> > ----------------------------------------------------
> > This mail has been sent using Alpikom webmail system
> > http://www.alpikom.it
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> 
> 
> 




----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it

[CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[bepi]

Hi.

My system: Fedora 23, kernel-4.7.10-100.fc23.x86_64 btrfs-progs-4.4.1-1.fc23.x86_64

Testing the remote differential receive (via ssh and in local network) of 24
sequential snapshots, and simultaneously deleting the snapshot, (in the same
file system, but in a different subvolume), there has been an file access error,
and the file system has corrupt.

Both scrub, both recovery and clear_cache mount options, both btrfsck, have
failed, the file system is left in a state unusable.

After reformatting the filesystem, remote receive of 24 snapshots worked properly.

The file system is used exclusively for receive the snapshot, it is composed of
a single device.
The initial snapshot is a linux installation of 50Gb.


I think that there was a race condition between the receive and deletion of
snapshots (that were performed on two different subvolume).


Best regards.

gdb



----------------------------------------------------
This mail has been sent using Alpikom webmail system
http://www.alpikom.it

Re: [CORRUPTION FILESYSTEM] Corrupted and unrecoverable file system during the snapshot receive

[Giuseppe Della Bianca]

> Chris Murphy Sun, 18 Dec 2016 12:12:42 -0800

> I'd say backup the volume conventionally to get important data
> secured. Then use btrfs check --repair to fix the problems with the
> file system, and then try the btrfs send receive again.


> It most certainly is a bug, but it's not clear to me what the cause
> is; other than the kernel isn't handling the problem on-disk
> gracefully. Maybe repair will fix it. Another possibility is to move
> to kernel 4.9.0 and see if it still reproduces. Per usual, there's a
> ton of bug fixes in each kernel release.

> Otherwise I'm out of ideas.

> Chris Murphy


Thanks for the reply.


I have three partitions that receive the same snapshots (and a separate backup 
for important data).

The damaged filesystems do not lose data, but only (!) waste time.

The repair attempts have only worsened the problem.


I understand the software development problems.
I would just like to be able to help you to find the problem.


Have a nice day


gdb