* 3.14.0rc3: did not find backref in send_root
@ 2014-02-25 6:36 Marc MERLIN
2014-02-25 7:50 ` Wang Shilong
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Marc MERLIN @ 2014-02-25 6:36 UTC (permalink / raw)
To: linux-btrfs
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560
I'll try a scrub when I've finished my backup, but is there anything I
can run on the file I've found from the inode?
gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3
ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0
file.mp3
Anything else?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: 3.14.0rc3: did not find backref in send_root 2014-02-25 6:36 3.14.0rc3: did not find backref in send_root Marc MERLIN @ 2014-02-25 7:50 ` Wang Shilong 2014-02-25 17:30 ` 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error Marc MERLIN 2014-02-26 7:46 ` 3.14.0rc3: did not find backref in send_root Marc MERLIN 2014-05-06 6:10 ` David Brown 2 siblings, 1 reply; 10+ messages in thread From: Wang Shilong @ 2014-02-25 7:50 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs Hi Marc, This seems a regression which has been fixed by the following commit(only pushed into btrfs-next): https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 Thanks, Wang On 02/25/2014 02:36 PM, Marc MERLIN wrote: > I got this during a btrfs send: > BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560 > > I'll try a scrub when I've finished my backup, but is there anything I > can run on the file I've found from the inode? > > gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 > ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 > file.mp3 > > Anything else? > > Thanks, > Marc ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error 2014-02-25 7:50 ` Wang Shilong @ 2014-02-25 17:30 ` Marc MERLIN 2014-02-26 3:38 ` Wang Shilong 0 siblings, 1 reply; 10+ messages in thread From: Marc MERLIN @ 2014-02-25 17:30 UTC (permalink / raw) To: Wang Shilong; +Cc: linux-btrfs On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote: > Hi Marc, > > This seems a regression which has been fixed by the following > commit(only pushed into btrfs-next): > > https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 I'll revert this, thanks. Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 (on my laptop this time): + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57 + btrfs receive /mnt/btrfs_pool2// At subvol var_ro.20140225_09:27:57 At snapshot var_ro.20140225_09:27:57 ERROR: send ioctl failed with -5: Input/output error ERROR: unexpected EOF in stream. Is it another different regression? Currently it's only happening on one of my subvolumes. The other ones are still syncing ok. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error 2014-02-25 17:30 ` 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error Marc MERLIN @ 2014-02-26 3:38 ` Wang Shilong 0 siblings, 0 replies; 10+ messages in thread From: Wang Shilong @ 2014-02-26 3:38 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs Hi Marc, On 02/26/2014 01:30 AM, Marc MERLIN wrote: > On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote: >> Hi Marc, >> >> This seems a regression which has been fixed by the following >> commit(only pushed into btrfs-next): >> >> https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 > I'll revert this, thanks. > > Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 > (on my laptop this time): What is your previous version, give a double check whether the following commit exists: Btrfs: remove transaction from btrfs send(commitid: 41ce9970) This regression should exist in 3.14 Sine chris firstly pull big btrfs request for Linux 3.14. > > + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57 > + btrfs receive /mnt/btrfs_pool2// > At subvol var_ro.20140225_09:27:57 > At snapshot var_ro.20140225_09:27:57 > ERROR: send ioctl failed with -5: Input/output error > ERROR: unexpected EOF in stream. Did dmesg output the same things like before? Hm, we can esaily trigger that regression if there is snapshot and send running concurrently. (balance/scrub ...device operations can also trigger send failure.) Thanks, Wang > > Is it another different regression? > > Currently it's only happening on one of my subvolumes. The other ones > are still syncing ok. > > Thanks, > Marc ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: did not find backref in send_root 2014-02-25 6:36 3.14.0rc3: did not find backref in send_root Marc MERLIN 2014-02-25 7:50 ` Wang Shilong @ 2014-02-26 7:46 ` Marc MERLIN 2014-02-26 7:51 ` Wang Shilong 2014-05-06 6:10 ` David Brown 2 siblings, 1 reply; 10+ messages in thread From: Marc MERLIN @ 2014-02-26 7:46 UTC (permalink / raw) To: linux-btrfs, Wang Shilong On Wed, Feb 26, 2014 at 11:38:30AM +0800, Wang Shilong wrote: > Hi Marc, > > On 02/26/2014 01:30 AM, Marc MERLIN wrote: > >On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote: > >>Hi Marc, > >> > >>This seems a regression which has been fixed by the following > >>commit(only pushed into btrfs-next): > >> > >>https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 > >I'll revert this, thanks. > > > >Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 > >(on my laptop this time): > What is your previous version, give a double check whether the > following commit exists: > > Btrfs: remove transaction from btrfs send(commitid: 41ce9970) > > This regression should exist in 3.14 Sine chris firstly pull big > btrfs request for Linux 3.14. > > > >+ btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57 > >+ btrfs receive /mnt/btrfs_pool2// > >At subvol var_ro.20140225_09:27:57 > >At snapshot var_ro.20140225_09:27:57 > >ERROR: send ioctl failed with -5: Input/output error > >ERROR: unexpected EOF in stream. > Did dmesg output the same things like before? > > Hm, we can esaily trigger that regression if there is snapshot and > send running concurrently. > (balance/scrub ...device operations can also trigger send failure.) So I was confused because I got this on my server: On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote: > I got this during a btrfs send: > BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560 and that on my laptop: [82841.563339] BTRFS info (device dm-0): csum failed ino 2590 extent 2397949952 csum 3100992190 wanted 2786811954 mirror 0 I've applied your patch from https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 legolas:/mnt/btrfs_pool1# bash -vx /var/local/scr/btrfs-subvolume-backup -k 5 var /mnt/btrfs_pool2/ still failed on my laptop + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_23:28:47 + btrfs receive /mnt/btrfs_pool2// At subvol var_ro.20140225_23:28:47 At snapshot var_ro.20140225_23:28:47 ERROR: send ioctl failed with -5: Input/output error ERROR: unexpected EOF in stream. [ 216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off 192512 csum 3851586574 expected csum 1402824092 Then again, this seems to be my problem on the laptop: legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1 scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6 scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 seconds total bytes scrubbed: 451.64GiB with 2 errors error details: csum=2 corrected errors: 0, uncorrectable errors: 2, unverified errors: 0 Ok, so that's not a btrfs send problem. Just out of curiosity, how do I find out which inodes are compromized so that I can delete/restore them? Back to my server, I installed the new kernel, and I'm testing send again. So far it hasn't failed, so there is a chance the patch fixed the problem, but I'll only know more tomorrow after it's run longer. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: did not find backref in send_root 2014-02-26 7:46 ` 3.14.0rc3: did not find backref in send_root Marc MERLIN @ 2014-02-26 7:51 ` Wang Shilong 2014-02-26 17:16 ` Marc MERLIN 0 siblings, 1 reply; 10+ messages in thread From: Wang Shilong @ 2014-02-26 7:51 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On 02/26/2014 03:46 PM, Marc MERLIN wrote: > On Wed, Feb 26, 2014 at 11:38:30AM +0800, Wang Shilong wrote: >> Hi Marc, >> >> On 02/26/2014 01:30 AM, Marc MERLIN wrote: >>> On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote: >>>> Hi Marc, >>>> >>>> This seems a regression which has been fixed by the following >>>> commit(only pushed into btrfs-next): >>>> >>>> https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 >>> I'll revert this, thanks. >>> >>> Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 >>> (on my laptop this time): >> What is your previous version, give a double check whether the >> following commit exists: >> >> Btrfs: remove transaction from btrfs send(commitid: 41ce9970) >> >> This regression should exist in 3.14 Sine chris firstly pull big >> btrfs request for Linux 3.14. >>> + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57 >>> + btrfs receive /mnt/btrfs_pool2// >>> At subvol var_ro.20140225_09:27:57 >>> At snapshot var_ro.20140225_09:27:57 >>> ERROR: send ioctl failed with -5: Input/output error >>> ERROR: unexpected EOF in stream. >> Did dmesg output the same things like before? >> >> Hm, we can esaily trigger that regression if there is snapshot and >> send running concurrently. >> (balance/scrub ...device operations can also trigger send failure.) > So I was confused because I got this on my server: > On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote: >> I got this during a btrfs send: >> BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560 > and that on my laptop: > [82841.563339] BTRFS info (device dm-0): csum failed ino 2590 extent 2397949952 csum 3100992190 wanted 2786811954 mirror 0 > > I've applied your patch from > https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 > > legolas:/mnt/btrfs_pool1# bash -vx /var/local/scr/btrfs-subvolume-backup -k 5 var /mnt/btrfs_pool2/ > > still failed on my laptop > + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_23:28:47 > + btrfs receive /mnt/btrfs_pool2// > At subvol var_ro.20140225_23:28:47 > At snapshot var_ro.20140225_23:28:47 > ERROR: send ioctl failed with -5: Input/output error > ERROR: unexpected EOF in stream. > [ 216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off 192512 csum 3851586574 expected csum 1402824092 > > Then again, this seems to be my problem on the laptop: > legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1 > scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6 > scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 seconds > total bytes scrubbed: 451.64GiB with 2 errors > error details: csum=2 > corrected errors: 0, uncorrectable errors: 2, unverified errors: 0 > > Ok, so that's not a btrfs send problem. > Just out of curiosity, how do I find out which inodes are compromized so > that I can delete/restore them? You can use command "btrfs inspect-internal inode-resolve" which will print inode's corresponding path with it's inode id. For your case, something like this: btrfs inspect-internal inode-resolve -v 2326135 /mnt/btrfs_pool1 Thanks, Wang > > Back to my server, I installed the new kernel, and I'm testing send > again. So far it hasn't failed, so there is a chance the patch fixed the > problem, but I'll only know more tomorrow after it's run longer. > > Thanks, > Marc ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: did not find backref in send_root 2014-02-26 7:51 ` Wang Shilong @ 2014-02-26 17:16 ` Marc MERLIN 0 siblings, 0 replies; 10+ messages in thread From: Marc MERLIN @ 2014-02-26 17:16 UTC (permalink / raw) To: Wang Shilong; +Cc: linux-btrfs On Wed, Feb 26, 2014 at 03:51:37PM +0800, Wang Shilong wrote: > >I've applied your patch from > >https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8 I can confirm this fixed the btrfs send error on my server, thank you. > >At snapshot var_ro.20140225_23:28:47 > >ERROR: send ioctl failed with -5: Input/output error > >ERROR: unexpected EOF in stream. > >[ 216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off > >192512 csum 3851586574 expected csum 1402824092 > > > >Then again, this seems to be my problem on the laptop: > >legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1 > >scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6 > > scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 > > seconds > > total bytes scrubbed: 451.64GiB with 2 errors > > error details: csum=2 > > corrected errors: 0, uncorrectable errors: 2, unverified errors: 0 > > > >Ok, so that's not a btrfs send problem. > >Just out of curiosity, how do I find out which inodes are compromized so > >that I can delete/restore them? > You can use command "btrfs inspect-internal inode-resolve" which will > print inode's > corresponding path with it's inode id. Yes, sorry, I wasn't clear. I know how to do this. What I meant is that btrfs scrub tells me there are 2 errors and does not tell me which inode/subvolume the error are in. Aah, but they were in syslog, I just hadn't found them until now :) Ok, I'm all good then. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: did not find backref in send_root 2014-02-25 6:36 3.14.0rc3: did not find backref in send_root Marc MERLIN 2014-02-25 7:50 ` Wang Shilong 2014-02-26 7:46 ` 3.14.0rc3: did not find backref in send_root Marc MERLIN @ 2014-05-06 6:10 ` David Brown 2014-05-06 6:49 ` Blaz Repas 2014-05-10 8:13 ` David Brown 2 siblings, 2 replies; 10+ messages in thread From: David Brown @ 2014-05-06 6:10 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote: >I got this during a btrfs send: >BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560 > >I'll try a scrub when I've finished my backup, but is there anything I >can run on the file I've found from the inode? > >gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 >ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 >file.mp3 I've just seen this error: BTRFS error (device sda4): did not find backref in send_root. inode=411890, offset=307200, disk_byte=48100618240 found extent=48100618240 during a send between two snapshots I have. after moving to 3.14.2. I've seen it on two filesystems now since moving to 3.14. I have the two readonly snapshots if there is anything helpful I can figure out from them. Scrub reports no errors, but I don't seem to be able to back up anything now. David ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: did not find backref in send_root 2014-05-06 6:10 ` David Brown @ 2014-05-06 6:49 ` Blaz Repas 2014-05-10 8:13 ` David Brown 1 sibling, 0 replies; 10+ messages in thread From: Blaz Repas @ 2014-05-06 6:49 UTC (permalink / raw) To: linux-btrfs On 05/06/2014 08:10 AM, David Brown wrote: > On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote: >> I got this during a btrfs send: >> BTRFS error (device dm-2): did not find backref in send_root. >> inode=22672, offset=524288, disk_byte=1490517954560 found >> extent=1490517954560 >> >> I'll try a scrub when I've finished my backup, but is there anything I >> can run on the file I've found from the inode? >> >> gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v >> 22672 file.mp3 >> ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 >> file.mp3 > > I've just seen this error: > > BTRFS error (device sda4): did not find backref in send_root. > inode=411890, offset=307200, disk_byte=48100618240 found > extent=48100618240 > > during a send between two snapshots I have. > > after moving to 3.14.2. I've seen it on two filesystems now since > moving to 3.14. I have the two readonly snapshots if there is > anything helpful I can figure out from them. > > Scrub reports no errors, but I don't seem to be able to back up > anything now. > > David > -- I am also seeing this on 3.14.1 (on ArchLinux). Scrub also reports no errors. I could also not do a ful send. Balancing made it better for a while (I was able to send a full snapshot of one subvolume, but not another), but it did not help. Offline repairing the fs with btrfsck --repair also did not affect it. Blaz ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 3.14.0rc3: did not find backref in send_root 2014-05-06 6:10 ` David Brown 2014-05-06 6:49 ` Blaz Repas @ 2014-05-10 8:13 ` David Brown 1 sibling, 0 replies; 10+ messages in thread From: David Brown @ 2014-05-10 8:13 UTC (permalink / raw) To: Josef Bacik, Chris Mason; +Cc: Marc MERLIN, linux-btrfs On Mon, May 05, 2014 at 11:10:54PM -0700, David Brown wrote: >On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote: >>I got this during a btrfs send: >>BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560 >> >>I'll try a scrub when I've finished my backup, but is there anything I >>can run on the file I've found from the inode? >> >>gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 >>ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 >>file.mp3 > >I've just seen this error: > > BTRFS error (device sda4): did not find backref in send_root. inode=411890, offset=307200, disk_byte=48100618240 found extent=48100618240 > >during a send between two snapshots I have. > >after moving to 3.14.2. I've seen it on two filesystems now since >moving to 3.14. I have the two readonly snapshots if there is >anything helpful I can figure out from them. After bisecting it seems to be caused by commit 7ef81ac86c8a44ab9f4e6e04e1f4c9ea53615b8a Author: Josef Bacik <jbacik@fb.com> Date: Fri Jan 24 14:05:42 2014 -0500 Btrfs: only process as many file extents as there are refs The backref walking code will search down to the key it is looking for and then proceed to walk _all_ of the extents on the file until it hits the end. This is suboptimal with large files, we only need to look for as many extents as we have references for that inode. I have a testcase that creates a randomly written 4 gig file and before this patch it took 6min 30sec to do the initial send, with this patch it takes 2min 30sec to do the intial send. Thanks, Signed-off-by: Josef Bacik <jbacik@fb.com> Signed-off-by: Chris Mason <clm@fb.com> This appears to be fixed in 3.15-rc5, but I wonder if either the extra fixes, or a revert of the above should be applied to 3.14 stable? David ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2014-05-10 8:13 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-02-25 6:36 3.14.0rc3: did not find backref in send_root Marc MERLIN 2014-02-25 7:50 ` Wang Shilong 2014-02-25 17:30 ` 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error Marc MERLIN 2014-02-26 3:38 ` Wang Shilong 2014-02-26 7:46 ` 3.14.0rc3: did not find backref in send_root Marc MERLIN 2014-02-26 7:51 ` Wang Shilong 2014-02-26 17:16 ` Marc MERLIN 2014-05-06 6:10 ` David Brown 2014-05-06 6:49 ` Blaz Repas 2014-05-10 8:13 ` David Brown
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.