3.14.0rc3: did not find backref in send_root

3.14.0rc3: did not find backref in send_root

[Marc MERLIN]

I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560

I'll try a scrub when I've finished my backup, but is there anything I
can run on the file I've found from the inode?

gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve  -v 22672 file.mp3
ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0
file.mp3

Anything else?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 3.14.0rc3: did not find backref in send_root

[Wang Shilong]

Hi Marc,

This seems a regression which has been fixed by the following
commit(only pushed into btrfs-next):

https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8

Thanks,
Wang

On 02/25/2014 02:36 PM, Marc MERLIN wrote:
> I got this during a btrfs send:
> BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560
>
> I'll try a scrub when I've finished my backup, but is there anything I
> can run on the file I've found from the inode?
>
> gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve  -v 22672 file.mp3
> ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0
> file.mp3
>
> Anything else?
>
> Thanks,
> Marc

Re: 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error

[Marc MERLIN]

On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote:
> Hi Marc,
> 
> This seems a regression which has been fixed by the following
> commit(only pushed into btrfs-next):
> 
> https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8

I'll revert this, thanks.

Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3
(on my laptop this time):

+ btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57
+ btrfs receive /mnt/btrfs_pool2//
At subvol var_ro.20140225_09:27:57
At snapshot var_ro.20140225_09:27:57
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.

Is it another different regression?

Currently it's only happening on one of my subvolumes. The other ones
are still syncing ok.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error

[Wang Shilong]

Hi Marc,

On 02/26/2014 01:30 AM, Marc MERLIN wrote:
> On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote:
>> Hi Marc,
>>
>> This seems a regression which has been fixed by the following
>> commit(only pushed into btrfs-next):
>>
>> https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8
> I'll revert this, thanks.
>
> Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3
> (on my laptop this time):
What is your previous version, give a double check whether the following 
commit exists:

Btrfs: remove transaction from btrfs send(commitid: 41ce9970)

This regression should exist in 3.14 Sine chris firstly pull big btrfs 
request for Linux 3.14.
>
> + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57
> + btrfs receive /mnt/btrfs_pool2//
> At subvol var_ro.20140225_09:27:57
> At snapshot var_ro.20140225_09:27:57
> ERROR: send ioctl failed with -5: Input/output error
> ERROR: unexpected EOF in stream.
Did dmesg output the same things like before?

Hm, we can esaily trigger that regression if there is snapshot and send 
running concurrently.
(balance/scrub ...device operations can also trigger send failure.)

Thanks,
Wang
>
> Is it another different regression?
>
> Currently it's only happening on one of my subvolumes. The other ones
> are still syncing ok.
>
> Thanks,
> Marc

Re: 3.14.0rc3: did not find backref in send_root

[Marc MERLIN]

On Wed, Feb 26, 2014 at 11:38:30AM +0800, Wang Shilong wrote:
> Hi Marc,
> 
> On 02/26/2014 01:30 AM, Marc MERLIN wrote:
> >On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote:
> >>Hi Marc,
> >>
> >>This seems a regression which has been fixed by the following
> >>commit(only pushed into btrfs-next):
> >>
> >>https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8
> >I'll revert this, thanks.
> >
> >Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3
> >(on my laptop this time):
> What is your previous version, give a double check whether the
> following commit exists:
> 
> Btrfs: remove transaction from btrfs send(commitid: 41ce9970)
> 
> This regression should exist in 3.14 Sine chris firstly pull big
> btrfs request for Linux 3.14.
> >
> >+ btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57
> >+ btrfs receive /mnt/btrfs_pool2//
> >At subvol var_ro.20140225_09:27:57
> >At snapshot var_ro.20140225_09:27:57
> >ERROR: send ioctl failed with -5: Input/output error
> >ERROR: unexpected EOF in stream.
> Did dmesg output the same things like before?
> 
> Hm, we can esaily trigger that regression if there is snapshot and
> send running concurrently.
> (balance/scrub ...device operations can also trigger send failure.)

So I was confused because I got this on my server:
On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote:
> I got this during a btrfs send:
> BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560

and that on my laptop:
[82841.563339] BTRFS info (device dm-0): csum failed ino 2590 extent 2397949952 csum 3100992190 wanted 2786811954 mirror 0

I've applied your patch from 
https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8

legolas:/mnt/btrfs_pool1# bash -vx /var/local/scr/btrfs-subvolume-backup -k 5  var /mnt/btrfs_pool2/

still failed on my laptop
+ btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_23:28:47
+ btrfs receive /mnt/btrfs_pool2//
At subvol var_ro.20140225_23:28:47
At snapshot var_ro.20140225_23:28:47
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
[  216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off 192512 csum 3851586574 expected csum 1402824092

Then again, this seems to be my problem on the laptop:
legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1
scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6
	scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 seconds
	total bytes scrubbed: 451.64GiB with 2 errors
	error details: csum=2
	corrected errors: 0, uncorrectable errors: 2, unverified errors: 0

Ok, so that's not a btrfs send problem.
Just out of curiosity, how do I find out which inodes are compromized so
that I can delete/restore them?

Back to my server, I installed the new kernel, and I'm testing send
again. So far it hasn't failed, so there is a chance the patch fixed the
problem, but I'll only know more tomorrow after it's run longer.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: 3.14.0rc3: did not find backref in send_root

[Wang Shilong]

On 02/26/2014 03:46 PM, Marc MERLIN wrote:
> On Wed, Feb 26, 2014 at 11:38:30AM +0800, Wang Shilong wrote:
>> Hi Marc,
>>
>> On 02/26/2014 01:30 AM, Marc MERLIN wrote:
>>> On Tue, Feb 25, 2014 at 03:50:15PM +0800, Wang Shilong wrote:
>>>> Hi Marc,
>>>>
>>>> This seems a regression which has been fixed by the following
>>>> commit(only pushed into btrfs-next):
>>>>
>>>> https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8
>>> I'll revert this, thanks.
>>>
>>> Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3
>>> (on my laptop this time):
>> What is your previous version, give a double check whether the
>> following commit exists:
>>
>> Btrfs: remove transaction from btrfs send(commitid: 41ce9970)
>>
>> This regression should exist in 3.14 Sine chris firstly pull big
>> btrfs request for Linux 3.14.
>>> + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57
>>> + btrfs receive /mnt/btrfs_pool2//
>>> At subvol var_ro.20140225_09:27:57
>>> At snapshot var_ro.20140225_09:27:57
>>> ERROR: send ioctl failed with -5: Input/output error
>>> ERROR: unexpected EOF in stream.
>> Did dmesg output the same things like before?
>>
>> Hm, we can esaily trigger that regression if there is snapshot and
>> send running concurrently.
>> (balance/scrub ...device operations can also trigger send failure.)
> So I was confused because I got this on my server:
> On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote:
>> I got this during a btrfs send:
>> BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560
> and that on my laptop:
> [82841.563339] BTRFS info (device dm-0): csum failed ino 2590 extent 2397949952 csum 3100992190 wanted 2786811954 mirror 0
>
> I've applied your patch from
> https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8
>
> legolas:/mnt/btrfs_pool1# bash -vx /var/local/scr/btrfs-subvolume-backup -k 5  var /mnt/btrfs_pool2/
>
> still failed on my laptop
> + btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_23:28:47
> + btrfs receive /mnt/btrfs_pool2//
> At subvol var_ro.20140225_23:28:47
> At snapshot var_ro.20140225_23:28:47
> ERROR: send ioctl failed with -5: Input/output error
> ERROR: unexpected EOF in stream.
> [  216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off 192512 csum 3851586574 expected csum 1402824092
>
> Then again, this seems to be my problem on the laptop:
> legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1
> scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6
> 	scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 seconds
> 	total bytes scrubbed: 451.64GiB with 2 errors
> 	error details: csum=2
> 	corrected errors: 0, uncorrectable errors: 2, unverified errors: 0
>
> Ok, so that's not a btrfs send problem.
> Just out of curiosity, how do I find out which inodes are compromized so
> that I can delete/restore them?
You can use command  "btrfs inspect-internal inode-resolve" which will 
print inode's
corresponding path with it's inode id.

For your case, something like this:

  btrfs inspect-internal inode-resolve -v 2326135 /mnt/btrfs_pool1

Thanks,
Wang
>
> Back to my server, I installed the new kernel, and I'm testing send
> again. So far it hasn't failed, so there is a chance the patch fixed the
> problem, but I'll only know more tomorrow after it's run longer.
>
> Thanks,
> Marc

Re: 3.14.0rc3: did not find backref in send_root

[Marc MERLIN]

On Wed, Feb 26, 2014 at 03:51:37PM +0800, Wang Shilong wrote:
> >I've applied your patch from
> >https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8

I can confirm this fixed the btrfs send error on my server, thank you.

> >At snapshot var_ro.20140225_23:28:47
> >ERROR: send ioctl failed with -5: Input/output error
> >ERROR: unexpected EOF in stream.
> >[  216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off 
> >192512 csum 3851586574 expected csum 1402824092
> >
> >Then again, this seems to be my problem on the laptop:
> >legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1
> >scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6
> >	scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 
> >	seconds
> >	total bytes scrubbed: 451.64GiB with 2 errors
> >	error details: csum=2
> >	corrected errors: 0, uncorrectable errors: 2, unverified errors: 0
> >
> >Ok, so that's not a btrfs send problem.
> >Just out of curiosity, how do I find out which inodes are compromized so
> >that I can delete/restore them?
> You can use command  "btrfs inspect-internal inode-resolve" which will 
> print inode's
> corresponding path with it's inode id.

Yes, sorry, I wasn't clear. I know how to do this.
What I meant is that btrfs scrub tells me there are 2 errors and does not
tell me which inode/subvolume the error are in.

Aah, but they were in syslog, I just hadn't found them until now :)

Ok, I'm all good then.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: 3.14.0rc3: did not find backref in send_root

[David Brown]

On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote:
>I got this during a btrfs send:
>BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560
>
>I'll try a scrub when I've finished my backup, but is there anything I
>can run on the file I've found from the inode?
>
>gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve  -v 22672 file.mp3
>ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0
>file.mp3

I've just seen this error:

   BTRFS error (device sda4): did not find backref in send_root. inode=411890, offset=307200, disk_byte=48100618240 found extent=48100618240

during a send between two snapshots I have.

after moving to 3.14.2.  I've seen it on two filesystems now since
moving to 3.14.  I have the two readonly snapshots if there is
anything helpful I can figure out from them.

Scrub reports no errors, but I don't seem to be able to back up
anything now.

David

Re: 3.14.0rc3: did not find backref in send_root

[Blaz Repas]

On 05/06/2014 08:10 AM, David Brown wrote:
> On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote:
>> I got this during a btrfs send:
>> BTRFS error (device dm-2): did not find backref in send_root. 
>> inode=22672, offset=524288, disk_byte=1490517954560 found 
>> extent=1490517954560
>>
>> I'll try a scrub when I've finished my backup, but is there anything I
>> can run on the file I've found from the inode?
>>
>> gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve  -v 
>> 22672 file.mp3
>> ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0
>> file.mp3
>
> I've just seen this error:
>
>   BTRFS error (device sda4): did not find backref in send_root. 
> inode=411890, offset=307200, disk_byte=48100618240 found 
> extent=48100618240
>
> during a send between two snapshots I have.
>
> after moving to 3.14.2.  I've seen it on two filesystems now since
> moving to 3.14.  I have the two readonly snapshots if there is
> anything helpful I can figure out from them.
>
> Scrub reports no errors, but I don't seem to be able to back up
> anything now.
>
> David
> -- 

I am also seeing this on 3.14.1 (on ArchLinux). Scrub also reports no 
errors.
I could also not do a ful send. Balancing made it better for a while (I 
was able to send a full snapshot of one subvolume, but not another), but 
it did not help. Offline repairing the fs with btrfsck --repair also did 
not affect it.

Blaz

Re: 3.14.0rc3: did not find backref in send_root

[David Brown]

On Mon, May 05, 2014 at 11:10:54PM -0700, David Brown wrote:
>On Mon, Feb 24, 2014 at 10:36:52PM -0800, Marc MERLIN wrote:
>>I got this during a btrfs send:
>>BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560
>>
>>I'll try a scrub when I've finished my backup, but is there anything I
>>can run on the file I've found from the inode?
>>
>>gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve  -v 22672 file.mp3
>>ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0
>>file.mp3
>
>I've just seen this error:
>
>  BTRFS error (device sda4): did not find backref in send_root. inode=411890, offset=307200, disk_byte=48100618240 found extent=48100618240
>
>during a send between two snapshots I have.
>
>after moving to 3.14.2.  I've seen it on two filesystems now since
>moving to 3.14.  I have the two readonly snapshots if there is
>anything helpful I can figure out from them.

After bisecting it seems to be caused by

    commit 7ef81ac86c8a44ab9f4e6e04e1f4c9ea53615b8a
    Author: Josef Bacik <jbacik@fb.com>
    Date:   Fri Jan 24 14:05:42 2014 -0500

        Btrfs: only process as many file extents as there are refs

        The backref walking code will search down to the key it is looking for and then
        proceed to walk _all_ of the extents on the file until it hits the end.  This is
        suboptimal with large files, we only need to look for as many extents as we have
        references for that inode.  I have a testcase that creates a randomly written 4
        gig file and before this patch it took 6min 30sec to do the initial send, with
        this patch it takes 2min 30sec to do the intial send.  Thanks,

        Signed-off-by: Josef Bacik <jbacik@fb.com>
        Signed-off-by: Chris Mason <clm@fb.com>

This appears to be fixed in 3.15-rc5, but I wonder if either the extra
fixes, or a revert of the above should be applied to 3.14 stable?

David