btrfs receive fails with "failed to clone extents"

btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

Hi,

The problem is as the title states. Relevant logs from `btrfs receive -vvv`:

  mkfile o119493905-1537066-0
  rename o119493905-1537066-0 ->
shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
  utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
  clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
- source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
source offset=0 offset=0 length=131072
  ERROR: failed to clone extents to
shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
Invalid argument

stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
on the receiving end:

  File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
  Size: 145904 Blocks: 288 IO Block: 4096 regular file

Looks to me the range of clone is within the boundary of the source
file. Not sure why this failed?

Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
and btrfs-progs 5.13.1

-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Qu Wenruo]

On 2021/9/23 03:37, Yuxuan Shui wrote:
> Hi,
>
> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
>
>    mkfile o119493905-1537066-0
>    rename o119493905-1537066-0 ->
> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>    utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
>    clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> source offset=0 offset=0 length=131072
>    ERROR: failed to clone extents to
> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> Invalid argument
>
> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> on the receiving end:
>
>    File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>    Size: 145904 Blocks: 288 IO Block: 4096 regular file
>
> Looks to me the range of clone is within the boundary of the source
> file. Not sure why this failed?

The most common reason is, you have changed the parent subvolume from RO
to RW, and modified the parent subvolume, then converted it back to RO.

Btrfs should not allow such incremental send at all.

We're already working on such problem, but next time if you want to
modify a RO subvolume which could be the parent subvolume of incremental
send, please either do a snapshot then modify the snapshot, or just
don't do it.

Thanks,
Qu

>
> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> and btrfs-progs 5.13.1
>

Re: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

Hi,

On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/23 03:37, Yuxuan Shui wrote:
> > Hi,
> >
> > The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
> >
> >    mkfile o119493905-1537066-0
> >    rename o119493905-1537066-0 ->
> > shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >    utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
> >    clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> > - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> > source offset=0 offset=0 length=131072
> >    ERROR: failed to clone extents to
> > shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> > Invalid argument
> >
> > stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> > on the receiving end:
> >
> >    File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >    Size: 145904 Blocks: 288 IO Block: 4096 regular file
> >
> > Looks to me the range of clone is within the boundary of the source
> > file. Not sure why this failed?
>
> The most common reason is, you have changed the parent subvolume from RO
> to RW, and modified the parent subvolume, then converted it back to RO.

This is 100% not the case. I created these snapshots as RO right
before sending, and definitely haven't
changed them to RW ever.

>
> Btrfs should not allow such incremental send at all.
>
> We're already working on such problem, but next time if you want to
> modify a RO subvolume which could be the parent subvolume of incremental
> send, please either do a snapshot then modify the snapshot, or just
> don't do it.
>
> Thanks,
> Qu
>
> >
> > Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> > and btrfs-progs 5.13.1
> >



-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

On Thu, Sep 23, 2021 at 2:34 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
>
> Hi,
>
> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2021/9/23 03:37, Yuxuan Shui wrote:
> > > Hi,
> > >
> > > The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
> > >
> > >    mkfile o119493905-1537066-0
> > >    rename o119493905-1537066-0 ->
> > > shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> > >    utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
> > >    clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> > > - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> > > source offset=0 offset=0 length=131072
> > >    ERROR: failed to clone extents to
> > > shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> > > Invalid argument
> > >
> > > stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> > > on the receiving end:
> > >
> > >    File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> > >    Size: 145904 Blocks: 288 IO Block: 4096 regular file
> > >
> > > Looks to me the range of clone is within the boundary of the source
> > > file. Not sure why this failed?
> >
> > The most common reason is, you have changed the parent subvolume from RO
> > to RW, and modified the parent subvolume, then converted it back to RO.
>
> This is 100% not the case. I created these snapshots as RO right
> before sending, and definitely haven't
> changed them to RW ever.

Besides that, I straced the btrfs command and this clone ioctl
definitely looks valid, irregardless of whether the parent snapshot
has been changed or not. The length looks to be aligned (128k), and
the range is within the source file. Why did the clone fail?

>
> >
> > Btrfs should not allow such incremental send at all.
> >
> > We're already working on such problem, but next time if you want to
> > modify a RO subvolume which could be the parent subvolume of incremental
> > send, please either do a snapshot then modify the snapshot, or just
> > don't do it.
> >
> > Thanks,
> > Qu
> >
> > >
> > > Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> > > and btrfs-progs 5.13.1
> > >
>
>
>
> --
>
> Regards
> Yuxuan Shui



-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Qu Wenruo]

On 2021/9/23 09:40, Yuxuan Shui wrote:
> On Thu, Sep 23, 2021 at 2:34 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
>>
>> Hi,
>>
>> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2021/9/23 03:37, Yuxuan Shui wrote:
>>>> Hi,
>>>>
>>>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
>>>>
>>>>     mkfile o119493905-1537066-0
>>>>     rename o119493905-1537066-0 ->
>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>>>>     utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
>>>>     clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>>>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>>>> source offset=0 offset=0 length=131072
>>>>     ERROR: failed to clone extents to
>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
>>>> Invalid argument
>>>>
>>>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
>>>> on the receiving end:
>>>>
>>>>     File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>>>>     Size: 145904 Blocks: 288 IO Block: 4096 regular file
>>>>
>>>> Looks to me the range of clone is within the boundary of the source
>>>> file. Not sure why this failed?
>>>
>>> The most common reason is, you have changed the parent subvolume from RO
>>> to RW, and modified the parent subvolume, then converted it back to RO.
>>
>> This is 100% not the case. I created these snapshots as RO right
>> before sending, and definitely haven't
>> changed them to RW ever.
>
> Besides that, I straced the btrfs command and this clone ioctl
> definitely looks valid, irregardless of whether the parent snapshot
> has been changed or not. The length looks to be aligned (128k), and
> the range is within the source file. Why did the clone fail?

The clone source must not have certain bits like NODATACOW.

If non-incremental send stream works, then it's almost certain it's the
received UUID bug we're working on.

Thanks,
Qu

>
>>
>>>
>>> Btrfs should not allow such incremental send at all.
>>>
>>> We're already working on such problem, but next time if you want to
>>> modify a RO subvolume which could be the parent subvolume of incremental
>>> send, please either do a snapshot then modify the snapshot, or just
>>> don't do it.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
>>>> and btrfs-progs 5.13.1
>>>>
>>
>>
>>
>> --
>>
>> Regards
>> Yuxuan Shui
>
>
>

Re: btrfs receive fails with "failed to clone extents"

[Graham Cobb]

On 23/09/2021 02:34, Yuxuan Shui wrote:
> Hi,
> 
> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2021/9/23 03:37, Yuxuan Shui wrote:
>>> Hi,
>>>
>>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
>>>
>>>    mkfile o119493905-1537066-0
>>>    rename o119493905-1537066-0 ->
>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>>>    utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
>>>    clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>>> source offset=0 offset=0 length=131072
>>>    ERROR: failed to clone extents to
>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
>>> Invalid argument
>>>
>>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
>>> on the receiving end:
>>>
>>>    File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>>>    Size: 145904 Blocks: 288 IO Block: 4096 regular file
>>>
>>> Looks to me the range of clone is within the boundary of the source
>>> file. Not sure why this failed?
>>
>> The most common reason is, you have changed the parent subvolume from RO
>> to RW, and modified the parent subvolume, then converted it back to RO.
> 
> This is 100% not the case. I created these snapshots as RO right
> before sending, and definitely haven't
> changed them to RW ever.

The problem isn't with the snapshots on the sending side, it is with the
snapshots on the receiving side. Are you certain the snapshot on the
receiving end has not been touched in any way (in particular, never been
set to "RW" at any time)?

Graham

Fwd: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

Hi,

(Sorry I forgot to CC linux-btrfs)

On Thu, Sep 23, 2021 at 3:32 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/23 09:40, Yuxuan Shui wrote:
> > On Thu, Sep 23, 2021 at 2:34 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>
> >>>
> >>>
> >>> On 2021/9/23 03:37, Yuxuan Shui wrote:
> >>>> Hi,
> >>>>
> >>>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
> >>>>
> >>>>     mkfile o119493905-1537066-0
> >>>>     rename o119493905-1537066-0 ->
> >>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>>>     utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
> >>>>     clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>>> source offset=0 offset=0 length=131072
> >>>>     ERROR: failed to clone extents to
> >>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> >>>> Invalid argument
> >>>>
> >>>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> >>>> on the receiving end:
> >>>>
> >>>>     File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>>>     Size: 145904 Blocks: 288 IO Block: 4096 regular file
> >>>>
> >>>> Looks to me the range of clone is within the boundary of the source
> >>>> file. Not sure why this failed?
> >>>
> >>> The most common reason is, you have changed the parent subvolume from RO
> >>> to RW, and modified the parent subvolume, then converted it back to RO.
> >>
> >> This is 100% not the case. I created these snapshots as RO right
> >> before sending, and definitely haven't
> >> changed them to RW ever.
> >
> > Besides that, I straced the btrfs command and this clone ioctl
> > definitely looks valid, irregardless of whether the parent snapshot
> > has been changed or not. The length looks to be aligned (128k), and
> > the range is within the source file. Why did the clone fail?
>
> The clone source must not have certain bits like NODATACOW.

lsattr doesn't show anything. The entire file system is mounted with
nodatasum, though. But I assume if this bit is the problem, send won't
fail just on this particular file?

>
> If non-incremental send stream works, then it's almost certain it's the
> received UUID bug we're working on.

I haven't tried non-incremental. If by received UUID bug you meant
this one: https://lore.kernel.org/linux-btrfs/87blnsuv7m.fsf@gmail.com/T/
, then I don't think this is the one. I don't have duplicated received
UUIDs on either end.

>
> Thanks,
> Qu
>
> >
> >>
> >>>
> >>> Btrfs should not allow such incremental send at all.
> >>>
> >>> We're already working on such problem, but next time if you want to
> >>> modify a RO subvolume which could be the parent subvolume of incremental
> >>> send, please either do a snapshot then modify the snapshot, or just
> >>> don't do it.
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>>>
> >>>> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> >>>> and btrfs-progs 5.13.1
> >>>>
> >>
> >>
> >>
> >> --
> >>
> >> Regards
> >> Yuxuan Shui
> >
> >
> >



--

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

Hi,

On Thu, Sep 23, 2021 at 4:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/23 11:09, Yuxuan Shui wrote:
> > Hi,
> >
> > On Thu, Sep 23, 2021 at 3:32 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2021/9/23 09:40, Yuxuan Shui wrote:
> >>> On Thu, Sep 23, 2021 at 2:34 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 2021/9/23 03:37, Yuxuan Shui wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
> >>>>>>
> >>>>>>      mkfile o119493905-1537066-0
> >>>>>>      rename o119493905-1537066-0 ->
> >>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>>>>>      utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
> >>>>>>      clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>>>>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>>>>> source offset=0 offset=0 length=131072
> >>>>>>      ERROR: failed to clone extents to
> >>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> >>>>>> Invalid argument
> >>>>>>
> >>>>>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> >>>>>> on the receiving end:
> >>>>>>
> >>>>>>      File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>>>>>      Size: 145904 Blocks: 288 IO Block: 4096 regular file
> >>>>>>
> >>>>>> Looks to me the range of clone is within the boundary of the source
> >>>>>> file. Not sure why this failed?
> >>>>>
> >>>>> The most common reason is, you have changed the parent subvolume from RO
> >>>>> to RW, and modified the parent subvolume, then converted it back to RO.
> >>>>
> >>>> This is 100% not the case. I created these snapshots as RO right
> >>>> before sending, and definitely haven't
> >>>> changed them to RW ever.
> >>>
> >>> Besides that, I straced the btrfs command and this clone ioctl
> >>> definitely looks valid, irregardless of whether the parent snapshot
> >>> has been changed or not. The length looks to be aligned (128k), and
> >>> the range is within the source file. Why did the clone fail?
> >>
> >> The clone source must not have certain bits like NODATACOW.
> >
> > lsattr doesn't show anything. The entire file system is mounted with
> > nodatasum, though. But I assume if this bit is the problem, send won't
> > fail just on this particular file?
>
> If source and dest file has different NODATASUM flags (one has and the
> other doesn't), then it will also be rejected.
>
> NODATASUM flag won't show up in lsattr, thus you need to use "btrfs
> prop" to check.

I checked but I don't think "btrfs prop" shows the NODATASUM flag. The
flags it displays include ro, label and compression only.

>
> Considering you're mounting with NODATASUM, I guess that could the case.
> Your receive target is inheriting the NODATASUM flag, while the source
> file doesn't have the NODATASUM flag.


Interesting, but the clone is within the same subvolume that is been
received. Won't the whole subvolume get the same flag, since the file
system is mounted with nodatasum?

>
> If that's the case, I guess we may hit a new bug for receive, that we
> should also update the dest file's flags before doing reflink.
>
> Could you remove the partially received (and fialed) subvolume, remount
> the fs without nodatasum, then try again to see if that's the case?
> (It may need to change the destination directory too to remove the
> NODATASUM flag)
>
> Thanks,
> Qu
> >
> >>
> >> If non-incremental send stream works, then it's almost certain it's the
> >> received UUID bug we're working on.
> >
> > I haven't tried non-incremental. If by received UUID bug you meant
> > this one: https://lore.kernel.org/linux-btrfs/87blnsuv7m.fsf@gmail.com/T/
> > , then I don't think this is the one. I don't have duplicated received
> > UUIDs on either end.
> >
> >>
> >> Thanks,
> >> Qu
> >>
> >>>
> >>>>
> >>>>>
> >>>>> Btrfs should not allow such incremental send at all.
> >>>>>
> >>>>> We're already working on such problem, but next time if you want to
> >>>>> modify a RO subvolume which could be the parent subvolume of incremental
> >>>>> send, please either do a snapshot then modify the snapshot, or just
> >>>>> don't do it.
> >>>>>
> >>>>> Thanks,
> >>>>> Qu
> >>>>>
> >>>>>>
> >>>>>> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> >>>>>> and btrfs-progs 5.13.1
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Regards
> >>>> Yuxuan Shui
> >>>
> >>>
> >>>
> >
> >
> >



-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

On Thu, Sep 23, 2021 at 10:27 AM Graham Cobb <g.btrfs@cobb.uk.net> wrote:
>
> On 23/09/2021 02:34, Yuxuan Shui wrote:
> > Hi,
> >
> > On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>
> >>
> >>
> >> On 2021/9/23 03:37, Yuxuan Shui wrote:
> >>> Hi,
> >>>
> >>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
> >>>
> >>>    mkfile o119493905-1537066-0
> >>>    rename o119493905-1537066-0 ->
> >>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>>    utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
> >>>    clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>> source offset=0 offset=0 length=131072
> >>>    ERROR: failed to clone extents to
> >>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> >>> Invalid argument
> >>>
> >>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> >>> on the receiving end:
> >>>
> >>>    File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>>    Size: 145904 Blocks: 288 IO Block: 4096 regular file
> >>>
> >>> Looks to me the range of clone is within the boundary of the source
> >>> file. Not sure why this failed?
> >>
> >> The most common reason is, you have changed the parent subvolume from RO
> >> to RW, and modified the parent subvolume, then converted it back to RO.
> >
> > This is 100% not the case. I created these snapshots as RO right
> > before sending, and definitely haven't
> > changed them to RW ever.
>
> The problem isn't with the snapshots on the sending side, it is with the
> snapshots on the receiving side. Are you certain the snapshot on the
> receiving end has not been touched in any way (in particular, never been
> set to "RW" at any time)?

I can guarantee you that neither ends have ever been set to RW.

>
> Graham



-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

Hi,

On Thu, Sep 23, 2021 at 10:38 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
>
> Hi,
>
> On Thu, Sep 23, 2021 at 4:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2021/9/23 11:09, Yuxuan Shui wrote:
> > > Hi,
> > >
> > > On Thu, Sep 23, 2021 at 3:32 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>
> > >>
> > >>
> > >> On 2021/9/23 09:40, Yuxuan Shui wrote:
> > >>> On Thu, Sep 23, 2021 at 2:34 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On 2021/9/23 03:37, Yuxuan Shui wrote:
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
> > >>>>>>
> > >>>>>>      mkfile o119493905-1537066-0
> > >>>>>>      rename o119493905-1537066-0 ->
> > >>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> > >>>>>>      utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
> > >>>>>>      clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> > >>>>>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> > >>>>>> source offset=0 offset=0 length=131072
> > >>>>>>      ERROR: failed to clone extents to
> > >>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> > >>>>>> Invalid argument
> > >>>>>>
> > >>>>>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> > >>>>>> on the receiving end:
> > >>>>>>
> > >>>>>>      File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> > >>>>>>      Size: 145904 Blocks: 288 IO Block: 4096 regular file
> > >>>>>>
> > >>>>>> Looks to me the range of clone is within the boundary of the source
> > >>>>>> file. Not sure why this failed?
> > >>>>>
> > >>>>> The most common reason is, you have changed the parent subvolume from RO
> > >>>>> to RW, and modified the parent subvolume, then converted it back to RO.
> > >>>>
> > >>>> This is 100% not the case. I created these snapshots as RO right
> > >>>> before sending, and definitely haven't
> > >>>> changed them to RW ever.
> > >>>
> > >>> Besides that, I straced the btrfs command and this clone ioctl
> > >>> definitely looks valid, irregardless of whether the parent snapshot
> > >>> has been changed or not. The length looks to be aligned (128k), and
> > >>> the range is within the source file. Why did the clone fail?
> > >>
> > >> The clone source must not have certain bits like NODATACOW.
> > >
> > > lsattr doesn't show anything. The entire file system is mounted with
> > > nodatasum, though. But I assume if this bit is the problem, send won't
> > > fail just on this particular file?
> >
> > If source and dest file has different NODATASUM flags (one has and the
> > other doesn't), then it will also be rejected.
> >
> > NODATASUM flag won't show up in lsattr, thus you need to use "btrfs
> > prop" to check.
>
> I checked but I don't think "btrfs prop" shows the NODATASUM flag. The
> flags it displays include ro, label and compression only.
>
> >
> > Considering you're mounting with NODATASUM, I guess that could the case.
> > Your receive target is inheriting the NODATASUM flag, while the source
> > file doesn't have the NODATASUM flag.
>
>
> Interesting, but the clone is within the same subvolume that is been
> received. Won't the whole subvolume get the same flag, since the file
> system is mounted with nodatasum?
>
> >
> > If that's the case, I guess we may hit a new bug for receive, that we
> > should also update the dest file's flags before doing reflink.
> >
> > Could you remove the partially received (and fialed) subvolume, remount
> > the fs without nodatasum, then try again to see if that's the case?
> > (It may need to change the destination directory too to remove the
> > NODATASUM flag)

Yes, the send/receive worked without nodatasum.

> >
> > Thanks,
> > Qu
> > >
> > >>
> > >> If non-incremental send stream works, then it's almost certain it's the
> > >> received UUID bug we're working on.
> > >
> > > I haven't tried non-incremental. If by received UUID bug you meant
> > > this one: https://lore.kernel.org/linux-btrfs/87blnsuv7m.fsf@gmail.com/T/
> > > , then I don't think this is the one. I don't have duplicated received
> > > UUIDs on either end.
> > >
> > >>
> > >> Thanks,
> > >> Qu
> > >>
> > >>>
> > >>>>
> > >>>>>
> > >>>>> Btrfs should not allow such incremental send at all.
> > >>>>>
> > >>>>> We're already working on such problem, but next time if you want to
> > >>>>> modify a RO subvolume which could be the parent subvolume of incremental
> > >>>>> send, please either do a snapshot then modify the snapshot, or just
> > >>>>> don't do it.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Qu
> > >>>>>
> > >>>>>>
> > >>>>>> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> > >>>>>> and btrfs-progs 5.13.1
> > >>>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>>
> > >>>> Regards
> > >>>> Yuxuan Shui
> > >>>
> > >>>
> > >>>
> > >
> > >
> > >
>
>
>
> --
>
> Regards
> Yuxuan Shui



-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Qu Wenruo]

On 2021/9/23 17:43, Yuxuan Shui wrote:
> Hi,
> 
> On Thu, Sep 23, 2021 at 10:38 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
>>
>> Hi,
>>
>> On Thu, Sep 23, 2021 at 4:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2021/9/23 11:09, Yuxuan Shui wrote:
>>>> Hi,
>>>>
>>>> On Thu, Sep 23, 2021 at 3:32 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2021/9/23 09:40, Yuxuan Shui wrote:
>>>>>> On Thu, Sep 23, 2021 at 2:34 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2021/9/23 03:37, Yuxuan Shui wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
>>>>>>>>>
>>>>>>>>>       mkfile o119493905-1537066-0
>>>>>>>>>       rename o119493905-1537066-0 ->
>>>>>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>>>>>>>>>       utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
>>>>>>>>>       clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>>>>>>>>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>>>>>>>>> source offset=0 offset=0 length=131072
>>>>>>>>>       ERROR: failed to clone extents to
>>>>>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
>>>>>>>>> Invalid argument
>>>>>>>>>
>>>>>>>>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
>>>>>>>>> on the receiving end:
>>>>>>>>>
>>>>>>>>>       File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>>>>>>>>>       Size: 145904 Blocks: 288 IO Block: 4096 regular file
>>>>>>>>>
>>>>>>>>> Looks to me the range of clone is within the boundary of the source
>>>>>>>>> file. Not sure why this failed?
>>>>>>>>
>>>>>>>> The most common reason is, you have changed the parent subvolume from RO
>>>>>>>> to RW, and modified the parent subvolume, then converted it back to RO.
>>>>>>>
>>>>>>> This is 100% not the case. I created these snapshots as RO right
>>>>>>> before sending, and definitely haven't
>>>>>>> changed them to RW ever.
>>>>>>
>>>>>> Besides that, I straced the btrfs command and this clone ioctl
>>>>>> definitely looks valid, irregardless of whether the parent snapshot
>>>>>> has been changed or not. The length looks to be aligned (128k), and
>>>>>> the range is within the source file. Why did the clone fail?
>>>>>
>>>>> The clone source must not have certain bits like NODATACOW.
>>>>
>>>> lsattr doesn't show anything. The entire file system is mounted with
>>>> nodatasum, though. But I assume if this bit is the problem, send won't
>>>> fail just on this particular file?
>>>
>>> If source and dest file has different NODATASUM flags (one has and the
>>> other doesn't), then it will also be rejected.
>>>
>>> NODATASUM flag won't show up in lsattr, thus you need to use "btrfs
>>> prop" to check.
>>
>> I checked but I don't think "btrfs prop" shows the NODATASUM flag. The
>> flags it displays include ro, label and compression only.
>>
>>>
>>> Considering you're mounting with NODATASUM, I guess that could the case.
>>> Your receive target is inheriting the NODATASUM flag, while the source
>>> file doesn't have the NODATASUM flag.
>>
>>
>> Interesting, but the clone is within the same subvolume that is been
>> received. Won't the whole subvolume get the same flag, since the file
>> system is mounted with nodatasum?
>>
>>>
>>> If that's the case, I guess we may hit a new bug for receive, that we
>>> should also update the dest file's flags before doing reflink.
>>>
>>> Could you remove the partially received (and fialed) subvolume, remount
>>> the fs without nodatasum, then try again to see if that's the case?
>>> (It may need to change the destination directory too to remove the
>>> NODATASUM flag)
> 
> Yes, the send/receive worked without nodatasum.

Mind to provide the full send stream dump?

This indeed looks like a bug now.

Thanks,
Qu

> 
>>>
>>> Thanks,
>>> Qu
>>>>
>>>>>
>>>>> If non-incremental send stream works, then it's almost certain it's the
>>>>> received UUID bug we're working on.
>>>>
>>>> I haven't tried non-incremental. If by received UUID bug you meant
>>>> this one: https://lore.kernel.org/linux-btrfs/87blnsuv7m.fsf@gmail.com/T/
>>>> , then I don't think this is the one. I don't have duplicated received
>>>> UUIDs on either end.
>>>>
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Btrfs should not allow such incremental send at all.
>>>>>>>>
>>>>>>>> We're already working on such problem, but next time if you want to
>>>>>>>> modify a RO subvolume which could be the parent subvolume of incremental
>>>>>>>> send, please either do a snapshot then modify the snapshot, or just
>>>>>>>> don't do it.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
>>>>>>>>> and btrfs-progs 5.13.1
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Regards
>>>>>>> Yuxuan Shui
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>> --
>>
>> Regards
>> Yuxuan Shui
> 
> 
> 

Re: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

Hi,

On Thu, Sep 23, 2021 at 10:45 AM Qu Wenruo <wqu@suse.com> wrote:
>
>
>
> On 2021/9/23 17:43, Yuxuan Shui wrote:
> > Hi,
> >
> > On Thu, Sep 23, 2021 at 10:38 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> On Thu, Sep 23, 2021 at 4:28 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>
> >>>
> >>>
> >>> On 2021/9/23 11:09, Yuxuan Shui wrote:
> >>>> Hi,
> >>>>
> >>>> On Thu, Sep 23, 2021 at 3:32 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 2021/9/23 09:40, Yuxuan Shui wrote:
> >>>>>> On Thu, Sep 23, 2021 at 2:34 AM Yuxuan Shui <yshuiv7@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> On Thu, Sep 23, 2021 at 12:24 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2021/9/23 03:37, Yuxuan Shui wrote:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
> >>>>>>>>>
> >>>>>>>>>       mkfile o119493905-1537066-0
> >>>>>>>>>       rename o119493905-1537066-0 ->
> >>>>>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>>>>>>>>       utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
> >>>>>>>>>       clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> >>>>>>>>> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>>>>>>>> source offset=0 offset=0 length=131072
> >>>>>>>>>       ERROR: failed to clone extents to
> >>>>>>>>> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> >>>>>>>>> Invalid argument
> >>>>>>>>>
> >>>>>>>>> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> >>>>>>>>> on the receiving end:
> >>>>>>>>>
> >>>>>>>>>       File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> >>>>>>>>>       Size: 145904 Blocks: 288 IO Block: 4096 regular file
> >>>>>>>>>
> >>>>>>>>> Looks to me the range of clone is within the boundary of the source
> >>>>>>>>> file. Not sure why this failed?
> >>>>>>>>
> >>>>>>>> The most common reason is, you have changed the parent subvolume from RO
> >>>>>>>> to RW, and modified the parent subvolume, then converted it back to RO.
> >>>>>>>
> >>>>>>> This is 100% not the case. I created these snapshots as RO right
> >>>>>>> before sending, and definitely haven't
> >>>>>>> changed them to RW ever.
> >>>>>>
> >>>>>> Besides that, I straced the btrfs command and this clone ioctl
> >>>>>> definitely looks valid, irregardless of whether the parent snapshot
> >>>>>> has been changed or not. The length looks to be aligned (128k), and
> >>>>>> the range is within the source file. Why did the clone fail?
> >>>>>
> >>>>> The clone source must not have certain bits like NODATACOW.
> >>>>
> >>>> lsattr doesn't show anything. The entire file system is mounted with
> >>>> nodatasum, though. But I assume if this bit is the problem, send won't
> >>>> fail just on this particular file?
> >>>
> >>> If source and dest file has different NODATASUM flags (one has and the
> >>> other doesn't), then it will also be rejected.
> >>>
> >>> NODATASUM flag won't show up in lsattr, thus you need to use "btrfs
> >>> prop" to check.
> >>
> >> I checked but I don't think "btrfs prop" shows the NODATASUM flag. The
> >> flags it displays include ro, label and compression only.
> >>
> >>>
> >>> Considering you're mounting with NODATASUM, I guess that could the case.
> >>> Your receive target is inheriting the NODATASUM flag, while the source
> >>> file doesn't have the NODATASUM flag.
> >>
> >>
> >> Interesting, but the clone is within the same subvolume that is been
> >> received. Won't the whole subvolume get the same flag, since the file
> >> system is mounted with nodatasum?
> >>
> >>>
> >>> If that's the case, I guess we may hit a new bug for receive, that we
> >>> should also update the dest file's flags before doing reflink.
> >>>
> >>> Could you remove the partially received (and fialed) subvolume, remount
> >>> the fs without nodatasum, then try again to see if that's the case?
> >>> (It may need to change the destination directory too to remove the
> >>> NODATASUM flag)
> >
> > Yes, the send/receive worked without nodatasum.
>
> Mind to provide the full send stream dump?
>
> This indeed looks like a bug now.

Sorry, this might have data I am not allowed to share, and I don't
think I will be able to vet all the files in the stream.

But if you can let me know what I can do to help debugging, I could try that.

>
> Thanks,
> Qu
>
> >
> >>>
> >>> Thanks,
> >>> Qu
> >>>>
> >>>>>
> >>>>> If non-incremental send stream works, then it's almost certain it's the
> >>>>> received UUID bug we're working on.
> >>>>
> >>>> I haven't tried non-incremental. If by received UUID bug you meant
> >>>> this one: https://lore.kernel.org/linux-btrfs/87blnsuv7m.fsf@gmail.com/T/
> >>>> , then I don't think this is the one. I don't have duplicated received
> >>>> UUIDs on either end.
> >>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Qu
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Btrfs should not allow such incremental send at all.
> >>>>>>>>
> >>>>>>>> We're already working on such problem, but next time if you want to
> >>>>>>>> modify a RO subvolume which could be the parent subvolume of incremental
> >>>>>>>> send, please either do a snapshot then modify the snapshot, or just
> >>>>>>>> don't do it.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Qu
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> >>>>>>>>> and btrfs-progs 5.13.1
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Regards
> >>>>>>> Yuxuan Shui
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >>
> >> --
> >>
> >> Regards
> >> Yuxuan Shui
> >
> >
> >
>


-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Filipe Manana]

On Wed, Sep 22, 2021 at 8:40 PM Yuxuan Shui <yshuiv7@gmail.com> wrote:
>
> Hi,
>
> The problem is as the title states. Relevant logs from `btrfs receive -vvv`:
>
>   mkfile o119493905-1537066-0
>   rename o119493905-1537066-0 ->
> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
>   utimes shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include
>   clone shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h
> - source=shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
> source offset=0 offset=0 length=131072
>   ERROR: failed to clone extents to
> shui/programs/treeusage/target/release/build/zstd-sys-506c8effd111251c/out/include/zstd.h:
> Invalid argument
>
> stat of shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h,
> on the receiving end:
>
>   File: /mnt/backup/home/backup-32/shui/.cargo/registry/src/github.com-1ecc6299db9ec823/zstd-sys-1.6.1+zstd.1.5.0/zstd/lib/zstd.h
>   Size: 145904 Blocks: 288 IO Block: 4096 regular file
>
> Looks to me the range of clone is within the boundary of the source
> file. Not sure why this failed?
>
> Sending end has 5.14.6 and btrfs-progs 5.14, receiving end has 5.14.6
> and btrfs-progs 5.13.1

So, this may or may not be due to the case Darrel ran into recently:

https://lore.kernel.org/linux-btrfs/CAOaVUnV3L6RpcqJ5gaqzNXWXK0VMkEVXCdihawH1PgS6TiMchQ@mail.gmail.com/

If you follow the thread, there's a list of steps and things useful
for debugging there.

Also, to really check if you have snapshots with a duplicated
received_uuid, run the following on each filesystem (i.e. sending side
and receiving side):

btrfs subvolume list -puqR <mount point sending side>
btrfs subvolume list -puqR <mount point receiving side>

Finally, what's the exact command you are using to do the incremental
send? Are you passing 1 or more clone roots (-c option), or just -p?

Also, when you 'btrfs receive -vvv', what's the very first line?
It should be something like this:

receiving snapshot mysnap2 uuid=3208d767-fa97-3340-b5d5-33f64ba76458,
ctransid=7 parent_uuid=d96591a4-7fbf-d44e-a8a8-c46fd1c7380d,
parent_ctransid=6

The output of receive -vvv is not enough to take any conclusion yet.
The operation will be invalid if we end up trying to clone from the
same file and root, because source and destination offsets are the
same.
But if it's from a different root and file, or just from a different
root but the same file, then it may or may not be invalid, depending
on the source file size.

One way to help figure that out is to use one of debug patches (for
btrfs-progs) I pasted on that thread, which is also at
https://pastebin.com/raw/9CbN9C0H

Thanks.

>
> --
>
> Regards
> Yuxuan Shui



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: btrfs receive fails with "failed to clone extents"

[Qu Wenruo]

On 2021/9/23 18:08, Yuxuan Shui wrote:
> Hi,
> 
> On Thu, Sep 23, 2021 at 10:45 AM Qu Wenruo <wqu@suse.com> wrote:
>>
>> Mind to provide the full send stream dump?
>>
>> This indeed looks like a bug now.
> 
> Sorry, this might have data I am not allowed to share, and I don't
> think I will be able to vet all the files in the stream.
> 
> But if you can let me know what I can do to help debugging, I could try that.

That's totally understandable.

Surprisingly, I can't even find a proper way to get the nodatasum flag 
per-file.
Thus I guess things may go complex by using "btrfs ins dump-tree"

For the receive failure case, mind to provide the following info?

- The inode number of the clone source
   You can use 'stat' command to get the inode number:
   $ stat /mnt/btrfs/file  | grep Inode
   Device: 29h/41d	Inode: 257         Links: 1

- The tree dump of the clone source
   $ btrfs ins dump-tree -t <subvolid> <device> | \
     grep "(<INODE> INODE_ITEM 0) item" -A 5
   	item 4 key (257 INODE_ITEM 0) itemoff 15883 itemsize 160
		generation 7 transid 7 size 0 nbytes 0
		block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
		sequence 3 flags 0x0(none)
		atime 1632391968.333333415 (2021-09-23 18:12:48)
		ctime 1632391968.333333415 (2021-09-23 18:12:48)

    <subvolid> is the subvolume id of the clone source.

- The tree dump of the clone dest directory
   The inode number can be get using the same 'stat' command.
   Then pass it to the same "btrfs ins dump-tree" command, with
   <subvolid> <device> and <INODE> modified.

Then we can be sure what's causing the NODATASUM flag mismatch.

Thanks,
Qu

Re: btrfs receive fails with "failed to clone extents"

[Filipe Manana]

On Thu, Sep 23, 2021 at 11:20 AM Qu Wenruo <wqu@suse.com> wrote:
>
>
>
> On 2021/9/23 18:08, Yuxuan Shui wrote:
> > Hi,
> >
> > On Thu, Sep 23, 2021 at 10:45 AM Qu Wenruo <wqu@suse.com> wrote:
> >>
> >> Mind to provide the full send stream dump?
> >>
> >> This indeed looks like a bug now.
> >
> > Sorry, this might have data I am not allowed to share, and I don't
> > think I will be able to vet all the files in the stream.
> >
> > But if you can let me know what I can do to help debugging, I could try that.
>
> That's totally understandable.
>
> Surprisingly, I can't even find a proper way to get the nodatasum flag
> per-file.
> Thus I guess things may go complex by using "btrfs ins dump-tree"
>
> For the receive failure case, mind to provide the following info?
>
> - The inode number of the clone source
>    You can use 'stat' command to get the inode number:
>    $ stat /mnt/btrfs/file  | grep Inode
>    Device: 29h/41d      Inode: 257         Links: 1
>
> - The tree dump of the clone source
>    $ btrfs ins dump-tree -t <subvolid> <device> | \
>      grep "(<INODE> INODE_ITEM 0) item" -A 5
>         item 4 key (257 INODE_ITEM 0) itemoff 15883 itemsize 160
>                 generation 7 transid 7 size 0 nbytes 0
>                 block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>                 sequence 3 flags 0x0(none)
>                 atime 1632391968.333333415 (2021-09-23 18:12:48)
>                 ctime 1632391968.333333415 (2021-09-23 18:12:48)
>
>     <subvolid> is the subvolume id of the clone source.
>
> - The tree dump of the clone dest directory
>    The inode number can be get using the same 'stat' command.
>    Then pass it to the same "btrfs ins dump-tree" command, with
>    <subvolid> <device> and <INODE> modified.
>
> Then we can be sure what's causing the NODATASUM flag mismatch.

Catching up on the whole thread, that is indeed a possible cause of failure.

Someone can do something like:

1) mount fs without -o nodatacow and without -o nodatasum
2) receive snapshot A
3) umount fs
4) mount fs with -o nodatacow or -o nodatasum
5) start receiving an incremental stream that has A as parent and
snapshot B as the send snapshot
6) files created in B end up with the NODATASUM flag set
7) the send stream has a clone operation to clone from some file in A
to a file in B - this fails, as the file in A does not have the
nodatasum bit but the file in B has it

I'm thinking that for cases like this, we could use a flag for send to
tell it to not generate clone operations and do regular write commands
instead.

>
> Thanks,
> Qu
>


-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: btrfs receive fails with "failed to clone extents"

[Qu Wenruo]

On 2021/9/23 18:46, Filipe Manana wrote:
> On Thu, Sep 23, 2021 at 11:20 AM Qu Wenruo <wqu@suse.com> wrote:
>>
>>
>>
>> On 2021/9/23 18:08, Yuxuan Shui wrote:
>>> Hi,
>>>
>>> On Thu, Sep 23, 2021 at 10:45 AM Qu Wenruo <wqu@suse.com> wrote:
>>>>
>>>> Mind to provide the full send stream dump?
>>>>
>>>> This indeed looks like a bug now.
>>>
>>> Sorry, this might have data I am not allowed to share, and I don't
>>> think I will be able to vet all the files in the stream.
>>>
>>> But if you can let me know what I can do to help debugging, I could try that.
>>
>> That's totally understandable.
>>
>> Surprisingly, I can't even find a proper way to get the nodatasum flag
>> per-file.
>> Thus I guess things may go complex by using "btrfs ins dump-tree"
>>
>> For the receive failure case, mind to provide the following info?
>>
>> - The inode number of the clone source
>>     You can use 'stat' command to get the inode number:
>>     $ stat /mnt/btrfs/file  | grep Inode
>>     Device: 29h/41d      Inode: 257         Links: 1
>>
>> - The tree dump of the clone source
>>     $ btrfs ins dump-tree -t <subvolid> <device> | \
>>       grep "(<INODE> INODE_ITEM 0) item" -A 5
>>          item 4 key (257 INODE_ITEM 0) itemoff 15883 itemsize 160
>>                  generation 7 transid 7 size 0 nbytes 0
>>                  block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>>                  sequence 3 flags 0x0(none)
>>                  atime 1632391968.333333415 (2021-09-23 18:12:48)
>>                  ctime 1632391968.333333415 (2021-09-23 18:12:48)
>>
>>      <subvolid> is the subvolume id of the clone source.
>>
>> - The tree dump of the clone dest directory
>>     The inode number can be get using the same 'stat' command.
>>     Then pass it to the same "btrfs ins dump-tree" command, with
>>     <subvolid> <device> and <INODE> modified.
>>
>> Then we can be sure what's causing the NODATASUM flag mismatch.
>
> Catching up on the whole thread, that is indeed a possible cause of failure.
>
> Someone can do something like:
>
> 1) mount fs without -o nodatacow and without -o nodatasum
> 2) receive snapshot A
> 3) umount fs
> 4) mount fs with -o nodatacow or -o nodatasum
> 5) start receiving an incremental stream that has A as parent and
> snapshot B as the send snapshot
> 6) files created in B end up with the NODATASUM flag set
> 7) the send stream has a clone operation to clone from some file in A
> to a file in B - this fails, as the file in A does not have the
> nodatasum bit but the file in B has it

Exactly.

We can craft a test case for it (but without a fix for a while).

>
> I'm thinking that for cases like this, we could use a flag for send to
> tell it to not generate clone operations and do regular write commands
> instead.

My biggest concern here is, we should properly include the inode flags
in the send stream, and provide an interface to change btrfs specific
flags (currently only NODATASUM) at the receive side.

With that we can set the proper inode flags before writing/cloning, no
matter the mount option.

But this is a big project, involving changing send stream format,
introducing new ioctl interface.

Thanks,
Qu

>
>>
>> Thanks,
>> Qu
>>
>
>

Re: btrfs receive fails with "failed to clone extents"

[Filipe Manana]

On Thu, Sep 23, 2021 at 12:16 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/23 18:46, Filipe Manana wrote:
> > On Thu, Sep 23, 2021 at 11:20 AM Qu Wenruo <wqu@suse.com> wrote:
> >>
> >>
> >>
> >> On 2021/9/23 18:08, Yuxuan Shui wrote:
> >>> Hi,
> >>>
> >>> On Thu, Sep 23, 2021 at 10:45 AM Qu Wenruo <wqu@suse.com> wrote:
> >>>>
> >>>> Mind to provide the full send stream dump?
> >>>>
> >>>> This indeed looks like a bug now.
> >>>
> >>> Sorry, this might have data I am not allowed to share, and I don't
> >>> think I will be able to vet all the files in the stream.
> >>>
> >>> But if you can let me know what I can do to help debugging, I could try that.
> >>
> >> That's totally understandable.
> >>
> >> Surprisingly, I can't even find a proper way to get the nodatasum flag
> >> per-file.
> >> Thus I guess things may go complex by using "btrfs ins dump-tree"
> >>
> >> For the receive failure case, mind to provide the following info?
> >>
> >> - The inode number of the clone source
> >>     You can use 'stat' command to get the inode number:
> >>     $ stat /mnt/btrfs/file  | grep Inode
> >>     Device: 29h/41d      Inode: 257         Links: 1
> >>
> >> - The tree dump of the clone source
> >>     $ btrfs ins dump-tree -t <subvolid> <device> | \
> >>       grep "(<INODE> INODE_ITEM 0) item" -A 5
> >>          item 4 key (257 INODE_ITEM 0) itemoff 15883 itemsize 160
> >>                  generation 7 transid 7 size 0 nbytes 0
> >>                  block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> >>                  sequence 3 flags 0x0(none)
> >>                  atime 1632391968.333333415 (2021-09-23 18:12:48)
> >>                  ctime 1632391968.333333415 (2021-09-23 18:12:48)
> >>
> >>      <subvolid> is the subvolume id of the clone source.
> >>
> >> - The tree dump of the clone dest directory
> >>     The inode number can be get using the same 'stat' command.
> >>     Then pass it to the same "btrfs ins dump-tree" command, with
> >>     <subvolid> <device> and <INODE> modified.
> >>
> >> Then we can be sure what's causing the NODATASUM flag mismatch.
> >
> > Catching up on the whole thread, that is indeed a possible cause of failure.
> >
> > Someone can do something like:
> >
> > 1) mount fs without -o nodatacow and without -o nodatasum
> > 2) receive snapshot A
> > 3) umount fs
> > 4) mount fs with -o nodatacow or -o nodatasum
> > 5) start receiving an incremental stream that has A as parent and
> > snapshot B as the send snapshot
> > 6) files created in B end up with the NODATASUM flag set
> > 7) the send stream has a clone operation to clone from some file in A
> > to a file in B - this fails, as the file in A does not have the
> > nodatasum bit but the file in B has it
>
> Exactly.
>
> We can craft a test case for it (but without a fix for a while).
>
> >
> > I'm thinking that for cases like this, we could use a flag for send to
> > tell it to not generate clone operations and do regular write commands
> > instead.
>
> My biggest concern here is, we should properly include the inode flags
> in the send stream, and provide an interface to change btrfs specific
> flags (currently only NODATASUM) at the receive side.

That would require changing the send stream format.
Not sure if Omar's v2 stream allow for that.

>
> With that we can set the proper inode flags before writing/cloning, no
> matter the mount option.

Well, making the sending side force the receiving side to have the
flags that the sending has (or doesn't have), is not something I'm
sure that everyone wants.

Compression for example - the sending side has no compression, but the
receiving side was mounted with -o compress because it wants
compression.
Having the send side clear the compression flag on every file would be
surprising to many users.

The same may or may not apply to nodatacow or nodatasum. I think there
are people relying on the mount options of the receiving side taking
effect.

>
> But this is a big project, involving changing send stream format,
> introducing new ioctl interface.
>
> Thanks,
> Qu
>
> >
> >>
> >> Thanks,
> >> Qu
> >>
> >
> >



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: btrfs receive fails with "failed to clone extents"

[Yuxuan Shui]

Hello,

On Thu, Sep 23, 2021 at 12:16 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/9/23 18:46, Filipe Manana wrote:
> > On Thu, Sep 23, 2021 at 11:20 AM Qu Wenruo <wqu@suse.com> wrote:
> >>
> >>
> >>
> >> On 2021/9/23 18:08, Yuxuan Shui wrote:
> >>> Hi,
> >>>
> >>> On Thu, Sep 23, 2021 at 10:45 AM Qu Wenruo <wqu@suse.com> wrote:
> >>>>
> >>>> Mind to provide the full send stream dump?
> >>>>
> >>>> This indeed looks like a bug now.
> >>>
> >>> Sorry, this might have data I am not allowed to share, and I don't
> >>> think I will be able to vet all the files in the stream.
> >>>
> >>> But if you can let me know what I can do to help debugging, I could try that.
> >>
> >> That's totally understandable.
> >>
> >> Surprisingly, I can't even find a proper way to get the nodatasum flag
> >> per-file.
> >> Thus I guess things may go complex by using "btrfs ins dump-tree"
> >>
> >> For the receive failure case, mind to provide the following info?
> >>
> >> - The inode number of the clone source
> >>     You can use 'stat' command to get the inode number:
> >>     $ stat /mnt/btrfs/file  | grep Inode
> >>     Device: 29h/41d      Inode: 257         Links: 1
> >>
> >> - The tree dump of the clone source
> >>     $ btrfs ins dump-tree -t <subvolid> <device> | \
> >>       grep "(<INODE> INODE_ITEM 0) item" -A 5
> >>          item 4 key (257 INODE_ITEM 0) itemoff 15883 itemsize 160
> >>                  generation 7 transid 7 size 0 nbytes 0
> >>                  block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> >>                  sequence 3 flags 0x0(none)
> >>                  atime 1632391968.333333415 (2021-09-23 18:12:48)
> >>                  ctime 1632391968.333333415 (2021-09-23 18:12:48)
> >>
> >>      <subvolid> is the subvolume id of the clone source.
> >>
> >> - The tree dump of the clone dest directory
> >>     The inode number can be get using the same 'stat' command.
> >>     Then pass it to the same "btrfs ins dump-tree" command, with
> >>     <subvolid> <device> and <INODE> modified.
> >>
> >> Then we can be sure what's causing the NODATASUM flag mismatch.
> >
> > Catching up on the whole thread, that is indeed a possible cause of failure.
> >
> > Someone can do something like:
> >
> > 1) mount fs without -o nodatacow and without -o nodatasum
> > 2) receive snapshot A
> > 3) umount fs
> > 4) mount fs with -o nodatacow or -o nodatasum
> > 5) start receiving an incremental stream that has A as parent and
> > snapshot B as the send snapshot
> > 6) files created in B end up with the NODATASUM flag set
> > 7) the send stream has a clone operation to clone from some file in A
> > to a file in B - this fails, as the file in A does not have the
> > nodatasum bit but the file in B has it
>
> Exactly.
>
> We can craft a test case for it (but without a fix for a while).

As a quick fix, is it possible to convert existing files that are not
nodatasum to nodatasum?

>
> >
> > I'm thinking that for cases like this, we could use a flag for send to
> > tell it to not generate clone operations and do regular write commands
> > instead.
>
> My biggest concern here is, we should properly include the inode flags
> in the send stream, and provide an interface to change btrfs specific
> flags (currently only NODATASUM) at the receive side.
>
> With that we can set the proper inode flags before writing/cloning, no
> matter the mount option.
>
> But this is a big project, involving changing send stream format,
> introducing new ioctl interface.
>
> Thanks,
> Qu
>
> >
> >>
> >> Thanks,
> >> Qu
> >>
> >
> >

I have a dumb question. As an outsider, it seems to me that cloning
from a file with datasum to a file with nodatasum can have a well
defined semantic? i.e. say I clone from A (datasum) to B (nodatasum),
then reading from B should skip checksum verification, and modifying B
would write data without checksum, and leaving A with checksum.

Is this not practical to implement due to btrfs' internal structure?

Thanks!

-- 

Regards
Yuxuan Shui

Re: btrfs receive fails with "failed to clone extents"

[Qu Wenruo]

On 2021/9/24 17:47, Yuxuan Shui wrote:
> Hello,
>
> On Thu, Sep 23, 2021 at 12:16 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
[...]
>>
>> Exactly.
>>
>> We can craft a test case for it (but without a fix for a while).
>
> As a quick fix, is it possible to convert existing files that are not
> nodatasum to nodatasum?

Well, this is embarrassing now, I don't even know a way to show/set
per-file NODATASUM flag.

So I'm afraid it's not possible yet.

>
>>
>>>
>>> I'm thinking that for cases like this, we could use a flag for send to
>>> tell it to not generate clone operations and do regular write commands
>>> instead.
>>
>> My biggest concern here is, we should properly include the inode flags
>> in the send stream, and provide an interface to change btrfs specific
>> flags (currently only NODATASUM) at the receive side.
>>
>> With that we can set the proper inode flags before writing/cloning, no
>> matter the mount option.
>>
>> But this is a big project, involving changing send stream format,
>> introducing new ioctl interface.
>>
>> Thanks,
>> Qu
>>
>>>
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>
>>>
>
> I have a dumb question. As an outsider, it seems to me that cloning
> from a file with datasum to a file with nodatasum can have a well
> defined semantic? i.e. say I clone from A (datasum) to B (nodatasum),
> then reading from B should skip checksum verification, and modifying B
> would write data without checksum, and leaving A with checksum.

Clone means it will share the on-disk data, without doing real write.

Then the problem is, one inode says the data should has data checksum,
the other says the same data should has no data checksum.

This inconsistent status is prevent us from cloning in the first place.

>
> Is this not practical to implement due to btrfs' internal structure?

I discussed with Filipe, we're more or less fine to fix it in
btrfs-progs (receive side) by fallback back to buffered write (read the
data from the inode, write it back to the destination).

By this, we can workaround the problem as a quick (and maybe final) fix.

Thanks,
Qu

>
> Thanks!
>