cross-fs copy support

cross-fs copy support

[Joshi]

I was wondering about the cross-fs copy through copy_file_range.
It seems current implement has below check, that disables such copy.

1577         /* this could be relaxed once a method supports cross-fs copies */
1578         if (inode_in->i_sb != inode_out->i_sb)
1579                 return -EXDEV;

May I know what are the thoughts behind disabling cross-fs copy?
Code has the comment "once a method supports", but that leaves me
wondering exactly what 'method' is expected, and from whom.

I disabled the check, and copy across volumes seemed to work fine. At
least for a single file (1G size), with no data mismatch, and faster
speed than regular copy.

-- 
Joshi

Re: cross-fs copy support

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1519 bytes --]



On 2018/10/1 下午10:32, Joshi wrote:
> I was wondering about the cross-fs copy through copy_file_range.

The term "cross-fs" looks pretty confusing.

If you mean "cross-subvolume", then it should work without problem in btrfs.

If you mean reflink across two different file systems (not matter the
same fs type or not).
Then it's impossible to work.

Reflink (clone_file_range) works by inserting data pointers into the
filesystem other than really copying the data.
Thus if the source is outside of the fs, it's really impossible to work,
as the source pointer/data is completely out of control of the dest fs.

> It seems current implement has below check, that disables such copy.
> 
> 1577         /* this could be relaxed once a method supports cross-fs copies */
> 1578         if (inode_in->i_sb != inode_out->i_sb)
> 1579                 return -EXDEV;
> 
> May I know what are the thoughts behind disabling cross-fs copy?
> Code has the comment "once a method supports", but that leaves me
> wondering exactly what 'method' is expected, and from whom.
> 
> I disabled the check, and copy across volumes seemed to work fine. At
> least for a single file (1G size), with no data mismatch, and faster
> speed than regular copy.

Please provide the steps or script about how you did the reflink, in
case I misunderstand your "cross-fs" definition.

And just in case you're really doing cross filesystem reflink, please
also run "btrfs check" on both fs.

Thanks,
Qu



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: cross-fs copy support

[Joshi]

Sorry if my post was not clear enough. I picked the term "cross-fs"
from the implementation of vfs_copy_file_range.
Below snippet, inside vfs_copy_file_range -
1578         /* this could be relaxed once a method supports cross-fs copies */
1579         if (inode_in->i_sb != inode_out->i_sb)
1580                 return -EXDEV;

And I was not trying reflink for copy, rather I just used example code
for copy_file_range syscall -
http://man7.org/linux/man-pages/man2/copy_file_range.2.html
This code creates a new file first, and later issues "copy_file_range"
for copying. I was trying this sort of copy among files residing on
btrfs, ext4, and xfs.
Although vfs_copy_file_range internally uses clone_file_range (which
would be true for btrfs and xfs).
On Mon, Oct 1, 2018 at 8:18 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2018/10/1 下午10:32, Joshi wrote:
> > I was wondering about the cross-fs copy through copy_file_range.
>
> The term "cross-fs" looks pretty confusing.
>
> If you mean "cross-subvolume", then it should work without problem in btrfs.
>
> If you mean reflink across two different file systems (not matter the
> same fs type or not).
> Then it's impossible to work.
>
> Reflink (clone_file_range) works by inserting data pointers into the
> filesystem other than really copying the data.
> Thus if the source is outside of the fs, it's really impossible to work,
> as the source pointer/data is completely out of control of the dest fs.
>
> > It seems current implement has below check, that disables such copy.
> >
> > 1577         /* this could be relaxed once a method supports cross-fs copies */
> > 1578         if (inode_in->i_sb != inode_out->i_sb)
> > 1579                 return -EXDEV;
> >
> > May I know what are the thoughts behind disabling cross-fs copy?
> > Code has the comment "once a method supports", but that leaves me
> > wondering exactly what 'method' is expected, and from whom.
> >
> > I disabled the check, and copy across volumes seemed to work fine. At
> > least for a single file (1G size), with no data mismatch, and faster
> > speed than regular copy.
>
> Please provide the steps or script about how you did the reflink, in
> case I misunderstand your "cross-fs" definition.
>
> And just in case you're really doing cross filesystem reflink, please
> also run "btrfs check" on both fs.
>
> Thanks,
> Qu
>
>


-- 
Joshi

Re: cross-fs copy support

[Eric Sandeen]

[-- Attachment #1.1: Type: text/plain, Size: 1485 bytes --]

On 10/1/18 9:48 AM, Qu Wenruo wrote:
> 
> 
> On 2018/10/1 下午10:32, Joshi wrote:
>> I was wondering about the cross-fs copy through copy_file_range.
> 
> The term "cross-fs" looks pretty confusing.
> 
> If you mean "cross-subvolume", then it should work without problem in btrfs.
> 
> If you mean reflink across two different file systems (not matter the
> same fs type or not).
> Then it's impossible to work.

I believe Joshi is talking about vfs_copy_file_range() not
vfs_clone_file range(), although _copy_ does call _clone_ if it can.

> Reflink (clone_file_range) works by inserting data pointers into the
> filesystem other than really copying the data.
> Thus if the source is outside of the fs, it's really impossible to work,
> as the source pointer/data is completely out of control of the dest fs.

Yes, I would expect there to be problems with his modified kernel
for a filesystem that supports clone_file_range, because
vfs_copy_file_range() will clone if possible, and this should fail across
filesystems.

In general, though, I don't know for sure why we don't fall back to
do_splice_direct() across filesystems, although the filesystems that
implement their own ->copy_file_range ops may have their own,
further restrictions within their implementations.

This call /is/ documented in the manpage as only being valid for
files on the same filesystem, though:
http://man7.org/linux/man-pages/man2/copy_file_range.2.html

-Eric


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: cross-fs copy support

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

\x1d\b\x1d\x1dOn Oct 1, 2018, at 9:49 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> 
> 
> On 10/1/18 9:48 AM, Qu Wenruo wrote:
>> 
>> 
>> On 2018/10/1 下午10:32, Joshi wrote:
>>> I was wondering about the cross-fs copy through copy_file_range.
>> 
>> The term "cross-fs" looks pretty confusing.
>> 
>> If you mean "cross-subvolume", then it should work without problem in btrfs.
>> 
>> If you mean reflink across two different file systems (not matter the
>> same fs type or not).
>> Then it's impossible to work.
> 
> I believe Joshi is talking about vfs_copy_file_range() not
> vfs_clone_file range(), although _copy_ does call _clone_ if it can.
> 
>> Reflink (clone_file_range) works by inserting data pointers into the
>> filesystem other than really copying the data.
>> Thus if the source is outside of the fs, it's really impossible to work,
>> as the source pointer/data is completely out of control of the dest fs.
> 
> Yes, I would expect there to be problems with his modified kernel
> for a filesystem that supports clone_file_range, because
> vfs_copy_file_range() will clone if possible, and this should fail across
> filesystems.
> 
> In general, though, I don't know for sure why we don't fall back to
> do_splice_direct() across filesystems, although the filesystems that
> implement their own ->copy_file_range ops may have their own,
> further restrictions within their implementations.
> 
> This call /is/ documented in the manpage as only being valid for
> files on the same filesystem, though:
> http://man7.org/linux/man-pages/man2/copy_file_range.2.html

There was a patch to allow cross-mount copy for NFS, but it hasn't landed
yet.

Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: cross-fs copy support

[David Sterba]

On Mon, Oct 01, 2018 at 01:51:09PM -0600, Andreas Dilger wrote:
> > Yes, I would expect there to be problems with his modified kernel
> > for a filesystem that supports clone_file_range, because
> > vfs_copy_file_range() will clone if possible, and this should fail across
> > filesystems.
> > 
> > In general, though, I don't know for sure why we don't fall back to
> > do_splice_direct() across filesystems, although the filesystems that
> > implement their own ->copy_file_range ops may have their own,
> > further restrictions within their implementations.
> > 
> > This call /is/ documented in the manpage as only being valid for
> > files on the same filesystem, though:
> > http://man7.org/linux/man-pages/man2/copy_file_range.2.html
> 
> There was a patch to allow cross-mount copy for NFS, but it hasn't landed
> yet.

I found https://marc.info/?l=linux-nfs&m=144138779721907&w=2 that lifts
the VFS check (part of a series that can't be easily linked to).

The lack of cross-mount reflink (based on the copy_file_ragne) is often
confusing users, there are common setups that mount subvolumes
separately and reflinking between them would require mount of the
toplevel subvolume.

If there are 2 in-kernel users of the relaxed cross-mount copy, I think
this would help to push the series forward.

Re: cross-fs copy support

[Darrick J. Wong]

On Tue, Oct 02, 2018 at 10:15:44AM +0200, David Sterba wrote:
> On Mon, Oct 01, 2018 at 01:51:09PM -0600, Andreas Dilger wrote:
> > > Yes, I would expect there to be problems with his modified kernel
> > > for a filesystem that supports clone_file_range, because
> > > vfs_copy_file_range() will clone if possible, and this should fail across
> > > filesystems.
> > > 
> > > In general, though, I don't know for sure why we don't fall back to
> > > do_splice_direct() across filesystems, although the filesystems that
> > > implement their own ->copy_file_range ops may have their own,
> > > further restrictions within their implementations.
> > > 
> > > This call /is/ documented in the manpage as only being valid for
> > > files on the same filesystem, though:
> > > http://man7.org/linux/man-pages/man2/copy_file_range.2.html
> > 
> > There was a patch to allow cross-mount copy for NFS, but it hasn't landed
> > yet.
> 
> I found https://marc.info/?l=linux-nfs&m=144138779721907&w=2 that lifts
> the VFS check (part of a series that can't be easily linked to).
> 
> The lack of cross-mount reflink (based on the copy_file_ragne) is often
> confusing users, there are common setups that mount subvolumes
> separately and reflinking between them would require mount of the
> toplevel subvolume.
> 
> If there are 2 in-kernel users of the relaxed cross-mount copy, I think
> this would help to push the series forward.

I don't have any objection to cross-mountpoint same-filesystem clones,
though obviously we need to all agree that from now on the vfs /does/
support certain IO operations across mountpoints.

(I haven't any opinion on cross-filesystem copies, as XFS is incapable
of such things.)

--D

Re: cross-fs copy support

[J. Bruce Fields]

On Mon, Oct 01, 2018 at 01:51:09PM -0600, Andreas Dilger wrote:
> \x1d
\b\x1d
\x1d
On Oct 1, 2018, at 9:49 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> > Yes, I would expect there to be problems with his modified kernel
> > for a filesystem that supports clone_file_range, because
> > vfs_copy_file_range() will clone if possible, and this should fail across
> > filesystems.
> > 
> > In general, though, I don't know for sure why we don't fall back to
> > do_splice_direct() across filesystems, although the filesystems that
> > implement their own ->copy_file_range ops may have their own,
> > further restrictions within their implementations.
> > 
> > This call /is/ documented in the manpage as only being valid for
> > files on the same filesystem, though:
> > http://man7.org/linux/man-pages/man2/copy_file_range.2.html
> 
> There was a patch to allow cross-mount copy for NFS, but it hasn't landed
> yet.

I thought Christoph Hellwig vetoed it partly because he thought NFS
server-to-server copy is too complicated.  Which perhaps it is, but I
suspect we'll do it anyway because the benefit seems obvious.

--b.