Copy tools on Linux

Copy tools on Linux

[Steve French]

I have been looking at i/o patterns from various copy tools on Linux,
and it is pretty discouraging - I am hoping that I am forgetting an
important one that someone can point me to ...

Some general problems:
1) if source and target on the same file system it would be nice to
call the copy_file_range syscall (AFAIK only test tools call that),
although in some cases at least cp can do it for --reflink
2) if source and target on different file systems there are multiple problems
    a) smaller i/o  (rsync e.g. maxes at 128K!)
    b) no async parallelized writes sent down to the kernel so writes
get serialized (either through page cache, or some fs offer option to
disable it - but it still is one thread at a time)
    c) sparse file support is mediocre (although cp has some support
for it, and can call fiemap in some cases)
    d) for file systems that prefer setting the file size first (to
avoid metadata penalties with multiple extending writes) - AFAIK only
rsync offers that, but rsync is one of the slowest tools otherwise

I have looked at cp, dd, scp, rsync, gio, gcp ... are there others?

What I am looking for (and maybe we just need to patch cp and rsync
etc.) is more like what you see with other OS ...
1) options for large i/o sizes (network latencies in network/cluster
fs can be large, so prefer larger 1M or 8M in some cases I/Os)
2) parallelizing writes so not just one write in flight at a time
3) options to turn off the page cache (large number of large file
copies are not going to benefit from reuse of pages in the page cache
so going through the page cache may be suboptimal in that case)
4) option to set the file size first, and then fill in writes (so
non-extending writes)
5) sparse file support
(and it would also be nice to support copy_file_range syscall ... but
that is unrelated to the above)

Am I missing some magic tool?  Seems like Windows has various options
for copy tools - but looking at Linux i/o patterns from these tools
was pretty depressing - I am hoping that there are other choices.

-- 
Thanks,

Steve

Re: Copy tools on Linux

[Goldwyn Rodrigues]

Hi Steve,

On 06-29 21:37, Steve French wrote:
> I have been looking at i/o patterns from various copy tools on Linux,
> and it is pretty discouraging - I am hoping that I am forgetting an
> important one that someone can point me to ...
> 
> Some general problems:
> 1) if source and target on the same file system it would be nice to
> call the copy_file_range syscall (AFAIK only test tools call that),
> although in some cases at least cp can do it for --reflink

I have submitted a patch set for copy_file_range() across filesystems
which can atleast use splice() [1] as a part of enabling holes in
copy_file_range(), but it has not been incorporated so far.

> 2) if source and target on different file systems there are multiple problems
>     a) smaller i/o  (rsync e.g. maxes at 128K!)
>     b) no async parallelized writes sent down to the kernel so writes
> get serialized (either through page cache, or some fs offer option to
> disable it - but it still is one thread at a time)
>     c) sparse file support is mediocre (although cp has some support
> for it, and can call fiemap in some cases)
>     d) for file systems that prefer setting the file size first (to
> avoid metadata penalties with multiple extending writes) - AFAIK only
> rsync offers that, but rsync is one of the slowest tools otherwise
> 
> I have looked at cp, dd, scp, rsync, gio, gcp ... are there others?
> 
> What I am looking for (and maybe we just need to patch cp and rsync
> etc.) is more like what you see with other OS ...
> 1) options for large i/o sizes (network latencies in network/cluster
> fs can be large, so prefer larger 1M or 8M in some cases I/Os)

Unfortunately tools derive I/O size from stat.st_blksize which may be
pretty small for performing "efficient" I/O. However, the tools such as
cp also determine series of zeros to convert into holes. So for that
reason it works well. OTOH, that is not the most common case of tools,
which I agree could be made faster.


> 2) parallelizing writes so not just one write in flight at a time

What would the resultant file be in case of errors? Should the
destination file be considered partially copied? man cp does not cover
the case errors but currently it is assumed the file is partially copied
and correct until the point of error.

> 3) options to turn off the page cache (large number of large file
> copies are not going to benefit from reuse of pages in the page cache
> so going through the page cache may be suboptimal in that case)

In most cases, pagecache is faster than direct I/O. Yes, large files
may not benefit from it. But it would still be faster to use up the
memory and defer writebacks.

> 4) option to set the file size first, and then fill in writes (so
> non-extending writes)

File size or file allocation? How would you determine what file
size to set? Consider the case the source file is sparse. It can be
calculated, but needs more thought.

> 5) sparse file support
> (and it would also be nice to support copy_file_range syscall ... but
> that is unrelated to the above)

Yup, main objective of [1]

> 
> Am I missing some magic tool?  Seems like Windows has various options
> for copy tools - but looking at Linux i/o patterns from these tools
> was pretty depressing - I am hoping that there are other choices.
> 


[1] https://www.spinics.net/lists/linux-fsdevel/msg128450.html

-- 
Goldwyn

Re: Copy tools on Linux

[Steve French]

On Sat, Jun 30, 2018 at 8:13 AM Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>
> Hi Steve,
>
> On 06-29 21:37, Steve French wrote:
> > I have been looking at i/o patterns from various copy tools on Linux,
> > and it is pretty discouraging - I am hoping that I am forgetting an
> > important one that someone can point me to ...
> >
> > Some general problems:
> > 1) if source and target on the same file system it would be nice to
> > call the copy_file_range syscall (AFAIK only test tools call that),
> > although in some cases at least cp can do it for --reflink
>
> I have submitted a patch set for copy_file_range() across filesystems
> which can atleast use splice() [1] as a part of enabling holes in
> copy_file_range(), but it has not been incorporated so far.

Do you have a link to the patch?

> > 1) options for large i/o sizes (network latencies in network/cluster
> > fs can be large, so prefer larger 1M or 8M in some cases I/Os)
>
> Unfortunately tools derive I/O size from stat.st_blksize which may be
> pretty small for performing "efficient" I/O. However, the tools such as
> cp also determine series of zeros to convert into holes. So for that
> reason it works well. OTOH, that is not the most common case of tools,
> which I agree could be made faster.

dd is nice in that you can set i/o size (as can rsync) but seems sane to
allow rsize/wsize to be configurable

> > 2) parallelizing writes so not just one write in flight at a time
>
> What would the resultant file be in case of errors? Should the
> destination file be considered partially copied? man cp does not cover
> the case errors but currently it is assumed the file is partially copied
> and correct until the point of error.

Whether parallel i/o on one file, or multiple files, either will be
a huge help.  Just did a quick google search on the topic and it
pointed to a sysadmin article discussing one of the more
common copy tools on Windows, robocopy:

"Perhaps the most important switch to pay attention is /MT, which is a
feature that enables Robocopy to copy files in multi-threaded mode...
with multi-threaded enabled, you can copy multiple files at the same
time better utilizing the bandwidth and significantly speeding up the
process. If you don’t set a number when using the /MT switch, then the
default number will be 8, which means that Robocopy will try to copy
eight files at the same time. However, Robocopy supports 1 to 128
threads."

This seems sane - even if cp can't do it, having a tool that can
reasonably get at least four i/o in flight (perhaps for different
files, with only one i/o per file) would be huge help.

<snip>
> > 4) option to set the file size first, and then fill in writes (so
> > non-extending writes)
>
> File size or file allocation? How would you determine what file
> size to set? Consider the case the source file is sparse. It can be
> calculated, but needs more thought.

The goal here is to allow a copy option (as rsync does)
for target file systems where metadata sync is expensive
or expensive locking needed for setting end-of-file, set the filesize
early so it doesn't get reset 100s of times on extending writes

-- 
Thanks,

Steve

Re: Copy tools on Linux

[Goldwyn Rodrigues]

On 06-30 09:12, Steve French wrote:
> On Sat, Jun 30, 2018 at 8:13 AM Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> >
> > Hi Steve,
> >
> > On 06-29 21:37, Steve French wrote:
> > > I have been looking at i/o patterns from various copy tools on Linux,
> > > and it is pretty discouraging - I am hoping that I am forgetting an
> > > important one that someone can point me to ...
> > >
> > > Some general problems:
> > > 1) if source and target on the same file system it would be nice to
> > > call the copy_file_range syscall (AFAIK only test tools call that),
> > > although in some cases at least cp can do it for --reflink
> >
> > I have submitted a patch set for copy_file_range() across filesystems
> > which can atleast use splice() [1] as a part of enabling holes in
> > copy_file_range(), but it has not been incorporated so far.
> 
> Do you have a link to the patch?

I posted it in the last mail, as a link to reference [1]
https://www.spinics.net/lists/linux-fsdevel/msg128450.html

> 
> > > 1) options for large i/o sizes (network latencies in network/cluster
> > > fs can be large, so prefer larger 1M or 8M in some cases I/Os)
> >
> > Unfortunately tools derive I/O size from stat.st_blksize which may be
> > pretty small for performing "efficient" I/O. However, the tools such as
> > cp also determine series of zeros to convert into holes. So for that
> > reason it works well. OTOH, that is not the most common case of tools,
> > which I agree could be made faster.
> 
> dd is nice in that you can set i/o size (as can rsync) but seems sane to
> allow rsize/wsize to be configurable
> 
> > > 2) parallelizing writes so not just one write in flight at a time
> >
> > What would the resultant file be in case of errors? Should the
> > destination file be considered partially copied? man cp does not cover
> > the case errors but currently it is assumed the file is partially copied
> > and correct until the point of error.
> 
> Whether parallel i/o on one file, or multiple files, either will be
> a huge help.  Just did a quick google search on the topic and it
> pointed to a sysadmin article discussing one of the more
> common copy tools on Windows, robocopy:
> 
> "Perhaps the most important switch to pay attention is /MT, which is a
> feature that enables Robocopy to copy files in multi-threaded mode...
> with multi-threaded enabled, you can copy multiple files at the same
> time better utilizing the bandwidth and significantly speeding up the
> process. If you don’t set a number when using the /MT switch, then the
> default number will be 8, which means that Robocopy will try to copy
> eight files at the same time. However, Robocopy supports 1 to 128
> threads."
> 
> This seems sane - even if cp can't do it, having a tool that can
> reasonably get at least four i/o in flight (perhaps for different
> files, with only one i/o per file) would be huge help.

I see. Perhaps posting patches to cp would benefit the general folk.
I will look at the coreutils code as well.

> 
> <snip>
> > > 4) option to set the file size first, and then fill in writes (so
> > > non-extending writes)
> >
> > File size or file allocation? How would you determine what file
> > size to set? Consider the case the source file is sparse. It can be
> > calculated, but needs more thought.
> 
> The goal here is to allow a copy option (as rsync does)
> for target file systems where metadata sync is expensive
> or expensive locking needed for setting end-of-file, set the filesize
> early so it doesn't get reset 100s of times on extending writes
> 

This assumes copy succeeds all the way. What happens in case of errors?
A common way to check that file is successfully copied (though stupid)
is compare file sizes.

-- 
Goldwyn

Re: Copy tools on Linux

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 3212 bytes --]

On Jun 29, 2018, at 8:37 PM, Steve French <smfrench@gmail.com> wrote:
> 
> I have been looking at i/o patterns from various copy tools on Linux,
> and it is pretty discouraging - I am hoping that I am forgetting an
> important one that someone can point me to ...
> 
> Some general problems:
> 1) if source and target on the same file system it would be nice to
> call the copy_file_range syscall (AFAIK only test tools call that),
> although in some cases at least cp can do it for --reflink
> 2) if source and target on different file systems there are multiple problems
>    a) smaller i/o  (rsync e.g. maxes at 128K!)
>    b) no async parallelized writes sent down to the kernel so writes
> get serialized (either through page cache, or some fs offer option to
> disable it - but it still is one thread at a time)
>    c) sparse file support is mediocre (although cp has some support
> for it, and can call fiemap in some cases)
>    d) for file systems that prefer setting the file size first (to
> avoid metadata penalties with multiple extending writes) - AFAIK only
> rsync offers that, but rsync is one of the slowest tools otherwise
> 
> I have looked at cp, dd, scp, rsync, gio, gcp ... are there others?

In the HPC world, there are a number of parallel copy tools, like
MPI fileutils (https://github.com/hpc/mpifileutils) which can copy
single files in parallel, as well as whole directory trees.  As the
name implies, it uses MPI for running on multiple nodes, but it may
be possible to modify the tools to run multi-threaded on a single
node as well.


> What I am looking for (and maybe we just need to patch cp and rsync
> etc.) is more like what you see with other OS ...
> 1) options for large i/o sizes (network latencies in network/cluster
> fs can be large, so prefer larger 1M or 8M in some cases I/Os)

Lustre reports a blocksize of 2MB for stat(), which cp(1) uses.  It
probably wouldn't be a bad idea to have it use at least 1MB by default,
instead of only e.g. 4KB for local filesystems.

> 2) parallelizing writes so not just one write in flight at a time
> 3) options to turn off the page cache (large number of large file
> copies are not going to benefit from reuse of pages in the page cache
> so going through the page cache may be suboptimal in that case)

Writing to cache is usually faster than O_DIRECT, until you have dozens
of threads doing parallel writes, or you are using AIO.  Instead of
using O_DIRECT, using posix_fadvise(POSIX_FADV_DONTNEED) to drop the
source file from cache after it is finished is reasonable.  Dropping
the target file after some delay (when copying many files) would also
be useful.

> 4) option to set the file size first, and then fill in writes (so
> non-extending writes)

What about using fallocate()?

> 5) sparse file support
> (and it would also be nice to support copy_file_range syscall ... but
> that is unrelated to the above)
> 
> Am I missing some magic tool?  Seems like Windows has various options
> for copy tools - but looking at Linux i/o patterns from these tools
> was pretty depressing - I am hoping that there are other choices.

Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: Copy tools on Linux

[Dave Chinner]

On Fri, Jun 29, 2018 at 09:37:27PM -0500, Steve French wrote:
> I have been looking at i/o patterns from various copy tools on Linux,
> and it is pretty discouraging - I am hoping that I am forgetting an
> important one that someone can point me to ...
> 
> Some general problems:
> 1) if source and target on the same file system it would be nice to
> call the copy_file_range syscall (AFAIK only test tools call that),
> although in some cases at least cp can do it for --reflink

copy_file_range() should be made to do the right thing in as many
scnearios as we can document, and then switch userspace over to use
it at all times. Aggregate all the knowledge in one place, where we
know what the filesystem implementations are and can get hints to do
the right thing.

> 2) if source and target on different file systems there are multiple problems
>     a) smaller i/o  (rsync e.g. maxes at 128K!)
>     b) no async parallelized writes sent down to the kernel so writes
> get serialized (either through page cache, or some fs offer option to
> disable it - but it still is one thread at a time)

That because, historically, copying data into the page cache for
buffering has been orders of magnitude faster than actually doing
IO. These days, with pcie based storage, not so much. Indeed, for
bulk data copy on nvme based storage I wonder if we even need
buffered IO anymore...

>     c) sparse file support is mediocre (although cp has some support
> for it, and can call fiemap in some cases)

Using fiemap for this is broken and will lead to data corruption,
especially if you start parallelising IO to individual files.
SEEK_DATA/SEEK_HOLE should be used instead.

>     d) for file systems that prefer setting the file size first (to
> avoid metadata penalties with multiple extending writes) - AFAIK only
> rsync offers that, but rsync is one of the slowest tools otherwise

We don't want to do this for most local filesystems as it defeats
things like specualtive preallocation which are used to optimise IO
patterns and prevent file fragmentation when concurrent parallel
writes are done.

> I have looked at cp, dd, scp, rsync, gio, gcp ... are there others?
> 
> What I am looking for (and maybe we just need to patch cp and rsync
> etc.) is more like what you see with other OS ...
> 1) options for large i/o sizes (network latencies in network/cluster
> fs can be large, so prefer larger 1M or 8M in some cases I/Os)

-o largeio mount option on XFS will expose the stripe unit as the
iminimum efficient IO size returned in stat. IIRC there's another
combination that makes it emit the stripe width rather than stripe
unit.

> 2) parallelizing writes so not just one write in flight at a time

That won't make buffered IO any faster - writeback will still be the
bottleneck, and it already does allow many IOs to be in flight at
once.

> 3) options to turn off the page cache (large number of large file
> copies are not going to benefit from reuse of pages in the page cache
> so going through the page cache may be suboptimal in that case)

If you do this, you really need to use AIO+DIO to avoid blocking
(i.e. userspace can still be single threaded!), and need the
filesystem to be able to tell the app what optimal DIO sizes are
(e.g. XFS_IOC_DIOINFO has historically been used for this)

> 4) option to set the file size first, and then fill in writes (so
> non-extending writes)

Must only be an option, as will cause problems with buffered IO
because it defeats all the extending write optimisations that local
filesystems do. Same with using fallocate() to preallocate files -
this defeats all the anti-fragmentation and cross-file allocation
packing optimisations we do at writeback time that are enabled by
delayed allocation.

In general, fine-grained control of extent allocation in local
filesystems from userspace is a recipe for rapidly aging and
degrading filesystem performance. We want to avoid that as much as
possible - it sets us back at least 25 years in terms of file layout
and allocation algorithm sophistication.

> 5) sparse file support
> (and it would also be nice to support copy_file_range syscall ... but
> that is unrelated to the above)

make copy_file_range() handle that properly.

You also forgot all the benfits we'd get from parallelising
recursive directory walks and stat()ing inodes along the way (i.e.
the dir walk that rsync does to work out what it needs to copy).
Also, sorting the files to be copied based on something like inode
number rather than just operating on readdir order can improve
copying of large numbers of files substantially. Chris Mason
demonstrated this years ago:

https://oss.oracle.com/~mason/acp/

> Am I missing some magic tool?  Seems like Windows has various options
> for copy tools - but looking at Linux i/o patterns from these tools
> was pretty depressing - I am hoping that there are other choices.

Not that I know of.  The Linux IO tools need to dragged kicking and
screaming out of the 1980s. :/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Copy tools on Linux

[Steve French]

On Sat, Jun 30, 2018 at 7:10 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Fri, Jun 29, 2018 at 09:37:27PM -0500, Steve French wrote:
> > I have been looking at i/o patterns from various copy tools on Linux,
> > and it is pretty discouraging - I am hoping that I am forgetting an
> > important one that someone can point me to ...
> >
> > Some general problems:
> > 1) if source and target on the same file system it would be nice to
> > call the copy_file_range syscall (AFAIK only test tools call that),
> > although in some cases at least cp can do it for --reflink
>
> copy_file_range() should be made to do the right thing in as many
> scnearios as we can document, and then switch userspace over to use
> it at all times. Aggregate all the knowledge in one place, where we
> know what the filesystem implementations are and can get hints to do
> the right thing.

Strongly agree.  Looking at the code for what cp and rsync are trying
to in userspace - and doing it without all of the information available in
the VFS and fs layer and trying to do it across platforms - it is never
going to be as good as what the kernel could do.

On the otherhand, even in userspace, robocopy (among other Windows
tools) seems to have sane options and do a much better job of parallelizing,
but makes more sense in kernel.

> > 5) sparse file support
> > (and it would also be nice to support copy_file_range syscall ... but
> > that is unrelated to the above)
>
> make copy_file_range() handle that properly.
>
> You also forgot all the benfits we'd get from parallelising
> recursive directory walks and stat()ing inodes along the way (i.e.
> the dir walk that rsync does to work out what it needs to copy).

Very good point

> > Am I missing some magic tool?  Seems like Windows has various options
> > for copy tools - but looking at Linux i/o patterns from these tools
> > was pretty depressing - I am hoping that there are other choices.
>
> Not that I know of.  The Linux IO tools need to dragged kicking and
> screaming out of the 1980s. :/

Sure looks that way



-- 
Thanks,

Steve

Re: Copy tools on Linux

[Goldwyn Rodrigues]

On 07-01 10:10, Dave Chinner wrote:
> On Fri, Jun 29, 2018 at 09:37:27PM -0500, Steve French wrote:
> > I have been looking at i/o patterns from various copy tools on Linux,
> > and it is pretty discouraging - I am hoping that I am forgetting an
> > important one that someone can point me to ...
> > 
> > Some general problems:
> > 1) if source and target on the same file system it would be nice to
> > call the copy_file_range syscall (AFAIK only test tools call that),
> > although in some cases at least cp can do it for --reflink
> 
> copy_file_range() should be made to do the right thing in as many
> scnearios as we can document, and then switch userspace over to use
> it at all times. Aggregate all the knowledge in one place, where we
> know what the filesystem implementations are and can get hints to do
> the right thing.

We have discussed this earlier in
https://www.spinics.net/lists/linux-fsdevel/msg125401.html
and Christoph suggested that there is no point adding new flags
for holes. Do you have a different opinion?
The same is with coreutils maintainer where he suggested adding
flags to perform everything with respect to holes in the kernel.


-- 
Goldwyn

Re: Copy tools on Linux

[Dave Chinner]

On Sun, Jul 01, 2018 at 12:44:43PM -0500, Goldwyn Rodrigues wrote:
> On 07-01 10:10, Dave Chinner wrote:
> > On Fri, Jun 29, 2018 at 09:37:27PM -0500, Steve French wrote:
> > > I have been looking at i/o patterns from various copy tools on Linux,
> > > and it is pretty discouraging - I am hoping that I am forgetting an
> > > important one that someone can point me to ...
> > > 
> > > Some general problems:
> > > 1) if source and target on the same file system it would be nice to
> > > call the copy_file_range syscall (AFAIK only test tools call that),
> > > although in some cases at least cp can do it for --reflink
> > 
> > copy_file_range() should be made to do the right thing in as many
> > scnearios as we can document, and then switch userspace over to use
> > it at all times. Aggregate all the knowledge in one place, where we
> > know what the filesystem implementations are and can get hints to do
> > the right thing.
> 
> We have discussed this earlier in
> https://www.spinics.net/lists/linux-fsdevel/msg125401.html
> and Christoph suggested that there is no point adding new flags
> for holes. Do you have a different opinion?

I don't care. All I was suggesting is that we add flags to...

> The same is with coreutils maintainer where he suggested adding
> flags to perform everything with respect to holes in the kernel.

.... get adoption from the userspace tool maintainers who won't use
anything that doesn't exactly replace what they do now.

And that's the real problem - getting userspace to use all the
go-fast bits we provide. How long have we had the splice interfaces
to do efficient data copies without passing data through userspace?
You'd think using these APIs would be a no-brainer, but nothing uses
them at all. I suspect we're going to see the same thing with
copy_file_range() and similar recent advances in AIO+DIO APIs...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com