[LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[Steve French]

Current Linux copy tools have various problems compared to other
platforms - small I/O sizes (and not even configurable for most), lack
of parallel I/O for multi-file copies, inability to reduce metadata
updates by setting file size first, lack of cross mount (to the same
file system) copy optimizations, limited ability to handle the wide
variety of server side copy (and copy offload) mechanisms and various
error handling problems.   And copy tools rely less on the kernel file
system (vs. code in the user space tool) in Linux than would be
expected, in order to determine which optimizations to use.

Would like to discuss some of the observations about copy tools and
how we can move forward on improving the performance of common copy
operations.

-- 
Thanks,

Steve

Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote:
> 
> Current Linux copy tools have various problems compared to other
> platforms - small I/O sizes (and not even configurable for most),

Hmm, this comment puzzles me, since "cp" already uses s_blksize
returned for the file as the IO size?  Not sure if tar/rsync do
the same, but if they don't already use s_blksize they should.

> lack of parallel I/O for multi-file copies, inability to reduce metadata
> updates by setting file size first, lack of cross mount (to the same
> file system) copy optimizations, limited ability to handle the wide
> variety of server side copy (and copy offload) mechanisms and various
> error handling problems.   And copy tools rely less on the kernel file
> system (vs. code in the user space tool) in Linux than would be
> expected, in order to determine which optimizations to use.

The rest of these issues are definitely a concern.  It is worthwhile to
point out MPIFileUtils (https://github.com/hpc/mpifileutils) that already
solves a lot of these problems.  As the name suggests, it currently uses
MPI to run in parallel across multiple nodes, but it should be possible
to add a wrapper for the MPI calls in the library with fork()+exec() or
so and run multi-threaded on one node for parallel copy/find/sync/etc.

IMHO, it makes sense to try and optimize a single set of tools, rather
than adding yet another set of tools that are not widely used.  There
is also "mutils" (https://github.com/pkolano/mutil) which are patches
for GNU cp and md5sum, but they are less widely used vs. MPIFileUtils.

That said, most users are going to have GNU Fileutils installed, so the
best option is to add improvements directly into those tools if possible,
with the caveat that you will get a headache reading that code, and they
may object to including parallel extensions due to portability concerns.

> Would like to discuss some of the observations about copy tools and
> how we can move forward on improving the performance of common copy
> operations.

Unfortunately, I'm unable to attend LSF/MM this year, or this would
definitely be a topic of interest to me.

Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[Steve French]

On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote:
>
> On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote:
> >
> > Current Linux copy tools have various problems compared to other
> > platforms - small I/O sizes (and not even configurable for most),
>
> Hmm, this comment puzzles me, since "cp" already uses s_blksize
> returned for the file as the IO size?  Not sure if tar/rsync do
> the same, but if they don't already use s_blksize they should.

We report a block size of 64K in cifs, but I see cp ignoring that and
issuing 128K (the code in coreutils is a little hard to follow to see
why cp does this), but for long latency networks, larger I/O sizes are
preferred.   cifs.ko would want at least 4MB typically to be passed
down per request since over the wire it negotiates the rsize and wsize
(which typically ends up as 1MB or 4MB) and NFS often uses a 1MB
rsize/wsize over the wire so similar considerations would apply.


-- 
Thanks,

Steve

Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[Steve French]

On Fri, Feb 8, 2019 at 5:03 PM Steve French <smfrench@gmail.com> wrote:
>
> On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote:
> >
> > On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote:
> > >
> > > Current Linux copy tools have various problems compared to other
> > > platforms - small I/O sizes (and not even configurable for most),
> >
> > Hmm, this comment puzzles me, since "cp" already uses s_blksize
> > returned for the file as the IO size?  Not sure if tar/rsync do
> > the same, but if they don't already use s_blksize they should.

I did some experiments changing the block size returned from 1K to 64K to 1MB
and see no difference in the copy size used by cp (it was always 128K in all
the cases when caching is disabled)




-- 
Thanks,

Steve

Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 2830 bytes --]

On Feb 8, 2019, at 4:56 PM, Steve French <smfrench@gmail.com> wrote:
> 
> On Fri, Feb 8, 2019 at 5:03 PM Steve French <smfrench@gmail.com> wrote:
>> 
>> On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote:
>>> 
>>> On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote:
>>>> 
>>>> Current Linux copy tools have various problems compared to other
>>>> platforms - small I/O sizes (and not even configurable for most),
>>> 
>>> Hmm, this comment puzzles me, since "cp" already uses s_blksize
>>> returned for the file as the IO size?  Not sure if tar/rsync do
>>> the same, but if they don't already use s_blksize they should.
> 
> I did some experiments changing the block size returned from 1K to 64K to 1MB
> and see no difference in the copy size used by cp (it was always 128K in all
> the cases when caching is disabled)

Strange.  I just re-tested this on Lustre, in case something had changed in
GNU fileutils that I didn't notice, and it worked fine for me, using both
"cp --version = 8.4" on RHEL and "cp --version = 8.26" on Ubuntu:

$ dd if=/dev/urandom of=/tmp/foo bs=1M count=12
$ strace -v cp /tmp/foo /testfs/tmp
:
open("/tmp/foo", O_RDONLY)              = 3
fstat(3, {... st_blksize=4096, st_blocks=24576, st_size=12582912, ...}) = 0
open("/testfs/tmp/foo", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
fstat(4, { ... st_blksize=4194304, st_blocks=0, st_size=0, ...}) = 0
read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
:

Note the "st_blksize=4194304" for the target file returned by Lustre matches
the read and write buffer size used by "cp".  The same is true if Lustre is
the source file and not the target, so it probably picks the maximum of both:

open("/testfs/tmp/foo", O_RDONLY)     = 3
fstat(3, {... st_blksize=4194304, st_blocks=24576, st_size=12582912 ...}) = 0
open("/tmp/bar", O_WRONLY|O_TRUNC)      = 4
fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0 ...}) = 0
read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
:

Running the same command with /tmp as the target uses a smaller buffer size
matching the "st_blocks=32768" and correspondingly more read/write calls:

$ strace -v cp /tmp/foo /tmp/baz
:
open("/tmp/baz", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0, ...}) = 0
read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768
write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768
:

In this case, cp probably has some minimum buffer size it uses to avoid the
poor performance of using 4KB blocks.

Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[Steve French]

On Mon, Feb 11, 2019 at 2:32 AM Andreas Dilger <adilger@dilger.ca> wrote:
>
> On Feb 8, 2019, at 4:56 PM, Steve French <smfrench@gmail.com> wrote:
> >
> > On Fri, Feb 8, 2019 at 5:03 PM Steve French <smfrench@gmail.com> wrote:
> >>
> >> On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote:
> >>>
> >>> On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote:
<snip>
> > I did some experiments changing the block size returned from 1K to 64K to 1MB
> > and see no difference in the copy size used by cp (it was always 128K in all
> > the cases when caching is disabled)

I figured out the problem - I read your note as meaning s_blocksize (which not
st_blksize), ie the block size in the superblock not on the file.

Changing st_blksize (stat->blksize) to 4MB did lead to the better performance
(and large I/O matching the block size) for uncached cp


> Strange.  I just re-tested this on Lustre, in case something had changed in
> GNU fileutils that I didn't notice, and it worked fine for me, using both
> "cp --version = 8.4" on RHEL and "cp --version = 8.26" on Ubuntu:
>
> $ dd if=/dev/urandom of=/tmp/foo bs=1M count=12
> $ strace -v cp /tmp/foo /testfs/tmp
> :
> open("/tmp/foo", O_RDONLY)              = 3
> fstat(3, {... st_blksize=4096, st_blocks=24576, st_size=12582912, ...}) = 0
> open("/testfs/tmp/foo", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
> fstat(4, { ... st_blksize=4194304, st_blocks=0, st_size=0, ...}) = 0
> read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
> write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
> :
>
> Note the "st_blksize=4194304" for the target file returned by Lustre matches
> the read and write buffer size used by "cp".  The same is true if Lustre is
> the source file and not the target, so it probably picks the maximum of both:
>
> open("/testfs/tmp/foo", O_RDONLY)     = 3
> fstat(3, {... st_blksize=4194304, st_blocks=24576, st_size=12582912 ...}) = 0
> open("/tmp/bar", O_WRONLY|O_TRUNC)      = 4
> fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0 ...}) = 0
> read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
> write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304
> :
>
> Running the same command with /tmp as the target uses a smaller buffer size
> matching the "st_blocks=32768" and correspondingly more read/write calls:
>
> $ strace -v cp /tmp/foo /tmp/baz
> :
> open("/tmp/baz", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
> fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0, ...}) = 0
> read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768
> write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768
> :
>
> In this case, cp probably has some minimum buffer size it uses to avoid the
> poor performance of using 4KB blocks.

Yes - although the code is a little hard to follow it looks like 128K
in my system's version of cp (Ubuntu)


-- 
Thanks,

Steve

Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS

[L. A. Walsh]

[-- Attachment #1: Type: text/plain, Size: 1939 bytes --]

On 2/8/2019 2:37 PM, Andreas Dilger wrote:
> On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote:
>   
>> Current Linux copy tools have various problems compared to other
>> platforms - small I/O sizes (and not even configurable for most),
>>     

> Hmm, this comment puzzles me, since "cp" already uses s_blksize
> returned for the file as the IO size?  Not sure if tar/rsync do
> the same, but if they don't already use s_blksize they should.
>   

But not as the optimal IO size.  It returns 4K.  My RAID can write
12 disks in parallel or 48K.\
Running  cat for example.  Yet the optimal bytes get returned as 4K.

cat speed:
 /bin/time cat 4G 4G 4G 4G > 16G
0.17user 10.06system 1:52.84elapsed 9%CPU (0avgtext+0avgdata
8876maxresident)k
0inputs+0outputs (2230major+0minor)pagefaults 0swaps
Note, the 4G is copied 4 times, so really only a need to read
4G once so read 4G+write 16G = 181MB/s

in those cases look like cat is averaging about 290M/s(?)
cp is doing 16 read and write in 230MB/s

vs. using 'dd':
bin/iotest
Using bs=16.0M, count=64, iosize=1.0G
R:1073741824 bytes (1.0GB) copied, 1.41419 s, 724MB/s
W:1073741824 bytes (1.0GB) copied, 3.79678 s, 270MB/s

dd is using a 16M bufferl  Optimal is about 32M buff, in all cases
the utils read and wrote to the network.

No figure gives you back that you should use 16M

This is on Win7SP1x64 over 10Gb ethernet using 4.20.3 kernel and
some 4.x samba version.

I can't even find the physical IO size in /sys or /proc (sda uses
all 4K-sec disks).

So.....you won't get more than about 20-30% on reads.

Note it will take some work to put together a real test with cp.

cp used to be the fastest -- now it's not (I think they walk the
tree twice). 

Anyway, have been doing speed tests between Win+lin since XP days.

Double Note:  The 'dd' test is only testing network xfer speed
(no file I/O) as it copies from /dev/zero and outputs to /dev/null
on the far end.



[-- Attachment #2: iotest --]
[-- Type: text/plain, Size: 5662 bytes --]

#!/bin/bash 
# iotest v0.2 - lawalsh(at)tlinx.org : open usage allowed
# (c) 2014-2018
# vim=:SetNumberAndWidth

_prgpth="${0:?}"; _prg="${_prgpth##*/}"; _prgdr="${_prgpth%/$_prg}"
[[ -z $_prgdr || $_prg == $_prgdr ]] && $_prgdr="$PWD"
export PATH="$_prgdr:$_prgdr/lib:$PATH"
shopt -s expand_aliases extglob sourcepath ; set -o pipefail

#include stdalias
#include Types

Dd=$(type -P dd)

[[ $Dd ]] || { echo "Cannot find dd.  Cannot proceed.";  exit 1; }

alias my=declare sub=function int='my -i' array='my -a' 
alias map='my -A' intConst='int -x' string=my

# 1 num = block size
# num-num = range of block sizes to test; w/increment = "2x", so
# 4M-16M tests 4M, 8M, 16M
# 4M-12M test 4M, 8M, 12M
# count adjusted to xfer 4G, rounding up
#----

#all xfers are using 'devices' (/dev/zero for source, /dev/null for target)
# remote filenames "zero" and "null" should be setup to be remote devices

intConst K=1024 M=K*K G=M*K T=G*K
my sfxs="KMGT"
my prog='/bin/dd'

##
# Defaults for BS count and IOSIZE
##
check_parm () {
  my param=$1
  if [[ ${!param} =~ ^[0-9]+[$sfxs]$ ]]; then
    my unit=${!param:0-1}
    eval $param="${!param:0:0-1}*$unit"
  elif [[ ${!param} =~ ^[0-9]+$ ]]; then
    : # ok as is
  elif [[ ! ${!param} =~ ^[0-9]+\*[$sfxs]$ ]]; then
    printf >&2 "Invalid number format \"%s\", for \"%s\"\n" "${!param}" "$param"
    exit 1
  fi
}

my BS COUNT IOSIZE
for p in BS COUNT IOSIZE; do
  if [[ ${!p} ]]; then
   check_parm $p
   int $p=${!p}
 fi
done

int params=0
array -i coef=(0 0 0)
#: ${BS:=16*M}  ${COUNT:=64}
if ((${#BS})); then int BS=$BS; params+=1; coef[0]=1; fi
if ((${#COUNT})); then int COUNT=$COUNT; params+=1; coef[1]=1; fi
if ((${#IOSIZE})); then int IOSIZE=$IOSIZE; params+=1 coef[2]=1;  fi
int BS=${BS:-16*M}
int COUNT=${COUNT:-64}
int IOSIZE=${IOSIZE:-2*G}
if ((params>=3)); then 
  printf >&2 "Error: can only specify 2 params; 3rd is calculated\n"; exit 1;
elif ((params==2)); then
  if ((!coef[2])); then int IOSIZE=COUNT*BS
  elif ((!coef[1])); then int COUNT=IOSIZE/BS
  elif ((!coef[0])); then int BS=IOSIZE/COUNT
  fi
elif ((params==1)); then
  if ((coef[2])) ; then   # only size spec'd, use default for BS unless too big
    if ((BS>IOSIZE)); then BS=IOSIZE/COUNT
    else COUNT=IOSIZE/BS
    fi
  elif ((coef[1])); then BS=IOSIZE/COUNT
  else COUNT=IOSIZE/BS
  fi
fi

int IOSIZE=COUNT*BS
#echo "BS=$BS, COUNT=$COUNT, IOSIZE=$IOSIZE"

# desuffix  arg1 [arg2]
# desuffix  arg1 - num + suffix -> convert to int
#           arg2 - optional buff name (else print to stdout)
#           return 0 if no error
desuffix () {            #convert num+Suff => int store in optional Buff
  my str="${1:?}" ; shift
  my bufnam=""; (($#)) && bufnam=$1
  if [[ $str =~ ^([0-9]+)([KMGT])$ ]]; then 
    int num=${BASH_REMATCH[1]}*${BASH_REMATCH[2]}
    ((num)) || return 1
    if [[ $bufnam ]] ; then printf -v $bufnam "%d" "$num"
    else printf "%d" "$num" ; fi
  else 
    return $p 
  fi
}


hdisp () {
  my parm=${1:?}
  if [[ ! $parm =~ ^[0-9]+$ ]]; then echo >&2 "hdisp: parm1 is not num: \"$parm\""; fi
  int num=${1:?}; shift
  export PERL5OPT="$PERL5OPT -I/h/bin/lib"
  my buf="$(perl -e 'use Hout; use P; P "%s", human($ARGV[0]);' "$num")"
  #printf "hdisp-buf=%s\n" "$buf"
  string bufnam=""; (($#)) && bufnam=$1
  if [[ $bufnam ]] ; then printf -v $bufnam "%s" "$buf"
  else printf "%s" "$buf" ; fi
}

map args=([b]=setblocksize [i]=iosize )

setblocksize () { BS=$1 ; }

iosize () { IOSIZE=$1 ; }

check_params () {
  int num=0
  while (($#)) ; do
    my arg=$1 ; shift
    if [[ ${arg:0:1} == - ]]; then arg=${arg:1}; fi
    my switch=${args["$arg"]:-""}
    int iswitch=0
    if [[ ${switch:-""} ]]; then
      my val=$1; shift
      my nval=""
      int ival=0
      desuffix "$val" nval
      if [[ $nval ]]; then ival=0+nval; fi
      $switch $ival
    fi
  done
  COUNT=IOSIZE/BS
}

(($#)) && check_params "$@"

string testdir=/h

array reada=($testdir/zero /dev/null )
array writea=(/dev/zero $testdir/null oflag=direct conv=nocreat,notrunc)

sub dd_io  {
  local if="$1" of="$2"; shift 2
  nice --19 $Dd if="$if" of="$of" bs="$BS" count="$COUNT" iflag=fullblock\
      conv=nocreat "$@"
}

dd () {
  local if="$1" of="$2" ; parms="$3"; shift 2
  array out=()
  my ignore_RE='^[^ \t]+ records\ (in|out)$'
  readarray out < <(  dd_io "$if" "$of" "$@" |& { int s=$?
                        if ((s)); then echo >&2 "stat:$s"; else cat; fi ; }
                    )
  printf "%s\n" "${out[@]}"
  return 0
}
  
dd_format () { 
  my bytes btxt pnum1 suffp1 pnum2 suffp2 copt time rest
  while read bytes btxt pnum1 suffp1 pnum2 suffp2 copt time rest; do
    if ! [[ $time =~ ^[0-9] ]]; then
      copt=$pnum2; time=$suffp2;
    fi
    [[ $btxt == records ]] && continue
    [[ $bytes && $time ]] || continue
    my sizeb="" rateb=""
    my bps="$(echo -E "$bytes/$time" | bc)"
    hdisp "$bytes" sizeb
    hdisp "$bps" rateb
    my fmt="%d bytes (%sB) copied, %s s, %sB/s\n"
    printf "$fmt" "$bytes" "$sizeb" "$time" "$rateb"
  done 
}

onecycle () {
  echo -n "R:"; { dd "${reada[@]}" || exit $?; } | dd_format
  sync;sleep .1;sync
  echo -n "W:"; { dd "${writea[@]}" || exit $?; } | dd_format 
}

my bsbuf="" ios_buf=""

hdisp "$BS" bsbuf
hdisp "$IOSIZE" ios_buf

#if (($#)) ; then
# if [[ $1 == -h || $1 == -\? ]]; then
#   printf "Defaults: BS=16M, COUNT=%d, IOSIZE=%s" $Count $IOSize
#   printf "BlkSize, cnt & Total can be overriden with BS, COUNT & IOSIZE:\n"
#   printf "BS=16M (uses 1G as Total IOSIZE), or\n" "$_prg" :
#   printf "BS=1M COUNT=1K %s (uses 1G as Total IOSIZE), or\n" "$_prg" 
#   printf ""
# fi
#fi

printf "Using bs=%s, count=%s, iosize=%s\n" "$bsbuf" "$COUNT" "$ios_buf"


onecycle