* [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS @ 2019-02-08 15:19 Steve French 2019-02-08 22:37 ` Andreas Dilger 0 siblings, 1 reply; 7+ messages in thread From: Steve French @ 2019-02-08 15:19 UTC (permalink / raw) To: lsf-pc; +Cc: linux-fsdevel, CIFS, samba-technical Current Linux copy tools have various problems compared to other platforms - small I/O sizes (and not even configurable for most), lack of parallel I/O for multi-file copies, inability to reduce metadata updates by setting file size first, lack of cross mount (to the same file system) copy optimizations, limited ability to handle the wide variety of server side copy (and copy offload) mechanisms and various error handling problems. And copy tools rely less on the kernel file system (vs. code in the user space tool) in Linux than would be expected, in order to determine which optimizations to use. Would like to discuss some of the observations about copy tools and how we can move forward on improving the performance of common copy operations. -- Thanks, Steve ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS 2019-02-08 15:19 [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS Steve French @ 2019-02-08 22:37 ` Andreas Dilger 2019-02-08 23:03 ` Steve French 2019-02-11 23:22 ` L. A. Walsh 0 siblings, 2 replies; 7+ messages in thread From: Andreas Dilger @ 2019-02-08 22:37 UTC (permalink / raw) To: Steve French; +Cc: lsf-pc, linux-fsdevel, CIFS, samba-technical [-- Attachment #1: Type: text/plain, Size: 2187 bytes --] On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote: > > Current Linux copy tools have various problems compared to other > platforms - small I/O sizes (and not even configurable for most), Hmm, this comment puzzles me, since "cp" already uses s_blksize returned for the file as the IO size? Not sure if tar/rsync do the same, but if they don't already use s_blksize they should. > lack of parallel I/O for multi-file copies, inability to reduce metadata > updates by setting file size first, lack of cross mount (to the same > file system) copy optimizations, limited ability to handle the wide > variety of server side copy (and copy offload) mechanisms and various > error handling problems. And copy tools rely less on the kernel file > system (vs. code in the user space tool) in Linux than would be > expected, in order to determine which optimizations to use. The rest of these issues are definitely a concern. It is worthwhile to point out MPIFileUtils (https://github.com/hpc/mpifileutils) that already solves a lot of these problems. As the name suggests, it currently uses MPI to run in parallel across multiple nodes, but it should be possible to add a wrapper for the MPI calls in the library with fork()+exec() or so and run multi-threaded on one node for parallel copy/find/sync/etc. IMHO, it makes sense to try and optimize a single set of tools, rather than adding yet another set of tools that are not widely used. There is also "mutils" (https://github.com/pkolano/mutil) which are patches for GNU cp and md5sum, but they are less widely used vs. MPIFileUtils. That said, most users are going to have GNU Fileutils installed, so the best option is to add improvements directly into those tools if possible, with the caveat that you will get a headache reading that code, and they may object to including parallel extensions due to portability concerns. > Would like to discuss some of the observations about copy tools and > how we can move forward on improving the performance of common copy > operations. Unfortunately, I'm unable to attend LSF/MM this year, or this would definitely be a topic of interest to me. Cheers, Andreas [-- Attachment #2: Message signed with OpenPGP --] [-- Type: application/pgp-signature, Size: 873 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS 2019-02-08 22:37 ` Andreas Dilger @ 2019-02-08 23:03 ` Steve French 2019-02-08 23:56 ` Steve French 2019-02-11 23:22 ` L. A. Walsh 1 sibling, 1 reply; 7+ messages in thread From: Steve French @ 2019-02-08 23:03 UTC (permalink / raw) To: Andreas Dilger; +Cc: lsf-pc, linux-fsdevel, CIFS, samba-technical On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote: > > On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote: > > > > Current Linux copy tools have various problems compared to other > > platforms - small I/O sizes (and not even configurable for most), > > Hmm, this comment puzzles me, since "cp" already uses s_blksize > returned for the file as the IO size? Not sure if tar/rsync do > the same, but if they don't already use s_blksize they should. We report a block size of 64K in cifs, but I see cp ignoring that and issuing 128K (the code in coreutils is a little hard to follow to see why cp does this), but for long latency networks, larger I/O sizes are preferred. cifs.ko would want at least 4MB typically to be passed down per request since over the wire it negotiates the rsize and wsize (which typically ends up as 1MB or 4MB) and NFS often uses a 1MB rsize/wsize over the wire so similar considerations would apply. -- Thanks, Steve ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS 2019-02-08 23:03 ` Steve French @ 2019-02-08 23:56 ` Steve French 2019-02-11 8:32 ` Andreas Dilger 0 siblings, 1 reply; 7+ messages in thread From: Steve French @ 2019-02-08 23:56 UTC (permalink / raw) To: Andreas Dilger; +Cc: lsf-pc, linux-fsdevel, CIFS, samba-technical On Fri, Feb 8, 2019 at 5:03 PM Steve French <smfrench@gmail.com> wrote: > > On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote: > > > > On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote: > > > > > > Current Linux copy tools have various problems compared to other > > > platforms - small I/O sizes (and not even configurable for most), > > > > Hmm, this comment puzzles me, since "cp" already uses s_blksize > > returned for the file as the IO size? Not sure if tar/rsync do > > the same, but if they don't already use s_blksize they should. I did some experiments changing the block size returned from 1K to 64K to 1MB and see no difference in the copy size used by cp (it was always 128K in all the cases when caching is disabled) -- Thanks, Steve ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS 2019-02-08 23:56 ` Steve French @ 2019-02-11 8:32 ` Andreas Dilger 2019-02-11 17:43 ` Steve French 0 siblings, 1 reply; 7+ messages in thread From: Andreas Dilger @ 2019-02-11 8:32 UTC (permalink / raw) To: Steve French; +Cc: lsf-pc, linux-fsdevel, CIFS, samba-technical [-- Attachment #1: Type: text/plain, Size: 2830 bytes --] On Feb 8, 2019, at 4:56 PM, Steve French <smfrench@gmail.com> wrote: > > On Fri, Feb 8, 2019 at 5:03 PM Steve French <smfrench@gmail.com> wrote: >> >> On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote: >>> >>> On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote: >>>> >>>> Current Linux copy tools have various problems compared to other >>>> platforms - small I/O sizes (and not even configurable for most), >>> >>> Hmm, this comment puzzles me, since "cp" already uses s_blksize >>> returned for the file as the IO size? Not sure if tar/rsync do >>> the same, but if they don't already use s_blksize they should. > > I did some experiments changing the block size returned from 1K to 64K to 1MB > and see no difference in the copy size used by cp (it was always 128K in all > the cases when caching is disabled) Strange. I just re-tested this on Lustre, in case something had changed in GNU fileutils that I didn't notice, and it worked fine for me, using both "cp --version = 8.4" on RHEL and "cp --version = 8.26" on Ubuntu: $ dd if=/dev/urandom of=/tmp/foo bs=1M count=12 $ strace -v cp /tmp/foo /testfs/tmp : open("/tmp/foo", O_RDONLY) = 3 fstat(3, {... st_blksize=4096, st_blocks=24576, st_size=12582912, ...}) = 0 open("/testfs/tmp/foo", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4 fstat(4, { ... st_blksize=4194304, st_blocks=0, st_size=0, ...}) = 0 read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 : Note the "st_blksize=4194304" for the target file returned by Lustre matches the read and write buffer size used by "cp". The same is true if Lustre is the source file and not the target, so it probably picks the maximum of both: open("/testfs/tmp/foo", O_RDONLY) = 3 fstat(3, {... st_blksize=4194304, st_blocks=24576, st_size=12582912 ...}) = 0 open("/tmp/bar", O_WRONLY|O_TRUNC) = 4 fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0 ...}) = 0 read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 : Running the same command with /tmp as the target uses a smaller buffer size matching the "st_blocks=32768" and correspondingly more read/write calls: $ strace -v cp /tmp/foo /tmp/baz : open("/tmp/baz", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4 fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0, ...}) = 0 read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768 write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768 : In this case, cp probably has some minimum buffer size it uses to avoid the poor performance of using 4KB blocks. Cheers, Andreas [-- Attachment #2: Message signed with OpenPGP --] [-- Type: application/pgp-signature, Size: 873 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS 2019-02-11 8:32 ` Andreas Dilger @ 2019-02-11 17:43 ` Steve French 0 siblings, 0 replies; 7+ messages in thread From: Steve French @ 2019-02-11 17:43 UTC (permalink / raw) To: Andreas Dilger; +Cc: lsf-pc, linux-fsdevel, CIFS, samba-technical On Mon, Feb 11, 2019 at 2:32 AM Andreas Dilger <adilger@dilger.ca> wrote: > > On Feb 8, 2019, at 4:56 PM, Steve French <smfrench@gmail.com> wrote: > > > > On Fri, Feb 8, 2019 at 5:03 PM Steve French <smfrench@gmail.com> wrote: > >> > >> On Fri, Feb 8, 2019 at 4:37 PM Andreas Dilger <adilger@dilger.ca> wrote: > >>> > >>> On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote: <snip> > > I did some experiments changing the block size returned from 1K to 64K to 1MB > > and see no difference in the copy size used by cp (it was always 128K in all > > the cases when caching is disabled) I figured out the problem - I read your note as meaning s_blocksize (which not st_blksize), ie the block size in the superblock not on the file. Changing st_blksize (stat->blksize) to 4MB did lead to the better performance (and large I/O matching the block size) for uncached cp > Strange. I just re-tested this on Lustre, in case something had changed in > GNU fileutils that I didn't notice, and it worked fine for me, using both > "cp --version = 8.4" on RHEL and "cp --version = 8.26" on Ubuntu: > > $ dd if=/dev/urandom of=/tmp/foo bs=1M count=12 > $ strace -v cp /tmp/foo /testfs/tmp > : > open("/tmp/foo", O_RDONLY) = 3 > fstat(3, {... st_blksize=4096, st_blocks=24576, st_size=12582912, ...}) = 0 > open("/testfs/tmp/foo", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4 > fstat(4, { ... st_blksize=4194304, st_blocks=0, st_size=0, ...}) = 0 > read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 > write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 > : > > Note the "st_blksize=4194304" for the target file returned by Lustre matches > the read and write buffer size used by "cp". The same is true if Lustre is > the source file and not the target, so it probably picks the maximum of both: > > open("/testfs/tmp/foo", O_RDONLY) = 3 > fstat(3, {... st_blksize=4194304, st_blocks=24576, st_size=12582912 ...}) = 0 > open("/tmp/bar", O_WRONLY|O_TRUNC) = 4 > fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0 ...}) = 0 > read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 > write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 4194304) = 4194304 > : > > Running the same command with /tmp as the target uses a smaller buffer size > matching the "st_blocks=32768" and correspondingly more read/write calls: > > $ strace -v cp /tmp/foo /tmp/baz > : > open("/tmp/baz", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4 > fstat(4, {... st_blksize=4096, st_blocks=0, st_size=0, ...}) = 0 > read(3, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768 > write(4, "h\230#`\2\223\273\3423W\24\222:\2113w\327"..., 32768) = 32768 > : > > In this case, cp probably has some minimum buffer size it uses to avoid the > poor performance of using 4KB blocks. Yes - although the code is a little hard to follow it looks like 128K in my system's version of cp (Ubuntu) -- Thanks, Steve ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS 2019-02-08 22:37 ` Andreas Dilger 2019-02-08 23:03 ` Steve French @ 2019-02-11 23:22 ` L. A. Walsh 1 sibling, 0 replies; 7+ messages in thread From: L. A. Walsh @ 2019-02-11 23:22 UTC (permalink / raw) To: Andreas Dilger; +Cc: Steve French, lsf-pc, linux-fsdevel, CIFS, samba-technical [-- Attachment #1: Type: text/plain, Size: 1939 bytes --] On 2/8/2019 2:37 PM, Andreas Dilger wrote: > On Feb 8, 2019, at 8:19 AM, Steve French <smfrench@gmail.com> wrote: > >> Current Linux copy tools have various problems compared to other >> platforms - small I/O sizes (and not even configurable for most), >> > Hmm, this comment puzzles me, since "cp" already uses s_blksize > returned for the file as the IO size? Not sure if tar/rsync do > the same, but if they don't already use s_blksize they should. > But not as the optimal IO size. It returns 4K. My RAID can write 12 disks in parallel or 48K.\ Running cat for example. Yet the optimal bytes get returned as 4K. cat speed: /bin/time cat 4G 4G 4G 4G > 16G 0.17user 10.06system 1:52.84elapsed 9%CPU (0avgtext+0avgdata 8876maxresident)k 0inputs+0outputs (2230major+0minor)pagefaults 0swaps Note, the 4G is copied 4 times, so really only a need to read 4G once so read 4G+write 16G = 181MB/s in those cases look like cat is averaging about 290M/s(?) cp is doing 16 read and write in 230MB/s vs. using 'dd': bin/iotest Using bs=16.0M, count=64, iosize=1.0G R:1073741824 bytes (1.0GB) copied, 1.41419 s, 724MB/s W:1073741824 bytes (1.0GB) copied, 3.79678 s, 270MB/s dd is using a 16M bufferl Optimal is about 32M buff, in all cases the utils read and wrote to the network. No figure gives you back that you should use 16M This is on Win7SP1x64 over 10Gb ethernet using 4.20.3 kernel and some 4.x samba version. I can't even find the physical IO size in /sys or /proc (sda uses all 4K-sec disks). So.....you won't get more than about 20-30% on reads. Note it will take some work to put together a real test with cp. cp used to be the fastest -- now it's not (I think they walk the tree twice). Anyway, have been doing speed tests between Win+lin since XP days. Double Note: The 'dd' test is only testing network xfer speed (no file I/O) as it copies from /dev/zero and outputs to /dev/null on the far end. [-- Attachment #2: iotest --] [-- Type: text/plain, Size: 5662 bytes --] #!/bin/bash # iotest v0.2 - lawalsh(at)tlinx.org : open usage allowed # (c) 2014-2018 # vim=:SetNumberAndWidth _prgpth="${0:?}"; _prg="${_prgpth##*/}"; _prgdr="${_prgpth%/$_prg}" [[ -z $_prgdr || $_prg == $_prgdr ]] && $_prgdr="$PWD" export PATH="$_prgdr:$_prgdr/lib:$PATH" shopt -s expand_aliases extglob sourcepath ; set -o pipefail #include stdalias #include Types Dd=$(type -P dd) [[ $Dd ]] || { echo "Cannot find dd. Cannot proceed."; exit 1; } alias my=declare sub=function int='my -i' array='my -a' alias map='my -A' intConst='int -x' string=my # 1 num = block size # num-num = range of block sizes to test; w/increment = "2x", so # 4M-16M tests 4M, 8M, 16M # 4M-12M test 4M, 8M, 12M # count adjusted to xfer 4G, rounding up #---- #all xfers are using 'devices' (/dev/zero for source, /dev/null for target) # remote filenames "zero" and "null" should be setup to be remote devices intConst K=1024 M=K*K G=M*K T=G*K my sfxs="KMGT" my prog='/bin/dd' ## # Defaults for BS count and IOSIZE ## check_parm () { my param=$1 if [[ ${!param} =~ ^[0-9]+[$sfxs]$ ]]; then my unit=${!param:0-1} eval $param="${!param:0:0-1}*$unit" elif [[ ${!param} =~ ^[0-9]+$ ]]; then : # ok as is elif [[ ! ${!param} =~ ^[0-9]+\*[$sfxs]$ ]]; then printf >&2 "Invalid number format \"%s\", for \"%s\"\n" "${!param}" "$param" exit 1 fi } my BS COUNT IOSIZE for p in BS COUNT IOSIZE; do if [[ ${!p} ]]; then check_parm $p int $p=${!p} fi done int params=0 array -i coef=(0 0 0) #: ${BS:=16*M} ${COUNT:=64} if ((${#BS})); then int BS=$BS; params+=1; coef[0]=1; fi if ((${#COUNT})); then int COUNT=$COUNT; params+=1; coef[1]=1; fi if ((${#IOSIZE})); then int IOSIZE=$IOSIZE; params+=1 coef[2]=1; fi int BS=${BS:-16*M} int COUNT=${COUNT:-64} int IOSIZE=${IOSIZE:-2*G} if ((params>=3)); then printf >&2 "Error: can only specify 2 params; 3rd is calculated\n"; exit 1; elif ((params==2)); then if ((!coef[2])); then int IOSIZE=COUNT*BS elif ((!coef[1])); then int COUNT=IOSIZE/BS elif ((!coef[0])); then int BS=IOSIZE/COUNT fi elif ((params==1)); then if ((coef[2])) ; then # only size spec'd, use default for BS unless too big if ((BS>IOSIZE)); then BS=IOSIZE/COUNT else COUNT=IOSIZE/BS fi elif ((coef[1])); then BS=IOSIZE/COUNT else COUNT=IOSIZE/BS fi fi int IOSIZE=COUNT*BS #echo "BS=$BS, COUNT=$COUNT, IOSIZE=$IOSIZE" # desuffix arg1 [arg2] # desuffix arg1 - num + suffix -> convert to int # arg2 - optional buff name (else print to stdout) # return 0 if no error desuffix () { #convert num+Suff => int store in optional Buff my str="${1:?}" ; shift my bufnam=""; (($#)) && bufnam=$1 if [[ $str =~ ^([0-9]+)([KMGT])$ ]]; then int num=${BASH_REMATCH[1]}*${BASH_REMATCH[2]} ((num)) || return 1 if [[ $bufnam ]] ; then printf -v $bufnam "%d" "$num" else printf "%d" "$num" ; fi else return $p fi } hdisp () { my parm=${1:?} if [[ ! $parm =~ ^[0-9]+$ ]]; then echo >&2 "hdisp: parm1 is not num: \"$parm\""; fi int num=${1:?}; shift export PERL5OPT="$PERL5OPT -I/h/bin/lib" my buf="$(perl -e 'use Hout; use P; P "%s", human($ARGV[0]);' "$num")" #printf "hdisp-buf=%s\n" "$buf" string bufnam=""; (($#)) && bufnam=$1 if [[ $bufnam ]] ; then printf -v $bufnam "%s" "$buf" else printf "%s" "$buf" ; fi } map args=([b]=setblocksize [i]=iosize ) setblocksize () { BS=$1 ; } iosize () { IOSIZE=$1 ; } check_params () { int num=0 while (($#)) ; do my arg=$1 ; shift if [[ ${arg:0:1} == - ]]; then arg=${arg:1}; fi my switch=${args["$arg"]:-""} int iswitch=0 if [[ ${switch:-""} ]]; then my val=$1; shift my nval="" int ival=0 desuffix "$val" nval if [[ $nval ]]; then ival=0+nval; fi $switch $ival fi done COUNT=IOSIZE/BS } (($#)) && check_params "$@" string testdir=/h array reada=($testdir/zero /dev/null ) array writea=(/dev/zero $testdir/null oflag=direct conv=nocreat,notrunc) sub dd_io { local if="$1" of="$2"; shift 2 nice --19 $Dd if="$if" of="$of" bs="$BS" count="$COUNT" iflag=fullblock\ conv=nocreat "$@" } dd () { local if="$1" of="$2" ; parms="$3"; shift 2 array out=() my ignore_RE='^[^ \t]+ records\ (in|out)$' readarray out < <( dd_io "$if" "$of" "$@" |& { int s=$? if ((s)); then echo >&2 "stat:$s"; else cat; fi ; } ) printf "%s\n" "${out[@]}" return 0 } dd_format () { my bytes btxt pnum1 suffp1 pnum2 suffp2 copt time rest while read bytes btxt pnum1 suffp1 pnum2 suffp2 copt time rest; do if ! [[ $time =~ ^[0-9] ]]; then copt=$pnum2; time=$suffp2; fi [[ $btxt == records ]] && continue [[ $bytes && $time ]] || continue my sizeb="" rateb="" my bps="$(echo -E "$bytes/$time" | bc)" hdisp "$bytes" sizeb hdisp "$bps" rateb my fmt="%d bytes (%sB) copied, %s s, %sB/s\n" printf "$fmt" "$bytes" "$sizeb" "$time" "$rateb" done } onecycle () { echo -n "R:"; { dd "${reada[@]}" || exit $?; } | dd_format sync;sleep .1;sync echo -n "W:"; { dd "${writea[@]}" || exit $?; } | dd_format } my bsbuf="" ios_buf="" hdisp "$BS" bsbuf hdisp "$IOSIZE" ios_buf #if (($#)) ; then # if [[ $1 == -h || $1 == -\? ]]; then # printf "Defaults: BS=16M, COUNT=%d, IOSIZE=%s" $Count $IOSize # printf "BlkSize, cnt & Total can be overriden with BS, COUNT & IOSIZE:\n" # printf "BS=16M (uses 1G as Total IOSIZE), or\n" "$_prg" : # printf "BS=1M COUNT=1K %s (uses 1G as Total IOSIZE), or\n" "$_prg" # printf "" # fi #fi printf "Using bs=%s, count=%s, iosize=%s\n" "$bsbuf" "$COUNT" "$ios_buf" onecycle ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-02-11 23:46 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-02-08 15:19 [LSF/MM TOPIC] Enhancing Copy Tools for Linux FS Steve French 2019-02-08 22:37 ` Andreas Dilger 2019-02-08 23:03 ` Steve French 2019-02-08 23:56 ` Steve French 2019-02-11 8:32 ` Andreas Dilger 2019-02-11 17:43 ` Steve French 2019-02-11 23:22 ` L. A. Walsh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).