Is it possible to transfer a large file between two computers using RDMA UD?

[Terry Toole <toole@photodiagnostic.com>]

Hi,
Is it possible to transfer a large file, say 25GB, between two computers using
RDMA UD, and have an exact copy of the original file on the receiving side? My
understanding is that the order of the messages is not guaranteed with UD.
But I thought that if I only use one QP I could ensure that the ordering of the
data will be predictable.

This is a proof of concept at this time. My thought was to load the file into
memory on the client side and advance the address pointer after each
transfer.  The same scheme would be used on the receiving side. That is,
allocate a large region of memory and increment the address pointer
after each message is received.

I have been using the program ib_send_bw from perftest as a starting point.
After increasing the buffer size on the client side and advancing sg_list->addr
on the client side, I seem to be sending what I intend to send. At the present
time I am not yet advancing sg_list->addr on the server side.  When I print out
the first 100B of data after each transfer, I often see the data I
expect, but not
always.

The commands I've been using to launch my modified version of ib_send_bw are
client:
./ib_send_bw -R   -d mlx5_0 -c UD -i 1 -F -q 1 --mmap=myInputfile.dat
--report_gbits -n 1000 10.10.10.3

server:
./ib_send_bw -R -r 50 -d mlx5_0 -c UD -i 1 -q 1 -F -n 1000 --report_gbits

The "-n 1000" is just for debugging. It will be increased later.

Am I going about this in the correct way? Should I be able to copy a file using
RDMA UD? Any thoughts or suggestions would be most appreciated.

Thanks,

Terry