Questions about usage of io-uring in a network application

From: "Richard W.M. Jones" <rjones@redhat.com>
To: io-uring@vger.kernel.org
Subject: Questions about usage of io-uring in a network application
Date: Sun, 10 May 2020 09:00:34 +0100	[thread overview]
Message-ID: <20200510080034.GI3888@redhat.com> (raw)

Hopefully these questions are not too stupid, but I didn't find any
answers looking over the archives of this list or in the io_uring.pdf
document.

I have an existing C library which is the client part of a
client/server network service.  It's not exactly the same as HTTP, but
it could be thought of as like an HTTP network client where we may
hold a few separate TCP/IP connections open to the same server, and we
issue multiple requests in flight per TCP connection.  There is one
pthread per TCP connection usually.

I want to try to see if io_uring gives us any performance benefit by
seeing if we can use IORING_OP_SENDMSG/IORING_OP_RECVMSG +
IOSQE_IO_LINK, as an experiment.

(1) How many io_urings should we create?

One ring per process?  All of the connections share the same ring.
Presumably there would be a lot of contention because I suppose we'd
need to lock the ring while submitting requests from multiple threads.
If there are multiple independent libraries or separate of the program
all trying to use io_uring, should they try to share a single ring?

One ring per pthread?  It seems we could implement this without locks
using thread-local storage to hold the the io_uring fd.

One ring per physical CPU?  (Not sure how to implement this race-free
in userspace).

One ring per TCP connection?

(2) The existing API (which we cannot change) takes user-allocated
buffers for the data to read/write.  We don't know if these were
allocated using malloc, they might be statically allocated or even
come from something exotic like mmap of a file.  I understand that we
cannot register these buffers using IORING_REGISTER_BUFFERS.  But can
these be passed in the io_uring_sqe->addr field?  ie. Do the same
restrictions in IORING_REGISTER_BUFFERS also apply to the addr field?

Thanks,
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v