Keep getting the same buffer ID when RECV with IOSQE_BUFFER_SELECT

From: Daniele Salvatore Albano <d.albano@gmail.com>
To: io-uring@vger.kernel.org
Subject: Keep getting the same buffer ID when RECV with IOSQE_BUFFER_SELECT
Date: Fri, 3 Jul 2020 20:48:10 +0200	[thread overview]
Message-ID: <CAKq9yRg1NkEOei-G8JKMMo-cTCp128aPPONeLCGPFLqD5w+fkA@mail.gmail.com> (raw)

Hi,

I have recently started to play with io_uring and liburing but I am
facing an odd issue, of course initially I thought it was my code but
after further investigation and testing some other code (
https://github.com/frevib/io_uring-echo-server/tree/io-uring-op-provide-buffers
) I faced the same behaviour.

When using the IOSQE_BUFFER_SELECT with RECV I always get the first
read right but all the subsequent return a buffer id different from
what was used by the kernel.

The problem starts to happen only after io_uring_prep_provide_buffers
is invoked to put back the buffer, the bid set is the one from cflags
>> 16.

The logic is as follow:
- io_uring_prep_provide_buffers + io_uring_submit + io_uring_wait_cqe
initialize all the buffers at the beginning
- within io_uring_for_each_cqe, when accepting a new connection a recv
sqe is submitted with the IOSQE_BUFFER_SELECT flag
- within io_uring_for_each_cqe, when recv a send sqe is submitted
using as buffer the one specified in cflags >> 16
- within io_uring_for_each_cqe, when send a provide buffers for the
bid used to send the data and a recv sqes are submitted.

If I drop io_uring_prep_provide_buffers both in my code and in the
code I referenced above it just works, but of course at some point
there are no more buffers available.

To further debug the issue I reduced the amount of provided buffers
and started to print out the entire bufferset and I noticed that after
the first correct RECV the kernel stores the data in the first buffer
of the group id but always returns the last buffer id.
It is like after calling io_uring_prep_provide_buffers the information
on the kernel side gets corrupted, I tried to follow the logic on the
kernel side but there is nothing apparent that would make me
understand why I am facing this behaviour.

The original author of that code told me on SO that he wrote & tested
it on the kernel 5.6 + the provide buffers branch, I am facing this
issue with 5.7.6, 5.8-rc1 and 5.8-rc3. The liburing library is built
out of the branch, I didn't do too much testing with different
versions but I tried to figure out where the issue was for the last
week and within this period I have pulled multiple times the repo.

Any hint or suggestion?

Thanks!
Daniele