From: Pavel Begunkov <asml.silence@gmail.com>
To: io-uring@vger.kernel.org, netdev@vger.kernel.org,
bpf@vger.kernel.org, linux-kernel@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
Yonghong Song <yhs@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@kernel.org>,
Horst Schirmeier <horst.schirmeier@tu-dortmund.de>,
"Franz-B . Tuneke" <franz-bernhard.tuneke@tu-dortmund.de>,
Christian Dietrich <stettberger@dokucode.de>
Subject: [PATCH 17/23] io_uring: enable bpf to reap CQEs
Date: Wed, 19 May 2021 15:13:28 +0100 [thread overview]
Message-ID: <e12812fd4cab5db70f0ae290166d0bf3a1f89076.1621424513.git.asml.silence@gmail.com> (raw)
In-Reply-To: <cover.1621424513.git.asml.silence@gmail.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 48 ++++++++++++++++++++++++++++++++++++++++
include/uapi/linux/bpf.h | 1 +
2 files changed, 49 insertions(+)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 464d630904e2..7c165b2ce8e4 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -10394,6 +10394,42 @@ BPF_CALL_5(io_bpf_emit_cqe, struct io_bpf_ctx *, bpf_ctx,
return submitted ? 0 : -ENOMEM;
}
+BPF_CALL_4(io_bpf_reap_cqe, struct io_bpf_ctx *, bpf_ctx,
+ u32, cq_idx,
+ struct io_uring_cqe *, cqe_out,
+ u32, cqe_len)
+{
+ struct io_ring_ctx *ctx = bpf_ctx->ctx;
+ struct io_uring_cqe *cqe;
+ struct io_cqring *cq;
+ struct io_rings *r;
+ unsigned tail, head, mask;
+ int ret = -EINVAL;
+
+ if (unlikely(cqe_len != sizeof(*cqe_out)))
+ goto err;
+ if (unlikely(cq_idx >= ctx->cq_nr))
+ goto err;
+
+ cq = &ctx->cqs[cq_idx];
+ r = cq->rings;
+ tail = READ_ONCE(r->cq.tail);
+ head = smp_load_acquire(&r->cq.head);
+
+ ret = -ENOENT;
+ if (unlikely(tail == head))
+ goto err;
+
+ mask = cq->entries - 1;
+ cqe = &r->cqes[head & mask];
+ memcpy(cqe_out, cqe, sizeof(*cqe_out));
+ WRITE_ONCE(r->cq.head, head + 1);
+ return 0;
+err:
+ memset(cqe_out, 0, sizeof(*cqe_out));
+ return ret;
+}
+
const struct bpf_func_proto io_bpf_queue_sqe_proto = {
.func = io_bpf_queue_sqe,
.gpl_only = false,
@@ -10414,6 +10450,16 @@ const struct bpf_func_proto io_bpf_emit_cqe_proto = {
.arg5_type = ARG_ANYTHING,
};
+const struct bpf_func_proto io_bpf_reap_cqe_proto = {
+ .func = io_bpf_reap_cqe,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg2_type = ARG_ANYTHING,
+ .arg3_type = ARG_PTR_TO_UNINIT_MEM,
+ .arg4_type = ARG_CONST_SIZE,
+};
+
static const struct bpf_func_proto *
io_bpf_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
@@ -10424,6 +10470,8 @@ io_bpf_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
return prog->aux->sleepable ? &io_bpf_queue_sqe_proto : NULL;
case BPF_FUNC_iouring_emit_cqe:
return &io_bpf_emit_cqe_proto;
+ case BPF_FUNC_iouring_reap_cqe:
+ return &io_bpf_reap_cqe_proto;
default:
return bpf_base_func_proto(func_id);
}
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index c6b023be7848..7719ec4a33e7 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -4084,6 +4084,7 @@ union bpf_attr {
FN(check_mtu), \
FN(iouring_queue_sqe), \
FN(iouring_emit_cqe), \
+ FN(iouring_reap_cqe), \
/* */
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
--
2.31.1
next prev parent reply other threads:[~2021-05-19 14:15 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-19 14:13 [RFC v2 00/23] io_uring BPF requests Pavel Begunkov
2021-05-19 14:13 ` [PATCH 01/23] io_uring: shuffle rarely used ctx fields Pavel Begunkov
2021-05-20 21:46 ` Song Liu
2021-05-20 22:46 ` Pavel Begunkov
2021-05-19 14:13 ` [PATCH 02/23] io_uring: localise fixed resources fields Pavel Begunkov
2021-05-19 14:13 ` [PATCH 03/23] io_uring: remove dependency on ring->sq/cq_entries Pavel Begunkov
2021-05-19 14:13 ` [PATCH 04/23] io_uring: deduce cq_mask from cq_entries Pavel Begunkov
2021-05-19 14:13 ` [PATCH 05/23] io_uring: kill cached_cq_overflow Pavel Begunkov
2021-05-19 14:13 ` [PATCH 06/23] io_uring: rename io_get_cqring Pavel Begunkov
2021-05-19 14:13 ` [PATCH 07/23] io_uring: extract struct for CQ Pavel Begunkov
2021-05-19 14:13 ` [PATCH 08/23] io_uring: internally pass CQ indexes Pavel Begunkov
2021-05-19 14:13 ` [PATCH 09/23] io_uring: extract cq size helper Pavel Begunkov
2021-05-19 14:13 ` [PATCH 10/23] io_uring: add support for multiple CQs Pavel Begunkov
2021-05-19 14:13 ` [PATCH 11/23] io_uring: enable mmap'ing additional CQs Pavel Begunkov
2021-05-19 14:13 ` [PATCH 12/23] bpf: add IOURING program type Pavel Begunkov
2021-05-20 23:34 ` Song Liu
2021-05-21 0:56 ` Pavel Begunkov
2021-05-19 14:13 ` [PATCH 13/23] io_uring: implement bpf prog registration Pavel Begunkov
2021-05-20 23:45 ` Song Liu
2021-05-21 0:43 ` Pavel Begunkov
2021-05-19 14:13 ` [PATCH 14/23] io_uring: add support for bpf requests Pavel Begunkov
2021-05-21 0:42 ` Pavel Begunkov
2021-05-19 14:13 ` [PATCH 15/23] io_uring: enable BPF to submit SQEs Pavel Begunkov
2021-05-21 0:06 ` Song Liu
2021-05-21 1:07 ` Alexei Starovoitov
2021-05-21 9:33 ` Pavel Begunkov
2021-05-19 14:13 ` [PATCH 16/23] io_uring: enable bpf to submit CQEs Pavel Begunkov
2021-05-19 14:13 ` Pavel Begunkov [this message]
2021-05-19 14:13 ` [PATCH 18/23] libbpf: support io_uring Pavel Begunkov
2021-05-19 17:38 ` Andrii Nakryiko
2021-05-20 9:58 ` Pavel Begunkov
2021-05-20 17:23 ` Andrii Nakryiko
2021-05-19 14:13 ` [PATCH 19/23] io_uring: pass user_data to bpf executor Pavel Begunkov
2021-05-19 14:13 ` [PATCH 20/23] bpf: Add bpf_copy_to_user() helper Pavel Begunkov
2021-05-19 14:13 ` [PATCH 21/23] io_uring: wire bpf copy to user Pavel Begunkov
2021-05-19 14:13 ` [PATCH 22/23] io_uring: don't wait on CQ exclusively Pavel Begunkov
2021-05-19 14:13 ` [PATCH 23/23] io_uring: enable bpf reqs to wait for CQs Pavel Begunkov
2021-05-21 0:35 ` [RFC v2 00/23] io_uring BPF requests Song Liu
2021-05-21 0:58 ` Pavel Begunkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e12812fd4cab5db70f0ae290166d0bf3a1f89076.1621424513.git.asml.silence@gmail.com \
--to=asml.silence@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=axboe@kernel.dk \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=franz-bernhard.tuneke@tu-dortmund.de \
--cc=horst.schirmeier@tu-dortmund.de \
--cc=io-uring@vger.kernel.org \
--cc=john.fastabend@gmail.com \
--cc=kafai@fb.com \
--cc=kpsingh@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=songliubraving@fb.com \
--cc=stettberger@dokucode.de \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).