Hi Linus, Lots of cleanups in here, hardening the code and/or making it easier to read and fixing buts, but a core feature/change too adding support for real async buffered reads. With the latter in place, we just need buffered write async support and we're done relying on kthreads for the fast path. In detail: - Cleanup how memory accounting is done on ring setup/free (Bijan) - sq array offset calculation fixup (Dmitry) - Consistently handle blocking off O_DIRECT submission path (me) - Support proper async buffered reads, instead of relying on kthread offload for that. This uses the page waitqueue to drive retries from task_work, like we handle poll based retry. (me) - IO completion optimizations (me) - Fix race with accounting and ring fd install (me) - Support EPOLLEXCLUSIVE (Jiufei) - Get rid of the io_kiocb unionizing, made possible by shrinking other bits (Pavel) - Completion side cleanups (Pavel) - Cleanup REQ_F_ flags handling, and kill off many of them (Pavel) - Request environment grabbing cleanups (Pavel) - File and socket read/write cleanups (Pavel) - Improve kiocb_set_rw_flags() (Pavel) - Tons of fixes and cleanups (Pavel) - IORING_SQ_NEED_WAKEUP clear fix (Xiaoguang) This will throw a few merge conflicts. One is due to the IOCB_NOIO addition that happened late in 5.8-rc, the other is due to a change in for-5.9/block. Both are trivial to fixup, I'm attaching my merge resolution when I pulled it in locally. Please pull! The following changes since commit 4ae6dbd683860b9edc254ea8acf5e04b5ae242e5: io_uring: fix lockup in io_fail_links() (2020-07-24 12:51:33 -0600) are available in the Git repository at: git://git.kernel.dk/linux-block.git tags/for-5.9/io_uring-20200802 for you to fetch changes up to fa15bafb71fd7a4d6018dae87cfaf890fd4ab47f: io_uring: flip if handling after io_setup_async_rw (2020-08-01 11:02:57 -0600) ---------------------------------------------------------------- for-5.9/io_uring-20200802 ---------------------------------------------------------------- Bijan Mottahedeh (4): io_uring: add wrappers for memory accounting io_uring: rename ctx->account_mem field io_uring: report pinned memory usage io_uring: separate reporting of ring pages from registered pages Dan Carpenter (1): io_uring: fix a use after free in io_async_task_func() Dmitry Vyukov (1): io_uring: fix sq array offset calculation Jens Axboe (31): block: provide plug based way of signaling forced no-wait semantics io_uring: always plug for any number of IOs io_uring: catch -EIO from buffered issue request failure io_uring: re-issue block requests that failed because of resources mm: allow read-ahead with IOCB_NOWAIT set mm: abstract out wake_page_match() from wake_page_function() mm: add support for async page locking mm: support async buffered reads in generic_file_buffered_read() fs: add FMODE_BUF_RASYNC block: flag block devices as supporting IOCB_WAITQ xfs: flag files as supporting buffered async reads btrfs: flag files as supporting buffered async reads mm: add kiocb_wait_page_queue_init() helper io_uring: support true async buffered reads, if file provides it Merge branch 'async-buffered.8' into for-5.9/io_uring io_uring: provide generic io_req_complete() helper io_uring: add 'io_comp_state' to struct io_submit_state io_uring: pass down completion state on the issue side io_uring: pass in completion state to appropriate issue side handlers io_uring: enable READ/WRITE to use deferred completions io_uring: use task_work for links if possible Merge branch 'io_uring-5.8' into for-5.9/io_uring io_uring: clean up io_kill_linked_timeout() locking Merge branch 'io_uring-5.8' into for-5.9/io_uring io_uring: abstract out task work running io_uring: use new io_req_task_work_add() helper throughout io_uring: only call kfree() for a non-zero pointer io_uring: get rid of __req_need_defer() io_uring: remove dead 'ctx' argument and move forward declaration Merge branch 'io_uring-5.8' into for-5.9/io_uring io_uring: don't touch 'ctx' after installing file descriptor Jiufei Xue (2): io_uring: change the poll type to be 32-bits io_uring: use EPOLLEXCLUSIVE flag to aoid thundering herd type behavior Pavel Begunkov (90): io_uring: remove setting REQ_F_MUST_PUNT in rw io_uring: remove REQ_F_MUST_PUNT io_uring: set @poll->file after @poll init io_uring: kill NULL checks for submit state io_uring: fix NULL-mm for linked reqs io-wq: compact io-wq flags numbers io-wq: return next work from ->do_work() directly io_uring: fix req->work corruption io_uring: fix punting req w/o grabbed env io_uring: fix feeding io-wq with uninit reqs io_uring: don't mark link's head for_async io_uring: fix missing io_grab_files() io_uring: fix refs underflow in io_iopoll_queue() io_uring: remove inflight batching in free_many() io_uring: dismantle req early and remove need_iter io_uring: batch-free linked requests as well io_uring: cosmetic changes for batch free io_uring: kill REQ_F_LINK_NEXT io_uring: clean up req->result setting by rw io_uring: do task_work_run() during iopoll io_uring: fix iopoll -EAGAIN handling io_uring: fix missing wake_up io_rw_reissue() io_uring: deduplicate freeing linked timeouts io_uring: replace find_next() out param with ret io_uring: kill REQ_F_TIMEOUT io_uring: kill REQ_F_TIMEOUT_NOSEQ io_uring: fix potential use after free on fallback request free io_uring: don't pass def into io_req_work_grab_env io_uring: do init work in grab_env() io_uring: factor out grab_env() from defer_prep() io_uring: do grab_env() just before punting io_uring: don't fail iopoll requeue without ->mm io_uring: fix NULL mm in io_poll_task_func() io_uring: simplify io_async_task_func() io_uring: optimise io_req_find_next() fast check io_uring: fix missing ->mm on exit io_uring: fix mis-refcounting linked timeouts io_uring: keep queue_sqe()'s fail path separately io_uring: fix lost cqe->flags io_uring: don't delay iopoll'ed req completion io_uring: fix stopping iopoll'ing too early io_uring: briefly loose locks while reaping events io_uring: partially inline io_iopoll_getevents() io_uring: remove nr_events arg from iopoll_check() io_uring: don't burn CPU for iopoll on exit io_uring: rename sr->msg into umsg io_uring: use more specific type in rcv/snd msg cp io_uring: extract io_sendmsg_copy_hdr() io_uring: replace rw->task_work with rq->task_work io_uring: simplify io_req_map_rw() io_uring: add a helper for async rw iovec prep io_uring: follow **iovec idiom in io_import_iovec io_uring: share completion list w/ per-op space io_uring: rename ctx->poll into ctx->iopoll io_uring: use inflight_entry list for iopoll'ing io_uring: use completion list for CQ overflow io_uring: add req->timeout.list io_uring: remove init for unused list io_uring: use non-intrusive list for defer io_uring: remove sequence from io_kiocb io_uring: place cflags into completion data io_uring: inline io_req_work_grab_env() io_uring: remove empty cleanup of OP_OPEN* reqs io_uring: alloc ->io in io_req_defer_prep() io_uring/io-wq: move RLIMIT_FSIZE to io-wq io_uring: simplify file ref tracking in submission state io_uring: indent left {send,recv}[msg]() io_uring: remove extra checks in send/recv io_uring: don't forget cflags in io_recv() io_uring: free selected-bufs if error'ed io_uring: move BUFFER_SELECT check into *recv[msg] io_uring: extract io_put_kbuf() helper io_uring: don't open-code recv kbuf managment io_uring: don't miscount pinned memory io_uring: return locked and pinned page accounting tasks: add put_task_struct_many() io_uring: batch put_task_struct() io_uring: don't do opcode prep twice io_uring: deduplicate io_grab_files() calls io_uring: mark ->work uninitialised after cleanup io_uring: fix missing io_queue_linked_timeout() io-wq: update hash bits io_uring: de-unionise io_kiocb io_uring: deduplicate __io_complete_rw() io_uring: fix racy overflow count reporting io_uring: fix stalled deferred requests io_uring: consolidate *_check_overflow accounting io_uring: get rid of atomic FAA for cq_timeouts fs: optimise kiocb_set_rw_flags() io_uring: flip if handling after io_setup_async_rw Randy Dunlap (1): io_uring: fix function args for !CONFIG_NET Xiaoguang Wang (1): io_uring: clear IORING_SQ_NEED_WAKEUP after executing task works block/blk-core.c | 6 + fs/block_dev.c | 2 +- fs/btrfs/file.c | 2 +- fs/io-wq.c | 14 +- fs/io-wq.h | 11 +- fs/io_uring.c | 2588 +++++++++++++++++++++++------------------ fs/xfs/xfs_file.c | 2 +- include/linux/blkdev.h | 1 + include/linux/fs.h | 26 +- include/linux/pagemap.h | 75 ++ include/linux/sched/task.h | 6 + include/uapi/linux/io_uring.h | 4 +- mm/filemap.c | 110 +- tools/io_uring/liburing.h | 6 +- 14 files changed, 1658 insertions(+), 1195 deletions(-) -- Jens Axboe