Linux-Block Archive on lore.kernel.org
 help / color / Atom feed
From: Jens Axboe <axboe@kernel.dk>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-aio@kvack.org, linux-block@vger.kernel.org,
	linux-api@vger.kernel.org, hch@lst.de, jmoyer@redhat.com,
	avi@scylladb.com, jannh@google.com, viro@ZenIV.linux.org.uk
Subject: Re: [PATCH 05/19] Add io_uring IO interface
Date: Sun, 10 Feb 2019 07:19:08 -0700
Message-ID: <eb713e6b-0fa4-852b-5a94-8e159aea0603@kernel.dk> (raw)
In-Reply-To: <alpine.DEB.2.21.1902101048100.8784@nanos.tec.linutronix.de>

On 2/10/19 5:03 AM, Thomas Gleixner wrote:
> On Sat, 9 Feb 2019, Jens Axboe wrote:
>> +static void io_commit_cqring(struct io_ring_ctx *ctx)
>> +{
>> +	struct io_cq_ring *ring = ctx->cq_ring;
>> +
>> +	if (ctx->cached_cq_tail != READ_ONCE(ring->r.tail)) {
>> +		/* order cqe stores with ring update */
> 
> This lacks a reference to the matching rmb()
> 
>> +		smp_wmb();
>> +		WRITE_ONCE(ring->r.tail, ctx->cached_cq_tail);
>> +		/* write side barrier of tail update, app has read side */
> 
> That's a bit meager. Can you please document the barriers which are paired
> with user space barriers very elaborate?

Agree, I added a lengthier comment at the top of the file to explain the
relationship between the kernel and application barriers.

Also documented all of them explicitly now, or referenced the top
comment.

>> +		if (READ_ONCE(ring->r.head) != READ_ONCE(ring->r.tail))
>> +			break;
>> +
> 
> There are undocumented smp_wmb()'s in 'io_uring: Add submission polling' as
> well. It's really hard to tell where the corresponding barriers are and
> what they are supposed to order.

Fixed those too.

> Especially the barriers which are paired with user space barriers need some
> careful documentation. What are the side effects if user space is missing a
> barrier? Just user space seing unconsistent data or is there something
> which goes the other way round and might cause havoc in the kernel?

The only side effect is that the application may not proceed as quickly
as it should, if it fails to notice a completion event come in. It'll
lead to it calling io_cqring_wait() ultimately, through a system call,
which will then ensure that it makes progress. But it may not have
needed to do that system call at all, if it used the barriers correctly.

I've also added a reference to the liburing git repo which has examples
of how to do it.

Thanks for your comments!

-- 
Jens Axboe


  reply index

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-09 21:13 [PATCHSET v14] " Jens Axboe
2019-02-09 21:13 ` [PATCH 01/19] fs: add an iopoll method to struct file_operations Jens Axboe
2019-02-09 21:13 ` [PATCH 02/19] block: wire up block device iopoll method Jens Axboe
2019-02-09 21:13 ` [PATCH 03/19] block: add bio_set_polled() helper Jens Axboe
2019-02-09 21:13 ` [PATCH 04/19] iomap: wire up the iopoll method Jens Axboe
2019-02-09 21:13 ` [PATCH 05/19] Add io_uring IO interface Jens Axboe
2019-02-10 12:03   ` Thomas Gleixner
2019-02-10 14:19     ` Jens Axboe [this message]
2019-02-09 21:13 ` [PATCH 06/19] io_uring: add fsync support Jens Axboe
2019-02-09 21:13 ` [PATCH 07/19] io_uring: support for IO polling Jens Axboe
2019-02-09 21:13 ` [PATCH 08/19] fs: add fget_many() and fput_many() Jens Axboe
2019-02-09 21:13 ` [PATCH 09/19] io_uring: use fget/fput_many() for file references Jens Axboe
2019-02-09 21:13 ` [PATCH 10/19] io_uring: batch io_kiocb allocation Jens Axboe
2019-02-09 21:13 ` [PATCH 11/19] block: implement bio helper to add iter bvec pages to bio Jens Axboe
2019-02-09 21:13 ` [PATCH 12/19] io_uring: add support for pre-mapped user IO buffers Jens Axboe
2019-02-09 21:13 ` [PATCH 13/19] net: split out functions related to registering inflight socket files Jens Axboe
2019-02-09 21:13 ` [PATCH 14/19] io_uring: add file set registration Jens Axboe
2019-02-09 23:52   ` Matt Mullins
2019-02-10  0:47     ` Jens Axboe
     [not found]       ` <60e4c6a489549daad1fb2c5e8eee5496c668d79a.camel@fb.com>
2019-02-10  2:34         ` Jens Axboe
2019-02-10  2:57           ` Jens Axboe
2019-02-10 19:55             ` Matt Mullins
2019-02-09 21:13 ` [PATCH 15/19] io_uring: add submission polling Jens Axboe
2019-02-09 21:13 ` [PATCH 16/19] io_uring: add io_kiocb ref count Jens Axboe
2019-02-10  9:19   ` Hannes Reinecke
2019-02-09 21:13 ` [PATCH 17/19] io_uring: add support for IORING_OP_POLL Jens Axboe
2019-02-10  9:28   ` Hannes Reinecke
2019-02-09 21:13 ` [PATCH 18/19] io_uring: allow workqueue item to handle multiple buffered requests Jens Axboe
2019-02-10  9:31   ` Hannes Reinecke
2019-02-09 21:13 ` [PATCH 19/19] io_uring: add io_uring_event cache hit information Jens Axboe
2019-02-10  9:36   ` Hannes Reinecke
2019-02-10 13:39     ` Jens Axboe
  -- strict thread matches above, loose matches on Subject: below --
2019-02-11 19:00 [PATCHSET v15] io_uring IO interface Jens Axboe
2019-02-11 19:00 ` [PATCH 05/19] Add " Jens Axboe
2019-02-08 17:34 [PATCHSET v13] " Jens Axboe
2019-02-08 17:34 ` [PATCH 05/19] Add " Jens Axboe
2019-02-08 22:12   ` Jann Horn
2019-02-09  4:15     ` Jens Axboe
2019-02-12 21:42       ` Jann Horn
2019-02-12 22:03         ` Jens Axboe
2019-02-12 22:06           ` Jens Axboe
2019-02-12 22:40             ` Jann Horn
2019-02-12 22:45               ` Jens Axboe
2019-02-12 22:52                 ` Jens Axboe
2019-02-12 22:57                   ` Jann Horn
2019-02-12 23:00                     ` Jens Axboe
2019-02-12 23:11                       ` Jann Horn
2019-02-12 23:19                         ` Jens Axboe
2019-02-12 23:28                           ` Jann Horn
2019-02-12 23:46                             ` Jens Axboe
2019-02-12 23:53                               ` Jens Axboe
2019-02-13  0:07                                 ` Andy Lutomirski
2019-02-13  0:14                                   ` Jann Horn
2019-02-13  0:24                                   ` Jens Axboe
2019-02-09  9:35   ` Hannes Reinecke

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=eb713e6b-0fa4-852b-5a94-8e159aea0603@kernel.dk \
    --to=axboe@kernel.dk \
    --cc=avi@scylladb.com \
    --cc=hch@lst.de \
    --cc=jannh@google.com \
    --cc=jmoyer@redhat.com \
    --cc=linux-aio@kvack.org \
    --cc=linux-api@vger.kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=tglx@linutronix.de \
    --cc=viro@ZenIV.linux.org.uk \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Block Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-block/0 linux-block/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-block linux-block/ https://lore.kernel.org/linux-block \
		linux-block@vger.kernel.org linux-block@archiver.kernel.org
	public-inbox-index linux-block

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-block


AGPL code for this site: git clone https://public-inbox.org/ public-inbox