From: Hannes Reinecke <hare@suse.de>
To: Ilya Dryomov <idryomov@gmail.com>
Cc: Sage Weil <sage@redhat.com>, Daniel Disseldorp <ddiss@suse.com>,
Jens Axboe <axboe@kernel.dk>,
Ceph Development <ceph-devel@vger.kernel.org>,
linux-block <linux-block@vger.kernel.org>
Subject: Re: [PATCH 02/15] rbd: use READ_ONCE() when checking the mapping size
Date: Tue, 4 Feb 2020 08:05:35 +0100 [thread overview]
Message-ID: <43e755d9-d7c0-5d0c-436f-220a1c807f85@suse.de> (raw)
In-Reply-To: <CAOi1vP--6qWHtifpeBVWRYOP8J_CC+fvKOkG7Xdsjoxaa7mrDQ@mail.gmail.com>
On 2/3/20 5:50 PM, Ilya Dryomov wrote:
> On Fri, Jan 31, 2020 at 11:38 AM Hannes Reinecke <hare@suse.de> wrote:
>>
>> The mapping size is changed only very infrequently, so we don't
>> need to take the header mutex for checking; using READ_ONCE()
>> is sufficient here. And it avoids having to take a mutex in the
>> hot path.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>> drivers/block/rbd.c | 12 ++++++------
>> 1 file changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
>> index db80b964d8ea..792180548e89 100644
>> --- a/drivers/block/rbd.c
>> +++ b/drivers/block/rbd.c
>> @@ -4788,13 +4788,13 @@ static void rbd_queue_workfn(struct work_struct *work)
>>
>> blk_mq_start_request(rq);
>>
>> - down_read(&rbd_dev->header_rwsem);
>> - mapping_size = rbd_dev->mapping.size;
>> + mapping_size = READ_ONCE(rbd_dev->mapping.size);
>> if (op_type != OBJ_OP_READ) {
>> + down_read(&rbd_dev->header_rwsem);
>> snapc = rbd_dev->header.snapc;
>> ceph_get_snap_context(snapc);
>> + up_read(&rbd_dev->header_rwsem);
>> }
>> - up_read(&rbd_dev->header_rwsem);
>>
>> if (offset + length > mapping_size) {
>> rbd_warn(rbd_dev, "beyond EOD (%llu~%llu > %llu)", offset,
>> @@ -4981,9 +4981,9 @@ static int rbd_dev_refresh(struct rbd_device *rbd_dev)
>> u64 mapping_size;
>> int ret;
>>
>> - down_write(&rbd_dev->header_rwsem);
>> - mapping_size = rbd_dev->mapping.size;
>> + mapping_size = READ_ONCE(rbd_dev->mapping.size);
>>
>> + down_write(&rbd_dev->header_rwsem);
>> ret = rbd_dev_header_info(rbd_dev);
>> if (ret)
>> goto out;
>> @@ -4999,7 +4999,7 @@ static int rbd_dev_refresh(struct rbd_device *rbd_dev)
>> }
>>
>> rbd_assert(!rbd_is_snap(rbd_dev));
>> - rbd_dev->mapping.size = rbd_dev->header.image_size;
>> + WRITE_ONCE(rbd_dev->mapping.size, rbd_dev->header.image_size);
>>
>> out:
>> up_write(&rbd_dev->header_rwsem);
>
> Does this result in a measurable performance improvement?
>
> I'd rather not go down the READ/WRITE_ONCE path and continue using
> proper locking, especially given that it's only for reads. FWIW the
> plan is to replace header_rwsem with a spin lock, after refactoring
> header read-in code to use a private buffer instead of reading into
> rbd_dev directly.
>
Well ... Not sure if I like the spin_lock idea.
Thing is, the mapping size is evaluated exactly _once_ when assembling
the request. So any change to the mapping size just after we've read it
would go unnoticed.
Hence it should be possible to combine both approaches; use READ_ONCE()
to read the mapping size, but use a spin lock for updating it as you
suggested. That way we'll eliminate a lock in the hot path, but would be
getting safe updates.
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
next prev parent reply other threads:[~2020-02-04 7:05 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-31 10:37 [PATCH 00/15] rbd: switch to blk-mq Hannes Reinecke
2020-01-31 10:37 ` [PATCH 01/15] rbd: lock object request list Hannes Reinecke
2020-02-03 16:38 ` Ilya Dryomov
2020-01-31 10:37 ` [PATCH 02/15] rbd: use READ_ONCE() when checking the mapping size Hannes Reinecke
2020-02-03 16:50 ` Ilya Dryomov
2020-02-04 7:05 ` Hannes Reinecke [this message]
2020-01-31 10:37 ` [PATCH 03/15] rbd: reorder rbd_img_advance() Hannes Reinecke
2020-01-31 10:37 ` [PATCH 04/15] rbd: reorder switch statement in rbd_advance_read() Hannes Reinecke
2020-01-31 10:37 ` [PATCH 05/15] rbd: reorder switch statement in rbd_advance_write() Hannes Reinecke
2020-01-31 10:37 ` [PATCH 06/15] rbd: add 'done' state for rbd_obj_advance_copyup() Hannes Reinecke
2020-01-31 10:37 ` [PATCH 07/15] rbd: use callback for image request completion Hannes Reinecke
2020-02-03 17:13 ` Ilya Dryomov
2020-01-31 10:37 ` [PATCH 08/15] rbd: add debugging statements for the state machine Hannes Reinecke
2020-01-31 10:37 ` [PATCH 09/15] rbd: count pending object requests in-line Hannes Reinecke
2020-02-03 17:47 ` Ilya Dryomov
2020-02-04 6:59 ` Hannes Reinecke
2020-01-31 10:37 ` [PATCH 10/15] rbd: kill 'work_result' Hannes Reinecke
2020-01-31 10:37 ` [PATCH 11/15] rbd: drop state_mutex in __rbd_img_handle_request() Hannes Reinecke
2020-02-03 18:01 ` Ilya Dryomov
2020-01-31 10:37 ` [PATCH 12/15] rbd: kill img_request kref Hannes Reinecke
2020-01-31 10:37 ` [PATCH 13/15] rbd: schedule image_request after preparation Hannes Reinecke
2020-02-03 18:40 ` Ilya Dryomov
2020-01-31 10:37 ` [PATCH 14/15] rbd: embed image request as blk_mq request payload Hannes Reinecke
2020-01-31 10:37 ` [PATCH 15/15] rbd: switch to blk-mq Hannes Reinecke
2020-02-03 8:36 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=43e755d9-d7c0-5d0c-436f-220a1c807f85@suse.de \
--to=hare@suse.de \
--cc=axboe@kernel.dk \
--cc=ceph-devel@vger.kernel.org \
--cc=ddiss@suse.com \
--cc=idryomov@gmail.com \
--cc=linux-block@vger.kernel.org \
--cc=sage@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).