linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Kirill Tkhai <ktkhai@virtuozzo.com>
To: Miklos Szeredi <miklos@szeredi.hu>
Cc: linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH 6/6] fuse: Do not take fuse_conn::lock on fuse_request_send_background()
Date: Wed, 26 Sep 2018 18:18:30 +0300	[thread overview]
Message-ID: <6978de82-7681-144a-318c-daadce00c230@virtuozzo.com> (raw)
In-Reply-To: <20180926122537.GC23439@veci.piliscsaba.redhat.com>

On 26.09.2018 15:25, Miklos Szeredi wrote:
> On Mon, Aug 27, 2018 at 06:29:56PM +0300, Kirill Tkhai wrote:
>> Currently, we take fc->lock there only to check for fc->connected.
>> But this flag is changed only on connection abort, which is very
>> rare operation. Good thing looks to make fuse_request_send_background()
>> faster, while fuse_abort_conn() slowler.
>>
>> So, we make fuse_request_send_background() lockless and mark
>> (fc->connected == 1) region as RCU-protected. Abort function
>> just uses synchronize_sched() to wait till all pending background
>> requests is being queued, and then makes ordinary abort.
>>
>> Note, that synchronize_sched() is used instead of synchronize_rcu(),
>> since we want to check for fc->connected without rcu_dereference()
>> in fuse_request_send_background() (i.e., not to add memory barriers
>> to this hot path).
> 
> Apart from the inaccuracies in the above (_sched variant is for scheduling and
> NMI taking code; _sched variant requires rcu_dereference() as well;
> rcu_dereference() does not add barriers; rcu_dereference() is only for pointers,
> so we can't use it for an integer),

Writing this I was inspired by expand_fdtable(). Yes, the description confuses,
and we don't need rcu_dereference() since we do not touch memory pointed by __rcu
pointer, we have no pointer at all. synchronize_sched() guarantees:

  On systems with more than one CPU, when synchronize_sched() returns,
  each CPU is guaranteed to have executed a full memory barrier since the
  end of its last RCU-sched read-side critical section whose beginning
  preceded the call to synchronize_sched().

(and rcu_dereference() unfolds in smp_read_barrier_depends(), which I mean as
 added barriers)

But it does not so matter. I'm OK with the patch you updated.

>wouldn't it be simpler to just use bg_lock
> for checking ->connected, and lock bg_lock (as well as fc->lock) when setting
> ->connected?
> 
> Updated patch below (untested).

Tested it. Works for me.

Thanks,
Kirill

> 
> ---
> Subject: fuse: do not take fc->lock in fuse_request_send_background()
> From: Kirill Tkhai <ktkhai@virtuozzo.com>
> Date: Mon, 27 Aug 2018 18:29:56 +0300
> 
> Currently, we take fc->lock there only to check for fc->connected.
> But this flag is changed only on connection abort, which is very
> rare operation.
> 
> Signed-off-by: Kirill Tkhai <ktkhai@virtuozzo.com>
> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
> ---
>  fs/fuse/dev.c    |   46 +++++++++++++++++++++++-----------------------
>  fs/fuse/file.c   |    4 +++-
>  fs/fuse/fuse_i.h |    4 +---
>  3 files changed, 27 insertions(+), 27 deletions(-)
> 
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -574,42 +574,38 @@ ssize_t fuse_simple_request(struct fuse_
>  	return ret;
>  }
>  
> -/*
> - * Called under fc->lock
> - *
> - * fc->connected must have been checked previously
> - */
> -void fuse_request_send_background_nocheck(struct fuse_conn *fc,
> -					  struct fuse_req *req)
> +bool fuse_request_queue_background(struct fuse_conn *fc, struct fuse_req *req)
>  {
> -	BUG_ON(!test_bit(FR_BACKGROUND, &req->flags));
> +	bool queued = false;
> +
> +	WARN_ON(!test_bit(FR_BACKGROUND, &req->flags));
>  	if (!test_bit(FR_WAITING, &req->flags)) {
>  		__set_bit(FR_WAITING, &req->flags);
>  		atomic_inc(&fc->num_waiting);
>  	}
>  	__set_bit(FR_ISREPLY, &req->flags);
>  	spin_lock(&fc->bg_lock);
> -	fc->num_background++;
> -	if (fc->num_background == fc->max_background)
> -		fc->blocked = 1;
> -	if (fc->num_background == fc->congestion_threshold && fc->sb) {
> -		set_bdi_congested(fc->sb->s_bdi, BLK_RW_SYNC);
> -		set_bdi_congested(fc->sb->s_bdi, BLK_RW_ASYNC);
> +	if (likely(fc->connected)) {
> +		fc->num_background++;
> +		if (fc->num_background == fc->max_background)
> +			fc->blocked = 1;
> +		if (fc->num_background == fc->congestion_threshold && fc->sb) {
> +			set_bdi_congested(fc->sb->s_bdi, BLK_RW_SYNC);
> +			set_bdi_congested(fc->sb->s_bdi, BLK_RW_ASYNC);
> +		}
> +		list_add_tail(&req->list, &fc->bg_queue);
> +		flush_bg_queue(fc);
> +		queued = true;
>  	}
> -	list_add_tail(&req->list, &fc->bg_queue);
> -	flush_bg_queue(fc);
>  	spin_unlock(&fc->bg_lock);
> +
> +	return queued;
>  }
>  
>  void fuse_request_send_background(struct fuse_conn *fc, struct fuse_req *req)
>  {
> -	BUG_ON(!req->end);
> -	spin_lock(&fc->lock);
> -	if (fc->connected) {
> -		fuse_request_send_background_nocheck(fc, req);
> -		spin_unlock(&fc->lock);
> -	} else {
> -		spin_unlock(&fc->lock);
> +	WARN_ON(!req->end);
> +	if (!fuse_request_queue_background(fc, req)) {
>  		req->out.h.error = -ENOTCONN;
>  		req->end(fc, req);
>  		fuse_put_request(fc, req);
> @@ -2112,7 +2108,11 @@ void fuse_abort_conn(struct fuse_conn *f
>  		struct fuse_req *req, *next;
>  		LIST_HEAD(to_end);
>  
> +		/* Background queuing checks fc->connected under bg_lock */
> +		spin_lock(&fc->bg_lock);
>  		fc->connected = 0;
> +		spin_unlock(&fc->bg_lock);
> +
>  		fc->aborted = is_abort;
>  		fuse_set_initialized(fc);
>  		list_for_each_entry(fud, &fc->devices, entry) {
> --- a/fs/fuse/fuse_i.h
> +++ b/fs/fuse/fuse_i.h
> @@ -863,9 +863,7 @@ ssize_t fuse_simple_request(struct fuse_
>   * Send a request in the background
>   */
>  void fuse_request_send_background(struct fuse_conn *fc, struct fuse_req *req);
> -
> -void fuse_request_send_background_nocheck(struct fuse_conn *fc,
> -					  struct fuse_req *req);
> +bool fuse_request_queue_background(struct fuse_conn *fc, struct fuse_req *req);
>  
>  /* Abort all requests */
>  void fuse_abort_conn(struct fuse_conn *fc, bool is_abort);
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -1487,6 +1487,7 @@ __acquires(fc->lock)
>  	struct fuse_inode *fi = get_fuse_inode(req->inode);
>  	struct fuse_write_in *inarg = &req->misc.write.in;
>  	__u64 data_size = req->num_pages * PAGE_SIZE;
> +	bool queued;
>  
>  	if (!fc->connected)
>  		goto out_free;
> @@ -1502,7 +1503,8 @@ __acquires(fc->lock)
>  
>  	req->in.args[1].size = inarg->size;
>  	fi->writectr++;
> -	fuse_request_send_background_nocheck(fc, req);
> +	queued = fuse_request_queue_background(fc, req);
> +	WARN_ON(!queued);
>  	return;
>  
>   out_free:
> 

  reply	other threads:[~2018-09-26 21:32 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-08-27 15:29 [PATCH 0/6] Extract bg queue logic out fuse_conn::lock Kirill Tkhai
2018-08-27 15:29 ` [PATCH 1/6] fuse: Use list_first_entry() in flush_bg_queue() Kirill Tkhai
2018-08-27 15:29 ` [PATCH 2/6] fuse: Move clear_bit() up in request_end() Kirill Tkhai
2018-08-27 15:29 ` [PATCH 3/6] fuse: Underline congestion_threshold and max_background may be read w/o fc->lock Kirill Tkhai
2018-08-27 15:29 ` [PATCH 4/6] fuse: Lock fc->lock during changing num_background and congestion_threshold Kirill Tkhai
2018-08-27 15:29 ` [PATCH 5/6] fs: Introduce fuse_conn::bg_lock Kirill Tkhai
2018-08-27 15:29 ` [PATCH 6/6] fuse: Do not take fuse_conn::lock on fuse_request_send_background() Kirill Tkhai
2018-09-26 12:25   ` Miklos Szeredi
2018-09-26 15:18     ` Kirill Tkhai [this message]
2018-09-27  8:37       ` Kirill Tkhai
2018-09-27 11:25         ` Miklos Szeredi
2018-09-11 10:14 ` [PATCH 0/6] Extract bg queue logic out fuse_conn::lock Kirill Tkhai
2018-10-01  9:23   ` Miklos Szeredi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6978de82-7681-144a-318c-daadce00c230@virtuozzo.com \
    --to=ktkhai@virtuozzo.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=miklos@szeredi.hu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).