[PATCH] blktrace: re-write setting q->blk_trace

[PATCH] blktrace: re-write setting q->blk_trace

[Davidlohr Bueso]

This is really about simplifying the double xchg patterns into
a single cmpxchg, with the same logic. Other than the immediate
cleanup, there are some subtleties this change deals with:

(i) While the load of the old bt is fully ordered wrt everything,
ie:

       old_bt = xchg(&q->blk_trace, bt);             [barrier]
       if (old_bt)
	     (void) xchg(&q->blk_trace, old_bt);    [barrier]

blk_trace could still be changed between the xchg and the old_bt
load. Note that this description is merely theoretical and afaict
very small, but doing everything in a single context with cmpxchg
closes this potential race.

(ii) Ordering guarantees are obviously kept with cmpxchg.

Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
  kernel/trace/blktrace.c | 15 ++++-----------
  1 file changed, 4 insertions(+), 11 deletions(-)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index 90e72a0..4a30229 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -437,7 +437,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
		       struct block_device *bdev,
		       struct blk_user_trace_setup *buts)
  {
-	struct blk_trace *old_bt, *bt = NULL;
+	struct blk_trace *bt = NULL;
	struct dentry *dir = NULL;
	int ret;

@@ -519,11 +519,8 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
	bt->trace_state = Blktrace_setup;

	ret = -EBUSY;
-	old_bt = xchg(&q->blk_trace, bt);
-	if (old_bt) {
-		(void) xchg(&q->blk_trace, old_bt);
+	if (cmpxchg(&q->blk_trace, NULL, bt))
		goto err;
-	}

	if (atomic_inc_return(&blk_probes_ref) == 1)
		blk_register_tracepoints();
@@ -1481,7 +1478,7 @@ static int blk_trace_remove_queue(struct request_queue *q)
  static int blk_trace_setup_queue(struct request_queue *q,
				 struct block_device *bdev)
  {
-	struct blk_trace *old_bt, *bt = NULL;
+	struct blk_trace *bt = NULL;
	int ret = -ENOMEM;

	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
@@ -1497,12 +1494,8 @@ static int blk_trace_setup_queue(struct request_queue *q,

	blk_trace_setup_lba(bt, bdev);

-	old_bt = xchg(&q->blk_trace, bt);
-	if (old_bt != NULL) {
-		(void)xchg(&q->blk_trace, old_bt);
-		ret = -EBUSY;
+	if (cmpxchg(&q->blk_trace, NULL, bt))
		goto free_bt;
-	}

	if (atomic_inc_return(&blk_probes_ref) == 1)
		blk_register_tracepoints();
--
2.1.4

[PATCH v2] blktrace: re-write setting q->blk_trace

[Davidlohr Bueso]

This is really about simplifying the double xchg patterns into
a single cmpxchg, with the same logic. Other than the immediate
cleanup, there are some subtleties this change deals with:

(i) While the load of the old bt is fully ordered wrt everything,
ie:

       old_bt = xchg(&q->blk_trace, bt);             [barrier]
       if (old_bt)
	     (void) xchg(&q->blk_trace, old_bt);    [barrier]

blk_trace could still be changed between the xchg and the old_bt
load. Note that this description is merely theoretical and afaict
very small, but doing everything in a single context with cmpxchg
closes this potential race.

(ii) Ordering guarantees are obviously kept with cmpxchg.

Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---

v2: sorry I sent a stale version which didn't set EBUSY when the structure is
already initialized.

  kernel/trace/blktrace.c | 16 +++++-----------
  1 file changed, 5 insertions(+), 11 deletions(-)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index 90e72a0..e3a2618 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -437,7 +437,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
  		       struct block_device *bdev,
  		       struct blk_user_trace_setup *buts)
  {
-	struct blk_trace *old_bt, *bt = NULL;
+	struct blk_trace *bt = NULL;
  	struct dentry *dir = NULL;
  	int ret;
  
@@ -519,11 +519,8 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
  	bt->trace_state = Blktrace_setup;
  
  	ret = -EBUSY;
-	old_bt = xchg(&q->blk_trace, bt);
-	if (old_bt) {
-		(void) xchg(&q->blk_trace, old_bt);
+	if (cmpxchg(&q->blk_trace, NULL, bt))
  		goto err;
-	}
  
  	if (atomic_inc_return(&blk_probes_ref) == 1)
  		blk_register_tracepoints();
@@ -1481,7 +1478,7 @@ static int blk_trace_remove_queue(struct request_queue *q)
  static int blk_trace_setup_queue(struct request_queue *q,
  				 struct block_device *bdev)
  {
-	struct blk_trace *old_bt, *bt = NULL;
+	struct blk_trace *bt = NULL;
  	int ret = -ENOMEM;
  
  	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
@@ -1497,12 +1494,9 @@ static int blk_trace_setup_queue(struct request_queue *q,
  
  	blk_trace_setup_lba(bt, bdev);
  
-	old_bt = xchg(&q->blk_trace, bt);
-	if (old_bt != NULL) {
-		(void)xchg(&q->blk_trace, old_bt);
-		ret = -EBUSY;
+	ret = -EBUSY;
+	if (cmpxchg(&q->blk_trace, NULL, bt))
  		goto free_bt;
-	}
  
  	if (atomic_inc_return(&blk_probes_ref) == 1)
  		blk_register_tracepoints();
-- 
2.1.4

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Jeff Moyer]

Davidlohr Bueso <dave@stgolabs.net> writes:

> This is really about simplifying the double xchg patterns into
> a single cmpxchg, with the same logic. Other than the immediate
> cleanup, there are some subtleties this change deals with:
>
> (i) While the load of the old bt is fully ordered wrt everything,
> ie:
>
>       old_bt = xchg(&q->blk_trace, bt);             [barrier]
>       if (old_bt)
> 	     (void) xchg(&q->blk_trace, old_bt);    [barrier]
>
> blk_trace could still be changed between the xchg and the old_bt
> load. Note that this description is merely theoretical and afaict
> very small, but doing everything in a single context with cmpxchg
> closes this potential race.
>
> (ii) Ordering guarantees are obviously kept with cmpxchg.

Hi David,

The patch itself looks ok, but it doesn't seem to apply to a recent
kernel tree.  It appears as though it is white-space damaged.  Would you
mind re-sending it?

Thanks!
Jeff

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Davidlohr Bueso]

>The patch itself looks ok, but it doesn't seem to apply to a recent
>kernel tree.  It appears as though it is white-space damaged.  Would you
>mind re-sending it?

Hmm sorry about that. I thought I had based it against that day's tip/master.
Anyway, here is v3 which is against tip/master as of 338e29ba93639138fafb9fb5ba946fd99a512aae.

Thanks,
Davidlohr

8<--------------------------------------------------------------------------------
From: Davidlohr Bueso <dave@stgolabs.net>
Date: Thu, 29 Oct 2015 12:13:24 -0700
Subject: [PATCH -tip v3] blktrace: re-write setting q->blk_trace

This is really about simplifying the double xchg patterns into
a single cmpxchg, with the same logic. Other than the immediate
cleanup, there are some subtleties this change deals with:

(i) While the load of the old bt is fully ordered wrt everything,
ie:

        old_bt = xchg(&q->blk_trace, bt);             [barrier]
        if (old_bt)
	     (void) xchg(&q->blk_trace, old_bt);    [barrier]

blk_trace could still be changed between the xchg and the old_bt
load. Note that this description is merely theoretical and afaict
very small, but doing everything in a single context with cmpxchg
closes this potential race.

(ii) Ordering guarantees are obviously kept with cmpxchg.

(iii) Gets rid of the hacky-by-nature (void)xchg pattern.

Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
v3: rebased ontop of today's -tip.
     minor changelog addition.

  kernel/trace/blktrace.c | 16 +++++-----------
  1 file changed, 5 insertions(+), 11 deletions(-)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index 90e72a0..e3a2618 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -437,7 +437,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
		       struct block_device *bdev,
		       struct blk_user_trace_setup *buts)
  {
-	struct blk_trace *old_bt, *bt = NULL;
+	struct blk_trace *bt = NULL;
	struct dentry *dir = NULL;
	int ret;

@@ -519,11 +519,8 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
	bt->trace_state = Blktrace_setup;

	ret = -EBUSY;
-	old_bt = xchg(&q->blk_trace, bt);
-	if (old_bt) {
-		(void) xchg(&q->blk_trace, old_bt);
+	if (cmpxchg(&q->blk_trace, NULL, bt))
		goto err;
-	}

	if (atomic_inc_return(&blk_probes_ref) == 1)
		blk_register_tracepoints();
@@ -1481,7 +1478,7 @@ static int blk_trace_remove_queue(struct request_queue *q)
  static int blk_trace_setup_queue(struct request_queue *q,
				 struct block_device *bdev)
  {
-	struct blk_trace *old_bt, *bt = NULL;
+	struct blk_trace *bt = NULL;
	int ret = -ENOMEM;

	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
@@ -1497,12 +1494,9 @@ static int blk_trace_setup_queue(struct request_queue *q,

	blk_trace_setup_lba(bt, bdev);

-	old_bt = xchg(&q->blk_trace, bt);
-	if (old_bt != NULL) {
-		(void)xchg(&q->blk_trace, old_bt);
-		ret = -EBUSY;
+	ret = -EBUSY;
+	if (cmpxchg(&q->blk_trace, NULL, bt))
		goto free_bt;
-	}

	if (atomic_inc_return(&blk_probes_ref) == 1)
		blk_register_tracepoints();
--
2.1.4

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Jeff Moyer]

Davidlohr Bueso <dave@stgolabs.net> writes:

>>The patch itself looks ok, but it doesn't seem to apply to a recent
>>kernel tree.  It appears as though it is white-space damaged.  Would you
>>mind re-sending it?
>
> Hmm sorry about that. I thought I had based it against that day's tip/master.
> Anyway, here is v3 which is against tip/master as of 338e29ba93639138fafb9fb5ba946fd99a512aae.

Thanks.

Reviewed-by: Jeff Moyer <jmoyer@redhat.com>


> 8<--------------------------------------------------------------------------------
> From: Davidlohr Bueso <dave@stgolabs.net>
> Date: Thu, 29 Oct 2015 12:13:24 -0700
> Subject: [PATCH -tip v3] blktrace: re-write setting q->blk_trace
>
> This is really about simplifying the double xchg patterns into
> a single cmpxchg, with the same logic. Other than the immediate
> cleanup, there are some subtleties this change deals with:
>
> (i) While the load of the old bt is fully ordered wrt everything,
> ie:
>
>        old_bt = xchg(&q->blk_trace, bt);             [barrier]
>        if (old_bt)
> 	     (void) xchg(&q->blk_trace, old_bt);    [barrier]
>
> blk_trace could still be changed between the xchg and the old_bt
> load. Note that this description is merely theoretical and afaict
> very small, but doing everything in a single context with cmpxchg
> closes this potential race.
>
> (ii) Ordering guarantees are obviously kept with cmpxchg.
>
> (iii) Gets rid of the hacky-by-nature (void)xchg pattern.
>
> Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
> ---
> v3: rebased ontop of today's -tip.
>     minor changelog addition.
>
>  kernel/trace/blktrace.c | 16 +++++-----------
>  1 file changed, 5 insertions(+), 11 deletions(-)
>
> diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
> index 90e72a0..e3a2618 100644
> --- a/kernel/trace/blktrace.c
> +++ b/kernel/trace/blktrace.c
> @@ -437,7 +437,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
> 		       struct block_device *bdev,
> 		       struct blk_user_trace_setup *buts)
>  {
> -	struct blk_trace *old_bt, *bt = NULL;
> +	struct blk_trace *bt = NULL;
> 	struct dentry *dir = NULL;
> 	int ret;
>
> @@ -519,11 +519,8 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
> 	bt->trace_state = Blktrace_setup;
>
> 	ret = -EBUSY;
> -	old_bt = xchg(&q->blk_trace, bt);
> -	if (old_bt) {
> -		(void) xchg(&q->blk_trace, old_bt);
> +	if (cmpxchg(&q->blk_trace, NULL, bt))
> 		goto err;
> -	}
>
> 	if (atomic_inc_return(&blk_probes_ref) == 1)
> 		blk_register_tracepoints();
> @@ -1481,7 +1478,7 @@ static int blk_trace_remove_queue(struct request_queue *q)
>  static int blk_trace_setup_queue(struct request_queue *q,
> 				 struct block_device *bdev)
>  {
> -	struct blk_trace *old_bt, *bt = NULL;
> +	struct blk_trace *bt = NULL;
> 	int ret = -ENOMEM;
>
> 	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
> @@ -1497,12 +1494,9 @@ static int blk_trace_setup_queue(struct request_queue *q,
>
> 	blk_trace_setup_lba(bt, bdev);
>
> -	old_bt = xchg(&q->blk_trace, bt);
> -	if (old_bt != NULL) {
> -		(void)xchg(&q->blk_trace, old_bt);
> -		ret = -EBUSY;
> +	ret = -EBUSY;
> +	if (cmpxchg(&q->blk_trace, NULL, bt))
> 		goto free_bt;
> -	}
>
> 	if (atomic_inc_return(&blk_probes_ref) == 1)
> 		blk_register_tracepoints();
> --
> 2.1.4

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Jens Axboe]

On 10/30/2015 04:19 AM, Davidlohr Bueso wrote:
>
>> The patch itself looks ok, but it doesn't seem to apply to a recent
>> kernel tree.  It appears as though it is white-space damaged.  Would you
>> mind re-sending it?
>
> Hmm sorry about that. I thought I had based it against that day's
> tip/master.
> Anyway, here is v3 which is against tip/master as of
> 338e29ba93639138fafb9fb5ba946fd99a512aae.

Thanks, this is definitely cleaner. Your patch is still pretty mangled, 
though. I had to hand apply it. Please check the result:

http://git.kernel.dk/cgit/linux-block/commit/?h=for-4.4/core&id=cdea01b2bf98affb7e9c44530108a4a28535eee8

-- 
Jens Axboe

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Jeff Moyer]

Jens Axboe <axboe@kernel.dk> writes:

> On 10/30/2015 04:19 AM, Davidlohr Bueso wrote:
>>
>>> The patch itself looks ok, but it doesn't seem to apply to a recent
>>> kernel tree.  It appears as though it is white-space damaged.  Would you
>>> mind re-sending it?
>>
>> Hmm sorry about that. I thought I had based it against that day's
>> tip/master.
>> Anyway, here is v3 which is against tip/master as of
>> 338e29ba93639138fafb9fb5ba946fd99a512aae.
>
> Thanks, this is definitely cleaner. Your patch is still pretty
> mangled, though. I had to hand apply it. Please check the result:

Heh, and here I just assumed the problem was on my end.

> http://git.kernel.dk/cgit/linux-block/commit/?h=for-4.4/core&id=cdea01b2bf98affb7e9c44530108a4a28535eee8

That looks fine, except you're missing the 'R' in Reviewed-by.

Cheers,
Jeff

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Jens Axboe]

On 10/30/2015 05:46 AM, Jeff Moyer wrote:
> Jens Axboe <axboe@kernel.dk> writes:
>
>> On 10/30/2015 04:19 AM, Davidlohr Bueso wrote:
>>>
>>>> The patch itself looks ok, but it doesn't seem to apply to a recent
>>>> kernel tree.  It appears as though it is white-space damaged.  Would you
>>>> mind re-sending it?
>>>
>>> Hmm sorry about that. I thought I had based it against that day's
>>> tip/master.
>>> Anyway, here is v3 which is against tip/master as of
>>> 338e29ba93639138fafb9fb5ba946fd99a512aae.
>>
>> Thanks, this is definitely cleaner. Your patch is still pretty
>> mangled, though. I had to hand apply it. Please check the result:
>
> Heh, and here I just assumed the problem was on my end.
>
>> http://git.kernel.dk/cgit/linux-block/commit/?h=for-4.4/core&id=cdea01b2bf98affb7e9c44530108a4a28535eee8
>
> That looks fine, except you're missing the 'R' in Reviewed-by.

e-viewed. Electronically viewed by jeff?

-- 
Jens Axboe

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Jeff Moyer]

Jens Axboe <axboe@kernel.dk> writes:

> On 10/30/2015 05:46 AM, Jeff Moyer wrote:
>> Jens Axboe <axboe@kernel.dk> writes:
>>
>>> On 10/30/2015 04:19 AM, Davidlohr Bueso wrote:
>>>>
>>>>> The patch itself looks ok, but it doesn't seem to apply to a recent
>>>>> kernel tree.  It appears as though it is white-space damaged.  Would you
>>>>> mind re-sending it?
>>>>
>>>> Hmm sorry about that. I thought I had based it against that day's
>>>> tip/master.
>>>> Anyway, here is v3 which is against tip/master as of
>>>> 338e29ba93639138fafb9fb5ba946fd99a512aae.
>>>
>>> Thanks, this is definitely cleaner. Your patch is still pretty
>>> mangled, though. I had to hand apply it. Please check the result:
>>
>> Heh, and here I just assumed the problem was on my end.
>>
>>> http://git.kernel.dk/cgit/linux-block/commit/?h=for-4.4/core&id=cdea01b2bf98affb7e9c44530108a4a28535eee8
>>
>> That looks fine, except you're missing the 'R' in Reviewed-by.
>
> e-viewed. Electronically viewed by jeff?

Heh.  Well, that's not wrong!  ;-)

Re: [PATCH v2] blktrace: re-write setting q->blk_trace

[Davidlohr Bueso]

On Fri, 30 Oct 2015, Jens Axboe wrote:

>Thanks, this is definitely cleaner. Your patch is still pretty 
>mangled, though. I had to hand apply it. Please check the result:
>
>http://git.kernel.dk/cgit/linux-block/commit/?h=for-4.4/core&id=cdea01b2bf98affb7e9c44530108a4a28535eee8

Hmm I'm not sure what I'm doing wrong with the patch. This is the way
I always send them through emacs... Sorry for the trouble, the commit
above looks good (except the already mentioned eviewed tag :).

Thanks,
Davidlohr