[PATCH V2 0/2] block: remove unnecessary RESTART

[PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

Hi Jens,

The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
by itself, and not necessary to waste CPU to do the expensive RESTART.
And Roman Pen reported that this RESTART cuts half of IOPS in his case.

The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.


Ming Lei (2):
  blk-mq: don't handle TAG_SHARED in restart
  blk-mq: don't restart queue when .get_budget returns BLK_STS_RESOURCE

 block/blk-mq-sched.c | 123 ++++-----------------------------------------
 block/blk-mq-sched.h |   2 +-
 block/blk-mq.c       |   8 +---
 3 files changed, 27 insertions(+), 106 deletions(-)

-- 
2.9.5

[PATCH V2 1/2] blk-mq: don't handle TAG_SHARED in restart

[Ming Lei]

Now restart is used in the following cases, and TAG_SHARED is for
SCSI only.

1) .get_budget() returns BLK_STS_RESOURCE
- if resource in target/host level isn't satisfied, this SCSI device
will be added in shost->starved_list, and the whole queue will be rerun
(via SCSI's built-in RESTART) in scsi_end_request() after any request
initiated from this host/targe is completed. Forget to mention, host level
resource can't be an issue for blk-mq at all.

- the same is true if resource in the queue level isn't satisfied.

- if there isn't outstanding request on this queue, then SCSI's RESTART
can't work(blk-mq's can't work too), and the queue will be run after
SCSI_QUEUE_DELAY, and finally all starved sdevs will be handled by SCSI's
RESTART when this request is finished

2) scsi_dispatch_cmd() returns BLK_STS_RESOURCE
- if there isn't onprogressing request on this queue, the queue
will be run after SCSI_QUEUE_DELAY

- otherwise, SCSI's RESTART covers the rerun.

3) blk_mq_get_driver_tag() failed
- BLK_MQ_S_TAG_WAITING covers the cross-queue RESTART for driver
allocation.

In one word, SCSI's built-in RESTART is enough to cover the queue
rerun, and we don't need to pay special attention to TAG_SHARED wrt. restart.

In my test on scsi_debug(8 luns), this patch improves IOPS by 20% ~ 30% when
running I/O on these 8 luns concurrently.

Aslo Roman Pen reported the current RESTART is very expensive especialy
when there are lots of LUNs attached in one host, such as in his
test, RESTART causes half of IOPS be cut.

Fixes: https://marc.info/?l=linux-kernel&m=150832216727524&w=2
Fixes: 6d8c6c0f97ad ("blk-mq: Restart a single queue if tag sets are shared")
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 block/blk-mq-sched.c | 78 +++-------------------------------------------------
 1 file changed, 4 insertions(+), 74 deletions(-)

diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index df8581bb0a37..daab27feb653 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c
@@ -68,25 +68,17 @@ static void blk_mq_sched_mark_restart_hctx(struct blk_mq_hw_ctx *hctx)
 		set_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
 }
 
-static bool blk_mq_sched_restart_hctx(struct blk_mq_hw_ctx *hctx)
+void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx)
 {
 	if (!test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
-		return false;
-
-	if (hctx->flags & BLK_MQ_F_TAG_SHARED) {
-		struct request_queue *q = hctx->queue;
+		return;
 
-		if (test_and_clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
-			atomic_dec(&q->shared_hctx_restart);
-	} else
-		clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
+	clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
 
 	if (blk_mq_hctx_has_pending(hctx)) {
 		blk_mq_run_hw_queue(hctx, true);
-		return true;
+		return;
 	}
-
-	return false;
 }
 
 /* return true if hctx need to run again */
@@ -385,68 +377,6 @@ static bool blk_mq_sched_bypass_insert(struct blk_mq_hw_ctx *hctx,
 	return true;
 }
 
-/**
- * list_for_each_entry_rcu_rr - iterate in a round-robin fashion over rcu list
- * @pos:    loop cursor.
- * @skip:   the list element that will not be examined. Iteration starts at
- *          @skip->next.
- * @head:   head of the list to examine. This list must have at least one
- *          element, namely @skip.
- * @member: name of the list_head structure within typeof(*pos).
- */
-#define list_for_each_entry_rcu_rr(pos, skip, head, member)		\
-	for ((pos) = (skip);						\
-	     (pos = (pos)->member.next != (head) ? list_entry_rcu(	\
-			(pos)->member.next, typeof(*pos), member) :	\
-	      list_entry_rcu((pos)->member.next->next, typeof(*pos), member)), \
-	     (pos) != (skip); )
-
-/*
- * Called after a driver tag has been freed to check whether a hctx needs to
- * be restarted. Restarts @hctx if its tag set is not shared. Restarts hardware
- * queues in a round-robin fashion if the tag set of @hctx is shared with other
- * hardware queues.
- */
-void blk_mq_sched_restart(struct blk_mq_hw_ctx *const hctx)
-{
-	struct blk_mq_tags *const tags = hctx->tags;
-	struct blk_mq_tag_set *const set = hctx->queue->tag_set;
-	struct request_queue *const queue = hctx->queue, *q;
-	struct blk_mq_hw_ctx *hctx2;
-	unsigned int i, j;
-
-	if (set->flags & BLK_MQ_F_TAG_SHARED) {
-		/*
-		 * If this is 0, then we know that no hardware queues
-		 * have RESTART marked. We're done.
-		 */
-		if (!atomic_read(&queue->shared_hctx_restart))
-			return;
-
-		rcu_read_lock();
-		list_for_each_entry_rcu_rr(q, queue, &set->tag_list,
-					   tag_set_list) {
-			queue_for_each_hw_ctx(q, hctx2, i)
-				if (hctx2->tags == tags &&
-				    blk_mq_sched_restart_hctx(hctx2))
-					goto done;
-		}
-		j = hctx->queue_num + 1;
-		for (i = 0; i < queue->nr_hw_queues; i++, j++) {
-			if (j == queue->nr_hw_queues)
-				j = 0;
-			hctx2 = queue->queue_hw_ctx[j];
-			if (hctx2->tags == tags &&
-			    blk_mq_sched_restart_hctx(hctx2))
-				break;
-		}
-done:
-		rcu_read_unlock();
-	} else {
-		blk_mq_sched_restart_hctx(hctx);
-	}
-}
-
 /*
  * Add flush/fua to the queue. If we fail getting a driver tag, then
  * punt to the requeue list. Requeue will re-invoke us from a context
-- 
2.9.5

[PATCH V2 2/2] blk-mq: don't restart queue when .get_budget returns BLK_STS_RESOURCE

[Ming Lei]

SCSI restarts its queue in scsi_end_request() automatically, so we don't
need to handle this case in blk-mq.

Especailly any request won't be dequeued in this case, we needn't to
worry about IO hang caused by restart vs. dispatch.

Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 block/blk-mq-sched.c | 45 ++++++++++++++++++++-------------------------
 block/blk-mq-sched.h |  2 +-
 block/blk-mq.c       |  8 ++------
 3 files changed, 23 insertions(+), 32 deletions(-)

diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index daab27feb653..7775f6b12fa9 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c
@@ -81,8 +81,12 @@ void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx)
 	}
 }
 
-/* return true if hctx need to run again */
-static bool blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)
+/*
+ * Only SCSI implements .get_budget and .put_budget, and SCSI restarts
+ * its queue by itself in its completion handler, so we don't need to
+ * restart queue if .get_budget() returns BLK_STS_NO_RESOURCE.
+ */
+static void blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)
 {
 	struct request_queue *q = hctx->queue;
 	struct elevator_queue *e = q->elevator;
@@ -98,7 +102,7 @@ static bool blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)
 
 		ret = blk_mq_get_dispatch_budget(hctx);
 		if (ret == BLK_STS_RESOURCE)
-			return true;
+			break;
 
 		rq = e->type->ops.mq.dispatch_request(hctx);
 		if (!rq) {
@@ -116,8 +120,6 @@ static bool blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)
 		 */
 		list_add(&rq->queuelist, &rq_list);
 	} while (blk_mq_dispatch_rq_list(q, &rq_list, true));
-
-	return false;
 }
 
 static struct blk_mq_ctx *blk_mq_next_ctx(struct blk_mq_hw_ctx *hctx,
@@ -131,8 +133,12 @@ static struct blk_mq_ctx *blk_mq_next_ctx(struct blk_mq_hw_ctx *hctx,
 	return hctx->ctxs[idx];
 }
 
-/* return true if hctx need to run again */
-static bool blk_mq_do_dispatch_ctx(struct blk_mq_hw_ctx *hctx)
+/*
+ * Only SCSI implements .get_budget and .put_budget, and SCSI restarts
+ * its queue by itself in its completion handler, so we don't need to
+ * restart queue if .get_budget() returns BLK_STS_NO_RESOURCE.
+ */
+static void blk_mq_do_dispatch_ctx(struct blk_mq_hw_ctx *hctx)
 {
 	struct request_queue *q = hctx->queue;
 	LIST_HEAD(rq_list);
@@ -147,7 +153,7 @@ static bool blk_mq_do_dispatch_ctx(struct blk_mq_hw_ctx *hctx)
 
 		ret = blk_mq_get_dispatch_budget(hctx);
 		if (ret == BLK_STS_RESOURCE)
-			return true;
+			break;
 
 		rq = blk_mq_dequeue_from_ctx(hctx, ctx);
 		if (!rq) {
@@ -171,22 +177,19 @@ static bool blk_mq_do_dispatch_ctx(struct blk_mq_hw_ctx *hctx)
 	} while (blk_mq_dispatch_rq_list(q, &rq_list, true));
 
 	WRITE_ONCE(hctx->dispatch_from, ctx);
-
-	return false;
 }
 
 /* return true if hw queue need to be run again */
-bool blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
+void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
 {
 	struct request_queue *q = hctx->queue;
 	struct elevator_queue *e = q->elevator;
 	const bool has_sched_dispatch = e && e->type->ops.mq.dispatch_request;
 	LIST_HEAD(rq_list);
-	bool run_queue = false;
 
 	/* RCU or SRCU read lock is needed before checking quiesced flag */
 	if (unlikely(blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)))
-		return false;
+		return;
 
 	hctx->run++;
 
@@ -218,12 +221,12 @@ bool blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
 		blk_mq_sched_mark_restart_hctx(hctx);
 		if (blk_mq_dispatch_rq_list(q, &rq_list, false)) {
 			if (has_sched_dispatch)
-				run_queue = blk_mq_do_dispatch_sched(hctx);
+				blk_mq_do_dispatch_sched(hctx);
 			else
-				run_queue = blk_mq_do_dispatch_ctx(hctx);
+				blk_mq_do_dispatch_ctx(hctx);
 		}
 	} else if (has_sched_dispatch) {
-		run_queue = blk_mq_do_dispatch_sched(hctx);
+		blk_mq_do_dispatch_sched(hctx);
 	} else if (q->mq_ops->get_budget) {
 		/*
 		 * If we need to get budget before queuing request, we
@@ -233,19 +236,11 @@ bool blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
 		 * TODO: get more budgets, and dequeue more requests in
 		 * one time.
 		 */
-		run_queue = blk_mq_do_dispatch_ctx(hctx);
+		blk_mq_do_dispatch_ctx(hctx);
 	} else {
 		blk_mq_flush_busy_ctxs(hctx, &rq_list);
 		blk_mq_dispatch_rq_list(q, &rq_list, false);
 	}
-
-	if (run_queue && !blk_mq_sched_needs_restart(hctx) &&
-			!test_bit(BLK_MQ_S_TAG_WAITING, &hctx->state)) {
-		blk_mq_sched_mark_restart_hctx(hctx);
-		return true;
-	}
-
-	return false;
 }
 
 bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio,
diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h
index 2434061cc5b7..9267d0b7c197 100644
--- a/block/blk-mq-sched.h
+++ b/block/blk-mq-sched.h
@@ -22,7 +22,7 @@ void blk_mq_sched_insert_requests(struct request_queue *q,
 				  struct blk_mq_ctx *ctx,
 				  struct list_head *list, bool run_queue_async);
 
-bool blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx);
+void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx);
 
 int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e);
 void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e);
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 097ca3ece716..e4d2490f4e7e 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1226,7 +1226,6 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list,
 static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 {
 	int srcu_idx;
-	bool run_queue;
 
 	/*
 	 * We should be running this queue from one of the CPUs that
@@ -1243,18 +1242,15 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 
 	if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
 		rcu_read_lock();
-		run_queue = blk_mq_sched_dispatch_requests(hctx);
+		blk_mq_sched_dispatch_requests(hctx);
 		rcu_read_unlock();
 	} else {
 		might_sleep();
 
 		srcu_idx = srcu_read_lock(hctx->queue_rq_srcu);
-		run_queue = blk_mq_sched_dispatch_requests(hctx);
+		blk_mq_sched_dispatch_requests(hctx);
 		srcu_read_unlock(hctx->queue_rq_srcu, srcu_idx);
 	}
-
-	if (run_queue)
-		blk_mq_run_hw_queue(hctx, true);
 }
 
 /*
-- 
2.9.5

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTEwLTI3IGF0IDEyOjQzICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gVGhl
IDFzdCBwYXRjaCByZW1vdmVzIHRoZSBSRVNUQVJUIGZvciBUQUctU0hBUkVEIGJlY2F1c2UgU0NT
SSBoYW5kbGVzIGl0DQo+IGJ5IGl0c2VsZiwgYW5kIG5vdCBuZWNlc3NhcnkgdG8gd2FzdGUgQ1BV
IHRvIGRvIHRoZSBleHBlbnNpdmUgUkVTVEFSVC4NCj4gQW5kIFJvbWFuIFBlbiByZXBvcnRlZCB0
aGF0IHRoaXMgUkVTVEFSVCBjdXRzIGhhbGYgb2YgSU9QUyBpbiBoaXMgY2FzZS4NCj4gDQo+IFRo
ZSAybmQgcGF0Y2ggcmVtb3ZlcyB0aGUgUkVTVEFSVCB3aGVuIC5nZXRfYnVkZ2V0IHJldHVybnMg
QkxLX1NUU19SRVNPVVJDRSwNCj4gYW5kIHRoaXMgUkVTVEFSVCBpcyBoYW5kbGVkIGJ5IFNDU0kn
cyBSRVNUQVJUKHNjc2lfZW5kX3JlcXVlc3QoKSkgdG9vLg0KDQpIZWxsbyBNaW5nLA0KDQpUaGVy
ZSBhcmUgbW9yZSBibG9jayBkcml2ZXJzIHRoYW4gdGhlIFNDU0kgY29yZSB0aGF0IHNoYXJlIHRh
Z3MuIElmIHRoZQ0KcmVzdGFydCBtZWNoYW5pc20gaXMgcmVtb3ZlZCBmcm9tIHRoZSBibGstbXEg
Y29yZSwgZG9lcyB0aGF0IG1lYW4gdGhhdCBhbGwNCmJsb2NrIGRyaXZlcnMgdGhhdCBzaGFyZSB0
YWdzIHdpbGwgaGF2ZSB0byBmb2xsb3cgdGhlIGV4YW1wbGUgb2YgdGhlIFNDU0kNCmNvcmUgYW5k
IGltcGxlbWVudCBhIHJlc3RhcnQgbWVjaGFuaXNtIHRoZW1zZWx2ZXM/IEFzIGZhciBhcyBJIGtu
b3cgdGhlcmUNCmlzIGEgc3Ryb25nIHByZWZlcmVuY2UgaW4gdGhlIExpbnV4IGNvbW11bml0eSB0
byBpbXBsZW1lbnQgY29tbW9uIG1lY2hhbmlzbXMNCmluIHRoZSAoYmxvY2sgbGF5ZXIpIGNvcmUg
aW5zdGVhZCBvZiBpbiBkcml2ZXJzLiBJdCBzZWVtcyB0byBtZSBsaWtlIHlvdSBhcmUNCnByb3Bv
c2luZyB0aGUgb3Bwb3NpdGUsIG5hbWVseSByZW1vdmluZyBhIGdlbmVyYWwgbWVjaGFuaXNtIGZy
b20gdGhlIChibG9jaw0KbGF5ZXIpIGNvcmUgYW5kIG1vdmluZyBpdCBpbnRvIGEgZHJpdmVyLCBu
YW1lbHkgdGhlIFNDU0kgY29yZT8NCg0KQmFydC4=

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

Hello Bart,

On Fri, Oct 27, 2017 at 04:53:18AM +0000, Bart Van Assche wrote:
> On Fri, 2017-10-27 at 12:43 +0800, Ming Lei wrote:
> > The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> > by itself, and not necessary to waste CPU to do the expensive RESTART.
> > And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> > 
> > The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> > and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
> 
> Hello Ming,
> 
> There are more block drivers than the SCSI core that share tags. If the

Could you share us what the other in-tree driver which share tags is?

If there are really more, and when all can share similar RESTART mechanism, I
think that is the time for considering the common RESTART.

> restart mechanism is removed from the blk-mq core, does that mean that all
> block drivers that share tags will have to follow the example of the SCSI
> core and implement a restart mechanism themselves? As far as I know there

If there are such drivers, there should have been their own restart mechanism
which works for long time before blk-mq comes, and more importantly each driver
has much more knowledge than generic block layer to handle the restart,
such as SCSI's restart, that means driver's implementation may be more efficient.

Also the RESTART for TAG-SHARED may never function as expected wrt. SCSI.

And more importantly the block's RESTART for TAG-SHARED has caused big performance
issue for people.

IMO, the above two reasons are enough to remove the current RESTART for shared-tag.

> is a strong preference in the Linux community to implement common mechanisms
> in the (block layer) core instead of in drivers. It seems to me like you are
> proposing the opposite, namely removing a general mechanism from the (block
> layer) core and moving it into a driver, namely the SCSI core?

Actually SCSI's RESTART is very thin, and it is just the per-host starved_list.


Thanks,
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Roman Penyaev]

Hi Ming,

On Fri, Oct 27, 2017 at 7:38 AM, Ming Lei <ming.lei@redhat.com> wrote:
> Hello Bart,
>
> On Fri, Oct 27, 2017 at 04:53:18AM +0000, Bart Van Assche wrote:
>> On Fri, 2017-10-27 at 12:43 +0800, Ming Lei wrote:
>> > The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
>> > by itself, and not necessary to waste CPU to do the expensive RESTART.
>> > And Roman Pen reported that this RESTART cuts half of IOPS in his case.
>> >
>> > The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
>> > and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
>>
>> Hello Ming,
>>
>> There are more block drivers than the SCSI core that share tags. If the
>
> Could you share us what the other in-tree driver which share tags is?
>
> If there are really more, and when all can share similar RESTART mechanism, I
> think that is the time for considering the common RESTART.
>
>> restart mechanism is removed from the blk-mq core, does that mean that all
>> block drivers that share tags will have to follow the example of the SCSI
>> core and implement a restart mechanism themselves? As far as I know there
>
> If there are such drivers, there should have been their own restart mechanism
> which works for long time before blk-mq comes, and more importantly each driver
> has much more knowledge than generic block layer to handle the restart,
> such as SCSI's restart, that means driver's implementation may be more efficient.

Personally I would be happier if block layer would do restarts for me.  My eyes
are bleeding when I see chunks of my own code which do this juggling with hctx,
enqueuing or dequeuing from a percpu lists, etc inside IBNBD driver.

> Also the RESTART for TAG-SHARED may never function as expected wrt. SCSI.

E.g. new BLK_MQ_F_NO_RESTARTS flag can help to disable restarts when they
are not needed.

> And more importantly the block's RESTART for TAG-SHARED has caused big performance
> issue for people.

That's just a bug in code, not a in issue with restarts, which can be fixed
if we put hctx which are needed to be restarted in percpu lists and avoid
long loops and contentions.

--
Roman

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

Hi Roman,

On Fri, Oct 27, 2017 at 07:55:33PM +0200, Roman Penyaev wrote:
> Hi Ming,
> 
> On Fri, Oct 27, 2017 at 7:38 AM, Ming Lei <ming.lei@redhat.com> wrote:
> > Hello Bart,
> >
> > On Fri, Oct 27, 2017 at 04:53:18AM +0000, Bart Van Assche wrote:
> >> On Fri, 2017-10-27 at 12:43 +0800, Ming Lei wrote:
> >> > The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> >> > by itself, and not necessary to waste CPU to do the expensive RESTART.
> >> > And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> >> >
> >> > The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> >> > and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
> >>
> >> Hello Ming,
> >>
> >> There are more block drivers than the SCSI core that share tags. If the
> >
> > Could you share us what the other in-tree driver which share tags is?
> >
> > If there are really more, and when all can share similar RESTART mechanism, I
> > think that is the time for considering the common RESTART.
> >
> >> restart mechanism is removed from the blk-mq core, does that mean that all
> >> block drivers that share tags will have to follow the example of the SCSI
> >> core and implement a restart mechanism themselves? As far as I know there
> >
> > If there are such drivers, there should have been their own restart mechanism
> > which works for long time before blk-mq comes, and more importantly each driver
> > has much more knowledge than generic block layer to handle the restart,
> > such as SCSI's restart, that means driver's implementation may be more efficient.
> 
> Personally I would be happier if block layer would do restarts for me.  My eyes
> are bleeding when I see chunks of my own code which do this juggling with hctx,
> enqueuing or dequeuing from a percpu lists, etc inside IBNBD driver.

The issue is that the current block's RESTART for TAG-SHARED isn't efficient, and
never functions as expected. And no one uses that at all actually.

Firstly we have handled cross-queue RESTART via BLK_MQ_S_TAG_WAITING already when
getting driver tag.

Secondly as you see, SCSI handles this kind of RESTART much more efficient and
simply via one linked list, I believe your IBNBD can do this in similar way,
right? If you can't figure out one easy way to do that, I am happy to
provide one patchset[1](which was done via sbitmap weeks ago) to do that
efficiently in block layer, but not the current approach, when you upstream your
IBNBD patchset.

https://github.com/ming1/linux/commits/blk_mq_improve_restart_V1-rc1

Finally you reported that the RESTART for TAG-SHARED has degraded your
IO performance by half, and that can't be accepted, IMO. I found the
similar performance issue in my test too.

> 
> > Also the RESTART for TAG-SHARED may never function as expected wrt. SCSI.
> 
> E.g. new BLK_MQ_F_NO_RESTARTS flag can help to disable restarts when they
> are not needed.

This flag is only needed iff there are more than one in-tree drivers which need
the RESTART for TAG-SHARED.

Also I have mentioned the current RESTART for TAG-SHARED has big performance issue.

> 
> > And more importantly the block's RESTART for TAG-SHARED has caused big performance
> > issue for people.
> 
> That's just a bug in code, not a in issue with restarts, which can be fixed
> if we put hctx which are needed to be restarted in percpu lists and avoid
> long loops and contentions.

Again no in-tree driver uses RESTART for TAG-SHARED now, so not necessary to fix
anything in the RESTART for TAG-SHARED.

If there is driver which need this kind of RESTART in future, one more efficient
implementation can be figured out at that time, but not now. As you see,
I already have one.

Thanks,
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTEwLTI3IGF0IDEzOjM4ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gT24g
RnJpLCBPY3QgMjcsIDIwMTcgYXQgMDQ6NTM6MThBTSArMDAwMCwgQmFydCBWYW4gQXNzY2hlIHdy
b3RlOg0KPiA+IE9uIEZyaSwgMjAxNy0xMC0yNyBhdCAxMjo0MyArMDgwMCwgTWluZyBMZWkgd3Jv
dGU6DQo+ID4gPiBUaGUgMXN0IHBhdGNoIHJlbW92ZXMgdGhlIFJFU1RBUlQgZm9yIFRBRy1TSEFS
RUQgYmVjYXVzZSBTQ1NJIGhhbmRsZXMgaXQNCj4gPiA+IGJ5IGl0c2VsZiwgYW5kIG5vdCBuZWNl
c3NhcnkgdG8gd2FzdGUgQ1BVIHRvIGRvIHRoZSBleHBlbnNpdmUgUkVTVEFSVC4NCj4gPiA+IEFu
ZCBSb21hbiBQZW4gcmVwb3J0ZWQgdGhhdCB0aGlzIFJFU1RBUlQgY3V0cyBoYWxmIG9mIElPUFMg
aW4gaGlzIGNhc2UuDQo+ID4gPiANCj4gPiA+IFRoZSAybmQgcGF0Y2ggcmVtb3ZlcyB0aGUgUkVT
VEFSVCB3aGVuIC5nZXRfYnVkZ2V0IHJldHVybnMgQkxLX1NUU19SRVNPVVJDRSwNCj4gPiA+IGFu
ZCB0aGlzIFJFU1RBUlQgaXMgaGFuZGxlZCBieSBTQ1NJJ3MgUkVTVEFSVChzY3NpX2VuZF9yZXF1
ZXN0KCkpIHRvby4NCj4gPiANCj4gPiBUaGVyZSBhcmUgbW9yZSBibG9jayBkcml2ZXJzIHRoYW4g
dGhlIFNDU0kgY29yZSB0aGF0IHNoYXJlIHRhZ3MuIElmIHRoZQ0KPiANCj4gQ291bGQgeW91IHNo
YXJlIHVzIHdoYXQgdGhlIG90aGVyIGluLXRyZWUgZHJpdmVyIHdoaWNoIHNoYXJlIHRhZ3MgaXM/
DQoNCkkgdGhpbmsgdGhlIGZvbGxvd2luZyBpbi10cmVlIGRyaXZlcnMgc3VwcG9ydCBzaGFyZWQg
dGFncyAoaW4gYWxwaGFiZXRpY2FsDQpvcmRlcik6DQoqIG51bGxfYmxrLiBTZWUgYWxzbyB0aGUg
c2hhcmVkX3RhZ3Mga2VybmVsIG1vZHVsZSBwYXJhbWV0ZXIuDQoqIG52bWUuIFNlZSBhbHNvIG52
bWVfYWxsb2NfbnMoKS4NCiogc2NzaS1tcS4NCg0KUGxlYXNlIG5vdGUgdGhhdCB0aGUgcXVldWVf
cnEoKSBmdW5jdGlvbiBvZiAqYWxsKiB0aGVzZSBkcml2ZXJzIGNhbiByZXR1cm4NCkJMS19TVFNf
UkVTT1VSQ0UuDQoNCj4gQWN0dWFsbHkgU0NTSSdzIFJFU1RBUlQgaXMgdmVyeSB0aGluLCBhbmQg
aXQgaXMganVzdCB0aGUgcGVyLWhvc3Qgc3RhcnZlZF9saXN0Lg0KDQpEbyB5b3Ugc2VlIGFueSBy
ZWFzb24gdGhhdCB3b3VsZCBwcmV2ZW50IHRvIG1vdmUgdGhhdCBtZWNoYW5pc20gZnJvbSBzY3Np
LW1xDQppbnRvIHRoZSBibGstbXEgY29yZT8NCg0KVGhhbmtzLA0KDQpCYXJ0Lg==

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTEwLTI3IGF0IDE5OjU1ICswMjAwLCBSb21hbiBQZW55YWV2IHdyb3RlOg0K
PiBUaGF0J3MganVzdCBhIGJ1ZyBpbiBjb2RlLCBub3QgYSBpbiBpc3N1ZSB3aXRoIHJlc3RhcnRz
LCB3aGljaCBjYW4gYmUgZml4ZWQNCj4gaWYgd2UgcHV0IGhjdHggd2hpY2ggYXJlIG5lZWRlZCB0
byBiZSByZXN0YXJ0ZWQgaW4gcGVyY3B1IGxpc3RzIGFuZCBhdm9pZA0KPiBsb25nIGxvb3BzIGFu
ZCBjb250ZW50aW9ucy4NCg0KSGVsbG8gUm9tYW4sDQoNCkhhdmUgeW91IG5vdGljZWQgdGhhdCBy
ZWNlbnRseSAuZ2V0X2J1ZGdldCgpIGFuZCAucHV0X2J1ZGdldCgpIGNhbGxiYWNrcyBoYXZlDQpi
ZWVuIGFkZGVkIGludG8gc3RydWN0IGJsa19tcV9vcHM/IElmIGJsb2NrIGRyaXZlcnMgdGhhdCBz
dXBwb3J0IHNoYXJlZCB0YWdzDQppbXBsZW1lbnQgdGhlc2UgY2FsbGJhY2sgZnVuY3Rpb25zIHBy
b3Blcmx5IHRoZW4gbWF5YmUgYnVzeSBjb25kaXRpb25zIGJlY29tZQ0KaW5mcmVxdWVudCBlbm91
Z2ggdG8gdXNlIGEgc2luZ2xlICJoY3R4IGJ1c3kiIGxpc3QgcGVyIHRhZyBzZXQgaW5zdGVhZCBv
Zg0KcGVyLWNwdSAiaGN0eCBidXN5IiBsaXN0cy4NCg0KQmFydC4=

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Mon, Oct 30, 2017 at 08:24:57PM +0000, Bart Van Assche wrote:
> On Fri, 2017-10-27 at 13:38 +0800, Ming Lei wrote:
> > On Fri, Oct 27, 2017 at 04:53:18AM +0000, Bart Van Assche wrote:
> > > On Fri, 2017-10-27 at 12:43 +0800, Ming Lei wrote:
> > > > The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> > > > by itself, and not necessary to waste CPU to do the expensive RESTART.
> > > > And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> > > > 
> > > > The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> > > > and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
> > > 
> > > There are more block drivers than the SCSI core that share tags. If the
> > 
> > Could you share us what the other in-tree driver which share tags is?
> 
> I think the following in-tree drivers support shared tags (in alphabetical
> order):
> * null_blk. See also the shared_tags kernel module parameter.
> * nvme. See also nvme_alloc_ns().

For both null_blk and nvme, we don't need to deal with cross-queue RESTART,
because BLK_MQ_S_TAG_WAITING has handled it already.

> * scsi-mq.
> 
> Please note that the queue_rq() function of *all* these drivers can return
> BLK_STS_RESOURCE.

Yes, but this patch doesn't remove the RESTART for the current hctx, and
only removed the part of handling cross-queue RESTART.

> 
> > Actually SCSI's RESTART is very thin, and it is just the per-host starved_list.
> 
> Do you see any reason that would prevent to move that mechanism from scsi-mq
> into the blk-mq core?

I don't see it is required from SCSI's view since this way has been run
for SCSI for long time, and we don't have this requirement for other
in-tree drivers meantime.

Also as I mentioned before, driver has more knowledge than generic block
layer for making it more efficiently.

Thanks,
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Jens Axboe]

On 10/26/2017 10:43 PM, Ming Lei wrote:
> Hi Jens,
> 
> The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> by itself, and not necessary to waste CPU to do the expensive RESTART.
> And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> 
> The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.

What base is this against?

-- 
Jens Axboe

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Tue, Oct 31, 2017 at 02:29:32PM -0600, Jens Axboe wrote:
> On 10/26/2017 10:43 PM, Ming Lei wrote:
> > Hi Jens,
> > 
> > The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> > by itself, and not necessary to waste CPU to do the expensive RESTART.
> > And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> > 
> > The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> > and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
> 
> What base is this against?

The for-next branch of your block tree:

--
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Jens Axboe]

> On Oct 31, 2017, at 7:46 PM, Ming Lei <ming.lei@redhat.com> wrote:
>=20
>> On Tue, Oct 31, 2017 at 02:29:32PM -0600, Jens Axboe wrote:
>>> On 10/26/2017 10:43 PM, Ming Lei wrote:
>>> Hi Jens,
>>>=20
>>> The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it=

>>> by itself, and not necessary to waste CPU to do the expensive RESTART.
>>> And Roman Pen reported that this RESTART cuts half of IOPS in his case.
>>>=20
>>> The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOU=
RCE,
>>> and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
>>=20
>> What base is this against?
>=20
> The for-next branch of your block tree:

=46rom when? Doesn=E2=80=99t apply at all today.=20

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Tue, Oct 31, 2017 at 07:53:03PM -0600, Jens Axboe wrote:
> 
> > On Oct 31, 2017, at 7:46 PM, Ming Lei <ming.lei@redhat.com> wrote:
> > 
> >> On Tue, Oct 31, 2017 at 02:29:32PM -0600, Jens Axboe wrote:
> >>> On 10/26/2017 10:43 PM, Ming Lei wrote:
> >>> Hi Jens,
> >>> 
> >>> The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> >>> by itself, and not necessary to waste CPU to do the expensive RESTART.
> >>> And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> >>> 
> >>> The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> >>> and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
> >> 
> >> What base is this against?
> > 
> > The for-next branch of your block tree:
> 
> From when? Doesn’t apply at all today. 

I just tried today's for-next(top commit is 'MAINTAINERS: Remove Rafael from Opal maintainers.'),
and the two patches can be applied cleanly.

I guess you may try to apply the two patches against for-4.15/block,
which doesn't include the patchset of '[PATCH V10 0/8] blk-mq-sched: improve sequential I/O'[1]:

	https://marc.info/?t=150797316600002&r=1&w=2

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gVHVlLCAyMDE3LTEwLTMxIGF0IDA5OjQ3ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gT24g
TW9uLCBPY3QgMzAsIDIwMTcgYXQgMDg6MjQ6NTdQTSArMDAwMCwgQmFydCBWYW4gQXNzY2hlIHdy
b3RlOg0KPiA+IE9uIEZyaSwgMjAxNy0xMC0yNyBhdCAxMzozOCArMDgwMCwgTWluZyBMZWkgd3Jv
dGU6DQo+ID4gPiBPbiBGcmksIE9jdCAyNywgMjAxNyBhdCAwNDo1MzoxOEFNICswMDAwLCBCYXJ0
IFZhbiBBc3NjaGUgd3JvdGU6DQo+ID4gPiA+IE9uIEZyaSwgMjAxNy0xMC0yNyBhdCAxMjo0MyAr
MDgwMCwgTWluZyBMZWkgd3JvdGU6DQo+ID4gPiA+ID4gVGhlIDFzdCBwYXRjaCByZW1vdmVzIHRo
ZSBSRVNUQVJUIGZvciBUQUctU0hBUkVEIGJlY2F1c2UgU0NTSSBoYW5kbGVzIGl0DQo+ID4gPiA+
ID4gYnkgaXRzZWxmLCBhbmQgbm90IG5lY2Vzc2FyeSB0byB3YXN0ZSBDUFUgdG8gZG8gdGhlIGV4
cGVuc2l2ZSBSRVNUQVJULg0KPiA+ID4gPiA+IEFuZCBSb21hbiBQZW4gcmVwb3J0ZWQgdGhhdCB0
aGlzIFJFU1RBUlQgY3V0cyBoYWxmIG9mIElPUFMgaW4gaGlzIGNhc2UuDQo+ID4gPiA+ID4gDQo+
ID4gPiA+ID4gVGhlIDJuZCBwYXRjaCByZW1vdmVzIHRoZSBSRVNUQVJUIHdoZW4gLmdldF9idWRn
ZXQgcmV0dXJucyBCTEtfU1RTX1JFU09VUkNFLA0KPiA+ID4gPiA+IGFuZCB0aGlzIFJFU1RBUlQg
aXMgaGFuZGxlZCBieSBTQ1NJJ3MgUkVTVEFSVChzY3NpX2VuZF9yZXF1ZXN0KCkpIHRvby4NCj4g
PiA+ID4gDQo+ID4gPiA+IFRoZXJlIGFyZSBtb3JlIGJsb2NrIGRyaXZlcnMgdGhhbiB0aGUgU0NT
SSBjb3JlIHRoYXQgc2hhcmUgdGFncy4gSWYgdGhlDQo+ID4gPiANCj4gPiA+IENvdWxkIHlvdSBz
aGFyZSB1cyB3aGF0IHRoZSBvdGhlciBpbi10cmVlIGRyaXZlciB3aGljaCBzaGFyZSB0YWdzIGlz
Pw0KPiA+IA0KPiA+IEkgdGhpbmsgdGhlIGZvbGxvd2luZyBpbi10cmVlIGRyaXZlcnMgc3VwcG9y
dCBzaGFyZWQgdGFncyAoaW4gYWxwaGFiZXRpY2FsDQo+ID4gb3JkZXIpOg0KPiA+ICogbnVsbF9i
bGsuIFNlZSBhbHNvIHRoZSBzaGFyZWRfdGFncyBrZXJuZWwgbW9kdWxlIHBhcmFtZXRlci4NCj4g
PiAqIG52bWUuIFNlZSBhbHNvIG52bWVfYWxsb2NfbnMoKS4NCj4gPiAqIHNjc2ktbXEuDQo+IA0K
PiBGb3IgYm90aCBudWxsX2JsayBhbmQgbnZtZSwgd2UgZG9uJ3QgbmVlZCB0byBkZWFsIHdpdGgg
Y3Jvc3MtcXVldWUgUkVTVEFSVCwNCj4gYmVjYXVzZSBCTEtfTVFfU19UQUdfV0FJVElORyBoYXMg
aGFuZGxlZCBpdCBhbHJlYWR5Lg0KDQpibGtfbXFfZGlzcGF0Y2hfd2FpdF9hZGQoKSByZXR1cm5z
IGltbWVkaWF0ZWx5IGlmIEJMS19NUV9TX1RBR19XQUlUSU5HIGlzDQphbHJlYWR5IHNldC4gQXJl
IHlvdSByZWFsbHkgc3VyZSB0aGF0IHJlbW92aW5nIHRoZSByZXN0YXJ0IG1lY2hhbmlzbSBkb2Vz
bid0DQpicmVhayB0aGUgTlZNZSBkcml2ZXIgaWYgdGhlcmUgYXJlIG11bHRpcGxlIG5hbWVzcGFj
ZXM/DQoNCkJhcnQu

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Wed, Nov 01, 2017 at 03:54:09AM +0000, Bart Van Assche wrote:
> On Tue, 2017-10-31 at 09:47 +0800, Ming Lei wrote:
> > On Mon, Oct 30, 2017 at 08:24:57PM +0000, Bart Van Assche wrote:
> > > On Fri, 2017-10-27 at 13:38 +0800, Ming Lei wrote:
> > > > On Fri, Oct 27, 2017 at 04:53:18AM +0000, Bart Van Assche wrote:
> > > > > On Fri, 2017-10-27 at 12:43 +0800, Ming Lei wrote:
> > > > > > The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> > > > > > by itself, and not necessary to waste CPU to do the expensive RESTART.
> > > > > > And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> > > > > > 
> > > > > > The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> > > > > > and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
> > > > > 
> > > > > There are more block drivers than the SCSI core that share tags. If the
> > > > 
> > > > Could you share us what the other in-tree driver which share tags is?
> > > 
> > > I think the following in-tree drivers support shared tags (in alphabetical
> > > order):
> > > * null_blk. See also the shared_tags kernel module parameter.
> > > * nvme. See also nvme_alloc_ns().
> > > * scsi-mq.
> > 
> > For both null_blk and nvme, we don't need to deal with cross-queue RESTART,
> > because BLK_MQ_S_TAG_WAITING has handled it already.
> 
> blk_mq_dispatch_wait_add() returns immediately if BLK_MQ_S_TAG_WAITING is
> already set. Are you really sure that removing the restart mechanism doesn't

If this flag is set, that means blk_mq_dispatch_wake() will be run when
driver tag is available, so the hw queue will be run at that time.

> break the NVMe driver if there are multiple namespaces?

Did your commit 6d8c6c0f97ad ("blk-mq: Restart a single queue if tag sets are shared")
suppose to fix NVMe?

Please take a look at the cover letter:

	https://marc.info/?t=149158911500010&r=1&w=2


-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Jens Axboe]

On 10/31/2017 08:03 PM, Ming Lei wrote:
> On Tue, Oct 31, 2017 at 07:53:03PM -0600, Jens Axboe wrote:
>>
>>> On Oct 31, 2017, at 7:46 PM, Ming Lei <ming.lei@redhat.com> wrote:
>>>
>>>> On Tue, Oct 31, 2017 at 02:29:32PM -0600, Jens Axboe wrote:
>>>>> On 10/26/2017 10:43 PM, Ming Lei wrote:
>>>>> Hi Jens,
>>>>>
>>>>> The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
>>>>> by itself, and not necessary to waste CPU to do the expensive RESTART.
>>>>> And Roman Pen reported that this RESTART cuts half of IOPS in his case.
>>>>>
>>>>> The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
>>>>> and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
>>>>
>>>> What base is this against?
>>>
>>> The for-next branch of your block tree:
>>
>> From when? Doesn’t apply at all today. 
> 
> I just tried today's for-next(top commit is 'MAINTAINERS: Remove Rafael from Opal maintainers.'),
> and the two patches can be applied cleanly.
> 
> I guess you may try to apply the two patches against for-4.15/block,
> which doesn't include the patchset of '[PATCH V10 0/8] blk-mq-sched: improve sequential I/O'[1]:
> 
> 	https://marc.info/?t=150797316600002&r=1&w=2

I'm the idiot, for some reason that patchset got applied to for-next
directly. That's not supposed to happen, for-next is always just
a merge point. Fixed that up, and applied these two patches as
well.

-- 
Jens Axboe

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gV2VkLCAyMDE3LTExLTAxIGF0IDEyOjA4ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gT24g
V2VkLCBOb3YgMDEsIDIwMTcgYXQgMDM6NTQ6MDlBTSArMDAwMCwgQmFydCBWYW4gQXNzY2hlIHdy
b3RlOg0KPiA+IE9uIFR1ZSwgMjAxNy0xMC0zMSBhdCAwOTo0NyArMDgwMCwgTWluZyBMZWkgd3Jv
dGU6DQo+ID4gPiBPbiBNb24sIE9jdCAzMCwgMjAxNyBhdCAwODoyNDo1N1BNICswMDAwLCBCYXJ0
IFZhbiBBc3NjaGUgd3JvdGU6DQo+ID4gPiA+IE9uIEZyaSwgMjAxNy0xMC0yNyBhdCAxMzozOCAr
MDgwMCwgTWluZyBMZWkgd3JvdGU6DQo+ID4gPiA+ID4gT24gRnJpLCBPY3QgMjcsIDIwMTcgYXQg
MDQ6NTM6MThBTSArMDAwMCwgQmFydCBWYW4gQXNzY2hlIHdyb3RlOg0KPiA+ID4gPiA+ID4gT24g
RnJpLCAyMDE3LTEwLTI3IGF0IDEyOjQzICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gPiA+ID4g
PiA+ID4gVGhlIDFzdCBwYXRjaCByZW1vdmVzIHRoZSBSRVNUQVJUIGZvciBUQUctU0hBUkVEIGJl
Y2F1c2UgU0NTSSBoYW5kbGVzIGl0DQo+ID4gPiA+ID4gPiA+IGJ5IGl0c2VsZiwgYW5kIG5vdCBu
ZWNlc3NhcnkgdG8gd2FzdGUgQ1BVIHRvIGRvIHRoZSBleHBlbnNpdmUgUkVTVEFSVC4NCj4gPiA+
ID4gPiA+ID4gQW5kIFJvbWFuIFBlbiByZXBvcnRlZCB0aGF0IHRoaXMgUkVTVEFSVCBjdXRzIGhh
bGYgb2YgSU9QUyBpbiBoaXMgY2FzZS4NCj4gPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiA+IFRo
ZSAybmQgcGF0Y2ggcmVtb3ZlcyB0aGUgUkVTVEFSVCB3aGVuIC5nZXRfYnVkZ2V0IHJldHVybnMg
QkxLX1NUU19SRVNPVVJDRSwNCj4gPiA+ID4gPiA+ID4gYW5kIHRoaXMgUkVTVEFSVCBpcyBoYW5k
bGVkIGJ5IFNDU0kncyBSRVNUQVJUKHNjc2lfZW5kX3JlcXVlc3QoKSkgdG9vLg0KPiA+ID4gPiA+
ID4gDQo+ID4gPiA+ID4gPiBUaGVyZSBhcmUgbW9yZSBibG9jayBkcml2ZXJzIHRoYW4gdGhlIFND
U0kgY29yZSB0aGF0IHNoYXJlIHRhZ3MuIElmIHRoZQ0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IENv
dWxkIHlvdSBzaGFyZSB1cyB3aGF0IHRoZSBvdGhlciBpbi10cmVlIGRyaXZlciB3aGljaCBzaGFy
ZSB0YWdzIGlzPw0KPiA+ID4gPiANCj4gPiA+ID4gSSB0aGluayB0aGUgZm9sbG93aW5nIGluLXRy
ZWUgZHJpdmVycyBzdXBwb3J0IHNoYXJlZCB0YWdzIChpbiBhbHBoYWJldGljYWwNCj4gPiA+ID4g
b3JkZXIpOg0KPiA+ID4gPiAqIG51bGxfYmxrLiBTZWUgYWxzbyB0aGUgc2hhcmVkX3RhZ3Mga2Vy
bmVsIG1vZHVsZSBwYXJhbWV0ZXIuDQo+ID4gPiA+ICogbnZtZS4gU2VlIGFsc28gbnZtZV9hbGxv
Y19ucygpLg0KPiA+ID4gPiAqIHNjc2ktbXEuDQo+ID4gPiANCj4gPiA+IEZvciBib3RoIG51bGxf
YmxrIGFuZCBudm1lLCB3ZSBkb24ndCBuZWVkIHRvIGRlYWwgd2l0aCBjcm9zcy1xdWV1ZSBSRVNU
QVJULA0KPiA+ID4gYmVjYXVzZSBCTEtfTVFfU19UQUdfV0FJVElORyBoYXMgaGFuZGxlZCBpdCBh
bHJlYWR5Lg0KPiA+IA0KPiA+IGJsa19tcV9kaXNwYXRjaF93YWl0X2FkZCgpIHJldHVybnMgaW1t
ZWRpYXRlbHkgaWYgQkxLX01RX1NfVEFHX1dBSVRJTkcgaXMNCj4gPiBhbHJlYWR5IHNldC4gQXJl
IHlvdSByZWFsbHkgc3VyZSB0aGF0IHJlbW92aW5nIHRoZSByZXN0YXJ0IG1lY2hhbmlzbSBkb2Vz
bid0DQo+ID4gYnJlYWsgdGhlIE5WTWUgZHJpdmVyIGlmIHRoZXJlIGFyZSBtdWx0aXBsZSBuYW1l
c3BhY2VzPz4gDQo+IElmIHRoaXMgZmxhZyBpcyBzZXQsIHRoYXQgbWVhbnMgYmxrX21xX2Rpc3Bh
dGNoX3dha2UoKSB3aWxsIGJlIHJ1biB3aGVuDQo+IGRyaXZlciB0YWcgaXMgYXZhaWxhYmxlLCBz
byB0aGUgaHcgcXVldWUgd2lsbCBiZSBydW4gYXQgdGhhdCB0aW1lLg0KDQpUaGUgb25seSBodyBx
dWV1ZSB0aGF0IHdpbGwgYmUgcmVydW4gaWYgQkxLX01RX1NfVEFHX1dBSVRJTkcgaXMgc2V0IGlz
IHRoZSBodw0KcXVldWUgZm9yIHdoaWNoIHRoYXQgZmxhZyB3YXMgc2V0LiBTb3JyeSBidXQgSSBk
b24ndCB0aGluayB0aGF0IG1lY2hhbmlzbSBieQ0KaXRzZWxmIGlzIHN1ZmZpY2llbnQgdG8gYXZv
aWQgcXVldWUgbG9ja3VwcyB3aXRoIHRoZSBOVk1lIGRyaXZlciBpbiBjYXNlIG9mDQpzaGFyZWQg
bmFtZXNwYWNlcy4gU2VlIGFsc28gdGhlIGRlc2NyaXB0aW9uIG9mIGNvbW1pdCBkYTU1ZjJjYzc4
NDEgKCJibGstbXE6DQp1c2Ugc2JxIHdhaXQgcXVldWVzIGluc3RlYWQgb2YgcmVzdGFydCBmb3Ig
ZHJpdmVyIHRhZ3MiKS4NCg0KQmFydC4=

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Roman Penyaev]

On Mon, Oct 30, 2017 at 9:33 PM, Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
> On Fri, 2017-10-27 at 19:55 +0200, Roman Penyaev wrote:
>> That's just a bug in code, not a in issue with restarts, which can be fixed
>> if we put hctx which are needed to be restarted in percpu lists and avoid
>> long loops and contentions.
>
> Hello Roman,
>
> Have you noticed that recently .get_budget() and .put_budget() callbacks have
> been added into struct blk_mq_ops? If block drivers that support shared tags
> implement these callback functions properly then maybe busy conditions become
> infrequent enough to use a single "hctx busy" list per tag set instead of
> per-cpu "hctx busy" lists.

Hi Bart,

No, not yet.  Thanks, good tip.

--
Roman

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Wed, Nov 01, 2017 at 04:47:06PM +0000, Bart Van Assche wrote:
> On Wed, 2017-11-01 at 12:08 +0800, Ming Lei wrote:
> > On Wed, Nov 01, 2017 at 03:54:09AM +0000, Bart Van Assche wrote:
> > > On Tue, 2017-10-31 at 09:47 +0800, Ming Lei wrote:
> > > > On Mon, Oct 30, 2017 at 08:24:57PM +0000, Bart Van Assche wrote:
> > > > > On Fri, 2017-10-27 at 13:38 +0800, Ming Lei wrote:
> > > > > > On Fri, Oct 27, 2017 at 04:53:18AM +0000, Bart Van Assche wrote:
> > > > > > > On Fri, 2017-10-27 at 12:43 +0800, Ming Lei wrote:
> > > > > > > > The 1st patch removes the RESTART for TAG-SHARED because SCSI handles it
> > > > > > > > by itself, and not necessary to waste CPU to do the expensive RESTART.
> > > > > > > > And Roman Pen reported that this RESTART cuts half of IOPS in his case.
> > > > > > > > 
> > > > > > > > The 2nd patch removes the RESTART when .get_budget returns BLK_STS_RESOURCE,
> > > > > > > > and this RESTART is handled by SCSI's RESTART(scsi_end_request()) too.
> > > > > > > 
> > > > > > > There are more block drivers than the SCSI core that share tags. If the
> > > > > > 
> > > > > > Could you share us what the other in-tree driver which share tags is?
> > > > > 
> > > > > I think the following in-tree drivers support shared tags (in alphabetical
> > > > > order):
> > > > > * null_blk. See also the shared_tags kernel module parameter.
> > > > > * nvme. See also nvme_alloc_ns().
> > > > > * scsi-mq.
> > > > 
> > > > For both null_blk and nvme, we don't need to deal with cross-queue RESTART,
> > > > because BLK_MQ_S_TAG_WAITING has handled it already.
> > > 
> > > blk_mq_dispatch_wait_add() returns immediately if BLK_MQ_S_TAG_WAITING is
> > > already set. Are you really sure that removing the restart mechanism doesn't
> > > break the NVMe driver if there are multiple namespaces?> 
> > If this flag is set, that means blk_mq_dispatch_wake() will be run when
> > driver tag is available, so the hw queue will be run at that time.
> 
> The only hw queue that will be rerun if BLK_MQ_S_TAG_WAITING is set is the hw
> queue for which that flag was set. Sorry but I don't think that mechanism by
> itself is sufficient to avoid queue lockups with the NVMe driver in case of
> shared namespaces. See also the description of commit da55f2cc7841 ("blk-mq:

Why?

> use sbq wait queues instead of restart for driver tags").

This commit just make sure that the hctx of BLK_MQ_S_TAG_WAITING will be
run again once there is driver tag available, see its commit log simply:

    However, we can still use the struct sbitmap_queue wait queues with a
    custom callback instead of blocking. This has a few benefits:
    
    1. It avoids iterating over all hardware queues when completing an I/O,
       which the current restart code has to do.
    2. It benefits from the existing rolling wakeup code.
    3. It avoids punting to another thread just to have it block.
    
The commit has described clearly that we don't need to iterate over all hw
queue when completing an I/O.

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gVGh1LCAyMDE3LTExLTAyIGF0IDAwOjU5ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gVGhl
IGNvbW1pdCBoYXMgZGVzY3JpYmVkIGNsZWFybHkgdGhhdCB3ZSBkb24ndCBuZWVkIHRvIGl0ZXJh
dGUgb3ZlciBhbGwgaHcNCj4gcXVldWUgd2hlbiBjb21wbGV0aW5nIGFuIEkvTy4NCg0KWW91IHF1
b3RlZCBhIHNtYWxsIHBhcnQgb2YgdGhlIGRlc2NyaXB0aW9uIG9mIGNvbW1pdCBkYTU1ZjJjYzc4
NDEgYW5kIGxlZnQgb3V0DQp0aGUgcGFydCB0aGF0IHNob3dzIHRoYXQgd2hhdCB5b3Ugd3JvdGUg
YWJvdmUgaXMgbm90IGNvcnJlY3QuDQoNCkJhcnQu

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Wed, Nov 01, 2017 at 05:09:41PM +0000, Bart Van Assche wrote:
> On Thu, 2017-11-02 at 00:59 +0800, Ming Lei wrote:
> > The commit has described clearly that we don't need to iterate over all hw
> > queue when completing an I/O.
> 
> You quoted a small part of the description of commit da55f2cc7841 and left out
> the part that shows that what you wrote above is not correct.

If you mean the 1st part:

    Commit 50e1dab86aa2 ("blk-mq-sched: fix starvation for multiple hardware
    queues and shared tags") fixed one starvation issue for shared tags.
    However, we can still get into a situation where we fail to allocate a
    tag because all tags are allocated but we don't have any pending
    requests on any hardware queue.

That is just what this commit wants to address, then we don't need to
depend on restart from all hctx for fixing this starvation.

If you don't believe me, you can ask Omar and Jens to clarify.

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gV2VkLCAyMDE3LTExLTAxIGF0IDA4OjIxIC0wNjAwLCBKZW5zIEF4Ym9lIHdyb3RlOg0KPiBG
aXhlZCB0aGF0IHVwLCBhbmQgYXBwbGllZCB0aGVzZSB0d28gcGF0Y2hlcyBhcyB3ZWxsLg0KDQpI
ZWxsbyBKZW5zLA0KDQpSZWNlbnRseSBJIG5vdGljZWQgdGhhdCBhIHRlc3Qgc3lzdGVtIHNwb3Jh
ZGljYWxseSBoYW5ncyBkdXJpbmcgYm9vdCAoRGVsbA0KUG93ZXJFZGdlIFI3MjAgdGhhdCBib290
cyBmcm9tIGEgaGFyZCBkaXNrIGNvbm5lY3RlZCB0byBhIE1lZ2FSQUlEIFNBUyBhZGFwdGVyKQ0K
YW5kIGFsc28gdGhhdCBzcnAtdGVzdHMgc3lzdGVtYXRpY2FsbHkgaGFuZ3MuIFJldmVydGluZyB0
aGUgdHdvIHBhdGNoZXMgZnJvbQ0KdGhpcyBzZXJpZXMgZml4ZXMgYm90aCBpc3N1ZXMuIEknbSBu
b3Qgc3VyZSB0aGVyZSBpcyBhbm90aGVyIHNvbHV0aW9uIHRoYW4NCnJldmVydGluZyB0aGUgdHdv
IHBhdGNoZXMgZnJvbSB0aGlzIHNlcmllcy4NCg0KQmFydC4NCg0KDQpCVFcsIHRoZSBmb2xsb3dp
bmcgYXBwZWFyZWQgaW4gdGhlIGtlcm5lbCBsb2cgd2hlbiBJIHRyaWVkIHRvIHJ1biBzcnAtdGVz
dHMNCmFnYWluc3QgYSBrZXJuZWwgd2l0aCB0aGUgdHdvIHBhdGNoZXMgZnJvbSB0aGlzIHNlcmll
cyBhcHBsaWVkOg0KDQpJTkZPOiB0YXNrIGt3b3JrZXIvMTk6MToyMDkgYmxvY2tlZCBmb3IgbW9y
ZSB0aGFuIDQ4MCBzZWNvbmRzLg0KSU5GTzogdGFzayBrd29ya2VyLzE5OjE6MjA5IGJsb2NrZWQg
Zm9yIG1vcmUgdGhhbiA0ODAgc2Vjb25kcy4NCiAgICAgIFRhaW50ZWQ6IEcgICAgICAgIFcgICAg
ICAgNC4xNC4wLXJjNy1kYmcrICMxDQogICAgICBUYWludGVkOiBHICAgICAgICBXICAgICAgIDQu
MTQuMC1yYzctZGJnKyAjMQ0KImVjaG8gMCA+IC9wcm9jL3N5cy9rZXJuZWwvaHVuZ190YXNrX3Rp
bWVvdXRfc2VjcyIgZGlzYWJsZXMgdGhpcyBtZXNzYWdlLg0KImVjaG8gMCA+IC9wcm9jL3N5cy9r
ZXJuZWwvaHVuZ190YXNrX3RpbWVvdXRfc2VjcyIgZGlzYWJsZXMgdGhpcyBtZXNzYWdlLg0Ka3dv
cmtlci8xOToxICAgIEQgICAgMCAgIDIwOSAgICAgIDIgMHg4MDAwMDAwMA0Ka3dvcmtlci8xOTox
ICAgIEQgICAgMCAgIDIwOSAgICAgIDIgMHg4MDAwMDAwMA0KV29ya3F1ZXVlOiBzcnBfcmVtb3Zl
IHNycF9yZW1vdmVfd29yayBbaWJfc3JwXQ0KV29ya3F1ZXVlOiBzcnBfcmVtb3ZlIHNycF9yZW1v
dmVfd29yayBbaWJfc3JwXQ0KQ2FsbCBUcmFjZToNCkNhbGwgVHJhY2U6DQogX19zY2hlZHVsZSsw
eDJmYS8weGJiMA0KIF9fc2NoZWR1bGUrMHgyZmEvMHhiYjANCiBzY2hlZHVsZSsweDM2LzB4OTAN
CiBzY2hlZHVsZSsweDM2LzB4OTANCiBhc3luY19zeW5jaHJvbml6ZV9jb29raWVfZG9tYWluKzB4
ODgvMHgxMzANCiBhc3luY19zeW5jaHJvbml6ZV9jb29raWVfZG9tYWluKzB4ODgvMHgxMzANCiA/
IGZpbmlzaF93YWl0KzB4OTAvMHg5MA0KID8gZmluaXNoX3dhaXQrMHg5MC8weDkwDQogYXN5bmNf
c3luY2hyb25pemVfZnVsbF9kb21haW4rMHgxOC8weDIwDQogYXN5bmNfc3luY2hyb25pemVfZnVs
bF9kb21haW4rMHgxOC8weDIwDQogc2RfcmVtb3ZlKzB4NGQvMHhjMCBbc2RfbW9kXQ0KIHNkX3Jl
bW92ZSsweDRkLzB4YzAgW3NkX21vZF0NCiBkZXZpY2VfcmVsZWFzZV9kcml2ZXJfaW50ZXJuYWwr
MHgxNjAvMHgyMTANCiBkZXZpY2VfcmVsZWFzZV9kcml2ZXJfaW50ZXJuYWwrMHgxNjAvMHgyMTAN
CiBkZXZpY2VfcmVsZWFzZV9kcml2ZXIrMHgxMi8weDIwDQogZGV2aWNlX3JlbGVhc2VfZHJpdmVy
KzB4MTIvMHgyMA0KIGJ1c19yZW1vdmVfZGV2aWNlKzB4MTAwLzB4MTgwDQogYnVzX3JlbW92ZV9k
ZXZpY2UrMHgxMDAvMHgxODANCiBkZXZpY2VfZGVsKzB4MWQ4LzB4MzQwDQogZGV2aWNlX2RlbCsw
eDFkOC8weDM0MA0KIF9fc2NzaV9yZW1vdmVfZGV2aWNlKzB4ZmMvMHgxMzANCiBfX3Njc2lfcmVt
b3ZlX2RldmljZSsweGZjLzB4MTMwDQogc2NzaV9mb3JnZXRfaG9zdCsweDI1LzB4NzANCiBzY3Np
X2ZvcmdldF9ob3N0KzB4MjUvMHg3MA0KIHNjc2lfcmVtb3ZlX2hvc3QrMHg3OS8weDEyMA0KIHNj
c2lfcmVtb3ZlX2hvc3QrMHg3OS8weDEyMA0KIHNycF9yZW1vdmVfd29yaysweDkwLzB4MWQwIFtp
Yl9zcnBdDQogc3JwX3JlbW92ZV93b3JrKzB4OTAvMHgxZDAgW2liX3NycF0NCiBwcm9jZXNzX29u
ZV93b3JrKzB4MjBhLzB4NjYwDQogcHJvY2Vzc19vbmVfd29yaysweDIwYS8weDY2MA0KIHdvcmtl
cl90aHJlYWQrMHgzZC8weDNiMA0KIHdvcmtlcl90aHJlYWQrMHgzZC8weDNiMA0KIGt0aHJlYWQr
MHgxM2EvMHgxNTANCiBrdGhyZWFkKzB4MTNhLzB4MTUwDQogPyBwcm9jZXNzX29uZV93b3JrKzB4
NjYwLzB4NjYwDQogPyBwcm9jZXNzX29uZV93b3JrKzB4NjYwLzB4NjYwDQogPyBrdGhyZWFkX2Ny
ZWF0ZV9vbl9ub2RlKzB4NDAvMHg0MA0KID8ga3RocmVhZF9jcmVhdGVfb25fbm9kZSsweDQwLzB4
NDANCiByZXRfZnJvbV9mb3JrKzB4MjcvMHg0MA0KIHJldF9mcm9tX2ZvcmsrMHgyNy8weDQwDQoN
ClNob3dpbmcgYWxsIGxvY2tzIGhlbGQgaW4gdGhlIHN5c3RlbToNCg0KU2hvd2luZyBhbGwgbG9j
a3MgaGVsZCBpbiB0aGUgc3lzdGVtOg0KMSBsb2NrIGhlbGQgYnkga2h1bmd0YXNrZC8xNzA6DQox
IGxvY2sgaGVsZCBieSBraHVuZ3Rhc2tkLzE3MDoNCiAjMDogICh0YXNrbGlzdF9sb2NrKXsuKy4r
fSwgYXQ6IFs8ZmZmZmZmZmY4MTBjMTI1ZD5dIGRlYnVnX3Nob3dfYWxsX2xvY2tzKzB4M2QvMHgx
YTANCiAjMDogICh0YXNrbGlzdF9sb2NrKXsuKy4rfSwgYXQ6IFs8ZmZmZmZmZmY4MTBjMTI1ZD5d
IGRlYnVnX3Nob3dfYWxsX2xvY2tzKzB4M2QvMHgxYTANCjQgbG9ja3MgaGVsZCBieSBrd29ya2Vy
LzE5OjEvMjA5Og0KNCBsb2NrcyBoZWxkIGJ5IGt3b3JrZXIvMTk6MS8yMDk6DQogIzA6ICAoIiVz
Iigic3JwX3JlbW92ZSIpKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTA4M2I4NT5dIHByb2Nlc3Nf
b25lX3dvcmsrMHgxOTUvMHg2NjANCiAjMDogICgiJXMiKCJzcnBfcmVtb3ZlIikpeysuKy59LCBh
dDogWzxmZmZmZmZmZjgxMDgzYjg1Pl0gcHJvY2Vzc19vbmVfd29yaysweDE5NS8weDY2MA0KICMx
OiAgKCgmdGFyZ2V0LT5yZW1vdmVfd29yaykpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxMDgzYjg1
Pl0gcHJvY2Vzc19vbmVfd29yaysweDE5NS8weDY2MA0KICMxOiAgKCgmdGFyZ2V0LT5yZW1vdmVf
d29yaykpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxMDgzYjg1Pl0gcHJvY2Vzc19vbmVfd29yaysw
eDE5NS8weDY2MA0KICMyOiAgKCZzaG9zdC0+c2Nhbl9tdXRleCl7Ky4rLn0sIGF0OiBbPGZmZmZm
ZmZmODE0ODA3YmY+XSBzY3NpX3JlbW92ZV9ob3N0KzB4MWYvMHgxMjANCiAjMjogICgmc2hvc3Qt
PnNjYW5fbXV0ZXgpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxNDgwN2JmPl0gc2NzaV9yZW1vdmVf
aG9zdCsweDFmLzB4MTIwDQogIzM6ICAoJmRldi0+bXV0ZXgpey4uLi59LCBhdDogWzxmZmZmZmZm
ZjgxNDUwMWE5Pl0gZGV2aWNlX3JlbGVhc2VfZHJpdmVyX2ludGVybmFsKzB4MzkvMHgyMTANCiAj
MzogICgmZGV2LT5tdXRleCl7Li4uLn0sIGF0OiBbPGZmZmZmZmZmODE0NTAxYTk+XSBkZXZpY2Vf
cmVsZWFzZV9kcml2ZXJfaW50ZXJuYWwrMHgzOS8weDIxMA0KMiBsb2NrcyBoZWxkIGJ5IGt3b3Jr
ZXIvdTY2OjAvMTkyNzoNCjIgbG9ja3MgaGVsZCBieSBrd29ya2VyL3U2NjowLzE5Mjc6DQogIzA6
ICAoImV2ZW50c191bmJvdW5kIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODEwODNiODU+XSBwcm9j
ZXNzX29uZV93b3JrKzB4MTk1LzB4NjYwDQogIzA6ICAoImV2ZW50c191bmJvdW5kIil7Ky4rLn0s
IGF0OiBbPGZmZmZmZmZmODEwODNiODU+XSBwcm9jZXNzX29uZV93b3JrKzB4MTk1LzB4NjYwDQog
IzE6ICAoKCZlbnRyeS0+d29yaykpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxMDgzYjg1Pl0gcHJv
Y2Vzc19vbmVfd29yaysweDE5NS8weDY2MA0KICMxOiAgKCgmZW50cnktPndvcmspKXsrLisufSwg
YXQ6IFs8ZmZmZmZmZmY4MTA4M2I4NT5dIHByb2Nlc3Nfb25lX3dvcmsrMHgxOTUvMHg2NjANCjIg
bG9ja3MgaGVsZCBieSBrd29ya2VyLzU6MC8yMDQ3Og0KMiBsb2NrcyBoZWxkIGJ5IGt3b3JrZXIv
NTowLzIwNDc6DQogIzA6ICAoImthbHVhZCIpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxMDgzYjg1
Pl0gcHJvY2Vzc19vbmVfd29yaysweDE5NS8weDY2MA0KICMwOiAgKCJrYWx1YWQiKXsrLisufSwg
YXQ6IFs8ZmZmZmZmZmY4MTA4M2I4NT5dIHByb2Nlc3Nfb25lX3dvcmsrMHgxOTUvMHg2NjANCiAj
MTogICgoJigmcGctPnJ0cGdfd29yayktPndvcmspKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTA4
M2I4NT5dIHByb2Nlc3Nfb25lX3dvcmsrMHgxOTUvMHg2NjANCiAjMTogICgoJigmcGctPnJ0cGdf
d29yayktPndvcmspKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTA4M2I4NT5dIHByb2Nlc3Nfb25l
X3dvcmsrMHgxOTUvMHg2NjANCg0KDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT0NCg0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09DQoNCklORk86IHRhc2sga3dvcmtlci8xOToxOjIwOSBibG9ja2VkIGZvciBtb3JlIHRoYW4g
NDgwIHNlY29uZHMuDQpJTkZPOiB0YXNrIGt3b3JrZXIvMTk6MToyMDkgYmxvY2tlZCBmb3IgbW9y
ZSB0aGFuIDQ4MCBzZWNvbmRzLg0KICAgICAgVGFpbnRlZDogRyAgICAgICAgVyAgICAgICA0LjE0
LjAtcmM3LWRiZysgIzENCiAgICAgIFRhaW50ZWQ6IEcgICAgICAgIFcgICAgICAgNC4xNC4wLXJj
Ny1kYmcrICMxDQoiZWNobyAwID4gL3Byb2Mvc3lzL2tlcm5lbC9odW5nX3Rhc2tfdGltZW91dF9z
ZWNzIiBkaXNhYmxlcyB0aGlzIG1lc3NhZ2UuDQoiZWNobyAwID4gL3Byb2Mvc3lzL2tlcm5lbC9o
dW5nX3Rhc2tfdGltZW91dF9zZWNzIiBkaXNhYmxlcyB0aGlzIG1lc3NhZ2UuDQprd29ya2VyLzE5
OjEgICAgRCAgICAwICAgMjA5ICAgICAgMiAweDgwMDAwMDAwDQprd29ya2VyLzE5OjEgICAgRCAg
ICAwICAgMjA5ICAgICAgMiAweDgwMDAwMDAwDQpXb3JrcXVldWU6IHNycF9yZW1vdmUgc3JwX3Jl
bW92ZV93b3JrIFtpYl9zcnBdDQpXb3JrcXVldWU6IHNycF9yZW1vdmUgc3JwX3JlbW92ZV93b3Jr
IFtpYl9zcnBdDQpDYWxsIFRyYWNlOg0KQ2FsbCBUcmFjZToNCiBfX3NjaGVkdWxlKzB4MmZhLzB4
YmIwDQogX19zY2hlZHVsZSsweDJmYS8weGJiMA0KIHNjaGVkdWxlKzB4MzYvMHg5MA0KIHNjaGVk
dWxlKzB4MzYvMHg5MA0KIGFzeW5jX3N5bmNocm9uaXplX2Nvb2tpZV9kb21haW4rMHg4OC8weDEz
MA0KIGFzeW5jX3N5bmNocm9uaXplX2Nvb2tpZV9kb21haW4rMHg4OC8weDEzMA0KID8gZmluaXNo
X3dhaXQrMHg5MC8weDkwDQogPyBmaW5pc2hfd2FpdCsweDkwLzB4OTANCiBhc3luY19zeW5jaHJv
bml6ZV9mdWxsX2RvbWFpbisweDE4LzB4MjANCiBhc3luY19zeW5jaHJvbml6ZV9mdWxsX2RvbWFp
bisweDE4LzB4MjANCiBzZF9yZW1vdmUrMHg0ZC8weGMwIFtzZF9tb2RdDQogc2RfcmVtb3ZlKzB4
NGQvMHhjMCBbc2RfbW9kXQ0KIGRldmljZV9yZWxlYXNlX2RyaXZlcl9pbnRlcm5hbCsweDE2MC8w
eDIxMA0KIGRldmljZV9yZWxlYXNlX2RyaXZlcl9pbnRlcm5hbCsweDE2MC8weDIxMA0KIGRldmlj
ZV9yZWxlYXNlX2RyaXZlcisweDEyLzB4MjANCiBkZXZpY2VfcmVsZWFzZV9kcml2ZXIrMHgxMi8w
eDIwDQogYnVzX3JlbW92ZV9kZXZpY2UrMHgxMDAvMHgxODANCiBidXNfcmVtb3ZlX2RldmljZSsw
eDEwMC8weDE4MA0KIGRldmljZV9kZWwrMHgxZDgvMHgzNDANCiBkZXZpY2VfZGVsKzB4MWQ4LzB4
MzQwDQogX19zY3NpX3JlbW92ZV9kZXZpY2UrMHhmYy8weDEzMA0KIF9fc2NzaV9yZW1vdmVfZGV2
aWNlKzB4ZmMvMHgxMzANCiBzY3NpX2ZvcmdldF9ob3N0KzB4MjUvMHg3MA0KIHNjc2lfZm9yZ2V0
X2hvc3QrMHgyNS8weDcwDQogc2NzaV9yZW1vdmVfaG9zdCsweDc5LzB4MTIwDQogc2NzaV9yZW1v
dmVfaG9zdCsweDc5LzB4MTIwDQogc3JwX3JlbW92ZV93b3JrKzB4OTAvMHgxZDAgW2liX3NycF0N
CiBzcnBfcmVtb3ZlX3dvcmsrMHg5MC8weDFkMCBbaWJfc3JwXQ0KIHByb2Nlc3Nfb25lX3dvcmsr
MHgyMGEvMHg2NjANCiBwcm9jZXNzX29uZV93b3JrKzB4MjBhLzB4NjYwDQogd29ya2VyX3RocmVh
ZCsweDNkLzB4M2IwDQogd29ya2VyX3RocmVhZCsweDNkLzB4M2IwDQoga3RocmVhZCsweDEzYS8w
eDE1MA0KIGt0aHJlYWQrMHgxM2EvMHgxNTANCiA/IHByb2Nlc3Nfb25lX3dvcmsrMHg2NjAvMHg2
NjANCiA/IHByb2Nlc3Nfb25lX3dvcmsrMHg2NjAvMHg2NjANCiA/IGt0aHJlYWRfY3JlYXRlX29u
X25vZGUrMHg0MC8weDQwDQogPyBrdGhyZWFkX2NyZWF0ZV9vbl9ub2RlKzB4NDAvMHg0MA0KIHJl
dF9mcm9tX2ZvcmsrMHgyNy8weDQwDQogcmV0X2Zyb21fZm9yaysweDI3LzB4NDANCg0KU2hvd2lu
ZyBhbGwgbG9ja3MgaGVsZCBpbiB0aGUgc3lzdGVtOg0KDQpTaG93aW5nIGFsbCBsb2NrcyBoZWxk
IGluIHRoZSBzeXN0ZW06DQoxIGxvY2sgaGVsZCBieSBraHVuZ3Rhc2tkLzE3MDoNCjEgbG9jayBo
ZWxkIGJ5IGtodW5ndGFza2QvMTcwOg0KICMwOiAgKHRhc2tsaXN0X2xvY2spey4rLit9LCBhdDog
WzxmZmZmZmZmZjgxMGMxMjVkPl0gZGVidWdfc2hvd19hbGxfbG9ja3MrMHgzZC8weDFhMA0KICMw
OiAgKHRhc2tsaXN0X2xvY2spey4rLit9LCBhdDogWzxmZmZmZmZmZjgxMGMxMjVkPl0gZGVidWdf
c2hvd19hbGxfbG9ja3MrMHgzZC8weDFhMA0KNCBsb2NrcyBoZWxkIGJ5IGt3b3JrZXIvMTk6MS8y
MDk6DQo0IGxvY2tzIGhlbGQgYnkga3dvcmtlci8xOToxLzIwOToNCiAjMDogICgiJXMiKCJzcnBf
cmVtb3ZlIikpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxMDgzYjg1Pl0gcHJvY2Vzc19vbmVfd29y
aysweDE5NS8weDY2MA0KICMwOiAgKCIlcyIoInNycF9yZW1vdmUiKSl7Ky4rLn0sIGF0OiBbPGZm
ZmZmZmZmODEwODNiODU+XSBwcm9jZXNzX29uZV93b3JrKzB4MTk1LzB4NjYwDQogIzE6ICAoKCZ0
YXJnZXQtPnJlbW92ZV93b3JrKSl7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODEwODNiODU+XSBwcm9j
ZXNzX29uZV93b3JrKzB4MTk1LzB4NjYwDQogIzE6ICAoKCZ0YXJnZXQtPnJlbW92ZV93b3JrKSl7
Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODEwODNiODU+XSBwcm9jZXNzX29uZV93b3JrKzB4MTk1LzB4
NjYwDQogIzI6ICAoJnNob3N0LT5zY2FuX211dGV4KXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTQ4
MDdiZj5dIHNjc2lfcmVtb3ZlX2hvc3QrMHgxZi8weDEyMA0KICMyOiAgKCZzaG9zdC0+c2Nhbl9t
dXRleCl7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODE0ODA3YmY+XSBzY3NpX3JlbW92ZV9ob3N0KzB4
MWYvMHgxMjANCiAjMzogICgmZGV2LT5tdXRleCl7Li4uLn0sIGF0OiBbPGZmZmZmZmZmODE0NTAx
YTk+XSBkZXZpY2VfcmVsZWFzZV9kcml2ZXJfaW50ZXJuYWwrMHgzOS8weDIxMA0KICMzOiAgKCZk
ZXYtPm11dGV4KXsuLi4ufSwgYXQ6IFs8ZmZmZmZmZmY4MTQ1MDFhOT5dIGRldmljZV9yZWxlYXNl
X2RyaXZlcl9pbnRlcm5hbCsweDM5LzB4MjEwDQoyIGxvY2tzIGhlbGQgYnkga3dvcmtlci91NjY6
MC8xOTI3Og0KMiBsb2NrcyBoZWxkIGJ5IGt3b3JrZXIvdTY2OjAvMTkyNzoNCiAjMDogICgiZXZl
bnRzX3VuYm91bmQiKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTA4M2I4NT5dIHByb2Nlc3Nfb25l
X3dvcmsrMHgxOTUvMHg2NjANCiAjMDogICgiZXZlbnRzX3VuYm91bmQiKXsrLisufSwgYXQ6IFs8
ZmZmZmZmZmY4MTA4M2I4NT5dIHByb2Nlc3Nfb25lX3dvcmsrMHgxOTUvMHg2NjANCiAjMTogICgo
JmVudHJ5LT53b3JrKSl7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODEwODNiODU+XSBwcm9jZXNzX29u
ZV93b3JrKzB4MTk1LzB4NjYwDQogIzE6ICAoKCZlbnRyeS0+d29yaykpeysuKy59LCBhdDogWzxm
ZmZmZmZmZjgxMDgzYjg1Pl0gcHJvY2Vzc19vbmVfd29yaysweDE5NS8weDY2MA0KMiBsb2NrcyBo
ZWxkIGJ5IGt3b3JrZXIvNTowLzIwNDc6DQoyIGxvY2tzIGhlbGQgYnkga3dvcmtlci81OjAvMjA0
NzoNCiAjMDogICgia2FsdWFkIil7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmODEwODNiODU+XSBwcm9j
ZXNzX29uZV93b3JrKzB4MTk1LzB4NjYwDQogIzA6ICAoImthbHVhZCIpeysuKy59LCBhdDogWzxm
ZmZmZmZmZjgxMDgzYjg1Pl0gcHJvY2Vzc19vbmVfd29yaysweDE5NS8weDY2MA0KICMxOiAgKCgm
KCZwZy0+cnRwZ193b3JrKS0+d29yaykpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxMDgzYjg1Pl0g
cHJvY2Vzc19vbmVfd29yaysweDE5NS8weDY2MA0KICMxOiAgKCgmKCZwZy0+cnRwZ193b3JrKS0+
d29yaykpeysuKy59LCBhdDogWzxmZmZmZmZmZjgxMDgzYjg1Pl0gcHJvY2Vzc19vbmVfd29yaysw
eDE5NS8weDY2MA0KDQoNCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PQ0KDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0=

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Thu, Nov 02, 2017 at 03:57:05PM +0000, Bart Van Assche wrote:
> On Wed, 2017-11-01 at 08:21 -0600, Jens Axboe wrote:
> > Fixed that up, and applied these two patches as well.
> 
> Hello Jens,
> 
> Recently I noticed that a test system sporadically hangs during boot (Dell
> PowerEdge R720 that boots from a hard disk connected to a MegaRAID SAS adapter)
> and also that srp-tests systematically hangs. Reverting the two patches from
> this series fixes both issues. I'm not sure there is another solution than
> reverting the two patches from this series.

Then we need to find the root cause, instead of using sort of workaround
as before.

For SCSI, the restart is always run from scsi_end_request(), and this
kind of restart from all hctx isn't necessary at all.

> 
> Bart.
> 
> 
> BTW, the following appeared in the kernel log when I tried to run srp-tests
> against a kernel with the two patches from this series applied:
> 
> INFO: task kworker/19:1:209 blocked for more than 480 seconds.
> INFO: task kworker/19:1:209 blocked for more than 480 seconds.
>       Tainted: G        W       4.14.0-rc7-dbg+ #1
>       Tainted: G        W       4.14.0-rc7-dbg+ #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> kworker/19:1    D    0   209      2 0x80000000
> kworker/19:1    D    0   209      2 0x80000000
> Workqueue: srp_remove srp_remove_work [ib_srp]
> Workqueue: srp_remove srp_remove_work [ib_srp]
> Call Trace:
> Call Trace:
>  __schedule+0x2fa/0xbb0
>  __schedule+0x2fa/0xbb0
>  schedule+0x36/0x90
>  schedule+0x36/0x90
>  async_synchronize_cookie_domain+0x88/0x130
>  async_synchronize_cookie_domain+0x88/0x130
>  ? finish_wait+0x90/0x90
>  ? finish_wait+0x90/0x90
>  async_synchronize_full_domain+0x18/0x20
>  async_synchronize_full_domain+0x18/0x20
>  sd_remove+0x4d/0xc0 [sd_mod]
>  sd_remove+0x4d/0xc0 [sd_mod]
>  device_release_driver_internal+0x160/0x210
>  device_release_driver_internal+0x160/0x210
>  device_release_driver+0x12/0x20
>  device_release_driver+0x12/0x20
>  bus_remove_device+0x100/0x180
>  bus_remove_device+0x100/0x180
>  device_del+0x1d8/0x340
>  device_del+0x1d8/0x340
>  __scsi_remove_device+0xfc/0x130
>  __scsi_remove_device+0xfc/0x130
>  scsi_forget_host+0x25/0x70
>  scsi_forget_host+0x25/0x70
>  scsi_remove_host+0x79/0x120
>  scsi_remove_host+0x79/0x120
>  srp_remove_work+0x90/0x1d0 [ib_srp]
>  srp_remove_work+0x90/0x1d0 [ib_srp]
>  process_one_work+0x20a/0x660
>  process_one_work+0x20a/0x660
>  worker_thread+0x3d/0x3b0
>  worker_thread+0x3d/0x3b0
>  kthread+0x13a/0x150
>  kthread+0x13a/0x150
>  ? process_one_work+0x660/0x660
>  ? process_one_work+0x660/0x660
>  ? kthread_create_on_node+0x40/0x40
>  ? kthread_create_on_node+0x40/0x40
>  ret_from_fork+0x27/0x40
>  ret_from_fork+0x27/0x40
> 
> Showing all locks held in the system:
> 
> Showing all locks held in the system:
> 1 lock held by khungtaskd/170:
> 1 lock held by khungtaskd/170:
>  #0:  (tasklist_lock){.+.+}, at: [<ffffffff810c125d>] debug_show_all_locks+0x3d/0x1a0
>  #0:  (tasklist_lock){.+.+}, at: [<ffffffff810c125d>] debug_show_all_locks+0x3d/0x1a0
> 4 locks held by kworker/19:1/209:
> 4 locks held by kworker/19:1/209:
>  #0:  ("%s"("srp_remove")){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #0:  ("%s"("srp_remove")){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&target->remove_work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&target->remove_work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #2:  (&shost->scan_mutex){+.+.}, at: [<ffffffff814807bf>] scsi_remove_host+0x1f/0x120
>  #2:  (&shost->scan_mutex){+.+.}, at: [<ffffffff814807bf>] scsi_remove_host+0x1f/0x120
>  #3:  (&dev->mutex){....}, at: [<ffffffff814501a9>] device_release_driver_internal+0x39/0x210
>  #3:  (&dev->mutex){....}, at: [<ffffffff814501a9>] device_release_driver_internal+0x39/0x210
> 2 locks held by kworker/u66:0/1927:
> 2 locks held by kworker/u66:0/1927:
>  #0:  ("events_unbound"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #0:  ("events_unbound"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&entry->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&entry->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
> 2 locks held by kworker/5:0/2047:
> 2 locks held by kworker/5:0/2047:
>  #0:  ("kaluad"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #0:  ("kaluad"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&(&pg->rtpg_work)->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&(&pg->rtpg_work)->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
> 
> 
> =============================================
> 
> =============================================
> 
> INFO: task kworker/19:1:209 blocked for more than 480 seconds.
> INFO: task kworker/19:1:209 blocked for more than 480 seconds.
>       Tainted: G        W       4.14.0-rc7-dbg+ #1
>       Tainted: G        W       4.14.0-rc7-dbg+ #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> kworker/19:1    D    0   209      2 0x80000000
> kworker/19:1    D    0   209      2 0x80000000
> Workqueue: srp_remove srp_remove_work [ib_srp]
> Workqueue: srp_remove srp_remove_work [ib_srp]
> Call Trace:
> Call Trace:
>  __schedule+0x2fa/0xbb0
>  __schedule+0x2fa/0xbb0
>  schedule+0x36/0x90
>  schedule+0x36/0x90
>  async_synchronize_cookie_domain+0x88/0x130
>  async_synchronize_cookie_domain+0x88/0x130
>  ? finish_wait+0x90/0x90
>  ? finish_wait+0x90/0x90
>  async_synchronize_full_domain+0x18/0x20
>  async_synchronize_full_domain+0x18/0x20
>  sd_remove+0x4d/0xc0 [sd_mod]
>  sd_remove+0x4d/0xc0 [sd_mod]
>  device_release_driver_internal+0x160/0x210
>  device_release_driver_internal+0x160/0x210
>  device_release_driver+0x12/0x20
>  device_release_driver+0x12/0x20
>  bus_remove_device+0x100/0x180
>  bus_remove_device+0x100/0x180
>  device_del+0x1d8/0x340
>  device_del+0x1d8/0x340
>  __scsi_remove_device+0xfc/0x130
>  __scsi_remove_device+0xfc/0x130
>  scsi_forget_host+0x25/0x70
>  scsi_forget_host+0x25/0x70
>  scsi_remove_host+0x79/0x120
>  scsi_remove_host+0x79/0x120
>  srp_remove_work+0x90/0x1d0 [ib_srp]
>  srp_remove_work+0x90/0x1d0 [ib_srp]
>  process_one_work+0x20a/0x660
>  process_one_work+0x20a/0x660
>  worker_thread+0x3d/0x3b0
>  worker_thread+0x3d/0x3b0
>  kthread+0x13a/0x150
>  kthread+0x13a/0x150
>  ? process_one_work+0x660/0x660
>  ? process_one_work+0x660/0x660
>  ? kthread_create_on_node+0x40/0x40
>  ? kthread_create_on_node+0x40/0x40
>  ret_from_fork+0x27/0x40
>  ret_from_fork+0x27/0x40
> 
> Showing all locks held in the system:
> 
> Showing all locks held in the system:
> 1 lock held by khungtaskd/170:
> 1 lock held by khungtaskd/170:
>  #0:  (tasklist_lock){.+.+}, at: [<ffffffff810c125d>] debug_show_all_locks+0x3d/0x1a0
>  #0:  (tasklist_lock){.+.+}, at: [<ffffffff810c125d>] debug_show_all_locks+0x3d/0x1a0
> 4 locks held by kworker/19:1/209:
> 4 locks held by kworker/19:1/209:
>  #0:  ("%s"("srp_remove")){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #0:  ("%s"("srp_remove")){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&target->remove_work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&target->remove_work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #2:  (&shost->scan_mutex){+.+.}, at: [<ffffffff814807bf>] scsi_remove_host+0x1f/0x120
>  #2:  (&shost->scan_mutex){+.+.}, at: [<ffffffff814807bf>] scsi_remove_host+0x1f/0x120
>  #3:  (&dev->mutex){....}, at: [<ffffffff814501a9>] device_release_driver_internal+0x39/0x210
>  #3:  (&dev->mutex){....}, at: [<ffffffff814501a9>] device_release_driver_internal+0x39/0x210
> 2 locks held by kworker/u66:0/1927:
> 2 locks held by kworker/u66:0/1927:
>  #0:  ("events_unbound"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #0:  ("events_unbound"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&entry->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&entry->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
> 2 locks held by kworker/5:0/2047:
> 2 locks held by kworker/5:0/2047:
>  #0:  ("kaluad"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #0:  ("kaluad"){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&(&pg->rtpg_work)->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
>  #1:  ((&(&pg->rtpg_work)->work)){+.+.}, at: [<ffffffff81083b85>] process_one_work+0x195/0x660
> 
> 
> =============================================
> 
> =============================================

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDA3OjM4ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gT24g
VGh1LCBOb3YgMDIsIDIwMTcgYXQgMDM6NTc6MDVQTSArMDAwMCwgQmFydCBWYW4gQXNzY2hlIHdy
b3RlOg0KPiA+IE9uIFdlZCwgMjAxNy0xMS0wMSBhdCAwODoyMSAtMDYwMCwgSmVucyBBeGJvZSB3
cm90ZToNCj4gPiA+IEZpeGVkIHRoYXQgdXAsIGFuZCBhcHBsaWVkIHRoZXNlIHR3byBwYXRjaGVz
IGFzIHdlbGwuDQo+ID4gDQo+ID4gSGVsbG8gSmVucywNCj4gPiANCj4gPiBSZWNlbnRseSBJIG5v
dGljZWQgdGhhdCBhIHRlc3Qgc3lzdGVtIHNwb3JhZGljYWxseSBoYW5ncyBkdXJpbmcgYm9vdCAo
RGVsbA0KPiA+IFBvd2VyRWRnZSBSNzIwIHRoYXQgYm9vdHMgZnJvbSBhIGhhcmQgZGlzayBjb25u
ZWN0ZWQgdG8gYSBNZWdhUkFJRCBTQVMgYWRhcHRlcikNCj4gPiBhbmQgYWxzbyB0aGF0IHNycC10
ZXN0cyBzeXN0ZW1hdGljYWxseSBoYW5ncy4gUmV2ZXJ0aW5nIHRoZSB0d28gcGF0Y2hlcyBmcm9t
DQo+ID4gdGhpcyBzZXJpZXMgZml4ZXMgYm90aCBpc3N1ZXMuIEknbSBub3Qgc3VyZSB0aGVyZSBp
cyBhbm90aGVyIHNvbHV0aW9uIHRoYW4NCj4gPiByZXZlcnRpbmcgdGhlIHR3byBwYXRjaGVzIGZy
b20gdGhpcyBzZXJpZXMuDQo+IA0KPiBUaGVuIHdlIG5lZWQgdG8gZmluZCB0aGUgcm9vdCBjYXVz
ZSwgaW5zdGVhZCBvZiB1c2luZyBzb3J0IG9mIHdvcmthcm91bmQNCj4gYXMgYmVmb3JlLg0KPiAN
Cj4gRm9yIFNDU0ksIHRoZSByZXN0YXJ0IGlzIGFsd2F5cyBydW4gZnJvbSBzY3NpX2VuZF9yZXF1
ZXN0KCksIGFuZCB0aGlzDQo+IGtpbmQgb2YgcmVzdGFydCBmcm9tIGFsbCBoY3R4IGlzbid0IG5l
Y2Vzc2FyeSBhdCBhbGwuDQoNClllcywgd2UgbmVlZCB0byBmaW5kIHRoZSByb290IGNhdXNlLCBi
dXQgYWZ0ZXIgdGhlc2UgdHdvIHBhdGNoZXMgaGF2ZSBlaXRoZXINCmJlZW4gcmV2ZXJ0ZWQgb3Ig
aGF2ZSBiZWVuIHJlbW92ZWQgZnJvbSB0aGUgZm9yLTQuMTUgcXVldWUuIE90aGVyd2lzZSBpZiBh
DQpiaXNlY3QgbGFuZHMgaW4gdGhlIG1pZGRsZSBvZiB0aGVzZSB0d28gcGF0Y2hlcyBhbmQgdGhl
IGZpeCBmb3IgdGhlc2UgcGF0Y2hlcw0KdGhhdCB3b3VsZCBiZSB2ZXJ5IGZydXN0cmF0aW5nIGZv
ciB0aGUgcGVyc29uIHdobyBpcyBydW5uaW5nIHRoZSBiaXNlY3QuDQoNCkJhcnQu

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Thu, Nov 02, 2017 at 11:43:55PM +0000, Bart Van Assche wrote:
> On Fri, 2017-11-03 at 07:38 +0800, Ming Lei wrote:
> > On Thu, Nov 02, 2017 at 03:57:05PM +0000, Bart Van Assche wrote:
> > > On Wed, 2017-11-01 at 08:21 -0600, Jens Axboe wrote:
> > > > Fixed that up, and applied these two patches as well.
> > > 
> > > Hello Jens,
> > > 
> > > Recently I noticed that a test system sporadically hangs during boot (Dell
> > > PowerEdge R720 that boots from a hard disk connected to a MegaRAID SAS adapter)
> > > and also that srp-tests systematically hangs. Reverting the two patches from
> > > this series fixes both issues. I'm not sure there is another solution than
> > > reverting the two patches from this series.
> > 
> > Then we need to find the root cause, instead of using sort of workaround
> > as before.
> > 
> > For SCSI, the restart is always run from scsi_end_request(), and this
> > kind of restart from all hctx isn't necessary at all.
> 
> Yes, we need to find the root cause, but after these two patches have either
> been reverted or have been removed from the for-4.15 queue. Otherwise if a
> bisect lands in the middle of these two patches and the fix for these patches
> that would be very frustrating for the person who is running the bisect.

Could you please share your srp-tests script? I may find a IB/SRP system
to see if I can reproduce this issue and figure out one solution.

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDA3OjQ4ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gQ291
bGQgeW91IHBsZWFzZSBzaGFyZSB5b3VyIHNycC10ZXN0cyBzY3JpcHQ/IEkgbWF5IGZpbmQgYSBJ
Qi9TUlAgc3lzdGVtDQo+IHRvIHNlZSBpZiBJIGNhbiByZXByb2R1Y2UgdGhpcyBpc3N1ZSBhbmQg
ZmlndXJlIG91dCBvbmUgc29sdXRpb24uDQoNClBsZWFzZSBoYXZlIGEgbG9vayBhdCBodHRwczov
L2dpdGh1Yi5jb20vYnZhbmFzc2NoZS9zcnAtdGVzdC4NCg0KQmFydC4=

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Thu, Nov 02, 2017 at 11:54:57PM +0000, Bart Van Assche wrote:
> On Fri, 2017-11-03 at 07:48 +0800, Ming Lei wrote:
> > Could you please share your srp-tests script? I may find a IB/SRP system
> > to see if I can reproduce this issue and figure out one solution.
> 
> Please have a look at https://github.com/bvanassche/srp-test.

Could you provide me the exact command line for the reproduction?

I simply run ./run_tests, and looks it doesn't work.

[root@ srp-test]# ./run_tests
df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
Unmounting /home/ming/git/srp-test/mnt2 from
umount: /home/ming/git/srp-test/mnt2: mountpoint not found
df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
Error: unmounting /home/ming/git/srp-test/mnt2 failed


Thanks,
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDA4OjE1ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gT24g
VGh1LCBOb3YgMDIsIDIwMTcgYXQgMTE6NTQ6NTdQTSArMDAwMCwgQmFydCBWYW4gQXNzY2hlIHdy
b3RlOg0KPiA+IE9uIEZyaSwgMjAxNy0xMS0wMyBhdCAwNzo0OCArMDgwMCwgTWluZyBMZWkgd3Jv
dGU6DQo+ID4gPiBDb3VsZCB5b3UgcGxlYXNlIHNoYXJlIHlvdXIgc3JwLXRlc3RzIHNjcmlwdD8g
SSBtYXkgZmluZCBhIElCL1NSUCBzeXN0ZW0NCj4gPiA+IHRvIHNlZSBpZiBJIGNhbiByZXByb2R1
Y2UgdGhpcyBpc3N1ZSBhbmQgZmlndXJlIG91dCBvbmUgc29sdXRpb24uDQo+ID4gDQo+ID4gUGxl
YXNlIGhhdmUgYSBsb29rIGF0IGh0dHBzOi8vZ2l0aHViLmNvbS9idmFuYXNzY2hlL3NycC10ZXN0
Lg0KPiANCj4gQ291bGQgeW91IHByb3ZpZGUgbWUgdGhlIGV4YWN0IGNvbW1hbmQgbGluZSBmb3Ig
dGhlIHJlcHJvZHVjdGlvbj8NCj4gDQo+IEkgc2ltcGx5IHJ1biAuL3J1bl90ZXN0cywgYW5kIGxv
b2tzIGl0IGRvZXNuJ3Qgd29yay4NCj4gDQo+IFtyb290QCBzcnAtdGVzdF0jIC4vcnVuX3Rlc3Rz
DQo+IGRmOiDigJgvaG9tZS9taW5nL2dpdC9zcnAtdGVzdC9tbnQy4oCZOiBObyBzdWNoIGZpbGUg
b3IgZGlyZWN0b3J5DQo+IGRmOiDigJgvaG9tZS9taW5nL2dpdC9zcnAtdGVzdC9tbnQy4oCZOiBO
byBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5DQo+IFVubW91bnRpbmcgL2hvbWUvbWluZy9naXQvc3Jw
LXRlc3QvbW50MiBmcm9tDQo+IHVtb3VudDogL2hvbWUvbWluZy9naXQvc3JwLXRlc3QvbW50Mjog
bW91bnRwb2ludCBub3QgZm91bmQNCj4gZGY6IOKAmC9ob21lL21pbmcvZ2l0L3NycC10ZXN0L21u
dDLigJk6IE5vIHN1Y2ggZmlsZSBvciBkaXJlY3RvcnkNCj4gRXJyb3I6IHVubW91bnRpbmcgL2hv
bWUvbWluZy9naXQvc3JwLXRlc3QvbW50MiBmYWlsZWQNCg0KVGhpcyBtZWFucyB0aGF0IEkgaGF2
ZSB0byBhZGQgdGhlIGZvbGxvd2luZyB0byB0aGUgUkVBRE1FIGRvY3VtZW50Og0KDQpSdW4gIm1r
ZGlyIG1udDEgbW50MiBtbnQzIiBiZWZvcmUgc3RhcnRpbmcgdGhlIHRlc3RzLg0KDQpCYXJ0Lg==

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Laurence Oberman]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

Hi Ming
I have used Bart's tests on my test bed they should run fine.
Are you using my SRP setup.

Thanks
Laurence


On Nov 2, 2017 8:48 PM, "Bart Van Assche" <Bart.VanAssche@wdc.com> wrote:

> On Fri, 2017-11-03 at 08:15 +0800, Ming Lei wrote:
> > On Thu, Nov 02, 2017 at 11:54:57PM +0000, Bart Van Assche wrote:
> > > On Fri, 2017-11-03 at 07:48 +0800, Ming Lei wrote:
> > > > Could you please share your srp-tests script? I may find a IB/SRP
> system
> > > > to see if I can reproduce this issue and figure out one solution.
> > >
> > > Please have a look at https://github.com/bvanassche/srp-test.
> >
> > Could you provide me the exact command line for the reproduction?
> >
> > I simply run ./run_tests, and looks it doesn't work.
> >
> > [root@ srp-test]# ./run_tests
> > df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
> > df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
> > Unmounting /home/ming/git/srp-test/mnt2 from
> > umount: /home/ming/git/srp-test/mnt2: mountpoint not found
> > df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
> > Error: unmounting /home/ming/git/srp-test/mnt2 failed
>
> This means that I have to add the following to the README document:
>
> Run "mkdir mnt1 mnt2 mnt3" before starting the tests.
>
> Bart.

[-- Attachment #2: Type: text/html, Size: 1984 bytes --]

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

Hi Laurence,

On Thu, Nov 02, 2017 at 09:16:04PM -0400, Laurence Oberman wrote:
> Hi Ming
> I have used Bart's tests on my test bed they should run fine.
> Are you using my SRP setup.

Yeah, I am using your SRP setup.

Once the three directories are created, I saw the new failure,
and both ib/srp and dm-mpath has been working well now.

Laurence and Bart, do you have receipt for making the test
working first?

[root@ibclient srp-test]# ./run_tests
modprobe: FATAL: Module target_core_mod is in use.
[root@ibclient srp-test]# lsmod | grep target
iscsi_target_mod      294912  1 ib_isert
target_core_mod       352256  2 iscsi_target_mod,ib_isert
[root@ibclient srp-test]# lsblk | head -n 10
NAME                                   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdy                                     65:128  0   3.9G  0 disk
└─36001405b2b5c6c24c084b6fa4d55da2f    253:16   0   3.9G  0 mpath
sdam                                    66:96   0   3.9G  0 disk
└─360014055f563ebf2abc4609a00a16794    253:11   0   3.9G  0 mpath
sdbb                                    67:80   0   3.9G  0 disk
└─36001405f58163b828714988bd2e88358    253:17   0   3.9G  0 mpath
sdf                                      8:80   0   3.9G  0 disk
└─36001405fb1402260c1346688f23d07e0    253:7    0   3.9G  0 mpath
sdau                                    66:224  0   3.9G  0 disk

Thanks,
Ming
> 
> Thanks
> Laurence
> 
> 
> On Nov 2, 2017 8:48 PM, "Bart Van Assche" <Bart.VanAssche@wdc.com> wrote:
> 
> > On Fri, 2017-11-03 at 08:15 +0800, Ming Lei wrote:
> > > On Thu, Nov 02, 2017 at 11:54:57PM +0000, Bart Van Assche wrote:
> > > > On Fri, 2017-11-03 at 07:48 +0800, Ming Lei wrote:
> > > > > Could you please share your srp-tests script? I may find a IB/SRP
> > system
> > > > > to see if I can reproduce this issue and figure out one solution.
> > > >
> > > > Please have a look at https://github.com/bvanassche/srp-test.
> > >
> > > Could you provide me the exact command line for the reproduction?
> > >
> > > I simply run ./run_tests, and looks it doesn't work.
> > >
> > > [root@ srp-test]# ./run_tests
> > > df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
> > > df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
> > > Unmounting /home/ming/git/srp-test/mnt2 from
> > > umount: /home/ming/git/srp-test/mnt2: mountpoint not found
> > > df: ‘/home/ming/git/srp-test/mnt2’: No such file or directory
> > > Error: unmounting /home/ming/git/srp-test/mnt2 failed
> >
> > This means that I have to add the following to the README document:
> >
> > Run "mkdir mnt1 mnt2 mnt3" before starting the tests.
> >
> > Bart.

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDEwOjEyICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gW3Jv
b3RAaWJjbGllbnQgc3JwLXRlc3RdIyAuL3J1bl90ZXN0cw0KPiBtb2Rwcm9iZTogRkFUQUw6IE1v
ZHVsZSB0YXJnZXRfY29yZV9tb2QgaXMgaW4gdXNlLg0KDQpMSU8gbXVzdCBiZSB1bmxvYWRlZCBi
ZWZvcmUgc3JwLXRlc3Qgc29mdHdhcmUgaXMgc3RhcnRlZC4NCg0KQmFydC4=

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Fri, Nov 03, 2017 at 02:42:50AM +0000, Bart Van Assche wrote:
> On Fri, 2017-11-03 at 10:12 +0800, Ming Lei wrote:
> > [root@ibclient srp-test]# ./run_tests
> > modprobe: FATAL: Module target_core_mod is in use.
> 
> LIO must be unloaded before srp-test software is started.

Yeah, I can make this test kick off after running
'modprobe -fr ib_isert' first, but looks all tests failed without
any hang, could you check if that is the expected result?

	https://pastebin.com/VXe66Jpg

Also could you provide some output from debugfs when this hang happens?
such as:

	dispatch/busy/.. under hctxN

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Fri, Nov 03, 2017 at 02:42:50AM +0000, Bart Van Assche wrote:
> On Fri, 2017-11-03 at 10:12 +0800, Ming Lei wrote:
> > [root@ibclient srp-test]# ./run_tests
> > modprobe: FATAL: Module target_core_mod is in use.
> 
> LIO must be unloaded before srp-test software is started.

Hi Bart,

Even with help of Laurence, we still can't setup your srp-test
in our test environment today.

But we have run Laurence's usual 3 tests on IB/SRP with/without all
my following patches against V4.14-rc4, looks everything is fine, and
no I/O hang is observed.

        0001-blk-mq-sched-dispatch-from-scheduler-IFF-progress-is.patch
        0002-blk-mq-sched-move-actual-dispatching-into-one-helper.patch
        0003-sbitmap-introduce-__sbitmap_for_each_set.patch
        0004-block-kyber-check-if-there-are-requests-in-ctx-in-ky.patch
        0005-blk-mq-introduce-.get_budget-and-.put_budget-in-blk_.patch
        0006-blk-mq-sched-improve-dispatching-from-sw-queue.patch
        0007-scsi-allow-passing-in-null-rq-to-scsi_prep_state_che.patch
        0008-scsi-implement-.get_budget-and-.put_budget-for-blk-m.patch
        0009-blk-mq-don-t-handle-TAG_SHARED-in-restart.patch
        0010-blk-mq-don-t-restart-queue-when-.get_budget-returns-.patch

BTW, Laurence found there is kernel crash in his IB/SRP test when running
for-next branch of block tree, so we just test v4.14-rc4 w/wo my blk-mq patches.

And I looked at the SCSI's queue_rq code for a while, and only found
one issue which may cause IO hang, and the following patch may address
this issue, but not sure if it is same with your issue. Could you apply
this patch and see if your issue can be fixed?

BTW, it should be helpful to check the blk-mq debugfs related files
when your I/O hang happens, could you provide that info?

--

>From edcb243d9a6f3446bd9a9f95c00bed7616dd7368 Mon Sep 17 00:00:00 2001
From: Ming Lei <ming.lei@redhat.com>
Date: Fri, 3 Nov 2017 12:11:59 +0800
Subject: [PATCH] SCSI_MQ: fix IO hang in case of queue busy

We have to insert the rq back before checking .device_busy,
otherwise When IO completes just after the check and before
this req is added to hctx->dispatch, this queue may never get
chance to be run, then this IO may hang forever.

This patch introduces BLK_STS_RESOURCE_OK for handling this
issue.

Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 block/blk-mq.c            | 17 +++++++++++++++++
 drivers/scsi/scsi_lib.c   |  8 ++++++++
 include/linux/blk-mq.h    |  1 +
 include/linux/blk_types.h |  1 +
 4 files changed, 27 insertions(+)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index e4d2490f4e7e..e1e03576edca 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -660,6 +660,16 @@ static void __blk_mq_requeue_request(struct request *rq)
 	}
 }
 
+void blk_mq_reinsert_request_hctx(struct blk_mq_hw_ctx *hctx, struct request *rq)
+{
+	__blk_mq_requeue_request(rq);
+
+	spin_lock(&hctx->lock);
+	list_add_tail(&rq->queuelist, &hctx->dispatch);
+	spin_unlock(&hctx->lock);
+}
+EXPORT_SYMBOL(blk_mq_reinsert_request_hctx);
+
 void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list)
 {
 	__blk_mq_requeue_request(rq);
@@ -1165,6 +1175,12 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list,
 			list_add(&rq->queuelist, list);
 			__blk_mq_requeue_request(rq);
 			break;
+		} else if (ret == BLK_STS_RESOURCE_OK) {
+			/*
+			 * BLK_STS_RESOURCE_OK means driver handled this
+			 * STS_RESOURCE already, we just need to stop dispatch.
+			 */
+			break;
 		}
 
  fail_rq:
@@ -1656,6 +1672,7 @@ static void __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
 	ret = q->mq_ops->queue_rq(hctx, &bd);
 	switch (ret) {
 	case BLK_STS_OK:
+	case BLK_STS_RESOURCE_OK:
 		*cookie = new_cookie;
 		return;
 	case BLK_STS_RESOURCE:
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 7f218ef61900..0165c1caed82 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -2030,9 +2030,17 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 	case BLK_STS_OK:
 		break;
 	case BLK_STS_RESOURCE:
+		/*
+		 * We have to insert the rq back before checking .device_busy,
+		 * otherwise when IO completes just after the check and before
+		 * this req is added to hctx->dispatch, this queue may never get
+		 * chance to be run, then this IO may hang forever.
+		 */
+		blk_mq_reinsert_request_hctx(hctx, req);
 		if (atomic_read(&sdev->device_busy) == 0 &&
 		    !scsi_device_blocked(sdev))
 			blk_mq_delay_run_hw_queue(hctx, SCSI_QUEUE_DELAY);
+		ret = BLK_STS_RESOURCE_OK;
 		break;
 	default:
 		/*
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index e5e6becd57d3..4740f643d8c5 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -244,6 +244,7 @@ void blk_mq_start_request(struct request *rq);
 void blk_mq_end_request(struct request *rq, blk_status_t error);
 void __blk_mq_end_request(struct request *rq, blk_status_t error);
 
+void blk_mq_reinsert_request_hctx(struct blk_mq_hw_ctx *hctx, struct request *rq);
 void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list);
 void blk_mq_add_to_requeue_list(struct request *rq, bool at_head,
 				bool kick_requeue_list);
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 3385c89f402e..b630cc026a93 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -32,6 +32,7 @@ typedef u8 __bitwise blk_status_t;
 #define BLK_STS_PROTECTION	((__force blk_status_t)8)
 #define BLK_STS_RESOURCE	((__force blk_status_t)9)
 #define BLK_STS_IOERR		((__force blk_status_t)10)
+#define BLK_STS_RESOURCE_OK	((__force blk_status_t)11)
 
 /* hack for device mapper, don't use elsewhere: */
 #define BLK_STS_DM_REQUEUE    ((__force blk_status_t)11)
-- 
2.9.5


-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDExOjUwICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gT24g
RnJpLCBOb3YgMDMsIDIwMTcgYXQgMDI6NDI6NTBBTSArMDAwMCwgQmFydCBWYW4gQXNzY2hlIHdy
b3RlOg0KPiA+IE9uIEZyaSwgMjAxNy0xMS0wMyBhdCAxMDoxMiArMDgwMCwgTWluZyBMZWkgd3Jv
dGU6DQo+ID4gPiBbcm9vdEBpYmNsaWVudCBzcnAtdGVzdF0jIC4vcnVuX3Rlc3RzDQo+ID4gPiBt
b2Rwcm9iZTogRkFUQUw6IE1vZHVsZSB0YXJnZXRfY29yZV9tb2QgaXMgaW4gdXNlLg0KPiA+IA0K
PiA+IExJTyBtdXN0IGJlIHVubG9hZGVkIGJlZm9yZSBzcnAtdGVzdCBzb2Z0d2FyZSBpcyBzdGFy
dGVkLg0KPiANCj4gWWVhaCwgSSBjYW4gbWFrZSB0aGlzIHRlc3Qga2ljayBvZmYgYWZ0ZXIgcnVu
bmluZw0KPiAnbW9kcHJvYmUgLWZyIGliX2lzZXJ0JyBmaXJzdCwgYnV0IGxvb2tzIGFsbCB0ZXN0
cyBmYWlsZWQgd2l0aG91dA0KPiBhbnkgaGFuZywgY291bGQgeW91IGNoZWNrIGlmIHRoYXQgaXMg
dGhlIGV4cGVjdGVkIHJlc3VsdD8NCj4gDQo+IAlodHRwczovL3Bhc3RlYmluLmNvbS9WWGU2Nkpw
Zw0KDQpJIHNlZSBwbGVudHkgb2YgIi4uLiA+L3N5cy8uLi4vYWRkX3RhcmdldCBmYWlsZWQ6IENv
bm5lY3Rpb24gdGltZWQgb3V0Ig0KbWVzc2FnZXMuIFRoZSBzcnAtdGVzdCBzY3JpcHRzIHVzZSB0
aGUgbG9vcGJhY2sgZnVuY3Rpb25hbGl0eSBvZiBJQiBwb3J0cw0Kc28gb25lIG9yIG1vcmUgSUIg
cG9ydHMgbXVzdCBiZSBhY3RpdmUuIEhhdmUgeW91IGFscmVhZHkgY2hlY2tlZCB0aGUgSUINCnBv
cnRzIHN0YXRlLCBlLmcuIGJ5IHJ1bm5pbmcgaWJ2X2RldmluZm8/DQoNCj4gQWxzbyBjb3VsZCB5
b3UgcHJvdmlkZSBzb21lIG91dHB1dCBmcm9tIGRlYnVnZnMgd2hlbiB0aGlzIGhhbmcgaGFwcGVu
cz8NCj4gc3VjaCBhczoNCj4gDQo+IAlkaXNwYXRjaC9idXN5Ly4uIHVuZGVyIGhjdHhODQoNClRo
aXMgaXMgdGhlIG91dHB1dCB0aGF0IGFwcGVhcnMgb24gbXkgdGVzdCBzZXR1cCBpZiB0aGUgdHdv
IHBhdGNoZXMgZnJvbQ0KdGhlIHBhdGNoIHNlcmllcyAiYmxvY2s6IHJlbW92ZSB1bm5lY2Vzc2Fy
eSBSRVNUQVJUIiBhcmUgYXBwbGllZDoNCg0KIyB+YmFydC9zb2Z0d2FyZS9pbmZpbmliYW5kL3Ny
cC10ZXN0L3J1bl90ZXN0cyAtZiB4ZnMgLWQgLWUgZGVhZGxpbmUgLXIgNjAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIA0KVW5sb2FkZWQgdGhlIGliX3NycHQga2VybmVsIG1vZHVsZQ0KWmVy
by1pbml0aWFsaXppbmcgL2Rldi9yYW0wIC4uLiBkb25lDQpaZXJvLWluaXRpYWxpemluZyAvZGV2
L3JhbTEgLi4uIGRvbmUNClplcm8taW5pdGlhbGl6aW5nIC9kZXYvZGlzay9ieS1pZC9zY3NpLTMz
MzMzMzMzMDAwMDA0NjUwIC4uLiBkb25lDQpicnctcnctLS0tIDEgcm9vdCBkaXNrIDEsIDAgTm92
ICAzIDA4OjEwIC9kZXYvcmFtMA0KL2Rldi9yYW0wDQpicnctcnctLS0tIDEgcm9vdCBkaXNrIDEs
IDEgTm92ICAzIDA4OjEwIC9kZXYvcmFtMQ0KL2Rldi9yYW0xDQpscnd4cnd4cnd4IDEgcm9vdCBy
b290IDkgTm92ICAzIDA4OjExIC9kZXYvZGlzay9ieS1pZC9zY3NpLTMzMzMzMzMzMDAwMDA0NjUw
IC0+IC4uLy4uL3NkaQ0KL2Rldi9zZGkNCkNvbmZpZ3VyZWQgU1JQIHRhcmdldCBkcml2ZXINClJ1
bm5pbmcgdGVzdCAvaG9tZS9iYXJ0L3NvZnR3YXJlL2luZmluaWJhbmQvc3JwLXRlc3QvdGVzdHMv
MDEgLi4uDQppZF9leHQ9MHgwMDAyYzkwMzAwYTAyZDUwLGlvY19ndWlkPTB4MDAwMmM5MDMwMGEw
MmQ1MCxkZ2lkPWZlODAwMDAwMDAwMDAwMDAwMDAyYzkwMzAwYTAyZDUxLHBrZXk9N2ZmZixzZXJ2
aWNlX2lkPTB4MDAwMmM5MDMwMGEwMmQ1MCwgPi9zeXMvY2xhc3MvaW5maW5pYmFuZF9zcnAvDQpz
cnAtbWx4NF8wLTIvYWRkX3RhcmdldCBmYWlsZWQ6IEludmFsaWQgYXJndW1lbnQNCmlkX2V4dD0w
eDAwMDJjOTAzMDBhMDJkNTAsaW9jX2d1aWQ9MHgwMDAyYzkwMzAwYTAyZDUwLGRnaWQ9ZmU4MDAw
MDAwMDAwMDAwMDAwMDJjOTAzMDBhMDJkNTIscGtleT03ZmZmLHNlcnZpY2VfaWQ9MHgwMDAyYzkw
MzAwYTAyZDUwLCA+L3N5cy9jbGFzcy9pbmZpbmliYW5kX3NycC8NCnNycC1tbHg0XzAtMi9hZGRf
dGFyZ2V0IGZhaWxlZDogSW52YWxpZCBhcmd1bWVudA0KY291bnRfbHVucygpOiAzIDw+IDMNCkVy
cm9yOiB1bmxvYWRpbmcga2VybmVsIG1vZHVsZSBpYl9zcnAgZmFpbGVkDQpUZXN0IC9ob21lL2Jh
cnQvc29mdHdhcmUvaW5maW5pYmFuZC9zcnAtdGVzdC90ZXN0cy8wMSBmYWlsZWQNCiMgY2F0IC9z
eXMvbW9kdWxlL3Njc2lfbW9kL3BhcmFtZXRlcnMvdXNlX2Jsa19tcSAgDQpZDQojIGZpbmQgL3N5
cy9rZXJuZWwvZGVidWcvYmxvY2sgLW5hbWUgYnVzeSB8IHhhcmdzIGNhdA0KIyBkbWVzZyAtYyA+
L2Rldi9udWxsOyBlY2hvIHcgPiAvcHJvYy9zeXNycS10cmlnZ2VyOyBkbWVzZw0KWyAgNzA5LjMy
MTc1MF0gc3lzcnE6IFN5c1JxIDogU2hvdyBCbG9ja2VkIFN0YXRlDQpbICA3MDkuMzI3OTA2XSAg
IHRhc2sgICAgICAgICAgICAgICAgICAgICAgICBQQyBzdGFjayAgIHBpZCBmYXRoZXINClsgIDcw
OS4zMzU5MTBdIGt3b3JrZXIvNDoxICAgICBEICAgIDAgICAgNTQgICAgICAyIDB4ODAwMDAwMDAN
ClsgIDcwOS4zNDI4ODddIFdvcmtxdWV1ZTogc3JwX3JlbW92ZSBzcnBfcmVtb3ZlX3dvcmsgW2li
X3NycF0NClsgIDcwOS4zNDk4MDZdIENhbGwgVHJhY2U6DQpbICA3MDkuMzUzMzY1XSAgX19zY2hl
ZHVsZSsweDJmYS8weGJiMA0KWyAgNzA5LjM1Nzg2NV0gIHNjaGVkdWxlKzB4MzYvMHg5MA0KWyAg
NzA5LjM2MTk4NV0gIGFzeW5jX3N5bmNocm9uaXplX2Nvb2tpZV9kb21haW4rMHg4OC8weDEzMA0K
WyAgNzA5LjM2ODQyMl0gID8gZmluaXNoX3dhaXQrMHg5MC8weDkwDQpbICA3MDkuMzczMDMzXSAg
YXN5bmNfc3luY2hyb25pemVfZnVsbF9kb21haW4rMHgxOC8weDIwDQpbICA3MDkuMzc5MTg0XSAg
c2RfcmVtb3ZlKzB4NGQvMHhjMCBbc2RfbW9kXQ0KWyAgNzA5LjM4NDI4Ml0gIGRldmljZV9yZWxl
YXNlX2RyaXZlcl9pbnRlcm5hbCsweDE2MC8weDIxMA0KWyAgNzA5LjM5MDc1Ml0gIGRldmljZV9y
ZWxlYXNlX2RyaXZlcisweDEyLzB4MjANClsgIDcwOS4zOTYxMzBdICBidXNfcmVtb3ZlX2Rldmlj
ZSsweDEwMC8weDE4MA0KWyAgNzA5LjQwMTMzMl0gIGRldmljZV9kZWwrMHgxZDgvMHgzNDANClsg
IDcwOS40MDU4MjNdICBfX3Njc2lfcmVtb3ZlX2RldmljZSsweGZjLzB4MTMwDQpbICA3MDkuNDEx
MjIxXSAgc2NzaV9mb3JnZXRfaG9zdCsweDI1LzB4NzANClsgIDcwOS40MTYxMDhdICBzY3NpX3Jl
bW92ZV9ob3N0KzB4NzkvMHgxMjANClsgIDcwOS40MjExMTRdICBzcnBfcmVtb3ZlX3dvcmsrMHg5
MC8weDFkMCBbaWJfc3JwXQ0KWyAgNzA5LjQyNjg2N10gIHByb2Nlc3Nfb25lX3dvcmsrMHgyMGEv
MHg2NjANClsgIDcwOS40MzE5NzFdICB3b3JrZXJfdGhyZWFkKzB4M2QvMHgzYjANClsgIDcwOS40
MzY2NTVdICBrdGhyZWFkKzB4MTNhLzB4MTUwDQpbICA3MDkuNDQwODcyXSAgPyBwcm9jZXNzX29u
ZV93b3JrKzB4NjYwLzB4NjYwDQpbICA3MDkuNDQ2MTMzXSAgPyBrdGhyZWFkX2NyZWF0ZV9vbl9u
b2RlKzB4NDAvMHg0MA0KWyAgNzA5LjQ1MTgyNV0gIHJldF9mcm9tX2ZvcmsrMHgyNy8weDQwDQpb
ICA3MDkuNDU2NjAyXSBrd29ya2VyL3U2Njo2ICAgRCAgICAwICAgMzQxICAgICAgMiAweDgwMDAw
MDAwDQpbICA3MDkuNDYzMzcxXSBXb3JrcXVldWU6IGV2ZW50c191bmJvdW5kIGFzeW5jX3J1bl9l
bnRyeV9mbg0KWyAgNzA5LjQ2OTkxOV0gQ2FsbCBUcmFjZToNClsgIDcwOS40NzMzMjVdICBfX3Nj
aGVkdWxlKzB4MmZhLzB4YmIwDQpbICA3MDkuNDc3ODUwXSAgPyB0cmFjZV9oYXJkaXJxc19vbl9j
YWxsZXIrMHhmYi8weDE5MA0KWyAgNzA5LjQ4Mzg0N10gIHNjaGVkdWxlKzB4MzYvMHg5MA0KWyAg
NzA5LjQ4Nzk4Ml0gIHNjaGVkdWxlX3RpbWVvdXQrMHgyMmMvMHg1NzANClsgIDcwOS40OTMxMTVd
ICA/IGNhbGxfdGltZXJfZm4rMHgzMzAvMHgzMzANClsgIDcwOS40OTgxMzNdICA/IHdhaXRfZm9y
X2NvbXBsZXRpb25faW9fdGltZW91dCsweGY3LzB4MTgwDQpbICA3MDkuNTA0NzMyXSAgaW9fc2No
ZWR1bGVfdGltZW91dCsweDFlLzB4NTANClsgIDcwOS41MDk5MzhdICA/IGlvX3NjaGVkdWxlX3Rp
bWVvdXQrMHgxZS8weDUwDQpbICA3MDkuNTE1NDEzXSAgd2FpdF9mb3JfY29tcGxldGlvbl9pb190
aW1lb3V0KzB4MTFmLzB4MTgwDQpbICA3MDkuNTIxOTIwXSAgPyB3YWtlX3VwX3ErMHg4MC8weDgw
DQpbICA3MDkuNTI2Mzc2XSAgYmxrX2V4ZWN1dGVfcnErMHg4Ni8weGMwDQpbICA3MDkuNTMxMTM1
XSAgc2NzaV9leGVjdXRlKzB4ZGIvMHgxZjANClsgIDcwOS41MzYzMzZdICBzZF9yZXZhbGlkYXRl
X2Rpc2srMHhlZC8weDFjNzAgW3NkX21vZF0NClsgIDcwOS41NDMyODFdICBzZF9wcm9iZV9hc3lu
YysweGMzLzB4MWQwIFtzZF9tb2RdDQpbICA3MDkuNTQ5NTI0XSAgYXN5bmNfcnVuX2VudHJ5X2Zu
KzB4MzgvMHgxNjANClsgIDcwOS41NTU1MDNdICBwcm9jZXNzX29uZV93b3JrKzB4MjBhLzB4NjYw
DQpbICA3MDkuNTYxMjk5XSAgd29ya2VyX3RocmVhZCsweDNkLzB4M2IwDQpbICA3MDkuNTY2Njg4
XSAga3RocmVhZCsweDEzYS8weDE1MA0KWyAgNzA5LjU3MTYxM10gID8gcHJvY2Vzc19vbmVfd29y
aysweDY2MC8weDY2MA0KWyAgNzA5LjU3NzU2MV0gID8ga3RocmVhZF9jcmVhdGVfb25fbm9kZSsw
eDQwLzB4NDANClsgIDcwOS41ODM5MTBdICByZXRfZnJvbV9mb3JrKzB4MjcvMHg0MA0KWyAgNzA5
LjU4OTE3NF0ga3dvcmtlci81OjIgICAgIEQgICAgMCAgIDM4NiAgICAgIDIgMHg4MDAwMDAwMA0K
WyAgNzA5LjU5NjYxNl0gV29ya3F1ZXVlOiBrYWx1YWQgYWx1YV9ydHBnX3dvcmsgW3Njc2lfZGhf
YWx1YV0NClsgIDcwOS42MDQxMzBdIENhbGwgVHJhY2U6DQpbICA3MDkuNjA4MDk2XSAgX19zY2hl
ZHVsZSsweDJmYS8weGJiMA0KWyAgNzA5LjYxMzI2NV0gID8gdHJhY2VfaGFyZGlycXNfb25fY2Fs
bGVyKzB4ZmIvMHgxOTANClsgIDcwOS42MTk4NzldICBzY2hlZHVsZSsweDM2LzB4OTANClsgIDcw
OS42MjQ2ODRdICBzY2hlZHVsZV90aW1lb3V0KzB4MjJjLzB4NTcwDQpbICA3MDkuNjMwNDM3XSAg
PyBjYWxsX3RpbWVyX2ZuKzB4MzMwLzB4MzMwDQpbICA3MDkuNjM2MDU5XSAgPyB3YWl0X2Zvcl9j
b21wbGV0aW9uX2lvX3RpbWVvdXQrMHhmNy8weDE4MA0KWyAgNzA5LjY0MzI3MF0gIGlvX3NjaGVk
dWxlX3RpbWVvdXQrMHgxZS8weDUwDQpbICA3MDkuNjQ5MDgzXSAgPyBpb19zY2hlZHVsZV90aW1l
b3V0KzB4MWUvMHg1MA0KWyAgNzA5LjY1NTE1OV0gIHdhaXRfZm9yX2NvbXBsZXRpb25faW9fdGlt
ZW91dCsweDExZi8weDE4MA0KWyAgNzA5LjY2MjI4NF0gID8gd2FrZV91cF9xKzB4ODAvMHg4MA0K
WyAgNzA5LjY2NzMzNF0gIGJsa19leGVjdXRlX3JxKzB4ODYvMHhjMA0KWyAgNzA5LjY3MjcwNV0g
IHNjc2lfZXhlY3V0ZSsweGRiLzB4MWYwDQpbICA3MDkuNjc3OTM4XSAgYWx1YV9ydHBnX3dvcmsr
MHgyYjMvMHhlOGEgW3Njc2lfZGhfYWx1YV0NClsgIDcwOS42ODQ5ODNdICBwcm9jZXNzX29uZV93
b3JrKzB4MjBhLzB4NjYwDQpbICA3MDkuNjkwNzU5XSAgd29ya2VyX3RocmVhZCsweDNkLzB4M2Iw
DQpbICA3MDkuNjk2MTA3XSAga3RocmVhZCsweDEzYS8weDE1MA0KWyAgNzA5LjcwMDk4Nl0gID8g
cHJvY2Vzc19vbmVfd29yaysweDY2MC8weDY2MA0KWyAgNzA5LjcwNjkyMF0gID8ga3RocmVhZF9j
cmVhdGVfb25fbm9kZSsweDQwLzB4NDANClsgIDcwOS43MTMzMTZdICByZXRfZnJvbV9mb3JrKzB4
MjcvMHg0MA0KWyAgNzA5LjcxODgwOV0gc3lzdGVtZC11ZGV2ZCAgIEQgICAgMCAgMTkwNSAgICA0
NjMgMHg4MDAwMDEwNg0KWyAgNzA5LjcyNjI4NV0gQ2FsbCBUcmFjZToNClsgIDcwOS43MzAzMDld
ICBfX3NjaGVkdWxlKzB4MmZhLzB4YmIwDQpbICA3MDkuNzM1NDYyXSAgc2NoZWR1bGUrMHgzNi8w
eDkwDQpbICA3MDkuNzQwMjcwXSAgaW9fc2NoZWR1bGUrMHgxNi8weDQwDQpbICA3MDkuNzQ1MzY5
XSAgX19sb2NrX3BhZ2UrMHgxMGEvMHgxNDANClsgIDcwOS43NTA2OTFdICA/IHBhZ2VfY2FjaGVf
dHJlZV9pbnNlcnQrMHgxMDAvMHgxMDANClsgIDcwOS43NTcyODBdICB0cnVuY2F0ZV9pbm9kZV9w
YWdlc19yYW5nZSsweDRmZi8weDgwMA0KWyAgNzA5Ljc2NDA0NF0gID8gYnVmZmVyX2V4aXRfY3B1
X2RlYWQrMHg3MC8weDcwDQpbICA3MDkuNzcwMjg1XSAgPyBmaW5kX25leHRfYml0KzB4Yi8weDEw
DQpbICA3MDkuNzc1NjkxXSAgPyBjcHVtYXNrX25leHQrMHgxYS8weDIwDQpbICA3MDkuNzgxMDg5
XSAgPyBzbXBfY2FsbF9mdW5jdGlvbl9tYW55KzB4MWUzLzB4MjMwDQpbICA3MDkuNzg3NTk4XSAg
PyBvbl9lYWNoX2NwdV9tYXNrKzB4NTIvMHhhMA0KWyAgNzA5Ljc5MzM1M10gID8gbWFya19idWZm
ZXJfYXN5bmNfd3JpdGUrMHgyMC8weDIwDQpbICA3MDkuNzk5NzQxXSAgPyBidWZmZXJfZXhpdF9j
cHVfZGVhZCsweDcwLzB4NzANClsgIDcwOS44MDU4ODhdICA/IG9uX2VhY2hfY3B1X2NvbmQrMHg5
OS8weGUwDQpbICA3MDkuODExNjM2XSAgdHJ1bmNhdGVfaW5vZGVfcGFnZXMrMHgxNS8weDIwDQpb
ICA3MDkuODE3NTM2XSAga2lsbF9iZGV2KzB4MzUvMHg0MA0KWyAgNzA5LjgyMjQwMF0gIF9fYmxr
ZGV2X3B1dCsweDZkLzB4MWYwDQpbICA3MDkuODI3NjI1XSAgYmxrZGV2X3B1dCsweDRlLzB4MTMw
DQpbICA3MDkuODMyNjczXSAgYmxrZGV2X2Nsb3NlKzB4MjUvMHgzMA0KWyAgNzA5LjgzNzc3OV0g
IF9fZnB1dCsweGVkLzB4MWYwDQpbICA3MDkuODQyNDMwXSAgX19fX2ZwdXQrMHhlLzB4MTANClsg
IDcwOS44NDcwMzBdICB0YXNrX3dvcmtfcnVuKzB4OGIvMHhjMA0KWyAgNzA5Ljg1MjI1MV0gIGRv
X2V4aXQrMHgzOGQvMHhjNzANClsgIDcwOS44NTcwNDZdICA/IGdldF9zaWduYWwrMHgyZjYvMHg4
YzANClsgIDcwOS44NjIzNDddICBkb19ncm91cF9leGl0KzB4NTAvMHhkMA0KWyAgNzA5Ljg2NzUx
NV0gIGdldF9zaWduYWwrMHgyYWQvMHg4YzANClsgIDcwOS44NzI2MDFdICBkb19zaWduYWwrMHgy
OC8weDY4MA0KWyAgNzA5Ljg3NzQ1Ml0gID8gYmxrZGV2X3JlYWRfaXRlcisweDM1LzB4NDANClsg
IDcwOS44ODMxMDhdICA/IF9fdmZzX3JlYWQrMHhjYy8weDEzMA0KWyAgNzA5Ljg4ODIzN10gID8g
dmZzX3JlYWQrMHhhMC8weDE1MA0KWyAgNzA5Ljg5MzIwMF0gIGV4aXRfdG9fdXNlcm1vZGVfbG9v
cCsweDVhLzB4YTANClsgIDcwOS44OTkxMDldICBkb19zeXNjYWxsXzY0KzB4MTJlLzB4MTcwDQpb
ICA3MDkuOTA0NDc1XSAgZW50cnlfU1lTQ0FMTDY0X3Nsb3dfcGF0aCsweDI1LzB4MjUNClsgIDcw
OS45MTA3OTddIFJJUDogMDAzMzoweDdmNWRjNGUzMmQxMQ0KWyAgNzA5LjkxNTkxN10gUlNQOiAw
MDJiOjAwMDA3ZmZlYjBiMTA1ODggRUZMQUdTOiAwMDAwMDI0NiBPUklHX1JBWDogMDAwMDAwMDAw
MDAwMDAwMA0KWyAgNzA5LjkyNTU4OV0gUkFYOiBmZmZmZmZmZmZmZmZmZmZjIFJCWDogMDAwMDU2
MjMwZGE4NDQ5MCBSQ1g6IDAwMDA3ZjVkYzRlMzJkMTENClsgIDcwOS45MzQ3ODVdIFJEWDogMDAw
MDAwMDAwMDAwMDA0MCBSU0k6IDAwMDA1NjIzMGRhODQ0YjggUkRJOiAwMDAwMDAwMDAwMDAwMDA3
DQpbICA3MDkuOTQzOTg3XSBSQlA6IDAwMDA1NjIzMGRhYmMzYzAgUjA4OiAwMDAwMDAwMDAwMDAw
MDAwIFIwOTogMDAwMDU2MjMwZGE4NDQ5MA0KWyAgNzA5Ljk1MzIwMl0gUjEwOiAwMDAwMDAwMDAw
MDAwMDZmIFIxMTogMDAwMDAwMDAwMDAwMDI0NiBSMTI6IDAwMDAwMDAwOGQ1MzAwMDANClsgIDcw
OS45NjI0MTldIFIxMzogMDAwMDAwMDA4ZDUzMDA0MCBSMTQ6IDAwMDA1NjIzMGRhYmM0MTAgUjE1
OiAwMDAwMDAwMDAwMDAwMDQwDQo=

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDIzOjE4ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gU3Vi
amVjdDogW1BBVENIXSBTQ1NJX01ROiBmaXggSU8gaGFuZyBpbiBjYXNlIG9mIHF1ZXVlIGJ1c3kN
Cj4gDQo+IFdlIGhhdmUgdG8gaW5zZXJ0IHRoZSBycSBiYWNrIGJlZm9yZSBjaGVja2luZyAuZGV2
aWNlX2J1c3ksDQo+IG90aGVyd2lzZSBXaGVuIElPIGNvbXBsZXRlcyBqdXN0IGFmdGVyIHRoZSBj
aGVjayBhbmQgYmVmb3JlDQo+IHRoaXMgcmVxIGlzIGFkZGVkIHRvIGhjdHgtPmRpc3BhdGNoLCB0
aGlzIHF1ZXVlIG1heSBuZXZlciBnZXQNCj4gY2hhbmNlIHRvIGJlIHJ1biwgdGhlbiB0aGlzIElP
IG1heSBoYW5nIGZvcmV2ZXIuDQo+IA0KPiBUaGlzIHBhdGNoIGludHJvZHVjZXMgQkxLX1NUU19S
RVNPVVJDRV9PSyBmb3IgaGFuZGxpbmcgdGhpcw0KPiBpc3N1ZS4NCj4gDQo+IFNpZ25lZC1vZmYt
Ynk6IE1pbmcgTGVpIDxtaW5nLmxlaUByZWRoYXQuY29tPg0KPiAtLS0NCj4gIGJsb2NrL2Jsay1t
cS5jICAgICAgICAgICAgfCAxNyArKysrKysrKysrKysrKysrKw0KPiAgZHJpdmVycy9zY3NpL3Nj
c2lfbGliLmMgICB8ICA4ICsrKysrKysrDQo+ICBpbmNsdWRlL2xpbnV4L2Jsay1tcS5oICAgIHwg
IDEgKw0KPiAgaW5jbHVkZS9saW51eC9ibGtfdHlwZXMuaCB8ICAxICsNCj4gIDQgZmlsZXMgY2hh
bmdlZCwgMjcgaW5zZXJ0aW9ucygrKQ0KPiANCj4gZGlmZiAtLWdpdCBhL2Jsb2NrL2Jsay1tcS5j
IGIvYmxvY2svYmxrLW1xLmMNCj4gaW5kZXggZTRkMjQ5MGY0ZTdlLi5lMWUwMzU3NmVkY2EgMTAw
NjQ0DQo+IC0tLSBhL2Jsb2NrL2Jsay1tcS5jDQo+ICsrKyBiL2Jsb2NrL2Jsay1tcS5jDQo+IEBA
IC02NjAsNiArNjYwLDE2IEBAIHN0YXRpYyB2b2lkIF9fYmxrX21xX3JlcXVldWVfcmVxdWVzdChz
dHJ1Y3QgcmVxdWVzdCAqcnEpDQo+ICAJfQ0KPiAgfQ0KPiAgDQo+ICt2b2lkIGJsa19tcV9yZWlu
c2VydF9yZXF1ZXN0X2hjdHgoc3RydWN0IGJsa19tcV9od19jdHggKmhjdHgsIHN0cnVjdCByZXF1
ZXN0ICpycSkNCj4gK3sNCj4gKwlfX2Jsa19tcV9yZXF1ZXVlX3JlcXVlc3QocnEpOw0KPiArDQo+
ICsJc3Bpbl9sb2NrKCZoY3R4LT5sb2NrKTsNCj4gKwlsaXN0X2FkZF90YWlsKCZycS0+cXVldWVs
aXN0LCAmaGN0eC0+ZGlzcGF0Y2gpOw0KPiArCXNwaW5fdW5sb2NrKCZoY3R4LT5sb2NrKTsNCj4g
K30NCj4gK0VYUE9SVF9TWU1CT0woYmxrX21xX3JlaW5zZXJ0X3JlcXVlc3RfaGN0eCk7DQo+ICsN
Cj4gIHZvaWQgYmxrX21xX3JlcXVldWVfcmVxdWVzdChzdHJ1Y3QgcmVxdWVzdCAqcnEsIGJvb2wg
a2lja19yZXF1ZXVlX2xpc3QpDQo+ICB7DQo+ICAJX19ibGtfbXFfcmVxdWV1ZV9yZXF1ZXN0KHJx
KTsNCj4gQEAgLTExNjUsNiArMTE3NSwxMiBAQCBib29sIGJsa19tcV9kaXNwYXRjaF9ycV9saXN0
KHN0cnVjdCByZXF1ZXN0X3F1ZXVlICpxLCBzdHJ1Y3QgbGlzdF9oZWFkICpsaXN0LA0KPiAgCQkJ
bGlzdF9hZGQoJnJxLT5xdWV1ZWxpc3QsIGxpc3QpOw0KPiAgCQkJX19ibGtfbXFfcmVxdWV1ZV9y
ZXF1ZXN0KHJxKTsNCj4gIAkJCWJyZWFrOw0KPiArCQl9IGVsc2UgaWYgKHJldCA9PSBCTEtfU1RT
X1JFU09VUkNFX09LKSB7DQo+ICsJCQkvKg0KPiArCQkJICogQkxLX1NUU19SRVNPVVJDRV9PSyBt
ZWFucyBkcml2ZXIgaGFuZGxlZCB0aGlzDQo+ICsJCQkgKiBTVFNfUkVTT1VSQ0UgYWxyZWFkeSwg
d2UganVzdCBuZWVkIHRvIHN0b3AgZGlzcGF0Y2guDQo+ICsJCQkgKi8NCj4gKwkJCWJyZWFrOw0K
PiAgCQl9DQo+ICANCj4gICBmYWlsX3JxOg0KPiBAQCAtMTY1Niw2ICsxNjcyLDcgQEAgc3RhdGlj
IHZvaWQgX19ibGtfbXFfdHJ5X2lzc3VlX2RpcmVjdGx5KHN0cnVjdCBibGtfbXFfaHdfY3R4ICpo
Y3R4LA0KPiAgCXJldCA9IHEtPm1xX29wcy0+cXVldWVfcnEoaGN0eCwgJmJkKTsNCj4gIAlzd2l0
Y2ggKHJldCkgew0KPiAgCWNhc2UgQkxLX1NUU19PSzoNCj4gKwljYXNlIEJMS19TVFNfUkVTT1VS
Q0VfT0s6DQo+ICAJCSpjb29raWUgPSBuZXdfY29va2llOw0KPiAgCQlyZXR1cm47DQo+ICAJY2Fz
ZSBCTEtfU1RTX1JFU09VUkNFOg0KPiBkaWZmIC0tZ2l0IGEvZHJpdmVycy9zY3NpL3Njc2lfbGli
LmMgYi9kcml2ZXJzL3Njc2kvc2NzaV9saWIuYw0KPiBpbmRleCA3ZjIxOGVmNjE5MDAuLjAxNjVj
MWNhZWQ4MiAxMDA2NDQNCj4gLS0tIGEvZHJpdmVycy9zY3NpL3Njc2lfbGliLmMNCj4gKysrIGIv
ZHJpdmVycy9zY3NpL3Njc2lfbGliLmMNCj4gQEAgLTIwMzAsOSArMjAzMCwxNyBAQCBzdGF0aWMg
YmxrX3N0YXR1c190IHNjc2lfcXVldWVfcnEoc3RydWN0IGJsa19tcV9od19jdHggKmhjdHgsDQo+
ICAJY2FzZSBCTEtfU1RTX09LOg0KPiAgCQlicmVhazsNCj4gIAljYXNlIEJMS19TVFNfUkVTT1VS
Q0U6DQo+ICsJCS8qDQo+ICsJCSAqIFdlIGhhdmUgdG8gaW5zZXJ0IHRoZSBycSBiYWNrIGJlZm9y
ZSBjaGVja2luZyAuZGV2aWNlX2J1c3ksDQo+ICsJCSAqIG90aGVyd2lzZSB3aGVuIElPIGNvbXBs
ZXRlcyBqdXN0IGFmdGVyIHRoZSBjaGVjayBhbmQgYmVmb3JlDQo+ICsJCSAqIHRoaXMgcmVxIGlz
IGFkZGVkIHRvIGhjdHgtPmRpc3BhdGNoLCB0aGlzIHF1ZXVlIG1heSBuZXZlciBnZXQNCj4gKwkJ
ICogY2hhbmNlIHRvIGJlIHJ1biwgdGhlbiB0aGlzIElPIG1heSBoYW5nIGZvcmV2ZXIuDQo+ICsJ
CSAqLw0KPiArCQlibGtfbXFfcmVpbnNlcnRfcmVxdWVzdF9oY3R4KGhjdHgsIHJlcSk7DQo+ICAJ
CWlmIChhdG9taWNfcmVhZCgmc2Rldi0+ZGV2aWNlX2J1c3kpID09IDAgJiYNCj4gIAkJICAgICFz
Y3NpX2RldmljZV9ibG9ja2VkKHNkZXYpKQ0KPiAgCQkJYmxrX21xX2RlbGF5X3J1bl9od19xdWV1
ZShoY3R4LCBTQ1NJX1FVRVVFX0RFTEFZKTsNCj4gKwkJcmV0ID0gQkxLX1NUU19SRVNPVVJDRV9P
SzsNCj4gIAkJYnJlYWs7DQo+ICAJZGVmYXVsdDoNCj4gIAkJLyoNCj4gZGlmZiAtLWdpdCBhL2lu
Y2x1ZGUvbGludXgvYmxrLW1xLmggYi9pbmNsdWRlL2xpbnV4L2Jsay1tcS5oDQo+IGluZGV4IGU1
ZTZiZWNkNTdkMy4uNDc0MGY2NDNkOGM1IDEwMDY0NA0KPiAtLS0gYS9pbmNsdWRlL2xpbnV4L2Js
ay1tcS5oDQo+ICsrKyBiL2luY2x1ZGUvbGludXgvYmxrLW1xLmgNCj4gQEAgLTI0NCw2ICsyNDQs
NyBAQCB2b2lkIGJsa19tcV9zdGFydF9yZXF1ZXN0KHN0cnVjdCByZXF1ZXN0ICpycSk7DQo+ICB2
b2lkIGJsa19tcV9lbmRfcmVxdWVzdChzdHJ1Y3QgcmVxdWVzdCAqcnEsIGJsa19zdGF0dXNfdCBl
cnJvcik7DQo+ICB2b2lkIF9fYmxrX21xX2VuZF9yZXF1ZXN0KHN0cnVjdCByZXF1ZXN0ICpycSwg
YmxrX3N0YXR1c190IGVycm9yKTsNCj4gIA0KPiArdm9pZCBibGtfbXFfcmVpbnNlcnRfcmVxdWVz
dF9oY3R4KHN0cnVjdCBibGtfbXFfaHdfY3R4ICpoY3R4LCBzdHJ1Y3QgcmVxdWVzdCAqcnEpOw0K
PiAgdm9pZCBibGtfbXFfcmVxdWV1ZV9yZXF1ZXN0KHN0cnVjdCByZXF1ZXN0ICpycSwgYm9vbCBr
aWNrX3JlcXVldWVfbGlzdCk7DQo+ICB2b2lkIGJsa19tcV9hZGRfdG9fcmVxdWV1ZV9saXN0KHN0
cnVjdCByZXF1ZXN0ICpycSwgYm9vbCBhdF9oZWFkLA0KPiAgCQkJCWJvb2wga2lja19yZXF1ZXVl
X2xpc3QpOw0KPiBkaWZmIC0tZ2l0IGEvaW5jbHVkZS9saW51eC9ibGtfdHlwZXMuaCBiL2luY2x1
ZGUvbGludXgvYmxrX3R5cGVzLmgNCj4gaW5kZXggMzM4NWM4OWY0MDJlLi5iNjMwY2MwMjZhOTMg
MTAwNjQ0DQo+IC0tLSBhL2luY2x1ZGUvbGludXgvYmxrX3R5cGVzLmgNCj4gKysrIGIvaW5jbHVk
ZS9saW51eC9ibGtfdHlwZXMuaA0KPiBAQCAtMzIsNiArMzIsNyBAQCB0eXBlZGVmIHU4IF9fYml0
d2lzZSBibGtfc3RhdHVzX3Q7DQo+ICAjZGVmaW5lIEJMS19TVFNfUFJPVEVDVElPTgkoKF9fZm9y
Y2UgYmxrX3N0YXR1c190KTgpDQo+ICAjZGVmaW5lIEJMS19TVFNfUkVTT1VSQ0UJKChfX2ZvcmNl
IGJsa19zdGF0dXNfdCk5KQ0KPiAgI2RlZmluZSBCTEtfU1RTX0lPRVJSCQkoKF9fZm9yY2UgYmxr
X3N0YXR1c190KTEwKQ0KPiArI2RlZmluZSBCTEtfU1RTX1JFU09VUkNFX09LCSgoX19mb3JjZSBi
bGtfc3RhdHVzX3QpMTEpDQo+ICANCj4gIC8qIGhhY2sgZm9yIGRldmljZSBtYXBwZXIsIGRvbid0
IHVzZSBlbHNld2hlcmU6ICovDQo+ICAjZGVmaW5lIEJMS19TVFNfRE1fUkVRVUVVRSAgICAoKF9f
Zm9yY2UgYmxrX3N0YXR1c190KTExKQ0KDQpSdW5uaW5nIHRlc3Qgb25lIHdpdGggdGhpcyBwYXRj
aCBhcHBsaWVkIG9uIHRvcCBvZiBKZW5zJyBmb3ItbmV4dCBicmFuY2ggeWllbGRzDQp0aGUgc2Ft
ZSByZXN1bHQgYXMgd2l0aG91dCB0aGlzIHBhdGNoOg0KWyAuLi4gXQ0KRXJyb3I6IHVubG9hZGlu
ZyBrZXJuZWwgbW9kdWxlIGliX3NycCBmYWlsZWQNClRlc3QgL2hvbWUvYmFydC9zb2Z0d2FyZS9p
bmZpbmliYW5kL3NycC10ZXN0L3Rlc3RzLzAxIGZhaWxlZA0KIyBkbWVzZyAtYyA+L2Rldi9udWxs
OyBlY2hvIHcgPiAvcHJvYy9zeXNycS10cmlnZ2VyOyBkbWVzZw0KWyAgMzI1LjE2NzI5NV0gc3lz
cnE6IFN5c1JxIDogU2hvdyBCbG9ja2VkIFN0YXRlDQpbICAzMjUuMTczNjEyXSAgIHRhc2sgICAg
ICAgICAgICAgICAgICAgICAgICBQQyBzdGFjayAgIHBpZCBmYXRoZXINClsgIDMyNS4xODE5MDNd
IGt3b3JrZXIvMTA6MSAgICBEICAgIDAgICAxNjUgICAgICAyIDB4ODAwMDAwMDANClsgIDMyNS4x
ODkzOTZdIFdvcmtxdWV1ZTogc3JwX3JlbW92ZSBzcnBfcmVtb3ZlX3dvcmsgW2liX3NycF0NClsg
IDMyNS4xOTYyMjJdIENhbGwgVHJhY2U6DQpbICAzMjUuMTk5NjIyXSAgX19zY2hlZHVsZSsweDJm
YS8weGJiMA0KWyAgMzI1LjIwNDIyN10gIHNjaGVkdWxlKzB4MzYvMHg5MA0KWyAgMzI1LjIwODM5
MF0gIGFzeW5jX3N5bmNocm9uaXplX2Nvb2tpZV9kb21haW4rMHg4OC8weDEzMA0KWyAgMzI1LjIx
NDkxNV0gID8gZmluaXNoX3dhaXQrMHg5MC8weDkwDQpbICAzMjUuMjE5NTc0XSAgYXN5bmNfc3lu
Y2hyb25pemVfZnVsbF9kb21haW4rMHgxOC8weDIwDQpbICAzMjUuMjI1ODE2XSAgc2RfcmVtb3Zl
KzB4NGQvMHhjMCBbc2RfbW9kXQ0KWyAgMzI1LjIzMDk0OV0gIGRldmljZV9yZWxlYXNlX2RyaXZl
cl9pbnRlcm5hbCsweDE2MC8weDIxMA0KWyAgMzI1LjIzNzQ3OV0gIGRldmljZV9yZWxlYXNlX2Ry
aXZlcisweDEyLzB4MjANClsgIDMyNS4yNDI5MDRdICBidXNfcmVtb3ZlX2RldmljZSsweDEwMC8w
eDE4MA0KWyAgMzI1LjI0ODE2M10gIGRldmljZV9kZWwrMHgxZDgvMHgzNDANClsgIDMyNS4yNTI3
MzVdICBfX3Njc2lfcmVtb3ZlX2RldmljZSsweGZjLzB4MTMwDQpbICAzMjUuMjU4MTkyXSAgc2Nz
aV9mb3JnZXRfaG9zdCsweDI1LzB4NzANClsgIDMyNS4yNjMxNzNdICBzY3NpX3JlbW92ZV9ob3N0
KzB4NzkvMHgxMjANClsgIDMyNS4yNjgyMjddICBzcnBfcmVtb3ZlX3dvcmsrMHg5MC8weDFkMCBb
aWJfc3JwXQ0KWyAgMzI1LjI3NDA1OV0gIHByb2Nlc3Nfb25lX3dvcmsrMHgyMGEvMHg2NjANClsg
IDMyNS4yNzkyNDNdICB3b3JrZXJfdGhyZWFkKzB4M2QvMHgzYjANClsgIDMyNS4yODQwMDRdICBr
dGhyZWFkKzB4MTNhLzB4MTUwDQpbICAzMjUuMjg4MjMyXSAgPyBwcm9jZXNzX29uZV93b3JrKzB4
NjYwLzB4NjYwDQpbICAzMjUuMjkzNTc1XSAgPyBrdGhyZWFkX2NyZWF0ZV9vbl9ub2RlKzB4NDAv
MHg0MA0KWyAgMzI1LjI5OTMzNF0gIHJldF9mcm9tX2ZvcmsrMHgyNy8weDQwDQpbICAzMjUuMzA0
MDAyXSBrd29ya2VyLzU6MSAgICAgRCAgICAwICAgMTgwICAgICAgMiAweDgwMDAwMDAwDQpbICAz
MjUuMzEwNzY4XSBXb3JrcXVldWU6IGthbHVhZCBhbHVhX3J0cGdfd29yayBbc2NzaV9kaF9hbHVh
XQ0KWyAgMzI1LjMxNzY3NF0gQ2FsbCBUcmFjZToNClsgIDMyNS4zMjEwNDddICBfX3NjaGVkdWxl
KzB4MmZhLzB4YmIwDQpbICAzMjUuMzI1NjE1XSAgPyB0cmFjZV9oYXJkaXJxc19vbl9jYWxsZXIr
MHhmYi8weDE5MA0KWyAgMzI1LjMzMTYxMl0gIHNjaGVkdWxlKzB4MzYvMHg5MA0KWyAgMzI1LjMz
NTc5N10gIHNjaGVkdWxlX3RpbWVvdXQrMHgyMmMvMHg1NzANClsgIDMyNS4zNDA5MTddICA/IGNh
bGxfdGltZXJfZm4rMHgzMzAvMHgzMzANClsgIDMyNS4zNDU5NzJdICA/IHdhaXRfZm9yX2NvbXBs
ZXRpb25faW9fdGltZW91dCsweGY3LzB4MTgwDQpbICAzMjUuMzUyNTU4XSAgaW9fc2NoZWR1bGVf
dGltZW91dCsweDFlLzB4NTANClsgIDMyNS4zNTc4MDhdICA/IGlvX3NjaGVkdWxlX3RpbWVvdXQr
MHgxZS8weDUwDQpbICAzMjUuMzYzMjYyXSAgd2FpdF9mb3JfY29tcGxldGlvbl9pb190aW1lb3V0
KzB4MTFmLzB4MTgwDQpbICAzMjUuMzY5NzkzXSAgPyB3YWtlX3VwX3ErMHg4MC8weDgwDQpbICAz
MjUuMzc0MjY3XSAgYmxrX2V4ZWN1dGVfcnErMHg4Ni8weGMwDQpbICAzMjUuMzc5MDEyXSAgc2Nz
aV9leGVjdXRlKzB4ZGIvMHgxZjANClsgIDMyNS4zODM2NzhdICBhbHVhX3J0cGdfd29yaysweDJi
My8weGU4YSBbc2NzaV9kaF9hbHVhXQ0KWyAgMzI1LjM5MDEzMF0gIHByb2Nlc3Nfb25lX3dvcmsr
MHgyMGEvMHg2NjANClsgIDMyNS4zOTU4MThdICB3b3JrZXJfdGhyZWFkKzB4M2QvMHgzYjANClsg
IDMyNS40MDEyNDhdICBrdGhyZWFkKzB4MTNhLzB4MTUwDQpbICAzMjUuNDA2MDU2XSAgPyBwcm9j
ZXNzX29uZV93b3JrKzB4NjYwLzB4NjYwDQpbICAzMjUuNDEyMDcwXSAgPyBrdGhyZWFkX2NyZWF0
ZV9vbl9ub2RlKzB4NDAvMHg0MA0KWyAgMzI1LjQxODQ3NF0gIHJldF9mcm9tX2ZvcmsrMHgyNy8w
eDQwDQpbICAzMjUuNDIzODc0XSBrd29ya2VyL3U2NjoxICAgRCAgICAwICAgMzMxICAgICAgMiAw
eDgwMDAwMDAwDQpbICAzMjUuNDMxMzc0XSBXb3JrcXVldWU6IGV2ZW50c191bmJvdW5kIGFzeW5j
X3J1bl9lbnRyeV9mbg0KWyAgMzI1LjQzODY1MV0gQ2FsbCBUcmFjZToNClsgIDMyNS40NDI2NjJd
ICBfX3NjaGVkdWxlKzB4MmZhLzB4YmIwDQpbICAzMjUuNDQ3ODM1XSAgPyB0cmFjZV9oYXJkaXJx
c19vbl9jYWxsZXIrMHhmYi8weDE5MA0KWyAgMzI1LjQ1NDUyMl0gIHNjaGVkdWxlKzB4MzYvMHg5
MA0KWyAgMzI1LjQ1OTMzOV0gIHNjaGVkdWxlX3RpbWVvdXQrMHgyMmMvMHg1NzANClsgIDMyNS40
NjUwOTddICA/IGNhbGxfdGltZXJfZm4rMHgzMzAvMHgzMzANClsgIDMyNS40NzA4MDBdICA/IHdh
aXRfZm9yX2NvbXBsZXRpb25faW9fdGltZW91dCsweGY3LzB4MTgwDQpbICAzMjUuNDc4MDE3XSAg
aW9fc2NoZWR1bGVfdGltZW91dCsweDFlLzB4NTANClsgIDMyNS40ODM5MzVdICA/IGlvX3NjaGVk
dWxlX3RpbWVvdXQrMHgxZS8weDUwDQpbICAzMjUuNDkwMDE1XSAgd2FpdF9mb3JfY29tcGxldGlv
bl9pb190aW1lb3V0KzB4MTFmLzB4MTgwDQpbICAzMjUuNDk3MTM0XSAgPyB3YWtlX3VwX3ErMHg4
MC8weDgwDQpbICAzMjUuNTAyMjcyXSAgYmxrX2V4ZWN1dGVfcnErMHg4Ni8weGMwDQpbICAzMjUu
NTA3NjUyXSAgc2NzaV9leGVjdXRlKzB4ZGIvMHgxZjANClsgIDMyNS41MTI5NjBdICBzZF9yZXZh
bGlkYXRlX2Rpc2srMHhlZC8weDFjNzAgW3NkX21vZF0NClsgIDMyNS41MTk4NDZdICBzZF9wcm9i
ZV9hc3luYysweGMzLzB4MWQwIFtzZF9tb2RdDQpbICAzMjUuNTI2MTk5XSAgYXN5bmNfcnVuX2Vu
dHJ5X2ZuKzB4MzgvMHgxNjANClsgIDMyNS41MzIxMjFdICBwcm9jZXNzX29uZV93b3JrKzB4MjBh
LzB4NjYwDQpbICAzMjUuNTM3ODk3XSAgd29ya2VyX3RocmVhZCsweDNkLzB4M2IwDQpbICAzMjUu
NTQzMzE5XSAga3RocmVhZCsweDEzYS8weDE1MA0KWyAgMzI1LjU0ODIwMF0gID8gcHJvY2Vzc19v
bmVfd29yaysweDY2MC8weDY2MA0KWyAgMzI1LjU1NDIwNF0gID8ga3RocmVhZF9jcmVhdGVfb25f
bm9kZSsweDQwLzB4NDANClsgIDMyNS41NjA2MDVdICByZXRfZnJvbV9mb3JrKzB4MjcvMHg0MA==

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Fri, Nov 03, 2017 at 03:23:14PM +0000, Bart Van Assche wrote:
> On Fri, 2017-11-03 at 11:50 +0800, Ming Lei wrote:
> > On Fri, Nov 03, 2017 at 02:42:50AM +0000, Bart Van Assche wrote:
> > > On Fri, 2017-11-03 at 10:12 +0800, Ming Lei wrote:
> > > > [root@ibclient srp-test]# ./run_tests
> > > > modprobe: FATAL: Module target_core_mod is in use.
> > > 
> > > LIO must be unloaded before srp-test software is started.
> > 
> > Yeah, I can make this test kick off after running
> > 'modprobe -fr ib_isert' first, but looks all tests failed without
> > any hang, could you check if that is the expected result?
> > 
> > 	https://pastebin.com/VXe66Jpg
> 
> I see plenty of "... >/sys/.../add_target failed: Connection timed out"
> messages. The srp-test scripts use the loopback functionality of IB ports
> so one or more IB ports must be active. Have you already checked the IB
> ports state, e.g. by running ibv_devinfo?

[root@ibclient ~]# ibv_devinfo
hca_id:	mlx5_1
	transport:			InfiniBand (0)
	fw_ver:				12.20.1010
	node_guid:			7cfe:9003:0072:6ed3
	sys_image_guid:			7cfe:9003:0072:6ed2
	vendor_id:			0x02c9
	vendor_part_id:			4115
	hw_ver:				0x0
	board_id:			MT_2190110032
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			1
			port_lid:		1
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_0
	transport:			InfiniBand (0)
	fw_ver:				12.20.1010
	node_guid:			7cfe:9003:0072:6ed2
	sys_image_guid:			7cfe:9003:0072:6ed2
	vendor_id:			0x02c9
	vendor_part_id:			4115
	hw_ver:				0x0
	board_id:			MT_2190110032
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			4
			port_lid:		4
			port_lmc:		0x00
			link_layer:		InfiniBand

Forget to mention, there is failure when running 'make' under srp-test
because shellcheck package is missed in RHEL7. Can that be the issue
of test failure? If yes, could you provide a special version of srp-test
which doesn't depend on shellcheck?

> 
> > Also could you provide some output from debugfs when this hang happens?
> > such as:
> > 
> > 	dispatch/busy/.. under hctxN
> 
> This is the output that appears on my test setup if the two patches from
> the patch series "block: remove unnecessary RESTART" are applied:
> 
> # ~bart/software/infiniband/srp-test/run_tests -f xfs -d -e deadline -r 60                             
> Unloaded the ib_srpt kernel module
> Zero-initializing /dev/ram0 ... done
> Zero-initializing /dev/ram1 ... done
> Zero-initializing /dev/disk/by-id/scsi-33333333000004650 ... done
> brw-rw---- 1 root disk 1, 0 Nov  3 08:10 /dev/ram0
> /dev/ram0
> brw-rw---- 1 root disk 1, 1 Nov  3 08:10 /dev/ram1
> /dev/ram1
> lrwxrwxrwx 1 root root 9 Nov  3 08:11 /dev/disk/by-id/scsi-33333333000004650 -> ../../sdi
> /dev/sdi
> Configured SRP target driver
> Running test /home/bart/software/infiniband/srp-test/tests/01 ...
> id_ext=0x0002c90300a02d50,ioc_guid=0x0002c90300a02d50,dgid=fe800000000000000002c90300a02d51,pkey=7fff,service_id=0x0002c90300a02d50, >/sys/class/infiniband_srp/
> srp-mlx4_0-2/add_target failed: Invalid argument
> id_ext=0x0002c90300a02d50,ioc_guid=0x0002c90300a02d50,dgid=fe800000000000000002c90300a02d52,pkey=7fff,service_id=0x0002c90300a02d50, >/sys/class/infiniband_srp/
> srp-mlx4_0-2/add_target failed: Invalid argument
> count_luns(): 3 <> 3
> Error: unloading kernel module ib_srp failed
> Test /home/bart/software/infiniband/srp-test/tests/01 failed
> # cat /sys/module/scsi_mod/parameters/use_blk_mq  
> Y
> # find /sys/kernel/debug/block -name busy | xargs cat

Can you check 'dispatch' file, hctx state, tags and sched_tags?
And see where is the IO not dispatched?

> # dmesg -c >/dev/null; echo w > /proc/sysrq-trigger; dmesg
> [  709.321750] sysrq: SysRq : Show Blocked State
> [  709.327906]   task                        PC stack   pid father
...
> [  709.456602] kworker/u66:6   D    0   341      2 0x80000000
> [  709.463371] Workqueue: events_unbound async_run_entry_fn
> [  709.469919] Call Trace:
> [  709.473325]  __schedule+0x2fa/0xbb0
> [  709.477850]  ? trace_hardirqs_on_caller+0xfb/0x190
> [  709.483847]  schedule+0x36/0x90
> [  709.487982]  schedule_timeout+0x22c/0x570
> [  709.493115]  ? call_timer_fn+0x330/0x330
> [  709.498133]  ? wait_for_completion_io_timeout+0xf7/0x180
> [  709.504732]  io_schedule_timeout+0x1e/0x50
> [  709.509938]  ? io_schedule_timeout+0x1e/0x50
> [  709.515413]  wait_for_completion_io_timeout+0x11f/0x180
> [  709.521920]  ? wake_up_q+0x80/0x80
> [  709.526376]  blk_execute_rq+0x86/0xc0
> [  709.531135]  scsi_execute+0xdb/0x1f0
> [  709.536336]  sd_revalidate_disk+0xed/0x1c70 [sd_mod]
> [  709.543281]  sd_probe_async+0xc3/0x1d0 [sd_mod]
> [  709.549524]  async_run_entry_fn+0x38/0x160
> [  709.555503]  process_one_work+0x20a/0x660
> [  709.561299]  worker_thread+0x3d/0x3b0
> [  709.566688]  kthread+0x13a/0x150
> [  709.571613]  ? process_one_work+0x660/0x660
> [  709.577561]  ? kthread_create_on_node+0x40/0x40
> [  709.583910]  ret_from_fork+0x27/0x40

Looks hang in scsi_execute().

> [  709.589174] kworker/5:2     D    0   386      2 0x80000000
> [  709.596616] Workqueue: kaluad alua_rtpg_work [scsi_dh_alua]
> [  709.604130] Call Trace:
> [  709.608096]  __schedule+0x2fa/0xbb0
> [  709.613265]  ? trace_hardirqs_on_caller+0xfb/0x190
> [  709.619879]  schedule+0x36/0x90
> [  709.624684]  schedule_timeout+0x22c/0x570
> [  709.630437]  ? call_timer_fn+0x330/0x330
> [  709.636059]  ? wait_for_completion_io_timeout+0xf7/0x180
> [  709.643270]  io_schedule_timeout+0x1e/0x50
> [  709.649083]  ? io_schedule_timeout+0x1e/0x50
> [  709.655159]  wait_for_completion_io_timeout+0x11f/0x180
> [  709.662284]  ? wake_up_q+0x80/0x80
> [  709.667334]  blk_execute_rq+0x86/0xc0
> [  709.672705]  scsi_execute+0xdb/0x1f0
> [  709.677938]  alua_rtpg_work+0x2b3/0xe8a [scsi_dh_alua]
> [  709.684983]  process_one_work+0x20a/0x660
> [  709.690759]  worker_thread+0x3d/0x3b0
> [  709.696107]  kthread+0x13a/0x150
> [  709.700986]  ? process_one_work+0x660/0x660
> [  709.706920]  ? kthread_create_on_node+0x40/0x40
> [  709.713316]  ret_from_fork+0x27/0x40

Same with above.

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDIzOjE4ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gQlRX
LCBMYXVyZW5jZSBmb3VuZCB0aGVyZSBpcyBrZXJuZWwgY3Jhc2ggaW4gaGlzIElCL1NSUCB0ZXN0
IHdoZW4gcnVubmluZw0KPiBmb3ItbmV4dCBicmFuY2ggb2YgYmxvY2sgdHJlZSwgc28gd2UganVz
dCB0ZXN0IHY0LjE0LXJjNCB3L3dvIG15IGJsay1tcSBwYXRjaGVzLg0KDQpPbmUgZml4IGZvciBh
ICpzcG9yYWRpYyogaW5pdGlhdG9yIGNyYXNoIGhhcyBiZWVuIHF1ZXVlZCBmb3IgdGhlIHY0LjE1
IG1lcmdlDQp3aW5kb3cuIElmIHlvdSBoaXQgYW5vdGhlciBjcmFzaCwgcGxlYXNlIGxldCBtZSBr
bm93LiBTZWUgYWxzbw0KaHR0cHM6Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5l
bC9naXQvZGxlZGZvcmQvcmRtYS5naXQvY29tbWl0Lz9oPWsuby9mb3ItbmV4dCZpZD04YTBkMThj
NjIxMjFkM2M1NTRhODNlYjk2ZTI3NTI4NjFkODRkOTM3Lg0KDQpCYXJ0Lg==

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Bart Van Assche]

T24gRnJpLCAyMDE3LTExLTAzIGF0IDIzOjQ3ICswODAwLCBNaW5nIExlaSB3cm90ZToNCj4gRm9y
Z2V0IHRvIG1lbnRpb24sIHRoZXJlIGlzIGZhaWx1cmUgd2hlbiBydW5uaW5nICdtYWtlJyB1bmRl
ciBzcnAtdGVzdA0KPiBiZWNhdXNlIHNoZWxsY2hlY2sgcGFja2FnZSBpcyBtaXNzZWQgaW4gUkhF
TDcuIENhbiB0aGF0IGJlIHRoZSBpc3N1ZQ0KPiBvZiB0ZXN0IGZhaWx1cmU/IElmIHllcywgY291
bGQgeW91IHByb3ZpZGUgYSBzcGVjaWFsIHZlcnNpb24gb2Ygc3JwLXRlc3QNCj4gd2hpY2ggZG9l
c24ndCBkZXBlbmQgb24gc2hlbGxjaGVjaz8NCg0KVGhlIHNycC10ZXN0IHNvZnR3YXJlIGRvZXMg
bm90IGRlcGVuZCBvbiBzaGVsbGNoZWNrLiBUaGUgb25seSBwdXJwb3NlIG9mDQp0aGUgInNoZWxs
Y2hlY2siIHRhcmdldCBpbiB0aGUgc3JwLXRlc3QgTWFrZWZpbGUgaXMgdG8gdmVyaWZ5IHRoZSBz
eW50YXgNCm9mIHRoZSBzaGVsbCBzY3JpcHRzLiBUaGUgdGVzdHMgcnVuIGZpbmUgZXZlbiBpZiBz
aGVsbGNoZWNrIGlzIG1pc3NpbmcuDQoNCkJhcnQu

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Laurence Oberman]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

Hi
I had it working some time back. I am off today to take my son to the
doctor.
I will get Bart's test working again this weekend.
Thanks
Laurence

On Nov 3, 2017 11:51 AM, "Bart Van Assche" <Bart.VanAssche@wdc.com> wrote:

> On Fri, 2017-11-03 at 23:47 +0800, Ming Lei wrote:
> > Forget to mention, there is failure when running 'make' under srp-test
> > because shellcheck package is missed in RHEL7. Can that be the issue
> > of test failure? If yes, could you provide a special version of srp-test
> > which doesn't depend on shellcheck?
>
> The srp-test software does not depend on shellcheck. The only purpose of
> the "shellcheck" target in the srp-test Makefile is to verify the syntax
> of the shell scripts. The tests run fine even if shellcheck is missing.
>
> Bart.

[-- Attachment #2: Type: text/html, Size: 1214 bytes --]

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Fri, Nov 03, 2017 at 12:13:09PM -0400, Laurence Oberman wrote:
> Hi
> I had it working some time back. I am off today to take my son to the
> doctor.
> I will get Bart's test working again this weekend.

Hello Laurence and Bart,

Just found srp-test starts to work now with v4.14-rc4 kernel, and
the IO hang issue has been reproduced one time. BTW, the connection
failure happened on for-next of block tree.

Will investigate the issue this weekend.

Thanks,
Ming

> 
> On Nov 3, 2017 11:51 AM, "Bart Van Assche" <Bart.VanAssche@wdc.com> wrote:
> 
> > On Fri, 2017-11-03 at 23:47 +0800, Ming Lei wrote:
> > > Forget to mention, there is failure when running 'make' under srp-test
> > > because shellcheck package is missed in RHEL7. Can that be the issue
> > > of test failure? If yes, could you provide a special version of srp-test
> > > which doesn't depend on shellcheck?
> >
> > The srp-test software does not depend on shellcheck. The only purpose of
> > the "shellcheck" target in the srp-test Makefile is to verify the syntax
> > of the shell scripts. The tests run fine even if shellcheck is missing.
> >
> > Bart.

-- 
Ming

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Laurence Oberman]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

Hi Ming
OK.
However that newer test kernel this morning (not rc4) has issues. Even with
device discovery.

Let me know if you want me to work on it after you are done this weekend.

Thanks
Laurence

On Nov 3, 2017 12:47 PM, "Ming Lei" <ming.lei@redhat.com> wrote:

> On Fri, Nov 03, 2017 at 12:13:09PM -0400, Laurence Oberman wrote:
> > Hi
> > I had it working some time back. I am off today to take my son to the
> > doctor.
> > I will get Bart's test working again this weekend.
>
> Hello Laurence and Bart,
>
> Just found srp-test starts to work now with v4.14-rc4 kernel, and
> the IO hang issue has been reproduced one time. BTW, the connection
> failure happened on for-next of block tree.
>
> Will investigate the issue this weekend.
>
> Thanks,
> Ming
>
> >
> > On Nov 3, 2017 11:51 AM, "Bart Van Assche" <Bart.VanAssche@wdc.com>
> wrote:
> >
> > > On Fri, 2017-11-03 at 23:47 +0800, Ming Lei wrote:
> > > > Forget to mention, there is failure when running 'make' under
> srp-test
> > > > because shellcheck package is missed in RHEL7. Can that be the issue
> > > > of test failure? If yes, could you provide a special version of
> srp-test
> > > > which doesn't depend on shellcheck?
> > >
> > > The srp-test software does not depend on shellcheck. The only purpose
> of
> > > the "shellcheck" target in the srp-test Makefile is to verify the
> syntax
> > > of the shell scripts. The tests run fine even if shellcheck is missing.
> > >
> > > Bart.
>
> --
> Ming
>

[-- Attachment #2: Type: text/html, Size: 2190 bytes --]

Re: [PATCH V2 0/2] block: remove unnecessary RESTART

[Ming Lei]

On Sat, Nov 04, 2017 at 12:47:31AM +0800, Ming Lei wrote:
> On Fri, Nov 03, 2017 at 12:13:09PM -0400, Laurence Oberman wrote:
> > Hi
> > I had it working some time back. I am off today to take my son to the
> > doctor.
> > I will get Bart's test working again this weekend.
> 
> Hello Laurence and Bart,
> 
> Just found srp-test starts to work now with v4.14-rc4 kernel, and
> the IO hang issue has been reproduced one time. BTW, the connection
> failure happened on for-next of block tree.
> 
> Will investigate the issue this weekend.

Hi Bart,

Please test the following patch to see if your issue can be fixed:

https://marc.info/?l=linux-scsi&m=150976056219162&w=2

I run three times of your test 01, not see the hang issue any more
once this patch is applied.

Thanks,
Ming

> 
> > 
> > On Nov 3, 2017 11:51 AM, "Bart Van Assche" <Bart.VanAssche@wdc.com> wrote:
> > 
> > > On Fri, 2017-11-03 at 23:47 +0800, Ming Lei wrote:
> > > > Forget to mention, there is failure when running 'make' under srp-test
> > > > because shellcheck package is missed in RHEL7. Can that be the issue
> > > > of test failure? If yes, could you provide a special version of srp-test
> > > > which doesn't depend on shellcheck?
> > >
> > > The srp-test software does not depend on shellcheck. The only purpose of
> > > the "shellcheck" target in the srp-test Makefile is to verify the syntax
> > > of the shell scripts. The tests run fine even if shellcheck is missing.
> > >
> > > Bart.
> 
> -- 
> Ming

-- 
Ming