All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH -next v10 0/4] support concurrent sync io for bfq on a specail occasion
@ 2022-06-10  2:16 ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:16 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

There are some problem in our mail server, resend v9 because Paolo seems
didn't receive them(hopefully v10 will be fine).

Changes in v10:
 - Add reviewed-tag for patch 2

Changes in v9:
 - also update how many bfqqs have pending_reqs bfq_bfqq_move().
 - fix one language in patch 4
 - Add reviewed-tag for patch 1,3,4

Changes in v8:
 - Instead of using whether bfqq is busy, using whether bfqq has pending
 requests. As Paolo pointed out the former way is problematic.

Changes in v7:
 - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
 also retest this patchset on v5.18 to make sure functionality is
 correct.
 - move the updating of 'bfqd->busy_queues' into new apis

Changes in v6:
 - add reviewed-by tag for patch 1

Changes in v5:
 - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
 - fix wrong definition in patch 1
 - fix spelling mistake in patch 2: leaset -> least
 - update comments in patch 3
 - add reviewed-by tag in patch 2,3

Changes in v4:
 - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
   suggested by Jan Kara.
 - remove unused 'in_groups_with_pending_reqs',

Changes in v3:
 - remove the cleanup patch that is irrelevant now(I'll post it
   separately).
 - instead of hacking wr queues and using weights tree insertion/removal,
   using bfq_add/del_bfqq_busy() to count the number of groups
   (suggested by Jan Kara).

Changes in v2:
 - Use a different approch to count root group, which is much simple.

Currently, bfq can't handle sync io concurrently as long as they
are not issued from root group. This is because
'bfqd->num_groups_with_pending_reqs > 0' is always true in
bfq_asymmetric_scenario().

The way that bfqg is counted into 'num_groups_with_pending_reqs':

Before this patchset:
 1) root group will never be counted.
 2) Count if bfqg or it's child bfqgs have pending requests.
 3) Don't count if bfqg and it's child bfqgs complete all the requests.

After this patchset:
 1) root group is counted.
 2) Count if bfqg has pending requests.
 3) Don't count if bfqg complete all the requests.

With the above changes, concurrent sync io can be supported if only
one group is activated.

fio test script(startdelay is used to avoid queue merging):
[global]
filename=/dev/sda
allow_mounted_write=0
ioengine=psync
direct=1
ioscheduler=bfq
offset_increment=10g
group_reporting
rw=randwrite
bs=4k

[test1]
numjobs=1

[test2]
startdelay=1
numjobs=1

[test3]
startdelay=2
numjobs=1

[test4]
startdelay=3
numjobs=1

[test5]
startdelay=4
numjobs=1

[test6]
startdelay=5
numjobs=1

[test7]
startdelay=6
numjobs=1

[test8]
startdelay=7
numjobs=1

test result:
running fio on root cgroup
v5.18:	   112 Mib/s
v5.18-patched: 112 Mib/s

running fio on non-root cgroup
v5.18:	   51.2 Mib/s
v5.18-patched: 112 Mib/s

Note that I also test null_blk with "irqmode=2
completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
that service guarantees are still preserved.

Previous versions:
RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/


Yu Kuai (4):
  block, bfq: support to track if bfqq has pending requests
  block, bfq: record how many queues have pending requests
  block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  block, bfq: do not idle if only one group is activated

 block/bfq-cgroup.c  | 10 ++++++++++
 block/bfq-iosched.c | 47 +++------------------------------------------
 block/bfq-iosched.h | 21 +++++++++++---------
 block/bfq-wf2q.c    | 47 ++++++++++++++++++++++++++++++---------------
 4 files changed, 57 insertions(+), 68 deletions(-)

-- 
2.31.1


^ permalink raw reply	[flat|nested] 38+ messages in thread
* [PATCH -next v10 0/4] support concurrent sync io for bfq on a specail occasion
@ 2022-06-10  2:16 ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:16 UTC (permalink / raw)
  To: paolo.valente-QSEj5FYQhm4dnm+yROfE0A, jack-AlSwsSmVLrQ,
	cgroups-u79uwXL29TY76Z2rM5mHXA,
	linux-block-u79uwXL29TY76Z2rM5mHXA
  Cc: tj-DgEjT+Ai2ygdnm+yROfE0A, axboe-tSWWG44O7X1aa/9Udqfwiw,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	yukuai3-hv44wF8Li93QT0dZR+AlfA, yi.zhang-hv44wF8Li93QT0dZR+AlfA

There are some problem in our mail server, resend v9 because Paolo seems
didn't receive them(hopefully v10 will be fine).

Changes in v10:
 - Add reviewed-tag for patch 2

Changes in v9:
 - also update how many bfqqs have pending_reqs bfq_bfqq_move().
 - fix one language in patch 4
 - Add reviewed-tag for patch 1,3,4

Changes in v8:
 - Instead of using whether bfqq is busy, using whether bfqq has pending
 requests. As Paolo pointed out the former way is problematic.

Changes in v7:
 - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
 also retest this patchset on v5.18 to make sure functionality is
 correct.
 - move the updating of 'bfqd->busy_queues' into new apis

Changes in v6:
 - add reviewed-by tag for patch 1

Changes in v5:
 - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
 - fix wrong definition in patch 1
 - fix spelling mistake in patch 2: leaset -> least
 - update comments in patch 3
 - add reviewed-by tag in patch 2,3

Changes in v4:
 - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
   suggested by Jan Kara.
 - remove unused 'in_groups_with_pending_reqs',

Changes in v3:
 - remove the cleanup patch that is irrelevant now(I'll post it
   separately).
 - instead of hacking wr queues and using weights tree insertion/removal,
   using bfq_add/del_bfqq_busy() to count the number of groups
   (suggested by Jan Kara).

Changes in v2:
 - Use a different approch to count root group, which is much simple.

Currently, bfq can't handle sync io concurrently as long as they
are not issued from root group. This is because
'bfqd->num_groups_with_pending_reqs > 0' is always true in
bfq_asymmetric_scenario().

The way that bfqg is counted into 'num_groups_with_pending_reqs':

Before this patchset:
 1) root group will never be counted.
 2) Count if bfqg or it's child bfqgs have pending requests.
 3) Don't count if bfqg and it's child bfqgs complete all the requests.

After this patchset:
 1) root group is counted.
 2) Count if bfqg has pending requests.
 3) Don't count if bfqg complete all the requests.

With the above changes, concurrent sync io can be supported if only
one group is activated.

fio test script(startdelay is used to avoid queue merging):
[global]
filename=/dev/sda
allow_mounted_write=0
ioengine=psync
direct=1
ioscheduler=bfq
offset_increment=10g
group_reporting
rw=randwrite
bs=4k

[test1]
numjobs=1

[test2]
startdelay=1
numjobs=1

[test3]
startdelay=2
numjobs=1

[test4]
startdelay=3
numjobs=1

[test5]
startdelay=4
numjobs=1

[test6]
startdelay=5
numjobs=1

[test7]
startdelay=6
numjobs=1

[test8]
startdelay=7
numjobs=1

test result:
running fio on root cgroup
v5.18:	   112 Mib/s
v5.18-patched: 112 Mib/s

running fio on non-root cgroup
v5.18:	   51.2 Mib/s
v5.18-patched: 112 Mib/s

Note that I also test null_blk with "irqmode=2
completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
that service guarantees are still preserved.

Previous versions:
RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/


Yu Kuai (4):
  block, bfq: support to track if bfqq has pending requests
  block, bfq: record how many queues have pending requests
  block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  block, bfq: do not idle if only one group is activated

 block/bfq-cgroup.c  | 10 ++++++++++
 block/bfq-iosched.c | 47 +++------------------------------------------
 block/bfq-iosched.h | 21 +++++++++++---------
 block/bfq-wf2q.c    | 47 ++++++++++++++++++++++++++++++---------------
 4 files changed, 57 insertions(+), 68 deletions(-)

-- 
2.31.1


^ permalink raw reply	[flat|nested] 38+ messages in thread
* [PATCH -next v10 1/4] block, bfq: support to track if bfqq has pending requests
  2022-06-10  2:16 ` Yu Kuai
@ 2022-06-10  2:16   ` Yu Kuai
  -1 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:16 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

If entity belongs to bfqq, then entity->in_groups_with_pending_reqs
is not used currently. This patch use it to track if bfqq has pending
requests through callers of weights_tree insertion and removal.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
---
 block/bfq-iosched.c |  1 +
 block/bfq-iosched.h |  2 ++
 block/bfq-wf2q.c    | 24 ++++++++++++++++++++++--
 3 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0d46cb728bbf..0ec21018daba 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
 		 */
 		bfqq->budget_timeout = jiffies;
 
+		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
 		bfq_weights_tree_remove(bfqd, bfqq);
 	}
 
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
index ca8177d7bf7c..69c7d15417e5 100644
--- a/block/bfq-iosched.h
+++ b/block/bfq-iosched.h
@@ -1080,6 +1080,8 @@ void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 		       bool expiration);
 void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq);
+void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq);
+void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq);
 
 /* --------------- end of interface of B-WF2Q+ ---------------- */
 
diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
index f8eb340381cf..5704a0315cf2 100644
--- a/block/bfq-wf2q.c
+++ b/block/bfq-wf2q.c
@@ -1647,6 +1647,22 @@ void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 				    bfqq == bfqd->in_service_queue, expiration);
 }
 
+void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
+{
+	struct bfq_entity *entity = &bfqq->entity;
+
+	if (!entity->in_groups_with_pending_reqs)
+		entity->in_groups_with_pending_reqs = true;
+}
+
+void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
+{
+	struct bfq_entity *entity = &bfqq->entity;
+
+	if (entity->in_groups_with_pending_reqs)
+		entity->in_groups_with_pending_reqs = false;
+}
+
 /*
  * Called when the bfqq no longer has requests pending, remove it from
  * the service tree. As a special case, it can be invoked during an
@@ -1668,8 +1684,10 @@ void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 
 	bfq_deactivate_bfqq(bfqd, bfqq, true, expiration);
 
-	if (!bfqq->dispatched)
+	if (!bfqq->dispatched) {
+		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
 		bfq_weights_tree_remove(bfqd, bfqq);
+	}
 }
 
 /*
@@ -1684,10 +1702,12 @@ void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq)
 	bfq_mark_bfqq_busy(bfqq);
 	bfqd->busy_queues[bfqq->ioprio_class - 1]++;
 
-	if (!bfqq->dispatched)
+	if (!bfqq->dispatched) {
+		bfq_add_bfqq_in_groups_with_pending_reqs(bfqq);
 		if (bfqq->wr_coeff == 1)
 			bfq_weights_tree_add(bfqd, bfqq,
 					     &bfqd->queue_weights_tree);
+	}
 
 	if (bfqq->wr_coeff > 1)
 		bfqd->wr_busy_queues++;
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* [PATCH -next v10 1/4] block, bfq: support to track if bfqq has pending requests
@ 2022-06-10  2:16   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:16 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

If entity belongs to bfqq, then entity->in_groups_with_pending_reqs
is not used currently. This patch use it to track if bfqq has pending
requests through callers of weights_tree insertion and removal.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
---
 block/bfq-iosched.c |  1 +
 block/bfq-iosched.h |  2 ++
 block/bfq-wf2q.c    | 24 ++++++++++++++++++++++--
 3 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0d46cb728bbf..0ec21018daba 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
 		 */
 		bfqq->budget_timeout = jiffies;
 
+		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
 		bfq_weights_tree_remove(bfqd, bfqq);
 	}
 
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
index ca8177d7bf7c..69c7d15417e5 100644
--- a/block/bfq-iosched.h
+++ b/block/bfq-iosched.h
@@ -1080,6 +1080,8 @@ void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 		       bool expiration);
 void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq);
+void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq);
+void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq);
 
 /* --------------- end of interface of B-WF2Q+ ---------------- */
 
diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
index f8eb340381cf..5704a0315cf2 100644
--- a/block/bfq-wf2q.c
+++ b/block/bfq-wf2q.c
@@ -1647,6 +1647,22 @@ void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 				    bfqq == bfqd->in_service_queue, expiration);
 }
 
+void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
+{
+	struct bfq_entity *entity = &bfqq->entity;
+
+	if (!entity->in_groups_with_pending_reqs)
+		entity->in_groups_with_pending_reqs = true;
+}
+
+void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
+{
+	struct bfq_entity *entity = &bfqq->entity;
+
+	if (entity->in_groups_with_pending_reqs)
+		entity->in_groups_with_pending_reqs = false;
+}
+
 /*
  * Called when the bfqq no longer has requests pending, remove it from
  * the service tree. As a special case, it can be invoked during an
@@ -1668,8 +1684,10 @@ void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 
 	bfq_deactivate_bfqq(bfqd, bfqq, true, expiration);
 
-	if (!bfqq->dispatched)
+	if (!bfqq->dispatched) {
+		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
 		bfq_weights_tree_remove(bfqd, bfqq);
+	}
 }
 
 /*
@@ -1684,10 +1702,12 @@ void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq)
 	bfq_mark_bfqq_busy(bfqq);
 	bfqd->busy_queues[bfqq->ioprio_class - 1]++;
 
-	if (!bfqq->dispatched)
+	if (!bfqq->dispatched) {
+		bfq_add_bfqq_in_groups_with_pending_reqs(bfqq);
 		if (bfqq->wr_coeff == 1)
 			bfq_weights_tree_add(bfqd, bfqq,
 					     &bfqd->queue_weights_tree);
+	}
 
 	if (bfqq->wr_coeff > 1)
 		bfqd->wr_busy_queues++;
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* [PATCH -next v10 2/4] block, bfq: record how many queues have pending requests
  2022-06-10  2:16 ` Yu Kuai
@ 2022-06-10  2:16   ` Yu Kuai
  -1 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:16 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

Prepare to refactor the counting of 'num_groups_with_pending_reqs'.

Add a counter in bfq_group, update it while tracking if bfqq have pending
requests and when bfq_bfqq_move() is called.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
---
 block/bfq-cgroup.c  | 10 ++++++++++
 block/bfq-iosched.h |  1 +
 block/bfq-wf2q.c    | 12 ++++++++++--
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index 09574af83566..88c0c320ddf3 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -557,6 +557,7 @@ static void bfq_pd_init(struct blkg_policy_data *pd)
 				   */
 	bfqg->bfqd = bfqd;
 	bfqg->active_entities = 0;
+	bfqg->num_queues_with_pending_reqs = 0;
 	bfqg->online = true;
 	bfqg->rq_pos_tree = RB_ROOT;
 }
@@ -646,6 +647,7 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 {
 	struct bfq_entity *entity = &bfqq->entity;
 	struct bfq_group *old_parent = bfqq_group(bfqq);
+	bool has_pending_reqs = false;
 
 	/*
 	 * No point to move bfqq to the same group, which can happen when
@@ -666,6 +668,11 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 	 */
 	bfqq->ref++;
 
+	if (entity->in_groups_with_pending_reqs) {
+		has_pending_reqs = true;
+		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
+	}
+
 	/* If bfqq is empty, then bfq_bfqq_expire also invokes
 	 * bfq_del_bfqq_busy, thereby removing bfqq and its entity
 	 * from data structures related to current group. Otherwise we
@@ -693,6 +700,9 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 	/* pin down bfqg and its associated blkg  */
 	bfqg_and_blkg_get(bfqg);
 
+	if (has_pending_reqs)
+		bfq_add_bfqq_in_groups_with_pending_reqs(bfqq);
+
 	if (bfq_bfqq_busy(bfqq)) {
 		if (unlikely(!bfqd->nonrot_with_queueing))
 			bfq_pos_tree_add_move(bfqd, bfqq);
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
index 69c7d15417e5..de2446a9b7ab 100644
--- a/block/bfq-iosched.h
+++ b/block/bfq-iosched.h
@@ -943,6 +943,7 @@ struct bfq_group {
 	struct bfq_entity *my_entity;
 
 	int active_entities;
+	int num_queues_with_pending_reqs;
 
 	struct rb_root rq_pos_tree;
 
diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
index 5704a0315cf2..6f36f3fe5cc8 100644
--- a/block/bfq-wf2q.c
+++ b/block/bfq-wf2q.c
@@ -1651,16 +1651,24 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 {
 	struct bfq_entity *entity = &bfqq->entity;
 
-	if (!entity->in_groups_with_pending_reqs)
+	if (!entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = true;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
+#endif
+	}
 }
 
 void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 {
 	struct bfq_entity *entity = &bfqq->entity;
 
-	if (entity->in_groups_with_pending_reqs)
+	if (entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = false;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
+#endif
+	}
 }
 
 /*
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* [PATCH -next v10 2/4] block, bfq: record how many queues have pending requests
@ 2022-06-10  2:16   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:16 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

Prepare to refactor the counting of 'num_groups_with_pending_reqs'.

Add a counter in bfq_group, update it while tracking if bfqq have pending
requests and when bfq_bfqq_move() is called.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
---
 block/bfq-cgroup.c  | 10 ++++++++++
 block/bfq-iosched.h |  1 +
 block/bfq-wf2q.c    | 12 ++++++++++--
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index 09574af83566..88c0c320ddf3 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -557,6 +557,7 @@ static void bfq_pd_init(struct blkg_policy_data *pd)
 				   */
 	bfqg->bfqd = bfqd;
 	bfqg->active_entities = 0;
+	bfqg->num_queues_with_pending_reqs = 0;
 	bfqg->online = true;
 	bfqg->rq_pos_tree = RB_ROOT;
 }
@@ -646,6 +647,7 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 {
 	struct bfq_entity *entity = &bfqq->entity;
 	struct bfq_group *old_parent = bfqq_group(bfqq);
+	bool has_pending_reqs = false;
 
 	/*
 	 * No point to move bfqq to the same group, which can happen when
@@ -666,6 +668,11 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 	 */
 	bfqq->ref++;
 
+	if (entity->in_groups_with_pending_reqs) {
+		has_pending_reqs = true;
+		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
+	}
+
 	/* If bfqq is empty, then bfq_bfqq_expire also invokes
 	 * bfq_del_bfqq_busy, thereby removing bfqq and its entity
 	 * from data structures related to current group. Otherwise we
@@ -693,6 +700,9 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 	/* pin down bfqg and its associated blkg  */
 	bfqg_and_blkg_get(bfqg);
 
+	if (has_pending_reqs)
+		bfq_add_bfqq_in_groups_with_pending_reqs(bfqq);
+
 	if (bfq_bfqq_busy(bfqq)) {
 		if (unlikely(!bfqd->nonrot_with_queueing))
 			bfq_pos_tree_add_move(bfqd, bfqq);
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
index 69c7d15417e5..de2446a9b7ab 100644
--- a/block/bfq-iosched.h
+++ b/block/bfq-iosched.h
@@ -943,6 +943,7 @@ struct bfq_group {
 	struct bfq_entity *my_entity;
 
 	int active_entities;
+	int num_queues_with_pending_reqs;
 
 	struct rb_root rq_pos_tree;
 
diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
index 5704a0315cf2..6f36f3fe5cc8 100644
--- a/block/bfq-wf2q.c
+++ b/block/bfq-wf2q.c
@@ -1651,16 +1651,24 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 {
 	struct bfq_entity *entity = &bfqq->entity;
 
-	if (!entity->in_groups_with_pending_reqs)
+	if (!entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = true;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
+#endif
+	}
 }
 
 void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 {
 	struct bfq_entity *entity = &bfqq->entity;
 
-	if (entity->in_groups_with_pending_reqs)
+	if (entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = false;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
+#endif
+	}
 }
 
 /*
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-06-10  2:17   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:17 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

Currently, bfq can't handle sync io concurrently as long as they
are not issued from root group. This is because
'bfqd->num_groups_with_pending_reqs > 0' is always true in
bfq_asymmetric_scenario().

The way that bfqg is counted into 'num_groups_with_pending_reqs':

Before this patch:
 1) root group will never be counted.
 2) Count if bfqg or it's child bfqgs have pending requests.
 3) Don't count if bfqg and it's child bfqgs complete all the requests.

After this patch:
 1) root group is counted.
 2) Count if bfqg have pending requests.
 3) Don't count if bfqg complete all the requests.

With this change, the occasion that only one group is activated can be
detected, and next patch will support concurrent sync io in the
occasion.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
---
 block/bfq-iosched.c | 42 ------------------------------------------
 block/bfq-iosched.h | 18 +++++++++---------
 block/bfq-wf2q.c    | 19 ++++---------------
 3 files changed, 13 insertions(+), 66 deletions(-)

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0ec21018daba..03b04892440c 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
 void bfq_weights_tree_remove(struct bfq_data *bfqd,
 			     struct bfq_queue *bfqq)
 {
-	struct bfq_entity *entity = bfqq->entity.parent;
-
-	for_each_entity(entity) {
-		struct bfq_sched_data *sd = entity->my_sched_data;
-
-		if (sd->next_in_service || sd->in_service_entity) {
-			/*
-			 * entity is still active, because either
-			 * next_in_service or in_service_entity is not
-			 * NULL (see the comments on the definition of
-			 * next_in_service for details on why
-			 * in_service_entity must be checked too).
-			 *
-			 * As a consequence, its parent entities are
-			 * active as well, and thus this loop must
-			 * stop here.
-			 */
-			break;
-		}
-
-		/*
-		 * The decrement of num_groups_with_pending_reqs is
-		 * not performed immediately upon the deactivation of
-		 * entity, but it is delayed to when it also happens
-		 * that the first leaf descendant bfqq of entity gets
-		 * all its pending requests completed. The following
-		 * instructions perform this delayed decrement, if
-		 * needed. See the comments on
-		 * num_groups_with_pending_reqs for details.
-		 */
-		if (entity->in_groups_with_pending_reqs) {
-			entity->in_groups_with_pending_reqs = false;
-			bfqd->num_groups_with_pending_reqs--;
-		}
-	}
-
-	/*
-	 * Next function is invoked last, because it causes bfqq to be
-	 * freed if the following holds: bfqq is not in service and
-	 * has no dispatched request. DO NOT use bfqq after the next
-	 * function invocation.
-	 */
 	__bfq_weights_tree_remove(bfqd, bfqq,
 				  &bfqd->queue_weights_tree);
 }
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
index de2446a9b7ab..f0fce94583e4 100644
--- a/block/bfq-iosched.h
+++ b/block/bfq-iosched.h
@@ -496,27 +496,27 @@ struct bfq_data {
 	struct rb_root_cached queue_weights_tree;
 
 	/*
-	 * Number of groups with at least one descendant process that
+	 * Number of groups with at least one process that
 	 * has at least one request waiting for completion. Note that
 	 * this accounts for also requests already dispatched, but not
 	 * yet completed. Therefore this number of groups may differ
 	 * (be larger) than the number of active groups, as a group is
 	 * considered active only if its corresponding entity has
-	 * descendant queues with at least one request queued. This
+	 * queues with at least one request queued. This
 	 * number is used to decide whether a scenario is symmetric.
 	 * For a detailed explanation see comments on the computation
 	 * of the variable asymmetric_scenario in the function
 	 * bfq_better_to_idle().
 	 *
 	 * However, it is hard to compute this number exactly, for
-	 * groups with multiple descendant processes. Consider a group
-	 * that is inactive, i.e., that has no descendant process with
+	 * groups with multiple processes. Consider a group
+	 * that is inactive, i.e., that has no process with
 	 * pending I/O inside BFQ queues. Then suppose that
 	 * num_groups_with_pending_reqs is still accounting for this
-	 * group, because the group has descendant processes with some
+	 * group, because the group has processes with some
 	 * I/O request still in flight. num_groups_with_pending_reqs
 	 * should be decremented when the in-flight request of the
-	 * last descendant process is finally completed (assuming that
+	 * last process is finally completed (assuming that
 	 * nothing else has changed for the group in the meantime, in
 	 * terms of composition of the group and active/inactive state of child
 	 * groups and processes). To accomplish this, an additional
@@ -525,7 +525,7 @@ struct bfq_data {
 	 * we resort to the following tradeoff between simplicity and
 	 * accuracy: for an inactive group that is still counted in
 	 * num_groups_with_pending_reqs, we decrement
-	 * num_groups_with_pending_reqs when the first descendant
+	 * num_groups_with_pending_reqs when the first
 	 * process of the group remains with no request waiting for
 	 * completion.
 	 *
@@ -533,12 +533,12 @@ struct bfq_data {
 	 * carefulness: to avoid multiple decrements, we flag a group,
 	 * more precisely an entity representing a group, as still
 	 * counted in num_groups_with_pending_reqs when it becomes
-	 * inactive. Then, when the first descendant queue of the
+	 * inactive. Then, when the first queue of the
 	 * entity remains with no request waiting for completion,
 	 * num_groups_with_pending_reqs is decremented, and this flag
 	 * is reset. After this flag is reset for the entity,
 	 * num_groups_with_pending_reqs won't be decremented any
-	 * longer in case a new descendant queue of the entity remains
+	 * longer in case a new queue of the entity remains
 	 * with no request waiting for completion.
 	 */
 	unsigned int num_groups_with_pending_reqs;
diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
index 6f36f3fe5cc8..9c2842bedf97 100644
--- a/block/bfq-wf2q.c
+++ b/block/bfq-wf2q.c
@@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
 		entity->on_st_or_in_serv = true;
 	}
 
-#ifdef CONFIG_BFQ_GROUP_IOSCHED
-	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
-		struct bfq_group *bfqg =
-			container_of(entity, struct bfq_group, entity);
-		struct bfq_data *bfqd = bfqg->bfqd;
-
-		if (!entity->in_groups_with_pending_reqs) {
-			entity->in_groups_with_pending_reqs = true;
-			bfqd->num_groups_with_pending_reqs++;
-		}
-	}
-#endif
-
 	bfq_update_fin_time_enqueue(entity, st, backshifted);
 }
 
@@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 	if (!entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = true;
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
-		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
+		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
+			bfqq->bfqd->num_groups_with_pending_reqs++;
 #endif
 	}
 }
@@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 	if (entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = false;
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
-		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
+		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
+			bfqq->bfqd->num_groups_with_pending_reqs--;
 #endif
 	}
 }
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-06-10  2:17   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:17 UTC (permalink / raw)
  To: paolo.valente-QSEj5FYQhm4dnm+yROfE0A, jack-AlSwsSmVLrQ,
	cgroups-u79uwXL29TY76Z2rM5mHXA,
	linux-block-u79uwXL29TY76Z2rM5mHXA
  Cc: tj-DgEjT+Ai2ygdnm+yROfE0A, axboe-tSWWG44O7X1aa/9Udqfwiw,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	yukuai3-hv44wF8Li93QT0dZR+AlfA, yi.zhang-hv44wF8Li93QT0dZR+AlfA

Currently, bfq can't handle sync io concurrently as long as they
are not issued from root group. This is because
'bfqd->num_groups_with_pending_reqs > 0' is always true in
bfq_asymmetric_scenario().

The way that bfqg is counted into 'num_groups_with_pending_reqs':

Before this patch:
 1) root group will never be counted.
 2) Count if bfqg or it's child bfqgs have pending requests.
 3) Don't count if bfqg and it's child bfqgs complete all the requests.

After this patch:
 1) root group is counted.
 2) Count if bfqg have pending requests.
 3) Don't count if bfqg complete all the requests.

With this change, the occasion that only one group is activated can be
detected, and next patch will support concurrent sync io in the
occasion.

Signed-off-by: Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
Reviewed-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
---
 block/bfq-iosched.c | 42 ------------------------------------------
 block/bfq-iosched.h | 18 +++++++++---------
 block/bfq-wf2q.c    | 19 ++++---------------
 3 files changed, 13 insertions(+), 66 deletions(-)

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0ec21018daba..03b04892440c 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
 void bfq_weights_tree_remove(struct bfq_data *bfqd,
 			     struct bfq_queue *bfqq)
 {
-	struct bfq_entity *entity = bfqq->entity.parent;
-
-	for_each_entity(entity) {
-		struct bfq_sched_data *sd = entity->my_sched_data;
-
-		if (sd->next_in_service || sd->in_service_entity) {
-			/*
-			 * entity is still active, because either
-			 * next_in_service or in_service_entity is not
-			 * NULL (see the comments on the definition of
-			 * next_in_service for details on why
-			 * in_service_entity must be checked too).
-			 *
-			 * As a consequence, its parent entities are
-			 * active as well, and thus this loop must
-			 * stop here.
-			 */
-			break;
-		}
-
-		/*
-		 * The decrement of num_groups_with_pending_reqs is
-		 * not performed immediately upon the deactivation of
-		 * entity, but it is delayed to when it also happens
-		 * that the first leaf descendant bfqq of entity gets
-		 * all its pending requests completed. The following
-		 * instructions perform this delayed decrement, if
-		 * needed. See the comments on
-		 * num_groups_with_pending_reqs for details.
-		 */
-		if (entity->in_groups_with_pending_reqs) {
-			entity->in_groups_with_pending_reqs = false;
-			bfqd->num_groups_with_pending_reqs--;
-		}
-	}
-
-	/*
-	 * Next function is invoked last, because it causes bfqq to be
-	 * freed if the following holds: bfqq is not in service and
-	 * has no dispatched request. DO NOT use bfqq after the next
-	 * function invocation.
-	 */
 	__bfq_weights_tree_remove(bfqd, bfqq,
 				  &bfqd->queue_weights_tree);
 }
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
index de2446a9b7ab..f0fce94583e4 100644
--- a/block/bfq-iosched.h
+++ b/block/bfq-iosched.h
@@ -496,27 +496,27 @@ struct bfq_data {
 	struct rb_root_cached queue_weights_tree;
 
 	/*
-	 * Number of groups with at least one descendant process that
+	 * Number of groups with at least one process that
 	 * has at least one request waiting for completion. Note that
 	 * this accounts for also requests already dispatched, but not
 	 * yet completed. Therefore this number of groups may differ
 	 * (be larger) than the number of active groups, as a group is
 	 * considered active only if its corresponding entity has
-	 * descendant queues with at least one request queued. This
+	 * queues with at least one request queued. This
 	 * number is used to decide whether a scenario is symmetric.
 	 * For a detailed explanation see comments on the computation
 	 * of the variable asymmetric_scenario in the function
 	 * bfq_better_to_idle().
 	 *
 	 * However, it is hard to compute this number exactly, for
-	 * groups with multiple descendant processes. Consider a group
-	 * that is inactive, i.e., that has no descendant process with
+	 * groups with multiple processes. Consider a group
+	 * that is inactive, i.e., that has no process with
 	 * pending I/O inside BFQ queues. Then suppose that
 	 * num_groups_with_pending_reqs is still accounting for this
-	 * group, because the group has descendant processes with some
+	 * group, because the group has processes with some
 	 * I/O request still in flight. num_groups_with_pending_reqs
 	 * should be decremented when the in-flight request of the
-	 * last descendant process is finally completed (assuming that
+	 * last process is finally completed (assuming that
 	 * nothing else has changed for the group in the meantime, in
 	 * terms of composition of the group and active/inactive state of child
 	 * groups and processes). To accomplish this, an additional
@@ -525,7 +525,7 @@ struct bfq_data {
 	 * we resort to the following tradeoff between simplicity and
 	 * accuracy: for an inactive group that is still counted in
 	 * num_groups_with_pending_reqs, we decrement
-	 * num_groups_with_pending_reqs when the first descendant
+	 * num_groups_with_pending_reqs when the first
 	 * process of the group remains with no request waiting for
 	 * completion.
 	 *
@@ -533,12 +533,12 @@ struct bfq_data {
 	 * carefulness: to avoid multiple decrements, we flag a group,
 	 * more precisely an entity representing a group, as still
 	 * counted in num_groups_with_pending_reqs when it becomes
-	 * inactive. Then, when the first descendant queue of the
+	 * inactive. Then, when the first queue of the
 	 * entity remains with no request waiting for completion,
 	 * num_groups_with_pending_reqs is decremented, and this flag
 	 * is reset. After this flag is reset for the entity,
 	 * num_groups_with_pending_reqs won't be decremented any
-	 * longer in case a new descendant queue of the entity remains
+	 * longer in case a new queue of the entity remains
 	 * with no request waiting for completion.
 	 */
 	unsigned int num_groups_with_pending_reqs;
diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
index 6f36f3fe5cc8..9c2842bedf97 100644
--- a/block/bfq-wf2q.c
+++ b/block/bfq-wf2q.c
@@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
 		entity->on_st_or_in_serv = true;
 	}
 
-#ifdef CONFIG_BFQ_GROUP_IOSCHED
-	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
-		struct bfq_group *bfqg =
-			container_of(entity, struct bfq_group, entity);
-		struct bfq_data *bfqd = bfqg->bfqd;
-
-		if (!entity->in_groups_with_pending_reqs) {
-			entity->in_groups_with_pending_reqs = true;
-			bfqd->num_groups_with_pending_reqs++;
-		}
-	}
-#endif
-
 	bfq_update_fin_time_enqueue(entity, st, backshifted);
 }
 
@@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 	if (!entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = true;
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
-		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
+		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
+			bfqq->bfqd->num_groups_with_pending_reqs++;
 #endif
 	}
 }
@@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
 	if (entity->in_groups_with_pending_reqs) {
 		entity->in_groups_with_pending_reqs = false;
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
-		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
+		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
+			bfqq->bfqd->num_groups_with_pending_reqs--;
 #endif
 	}
 }
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* [PATCH -next v10 4/4] block, bfq: do not idle if only one group is activated
  2022-06-10  2:16 ` Yu Kuai
@ 2022-06-10  2:17   ` Yu Kuai
  -1 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:17 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

Now that root group is counted into 'num_groups_with_pending_reqs',
'num_groups_with_pending_reqs > 0' is always true in
bfq_asymmetric_scenario(). Thus change the condition to '> 1'.

On the other hand, this change can enable concurrent sync io if only
one group is activated.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
---
 block/bfq-iosched.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 03b04892440c..a2aa243505dc 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -820,7 +820,7 @@ bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq)
  * much easier to maintain the needed state:
  * 1) all active queues have the same weight,
  * 2) all active queues belong to the same I/O-priority class,
- * 3) there are no active groups.
+ * 3) there is at most one active group.
  * In particular, the last condition is always true if hierarchical
  * support or the cgroups interface are not enabled, thus no state
  * needs to be maintained in this case.
@@ -852,7 +852,7 @@ static bool bfq_asymmetric_scenario(struct bfq_data *bfqd,
 
 	return varied_queue_weights || multiple_classes_busy
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
-	       || bfqd->num_groups_with_pending_reqs > 0
+	       || bfqd->num_groups_with_pending_reqs > 1
 #endif
 		;
 }
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* [PATCH -next v10 4/4] block, bfq: do not idle if only one group is activated
@ 2022-06-10  2:17   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-10  2:17 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yukuai3, yi.zhang

Now that root group is counted into 'num_groups_with_pending_reqs',
'num_groups_with_pending_reqs > 0' is always true in
bfq_asymmetric_scenario(). Thus change the condition to '> 1'.

On the other hand, this change can enable concurrent sync io if only
one group is activated.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
---
 block/bfq-iosched.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 03b04892440c..a2aa243505dc 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -820,7 +820,7 @@ bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq)
  * much easier to maintain the needed state:
  * 1) all active queues have the same weight,
  * 2) all active queues belong to the same I/O-priority class,
- * 3) there are no active groups.
+ * 3) there is at most one active group.
  * In particular, the last condition is always true if hierarchical
  * support or the cgroups interface are not enabled, thus no state
  * needs to be maintained in this case.
@@ -852,7 +852,7 @@ static bool bfq_asymmetric_scenario(struct bfq_data *bfqd,
 
 	return varied_queue_weights || multiple_classes_busy
 #ifdef CONFIG_BFQ_GROUP_IOSCHED
-	       || bfqd->num_groups_with_pending_reqs > 0
+	       || bfqd->num_groups_with_pending_reqs > 1
 #endif
 		;
 }
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 0/4] support concurrent sync io for bfq on a specail occasion
@ 2022-06-17  1:12   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-17  1:12 UTC (permalink / raw)
  To: paolo.valente, jack, cgroups, linux-block
  Cc: tj, axboe, linux-kernel, yi.zhang

Hi, Paolo

Can you take a look at this patchset?

Thanks,
Kuai

在 2022/06/10 10:16, Yu Kuai 写道:
> There are some problem in our mail server, resend v9 because Paolo seems
> didn't receive them(hopefully v10 will be fine).
> 
> Changes in v10:
>   - Add reviewed-tag for patch 2
> 
> Changes in v9:
>   - also update how many bfqqs have pending_reqs bfq_bfqq_move().
>   - fix one language in patch 4
>   - Add reviewed-tag for patch 1,3,4
> 
> Changes in v8:
>   - Instead of using whether bfqq is busy, using whether bfqq has pending
>   requests. As Paolo pointed out the former way is problematic.
> 
> Changes in v7:
>   - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
>   also retest this patchset on v5.18 to make sure functionality is
>   correct.
>   - move the updating of 'bfqd->busy_queues' into new apis
> 
> Changes in v6:
>   - add reviewed-by tag for patch 1
> 
> Changes in v5:
>   - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
>   - fix wrong definition in patch 1
>   - fix spelling mistake in patch 2: leaset -> least
>   - update comments in patch 3
>   - add reviewed-by tag in patch 2,3
> 
> Changes in v4:
>   - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
>     suggested by Jan Kara.
>   - remove unused 'in_groups_with_pending_reqs',
> 
> Changes in v3:
>   - remove the cleanup patch that is irrelevant now(I'll post it
>     separately).
>   - instead of hacking wr queues and using weights tree insertion/removal,
>     using bfq_add/del_bfqq_busy() to count the number of groups
>     (suggested by Jan Kara).
> 
> Changes in v2:
>   - Use a different approch to count root group, which is much simple.
> 
> Currently, bfq can't handle sync io concurrently as long as they
> are not issued from root group. This is because
> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
> bfq_asymmetric_scenario().
> 
> The way that bfqg is counted into 'num_groups_with_pending_reqs':
> 
> Before this patchset:
>   1) root group will never be counted.
>   2) Count if bfqg or it's child bfqgs have pending requests.
>   3) Don't count if bfqg and it's child bfqgs complete all the requests.
> 
> After this patchset:
>   1) root group is counted.
>   2) Count if bfqg has pending requests.
>   3) Don't count if bfqg complete all the requests.
> 
> With the above changes, concurrent sync io can be supported if only
> one group is activated.
> 
> fio test script(startdelay is used to avoid queue merging):
> [global]
> filename=/dev/sda
> allow_mounted_write=0
> ioengine=psync
> direct=1
> ioscheduler=bfq
> offset_increment=10g
> group_reporting
> rw=randwrite
> bs=4k
> 
> [test1]
> numjobs=1
> 
> [test2]
> startdelay=1
> numjobs=1
> 
> [test3]
> startdelay=2
> numjobs=1
> 
> [test4]
> startdelay=3
> numjobs=1
> 
> [test5]
> startdelay=4
> numjobs=1
> 
> [test6]
> startdelay=5
> numjobs=1
> 
> [test7]
> startdelay=6
> numjobs=1
> 
> [test8]
> startdelay=7
> numjobs=1
> 
> test result:
> running fio on root cgroup
> v5.18:	   112 Mib/s
> v5.18-patched: 112 Mib/s
> 
> running fio on non-root cgroup
> v5.18:	   51.2 Mib/s
> v5.18-patched: 112 Mib/s
> 
> Note that I also test null_blk with "irqmode=2
> completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
> that service guarantees are still preserved.
> 
> Previous versions:
> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/
> 
> 
> Yu Kuai (4):
>    block, bfq: support to track if bfqq has pending requests
>    block, bfq: record how many queues have pending requests
>    block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>    block, bfq: do not idle if only one group is activated
> 
>   block/bfq-cgroup.c  | 10 ++++++++++
>   block/bfq-iosched.c | 47 +++------------------------------------------
>   block/bfq-iosched.h | 21 +++++++++++---------
>   block/bfq-wf2q.c    | 47 ++++++++++++++++++++++++++++++---------------
>   4 files changed, 57 insertions(+), 68 deletions(-)
> 

^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 0/4] support concurrent sync io for bfq on a specail occasion
@ 2022-06-17  1:12   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-17  1:12 UTC (permalink / raw)
  To: paolo.valente-QSEj5FYQhm4dnm+yROfE0A, jack-AlSwsSmVLrQ,
	cgroups-u79uwXL29TY76Z2rM5mHXA,
	linux-block-u79uwXL29TY76Z2rM5mHXA
  Cc: tj-DgEjT+Ai2ygdnm+yROfE0A, axboe-tSWWG44O7X1aa/9Udqfwiw,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	yi.zhang-hv44wF8Li93QT0dZR+AlfA

Hi, Paolo

Can you take a look at this patchset?

Thanks,
Kuai

ÔÚ 2022/06/10 10:16, Yu Kuai дµÀ:
> There are some problem in our mail server, resend v9 because Paolo seems
> didn't receive them(hopefully v10 will be fine).
> 
> Changes in v10:
>   - Add reviewed-tag for patch 2
> 
> Changes in v9:
>   - also update how many bfqqs have pending_reqs bfq_bfqq_move().
>   - fix one language in patch 4
>   - Add reviewed-tag for patch 1,3,4
> 
> Changes in v8:
>   - Instead of using whether bfqq is busy, using whether bfqq has pending
>   requests. As Paolo pointed out the former way is problematic.
> 
> Changes in v7:
>   - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
>   also retest this patchset on v5.18 to make sure functionality is
>   correct.
>   - move the updating of 'bfqd->busy_queues' into new apis
> 
> Changes in v6:
>   - add reviewed-by tag for patch 1
> 
> Changes in v5:
>   - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
>   - fix wrong definition in patch 1
>   - fix spelling mistake in patch 2: leaset -> least
>   - update comments in patch 3
>   - add reviewed-by tag in patch 2,3
> 
> Changes in v4:
>   - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
>     suggested by Jan Kara.
>   - remove unused 'in_groups_with_pending_reqs',
> 
> Changes in v3:
>   - remove the cleanup patch that is irrelevant now(I'll post it
>     separately).
>   - instead of hacking wr queues and using weights tree insertion/removal,
>     using bfq_add/del_bfqq_busy() to count the number of groups
>     (suggested by Jan Kara).
> 
> Changes in v2:
>   - Use a different approch to count root group, which is much simple.
> 
> Currently, bfq can't handle sync io concurrently as long as they
> are not issued from root group. This is because
> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
> bfq_asymmetric_scenario().
> 
> The way that bfqg is counted into 'num_groups_with_pending_reqs':
> 
> Before this patchset:
>   1) root group will never be counted.
>   2) Count if bfqg or it's child bfqgs have pending requests.
>   3) Don't count if bfqg and it's child bfqgs complete all the requests.
> 
> After this patchset:
>   1) root group is counted.
>   2) Count if bfqg has pending requests.
>   3) Don't count if bfqg complete all the requests.
> 
> With the above changes, concurrent sync io can be supported if only
> one group is activated.
> 
> fio test script(startdelay is used to avoid queue merging):
> [global]
> filename=/dev/sda
> allow_mounted_write=0
> ioengine=psync
> direct=1
> ioscheduler=bfq
> offset_increment=10g
> group_reporting
> rw=randwrite
> bs=4k
> 
> [test1]
> numjobs=1
> 
> [test2]
> startdelay=1
> numjobs=1
> 
> [test3]
> startdelay=2
> numjobs=1
> 
> [test4]
> startdelay=3
> numjobs=1
> 
> [test5]
> startdelay=4
> numjobs=1
> 
> [test6]
> startdelay=5
> numjobs=1
> 
> [test7]
> startdelay=6
> numjobs=1
> 
> [test8]
> startdelay=7
> numjobs=1
> 
> test result:
> running fio on root cgroup
> v5.18:	   112 Mib/s
> v5.18-patched: 112 Mib/s
> 
> running fio on non-root cgroup
> v5.18:	   51.2 Mib/s
> v5.18-patched: 112 Mib/s
> 
> Note that I also test null_blk with "irqmode=2
> completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
> that service guarantees are still preserved.
> 
> Previous versions:
> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
> 
> 
> Yu Kuai (4):
>    block, bfq: support to track if bfqq has pending requests
>    block, bfq: record how many queues have pending requests
>    block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>    block, bfq: do not idle if only one group is activated
> 
>   block/bfq-cgroup.c  | 10 ++++++++++
>   block/bfq-iosched.c | 47 +++------------------------------------------
>   block/bfq-iosched.h | 21 +++++++++++---------
>   block/bfq-wf2q.c    | 47 ++++++++++++++++++++++++++++++---------------
>   4 files changed, 57 insertions(+), 68 deletions(-)
> 

^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-06-23 15:32     ` Paolo Valente
  0 siblings, 0 replies; 38+ messages in thread
From: Paolo Valente @ 2022-06-23 15:32 UTC (permalink / raw)
  To: Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe,
	linux-kernel, yi.zhang

Sorry for the delay.

> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com> ha scritto:
> 
> Currently, bfq can't handle sync io concurrently as long as they
> are not issued from root group. This is because
> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
> bfq_asymmetric_scenario().
> 
> The way that bfqg is counted into 'num_groups_with_pending_reqs':
> 
> Before this patch:
> 1) root group will never be counted.
> 2) Count if bfqg or it's child bfqgs have pending requests.
> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
> 
> After this patch:
> 1) root group is counted.
> 2) Count if bfqg have pending requests.
> 3) Don't count if bfqg complete all the requests.
> 
> With this change, the occasion that only one group is activated can be
> detected, and next patch will support concurrent sync io in the
> occasion.
> 
> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
> Reviewed-by: Jan Kara <jack@suse.cz>
> ---
> block/bfq-iosched.c | 42 ------------------------------------------
> block/bfq-iosched.h | 18 +++++++++---------
> block/bfq-wf2q.c    | 19 ++++---------------
> 3 files changed, 13 insertions(+), 66 deletions(-)
> 
> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
> index 0ec21018daba..03b04892440c 100644
> --- a/block/bfq-iosched.c
> +++ b/block/bfq-iosched.c
> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
> void bfq_weights_tree_remove(struct bfq_data *bfqd,
> 			     struct bfq_queue *bfqq)
> {
> -	struct bfq_entity *entity = bfqq->entity.parent;
> -
> -	for_each_entity(entity) {
> -		struct bfq_sched_data *sd = entity->my_sched_data;
> -
> -		if (sd->next_in_service || sd->in_service_entity) {
> -			/*
> -			 * entity is still active, because either
> -			 * next_in_service or in_service_entity is not
> -			 * NULL (see the comments on the definition of
> -			 * next_in_service for details on why
> -			 * in_service_entity must be checked too).
> -			 *
> -			 * As a consequence, its parent entities are
> -			 * active as well, and thus this loop must
> -			 * stop here.
> -			 */
> -			break;
> -		}
> -
> -		/*
> -		 * The decrement of num_groups_with_pending_reqs is
> -		 * not performed immediately upon the deactivation of
> -		 * entity, but it is delayed to when it also happens
> -		 * that the first leaf descendant bfqq of entity gets
> -		 * all its pending requests completed. The following
> -		 * instructions perform this delayed decrement, if
> -		 * needed. See the comments on
> -		 * num_groups_with_pending_reqs for details.
> -		 */
> -		if (entity->in_groups_with_pending_reqs) {
> -			entity->in_groups_with_pending_reqs = false;
> -			bfqd->num_groups_with_pending_reqs--;
> -		}
> -	}

With this part removed, I'm missing how you handle the following
sequence of events:
1.  a queue Q becomes non busy but still has dispatched requests, so
it must not be removed from the counter of queues with pending reqs
yet
2.  the last request of Q is completed with Q being still idle (non
busy).  At this point Q must be removed from the counter.  It seems to
me that this case is not handled any longer

Additional comment: if your changes do not cpus the problem above,
then this function only invokes __bfq_weights_tree_remove.  So what's
the point in keeping this function)

> -
> -	/*
> -	 * Next function is invoked last, because it causes bfqq to be
> -	 * freed if the following holds: bfqq is not in service and
> -	 * has no dispatched request. DO NOT use bfqq after the next
> -	 * function invocation.
> -	 */

I would really love it if you leave this comment.  I added it after
suffering a lot for a nasty UAF.  Of course the first sentence may
need to be adjusted if the code that precedes it is to be removed.

Thanks,
Paolo


> 	__bfq_weights_tree_remove(bfqd, bfqq,
> 				  &bfqd->queue_weights_tree);
> }
> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
> index de2446a9b7ab..f0fce94583e4 100644
> --- a/block/bfq-iosched.h
> +++ b/block/bfq-iosched.h
> @@ -496,27 +496,27 @@ struct bfq_data {
> 	struct rb_root_cached queue_weights_tree;
> 
> 	/*
> -	 * Number of groups with at least one descendant process that
> +	 * Number of groups with at least one process that
> 	 * has at least one request waiting for completion. Note that
> 	 * this accounts for also requests already dispatched, but not
> 	 * yet completed. Therefore this number of groups may differ
> 	 * (be larger) than the number of active groups, as a group is
> 	 * considered active only if its corresponding entity has
> -	 * descendant queues with at least one request queued. This
> +	 * queues with at least one request queued. This
> 	 * number is used to decide whether a scenario is symmetric.
> 	 * For a detailed explanation see comments on the computation
> 	 * of the variable asymmetric_scenario in the function
> 	 * bfq_better_to_idle().
> 	 *
> 	 * However, it is hard to compute this number exactly, for
> -	 * groups with multiple descendant processes. Consider a group
> -	 * that is inactive, i.e., that has no descendant process with
> +	 * groups with multiple processes. Consider a group
> +	 * that is inactive, i.e., that has no process with
> 	 * pending I/O inside BFQ queues. Then suppose that
> 	 * num_groups_with_pending_reqs is still accounting for this
> -	 * group, because the group has descendant processes with some
> +	 * group, because the group has processes with some
> 	 * I/O request still in flight. num_groups_with_pending_reqs
> 	 * should be decremented when the in-flight request of the
> -	 * last descendant process is finally completed (assuming that
> +	 * last process is finally completed (assuming that
> 	 * nothing else has changed for the group in the meantime, in
> 	 * terms of composition of the group and active/inactive state of child
> 	 * groups and processes). To accomplish this, an additional
> @@ -525,7 +525,7 @@ struct bfq_data {
> 	 * we resort to the following tradeoff between simplicity and
> 	 * accuracy: for an inactive group that is still counted in
> 	 * num_groups_with_pending_reqs, we decrement
> -	 * num_groups_with_pending_reqs when the first descendant
> +	 * num_groups_with_pending_reqs when the first
> 	 * process of the group remains with no request waiting for
> 	 * completion.
> 	 *
> @@ -533,12 +533,12 @@ struct bfq_data {
> 	 * carefulness: to avoid multiple decrements, we flag a group,
> 	 * more precisely an entity representing a group, as still
> 	 * counted in num_groups_with_pending_reqs when it becomes
> -	 * inactive. Then, when the first descendant queue of the
> +	 * inactive. Then, when the first queue of the
> 	 * entity remains with no request waiting for completion,
> 	 * num_groups_with_pending_reqs is decremented, and this flag
> 	 * is reset. After this flag is reset for the entity,
> 	 * num_groups_with_pending_reqs won't be decremented any
> -	 * longer in case a new descendant queue of the entity remains
> +	 * longer in case a new queue of the entity remains
> 	 * with no request waiting for completion.
> 	 */
> 	unsigned int num_groups_with_pending_reqs;
> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
> index 6f36f3fe5cc8..9c2842bedf97 100644
> --- a/block/bfq-wf2q.c
> +++ b/block/bfq-wf2q.c
> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
> 		entity->on_st_or_in_serv = true;
> 	}
> 
> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
> -	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
> -		struct bfq_group *bfqg =
> -			container_of(entity, struct bfq_group, entity);
> -		struct bfq_data *bfqd = bfqg->bfqd;
> -
> -		if (!entity->in_groups_with_pending_reqs) {
> -			entity->in_groups_with_pending_reqs = true;
> -			bfqd->num_groups_with_pending_reqs++;
> -		}
> -	}
> -#endif
> -
> 	bfq_update_fin_time_enqueue(entity, st, backshifted);
> }
> 
> @@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
> 	if (!entity->in_groups_with_pending_reqs) {
> 		entity->in_groups_with_pending_reqs = true;
> #ifdef CONFIG_BFQ_GROUP_IOSCHED
> -		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
> +		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
> +			bfqq->bfqd->num_groups_with_pending_reqs++;
> #endif
> 	}
> }
> @@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
> 	if (entity->in_groups_with_pending_reqs) {
> 		entity->in_groups_with_pending_reqs = false;
> #ifdef CONFIG_BFQ_GROUP_IOSCHED
> -		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
> +		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
> +			bfqq->bfqd->num_groups_with_pending_reqs--;
> #endif
> 	}
> }
> -- 
> 2.31.1
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-06-23 15:32     ` Paolo Valente
  0 siblings, 0 replies; 38+ messages in thread
From: Paolo Valente @ 2022-06-23 15:32 UTC (permalink / raw)
  To: Yu Kuai
  Cc: Jan Kara, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-block, Tejun Heo,
	Jens Axboe, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	yi.zhang-hv44wF8Li93QT0dZR+AlfA

Sorry for the delay.

> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org> ha scritto:
> 
> Currently, bfq can't handle sync io concurrently as long as they
> are not issued from root group. This is because
> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
> bfq_asymmetric_scenario().
> 
> The way that bfqg is counted into 'num_groups_with_pending_reqs':
> 
> Before this patch:
> 1) root group will never be counted.
> 2) Count if bfqg or it's child bfqgs have pending requests.
> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
> 
> After this patch:
> 1) root group is counted.
> 2) Count if bfqg have pending requests.
> 3) Don't count if bfqg complete all the requests.
> 
> With this change, the occasion that only one group is activated can be
> detected, and next patch will support concurrent sync io in the
> occasion.
> 
> Signed-off-by: Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
> Reviewed-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
> ---
> block/bfq-iosched.c | 42 ------------------------------------------
> block/bfq-iosched.h | 18 +++++++++---------
> block/bfq-wf2q.c    | 19 ++++---------------
> 3 files changed, 13 insertions(+), 66 deletions(-)
> 
> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
> index 0ec21018daba..03b04892440c 100644
> --- a/block/bfq-iosched.c
> +++ b/block/bfq-iosched.c
> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
> void bfq_weights_tree_remove(struct bfq_data *bfqd,
> 			     struct bfq_queue *bfqq)
> {
> -	struct bfq_entity *entity = bfqq->entity.parent;
> -
> -	for_each_entity(entity) {
> -		struct bfq_sched_data *sd = entity->my_sched_data;
> -
> -		if (sd->next_in_service || sd->in_service_entity) {
> -			/*
> -			 * entity is still active, because either
> -			 * next_in_service or in_service_entity is not
> -			 * NULL (see the comments on the definition of
> -			 * next_in_service for details on why
> -			 * in_service_entity must be checked too).
> -			 *
> -			 * As a consequence, its parent entities are
> -			 * active as well, and thus this loop must
> -			 * stop here.
> -			 */
> -			break;
> -		}
> -
> -		/*
> -		 * The decrement of num_groups_with_pending_reqs is
> -		 * not performed immediately upon the deactivation of
> -		 * entity, but it is delayed to when it also happens
> -		 * that the first leaf descendant bfqq of entity gets
> -		 * all its pending requests completed. The following
> -		 * instructions perform this delayed decrement, if
> -		 * needed. See the comments on
> -		 * num_groups_with_pending_reqs for details.
> -		 */
> -		if (entity->in_groups_with_pending_reqs) {
> -			entity->in_groups_with_pending_reqs = false;
> -			bfqd->num_groups_with_pending_reqs--;
> -		}
> -	}

With this part removed, I'm missing how you handle the following
sequence of events:
1.  a queue Q becomes non busy but still has dispatched requests, so
it must not be removed from the counter of queues with pending reqs
yet
2.  the last request of Q is completed with Q being still idle (non
busy).  At this point Q must be removed from the counter.  It seems to
me that this case is not handled any longer

Additional comment: if your changes do not cpus the problem above,
then this function only invokes __bfq_weights_tree_remove.  So what's
the point in keeping this function)

> -
> -	/*
> -	 * Next function is invoked last, because it causes bfqq to be
> -	 * freed if the following holds: bfqq is not in service and
> -	 * has no dispatched request. DO NOT use bfqq after the next
> -	 * function invocation.
> -	 */

I would really love it if you leave this comment.  I added it after
suffering a lot for a nasty UAF.  Of course the first sentence may
need to be adjusted if the code that precedes it is to be removed.

Thanks,
Paolo


> 	__bfq_weights_tree_remove(bfqd, bfqq,
> 				  &bfqd->queue_weights_tree);
> }
> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
> index de2446a9b7ab..f0fce94583e4 100644
> --- a/block/bfq-iosched.h
> +++ b/block/bfq-iosched.h
> @@ -496,27 +496,27 @@ struct bfq_data {
> 	struct rb_root_cached queue_weights_tree;
> 
> 	/*
> -	 * Number of groups with at least one descendant process that
> +	 * Number of groups with at least one process that
> 	 * has at least one request waiting for completion. Note that
> 	 * this accounts for also requests already dispatched, but not
> 	 * yet completed. Therefore this number of groups may differ
> 	 * (be larger) than the number of active groups, as a group is
> 	 * considered active only if its corresponding entity has
> -	 * descendant queues with at least one request queued. This
> +	 * queues with at least one request queued. This
> 	 * number is used to decide whether a scenario is symmetric.
> 	 * For a detailed explanation see comments on the computation
> 	 * of the variable asymmetric_scenario in the function
> 	 * bfq_better_to_idle().
> 	 *
> 	 * However, it is hard to compute this number exactly, for
> -	 * groups with multiple descendant processes. Consider a group
> -	 * that is inactive, i.e., that has no descendant process with
> +	 * groups with multiple processes. Consider a group
> +	 * that is inactive, i.e., that has no process with
> 	 * pending I/O inside BFQ queues. Then suppose that
> 	 * num_groups_with_pending_reqs is still accounting for this
> -	 * group, because the group has descendant processes with some
> +	 * group, because the group has processes with some
> 	 * I/O request still in flight. num_groups_with_pending_reqs
> 	 * should be decremented when the in-flight request of the
> -	 * last descendant process is finally completed (assuming that
> +	 * last process is finally completed (assuming that
> 	 * nothing else has changed for the group in the meantime, in
> 	 * terms of composition of the group and active/inactive state of child
> 	 * groups and processes). To accomplish this, an additional
> @@ -525,7 +525,7 @@ struct bfq_data {
> 	 * we resort to the following tradeoff between simplicity and
> 	 * accuracy: for an inactive group that is still counted in
> 	 * num_groups_with_pending_reqs, we decrement
> -	 * num_groups_with_pending_reqs when the first descendant
> +	 * num_groups_with_pending_reqs when the first
> 	 * process of the group remains with no request waiting for
> 	 * completion.
> 	 *
> @@ -533,12 +533,12 @@ struct bfq_data {
> 	 * carefulness: to avoid multiple decrements, we flag a group,
> 	 * more precisely an entity representing a group, as still
> 	 * counted in num_groups_with_pending_reqs when it becomes
> -	 * inactive. Then, when the first descendant queue of the
> +	 * inactive. Then, when the first queue of the
> 	 * entity remains with no request waiting for completion,
> 	 * num_groups_with_pending_reqs is decremented, and this flag
> 	 * is reset. After this flag is reset for the entity,
> 	 * num_groups_with_pending_reqs won't be decremented any
> -	 * longer in case a new descendant queue of the entity remains
> +	 * longer in case a new queue of the entity remains
> 	 * with no request waiting for completion.
> 	 */
> 	unsigned int num_groups_with_pending_reqs;
> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
> index 6f36f3fe5cc8..9c2842bedf97 100644
> --- a/block/bfq-wf2q.c
> +++ b/block/bfq-wf2q.c
> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
> 		entity->on_st_or_in_serv = true;
> 	}
> 
> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
> -	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
> -		struct bfq_group *bfqg =
> -			container_of(entity, struct bfq_group, entity);
> -		struct bfq_data *bfqd = bfqg->bfqd;
> -
> -		if (!entity->in_groups_with_pending_reqs) {
> -			entity->in_groups_with_pending_reqs = true;
> -			bfqd->num_groups_with_pending_reqs++;
> -		}
> -	}
> -#endif
> -
> 	bfq_update_fin_time_enqueue(entity, st, backshifted);
> }
> 
> @@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
> 	if (!entity->in_groups_with_pending_reqs) {
> 		entity->in_groups_with_pending_reqs = true;
> #ifdef CONFIG_BFQ_GROUP_IOSCHED
> -		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
> +		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
> +			bfqq->bfqd->num_groups_with_pending_reqs++;
> #endif
> 	}
> }
> @@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
> 	if (entity->in_groups_with_pending_reqs) {
> 		entity->in_groups_with_pending_reqs = false;
> #ifdef CONFIG_BFQ_GROUP_IOSCHED
> -		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
> +		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
> +			bfqq->bfqd->num_groups_with_pending_reqs--;
> #endif
> 	}
> }
> -- 
> 2.31.1
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-06-23 15:32     ` Paolo Valente
@ 2022-06-24  1:26       ` Yu Kuai
  -1 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-24  1:26 UTC (permalink / raw)
  To: Paolo Valente
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe,
	linux-kernel, yi.zhang

在 2022/06/23 23:32, Paolo Valente 写道:
> Sorry for the delay.
> 
>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com> ha scritto:
>>
>> Currently, bfq can't handle sync io concurrently as long as they
>> are not issued from root group. This is because
>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>> bfq_asymmetric_scenario().
>>
>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>
>> Before this patch:
>> 1) root group will never be counted.
>> 2) Count if bfqg or it's child bfqgs have pending requests.
>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>
>> After this patch:
>> 1) root group is counted.
>> 2) Count if bfqg have pending requests.
>> 3) Don't count if bfqg complete all the requests.
>>
>> With this change, the occasion that only one group is activated can be
>> detected, and next patch will support concurrent sync io in the
>> occasion.
>>
>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>> Reviewed-by: Jan Kara <jack@suse.cz>
>> ---
>> block/bfq-iosched.c | 42 ------------------------------------------
>> block/bfq-iosched.h | 18 +++++++++---------
>> block/bfq-wf2q.c    | 19 ++++---------------
>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>
>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>> index 0ec21018daba..03b04892440c 100644
>> --- a/block/bfq-iosched.c
>> +++ b/block/bfq-iosched.c
>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>> 			     struct bfq_queue *bfqq)
>> {
>> -	struct bfq_entity *entity = bfqq->entity.parent;
>> -
>> -	for_each_entity(entity) {
>> -		struct bfq_sched_data *sd = entity->my_sched_data;
>> -
>> -		if (sd->next_in_service || sd->in_service_entity) {
>> -			/*
>> -			 * entity is still active, because either
>> -			 * next_in_service or in_service_entity is not
>> -			 * NULL (see the comments on the definition of
>> -			 * next_in_service for details on why
>> -			 * in_service_entity must be checked too).
>> -			 *
>> -			 * As a consequence, its parent entities are
>> -			 * active as well, and thus this loop must
>> -			 * stop here.
>> -			 */
>> -			break;
>> -		}
>> -
>> -		/*
>> -		 * The decrement of num_groups_with_pending_reqs is
>> -		 * not performed immediately upon the deactivation of
>> -		 * entity, but it is delayed to when it also happens
>> -		 * that the first leaf descendant bfqq of entity gets
>> -		 * all its pending requests completed. The following
>> -		 * instructions perform this delayed decrement, if
>> -		 * needed. See the comments on
>> -		 * num_groups_with_pending_reqs for details.
>> -		 */
>> -		if (entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = false;
>> -			bfqd->num_groups_with_pending_reqs--;
>> -		}
>> -	}
> 
> With this part removed, I'm missing how you handle the following
> sequence of events:
> 1.  a queue Q becomes non busy but still has dispatched requests, so
> it must not be removed from the counter of queues with pending reqs
> yet
> 2.  the last request of Q is completed with Q being still idle (non
> busy).  At this point Q must be removed from the counter.  It seems to
> me that this case is not handled any longer
> 
Hi, Paolo

1) At first, patch 1 support to track if bfqq has pending requests, it's
done by setting the flag 'entity->in_groups_with_pending_reqs' when the
first request is inserted to bfqq, and it's cleared when the last
request is completed.

2) Then, patch 2 add a counter in bfqg: how many bfqqs have pending
requests, which is updated while tracking if bfqq has pending requests.

3) Finally, patch 3 tracks 'num_groups_with_pending_reqs' based on the
new counter in patch 2:
  - if the counter(how many bfqqs have pending requests) increased from 0
    to 0, increase 'num_groups_with_pending_reqs'.
  - if the counter is decreased from 1 to 0, decrease
    'num_groups_with_pending_reqs'

> Additional comment: if your changes do not cpus the problem above,
> then this function only invokes __bfq_weights_tree_remove.  So what's
> the point in keeping this function)

If this patchset is applied, there are following cleanup patches to
remove this function.

multiple cleanup patches for bfq:
https://lore.kernel.org/all/20220528095958.270455-1-yukuai3@huawei.com/
> 
>> -
>> -	/*
>> -	 * Next function is invoked last, because it causes bfqq to be
>> -	 * freed if the following holds: bfqq is not in service and
>> -	 * has no dispatched request. DO NOT use bfqq after the next
>> -	 * function invocation.
>> -	 */
> 
> I would really love it if you leave this comment.  I added it after
> suffering a lot for a nasty UAF.  Of course the first sentence may
> need to be adjusted if the code that precedes it is to be removed.
> 

Same as above, if this patch is applied, this function will be gone.

Thanks,
Kuai
> Thanks,
> Paolo
> 
> 
>> 	__bfq_weights_tree_remove(bfqd, bfqq,
>> 				  &bfqd->queue_weights_tree);
>> }
>> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
>> index de2446a9b7ab..f0fce94583e4 100644
>> --- a/block/bfq-iosched.h
>> +++ b/block/bfq-iosched.h
>> @@ -496,27 +496,27 @@ struct bfq_data {
>> 	struct rb_root_cached queue_weights_tree;
>>
>> 	/*
>> -	 * Number of groups with at least one descendant process that
>> +	 * Number of groups with at least one process that
>> 	 * has at least one request waiting for completion. Note that
>> 	 * this accounts for also requests already dispatched, but not
>> 	 * yet completed. Therefore this number of groups may differ
>> 	 * (be larger) than the number of active groups, as a group is
>> 	 * considered active only if its corresponding entity has
>> -	 * descendant queues with at least one request queued. This
>> +	 * queues with at least one request queued. This
>> 	 * number is used to decide whether a scenario is symmetric.
>> 	 * For a detailed explanation see comments on the computation
>> 	 * of the variable asymmetric_scenario in the function
>> 	 * bfq_better_to_idle().
>> 	 *
>> 	 * However, it is hard to compute this number exactly, for
>> -	 * groups with multiple descendant processes. Consider a group
>> -	 * that is inactive, i.e., that has no descendant process with
>> +	 * groups with multiple processes. Consider a group
>> +	 * that is inactive, i.e., that has no process with
>> 	 * pending I/O inside BFQ queues. Then suppose that
>> 	 * num_groups_with_pending_reqs is still accounting for this
>> -	 * group, because the group has descendant processes with some
>> +	 * group, because the group has processes with some
>> 	 * I/O request still in flight. num_groups_with_pending_reqs
>> 	 * should be decremented when the in-flight request of the
>> -	 * last descendant process is finally completed (assuming that
>> +	 * last process is finally completed (assuming that
>> 	 * nothing else has changed for the group in the meantime, in
>> 	 * terms of composition of the group and active/inactive state of child
>> 	 * groups and processes). To accomplish this, an additional
>> @@ -525,7 +525,7 @@ struct bfq_data {
>> 	 * we resort to the following tradeoff between simplicity and
>> 	 * accuracy: for an inactive group that is still counted in
>> 	 * num_groups_with_pending_reqs, we decrement
>> -	 * num_groups_with_pending_reqs when the first descendant
>> +	 * num_groups_with_pending_reqs when the first
>> 	 * process of the group remains with no request waiting for
>> 	 * completion.
>> 	 *
>> @@ -533,12 +533,12 @@ struct bfq_data {
>> 	 * carefulness: to avoid multiple decrements, we flag a group,
>> 	 * more precisely an entity representing a group, as still
>> 	 * counted in num_groups_with_pending_reqs when it becomes
>> -	 * inactive. Then, when the first descendant queue of the
>> +	 * inactive. Then, when the first queue of the
>> 	 * entity remains with no request waiting for completion,
>> 	 * num_groups_with_pending_reqs is decremented, and this flag
>> 	 * is reset. After this flag is reset for the entity,
>> 	 * num_groups_with_pending_reqs won't be decremented any
>> -	 * longer in case a new descendant queue of the entity remains
>> +	 * longer in case a new queue of the entity remains
>> 	 * with no request waiting for completion.
>> 	 */
>> 	unsigned int num_groups_with_pending_reqs;
>> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
>> index 6f36f3fe5cc8..9c2842bedf97 100644
>> --- a/block/bfq-wf2q.c
>> +++ b/block/bfq-wf2q.c
>> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
>> 		entity->on_st_or_in_serv = true;
>> 	}
>>
>> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
>> -		struct bfq_group *bfqg =
>> -			container_of(entity, struct bfq_group, entity);
>> -		struct bfq_data *bfqd = bfqg->bfqd;
>> -
>> -		if (!entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = true;
>> -			bfqd->num_groups_with_pending_reqs++;
>> -		}
>> -	}
>> -#endif
>> -
>> 	bfq_update_fin_time_enqueue(entity, st, backshifted);
>> }
>>
>> @@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (!entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = true;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
>> +		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
>> +			bfqq->bfqd->num_groups_with_pending_reqs++;
>> #endif
>> 	}
>> }
>> @@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = false;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
>> +		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
>> +			bfqq->bfqd->num_groups_with_pending_reqs--;
>> #endif
>> 	}
>> }
>> -- 
>> 2.31.1
>>
> 
> .
> 

^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-06-24  1:26       ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-24  1:26 UTC (permalink / raw)
  To: Paolo Valente
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe,
	linux-kernel, yi.zhang

ÔÚ 2022/06/23 23:32, Paolo Valente дµÀ:
> Sorry for the delay.
> 
>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com> ha scritto:
>>
>> Currently, bfq can't handle sync io concurrently as long as they
>> are not issued from root group. This is because
>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>> bfq_asymmetric_scenario().
>>
>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>
>> Before this patch:
>> 1) root group will never be counted.
>> 2) Count if bfqg or it's child bfqgs have pending requests.
>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>
>> After this patch:
>> 1) root group is counted.
>> 2) Count if bfqg have pending requests.
>> 3) Don't count if bfqg complete all the requests.
>>
>> With this change, the occasion that only one group is activated can be
>> detected, and next patch will support concurrent sync io in the
>> occasion.
>>
>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>> Reviewed-by: Jan Kara <jack@suse.cz>
>> ---
>> block/bfq-iosched.c | 42 ------------------------------------------
>> block/bfq-iosched.h | 18 +++++++++---------
>> block/bfq-wf2q.c    | 19 ++++---------------
>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>
>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>> index 0ec21018daba..03b04892440c 100644
>> --- a/block/bfq-iosched.c
>> +++ b/block/bfq-iosched.c
>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>> 			     struct bfq_queue *bfqq)
>> {
>> -	struct bfq_entity *entity = bfqq->entity.parent;
>> -
>> -	for_each_entity(entity) {
>> -		struct bfq_sched_data *sd = entity->my_sched_data;
>> -
>> -		if (sd->next_in_service || sd->in_service_entity) {
>> -			/*
>> -			 * entity is still active, because either
>> -			 * next_in_service or in_service_entity is not
>> -			 * NULL (see the comments on the definition of
>> -			 * next_in_service for details on why
>> -			 * in_service_entity must be checked too).
>> -			 *
>> -			 * As a consequence, its parent entities are
>> -			 * active as well, and thus this loop must
>> -			 * stop here.
>> -			 */
>> -			break;
>> -		}
>> -
>> -		/*
>> -		 * The decrement of num_groups_with_pending_reqs is
>> -		 * not performed immediately upon the deactivation of
>> -		 * entity, but it is delayed to when it also happens
>> -		 * that the first leaf descendant bfqq of entity gets
>> -		 * all its pending requests completed. The following
>> -		 * instructions perform this delayed decrement, if
>> -		 * needed. See the comments on
>> -		 * num_groups_with_pending_reqs for details.
>> -		 */
>> -		if (entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = false;
>> -			bfqd->num_groups_with_pending_reqs--;
>> -		}
>> -	}
> 
> With this part removed, I'm missing how you handle the following
> sequence of events:
> 1.  a queue Q becomes non busy but still has dispatched requests, so
> it must not be removed from the counter of queues with pending reqs
> yet
> 2.  the last request of Q is completed with Q being still idle (non
> busy).  At this point Q must be removed from the counter.  It seems to
> me that this case is not handled any longer
> 
Hi, Paolo

1) At first, patch 1 support to track if bfqq has pending requests, it's
done by setting the flag 'entity->in_groups_with_pending_reqs' when the
first request is inserted to bfqq, and it's cleared when the last
request is completed.

2) Then, patch 2 add a counter in bfqg: how many bfqqs have pending
requests, which is updated while tracking if bfqq has pending requests.

3) Finally, patch 3 tracks 'num_groups_with_pending_reqs' based on the
new counter in patch 2:
  - if the counter(how many bfqqs have pending requests) increased from 0
    to 0, increase 'num_groups_with_pending_reqs'.
  - if the counter is decreased from 1 to 0, decrease
    'num_groups_with_pending_reqs'

> Additional comment: if your changes do not cpus the problem above,
> then this function only invokes __bfq_weights_tree_remove.  So what's
> the point in keeping this function)

If this patchset is applied, there are following cleanup patches to
remove this function.

multiple cleanup patches for bfq:
https://lore.kernel.org/all/20220528095958.270455-1-yukuai3@huawei.com/
> 
>> -
>> -	/*
>> -	 * Next function is invoked last, because it causes bfqq to be
>> -	 * freed if the following holds: bfqq is not in service and
>> -	 * has no dispatched request. DO NOT use bfqq after the next
>> -	 * function invocation.
>> -	 */
> 
> I would really love it if you leave this comment.  I added it after
> suffering a lot for a nasty UAF.  Of course the first sentence may
> need to be adjusted if the code that precedes it is to be removed.
> 

Same as above, if this patch is applied, this function will be gone.

Thanks,
Kuai
> Thanks,
> Paolo
> 
> 
>> 	__bfq_weights_tree_remove(bfqd, bfqq,
>> 				  &bfqd->queue_weights_tree);
>> }
>> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
>> index de2446a9b7ab..f0fce94583e4 100644
>> --- a/block/bfq-iosched.h
>> +++ b/block/bfq-iosched.h
>> @@ -496,27 +496,27 @@ struct bfq_data {
>> 	struct rb_root_cached queue_weights_tree;
>>
>> 	/*
>> -	 * Number of groups with at least one descendant process that
>> +	 * Number of groups with at least one process that
>> 	 * has at least one request waiting for completion. Note that
>> 	 * this accounts for also requests already dispatched, but not
>> 	 * yet completed. Therefore this number of groups may differ
>> 	 * (be larger) than the number of active groups, as a group is
>> 	 * considered active only if its corresponding entity has
>> -	 * descendant queues with at least one request queued. This
>> +	 * queues with at least one request queued. This
>> 	 * number is used to decide whether a scenario is symmetric.
>> 	 * For a detailed explanation see comments on the computation
>> 	 * of the variable asymmetric_scenario in the function
>> 	 * bfq_better_to_idle().
>> 	 *
>> 	 * However, it is hard to compute this number exactly, for
>> -	 * groups with multiple descendant processes. Consider a group
>> -	 * that is inactive, i.e., that has no descendant process with
>> +	 * groups with multiple processes. Consider a group
>> +	 * that is inactive, i.e., that has no process with
>> 	 * pending I/O inside BFQ queues. Then suppose that
>> 	 * num_groups_with_pending_reqs is still accounting for this
>> -	 * group, because the group has descendant processes with some
>> +	 * group, because the group has processes with some
>> 	 * I/O request still in flight. num_groups_with_pending_reqs
>> 	 * should be decremented when the in-flight request of the
>> -	 * last descendant process is finally completed (assuming that
>> +	 * last process is finally completed (assuming that
>> 	 * nothing else has changed for the group in the meantime, in
>> 	 * terms of composition of the group and active/inactive state of child
>> 	 * groups and processes). To accomplish this, an additional
>> @@ -525,7 +525,7 @@ struct bfq_data {
>> 	 * we resort to the following tradeoff between simplicity and
>> 	 * accuracy: for an inactive group that is still counted in
>> 	 * num_groups_with_pending_reqs, we decrement
>> -	 * num_groups_with_pending_reqs when the first descendant
>> +	 * num_groups_with_pending_reqs when the first
>> 	 * process of the group remains with no request waiting for
>> 	 * completion.
>> 	 *
>> @@ -533,12 +533,12 @@ struct bfq_data {
>> 	 * carefulness: to avoid multiple decrements, we flag a group,
>> 	 * more precisely an entity representing a group, as still
>> 	 * counted in num_groups_with_pending_reqs when it becomes
>> -	 * inactive. Then, when the first descendant queue of the
>> +	 * inactive. Then, when the first queue of the
>> 	 * entity remains with no request waiting for completion,
>> 	 * num_groups_with_pending_reqs is decremented, and this flag
>> 	 * is reset. After this flag is reset for the entity,
>> 	 * num_groups_with_pending_reqs won't be decremented any
>> -	 * longer in case a new descendant queue of the entity remains
>> +	 * longer in case a new queue of the entity remains
>> 	 * with no request waiting for completion.
>> 	 */
>> 	unsigned int num_groups_with_pending_reqs;
>> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
>> index 6f36f3fe5cc8..9c2842bedf97 100644
>> --- a/block/bfq-wf2q.c
>> +++ b/block/bfq-wf2q.c
>> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
>> 		entity->on_st_or_in_serv = true;
>> 	}
>>
>> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
>> -		struct bfq_group *bfqg =
>> -			container_of(entity, struct bfq_group, entity);
>> -		struct bfq_data *bfqd = bfqg->bfqd;
>> -
>> -		if (!entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = true;
>> -			bfqd->num_groups_with_pending_reqs++;
>> -		}
>> -	}
>> -#endif
>> -
>> 	bfq_update_fin_time_enqueue(entity, st, backshifted);
>> }
>>
>> @@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (!entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = true;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
>> +		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
>> +			bfqq->bfqd->num_groups_with_pending_reqs++;
>> #endif
>> 	}
>> }
>> @@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = false;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
>> +		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
>> +			bfqq->bfqd->num_groups_with_pending_reqs--;
>> #endif
>> 	}
>> }
>> -- 
>> 2.31.1
>>
> 
> .
> 

^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-06-25  8:10         ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-25  8:10 UTC (permalink / raw)
  To: Paolo Valente
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe,
	linux-kernel, yi.zhang

在 2022/06/24 9:26, Yu Kuai 写道:
> 在 2022/06/23 23:32, Paolo Valente 写道:
>> Sorry for the delay.
>>
>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com> 
>>> ha scritto:
>>>
>>> Currently, bfq can't handle sync io concurrently as long as they
>>> are not issued from root group. This is because
>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>> bfq_asymmetric_scenario().
>>>
>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>
>>> Before this patch:
>>> 1) root group will never be counted.
>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>
>>> After this patch:
>>> 1) root group is counted.
>>> 2) Count if bfqg have pending requests.
>>> 3) Don't count if bfqg complete all the requests.
>>>
>>> With this change, the occasion that only one group is activated can be
>>> detected, and next patch will support concurrent sync io in the
>>> occasion.
>>>
>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>>> Reviewed-by: Jan Kara <jack@suse.cz>
>>> ---
>>> block/bfq-iosched.c | 42 ------------------------------------------
>>> block/bfq-iosched.h | 18 +++++++++---------
>>> block/bfq-wf2q.c    | 19 ++++---------------
>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>
>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>> index 0ec21018daba..03b04892440c 100644
>>> --- a/block/bfq-iosched.c
>>> +++ b/block/bfq-iosched.c
>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data 
>>> *bfqd,
>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>                  struct bfq_queue *bfqq)
>>> {
>>> -    struct bfq_entity *entity = bfqq->entity.parent;
>>> -
>>> -    for_each_entity(entity) {
>>> -        struct bfq_sched_data *sd = entity->my_sched_data;
>>> -
>>> -        if (sd->next_in_service || sd->in_service_entity) {
>>> -            /*
>>> -             * entity is still active, because either
>>> -             * next_in_service or in_service_entity is not
>>> -             * NULL (see the comments on the definition of
>>> -             * next_in_service for details on why
>>> -             * in_service_entity must be checked too).
>>> -             *
>>> -             * As a consequence, its parent entities are
>>> -             * active as well, and thus this loop must
>>> -             * stop here.
>>> -             */
>>> -            break;
>>> -        }
>>> -
>>> -        /*
>>> -         * The decrement of num_groups_with_pending_reqs is
>>> -         * not performed immediately upon the deactivation of
>>> -         * entity, but it is delayed to when it also happens
>>> -         * that the first leaf descendant bfqq of entity gets
>>> -         * all its pending requests completed. The following
>>> -         * instructions perform this delayed decrement, if
>>> -         * needed. See the comments on
>>> -         * num_groups_with_pending_reqs for details.
>>> -         */
>>> -        if (entity->in_groups_with_pending_reqs) {
>>> -            entity->in_groups_with_pending_reqs = false;
>>> -            bfqd->num_groups_with_pending_reqs--;
>>> -        }
>>> -    }
>>
>> With this part removed, I'm missing how you handle the following
>> sequence of events:
>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>> it must not be removed from the counter of queues with pending reqs
>> yet
>> 2.  the last request of Q is completed with Q being still idle (non
>> busy).  At this point Q must be removed from the counter.  It seems to
>> me that this case is not handled any longer
>>
> Hi, Paolo
> 
> 1) At first, patch 1 support to track if bfqq has pending requests, it's
> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
> first request is inserted to bfqq, and it's cleared when the last
> request is completed.
> 
> 2) Then, patch 2 add a counter in bfqg: how many bfqqs have pending
> requests, which is updated while tracking if bfqq has pending requests.
> 
> 3) Finally, patch 3 tracks 'num_groups_with_pending_reqs' based on the
> new counter in patch 2:
>   - if the counter(how many bfqqs have pending requests) increased from 0
>     to 0, increase 'num_groups_with_pending_reqs'.
Hi, Paolo

Sorry that I made a mistake here:
increased from 0 to 0 -> increased from 0 to 1.

look forward to your reply
Kuai
>   - if the counter is decreased from 1 to 0, decrease
>     'num_groups_with_pending_reqs'
> 
>> Additional comment: if your changes do not cpus the problem above,
>> then this function only invokes __bfq_weights_tree_remove.  So what's
>> the point in keeping this function)
> 
> If this patchset is applied, there are following cleanup patches to
> remove this function.
> 
> multiple cleanup patches for bfq:
> https://lore.kernel.org/all/20220528095958.270455-1-yukuai3@huawei.com/
>>
>>> -
>>> -    /*
>>> -     * Next function is invoked last, because it causes bfqq to be
>>> -     * freed if the following holds: bfqq is not in service and
>>> -     * has no dispatched request. DO NOT use bfqq after the next
>>> -     * function invocation.
>>> -     */
>>
>> I would really love it if you leave this comment.  I added it after
>> suffering a lot for a nasty UAF.  Of course the first sentence may
>> need to be adjusted if the code that precedes it is to be removed.
>>
> 
> Same as above, if this patch is applied, this function will be gone.
> 
> Thanks,
> Kuai
>> Thanks,
>> Paolo
>>
>>
>>>     __bfq_weights_tree_remove(bfqd, bfqq,
>>>                   &bfqd->queue_weights_tree);
>>> }
>>> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
>>> index de2446a9b7ab..f0fce94583e4 100644
>>> --- a/block/bfq-iosched.h
>>> +++ b/block/bfq-iosched.h
>>> @@ -496,27 +496,27 @@ struct bfq_data {
>>>     struct rb_root_cached queue_weights_tree;
>>>
>>>     /*
>>> -     * Number of groups with at least one descendant process that
>>> +     * Number of groups with at least one process that
>>>      * has at least one request waiting for completion. Note that
>>>      * this accounts for also requests already dispatched, but not
>>>      * yet completed. Therefore this number of groups may differ
>>>      * (be larger) than the number of active groups, as a group is
>>>      * considered active only if its corresponding entity has
>>> -     * descendant queues with at least one request queued. This
>>> +     * queues with at least one request queued. This
>>>      * number is used to decide whether a scenario is symmetric.
>>>      * For a detailed explanation see comments on the computation
>>>      * of the variable asymmetric_scenario in the function
>>>      * bfq_better_to_idle().
>>>      *
>>>      * However, it is hard to compute this number exactly, for
>>> -     * groups with multiple descendant processes. Consider a group
>>> -     * that is inactive, i.e., that has no descendant process with
>>> +     * groups with multiple processes. Consider a group
>>> +     * that is inactive, i.e., that has no process with
>>>      * pending I/O inside BFQ queues. Then suppose that
>>>      * num_groups_with_pending_reqs is still accounting for this
>>> -     * group, because the group has descendant processes with some
>>> +     * group, because the group has processes with some
>>>      * I/O request still in flight. num_groups_with_pending_reqs
>>>      * should be decremented when the in-flight request of the
>>> -     * last descendant process is finally completed (assuming that
>>> +     * last process is finally completed (assuming that
>>>      * nothing else has changed for the group in the meantime, in
>>>      * terms of composition of the group and active/inactive state of 
>>> child
>>>      * groups and processes). To accomplish this, an additional
>>> @@ -525,7 +525,7 @@ struct bfq_data {
>>>      * we resort to the following tradeoff between simplicity and
>>>      * accuracy: for an inactive group that is still counted in
>>>      * num_groups_with_pending_reqs, we decrement
>>> -     * num_groups_with_pending_reqs when the first descendant
>>> +     * num_groups_with_pending_reqs when the first
>>>      * process of the group remains with no request waiting for
>>>      * completion.
>>>      *
>>> @@ -533,12 +533,12 @@ struct bfq_data {
>>>      * carefulness: to avoid multiple decrements, we flag a group,
>>>      * more precisely an entity representing a group, as still
>>>      * counted in num_groups_with_pending_reqs when it becomes
>>> -     * inactive. Then, when the first descendant queue of the
>>> +     * inactive. Then, when the first queue of the
>>>      * entity remains with no request waiting for completion,
>>>      * num_groups_with_pending_reqs is decremented, and this flag
>>>      * is reset. After this flag is reset for the entity,
>>>      * num_groups_with_pending_reqs won't be decremented any
>>> -     * longer in case a new descendant queue of the entity remains
>>> +     * longer in case a new queue of the entity remains
>>>      * with no request waiting for completion.
>>>      */
>>>     unsigned int num_groups_with_pending_reqs;
>>> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
>>> index 6f36f3fe5cc8..9c2842bedf97 100644
>>> --- a/block/bfq-wf2q.c
>>> +++ b/block/bfq-wf2q.c
>>> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct 
>>> bfq_entity *entity,
>>>         entity->on_st_or_in_serv = true;
>>>     }
>>>
>>> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
>>> -    if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
>>> -        struct bfq_group *bfqg =
>>> -            container_of(entity, struct bfq_group, entity);
>>> -        struct bfq_data *bfqd = bfqg->bfqd;
>>> -
>>> -        if (!entity->in_groups_with_pending_reqs) {
>>> -            entity->in_groups_with_pending_reqs = true;
>>> -            bfqd->num_groups_with_pending_reqs++;
>>> -        }
>>> -    }
>>> -#endif
>>> -
>>>     bfq_update_fin_time_enqueue(entity, st, backshifted);
>>> }
>>>
>>> @@ -1654,7 +1641,8 @@ void 
>>> bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>>>     if (!entity->in_groups_with_pending_reqs) {
>>>         entity->in_groups_with_pending_reqs = true;
>>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>>> -        bfqq_group(bfqq)->num_queues_with_pending_reqs++;
>>> +        if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
>>> +            bfqq->bfqd->num_groups_with_pending_reqs++;
>>> #endif
>>>     }
>>> }
>>> @@ -1666,7 +1654,8 @@ void 
>>> bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>>>     if (entity->in_groups_with_pending_reqs) {
>>>         entity->in_groups_with_pending_reqs = false;
>>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>>> -        bfqq_group(bfqq)->num_queues_with_pending_reqs--;
>>> +        if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
>>> +            bfqq->bfqd->num_groups_with_pending_reqs--;
>>> #endif
>>>     }
>>> }
>>> -- 
>>> 2.31.1
>>>
>>
>> .
>>

^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-06-25  8:10         ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-06-25  8:10 UTC (permalink / raw)
  To: Paolo Valente
  Cc: Jan Kara, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-block, Tejun Heo,
	Jens Axboe, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
	yi.zhang-hv44wF8Li93QT0dZR+AlfA

在 2022/06/24 9:26, Yu Kuai 写道:
> 在 2022/06/23 23:32, Paolo Valente 写道:
>> Sorry for the delay.
>>
>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org> 
>>> ha scritto:
>>>
>>> Currently, bfq can't handle sync io concurrently as long as they
>>> are not issued from root group. This is because
>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>> bfq_asymmetric_scenario().
>>>
>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>
>>> Before this patch:
>>> 1) root group will never be counted.
>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>
>>> After this patch:
>>> 1) root group is counted.
>>> 2) Count if bfqg have pending requests.
>>> 3) Don't count if bfqg complete all the requests.
>>>
>>> With this change, the occasion that only one group is activated can be
>>> detected, and next patch will support concurrent sync io in the
>>> occasion.
>>>
>>> Signed-off-by: Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>
>>> Reviewed-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
>>> ---
>>> block/bfq-iosched.c | 42 ------------------------------------------
>>> block/bfq-iosched.h | 18 +++++++++---------
>>> block/bfq-wf2q.c    | 19 ++++---------------
>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>
>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>> index 0ec21018daba..03b04892440c 100644
>>> --- a/block/bfq-iosched.c
>>> +++ b/block/bfq-iosched.c
>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data 
>>> *bfqd,
>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>                  struct bfq_queue *bfqq)
>>> {
>>> -    struct bfq_entity *entity = bfqq->entity.parent;
>>> -
>>> -    for_each_entity(entity) {
>>> -        struct bfq_sched_data *sd = entity->my_sched_data;
>>> -
>>> -        if (sd->next_in_service || sd->in_service_entity) {
>>> -            /*
>>> -             * entity is still active, because either
>>> -             * next_in_service or in_service_entity is not
>>> -             * NULL (see the comments on the definition of
>>> -             * next_in_service for details on why
>>> -             * in_service_entity must be checked too).
>>> -             *
>>> -             * As a consequence, its parent entities are
>>> -             * active as well, and thus this loop must
>>> -             * stop here.
>>> -             */
>>> -            break;
>>> -        }
>>> -
>>> -        /*
>>> -         * The decrement of num_groups_with_pending_reqs is
>>> -         * not performed immediately upon the deactivation of
>>> -         * entity, but it is delayed to when it also happens
>>> -         * that the first leaf descendant bfqq of entity gets
>>> -         * all its pending requests completed. The following
>>> -         * instructions perform this delayed decrement, if
>>> -         * needed. See the comments on
>>> -         * num_groups_with_pending_reqs for details.
>>> -         */
>>> -        if (entity->in_groups_with_pending_reqs) {
>>> -            entity->in_groups_with_pending_reqs = false;
>>> -            bfqd->num_groups_with_pending_reqs--;
>>> -        }
>>> -    }
>>
>> With this part removed, I'm missing how you handle the following
>> sequence of events:
>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>> it must not be removed from the counter of queues with pending reqs
>> yet
>> 2.  the last request of Q is completed with Q being still idle (non
>> busy).  At this point Q must be removed from the counter.  It seems to
>> me that this case is not handled any longer
>>
> Hi, Paolo
> 
> 1) At first, patch 1 support to track if bfqq has pending requests, it's
> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
> first request is inserted to bfqq, and it's cleared when the last
> request is completed.
> 
> 2) Then, patch 2 add a counter in bfqg: how many bfqqs have pending
> requests, which is updated while tracking if bfqq has pending requests.
> 
> 3) Finally, patch 3 tracks 'num_groups_with_pending_reqs' based on the
> new counter in patch 2:
>   - if the counter(how many bfqqs have pending requests) increased from 0
>     to 0, increase 'num_groups_with_pending_reqs'.
Hi, Paolo

Sorry that I made a mistake here:
increased from 0 to 0 -> increased from 0 to 1.

look forward to your reply
Kuai
>   - if the counter is decreased from 1 to 0, decrease
>     'num_groups_with_pending_reqs'
> 
>> Additional comment: if your changes do not cpus the problem above,
>> then this function only invokes __bfq_weights_tree_remove.  So what's
>> the point in keeping this function)
> 
> If this patchset is applied, there are following cleanup patches to
> remove this function.
> 
> multiple cleanup patches for bfq:
> https://lore.kernel.org/all/20220528095958.270455-1-yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org/
>>
>>> -
>>> -    /*
>>> -     * Next function is invoked last, because it causes bfqq to be
>>> -     * freed if the following holds: bfqq is not in service and
>>> -     * has no dispatched request. DO NOT use bfqq after the next
>>> -     * function invocation.
>>> -     */
>>
>> I would really love it if you leave this comment.  I added it after
>> suffering a lot for a nasty UAF.  Of course the first sentence may
>> need to be adjusted if the code that precedes it is to be removed.
>>
> 
> Same as above, if this patch is applied, this function will be gone.
> 
> Thanks,
> Kuai
>> Thanks,
>> Paolo
>>
>>
>>>     __bfq_weights_tree_remove(bfqd, bfqq,
>>>                   &bfqd->queue_weights_tree);
>>> }
>>> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
>>> index de2446a9b7ab..f0fce94583e4 100644
>>> --- a/block/bfq-iosched.h
>>> +++ b/block/bfq-iosched.h
>>> @@ -496,27 +496,27 @@ struct bfq_data {
>>>     struct rb_root_cached queue_weights_tree;
>>>
>>>     /*
>>> -     * Number of groups with at least one descendant process that
>>> +     * Number of groups with at least one process that
>>>      * has at least one request waiting for completion. Note that
>>>      * this accounts for also requests already dispatched, but not
>>>      * yet completed. Therefore this number of groups may differ
>>>      * (be larger) than the number of active groups, as a group is
>>>      * considered active only if its corresponding entity has
>>> -     * descendant queues with at least one request queued. This
>>> +     * queues with at least one request queued. This
>>>      * number is used to decide whether a scenario is symmetric.
>>>      * For a detailed explanation see comments on the computation
>>>      * of the variable asymmetric_scenario in the function
>>>      * bfq_better_to_idle().
>>>      *
>>>      * However, it is hard to compute this number exactly, for
>>> -     * groups with multiple descendant processes. Consider a group
>>> -     * that is inactive, i.e., that has no descendant process with
>>> +     * groups with multiple processes. Consider a group
>>> +     * that is inactive, i.e., that has no process with
>>>      * pending I/O inside BFQ queues. Then suppose that
>>>      * num_groups_with_pending_reqs is still accounting for this
>>> -     * group, because the group has descendant processes with some
>>> +     * group, because the group has processes with some
>>>      * I/O request still in flight. num_groups_with_pending_reqs
>>>      * should be decremented when the in-flight request of the
>>> -     * last descendant process is finally completed (assuming that
>>> +     * last process is finally completed (assuming that
>>>      * nothing else has changed for the group in the meantime, in
>>>      * terms of composition of the group and active/inactive state of 
>>> child
>>>      * groups and processes). To accomplish this, an additional
>>> @@ -525,7 +525,7 @@ struct bfq_data {
>>>      * we resort to the following tradeoff between simplicity and
>>>      * accuracy: for an inactive group that is still counted in
>>>      * num_groups_with_pending_reqs, we decrement
>>> -     * num_groups_with_pending_reqs when the first descendant
>>> +     * num_groups_with_pending_reqs when the first
>>>      * process of the group remains with no request waiting for
>>>      * completion.
>>>      *
>>> @@ -533,12 +533,12 @@ struct bfq_data {
>>>      * carefulness: to avoid multiple decrements, we flag a group,
>>>      * more precisely an entity representing a group, as still
>>>      * counted in num_groups_with_pending_reqs when it becomes
>>> -     * inactive. Then, when the first descendant queue of the
>>> +     * inactive. Then, when the first queue of the
>>>      * entity remains with no request waiting for completion,
>>>      * num_groups_with_pending_reqs is decremented, and this flag
>>>      * is reset. After this flag is reset for the entity,
>>>      * num_groups_with_pending_reqs won't be decremented any
>>> -     * longer in case a new descendant queue of the entity remains
>>> +     * longer in case a new queue of the entity remains
>>>      * with no request waiting for completion.
>>>      */
>>>     unsigned int num_groups_with_pending_reqs;
>>> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
>>> index 6f36f3fe5cc8..9c2842bedf97 100644
>>> --- a/block/bfq-wf2q.c
>>> +++ b/block/bfq-wf2q.c
>>> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct 
>>> bfq_entity *entity,
>>>         entity->on_st_or_in_serv = true;
>>>     }
>>>
>>> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
>>> -    if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
>>> -        struct bfq_group *bfqg =
>>> -            container_of(entity, struct bfq_group, entity);
>>> -        struct bfq_data *bfqd = bfqg->bfqd;
>>> -
>>> -        if (!entity->in_groups_with_pending_reqs) {
>>> -            entity->in_groups_with_pending_reqs = true;
>>> -            bfqd->num_groups_with_pending_reqs++;
>>> -        }
>>> -    }
>>> -#endif
>>> -
>>>     bfq_update_fin_time_enqueue(entity, st, backshifted);
>>> }
>>>
>>> @@ -1654,7 +1641,8 @@ void 
>>> bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>>>     if (!entity->in_groups_with_pending_reqs) {
>>>         entity->in_groups_with_pending_reqs = true;
>>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>>> -        bfqq_group(bfqq)->num_queues_with_pending_reqs++;
>>> +        if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
>>> +            bfqq->bfqd->num_groups_with_pending_reqs++;
>>> #endif
>>>     }
>>> }
>>> @@ -1666,7 +1654,8 @@ void 
>>> bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>>>     if (entity->in_groups_with_pending_reqs) {
>>>         entity->in_groups_with_pending_reqs = false;
>>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>>> -        bfqq_group(bfqq)->num_queues_with_pending_reqs--;
>>> +        if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
>>> +            bfqq->bfqd->num_groups_with_pending_reqs--;
>>> #endif
>>>     }
>>> }
>>> -- 
>>> 2.31.1
>>>
>>
>> .
>>

^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-06-23 15:32     ` Paolo Valente
@ 2022-07-12 13:30       ` Yu Kuai
  -1 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-07-12 13:30 UTC (permalink / raw)
  To: Paolo Valente
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe,
	linux-kernel, yi.zhang

Hi!

I'm copying my reply with new mail address, because Paolo seems
didn't receive my reply.

在 2022/06/23 23:32, Paolo Valente 写道:
> Sorry for the delay.
> 
>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com> ha scritto:
>>
>> Currently, bfq can't handle sync io concurrently as long as they
>> are not issued from root group. This is because
>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>> bfq_asymmetric_scenario().
>>
>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>
>> Before this patch:
>> 1) root group will never be counted.
>> 2) Count if bfqg or it's child bfqgs have pending requests.
>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>
>> After this patch:
>> 1) root group is counted.
>> 2) Count if bfqg have pending requests.
>> 3) Don't count if bfqg complete all the requests.
>>
>> With this change, the occasion that only one group is activated can be
>> detected, and next patch will support concurrent sync io in the
>> occasion.
>>
>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>> Reviewed-by: Jan Kara <jack@suse.cz>
>> ---
>> block/bfq-iosched.c | 42 ------------------------------------------
>> block/bfq-iosched.h | 18 +++++++++---------
>> block/bfq-wf2q.c    | 19 ++++---------------
>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>
>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>> index 0ec21018daba..03b04892440c 100644
>> --- a/block/bfq-iosched.c
>> +++ b/block/bfq-iosched.c
>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>> 			     struct bfq_queue *bfqq)
>> {
>> -	struct bfq_entity *entity = bfqq->entity.parent;
>> -
>> -	for_each_entity(entity) {
>> -		struct bfq_sched_data *sd = entity->my_sched_data;
>> -
>> -		if (sd->next_in_service || sd->in_service_entity) {
>> -			/*
>> -			 * entity is still active, because either
>> -			 * next_in_service or in_service_entity is not
>> -			 * NULL (see the comments on the definition of
>> -			 * next_in_service for details on why
>> -			 * in_service_entity must be checked too).
>> -			 *
>> -			 * As a consequence, its parent entities are
>> -			 * active as well, and thus this loop must
>> -			 * stop here.
>> -			 */
>> -			break;
>> -		}
>> -
>> -		/*
>> -		 * The decrement of num_groups_with_pending_reqs is
>> -		 * not performed immediately upon the deactivation of
>> -		 * entity, but it is delayed to when it also happens
>> -		 * that the first leaf descendant bfqq of entity gets
>> -		 * all its pending requests completed. The following
>> -		 * instructions perform this delayed decrement, if
>> -		 * needed. See the comments on
>> -		 * num_groups_with_pending_reqs for details.
>> -		 */
>> -		if (entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = false;
>> -			bfqd->num_groups_with_pending_reqs--;
>> -		}
>> -	}
> 
> With this part removed, I'm missing how you handle the following
> sequence of events:
> 1.  a queue Q becomes non busy but still has dispatched requests, so
> it must not be removed from the counter of queues with pending reqs
> yet
> 2.  the last request of Q is completed with Q being still idle (non
> busy).  At this point Q must be removed from the counter.  It seems to
> me that this case is not handled any longer
> 
Hi, Paolo

1) At first, patch 1 support to track if bfqq has pending requests, it's
done by setting the flag 'entity->in_groups_with_pending_reqs' when the
first request is inserted to bfqq, and it's cleared when the last
request is completed(based on weights_tree insertion and removal).

2) Then, patch 2 add a counter in bfqg: how many bfqqs have pending
requests, which is updated while tracking if bfqq has pending requests.

3) Finally, patch 3 tracks 'num_groups_with_pending_reqs' based on the
new counter in patch 2:
  - if the counter(how many bfqqs have pending requests) increased from 0
    to 1, increase 'num_groups_with_pending_reqs'.
  - if the counter is decreased from 1 to 0, decrease
    'num_groups_with_pending_reqs'

> Additional comment: if your changes do not cpus the problem above,
> then this function only invokes __bfq_weights_tree_remove.  So what's
> the point in keeping this function)
> 
If this patchset is applied, there are following cleanup patches to
remove this function.

multiple cleanup patches for bfq:
https://lore.kernel.org/all/20220528095958.270455-1-yukuai3@huawei.com/
>> -
>> -	/*
>> -	 * Next function is invoked last, because it causes bfqq to be
>> -	 * freed if the following holds: bfqq is not in service and
>> -	 * has no dispatched request. DO NOT use bfqq after the next
>> -	 * function invocation.
>> -	 */
> 
> I would really love it if you leave this comment.  I added it after
> suffering a lot for a nasty UAF.  Of course the first sentence may
> need to be adjusted if the code that precedes it is to be removed.
> 
Same as above, if this patch is applied, this function will be gone.

Thanks,
Kuai
> Thanks,
> Paolo
> 
> 
>> 	__bfq_weights_tree_remove(bfqd, bfqq,
>> 				  &bfqd->queue_weights_tree);
>> }
>> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
>> index de2446a9b7ab..f0fce94583e4 100644
>> --- a/block/bfq-iosched.h
>> +++ b/block/bfq-iosched.h
>> @@ -496,27 +496,27 @@ struct bfq_data {
>> 	struct rb_root_cached queue_weights_tree;
>>
>> 	/*
>> -	 * Number of groups with at least one descendant process that
>> +	 * Number of groups with at least one process that
>> 	 * has at least one request waiting for completion. Note that
>> 	 * this accounts for also requests already dispatched, but not
>> 	 * yet completed. Therefore this number of groups may differ
>> 	 * (be larger) than the number of active groups, as a group is
>> 	 * considered active only if its corresponding entity has
>> -	 * descendant queues with at least one request queued. This
>> +	 * queues with at least one request queued. This
>> 	 * number is used to decide whether a scenario is symmetric.
>> 	 * For a detailed explanation see comments on the computation
>> 	 * of the variable asymmetric_scenario in the function
>> 	 * bfq_better_to_idle().
>> 	 *
>> 	 * However, it is hard to compute this number exactly, for
>> -	 * groups with multiple descendant processes. Consider a group
>> -	 * that is inactive, i.e., that has no descendant process with
>> +	 * groups with multiple processes. Consider a group
>> +	 * that is inactive, i.e., that has no process with
>> 	 * pending I/O inside BFQ queues. Then suppose that
>> 	 * num_groups_with_pending_reqs is still accounting for this
>> -	 * group, because the group has descendant processes with some
>> +	 * group, because the group has processes with some
>> 	 * I/O request still in flight. num_groups_with_pending_reqs
>> 	 * should be decremented when the in-flight request of the
>> -	 * last descendant process is finally completed (assuming that
>> +	 * last process is finally completed (assuming that
>> 	 * nothing else has changed for the group in the meantime, in
>> 	 * terms of composition of the group and active/inactive state of child
>> 	 * groups and processes). To accomplish this, an additional
>> @@ -525,7 +525,7 @@ struct bfq_data {
>> 	 * we resort to the following tradeoff between simplicity and
>> 	 * accuracy: for an inactive group that is still counted in
>> 	 * num_groups_with_pending_reqs, we decrement
>> -	 * num_groups_with_pending_reqs when the first descendant
>> +	 * num_groups_with_pending_reqs when the first
>> 	 * process of the group remains with no request waiting for
>> 	 * completion.
>> 	 *
>> @@ -533,12 +533,12 @@ struct bfq_data {
>> 	 * carefulness: to avoid multiple decrements, we flag a group,
>> 	 * more precisely an entity representing a group, as still
>> 	 * counted in num_groups_with_pending_reqs when it becomes
>> -	 * inactive. Then, when the first descendant queue of the
>> +	 * inactive. Then, when the first queue of the
>> 	 * entity remains with no request waiting for completion,
>> 	 * num_groups_with_pending_reqs is decremented, and this flag
>> 	 * is reset. After this flag is reset for the entity,
>> 	 * num_groups_with_pending_reqs won't be decremented any
>> -	 * longer in case a new descendant queue of the entity remains
>> +	 * longer in case a new queue of the entity remains
>> 	 * with no request waiting for completion.
>> 	 */
>> 	unsigned int num_groups_with_pending_reqs;
>> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
>> index 6f36f3fe5cc8..9c2842bedf97 100644
>> --- a/block/bfq-wf2q.c
>> +++ b/block/bfq-wf2q.c
>> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
>> 		entity->on_st_or_in_serv = true;
>> 	}
>>
>> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
>> -		struct bfq_group *bfqg =
>> -			container_of(entity, struct bfq_group, entity);
>> -		struct bfq_data *bfqd = bfqg->bfqd;
>> -
>> -		if (!entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = true;
>> -			bfqd->num_groups_with_pending_reqs++;
>> -		}
>> -	}
>> -#endif
>> -
>> 	bfq_update_fin_time_enqueue(entity, st, backshifted);
>> }
>>
>> @@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (!entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = true;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
>> +		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
>> +			bfqq->bfqd->num_groups_with_pending_reqs++;
>> #endif
>> 	}
>> }
>> @@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = false;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
>> +		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
>> +			bfqq->bfqd->num_groups_with_pending_reqs--;
>> #endif
>> 	}
>> }
>> -- 
>> 2.31.1
>>
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-07-12 13:30       ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-07-12 13:30 UTC (permalink / raw)
  To: Paolo Valente
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe,
	linux-kernel, yi.zhang

Hi!

I'm copying my reply with new mail address, because Paolo seems
didn't receive my reply.

ÔÚ 2022/06/23 23:32, Paolo Valente дµÀ:
> Sorry for the delay.
> 
>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com> ha scritto:
>>
>> Currently, bfq can't handle sync io concurrently as long as they
>> are not issued from root group. This is because
>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>> bfq_asymmetric_scenario().
>>
>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>
>> Before this patch:
>> 1) root group will never be counted.
>> 2) Count if bfqg or it's child bfqgs have pending requests.
>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>
>> After this patch:
>> 1) root group is counted.
>> 2) Count if bfqg have pending requests.
>> 3) Don't count if bfqg complete all the requests.
>>
>> With this change, the occasion that only one group is activated can be
>> detected, and next patch will support concurrent sync io in the
>> occasion.
>>
>> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
>> Reviewed-by: Jan Kara <jack@suse.cz>
>> ---
>> block/bfq-iosched.c | 42 ------------------------------------------
>> block/bfq-iosched.h | 18 +++++++++---------
>> block/bfq-wf2q.c    | 19 ++++---------------
>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>
>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>> index 0ec21018daba..03b04892440c 100644
>> --- a/block/bfq-iosched.c
>> +++ b/block/bfq-iosched.c
>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>> 			     struct bfq_queue *bfqq)
>> {
>> -	struct bfq_entity *entity = bfqq->entity.parent;
>> -
>> -	for_each_entity(entity) {
>> -		struct bfq_sched_data *sd = entity->my_sched_data;
>> -
>> -		if (sd->next_in_service || sd->in_service_entity) {
>> -			/*
>> -			 * entity is still active, because either
>> -			 * next_in_service or in_service_entity is not
>> -			 * NULL (see the comments on the definition of
>> -			 * next_in_service for details on why
>> -			 * in_service_entity must be checked too).
>> -			 *
>> -			 * As a consequence, its parent entities are
>> -			 * active as well, and thus this loop must
>> -			 * stop here.
>> -			 */
>> -			break;
>> -		}
>> -
>> -		/*
>> -		 * The decrement of num_groups_with_pending_reqs is
>> -		 * not performed immediately upon the deactivation of
>> -		 * entity, but it is delayed to when it also happens
>> -		 * that the first leaf descendant bfqq of entity gets
>> -		 * all its pending requests completed. The following
>> -		 * instructions perform this delayed decrement, if
>> -		 * needed. See the comments on
>> -		 * num_groups_with_pending_reqs for details.
>> -		 */
>> -		if (entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = false;
>> -			bfqd->num_groups_with_pending_reqs--;
>> -		}
>> -	}
> 
> With this part removed, I'm missing how you handle the following
> sequence of events:
> 1.  a queue Q becomes non busy but still has dispatched requests, so
> it must not be removed from the counter of queues with pending reqs
> yet
> 2.  the last request of Q is completed with Q being still idle (non
> busy).  At this point Q must be removed from the counter.  It seems to
> me that this case is not handled any longer
> 
Hi, Paolo

1) At first, patch 1 support to track if bfqq has pending requests, it's
done by setting the flag 'entity->in_groups_with_pending_reqs' when the
first request is inserted to bfqq, and it's cleared when the last
request is completed(based on weights_tree insertion and removal).

2) Then, patch 2 add a counter in bfqg: how many bfqqs have pending
requests, which is updated while tracking if bfqq has pending requests.

3) Finally, patch 3 tracks 'num_groups_with_pending_reqs' based on the
new counter in patch 2:
  - if the counter(how many bfqqs have pending requests) increased from 0
    to 1, increase 'num_groups_with_pending_reqs'.
  - if the counter is decreased from 1 to 0, decrease
    'num_groups_with_pending_reqs'

> Additional comment: if your changes do not cpus the problem above,
> then this function only invokes __bfq_weights_tree_remove.  So what's
> the point in keeping this function)
> 
If this patchset is applied, there are following cleanup patches to
remove this function.

multiple cleanup patches for bfq:
https://lore.kernel.org/all/20220528095958.270455-1-yukuai3@huawei.com/
>> -
>> -	/*
>> -	 * Next function is invoked last, because it causes bfqq to be
>> -	 * freed if the following holds: bfqq is not in service and
>> -	 * has no dispatched request. DO NOT use bfqq after the next
>> -	 * function invocation.
>> -	 */
> 
> I would really love it if you leave this comment.  I added it after
> suffering a lot for a nasty UAF.  Of course the first sentence may
> need to be adjusted if the code that precedes it is to be removed.
> 
Same as above, if this patch is applied, this function will be gone.

Thanks,
Kuai
> Thanks,
> Paolo
> 
> 
>> 	__bfq_weights_tree_remove(bfqd, bfqq,
>> 				  &bfqd->queue_weights_tree);
>> }
>> diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
>> index de2446a9b7ab..f0fce94583e4 100644
>> --- a/block/bfq-iosched.h
>> +++ b/block/bfq-iosched.h
>> @@ -496,27 +496,27 @@ struct bfq_data {
>> 	struct rb_root_cached queue_weights_tree;
>>
>> 	/*
>> -	 * Number of groups with at least one descendant process that
>> +	 * Number of groups with at least one process that
>> 	 * has at least one request waiting for completion. Note that
>> 	 * this accounts for also requests already dispatched, but not
>> 	 * yet completed. Therefore this number of groups may differ
>> 	 * (be larger) than the number of active groups, as a group is
>> 	 * considered active only if its corresponding entity has
>> -	 * descendant queues with at least one request queued. This
>> +	 * queues with at least one request queued. This
>> 	 * number is used to decide whether a scenario is symmetric.
>> 	 * For a detailed explanation see comments on the computation
>> 	 * of the variable asymmetric_scenario in the function
>> 	 * bfq_better_to_idle().
>> 	 *
>> 	 * However, it is hard to compute this number exactly, for
>> -	 * groups with multiple descendant processes. Consider a group
>> -	 * that is inactive, i.e., that has no descendant process with
>> +	 * groups with multiple processes. Consider a group
>> +	 * that is inactive, i.e., that has no process with
>> 	 * pending I/O inside BFQ queues. Then suppose that
>> 	 * num_groups_with_pending_reqs is still accounting for this
>> -	 * group, because the group has descendant processes with some
>> +	 * group, because the group has processes with some
>> 	 * I/O request still in flight. num_groups_with_pending_reqs
>> 	 * should be decremented when the in-flight request of the
>> -	 * last descendant process is finally completed (assuming that
>> +	 * last process is finally completed (assuming that
>> 	 * nothing else has changed for the group in the meantime, in
>> 	 * terms of composition of the group and active/inactive state of child
>> 	 * groups and processes). To accomplish this, an additional
>> @@ -525,7 +525,7 @@ struct bfq_data {
>> 	 * we resort to the following tradeoff between simplicity and
>> 	 * accuracy: for an inactive group that is still counted in
>> 	 * num_groups_with_pending_reqs, we decrement
>> -	 * num_groups_with_pending_reqs when the first descendant
>> +	 * num_groups_with_pending_reqs when the first
>> 	 * process of the group remains with no request waiting for
>> 	 * completion.
>> 	 *
>> @@ -533,12 +533,12 @@ struct bfq_data {
>> 	 * carefulness: to avoid multiple decrements, we flag a group,
>> 	 * more precisely an entity representing a group, as still
>> 	 * counted in num_groups_with_pending_reqs when it becomes
>> -	 * inactive. Then, when the first descendant queue of the
>> +	 * inactive. Then, when the first queue of the
>> 	 * entity remains with no request waiting for completion,
>> 	 * num_groups_with_pending_reqs is decremented, and this flag
>> 	 * is reset. After this flag is reset for the entity,
>> 	 * num_groups_with_pending_reqs won't be decremented any
>> -	 * longer in case a new descendant queue of the entity remains
>> +	 * longer in case a new queue of the entity remains
>> 	 * with no request waiting for completion.
>> 	 */
>> 	unsigned int num_groups_with_pending_reqs;
>> diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
>> index 6f36f3fe5cc8..9c2842bedf97 100644
>> --- a/block/bfq-wf2q.c
>> +++ b/block/bfq-wf2q.c
>> @@ -984,19 +984,6 @@ static void __bfq_activate_entity(struct bfq_entity *entity,
>> 		entity->on_st_or_in_serv = true;
>> 	}
>>
>> -#ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -	if (!bfq_entity_to_bfqq(entity)) { /* bfq_group */
>> -		struct bfq_group *bfqg =
>> -			container_of(entity, struct bfq_group, entity);
>> -		struct bfq_data *bfqd = bfqg->bfqd;
>> -
>> -		if (!entity->in_groups_with_pending_reqs) {
>> -			entity->in_groups_with_pending_reqs = true;
>> -			bfqd->num_groups_with_pending_reqs++;
>> -		}
>> -	}
>> -#endif
>> -
>> 	bfq_update_fin_time_enqueue(entity, st, backshifted);
>> }
>>
>> @@ -1654,7 +1641,8 @@ void bfq_add_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (!entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = true;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs++;
>> +		if (!(bfqq_group(bfqq)->num_queues_with_pending_reqs++))
>> +			bfqq->bfqd->num_groups_with_pending_reqs++;
>> #endif
>> 	}
>> }
>> @@ -1666,7 +1654,8 @@ void bfq_del_bfqq_in_groups_with_pending_reqs(struct bfq_queue *bfqq)
>> 	if (entity->in_groups_with_pending_reqs) {
>> 		entity->in_groups_with_pending_reqs = false;
>> #ifdef CONFIG_BFQ_GROUP_IOSCHED
>> -		bfqq_group(bfqq)->num_queues_with_pending_reqs--;
>> +		if (!(--bfqq_group(bfqq)->num_queues_with_pending_reqs))
>> +			bfqq->bfqd->num_groups_with_pending_reqs--;
>> #endif
>> 	}
>> }
>> -- 
>> 2.31.1
>>
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
       [not found]       ` <C2CF100A-9A7C-4300-9A70-1295BC939C66@unimore.it>
@ 2022-07-20 11:38         ` Yu Kuai
  2022-07-27 12:11           ` Yu Kuai
  0 siblings, 1 reply; 38+ messages in thread
From: Yu Kuai @ 2022-07-20 11:38 UTC (permalink / raw)
  To: Paolo VALENTE, Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML, yi.zhang

Hi

在 2022/07/20 19:24, Paolo VALENTE 写道:
> 
> 
>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>
>> Hi!
>>
>> I'm copying my reply with new mail address, because Paolo seems
>> didn't receive my reply.
>>
>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>> Sorry for the delay.
>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com 
>>>> <mailto:yukuai3@huawei.com>> ha scritto:
>>>>
>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>> are not issued from root group. This is because
>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>> bfq_asymmetric_scenario().
>>>>
>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>
>>>> Before this patch:
>>>> 1) root group will never be counted.
>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>>
>>>> After this patch:
>>>> 1) root group is counted.
>>>> 2) Count if bfqg have pending requests.
>>>> 3) Don't count if bfqg complete all the requests.
>>>>
>>>> With this change, the occasion that only one group is activated can be
>>>> detected, and next patch will support concurrent sync io in the
>>>> occasion.
>>>>
>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com>>
>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz>>
>>>> ---
>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>
>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>> index 0ec21018daba..03b04892440c 100644
>>>> --- a/block/bfq-iosched.c
>>>> +++ b/block/bfq-iosched.c
>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data 
>>>> *bfqd,
>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>     struct bfq_queue *bfqq)
>>>> {
>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>> -
>>>> -for_each_entity(entity) {
>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>> -
>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>> -/*
>>>> -* entity is still active, because either
>>>> -* next_in_service or in_service_entity is not
>>>> -* NULL (see the comments on the definition of
>>>> -* next_in_service for details on why
>>>> -* in_service_entity must be checked too).
>>>> -*
>>>> -* As a consequence, its parent entities are
>>>> -* active as well, and thus this loop must
>>>> -* stop here.
>>>> -*/
>>>> -break;
>>>> -}
>>>> -
>>>> -/*
>>>> -* The decrement of num_groups_with_pending_reqs is
>>>> -* not performed immediately upon the deactivation of
>>>> -* entity, but it is delayed to when it also happens
>>>> -* that the first leaf descendant bfqq of entity gets
>>>> -* all its pending requests completed. The following
>>>> -* instructions perform this delayed decrement, if
>>>> -* needed. See the comments on
>>>> -* num_groups_with_pending_reqs for details.
>>>> -*/
>>>> -if (entity->in_groups_with_pending_reqs) {
>>>> -entity->in_groups_with_pending_reqs = false;
>>>> -bfqd->num_groups_with_pending_reqs--;
>>>> -}
>>>> -}
>>> With this part removed, I'm missing how you handle the following
>>> sequence of events:
>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>> it must not be removed from the counter of queues with pending reqs
>>> yet
>>> 2.  the last request of Q is completed with Q being still idle (non
>>> busy).  At this point Q must be removed from the counter.  It seems to
>>> me that this case is not handled any longer
>> Hi, Paolo
>>
>> 1) At first, patch 1 support to track if bfqq has pending requests, it's
>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>> first request is inserted to bfqq, and it's cleared when the last
>> request is completed(based on weights_tree insertion and removal).
>>
> 
> In patch 1 I don't see the flag cleared for the request-completion event :(
> 
> The piece of code involved is this:
> 
> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
> bfq_data *bfqd)
> {
> u64 now_ns;
> u32 delta_us;
> 
> bfq_update_hw_tag(bfqd);
> 
> bfqd->rq_in_driver[bfqq->actuator_idx]--;
> bfqd->tot_rq_in_driver--;
> bfqq->dispatched--;
> 
> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
> /*
> * Set budget_timeout (which we overload to store the
> * time at which the queue remains with no backlog and
> * no outstanding request; used by the weight-raising
> * mechanism).
> */
> bfqq->budget_timeout = jiffies;
> 
> bfq_weights_tree_remove(bfqd, bfqq);
> }
> ...
> 
> Am I missing something?

I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
to clear the flag, and it's called both from bfq_del_bfqq_busy() and
bfq_completed_request(). I think you may miss the later:

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0d46cb728bbf..0ec21018daba 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue 
*bfqq, struct bfq_data *bfqd)
  		 */
  		bfqq->budget_timeout = jiffies;

+		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
  		bfq_weights_tree_remove(bfqd, bfqq);
  	}

Thanks,
Kuai
> 
> Thanks,
> Paolo


^ permalink raw reply related	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-07-20 11:38         ` Yu Kuai
@ 2022-07-27 12:11           ` Yu Kuai
  2022-08-05 11:20               ` Yu Kuai
  2022-08-10 10:49             ` Paolo Valente
  0 siblings, 2 replies; 38+ messages in thread
From: Yu Kuai @ 2022-07-27 12:11 UTC (permalink / raw)
  To: Paolo VALENTE
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)

Hi, Paolo

Are you still interested in this patchset?

在 2022/07/20 19:38, Yu Kuai 写道:
> Hi
>
> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>
>>
>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>
>>> Hi!
>>>
>>> I'm copying my reply with new mail address, because Paolo seems
>>> didn't receive my reply.
>>>
>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>> Sorry for the delay.
>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com 
>>>>> <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>
>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>> are not issued from root group. This is because
>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>> bfq_asymmetric_scenario().
>>>>>
>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>
>>>>> Before this patch:
>>>>> 1) root group will never be counted.
>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the 
>>>>> requests.
>>>>>
>>>>> After this patch:
>>>>> 1) root group is counted.
>>>>> 2) Count if bfqg have pending requests.
>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>
>>>>> With this change, the occasion that only one group is activated 
>>>>> can be
>>>>> detected, and next patch will support concurrent sync io in the
>>>>> occasion.
>>>>>
>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com 
>>>>> <mailto:yukuai3@huawei.com>>
>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz>>
>>>>> ---
>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>
>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>> index 0ec21018daba..03b04892440c 100644
>>>>> --- a/block/bfq-iosched.c
>>>>> +++ b/block/bfq-iosched.c
>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct 
>>>>> bfq_data *bfqd,
>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>     struct bfq_queue *bfqq)
>>>>> {
>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>> -
>>>>> -for_each_entity(entity) {
>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>> -
>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>> -/*
>>>>> -* entity is still active, because either
>>>>> -* next_in_service or in_service_entity is not
>>>>> -* NULL (see the comments on the definition of
>>>>> -* next_in_service for details on why
>>>>> -* in_service_entity must be checked too).
>>>>> -*
>>>>> -* As a consequence, its parent entities are
>>>>> -* active as well, and thus this loop must
>>>>> -* stop here.
>>>>> -*/
>>>>> -break;
>>>>> -}
>>>>> -
>>>>> -/*
>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>> -* not performed immediately upon the deactivation of
>>>>> -* entity, but it is delayed to when it also happens
>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>> -* all its pending requests completed. The following
>>>>> -* instructions perform this delayed decrement, if
>>>>> -* needed. See the comments on
>>>>> -* num_groups_with_pending_reqs for details.
>>>>> -*/
>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>> -}
>>>>> -}
>>>> With this part removed, I'm missing how you handle the following
>>>> sequence of events:
>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>> it must not be removed from the counter of queues with pending reqs
>>>> yet
>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>> me that this case is not handled any longer
>>> Hi, Paolo
>>>
>>> 1) At first, patch 1 support to track if bfqq has pending requests, 
>>> it's
>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>> first request is inserted to bfqq, and it's cleared when the last
>>> request is completed(based on weights_tree insertion and removal).
>>>
>>
>> In patch 1 I don't see the flag cleared for the request-completion 
>> event :(
>>
>> The piece of code involved is this:
>>
>> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
>> bfq_data *bfqd)
>> {
>> u64 now_ns;
>> u32 delta_us;
>>
>> bfq_update_hw_tag(bfqd);
>>
>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>> bfqd->tot_rq_in_driver--;
>> bfqq->dispatched--;
>>
>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>> /*
>> * Set budget_timeout (which we overload to store the
>> * time at which the queue remains with no backlog and
>> * no outstanding request; used by the weight-raising
>> * mechanism).
>> */
>> bfqq->budget_timeout = jiffies;
>>
>> bfq_weights_tree_remove(bfqd, bfqq);
>> }
>> ...
>>
>> Am I missing something?
>
> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
> bfq_completed_request(). I think you may miss the later:
>
> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
> index 0d46cb728bbf..0ec21018daba 100644
> --- a/block/bfq-iosched.c
> +++ b/block/bfq-iosched.c
> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct 
> bfq_queue *bfqq, struct bfq_data *bfqd)
>           */
>          bfqq->budget_timeout = jiffies;
>
> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>          bfq_weights_tree_remove(bfqd, bfqq);
>      }
>
> Thanks,
> Kuai
>>
>> Thanks,
>> Paolo


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-05 11:20               ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-05 11:20 UTC (permalink / raw)
  To: Yu Kuai, Paolo VALENTE
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)


在 2022/07/27 20:11, Yu Kuai 写道:
> Hi, Paolo
> 
> Are you still interested in this patchset?

Friendly ping...
> 
> 在 2022/07/20 19:38, Yu Kuai 写道:
>> Hi
>>
>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>
>>>
>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>>>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>
>>>> Hi!
>>>>
>>>> I'm copying my reply with new mail address, because Paolo seems
>>>> didn't receive my reply.
>>>>
>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>> Sorry for the delay.
>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com 
>>>>>> <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>>
>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>> are not issued from root group. This is because
>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>> bfq_asymmetric_scenario().
>>>>>>
>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>
>>>>>> Before this patch:
>>>>>> 1) root group will never be counted.
>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the 
>>>>>> requests.
>>>>>>
>>>>>> After this patch:
>>>>>> 1) root group is counted.
>>>>>> 2) Count if bfqg have pending requests.
>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>
>>>>>> With this change, the occasion that only one group is activated 
>>>>>> can be
>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>> occasion.
>>>>>>
>>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com 
>>>>>> <mailto:yukuai3@huawei.com>>
>>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz>>
>>>>>> ---
>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>
>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>> --- a/block/bfq-iosched.c
>>>>>> +++ b/block/bfq-iosched.c
>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct 
>>>>>> bfq_data *bfqd,
>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>     struct bfq_queue *bfqq)
>>>>>> {
>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>> -
>>>>>> -for_each_entity(entity) {
>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>> -
>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>> -/*
>>>>>> -* entity is still active, because either
>>>>>> -* next_in_service or in_service_entity is not
>>>>>> -* NULL (see the comments on the definition of
>>>>>> -* next_in_service for details on why
>>>>>> -* in_service_entity must be checked too).
>>>>>> -*
>>>>>> -* As a consequence, its parent entities are
>>>>>> -* active as well, and thus this loop must
>>>>>> -* stop here.
>>>>>> -*/
>>>>>> -break;
>>>>>> -}
>>>>>> -
>>>>>> -/*
>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>> -* not performed immediately upon the deactivation of
>>>>>> -* entity, but it is delayed to when it also happens
>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>> -* all its pending requests completed. The following
>>>>>> -* instructions perform this delayed decrement, if
>>>>>> -* needed. See the comments on
>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>> -*/
>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>> -}
>>>>>> -}
>>>>> With this part removed, I'm missing how you handle the following
>>>>> sequence of events:
>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>> it must not be removed from the counter of queues with pending reqs
>>>>> yet
>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>>> me that this case is not handled any longer
>>>> Hi, Paolo
>>>>
>>>> 1) At first, patch 1 support to track if bfqq has pending requests, 
>>>> it's
>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>> first request is inserted to bfqq, and it's cleared when the last
>>>> request is completed(based on weights_tree insertion and removal).
>>>>
>>>
>>> In patch 1 I don't see the flag cleared for the request-completion 
>>> event :(
>>>
>>> The piece of code involved is this:
>>>
>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
>>> bfq_data *bfqd)
>>> {
>>> u64 now_ns;
>>> u32 delta_us;
>>>
>>> bfq_update_hw_tag(bfqd);
>>>
>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>> bfqd->tot_rq_in_driver--;
>>> bfqq->dispatched--;
>>>
>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>> /*
>>> * Set budget_timeout (which we overload to store the
>>> * time at which the queue remains with no backlog and
>>> * no outstanding request; used by the weight-raising
>>> * mechanism).
>>> */
>>> bfqq->budget_timeout = jiffies;
>>>
>>> bfq_weights_tree_remove(bfqd, bfqq);
>>> }
>>> ...
>>>
>>> Am I missing something?
>>
>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>> bfq_completed_request(). I think you may miss the later:
>>
>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>> index 0d46cb728bbf..0ec21018daba 100644
>> --- a/block/bfq-iosched.c
>> +++ b/block/bfq-iosched.c
>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct 
>> bfq_queue *bfqq, struct bfq_data *bfqd)
>>           */
>>          bfqq->budget_timeout = jiffies;
>>
>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>          bfq_weights_tree_remove(bfqd, bfqq);
>>      }
>>
>> Thanks,
>> Kuai
>>>
>>> Thanks,
>>> Paolo
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-05 11:20               ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-05 11:20 UTC (permalink / raw)
  To: Yu Kuai, Paolo VALENTE
  Cc: Jan Kara, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-block, Tejun Heo,
	Jens Axboe, LKML, yi.zhang-hv44wF8Li93QT0dZR+AlfA, yukuai (C)


在 2022/07/27 20:11, Yu Kuai 写道:
> Hi, Paolo
> 
> Are you still interested in this patchset?

Friendly ping...
> 
> 在 2022/07/20 19:38, Yu Kuai 写道:
>> Hi
>>
>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>
>>>
>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>>>> <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org <mailto:yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org>> ha scritto:
>>>>
>>>> Hi!
>>>>
>>>> I'm copying my reply with new mail address, because Paolo seems
>>>> didn't receive my reply.
>>>>
>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>> Sorry for the delay.
>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org 
>>>>>> <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>> ha scritto:
>>>>>>
>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>> are not issued from root group. This is because
>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>> bfq_asymmetric_scenario().
>>>>>>
>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>
>>>>>> Before this patch:
>>>>>> 1) root group will never be counted.
>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the 
>>>>>> requests.
>>>>>>
>>>>>> After this patch:
>>>>>> 1) root group is counted.
>>>>>> 2) Count if bfqg have pending requests.
>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>
>>>>>> With this change, the occasion that only one group is activated 
>>>>>> can be
>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>> occasion.
>>>>>>
>>>>>> Signed-off-by: Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org 
>>>>>> <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>>
>>>>>> Reviewed-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org <mailto:jack-AlSwsSmVLrQ@public.gmane.org>>
>>>>>> ---
>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>
>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>> --- a/block/bfq-iosched.c
>>>>>> +++ b/block/bfq-iosched.c
>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct 
>>>>>> bfq_data *bfqd,
>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>     struct bfq_queue *bfqq)
>>>>>> {
>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>> -
>>>>>> -for_each_entity(entity) {
>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>> -
>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>> -/*
>>>>>> -* entity is still active, because either
>>>>>> -* next_in_service or in_service_entity is not
>>>>>> -* NULL (see the comments on the definition of
>>>>>> -* next_in_service for details on why
>>>>>> -* in_service_entity must be checked too).
>>>>>> -*
>>>>>> -* As a consequence, its parent entities are
>>>>>> -* active as well, and thus this loop must
>>>>>> -* stop here.
>>>>>> -*/
>>>>>> -break;
>>>>>> -}
>>>>>> -
>>>>>> -/*
>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>> -* not performed immediately upon the deactivation of
>>>>>> -* entity, but it is delayed to when it also happens
>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>> -* all its pending requests completed. The following
>>>>>> -* instructions perform this delayed decrement, if
>>>>>> -* needed. See the comments on
>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>> -*/
>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>> -}
>>>>>> -}
>>>>> With this part removed, I'm missing how you handle the following
>>>>> sequence of events:
>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>> it must not be removed from the counter of queues with pending reqs
>>>>> yet
>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>>> me that this case is not handled any longer
>>>> Hi, Paolo
>>>>
>>>> 1) At first, patch 1 support to track if bfqq has pending requests, 
>>>> it's
>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>> first request is inserted to bfqq, and it's cleared when the last
>>>> request is completed(based on weights_tree insertion and removal).
>>>>
>>>
>>> In patch 1 I don't see the flag cleared for the request-completion 
>>> event :(
>>>
>>> The piece of code involved is this:
>>>
>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
>>> bfq_data *bfqd)
>>> {
>>> u64 now_ns;
>>> u32 delta_us;
>>>
>>> bfq_update_hw_tag(bfqd);
>>>
>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>> bfqd->tot_rq_in_driver--;
>>> bfqq->dispatched--;
>>>
>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>> /*
>>> * Set budget_timeout (which we overload to store the
>>> * time at which the queue remains with no backlog and
>>> * no outstanding request; used by the weight-raising
>>> * mechanism).
>>> */
>>> bfqq->budget_timeout = jiffies;
>>>
>>> bfq_weights_tree_remove(bfqd, bfqq);
>>> }
>>> ...
>>>
>>> Am I missing something?
>>
>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>> bfq_completed_request(). I think you may miss the later:
>>
>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>> index 0d46cb728bbf..0ec21018daba 100644
>> --- a/block/bfq-iosched.c
>> +++ b/block/bfq-iosched.c
>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct 
>> bfq_queue *bfqq, struct bfq_data *bfqd)
>>           */
>>          bfqq->budget_timeout = jiffies;
>>
>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>          bfq_weights_tree_remove(bfqd, bfqq);
>>      }
>>
>> Thanks,
>> Kuai
>>>
>>> Thanks,
>>> Paolo
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-07-27 12:11           ` Yu Kuai
  2022-08-05 11:20               ` Yu Kuai
@ 2022-08-10 10:49             ` Paolo Valente
  2022-08-11  1:19                 ` Yu Kuai
  1 sibling, 1 reply; 38+ messages in thread
From: Paolo Valente @ 2022-08-10 10:49 UTC (permalink / raw)
  To: Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)



> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> 
> Hi, Paolo
> 

hi

> Are you still interested in this patchset?
> 

Yes. Sorry for replying very late again.

Probably the last fix that you suggest is enough, but I'm a little bit
concerned that it may be a little hasty.  In fact, before this fix, we
exchanged several messages, and I didn't seem to be very good at
convincing you about the need to keep into account also in-service
I/O.  So, my question is: are you sure that now you have a
clear/complete understanding of this non-trivial matter?
Consequently, are we sure that this last fix is most certainly all we
need?  Of course, I will check on my own, but if you reassure me on
this point, I will feel more confident.

Thanks,
Paolo

> 在 2022/07/20 19:38, Yu Kuai 写道:
>> Hi
>> 
>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>> 
>>> 
>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>> 
>>>> Hi!
>>>> 
>>>> I'm copying my reply with new mail address, because Paolo seems
>>>> didn't receive my reply.
>>>> 
>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>> Sorry for the delay.
>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>> 
>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>> are not issued from root group. This is because
>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>> bfq_asymmetric_scenario().
>>>>>> 
>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>> 
>>>>>> Before this patch:
>>>>>> 1) root group will never be counted.
>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>>>> 
>>>>>> After this patch:
>>>>>> 1) root group is counted.
>>>>>> 2) Count if bfqg have pending requests.
>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>> 
>>>>>> With this change, the occasion that only one group is activated can be
>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>> occasion.
>>>>>> 
>>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com>>
>>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz>>
>>>>>> ---
>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>> 
>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>> --- a/block/bfq-iosched.c
>>>>>> +++ b/block/bfq-iosched.c
>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>     struct bfq_queue *bfqq)
>>>>>> {
>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>> -
>>>>>> -for_each_entity(entity) {
>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>> -
>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>> -/*
>>>>>> -* entity is still active, because either
>>>>>> -* next_in_service or in_service_entity is not
>>>>>> -* NULL (see the comments on the definition of
>>>>>> -* next_in_service for details on why
>>>>>> -* in_service_entity must be checked too).
>>>>>> -*
>>>>>> -* As a consequence, its parent entities are
>>>>>> -* active as well, and thus this loop must
>>>>>> -* stop here.
>>>>>> -*/
>>>>>> -break;
>>>>>> -}
>>>>>> -
>>>>>> -/*
>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>> -* not performed immediately upon the deactivation of
>>>>>> -* entity, but it is delayed to when it also happens
>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>> -* all its pending requests completed. The following
>>>>>> -* instructions perform this delayed decrement, if
>>>>>> -* needed. See the comments on
>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>> -*/
>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>> -}
>>>>>> -}
>>>>> With this part removed, I'm missing how you handle the following
>>>>> sequence of events:
>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>> it must not be removed from the counter of queues with pending reqs
>>>>> yet
>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>>> me that this case is not handled any longer
>>>> Hi, Paolo
>>>> 
>>>> 1) At first, patch 1 support to track if bfqq has pending requests, it's
>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>> first request is inserted to bfqq, and it's cleared when the last
>>>> request is completed(based on weights_tree insertion and removal).
>>>> 
>>> 
>>> In patch 1 I don't see the flag cleared for the request-completion event :(
>>> 
>>> The piece of code involved is this:
>>> 
>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>> {
>>> u64 now_ns;
>>> u32 delta_us;
>>> 
>>> bfq_update_hw_tag(bfqd);
>>> 
>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>> bfqd->tot_rq_in_driver--;
>>> bfqq->dispatched--;
>>> 
>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>> /*
>>> * Set budget_timeout (which we overload to store the
>>> * time at which the queue remains with no backlog and
>>> * no outstanding request; used by the weight-raising
>>> * mechanism).
>>> */
>>> bfqq->budget_timeout = jiffies;
>>> 
>>> bfq_weights_tree_remove(bfqd, bfqq);
>>> }
>>> ...
>>> 
>>> Am I missing something?
>> 
>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>> bfq_completed_request(). I think you may miss the later:
>> 
>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>> index 0d46cb728bbf..0ec21018daba 100644
>> --- a/block/bfq-iosched.c
>> +++ b/block/bfq-iosched.c
>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>           */
>>          bfqq->budget_timeout = jiffies;
>> 
>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>          bfq_weights_tree_remove(bfqd, bfqq);
>>      }
>> 
>> Thanks,
>> Kuai
>>> 
>>> Thanks,
>>> Paolo
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-11  1:19                 ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-11  1:19 UTC (permalink / raw)
  To: Paolo Valente, Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai3@huawei.com >> yukuai (C)

Hi, Paolo

在 2022/08/10 18:49, Paolo Valente 写道:
> 
> 
>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>
>> Hi, Paolo
>>
> 
> hi
> 
>> Are you still interested in this patchset?
>>
> 
> Yes. Sorry for replying very late again.
> 
> Probably the last fix that you suggest is enough, but I'm a little bit
> concerned that it may be a little hasty.  In fact, before this fix, we
> exchanged several messages, and I didn't seem to be very good at
> convincing you about the need to keep into account also in-service
> I/O.  So, my question is: are you sure that now you have a

I'm confused here, I'm pretty aware that in-service I/O(as said pending
requests is the patchset) should be counted, as you suggested in v7, are
you still thinking that the way in this patchset is problematic?

I'll try to explain again that how to track is bfqq has pending pending
requests, please let me know if you still think there are some problems:

patch 1 support to track if bfqq has pending requests, it's
done by setting the flag 'entity->in_groups_with_pending_reqs' when the
first request is inserted to bfqq, and it's cleared when the last
request is completed. specifically the flag is set in
bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
both in bfq_completed_request() and bfq_del_bfqq_busy() when
'bfqq->diapatched' is false.

Thanks,
Kuai
> clear/complete understanding of this non-trivial matter?
> Consequently, are we sure that this last fix is most certainly all we
> need?  Of course, I will check on my own, but if you reassure me on
> this point, I will feel more confident.
> 
> Thanks,
> Paolo
> 
>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>> Hi
>>>
>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>
>>>>
>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>
>>>>> Hi!
>>>>>
>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>> didn't receive my reply.
>>>>>
>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>> Sorry for the delay.
>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>>>
>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>> are not issued from root group. This is because
>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>> bfq_asymmetric_scenario().
>>>>>>>
>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>
>>>>>>> Before this patch:
>>>>>>> 1) root group will never be counted.
>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>>>>>
>>>>>>> After this patch:
>>>>>>> 1) root group is counted.
>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>
>>>>>>> With this change, the occasion that only one group is activated can be
>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>> occasion.
>>>>>>>
>>>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com>>
>>>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz>>
>>>>>>> ---
>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>
>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>> --- a/block/bfq-iosched.c
>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>      struct bfq_queue *bfqq)
>>>>>>> {
>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>> -
>>>>>>> -for_each_entity(entity) {
>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>> -
>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>> -/*
>>>>>>> -* entity is still active, because either
>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>> -* NULL (see the comments on the definition of
>>>>>>> -* next_in_service for details on why
>>>>>>> -* in_service_entity must be checked too).
>>>>>>> -*
>>>>>>> -* As a consequence, its parent entities are
>>>>>>> -* active as well, and thus this loop must
>>>>>>> -* stop here.
>>>>>>> -*/
>>>>>>> -break;
>>>>>>> -}
>>>>>>> -
>>>>>>> -/*
>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>> -* all its pending requests completed. The following
>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>> -* needed. See the comments on
>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>> -*/
>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>> -}
>>>>>>> -}
>>>>>> With this part removed, I'm missing how you handle the following
>>>>>> sequence of events:
>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>> yet
>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>>>> me that this case is not handled any longer
>>>>> Hi, Paolo
>>>>>
>>>>> 1) At first, patch 1 support to track if bfqq has pending requests, it's
>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>
>>>>
>>>> In patch 1 I don't see the flag cleared for the request-completion event :(
>>>>
>>>> The piece of code involved is this:
>>>>
>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>> {
>>>> u64 now_ns;
>>>> u32 delta_us;
>>>>
>>>> bfq_update_hw_tag(bfqd);
>>>>
>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>> bfqd->tot_rq_in_driver--;
>>>> bfqq->dispatched--;
>>>>
>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>> /*
>>>> * Set budget_timeout (which we overload to store the
>>>> * time at which the queue remains with no backlog and
>>>> * no outstanding request; used by the weight-raising
>>>> * mechanism).
>>>> */
>>>> bfqq->budget_timeout = jiffies;
>>>>
>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>> }
>>>> ...
>>>>
>>>> Am I missing something?
>>>
>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>> bfq_completed_request(). I think you may miss the later:
>>>
>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>> index 0d46cb728bbf..0ec21018daba 100644
>>> --- a/block/bfq-iosched.c
>>> +++ b/block/bfq-iosched.c
>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>            */
>>>           bfqq->budget_timeout = jiffies;
>>>
>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>           bfq_weights_tree_remove(bfqd, bfqq);
>>>       }
>>>
>>> Thanks,
>>> Kuai
>>>>
>>>> Thanks,
>>>> Paolo
>>
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-11  1:19                 ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-11  1:19 UTC (permalink / raw)
  To: Paolo Valente, Yu Kuai
  Cc: Jan Kara, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-block, Tejun Heo,
	Jens Axboe, LKML, yi.zhang-hv44wF8Li93QT0dZR+AlfA,
	yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org >> yukuai
	(C)

Hi, Paolo

在 2022/08/10 18:49, Paolo Valente 写道:
> 
> 
>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org> ha scritto:
>>
>> Hi, Paolo
>>
> 
> hi
> 
>> Are you still interested in this patchset?
>>
> 
> Yes. Sorry for replying very late again.
> 
> Probably the last fix that you suggest is enough, but I'm a little bit
> concerned that it may be a little hasty.  In fact, before this fix, we
> exchanged several messages, and I didn't seem to be very good at
> convincing you about the need to keep into account also in-service
> I/O.  So, my question is: are you sure that now you have a

I'm confused here, I'm pretty aware that in-service I/O(as said pending
requests is the patchset) should be counted, as you suggested in v7, are
you still thinking that the way in this patchset is problematic?

I'll try to explain again that how to track is bfqq has pending pending
requests, please let me know if you still think there are some problems:

patch 1 support to track if bfqq has pending requests, it's
done by setting the flag 'entity->in_groups_with_pending_reqs' when the
first request is inserted to bfqq, and it's cleared when the last
request is completed. specifically the flag is set in
bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
both in bfq_completed_request() and bfq_del_bfqq_busy() when
'bfqq->diapatched' is false.

Thanks,
Kuai
> clear/complete understanding of this non-trivial matter?
> Consequently, are we sure that this last fix is most certainly all we
> need?  Of course, I will check on my own, but if you reassure me on
> this point, I will feel more confident.
> 
> Thanks,
> Paolo
> 
>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>> Hi
>>>
>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>
>>>>
>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org <mailto:yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org>> ha scritto:
>>>>>
>>>>> Hi!
>>>>>
>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>> didn't receive my reply.
>>>>>
>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>> Sorry for the delay.
>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>> ha scritto:
>>>>>>>
>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>> are not issued from root group. This is because
>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>> bfq_asymmetric_scenario().
>>>>>>>
>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>
>>>>>>> Before this patch:
>>>>>>> 1) root group will never be counted.
>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>>>>>
>>>>>>> After this patch:
>>>>>>> 1) root group is counted.
>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>
>>>>>>> With this change, the occasion that only one group is activated can be
>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>> occasion.
>>>>>>>
>>>>>>> Signed-off-by: Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>>
>>>>>>> Reviewed-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org <mailto:jack-AlSwsSmVLrQ@public.gmane.org>>
>>>>>>> ---
>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>
>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>> --- a/block/bfq-iosched.c
>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>      struct bfq_queue *bfqq)
>>>>>>> {
>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>> -
>>>>>>> -for_each_entity(entity) {
>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>> -
>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>> -/*
>>>>>>> -* entity is still active, because either
>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>> -* NULL (see the comments on the definition of
>>>>>>> -* next_in_service for details on why
>>>>>>> -* in_service_entity must be checked too).
>>>>>>> -*
>>>>>>> -* As a consequence, its parent entities are
>>>>>>> -* active as well, and thus this loop must
>>>>>>> -* stop here.
>>>>>>> -*/
>>>>>>> -break;
>>>>>>> -}
>>>>>>> -
>>>>>>> -/*
>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>> -* all its pending requests completed. The following
>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>> -* needed. See the comments on
>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>> -*/
>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>> -}
>>>>>>> -}
>>>>>> With this part removed, I'm missing how you handle the following
>>>>>> sequence of events:
>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>> yet
>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>>>> me that this case is not handled any longer
>>>>> Hi, Paolo
>>>>>
>>>>> 1) At first, patch 1 support to track if bfqq has pending requests, it's
>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>
>>>>
>>>> In patch 1 I don't see the flag cleared for the request-completion event :(
>>>>
>>>> The piece of code involved is this:
>>>>
>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>> {
>>>> u64 now_ns;
>>>> u32 delta_us;
>>>>
>>>> bfq_update_hw_tag(bfqd);
>>>>
>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>> bfqd->tot_rq_in_driver--;
>>>> bfqq->dispatched--;
>>>>
>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>> /*
>>>> * Set budget_timeout (which we overload to store the
>>>> * time at which the queue remains with no backlog and
>>>> * no outstanding request; used by the weight-raising
>>>> * mechanism).
>>>> */
>>>> bfqq->budget_timeout = jiffies;
>>>>
>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>> }
>>>> ...
>>>>
>>>> Am I missing something?
>>>
>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>> bfq_completed_request(). I think you may miss the later:
>>>
>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>> index 0d46cb728bbf..0ec21018daba 100644
>>> --- a/block/bfq-iosched.c
>>> +++ b/block/bfq-iosched.c
>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>            */
>>>           bfqq->budget_timeout = jiffies;
>>>
>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>           bfq_weights_tree_remove(bfqd, bfqq);
>>>       }
>>>
>>> Thanks,
>>> Kuai
>>>>
>>>> Thanks,
>>>> Paolo
>>
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-25 12:14                   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-25 12:14 UTC (permalink / raw)
  To: Yu Kuai, Paolo Valente
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)



在 2022/08/11 9:19, Yu Kuai 写道:
> Hi, Paolo
> 
> 在 2022/08/10 18:49, Paolo Valente 写道:
>>
>>
>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai 
>>> <yukuai1@huaweicloud.com> ha scritto:
>>>
>>> Hi, Paolo
>>>
>>
>> hi
>>
>>> Are you still interested in this patchset?
>>>
>>
>> Yes. Sorry for replying very late again.
>>
>> Probably the last fix that you suggest is enough, but I'm a little bit
>> concerned that it may be a little hasty.  In fact, before this fix, we
>> exchanged several messages, and I didn't seem to be very good at
>> convincing you about the need to keep into account also in-service
>> I/O.  So, my question is: are you sure that now you have a
> 
> I'm confused here, I'm pretty aware that in-service I/O(as said pending
> requests is the patchset) should be counted, as you suggested in v7, are
> you still thinking that the way in this patchset is problematic?
> 
> I'll try to explain again that how to track is bfqq has pending pending
> requests, please let me know if you still think there are some problems:
> 
> patch 1 support to track if bfqq has pending requests, it's
> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
> first request is inserted to bfqq, and it's cleared when the last
> request is completed. specifically the flag is set in
> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
> both in bfq_completed_request() and bfq_del_bfqq_busy() when
> 'bfqq->diapatched' is false.

Hi, Paolo

Can you please have a check if patch 1 is ok?

Thanks,
Kuai
> 
> Thanks,
> Kuai
>> clear/complete understanding of this non-trivial matter?
>> Consequently, are we sure that this last fix is most certainly all we
>> need?  Of course, I will check on my own, but if you reassure me on
>> this point, I will feel more confident.
>>
>> Thanks,
>> Paolo
>>
>>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>>> Hi
>>>>
>>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>>
>>>>>
>>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>>>>>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha 
>>>>>> scritto:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>>> didn't receive my reply.
>>>>>>
>>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>>> Sorry for the delay.
>>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai 
>>>>>>>> <yukuai3@huawei.com <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>>>>
>>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>>> are not issued from root group. This is because
>>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>>> bfq_asymmetric_scenario().
>>>>>>>>
>>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>>
>>>>>>>> Before this patch:
>>>>>>>> 1) root group will never be counted.
>>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the 
>>>>>>>> requests.
>>>>>>>>
>>>>>>>> After this patch:
>>>>>>>> 1) root group is counted.
>>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>>
>>>>>>>> With this change, the occasion that only one group is activated 
>>>>>>>> can be
>>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>>> occasion.
>>>>>>>>
>>>>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com 
>>>>>>>> <mailto:yukuai3@huawei.com>>
>>>>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz>>
>>>>>>>> ---
>>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>>> --- a/block/bfq-iosched.c
>>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct 
>>>>>>>> bfq_data *bfqd,
>>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>      struct bfq_queue *bfqq)
>>>>>>>> {
>>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>>> -
>>>>>>>> -for_each_entity(entity) {
>>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>>> -
>>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>>> -/*
>>>>>>>> -* entity is still active, because either
>>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>>> -* NULL (see the comments on the definition of
>>>>>>>> -* next_in_service for details on why
>>>>>>>> -* in_service_entity must be checked too).
>>>>>>>> -*
>>>>>>>> -* As a consequence, its parent entities are
>>>>>>>> -* active as well, and thus this loop must
>>>>>>>> -* stop here.
>>>>>>>> -*/
>>>>>>>> -break;
>>>>>>>> -}
>>>>>>>> -
>>>>>>>> -/*
>>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>>> -* all its pending requests completed. The following
>>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>>> -* needed. See the comments on
>>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>>> -*/
>>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>>> -}
>>>>>>>> -}
>>>>>>> With this part removed, I'm missing how you handle the following
>>>>>>> sequence of events:
>>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>>> yet
>>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>>> busy).  At this point Q must be removed from the counter.  It 
>>>>>>> seems to
>>>>>>> me that this case is not handled any longer
>>>>>> Hi, Paolo
>>>>>>
>>>>>> 1) At first, patch 1 support to track if bfqq has pending 
>>>>>> requests, it's
>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' 
>>>>>> when the
>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>>
>>>>>
>>>>> In patch 1 I don't see the flag cleared for the request-completion 
>>>>> event :(
>>>>>
>>>>> The piece of code involved is this:
>>>>>
>>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
>>>>> bfq_data *bfqd)
>>>>> {
>>>>> u64 now_ns;
>>>>> u32 delta_us;
>>>>>
>>>>> bfq_update_hw_tag(bfqd);
>>>>>
>>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>>> bfqd->tot_rq_in_driver--;
>>>>> bfqq->dispatched--;
>>>>>
>>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>>> /*
>>>>> * Set budget_timeout (which we overload to store the
>>>>> * time at which the queue remains with no backlog and
>>>>> * no outstanding request; used by the weight-raising
>>>>> * mechanism).
>>>>> */
>>>>> bfqq->budget_timeout = jiffies;
>>>>>
>>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>>> }
>>>>> ...
>>>>>
>>>>> Am I missing something?
>>>>
>>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>>> bfq_completed_request(). I think you may miss the later:
>>>>
>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>> index 0d46cb728bbf..0ec21018daba 100644
>>>> --- a/block/bfq-iosched.c
>>>> +++ b/block/bfq-iosched.c
>>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct 
>>>> bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>            */
>>>>           bfqq->budget_timeout = jiffies;
>>>>
>>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>>           bfq_weights_tree_remove(bfqd, bfqq);
>>>>       }
>>>>
>>>> Thanks,
>>>> Kuai
>>>>>
>>>>> Thanks,
>>>>> Paolo
>>>
>>
>> .
>>
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-25 12:14                   ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-25 12:14 UTC (permalink / raw)
  To: Yu Kuai, Paolo Valente
  Cc: Jan Kara, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-block, Tejun Heo,
	Jens Axboe, LKML, yi.zhang-hv44wF8Li93QT0dZR+AlfA, yukuai (C)



在 2022/08/11 9:19, Yu Kuai 写道:
> Hi, Paolo
> 
> 在 2022/08/10 18:49, Paolo Valente 写道:
>>
>>
>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai 
>>> <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org> ha scritto:
>>>
>>> Hi, Paolo
>>>
>>
>> hi
>>
>>> Are you still interested in this patchset?
>>>
>>
>> Yes. Sorry for replying very late again.
>>
>> Probably the last fix that you suggest is enough, but I'm a little bit
>> concerned that it may be a little hasty.  In fact, before this fix, we
>> exchanged several messages, and I didn't seem to be very good at
>> convincing you about the need to keep into account also in-service
>> I/O.  So, my question is: are you sure that now you have a
> 
> I'm confused here, I'm pretty aware that in-service I/O(as said pending
> requests is the patchset) should be counted, as you suggested in v7, are
> you still thinking that the way in this patchset is problematic?
> 
> I'll try to explain again that how to track is bfqq has pending pending
> requests, please let me know if you still think there are some problems:
> 
> patch 1 support to track if bfqq has pending requests, it's
> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
> first request is inserted to bfqq, and it's cleared when the last
> request is completed. specifically the flag is set in
> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
> both in bfq_completed_request() and bfq_del_bfqq_busy() when
> 'bfqq->diapatched' is false.

Hi, Paolo

Can you please have a check if patch 1 is ok?

Thanks,
Kuai
> 
> Thanks,
> Kuai
>> clear/complete understanding of this non-trivial matter?
>> Consequently, are we sure that this last fix is most certainly all we
>> need?  Of course, I will check on my own, but if you reassure me on
>> this point, I will feel more confident.
>>
>> Thanks,
>> Paolo
>>
>>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>>> Hi
>>>>
>>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>>
>>>>>
>>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>>>>>> <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org <mailto:yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org>> ha 
>>>>>> scritto:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>>> didn't receive my reply.
>>>>>>
>>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>>> Sorry for the delay.
>>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai 
>>>>>>>> <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>> ha scritto:
>>>>>>>>
>>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>>> are not issued from root group. This is because
>>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>>> bfq_asymmetric_scenario().
>>>>>>>>
>>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>>
>>>>>>>> Before this patch:
>>>>>>>> 1) root group will never be counted.
>>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the 
>>>>>>>> requests.
>>>>>>>>
>>>>>>>> After this patch:
>>>>>>>> 1) root group is counted.
>>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>>
>>>>>>>> With this change, the occasion that only one group is activated 
>>>>>>>> can be
>>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>>> occasion.
>>>>>>>>
>>>>>>>> Signed-off-by: Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org 
>>>>>>>> <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>>
>>>>>>>> Reviewed-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org <mailto:jack-AlSwsSmVLrQ@public.gmane.org>>
>>>>>>>> ---
>>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>>> --- a/block/bfq-iosched.c
>>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct 
>>>>>>>> bfq_data *bfqd,
>>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>      struct bfq_queue *bfqq)
>>>>>>>> {
>>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>>> -
>>>>>>>> -for_each_entity(entity) {
>>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>>> -
>>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>>> -/*
>>>>>>>> -* entity is still active, because either
>>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>>> -* NULL (see the comments on the definition of
>>>>>>>> -* next_in_service for details on why
>>>>>>>> -* in_service_entity must be checked too).
>>>>>>>> -*
>>>>>>>> -* As a consequence, its parent entities are
>>>>>>>> -* active as well, and thus this loop must
>>>>>>>> -* stop here.
>>>>>>>> -*/
>>>>>>>> -break;
>>>>>>>> -}
>>>>>>>> -
>>>>>>>> -/*
>>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>>> -* all its pending requests completed. The following
>>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>>> -* needed. See the comments on
>>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>>> -*/
>>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>>> -}
>>>>>>>> -}
>>>>>>> With this part removed, I'm missing how you handle the following
>>>>>>> sequence of events:
>>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>>> yet
>>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>>> busy).  At this point Q must be removed from the counter.  It 
>>>>>>> seems to
>>>>>>> me that this case is not handled any longer
>>>>>> Hi, Paolo
>>>>>>
>>>>>> 1) At first, patch 1 support to track if bfqq has pending 
>>>>>> requests, it's
>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' 
>>>>>> when the
>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>>
>>>>>
>>>>> In patch 1 I don't see the flag cleared for the request-completion 
>>>>> event :(
>>>>>
>>>>> The piece of code involved is this:
>>>>>
>>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
>>>>> bfq_data *bfqd)
>>>>> {
>>>>> u64 now_ns;
>>>>> u32 delta_us;
>>>>>
>>>>> bfq_update_hw_tag(bfqd);
>>>>>
>>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>>> bfqd->tot_rq_in_driver--;
>>>>> bfqq->dispatched--;
>>>>>
>>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>>> /*
>>>>> * Set budget_timeout (which we overload to store the
>>>>> * time at which the queue remains with no backlog and
>>>>> * no outstanding request; used by the weight-raising
>>>>> * mechanism).
>>>>> */
>>>>> bfqq->budget_timeout = jiffies;
>>>>>
>>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>>> }
>>>>> ...
>>>>>
>>>>> Am I missing something?
>>>>
>>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>>> bfq_completed_request(). I think you may miss the later:
>>>>
>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>> index 0d46cb728bbf..0ec21018daba 100644
>>>> --- a/block/bfq-iosched.c
>>>> +++ b/block/bfq-iosched.c
>>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct 
>>>> bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>            */
>>>>           bfqq->budget_timeout = jiffies;
>>>>
>>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>>           bfq_weights_tree_remove(bfqd, bfqq);
>>>>       }
>>>>
>>>> Thanks,
>>>> Kuai
>>>>>
>>>>> Thanks,
>>>>> Paolo
>>>
>>
>> .
>>
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-26  2:34                     ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-26  2:34 UTC (permalink / raw)
  To: Paolo Valente, Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)

Hi, Paolo!

在 2022/08/25 22:59, Paolo Valente 写道:
> 
> 
>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai 
>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>
>> Hi, Paolo
>>
>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai 
>>>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>
>>>> Hi, Paolo
>>>>
>>> hi
>>>> Are you still interested in this patchset?
>>>>
>>> Yes. Sorry for replying very late again.
>>> Probably the last fix that you suggest is enough, but I'm a little bit
>>> concerned that it may be a little hasty.  In fact, before this fix, we
>>> exchanged several messages, and I didn't seem to be very good at
>>> convincing you about the need to keep into account also in-service
>>> I/O.  So, my question is: are you sure that now you have a
>>
>> I'm confused here, I'm pretty aware that in-service I/O(as said pending
>> requests is the patchset) should be counted, as you suggested in v7, are
>> you still thinking that the way in this patchset is problematic?
>>
>> I'll try to explain again that how to track is bfqq has pending pending
>> requests, please let me know if you still think there are some problems:
>>
>> patch 1 support to track if bfqq has pending requests, it's
>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>> first request is inserted to bfqq, and it's cleared when the last
>> request is completed. specifically the flag is set in
>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>> 'bfqq->diapatched' is false.
>>
> 
> This general description seems correct to me. Have you already sent a 
> new version of your patchset?

It's glad that we finially on the same page here.

Please take a look at patch 1, which already impelement the above
descriptions, it seems to me there is no need to send a new version
for now. If you think there are still some other problems, please let
me know.

Thanks,
Kuai
> 
> Thanks,
> Paolo
> 
>> Thanks,
>> Kuai
>>> clear/complete understanding of this non-trivial matter?
>>> Consequently, are we sure that this last fix is most certainly all we
>>> need?  Of course, I will check on my own, but if you reassure me on
>>> this point, I will feel more confident.
>>> Thanks,
>>> Paolo
>>>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>>>> Hi
>>>>>
>>>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>>>
>>>>>>
>>>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>>>>>>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com> 
>>>>>>> <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>>>> didn't receive my reply.
>>>>>>>
>>>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>>>> Sorry for the delay.
>>>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai 
>>>>>>>>> <yukuai3@huawei.com <mailto:yukuai3@huawei.com> 
>>>>>>>>> <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>>>>>
>>>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>>>> are not issued from root group. This is because
>>>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>>>> bfq_asymmetric_scenario().
>>>>>>>>>
>>>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>>>
>>>>>>>>> Before this patch:
>>>>>>>>> 1) root group will never be counted.
>>>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the 
>>>>>>>>> requests.
>>>>>>>>>
>>>>>>>>> After this patch:
>>>>>>>>> 1) root group is counted.
>>>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>>>
>>>>>>>>> With this change, the occasion that only one group is activated 
>>>>>>>>> can be
>>>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>>>> occasion.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com 
>>>>>>>>> <mailto:yukuai3@huawei.com> <mailto:yukuai3@huawei.com>>
>>>>>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz> 
>>>>>>>>> <mailto:jack@suse.cz>>
>>>>>>>>> ---
>>>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>>>> --- a/block/bfq-iosched.c
>>>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct 
>>>>>>>>> bfq_data *bfqd,
>>>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>>     struct bfq_queue *bfqq)
>>>>>>>>> {
>>>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>>>> -
>>>>>>>>> -for_each_entity(entity) {
>>>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>>>> -
>>>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>>>> -/*
>>>>>>>>> -* entity is still active, because either
>>>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>>>> -* NULL (see the comments on the definition of
>>>>>>>>> -* next_in_service for details on why
>>>>>>>>> -* in_service_entity must be checked too).
>>>>>>>>> -*
>>>>>>>>> -* As a consequence, its parent entities are
>>>>>>>>> -* active as well, and thus this loop must
>>>>>>>>> -* stop here.
>>>>>>>>> -*/
>>>>>>>>> -break;
>>>>>>>>> -}
>>>>>>>>> -
>>>>>>>>> -/*
>>>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>>>> -* all its pending requests completed. The following
>>>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>>>> -* needed. See the comments on
>>>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>>>> -*/
>>>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>>>> -}
>>>>>>>>> -}
>>>>>>>> With this part removed, I'm missing how you handle the following
>>>>>>>> sequence of events:
>>>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>>>> yet
>>>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>>>> busy).  At this point Q must be removed from the counter.  It 
>>>>>>>> seems to
>>>>>>>> me that this case is not handled any longer
>>>>>>> Hi, Paolo
>>>>>>>
>>>>>>> 1) At first, patch 1 support to track if bfqq has pending 
>>>>>>> requests, it's
>>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' 
>>>>>>> when the
>>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>>>
>>>>>>
>>>>>> In patch 1 I don't see the flag cleared for the request-completion 
>>>>>> event :(
>>>>>>
>>>>>> The piece of code involved is this:
>>>>>>
>>>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
>>>>>> bfq_data *bfqd)
>>>>>> {
>>>>>> u64 now_ns;
>>>>>> u32 delta_us;
>>>>>>
>>>>>> bfq_update_hw_tag(bfqd);
>>>>>>
>>>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>>>> bfqd->tot_rq_in_driver--;
>>>>>> bfqq->dispatched--;
>>>>>>
>>>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>>>> /*
>>>>>> * Set budget_timeout (which we overload to store the
>>>>>> * time at which the queue remains with no backlog and
>>>>>> * no outstanding request; used by the weight-raising
>>>>>> * mechanism).
>>>>>> */
>>>>>> bfqq->budget_timeout = jiffies;
>>>>>>
>>>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>>>> }
>>>>>> ...
>>>>>>
>>>>>> Am I missing something?
>>>>>
>>>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>>>> bfq_completed_request(). I think you may miss the later:
>>>>>
>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>> index 0d46cb728bbf..0ec21018daba 100644
>>>>> --- a/block/bfq-iosched.c
>>>>> +++ b/block/bfq-iosched.c
>>>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct 
>>>>> bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>>           */
>>>>>          bfqq->budget_timeout = jiffies;
>>>>>
>>>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>>>          bfq_weights_tree_remove(bfqd, bfqq);
>>>>>      }
>>>>>
>>>>> Thanks,
>>>>> Kuai
>>>>>>
>>>>>> Thanks,
>>>>>> Paolo
>>>>
>>> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
@ 2022-08-26  2:34                     ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-08-26  2:34 UTC (permalink / raw)
  To: Paolo Valente, Yu Kuai
  Cc: Jan Kara, cgroups-u79uwXL29TY76Z2rM5mHXA, linux-block, Tejun Heo,
	Jens Axboe, LKML, yi.zhang-hv44wF8Li93QT0dZR+AlfA, yukuai (C)

Hi, Paolo!

在 2022/08/25 22:59, Paolo Valente 写道:
> 
> 
>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai 
>> <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org <mailto:yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org>> ha scritto:
>>
>> Hi, Paolo
>>
>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai 
>>>> <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org <mailto:yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org>> ha scritto:
>>>>
>>>> Hi, Paolo
>>>>
>>> hi
>>>> Are you still interested in this patchset?
>>>>
>>> Yes. Sorry for replying very late again.
>>> Probably the last fix that you suggest is enough, but I'm a little bit
>>> concerned that it may be a little hasty.  In fact, before this fix, we
>>> exchanged several messages, and I didn't seem to be very good at
>>> convincing you about the need to keep into account also in-service
>>> I/O.  So, my question is: are you sure that now you have a
>>
>> I'm confused here, I'm pretty aware that in-service I/O(as said pending
>> requests is the patchset) should be counted, as you suggested in v7, are
>> you still thinking that the way in this patchset is problematic?
>>
>> I'll try to explain again that how to track is bfqq has pending pending
>> requests, please let me know if you still think there are some problems:
>>
>> patch 1 support to track if bfqq has pending requests, it's
>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>> first request is inserted to bfqq, and it's cleared when the last
>> request is completed. specifically the flag is set in
>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>> 'bfqq->diapatched' is false.
>>
> 
> This general description seems correct to me. Have you already sent a 
> new version of your patchset?

It's glad that we finially on the same page here.

Please take a look at patch 1, which already impelement the above
descriptions, it seems to me there is no need to send a new version
for now. If you think there are still some other problems, please let
me know.

Thanks,
Kuai
> 
> Thanks,
> Paolo
> 
>> Thanks,
>> Kuai
>>> clear/complete understanding of this non-trivial matter?
>>> Consequently, are we sure that this last fix is most certainly all we
>>> need?  Of course, I will check on my own, but if you reassure me on
>>> this point, I will feel more confident.
>>> Thanks,
>>> Paolo
>>>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>>>> Hi
>>>>>
>>>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>>>
>>>>>>
>>>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai 
>>>>>>> <yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org <mailto:yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org> 
>>>>>>> <mailto:yukuai1-XF6JlduFytWkHkcT6e4Xnw@public.gmane.org>> ha scritto:
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>>>> didn't receive my reply.
>>>>>>>
>>>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>>>> Sorry for the delay.
>>>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai 
>>>>>>>>> <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org> 
>>>>>>>>> <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>> ha scritto:
>>>>>>>>>
>>>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>>>> are not issued from root group. This is because
>>>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>>>> bfq_asymmetric_scenario().
>>>>>>>>>
>>>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>>>
>>>>>>>>> Before this patch:
>>>>>>>>> 1) root group will never be counted.
>>>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the 
>>>>>>>>> requests.
>>>>>>>>>
>>>>>>>>> After this patch:
>>>>>>>>> 1) root group is counted.
>>>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>>>
>>>>>>>>> With this change, the occasion that only one group is activated 
>>>>>>>>> can be
>>>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>>>> occasion.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Yu Kuai <yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org 
>>>>>>>>> <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org> <mailto:yukuai3-hv44wF8Li93QT0dZR+AlfA@public.gmane.org>>
>>>>>>>>> Reviewed-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org <mailto:jack-AlSwsSmVLrQ@public.gmane.org> 
>>>>>>>>> <mailto:jack-AlSwsSmVLrQ@public.gmane.org>>
>>>>>>>>> ---
>>>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>>>> --- a/block/bfq-iosched.c
>>>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct 
>>>>>>>>> bfq_data *bfqd,
>>>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>>     struct bfq_queue *bfqq)
>>>>>>>>> {
>>>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>>>> -
>>>>>>>>> -for_each_entity(entity) {
>>>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>>>> -
>>>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>>>> -/*
>>>>>>>>> -* entity is still active, because either
>>>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>>>> -* NULL (see the comments on the definition of
>>>>>>>>> -* next_in_service for details on why
>>>>>>>>> -* in_service_entity must be checked too).
>>>>>>>>> -*
>>>>>>>>> -* As a consequence, its parent entities are
>>>>>>>>> -* active as well, and thus this loop must
>>>>>>>>> -* stop here.
>>>>>>>>> -*/
>>>>>>>>> -break;
>>>>>>>>> -}
>>>>>>>>> -
>>>>>>>>> -/*
>>>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>>>> -* all its pending requests completed. The following
>>>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>>>> -* needed. See the comments on
>>>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>>>> -*/
>>>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>>>> -}
>>>>>>>>> -}
>>>>>>>> With this part removed, I'm missing how you handle the following
>>>>>>>> sequence of events:
>>>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>>>> yet
>>>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>>>> busy).  At this point Q must be removed from the counter.  It 
>>>>>>>> seems to
>>>>>>>> me that this case is not handled any longer
>>>>>>> Hi, Paolo
>>>>>>>
>>>>>>> 1) At first, patch 1 support to track if bfqq has pending 
>>>>>>> requests, it's
>>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' 
>>>>>>> when the
>>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>>>
>>>>>>
>>>>>> In patch 1 I don't see the flag cleared for the request-completion 
>>>>>> event :(
>>>>>>
>>>>>> The piece of code involved is this:
>>>>>>
>>>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct 
>>>>>> bfq_data *bfqd)
>>>>>> {
>>>>>> u64 now_ns;
>>>>>> u32 delta_us;
>>>>>>
>>>>>> bfq_update_hw_tag(bfqd);
>>>>>>
>>>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>>>> bfqd->tot_rq_in_driver--;
>>>>>> bfqq->dispatched--;
>>>>>>
>>>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>>>> /*
>>>>>> * Set budget_timeout (which we overload to store the
>>>>>> * time at which the queue remains with no backlog and
>>>>>> * no outstanding request; used by the weight-raising
>>>>>> * mechanism).
>>>>>> */
>>>>>> bfqq->budget_timeout = jiffies;
>>>>>>
>>>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>>>> }
>>>>>> ...
>>>>>>
>>>>>> Am I missing something?
>>>>>
>>>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>>>> bfq_completed_request(). I think you may miss the later:
>>>>>
>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>> index 0d46cb728bbf..0ec21018daba 100644
>>>>> --- a/block/bfq-iosched.c
>>>>> +++ b/block/bfq-iosched.c
>>>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct 
>>>>> bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>>           */
>>>>>          bfqq->budget_timeout = jiffies;
>>>>>
>>>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>>>          bfq_weights_tree_remove(bfqd, bfqq);
>>>>>      }
>>>>>
>>>>> Thanks,
>>>>> Kuai
>>>>>>
>>>>>> Thanks,
>>>>>> Paolo
>>>>
>>> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-08-26  2:34                     ` Yu Kuai
  (?)
@ 2022-09-06  9:37                     ` Paolo Valente
  2022-09-07  1:16                       ` Yu Kuai
  -1 siblings, 1 reply; 38+ messages in thread
From: Paolo Valente @ 2022-09-06  9:37 UTC (permalink / raw)
  To: Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)



> Il giorno 26 ago 2022, alle ore 04:34, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> 
> Hi, Paolo!
> 
> 在 2022/08/25 22:59, Paolo Valente 写道:
>>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>> 
>>> Hi, Paolo
>>> 
>>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>> 
>>>>> Hi, Paolo
>>>>> 
>>>> hi
>>>>> Are you still interested in this patchset?
>>>>> 
>>>> Yes. Sorry for replying very late again.
>>>> Probably the last fix that you suggest is enough, but I'm a little bit
>>>> concerned that it may be a little hasty.  In fact, before this fix, we
>>>> exchanged several messages, and I didn't seem to be very good at
>>>> convincing you about the need to keep into account also in-service
>>>> I/O.  So, my question is: are you sure that now you have a
>>> 
>>> I'm confused here, I'm pretty aware that in-service I/O(as said pending
>>> requests is the patchset) should be counted, as you suggested in v7, are
>>> you still thinking that the way in this patchset is problematic?
>>> 
>>> I'll try to explain again that how to track is bfqq has pending pending
>>> requests, please let me know if you still think there are some problems:
>>> 
>>> patch 1 support to track if bfqq has pending requests, it's
>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>> first request is inserted to bfqq, and it's cleared when the last
>>> request is completed. specifically the flag is set in
>>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>>> 'bfqq->diapatched' is false.
>>> 
>> This general description seems correct to me. Have you already sent a new version of your patchset?
> 
> It's glad that we finially on the same page here.
> 

Yep. Sorry for my chronicle delay.

> Please take a look at patch 1, which already impelement the above
> descriptions, it seems to me there is no need to send a new version
> for now. If you think there are still some other problems, please let
> me know.
> 

Patch 1 seems ok to me. I seem to have only one pending comment on this patch (3/4) instead. Let me paste previous stuff here for your convenience:

>> 
>> -	/*
>> -	 * Next function is invoked last, because it causes bfqq to be
>> -	 * freed if the following holds: bfqq is not in service and
>> -	 * has no dispatched request. DO NOT use bfqq after the next
>> -	 * function invocation.
>> -	 */
> I would really love it if you leave this comment.  I added it after
> suffering a lot for a nasty UAF.  Of course the first sentence may
> need to be adjusted if the code that precedes it is to be removed.
> Same as above, if this patch is applied, this function will be gone.

yes, but this comment now must be moved forward.

Looking forward for a new complete version, for a new review.  I'll do
my best to reply quicker.

Thanks,
Paolo





> Thanks,
> Kuai
>> Thanks,
>> Paolo
>>> Thanks,
>>> Kuai
>>>> clear/complete understanding of this non-trivial matter?
>>>> Consequently, are we sure that this last fix is most certainly all we
>>>> need?  Of course, I will check on my own, but if you reassure me on
>>>> this point, I will feel more confident.
>>>> Thanks,
>>>> Paolo
>>>>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>>>>> Hi
>>>>>> 
>>>>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>>>> 
>>>>>>> 
>>>>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com> <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>> 
>>>>>>>> Hi!
>>>>>>>> 
>>>>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>>>>> didn't receive my reply.
>>>>>>>> 
>>>>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>>>>> Sorry for the delay.
>>>>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com> <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>>>>>> 
>>>>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>>>>> are not issued from root group. This is because
>>>>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>>>>> bfq_asymmetric_scenario().
>>>>>>>>>> 
>>>>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>>>> 
>>>>>>>>>> Before this patch:
>>>>>>>>>> 1) root group will never be counted.
>>>>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>>>>>>>> 
>>>>>>>>>> After this patch:
>>>>>>>>>> 1) root group is counted.
>>>>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>>>> 
>>>>>>>>>> With this change, the occasion that only one group is activated can be
>>>>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>>>>> occasion.
>>>>>>>>>> 
>>>>>>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com> <mailto:yukuai3@huawei.com>>
>>>>>>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz> <mailto:jack@suse.cz>>
>>>>>>>>>> ---
>>>>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>>>> 
>>>>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>>>>> --- a/block/bfq-iosched.c
>>>>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>>>     struct bfq_queue *bfqq)
>>>>>>>>>> {
>>>>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>>>>> -
>>>>>>>>>> -for_each_entity(entity) {
>>>>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>>>>> -
>>>>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>>>>> -/*
>>>>>>>>>> -* entity is still active, because either
>>>>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>>>>> -* NULL (see the comments on the definition of
>>>>>>>>>> -* next_in_service for details on why
>>>>>>>>>> -* in_service_entity must be checked too).
>>>>>>>>>> -*
>>>>>>>>>> -* As a consequence, its parent entities are
>>>>>>>>>> -* active as well, and thus this loop must
>>>>>>>>>> -* stop here.
>>>>>>>>>> -*/
>>>>>>>>>> -break;
>>>>>>>>>> -}
>>>>>>>>>> -
>>>>>>>>>> -/*
>>>>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>>>>> -* all its pending requests completed. The following
>>>>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>>>>> -* needed. See the comments on
>>>>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>>>>> -*/
>>>>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>>>>> -}
>>>>>>>>>> -}
>>>>>>>>> With this part removed, I'm missing how you handle the following
>>>>>>>>> sequence of events:
>>>>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>>>>> yet
>>>>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>>>>>>> me that this case is not handled any longer
>>>>>>>> Hi, Paolo
>>>>>>>> 
>>>>>>>> 1) At first, patch 1 support to track if bfqq has pending requests, it's
>>>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>>>> 
>>>>>>> 
>>>>>>> In patch 1 I don't see the flag cleared for the request-completion event :(
>>>>>>> 
>>>>>>> The piece of code involved is this:
>>>>>>> 
>>>>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>>>> {
>>>>>>> u64 now_ns;
>>>>>>> u32 delta_us;
>>>>>>> 
>>>>>>> bfq_update_hw_tag(bfqd);
>>>>>>> 
>>>>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>>>>> bfqd->tot_rq_in_driver--;
>>>>>>> bfqq->dispatched--;
>>>>>>> 
>>>>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>>>>> /*
>>>>>>> * Set budget_timeout (which we overload to store the
>>>>>>> * time at which the queue remains with no backlog and
>>>>>>> * no outstanding request; used by the weight-raising
>>>>>>> * mechanism).
>>>>>>> */
>>>>>>> bfqq->budget_timeout = jiffies;
>>>>>>> 
>>>>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>>>>> }
>>>>>>> ...
>>>>>>> 
>>>>>>> Am I missing something?
>>>>>> 
>>>>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>>>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>>>>> bfq_completed_request(). I think you may miss the later:
>>>>>> 
>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>> index 0d46cb728bbf..0ec21018daba 100644
>>>>>> --- a/block/bfq-iosched.c
>>>>>> +++ b/block/bfq-iosched.c
>>>>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>>>           */
>>>>>>          bfqq->budget_timeout = jiffies;
>>>>>> 
>>>>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>>>>          bfq_weights_tree_remove(bfqd, bfqq);
>>>>>>      }
>>>>>> 
>>>>>> Thanks,
>>>>>> Kuai
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Paolo
>>>>> 
>>>> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-09-06  9:37                     ` Paolo Valente
@ 2022-09-07  1:16                       ` Yu Kuai
  2022-09-14  1:55                         ` Yu Kuai
  0 siblings, 1 reply; 38+ messages in thread
From: Yu Kuai @ 2022-09-07  1:16 UTC (permalink / raw)
  To: Paolo Valente, Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)

Hi, Paolo!

在 2022/09/06 17:37, Paolo Valente 写道:
> 
> 
>> Il giorno 26 ago 2022, alle ore 04:34, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>
>> Hi, Paolo!
>>
>> 在 2022/08/25 22:59, Paolo Valente 写道:
>>>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>
>>>> Hi, Paolo
>>>>
>>>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>
>>>>>> Hi, Paolo
>>>>>>
>>>>> hi
>>>>>> Are you still interested in this patchset?
>>>>>>
>>>>> Yes. Sorry for replying very late again.
>>>>> Probably the last fix that you suggest is enough, but I'm a little bit
>>>>> concerned that it may be a little hasty.  In fact, before this fix, we
>>>>> exchanged several messages, and I didn't seem to be very good at
>>>>> convincing you about the need to keep into account also in-service
>>>>> I/O.  So, my question is: are you sure that now you have a
>>>>
>>>> I'm confused here, I'm pretty aware that in-service I/O(as said pending
>>>> requests is the patchset) should be counted, as you suggested in v7, are
>>>> you still thinking that the way in this patchset is problematic?
>>>>
>>>> I'll try to explain again that how to track is bfqq has pending pending
>>>> requests, please let me know if you still think there are some problems:
>>>>
>>>> patch 1 support to track if bfqq has pending requests, it's
>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>> first request is inserted to bfqq, and it's cleared when the last
>>>> request is completed. specifically the flag is set in
>>>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>>>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>>>> 'bfqq->diapatched' is false.
>>>>
>>> This general description seems correct to me. Have you already sent a new version of your patchset?
>>
>> It's glad that we finially on the same page here.
>>
> 
> Yep. Sorry for my chronicle delay.

Better late than never 😁
> 
>> Please take a look at patch 1, which already impelement the above
>> descriptions, it seems to me there is no need to send a new version
>> for now. If you think there are still some other problems, please let
>> me know.
>>
> 
> Patch 1 seems ok to me. I seem to have only one pending comment on this patch (3/4) instead. Let me paste previous stuff here for your convenience:
That sounds good.

> 
>>>
>>> -	/*
>>> -	 * Next function is invoked last, because it causes bfqq to be
>>> -	 * freed if the following holds: bfqq is not in service and
>>> -	 * has no dispatched request. DO NOT use bfqq after the next
>>> -	 * function invocation.
>>> -	 */
>> I would really love it if you leave this comment.  I added it after
>> suffering a lot for a nasty UAF.  Of course the first sentence may
>> need to be adjusted if the code that precedes it is to be removed.
>> Same as above, if this patch is applied, this function will be gone.
> 
> yes, but this comment now must be moved forward.
> 
> Looking forward for a new complete version, for a new review.  I'll do
> my best to reply quicker.
I'll send a new version soon, perhaps I'll also add the following
cleanups to this patchset.

Thanks,
Kuai
> 
> Thanks,
> Paolo
> 
> 
> 
> 
> 
>> Thanks,
>> Kuai
>>> Thanks,
>>> Paolo
>>>> Thanks,
>>>> Kuai
>>>>> clear/complete understanding of this non-trivial matter?
>>>>> Consequently, are we sure that this last fix is most certainly all we
>>>>> need?  Of course, I will check on my own, but if you reassure me on
>>>>> this point, I will feel more confident.
>>>>> Thanks,
>>>>> Paolo
>>>>>> 在 2022/07/20 19:38, Yu Kuai 写道:
>>>>>>> Hi
>>>>>>>
>>>>>>> 在 2022/07/20 19:24, Paolo VALENTE 写道:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Il giorno 12 lug 2022, alle ore 15:30, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com> <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> I'm copying my reply with new mail address, because Paolo seems
>>>>>>>>> didn't receive my reply.
>>>>>>>>>
>>>>>>>>> 在 2022/06/23 23:32, Paolo Valente 写道:
>>>>>>>>>> Sorry for the delay.
>>>>>>>>>>> Il giorno 10 giu 2022, alle ore 04:17, Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com> <mailto:yukuai3@huawei.com>> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>>>>>>>>> are not issued from root group. This is because
>>>>>>>>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>>>>>>>>> bfq_asymmetric_scenario().
>>>>>>>>>>>
>>>>>>>>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>>>>>>>>
>>>>>>>>>>> Before this patch:
>>>>>>>>>>> 1) root group will never be counted.
>>>>>>>>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>>>>>>>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>>>>>>>>>
>>>>>>>>>>> After this patch:
>>>>>>>>>>> 1) root group is counted.
>>>>>>>>>>> 2) Count if bfqg have pending requests.
>>>>>>>>>>> 3) Don't count if bfqg complete all the requests.
>>>>>>>>>>>
>>>>>>>>>>> With this change, the occasion that only one group is activated can be
>>>>>>>>>>> detected, and next patch will support concurrent sync io in the
>>>>>>>>>>> occasion.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Yu Kuai <yukuai3@huawei.com <mailto:yukuai3@huawei.com> <mailto:yukuai3@huawei.com>>
>>>>>>>>>>> Reviewed-by: Jan Kara <jack@suse.cz <mailto:jack@suse.cz> <mailto:jack@suse.cz>>
>>>>>>>>>>> ---
>>>>>>>>>>> block/bfq-iosched.c | 42 ------------------------------------------
>>>>>>>>>>> block/bfq-iosched.h | 18 +++++++++---------
>>>>>>>>>>> block/bfq-wf2q.c    | 19 ++++---------------
>>>>>>>>>>> 3 files changed, 13 insertions(+), 66 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>>>>>> index 0ec21018daba..03b04892440c 100644
>>>>>>>>>>> --- a/block/bfq-iosched.c
>>>>>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>>>>>> @@ -970,48 +970,6 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>>>> void bfq_weights_tree_remove(struct bfq_data *bfqd,
>>>>>>>>>>>      struct bfq_queue *bfqq)
>>>>>>>>>>> {
>>>>>>>>>>> -struct bfq_entity *entity = bfqq->entity.parent;
>>>>>>>>>>> -
>>>>>>>>>>> -for_each_entity(entity) {
>>>>>>>>>>> -struct bfq_sched_data *sd = entity->my_sched_data;
>>>>>>>>>>> -
>>>>>>>>>>> -if (sd->next_in_service || sd->in_service_entity) {
>>>>>>>>>>> -/*
>>>>>>>>>>> -* entity is still active, because either
>>>>>>>>>>> -* next_in_service or in_service_entity is not
>>>>>>>>>>> -* NULL (see the comments on the definition of
>>>>>>>>>>> -* next_in_service for details on why
>>>>>>>>>>> -* in_service_entity must be checked too).
>>>>>>>>>>> -*
>>>>>>>>>>> -* As a consequence, its parent entities are
>>>>>>>>>>> -* active as well, and thus this loop must
>>>>>>>>>>> -* stop here.
>>>>>>>>>>> -*/
>>>>>>>>>>> -break;
>>>>>>>>>>> -}
>>>>>>>>>>> -
>>>>>>>>>>> -/*
>>>>>>>>>>> -* The decrement of num_groups_with_pending_reqs is
>>>>>>>>>>> -* not performed immediately upon the deactivation of
>>>>>>>>>>> -* entity, but it is delayed to when it also happens
>>>>>>>>>>> -* that the first leaf descendant bfqq of entity gets
>>>>>>>>>>> -* all its pending requests completed. The following
>>>>>>>>>>> -* instructions perform this delayed decrement, if
>>>>>>>>>>> -* needed. See the comments on
>>>>>>>>>>> -* num_groups_with_pending_reqs for details.
>>>>>>>>>>> -*/
>>>>>>>>>>> -if (entity->in_groups_with_pending_reqs) {
>>>>>>>>>>> -entity->in_groups_with_pending_reqs = false;
>>>>>>>>>>> -bfqd->num_groups_with_pending_reqs--;
>>>>>>>>>>> -}
>>>>>>>>>>> -}
>>>>>>>>>> With this part removed, I'm missing how you handle the following
>>>>>>>>>> sequence of events:
>>>>>>>>>> 1.  a queue Q becomes non busy but still has dispatched requests, so
>>>>>>>>>> it must not be removed from the counter of queues with pending reqs
>>>>>>>>>> yet
>>>>>>>>>> 2.  the last request of Q is completed with Q being still idle (non
>>>>>>>>>> busy).  At this point Q must be removed from the counter.  It seems to
>>>>>>>>>> me that this case is not handled any longer
>>>>>>>>> Hi, Paolo
>>>>>>>>>
>>>>>>>>> 1) At first, patch 1 support to track if bfqq has pending requests, it's
>>>>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>>>>> request is completed(based on weights_tree insertion and removal).
>>>>>>>>>
>>>>>>>>
>>>>>>>> In patch 1 I don't see the flag cleared for the request-completion event :(
>>>>>>>>
>>>>>>>> The piece of code involved is this:
>>>>>>>>
>>>>>>>> static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>>>>> {
>>>>>>>> u64 now_ns;
>>>>>>>> u32 delta_us;
>>>>>>>>
>>>>>>>> bfq_update_hw_tag(bfqd);
>>>>>>>>
>>>>>>>> bfqd->rq_in_driver[bfqq->actuator_idx]--;
>>>>>>>> bfqd->tot_rq_in_driver--;
>>>>>>>> bfqq->dispatched--;
>>>>>>>>
>>>>>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
>>>>>>>> /*
>>>>>>>> * Set budget_timeout (which we overload to store the
>>>>>>>> * time at which the queue remains with no backlog and
>>>>>>>> * no outstanding request; used by the weight-raising
>>>>>>>> * mechanism).
>>>>>>>> */
>>>>>>>> bfqq->budget_timeout = jiffies;
>>>>>>>>
>>>>>>>> bfq_weights_tree_remove(bfqd, bfqq);
>>>>>>>> }
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Am I missing something?
>>>>>>>
>>>>>>> I add a new api bfq_del_bfqq_in_groups_with_pending_reqs() in patch 1
>>>>>>> to clear the flag, and it's called both from bfq_del_bfqq_busy() and
>>>>>>> bfq_completed_request(). I think you may miss the later:
>>>>>>>
>>>>>>> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
>>>>>>> index 0d46cb728bbf..0ec21018daba 100644
>>>>>>> --- a/block/bfq-iosched.c
>>>>>>> +++ b/block/bfq-iosched.c
>>>>>>> @@ -6263,6 +6263,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
>>>>>>>            */
>>>>>>>           bfqq->budget_timeout = jiffies;
>>>>>>>
>>>>>>> +        bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);
>>>>>>>           bfq_weights_tree_remove(bfqd, bfqq);
>>>>>>>       }
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kuai
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Paolo
>>>>>>
>>>>> .
>>
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-09-07  1:16                       ` Yu Kuai
@ 2022-09-14  1:55                         ` Yu Kuai
  2022-09-14  7:50                           ` Paolo VALENTE
  0 siblings, 1 reply; 38+ messages in thread
From: Yu Kuai @ 2022-09-14  1:55 UTC (permalink / raw)
  To: Yu Kuai, Paolo Valente, Jan Kara
  Cc: cgroups, linux-block, Tejun Heo, Jens Axboe, LKML, yi.zhang, yukuai (C)



在 2022/09/07 9:16, Yu Kuai 写道:
> Hi, Paolo!
> 
> 在 2022/09/06 17:37, Paolo Valente 写道:
>>
>>
>>> Il giorno 26 ago 2022, alle ore 04:34, Yu Kuai 
>>> <yukuai1@huaweicloud.com> ha scritto:
>>>
>>> Hi, Paolo!
>>>
>>> 在 2022/08/25 22:59, Paolo Valente 写道:
>>>>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai 
>>>>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>
>>>>> Hi, Paolo
>>>>>
>>>>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai 
>>>>>>> <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha 
>>>>>>> scritto:
>>>>>>>
>>>>>>> Hi, Paolo
>>>>>>>
>>>>>> hi
>>>>>>> Are you still interested in this patchset?
>>>>>>>
>>>>>> Yes. Sorry for replying very late again.
>>>>>> Probably the last fix that you suggest is enough, but I'm a little 
>>>>>> bit
>>>>>> concerned that it may be a little hasty.  In fact, before this 
>>>>>> fix, we
>>>>>> exchanged several messages, and I didn't seem to be very good at
>>>>>> convincing you about the need to keep into account also in-service
>>>>>> I/O.  So, my question is: are you sure that now you have a
>>>>>
>>>>> I'm confused here, I'm pretty aware that in-service I/O(as said 
>>>>> pending
>>>>> requests is the patchset) should be counted, as you suggested in 
>>>>> v7, are
>>>>> you still thinking that the way in this patchset is problematic?
>>>>>
>>>>> I'll try to explain again that how to track is bfqq has pending 
>>>>> pending
>>>>> requests, please let me know if you still think there are some 
>>>>> problems:
>>>>>
>>>>> patch 1 support to track if bfqq has pending requests, it's
>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when 
>>>>> the
>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>> request is completed. specifically the flag is set in
>>>>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>>>>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>>>>> 'bfqq->diapatched' is false.
>>>>>
>>>> This general description seems correct to me. Have you already sent 
>>>> a new version of your patchset?
>>>
>>> It's glad that we finially on the same page here.
>>>
>>
>> Yep. Sorry for my chronicle delay.
> 
> Better late than never 😁
>>
>>> Please take a look at patch 1, which already impelement the above
>>> descriptions, it seems to me there is no need to send a new version
>>> for now. If you think there are still some other problems, please let
>>> me know.
>>>
>>
>> Patch 1 seems ok to me. I seem to have only one pending comment on 
>> this patch (3/4) instead. Let me paste previous stuff here for your 
>> convenience:
> That sounds good.
> 
>>
>>>>
>>>> -    /*
>>>> -     * Next function is invoked last, because it causes bfqq to be
>>>> -     * freed if the following holds: bfqq is not in service and
>>>> -     * has no dispatched request. DO NOT use bfqq after the next
>>>> -     * function invocation.
>>>> -     */
>>> I would really love it if you leave this comment.  I added it after
>>> suffering a lot for a nasty UAF.  Of course the first sentence may
>>> need to be adjusted if the code that precedes it is to be removed.
>>> Same as above, if this patch is applied, this function will be gone.

Hi, I'm curious while I'm trying to add the comment, before this
patchset, can bfqq be freed when bfq_weights_tree_remove is called?

bfq_completed_request
  bfqq->dispatched--
  if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq))
   bfq_weights_tree_remove(bfqd, bfqq);

  // continue to use bfqq

It seems to me this is problematic if so, because bfqq is used after
bfq_weights_tree_remove() is called.

Thanks,
Kuai


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-09-14  1:55                         ` Yu Kuai
@ 2022-09-14  7:50                           ` Paolo VALENTE
  2022-09-14  8:15                             ` Yu Kuai
  0 siblings, 1 reply; 38+ messages in thread
From: Paolo VALENTE @ 2022-09-14  7:50 UTC (permalink / raw)
  To: Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)



> Il giorno 14 set 2022, alle ore 03:55, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> 
> 
> 
> 在 2022/09/07 9:16, Yu Kuai 写道:
>> Hi, Paolo!
>> 在 2022/09/06 17:37, Paolo Valente 写道:
>>> 
>>> 
>>>> Il giorno 26 ago 2022, alle ore 04:34, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>>> 
>>>> Hi, Paolo!
>>>> 
>>>> 在 2022/08/25 22:59, Paolo Valente 写道:
>>>>>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>> 
>>>>>> Hi, Paolo
>>>>>> 
>>>>>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>>>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>> 
>>>>>>>> Hi, Paolo
>>>>>>>> 
>>>>>>> hi
>>>>>>>> Are you still interested in this patchset?
>>>>>>>> 
>>>>>>> Yes. Sorry for replying very late again.
>>>>>>> Probably the last fix that you suggest is enough, but I'm a little bit
>>>>>>> concerned that it may be a little hasty.  In fact, before this fix, we
>>>>>>> exchanged several messages, and I didn't seem to be very good at
>>>>>>> convincing you about the need to keep into account also in-service
>>>>>>> I/O.  So, my question is: are you sure that now you have a
>>>>>> 
>>>>>> I'm confused here, I'm pretty aware that in-service I/O(as said pending
>>>>>> requests is the patchset) should be counted, as you suggested in v7, are
>>>>>> you still thinking that the way in this patchset is problematic?
>>>>>> 
>>>>>> I'll try to explain again that how to track is bfqq has pending pending
>>>>>> requests, please let me know if you still think there are some problems:
>>>>>> 
>>>>>> patch 1 support to track if bfqq has pending requests, it's
>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>> request is completed. specifically the flag is set in
>>>>>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>>>>>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>>>>>> 'bfqq->diapatched' is false.
>>>>>> 
>>>>> This general description seems correct to me. Have you already sent a new version of your patchset?
>>>> 
>>>> It's glad that we finially on the same page here.
>>>> 
>>> 
>>> Yep. Sorry for my chronicle delay.
>> Better late than never 😁
>>> 
>>>> Please take a look at patch 1, which already impelement the above
>>>> descriptions, it seems to me there is no need to send a new version
>>>> for now. If you think there are still some other problems, please let
>>>> me know.
>>>> 
>>> 
>>> Patch 1 seems ok to me. I seem to have only one pending comment on this patch (3/4) instead. Let me paste previous stuff here for your convenience:
>> That sounds good.
>>> 
>>>>> 
>>>>> -    /*
>>>>> -     * Next function is invoked last, because it causes bfqq to be
>>>>> -     * freed if the following holds: bfqq is not in service and
>>>>> -     * has no dispatched request. DO NOT use bfqq after the next
>>>>> -     * function invocation.
>>>>> -     */
>>>> I would really love it if you leave this comment.  I added it after
>>>> suffering a lot for a nasty UAF.  Of course the first sentence may
>>>> need to be adjusted if the code that precedes it is to be removed.
>>>> Same as above, if this patch is applied, this function will be gone.
> 
> Hi, I'm curious while I'm trying to add the comment, before this
> patchset, can bfqq be freed when bfq_weights_tree_remove is called?
> 
> bfq_completed_request
> bfqq->dispatched--
> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq))
>  bfq_weights_tree_remove(bfqd, bfqq);
> 
> // continue to use bfqq
> 
> It seems to me this is problematic if so, because bfqq is used after
> bfq_weights_tree_remove() is called.
> 

It is.  Yet, IIRC, I verified that bfqq was not used after that free,
and I added that comment as a heads-up.  What is a scenario (before
your pending modifications) where this use-after-free happens?

Thanks,
Paolo

> Thanks,
> Kuai


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-09-14  7:50                           ` Paolo VALENTE
@ 2022-09-14  8:15                             ` Yu Kuai
  2022-09-14  9:00                               ` Jan Kara
  0 siblings, 1 reply; 38+ messages in thread
From: Yu Kuai @ 2022-09-14  8:15 UTC (permalink / raw)
  To: Paolo VALENTE, Yu Kuai
  Cc: Jan Kara, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)

Hi, Paolo

在 2022/09/14 15:50, Paolo VALENTE 写道:
> 
> 
>> Il giorno 14 set 2022, alle ore 03:55, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>
>>
>>
>> 在 2022/09/07 9:16, Yu Kuai 写道:
>>> Hi, Paolo!
>>> 在 2022/09/06 17:37, Paolo Valente 写道:
>>>>
>>>>
>>>>> Il giorno 26 ago 2022, alle ore 04:34, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>>>>
>>>>> Hi, Paolo!
>>>>>
>>>>> 在 2022/08/25 22:59, Paolo Valente 写道:
>>>>>>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>
>>>>>>> Hi, Paolo
>>>>>>>
>>>>>>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>>>>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>>>
>>>>>>>>> Hi, Paolo
>>>>>>>>>
>>>>>>>> hi
>>>>>>>>> Are you still interested in this patchset?
>>>>>>>>>
>>>>>>>> Yes. Sorry for replying very late again.
>>>>>>>> Probably the last fix that you suggest is enough, but I'm a little bit
>>>>>>>> concerned that it may be a little hasty.  In fact, before this fix, we
>>>>>>>> exchanged several messages, and I didn't seem to be very good at
>>>>>>>> convincing you about the need to keep into account also in-service
>>>>>>>> I/O.  So, my question is: are you sure that now you have a
>>>>>>>
>>>>>>> I'm confused here, I'm pretty aware that in-service I/O(as said pending
>>>>>>> requests is the patchset) should be counted, as you suggested in v7, are
>>>>>>> you still thinking that the way in this patchset is problematic?
>>>>>>>
>>>>>>> I'll try to explain again that how to track is bfqq has pending pending
>>>>>>> requests, please let me know if you still think there are some problems:
>>>>>>>
>>>>>>> patch 1 support to track if bfqq has pending requests, it's
>>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>>> request is completed. specifically the flag is set in
>>>>>>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>>>>>>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>>>>>>> 'bfqq->diapatched' is false.
>>>>>>>
>>>>>> This general description seems correct to me. Have you already sent a new version of your patchset?
>>>>>
>>>>> It's glad that we finially on the same page here.
>>>>>
>>>>
>>>> Yep. Sorry for my chronicle delay.
>>> Better late than never 😁
>>>>
>>>>> Please take a look at patch 1, which already impelement the above
>>>>> descriptions, it seems to me there is no need to send a new version
>>>>> for now. If you think there are still some other problems, please let
>>>>> me know.
>>>>>
>>>>
>>>> Patch 1 seems ok to me. I seem to have only one pending comment on this patch (3/4) instead. Let me paste previous stuff here for your convenience:
>>> That sounds good.
>>>>
>>>>>>
>>>>>> -    /*
>>>>>> -     * Next function is invoked last, because it causes bfqq to be
>>>>>> -     * freed if the following holds: bfqq is not in service and
>>>>>> -     * has no dispatched request. DO NOT use bfqq after the next
>>>>>> -     * function invocation.
>>>>>> -     */
>>>>> I would really love it if you leave this comment.  I added it after
>>>>> suffering a lot for a nasty UAF.  Of course the first sentence may
>>>>> need to be adjusted if the code that precedes it is to be removed.
>>>>> Same as above, if this patch is applied, this function will be gone.
>>
>> Hi, I'm curious while I'm trying to add the comment, before this
>> patchset, can bfqq be freed when bfq_weights_tree_remove is called?
>>
>> bfq_completed_request
>> bfqq->dispatched--
>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq))
>>   bfq_weights_tree_remove(bfqd, bfqq);
>>
>> // continue to use bfqq
>>
>> It seems to me this is problematic if so, because bfqq is used after
>> bfq_weights_tree_remove() is called.
>>
> 
> It is.  Yet, IIRC, I verified that bfqq was not used after that free,
> and I added that comment as a heads-up.  What is a scenario (before
> your pending modifications) where this use-after-free happens?
> 

No, it never happens, I just notice it because it'll be weird if I
place the comment where bfq_weights_tree_remove() is called, since bfqq
will still be accessed.

If the suituation that the comment says is possible, perhaps we should
move bfq_weights_tree_remove() to the last of bfq_completed_request().
However, it seems that we haven't meet the problem for quite a long
time...

Thanks,
Kuai

> Thanks,
> Paolo
> 
>> Thanks,
>> Kuai
> 
> .
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-09-14  8:15                             ` Yu Kuai
@ 2022-09-14  9:00                               ` Jan Kara
  2022-09-15  1:18                                 ` Yu Kuai
  0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2022-09-14  9:00 UTC (permalink / raw)
  To: Yu Kuai
  Cc: Paolo VALENTE, Jan Kara, cgroups, linux-block, Tejun Heo,
	Jens Axboe, LKML, yi.zhang, yukuai (C)

Hi guys!

On Wed 14-09-22 16:15:26, Yu Kuai wrote:
> 在 2022/09/14 15:50, Paolo VALENTE 写道:
> > 
> > 
> > > Il giorno 14 set 2022, alle ore 03:55, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> > > 
> > > 
> > > 
> > > 在 2022/09/07 9:16, Yu Kuai 写道:
> > > > Hi, Paolo!
> > > > 在 2022/09/06 17:37, Paolo Valente 写道:
> > > > > 
> > > > > 
> > > > > > Il giorno 26 ago 2022, alle ore 04:34, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> > > > > > 
> > > > > > Hi, Paolo!
> > > > > > 
> > > > > > 在 2022/08/25 22:59, Paolo Valente 写道:
> > > > > > > > Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
> > > > > > > > 
> > > > > > > > Hi, Paolo
> > > > > > > > 
> > > > > > > > 在 2022/08/10 18:49, Paolo Valente 写道:
> > > > > > > > > > Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
> > > > > > > > > > 
> > > > > > > > > > Hi, Paolo
> > > > > > > > > > 
> > > > > > > > > hi
> > > > > > > > > > Are you still interested in this patchset?
> > > > > > > > > > 
> > > > > > > > > Yes. Sorry for replying very late again.
> > > > > > > > > Probably the last fix that you suggest is enough, but I'm a little bit
> > > > > > > > > concerned that it may be a little hasty.  In fact, before this fix, we
> > > > > > > > > exchanged several messages, and I didn't seem to be very good at
> > > > > > > > > convincing you about the need to keep into account also in-service
> > > > > > > > > I/O.  So, my question is: are you sure that now you have a
> > > > > > > > 
> > > > > > > > I'm confused here, I'm pretty aware that in-service I/O(as said pending
> > > > > > > > requests is the patchset) should be counted, as you suggested in v7, are
> > > > > > > > you still thinking that the way in this patchset is problematic?
> > > > > > > > 
> > > > > > > > I'll try to explain again that how to track is bfqq has pending pending
> > > > > > > > requests, please let me know if you still think there are some problems:
> > > > > > > > 
> > > > > > > > patch 1 support to track if bfqq has pending requests, it's
> > > > > > > > done by setting the flag 'entity->in_groups_with_pending_reqs' when the
> > > > > > > > first request is inserted to bfqq, and it's cleared when the last
> > > > > > > > request is completed. specifically the flag is set in
> > > > > > > > bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
> > > > > > > > both in bfq_completed_request() and bfq_del_bfqq_busy() when
> > > > > > > > 'bfqq->diapatched' is false.
> > > > > > > > 
> > > > > > > This general description seems correct to me. Have you already sent a new version of your patchset?
> > > > > > 
> > > > > > It's glad that we finially on the same page here.
> > > > > > 
> > > > > 
> > > > > Yep. Sorry for my chronicle delay.
> > > > Better late than never 😁
> > > > > 
> > > > > > Please take a look at patch 1, which already impelement the above
> > > > > > descriptions, it seems to me there is no need to send a new version
> > > > > > for now. If you think there are still some other problems, please let
> > > > > > me know.
> > > > > > 
> > > > > 
> > > > > Patch 1 seems ok to me. I seem to have only one pending comment on this patch (3/4) instead. Let me paste previous stuff here for your convenience:
> > > > That sounds good.
> > > > > 
> > > > > > > 
> > > > > > > -    /*
> > > > > > > -     * Next function is invoked last, because it causes bfqq to be
> > > > > > > -     * freed if the following holds: bfqq is not in service and
> > > > > > > -     * has no dispatched request. DO NOT use bfqq after the next
> > > > > > > -     * function invocation.
> > > > > > > -     */
> > > > > > I would really love it if you leave this comment.  I added it after
> > > > > > suffering a lot for a nasty UAF.  Of course the first sentence may
> > > > > > need to be adjusted if the code that precedes it is to be removed.
> > > > > > Same as above, if this patch is applied, this function will be gone.
> > > 
> > > Hi, I'm curious while I'm trying to add the comment, before this
> > > patchset, can bfqq be freed when bfq_weights_tree_remove is called?
> > > 
> > > bfq_completed_request
> > > bfqq->dispatched--
> > > if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq))
> > >   bfq_weights_tree_remove(bfqd, bfqq);
> > > 
> > > // continue to use bfqq
> > > 
> > > It seems to me this is problematic if so, because bfqq is used after
> > > bfq_weights_tree_remove() is called.
> > > 
> > 
> > It is.  Yet, IIRC, I verified that bfqq was not used after that free,
> > and I added that comment as a heads-up.  What is a scenario (before
> > your pending modifications) where this use-after-free happens?
> > 
> 
> No, it never happens, I just notice it because it'll be weird if I
> place the comment where bfq_weights_tree_remove() is called, since bfqq
> will still be accessed.
> 
> If the suituation that the comment says is possible, perhaps we should
> move bfq_weights_tree_remove() to the last of bfq_completed_request().
> However, it seems that we haven't meet the problem for quite a long
> time...

I'm bit confused which comment you are speaking about but
bfq_completed_request() gets called only from bfq_finish_requeue_request()
and the request itself still holds a reference to bfqq. Only later in
bfq_finish_requeue_request() when we do:

	bfqq_request_freed(bfqq);
	bfq_put_queue(bfqq);

bfqq can get freed.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 38+ messages in thread
* Re: [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  2022-09-14  9:00                               ` Jan Kara
@ 2022-09-15  1:18                                 ` Yu Kuai
  0 siblings, 0 replies; 38+ messages in thread
From: Yu Kuai @ 2022-09-15  1:18 UTC (permalink / raw)
  To: Jan Kara, Yu Kuai
  Cc: Paolo VALENTE, cgroups, linux-block, Tejun Heo, Jens Axboe, LKML,
	yi.zhang, yukuai (C)

Hi,

在 2022/09/14 17:00, Jan Kara 写道:
> Hi guys!
> 
> On Wed 14-09-22 16:15:26, Yu Kuai wrote:
>> 在 2022/09/14 15:50, Paolo VALENTE 写道:
>>>
>>>
>>>> Il giorno 14 set 2022, alle ore 03:55, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>>>
>>>>
>>>>
>>>> 在 2022/09/07 9:16, Yu Kuai 写道:
>>>>> Hi, Paolo!
>>>>> 在 2022/09/06 17:37, Paolo Valente 写道:
>>>>>>
>>>>>>
>>>>>>> Il giorno 26 ago 2022, alle ore 04:34, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>>>>>>
>>>>>>> Hi, Paolo!
>>>>>>>
>>>>>>> 在 2022/08/25 22:59, Paolo Valente 写道:
>>>>>>>>> Il giorno 11 ago 2022, alle ore 03:19, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>>>
>>>>>>>>> Hi, Paolo
>>>>>>>>>
>>>>>>>>> 在 2022/08/10 18:49, Paolo Valente 写道:
>>>>>>>>>>> Il giorno 27 lug 2022, alle ore 14:11, Yu Kuai <yukuai1@huaweicloud.com <mailto:yukuai1@huaweicloud.com>> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> Hi, Paolo
>>>>>>>>>>>
>>>>>>>>>> hi
>>>>>>>>>>> Are you still interested in this patchset?
>>>>>>>>>>>
>>>>>>>>>> Yes. Sorry for replying very late again.
>>>>>>>>>> Probably the last fix that you suggest is enough, but I'm a little bit
>>>>>>>>>> concerned that it may be a little hasty.  In fact, before this fix, we
>>>>>>>>>> exchanged several messages, and I didn't seem to be very good at
>>>>>>>>>> convincing you about the need to keep into account also in-service
>>>>>>>>>> I/O.  So, my question is: are you sure that now you have a
>>>>>>>>>
>>>>>>>>> I'm confused here, I'm pretty aware that in-service I/O(as said pending
>>>>>>>>> requests is the patchset) should be counted, as you suggested in v7, are
>>>>>>>>> you still thinking that the way in this patchset is problematic?
>>>>>>>>>
>>>>>>>>> I'll try to explain again that how to track is bfqq has pending pending
>>>>>>>>> requests, please let me know if you still think there are some problems:
>>>>>>>>>
>>>>>>>>> patch 1 support to track if bfqq has pending requests, it's
>>>>>>>>> done by setting the flag 'entity->in_groups_with_pending_reqs' when the
>>>>>>>>> first request is inserted to bfqq, and it's cleared when the last
>>>>>>>>> request is completed. specifically the flag is set in
>>>>>>>>> bfq_add_bfqq_busy() when 'bfqq->dispatched' if false, and it's cleared
>>>>>>>>> both in bfq_completed_request() and bfq_del_bfqq_busy() when
>>>>>>>>> 'bfqq->diapatched' is false.
>>>>>>>>>
>>>>>>>> This general description seems correct to me. Have you already sent a new version of your patchset?
>>>>>>>
>>>>>>> It's glad that we finially on the same page here.
>>>>>>>
>>>>>>
>>>>>> Yep. Sorry for my chronicle delay.
>>>>> Better late than never 😁
>>>>>>
>>>>>>> Please take a look at patch 1, which already impelement the above
>>>>>>> descriptions, it seems to me there is no need to send a new version
>>>>>>> for now. If you think there are still some other problems, please let
>>>>>>> me know.
>>>>>>>
>>>>>>
>>>>>> Patch 1 seems ok to me. I seem to have only one pending comment on this patch (3/4) instead. Let me paste previous stuff here for your convenience:
>>>>> That sounds good.
>>>>>>
>>>>>>>>
>>>>>>>> -    /*
>>>>>>>> -     * Next function is invoked last, because it causes bfqq to be
>>>>>>>> -     * freed if the following holds: bfqq is not in service and
>>>>>>>> -     * has no dispatched request. DO NOT use bfqq after the next
>>>>>>>> -     * function invocation.
>>>>>>>> -     */
>>>>>>> I would really love it if you leave this comment.  I added it after
>>>>>>> suffering a lot for a nasty UAF.  Of course the first sentence may
>>>>>>> need to be adjusted if the code that precedes it is to be removed.
>>>>>>> Same as above, if this patch is applied, this function will be gone.
>>>>
>>>> Hi, I'm curious while I'm trying to add the comment, before this
>>>> patchset, can bfqq be freed when bfq_weights_tree_remove is called?
>>>>
>>>> bfq_completed_request
>>>> bfqq->dispatched--
>>>> if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq))
>>>>    bfq_weights_tree_remove(bfqd, bfqq);
>>>>
>>>> // continue to use bfqq
>>>>
>>>> It seems to me this is problematic if so, because bfqq is used after
>>>> bfq_weights_tree_remove() is called.
>>>>
>>>
>>> It is.  Yet, IIRC, I verified that bfqq was not used after that free,
>>> and I added that comment as a heads-up.  What is a scenario (before
>>> your pending modifications) where this use-after-free happens?
>>>
>>
>> No, it never happens, I just notice it because it'll be weird if I
>> place the comment where bfq_weights_tree_remove() is called, since bfqq
>> will still be accessed.
>>
>> If the suituation that the comment says is possible, perhaps we should
>> move bfq_weights_tree_remove() to the last of bfq_completed_request().
>> However, it seems that we haven't meet the problem for quite a long
>> time...
> 
> I'm bit confused which comment you are speaking about but
> bfq_completed_request() gets called only from bfq_finish_requeue_request()
> and the request itself still holds a reference to bfqq. Only later in
> bfq_finish_requeue_request() when we do:
> 
> 	bfqq_request_freed(bfqq);
> 	bfq_put_queue(bfqq);
> 
> bfqq can get freed.

Yes, you're right. Then I think the only place that
bfq_weights_tree_remove() can free bfqq is from bfq_del_bfqq_busy().
I'll move the following comment with a little adjustment here, which is
from bfq_weights_tree_remove() before this patchset:

         /*
         ┊* Next function is invoked last, because it causes bfqq to be
         ┊* freed. DO NOT use bfqq after the next function invocation.
         ┊*/

Thanks,
Kuai

> 
> 								Honza
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread
end of thread, other threads:[~2022-09-15  1:18 UTC | newest]

Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-10  2:16 [PATCH -next v10 0/4] support concurrent sync io for bfq on a specail occasion Yu Kuai
2022-06-10  2:16 ` Yu Kuai
2022-06-10  2:16 ` [PATCH -next v10 1/4] block, bfq: support to track if bfqq has pending requests Yu Kuai
2022-06-10  2:16   ` Yu Kuai
2022-06-10  2:16 ` [PATCH -next v10 2/4] block, bfq: record how many queues have " Yu Kuai
2022-06-10  2:16   ` Yu Kuai
2022-06-10  2:17 ` [PATCH -next v10 3/4] block, bfq: refactor the counting of 'num_groups_with_pending_reqs' Yu Kuai
2022-06-10  2:17   ` Yu Kuai
2022-06-23 15:32   ` Paolo Valente
2022-06-23 15:32     ` Paolo Valente
2022-06-24  1:26     ` Yu Kuai
2022-06-24  1:26       ` Yu Kuai
2022-06-25  8:10       ` Yu Kuai
2022-06-25  8:10         ` Yu Kuai
2022-07-12 13:30     ` Yu Kuai
2022-07-12 13:30       ` Yu Kuai
     [not found]       ` <C2CF100A-9A7C-4300-9A70-1295BC939C66@unimore.it>
2022-07-20 11:38         ` Yu Kuai
2022-07-27 12:11           ` Yu Kuai
2022-08-05 11:20             ` Yu Kuai
2022-08-05 11:20               ` Yu Kuai
2022-08-10 10:49             ` Paolo Valente
2022-08-11  1:19               ` Yu Kuai
2022-08-11  1:19                 ` Yu Kuai
2022-08-25 12:14                 ` Yu Kuai
2022-08-25 12:14                   ` Yu Kuai
     [not found]                 ` <D89DCF20-27D8-4F8F-B8B0-FD193FC4F18D@unimore.it>
2022-08-26  2:34                   ` Yu Kuai
2022-08-26  2:34                     ` Yu Kuai
2022-09-06  9:37                     ` Paolo Valente
2022-09-07  1:16                       ` Yu Kuai
2022-09-14  1:55                         ` Yu Kuai
2022-09-14  7:50                           ` Paolo VALENTE
2022-09-14  8:15                             ` Yu Kuai
2022-09-14  9:00                               ` Jan Kara
2022-09-15  1:18                                 ` Yu Kuai
2022-06-10  2:17 ` [PATCH -next v10 4/4] block, bfq: do not idle if only one group is activated Yu Kuai
2022-06-10  2:17   ` Yu Kuai
2022-06-17  1:12 ` [PATCH -next v10 0/4] support concurrent sync io for bfq on a specail occasion Yu Kuai
2022-06-17  1:12   ` Yu Kuai

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.