[RFC V5 0/16] blk: blk-rq-qos framework refactor and cleanup

[RFC V5 0/16] blk: blk-rq-qos framework refactor and cleanup

[Wang Jianchao (Kuaishou)]

Hi Jens

This patchset is working on our v4.18 kernel to provide the capacity
for upgrade blk-rq-qos policies on the fly w/o needing to restart
machine and applications.

In previous discussion, the modularization has been rid of. Please
refer to the link:
https://www.spinics.net/lists/linux-block/msg79958.html

Right now, patchset is to make blk-rq-qos policies pluggable through
a new sysfs interface, /sys/block/xxx/queue/qos. Then we can open/close
policy per device and needn't to iterate the rqos list and allocate
memory for them if we don't use.

The blktest has passed along with a new test case which switches
rqos policies with running fio in cgroups. Refer to following link,

https://patchwork.kernel.org/project/linux-block/patch/20220224085340.53854-1-jianchao.wan9@gmail.com/

When do the test above, I find the queue freezing mechanism has
not been able to protect the whole blk_mq_submit_bio(). I guess it
should be for saving some cpu cyles. But queue freezing is the basis
for patchset. So I add it back as the 1st patch which maybe unacceptable.

The 2nd ~ 8th provide the new API and adapt the code of blk-wbt,
blk-iolatency, blk-iocost and blk-ioprio to use the new API. And
finally, the /sys/block/xxx/queue/qos is exposed.

The following patches on blk-wbt may break its original assumpation.

The 9th patch is to expose 'wbt_lat_nsec' when 'wbt' police is opened.
And some code cleanup is also there as some interfaces needn't to be
exposed.

The 10th patch gets rid of the following actions, enable wbt by default,
disable wbt when bfq is enabled. The user should be responsible for
selecting different policies.

The 11th patch gets rid of the rqos debugfs interfaces registeration
when queue is initialized as there is not any default policies any more.

The 12th patch rename blk-rq-qos's queue_depth_changed callback to
setting_changed, then we can involve the cache state changing.

The 13th patch involve the cache state changing into setting_changed callback,
then the last exposed interface of wbt is get rid of.

The 14th patch cleanup the blk-wbt.h as there is nothing need to be exposed.

The 15th patch moves the rqos debugfs registeration interfaces to blk-rq-qos.c

The 16th patch adds a new CONFIG_BLK_RQ_QOS to control the whole blk-rq-qos code.


Changes since v4:
 - add 1st patch to protect the whole blk_mq_submit_bio path.
 - add sysfs_lock around the rq_qos_queue_depth_changed()
 - add qos_lock to protect the rqos list
 - add patches to clean up the blk-wbt
 - add new config to control blk-rq-qos's code

Changes since v3:
 - Fix some code sytle issues
 - Rename policies to blk-wbt, blk-iolat, blk-iocost, blk-ioprio

Changes since v2:
 Refactor the patches,
 - patch01 only provide new interfaces and not export the sysfs interface.
 - patch02 adapt the wbt to new interface, reserve the previous action
   when write to wbt_lat
 - export sysfs interface in last patch when all of the blk-rqos policies
   are preprared well.

Changes since v1:
 - Just make iocost and iolatency pluggable, then we needn't to export
   those interfaces
 - Remove the iostat rqos policy
 - Rename module of blk-ioprio to io-prio to avoid rename ioprio.c file

Wang Jianchao(16)
	blk: make the whole blk_mq_submit_bio under
    blk/rq-qos: prepare to make blk-rq-qos pluggable
    blk-wbt: make wbt pluggable
    blk-iolatency: make iolatency pluggable
    blk-iocost: make iocost pluggable
    blk-ioprio: make ioprio pluggable
    blk/rq-qos: get rid of unused interfaces of rqos
    blk/rq-qos: export the sysfs for switching qos
    blk-wbt: show wbt_lat_us sysfs interface only when wbt
    blk-wbt: get rid of wbt_enable/disable_default()
    blk/rq-qos: get rid of debugfs register in
    blk/rq-qos: change queue_depth_changed to
    blk-wbt: move cache setting to
    blk-wbt: cleanup the blk-wbt.h
    blk/rq-qos: move the rqos debugfs code to blk-rq-qos.c
    blk/rq-qos: add config to control the whole blk-rq-qos

 15 files changed, 488 insertions(+), 172 deletions(-)
 block/Kconfig              |   7 ++
 block/Makefile             |   3 +-
 block/bfq-iosched.c        |   2 -
 block/blk-cgroup.c         |  11 --
 block/blk-core.c           |   1 +
 block/blk-iocost.c         |  83 +++++++------
 block/blk-iolatency.c      |  44 ++++---
 block/blk-ioprio.c         |  53 +++++----
 block/blk-ioprio.h         |  19 ---
 block/blk-mq-debugfs.c     |  67 +----------
 block/blk-mq-debugfs.h     |  17 +--
 block/blk-mq-sched.c       |   1 -
 block/blk-mq.c             |  22 ++--
 block/blk-rq-qos.c         | 285 +++++++++++++++++++++++++++++++++++++++++++-
 block/blk-rq-qos.h         | 124 +++++++------------
 block/blk-settings.c       |   6 +-
 block/blk-sysfs.c          |  82 ++-----------
 block/blk-wbt.c            | 288 ++++++++++++++++++++++++++++-----------------
 block/blk-wbt.h            | 134 ---------------------
 block/blk.h                |  12 +-
 block/elevator.c           |   3 -
 include/linux/blkdev.h     |   7 +-
 include/trace/events/wbt.h |   3 +-
 23 files changed, 668 insertions(+), 606 deletions(-)

[RFC V5 01/16] blk: make the whole blk_mq_submit_bio under q_usage_counter

[Wang Jianchao (Kuaishou)]

This is to protect the rqos list against the rqos open/close. We
need to drain all of the caller of blk_mq_submit_bio() before
we can operate the rqos list.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-mq.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 1adfe4824ef5..3c1cd32c72fd 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2723,8 +2723,7 @@ static struct request *blk_mq_get_new_requests(struct request_queue *q,
 	};
 	struct request *rq;
 
-	if (unlikely(bio_queue_enter(bio)))
-		return NULL;
+	percpu_ref_get(&q->q_usage_counter);
 
 	if (plug) {
 		data.nr_tags = plug->nr_ios;
@@ -2789,15 +2788,18 @@ void blk_mq_submit_bio(struct bio *bio)
 	if (unlikely(!blk_crypto_bio_prep(&bio)))
 		return;
 
+	if (unlikely(bio_queue_enter(bio)))
+		return;
+
 	blk_queue_bounce(q, &bio);
 	if (blk_may_split(q, bio))
 		__blk_queue_split(q, &bio, &nr_segs);
 
 	if (!bio_integrity_prep(bio))
-		return;
+		goto exit;
 
 	if (blk_mq_attempt_bio_merge(q, bio, nr_segs))
-		return;
+		goto exit;
 
 	rq_qos_throttle(q, bio);
 
@@ -2805,7 +2807,7 @@ void blk_mq_submit_bio(struct bio *bio)
 	if (!rq) {
 		rq = blk_mq_get_new_requests(q, plug, bio);
 		if (unlikely(!rq))
-			return;
+			goto exit;
 	}
 
 	trace_block_getrq(bio);
@@ -2819,12 +2821,12 @@ void blk_mq_submit_bio(struct bio *bio)
 		bio->bi_status = ret;
 		bio_endio(bio);
 		blk_mq_free_request(rq);
-		return;
+		goto exit;
 	}
 
 	if (op_is_flush(bio->bi_opf)) {
 		blk_insert_flush(rq);
-		return;
+		goto exit;
 	}
 
 	if (plug)
@@ -2836,6 +2838,8 @@ void blk_mq_submit_bio(struct bio *bio)
 	else
 		blk_mq_run_dispatch_ops(rq->q,
 				blk_mq_try_issue_directly(rq->mq_hctx, rq));
+exit:
+	blk_queue_exit(q);
 }
 
 /**
-- 
2.17.1

[RFC V5 02/16] blk/rq-qos: prepare to make blk-rq-qos pluggable

[Wang Jianchao (Kuaishou)]

This patch makes blk-rq-qos policies pluggable as following,
(1) Add code to maintain the rq_qos_ops. A rq-qos policy need to
    register itself with rq_qos_register(). The original enum
    rq_qos_id will be removed in following patch. They will use
    a dynamic id maintained by rq_qos_ida.
(2) Add .init callback into rq_qos_ops. We use it to initialize the
    resource.
(3) Add /sys/block/x/queue/qos
    We can use '+name' or "-name" to open or close the blk-rq-qos
    policy.

This patch mainly prepare help interfaces and no functional changes.
Following patches will adpat the code of wbt, iolatency, iocost and
ioprio to make them pluggable one by one. And after that, the sysfs
interface /sys/block/xxx/queue/qos will be exported.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-core.c       |   1 +
 block/blk-mq-debugfs.c |  12 +-
 block/blk-rq-qos.c     | 246 ++++++++++++++++++++++++++++++++++++++++-
 block/blk-rq-qos.h     |  25 ++++-
 include/linux/blkdev.h |   1 +
 5 files changed, 280 insertions(+), 5 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index d93e3bb9a769..448d153e515b 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -485,6 +485,7 @@ struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu)
 	mutex_init(&q->sysfs_lock);
 	mutex_init(&q->sysfs_dir_lock);
 	spin_lock_init(&q->queue_lock);
+	spin_lock_init(&q->rqos_lock);
 
 	init_waitqueue_head(&q->mq_freeze_wq);
 	mutex_init(&q->mq_freeze_lock);
diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 3a790eb4995c..24d47bc90b4a 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -729,7 +729,10 @@ void blk_mq_debugfs_register(struct request_queue *q)
 
 	if (q->rq_qos) {
 		struct rq_qos *rqos = q->rq_qos;
-
+		/*
+		 * Queue has not been registered right now, it is safe to
+		 * iterate the rqos w/o lock
+		 */
 		while (rqos) {
 			blk_mq_debugfs_register_rqos(rqos);
 			rqos = rqos->next;
@@ -844,7 +847,12 @@ void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos)
 void blk_mq_debugfs_register_rqos(struct rq_qos *rqos)
 {
 	struct request_queue *q = rqos->q;
-	const char *dir_name = rq_qos_id_to_name(rqos->id);
+	const char *dir_name;
+
+	if (rqos->ops->name)
+		dir_name = rqos->ops->name;
+	else
+		dir_name = rq_qos_id_to_name(rqos->id);
 
 	if (rqos->debugfs_dir || !rqos->ops->debugfs_attrs)
 		return;
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index e83af7bc7591..c43436c2ead1 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -2,6 +2,9 @@
 
 #include "blk-rq-qos.h"
 
+static DEFINE_IDR(rq_qos_idr);
+static DEFINE_MUTEX(rq_qos_mutex);
+
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -294,11 +297,250 @@ void rq_qos_wait(struct rq_wait *rqw, void *private_data,
 
 void rq_qos_exit(struct request_queue *q)
 {
-	blk_mq_debugfs_unregister_queue_rqos(q);
-
+	/*
+	 * Queue must have been unregistered here, it is safe to
+	 * iterate the list w/o lock
+	 */
 	while (q->rq_qos) {
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
 	}
+	blk_mq_debugfs_unregister_queue_rqos(q);
+}
+
+static struct rq_qos *rq_qos_by_name(struct request_queue *q,
+		const char *name)
+{
+	struct rq_qos *rqos;
+
+	for (rqos = q->rq_qos; rqos; rqos = rqos->next) {
+		if (!rqos->ops->name)
+			continue;
+
+		if (!strncmp(rqos->ops->name, name,
+					strlen(rqos->ops->name)))
+			return rqos;
+	}
+	return NULL;
+}
+
+/*
+ * This helper interface is to keep rqos alive against
+ * rqos switching. rq_qos_deactivate() drains all of
+ * rq_qos_get() before it frees the rqos structure.
+ */
+struct rq_qos *rq_qos_get(struct request_queue *q, int id)
+{
+	struct rq_qos *rqos;
+
+	spin_lock_irq(&q->rqos_lock);
+	for (rqos = q->rq_qos; rqos; rqos = rqos->next) {
+		if (rqos->id == id) {
+			break;
+		}
+	}
+	if (rqos && rqos->dying)
+		rqos = NULL;
+	if (rqos)
+		refcount_inc(&rqos->ref);
+	spin_unlock_irq(&q->rqos_lock);
+	return rqos;
+}
+
+void rq_qos_put(struct rq_qos *rqos)
+{
+	struct request_queue *q = rqos->q;
+
+	spin_lock_irq(&q->rqos_lock);
+	refcount_dec(&rqos->ref);
+	if (rqos->dying)
+		wake_up(&rqos->waitq);
+	spin_unlock_irq(&q->rqos_lock);
+}
+
+void rq_qos_activate(struct request_queue *q,
+		struct rq_qos *rqos, const struct rq_qos_ops *ops)
+{
+	struct rq_qos *pos;
+
+	rqos->dying = false;
+	refcount_set(&rqos->ref, 1);
+	init_waitqueue_head(&rqos->waitq);
+	rqos->id = ops->id;
+	rqos->ops = ops;
+	rqos->q = q;
+	rqos->next = NULL;
+
+	spin_lock_irq(&q->rqos_lock);
+	pos = q->rq_qos;
+	if (pos) {
+		while (pos->next)
+			pos = pos->next;
+		pos->next = rqos;
+	} else {
+		q->rq_qos = rqos;
+	}
+	spin_unlock_irq(&q->rqos_lock);
+
+	if (rqos->ops->debugfs_attrs)
+		blk_mq_debugfs_register_rqos(rqos);
+}
+
+void rq_qos_deactivate(struct rq_qos *rqos)
+{
+	struct request_queue *q = rqos->q;
+	struct rq_qos **cur;
+
+	spin_lock_irq(&q->rqos_lock);
+	rqos->dying = true;
+	/*
+	 * Drain all of the usage of get/put_rqos()
+	 */
+	wait_event_lock_irq(rqos->waitq,
+		refcount_read(&rqos->ref) == 1, q->rqos_lock);
+	for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {
+		if (*cur == rqos) {
+			*cur = rqos->next;
+			break;
+		}
+	}
+	spin_unlock_irq(&q->rqos_lock);
+	blk_mq_debugfs_unregister_rqos(rqos);
+}
+
+static struct rq_qos_ops *rq_qos_op_find(const char *name)
+{
+	struct rq_qos_ops *pos;
+	int id;
+
+	idr_for_each_entry(&rq_qos_idr, pos, id) {
+		if (!strncmp(pos->name, name, strlen(pos->name)))
+			return pos;
+	}
+
+	return NULL;
+}
+
+int rq_qos_register(struct rq_qos_ops *ops)
+{
+	int ret, start;
+
+	mutex_lock(&rq_qos_mutex);
+
+	if (rq_qos_op_find(ops->name)) {
+		ret = -EEXIST;
+		goto out;
+	}
+
+	start = RQ_QOS_IOPRIO + 1;
+	ret = idr_alloc(&rq_qos_idr, ops, start, INT_MAX, GFP_KERNEL);
+	if (ret < 0)
+		goto out;
+
+	ops->id = ret;
+	ret = 0;
+out:
+	mutex_unlock(&rq_qos_mutex);
+	return ret;
+}
+
+void rq_qos_unregister(struct rq_qos_ops *ops)
+{
+	mutex_lock(&rq_qos_mutex);
+	idr_remove(&rq_qos_idr, ops->id);
+	mutex_unlock(&rq_qos_mutex);
+}
+
+ssize_t queue_qos_show(struct request_queue *q, char *buf)
+{
+	struct rq_qos_ops *ops;
+	struct rq_qos *rqos;
+	int id, ret = 0;
+
+	mutex_lock(&rq_qos_mutex);
+	/*
+	 * Show the policies in the order of being invoked.
+	 * rqos_lock is not needed here as the sysfs_lock is
+	 * protected us from the queue_qos_store()
+	 */
+	for (rqos = q->rq_qos; rqos; rqos = rqos->next) {
+		if (!rqos->ops->name)
+			continue;
+		ret += sprintf(buf + ret, "[%s] ", rqos->ops->name);
+	}
+	idr_for_each_entry(&rq_qos_idr, ops, id) {
+		if (!rq_qos_by_name(q, ops->name))
+			ret += sprintf(buf + ret, "%s ", ops->name);
+	}
+
+	ret--; /* overwrite the last space */
+	ret += sprintf(buf + ret, "\n");
+	mutex_unlock(&rq_qos_mutex);
+
+	return ret;
+}
+
+ssize_t queue_qos_store(struct request_queue *q, const char *page,
+			  size_t count)
+{
+	const struct rq_qos_ops *ops;
+	struct rq_qos *rqos;
+	const char *qosname;
+	char *buf;
+	bool add;
+	int ret;
+
+	if (!blk_queue_registered(q))
+		return -ENOENT;
+
+	buf = kstrdup(page, GFP_KERNEL);
+	if (!buf)
+		return -ENOMEM;
+
+	buf = strim(buf);
+	if (buf[0] != '+' && buf[0] != '-') {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	add = buf[0] == '+';
+	qosname = buf + 1;
+
+	rqos = rq_qos_by_name(q, qosname);
+	if ((buf[0] == '+' && rqos)) {
+		ret = -EEXIST;
+		goto out;
+	}
+
+	if ((buf[0] == '-' && !rqos)) {
+		ret = -ENODEV;
+		goto out;
+	}
+
+	if (add) {
+		mutex_lock(&rq_qos_mutex);
+		ops = rq_qos_op_find(qosname);
+		mutex_unlock(&rq_qos_mutex);
+		if (!ops) {
+			ret = -EINVAL;
+			goto out;
+		}
+	} else {
+		ops = rqos->ops;
+	}
+
+	blk_mq_freeze_queue(q);
+	blk_mq_quiesce_queue(q);
+	if (add) {
+		ret = ops->init(q);
+	} else {
+		ops->exit(rqos);
+		ret = 0;
+	}
+	blk_mq_unquiesce_queue(q);
+	blk_mq_unfreeze_queue(q);
+out:
+	kfree(buf);
+	return ret ? ret : count;
 }
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 3cfbc8668cba..aa6ef12637a0 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -26,9 +26,12 @@ struct rq_wait {
 };
 
 struct rq_qos {
-	struct rq_qos_ops *ops;
+	const struct rq_qos_ops *ops;
 	struct request_queue *q;
 	enum rq_qos_id id;
+	refcount_t ref;
+	wait_queue_head_t waitq;
+	bool dying;
 	struct rq_qos *next;
 #ifdef CONFIG_BLK_DEBUG_FS
 	struct dentry *debugfs_dir;
@@ -36,6 +39,8 @@ struct rq_qos {
 };
 
 struct rq_qos_ops {
+	const char *name;
+	int id;
 	void (*throttle)(struct rq_qos *, struct bio *);
 	void (*track)(struct rq_qos *, struct request *, struct bio *);
 	void (*merge)(struct rq_qos *, struct request *, struct bio *);
@@ -46,6 +51,7 @@ struct rq_qos_ops {
 	void (*cleanup)(struct rq_qos *, struct bio *);
 	void (*queue_depth_changed)(struct rq_qos *);
 	void (*exit)(struct rq_qos *);
+	int (*init)(struct request_queue *);
 	const struct blk_mq_debugfs_attr *debugfs_attrs;
 };
 
@@ -132,6 +138,17 @@ static inline void rq_qos_del(struct request_queue *q, struct rq_qos *rqos)
 	blk_mq_debugfs_unregister_rqos(rqos);
 }
 
+int rq_qos_register(struct rq_qos_ops *ops);
+void rq_qos_unregister(struct rq_qos_ops *ops);
+void rq_qos_activate(struct request_queue *q,
+		struct rq_qos *rqos, const struct rq_qos_ops *ops);
+void rq_qos_deactivate(struct rq_qos *rqos);
+ssize_t queue_qos_show(struct request_queue *q, char *buf);
+ssize_t queue_qos_store(struct request_queue *q, const char *page,
+			  size_t count);
+struct rq_qos *rq_qos_get(struct request_queue *q, int id);
+void rq_qos_put(struct rq_qos *rqos);
+
 typedef bool (acquire_inflight_cb_t)(struct rq_wait *rqw, void *private_data);
 typedef void (cleanup_cb_t)(struct rq_wait *rqw, void *private_data);
 
@@ -211,8 +228,14 @@ static inline void rq_qos_merge(struct request_queue *q, struct request *rq,
 
 static inline void rq_qos_queue_depth_changed(struct request_queue *q)
 {
+	/*
+	 * It is called by external module, protect the rqos list with
+	 * sysfs_lock against qos switching
+	 */
+	mutex_lock(&q->sysfs_lock);
 	if (q->rq_qos)
 		__rq_qos_queue_depth_changed(q->rq_qos);
+	mutex_unlock(&q->sysfs_lock);
 }
 
 void rq_qos_exit(struct request_queue *);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index f35aea98bc35..126aac824ccc 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -230,6 +230,7 @@ struct request_queue {
 	int			id;
 
 	spinlock_t		queue_lock;
+	spinlock_t		rqos_lock;
 
 	struct gendisk		*disk;
 
-- 
2.17.1

[RFC V5 03/16] blk-wbt: make wbt pluggable

[Wang Jianchao (Kuaishou)]

This patch makes wbt pluggable through /sys/block/xxx/queue/qos.
Some queue_lock/unlock is added to protect rq_qos_by_id() in
wbt_rq_qos(). By default, wbt is enabled which is same with previous
code.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-mq-debugfs.c |  2 --
 block/blk-rq-qos.h     |  6 -----
 block/blk-sysfs.c      |  6 ++---
 block/blk-wbt.c        | 56 +++++++++++++++++++++++++++++++++++++-----
 block/blk-wbt.h        |  6 +++++
 5 files changed, 58 insertions(+), 18 deletions(-)

diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 24d47bc90b4a..073ca7fca9b3 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -826,8 +826,6 @@ void blk_mq_debugfs_unregister_sched(struct request_queue *q)
 static const char *rq_qos_id_to_name(enum rq_qos_id id)
 {
 	switch (id) {
-	case RQ_QOS_WBT:
-		return "wbt";
 	case RQ_QOS_LATENCY:
 		return "latency";
 	case RQ_QOS_COST:
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index aa6ef12637a0..8ce3f48b16b9 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -14,7 +14,6 @@
 struct blk_mq_debugfs_attr;
 
 enum rq_qos_id {
-	RQ_QOS_WBT,
 	RQ_QOS_LATENCY,
 	RQ_QOS_COST,
 	RQ_QOS_IOPRIO,
@@ -76,11 +75,6 @@ static inline struct rq_qos *rq_qos_id(struct request_queue *q,
 	return rqos;
 }
 
-static inline struct rq_qos *wbt_rq_qos(struct request_queue *q)
-{
-	return rq_qos_id(q, RQ_QOS_WBT);
-}
-
 static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q)
 {
 	return rq_qos_id(q, RQ_QOS_LATENCY);
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 9f32882ceb2f..c713d6b305cc 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -463,7 +463,7 @@ static ssize_t queue_io_timeout_store(struct request_queue *q, const char *page,
 
 static ssize_t queue_wb_lat_show(struct request_queue *q, char *page)
 {
-	if (!wbt_rq_qos(q))
+	if (!wbt_opened(q))
 		return -EINVAL;
 
 	return sprintf(page, "%llu\n", div_u64(wbt_get_min_lat(q), 1000));
@@ -472,7 +472,6 @@ static ssize_t queue_wb_lat_show(struct request_queue *q, char *page)
 static ssize_t queue_wb_lat_store(struct request_queue *q, const char *page,
 				  size_t count)
 {
-	struct rq_qos *rqos;
 	ssize_t ret;
 	s64 val;
 
@@ -482,8 +481,7 @@ static ssize_t queue_wb_lat_store(struct request_queue *q, const char *page,
 	if (val < -1)
 		return -EINVAL;
 
-	rqos = wbt_rq_qos(q);
-	if (!rqos) {
+	if (!wbt_opened(q)) {
 		ret = wbt_init(q);
 		if (ret)
 			return ret;
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index 0c119be0e813..44a920d83475 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -31,6 +31,22 @@
 #define CREATE_TRACE_POINTS
 #include <trace/events/wbt.h>
 
+static struct rq_qos_ops wbt_rqos_ops;
+
+static inline struct rq_qos *wbt_rq_qos(struct request_queue *q)
+{
+	return rq_qos_get(q, wbt_rqos_ops.id);
+}
+
+bool wbt_opened(struct request_queue *q)
+{
+	struct rq_qos *rqos = wbt_rq_qos(q);
+
+	rq_qos_put(rqos);
+
+	return !!rqos;
+}
+
 static inline void wbt_clear_state(struct request *rq)
 {
 	rq->wbt_flags = 0;
@@ -425,19 +441,27 @@ static void wbt_update_limits(struct rq_wb *rwb)
 u64 wbt_get_min_lat(struct request_queue *q)
 {
 	struct rq_qos *rqos = wbt_rq_qos(q);
+	u64 nsec;
+
 	if (!rqos)
 		return 0;
-	return RQWB(rqos)->min_lat_nsec;
+
+	nsec = RQWB(rqos)->min_lat_nsec;
+	rq_qos_put(rqos);
+
+	return  nsec;
 }
 
 void wbt_set_min_lat(struct request_queue *q, u64 val)
 {
 	struct rq_qos *rqos = wbt_rq_qos(q);
+
 	if (!rqos)
 		return;
 	RQWB(rqos)->min_lat_nsec = val;
 	RQWB(rqos)->enable_state = WBT_STATE_ON_MANUAL;
 	wbt_update_limits(RQWB(rqos));
+	rq_qos_put(rqos);
 }
 
 
@@ -629,8 +653,11 @@ static void wbt_requeue(struct rq_qos *rqos, struct request *rq)
 void wbt_set_write_cache(struct request_queue *q, bool write_cache_on)
 {
 	struct rq_qos *rqos = wbt_rq_qos(q);
-	if (rqos)
+
+	if (rqos) {
 		RQWB(rqos)->wc = write_cache_on;
+		rq_qos_put(rqos);
+	}
 }
 
 /*
@@ -644,6 +671,8 @@ void wbt_enable_default(struct request_queue *q)
 	if (rqos) {
 		if (RQWB(rqos)->enable_state == WBT_STATE_OFF_DEFAULT)
 			RQWB(rqos)->enable_state = WBT_STATE_ON_DEFAULT;
+
+		rq_qos_put(rqos);
 		return;
 	}
 
@@ -692,6 +721,7 @@ static void wbt_exit(struct rq_qos *rqos)
 	struct rq_wb *rwb = RQWB(rqos);
 	struct request_queue *q = rqos->q;
 
+	rq_qos_deactivate(rqos);
 	blk_stat_remove_callback(q, rwb->cb);
 	blk_stat_free_callback(rwb->cb);
 	kfree(rwb);
@@ -704,13 +734,16 @@ void wbt_disable_default(struct request_queue *q)
 {
 	struct rq_qos *rqos = wbt_rq_qos(q);
 	struct rq_wb *rwb;
+
 	if (!rqos)
 		return;
+
 	rwb = RQWB(rqos);
 	if (rwb->enable_state == WBT_STATE_ON_DEFAULT) {
 		blk_stat_deactivate(rwb->cb);
 		rwb->enable_state = WBT_STATE_OFF_DEFAULT;
 	}
+	rq_qos_put(rqos);
 }
 EXPORT_SYMBOL_GPL(wbt_disable_default);
 
@@ -803,6 +836,7 @@ static const struct blk_mq_debugfs_attr wbt_debugfs_attrs[] = {
 #endif
 
 static struct rq_qos_ops wbt_rqos_ops = {
+	.name = "wbt",
 	.throttle = wbt_wait,
 	.issue = wbt_issue,
 	.track = wbt_track,
@@ -811,6 +845,7 @@ static struct rq_qos_ops wbt_rqos_ops = {
 	.cleanup = wbt_cleanup,
 	.queue_depth_changed = wbt_queue_depth_changed,
 	.exit = wbt_exit,
+	.init = wbt_init,
 #ifdef CONFIG_BLK_DEBUG_FS
 	.debugfs_attrs = wbt_debugfs_attrs,
 #endif
@@ -834,9 +869,6 @@ int wbt_init(struct request_queue *q)
 	for (i = 0; i < WBT_NUM_RWQ; i++)
 		rq_wait_init(&rwb->rq_wait[i]);
 
-	rwb->rqos.id = RQ_QOS_WBT;
-	rwb->rqos.ops = &wbt_rqos_ops;
-	rwb->rqos.q = q;
 	rwb->last_comp = rwb->last_issue = jiffies;
 	rwb->win_nsec = RWB_WINDOW_NSEC;
 	rwb->enable_state = WBT_STATE_ON_DEFAULT;
@@ -846,7 +878,7 @@ int wbt_init(struct request_queue *q)
 	/*
 	 * Assign rwb and add the stats callback.
 	 */
-	rq_qos_add(q, &rwb->rqos);
+	rq_qos_activate(q, &rwb->rqos, &wbt_rqos_ops);
 	blk_stat_add_callback(q, rwb->cb);
 
 	rwb->min_lat_nsec = wbt_default_latency_nsec(q);
@@ -856,3 +888,15 @@ int wbt_init(struct request_queue *q)
 
 	return 0;
 }
+
+static __init int wbt_mod_init(void)
+{
+	return rq_qos_register(&wbt_rqos_ops);
+}
+
+static __exit void wbt_mod_exit(void)
+{
+	return rq_qos_unregister(&wbt_rqos_ops);
+}
+module_init(wbt_mod_init);
+module_exit(wbt_mod_exit);
diff --git a/block/blk-wbt.h b/block/blk-wbt.h
index 2eb01becde8c..f128dcf662db 100644
--- a/block/blk-wbt.h
+++ b/block/blk-wbt.h
@@ -88,6 +88,7 @@ static inline unsigned int wbt_inflight(struct rq_wb *rwb)
 
 #ifdef CONFIG_BLK_WBT
 
+bool wbt_opened(struct request_queue *q);
 int wbt_init(struct request_queue *);
 void wbt_disable_default(struct request_queue *);
 void wbt_enable_default(struct request_queue *);
@@ -101,6 +102,11 @@ u64 wbt_default_latency_nsec(struct request_queue *);
 
 #else
 
+bool wbt_opened(struct request_queue *q)
+{
+	return false;
+}
+
 static inline void wbt_track(struct request *rq, enum wbt_flags flags)
 {
 }
-- 
2.17.1

[RFC V5 04/16] blk-iolatency: make iolatency pluggable

[Wang Jianchao (Kuaishou)]

Use new blk-rq-qos API to make blk-iolatency pluggable.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-cgroup.c     |  6 ------
 block/blk-iolatency.c  | 44 +++++++++++++++++++++++++++---------------
 block/blk-mq-debugfs.c |  2 --
 block/blk-rq-qos.h     |  6 ------
 block/blk.h            |  6 ------
 5 files changed, 28 insertions(+), 36 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 650f7e27989f..3ae2aa557aef 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1203,12 +1203,6 @@ int blkcg_init_queue(struct request_queue *q)
 	if (ret)
 		goto err_destroy_all;
 
-	ret = blk_iolatency_init(q);
-	if (ret) {
-		blk_throtl_exit(q);
-		goto err_destroy_all;
-	}
-
 	return 0;
 
 err_destroy_all:
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index 6593c7123b97..1459b606e995 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -90,6 +90,8 @@ struct blk_iolatency {
 	atomic_t enabled;
 };
 
+static struct rq_qos_ops blkcg_iolatency_ops;
+
 static inline struct blk_iolatency *BLKIOLATENCY(struct rq_qos *rqos)
 {
 	return container_of(rqos, struct blk_iolatency, rqos);
@@ -646,15 +648,10 @@ static void blkcg_iolatency_exit(struct rq_qos *rqos)
 
 	del_timer_sync(&blkiolat->timer);
 	blkcg_deactivate_policy(rqos->q, &blkcg_policy_iolatency);
+	rq_qos_deactivate(rqos);
 	kfree(blkiolat);
 }
 
-static struct rq_qos_ops blkcg_iolatency_ops = {
-	.throttle = blkcg_iolatency_throttle,
-	.done_bio = blkcg_iolatency_done_bio,
-	.exit = blkcg_iolatency_exit,
-};
-
 static void blkiolatency_timer_fn(struct timer_list *t)
 {
 	struct blk_iolatency *blkiolat = from_timer(blkiolat, t, timer);
@@ -716,7 +713,7 @@ static void blkiolatency_timer_fn(struct timer_list *t)
 	rcu_read_unlock();
 }
 
-int blk_iolatency_init(struct request_queue *q)
+static int blk_iolatency_init(struct request_queue *q)
 {
 	struct blk_iolatency *blkiolat;
 	struct rq_qos *rqos;
@@ -727,15 +724,10 @@ int blk_iolatency_init(struct request_queue *q)
 		return -ENOMEM;
 
 	rqos = &blkiolat->rqos;
-	rqos->id = RQ_QOS_LATENCY;
-	rqos->ops = &blkcg_iolatency_ops;
-	rqos->q = q;
-
-	rq_qos_add(q, rqos);
-
+	rq_qos_activate(q, rqos, &blkcg_iolatency_ops);
 	ret = blkcg_activate_policy(q, &blkcg_policy_iolatency);
 	if (ret) {
-		rq_qos_del(q, rqos);
+		rq_qos_deactivate(rqos);
 		kfree(blkiolat);
 		return ret;
 	}
@@ -745,6 +737,14 @@ int blk_iolatency_init(struct request_queue *q)
 	return 0;
 }
 
+static struct rq_qos_ops blkcg_iolatency_ops = {
+	.name = "iolat",
+	.throttle = blkcg_iolatency_throttle,
+	.done_bio = blkcg_iolatency_done_bio,
+	.exit = blkcg_iolatency_exit,
+	.init = blk_iolatency_init,
+};
+
 /*
  * return 1 for enabling iolatency, return -1 for disabling iolatency, otherwise
  * return 0.
@@ -962,7 +962,7 @@ static void iolatency_pd_init(struct blkg_policy_data *pd)
 {
 	struct iolatency_grp *iolat = pd_to_lat(pd);
 	struct blkcg_gq *blkg = lat_to_blkg(iolat);
-	struct rq_qos *rqos = blkcg_rq_qos(blkg->q);
+	struct rq_qos *rqos = rq_qos_get(blkg->q, blkcg_iolatency_ops.id);
 	struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos);
 	u64 now = ktime_to_ns(ktime_get());
 	int cpu;
@@ -1016,6 +1016,7 @@ static void iolatency_pd_offline(struct blkg_policy_data *pd)
 	if (ret == -1)
 		atomic_dec(&blkiolat->enabled);
 	iolatency_clear_scaling(blkg);
+	rq_qos_put(&blkiolat->rqos);
 }
 
 static void iolatency_pd_free(struct blkg_policy_data *pd)
@@ -1046,12 +1047,23 @@ static struct blkcg_policy blkcg_policy_iolatency = {
 
 static int __init iolatency_init(void)
 {
-	return blkcg_policy_register(&blkcg_policy_iolatency);
+	int ret;
+
+	ret = rq_qos_register(&blkcg_iolatency_ops);
+	if (ret)
+		return ret;
+
+	ret = blkcg_policy_register(&blkcg_policy_iolatency);
+	if (ret)
+		rq_qos_unregister(&blkcg_iolatency_ops);
+
+	return ret;
 }
 
 static void __exit iolatency_exit(void)
 {
 	blkcg_policy_unregister(&blkcg_policy_iolatency);
+	rq_qos_unregister(&blkcg_iolatency_ops);
 }
 
 module_init(iolatency_init);
diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 073ca7fca9b3..102bc5814cda 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -826,8 +826,6 @@ void blk_mq_debugfs_unregister_sched(struct request_queue *q)
 static const char *rq_qos_id_to_name(enum rq_qos_id id)
 {
 	switch (id) {
-	case RQ_QOS_LATENCY:
-		return "latency";
 	case RQ_QOS_COST:
 		return "cost";
 	case RQ_QOS_IOPRIO:
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 8ce3f48b16b9..ecd1d1692a7d 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -14,7 +14,6 @@
 struct blk_mq_debugfs_attr;
 
 enum rq_qos_id {
-	RQ_QOS_LATENCY,
 	RQ_QOS_COST,
 	RQ_QOS_IOPRIO,
 };
@@ -75,11 +74,6 @@ static inline struct rq_qos *rq_qos_id(struct request_queue *q,
 	return rqos;
 }
 
-static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q)
-{
-	return rq_qos_id(q, RQ_QOS_LATENCY);
-}
-
 static inline void rq_wait_init(struct rq_wait *rq_wait)
 {
 	atomic_set(&rq_wait->inflight, 0);
diff --git a/block/blk.h b/block/blk.h
index 8bd43b3ad33d..1a314257b6a3 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -400,12 +400,6 @@ static inline void blk_queue_bounce(struct request_queue *q, struct bio **bio)
 		__blk_queue_bounce(q, bio);	
 }
 
-#ifdef CONFIG_BLK_CGROUP_IOLATENCY
-extern int blk_iolatency_init(struct request_queue *q);
-#else
-static inline int blk_iolatency_init(struct request_queue *q) { return 0; }
-#endif
-
 struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp);
 
 #ifdef CONFIG_BLK_DEV_ZONED
-- 
2.17.1

[RFC V5 05/16] blk-iocost: make iocost pluggable

[Wang Jianchao (Kuaishou)]

Use new blk-rq-qos API to make blk-iocost pluggable.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-iocost.c     | 80 ++++++++++++++++++++++++------------------
 block/blk-mq-debugfs.c |  2 --
 block/blk-rq-qos.h     |  1 -
 3 files changed, 45 insertions(+), 38 deletions(-)

diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 769b64394298..886c03383444 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -660,10 +660,7 @@ static struct ioc *rqos_to_ioc(struct rq_qos *rqos)
 	return container_of(rqos, struct ioc, rqos);
 }
 
-static struct ioc *q_to_ioc(struct request_queue *q)
-{
-	return rqos_to_ioc(rq_qos_id(q, RQ_QOS_COST));
-}
+static struct rq_qos_ops ioc_rqos_ops;
 
 static const char *q_name(struct request_queue *q)
 {
@@ -2810,6 +2807,7 @@ static void ioc_rqos_exit(struct rq_qos *rqos)
 	struct ioc *ioc = rqos_to_ioc(rqos);
 
 	blkcg_deactivate_policy(rqos->q, &blkcg_policy_iocost);
+	rq_qos_deactivate(rqos);
 
 	spin_lock_irq(&ioc->lock);
 	ioc->running = IOC_STOP;
@@ -2820,15 +2818,6 @@ static void ioc_rqos_exit(struct rq_qos *rqos)
 	kfree(ioc);
 }
 
-static struct rq_qos_ops ioc_rqos_ops = {
-	.throttle = ioc_rqos_throttle,
-	.merge = ioc_rqos_merge,
-	.done_bio = ioc_rqos_done_bio,
-	.done = ioc_rqos_done,
-	.queue_depth_changed = ioc_rqos_queue_depth_changed,
-	.exit = ioc_rqos_exit,
-};
-
 static int blk_iocost_init(struct request_queue *q)
 {
 	struct ioc *ioc;
@@ -2856,10 +2845,7 @@ static int blk_iocost_init(struct request_queue *q)
 	}
 
 	rqos = &ioc->rqos;
-	rqos->id = RQ_QOS_COST;
-	rqos->ops = &ioc_rqos_ops;
-	rqos->q = q;
-
+	rq_qos_activate(q, rqos, &ioc_rqos_ops);
 	spin_lock_init(&ioc->lock);
 	timer_setup(&ioc->timer, ioc_timer_fn, 0);
 	INIT_LIST_HEAD(&ioc->active_iocgs);
@@ -2883,10 +2869,9 @@ static int blk_iocost_init(struct request_queue *q)
 	 * called before policy activation completion, can't assume that the
 	 * target bio has an iocg associated and need to test for NULL iocg.
 	 */
-	rq_qos_add(q, rqos);
 	ret = blkcg_activate_policy(q, &blkcg_policy_iocost);
 	if (ret) {
-		rq_qos_del(q, rqos);
+		rq_qos_deactivate(rqos);
 		free_percpu(ioc->pcpu_stat);
 		kfree(ioc);
 		return ret;
@@ -2894,6 +2879,17 @@ static int blk_iocost_init(struct request_queue *q)
 	return 0;
 }
 
+static struct rq_qos_ops ioc_rqos_ops = {
+	.name = "iocost",
+	.throttle = ioc_rqos_throttle,
+	.merge = ioc_rqos_merge,
+	.done_bio = ioc_rqos_done_bio,
+	.done = ioc_rqos_done,
+	.queue_depth_changed = ioc_rqos_queue_depth_changed,
+	.exit = ioc_rqos_exit,
+	.init = blk_iocost_init,
+};
+
 static struct blkcg_policy_data *ioc_cpd_alloc(gfp_t gfp)
 {
 	struct ioc_cgrp *iocc;
@@ -2934,7 +2930,7 @@ static void ioc_pd_init(struct blkg_policy_data *pd)
 {
 	struct ioc_gq *iocg = pd_to_iocg(pd);
 	struct blkcg_gq *blkg = pd_to_blkg(&iocg->pd);
-	struct ioc *ioc = q_to_ioc(blkg->q);
+	struct ioc *ioc = rqos_to_ioc(rq_qos_get(blkg->q, ioc_rqos_ops.id));
 	struct ioc_now now;
 	struct blkcg_gq *tblkg;
 	unsigned long flags;
@@ -2990,6 +2986,7 @@ static void ioc_pd_free(struct blkg_policy_data *pd)
 		spin_unlock_irqrestore(&ioc->lock, flags);
 
 		hrtimer_cancel(&iocg->waitq_timer);
+		rq_qos_put(&ioc->rqos);
 	}
 	free_percpu(iocg->pcpu_stat);
 	kfree(iocg);
@@ -3162,6 +3159,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 			     size_t nbytes, loff_t off)
 {
 	struct block_device *bdev;
+	struct rq_qos *rqos;
 	struct ioc *ioc;
 	u32 qos[NR_QOS_PARAMS];
 	bool enable, user;
@@ -3172,14 +3170,12 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 	if (IS_ERR(bdev))
 		return PTR_ERR(bdev);
 
-	ioc = q_to_ioc(bdev_get_queue(bdev));
-	if (!ioc) {
-		ret = blk_iocost_init(bdev_get_queue(bdev));
-		if (ret)
-			goto err;
-		ioc = q_to_ioc(bdev_get_queue(bdev));
+	rqos = rq_qos_get(bdev_get_queue(bdev), ioc_rqos_ops.id);
+	if (!rqos) {
+		ret = -EOPNOTSUPP;
+		goto err;
 	}
-
+	ioc = rqos_to_ioc(rqos);
 	spin_lock_irq(&ioc->lock);
 	memcpy(qos, ioc->params.qos, sizeof(qos));
 	enable = ioc->enabled;
@@ -3272,10 +3268,12 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 	ioc_refresh_params(ioc, true);
 	spin_unlock_irq(&ioc->lock);
 
+	rq_qos_put(rqos);
 	blkdev_put_no_open(bdev);
 	return nbytes;
 einval:
 	ret = -EINVAL;
+	rq_qos_put(rqos);
 err:
 	blkdev_put_no_open(bdev);
 	return ret;
@@ -3329,6 +3327,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 				    size_t nbytes, loff_t off)
 {
 	struct block_device *bdev;
+	struct rq_qos *rqos;
 	struct ioc *ioc;
 	u64 u[NR_I_LCOEFS];
 	bool user;
@@ -3339,14 +3338,12 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 	if (IS_ERR(bdev))
 		return PTR_ERR(bdev);
 
-	ioc = q_to_ioc(bdev_get_queue(bdev));
-	if (!ioc) {
-		ret = blk_iocost_init(bdev_get_queue(bdev));
-		if (ret)
-			goto err;
-		ioc = q_to_ioc(bdev_get_queue(bdev));
+	rqos = rq_qos_get(bdev_get_queue(bdev), ioc_rqos_ops.id);
+	if (!rqos) {
+		ret = -EOPNOTSUPP;
+		goto err;
 	}
-
+	ioc = rqos_to_ioc(rqos);
 	spin_lock_irq(&ioc->lock);
 	memcpy(u, ioc->params.i_lcoefs, sizeof(u));
 	user = ioc->user_cost_model;
@@ -3397,11 +3394,13 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
 	ioc_refresh_params(ioc, true);
 	spin_unlock_irq(&ioc->lock);
 
+	rq_qos_put(rqos);
 	blkdev_put_no_open(bdev);
 	return nbytes;
 
 einval:
 	ret = -EINVAL;
+	rq_qos_put(rqos);
 err:
 	blkdev_put_no_open(bdev);
 	return ret;
@@ -3441,12 +3440,23 @@ static struct blkcg_policy blkcg_policy_iocost = {
 
 static int __init ioc_init(void)
 {
-	return blkcg_policy_register(&blkcg_policy_iocost);
+	int ret;
+
+	ret = rq_qos_register(&ioc_rqos_ops);
+	if (ret)
+		return ret;
+
+	ret = blkcg_policy_register(&blkcg_policy_iocost);
+	if (ret)
+		rq_qos_unregister(&ioc_rqos_ops);
+
+	return ret;
 }
 
 static void __exit ioc_exit(void)
 {
 	blkcg_policy_unregister(&blkcg_policy_iocost);
+	rq_qos_unregister(&ioc_rqos_ops);
 }
 
 module_init(ioc_init);
diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 102bc5814cda..27ca6185ad59 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -826,8 +826,6 @@ void blk_mq_debugfs_unregister_sched(struct request_queue *q)
 static const char *rq_qos_id_to_name(enum rq_qos_id id)
 {
 	switch (id) {
-	case RQ_QOS_COST:
-		return "cost";
 	case RQ_QOS_IOPRIO:
 		return "ioprio";
 	}
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index ecd1d1692a7d..ab95a366710d 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -14,7 +14,6 @@
 struct blk_mq_debugfs_attr;
 
 enum rq_qos_id {
-	RQ_QOS_COST,
 	RQ_QOS_IOPRIO,
 };
 
-- 
2.17.1

[RFC V5 06/16] blk-ioprio: make ioprio pluggable

[Wang Jianchao (Kuaishou)]

Use new blk-rq-qos API to make blk-ioprio pluggable.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-cgroup.c     |  5 ----
 block/blk-ioprio.c     | 53 ++++++++++++++++++++++++------------------
 block/blk-ioprio.h     | 19 ---------------
 block/blk-mq-debugfs.c |  4 ----
 block/blk-rq-qos.c     |  2 +-
 block/blk-rq-qos.h     |  2 +-
 6 files changed, 33 insertions(+), 52 deletions(-)
 delete mode 100644 block/blk-ioprio.h

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 3ae2aa557aef..f617f7ba311d 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -32,7 +32,6 @@
 #include <linux/psi.h>
 #include <linux/part_stat.h>
 #include "blk.h"
-#include "blk-ioprio.h"
 #include "blk-throttle.h"
 
 /*
@@ -1195,10 +1194,6 @@ int blkcg_init_queue(struct request_queue *q)
 	if (preloaded)
 		radix_tree_preload_end();
 
-	ret = blk_ioprio_init(q);
-	if (ret)
-		goto err_destroy_all;
-
 	ret = blk_throtl_init(q);
 	if (ret)
 		goto err_destroy_all;
diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c
index 2e7f10e1c03f..947aae54d295 100644
--- a/block/blk-ioprio.c
+++ b/block/blk-ioprio.c
@@ -17,7 +17,6 @@
 #include <linux/blk_types.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
-#include "blk-ioprio.h"
 #include "blk-rq-qos.h"
 
 /**
@@ -45,6 +44,7 @@ static const char *policy_name[] = {
 };
 
 static struct blkcg_policy ioprio_policy;
+static struct rq_qos_ops blkcg_ioprio_ops;
 
 /**
  * struct ioprio_blkg - Per (cgroup, request queue) data.
@@ -216,15 +216,11 @@ static void blkcg_ioprio_exit(struct rq_qos *rqos)
 		container_of(rqos, typeof(*blkioprio_blkg), rqos);
 
 	blkcg_deactivate_policy(rqos->q, &ioprio_policy);
+	rq_qos_deactivate(rqos);
 	kfree(blkioprio_blkg);
 }
 
-static struct rq_qos_ops blkcg_ioprio_ops = {
-	.track	= blkcg_ioprio_track,
-	.exit	= blkcg_ioprio_exit,
-};
-
-int blk_ioprio_init(struct request_queue *q)
+static int blk_ioprio_init(struct request_queue *q)
 {
 	struct blk_ioprio *blkioprio_blkg;
 	struct rq_qos *rqos;
@@ -234,35 +230,48 @@ int blk_ioprio_init(struct request_queue *q)
 	if (!blkioprio_blkg)
 		return -ENOMEM;
 
+	/*
+	 * No need to worry ioprio_blkcg_from_css return NULL as
+	 * the queue is frozen right now.
+	 */
+	rqos = &blkioprio_blkg->rqos;
+	rq_qos_activate(q, rqos, &blkcg_ioprio_ops);
+
 	ret = blkcg_activate_policy(q, &ioprio_policy);
 	if (ret) {
+		rq_qos_deactivate(rqos);
 		kfree(blkioprio_blkg);
-		return ret;
 	}
 
-	rqos = &blkioprio_blkg->rqos;
-	rqos->id = RQ_QOS_IOPRIO;
-	rqos->ops = &blkcg_ioprio_ops;
-	rqos->q = q;
-
-	/*
-	 * Registering the rq-qos policy after activating the blk-cgroup
-	 * policy guarantees that ioprio_blkcg_from_bio(bio) != NULL in the
-	 * rq-qos callbacks.
-	 */
-	rq_qos_add(q, rqos);
-
-	return 0;
+	return ret;
 }
 
+static struct rq_qos_ops blkcg_ioprio_ops = {
+	.name	= "ioprio",
+	.track	= blkcg_ioprio_track,
+	.exit	= blkcg_ioprio_exit,
+	.init	= blk_ioprio_init,
+};
+
 static int __init ioprio_init(void)
 {
-	return blkcg_policy_register(&ioprio_policy);
+	int ret;
+
+	ret = rq_qos_register(&blkcg_ioprio_ops);
+	if (ret)
+		return ret;
+
+	ret = blkcg_policy_register(&ioprio_policy);
+	if (ret)
+		rq_qos_unregister(&blkcg_ioprio_ops);
+
+	return ret;
 }
 
 static void __exit ioprio_exit(void)
 {
 	blkcg_policy_unregister(&ioprio_policy);
+	rq_qos_unregister(&blkcg_ioprio_ops);
 }
 
 module_init(ioprio_init);
diff --git a/block/blk-ioprio.h b/block/blk-ioprio.h
deleted file mode 100644
index a7785c2f1aea..000000000000
--- a/block/blk-ioprio.h
+++ /dev/null
@@ -1,19 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-
-#ifndef _BLK_IOPRIO_H_
-#define _BLK_IOPRIO_H_
-
-#include <linux/kconfig.h>
-
-struct request_queue;
-
-#ifdef CONFIG_BLK_CGROUP_IOPRIO
-int blk_ioprio_init(struct request_queue *q);
-#else
-static inline int blk_ioprio_init(struct request_queue *q)
-{
-	return 0;
-}
-#endif
-
-#endif /* _BLK_IOPRIO_H_ */
diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 27ca6185ad59..a35f2d6dd422 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -825,10 +825,6 @@ void blk_mq_debugfs_unregister_sched(struct request_queue *q)
 
 static const char *rq_qos_id_to_name(enum rq_qos_id id)
 {
-	switch (id) {
-	case RQ_QOS_IOPRIO:
-		return "ioprio";
-	}
 	return "unknown";
 }
 
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index c43436c2ead1..03f2e654cd17 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -433,7 +433,7 @@ int rq_qos_register(struct rq_qos_ops *ops)
 		goto out;
 	}
 
-	start = RQ_QOS_IOPRIO + 1;
+	start = 1;
 	ret = idr_alloc(&rq_qos_idr, ops, start, INT_MAX, GFP_KERNEL);
 	if (ret < 0)
 		goto out;
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index ab95a366710d..5a26eb128456 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -14,7 +14,7 @@
 struct blk_mq_debugfs_attr;
 
 enum rq_qos_id {
-	RQ_QOS_IOPRIO,
+	RQ_QOS_UNUSED,
 };
 
 struct rq_wait {
-- 
2.17.1

[RFC V5 07/16] blk/rq-qos: get rid of unused interfaces of rqos

[Wang Jianchao (Kuaishou)]

No functional changes here

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-mq-debugfs.c | 13 +--------
 block/blk-rq-qos.h     | 63 +-----------------------------------------
 2 files changed, 2 insertions(+), 74 deletions(-)

diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index a35f2d6dd422..7e3c3cc1b1a9 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -823,11 +823,6 @@ void blk_mq_debugfs_unregister_sched(struct request_queue *q)
 	q->sched_debugfs_dir = NULL;
 }
 
-static const char *rq_qos_id_to_name(enum rq_qos_id id)
-{
-	return "unknown";
-}
-
 void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos)
 {
 	debugfs_remove_recursive(rqos->debugfs_dir);
@@ -837,12 +832,6 @@ void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos)
 void blk_mq_debugfs_register_rqos(struct rq_qos *rqos)
 {
 	struct request_queue *q = rqos->q;
-	const char *dir_name;
-
-	if (rqos->ops->name)
-		dir_name = rqos->ops->name;
-	else
-		dir_name = rq_qos_id_to_name(rqos->id);
 
 	if (rqos->debugfs_dir || !rqos->ops->debugfs_attrs)
 		return;
@@ -851,7 +840,7 @@ void blk_mq_debugfs_register_rqos(struct rq_qos *rqos)
 		q->rqos_debugfs_dir = debugfs_create_dir("rqos",
 							 q->debugfs_dir);
 
-	rqos->debugfs_dir = debugfs_create_dir(dir_name,
+	rqos->debugfs_dir = debugfs_create_dir(rqos->ops->name,
 					       rqos->q->rqos_debugfs_dir);
 
 	debugfs_create_files(rqos->debugfs_dir, rqos, rqos->ops->debugfs_attrs);
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 5a26eb128456..58aba16c7d32 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -13,10 +13,6 @@
 
 struct blk_mq_debugfs_attr;
 
-enum rq_qos_id {
-	RQ_QOS_UNUSED,
-};
-
 struct rq_wait {
 	wait_queue_head_t wait;
 	atomic_t inflight;
@@ -25,7 +21,7 @@ struct rq_wait {
 struct rq_qos {
 	const struct rq_qos_ops *ops;
 	struct request_queue *q;
-	enum rq_qos_id id;
+	int id;
 	refcount_t ref;
 	wait_queue_head_t waitq;
 	bool dying;
@@ -62,69 +58,12 @@ struct rq_depth {
 	unsigned int default_depth;
 };
 
-static inline struct rq_qos *rq_qos_id(struct request_queue *q,
-				       enum rq_qos_id id)
-{
-	struct rq_qos *rqos;
-	for (rqos = q->rq_qos; rqos; rqos = rqos->next) {
-		if (rqos->id == id)
-			break;
-	}
-	return rqos;
-}
-
 static inline void rq_wait_init(struct rq_wait *rq_wait)
 {
 	atomic_set(&rq_wait->inflight, 0);
 	init_waitqueue_head(&rq_wait->wait);
 }
 
-static inline void rq_qos_add(struct request_queue *q, struct rq_qos *rqos)
-{
-	/*
-	 * No IO can be in-flight when adding rqos, so freeze queue, which
-	 * is fine since we only support rq_qos for blk-mq queue.
-	 *
-	 * Reuse ->queue_lock for protecting against other concurrent
-	 * rq_qos adding/deleting
-	 */
-	blk_mq_freeze_queue(q);
-
-	spin_lock_irq(&q->queue_lock);
-	rqos->next = q->rq_qos;
-	q->rq_qos = rqos;
-	spin_unlock_irq(&q->queue_lock);
-
-	blk_mq_unfreeze_queue(q);
-
-	if (rqos->ops->debugfs_attrs)
-		blk_mq_debugfs_register_rqos(rqos);
-}
-
-static inline void rq_qos_del(struct request_queue *q, struct rq_qos *rqos)
-{
-	struct rq_qos **cur;
-
-	/*
-	 * See comment in rq_qos_add() about freezing queue & using
-	 * ->queue_lock.
-	 */
-	blk_mq_freeze_queue(q);
-
-	spin_lock_irq(&q->queue_lock);
-	for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {
-		if (*cur == rqos) {
-			*cur = rqos->next;
-			break;
-		}
-	}
-	spin_unlock_irq(&q->queue_lock);
-
-	blk_mq_unfreeze_queue(q);
-
-	blk_mq_debugfs_unregister_rqos(rqos);
-}
-
 int rq_qos_register(struct rq_qos_ops *ops);
 void rq_qos_unregister(struct rq_qos_ops *ops);
 void rq_qos_activate(struct request_queue *q,
-- 
2.17.1

[RFC V5 08/16] blk/rq-qos: export the sysfs for switching qos

[Wang Jianchao (Kuaishou)]

All of the blk-rq-qos policies has been changed to use the new
interfaces. Thus we can export the sysfs interface, namely
/sys/block/xxx/queue/qos

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-sysfs.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index c713d6b305cc..052b5f03fc73 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -572,6 +572,7 @@ QUEUE_RO_ENTRY(queue_max_segments, "max_segments");
 QUEUE_RO_ENTRY(queue_max_integrity_segments, "max_integrity_segments");
 QUEUE_RO_ENTRY(queue_max_segment_size, "max_segment_size");
 QUEUE_RW_ENTRY(elv_iosched, "scheduler");
+QUEUE_RW_ENTRY(queue_qos, "qos");
 
 QUEUE_RO_ENTRY(queue_logical_block_size, "logical_block_size");
 QUEUE_RO_ENTRY(queue_physical_block_size, "physical_block_size");
@@ -631,6 +632,7 @@ static struct attribute *queue_attrs[] = {
 	&queue_max_integrity_segments_entry.attr,
 	&queue_max_segment_size_entry.attr,
 	&elv_iosched_entry.attr,
+	&queue_qos_entry.attr,
 	&queue_hw_sector_size_entry.attr,
 	&queue_logical_block_size_entry.attr,
 	&queue_physical_block_size_entry.attr,
-- 
2.17.1

[RFC V5 09/16] blk-wbt: show wbt_lat_us sysfs interface only when wbt is opened

[Wang Jianchao (Kuaishou)]

Add 'wbt_lat_us' to block device' queue sysfs only when we open
'wbt' blk-rq-qos policy. And do some code cleanup as we needn't
to export those wbt interfaces.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-sysfs.c |  71 ----------------
 block/blk-wbt.c   | 201 +++++++++++++++++++++++++++-------------------
 block/blk-wbt.h   |  28 -------
 block/blk.h       |   6 ++
 4 files changed, 125 insertions(+), 181 deletions(-)

diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 052b5f03fc73..1b321edd7ba7 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -20,12 +20,6 @@
 #include "blk-wbt.h"
 #include "blk-throttle.h"
 
-struct queue_sysfs_entry {
-	struct attribute attr;
-	ssize_t (*show)(struct request_queue *, char *);
-	ssize_t (*store)(struct request_queue *, const char *, size_t);
-};
-
 static ssize_t
 queue_var_show(unsigned long var, char *page)
 {
@@ -47,19 +41,6 @@ queue_var_store(unsigned long *var, const char *page, size_t count)
 	return count;
 }
 
-static ssize_t queue_var_store64(s64 *var, const char *page)
-{
-	int err;
-	s64 v;
-
-	err = kstrtos64(page, 10, &v);
-	if (err < 0)
-		return err;
-
-	*var = v;
-	return 0;
-}
-
 static ssize_t queue_requests_show(struct request_queue *q, char *page)
 {
 	return queue_var_show(q->nr_requests, page);
@@ -461,56 +442,6 @@ static ssize_t queue_io_timeout_store(struct request_queue *q, const char *page,
 	return count;
 }
 
-static ssize_t queue_wb_lat_show(struct request_queue *q, char *page)
-{
-	if (!wbt_opened(q))
-		return -EINVAL;
-
-	return sprintf(page, "%llu\n", div_u64(wbt_get_min_lat(q), 1000));
-}
-
-static ssize_t queue_wb_lat_store(struct request_queue *q, const char *page,
-				  size_t count)
-{
-	ssize_t ret;
-	s64 val;
-
-	ret = queue_var_store64(&val, page);
-	if (ret < 0)
-		return ret;
-	if (val < -1)
-		return -EINVAL;
-
-	if (!wbt_opened(q)) {
-		ret = wbt_init(q);
-		if (ret)
-			return ret;
-	}
-
-	if (val == -1)
-		val = wbt_default_latency_nsec(q);
-	else if (val >= 0)
-		val *= 1000ULL;
-
-	if (wbt_get_min_lat(q) == val)
-		return count;
-
-	/*
-	 * Ensure that the queue is idled, in case the latency update
-	 * ends up either enabling or disabling wbt completely. We can't
-	 * have IO inflight if that happens.
-	 */
-	blk_mq_freeze_queue(q);
-	blk_mq_quiesce_queue(q);
-
-	wbt_set_min_lat(q, val);
-
-	blk_mq_unquiesce_queue(q);
-	blk_mq_unfreeze_queue(q);
-
-	return count;
-}
-
 static ssize_t queue_wc_show(struct request_queue *q, char *page)
 {
 	if (test_bit(QUEUE_FLAG_WC, &q->queue_flags))
@@ -604,7 +535,6 @@ QUEUE_RW_ENTRY(queue_wc, "write_cache");
 QUEUE_RO_ENTRY(queue_fua, "fua");
 QUEUE_RO_ENTRY(queue_dax, "dax");
 QUEUE_RW_ENTRY(queue_io_timeout, "io_timeout");
-QUEUE_RW_ENTRY(queue_wb_lat, "wbt_lat_usec");
 QUEUE_RO_ENTRY(queue_virt_boundary_mask, "virt_boundary_mask");
 
 #ifdef CONFIG_BLK_DEV_THROTTLING_LOW
@@ -661,7 +591,6 @@ static struct attribute *queue_attrs[] = {
 	&queue_wc_entry.attr,
 	&queue_fua_entry.attr,
 	&queue_dax_entry.attr,
-	&queue_wb_lat_entry.attr,
 	&queue_poll_delay_entry.attr,
 	&queue_io_timeout_entry.attr,
 #ifdef CONFIG_BLK_DEV_THROTTLING_LOW
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index 44a920d83475..bbec3b394273 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -24,7 +24,9 @@
 #include <linux/slab.h>
 #include <linux/backing-dev.h>
 #include <linux/swap.h>
+#include <linux/sysfs.h>
 
+#include "blk.h"
 #include "blk-wbt.h"
 #include "blk-rq-qos.h"
 
@@ -38,15 +40,6 @@ static inline struct rq_qos *wbt_rq_qos(struct request_queue *q)
 	return rq_qos_get(q, wbt_rqos_ops.id);
 }
 
-bool wbt_opened(struct request_queue *q)
-{
-	struct rq_qos *rqos = wbt_rq_qos(q);
-
-	rq_qos_put(rqos);
-
-	return !!rqos;
-}
-
 static inline void wbt_clear_state(struct request *rq)
 {
 	rq->wbt_flags = 0;
@@ -438,33 +431,6 @@ static void wbt_update_limits(struct rq_wb *rwb)
 	rwb_wake_all(rwb);
 }
 
-u64 wbt_get_min_lat(struct request_queue *q)
-{
-	struct rq_qos *rqos = wbt_rq_qos(q);
-	u64 nsec;
-
-	if (!rqos)
-		return 0;
-
-	nsec = RQWB(rqos)->min_lat_nsec;
-	rq_qos_put(rqos);
-
-	return  nsec;
-}
-
-void wbt_set_min_lat(struct request_queue *q, u64 val)
-{
-	struct rq_qos *rqos = wbt_rq_qos(q);
-
-	if (!rqos)
-		return;
-	RQWB(rqos)->min_lat_nsec = val;
-	RQWB(rqos)->enable_state = WBT_STATE_ON_MANUAL;
-	wbt_update_limits(RQWB(rqos));
-	rq_qos_put(rqos);
-}
-
-
 static bool close_io(struct rq_wb *rwb)
 {
 	const unsigned long now = jiffies;
@@ -660,32 +626,7 @@ void wbt_set_write_cache(struct request_queue *q, bool write_cache_on)
 	}
 }
 
-/*
- * Enable wbt if defaults are configured that way
- */
-void wbt_enable_default(struct request_queue *q)
-{
-	struct rq_qos *rqos = wbt_rq_qos(q);
-
-	/* Throttling already enabled? */
-	if (rqos) {
-		if (RQWB(rqos)->enable_state == WBT_STATE_OFF_DEFAULT)
-			RQWB(rqos)->enable_state = WBT_STATE_ON_DEFAULT;
-
-		rq_qos_put(rqos);
-		return;
-	}
-
-	/* Queue not registered? Maybe shutting down... */
-	if (!blk_queue_registered(q))
-		return;
-
-	if (queue_is_mq(q) && IS_ENABLED(CONFIG_BLK_WBT_MQ))
-		wbt_init(q);
-}
-EXPORT_SYMBOL_GPL(wbt_enable_default);
-
-u64 wbt_default_latency_nsec(struct request_queue *q)
+static u64 wbt_default_latency_nsec(struct request_queue *q)
 {
 	/*
 	 * We default to 2msec for non-rotational storage, and 75msec
@@ -716,7 +657,67 @@ static void wbt_queue_depth_changed(struct rq_qos *rqos)
 	wbt_update_limits(RQWB(rqos));
 }
 
-static void wbt_exit(struct rq_qos *rqos)
+static ssize_t wbt_lat_show(struct request_queue *q, char *page)
+{
+	struct rq_qos *rqos = wbt_rq_qos(q);
+	ssize_t ret;
+
+	ret = sprintf(page, "%llu\n", div_u64(RQWB(rqos)->min_lat_nsec, 1000));
+
+	rq_qos_put(rqos);
+
+	return ret;
+}
+
+static ssize_t wbt_lat_store(struct request_queue *q, const char *page,
+				  size_t count)
+{
+	struct rq_qos *rqos;
+	struct rq_wb *rwb;
+	ssize_t ret;
+	s64 val;
+
+	ret = kstrtos64(page, 10, &val);
+	if (ret < 0)
+		return ret;
+
+	if (val < -1)
+		return -EINVAL;
+
+	if (val == -1)
+		val = wbt_default_latency_nsec(q);
+	else if (val >= 0)
+		val *= 1000ULL;
+
+	rqos = wbt_rq_qos(q);
+	rwb = RQWB(rqos);
+
+	if (rwb->min_lat_nsec == val)
+		goto out;
+	/*
+	 * Ensure that the queue is idled, in case the latency update
+	 * ends up either enabling or disabling wbt completely. We can't
+	 * have IO inflight if that happens.
+	 */
+	blk_mq_freeze_queue(q);
+	blk_mq_quiesce_queue(q);
+
+	rwb->min_lat_nsec = val;
+	rwb->enable_state = WBT_STATE_ON_MANUAL;
+	wbt_update_limits(rwb);
+
+	blk_mq_unquiesce_queue(q);
+	blk_mq_unfreeze_queue(q);
+
+out:
+	rq_qos_put(rqos);
+	return count;
+}
+
+static struct queue_sysfs_entry wbt_attr =
+	__ATTR(wbt_lat_usec, 0644, wbt_lat_show, wbt_lat_store);
+
+static void __wbt_exit(struct rq_qos *rqos)
 {
 	struct rq_wb *rwb = RQWB(rqos);
 	struct request_queue *q = rqos->q;
@@ -727,6 +728,12 @@ static void wbt_exit(struct rq_qos *rqos)
 	kfree(rwb);
 }
 
+static void wbt_exit(struct rq_qos *rqos)
+{
+	sysfs_remove_file(&rqos->q->kobj, &wbt_attr.attr);
+	__wbt_exit(rqos);
+}
+
 /*
  * Disable wbt, if enabled by default.
  */
@@ -835,26 +842,10 @@ static const struct blk_mq_debugfs_attr wbt_debugfs_attrs[] = {
 };
 #endif
 
-static struct rq_qos_ops wbt_rqos_ops = {
-	.name = "wbt",
-	.throttle = wbt_wait,
-	.issue = wbt_issue,
-	.track = wbt_track,
-	.requeue = wbt_requeue,
-	.done = wbt_done,
-	.cleanup = wbt_cleanup,
-	.queue_depth_changed = wbt_queue_depth_changed,
-	.exit = wbt_exit,
-	.init = wbt_init,
-#ifdef CONFIG_BLK_DEBUG_FS
-	.debugfs_attrs = wbt_debugfs_attrs,
-#endif
-};
-
-int wbt_init(struct request_queue *q)
+static int wbt_init(struct request_queue *q)
 {
 	struct rq_wb *rwb;
-	int i;
+	int i, ret;
 
 	rwb = kzalloc(sizeof(*rwb), GFP_KERNEL);
 	if (!rwb)
@@ -886,9 +877,54 @@ int wbt_init(struct request_queue *q)
 	wbt_queue_depth_changed(&rwb->rqos);
 	wbt_set_write_cache(q, test_bit(QUEUE_FLAG_WC, &q->queue_flags));
 
-	return 0;
+	ret = sysfs_create_file(&q->kobj, &wbt_attr.attr);
+	if (ret < 0)
+		__wbt_exit(&rwb->rqos);
+
+	return ret;
 }
 
+static struct rq_qos_ops wbt_rqos_ops = {
+	.name = "wbt",
+	.throttle = wbt_wait,
+	.issue = wbt_issue,
+	.track = wbt_track,
+	.requeue = wbt_requeue,
+	.done = wbt_done,
+	.cleanup = wbt_cleanup,
+	.queue_depth_changed = wbt_queue_depth_changed,
+	.exit = wbt_exit,
+	.init = wbt_init,
+#ifdef CONFIG_BLK_DEBUG_FS
+	.debugfs_attrs = wbt_debugfs_attrs,
+#endif
+};
+
+/*
+ * Enable wbt if defaults are configured that way
+ */
+void wbt_enable_default(struct request_queue *q)
+{
+	struct rq_qos *rqos = wbt_rq_qos(q);
+
+	/* Throttling already enabled? */
+	if (rqos) {
+		if (RQWB(rqos)->enable_state == WBT_STATE_OFF_DEFAULT)
+			RQWB(rqos)->enable_state = WBT_STATE_ON_DEFAULT;
+
+		rq_qos_put(rqos);
+		return;
+	}
+
+	/* Queue not registered? Maybe shutting down... */
+	if (!blk_queue_registered(q))
+		return;
+
+	if (queue_is_mq(q) && IS_ENABLED(CONFIG_BLK_WBT_MQ))
+		wbt_init(q);
+}
+EXPORT_SYMBOL_GPL(wbt_enable_default);
+
 static __init int wbt_mod_init(void)
 {
 	return rq_qos_register(&wbt_rqos_ops);
@@ -898,5 +934,6 @@ static __exit void wbt_mod_exit(void)
 {
 	return rq_qos_unregister(&wbt_rqos_ops);
 }
+
 module_init(wbt_mod_init);
 module_exit(wbt_mod_exit);
diff --git a/block/blk-wbt.h b/block/blk-wbt.h
index f128dcf662db..b9ed3f373b5a 100644
--- a/block/blk-wbt.h
+++ b/block/blk-wbt.h
@@ -88,32 +88,16 @@ static inline unsigned int wbt_inflight(struct rq_wb *rwb)
 
 #ifdef CONFIG_BLK_WBT
 
-bool wbt_opened(struct request_queue *q);
-int wbt_init(struct request_queue *);
 void wbt_disable_default(struct request_queue *);
 void wbt_enable_default(struct request_queue *);
 
-u64 wbt_get_min_lat(struct request_queue *q);
-void wbt_set_min_lat(struct request_queue *q, u64 val);
-
 void wbt_set_write_cache(struct request_queue *, bool);
 
-u64 wbt_default_latency_nsec(struct request_queue *);
-
 #else
 
-bool wbt_opened(struct request_queue *q)
-{
-	return false;
-}
-
 static inline void wbt_track(struct request *rq, enum wbt_flags flags)
 {
 }
-static inline int wbt_init(struct request_queue *q)
-{
-	return -EINVAL;
-}
 static inline void wbt_disable_default(struct request_queue *q)
 {
 }
@@ -123,18 +107,6 @@ static inline void wbt_enable_default(struct request_queue *q)
 static inline void wbt_set_write_cache(struct request_queue *q, bool wc)
 {
 }
-static inline u64 wbt_get_min_lat(struct request_queue *q)
-{
-	return 0;
-}
-static inline void wbt_set_min_lat(struct request_queue *q, u64 val)
-{
-}
-static inline u64 wbt_default_latency_nsec(struct request_queue *q)
-{
-	return 0;
-}
-
 #endif /* CONFIG_BLK_WBT */
 
 #endif
diff --git a/block/blk.h b/block/blk.h
index 1a314257b6a3..2ec7510c080c 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -14,6 +14,12 @@ struct elevator_type;
 
 extern struct dentry *blk_debugfs_root;
 
+struct queue_sysfs_entry {
+	struct attribute attr;
+	ssize_t (*show)(struct request_queue *, char *);
+	ssize_t (*store)(struct request_queue *, const char *, size_t);
+};
+
 struct blk_flush_queue {
 	unsigned int		flush_pending_idx:1;
 	unsigned int		flush_running_idx:1;
-- 
2.17.1

[RFC V5 10/16] blk-wbt: get rid of wbt_enable/disable_default()

[Wang Jianchao (Kuaishou)]

wbt_enable_default() is used to open wbt feature by default.
wbt_disable_default() is only employed by bfq which has similar
writeback throttling mechanism. This kind of policy should be
selected by the user instead of being set in kernel. This patch
get rid of the interfaces and let the user configure appropricate
blk-rq-qos policy by themselves.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/bfq-iosched.c |  2 --
 block/blk-sysfs.c   |  3 +--
 block/blk-wbt.c     | 60 +--------------------------------------------
 block/blk-wbt.h     | 22 -----------------
 block/elevator.c    |  3 ---
 5 files changed, 2 insertions(+), 88 deletions(-)

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index 0c612a911696..adbe5ff6c5e8 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -132,7 +132,6 @@
 #include "blk-mq-tag.h"
 #include "blk-mq-sched.h"
 #include "bfq-iosched.h"
-#include "blk-wbt.h"
 
 #define BFQ_BFQQ_FNS(name)						\
 void bfq_mark_bfqq_##name(struct bfq_queue *bfqq)			\
@@ -7158,7 +7157,6 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e)
 	bfq_init_root_group(bfqd->root_group, bfqd);
 	bfq_init_entity(&bfqd->oom_bfqq.entity, bfqd->root_group);
 
-	wbt_disable_default(q);
 	return 0;
 
 out_free:
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 1b321edd7ba7..4563283dae07 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -17,8 +17,8 @@
 #include "blk-mq.h"
 #include "blk-mq-debugfs.h"
 #include "blk-mq-sched.h"
-#include "blk-wbt.h"
 #include "blk-throttle.h"
+#include "blk-rq-qos.h"
 
 static ssize_t
 queue_var_show(unsigned long var, char *page)
@@ -810,7 +810,6 @@ int blk_register_queue(struct gendisk *disk)
 	}
 
 	blk_queue_flag_set(QUEUE_FLAG_REGISTERED, q);
-	wbt_enable_default(q);
 	blk_throtl_register_queue(q);
 
 	/* Now everything is ready and send out KOBJ_ADD uevent */
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index bbec3b394273..6c0a85ddb107 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -86,8 +86,7 @@ enum {
 
 static inline bool rwb_enabled(struct rq_wb *rwb)
 {
-	return rwb && rwb->enable_state != WBT_STATE_OFF_DEFAULT &&
-		      rwb->wb_normal != 0;
+	return rwb && rwb->wb_normal != 0;
 }
 
 static void wb_timestamp(struct rq_wb *rwb, unsigned long *var)
@@ -703,7 +702,6 @@ static ssize_t wbt_lat_store(struct request_queue *q, const char *page,
 	blk_mq_quiesce_queue(q);
 
 	rwb->min_lat_nsec = val;
-	rwb->enable_state = WBT_STATE_ON_MANUAL;
 	wbt_update_limits(rwb);
 
 	blk_mq_unquiesce_queue(q);
@@ -734,26 +732,6 @@ static void wbt_exit(struct rq_qos *rqos)
 	__wbt_exit(rqos);
 }
 
-/*
- * Disable wbt, if enabled by default.
- */
-void wbt_disable_default(struct request_queue *q)
-{
-	struct rq_qos *rqos = wbt_rq_qos(q);
-	struct rq_wb *rwb;
-
-	if (!rqos)
-		return;
-
-	rwb = RQWB(rqos);
-	if (rwb->enable_state == WBT_STATE_ON_DEFAULT) {
-		blk_stat_deactivate(rwb->cb);
-		rwb->enable_state = WBT_STATE_OFF_DEFAULT;
-	}
-	rq_qos_put(rqos);
-}
-EXPORT_SYMBOL_GPL(wbt_disable_default);
-
 #ifdef CONFIG_BLK_DEBUG_FS
 static int wbt_curr_win_nsec_show(void *data, struct seq_file *m)
 {
@@ -764,15 +742,6 @@ static int wbt_curr_win_nsec_show(void *data, struct seq_file *m)
 	return 0;
 }
 
-static int wbt_enabled_show(void *data, struct seq_file *m)
-{
-	struct rq_qos *rqos = data;
-	struct rq_wb *rwb = RQWB(rqos);
-
-	seq_printf(m, "%d\n", rwb->enable_state);
-	return 0;
-}
-
 static int wbt_id_show(void *data, struct seq_file *m)
 {
 	struct rq_qos *rqos = data;
@@ -831,7 +800,6 @@ static int wbt_background_show(void *data, struct seq_file *m)
 
 static const struct blk_mq_debugfs_attr wbt_debugfs_attrs[] = {
 	{"curr_win_nsec", 0400, wbt_curr_win_nsec_show},
-	{"enabled", 0400, wbt_enabled_show},
 	{"id", 0400, wbt_id_show},
 	{"inflight", 0400, wbt_inflight_show},
 	{"min_lat_nsec", 0400, wbt_min_lat_nsec_show},
@@ -862,7 +830,6 @@ static int wbt_init(struct request_queue *q)
 
 	rwb->last_comp = rwb->last_issue = jiffies;
 	rwb->win_nsec = RWB_WINDOW_NSEC;
-	rwb->enable_state = WBT_STATE_ON_DEFAULT;
 	rwb->wc = 1;
 	rwb->rq_depth.default_depth = RWB_DEF_DEPTH;
 
@@ -900,31 +867,6 @@ static struct rq_qos_ops wbt_rqos_ops = {
 #endif
 };
 
-/*
- * Enable wbt if defaults are configured that way
- */
-void wbt_enable_default(struct request_queue *q)
-{
-	struct rq_qos *rqos = wbt_rq_qos(q);
-
-	/* Throttling already enabled? */
-	if (rqos) {
-		if (RQWB(rqos)->enable_state == WBT_STATE_OFF_DEFAULT)
-			RQWB(rqos)->enable_state = WBT_STATE_ON_DEFAULT;
-
-		rq_qos_put(rqos);
-		return;
-	}
-
-	/* Queue not registered? Maybe shutting down... */
-	if (!blk_queue_registered(q))
-		return;
-
-	if (queue_is_mq(q) && IS_ENABLED(CONFIG_BLK_WBT_MQ))
-		wbt_init(q);
-}
-EXPORT_SYMBOL_GPL(wbt_enable_default);
-
 static __init int wbt_mod_init(void)
 {
 	return rq_qos_register(&wbt_rqos_ops);
diff --git a/block/blk-wbt.h b/block/blk-wbt.h
index b9ed3f373b5a..fe26f78390bd 100644
--- a/block/blk-wbt.h
+++ b/block/blk-wbt.h
@@ -27,16 +27,6 @@ enum {
 	WBT_NUM_RWQ,
 };
 
-/*
- * Enable states. Either off, or on by default (done at init time),
- * or on through manual setup in sysfs.
- */
-enum {
-	WBT_STATE_ON_DEFAULT	= 1,
-	WBT_STATE_ON_MANUAL	= 2,
-	WBT_STATE_OFF_DEFAULT
-};
-
 struct rq_wb {
 	/*
 	 * Settings that govern how we throttle
@@ -44,8 +34,6 @@ struct rq_wb {
 	unsigned int wb_background;		/* background writeback */
 	unsigned int wb_normal;			/* normal writeback */
 
-	short enable_state;			/* WBT_STATE_* */
-
 	/*
 	 * Number of consecutive periods where we don't have enough
 	 * information to make a firm scale up/down decision.
@@ -85,12 +73,8 @@ static inline unsigned int wbt_inflight(struct rq_wb *rwb)
 	return ret;
 }
 
-
 #ifdef CONFIG_BLK_WBT
 
-void wbt_disable_default(struct request_queue *);
-void wbt_enable_default(struct request_queue *);
-
 void wbt_set_write_cache(struct request_queue *, bool);
 
 #else
@@ -98,12 +82,6 @@ void wbt_set_write_cache(struct request_queue *, bool);
 static inline void wbt_track(struct request *rq, enum wbt_flags flags)
 {
 }
-static inline void wbt_disable_default(struct request_queue *q)
-{
-}
-static inline void wbt_enable_default(struct request_queue *q)
-{
-}
 static inline void wbt_set_write_cache(struct request_queue *q, bool wc)
 {
 }
diff --git a/block/elevator.c b/block/elevator.c
index ec98aed39c4f..0da6fcf08d7b 100644
--- a/block/elevator.c
+++ b/block/elevator.c
@@ -43,7 +43,6 @@
 #include "blk.h"
 #include "blk-mq-sched.h"
 #include "blk-pm.h"
-#include "blk-wbt.h"
 
 static DEFINE_SPINLOCK(elv_list_lock);
 static LIST_HEAD(elv_list);
@@ -525,8 +524,6 @@ void elv_unregister_queue(struct request_queue *q)
 		kobject_del(&e->kobj);
 
 		e->registered = 0;
-		/* Re-enable throttling in case elevator disabled it */
-		wbt_enable_default(q);
 	}
 }
 
-- 
2.17.1

[RFC V5 11/16] blk/rq-qos: get rid of debugfs register in blk_mq_debugfs_register

[Wang Jianchao (Kuaishou)]

There is not any enabled rqos policy when queue is initialized,
get rid of the coding for registering debugfs.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-mq-debugfs.c | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 7e3c3cc1b1a9..4c9dafd16a08 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -726,18 +726,6 @@ void blk_mq_debugfs_register(struct request_queue *q)
 		if (q->elevator && !hctx->sched_debugfs_dir)
 			blk_mq_debugfs_register_sched_hctx(q, hctx);
 	}
-
-	if (q->rq_qos) {
-		struct rq_qos *rqos = q->rq_qos;
-		/*
-		 * Queue has not been registered right now, it is safe to
-		 * iterate the rqos w/o lock
-		 */
-		while (rqos) {
-			blk_mq_debugfs_register_rqos(rqos);
-			rqos = rqos->next;
-		}
-	}
 }
 
 void blk_mq_debugfs_unregister(struct request_queue *q)
-- 
2.17.1

[RFC V5 12/16] blk/rq-qos: change queue_depth_changed to setting_changed

[Wang Jianchao (Kuaishou)]

Change the callback .queue_depth_changed() to .setting_changed().
Then all of the queue setting changes can be involved.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-iocost.c   |  4 ++--
 block/blk-rq-qos.c   |  6 +++---
 block/blk-rq-qos.h   |  8 ++++----
 block/blk-settings.c |  2 +-
 block/blk-wbt.c      | 12 +++++++-----
 5 files changed, 17 insertions(+), 15 deletions(-)

diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 886c03383444..ad1d4d9762bd 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2793,7 +2793,7 @@ static void ioc_rqos_done(struct rq_qos *rqos, struct request *rq)
 	put_cpu_ptr(ccs);
 }
 
-static void ioc_rqos_queue_depth_changed(struct rq_qos *rqos)
+static void ioc_rqos_setting_changed(struct rq_qos *rqos)
 {
 	struct ioc *ioc = rqos_to_ioc(rqos);
 
@@ -2885,7 +2885,7 @@ static struct rq_qos_ops ioc_rqos_ops = {
 	.merge = ioc_rqos_merge,
 	.done_bio = ioc_rqos_done_bio,
 	.done = ioc_rqos_done,
-	.queue_depth_changed = ioc_rqos_queue_depth_changed,
+	.setting_changed = ioc_rqos_setting_changed,
 	.exit = ioc_rqos_exit,
 	.init = blk_iocost_init,
 };
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index 03f2e654cd17..236ecb2af845 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -104,11 +104,11 @@ void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio)
 	} while (rqos);
 }
 
-void __rq_qos_queue_depth_changed(struct rq_qos *rqos)
+void __rq_qos_setting_changed(struct rq_qos *rqos)
 {
 	do {
-		if (rqos->ops->queue_depth_changed)
-			rqos->ops->queue_depth_changed(rqos);
+		if (rqos->ops->setting_changed)
+			rqos->ops->setting_changed(rqos);
 		rqos = rqos->next;
 	} while (rqos);
 }
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 58aba16c7d32..3d7df14edde3 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -42,7 +42,7 @@ struct rq_qos_ops {
 	void (*done)(struct rq_qos *, struct request *);
 	void (*done_bio)(struct rq_qos *, struct bio *);
 	void (*cleanup)(struct rq_qos *, struct bio *);
-	void (*queue_depth_changed)(struct rq_qos *);
+	void (*setting_changed)(struct rq_qos *);
 	void (*exit)(struct rq_qos *);
 	int (*init)(struct request_queue *);
 	const struct blk_mq_debugfs_attr *debugfs_attrs;
@@ -94,7 +94,7 @@ void __rq_qos_throttle(struct rq_qos *rqos, struct bio *bio);
 void __rq_qos_track(struct rq_qos *rqos, struct request *rq, struct bio *bio);
 void __rq_qos_merge(struct rq_qos *rqos, struct request *rq, struct bio *bio);
 void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio);
-void __rq_qos_queue_depth_changed(struct rq_qos *rqos);
+void __rq_qos_setting_changed(struct rq_qos *rqos);
 
 static inline void rq_qos_cleanup(struct request_queue *q, struct bio *bio)
 {
@@ -152,7 +152,7 @@ static inline void rq_qos_merge(struct request_queue *q, struct request *rq,
 		__rq_qos_merge(q->rq_qos, rq, bio);
 }
 
-static inline void rq_qos_queue_depth_changed(struct request_queue *q)
+static inline void rq_qos_setting_changed(struct request_queue *q)
 {
 	/*
 	 * It is called by external module, protect the rqos list with
@@ -160,7 +160,7 @@ static inline void rq_qos_queue_depth_changed(struct request_queue *q)
 	 */
 	mutex_lock(&q->sysfs_lock);
 	if (q->rq_qos)
-		__rq_qos_queue_depth_changed(q->rq_qos);
+		__rq_qos_setting_changed(q->rq_qos);
 	mutex_unlock(&q->sysfs_lock);
 }
 
diff --git a/block/blk-settings.c b/block/blk-settings.c
index b880c70e22e4..bceb1925e978 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -777,7 +777,7 @@ EXPORT_SYMBOL(blk_queue_update_dma_alignment);
 void blk_set_queue_depth(struct request_queue *q, unsigned int depth)
 {
 	q->queue_depth = depth;
-	rq_qos_queue_depth_changed(q);
+	rq_qos_setting_changed(q);
 }
 EXPORT_SYMBOL(blk_set_queue_depth);
 
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index 6c0a85ddb107..3a3a683009c4 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -650,10 +650,12 @@ static int wbt_data_dir(const struct request *rq)
 	return -1;
 }
 
-static void wbt_queue_depth_changed(struct rq_qos *rqos)
+static void wbt_setting_changed(struct rq_qos *rqos)
 {
-	RQWB(rqos)->rq_depth.queue_depth = blk_queue_depth(rqos->q);
-	wbt_update_limits(RQWB(rqos));
+	if (RQWB(rqos)->rq_depth.queue_depth != blk_queue_depth(rqos->q)) {
+		RQWB(rqos)->rq_depth.queue_depth = blk_queue_depth(rqos->q);
+		wbt_update_limits(RQWB(rqos));
+	}
 }
 
 static ssize_t wbt_lat_show(struct request_queue *q, char *page)
@@ -841,7 +843,7 @@ static int wbt_init(struct request_queue *q)
 
 	rwb->min_lat_nsec = wbt_default_latency_nsec(q);
 
-	wbt_queue_depth_changed(&rwb->rqos);
+	wbt_setting_changed(&rwb->rqos);
 	wbt_set_write_cache(q, test_bit(QUEUE_FLAG_WC, &q->queue_flags));
 
 	ret = sysfs_create_file(&q->kobj, &wbt_attr.attr);
@@ -859,7 +861,7 @@ static struct rq_qos_ops wbt_rqos_ops = {
 	.requeue = wbt_requeue,
 	.done = wbt_done,
 	.cleanup = wbt_cleanup,
-	.queue_depth_changed = wbt_queue_depth_changed,
+	.setting_changed = wbt_setting_changed,
 	.exit = wbt_exit,
 	.init = wbt_init,
 #ifdef CONFIG_BLK_DEBUG_FS
-- 
2.17.1

[RFC V5 13/16] blk-wbt: move cache setting to rq_qos_setting_changed()

[Wang Jianchao (Kuaishou)]

Move cache setting to rq_qos_setting_changed then we can avoid
to export a standalone wbt interface to blk_queue_write_cache().

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-settings.c |  4 ++--
 block/blk-wbt.c      | 13 ++-----------
 block/blk-wbt.h      |  7 -------
 3 files changed, 4 insertions(+), 20 deletions(-)

diff --git a/block/blk-settings.c b/block/blk-settings.c
index bceb1925e978..124bf5cc623f 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -16,7 +16,7 @@
 #include <linux/dma-mapping.h>
 
 #include "blk.h"
-#include "blk-wbt.h"
+#include "blk-rq-qos.h"
 
 void blk_queue_rq_timeout(struct request_queue *q, unsigned int timeout)
 {
@@ -800,7 +800,7 @@ void blk_queue_write_cache(struct request_queue *q, bool wc, bool fua)
 	else
 		blk_queue_flag_clear(QUEUE_FLAG_FUA, q);
 
-	wbt_set_write_cache(q, test_bit(QUEUE_FLAG_WC, &q->queue_flags));
+	rq_qos_setting_changed(q);
 }
 EXPORT_SYMBOL_GPL(blk_queue_write_cache);
 
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index 3a3a683009c4..83e05417e25f 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -615,16 +615,6 @@ static void wbt_requeue(struct rq_qos *rqos, struct request *rq)
 	}
 }
 
-void wbt_set_write_cache(struct request_queue *q, bool write_cache_on)
-{
-	struct rq_qos *rqos = wbt_rq_qos(q);
-
-	if (rqos) {
-		RQWB(rqos)->wc = write_cache_on;
-		rq_qos_put(rqos);
-	}
-}
-
 static u64 wbt_default_latency_nsec(struct request_queue *q)
 {
 	/*
@@ -656,6 +646,7 @@ static void wbt_setting_changed(struct rq_qos *rqos)
 		RQWB(rqos)->rq_depth.queue_depth = blk_queue_depth(rqos->q);
 		wbt_update_limits(RQWB(rqos));
 	}
+	RQWB(rqos)->wc = test_bit(QUEUE_FLAG_WC, &rqos->q->queue_flags);
 }
 
 static ssize_t wbt_lat_show(struct request_queue *q, char *page)
@@ -844,7 +835,7 @@ static int wbt_init(struct request_queue *q)
 	rwb->min_lat_nsec = wbt_default_latency_nsec(q);
 
 	wbt_setting_changed(&rwb->rqos);
-	wbt_set_write_cache(q, test_bit(QUEUE_FLAG_WC, &q->queue_flags));
+	rwb->wc = test_bit(QUEUE_FLAG_WC, &q->queue_flags);
 
 	ret = sysfs_create_file(&q->kobj, &wbt_attr.attr);
 	if (ret < 0)
diff --git a/block/blk-wbt.h b/block/blk-wbt.h
index fe26f78390bd..eb837dfd0ace 100644
--- a/block/blk-wbt.h
+++ b/block/blk-wbt.h
@@ -74,17 +74,10 @@ static inline unsigned int wbt_inflight(struct rq_wb *rwb)
 }
 
 #ifdef CONFIG_BLK_WBT
-
-void wbt_set_write_cache(struct request_queue *, bool);
-
 #else
-
 static inline void wbt_track(struct request *rq, enum wbt_flags flags)
 {
 }
-static inline void wbt_set_write_cache(struct request_queue *q, bool wc)
-{
-}
 #endif /* CONFIG_BLK_WBT */
 
 #endif
-- 
2.17.1

[RFC V5 14/16] blk-wbt: cleanup the blk-wbt.h

[Wang Jianchao (Kuaishou)]

There is not any wbt functions that need to be external, cleanup
the blk-wbt.h

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-iocost.c         |  1 -
 block/blk-mq-sched.c       |  1 -
 block/blk-wbt.c            | 64 ++++++++++++++++++++++++++++-
 block/blk-wbt.h            | 83 --------------------------------------
 include/trace/events/wbt.h |  3 +-
 5 files changed, 65 insertions(+), 87 deletions(-)
 delete mode 100644 block/blk-wbt.h

diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index ad1d4d9762bd..c7142b60bb1a 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -183,7 +183,6 @@
 #include <asm/local64.h>
 #include "blk-rq-qos.h"
 #include "blk-stat.h"
-#include "blk-wbt.h"
 
 #ifdef CONFIG_TRACEPOINTS
 
diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index 55488ba97823..25f65a5d5bb7 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c
@@ -16,7 +16,6 @@
 #include "blk-mq-debugfs.h"
 #include "blk-mq-sched.h"
 #include "blk-mq-tag.h"
-#include "blk-wbt.h"
 
 /*
  * Mark a hardware queue as needing a restart. For shared queues, maintain
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index 83e05417e25f..33072118929f 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -27,14 +27,76 @@
 #include <linux/sysfs.h>
 
 #include "blk.h"
-#include "blk-wbt.h"
+#include "blk-stat.h"
 #include "blk-rq-qos.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/wbt.h>
 
+enum wbt_flags {
+	WBT_TRACKED		= 1,	/* write, tracked for throttling */
+	WBT_READ		= 2,	/* read */
+	WBT_KSWAPD		= 4,	/* write, from kswapd */
+	WBT_DISCARD		= 8,	/* discard */
+
+	WBT_NR_BITS		= 4,	/* number of bits */
+};
+
+enum {
+	WBT_RWQ_BG		= 0,
+	WBT_RWQ_KSWAPD,
+	WBT_RWQ_DISCARD,
+	WBT_NUM_RWQ,
+};
+
+struct rq_wb {
+	/*
+	 * Settings that govern how we throttle
+	 */
+	unsigned int wb_background;		/* background writeback */
+	unsigned int wb_normal;			/* normal writeback */
+
+	/*
+	 * Number of consecutive periods where we don't have enough
+	 * information to make a firm scale up/down decision.
+	 */
+	unsigned int unknown_cnt;
+
+	u64 win_nsec;				/* default window size */
+	u64 cur_win_nsec;			/* current window size */
+
+	struct blk_stat_callback *cb;
+
+	u64 sync_issue;
+	void *sync_cookie;
+
+	unsigned int wc;
+
+	unsigned long last_issue;		/* last non-throttled issue */
+	unsigned long last_comp;		/* last non-throttled comp */
+	unsigned long min_lat_nsec;
+	struct rq_qos rqos;
+	struct rq_wait rq_wait[WBT_NUM_RWQ];
+	struct rq_depth rq_depth;
+};
+
 static struct rq_qos_ops wbt_rqos_ops;
 
+static inline struct rq_wb *RQWB(struct rq_qos *rqos)
+{
+	return container_of(rqos, struct rq_wb, rqos);
+}
+
+static inline unsigned int wbt_inflight(struct rq_wb *rwb)
+{
+	unsigned int i, ret = 0;
+
+	for (i = 0; i < WBT_NUM_RWQ; i++)
+		ret += atomic_read(&rwb->rq_wait[i].inflight);
+
+	return ret;
+}
+
 static inline struct rq_qos *wbt_rq_qos(struct request_queue *q)
 {
 	return rq_qos_get(q, wbt_rqos_ops.id);
diff --git a/block/blk-wbt.h b/block/blk-wbt.h
deleted file mode 100644
index eb837dfd0ace..000000000000
--- a/block/blk-wbt.h
+++ /dev/null
@@ -1,83 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef WB_THROTTLE_H
-#define WB_THROTTLE_H
-
-#include <linux/kernel.h>
-#include <linux/atomic.h>
-#include <linux/wait.h>
-#include <linux/timer.h>
-#include <linux/ktime.h>
-
-#include "blk-stat.h"
-#include "blk-rq-qos.h"
-
-enum wbt_flags {
-	WBT_TRACKED		= 1,	/* write, tracked for throttling */
-	WBT_READ		= 2,	/* read */
-	WBT_KSWAPD		= 4,	/* write, from kswapd */
-	WBT_DISCARD		= 8,	/* discard */
-
-	WBT_NR_BITS		= 4,	/* number of bits */
-};
-
-enum {
-	WBT_RWQ_BG		= 0,
-	WBT_RWQ_KSWAPD,
-	WBT_RWQ_DISCARD,
-	WBT_NUM_RWQ,
-};
-
-struct rq_wb {
-	/*
-	 * Settings that govern how we throttle
-	 */
-	unsigned int wb_background;		/* background writeback */
-	unsigned int wb_normal;			/* normal writeback */
-
-	/*
-	 * Number of consecutive periods where we don't have enough
-	 * information to make a firm scale up/down decision.
-	 */
-	unsigned int unknown_cnt;
-
-	u64 win_nsec;				/* default window size */
-	u64 cur_win_nsec;			/* current window size */
-
-	struct blk_stat_callback *cb;
-
-	u64 sync_issue;
-	void *sync_cookie;
-
-	unsigned int wc;
-
-	unsigned long last_issue;		/* last non-throttled issue */
-	unsigned long last_comp;		/* last non-throttled comp */
-	unsigned long min_lat_nsec;
-	struct rq_qos rqos;
-	struct rq_wait rq_wait[WBT_NUM_RWQ];
-	struct rq_depth rq_depth;
-};
-
-static inline struct rq_wb *RQWB(struct rq_qos *rqos)
-{
-	return container_of(rqos, struct rq_wb, rqos);
-}
-
-static inline unsigned int wbt_inflight(struct rq_wb *rwb)
-{
-	unsigned int i, ret = 0;
-
-	for (i = 0; i < WBT_NUM_RWQ; i++)
-		ret += atomic_read(&rwb->rq_wait[i].inflight);
-
-	return ret;
-}
-
-#ifdef CONFIG_BLK_WBT
-#else
-static inline void wbt_track(struct request *rq, enum wbt_flags flags)
-{
-}
-#endif /* CONFIG_BLK_WBT */
-
-#endif
diff --git a/include/trace/events/wbt.h b/include/trace/events/wbt.h
index 9c66e59d859c..b62e623ed067 100644
--- a/include/trace/events/wbt.h
+++ b/include/trace/events/wbt.h
@@ -6,7 +6,8 @@
 #define _TRACE_WBT_H
 
 #include <linux/tracepoint.h>
-#include "../../../block/blk-wbt.h"
+#include "../../../block/blk-stat.h"
+#include "../../../block/blk-rq-qos.h"
 
 /**
  * wbt_stat - trace stats for blk_wb
-- 
2.17.1

[RFC V5 15/16] blk/rq-qos: move the rqos debugfs code to blk-rq-qos.c

[Wang Jianchao (Kuaishou)]

This is no functional change in this patch.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/blk-mq-debugfs.c | 42 ++++++------------------------------------
 block/blk-mq-debugfs.h | 17 ++++-------------
 block/blk-rq-qos.c     | 38 +++++++++++++++++++++++++++++++++++---
 block/blk-rq-qos.h     |  1 +
 4 files changed, 46 insertions(+), 52 deletions(-)

diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 4c9dafd16a08..71e6a097361b 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -13,7 +13,6 @@
 #include "blk-mq-debugfs.h"
 #include "blk-mq-sched.h"
 #include "blk-mq-tag.h"
-#include "blk-rq-qos.h"
 
 static void print_stat(struct seq_file *m, struct blk_rq_stat *stat)
 {
@@ -691,7 +690,7 @@ static const struct blk_mq_debugfs_attr blk_mq_debugfs_ctx_attrs[] = {
 	{},
 };
 
-static void debugfs_create_files(struct dentry *parent, void *data,
+void blk_mq_debugfs_create_files(struct dentry *parent, void *data,
 				 const struct blk_mq_debugfs_attr *attr)
 {
 	if (IS_ERR_OR_NULL(parent))
@@ -709,7 +708,7 @@ void blk_mq_debugfs_register(struct request_queue *q)
 	struct blk_mq_hw_ctx *hctx;
 	int i;
 
-	debugfs_create_files(q->debugfs_dir, q, blk_mq_debugfs_queue_attrs);
+	blk_mq_debugfs_create_files(q->debugfs_dir, q, blk_mq_debugfs_queue_attrs);
 
 	/*
 	 * blk_mq_init_sched() attempted to do this already, but q->debugfs_dir
@@ -742,7 +741,7 @@ static void blk_mq_debugfs_register_ctx(struct blk_mq_hw_ctx *hctx,
 	snprintf(name, sizeof(name), "cpu%u", ctx->cpu);
 	ctx_dir = debugfs_create_dir(name, hctx->debugfs_dir);
 
-	debugfs_create_files(ctx_dir, ctx, blk_mq_debugfs_ctx_attrs);
+	blk_mq_debugfs_create_files(ctx_dir, ctx, blk_mq_debugfs_ctx_attrs);
 }
 
 void blk_mq_debugfs_register_hctx(struct request_queue *q,
@@ -755,7 +754,7 @@ void blk_mq_debugfs_register_hctx(struct request_queue *q,
 	snprintf(name, sizeof(name), "hctx%u", hctx->queue_num);
 	hctx->debugfs_dir = debugfs_create_dir(name, q->debugfs_dir);
 
-	debugfs_create_files(hctx->debugfs_dir, hctx, blk_mq_debugfs_hctx_attrs);
+	blk_mq_debugfs_create_files(hctx->debugfs_dir, hctx, blk_mq_debugfs_hctx_attrs);
 
 	hctx_for_each_ctx(hctx, ctx, i)
 		blk_mq_debugfs_register_ctx(hctx, ctx);
@@ -802,7 +801,7 @@ void blk_mq_debugfs_register_sched(struct request_queue *q)
 
 	q->sched_debugfs_dir = debugfs_create_dir("sched", q->debugfs_dir);
 
-	debugfs_create_files(q->sched_debugfs_dir, q, e->queue_debugfs_attrs);
+	blk_mq_debugfs_create_files(q->sched_debugfs_dir, q, e->queue_debugfs_attrs);
 }
 
 void blk_mq_debugfs_unregister_sched(struct request_queue *q)
@@ -811,35 +810,6 @@ void blk_mq_debugfs_unregister_sched(struct request_queue *q)
 	q->sched_debugfs_dir = NULL;
 }
 
-void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos)
-{
-	debugfs_remove_recursive(rqos->debugfs_dir);
-	rqos->debugfs_dir = NULL;
-}
-
-void blk_mq_debugfs_register_rqos(struct rq_qos *rqos)
-{
-	struct request_queue *q = rqos->q;
-
-	if (rqos->debugfs_dir || !rqos->ops->debugfs_attrs)
-		return;
-
-	if (!q->rqos_debugfs_dir)
-		q->rqos_debugfs_dir = debugfs_create_dir("rqos",
-							 q->debugfs_dir);
-
-	rqos->debugfs_dir = debugfs_create_dir(rqos->ops->name,
-					       rqos->q->rqos_debugfs_dir);
-
-	debugfs_create_files(rqos->debugfs_dir, rqos, rqos->ops->debugfs_attrs);
-}
-
-void blk_mq_debugfs_unregister_queue_rqos(struct request_queue *q)
-{
-	debugfs_remove_recursive(q->rqos_debugfs_dir);
-	q->rqos_debugfs_dir = NULL;
-}
-
 void blk_mq_debugfs_register_sched_hctx(struct request_queue *q,
 					struct blk_mq_hw_ctx *hctx)
 {
@@ -858,7 +828,7 @@ void blk_mq_debugfs_register_sched_hctx(struct request_queue *q,
 
 	hctx->sched_debugfs_dir = debugfs_create_dir("sched",
 						     hctx->debugfs_dir);
-	debugfs_create_files(hctx->sched_debugfs_dir, hctx,
+	blk_mq_debugfs_create_files(hctx->sched_debugfs_dir, hctx,
 			     e->hctx_debugfs_attrs);
 }
 
diff --git a/block/blk-mq-debugfs.h b/block/blk-mq-debugfs.h
index a68aa6041a10..ed805dca934c 100644
--- a/block/blk-mq-debugfs.h
+++ b/block/blk-mq-debugfs.h
@@ -31,10 +31,8 @@ void blk_mq_debugfs_unregister_sched(struct request_queue *q);
 void blk_mq_debugfs_register_sched_hctx(struct request_queue *q,
 				       struct blk_mq_hw_ctx *hctx);
 void blk_mq_debugfs_unregister_sched_hctx(struct blk_mq_hw_ctx *hctx);
-
-void blk_mq_debugfs_register_rqos(struct rq_qos *rqos);
-void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos);
-void blk_mq_debugfs_unregister_queue_rqos(struct request_queue *q);
+void blk_mq_debugfs_create_files(struct dentry *parent, void *data,
+				 const struct blk_mq_debugfs_attr *attr);
 #else
 static inline void blk_mq_debugfs_register(struct request_queue *q)
 {
@@ -78,15 +76,8 @@ static inline void blk_mq_debugfs_unregister_sched_hctx(struct blk_mq_hw_ctx *hc
 {
 }
 
-static inline void blk_mq_debugfs_register_rqos(struct rq_qos *rqos)
-{
-}
-
-static inline void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos)
-{
-}
-
-static inline void blk_mq_debugfs_unregister_queue_rqos(struct request_queue *q)
+void blk_mq_debugfs_create_files(struct dentry *parent, void *data,
+				 const struct blk_mq_debugfs_attr *attr)
 {
 }
 #endif
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index 236ecb2af845..0572dd584c53 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -295,6 +295,31 @@ void rq_qos_wait(struct rq_wait *rqw, void *private_data,
 	finish_wait(&rqw->wait, &data.wq);
 }
 
+#ifdef CONFIG_BLK_DEBUG_FS
+static void rq_qos_debugfs_unregister(struct rq_qos *rqos)
+{
+	debugfs_remove_recursive(rqos->debugfs_dir);
+	rqos->debugfs_dir = NULL;
+}
+
+static void rq_qos_debugfs_register(struct rq_qos *rqos)
+{
+	struct request_queue *q = rqos->q;
+
+	if (rqos->debugfs_dir || !rqos->ops->debugfs_attrs)
+		return;
+
+	if (!q->rqos_debugfs_dir)
+		q->rqos_debugfs_dir = debugfs_create_dir("rqos",
+							 q->debugfs_dir);
+
+	rqos->debugfs_dir = debugfs_create_dir(rqos->ops->name,
+					       rqos->q->rqos_debugfs_dir);
+
+	blk_mq_debugfs_create_files(rqos->debugfs_dir, rqos, rqos->ops->debugfs_attrs);
+}
+#endif
+
 void rq_qos_exit(struct request_queue *q)
 {
 	/*
@@ -306,7 +331,10 @@ void rq_qos_exit(struct request_queue *q)
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
 	}
-	blk_mq_debugfs_unregister_queue_rqos(q);
+#ifdef CONFIG_BLK_DEBUG_FS
+	debugfs_remove_recursive(q->rqos_debugfs_dir);
+	q->rqos_debugfs_dir = NULL;
+#endif
 }
 
 static struct rq_qos *rq_qos_by_name(struct request_queue *q,
@@ -383,8 +411,10 @@ void rq_qos_activate(struct request_queue *q,
 	}
 	spin_unlock_irq(&q->rqos_lock);
 
+#ifdef CONFIG_BLK_DEBUG_FS
 	if (rqos->ops->debugfs_attrs)
-		blk_mq_debugfs_register_rqos(rqos);
+		rq_qos_debugfs_register(rqos);
+#endif
 }
 
 void rq_qos_deactivate(struct rq_qos *rqos)
@@ -406,7 +436,9 @@ void rq_qos_deactivate(struct rq_qos *rqos)
 		}
 	}
 	spin_unlock_irq(&q->rqos_lock);
-	blk_mq_debugfs_unregister_rqos(rqos);
+#ifdef CONFIG_BLK_DEBUG_FS
+	rq_qos_debugfs_unregister(rqos);
+#endif
 }
 
 static struct rq_qos_ops *rq_qos_op_find(const char *name)
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 3d7df14edde3..463c64d28e8b 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -8,6 +8,7 @@
 #include <linux/atomic.h>
 #include <linux/wait.h>
 #include <linux/blk-mq.h>
+#include <linux/debugfs.h>
 
 #include "blk-mq-debugfs.h"
 
-- 
2.17.1

[RFC V5 16/16] blk/rq-qos: add config to control the whole blk-rq-qos framework

[Wang Jianchao (Kuaishou)]

This patch makes the whole blk-rq-qos framework be able to be
configured by CONFIG_BLK_RQ_QOS. There is no functional changes
in this patch.

Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>
---
 block/Kconfig          |  7 +++++++
 block/Makefile         |  3 ++-
 block/blk-mq.c         |  4 +---
 block/blk-rq-qos.c     |  1 +
 block/blk-rq-qos.h     | 16 ++++++++++++++++
 block/blk-sysfs.c      |  4 ++++
 include/linux/blkdev.h |  6 ++++--
 7 files changed, 35 insertions(+), 6 deletions(-)

diff --git a/block/Kconfig b/block/Kconfig
index d5d4197b7ed2..b9d2a2dcbe56 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -38,6 +38,9 @@ config BLK_DEV_BSG_COMMON
 config BLK_ICQ
 	bool
 
+config BLK_RQ_QOS
+	bool
+
 config BLK_DEV_BSGLIB
 	bool "Block layer SG support v4 helper lib"
 	select BLK_DEV_BSG_COMMON
@@ -99,6 +102,7 @@ config BLK_DEV_THROTTLING_LOW
 
 config BLK_WBT
 	bool "Enable support for block device writeback throttling"
+	select BLK_RQ_QOS
 	help
 	Enabling this option enables the block layer to throttle buffered
 	background writeback from the VM, making it more smooth and having
@@ -116,6 +120,7 @@ config BLK_WBT_MQ
 config BLK_CGROUP_IOLATENCY
 	bool "Enable support for latency based cgroup IO protection"
 	depends on BLK_CGROUP
+	select BLK_RQ_QOS
 	help
 	Enabling this option enables the .latency interface for IO throttling.
 	The IO controller will attempt to maintain average IO latencies below
@@ -138,6 +143,7 @@ config BLK_CGROUP_IOCOST
 	depends on BLK_CGROUP
 	select BLK_RQ_IO_DATA_LEN
 	select BLK_RQ_ALLOC_TIME
+	select BLK_RQ_QOS
 	help
 	Enabling this option enables the .weight interface for cost
 	model based proportional IO control.  The IO controller
@@ -147,6 +153,7 @@ config BLK_CGROUP_IOCOST
 config BLK_CGROUP_IOPRIO
 	bool "Cgroup I/O controller for assigning an I/O priority class"
 	depends on BLK_CGROUP
+	select BLK_RQ_QOS
 	help
 	Enable the .prio interface for assigning an I/O priority class to
 	requests. The I/O priority class affects the order in which an I/O
diff --git a/block/Makefile b/block/Makefile
index f38eaa612929..07af1b2f6fe0 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -8,9 +8,10 @@ obj-y		:= bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \
 			blk-merge.o blk-timeout.o \
 			blk-lib.o blk-mq.o blk-mq-tag.o blk-stat.o \
 			blk-mq-sysfs.o blk-mq-cpumap.o blk-mq-sched.o ioctl.o \
-			genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \
+			genhd.o ioprio.o badblocks.o partitions/ \
 			disk-events.o blk-ia-ranges.o
 
+obj-$(CONFIG_BLK_RQ_QOS) 	+= blk-rq-qos.o
 obj-$(CONFIG_BOUNCE)		+= bounce.o
 obj-$(CONFIG_BLK_DEV_BSG_COMMON) += bsg.o
 obj-$(CONFIG_BLK_DEV_BSGLIB)	+= bsg-lib.o
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 3c1cd32c72fd..9780bfb25046 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1248,12 +1248,10 @@ EXPORT_SYMBOL(blk_execute_rq);
 
 static void __blk_mq_requeue_request(struct request *rq)
 {
-	struct request_queue *q = rq->q;
-
 	blk_mq_put_driver_tag(rq);
 
 	trace_block_rq_requeue(rq);
-	rq_qos_requeue(q, rq);
+	rq_qos_requeue(rq->q, rq);
 
 	if (blk_mq_request_started(rq)) {
 		WRITE_ONCE(rq->state, MQ_RQ_IDLE);
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index 0572dd584c53..92df1dca74cc 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 
+#include <linux/debugfs.h>
 #include "blk-rq-qos.h"
 
 static DEFINE_IDR(rq_qos_idr);
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 463c64d28e8b..dc7071139247 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -2,6 +2,8 @@
 #ifndef RQ_QOS_H
 #define RQ_QOS_H
 
+#ifdef CONFIG_BLK_RQ_QOS
+
 #include <linux/kernel.h>
 #include <linux/blkdev.h>
 #include <linux/blk_types.h>
@@ -167,4 +169,18 @@ static inline void rq_qos_setting_changed(struct request_queue *q)
 
 void rq_qos_exit(struct request_queue *);
 
+#else
+
+#define rq_qos_cleanup(q, bio)
+#define rq_qos_done(q, rq)
+#define rq_qos_issue(q, rq)
+#define rq_qos_requeue(q, rq)
+#define rq_qos_done_bio(q, bio)
+#define rq_qos_throttle(q, bio)
+#define rq_qos_track(q, rq, bio)
+#define rq_qos_merge(q, rq, bio)
+#define rq_qos_setting_changed(q)
+#define rq_qos_exit(q)
+
+#endif /* CONFIG_BLK_RQ_QOS */
 #endif
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 4563283dae07..784b4d8416d5 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -503,7 +503,9 @@ QUEUE_RO_ENTRY(queue_max_segments, "max_segments");
 QUEUE_RO_ENTRY(queue_max_integrity_segments, "max_integrity_segments");
 QUEUE_RO_ENTRY(queue_max_segment_size, "max_segment_size");
 QUEUE_RW_ENTRY(elv_iosched, "scheduler");
+#ifdef CONFIG_BLK_RQ_QOS
 QUEUE_RW_ENTRY(queue_qos, "qos");
+#endif
 
 QUEUE_RO_ENTRY(queue_logical_block_size, "logical_block_size");
 QUEUE_RO_ENTRY(queue_physical_block_size, "physical_block_size");
@@ -562,7 +564,9 @@ static struct attribute *queue_attrs[] = {
 	&queue_max_integrity_segments_entry.attr,
 	&queue_max_segment_size_entry.attr,
 	&elv_iosched_entry.attr,
+#ifdef CONFIG_BLK_RQ_QOS
 	&queue_qos_entry.attr,
+#endif
 	&queue_hw_sector_size_entry.attr,
 	&queue_logical_block_size_entry.attr,
 	&queue_physical_block_size_entry.attr,
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 126aac824ccc..8d6dff40fe73 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -194,8 +194,9 @@ struct request_queue {
 	struct percpu_ref	q_usage_counter;
 
 	struct blk_queue_stats	*stats;
+#ifdef CONFIG_BLK_RQ_QOS
 	struct rq_qos		*rq_qos;
-
+#endif
 	const struct blk_mq_ops	*mq_ops;
 
 	/* sw queues */
@@ -230,8 +231,9 @@ struct request_queue {
 	int			id;
 
 	spinlock_t		queue_lock;
+#ifdef CONFIG_BLK_RQ_QOS
 	spinlock_t		rqos_lock;
-
+#endif
 	struct gendisk		*disk;
 
 	/*
-- 
2.17.1

Re: [RFC V5 0/16] blk: blk-rq-qos framework refactor and cleanup

[Christoph Hellwig]

On Thu, Feb 24, 2022 at 05:06:38PM +0800, Wang Jianchao (Kuaishou) wrote:
> Hi Jens
> 
> This patchset is working on our v4.18 kernel to provide the capacity
> for upgrade blk-rq-qos policies on the fly w/o needing to restart
> machine and applications.

Why would you upgrade kernel code at runtime?  And how is what you do
on some randm old kernel relevant for mainline?

Re: [RFC V5 01/16] blk: make the whole blk_mq_submit_bio under q_usage_counter

[Christoph Hellwig]

On Thu, Feb 24, 2022 at 05:06:39PM +0800, Wang Jianchao (Kuaishou) wrote:
> This is to protect the rqos list against the rqos open/close. We
> need to drain all of the caller of blk_mq_submit_bio() before
> we can operate the rqos list.

This means ever I/O now has to d an extra refcount roundtrip.

Re: [RFC V5 02/16] blk/rq-qos: prepare to make blk-rq-qos pluggable

[Christoph Hellwig]

On Thu, Feb 24, 2022 at 05:06:40PM +0800, Wang Jianchao (Kuaishou) wrote:
> This patch makes blk-rq-qos policies pluggable as following,
> (1) Add code to maintain the rq_qos_ops. A rq-qos policy need to
>     register itself with rq_qos_register(). The original enum
>     rq_qos_id will be removed in following patch. They will use
>     a dynamic id maintained by rq_qos_ida.
> (2) Add .init callback into rq_qos_ops. We use it to initialize the
>     resource.
> (3) Add /sys/block/x/queue/qos
>     We can use '+name' or "-name" to open or close the blk-rq-qos
>     policy.

This sunds like most of these should be separate patches.

>  {
>  	struct request_queue *q = rqos->q;
> -	const char *dir_name = rq_qos_id_to_name(rqos->id);
> +	const char *dir_name;
> +
> +	if (rqos->ops->name)
> +		dir_name = rqos->ops->name;
> +	else
> +		dir_name = rq_qos_id_to_name(rqos->id);

Plase split out a patch to add the name to all the ops and remove
rq_qos_id_to_name, which can be at the beginning of the series.

> +	spin_lock_irq(&q->rqos_lock);
> +	pos = q->rq_qos;
> +	if (pos) {
> +		while (pos->next)
> +			pos = pos->next;
> +		pos->next = rqos;
> +	} else {
> +		q->rq_qos = rqos;
> +	}

I think another really useful prep patch would be to switch the queue
linkage to use a hlist_head instead of all this open coded list
manipulation.

> +	rqos = rq_qos_by_name(q, qosname);
> +	if ((buf[0] == '+' && rqos)) {
> +		ret = -EEXIST;
> +		goto out;
> +	}
> +
> +	if ((buf[0] == '-' && !rqos)) {

These two conditionals have pretty odd extra braces.

Re: [RFC V5 10/16] blk-wbt: get rid of wbt_enable/disable_default()

[Christoph Hellwig]

On Thu, Feb 24, 2022 at 05:06:48PM +0800, Wang Jianchao (Kuaishou) wrote:
> wbt_enable_default() is used to open wbt feature by default.
> wbt_disable_default() is only employed by bfq which has similar
> writeback throttling mechanism. This kind of policy should be
> selected by the user instead of being set in kernel. This patch
> get rid of the interfaces and let the user configure appropricate
> blk-rq-qos policy by themselves.

This drastically changes the user visible behavior.

Re: [RFC V5 02/16] blk/rq-qos: prepare to make blk-rq-qos pluggable

[Tejun Heo]

On Thu, Feb 24, 2022 at 05:06:40PM +0800, Wang Jianchao (Kuaishou) wrote:
> This patch makes blk-rq-qos policies pluggable as following,
> (1) Add code to maintain the rq_qos_ops. A rq-qos policy need to
>     register itself with rq_qos_register(). The original enum
>     rq_qos_id will be removed in following patch. They will use
>     a dynamic id maintained by rq_qos_ida.
> (2) Add .init callback into rq_qos_ops. We use it to initialize the
>     resource.
> (3) Add /sys/block/x/queue/qos
>     We can use '+name' or "-name" to open or close the blk-rq-qos
>     policy.
> 
> This patch mainly prepare help interfaces and no functional changes.
> Following patches will adpat the code of wbt, iolatency, iocost and
> ioprio to make them pluggable one by one. And after that, the sysfs
> interface /sys/block/xxx/queue/qos will be exported.
> 
> Signed-off-by: Wang Jianchao (Kuaishou) <jianchao.wan9@gmail.com>

As discussed before, the addition of new interface and custom module loading
mechanism is unnecessary and undesirable.

Nacked-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun

Re: [RFC V5 10/16] blk-wbt: get rid of wbt_enable/disable_default()

[Tejun Heo]

On Thu, Feb 24, 2022 at 05:06:48PM +0800, Wang Jianchao (Kuaishou) wrote:
> wbt_enable_default() is used to open wbt feature by default.
> wbt_disable_default() is only employed by bfq which has similar
> writeback throttling mechanism. This kind of policy should be
> selected by the user instead of being set in kernel. This patch
> get rid of the interfaces and let the user configure appropricate
> blk-rq-qos policy by themselves.

Some combinations simply don't work well together. The kernel should be able
to provide a reasonble default behavior.

Nacked-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun

Re: [RFC V5 01/16] blk: make the whole blk_mq_submit_bio under q_usage_counter

[Wang Jianchao]

On 2022/2/24 9:38 下午, Christoph Hellwig wrote:
> On Thu, Feb 24, 2022 at 05:06:39PM +0800, Wang Jianchao (Kuaishou) wrote:
>> This is to protect the rqos list against the rqos open/close. We
>> need to drain all of the caller of blk_mq_submit_bio() before
>> we can operate the rqos list.
> 
> This means ever I/O now has to d an extra refcount roundtrip.

If we want to save cpu cycles for the very fast device, why not introduce
another blk_mq_submit_bio_fast() which can reduce the code drastically.

Thanks
Jianchao