[PATCH 1/2] blk-iocost: Add iocg idle state tracepoint

[PATCH 1/2] blk-iocost: Add iocg idle state tracepoint

[Baolin Wang]

It will be helpful to trace the iocg's whole state, including active and
idle state. And we can easily expand the original iocost_iocg_activate
trace event to support a state trace class, including active and idle
state tracing.

Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
---
 block/blk-iocost.c            |  3 +++
 include/trace/events/iocost.h | 16 +++++++++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index ffa418c..ac6078a 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2185,6 +2185,9 @@ static int ioc_check_iocgs(struct ioc *ioc, struct ioc_now *now)
 							    WEIGHT_ONE);
 			}
 
+			TRACE_IOCG_PATH(iocg_idle, iocg, now,
+					atomic64_read(&iocg->active_period),
+					atomic64_read(&ioc->cur_period), vtime);
 			__propagate_weights(iocg, 0, 0, false, now);
 			list_del_init(&iocg->active_list);
 		}
diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h
index 0b68699..e282ce0 100644
--- a/include/trace/events/iocost.h
+++ b/include/trace/events/iocost.h
@@ -11,7 +11,7 @@
 
 #include <linux/tracepoint.h>
 
-TRACE_EVENT(iocost_iocg_activate,
+DECLARE_EVENT_CLASS(iocost_iocg_state,
 
 	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
 		u64 last_period, u64 cur_period, u64 vtime),
@@ -59,6 +59,20 @@
 	)
 );
 
+DEFINE_EVENT(iocost_iocg_state, iocost_iocg_activate,
+	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
+		 u64 last_period, u64 cur_period, u64 vtime),
+
+	TP_ARGS(iocg, path, now, last_period, cur_period, vtime)
+);
+
+DEFINE_EVENT(iocost_iocg_state, iocost_iocg_idle,
+	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
+		 u64 last_period, u64 cur_period, u64 vtime),
+
+	TP_ARGS(iocg, path, now, last_period, cur_period, vtime)
+);
+
 DECLARE_EVENT_CLASS(iocg_inuse_update,
 
 	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
-- 
1.8.3.1

[PATCH 2/2] blk-iocost: Use alloc_percpu_gfp() to simplify the code

[Baolin Wang]

Use alloc_percpu_gfp() with __GFP_ZERO flag, which can remove
some explicit initialization code.

Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
---
 block/blk-iocost.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index ac6078a..52ce2e3 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2819,28 +2819,19 @@ static int blk_iocost_init(struct request_queue *q)
 {
 	struct ioc *ioc;
 	struct rq_qos *rqos;
-	int i, cpu, ret;
+	int ret;
+	gfp_t gfp = GFP_KERNEL | __GFP_ZERO;
 
 	ioc = kzalloc(sizeof(*ioc), GFP_KERNEL);
 	if (!ioc)
 		return -ENOMEM;
 
-	ioc->pcpu_stat = alloc_percpu(struct ioc_pcpu_stat);
+	ioc->pcpu_stat = alloc_percpu_gfp(struct ioc_pcpu_stat, gfp);
 	if (!ioc->pcpu_stat) {
 		kfree(ioc);
 		return -ENOMEM;
 	}
 
-	for_each_possible_cpu(cpu) {
-		struct ioc_pcpu_stat *ccs = per_cpu_ptr(ioc->pcpu_stat, cpu);
-
-		for (i = 0; i < ARRAY_SIZE(ccs->missed); i++) {
-			local_set(&ccs->missed[i].nr_met, 0);
-			local_set(&ccs->missed[i].nr_missed, 0);
-		}
-		local64_set(&ccs->rq_wait_ns, 0);
-	}
-
 	rqos = &ioc->rqos;
 	rqos->id = RQ_QOS_COST;
 	rqos->ops = &ioc_rqos_ops;
-- 
1.8.3.1

Re: [PATCH 2/2] blk-iocost: Use alloc_percpu_gfp() to simplify the code

[Tejun Heo]

Hello,

On Thu, Dec 10, 2020 at 06:56:45PM +0800, Baolin Wang wrote:
> Use alloc_percpu_gfp() with __GFP_ZERO flag, which can remove
> some explicit initialization code.

__GFP_ZERO is implicit for percpu allocations and local[64]_t's initial
states aren't guaranteed to be all zeros on different archs.

Thanks.

-- 
tejun

Re: [PATCH 1/2] blk-iocost: Add iocg idle state tracepoint

[Tejun Heo]

On Thu, Dec 10, 2020 at 06:56:44PM +0800, Baolin Wang wrote:
> It will be helpful to trace the iocg's whole state, including active and
> idle state. And we can easily expand the original iocost_iocg_activate
> trace event to support a state trace class, including active and idle
> state tracing.
> 
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun

Re: [PATCH 2/2] blk-iocost: Use alloc_percpu_gfp() to simplify the code

[Baolin Wang]

Hi Tejun,

> Hello,
> 
> On Thu, Dec 10, 2020 at 06:56:45PM +0800, Baolin Wang wrote:
>> Use alloc_percpu_gfp() with __GFP_ZERO flag, which can remove
>> some explicit initialization code.
> 
> __GFP_ZERO is implicit for percpu allocations and local[64]_t's initial
> states aren't guaranteed to be all zeros on different archs.

Thanks for teaching me this, at least I did not get this from the 
local_ops Documentation before. Just out of curiosity, these local[64]_t 
variables are also allocated from budy allocator ultimately, why they 
can not be initialized to zeros on some ARCHs with __GFP_ZERO? Could you 
elaborate on about this restriction? Thanks.

By the way, seems the kyber-iosched has the same issue, since the 
'struct kyber_cpu_latency' also contains an atomic_t variable.

	kqd->cpu_latency = alloc_percpu_gfp(struct kyber_cpu_latency,
					    GFP_KERNEL | __GFP_ZERO);
	if (!kqd->cpu_latency)
		goto err_kqd;

Re: [PATCH 2/2] blk-iocost: Use alloc_percpu_gfp() to simplify the code

[Tejun Heo]

Hello,

On Fri, Dec 11, 2020 at 03:13:29PM +0800, Baolin Wang wrote:
> Thanks for teaching me this, at least I did not get this from the local_ops
> Documentation before. Just out of curiosity, these local[64]_t variables are
> also allocated from budy allocator ultimately, why they can not be
> initialized to zeros on some ARCHs with __GFP_ZERO? Could you elaborate on
> about this restriction? Thanks.

* It's highly unlikely but theoretically possible that some arch might need
  more than raw value to implement local semantics.

* People might wanna add debug annotations which may require more than just
  the raw value.

> By the way, seems the kyber-iosched has the same issue, since the 'struct
> kyber_cpu_latency' also contains an atomic_t variable.
> 
> 	kqd->cpu_latency = alloc_percpu_gfp(struct kyber_cpu_latency,
> 					    GFP_KERNEL | __GFP_ZERO);
> 	if (!kqd->cpu_latency)
> 		goto err_kqd;

Yeah, the lines become blurry when all existing usages are fine with zeroing
and we do end up needing to clean up those when the need arises (e.g. we
used to zero some spinlocks too). It's just a better form to stick with
initializers when they are provided.

Thanks.

-- 
tejun

Re: [PATCH 2/2] blk-iocost: Use alloc_percpu_gfp() to simplify the code

[Baolin Wang]

Hi Tejun,

> Hello,
> 
> On Fri, Dec 11, 2020 at 03:13:29PM +0800, Baolin Wang wrote:
>> Thanks for teaching me this, at least I did not get this from the local_ops
>> Documentation before. Just out of curiosity, these local[64]_t variables are
>> also allocated from budy allocator ultimately, why they can not be
>> initialized to zeros on some ARCHs with __GFP_ZERO? Could you elaborate on
>> about this restriction? Thanks.
> 
> * It's highly unlikely but theoretically possible that some arch might need
>    more than raw value to implement local semantics.
> 
> * People might wanna add debug annotations which may require more than just
>    the raw value.

Thanks for your explanation. It will be helpful to add these comments 
for the code in case someone else wants to do the same thing like this 
patch in future. I can send a patch to add these comments if you have no 
objection.

>> By the way, seems the kyber-iosched has the same issue, since the 'struct
>> kyber_cpu_latency' also contains an atomic_t variable.
>>
>> 	kqd->cpu_latency = alloc_percpu_gfp(struct kyber_cpu_latency,
>> 					    GFP_KERNEL | __GFP_ZERO);
>> 	if (!kqd->cpu_latency)
>> 		goto err_kqd;
> 
> Yeah, the lines become blurry when all existing usages are fine with zeroing
> and we do end up needing to clean up those when the need arises (e.g. we
> used to zero some spinlocks too). It's just a better form to stick with
> initializers when they are provided.

OK. Sounds it is worth sending a patch to initialize this structure 
explicitly to keep consistent.

Re: [PATCH 1/2] blk-iocost: Add iocg idle state tracepoint

[Baolin Wang]

Hi Jens,

> It will be helpful to trace the iocg's whole state, including active and
> idle state. And we can easily expand the original iocost_iocg_activate
> trace event to support a state trace class, including active and idle
> state tracing.
> 
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Could you pick up patch 1 which was already acked by Tejun. Thanks.

> ---
>   block/blk-iocost.c            |  3 +++
>   include/trace/events/iocost.h | 16 +++++++++++++++-
>   2 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/block/blk-iocost.c b/block/blk-iocost.c
> index ffa418c..ac6078a 100644
> --- a/block/blk-iocost.c
> +++ b/block/blk-iocost.c
> @@ -2185,6 +2185,9 @@ static int ioc_check_iocgs(struct ioc *ioc, struct ioc_now *now)
>   							    WEIGHT_ONE);
>   			}
>   
> +			TRACE_IOCG_PATH(iocg_idle, iocg, now,
> +					atomic64_read(&iocg->active_period),
> +					atomic64_read(&ioc->cur_period), vtime);
>   			__propagate_weights(iocg, 0, 0, false, now);
>   			list_del_init(&iocg->active_list);
>   		}
> diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h
> index 0b68699..e282ce0 100644
> --- a/include/trace/events/iocost.h
> +++ b/include/trace/events/iocost.h
> @@ -11,7 +11,7 @@
>   
>   #include <linux/tracepoint.h>
>   
> -TRACE_EVENT(iocost_iocg_activate,
> +DECLARE_EVENT_CLASS(iocost_iocg_state,
>   
>   	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
>   		u64 last_period, u64 cur_period, u64 vtime),
> @@ -59,6 +59,20 @@
>   	)
>   );
>   
> +DEFINE_EVENT(iocost_iocg_state, iocost_iocg_activate,
> +	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
> +		 u64 last_period, u64 cur_period, u64 vtime),
> +
> +	TP_ARGS(iocg, path, now, last_period, cur_period, vtime)
> +);
> +
> +DEFINE_EVENT(iocost_iocg_state, iocost_iocg_idle,
> +	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
> +		 u64 last_period, u64 cur_period, u64 vtime),
> +
> +	TP_ARGS(iocg, path, now, last_period, cur_period, vtime)
> +);
> +
>   DECLARE_EVENT_CLASS(iocg_inuse_update,
>   
>   	TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
> 

Re: [PATCH 1/2] blk-iocost: Add iocg idle state tracepoint

[Jens Axboe]

On 12/17/20 12:57 AM, Baolin Wang wrote:
> Hi Jens,
> 
>> It will be helpful to trace the iocg's whole state, including active and
>> idle state. And we can easily expand the original iocost_iocg_activate
>> trace event to support a state trace class, including active and idle
>> state tracing.
>>
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> 
> Could you pick up patch 1 which was already acked by Tejun. Thanks.

Done, thanks.

-- 
Jens Axboe