[PATCH 0/4] ceph: add IO size metric support

[PATCH 0/4] ceph: add IO size metric support

[xiubli]

From: Xiubo Li <xiubli@redhat.com>

Currently it will show as the following:

item          total       avg_sz(bytes)   min_sz(bytes)   max_sz(bytes)  total_sz(bytes)
----------------------------------------------------------------------------------------
read          1           10240           10240           10240           10240
write         1           10240           10240           10240           10240



Xiubo Li (4):
  ceph: rename the metric helpers
  ceph: update the __update_latency helper
  ceph: avoid count the same request twice or more
  ceph: add IO size metrics support

 fs/ceph/addr.c       |  20 +++----
 fs/ceph/debugfs.c    |  49 +++++++++++++----
 fs/ceph/file.c       |  47 ++++++++--------
 fs/ceph/mds_client.c |   2 +-
 fs/ceph/metric.c     | 126 ++++++++++++++++++++++++++++++++-----------
 fs/ceph/metric.h     |  22 +++++---
 6 files changed, 184 insertions(+), 82 deletions(-)

-- 
2.27.0

[PATCH 1/4] ceph: rename the metric helpers

[xiubli]

From: Xiubo Li <xiubli@redhat.com>

Prepare for the size metrics patches followed.

URL: https://tracker.ceph.com/issues/49913
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
 fs/ceph/addr.c       |  8 ++++----
 fs/ceph/debugfs.c    | 12 ++++++------
 fs/ceph/file.c       | 12 ++++++------
 fs/ceph/mds_client.c |  2 +-
 fs/ceph/metric.c     | 24 ++++++++++++------------
 fs/ceph/metric.h     | 12 ++++++------
 6 files changed, 35 insertions(+), 35 deletions(-)

diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index b476133353ae..7c2802758d0e 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -226,7 +226,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
 	int num_pages;
 	int err = req->r_result;
 
-	ceph_update_read_latency(&fsc->mdsc->metric, req->r_start_latency,
+	ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
 				 req->r_end_latency, err);
 
 	dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
@@ -560,7 +560,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
 	if (!err)
 		err = ceph_osdc_wait_request(osdc, req);
 
-	ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
 				  req->r_end_latency, err);
 
 	ceph_osdc_put_request(req);
@@ -648,7 +648,7 @@ static void writepages_finish(struct ceph_osd_request *req)
 		ceph_clear_error_write(ci);
 	}
 
-	ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
 				  req->r_end_latency, rc);
 
 	/*
@@ -1719,7 +1719,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
 	if (!err)
 		err = ceph_osdc_wait_request(&fsc->client->osdc, req);
 
-	ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
 				  req->r_end_latency, err);
 
 out_put:
diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
index 66989c880adb..425f3356332a 100644
--- a/fs/ceph/debugfs.c
+++ b/fs/ceph/debugfs.c
@@ -162,34 +162,34 @@ static int metric_show(struct seq_file *s, void *p)
 	seq_printf(s, "item          total       avg_lat(us)     min_lat(us)     max_lat(us)     stdev(us)\n");
 	seq_printf(s, "-----------------------------------------------------------------------------------\n");
 
-	spin_lock(&m->read_latency_lock);
+	spin_lock(&m->read_metric_lock);
 	total = m->total_reads;
 	sum = m->read_latency_sum;
 	avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
 	min = m->read_latency_min;
 	max = m->read_latency_max;
 	sq = m->read_latency_sq_sum;
-	spin_unlock(&m->read_latency_lock);
+	spin_unlock(&m->read_metric_lock);
 	CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
 
-	spin_lock(&m->write_latency_lock);
+	spin_lock(&m->write_metric_lock);
 	total = m->total_writes;
 	sum = m->write_latency_sum;
 	avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
 	min = m->write_latency_min;
 	max = m->write_latency_max;
 	sq = m->write_latency_sq_sum;
-	spin_unlock(&m->write_latency_lock);
+	spin_unlock(&m->write_metric_lock);
 	CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
 
-	spin_lock(&m->metadata_latency_lock);
+	spin_lock(&m->metadata_metric_lock);
 	total = m->total_metadatas;
 	sum = m->metadata_latency_sum;
 	avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
 	min = m->metadata_latency_min;
 	max = m->metadata_latency_max;
 	sq = m->metadata_latency_sq_sum;
-	spin_unlock(&m->metadata_latency_lock);
+	spin_unlock(&m->metadata_metric_lock);
 	CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
 
 	seq_printf(s, "\n");
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index a6ef1d143308..a27aabcb0e0b 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -895,7 +895,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
 		if (!ret)
 			ret = ceph_osdc_wait_request(osdc, req);
 
-		ceph_update_read_latency(&fsc->mdsc->metric,
+		ceph_update_read_metrics(&fsc->mdsc->metric,
 					 req->r_start_latency,
 					 req->r_end_latency,
 					 ret);
@@ -1040,10 +1040,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
 	/* r_start_latency == 0 means the request was not submitted */
 	if (req->r_start_latency) {
 		if (aio_req->write)
-			ceph_update_write_latency(metric, req->r_start_latency,
+			ceph_update_write_metrics(metric, req->r_start_latency,
 						  req->r_end_latency, rc);
 		else
-			ceph_update_read_latency(metric, req->r_start_latency,
+			ceph_update_read_metrics(metric, req->r_start_latency,
 						 req->r_end_latency, rc);
 	}
 
@@ -1293,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
 			ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
 
 		if (write)
-			ceph_update_write_latency(metric, req->r_start_latency,
+			ceph_update_write_metrics(metric, req->r_start_latency,
 						  req->r_end_latency, ret);
 		else
-			ceph_update_read_latency(metric, req->r_start_latency,
+			ceph_update_read_metrics(metric, req->r_start_latency,
 						 req->r_end_latency, ret);
 
 		size = i_size_read(inode);
@@ -1470,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 		if (!ret)
 			ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
 
-		ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+		ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
 					  req->r_end_latency, ret);
 out:
 		ceph_osdc_put_request(req);
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index d87bd852ed96..73ecb7d128c9 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -3306,7 +3306,7 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg)
 	/* kick calling process */
 	complete_request(mdsc, req);
 
-	ceph_update_metadata_latency(&mdsc->metric, req->r_start_latency,
+	ceph_update_metadata_metrics(&mdsc->metric, req->r_start_latency,
 				     req->r_end_latency, err);
 out:
 	ceph_mdsc_put_request(req);
diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
index 5ec94bd4c1de..75d309f2fb0c 100644
--- a/fs/ceph/metric.c
+++ b/fs/ceph/metric.c
@@ -183,21 +183,21 @@ int ceph_metric_init(struct ceph_client_metric *m)
 	if (ret)
 		goto err_i_caps_mis;
 
-	spin_lock_init(&m->read_latency_lock);
+	spin_lock_init(&m->read_metric_lock);
 	m->read_latency_sq_sum = 0;
 	m->read_latency_min = KTIME_MAX;
 	m->read_latency_max = 0;
 	m->total_reads = 0;
 	m->read_latency_sum = 0;
 
-	spin_lock_init(&m->write_latency_lock);
+	spin_lock_init(&m->write_metric_lock);
 	m->write_latency_sq_sum = 0;
 	m->write_latency_min = KTIME_MAX;
 	m->write_latency_max = 0;
 	m->total_writes = 0;
 	m->write_latency_sum = 0;
 
-	spin_lock_init(&m->metadata_latency_lock);
+	spin_lock_init(&m->metadata_metric_lock);
 	m->metadata_latency_sq_sum = 0;
 	m->metadata_latency_min = KTIME_MAX;
 	m->metadata_latency_max = 0;
@@ -274,7 +274,7 @@ static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
 	*sq_sump += sq;
 }
 
-void ceph_update_read_latency(struct ceph_client_metric *m,
+void ceph_update_read_metrics(struct ceph_client_metric *m,
 			      ktime_t r_start, ktime_t r_end,
 			      int rc)
 {
@@ -283,14 +283,14 @@ void ceph_update_read_latency(struct ceph_client_metric *m,
 	if (unlikely(rc < 0 && rc != -ENOENT && rc != -ETIMEDOUT))
 		return;
 
-	spin_lock(&m->read_latency_lock);
+	spin_lock(&m->read_metric_lock);
 	__update_latency(&m->total_reads, &m->read_latency_sum,
 			 &m->read_latency_min, &m->read_latency_max,
 			 &m->read_latency_sq_sum, lat);
-	spin_unlock(&m->read_latency_lock);
+	spin_unlock(&m->read_metric_lock);
 }
 
-void ceph_update_write_latency(struct ceph_client_metric *m,
+void ceph_update_write_metrics(struct ceph_client_metric *m,
 			       ktime_t r_start, ktime_t r_end,
 			       int rc)
 {
@@ -299,14 +299,14 @@ void ceph_update_write_latency(struct ceph_client_metric *m,
 	if (unlikely(rc && rc != -ETIMEDOUT))
 		return;
 
-	spin_lock(&m->write_latency_lock);
+	spin_lock(&m->write_metric_lock);
 	__update_latency(&m->total_writes, &m->write_latency_sum,
 			 &m->write_latency_min, &m->write_latency_max,
 			 &m->write_latency_sq_sum, lat);
-	spin_unlock(&m->write_latency_lock);
+	spin_unlock(&m->write_metric_lock);
 }
 
-void ceph_update_metadata_latency(struct ceph_client_metric *m,
+void ceph_update_metadata_metrics(struct ceph_client_metric *m,
 				  ktime_t r_start, ktime_t r_end,
 				  int rc)
 {
@@ -315,9 +315,9 @@ void ceph_update_metadata_latency(struct ceph_client_metric *m,
 	if (unlikely(rc && rc != -ENOENT))
 		return;
 
-	spin_lock(&m->metadata_latency_lock);
+	spin_lock(&m->metadata_metric_lock);
 	__update_latency(&m->total_metadatas, &m->metadata_latency_sum,
 			 &m->metadata_latency_min, &m->metadata_latency_max,
 			 &m->metadata_latency_sq_sum, lat);
-	spin_unlock(&m->metadata_latency_lock);
+	spin_unlock(&m->metadata_metric_lock);
 }
diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
index af6038ff39d4..57b5f0ec38be 100644
--- a/fs/ceph/metric.h
+++ b/fs/ceph/metric.h
@@ -108,21 +108,21 @@ struct ceph_client_metric {
 	struct percpu_counter i_caps_hit;
 	struct percpu_counter i_caps_mis;
 
-	spinlock_t read_latency_lock;
+	spinlock_t read_metric_lock;
 	u64 total_reads;
 	ktime_t read_latency_sum;
 	ktime_t read_latency_sq_sum;
 	ktime_t read_latency_min;
 	ktime_t read_latency_max;
 
-	spinlock_t write_latency_lock;
+	spinlock_t write_metric_lock;
 	u64 total_writes;
 	ktime_t write_latency_sum;
 	ktime_t write_latency_sq_sum;
 	ktime_t write_latency_min;
 	ktime_t write_latency_max;
 
-	spinlock_t metadata_latency_lock;
+	spinlock_t metadata_metric_lock;
 	u64 total_metadatas;
 	ktime_t metadata_latency_sum;
 	ktime_t metadata_latency_sq_sum;
@@ -162,13 +162,13 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
 	percpu_counter_inc(&m->i_caps_mis);
 }
 
-extern void ceph_update_read_latency(struct ceph_client_metric *m,
+extern void ceph_update_read_metrics(struct ceph_client_metric *m,
 				     ktime_t r_start, ktime_t r_end,
 				     int rc);
-extern void ceph_update_write_latency(struct ceph_client_metric *m,
+extern void ceph_update_write_metrics(struct ceph_client_metric *m,
 				      ktime_t r_start, ktime_t r_end,
 				      int rc);
-extern void ceph_update_metadata_latency(struct ceph_client_metric *m,
+extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
 				         ktime_t r_start, ktime_t r_end,
 					 int rc);
 #endif /* _FS_CEPH_MDS_METRIC_H */
-- 
2.27.0

[PATCH 2/4] ceph: update the __update_latency helper

[xiubli]

From: Xiubo Li <xiubli@redhat.com>

Let the __update_latency() helper choose the correcsponding members
according to the metric_type.

URL: https://tracker.ceph.com/issues/49913
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
 fs/ceph/metric.c | 58 +++++++++++++++++++++++++++++++++++-------------
 1 file changed, 42 insertions(+), 16 deletions(-)

diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
index 75d309f2fb0c..d5560ff99a9d 100644
--- a/fs/ceph/metric.c
+++ b/fs/ceph/metric.c
@@ -249,19 +249,51 @@ void ceph_metric_destroy(struct ceph_client_metric *m)
 		ceph_put_mds_session(m->session);
 }
 
-static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
-				    ktime_t *min, ktime_t *max,
-				    ktime_t *sq_sump, ktime_t lat)
+typedef enum {
+	CEPH_METRIC_READ,
+	CEPH_METRIC_WRITE,
+	CEPH_METRIC_METADATA,
+} metric_type;
+
+static inline void __update_latency(struct ceph_client_metric *m,
+				    metric_type type, ktime_t lat)
 {
+	ktime_t *totalp, *minp, *maxp, *lsump, *sq_sump;
 	ktime_t total, avg, sq, lsum;
 
+	switch (type) {
+	case CEPH_METRIC_READ:
+		totalp = &m->total_reads;
+		lsump = &m->read_latency_sum;
+		minp = &m->read_latency_min;
+		maxp = &m->read_latency_max;
+		sq_sump = &m->read_latency_sq_sum;
+		break;
+	case CEPH_METRIC_WRITE:
+		totalp = &m->total_writes;
+		lsump = &m->write_latency_sum;
+		minp = &m->write_latency_min;
+		maxp = &m->write_latency_max;
+		sq_sump = &m->write_latency_sq_sum;
+		break;
+	case CEPH_METRIC_METADATA:
+		totalp = &m->total_metadatas;
+		lsump = &m->metadata_latency_sum;
+		minp = &m->metadata_latency_min;
+		maxp = &m->metadata_latency_max;
+		sq_sump = &m->metadata_latency_sq_sum;
+		break;
+	default:
+		return;
+	}
+
 	total = ++(*totalp);
 	lsum = (*lsump += lat);
 
-	if (unlikely(lat < *min))
-		*min = lat;
-	if (unlikely(lat > *max))
-		*max = lat;
+	if (unlikely(lat < *minp))
+		*minp = lat;
+	if (unlikely(lat > *maxp))
+		*maxp = lat;
 
 	if (unlikely(total == 1))
 		return;
@@ -284,9 +316,7 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
 		return;
 
 	spin_lock(&m->read_metric_lock);
-	__update_latency(&m->total_reads, &m->read_latency_sum,
-			 &m->read_latency_min, &m->read_latency_max,
-			 &m->read_latency_sq_sum, lat);
+	__update_latency(m, CEPH_METRIC_READ, lat);
 	spin_unlock(&m->read_metric_lock);
 }
 
@@ -300,9 +330,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
 		return;
 
 	spin_lock(&m->write_metric_lock);
-	__update_latency(&m->total_writes, &m->write_latency_sum,
-			 &m->write_latency_min, &m->write_latency_max,
-			 &m->write_latency_sq_sum, lat);
+	__update_latency(m, CEPH_METRIC_WRITE, lat);
 	spin_unlock(&m->write_metric_lock);
 }
 
@@ -316,8 +344,6 @@ void ceph_update_metadata_metrics(struct ceph_client_metric *m,
 		return;
 
 	spin_lock(&m->metadata_metric_lock);
-	__update_latency(&m->total_metadatas, &m->metadata_latency_sum,
-			 &m->metadata_latency_min, &m->metadata_latency_max,
-			 &m->metadata_latency_sq_sum, lat);
+	__update_latency(m, CEPH_METRIC_METADATA, lat);
 	spin_unlock(&m->metadata_metric_lock);
 }
-- 
2.27.0

[PATCH 3/4] ceph: avoid count the same request twice or more

[xiubli]

From: Xiubo Li <xiubli@redhat.com>

If the request will retry, skip updating the latency metric.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
 fs/ceph/file.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index a27aabcb0e0b..31542eac7e59 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -1037,16 +1037,6 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
 	dout("ceph_aio_complete_req %p rc %d bytes %u\n",
 	     inode, rc, osd_data->bvec_pos.iter.bi_size);
 
-	/* r_start_latency == 0 means the request was not submitted */
-	if (req->r_start_latency) {
-		if (aio_req->write)
-			ceph_update_write_metrics(metric, req->r_start_latency,
-						  req->r_end_latency, rc);
-		else
-			ceph_update_read_metrics(metric, req->r_start_latency,
-						 req->r_end_latency, rc);
-	}
-
 	if (rc == -EOLDSNAPC) {
 		struct ceph_aio_work *aio_work;
 		BUG_ON(!aio_req->write);
@@ -1089,6 +1079,16 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
 		}
 	}
 
+	/* r_start_latency == 0 means the request was not submitted */
+	if (req->r_start_latency) {
+		if (aio_req->write)
+			ceph_update_write_metrics(metric, req->r_start_latency,
+						  req->r_end_latency, rc);
+		else
+			ceph_update_read_metrics(metric, req->r_start_latency,
+						 req->r_end_latency, rc);
+	}
+
 	put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
 		  aio_req->should_dirty);
 	ceph_osdc_put_request(req);
-- 
2.27.0

[PATCH 4/4] ceph: add IO size metrics support

[xiubli]

From: Xiubo Li <xiubli@redhat.com>

This will collect IO's total size and then calculate the average
size, and also will collect the min/max IO sizes.

The debugfs will show the size metrics in byte and will let the
userspace applications to switch to what they need.

URL: https://tracker.ceph.com/issues/49913
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
 fs/ceph/addr.c    | 14 ++++++++------
 fs/ceph/debugfs.c | 37 +++++++++++++++++++++++++++++++++----
 fs/ceph/file.c    | 23 +++++++++++------------
 fs/ceph/metric.c  | 44 ++++++++++++++++++++++++++++++++++++++++++--
 fs/ceph/metric.h  | 10 ++++++++--
 5 files changed, 102 insertions(+), 26 deletions(-)

diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 7c2802758d0e..d8a3624bc81d 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -227,7 +227,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
 	int err = req->r_result;
 
 	ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
-				 req->r_end_latency, err);
+				 req->r_end_latency, osd_data->length, err);
 
 	dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
 	     subreq->len, i_size_read(req->r_inode));
@@ -561,7 +561,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
 		err = ceph_osdc_wait_request(osdc, req);
 
 	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
-				  req->r_end_latency, err);
+				  req->r_end_latency, len, err);
 
 	ceph_osdc_put_request(req);
 	if (err == 0)
@@ -636,6 +636,7 @@ static void writepages_finish(struct ceph_osd_request *req)
 	struct ceph_snap_context *snapc = req->r_snapc;
 	struct address_space *mapping = inode->i_mapping;
 	struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
+	unsigned int len = 0;
 	bool remove_page;
 
 	dout("writepages_finish %p rc %d\n", inode, rc);
@@ -648,9 +649,6 @@ static void writepages_finish(struct ceph_osd_request *req)
 		ceph_clear_error_write(ci);
 	}
 
-	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
-				  req->r_end_latency, rc);
-
 	/*
 	 * We lost the cache cap, need to truncate the page before
 	 * it is unlocked, otherwise we'd truncate it later in the
@@ -667,6 +665,7 @@ static void writepages_finish(struct ceph_osd_request *req)
 
 		osd_data = osd_req_op_extent_osd_data(req, i);
 		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+		len += osd_data->length;
 		num_pages = calc_pages_for((u64)osd_data->alignment,
 					   (u64)osd_data->length);
 		total_pages += num_pages;
@@ -699,6 +698,9 @@ static void writepages_finish(struct ceph_osd_request *req)
 		release_pages(osd_data->pages, num_pages);
 	}
 
+	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
+				  req->r_end_latency, len, rc);
+
 	ceph_put_wrbuffer_cap_refs(ci, total_pages, snapc);
 
 	osd_data = osd_req_op_extent_osd_data(req, 0);
@@ -1720,7 +1722,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
 		err = ceph_osdc_wait_request(&fsc->client->osdc, req);
 
 	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
-				  req->r_end_latency, err);
+				  req->r_end_latency, len, err);
 
 out_put:
 	ceph_osdc_put_request(req);
diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
index 425f3356332a..38b78b45811f 100644
--- a/fs/ceph/debugfs.c
+++ b/fs/ceph/debugfs.c
@@ -127,7 +127,7 @@ static int mdsc_show(struct seq_file *s, void *p)
 	return 0;
 }
 
-#define CEPH_METRIC_SHOW(name, total, avg, min, max, sq) {		\
+#define CEPH_LAT_METRIC_SHOW(name, total, avg, min, max, sq) {		\
 	s64 _total, _avg, _min, _max, _sq, _st;				\
 	_avg = ktime_to_us(avg);					\
 	_min = ktime_to_us(min == KTIME_MAX ? 0 : min);			\
@@ -140,6 +140,12 @@ static int mdsc_show(struct seq_file *s, void *p)
 		   name, total, _avg, _min, _max, _st);			\
 }
 
+#define CEPH_SZ_METRIC_SHOW(name, total, avg, min, max, sum) {		\
+	u64 _min = min == U64_MAX ? 0 : min;				\
+	seq_printf(s, "%-14s%-12lld%-16llu%-16llu%-16llu%llu\n",	\
+		   name, total, avg, _min, max, sum);			\
+}
+
 static int metric_show(struct seq_file *s, void *p)
 {
 	struct ceph_fs_client *fsc = s->private;
@@ -147,6 +153,7 @@ static int metric_show(struct seq_file *s, void *p)
 	struct ceph_client_metric *m = &mdsc->metric;
 	int nr_caps = 0;
 	s64 total, sum, avg, min, max, sq;
+	u64 sum_sz, avg_sz, min_sz, max_sz;
 
 	sum = percpu_counter_sum(&m->total_inodes);
 	seq_printf(s, "item                               total\n");
@@ -170,7 +177,7 @@ static int metric_show(struct seq_file *s, void *p)
 	max = m->read_latency_max;
 	sq = m->read_latency_sq_sum;
 	spin_unlock(&m->read_metric_lock);
-	CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
+	CEPH_LAT_METRIC_SHOW("read", total, avg, min, max, sq);
 
 	spin_lock(&m->write_metric_lock);
 	total = m->total_writes;
@@ -180,7 +187,7 @@ static int metric_show(struct seq_file *s, void *p)
 	max = m->write_latency_max;
 	sq = m->write_latency_sq_sum;
 	spin_unlock(&m->write_metric_lock);
-	CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
+	CEPH_LAT_METRIC_SHOW("write", total, avg, min, max, sq);
 
 	spin_lock(&m->metadata_metric_lock);
 	total = m->total_metadatas;
@@ -190,7 +197,29 @@ static int metric_show(struct seq_file *s, void *p)
 	max = m->metadata_latency_max;
 	sq = m->metadata_latency_sq_sum;
 	spin_unlock(&m->metadata_metric_lock);
-	CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
+	CEPH_LAT_METRIC_SHOW("metadata", total, avg, min, max, sq);
+
+	seq_printf(s, "\n");
+	seq_printf(s, "item          total       avg_sz(bytes)   min_sz(bytes)   max_sz(bytes)  total_sz(bytes)\n");
+	seq_printf(s, "----------------------------------------------------------------------------------------\n");
+
+	spin_lock(&m->read_metric_lock);
+	total = m->total_reads;
+	sum_sz = m->read_size_sum;
+	avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
+	min_sz = m->read_size_min;
+	max_sz = m->read_size_max;
+	spin_unlock(&m->read_metric_lock);
+	CEPH_SZ_METRIC_SHOW("read", total, avg_sz, min_sz, max_sz, sum_sz);
+
+	spin_lock(&m->write_metric_lock);
+	total = m->total_writes;
+	sum_sz = m->write_size_sum;
+	avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
+	min_sz = m->write_size_min;
+	max_sz = m->write_size_max;
+	spin_unlock(&m->write_metric_lock);
+	CEPH_SZ_METRIC_SHOW("write", total, avg_sz, min_sz, max_sz, sum_sz);
 
 	seq_printf(s, "\n");
 	seq_printf(s, "item          total           miss            hit\n");
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 31542eac7e59..db43d2d013b9 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -898,7 +898,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
 		ceph_update_read_metrics(&fsc->mdsc->metric,
 					 req->r_start_latency,
 					 req->r_end_latency,
-					 ret);
+					 len, ret);
 
 		ceph_osdc_put_request(req);
 
@@ -1030,12 +1030,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
 	struct ceph_aio_request *aio_req = req->r_priv;
 	struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0);
 	struct ceph_client_metric *metric = &ceph_sb_to_mdsc(inode->i_sb)->metric;
+	unsigned int len = osd_data->bvec_pos.iter.bi_size;
 
 	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS);
 	BUG_ON(!osd_data->num_bvecs);
 
-	dout("ceph_aio_complete_req %p rc %d bytes %u\n",
-	     inode, rc, osd_data->bvec_pos.iter.bi_size);
+	dout("ceph_aio_complete_req %p rc %d bytes %u\n", inode, rc, len);
 
 	if (rc == -EOLDSNAPC) {
 		struct ceph_aio_work *aio_work;
@@ -1053,9 +1053,9 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
 	} else if (!aio_req->write) {
 		if (rc == -ENOENT)
 			rc = 0;
-		if (rc >= 0 && osd_data->bvec_pos.iter.bi_size > rc) {
+		if (rc >= 0 && len > rc) {
 			struct iov_iter i;
-			int zlen = osd_data->bvec_pos.iter.bi_size - rc;
+			int zlen = len - rc;
 
 			/*
 			 * If read is satisfied by single OSD request,
@@ -1072,8 +1072,7 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
 			}
 
 			iov_iter_bvec(&i, READ, osd_data->bvec_pos.bvecs,
-				      osd_data->num_bvecs,
-				      osd_data->bvec_pos.iter.bi_size);
+				      osd_data->num_bvecs, len);
 			iov_iter_advance(&i, rc);
 			iov_iter_zero(zlen, &i);
 		}
@@ -1083,10 +1082,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
 	if (req->r_start_latency) {
 		if (aio_req->write)
 			ceph_update_write_metrics(metric, req->r_start_latency,
-						  req->r_end_latency, rc);
+						  req->r_end_latency, len, rc);
 		else
 			ceph_update_read_metrics(metric, req->r_start_latency,
-						 req->r_end_latency, rc);
+						 req->r_end_latency, len, rc);
 	}
 
 	put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
@@ -1294,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
 
 		if (write)
 			ceph_update_write_metrics(metric, req->r_start_latency,
-						  req->r_end_latency, ret);
+						  req->r_end_latency, len, ret);
 		else
 			ceph_update_read_metrics(metric, req->r_start_latency,
-						 req->r_end_latency, ret);
+						 req->r_end_latency, len, ret);
 
 		size = i_size_read(inode);
 		if (!write) {
@@ -1471,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 			ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
 
 		ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
-					  req->r_end_latency, ret);
+					  req->r_end_latency, len, ret);
 out:
 		ceph_osdc_put_request(req);
 		if (ret != 0) {
diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
index d5560ff99a9d..ff3c9d5cf9ff 100644
--- a/fs/ceph/metric.c
+++ b/fs/ceph/metric.c
@@ -189,6 +189,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
 	m->read_latency_max = 0;
 	m->total_reads = 0;
 	m->read_latency_sum = 0;
+	m->read_size_min = U64_MAX;
+	m->read_size_max = 0;
+	m->read_size_sum = 0;
 
 	spin_lock_init(&m->write_metric_lock);
 	m->write_latency_sq_sum = 0;
@@ -196,6 +199,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
 	m->write_latency_max = 0;
 	m->total_writes = 0;
 	m->write_latency_sum = 0;
+	m->write_size_min = U64_MAX;
+	m->write_size_max = 0;
+	m->write_size_sum = 0;
 
 	spin_lock_init(&m->metadata_metric_lock);
 	m->metadata_latency_sq_sum = 0;
@@ -306,9 +312,41 @@ static inline void __update_latency(struct ceph_client_metric *m,
 	*sq_sump += sq;
 }
 
+static inline void __update_size(struct ceph_client_metric *m,
+				 metric_type type, unsigned int size)
+{
+	ktime_t total;
+	u64 *minp, *maxp, *sump;
+
+	switch (type) {
+	case CEPH_METRIC_READ:
+		total = m->total_reads;
+		sump = &m->read_size_sum;
+		minp = &m->read_size_min;
+		maxp = &m->read_size_max;
+		break;
+	case CEPH_METRIC_WRITE:
+		total = m->total_writes;
+		sump = &m->write_size_sum;
+		minp = &m->write_size_min;
+		maxp = &m->write_size_max;
+		break;
+	case CEPH_METRIC_METADATA:
+	default:
+		return;
+	}
+
+	*sump += size;
+
+	if (unlikely(size < *minp))
+		*minp = size;
+	if (unlikely(size > *maxp))
+		*maxp = size;
+}
+
 void ceph_update_read_metrics(struct ceph_client_metric *m,
 			      ktime_t r_start, ktime_t r_end,
-			      int rc)
+			      unsigned int size, int rc)
 {
 	ktime_t lat = ktime_sub(r_end, r_start);
 
@@ -317,12 +355,13 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
 
 	spin_lock(&m->read_metric_lock);
 	__update_latency(m, CEPH_METRIC_READ, lat);
+	__update_size(m, CEPH_METRIC_READ, size);
 	spin_unlock(&m->read_metric_lock);
 }
 
 void ceph_update_write_metrics(struct ceph_client_metric *m,
 			       ktime_t r_start, ktime_t r_end,
-			       int rc)
+			       unsigned int size, int rc)
 {
 	ktime_t lat = ktime_sub(r_end, r_start);
 
@@ -331,6 +370,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
 
 	spin_lock(&m->write_metric_lock);
 	__update_latency(m, CEPH_METRIC_WRITE, lat);
+	__update_size(m, CEPH_METRIC_WRITE, size);
 	spin_unlock(&m->write_metric_lock);
 }
 
diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
index 57b5f0ec38be..64651b6ac886 100644
--- a/fs/ceph/metric.h
+++ b/fs/ceph/metric.h
@@ -110,6 +110,9 @@ struct ceph_client_metric {
 
 	spinlock_t read_metric_lock;
 	u64 total_reads;
+	u64 read_size_sum;
+	u64 read_size_min;
+	u64 read_size_max;
 	ktime_t read_latency_sum;
 	ktime_t read_latency_sq_sum;
 	ktime_t read_latency_min;
@@ -117,6 +120,9 @@ struct ceph_client_metric {
 
 	spinlock_t write_metric_lock;
 	u64 total_writes;
+	u64 write_size_sum;
+	u64 write_size_min;
+	u64 write_size_max;
 	ktime_t write_latency_sum;
 	ktime_t write_latency_sq_sum;
 	ktime_t write_latency_min;
@@ -164,10 +170,10 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
 
 extern void ceph_update_read_metrics(struct ceph_client_metric *m,
 				     ktime_t r_start, ktime_t r_end,
-				     int rc);
+				     unsigned int size, int rc);
 extern void ceph_update_write_metrics(struct ceph_client_metric *m,
 				      ktime_t r_start, ktime_t r_end,
-				      int rc);
+				      unsigned int size, int rc);
 extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
 				         ktime_t r_start, ktime_t r_end,
 					 int rc);
-- 
2.27.0

Re: [PATCH 4/4] ceph: add IO size metrics support

[Jeff Layton]

On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
> From: Xiubo Li <xiubli@redhat.com>
> 
> This will collect IO's total size and then calculate the average
> size, and also will collect the min/max IO sizes.
> 
> The debugfs will show the size metrics in byte and will let the
> userspace applications to switch to what they need.
> 
> URL: https://tracker.ceph.com/issues/49913
> Signed-off-by: Xiubo Li <xiubli@redhat.com>
> ---
>  fs/ceph/addr.c    | 14 ++++++++------
>  fs/ceph/debugfs.c | 37 +++++++++++++++++++++++++++++++++----
>  fs/ceph/file.c    | 23 +++++++++++------------
>  fs/ceph/metric.c  | 44 ++++++++++++++++++++++++++++++++++++++++++--
>  fs/ceph/metric.h  | 10 ++++++++--
>  5 files changed, 102 insertions(+), 26 deletions(-)
> 
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index 7c2802758d0e..d8a3624bc81d 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -227,7 +227,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
>  	int err = req->r_result;
>  
> 
> 
> 
>  	ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
> -				 req->r_end_latency, err);
> +				 req->r_end_latency, osd_data->length, err);
>  
> 
> 
> 
>  	dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
>  	     subreq->len, i_size_read(req->r_inode));
> @@ -561,7 +561,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
>  		err = ceph_osdc_wait_request(osdc, req);
>  
> 
> 
> 
>  	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> -				  req->r_end_latency, err);
> +				  req->r_end_latency, len, err);
>  
> 
> 
> 
>  	ceph_osdc_put_request(req);
>  	if (err == 0)
> @@ -636,6 +636,7 @@ static void writepages_finish(struct ceph_osd_request *req)
>  	struct ceph_snap_context *snapc = req->r_snapc;
>  	struct address_space *mapping = inode->i_mapping;
>  	struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
> +	unsigned int len = 0;
>  	bool remove_page;
>  
> 
> 
> 
>  	dout("writepages_finish %p rc %d\n", inode, rc);
> @@ -648,9 +649,6 @@ static void writepages_finish(struct ceph_osd_request *req)
>  		ceph_clear_error_write(ci);
>  	}
>  
> 
> 
> 
> -	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> -				  req->r_end_latency, rc);
> -
>  	/*
>  	 * We lost the cache cap, need to truncate the page before
>  	 * it is unlocked, otherwise we'd truncate it later in the
> @@ -667,6 +665,7 @@ static void writepages_finish(struct ceph_osd_request *req)
>  
> 
> 
> 
>  		osd_data = osd_req_op_extent_osd_data(req, i);
>  		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
> +		len += osd_data->length;
>  		num_pages = calc_pages_for((u64)osd_data->alignment,
>  					   (u64)osd_data->length);
>  		total_pages += num_pages;
> @@ -699,6 +698,9 @@ static void writepages_finish(struct ceph_osd_request *req)
>  		release_pages(osd_data->pages, num_pages);
>  	}
>  
> 
> 
> 
> +	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> +				  req->r_end_latency, len, rc);
> +
>  	ceph_put_wrbuffer_cap_refs(ci, total_pages, snapc);
>  
> 
> 
> 
>  	osd_data = osd_req_op_extent_osd_data(req, 0);
> @@ -1720,7 +1722,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
>  		err = ceph_osdc_wait_request(&fsc->client->osdc, req);
>  
> 
> 
> 
>  	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> -				  req->r_end_latency, err);
> +				  req->r_end_latency, len, err);
>  
> 
> 
> 
>  out_put:
>  	ceph_osdc_put_request(req);
> diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
> index 425f3356332a..38b78b45811f 100644
> --- a/fs/ceph/debugfs.c
> +++ b/fs/ceph/debugfs.c
> @@ -127,7 +127,7 @@ static int mdsc_show(struct seq_file *s, void *p)
>  	return 0;
>  }
>  
> 
> 
> 
> -#define CEPH_METRIC_SHOW(name, total, avg, min, max, sq) {		\
> +#define CEPH_LAT_METRIC_SHOW(name, total, avg, min, max, sq) {		\
>  	s64 _total, _avg, _min, _max, _sq, _st;				\
>  	_avg = ktime_to_us(avg);					\
>  	_min = ktime_to_us(min == KTIME_MAX ? 0 : min);			\
> @@ -140,6 +140,12 @@ static int mdsc_show(struct seq_file *s, void *p)
>  		   name, total, _avg, _min, _max, _st);			\
>  }
>  
> 
> 
> 
> +#define CEPH_SZ_METRIC_SHOW(name, total, avg, min, max, sum) {		\
> +	u64 _min = min == U64_MAX ? 0 : min;				\
> +	seq_printf(s, "%-14s%-12lld%-16llu%-16llu%-16llu%llu\n",	\
> +		   name, total, avg, _min, max, sum);			\
> +}
> +
>  static int metric_show(struct seq_file *s, void *p)
>  {
>  	struct ceph_fs_client *fsc = s->private;
> @@ -147,6 +153,7 @@ static int metric_show(struct seq_file *s, void *p)
>  	struct ceph_client_metric *m = &mdsc->metric;
>  	int nr_caps = 0;
>  	s64 total, sum, avg, min, max, sq;
> +	u64 sum_sz, avg_sz, min_sz, max_sz;
>  
> 
> 
> 
>  	sum = percpu_counter_sum(&m->total_inodes);
>  	seq_printf(s, "item                               total\n");
> @@ -170,7 +177,7 @@ static int metric_show(struct seq_file *s, void *p)
>  	max = m->read_latency_max;
>  	sq = m->read_latency_sq_sum;
>  	spin_unlock(&m->read_metric_lock);
> -	CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
> +	CEPH_LAT_METRIC_SHOW("read", total, avg, min, max, sq);
>  
> 
> 
> 
>  	spin_lock(&m->write_metric_lock);
>  	total = m->total_writes;
> @@ -180,7 +187,7 @@ static int metric_show(struct seq_file *s, void *p)
>  	max = m->write_latency_max;
>  	sq = m->write_latency_sq_sum;
>  	spin_unlock(&m->write_metric_lock);
> -	CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
> +	CEPH_LAT_METRIC_SHOW("write", total, avg, min, max, sq);
>  
> 
> 
> 
>  	spin_lock(&m->metadata_metric_lock);
>  	total = m->total_metadatas;
> @@ -190,7 +197,29 @@ static int metric_show(struct seq_file *s, void *p)
>  	max = m->metadata_latency_max;
>  	sq = m->metadata_latency_sq_sum;
>  	spin_unlock(&m->metadata_metric_lock);
> -	CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
> +	CEPH_LAT_METRIC_SHOW("metadata", total, avg, min, max, sq);
> +
> +	seq_printf(s, "\n");
> +	seq_printf(s, "item          total       avg_sz(bytes)   min_sz(bytes)   max_sz(bytes)  total_sz(bytes)\n");
> +	seq_printf(s, "----------------------------------------------------------------------------------------\n");
> +
> +	spin_lock(&m->read_metric_lock);
> +	total = m->total_reads;
> +	sum_sz = m->read_size_sum;
> +	avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
> +	min_sz = m->read_size_min;
> +	max_sz = m->read_size_max;
> +	spin_unlock(&m->read_metric_lock);
> +	CEPH_SZ_METRIC_SHOW("read", total, avg_sz, min_sz, max_sz, sum_sz);
> +
> +	spin_lock(&m->write_metric_lock);
> +	total = m->total_writes;
> +	sum_sz = m->write_size_sum;
> +	avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
> +	min_sz = m->write_size_min;
> +	max_sz = m->write_size_max;
> +	spin_unlock(&m->write_metric_lock);
> +	CEPH_SZ_METRIC_SHOW("write", total, avg_sz, min_sz, max_sz, sum_sz);
>  
> 
> 
> 
>  	seq_printf(s, "\n");
>  	seq_printf(s, "item          total           miss            hit\n");
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 31542eac7e59..db43d2d013b9 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -898,7 +898,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
>  		ceph_update_read_metrics(&fsc->mdsc->metric,
>  					 req->r_start_latency,
>  					 req->r_end_latency,
> -					 ret);
> +					 len, ret);
>  
> 
> 
> 
>  		ceph_osdc_put_request(req);
>  
> 
> 
> 
> @@ -1030,12 +1030,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>  	struct ceph_aio_request *aio_req = req->r_priv;
>  	struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0);
>  	struct ceph_client_metric *metric = &ceph_sb_to_mdsc(inode->i_sb)->metric;
> +	unsigned int len = osd_data->bvec_pos.iter.bi_size;
>  
> 
> 
> 
>  	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS);
>  	BUG_ON(!osd_data->num_bvecs);
>  
> 
> 
> 
> -	dout("ceph_aio_complete_req %p rc %d bytes %u\n",
> -	     inode, rc, osd_data->bvec_pos.iter.bi_size);
> +	dout("ceph_aio_complete_req %p rc %d bytes %u\n", inode, rc, len);
>  
> 
> 
> 
>  	if (rc == -EOLDSNAPC) {
>  		struct ceph_aio_work *aio_work;
> @@ -1053,9 +1053,9 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>  	} else if (!aio_req->write) {
>  		if (rc == -ENOENT)
>  			rc = 0;
> -		if (rc >= 0 && osd_data->bvec_pos.iter.bi_size > rc) {
> +		if (rc >= 0 && len > rc) {
>  			struct iov_iter i;
> -			int zlen = osd_data->bvec_pos.iter.bi_size - rc;
> +			int zlen = len - rc;
>  
> 
> 
> 
>  			/*
>  			 * If read is satisfied by single OSD request,
> @@ -1072,8 +1072,7 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>  			}
>  
> 
> 
> 
>  			iov_iter_bvec(&i, READ, osd_data->bvec_pos.bvecs,
> -				      osd_data->num_bvecs,
> -				      osd_data->bvec_pos.iter.bi_size);
> +				      osd_data->num_bvecs, len);
>  			iov_iter_advance(&i, rc);
>  			iov_iter_zero(zlen, &i);
>  		}
> @@ -1083,10 +1082,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>  	if (req->r_start_latency) {
>  		if (aio_req->write)
>  			ceph_update_write_metrics(metric, req->r_start_latency,
> -						  req->r_end_latency, rc);
> +						  req->r_end_latency, len, rc);
>  		else
>  			ceph_update_read_metrics(metric, req->r_start_latency,
> -						 req->r_end_latency, rc);
> +						 req->r_end_latency, len, rc);
>  	}
>  
> 
> 
> 
>  	put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
> @@ -1294,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
>  
> 
> 
> 
>  		if (write)
>  			ceph_update_write_metrics(metric, req->r_start_latency,
> -						  req->r_end_latency, ret);
> +						  req->r_end_latency, len, ret);
>  		else
>  			ceph_update_read_metrics(metric, req->r_start_latency,
> -						 req->r_end_latency, ret);
> +						 req->r_end_latency, len, ret);
>  
> 
> 
> 
>  		size = i_size_read(inode);
>  		if (!write) {
> @@ -1471,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
>  			ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
>  
> 
> 
> 
>  		ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> -					  req->r_end_latency, ret);
> +					  req->r_end_latency, len, ret);
>  out:
>  		ceph_osdc_put_request(req);
>  		if (ret != 0) {
> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
> index d5560ff99a9d..ff3c9d5cf9ff 100644
> --- a/fs/ceph/metric.c
> +++ b/fs/ceph/metric.c
> @@ -189,6 +189,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
>  	m->read_latency_max = 0;
>  	m->total_reads = 0;
>  	m->read_latency_sum = 0;
> +	m->read_size_min = U64_MAX;
> +	m->read_size_max = 0;
> +	m->read_size_sum = 0;
>  
> 
> 
> 
>  	spin_lock_init(&m->write_metric_lock);
>  	m->write_latency_sq_sum = 0;
> @@ -196,6 +199,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
>  	m->write_latency_max = 0;
>  	m->total_writes = 0;
>  	m->write_latency_sum = 0;
> +	m->write_size_min = U64_MAX;
> +	m->write_size_max = 0;
> +	m->write_size_sum = 0;
>  
> 
> 
> 
>  	spin_lock_init(&m->metadata_metric_lock);
>  	m->metadata_latency_sq_sum = 0;
> @@ -306,9 +312,41 @@ static inline void __update_latency(struct ceph_client_metric *m,
>  	*sq_sump += sq;
>  }
>  
> 
> 
> 
> +static inline void __update_size(struct ceph_client_metric *m,
> +				 metric_type type, unsigned int size)
> +{
> +	ktime_t total;
> +	u64 *minp, *maxp, *sump;
> +
> +	switch (type) {
> +	case CEPH_METRIC_READ:
> +		total = m->total_reads;
> +		sump = &m->read_size_sum;
> +		minp = &m->read_size_min;
> +		maxp = &m->read_size_max;
> +		break;
> +	case CEPH_METRIC_WRITE:
> +		total = m->total_writes;

"total" and "sump" are unused in this function, aside from the
assignment.

> +		sump = &m->write_size_sum;
> +		minp = &m->write_size_min;
> +		maxp = &m->write_size_max;
> +		break;
> +	case CEPH_METRIC_METADATA:
> +	default:
> +		return;
> +	}
> +
> +	*sump += size;
> +
> +	if (unlikely(size < *minp))
> +		*minp = size;
> +	if (unlikely(size > *maxp))
> +		*maxp = size;
> +}
> +
>  void ceph_update_read_metrics(struct ceph_client_metric *m,
>  			      ktime_t r_start, ktime_t r_end,
> -			      int rc)
> +			      unsigned int size, int rc)
>  {
>  	ktime_t lat = ktime_sub(r_end, r_start);
>  
> 
> 
> 
> @@ -317,12 +355,13 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
>  
> 
> 
> 
>  	spin_lock(&m->read_metric_lock);
>  	__update_latency(m, CEPH_METRIC_READ, lat);
> +	__update_size(m, CEPH_METRIC_READ, size);
>  	spin_unlock(&m->read_metric_lock);
>  }
>  
> 
> 
> 
>  void ceph_update_write_metrics(struct ceph_client_metric *m,
>  			       ktime_t r_start, ktime_t r_end,
> -			       int rc)
> +			       unsigned int size, int rc)
>  {
>  	ktime_t lat = ktime_sub(r_end, r_start);
>  
> 
> 
> 
> @@ -331,6 +370,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
>  
> 
> 
> 
>  	spin_lock(&m->write_metric_lock);
>  	__update_latency(m, CEPH_METRIC_WRITE, lat);
> +	__update_size(m, CEPH_METRIC_WRITE, size);
>  	spin_unlock(&m->write_metric_lock);
>  }
>  
> 
> 
> 
> diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
> index 57b5f0ec38be..64651b6ac886 100644
> --- a/fs/ceph/metric.h
> +++ b/fs/ceph/metric.h
> @@ -110,6 +110,9 @@ struct ceph_client_metric {
>  
> 
> 
> 
>  	spinlock_t read_metric_lock;
>  	u64 total_reads;
> +	u64 read_size_sum;
> +	u64 read_size_min;
> +	u64 read_size_max;
>  	ktime_t read_latency_sum;
>  	ktime_t read_latency_sq_sum;
>  	ktime_t read_latency_min;
> @@ -117,6 +120,9 @@ struct ceph_client_metric {
>  
> 
> 
> 
>  	spinlock_t write_metric_lock;
>  	u64 total_writes;
> +	u64 write_size_sum;
> +	u64 write_size_min;
> +	u64 write_size_max;
>  	ktime_t write_latency_sum;
>  	ktime_t write_latency_sq_sum;
>  	ktime_t write_latency_min;
> @@ -164,10 +170,10 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
>  
> 
> 
> 
>  extern void ceph_update_read_metrics(struct ceph_client_metric *m,
>  				     ktime_t r_start, ktime_t r_end,
> -				     int rc);
> +				     unsigned int size, int rc);
>  extern void ceph_update_write_metrics(struct ceph_client_metric *m,
>  				      ktime_t r_start, ktime_t r_end,
> -				      int rc);
> +				      unsigned int size, int rc);
>  extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
>  				         ktime_t r_start, ktime_t r_end,
>  					 int rc);

-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH 2/4] ceph: update the __update_latency helper

[Jeff Layton]

On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
> From: Xiubo Li <xiubli@redhat.com>
> 
> Let the __update_latency() helper choose the correcsponding members
> according to the metric_type.
> 
> URL: https://tracker.ceph.com/issues/49913
> Signed-off-by: Xiubo Li <xiubli@redhat.com>
> ---
>  fs/ceph/metric.c | 58 +++++++++++++++++++++++++++++++++++-------------
>  1 file changed, 42 insertions(+), 16 deletions(-)
> 
> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
> index 75d309f2fb0c..d5560ff99a9d 100644
> --- a/fs/ceph/metric.c
> +++ b/fs/ceph/metric.c
> @@ -249,19 +249,51 @@ void ceph_metric_destroy(struct ceph_client_metric *m)
>  		ceph_put_mds_session(m->session);
>  }
>  
> 
> 
> 
> 
> 
> 
> 
> -static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
> -				    ktime_t *min, ktime_t *max,
> -				    ktime_t *sq_sump, ktime_t lat)
> +typedef enum {
> +	CEPH_METRIC_READ,
> +	CEPH_METRIC_WRITE,
> +	CEPH_METRIC_METADATA,
> +} metric_type;
> +
> +static inline void __update_latency(struct ceph_client_metric *m,
> +				    metric_type type, ktime_t lat)
>  {
> +	ktime_t *totalp, *minp, *maxp, *lsump, *sq_sump;
>  	ktime_t total, avg, sq, lsum;
>  
> 
> 
> 
> 
> 
> 
> 
> +	switch (type) {
> +	case CEPH_METRIC_READ:
> +		totalp = &m->total_reads;
> +		lsump = &m->read_latency_sum;
> +		minp = &m->read_latency_min;
> +		maxp = &m->read_latency_max;
> +		sq_sump = &m->read_latency_sq_sum;
> +		break;
> +	case CEPH_METRIC_WRITE:
> +		totalp = &m->total_writes;
> +		lsump = &m->write_latency_sum;
> +		minp = &m->write_latency_min;
> +		maxp = &m->write_latency_max;
> +		sq_sump = &m->write_latency_sq_sum;
> +		break;
> +	case CEPH_METRIC_METADATA:
> +		totalp = &m->total_metadatas;
> +		lsump = &m->metadata_latency_sum;
> +		minp = &m->metadata_latency_min;
> +		maxp = &m->metadata_latency_max;
> +		sq_sump = &m->metadata_latency_sq_sum;
> +		break;
> +	default:
> +		return;
> +	}
> +
>  	total = ++(*totalp);

Why are you adding one to *totalp above? Is that to avoid it being 0? 

>  	lsum = (*lsump += lat);
>  
> 

^^^
Instead of doing all of the above with pointers, why not just add to
total and lsum directly inside the switch statement? This seems like a
lot of pointless indirection.

> 
> 
> 
> 
> 
> -	if (unlikely(lat < *min))
> -		*min = lat;
> -	if (unlikely(lat > *max))
> -		*max = lat;
> +	if (unlikely(lat < *minp))
> +		*minp = lat;
> +	if (unlikely(lat > *maxp))
> +		*maxp = lat;
>  
> 
> 
> 
>  	if (unlikely(total == 1))
>  		return;
> @@ -284,9 +316,7 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
>  		return;
>  
> 
> 
> 
>  	spin_lock(&m->read_metric_lock);
> -	__update_latency(&m->total_reads, &m->read_latency_sum,
> -			 &m->read_latency_min, &m->read_latency_max,
> -			 &m->read_latency_sq_sum, lat);
> +	__update_latency(m, CEPH_METRIC_READ, lat);
>  	spin_unlock(&m->read_metric_lock);
>  }
>  
> 
> 
> 
> @@ -300,9 +330,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
>  		return;
>  
> 
> 
> 
>  	spin_lock(&m->write_metric_lock);
> -	__update_latency(&m->total_writes, &m->write_latency_sum,
> -			 &m->write_latency_min, &m->write_latency_max,
> -			 &m->write_latency_sq_sum, lat);
> +	__update_latency(m, CEPH_METRIC_WRITE, lat);
>  	spin_unlock(&m->write_metric_lock);
>  }
>  
> 
> 
> 
> @@ -316,8 +344,6 @@ void ceph_update_metadata_metrics(struct ceph_client_metric *m,
>  		return;
>  
> 
> 
> 
>  	spin_lock(&m->metadata_metric_lock);
> -	__update_latency(&m->total_metadatas, &m->metadata_latency_sum,
> -			 &m->metadata_latency_min, &m->metadata_latency_max,
> -			 &m->metadata_latency_sq_sum, lat);
> +	__update_latency(m, CEPH_METRIC_METADATA, lat);
>  	spin_unlock(&m->metadata_metric_lock);
>  }

-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH 2/4] ceph: update the __update_latency helper

[Xiubo Li]

On 2021/3/23 20:34, Jeff Layton wrote:
> On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
>> From: Xiubo Li <xiubli@redhat.com>
>>
>> Let the __update_latency() helper choose the correcsponding members
>> according to the metric_type.
>>
>> URL: https://tracker.ceph.com/issues/49913
>> Signed-off-by: Xiubo Li <xiubli@redhat.com>
>> ---
>>   fs/ceph/metric.c | 58 +++++++++++++++++++++++++++++++++++-------------
>>   1 file changed, 42 insertions(+), 16 deletions(-)
>>
>> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
>> index 75d309f2fb0c..d5560ff99a9d 100644
>> --- a/fs/ceph/metric.c
>> +++ b/fs/ceph/metric.c
>> @@ -249,19 +249,51 @@ void ceph_metric_destroy(struct ceph_client_metric *m)
>>   		ceph_put_mds_session(m->session);
>>   }
>>   
>>
>>
>>
>>
>>
>>
>>
>> -static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
>> -				    ktime_t *min, ktime_t *max,
>> -				    ktime_t *sq_sump, ktime_t lat)
>> +typedef enum {
>> +	CEPH_METRIC_READ,
>> +	CEPH_METRIC_WRITE,
>> +	CEPH_METRIC_METADATA,
>> +} metric_type;
>> +
>> +static inline void __update_latency(struct ceph_client_metric *m,
>> +				    metric_type type, ktime_t lat)
>>   {
>> +	ktime_t *totalp, *minp, *maxp, *lsump, *sq_sump;
>>   	ktime_t total, avg, sq, lsum;
>>   
>>
>>
>>
>>
>>
>>
>>
>> +	switch (type) {
>> +	case CEPH_METRIC_READ:
>> +		totalp = &m->total_reads;
>> +		lsump = &m->read_latency_sum;
>> +		minp = &m->read_latency_min;
>> +		maxp = &m->read_latency_max;
>> +		sq_sump = &m->read_latency_sq_sum;
>> +		break;
>> +	case CEPH_METRIC_WRITE:
>> +		totalp = &m->total_writes;
>> +		lsump = &m->write_latency_sum;
>> +		minp = &m->write_latency_min;
>> +		maxp = &m->write_latency_max;
>> +		sq_sump = &m->write_latency_sq_sum;
>> +		break;
>> +	case CEPH_METRIC_METADATA:
>> +		totalp = &m->total_metadatas;
>> +		lsump = &m->metadata_latency_sum;
>> +		minp = &m->metadata_latency_min;
>> +		maxp = &m->metadata_latency_max;
>> +		sq_sump = &m->metadata_latency_sq_sum;
>> +		break;
>> +	default:
>> +		return;
>> +	}
>> +
>>   	total = ++(*totalp);
> Why are you adding one to *totalp above? Is that to avoid it being 0?

No, in the old code we will count the 
total_reads/total_writes/total_metadatas for each call of the 
ceph_update_{read/write/metadata}_latency() helpers. And the same here.


>>   	lsum = (*lsump += lat);
>>   
>>
> ^^^
> Instead of doing all of the above with pointers, why not just add to
> total and lsum directly inside the switch statement? This seems like a
> lot of pointless indirection.

Okay, sounds good, will change it.


>>
>>
>>
>>
>> -	if (unlikely(lat < *min))
>> -		*min = lat;
>> -	if (unlikely(lat > *max))
>> -		*max = lat;
>> +	if (unlikely(lat < *minp))
>> +		*minp = lat;
>> +	if (unlikely(lat > *maxp))
>> +		*maxp = lat;
>>   
>>
>>
>>
>>   	if (unlikely(total == 1))
>>   		return;
>> @@ -284,9 +316,7 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
>>   		return;
>>   
>>
>>
>>
>>   	spin_lock(&m->read_metric_lock);
>> -	__update_latency(&m->total_reads, &m->read_latency_sum,
>> -			 &m->read_latency_min, &m->read_latency_max,
>> -			 &m->read_latency_sq_sum, lat);
>> +	__update_latency(m, CEPH_METRIC_READ, lat);
>>   	spin_unlock(&m->read_metric_lock);
>>   }
>>   
>>
>>
>>
>> @@ -300,9 +330,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
>>   		return;
>>   
>>
>>
>>
>>   	spin_lock(&m->write_metric_lock);
>> -	__update_latency(&m->total_writes, &m->write_latency_sum,
>> -			 &m->write_latency_min, &m->write_latency_max,
>> -			 &m->write_latency_sq_sum, lat);
>> +	__update_latency(m, CEPH_METRIC_WRITE, lat);
>>   	spin_unlock(&m->write_metric_lock);
>>   }
>>   
>>
>>
>>
>> @@ -316,8 +344,6 @@ void ceph_update_metadata_metrics(struct ceph_client_metric *m,
>>   		return;
>>   
>>
>>
>>
>>   	spin_lock(&m->metadata_metric_lock);
>> -	__update_latency(&m->total_metadatas, &m->metadata_latency_sum,
>> -			 &m->metadata_latency_min, &m->metadata_latency_max,
>> -			 &m->metadata_latency_sq_sum, lat);
>> +	__update_latency(m, CEPH_METRIC_METADATA, lat);
>>   	spin_unlock(&m->metadata_metric_lock);
>>   }

Re: [PATCH 4/4] ceph: add IO size metrics support

[Xiubo Li]

On 2021/3/23 20:29, Jeff Layton wrote:
> On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
>> From: Xiubo Li <xiubli@redhat.com>
>>
>> This will collect IO's total size and then calculate the average
>> size, and also will collect the min/max IO sizes.
>>
>> The debugfs will show the size metrics in byte and will let the
>> userspace applications to switch to what they need.
>>
>> URL: https://tracker.ceph.com/issues/49913
>> Signed-off-by: Xiubo Li <xiubli@redhat.com>
>> ---
>>   fs/ceph/addr.c    | 14 ++++++++------
>>   fs/ceph/debugfs.c | 37 +++++++++++++++++++++++++++++++++----
>>   fs/ceph/file.c    | 23 +++++++++++------------
>>   fs/ceph/metric.c  | 44 ++++++++++++++++++++++++++++++++++++++++++--
>>   fs/ceph/metric.h  | 10 ++++++++--
>>   5 files changed, 102 insertions(+), 26 deletions(-)
>>
>> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
>> index 7c2802758d0e..d8a3624bc81d 100644
>> --- a/fs/ceph/addr.c
>> +++ b/fs/ceph/addr.c
>> @@ -227,7 +227,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
>>   	int err = req->r_result;
>>   
>>
>>
>>
>>   	ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> -				 req->r_end_latency, err);
>> +				 req->r_end_latency, osd_data->length, err);
>>   
>>
>>
>>
>>   	dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
>>   	     subreq->len, i_size_read(req->r_inode));
>> @@ -561,7 +561,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
>>   		err = ceph_osdc_wait_request(osdc, req);
>>   
>>
>>
>>
>>   	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> -				  req->r_end_latency, err);
>> +				  req->r_end_latency, len, err);
>>   
>>
>>
>>
>>   	ceph_osdc_put_request(req);
>>   	if (err == 0)
>> @@ -636,6 +636,7 @@ static void writepages_finish(struct ceph_osd_request *req)
>>   	struct ceph_snap_context *snapc = req->r_snapc;
>>   	struct address_space *mapping = inode->i_mapping;
>>   	struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
>> +	unsigned int len = 0;
>>   	bool remove_page;
>>   
>>
>>
>>
>>   	dout("writepages_finish %p rc %d\n", inode, rc);
>> @@ -648,9 +649,6 @@ static void writepages_finish(struct ceph_osd_request *req)
>>   		ceph_clear_error_write(ci);
>>   	}
>>   
>>
>>
>>
>> -	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> -				  req->r_end_latency, rc);
>> -
>>   	/*
>>   	 * We lost the cache cap, need to truncate the page before
>>   	 * it is unlocked, otherwise we'd truncate it later in the
>> @@ -667,6 +665,7 @@ static void writepages_finish(struct ceph_osd_request *req)
>>   
>>
>>
>>
>>   		osd_data = osd_req_op_extent_osd_data(req, i);
>>   		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
>> +		len += osd_data->length;
>>   		num_pages = calc_pages_for((u64)osd_data->alignment,
>>   					   (u64)osd_data->length);
>>   		total_pages += num_pages;
>> @@ -699,6 +698,9 @@ static void writepages_finish(struct ceph_osd_request *req)
>>   		release_pages(osd_data->pages, num_pages);
>>   	}
>>   
>>
>>
>>
>> +	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> +				  req->r_end_latency, len, rc);
>> +
>>   	ceph_put_wrbuffer_cap_refs(ci, total_pages, snapc);
>>   
>>
>>
>>
>>   	osd_data = osd_req_op_extent_osd_data(req, 0);
>> @@ -1720,7 +1722,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
>>   		err = ceph_osdc_wait_request(&fsc->client->osdc, req);
>>   
>>
>>
>>
>>   	ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> -				  req->r_end_latency, err);
>> +				  req->r_end_latency, len, err);
>>   
>>
>>
>>
>>   out_put:
>>   	ceph_osdc_put_request(req);
>> diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
>> index 425f3356332a..38b78b45811f 100644
>> --- a/fs/ceph/debugfs.c
>> +++ b/fs/ceph/debugfs.c
>> @@ -127,7 +127,7 @@ static int mdsc_show(struct seq_file *s, void *p)
>>   	return 0;
>>   }
>>   
>>
>>
>>
>> -#define CEPH_METRIC_SHOW(name, total, avg, min, max, sq) {		\
>> +#define CEPH_LAT_METRIC_SHOW(name, total, avg, min, max, sq) {		\
>>   	s64 _total, _avg, _min, _max, _sq, _st;				\
>>   	_avg = ktime_to_us(avg);					\
>>   	_min = ktime_to_us(min == KTIME_MAX ? 0 : min);			\
>> @@ -140,6 +140,12 @@ static int mdsc_show(struct seq_file *s, void *p)
>>   		   name, total, _avg, _min, _max, _st);			\
>>   }
>>   
>>
>>
>>
>> +#define CEPH_SZ_METRIC_SHOW(name, total, avg, min, max, sum) {		\
>> +	u64 _min = min == U64_MAX ? 0 : min;				\
>> +	seq_printf(s, "%-14s%-12lld%-16llu%-16llu%-16llu%llu\n",	\
>> +		   name, total, avg, _min, max, sum);			\
>> +}
>> +
>>   static int metric_show(struct seq_file *s, void *p)
>>   {
>>   	struct ceph_fs_client *fsc = s->private;
>> @@ -147,6 +153,7 @@ static int metric_show(struct seq_file *s, void *p)
>>   	struct ceph_client_metric *m = &mdsc->metric;
>>   	int nr_caps = 0;
>>   	s64 total, sum, avg, min, max, sq;
>> +	u64 sum_sz, avg_sz, min_sz, max_sz;
>>   
>>
>>
>>
>>   	sum = percpu_counter_sum(&m->total_inodes);
>>   	seq_printf(s, "item                               total\n");
>> @@ -170,7 +177,7 @@ static int metric_show(struct seq_file *s, void *p)
>>   	max = m->read_latency_max;
>>   	sq = m->read_latency_sq_sum;
>>   	spin_unlock(&m->read_metric_lock);
>> -	CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
>> +	CEPH_LAT_METRIC_SHOW("read", total, avg, min, max, sq);
>>   
>>
>>
>>
>>   	spin_lock(&m->write_metric_lock);
>>   	total = m->total_writes;
>> @@ -180,7 +187,7 @@ static int metric_show(struct seq_file *s, void *p)
>>   	max = m->write_latency_max;
>>   	sq = m->write_latency_sq_sum;
>>   	spin_unlock(&m->write_metric_lock);
>> -	CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
>> +	CEPH_LAT_METRIC_SHOW("write", total, avg, min, max, sq);
>>   
>>
>>
>>
>>   	spin_lock(&m->metadata_metric_lock);
>>   	total = m->total_metadatas;
>> @@ -190,7 +197,29 @@ static int metric_show(struct seq_file *s, void *p)
>>   	max = m->metadata_latency_max;
>>   	sq = m->metadata_latency_sq_sum;
>>   	spin_unlock(&m->metadata_metric_lock);
>> -	CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
>> +	CEPH_LAT_METRIC_SHOW("metadata", total, avg, min, max, sq);
>> +
>> +	seq_printf(s, "\n");
>> +	seq_printf(s, "item          total       avg_sz(bytes)   min_sz(bytes)   max_sz(bytes)  total_sz(bytes)\n");
>> +	seq_printf(s, "----------------------------------------------------------------------------------------\n");
>> +
>> +	spin_lock(&m->read_metric_lock);
>> +	total = m->total_reads;
>> +	sum_sz = m->read_size_sum;
>> +	avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
>> +	min_sz = m->read_size_min;
>> +	max_sz = m->read_size_max;
>> +	spin_unlock(&m->read_metric_lock);
>> +	CEPH_SZ_METRIC_SHOW("read", total, avg_sz, min_sz, max_sz, sum_sz);
>> +
>> +	spin_lock(&m->write_metric_lock);
>> +	total = m->total_writes;
>> +	sum_sz = m->write_size_sum;
>> +	avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
>> +	min_sz = m->write_size_min;
>> +	max_sz = m->write_size_max;
>> +	spin_unlock(&m->write_metric_lock);
>> +	CEPH_SZ_METRIC_SHOW("write", total, avg_sz, min_sz, max_sz, sum_sz);
>>   
>>
>>
>>
>>   	seq_printf(s, "\n");
>>   	seq_printf(s, "item          total           miss            hit\n");
>> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
>> index 31542eac7e59..db43d2d013b9 100644
>> --- a/fs/ceph/file.c
>> +++ b/fs/ceph/file.c
>> @@ -898,7 +898,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
>>   		ceph_update_read_metrics(&fsc->mdsc->metric,
>>   					 req->r_start_latency,
>>   					 req->r_end_latency,
>> -					 ret);
>> +					 len, ret);
>>   
>>
>>
>>
>>   		ceph_osdc_put_request(req);
>>   
>>
>>
>>
>> @@ -1030,12 +1030,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>>   	struct ceph_aio_request *aio_req = req->r_priv;
>>   	struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0);
>>   	struct ceph_client_metric *metric = &ceph_sb_to_mdsc(inode->i_sb)->metric;
>> +	unsigned int len = osd_data->bvec_pos.iter.bi_size;
>>   
>>
>>
>>
>>   	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS);
>>   	BUG_ON(!osd_data->num_bvecs);
>>   
>>
>>
>>
>> -	dout("ceph_aio_complete_req %p rc %d bytes %u\n",
>> -	     inode, rc, osd_data->bvec_pos.iter.bi_size);
>> +	dout("ceph_aio_complete_req %p rc %d bytes %u\n", inode, rc, len);
>>   
>>
>>
>>
>>   	if (rc == -EOLDSNAPC) {
>>   		struct ceph_aio_work *aio_work;
>> @@ -1053,9 +1053,9 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>>   	} else if (!aio_req->write) {
>>   		if (rc == -ENOENT)
>>   			rc = 0;
>> -		if (rc >= 0 && osd_data->bvec_pos.iter.bi_size > rc) {
>> +		if (rc >= 0 && len > rc) {
>>   			struct iov_iter i;
>> -			int zlen = osd_data->bvec_pos.iter.bi_size - rc;
>> +			int zlen = len - rc;
>>   
>>
>>
>>
>>   			/*
>>   			 * If read is satisfied by single OSD request,
>> @@ -1072,8 +1072,7 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>>   			}
>>   
>>
>>
>>
>>   			iov_iter_bvec(&i, READ, osd_data->bvec_pos.bvecs,
>> -				      osd_data->num_bvecs,
>> -				      osd_data->bvec_pos.iter.bi_size);
>> +				      osd_data->num_bvecs, len);
>>   			iov_iter_advance(&i, rc);
>>   			iov_iter_zero(zlen, &i);
>>   		}
>> @@ -1083,10 +1082,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>>   	if (req->r_start_latency) {
>>   		if (aio_req->write)
>>   			ceph_update_write_metrics(metric, req->r_start_latency,
>> -						  req->r_end_latency, rc);
>> +						  req->r_end_latency, len, rc);
>>   		else
>>   			ceph_update_read_metrics(metric, req->r_start_latency,
>> -						 req->r_end_latency, rc);
>> +						 req->r_end_latency, len, rc);
>>   	}
>>   
>>
>>
>>
>>   	put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
>> @@ -1294,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
>>   
>>
>>
>>
>>   		if (write)
>>   			ceph_update_write_metrics(metric, req->r_start_latency,
>> -						  req->r_end_latency, ret);
>> +						  req->r_end_latency, len, ret);
>>   		else
>>   			ceph_update_read_metrics(metric, req->r_start_latency,
>> -						 req->r_end_latency, ret);
>> +						 req->r_end_latency, len, ret);
>>   
>>
>>
>>
>>   		size = i_size_read(inode);
>>   		if (!write) {
>> @@ -1471,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
>>   			ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
>>   
>>
>>
>>
>>   		ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> -					  req->r_end_latency, ret);
>> +					  req->r_end_latency, len, ret);
>>   out:
>>   		ceph_osdc_put_request(req);
>>   		if (ret != 0) {
>> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
>> index d5560ff99a9d..ff3c9d5cf9ff 100644
>> --- a/fs/ceph/metric.c
>> +++ b/fs/ceph/metric.c
>> @@ -189,6 +189,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
>>   	m->read_latency_max = 0;
>>   	m->total_reads = 0;
>>   	m->read_latency_sum = 0;
>> +	m->read_size_min = U64_MAX;
>> +	m->read_size_max = 0;
>> +	m->read_size_sum = 0;
>>   
>>
>>
>>
>>   	spin_lock_init(&m->write_metric_lock);
>>   	m->write_latency_sq_sum = 0;
>> @@ -196,6 +199,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
>>   	m->write_latency_max = 0;
>>   	m->total_writes = 0;
>>   	m->write_latency_sum = 0;
>> +	m->write_size_min = U64_MAX;
>> +	m->write_size_max = 0;
>> +	m->write_size_sum = 0;
>>   
>>
>>
>>
>>   	spin_lock_init(&m->metadata_metric_lock);
>>   	m->metadata_latency_sq_sum = 0;
>> @@ -306,9 +312,41 @@ static inline void __update_latency(struct ceph_client_metric *m,
>>   	*sq_sump += sq;
>>   }
>>   
>>
>>
>>
>> +static inline void __update_size(struct ceph_client_metric *m,
>> +				 metric_type type, unsigned int size)
>> +{
>> +	ktime_t total;
>> +	u64 *minp, *maxp, *sump;
>> +
>> +	switch (type) {
>> +	case CEPH_METRIC_READ:
>> +		total = m->total_reads;
>> +		sump = &m->read_size_sum;
>> +		minp = &m->read_size_min;
>> +		maxp = &m->read_size_max;
>> +		break;
>> +	case CEPH_METRIC_WRITE:
>> +		total = m->total_writes;
> "total" and "sump" are unused in this function, aside from the
> assignment.

Will fix it.

Just assuming to add the read/write IO speeds here, will it make sense ?


>> +		sump = &m->write_size_sum;
>> +		minp = &m->write_size_min;
>> +		maxp = &m->write_size_max;
>> +		break;
>> +	case CEPH_METRIC_METADATA:
>> +	default:
>> +		return;
>> +	}
>> +
>> +	*sump += size;
>> +
>> +	if (unlikely(size < *minp))
>> +		*minp = size;
>> +	if (unlikely(size > *maxp))
>> +		*maxp = size;
>> +}
>> +
>>   void ceph_update_read_metrics(struct ceph_client_metric *m,
>>   			      ktime_t r_start, ktime_t r_end,
>> -			      int rc)
>> +			      unsigned int size, int rc)
>>   {
>>   	ktime_t lat = ktime_sub(r_end, r_start);
>>   
>>
>>
>>
>> @@ -317,12 +355,13 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
>>   
>>
>>
>>
>>   	spin_lock(&m->read_metric_lock);
>>   	__update_latency(m, CEPH_METRIC_READ, lat);
>> +	__update_size(m, CEPH_METRIC_READ, size);
>>   	spin_unlock(&m->read_metric_lock);
>>   }
>>   
>>
>>
>>
>>   void ceph_update_write_metrics(struct ceph_client_metric *m,
>>   			       ktime_t r_start, ktime_t r_end,
>> -			       int rc)
>> +			       unsigned int size, int rc)
>>   {
>>   	ktime_t lat = ktime_sub(r_end, r_start);
>>   
>>
>>
>>
>> @@ -331,6 +370,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
>>   
>>
>>
>>
>>   	spin_lock(&m->write_metric_lock);
>>   	__update_latency(m, CEPH_METRIC_WRITE, lat);
>> +	__update_size(m, CEPH_METRIC_WRITE, size);
>>   	spin_unlock(&m->write_metric_lock);
>>   }
>>   
>>
>>
>>
>> diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
>> index 57b5f0ec38be..64651b6ac886 100644
>> --- a/fs/ceph/metric.h
>> +++ b/fs/ceph/metric.h
>> @@ -110,6 +110,9 @@ struct ceph_client_metric {
>>   
>>
>>
>>
>>   	spinlock_t read_metric_lock;
>>   	u64 total_reads;
>> +	u64 read_size_sum;
>> +	u64 read_size_min;
>> +	u64 read_size_max;
>>   	ktime_t read_latency_sum;
>>   	ktime_t read_latency_sq_sum;
>>   	ktime_t read_latency_min;
>> @@ -117,6 +120,9 @@ struct ceph_client_metric {
>>   
>>
>>
>>
>>   	spinlock_t write_metric_lock;
>>   	u64 total_writes;
>> +	u64 write_size_sum;
>> +	u64 write_size_min;
>> +	u64 write_size_max;
>>   	ktime_t write_latency_sum;
>>   	ktime_t write_latency_sq_sum;
>>   	ktime_t write_latency_min;
>> @@ -164,10 +170,10 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
>>   
>>
>>
>>
>>   extern void ceph_update_read_metrics(struct ceph_client_metric *m,
>>   				     ktime_t r_start, ktime_t r_end,
>> -				     int rc);
>> +				     unsigned int size, int rc);
>>   extern void ceph_update_write_metrics(struct ceph_client_metric *m,
>>   				      ktime_t r_start, ktime_t r_end,
>> -				      int rc);
>> +				      unsigned int size, int rc);
>>   extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
>>   				         ktime_t r_start, ktime_t r_end,
>>   					 int rc);

Re: [PATCH 0/4] ceph: add IO size metric support

[Jeff Layton]

On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
> From: Xiubo Li <xiubli@redhat.com>
> 
> Currently it will show as the following:
> 
> item          total       avg_sz(bytes)   min_sz(bytes)   max_sz(bytes)  total_sz(bytes)
> ----------------------------------------------------------------------------------------
> read          1           10240           10240           10240           10240
> write         1           10240           10240           10240           10240
> 
> 
> 
> Xiubo Li (4):
>   ceph: rename the metric helpers
>   ceph: update the __update_latency helper
>   ceph: avoid count the same request twice or more
>   ceph: add IO size metrics support
> 
>  fs/ceph/addr.c       |  20 +++----
>  fs/ceph/debugfs.c    |  49 +++++++++++++----
>  fs/ceph/file.c       |  47 ++++++++--------
>  fs/ceph/mds_client.c |   2 +-
>  fs/ceph/metric.c     | 126 ++++++++++++++++++++++++++++++++-----------
>  fs/ceph/metric.h     |  22 +++++---
>  6 files changed, 184 insertions(+), 82 deletions(-)
> 

I've gone ahead and merged patches 1 and 3 from this series into
ceph-client/testing. 1 was just a trivial renaming that we might as well
get out of the way, and 3 looked like a (minor) bugfix. The other two
still need a bit of work (but nothing major).

Cheers,
-- 
Jeff Layton <jlayton@kernel.org>

Re: [PATCH 0/4] ceph: add IO size metric support

[Xiubo Li]

On 2021/3/24 23:06, Jeff Layton wrote:
> On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
>> From: Xiubo Li <xiubli@redhat.com>
>>
>> Currently it will show as the following:
>>
>> item          total       avg_sz(bytes)   min_sz(bytes)   max_sz(bytes)  total_sz(bytes)
>> ----------------------------------------------------------------------------------------
>> read          1           10240           10240           10240           10240
>> write         1           10240           10240           10240           10240
>>
>>
>>
>> Xiubo Li (4):
>>    ceph: rename the metric helpers
>>    ceph: update the __update_latency helper
>>    ceph: avoid count the same request twice or more
>>    ceph: add IO size metrics support
>>
>>   fs/ceph/addr.c       |  20 +++----
>>   fs/ceph/debugfs.c    |  49 +++++++++++++----
>>   fs/ceph/file.c       |  47 ++++++++--------
>>   fs/ceph/mds_client.c |   2 +-
>>   fs/ceph/metric.c     | 126 ++++++++++++++++++++++++++++++++-----------
>>   fs/ceph/metric.h     |  22 +++++---
>>   6 files changed, 184 insertions(+), 82 deletions(-)
>>
> I've gone ahead and merged patches 1 and 3 from this series into
> ceph-client/testing. 1 was just a trivial renaming that we might as well
> get out of the way, and 3 looked like a (minor) bugfix. The other two
> still need a bit of work (but nothing major).

Sure, will fix them and post the v2 later.

Thanks Jeff.


> Cheers,