* [PATCH 0/4] ceph: add IO size metric support
@ 2021-03-22 12:28 xiubli
2021-03-22 12:28 ` [PATCH 1/4] ceph: rename the metric helpers xiubli
` (4 more replies)
0 siblings, 5 replies; 11+ messages in thread
From: xiubli @ 2021-03-22 12:28 UTC (permalink / raw)
To: jlayton; +Cc: idryomov, pdonnell, ceph-devel, Xiubo Li
From: Xiubo Li <xiubli@redhat.com>
Currently it will show as the following:
item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)
----------------------------------------------------------------------------------------
read 1 10240 10240 10240 10240
write 1 10240 10240 10240 10240
Xiubo Li (4):
ceph: rename the metric helpers
ceph: update the __update_latency helper
ceph: avoid count the same request twice or more
ceph: add IO size metrics support
fs/ceph/addr.c | 20 +++----
fs/ceph/debugfs.c | 49 +++++++++++++----
fs/ceph/file.c | 47 ++++++++--------
fs/ceph/mds_client.c | 2 +-
fs/ceph/metric.c | 126 ++++++++++++++++++++++++++++++++-----------
fs/ceph/metric.h | 22 +++++---
6 files changed, 184 insertions(+), 82 deletions(-)
--
2.27.0
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/4] ceph: rename the metric helpers
2021-03-22 12:28 [PATCH 0/4] ceph: add IO size metric support xiubli
@ 2021-03-22 12:28 ` xiubli
2021-03-22 12:28 ` [PATCH 2/4] ceph: update the __update_latency helper xiubli
` (3 subsequent siblings)
4 siblings, 0 replies; 11+ messages in thread
From: xiubli @ 2021-03-22 12:28 UTC (permalink / raw)
To: jlayton; +Cc: idryomov, pdonnell, ceph-devel, Xiubo Li
From: Xiubo Li <xiubli@redhat.com>
Prepare for the size metrics patches followed.
URL: https://tracker.ceph.com/issues/49913
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
fs/ceph/addr.c | 8 ++++----
fs/ceph/debugfs.c | 12 ++++++------
fs/ceph/file.c | 12 ++++++------
fs/ceph/mds_client.c | 2 +-
fs/ceph/metric.c | 24 ++++++++++++------------
fs/ceph/metric.h | 12 ++++++------
6 files changed, 35 insertions(+), 35 deletions(-)
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index b476133353ae..7c2802758d0e 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -226,7 +226,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
int num_pages;
int err = req->r_result;
- ceph_update_read_latency(&fsc->mdsc->metric, req->r_start_latency,
+ ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
req->r_end_latency, err);
dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
@@ -560,7 +560,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
if (!err)
err = ceph_osdc_wait_request(osdc, req);
- ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+ ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
req->r_end_latency, err);
ceph_osdc_put_request(req);
@@ -648,7 +648,7 @@ static void writepages_finish(struct ceph_osd_request *req)
ceph_clear_error_write(ci);
}
- ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+ ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
req->r_end_latency, rc);
/*
@@ -1719,7 +1719,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
if (!err)
err = ceph_osdc_wait_request(&fsc->client->osdc, req);
- ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+ ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
req->r_end_latency, err);
out_put:
diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
index 66989c880adb..425f3356332a 100644
--- a/fs/ceph/debugfs.c
+++ b/fs/ceph/debugfs.c
@@ -162,34 +162,34 @@ static int metric_show(struct seq_file *s, void *p)
seq_printf(s, "item total avg_lat(us) min_lat(us) max_lat(us) stdev(us)\n");
seq_printf(s, "-----------------------------------------------------------------------------------\n");
- spin_lock(&m->read_latency_lock);
+ spin_lock(&m->read_metric_lock);
total = m->total_reads;
sum = m->read_latency_sum;
avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
min = m->read_latency_min;
max = m->read_latency_max;
sq = m->read_latency_sq_sum;
- spin_unlock(&m->read_latency_lock);
+ spin_unlock(&m->read_metric_lock);
CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
- spin_lock(&m->write_latency_lock);
+ spin_lock(&m->write_metric_lock);
total = m->total_writes;
sum = m->write_latency_sum;
avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
min = m->write_latency_min;
max = m->write_latency_max;
sq = m->write_latency_sq_sum;
- spin_unlock(&m->write_latency_lock);
+ spin_unlock(&m->write_metric_lock);
CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
- spin_lock(&m->metadata_latency_lock);
+ spin_lock(&m->metadata_metric_lock);
total = m->total_metadatas;
sum = m->metadata_latency_sum;
avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
min = m->metadata_latency_min;
max = m->metadata_latency_max;
sq = m->metadata_latency_sq_sum;
- spin_unlock(&m->metadata_latency_lock);
+ spin_unlock(&m->metadata_metric_lock);
CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
seq_printf(s, "\n");
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index a6ef1d143308..a27aabcb0e0b 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -895,7 +895,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
if (!ret)
ret = ceph_osdc_wait_request(osdc, req);
- ceph_update_read_latency(&fsc->mdsc->metric,
+ ceph_update_read_metrics(&fsc->mdsc->metric,
req->r_start_latency,
req->r_end_latency,
ret);
@@ -1040,10 +1040,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
/* r_start_latency == 0 means the request was not submitted */
if (req->r_start_latency) {
if (aio_req->write)
- ceph_update_write_latency(metric, req->r_start_latency,
+ ceph_update_write_metrics(metric, req->r_start_latency,
req->r_end_latency, rc);
else
- ceph_update_read_latency(metric, req->r_start_latency,
+ ceph_update_read_metrics(metric, req->r_start_latency,
req->r_end_latency, rc);
}
@@ -1293,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
if (write)
- ceph_update_write_latency(metric, req->r_start_latency,
+ ceph_update_write_metrics(metric, req->r_start_latency,
req->r_end_latency, ret);
else
- ceph_update_read_latency(metric, req->r_start_latency,
+ ceph_update_read_metrics(metric, req->r_start_latency,
req->r_end_latency, ret);
size = i_size_read(inode);
@@ -1470,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
if (!ret)
ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
- ceph_update_write_latency(&fsc->mdsc->metric, req->r_start_latency,
+ ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
req->r_end_latency, ret);
out:
ceph_osdc_put_request(req);
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index d87bd852ed96..73ecb7d128c9 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -3306,7 +3306,7 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg)
/* kick calling process */
complete_request(mdsc, req);
- ceph_update_metadata_latency(&mdsc->metric, req->r_start_latency,
+ ceph_update_metadata_metrics(&mdsc->metric, req->r_start_latency,
req->r_end_latency, err);
out:
ceph_mdsc_put_request(req);
diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
index 5ec94bd4c1de..75d309f2fb0c 100644
--- a/fs/ceph/metric.c
+++ b/fs/ceph/metric.c
@@ -183,21 +183,21 @@ int ceph_metric_init(struct ceph_client_metric *m)
if (ret)
goto err_i_caps_mis;
- spin_lock_init(&m->read_latency_lock);
+ spin_lock_init(&m->read_metric_lock);
m->read_latency_sq_sum = 0;
m->read_latency_min = KTIME_MAX;
m->read_latency_max = 0;
m->total_reads = 0;
m->read_latency_sum = 0;
- spin_lock_init(&m->write_latency_lock);
+ spin_lock_init(&m->write_metric_lock);
m->write_latency_sq_sum = 0;
m->write_latency_min = KTIME_MAX;
m->write_latency_max = 0;
m->total_writes = 0;
m->write_latency_sum = 0;
- spin_lock_init(&m->metadata_latency_lock);
+ spin_lock_init(&m->metadata_metric_lock);
m->metadata_latency_sq_sum = 0;
m->metadata_latency_min = KTIME_MAX;
m->metadata_latency_max = 0;
@@ -274,7 +274,7 @@ static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
*sq_sump += sq;
}
-void ceph_update_read_latency(struct ceph_client_metric *m,
+void ceph_update_read_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
int rc)
{
@@ -283,14 +283,14 @@ void ceph_update_read_latency(struct ceph_client_metric *m,
if (unlikely(rc < 0 && rc != -ENOENT && rc != -ETIMEDOUT))
return;
- spin_lock(&m->read_latency_lock);
+ spin_lock(&m->read_metric_lock);
__update_latency(&m->total_reads, &m->read_latency_sum,
&m->read_latency_min, &m->read_latency_max,
&m->read_latency_sq_sum, lat);
- spin_unlock(&m->read_latency_lock);
+ spin_unlock(&m->read_metric_lock);
}
-void ceph_update_write_latency(struct ceph_client_metric *m,
+void ceph_update_write_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
int rc)
{
@@ -299,14 +299,14 @@ void ceph_update_write_latency(struct ceph_client_metric *m,
if (unlikely(rc && rc != -ETIMEDOUT))
return;
- spin_lock(&m->write_latency_lock);
+ spin_lock(&m->write_metric_lock);
__update_latency(&m->total_writes, &m->write_latency_sum,
&m->write_latency_min, &m->write_latency_max,
&m->write_latency_sq_sum, lat);
- spin_unlock(&m->write_latency_lock);
+ spin_unlock(&m->write_metric_lock);
}
-void ceph_update_metadata_latency(struct ceph_client_metric *m,
+void ceph_update_metadata_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
int rc)
{
@@ -315,9 +315,9 @@ void ceph_update_metadata_latency(struct ceph_client_metric *m,
if (unlikely(rc && rc != -ENOENT))
return;
- spin_lock(&m->metadata_latency_lock);
+ spin_lock(&m->metadata_metric_lock);
__update_latency(&m->total_metadatas, &m->metadata_latency_sum,
&m->metadata_latency_min, &m->metadata_latency_max,
&m->metadata_latency_sq_sum, lat);
- spin_unlock(&m->metadata_latency_lock);
+ spin_unlock(&m->metadata_metric_lock);
}
diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
index af6038ff39d4..57b5f0ec38be 100644
--- a/fs/ceph/metric.h
+++ b/fs/ceph/metric.h
@@ -108,21 +108,21 @@ struct ceph_client_metric {
struct percpu_counter i_caps_hit;
struct percpu_counter i_caps_mis;
- spinlock_t read_latency_lock;
+ spinlock_t read_metric_lock;
u64 total_reads;
ktime_t read_latency_sum;
ktime_t read_latency_sq_sum;
ktime_t read_latency_min;
ktime_t read_latency_max;
- spinlock_t write_latency_lock;
+ spinlock_t write_metric_lock;
u64 total_writes;
ktime_t write_latency_sum;
ktime_t write_latency_sq_sum;
ktime_t write_latency_min;
ktime_t write_latency_max;
- spinlock_t metadata_latency_lock;
+ spinlock_t metadata_metric_lock;
u64 total_metadatas;
ktime_t metadata_latency_sum;
ktime_t metadata_latency_sq_sum;
@@ -162,13 +162,13 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
percpu_counter_inc(&m->i_caps_mis);
}
-extern void ceph_update_read_latency(struct ceph_client_metric *m,
+extern void ceph_update_read_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
int rc);
-extern void ceph_update_write_latency(struct ceph_client_metric *m,
+extern void ceph_update_write_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
int rc);
-extern void ceph_update_metadata_latency(struct ceph_client_metric *m,
+extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
int rc);
#endif /* _FS_CEPH_MDS_METRIC_H */
--
2.27.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 2/4] ceph: update the __update_latency helper
2021-03-22 12:28 [PATCH 0/4] ceph: add IO size metric support xiubli
2021-03-22 12:28 ` [PATCH 1/4] ceph: rename the metric helpers xiubli
@ 2021-03-22 12:28 ` xiubli
2021-03-23 12:34 ` Jeff Layton
2021-03-22 12:28 ` [PATCH 3/4] ceph: avoid count the same request twice or more xiubli
` (2 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: xiubli @ 2021-03-22 12:28 UTC (permalink / raw)
To: jlayton; +Cc: idryomov, pdonnell, ceph-devel, Xiubo Li
From: Xiubo Li <xiubli@redhat.com>
Let the __update_latency() helper choose the correcsponding members
according to the metric_type.
URL: https://tracker.ceph.com/issues/49913
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
fs/ceph/metric.c | 58 +++++++++++++++++++++++++++++++++++-------------
1 file changed, 42 insertions(+), 16 deletions(-)
diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
index 75d309f2fb0c..d5560ff99a9d 100644
--- a/fs/ceph/metric.c
+++ b/fs/ceph/metric.c
@@ -249,19 +249,51 @@ void ceph_metric_destroy(struct ceph_client_metric *m)
ceph_put_mds_session(m->session);
}
-static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
- ktime_t *min, ktime_t *max,
- ktime_t *sq_sump, ktime_t lat)
+typedef enum {
+ CEPH_METRIC_READ,
+ CEPH_METRIC_WRITE,
+ CEPH_METRIC_METADATA,
+} metric_type;
+
+static inline void __update_latency(struct ceph_client_metric *m,
+ metric_type type, ktime_t lat)
{
+ ktime_t *totalp, *minp, *maxp, *lsump, *sq_sump;
ktime_t total, avg, sq, lsum;
+ switch (type) {
+ case CEPH_METRIC_READ:
+ totalp = &m->total_reads;
+ lsump = &m->read_latency_sum;
+ minp = &m->read_latency_min;
+ maxp = &m->read_latency_max;
+ sq_sump = &m->read_latency_sq_sum;
+ break;
+ case CEPH_METRIC_WRITE:
+ totalp = &m->total_writes;
+ lsump = &m->write_latency_sum;
+ minp = &m->write_latency_min;
+ maxp = &m->write_latency_max;
+ sq_sump = &m->write_latency_sq_sum;
+ break;
+ case CEPH_METRIC_METADATA:
+ totalp = &m->total_metadatas;
+ lsump = &m->metadata_latency_sum;
+ minp = &m->metadata_latency_min;
+ maxp = &m->metadata_latency_max;
+ sq_sump = &m->metadata_latency_sq_sum;
+ break;
+ default:
+ return;
+ }
+
total = ++(*totalp);
lsum = (*lsump += lat);
- if (unlikely(lat < *min))
- *min = lat;
- if (unlikely(lat > *max))
- *max = lat;
+ if (unlikely(lat < *minp))
+ *minp = lat;
+ if (unlikely(lat > *maxp))
+ *maxp = lat;
if (unlikely(total == 1))
return;
@@ -284,9 +316,7 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
return;
spin_lock(&m->read_metric_lock);
- __update_latency(&m->total_reads, &m->read_latency_sum,
- &m->read_latency_min, &m->read_latency_max,
- &m->read_latency_sq_sum, lat);
+ __update_latency(m, CEPH_METRIC_READ, lat);
spin_unlock(&m->read_metric_lock);
}
@@ -300,9 +330,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
return;
spin_lock(&m->write_metric_lock);
- __update_latency(&m->total_writes, &m->write_latency_sum,
- &m->write_latency_min, &m->write_latency_max,
- &m->write_latency_sq_sum, lat);
+ __update_latency(m, CEPH_METRIC_WRITE, lat);
spin_unlock(&m->write_metric_lock);
}
@@ -316,8 +344,6 @@ void ceph_update_metadata_metrics(struct ceph_client_metric *m,
return;
spin_lock(&m->metadata_metric_lock);
- __update_latency(&m->total_metadatas, &m->metadata_latency_sum,
- &m->metadata_latency_min, &m->metadata_latency_max,
- &m->metadata_latency_sq_sum, lat);
+ __update_latency(m, CEPH_METRIC_METADATA, lat);
spin_unlock(&m->metadata_metric_lock);
}
--
2.27.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 3/4] ceph: avoid count the same request twice or more
2021-03-22 12:28 [PATCH 0/4] ceph: add IO size metric support xiubli
2021-03-22 12:28 ` [PATCH 1/4] ceph: rename the metric helpers xiubli
2021-03-22 12:28 ` [PATCH 2/4] ceph: update the __update_latency helper xiubli
@ 2021-03-22 12:28 ` xiubli
2021-03-22 12:28 ` [PATCH 4/4] ceph: add IO size metrics support xiubli
2021-03-24 15:06 ` [PATCH 0/4] ceph: add IO size metric support Jeff Layton
4 siblings, 0 replies; 11+ messages in thread
From: xiubli @ 2021-03-22 12:28 UTC (permalink / raw)
To: jlayton; +Cc: idryomov, pdonnell, ceph-devel, Xiubo Li
From: Xiubo Li <xiubli@redhat.com>
If the request will retry, skip updating the latency metric.
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
fs/ceph/file.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index a27aabcb0e0b..31542eac7e59 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -1037,16 +1037,6 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
dout("ceph_aio_complete_req %p rc %d bytes %u\n",
inode, rc, osd_data->bvec_pos.iter.bi_size);
- /* r_start_latency == 0 means the request was not submitted */
- if (req->r_start_latency) {
- if (aio_req->write)
- ceph_update_write_metrics(metric, req->r_start_latency,
- req->r_end_latency, rc);
- else
- ceph_update_read_metrics(metric, req->r_start_latency,
- req->r_end_latency, rc);
- }
-
if (rc == -EOLDSNAPC) {
struct ceph_aio_work *aio_work;
BUG_ON(!aio_req->write);
@@ -1089,6 +1079,16 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
}
}
+ /* r_start_latency == 0 means the request was not submitted */
+ if (req->r_start_latency) {
+ if (aio_req->write)
+ ceph_update_write_metrics(metric, req->r_start_latency,
+ req->r_end_latency, rc);
+ else
+ ceph_update_read_metrics(metric, req->r_start_latency,
+ req->r_end_latency, rc);
+ }
+
put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
aio_req->should_dirty);
ceph_osdc_put_request(req);
--
2.27.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 4/4] ceph: add IO size metrics support
2021-03-22 12:28 [PATCH 0/4] ceph: add IO size metric support xiubli
` (2 preceding siblings ...)
2021-03-22 12:28 ` [PATCH 3/4] ceph: avoid count the same request twice or more xiubli
@ 2021-03-22 12:28 ` xiubli
2021-03-23 12:29 ` Jeff Layton
2021-03-24 15:06 ` [PATCH 0/4] ceph: add IO size metric support Jeff Layton
4 siblings, 1 reply; 11+ messages in thread
From: xiubli @ 2021-03-22 12:28 UTC (permalink / raw)
To: jlayton; +Cc: idryomov, pdonnell, ceph-devel, Xiubo Li
From: Xiubo Li <xiubli@redhat.com>
This will collect IO's total size and then calculate the average
size, and also will collect the min/max IO sizes.
The debugfs will show the size metrics in byte and will let the
userspace applications to switch to what they need.
URL: https://tracker.ceph.com/issues/49913
Signed-off-by: Xiubo Li <xiubli@redhat.com>
---
fs/ceph/addr.c | 14 ++++++++------
fs/ceph/debugfs.c | 37 +++++++++++++++++++++++++++++++++----
fs/ceph/file.c | 23 +++++++++++------------
fs/ceph/metric.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
fs/ceph/metric.h | 10 ++++++++--
5 files changed, 102 insertions(+), 26 deletions(-)
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 7c2802758d0e..d8a3624bc81d 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -227,7 +227,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
int err = req->r_result;
ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
- req->r_end_latency, err);
+ req->r_end_latency, osd_data->length, err);
dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
subreq->len, i_size_read(req->r_inode));
@@ -561,7 +561,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
err = ceph_osdc_wait_request(osdc, req);
ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
- req->r_end_latency, err);
+ req->r_end_latency, len, err);
ceph_osdc_put_request(req);
if (err == 0)
@@ -636,6 +636,7 @@ static void writepages_finish(struct ceph_osd_request *req)
struct ceph_snap_context *snapc = req->r_snapc;
struct address_space *mapping = inode->i_mapping;
struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
+ unsigned int len = 0;
bool remove_page;
dout("writepages_finish %p rc %d\n", inode, rc);
@@ -648,9 +649,6 @@ static void writepages_finish(struct ceph_osd_request *req)
ceph_clear_error_write(ci);
}
- ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
- req->r_end_latency, rc);
-
/*
* We lost the cache cap, need to truncate the page before
* it is unlocked, otherwise we'd truncate it later in the
@@ -667,6 +665,7 @@ static void writepages_finish(struct ceph_osd_request *req)
osd_data = osd_req_op_extent_osd_data(req, i);
BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+ len += osd_data->length;
num_pages = calc_pages_for((u64)osd_data->alignment,
(u64)osd_data->length);
total_pages += num_pages;
@@ -699,6 +698,9 @@ static void writepages_finish(struct ceph_osd_request *req)
release_pages(osd_data->pages, num_pages);
}
+ ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
+ req->r_end_latency, len, rc);
+
ceph_put_wrbuffer_cap_refs(ci, total_pages, snapc);
osd_data = osd_req_op_extent_osd_data(req, 0);
@@ -1720,7 +1722,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
err = ceph_osdc_wait_request(&fsc->client->osdc, req);
ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
- req->r_end_latency, err);
+ req->r_end_latency, len, err);
out_put:
ceph_osdc_put_request(req);
diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
index 425f3356332a..38b78b45811f 100644
--- a/fs/ceph/debugfs.c
+++ b/fs/ceph/debugfs.c
@@ -127,7 +127,7 @@ static int mdsc_show(struct seq_file *s, void *p)
return 0;
}
-#define CEPH_METRIC_SHOW(name, total, avg, min, max, sq) { \
+#define CEPH_LAT_METRIC_SHOW(name, total, avg, min, max, sq) { \
s64 _total, _avg, _min, _max, _sq, _st; \
_avg = ktime_to_us(avg); \
_min = ktime_to_us(min == KTIME_MAX ? 0 : min); \
@@ -140,6 +140,12 @@ static int mdsc_show(struct seq_file *s, void *p)
name, total, _avg, _min, _max, _st); \
}
+#define CEPH_SZ_METRIC_SHOW(name, total, avg, min, max, sum) { \
+ u64 _min = min == U64_MAX ? 0 : min; \
+ seq_printf(s, "%-14s%-12lld%-16llu%-16llu%-16llu%llu\n", \
+ name, total, avg, _min, max, sum); \
+}
+
static int metric_show(struct seq_file *s, void *p)
{
struct ceph_fs_client *fsc = s->private;
@@ -147,6 +153,7 @@ static int metric_show(struct seq_file *s, void *p)
struct ceph_client_metric *m = &mdsc->metric;
int nr_caps = 0;
s64 total, sum, avg, min, max, sq;
+ u64 sum_sz, avg_sz, min_sz, max_sz;
sum = percpu_counter_sum(&m->total_inodes);
seq_printf(s, "item total\n");
@@ -170,7 +177,7 @@ static int metric_show(struct seq_file *s, void *p)
max = m->read_latency_max;
sq = m->read_latency_sq_sum;
spin_unlock(&m->read_metric_lock);
- CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
+ CEPH_LAT_METRIC_SHOW("read", total, avg, min, max, sq);
spin_lock(&m->write_metric_lock);
total = m->total_writes;
@@ -180,7 +187,7 @@ static int metric_show(struct seq_file *s, void *p)
max = m->write_latency_max;
sq = m->write_latency_sq_sum;
spin_unlock(&m->write_metric_lock);
- CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
+ CEPH_LAT_METRIC_SHOW("write", total, avg, min, max, sq);
spin_lock(&m->metadata_metric_lock);
total = m->total_metadatas;
@@ -190,7 +197,29 @@ static int metric_show(struct seq_file *s, void *p)
max = m->metadata_latency_max;
sq = m->metadata_latency_sq_sum;
spin_unlock(&m->metadata_metric_lock);
- CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
+ CEPH_LAT_METRIC_SHOW("metadata", total, avg, min, max, sq);
+
+ seq_printf(s, "\n");
+ seq_printf(s, "item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)\n");
+ seq_printf(s, "----------------------------------------------------------------------------------------\n");
+
+ spin_lock(&m->read_metric_lock);
+ total = m->total_reads;
+ sum_sz = m->read_size_sum;
+ avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
+ min_sz = m->read_size_min;
+ max_sz = m->read_size_max;
+ spin_unlock(&m->read_metric_lock);
+ CEPH_SZ_METRIC_SHOW("read", total, avg_sz, min_sz, max_sz, sum_sz);
+
+ spin_lock(&m->write_metric_lock);
+ total = m->total_writes;
+ sum_sz = m->write_size_sum;
+ avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
+ min_sz = m->write_size_min;
+ max_sz = m->write_size_max;
+ spin_unlock(&m->write_metric_lock);
+ CEPH_SZ_METRIC_SHOW("write", total, avg_sz, min_sz, max_sz, sum_sz);
seq_printf(s, "\n");
seq_printf(s, "item total miss hit\n");
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index 31542eac7e59..db43d2d013b9 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -898,7 +898,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
ceph_update_read_metrics(&fsc->mdsc->metric,
req->r_start_latency,
req->r_end_latency,
- ret);
+ len, ret);
ceph_osdc_put_request(req);
@@ -1030,12 +1030,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
struct ceph_aio_request *aio_req = req->r_priv;
struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0);
struct ceph_client_metric *metric = &ceph_sb_to_mdsc(inode->i_sb)->metric;
+ unsigned int len = osd_data->bvec_pos.iter.bi_size;
BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS);
BUG_ON(!osd_data->num_bvecs);
- dout("ceph_aio_complete_req %p rc %d bytes %u\n",
- inode, rc, osd_data->bvec_pos.iter.bi_size);
+ dout("ceph_aio_complete_req %p rc %d bytes %u\n", inode, rc, len);
if (rc == -EOLDSNAPC) {
struct ceph_aio_work *aio_work;
@@ -1053,9 +1053,9 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
} else if (!aio_req->write) {
if (rc == -ENOENT)
rc = 0;
- if (rc >= 0 && osd_data->bvec_pos.iter.bi_size > rc) {
+ if (rc >= 0 && len > rc) {
struct iov_iter i;
- int zlen = osd_data->bvec_pos.iter.bi_size - rc;
+ int zlen = len - rc;
/*
* If read is satisfied by single OSD request,
@@ -1072,8 +1072,7 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
}
iov_iter_bvec(&i, READ, osd_data->bvec_pos.bvecs,
- osd_data->num_bvecs,
- osd_data->bvec_pos.iter.bi_size);
+ osd_data->num_bvecs, len);
iov_iter_advance(&i, rc);
iov_iter_zero(zlen, &i);
}
@@ -1083,10 +1082,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
if (req->r_start_latency) {
if (aio_req->write)
ceph_update_write_metrics(metric, req->r_start_latency,
- req->r_end_latency, rc);
+ req->r_end_latency, len, rc);
else
ceph_update_read_metrics(metric, req->r_start_latency,
- req->r_end_latency, rc);
+ req->r_end_latency, len, rc);
}
put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
@@ -1294,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
if (write)
ceph_update_write_metrics(metric, req->r_start_latency,
- req->r_end_latency, ret);
+ req->r_end_latency, len, ret);
else
ceph_update_read_metrics(metric, req->r_start_latency,
- req->r_end_latency, ret);
+ req->r_end_latency, len, ret);
size = i_size_read(inode);
if (!write) {
@@ -1471,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
- req->r_end_latency, ret);
+ req->r_end_latency, len, ret);
out:
ceph_osdc_put_request(req);
if (ret != 0) {
diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
index d5560ff99a9d..ff3c9d5cf9ff 100644
--- a/fs/ceph/metric.c
+++ b/fs/ceph/metric.c
@@ -189,6 +189,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
m->read_latency_max = 0;
m->total_reads = 0;
m->read_latency_sum = 0;
+ m->read_size_min = U64_MAX;
+ m->read_size_max = 0;
+ m->read_size_sum = 0;
spin_lock_init(&m->write_metric_lock);
m->write_latency_sq_sum = 0;
@@ -196,6 +199,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
m->write_latency_max = 0;
m->total_writes = 0;
m->write_latency_sum = 0;
+ m->write_size_min = U64_MAX;
+ m->write_size_max = 0;
+ m->write_size_sum = 0;
spin_lock_init(&m->metadata_metric_lock);
m->metadata_latency_sq_sum = 0;
@@ -306,9 +312,41 @@ static inline void __update_latency(struct ceph_client_metric *m,
*sq_sump += sq;
}
+static inline void __update_size(struct ceph_client_metric *m,
+ metric_type type, unsigned int size)
+{
+ ktime_t total;
+ u64 *minp, *maxp, *sump;
+
+ switch (type) {
+ case CEPH_METRIC_READ:
+ total = m->total_reads;
+ sump = &m->read_size_sum;
+ minp = &m->read_size_min;
+ maxp = &m->read_size_max;
+ break;
+ case CEPH_METRIC_WRITE:
+ total = m->total_writes;
+ sump = &m->write_size_sum;
+ minp = &m->write_size_min;
+ maxp = &m->write_size_max;
+ break;
+ case CEPH_METRIC_METADATA:
+ default:
+ return;
+ }
+
+ *sump += size;
+
+ if (unlikely(size < *minp))
+ *minp = size;
+ if (unlikely(size > *maxp))
+ *maxp = size;
+}
+
void ceph_update_read_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
- int rc)
+ unsigned int size, int rc)
{
ktime_t lat = ktime_sub(r_end, r_start);
@@ -317,12 +355,13 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
spin_lock(&m->read_metric_lock);
__update_latency(m, CEPH_METRIC_READ, lat);
+ __update_size(m, CEPH_METRIC_READ, size);
spin_unlock(&m->read_metric_lock);
}
void ceph_update_write_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
- int rc)
+ unsigned int size, int rc)
{
ktime_t lat = ktime_sub(r_end, r_start);
@@ -331,6 +370,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
spin_lock(&m->write_metric_lock);
__update_latency(m, CEPH_METRIC_WRITE, lat);
+ __update_size(m, CEPH_METRIC_WRITE, size);
spin_unlock(&m->write_metric_lock);
}
diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
index 57b5f0ec38be..64651b6ac886 100644
--- a/fs/ceph/metric.h
+++ b/fs/ceph/metric.h
@@ -110,6 +110,9 @@ struct ceph_client_metric {
spinlock_t read_metric_lock;
u64 total_reads;
+ u64 read_size_sum;
+ u64 read_size_min;
+ u64 read_size_max;
ktime_t read_latency_sum;
ktime_t read_latency_sq_sum;
ktime_t read_latency_min;
@@ -117,6 +120,9 @@ struct ceph_client_metric {
spinlock_t write_metric_lock;
u64 total_writes;
+ u64 write_size_sum;
+ u64 write_size_min;
+ u64 write_size_max;
ktime_t write_latency_sum;
ktime_t write_latency_sq_sum;
ktime_t write_latency_min;
@@ -164,10 +170,10 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
extern void ceph_update_read_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
- int rc);
+ unsigned int size, int rc);
extern void ceph_update_write_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
- int rc);
+ unsigned int size, int rc);
extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
ktime_t r_start, ktime_t r_end,
int rc);
--
2.27.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 4/4] ceph: add IO size metrics support
2021-03-22 12:28 ` [PATCH 4/4] ceph: add IO size metrics support xiubli
@ 2021-03-23 12:29 ` Jeff Layton
2021-03-23 13:17 ` Xiubo Li
0 siblings, 1 reply; 11+ messages in thread
From: Jeff Layton @ 2021-03-23 12:29 UTC (permalink / raw)
To: xiubli; +Cc: idryomov, pdonnell, ceph-devel
On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
> From: Xiubo Li <xiubli@redhat.com>
>
> This will collect IO's total size and then calculate the average
> size, and also will collect the min/max IO sizes.
>
> The debugfs will show the size metrics in byte and will let the
> userspace applications to switch to what they need.
>
> URL: https://tracker.ceph.com/issues/49913
> Signed-off-by: Xiubo Li <xiubli@redhat.com>
> ---
> fs/ceph/addr.c | 14 ++++++++------
> fs/ceph/debugfs.c | 37 +++++++++++++++++++++++++++++++++----
> fs/ceph/file.c | 23 +++++++++++------------
> fs/ceph/metric.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
> fs/ceph/metric.h | 10 ++++++++--
> 5 files changed, 102 insertions(+), 26 deletions(-)
>
> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
> index 7c2802758d0e..d8a3624bc81d 100644
> --- a/fs/ceph/addr.c
> +++ b/fs/ceph/addr.c
> @@ -227,7 +227,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
> int err = req->r_result;
>
>
>
>
> ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
> - req->r_end_latency, err);
> + req->r_end_latency, osd_data->length, err);
>
>
>
>
> dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
> subreq->len, i_size_read(req->r_inode));
> @@ -561,7 +561,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
> err = ceph_osdc_wait_request(osdc, req);
>
>
>
>
> ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> - req->r_end_latency, err);
> + req->r_end_latency, len, err);
>
>
>
>
> ceph_osdc_put_request(req);
> if (err == 0)
> @@ -636,6 +636,7 @@ static void writepages_finish(struct ceph_osd_request *req)
> struct ceph_snap_context *snapc = req->r_snapc;
> struct address_space *mapping = inode->i_mapping;
> struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
> + unsigned int len = 0;
> bool remove_page;
>
>
>
>
> dout("writepages_finish %p rc %d\n", inode, rc);
> @@ -648,9 +649,6 @@ static void writepages_finish(struct ceph_osd_request *req)
> ceph_clear_error_write(ci);
> }
>
>
>
>
> - ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> - req->r_end_latency, rc);
> -
> /*
> * We lost the cache cap, need to truncate the page before
> * it is unlocked, otherwise we'd truncate it later in the
> @@ -667,6 +665,7 @@ static void writepages_finish(struct ceph_osd_request *req)
>
>
>
>
> osd_data = osd_req_op_extent_osd_data(req, i);
> BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
> + len += osd_data->length;
> num_pages = calc_pages_for((u64)osd_data->alignment,
> (u64)osd_data->length);
> total_pages += num_pages;
> @@ -699,6 +698,9 @@ static void writepages_finish(struct ceph_osd_request *req)
> release_pages(osd_data->pages, num_pages);
> }
>
>
>
>
> + ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> + req->r_end_latency, len, rc);
> +
> ceph_put_wrbuffer_cap_refs(ci, total_pages, snapc);
>
>
>
>
> osd_data = osd_req_op_extent_osd_data(req, 0);
> @@ -1720,7 +1722,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
> err = ceph_osdc_wait_request(&fsc->client->osdc, req);
>
>
>
>
> ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> - req->r_end_latency, err);
> + req->r_end_latency, len, err);
>
>
>
>
> out_put:
> ceph_osdc_put_request(req);
> diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
> index 425f3356332a..38b78b45811f 100644
> --- a/fs/ceph/debugfs.c
> +++ b/fs/ceph/debugfs.c
> @@ -127,7 +127,7 @@ static int mdsc_show(struct seq_file *s, void *p)
> return 0;
> }
>
>
>
>
> -#define CEPH_METRIC_SHOW(name, total, avg, min, max, sq) { \
> +#define CEPH_LAT_METRIC_SHOW(name, total, avg, min, max, sq) { \
> s64 _total, _avg, _min, _max, _sq, _st; \
> _avg = ktime_to_us(avg); \
> _min = ktime_to_us(min == KTIME_MAX ? 0 : min); \
> @@ -140,6 +140,12 @@ static int mdsc_show(struct seq_file *s, void *p)
> name, total, _avg, _min, _max, _st); \
> }
>
>
>
>
> +#define CEPH_SZ_METRIC_SHOW(name, total, avg, min, max, sum) { \
> + u64 _min = min == U64_MAX ? 0 : min; \
> + seq_printf(s, "%-14s%-12lld%-16llu%-16llu%-16llu%llu\n", \
> + name, total, avg, _min, max, sum); \
> +}
> +
> static int metric_show(struct seq_file *s, void *p)
> {
> struct ceph_fs_client *fsc = s->private;
> @@ -147,6 +153,7 @@ static int metric_show(struct seq_file *s, void *p)
> struct ceph_client_metric *m = &mdsc->metric;
> int nr_caps = 0;
> s64 total, sum, avg, min, max, sq;
> + u64 sum_sz, avg_sz, min_sz, max_sz;
>
>
>
>
> sum = percpu_counter_sum(&m->total_inodes);
> seq_printf(s, "item total\n");
> @@ -170,7 +177,7 @@ static int metric_show(struct seq_file *s, void *p)
> max = m->read_latency_max;
> sq = m->read_latency_sq_sum;
> spin_unlock(&m->read_metric_lock);
> - CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
> + CEPH_LAT_METRIC_SHOW("read", total, avg, min, max, sq);
>
>
>
>
> spin_lock(&m->write_metric_lock);
> total = m->total_writes;
> @@ -180,7 +187,7 @@ static int metric_show(struct seq_file *s, void *p)
> max = m->write_latency_max;
> sq = m->write_latency_sq_sum;
> spin_unlock(&m->write_metric_lock);
> - CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
> + CEPH_LAT_METRIC_SHOW("write", total, avg, min, max, sq);
>
>
>
>
> spin_lock(&m->metadata_metric_lock);
> total = m->total_metadatas;
> @@ -190,7 +197,29 @@ static int metric_show(struct seq_file *s, void *p)
> max = m->metadata_latency_max;
> sq = m->metadata_latency_sq_sum;
> spin_unlock(&m->metadata_metric_lock);
> - CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
> + CEPH_LAT_METRIC_SHOW("metadata", total, avg, min, max, sq);
> +
> + seq_printf(s, "\n");
> + seq_printf(s, "item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)\n");
> + seq_printf(s, "----------------------------------------------------------------------------------------\n");
> +
> + spin_lock(&m->read_metric_lock);
> + total = m->total_reads;
> + sum_sz = m->read_size_sum;
> + avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
> + min_sz = m->read_size_min;
> + max_sz = m->read_size_max;
> + spin_unlock(&m->read_metric_lock);
> + CEPH_SZ_METRIC_SHOW("read", total, avg_sz, min_sz, max_sz, sum_sz);
> +
> + spin_lock(&m->write_metric_lock);
> + total = m->total_writes;
> + sum_sz = m->write_size_sum;
> + avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
> + min_sz = m->write_size_min;
> + max_sz = m->write_size_max;
> + spin_unlock(&m->write_metric_lock);
> + CEPH_SZ_METRIC_SHOW("write", total, avg_sz, min_sz, max_sz, sum_sz);
>
>
>
>
> seq_printf(s, "\n");
> seq_printf(s, "item total miss hit\n");
> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
> index 31542eac7e59..db43d2d013b9 100644
> --- a/fs/ceph/file.c
> +++ b/fs/ceph/file.c
> @@ -898,7 +898,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
> ceph_update_read_metrics(&fsc->mdsc->metric,
> req->r_start_latency,
> req->r_end_latency,
> - ret);
> + len, ret);
>
>
>
>
> ceph_osdc_put_request(req);
>
>
>
>
> @@ -1030,12 +1030,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
> struct ceph_aio_request *aio_req = req->r_priv;
> struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0);
> struct ceph_client_metric *metric = &ceph_sb_to_mdsc(inode->i_sb)->metric;
> + unsigned int len = osd_data->bvec_pos.iter.bi_size;
>
>
>
>
> BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS);
> BUG_ON(!osd_data->num_bvecs);
>
>
>
>
> - dout("ceph_aio_complete_req %p rc %d bytes %u\n",
> - inode, rc, osd_data->bvec_pos.iter.bi_size);
> + dout("ceph_aio_complete_req %p rc %d bytes %u\n", inode, rc, len);
>
>
>
>
> if (rc == -EOLDSNAPC) {
> struct ceph_aio_work *aio_work;
> @@ -1053,9 +1053,9 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
> } else if (!aio_req->write) {
> if (rc == -ENOENT)
> rc = 0;
> - if (rc >= 0 && osd_data->bvec_pos.iter.bi_size > rc) {
> + if (rc >= 0 && len > rc) {
> struct iov_iter i;
> - int zlen = osd_data->bvec_pos.iter.bi_size - rc;
> + int zlen = len - rc;
>
>
>
>
> /*
> * If read is satisfied by single OSD request,
> @@ -1072,8 +1072,7 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
> }
>
>
>
>
> iov_iter_bvec(&i, READ, osd_data->bvec_pos.bvecs,
> - osd_data->num_bvecs,
> - osd_data->bvec_pos.iter.bi_size);
> + osd_data->num_bvecs, len);
> iov_iter_advance(&i, rc);
> iov_iter_zero(zlen, &i);
> }
> @@ -1083,10 +1082,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
> if (req->r_start_latency) {
> if (aio_req->write)
> ceph_update_write_metrics(metric, req->r_start_latency,
> - req->r_end_latency, rc);
> + req->r_end_latency, len, rc);
> else
> ceph_update_read_metrics(metric, req->r_start_latency,
> - req->r_end_latency, rc);
> + req->r_end_latency, len, rc);
> }
>
>
>
>
> put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
> @@ -1294,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
>
>
>
>
> if (write)
> ceph_update_write_metrics(metric, req->r_start_latency,
> - req->r_end_latency, ret);
> + req->r_end_latency, len, ret);
> else
> ceph_update_read_metrics(metric, req->r_start_latency,
> - req->r_end_latency, ret);
> + req->r_end_latency, len, ret);
>
>
>
>
> size = i_size_read(inode);
> if (!write) {
> @@ -1471,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
> ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
>
>
>
>
> ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
> - req->r_end_latency, ret);
> + req->r_end_latency, len, ret);
> out:
> ceph_osdc_put_request(req);
> if (ret != 0) {
> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
> index d5560ff99a9d..ff3c9d5cf9ff 100644
> --- a/fs/ceph/metric.c
> +++ b/fs/ceph/metric.c
> @@ -189,6 +189,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
> m->read_latency_max = 0;
> m->total_reads = 0;
> m->read_latency_sum = 0;
> + m->read_size_min = U64_MAX;
> + m->read_size_max = 0;
> + m->read_size_sum = 0;
>
>
>
>
> spin_lock_init(&m->write_metric_lock);
> m->write_latency_sq_sum = 0;
> @@ -196,6 +199,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
> m->write_latency_max = 0;
> m->total_writes = 0;
> m->write_latency_sum = 0;
> + m->write_size_min = U64_MAX;
> + m->write_size_max = 0;
> + m->write_size_sum = 0;
>
>
>
>
> spin_lock_init(&m->metadata_metric_lock);
> m->metadata_latency_sq_sum = 0;
> @@ -306,9 +312,41 @@ static inline void __update_latency(struct ceph_client_metric *m,
> *sq_sump += sq;
> }
>
>
>
>
> +static inline void __update_size(struct ceph_client_metric *m,
> + metric_type type, unsigned int size)
> +{
> + ktime_t total;
> + u64 *minp, *maxp, *sump;
> +
> + switch (type) {
> + case CEPH_METRIC_READ:
> + total = m->total_reads;
> + sump = &m->read_size_sum;
> + minp = &m->read_size_min;
> + maxp = &m->read_size_max;
> + break;
> + case CEPH_METRIC_WRITE:
> + total = m->total_writes;
"total" and "sump" are unused in this function, aside from the
assignment.
> + sump = &m->write_size_sum;
> + minp = &m->write_size_min;
> + maxp = &m->write_size_max;
> + break;
> + case CEPH_METRIC_METADATA:
> + default:
> + return;
> + }
> +
> + *sump += size;
> +
> + if (unlikely(size < *minp))
> + *minp = size;
> + if (unlikely(size > *maxp))
> + *maxp = size;
> +}
> +
> void ceph_update_read_metrics(struct ceph_client_metric *m,
> ktime_t r_start, ktime_t r_end,
> - int rc)
> + unsigned int size, int rc)
> {
> ktime_t lat = ktime_sub(r_end, r_start);
>
>
>
>
> @@ -317,12 +355,13 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
>
>
>
>
> spin_lock(&m->read_metric_lock);
> __update_latency(m, CEPH_METRIC_READ, lat);
> + __update_size(m, CEPH_METRIC_READ, size);
> spin_unlock(&m->read_metric_lock);
> }
>
>
>
>
> void ceph_update_write_metrics(struct ceph_client_metric *m,
> ktime_t r_start, ktime_t r_end,
> - int rc)
> + unsigned int size, int rc)
> {
> ktime_t lat = ktime_sub(r_end, r_start);
>
>
>
>
> @@ -331,6 +370,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
>
>
>
>
> spin_lock(&m->write_metric_lock);
> __update_latency(m, CEPH_METRIC_WRITE, lat);
> + __update_size(m, CEPH_METRIC_WRITE, size);
> spin_unlock(&m->write_metric_lock);
> }
>
>
>
>
> diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
> index 57b5f0ec38be..64651b6ac886 100644
> --- a/fs/ceph/metric.h
> +++ b/fs/ceph/metric.h
> @@ -110,6 +110,9 @@ struct ceph_client_metric {
>
>
>
>
> spinlock_t read_metric_lock;
> u64 total_reads;
> + u64 read_size_sum;
> + u64 read_size_min;
> + u64 read_size_max;
> ktime_t read_latency_sum;
> ktime_t read_latency_sq_sum;
> ktime_t read_latency_min;
> @@ -117,6 +120,9 @@ struct ceph_client_metric {
>
>
>
>
> spinlock_t write_metric_lock;
> u64 total_writes;
> + u64 write_size_sum;
> + u64 write_size_min;
> + u64 write_size_max;
> ktime_t write_latency_sum;
> ktime_t write_latency_sq_sum;
> ktime_t write_latency_min;
> @@ -164,10 +170,10 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
>
>
>
>
> extern void ceph_update_read_metrics(struct ceph_client_metric *m,
> ktime_t r_start, ktime_t r_end,
> - int rc);
> + unsigned int size, int rc);
> extern void ceph_update_write_metrics(struct ceph_client_metric *m,
> ktime_t r_start, ktime_t r_end,
> - int rc);
> + unsigned int size, int rc);
> extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
> ktime_t r_start, ktime_t r_end,
> int rc);
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/4] ceph: update the __update_latency helper
2021-03-22 12:28 ` [PATCH 2/4] ceph: update the __update_latency helper xiubli
@ 2021-03-23 12:34 ` Jeff Layton
2021-03-23 13:14 ` Xiubo Li
0 siblings, 1 reply; 11+ messages in thread
From: Jeff Layton @ 2021-03-23 12:34 UTC (permalink / raw)
To: xiubli; +Cc: idryomov, pdonnell, ceph-devel
On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
> From: Xiubo Li <xiubli@redhat.com>
>
> Let the __update_latency() helper choose the correcsponding members
> according to the metric_type.
>
> URL: https://tracker.ceph.com/issues/49913
> Signed-off-by: Xiubo Li <xiubli@redhat.com>
> ---
> fs/ceph/metric.c | 58 +++++++++++++++++++++++++++++++++++-------------
> 1 file changed, 42 insertions(+), 16 deletions(-)
>
> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
> index 75d309f2fb0c..d5560ff99a9d 100644
> --- a/fs/ceph/metric.c
> +++ b/fs/ceph/metric.c
> @@ -249,19 +249,51 @@ void ceph_metric_destroy(struct ceph_client_metric *m)
> ceph_put_mds_session(m->session);
> }
>
>
>
>
>
>
>
>
> -static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
> - ktime_t *min, ktime_t *max,
> - ktime_t *sq_sump, ktime_t lat)
> +typedef enum {
> + CEPH_METRIC_READ,
> + CEPH_METRIC_WRITE,
> + CEPH_METRIC_METADATA,
> +} metric_type;
> +
> +static inline void __update_latency(struct ceph_client_metric *m,
> + metric_type type, ktime_t lat)
> {
> + ktime_t *totalp, *minp, *maxp, *lsump, *sq_sump;
> ktime_t total, avg, sq, lsum;
>
>
>
>
>
>
>
>
> + switch (type) {
> + case CEPH_METRIC_READ:
> + totalp = &m->total_reads;
> + lsump = &m->read_latency_sum;
> + minp = &m->read_latency_min;
> + maxp = &m->read_latency_max;
> + sq_sump = &m->read_latency_sq_sum;
> + break;
> + case CEPH_METRIC_WRITE:
> + totalp = &m->total_writes;
> + lsump = &m->write_latency_sum;
> + minp = &m->write_latency_min;
> + maxp = &m->write_latency_max;
> + sq_sump = &m->write_latency_sq_sum;
> + break;
> + case CEPH_METRIC_METADATA:
> + totalp = &m->total_metadatas;
> + lsump = &m->metadata_latency_sum;
> + minp = &m->metadata_latency_min;
> + maxp = &m->metadata_latency_max;
> + sq_sump = &m->metadata_latency_sq_sum;
> + break;
> + default:
> + return;
> + }
> +
> total = ++(*totalp);
Why are you adding one to *totalp above? Is that to avoid it being 0?
> lsum = (*lsump += lat);
>
>
^^^
Instead of doing all of the above with pointers, why not just add to
total and lsum directly inside the switch statement? This seems like a
lot of pointless indirection.
>
>
>
>
>
> - if (unlikely(lat < *min))
> - *min = lat;
> - if (unlikely(lat > *max))
> - *max = lat;
> + if (unlikely(lat < *minp))
> + *minp = lat;
> + if (unlikely(lat > *maxp))
> + *maxp = lat;
>
>
>
>
> if (unlikely(total == 1))
> return;
> @@ -284,9 +316,7 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
> return;
>
>
>
>
> spin_lock(&m->read_metric_lock);
> - __update_latency(&m->total_reads, &m->read_latency_sum,
> - &m->read_latency_min, &m->read_latency_max,
> - &m->read_latency_sq_sum, lat);
> + __update_latency(m, CEPH_METRIC_READ, lat);
> spin_unlock(&m->read_metric_lock);
> }
>
>
>
>
> @@ -300,9 +330,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
> return;
>
>
>
>
> spin_lock(&m->write_metric_lock);
> - __update_latency(&m->total_writes, &m->write_latency_sum,
> - &m->write_latency_min, &m->write_latency_max,
> - &m->write_latency_sq_sum, lat);
> + __update_latency(m, CEPH_METRIC_WRITE, lat);
> spin_unlock(&m->write_metric_lock);
> }
>
>
>
>
> @@ -316,8 +344,6 @@ void ceph_update_metadata_metrics(struct ceph_client_metric *m,
> return;
>
>
>
>
> spin_lock(&m->metadata_metric_lock);
> - __update_latency(&m->total_metadatas, &m->metadata_latency_sum,
> - &m->metadata_latency_min, &m->metadata_latency_max,
> - &m->metadata_latency_sq_sum, lat);
> + __update_latency(m, CEPH_METRIC_METADATA, lat);
> spin_unlock(&m->metadata_metric_lock);
> }
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/4] ceph: update the __update_latency helper
2021-03-23 12:34 ` Jeff Layton
@ 2021-03-23 13:14 ` Xiubo Li
0 siblings, 0 replies; 11+ messages in thread
From: Xiubo Li @ 2021-03-23 13:14 UTC (permalink / raw)
To: Jeff Layton; +Cc: idryomov, pdonnell, ceph-devel
On 2021/3/23 20:34, Jeff Layton wrote:
> On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
>> From: Xiubo Li <xiubli@redhat.com>
>>
>> Let the __update_latency() helper choose the correcsponding members
>> according to the metric_type.
>>
>> URL: https://tracker.ceph.com/issues/49913
>> Signed-off-by: Xiubo Li <xiubli@redhat.com>
>> ---
>> fs/ceph/metric.c | 58 +++++++++++++++++++++++++++++++++++-------------
>> 1 file changed, 42 insertions(+), 16 deletions(-)
>>
>> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
>> index 75d309f2fb0c..d5560ff99a9d 100644
>> --- a/fs/ceph/metric.c
>> +++ b/fs/ceph/metric.c
>> @@ -249,19 +249,51 @@ void ceph_metric_destroy(struct ceph_client_metric *m)
>> ceph_put_mds_session(m->session);
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>> -static inline void __update_latency(ktime_t *totalp, ktime_t *lsump,
>> - ktime_t *min, ktime_t *max,
>> - ktime_t *sq_sump, ktime_t lat)
>> +typedef enum {
>> + CEPH_METRIC_READ,
>> + CEPH_METRIC_WRITE,
>> + CEPH_METRIC_METADATA,
>> +} metric_type;
>> +
>> +static inline void __update_latency(struct ceph_client_metric *m,
>> + metric_type type, ktime_t lat)
>> {
>> + ktime_t *totalp, *minp, *maxp, *lsump, *sq_sump;
>> ktime_t total, avg, sq, lsum;
>>
>>
>>
>>
>>
>>
>>
>>
>> + switch (type) {
>> + case CEPH_METRIC_READ:
>> + totalp = &m->total_reads;
>> + lsump = &m->read_latency_sum;
>> + minp = &m->read_latency_min;
>> + maxp = &m->read_latency_max;
>> + sq_sump = &m->read_latency_sq_sum;
>> + break;
>> + case CEPH_METRIC_WRITE:
>> + totalp = &m->total_writes;
>> + lsump = &m->write_latency_sum;
>> + minp = &m->write_latency_min;
>> + maxp = &m->write_latency_max;
>> + sq_sump = &m->write_latency_sq_sum;
>> + break;
>> + case CEPH_METRIC_METADATA:
>> + totalp = &m->total_metadatas;
>> + lsump = &m->metadata_latency_sum;
>> + minp = &m->metadata_latency_min;
>> + maxp = &m->metadata_latency_max;
>> + sq_sump = &m->metadata_latency_sq_sum;
>> + break;
>> + default:
>> + return;
>> + }
>> +
>> total = ++(*totalp);
> Why are you adding one to *totalp above? Is that to avoid it being 0?
No, in the old code we will count the
total_reads/total_writes/total_metadatas for each call of the
ceph_update_{read/write/metadata}_latency() helpers. And the same here.
>> lsum = (*lsump += lat);
>>
>>
> ^^^
> Instead of doing all of the above with pointers, why not just add to
> total and lsum directly inside the switch statement? This seems like a
> lot of pointless indirection.
Okay, sounds good, will change it.
>>
>>
>>
>>
>> - if (unlikely(lat < *min))
>> - *min = lat;
>> - if (unlikely(lat > *max))
>> - *max = lat;
>> + if (unlikely(lat < *minp))
>> + *minp = lat;
>> + if (unlikely(lat > *maxp))
>> + *maxp = lat;
>>
>>
>>
>>
>> if (unlikely(total == 1))
>> return;
>> @@ -284,9 +316,7 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
>> return;
>>
>>
>>
>>
>> spin_lock(&m->read_metric_lock);
>> - __update_latency(&m->total_reads, &m->read_latency_sum,
>> - &m->read_latency_min, &m->read_latency_max,
>> - &m->read_latency_sq_sum, lat);
>> + __update_latency(m, CEPH_METRIC_READ, lat);
>> spin_unlock(&m->read_metric_lock);
>> }
>>
>>
>>
>>
>> @@ -300,9 +330,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
>> return;
>>
>>
>>
>>
>> spin_lock(&m->write_metric_lock);
>> - __update_latency(&m->total_writes, &m->write_latency_sum,
>> - &m->write_latency_min, &m->write_latency_max,
>> - &m->write_latency_sq_sum, lat);
>> + __update_latency(m, CEPH_METRIC_WRITE, lat);
>> spin_unlock(&m->write_metric_lock);
>> }
>>
>>
>>
>>
>> @@ -316,8 +344,6 @@ void ceph_update_metadata_metrics(struct ceph_client_metric *m,
>> return;
>>
>>
>>
>>
>> spin_lock(&m->metadata_metric_lock);
>> - __update_latency(&m->total_metadatas, &m->metadata_latency_sum,
>> - &m->metadata_latency_min, &m->metadata_latency_max,
>> - &m->metadata_latency_sq_sum, lat);
>> + __update_latency(m, CEPH_METRIC_METADATA, lat);
>> spin_unlock(&m->metadata_metric_lock);
>> }
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 4/4] ceph: add IO size metrics support
2021-03-23 12:29 ` Jeff Layton
@ 2021-03-23 13:17 ` Xiubo Li
0 siblings, 0 replies; 11+ messages in thread
From: Xiubo Li @ 2021-03-23 13:17 UTC (permalink / raw)
To: Jeff Layton; +Cc: idryomov, pdonnell, ceph-devel
On 2021/3/23 20:29, Jeff Layton wrote:
> On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
>> From: Xiubo Li <xiubli@redhat.com>
>>
>> This will collect IO's total size and then calculate the average
>> size, and also will collect the min/max IO sizes.
>>
>> The debugfs will show the size metrics in byte and will let the
>> userspace applications to switch to what they need.
>>
>> URL: https://tracker.ceph.com/issues/49913
>> Signed-off-by: Xiubo Li <xiubli@redhat.com>
>> ---
>> fs/ceph/addr.c | 14 ++++++++------
>> fs/ceph/debugfs.c | 37 +++++++++++++++++++++++++++++++++----
>> fs/ceph/file.c | 23 +++++++++++------------
>> fs/ceph/metric.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
>> fs/ceph/metric.h | 10 ++++++++--
>> 5 files changed, 102 insertions(+), 26 deletions(-)
>>
>> diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
>> index 7c2802758d0e..d8a3624bc81d 100644
>> --- a/fs/ceph/addr.c
>> +++ b/fs/ceph/addr.c
>> @@ -227,7 +227,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
>> int err = req->r_result;
>>
>>
>>
>>
>> ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> - req->r_end_latency, err);
>> + req->r_end_latency, osd_data->length, err);
>>
>>
>>
>>
>> dout("%s: result %d subreq->len=%zu i_size=%lld\n", __func__, req->r_result,
>> subreq->len, i_size_read(req->r_inode));
>> @@ -561,7 +561,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)
>> err = ceph_osdc_wait_request(osdc, req);
>>
>>
>>
>>
>> ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> - req->r_end_latency, err);
>> + req->r_end_latency, len, err);
>>
>>
>>
>>
>> ceph_osdc_put_request(req);
>> if (err == 0)
>> @@ -636,6 +636,7 @@ static void writepages_finish(struct ceph_osd_request *req)
>> struct ceph_snap_context *snapc = req->r_snapc;
>> struct address_space *mapping = inode->i_mapping;
>> struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
>> + unsigned int len = 0;
>> bool remove_page;
>>
>>
>>
>>
>> dout("writepages_finish %p rc %d\n", inode, rc);
>> @@ -648,9 +649,6 @@ static void writepages_finish(struct ceph_osd_request *req)
>> ceph_clear_error_write(ci);
>> }
>>
>>
>>
>>
>> - ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> - req->r_end_latency, rc);
>> -
>> /*
>> * We lost the cache cap, need to truncate the page before
>> * it is unlocked, otherwise we'd truncate it later in the
>> @@ -667,6 +665,7 @@ static void writepages_finish(struct ceph_osd_request *req)
>>
>>
>>
>>
>> osd_data = osd_req_op_extent_osd_data(req, i);
>> BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
>> + len += osd_data->length;
>> num_pages = calc_pages_for((u64)osd_data->alignment,
>> (u64)osd_data->length);
>> total_pages += num_pages;
>> @@ -699,6 +698,9 @@ static void writepages_finish(struct ceph_osd_request *req)
>> release_pages(osd_data->pages, num_pages);
>> }
>>
>>
>>
>>
>> + ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> + req->r_end_latency, len, rc);
>> +
>> ceph_put_wrbuffer_cap_refs(ci, total_pages, snapc);
>>
>>
>>
>>
>> osd_data = osd_req_op_extent_osd_data(req, 0);
>> @@ -1720,7 +1722,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)
>> err = ceph_osdc_wait_request(&fsc->client->osdc, req);
>>
>>
>>
>>
>> ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> - req->r_end_latency, err);
>> + req->r_end_latency, len, err);
>>
>>
>>
>>
>> out_put:
>> ceph_osdc_put_request(req);
>> diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
>> index 425f3356332a..38b78b45811f 100644
>> --- a/fs/ceph/debugfs.c
>> +++ b/fs/ceph/debugfs.c
>> @@ -127,7 +127,7 @@ static int mdsc_show(struct seq_file *s, void *p)
>> return 0;
>> }
>>
>>
>>
>>
>> -#define CEPH_METRIC_SHOW(name, total, avg, min, max, sq) { \
>> +#define CEPH_LAT_METRIC_SHOW(name, total, avg, min, max, sq) { \
>> s64 _total, _avg, _min, _max, _sq, _st; \
>> _avg = ktime_to_us(avg); \
>> _min = ktime_to_us(min == KTIME_MAX ? 0 : min); \
>> @@ -140,6 +140,12 @@ static int mdsc_show(struct seq_file *s, void *p)
>> name, total, _avg, _min, _max, _st); \
>> }
>>
>>
>>
>>
>> +#define CEPH_SZ_METRIC_SHOW(name, total, avg, min, max, sum) { \
>> + u64 _min = min == U64_MAX ? 0 : min; \
>> + seq_printf(s, "%-14s%-12lld%-16llu%-16llu%-16llu%llu\n", \
>> + name, total, avg, _min, max, sum); \
>> +}
>> +
>> static int metric_show(struct seq_file *s, void *p)
>> {
>> struct ceph_fs_client *fsc = s->private;
>> @@ -147,6 +153,7 @@ static int metric_show(struct seq_file *s, void *p)
>> struct ceph_client_metric *m = &mdsc->metric;
>> int nr_caps = 0;
>> s64 total, sum, avg, min, max, sq;
>> + u64 sum_sz, avg_sz, min_sz, max_sz;
>>
>>
>>
>>
>> sum = percpu_counter_sum(&m->total_inodes);
>> seq_printf(s, "item total\n");
>> @@ -170,7 +177,7 @@ static int metric_show(struct seq_file *s, void *p)
>> max = m->read_latency_max;
>> sq = m->read_latency_sq_sum;
>> spin_unlock(&m->read_metric_lock);
>> - CEPH_METRIC_SHOW("read", total, avg, min, max, sq);
>> + CEPH_LAT_METRIC_SHOW("read", total, avg, min, max, sq);
>>
>>
>>
>>
>> spin_lock(&m->write_metric_lock);
>> total = m->total_writes;
>> @@ -180,7 +187,7 @@ static int metric_show(struct seq_file *s, void *p)
>> max = m->write_latency_max;
>> sq = m->write_latency_sq_sum;
>> spin_unlock(&m->write_metric_lock);
>> - CEPH_METRIC_SHOW("write", total, avg, min, max, sq);
>> + CEPH_LAT_METRIC_SHOW("write", total, avg, min, max, sq);
>>
>>
>>
>>
>> spin_lock(&m->metadata_metric_lock);
>> total = m->total_metadatas;
>> @@ -190,7 +197,29 @@ static int metric_show(struct seq_file *s, void *p)
>> max = m->metadata_latency_max;
>> sq = m->metadata_latency_sq_sum;
>> spin_unlock(&m->metadata_metric_lock);
>> - CEPH_METRIC_SHOW("metadata", total, avg, min, max, sq);
>> + CEPH_LAT_METRIC_SHOW("metadata", total, avg, min, max, sq);
>> +
>> + seq_printf(s, "\n");
>> + seq_printf(s, "item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)\n");
>> + seq_printf(s, "----------------------------------------------------------------------------------------\n");
>> +
>> + spin_lock(&m->read_metric_lock);
>> + total = m->total_reads;
>> + sum_sz = m->read_size_sum;
>> + avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
>> + min_sz = m->read_size_min;
>> + max_sz = m->read_size_max;
>> + spin_unlock(&m->read_metric_lock);
>> + CEPH_SZ_METRIC_SHOW("read", total, avg_sz, min_sz, max_sz, sum_sz);
>> +
>> + spin_lock(&m->write_metric_lock);
>> + total = m->total_writes;
>> + sum_sz = m->write_size_sum;
>> + avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
>> + min_sz = m->write_size_min;
>> + max_sz = m->write_size_max;
>> + spin_unlock(&m->write_metric_lock);
>> + CEPH_SZ_METRIC_SHOW("write", total, avg_sz, min_sz, max_sz, sum_sz);
>>
>>
>>
>>
>> seq_printf(s, "\n");
>> seq_printf(s, "item total miss hit\n");
>> diff --git a/fs/ceph/file.c b/fs/ceph/file.c
>> index 31542eac7e59..db43d2d013b9 100644
>> --- a/fs/ceph/file.c
>> +++ b/fs/ceph/file.c
>> @@ -898,7 +898,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,
>> ceph_update_read_metrics(&fsc->mdsc->metric,
>> req->r_start_latency,
>> req->r_end_latency,
>> - ret);
>> + len, ret);
>>
>>
>>
>>
>> ceph_osdc_put_request(req);
>>
>>
>>
>>
>> @@ -1030,12 +1030,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>> struct ceph_aio_request *aio_req = req->r_priv;
>> struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0);
>> struct ceph_client_metric *metric = &ceph_sb_to_mdsc(inode->i_sb)->metric;
>> + unsigned int len = osd_data->bvec_pos.iter.bi_size;
>>
>>
>>
>>
>> BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS);
>> BUG_ON(!osd_data->num_bvecs);
>>
>>
>>
>>
>> - dout("ceph_aio_complete_req %p rc %d bytes %u\n",
>> - inode, rc, osd_data->bvec_pos.iter.bi_size);
>> + dout("ceph_aio_complete_req %p rc %d bytes %u\n", inode, rc, len);
>>
>>
>>
>>
>> if (rc == -EOLDSNAPC) {
>> struct ceph_aio_work *aio_work;
>> @@ -1053,9 +1053,9 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>> } else if (!aio_req->write) {
>> if (rc == -ENOENT)
>> rc = 0;
>> - if (rc >= 0 && osd_data->bvec_pos.iter.bi_size > rc) {
>> + if (rc >= 0 && len > rc) {
>> struct iov_iter i;
>> - int zlen = osd_data->bvec_pos.iter.bi_size - rc;
>> + int zlen = len - rc;
>>
>>
>>
>>
>> /*
>> * If read is satisfied by single OSD request,
>> @@ -1072,8 +1072,7 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>> }
>>
>>
>>
>>
>> iov_iter_bvec(&i, READ, osd_data->bvec_pos.bvecs,
>> - osd_data->num_bvecs,
>> - osd_data->bvec_pos.iter.bi_size);
>> + osd_data->num_bvecs, len);
>> iov_iter_advance(&i, rc);
>> iov_iter_zero(zlen, &i);
>> }
>> @@ -1083,10 +1082,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)
>> if (req->r_start_latency) {
>> if (aio_req->write)
>> ceph_update_write_metrics(metric, req->r_start_latency,
>> - req->r_end_latency, rc);
>> + req->r_end_latency, len, rc);
>> else
>> ceph_update_read_metrics(metric, req->r_start_latency,
>> - req->r_end_latency, rc);
>> + req->r_end_latency, len, rc);
>> }
>>
>>
>>
>>
>> put_bvecs(osd_data->bvec_pos.bvecs, osd_data->num_bvecs,
>> @@ -1294,10 +1293,10 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,
>>
>>
>>
>>
>> if (write)
>> ceph_update_write_metrics(metric, req->r_start_latency,
>> - req->r_end_latency, ret);
>> + req->r_end_latency, len, ret);
>> else
>> ceph_update_read_metrics(metric, req->r_start_latency,
>> - req->r_end_latency, ret);
>> + req->r_end_latency, len, ret);
>>
>>
>>
>>
>> size = i_size_read(inode);
>> if (!write) {
>> @@ -1471,7 +1470,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
>> ret = ceph_osdc_wait_request(&fsc->client->osdc, req);
>>
>>
>>
>>
>> ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency,
>> - req->r_end_latency, ret);
>> + req->r_end_latency, len, ret);
>> out:
>> ceph_osdc_put_request(req);
>> if (ret != 0) {
>> diff --git a/fs/ceph/metric.c b/fs/ceph/metric.c
>> index d5560ff99a9d..ff3c9d5cf9ff 100644
>> --- a/fs/ceph/metric.c
>> +++ b/fs/ceph/metric.c
>> @@ -189,6 +189,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
>> m->read_latency_max = 0;
>> m->total_reads = 0;
>> m->read_latency_sum = 0;
>> + m->read_size_min = U64_MAX;
>> + m->read_size_max = 0;
>> + m->read_size_sum = 0;
>>
>>
>>
>>
>> spin_lock_init(&m->write_metric_lock);
>> m->write_latency_sq_sum = 0;
>> @@ -196,6 +199,9 @@ int ceph_metric_init(struct ceph_client_metric *m)
>> m->write_latency_max = 0;
>> m->total_writes = 0;
>> m->write_latency_sum = 0;
>> + m->write_size_min = U64_MAX;
>> + m->write_size_max = 0;
>> + m->write_size_sum = 0;
>>
>>
>>
>>
>> spin_lock_init(&m->metadata_metric_lock);
>> m->metadata_latency_sq_sum = 0;
>> @@ -306,9 +312,41 @@ static inline void __update_latency(struct ceph_client_metric *m,
>> *sq_sump += sq;
>> }
>>
>>
>>
>>
>> +static inline void __update_size(struct ceph_client_metric *m,
>> + metric_type type, unsigned int size)
>> +{
>> + ktime_t total;
>> + u64 *minp, *maxp, *sump;
>> +
>> + switch (type) {
>> + case CEPH_METRIC_READ:
>> + total = m->total_reads;
>> + sump = &m->read_size_sum;
>> + minp = &m->read_size_min;
>> + maxp = &m->read_size_max;
>> + break;
>> + case CEPH_METRIC_WRITE:
>> + total = m->total_writes;
> "total" and "sump" are unused in this function, aside from the
> assignment.
Will fix it.
Just assuming to add the read/write IO speeds here, will it make sense ?
>> + sump = &m->write_size_sum;
>> + minp = &m->write_size_min;
>> + maxp = &m->write_size_max;
>> + break;
>> + case CEPH_METRIC_METADATA:
>> + default:
>> + return;
>> + }
>> +
>> + *sump += size;
>> +
>> + if (unlikely(size < *minp))
>> + *minp = size;
>> + if (unlikely(size > *maxp))
>> + *maxp = size;
>> +}
>> +
>> void ceph_update_read_metrics(struct ceph_client_metric *m,
>> ktime_t r_start, ktime_t r_end,
>> - int rc)
>> + unsigned int size, int rc)
>> {
>> ktime_t lat = ktime_sub(r_end, r_start);
>>
>>
>>
>>
>> @@ -317,12 +355,13 @@ void ceph_update_read_metrics(struct ceph_client_metric *m,
>>
>>
>>
>>
>> spin_lock(&m->read_metric_lock);
>> __update_latency(m, CEPH_METRIC_READ, lat);
>> + __update_size(m, CEPH_METRIC_READ, size);
>> spin_unlock(&m->read_metric_lock);
>> }
>>
>>
>>
>>
>> void ceph_update_write_metrics(struct ceph_client_metric *m,
>> ktime_t r_start, ktime_t r_end,
>> - int rc)
>> + unsigned int size, int rc)
>> {
>> ktime_t lat = ktime_sub(r_end, r_start);
>>
>>
>>
>>
>> @@ -331,6 +370,7 @@ void ceph_update_write_metrics(struct ceph_client_metric *m,
>>
>>
>>
>>
>> spin_lock(&m->write_metric_lock);
>> __update_latency(m, CEPH_METRIC_WRITE, lat);
>> + __update_size(m, CEPH_METRIC_WRITE, size);
>> spin_unlock(&m->write_metric_lock);
>> }
>>
>>
>>
>>
>> diff --git a/fs/ceph/metric.h b/fs/ceph/metric.h
>> index 57b5f0ec38be..64651b6ac886 100644
>> --- a/fs/ceph/metric.h
>> +++ b/fs/ceph/metric.h
>> @@ -110,6 +110,9 @@ struct ceph_client_metric {
>>
>>
>>
>>
>> spinlock_t read_metric_lock;
>> u64 total_reads;
>> + u64 read_size_sum;
>> + u64 read_size_min;
>> + u64 read_size_max;
>> ktime_t read_latency_sum;
>> ktime_t read_latency_sq_sum;
>> ktime_t read_latency_min;
>> @@ -117,6 +120,9 @@ struct ceph_client_metric {
>>
>>
>>
>>
>> spinlock_t write_metric_lock;
>> u64 total_writes;
>> + u64 write_size_sum;
>> + u64 write_size_min;
>> + u64 write_size_max;
>> ktime_t write_latency_sum;
>> ktime_t write_latency_sq_sum;
>> ktime_t write_latency_min;
>> @@ -164,10 +170,10 @@ static inline void ceph_update_cap_mis(struct ceph_client_metric *m)
>>
>>
>>
>>
>> extern void ceph_update_read_metrics(struct ceph_client_metric *m,
>> ktime_t r_start, ktime_t r_end,
>> - int rc);
>> + unsigned int size, int rc);
>> extern void ceph_update_write_metrics(struct ceph_client_metric *m,
>> ktime_t r_start, ktime_t r_end,
>> - int rc);
>> + unsigned int size, int rc);
>> extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
>> ktime_t r_start, ktime_t r_end,
>> int rc);
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/4] ceph: add IO size metric support
2021-03-22 12:28 [PATCH 0/4] ceph: add IO size metric support xiubli
` (3 preceding siblings ...)
2021-03-22 12:28 ` [PATCH 4/4] ceph: add IO size metrics support xiubli
@ 2021-03-24 15:06 ` Jeff Layton
2021-03-25 0:42 ` Xiubo Li
4 siblings, 1 reply; 11+ messages in thread
From: Jeff Layton @ 2021-03-24 15:06 UTC (permalink / raw)
To: xiubli; +Cc: idryomov, pdonnell, ceph-devel
On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
> From: Xiubo Li <xiubli@redhat.com>
>
> Currently it will show as the following:
>
> item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)
> ----------------------------------------------------------------------------------------
> read 1 10240 10240 10240 10240
> write 1 10240 10240 10240 10240
>
>
>
> Xiubo Li (4):
> ceph: rename the metric helpers
> ceph: update the __update_latency helper
> ceph: avoid count the same request twice or more
> ceph: add IO size metrics support
>
> fs/ceph/addr.c | 20 +++----
> fs/ceph/debugfs.c | 49 +++++++++++++----
> fs/ceph/file.c | 47 ++++++++--------
> fs/ceph/mds_client.c | 2 +-
> fs/ceph/metric.c | 126 ++++++++++++++++++++++++++++++++-----------
> fs/ceph/metric.h | 22 +++++---
> 6 files changed, 184 insertions(+), 82 deletions(-)
>
I've gone ahead and merged patches 1 and 3 from this series into
ceph-client/testing. 1 was just a trivial renaming that we might as well
get out of the way, and 3 looked like a (minor) bugfix. The other two
still need a bit of work (but nothing major).
Cheers,
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/4] ceph: add IO size metric support
2021-03-24 15:06 ` [PATCH 0/4] ceph: add IO size metric support Jeff Layton
@ 2021-03-25 0:42 ` Xiubo Li
0 siblings, 0 replies; 11+ messages in thread
From: Xiubo Li @ 2021-03-25 0:42 UTC (permalink / raw)
To: Jeff Layton; +Cc: idryomov, pdonnell, ceph-devel
On 2021/3/24 23:06, Jeff Layton wrote:
> On Mon, 2021-03-22 at 20:28 +0800, xiubli@redhat.com wrote:
>> From: Xiubo Li <xiubli@redhat.com>
>>
>> Currently it will show as the following:
>>
>> item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)
>> ----------------------------------------------------------------------------------------
>> read 1 10240 10240 10240 10240
>> write 1 10240 10240 10240 10240
>>
>>
>>
>> Xiubo Li (4):
>> ceph: rename the metric helpers
>> ceph: update the __update_latency helper
>> ceph: avoid count the same request twice or more
>> ceph: add IO size metrics support
>>
>> fs/ceph/addr.c | 20 +++----
>> fs/ceph/debugfs.c | 49 +++++++++++++----
>> fs/ceph/file.c | 47 ++++++++--------
>> fs/ceph/mds_client.c | 2 +-
>> fs/ceph/metric.c | 126 ++++++++++++++++++++++++++++++++-----------
>> fs/ceph/metric.h | 22 +++++---
>> 6 files changed, 184 insertions(+), 82 deletions(-)
>>
> I've gone ahead and merged patches 1 and 3 from this series into
> ceph-client/testing. 1 was just a trivial renaming that we might as well
> get out of the way, and 3 looked like a (minor) bugfix. The other two
> still need a bit of work (but nothing major).
Sure, will fix them and post the v2 later.
Thanks Jeff.
> Cheers,
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-03-25 0:44 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-22 12:28 [PATCH 0/4] ceph: add IO size metric support xiubli
2021-03-22 12:28 ` [PATCH 1/4] ceph: rename the metric helpers xiubli
2021-03-22 12:28 ` [PATCH 2/4] ceph: update the __update_latency helper xiubli
2021-03-23 12:34 ` Jeff Layton
2021-03-23 13:14 ` Xiubo Li
2021-03-22 12:28 ` [PATCH 3/4] ceph: avoid count the same request twice or more xiubli
2021-03-22 12:28 ` [PATCH 4/4] ceph: add IO size metrics support xiubli
2021-03-23 12:29 ` Jeff Layton
2021-03-23 13:17 ` Xiubo Li
2021-03-24 15:06 ` [PATCH 0/4] ceph: add IO size metric support Jeff Layton
2021-03-25 0:42 ` Xiubo Li
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).