* Re: measure performance / latency in blustore
[not found] ` <71b773a4-18a1-c206-2fe4-83b51cdf368c-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
@ 2017-12-14 7:48 ` Stefan Priebe - Profihost AG
[not found] ` <231b2674-d2b6-b66a-ae29-72c0f2a7f1a0-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
0 siblings, 1 reply; 5+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-12-14 7:48 UTC (permalink / raw)
To: ceph-users-idqoXFIVOFJgJs9I8MT0rw, ceph-devel-u79uwXL29TY76Z2rM5mHXA
Hello,
Am 21.11.2017 um 11:06 schrieb Stefan Priebe - Profihost AG:
> Hello,
>
> to measure performance / latency for filestore we used:
> filestore:apply_latency
> filestore:commitcycle_latency
> filestore:journal_latency
> filestore:queue_transaction_latency_avg
>
> What are the correct ones for bluestore?
really nobody? Does nobody track the latency under bluestore?
Thanks,
Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: measure performance / latency in blustore
[not found] ` <231b2674-d2b6-b66a-ae29-72c0f2a7f1a0-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
@ 2017-12-14 12:22 ` Sage Weil
2017-12-14 13:59 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 5+ messages in thread
From: Sage Weil @ 2017-12-14 12:22 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG
Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, ceph-devel-u79uwXL29TY76Z2rM5mHXA
On Thu, 14 Dec 2017, Stefan Priebe - Profihost AG wrote:
> Hello,
>
> Am 21.11.2017 um 11:06 schrieb Stefan Priebe - Profihost AG:
> > Hello,
> >
> > to measure performance / latency for filestore we used:
> > filestore:apply_latency
> > filestore:commitcycle_latency
> > filestore:journal_latency
> > filestore:queue_transaction_latency_avg
> >
> > What are the correct ones for bluestore?
>
> really nobody? Does nobody track the latency under bluestore?
I forget the long names off the top of my head, but the interesting
latency measures are marked with a low priority and come up (with a wide
terminal) when you do 'ceph daemonperf osd.N'. You can see the metrics,
pririties, and descriptions with 'ceph daemon osd.N perf schema'.
sage
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: measure performance / latency in blustore
2017-12-14 12:22 ` Sage Weil
@ 2017-12-14 13:59 ` Stefan Priebe - Profihost AG
2017-12-14 14:02 ` Sage Weil
0 siblings, 1 reply; 5+ messages in thread
From: Stefan Priebe - Profihost AG @ 2017-12-14 13:59 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-users, ceph-devel
Am 14.12.2017 um 13:22 schrieb Sage Weil:
> On Thu, 14 Dec 2017, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> Am 21.11.2017 um 11:06 schrieb Stefan Priebe - Profihost AG:
>>> Hello,
>>>
>>> to measure performance / latency for filestore we used:
>>> filestore:apply_latency
>>> filestore:commitcycle_latency
>>> filestore:journal_latency
>>> filestore:queue_transaction_latency_avg
>>>
>>> What are the correct ones for bluestore?
>>
>> really nobody? Does nobody track the latency under bluestore?
>
> I forget the long names off the top of my head, but the interesting
> latency measures are marked with a low priority and come up (with a wide
> terminal) when you do 'ceph daemonperf osd.N'. You can see the metrics,
> pririties, and descriptions with 'ceph daemon osd.N perf schema'.
uhuh very long list. Any idea which ones are relevant?
ceph daemon osd.8 perf dump
shows me a lot of stuff and even a lot of wait or latancy values.
ceph daemon osd.8 perf dump | egrep "wait|lat"
"kv_flush_lat": {
"kv_commit_lat": {
"kv_lat": {
"state_prepare_lat": {
"state_aio_wait_lat": {
"state_io_done_lat": {
"state_kv_queued_lat": {
"state_kv_commiting_lat": {
"state_kv_done_lat": {
"state_deferred_queued_lat": {
"state_deferred_aio_wait_lat": {
"state_deferred_cleanup_lat": {
"state_finishing_lat": {
"state_done_lat": {
"throttle_lat": {
"submit_lat": {
"commit_lat": {
"read_lat": {
"read_onode_meta_lat": {
"read_wait_aio_lat": {
"compress_lat": {
"decompress_lat": {
"csum_lat": {
"complete_latency": {
"complete_latency": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"op_latency": {
"op_process_latency": {
"op_prepare_latency": {
"op_r_latency": {
"op_r_process_latency": {
"op_r_prepare_latency": {
"op_w_latency": {
"op_w_process_latency": {
"op_w_prepare_latency": {
"op_rw_latency": {
"op_rw_process_latency": {
"op_rw_prepare_latency": {
"op_before_queue_op_lat": {
"op_before_dequeue_op_lat": {
"subop_latency": {
"subop_w_latency": {
"subop_pull_latency": {
"subop_push_latency": {
"osd_tier_flush_lat": {
"osd_tier_promote_lat": {
"osd_tier_r_lat": {
"initial_latency": {
"started_latency": {
"reset_latency": {
"start_latency": {
"primary_latency": {
"peering_latency": {
"backfilling_latency": {
"waitremotebackfillreserved_latency": {
"waitlocalbackfillreserved_latency": {
"notbackfilling_latency": {
"repnotrecovering_latency": {
"repwaitrecoveryreserved_latency": {
"repwaitbackfillreserved_latency": {
"reprecovering_latency": {
"activating_latency": {
"waitlocalrecoveryreserved_latency": {
"waitremoterecoveryreserved_latency": {
"recovering_latency": {
"recovered_latency": {
"clean_latency": {
"active_latency": {
"replicaactive_latency": {
"stray_latency": {
"getinfo_latency": {
"getlog_latency": {
"waitactingchange_latency": {
"incomplete_latency": {
"down_latency": {
"getmissing_latency": {
"waitupthru_latency": {
"notrecovering_latency": {
"get_latency": {
"submit_latency": {
"submit_sync_latency": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
"wait": {
Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: measure performance / latency in blustore
2017-12-14 13:59 ` Stefan Priebe - Profihost AG
@ 2017-12-14 14:02 ` Sage Weil
2018-02-07 14:07 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 5+ messages in thread
From: Sage Weil @ 2017-12-14 14:02 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: ceph-users, ceph-devel
On Thu, 14 Dec 2017, Stefan Priebe - Profihost AG wrote:
>
> Am 14.12.2017 um 13:22 schrieb Sage Weil:
> > On Thu, 14 Dec 2017, Stefan Priebe - Profihost AG wrote:
> >> Hello,
> >>
> >> Am 21.11.2017 um 11:06 schrieb Stefan Priebe - Profihost AG:
> >>> Hello,
> >>>
> >>> to measure performance / latency for filestore we used:
> >>> filestore:apply_latency
> >>> filestore:commitcycle_latency
> >>> filestore:journal_latency
> >>> filestore:queue_transaction_latency_avg
> >>>
> >>> What are the correct ones for bluestore?
> >>
> >> really nobody? Does nobody track the latency under bluestore?
> >
> > I forget the long names off the top of my head, but the interesting
> > latency measures are marked with a low priority and come up (with a wide
> > terminal) when you do 'ceph daemonperf osd.N'. You can see the metrics,
> > pririties, and descriptions with 'ceph daemon osd.N perf schema'.
>
> uhuh very long list. Any idea which ones are relevant?
>
> ceph daemon osd.8 perf dump
If you do 'perf schema' you'll see a 'priority' property that calls out
the important ones. That's how the daemonperf command decides which ones
to show (based on terminal width and priorities).
sage
>
> shows me a lot of stuff and even a lot of wait or latancy values.
>
> ceph daemon osd.8 perf dump | egrep "wait|lat"
> "kv_flush_lat": {
> "kv_commit_lat": {
> "kv_lat": {
> "state_prepare_lat": {
> "state_aio_wait_lat": {
> "state_io_done_lat": {
> "state_kv_queued_lat": {
> "state_kv_commiting_lat": {
> "state_kv_done_lat": {
> "state_deferred_queued_lat": {
> "state_deferred_aio_wait_lat": {
> "state_deferred_cleanup_lat": {
> "state_finishing_lat": {
> "state_done_lat": {
> "throttle_lat": {
> "submit_lat": {
> "commit_lat": {
> "read_lat": {
> "read_onode_meta_lat": {
> "read_wait_aio_lat": {
> "compress_lat": {
> "decompress_lat": {
> "csum_lat": {
> "complete_latency": {
> "complete_latency": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "op_latency": {
> "op_process_latency": {
> "op_prepare_latency": {
> "op_r_latency": {
> "op_r_process_latency": {
> "op_r_prepare_latency": {
> "op_w_latency": {
> "op_w_process_latency": {
> "op_w_prepare_latency": {
> "op_rw_latency": {
> "op_rw_process_latency": {
> "op_rw_prepare_latency": {
> "op_before_queue_op_lat": {
> "op_before_dequeue_op_lat": {
> "subop_latency": {
> "subop_w_latency": {
> "subop_pull_latency": {
> "subop_push_latency": {
> "osd_tier_flush_lat": {
> "osd_tier_promote_lat": {
> "osd_tier_r_lat": {
> "initial_latency": {
> "started_latency": {
> "reset_latency": {
> "start_latency": {
> "primary_latency": {
> "peering_latency": {
> "backfilling_latency": {
> "waitremotebackfillreserved_latency": {
> "waitlocalbackfillreserved_latency": {
> "notbackfilling_latency": {
> "repnotrecovering_latency": {
> "repwaitrecoveryreserved_latency": {
> "repwaitbackfillreserved_latency": {
> "reprecovering_latency": {
> "activating_latency": {
> "waitlocalrecoveryreserved_latency": {
> "waitremoterecoveryreserved_latency": {
> "recovering_latency": {
> "recovered_latency": {
> "clean_latency": {
> "active_latency": {
> "replicaactive_latency": {
> "stray_latency": {
> "getinfo_latency": {
> "getlog_latency": {
> "waitactingchange_latency": {
> "incomplete_latency": {
> "down_latency": {
> "getmissing_latency": {
> "waitupthru_latency": {
> "notrecovering_latency": {
> "get_latency": {
> "submit_latency": {
> "submit_sync_latency": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
> "wait": {
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: measure performance / latency in blustore
2017-12-14 14:02 ` Sage Weil
@ 2018-02-07 14:07 ` Stefan Priebe - Profihost AG
0 siblings, 0 replies; 5+ messages in thread
From: Stefan Priebe - Profihost AG @ 2018-02-07 14:07 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-users, ceph-devel
Hi,
Am 14.12.2017 um 15:02 schrieb Sage Weil:
> On Thu, 14 Dec 2017, Stefan Priebe - Profihost AG wrote:
>>
>> Am 14.12.2017 um 13:22 schrieb Sage Weil:
>>> On Thu, 14 Dec 2017, Stefan Priebe - Profihost AG wrote:
>>>> Hello,
>>>>
>>>> Am 21.11.2017 um 11:06 schrieb Stefan Priebe - Profihost AG:
>>>>> Hello,
>>>>>
>>>>> to measure performance / latency for filestore we used:
>>>>> filestore:apply_latency
>>>>> filestore:commitcycle_latency
>>>>> filestore:journal_latency
>>>>> filestore:queue_transaction_latency_avg
>>>>>
>>>>> What are the correct ones for bluestore?
>>>>
>>>> really nobody? Does nobody track the latency under bluestore?
>>>
>>> I forget the long names off the top of my head, but the interesting
>>> latency measures are marked with a low priority and come up (with a wide
>>> terminal) when you do 'ceph daemonperf osd.N'. You can see the metrics,
>>> pririties, and descriptions with 'ceph daemon osd.N perf schema'.
>>
>> uhuh very long list. Any idea which ones are relevant?
>>
>> ceph daemon osd.8 perf dump
>
> If you do 'perf schema' you'll see a 'priority' property that calls out
> the important ones. That's how the daemonperf command decides which ones
> to show (based on terminal width and priorities).
>
> sage
So i would start using:
"throttle_lat": {
"type": 5,
"priority": 10
"submit_lat": {
"type": 5,
"priority": 10
"commit_lat": {
"type": 5,
"priority": 10
"read_lat": {
"type": 5,
"priority": 10
But i'm not able to find any hints about those values and their meaning.
Greets,
Stefan
>
>
> >
>> shows me a lot of stuff and even a lot of wait or latancy values.
>>
>> ceph daemon osd.8 perf dump | egrep "wait|lat"
>> "kv_flush_lat": {
>> "kv_commit_lat": {
>> "kv_lat": {
>> "state_prepare_lat": {
>> "state_aio_wait_lat": {
>> "state_io_done_lat": {
>> "state_kv_queued_lat": {
>> "state_kv_commiting_lat": {
>> "state_kv_done_lat": {
>> "state_deferred_queued_lat": {
>> "state_deferred_aio_wait_lat": {
>> "state_deferred_cleanup_lat": {
>> "state_finishing_lat": {
>> "state_done_lat": {
>> "throttle_lat": {
>> "submit_lat": {
>> "commit_lat": {
>> "read_lat": {
>> "read_onode_meta_lat": {
>> "read_wait_aio_lat": {
>> "compress_lat": {
>> "decompress_lat": {
>> "csum_lat": {
>> "complete_latency": {
>> "complete_latency": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "op_latency": {
>> "op_process_latency": {
>> "op_prepare_latency": {
>> "op_r_latency": {
>> "op_r_process_latency": {
>> "op_r_prepare_latency": {
>> "op_w_latency": {
>> "op_w_process_latency": {
>> "op_w_prepare_latency": {
>> "op_rw_latency": {
>> "op_rw_process_latency": {
>> "op_rw_prepare_latency": {
>> "op_before_queue_op_lat": {
>> "op_before_dequeue_op_lat": {
>> "subop_latency": {
>> "subop_w_latency": {
>> "subop_pull_latency": {
>> "subop_push_latency": {
>> "osd_tier_flush_lat": {
>> "osd_tier_promote_lat": {
>> "osd_tier_r_lat": {
>> "initial_latency": {
>> "started_latency": {
>> "reset_latency": {
>> "start_latency": {
>> "primary_latency": {
>> "peering_latency": {
>> "backfilling_latency": {
>> "waitremotebackfillreserved_latency": {
>> "waitlocalbackfillreserved_latency": {
>> "notbackfilling_latency": {
>> "repnotrecovering_latency": {
>> "repwaitrecoveryreserved_latency": {
>> "repwaitbackfillreserved_latency": {
>> "reprecovering_latency": {
>> "activating_latency": {
>> "waitlocalrecoveryreserved_latency": {
>> "waitremoterecoveryreserved_latency": {
>> "recovering_latency": {
>> "recovered_latency": {
>> "clean_latency": {
>> "active_latency": {
>> "replicaactive_latency": {
>> "stray_latency": {
>> "getinfo_latency": {
>> "getlog_latency": {
>> "waitactingchange_latency": {
>> "incomplete_latency": {
>> "down_latency": {
>> "getmissing_latency": {
>> "waitupthru_latency": {
>> "notrecovering_latency": {
>> "get_latency": {
>> "submit_latency": {
>> "submit_sync_latency": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>> "wait": {
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-02-07 14:07 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <71b773a4-18a1-c206-2fe4-83b51cdf368c@profihost.ag>
[not found] ` <71b773a4-18a1-c206-2fe4-83b51cdf368c-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2017-12-14 7:48 ` measure performance / latency in blustore Stefan Priebe - Profihost AG
[not found] ` <231b2674-d2b6-b66a-ae29-72c0f2a7f1a0-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2017-12-14 12:22 ` Sage Weil
2017-12-14 13:59 ` Stefan Priebe - Profihost AG
2017-12-14 14:02 ` Sage Weil
2018-02-07 14:07 ` Stefan Priebe - Profihost AG
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.