Re: Show and Tell: Grafana cluster dashboard

Re: Show and Tell: Grafana cluster dashboard

[Kurt Bauer]

[-- Attachment #1.1: Type: text/plain, Size: 1181 bytes --]

Hi Jan,

first of all thanks for this dashboard.
A few comments:
-) 'vonage-status-panel' is needed, which isn't mentioned in the ReadMe
-) Using ceph 12.2.4 the mon metric for me is apparently called 
'ceph_mon_quorum_count' not 'ceph_mon_quorum_status'

And a question:
Is there a way to get the Cluster IOPS with prometheus metrics? I did 
this with collectd, but can't find a suitable metric from ceph-mgr.

Best regards,
Kurt

> Jan Fajerski <mailto:jfajerski-IBi9RG/b67k@public.gmane.org>
> 7. May 2018 at 12:32
> Hi all,
> I'd like to request comments and feedback about a Grafana dashboard 
> for Ceph cluster monitoring.
>
> https://youtu.be/HJquM127wMY
>
> https://github.com/ceph/ceph/pull/21850
>
> The goal is to eventually have a set of default dashboards in the Ceph 
> repository that offer decent monitoring for clusters of various (maybe 
> even all) sizes and applications, or at least serve as a starting 
> point for customizations.
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #1.2: Type: text/html, Size: 2665 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Show and Tell: Grafana cluster dashboard

[Jan Fajerski]

On Mon, May 07, 2018 at 02:05:59PM +0200, Kurt Bauer wrote:
>   Hi Jan,
>   first of all thanks for this dashboard.
>   A few comments:
>   -) 'vonage-status-panel' is needed, which isn't mentioned in the ReadMe
Yes, my bad. Will update the README
>   -) Using ceph 12.2.4 the mon metric for me is apparently called
>   'ceph_mon_quorum_count' not 'ceph_mon_quorum_status'
I'll also add to the readme: The dashboard is based on Ceph Mimic.
>   And a question:
>   Is there a way to get the Cluster IOPS with prometheus metrics? I did
>   this with collectd, but can't find a suitable metric from ceph-mgr.
Yes...at least in Mimic the metrics are called ceph_osd_op[_r,_w,_rw]
>   Best regards,
>   Kurt
>
>   [1]Jan Fajerski
>   7. May 2018 at 12:32
>
>   Hi all,
>   I'd like to request comments and feedback about a Grafana dashboard for
>   Ceph cluster monitoring.
>   [2]https://youtu.be/HJquM127wMY
>   [3]https://github.com/ceph/ceph/pull/21850
>   The goal is to eventually have a set of default dashboards in the Ceph
>   repository that offer decent monitoring for clusters of various (maybe
>   even all) sizes and applications, or at least serve as a starting point
>   for customizations.
>   --
>   To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>   in
>   the body of a message to [4]majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>   More majordomo info at  [5]http://vger.kernel.org/majordomo-info.html
>
>References
>
>   1. mailto:jfajerski-IBi9RG/b67k@public.gmane.org
>   2. https://youtu.be/HJquM127wMY
>   3. https://github.com/ceph/ceph/pull/21850
>   4. mailto:majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>   5. http://vger.kernel.org/majordomo-info.html

>_______________________________________________
>ceph-users mailing list
>ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
Jan Fajerski
Engineer Enterprise Storage
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)

Re: Show and Tell: Grafana cluster dashboard

[Kurt Bauer]

> Jan Fajerski <mailto:jfajerski-IBi9RG/b67k@public.gmane.org>
> 7. May 2018 at 14:21
> On Mon, May 07, 2018 at 02:05:59PM +0200, Kurt Bauer wrote:
>>   Hi Jan,
>>   first of all thanks for this dashboard.
>>   A few comments:
>>   -) 'vonage-status-panel' is needed, which isn't mentioned in the 
>> ReadMe
> Yes, my bad. Will update the README
>>   -) Using ceph 12.2.4 the mon metric for me is apparently called
>>   'ceph_mon_quorum_count' not 'ceph_mon_quorum_status'
> I'll also add to the readme: The dashboard is based on Ceph Mimic.
>>   And a question:
>>   Is there a way to get the Cluster IOPS with prometheus metrics? I did
>>   this with collectd, but can't find a suitable metric from ceph-mgr.
> Yes...at least in Mimic the metrics are called ceph_osd_op[_r,_w,_rw]
Thanks, these metrics are in Luminous too. I seem unable to find some 
sort of register, to see which metrics mean what. Some are quite 
obvious, but others are a mystery. Does smth. like that exist somewhere?

Thanks.

>>   Best regards,
>>   Kurt
>>
>>   [1]Jan Fajerski
>>   7. May 2018 at 12:32
>>
>>   Hi all,
>>   I'd like to request comments and feedback about a Grafana dashboard 
>> for
>>   Ceph cluster monitoring.
>>   [2]https://youtu.be/HJquM127wMY
>>   [3]https://github.com/ceph/ceph/pull/21850
>>   The goal is to eventually have a set of default dashboards in the Ceph
>>   repository that offer decent monitoring for clusters of various (maybe
>>   even all) sizes and applications, or at least serve as a starting 
>> point
>>   for customizations.
>>   --
>>   To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>   in
>>   the body of a message to [4]majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>   More majordomo info at [5]http://vger.kernel.org/majordomo-info.html
>>
>> References
>>
>>   1. mailto:jfajerski-IBi9RG/b67k@public.gmane.org
>>   2. https://youtu.be/HJquM127wMY
>>   3. https://github.com/ceph/ceph/pull/21850
>>   4. mailto:majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>   5. http://vger.kernel.org/majordomo-info.html
>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> Kurt Bauer <mailto:kurt.bauer-4JhlDu4IDl0juwv8T7myQQ@public.gmane.org>
> 7. May 2018 at 14:05
> Hi Jan,
>
> first of all thanks for this dashboard.
> A few comments:
> -) 'vonage-status-panel' is needed, which isn't mentioned in the ReadMe
> -) Using ceph 12.2.4 the mon metric for me is apparently called 
> 'ceph_mon_quorum_count' not 'ceph_mon_quorum_status'
>
> And a question:
> Is there a way to get the Cluster IOPS with prometheus metrics? I did 
> this with collectd, but can't find a suitable metric from ceph-mgr.
>
> Best regards,
> Kurt
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 

Kurt Bauer<kurt.bauer-4JhlDu4IDl0juwv8T7myQQ@public.gmane.org>
Vienna University Computer Center - ACOnet - VIX
Universitaetsstrasse 7, A-1010 Vienna, Austria, Europe
Tel: ++431 4277  - 14070 (Fax: - 814070)  KB1970-RIPE

Re: Show and Tell: Grafana cluster dashboard

[Jan Fajerski]

On Mon, May 07, 2018 at 02:45:14PM +0200, Kurt Bauer wrote:
>
>
>>Jan Fajerski <mailto:jfajerski-IBi9RG/b67k@public.gmane.org>
>>7. May 2018 at 14:21
>>On Mon, May 07, 2018 at 02:05:59PM +0200, Kurt Bauer wrote:
>>>  Hi Jan,
>>>  first of all thanks for this dashboard.
>>>  A few comments:
>>>  -) 'vonage-status-panel' is needed, which isn't mentioned in the 
>>>ReadMe
>>Yes, my bad. Will update the README
>>>  -) Using ceph 12.2.4 the mon metric for me is apparently called
>>>  'ceph_mon_quorum_count' not 'ceph_mon_quorum_status'
>>I'll also add to the readme: The dashboard is based on Ceph Mimic.
>>>  And a question:
>>>  Is there a way to get the Cluster IOPS with prometheus metrics? I did
>>>  this with collectd, but can't find a suitable metric from ceph-mgr.
>>Yes...at least in Mimic the metrics are called ceph_osd_op[_r,_w,_rw]
>Thanks, these metrics are in Luminous too. I seem unable to find some 
>sort of register, to see which metrics mean what. Some are quite 
>obvious, but others are a mystery. Does smth. like that exist 
>somewhere?
Not yet.
Most daemon specific metric names (like ceph_osd_op[_r,_w,_rw) are derived 
directly from the respective perf counter names. The plugin exports all perf 
counters with PRIO_INTERESTING or higher (iirc).
An automatically created index would certainly be feasible.
>
>Thanks.
>
>>>  Best regards,
>>>  Kurt
>>>
>>>  [1]Jan Fajerski
>>>  7. May 2018 at 12:32
>>>
>>>  Hi all,
>>>  I'd like to request comments and feedback about a Grafana 
>>>dashboard for
>>>  Ceph cluster monitoring.
>>>  [2]https://youtu.be/HJquM127wMY
>>>  [3]https://github.com/ceph/ceph/pull/21850
>>>  The goal is to eventually have a set of default dashboards in the Ceph
>>>  repository that offer decent monitoring for clusters of various (maybe
>>>  even all) sizes and applications, or at least serve as a 
>>>starting point
>>>  for customizations.
>>>  --
>>>  To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>  in
>>>  the body of a message to [4]majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>  More majordomo info at [5]http://vger.kernel.org/majordomo-info.html
>>>
>>>References
>>>
>>>  1. mailto:jfajerski-IBi9RG/b67k@public.gmane.org
>>>  2. https://youtu.be/HJquM127wMY
>>>  3. https://github.com/ceph/ceph/pull/21850
>>>  4. mailto:majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>  5. http://vger.kernel.org/majordomo-info.html
>>
>>>_______________________________________________
>>>ceph-users mailing list
>>>ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>Kurt Bauer <mailto:kurt.bauer-4JhlDu4IDl0juwv8T7myQQ@public.gmane.org>
>>7. May 2018 at 14:05
>>Hi Jan,
>>
>>first of all thanks for this dashboard.
>>A few comments:
>>-) 'vonage-status-panel' is needed, which isn't mentioned in the ReadMe
>>-) Using ceph 12.2.4 the mon metric for me is apparently called 
>>'ceph_mon_quorum_count' not 'ceph_mon_quorum_status'
>>
>>And a question:
>>Is there a way to get the Cluster IOPS with prometheus metrics? I 
>>did this with collectd, but can't find a suitable metric from 
>>ceph-mgr.
>>
>>Best regards,
>>Kurt
>>
>>
>>
>>
>>_______________________________________________
>>ceph-users mailing list
>>ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>-- 
>
>Kurt Bauer<kurt.bauer-4JhlDu4IDl0juwv8T7myQQ@public.gmane.org>
>Vienna University Computer Center - ACOnet - VIX
>Universitaetsstrasse 7, A-1010 Vienna, Austria, Europe
>Tel: ++431 4277  - 14070 (Fax: - 814070)  KB1970-RIPE
>
>_______________________________________________
>ceph-users mailing list
>ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

-- 
Jan Fajerski
Engineer Enterprise Storage
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)

Re: Show and Tell: Grafana cluster dashboard

[Reed Dier]

I’ll +1 on InfluxDB rather than Prometheus, though I think having a version for each infrastructure path would be best.
I’m sure plenty here have existing InfluxDB infrastructure as their TSDB of choice, and moving to Prometheus would be less advantageous.

Conversely, I’m sure all of the Prometheus folks would be less inclined to move to InfluxDB for TSDB, so I think supporting both paths would be the best choice.

Reed

> On May 7, 2018, at 3:06 AM, Marc Roos <M.Roos@f1-outsourcing.eu> wrote:
> 
> 
> Looks nice 
> 
> - I rather have some dashboards with collectd/influxdb.
> - Take into account bigger tv/screens eg 65" uhd. I am putting more 
> stats on them than viewing them locally in a webbrowser.
> - What is to be considered most important to have on your ceph 
> dashboard? As a newbie I find it difficult to determine what is 
> important to monitor.
> - Maybe also some docs on what metrics you have taken and argumentation 
> on how you used them (could be usefull if one wants to modify the 
> dashboard for some other backend)
> 
> Ceph performance counters description.
> https://access.redhat.com/documentation/en/red-hat-ceph-storage/1.3/paged/administration-guide/chapter-9-performance-counters
> 
> 
> -----Original Message-----
> From: Jan Fajerski [mailto:jfajerski@suse.com] 
> Sent: maandag 7 mei 2018 12:32
> To: ceph-devel
> Cc: ceph-users
> Subject: [ceph-users] Show and Tell: Grafana cluster dashboard
> 
> Hi all,
> I'd like to request comments and feedback about a Grafana dashboard for 
> Ceph cluster monitoring.
> 
> https://youtu.be/HJquM127wMY
> 
> https://github.com/ceph/ceph/pull/21850
> 
> The goal is to eventually have a set of default dashboards in the Ceph 
> repository that offer decent monitoring for clusters of various (maybe 
> even all) sizes and applications, or at least serve as a starting point 
> for customizations.
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Show and Tell: Grafana cluster dashboard

[Kai Wagner]

[-- Attachment #1.1.1: Type: text/plain, Size: 604 bytes --]

Looks very good. Is it anyhow possible to display the reason why a
cluster is in an error or warning state? Thinking about the output from
ceph -s if this could by shown in case there's a failure. I think this
will not be provided by default but wondering if it's possible to add.

Kai

On 05/07/2018 04:53 PM, Reed Dier wrote:
> I think supporting both paths would be the best choice.
That's the way we should go. Supporting both or in general as much as
possible (try to be generic)

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Show and Tell: Grafana cluster dashboard

[Lenz Grimmer]

[-- Attachment #1.1.1: Type: text/plain, Size: 1120 bytes --]

On 05/08/2018 07:21 AM, Kai Wagner wrote:

> Looks very good. Is it anyhow possible to display the reason why a
> cluster is in an error or warning state? Thinking about the output from
> ceph -s if this could by shown in case there's a failure. I think this
> will not be provided by default but wondering if it's possible to add.

Sorry for the late reply. We actually discussed this aspect during one
of the calls we had when discussing the Grafana dashboard integration
into the Ceph Manager Dashboard. Such kind of state information is
somewhat difficult to track and visualize using Prometheus/Grafana (or
any other TSDB, FWIW), as you can't store the actual reasons for why the
cluster is in HEALTH_WARN or HEALTH_ERR, for example.

We are therefore considering displaying this information in the form of
"native" widgets on the Manager Dashboard, and using the Grafana
dashboards for visualizing the other more suitable performance metrics.

Lenz

-- 
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nuernberg (Germany)
GF:Felix Imendörffer,Jane Smithard,Graham Norton,HRB 21284 (AG Nürnberg)


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Show and Tell: Grafana cluster dashboard

[Jan Fajerski]

Hi all,
I'd like to request comments and feedback about a Grafana dashboard for Ceph 
cluster monitoring.

https://youtu.be/HJquM127wMY

https://github.com/ceph/ceph/pull/21850

The goal is to eventually have a set of default dashboards in the Ceph 
repository that offer decent monitoring for clusters of various (maybe even all) 
sizes and applications, or at least serve as a starting point for 
customizations.

Re: Show and Tell: Grafana cluster dashboard

[Marc Roos]

Looks nice 

- I rather have some dashboards with collectd/influxdb.
- Take into account bigger tv/screens eg 65" uhd. I am putting more 
stats on them than viewing them locally in a webbrowser.
- What is to be considered most important to have on your ceph 
dashboard? As a newbie I find it difficult to determine what is 
important to monitor.
- Maybe also some docs on what metrics you have taken and argumentation 
on how you used them (could be usefull if one wants to modify the 
dashboard for some other backend)

Ceph performance counters description.
https://access.redhat.com/documentation/en/red-hat-ceph-storage/1.3/paged/administration-guide/chapter-9-performance-counters


-----Original Message-----
From: Jan Fajerski [mailto:jfajerski-IBi9RG/b67k@public.gmane.org] 
Sent: maandag 7 mei 2018 12:32
To: ceph-devel
Cc: ceph-users
Subject: [ceph-users] Show and Tell: Grafana cluster dashboard

Hi all,
I'd like to request comments and feedback about a Grafana dashboard for 
Ceph cluster monitoring.

https://youtu.be/HJquM127wMY

https://github.com/ceph/ceph/pull/21850

The goal is to eventually have a set of default dashboards in the Ceph 
repository that offer decent monitoring for clusters of various (maybe 
even all) sizes and applications, or at least serve as a starting point 
for customizations.
_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com