[PATCH 0/4] dm-latency: Introduction

[PATCH 0/4] dm-latency: Introduction

[Coly Li]

From: Coly Li <bosong.ly@alibaba-inc.com>

Dm-latency patch set is an effort to measure hard disk I/O latency on
top of device mapper layer. The original motivation of I/O latency
measurement was to predict hard disk failure by machine learning method,
I/O latency information was one of the inputs sent to machine learning
model.

This patch set was written in Aug~Sep 2013, I deployed it on many
servers of Alibaba cloud infrastructure. After running for weeks, some
interesting data about hard disk I/O latency was observed. In 2013, I
gave a talk on OpenSuSE Conference about this topic
(http://blog.coly.li/docs/osc13-coly.pdf).

When generating time stamp for I/O request, clock source is a global
unique resource which is protected by spin-locks. Dm-latency was tested
on SAS/SATA hard disk and SATA SSD, it worked well as expected. Running
dm-latency on PCI-e or NVMe SSD should work (I didn't test), but there
will be spin-lock scalability issue, when accessing clock source for
time stamping.

Dm-latency is good for I/O latency measurement to hard disk based
storage, no matter local or distributed storage via network. For PCI-e
or NVMe SSD, I suggest people to look for device provided statistic
information, if there is.

The code is very simple, there is no resource allocation/destory, no
spin_lock/spin_unlock. The patch set gets merged into Alibaba kernel
more than 1 year, no bug reported in last 12 months.

This patch set has 4 patches,
- [PATCH 1/4] dm-latency: move struct mapped_device from dm.c to dm.h
- [PATCH 2/4] dm-latency: add I/O latency measurement in device mapper
- [PATCH 3/4] dm-latency: add sysfs interface
- [PATCH 4/4] dm-latency: add reset function to dm-latency in sysfs
interface
All these patches are rebased on Linux 4.0-rc1.

Today Laurence Oberman from Redhat sent me an email asking whether this
patch set is upstream merged, because he is thinking of pulling this
patch set into their kernel. I'd like to maintain this patch set, hope
it could be merged.

Thanks in advance.

Coly Li

Re: [PATCH 0/4] dm-latency: Introduction

[Mikulas Patocka]

On Fri, 27 Feb 2015, Bryn M. Reeves wrote:

> On Thu, Feb 26, 2015 at 02:45:43PM -0500, Laurence Oberman wrote:
> > For my particular use case its about providing the ability to warn on latencies being seen for multipath devices based on a given threshold.
> > Of course this can simply be a userspace tool using what we already expose and do the calculations to make it work.
> > When we have these latencies we then focus on which SAN path may or may not be contributing.
> > Within multipathd we can already configure service time as a load balancer, perhaps we can do the monitoring in the same place.
> > i.e. warn on service time above a threshold.
> 
> One limitation here is that the dm-mpath target is request-based.
> Currently dm-statistics are only available for bio-based targets. This
> means that to obtain fine-grained stats for multipath devices we need to
> insert a linear layer on top of the dm-mpath device.

Someone could hack dm statistics to work on request based targets.

Mikulas

Re: [PATCH 0/4] dm-latency: Introduction

[Bryn M. Reeves]

On Thu, Feb 26, 2015 at 02:45:43PM -0500, Laurence Oberman wrote:
> For my particular use case its about providing the ability to warn on latencies being seen for multipath devices based on a given threshold.
> Of course this can simply be a userspace tool using what we already expose and do the calculations to make it work.
> When we have these latencies we then focus on which SAN path may or may not be contributing.
> Within multipathd we can already configure service time as a load balancer, perhaps we can do the monitoring in the same place.
> i.e. warn on service time above a threshold.

One limitation here is that the dm-mpath target is request-based.
Currently dm-statistics are only available for bio-based targets. This
means that to obtain fine-grained stats for multipath devices we need to
insert a linear layer on top of the dm-mpath device.

If we wanted to take this further and apply counters to each underlying
path then that would require a similar layer to be added between the sd
devices and the dm-mpath map.

> When a customer says "I currently use the following xxxx for multipath
> on RHEL however I want to go to native multipathing, but you don't
> provide the monitoring I need" I want to work to an enhancement.

Average wait times are easily obtained from the current kernel counter
set but (aside from the ability to hone in on subsections of the device)
this doesn't buy you that much beyond vanilla iostat.

Regards,
Bryn.

Re: [PATCH 0/4] dm-latency: Introduction

[Laurence Oberman]

For my particular use case its about providing the ability to warn on latencies being seen for multipath devices based on a given threshold.
Of course this can simply be a userspace tool using what we already expose and do the calculations to make it work.
When we have these latencies we then focus on which SAN path may or may not be contributing.
Within multipathd we can already configure service time as a load balancer, perhaps we can do the monitoring in the same place.
i.e. warn on service time above a threshold.

When a customer says "I currently use the following xxxx for multipath  on RHEL however I want to go to native multipathing, but you don't provide the monitoring I need" I want to work to an enhancement.

For example I had Ben add the ability just recently to multipath to control re-establishing path usage based on path health when the path returns to mimic what DMP can do to avoid recovery due to path flapping. 

Thanks

Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Mikulas Patocka" <mpatocka@redhat.com>
To: "Bryn M. Reeves" <bmr@redhat.com>
Cc: "device-mapper development" <dm-devel@redhat.com>, "Tao Ma" <boyu.mt@taobao.com>, "Robin Dong" <sanbai@alibaba-inc.com>, "Laurence Oberman" <loberman@redhat.com>, "Coly Li" <colyli@gmail.com>, "Alasdair Kergon" <agk@redhat.com>
Sent: Thursday, February 26, 2015 2:34:40 PM
Subject: Re: [dm-devel] [PATCH 0/4] dm-latency: Introduction



On Thu, 26 Feb 2015, Bryn M. Reeves wrote:

> On Thu, Feb 26, 2015 at 11:49:28AM -0500, Mikulas Patocka wrote:
> > We have already dm-statistics that counts various events - see 
> > Documentation/device-mapper/statistics.txt. It counts the nubmer of 
> > requests and the time spent servicing each request, thus you can 
> > calculate average latency from these values.
> 
> Right: average service time (as reported by iostat etc.) is easily derived
> from the existing stats.
> 
> Does the separate latency accounting buy anything additional?
>
> > Please look at dm-statistics to see if it fits your purpose. If you need 
> > additional information not provided by dm-statistics, it would be better 
> > to extend the statistics code rather than introduce new "latency" 
> > infrastructure.
> 
> Agreed; I'm working on userspace support for dm-statistics at the moment
> and if there is a need for these additional measurements I would greatly
> prefer to consume them as additional fields in the existing dm-stats
> counter set.
> 
> This also has the advantage of benefiting from the existing step and
> area support allowing a device to be subdivided into discrete stats
> regions.
> 
> Regards,
> Bryn.

Coly's paper (http://blog.coly.li/docs/osc13-coly.pdf) shows that they 
take histogram of latencies and use it to predict disk failure.

That could be easily added to dm-statistics.

Average latency alone can't be used to predict disk failure because 
average latency depends on the type of workload (for example - sequantial 
or nearly sequential requests have much lower latency than random 
requests).

I'd like to know if we need separate histogram per region, or if it is 
sufficient to have a histogram per device. dm-latency has no regions, it 
has a histogram for the whole device. The histogram-per-region would 
consume more memory, I'm interested if there is some reasonable use case 
for that.

Mikulas

Re: [PATCH 0/4] dm-latency: Introduction

[Mikulas Patocka]

On Thu, 26 Feb 2015, Bryn M. Reeves wrote:

> On Thu, Feb 26, 2015 at 11:49:28AM -0500, Mikulas Patocka wrote:
> > We have already dm-statistics that counts various events - see 
> > Documentation/device-mapper/statistics.txt. It counts the nubmer of 
> > requests and the time spent servicing each request, thus you can 
> > calculate average latency from these values.
> 
> Right: average service time (as reported by iostat etc.) is easily derived
> from the existing stats.
> 
> Does the separate latency accounting buy anything additional?
>
> > Please look at dm-statistics to see if it fits your purpose. If you need 
> > additional information not provided by dm-statistics, it would be better 
> > to extend the statistics code rather than introduce new "latency" 
> > infrastructure.
> 
> Agreed; I'm working on userspace support for dm-statistics at the moment
> and if there is a need for these additional measurements I would greatly
> prefer to consume them as additional fields in the existing dm-stats
> counter set.
> 
> This also has the advantage of benefiting from the existing step and
> area support allowing a device to be subdivided into discrete stats
> regions.
> 
> Regards,
> Bryn.

Coly's paper (http://blog.coly.li/docs/osc13-coly.pdf) shows that they 
take histogram of latencies and use it to predict disk failure.

That could be easily added to dm-statistics.

Average latency alone can't be used to predict disk failure because 
average latency depends on the type of workload (for example - sequantial 
or nearly sequential requests have much lower latency than random 
requests).

I'd like to know if we need separate histogram per region, or if it is 
sufficient to have a histogram per device. dm-latency has no regions, it 
has a histogram for the whole device. The histogram-per-region would 
consume more memory, I'm interested if there is some reasonable use case 
for that.

Mikulas

Re: [PATCH 0/4] dm-latency: Introduction

[Bryn M. Reeves]

On Thu, Feb 26, 2015 at 12:00:47PM -0500, Laurence Oberman wrote:
> Mikulas
> Thanks
> This came from customer asking for support for similar functionality in proprietary solutions such as PowerPath and HDLM.
> I had seen the information from Coly Li and asked if he could submit for comments.
> I will look into what can be done with what is is /sys/block/dm-xxx/stat

Those are just the basic diskstats counters (similar to /proc/diskstats).

You want to look at the stuff described in Documentation/device-mapper/dm-statistics.txt

Regards,
Bryn.

Re: [PATCH 0/4] dm-latency: Introduction

[Bryn M. Reeves]

On Thu, Feb 26, 2015 at 11:49:28AM -0500, Mikulas Patocka wrote:
> We have already dm-statistics that counts various events - see 
> Documentation/device-mapper/statistics.txt. It counts the nubmer of 
> requests and the time spent servicing each request, thus you can 
> calculate average latency from these values.

Right: average service time (as reported by iostat etc.) is easily derived
from the existing stats.

Does the separate latency accounting buy anything additional?

> Please look at dm-statistics to see if it fits your purpose. If you need 
> additional information not provided by dm-statistics, it would be better 
> to extend the statistics code rather than introduce new "latency" 
> infrastructure.

Agreed; I'm working on userspace support for dm-statistics at the moment
and if there is a need for these additional measurements I would greatly
prefer to consume them as additional fields in the existing dm-stats
counter set.

This also has the advantage of benefiting from the existing step and
area support allowing a device to be subdivided into discrete stats
regions.

Regards,
Bryn.

Re: [PATCH 0/4] dm-latency: Introduction

[Laurence Oberman]

Mikulas
Thanks
This came from customer asking for support for similar functionality in proprietary solutions such as PowerPath and HDLM.
I had seen the information from Coly Li and asked if he could submit for comments.
I will look into what can be done with what is is /sys/block/dm-xxx/stat


Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Mikulas Patocka" <mpatocka@redhat.com>
To: "device-mapper development" <dm-devel@redhat.com>, "Coly Li" <colyli@gmail.com>
Cc: "Tao Ma" <boyu.mt@taobao.com>, "Robin Dong" <sanbai@alibaba-inc.com>, "Laurence Oberman" <loberman@redhat.com>, "Alasdair Kergon" <agk@redhat.com>
Sent: Thursday, February 26, 2015 11:49:28 AM
Subject: Re: [dm-devel] [PATCH 0/4] dm-latency: Introduction

Hi

We have already dm-statistics that counts various events - see 
Documentation/device-mapper/statistics.txt. It counts the nubmer of 
requests and the time spent servicing each request, thus you can 
calculate average latency from these values.

Please look at dm-statistics to see if it fits your purpose. If you need 
additional information not provided by dm-statistics, it would be better 
to extend the statistics code rather than introduce new "latency" 
infrastructure.

Mikulas


On Thu, 26 Feb 2015, Coly Li wrote:

> From: Coly Li <bosong.ly@alibaba-inc.com>
> 
> Dm-latency patch set is an effort to measure hard disk I/O latency on
> top of device mapper layer. The original motivation of I/O latency
> measurement was to predict hard disk failure by machine learning method,
> I/O latency information was one of the inputs sent to machine learning
> model.
> 
> This patch set was written in Aug~Sep 2013, I deployed it on many
> servers of Alibaba cloud infrastructure. After running for weeks, some
> interesting data about hard disk I/O latency was observed. In 2013, I
> gave a talk on OpenSuSE Conference about this topic
> (http://blog.coly.li/docs/osc13-coly.pdf).
> 
> When generating time stamp for I/O request, clock source is a global
> unique resource which is protected by spin-locks. Dm-latency was tested
> on SAS/SATA hard disk and SATA SSD, it worked well as expected. Running
> dm-latency on PCI-e or NVMe SSD should work (I didn't test), but there
> will be spin-lock scalability issue, when accessing clock source for
> time stamping.
> 
> Dm-latency is good for I/O latency measurement to hard disk based
> storage, no matter local or distributed storage via network. For PCI-e
> or NVMe SSD, I suggest people to look for device provided statistic
> information, if there is.
> 
> The code is very simple, there is no resource allocation/destory, no
> spin_lock/spin_unlock. The patch set gets merged into Alibaba kernel
> more than 1 year, no bug reported in last 12 months.
> 
> This patch set has 4 patches,
> - [PATCH 1/4] dm-latency: move struct mapped_device from dm.c to dm.h
> - [PATCH 2/4] dm-latency: add I/O latency measurement in device mapper
> - [PATCH 3/4] dm-latency: add sysfs interface
> - [PATCH 4/4] dm-latency: add reset function to dm-latency in sysfs
> interface
> All these patches are rebased on Linux 4.0-rc1.
> 
> Today Laurence Oberman from Redhat sent me an email asking whether this
> patch set is upstream merged, because he is thinking of pulling this
> patch set into their kernel. I'd like to maintain this patch set, hope
> it could be merged.
> 
> Thanks in advance.
> 
> Coly Li
> 
> 
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

Re: [PATCH 0/4] dm-latency: Introduction

[Mikulas Patocka]

Hi

We have already dm-statistics that counts various events - see 
Documentation/device-mapper/statistics.txt. It counts the nubmer of 
requests and the time spent servicing each request, thus you can 
calculate average latency from these values.

Please look at dm-statistics to see if it fits your purpose. If you need 
additional information not provided by dm-statistics, it would be better 
to extend the statistics code rather than introduce new "latency" 
infrastructure.

Mikulas


On Thu, 26 Feb 2015, Coly Li wrote:

> From: Coly Li <bosong.ly@alibaba-inc.com>
> 
> Dm-latency patch set is an effort to measure hard disk I/O latency on
> top of device mapper layer. The original motivation of I/O latency
> measurement was to predict hard disk failure by machine learning method,
> I/O latency information was one of the inputs sent to machine learning
> model.
> 
> This patch set was written in Aug~Sep 2013, I deployed it on many
> servers of Alibaba cloud infrastructure. After running for weeks, some
> interesting data about hard disk I/O latency was observed. In 2013, I
> gave a talk on OpenSuSE Conference about this topic
> (http://blog.coly.li/docs/osc13-coly.pdf).
> 
> When generating time stamp for I/O request, clock source is a global
> unique resource which is protected by spin-locks. Dm-latency was tested
> on SAS/SATA hard disk and SATA SSD, it worked well as expected. Running
> dm-latency on PCI-e or NVMe SSD should work (I didn't test), but there
> will be spin-lock scalability issue, when accessing clock source for
> time stamping.
> 
> Dm-latency is good for I/O latency measurement to hard disk based
> storage, no matter local or distributed storage via network. For PCI-e
> or NVMe SSD, I suggest people to look for device provided statistic
> information, if there is.
> 
> The code is very simple, there is no resource allocation/destory, no
> spin_lock/spin_unlock. The patch set gets merged into Alibaba kernel
> more than 1 year, no bug reported in last 12 months.
> 
> This patch set has 4 patches,
> - [PATCH 1/4] dm-latency: move struct mapped_device from dm.c to dm.h
> - [PATCH 2/4] dm-latency: add I/O latency measurement in device mapper
> - [PATCH 3/4] dm-latency: add sysfs interface
> - [PATCH 4/4] dm-latency: add reset function to dm-latency in sysfs
> interface
> All these patches are rebased on Linux 4.0-rc1.
> 
> Today Laurence Oberman from Redhat sent me an email asking whether this
> patch set is upstream merged, because he is thinking of pulling this
> patch set into their kernel. I'd like to maintain this patch set, hope
> it could be merged.
> 
> Thanks in advance.
> 
> Coly Li
> 
> 
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

[PATCH 0/4] dm-latency: Introduction

[Coly Li]

From: Coly Li <bosong.ly@alibaba-inc.com>

Dm-latency patch set is an effort to measure hard disk I/O latency on
top of device mapper layer. The original motivation of I/O latency
measurement was to predict hard disk failure by machine learning method,
I/O latency information was one of the inputs sent to machine learning
model.

This patch set was written in Aug~Sep 2013, I deployed it on many
servers of Alibaba cloud infrastructure. After running for weeks, some
interesting data about hard disk I/O latency was observed. In 2013, I
gave a talk on OpenSuSE Conference about this topic
(http://blog.coly.li/docs/osc13-coly.pdf).

When generating time stamp for I/O request, clock source is a global
unique resource which is protected by spin-locks. Dm-latency was tested
on SAS/SATA hard disk and SATA SSD, it worked well as expected. Running
dm-latency on PCI-e or NVMe SSD should work (I didn't test), but there
will be spin-lock scalability issue, when accessing clock source for
time stamping.

Dm-latency is good for I/O latency measurement to hard disk based
storage, no matter local or distributed storage via network. For PCI-e
or NVMe SSD, I suggest people to look for device provided statistic
information, if there is.

The code is very simple, there is no resource allocation/destory, no
spin_lock/spin_unlock. The patch set gets merged into Alibaba kernel
more than 1 year, no bug reported in last 12 months.

This patch set has 4 patches,
- [PATCH 1/4] dm-latency: move struct mapped_device from dm.c to dm.h
- [PATCH 2/4] dm-latency: add I/O latency measurement in device mapper
- [PATCH 3/4] dm-latency: add sysfs interface
- [PATCH 4/4] dm-latency: add reset function to dm-latency in sysfs
interface
All these patches are rebased on Linux 4.0-rc1.

Today Laurence Oberman from Redhat sent me an email asking whether this
patch set is upstream merged, because he is thinking of pulling this
patch set into their kernel. I'd like to maintain this patch set, hope
it could be merged.

Thanks in advance.

Coly Li