Bad SSD performance with recent kernels

Bad SSD performance with recent kernels

[Herbert Poetzl]

Dear Community!

Recently I decided to update the kernel on a Dell Laptop to
a more recent version than 2.6.38.x, but experienced bad
I/O performance with the new kernel, so I started to dig
a little deeper and ended up with the following test:

  1) download and extract kernel (on 2.6.38.8)
  2) make defconfig
  3) make localmodconfig
  4) make
  5) make modules_install install

Then I booted each kernel in single user and ran the following
test script:

  echo noop >/sys/class/block/sda/queue/scheduler
  for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
  /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
        ionice -c0 nice -20 \
        dd if=/dev/sda of=/dev/null bs=1M count=20480

  echo deadline >/sys/class/block/sda/queue/scheduler
  for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
  /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
        ionice -c0 nice -20 \
        dd if=/dev/sda of=/dev/null bs=1M count=20480

  echo cfq >/sys/class/block/sda/queue/scheduler
  for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
  /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
        ionice -c0 nice -20 \
        dd if=/dev/sda of=/dev/null bs=1M count=20480


note that the Laptop is a relatively modern Latitude E6400
with a Samsung 830 Series 256GB SSD

here are the surprising results:

@ linux 2.6.38.8
  248 MB/s  real = 86.74, user = 0.01, sys = 21.65, 24% cpu
  248 MB/s  real = 86.81, user = 0.02, sys = 21.75, 25% cpu
  251 MB/s  real = 85.63, user = 0.01, sys = 22.24, 25% cpu

@ linux 2.6.39.4
 49.0 MB/s  real = 438.79, user = 0.01, sys = 19.79, 4% cpu
 25.7 MB/s  real = 836.70, user = 0.02, sys = 18.39, 2% cpu
 27.7 MB/s  real = 776.53, user = 0.01, sys = 16.03, 2% cpu

@ linux 3.0.18
 48.9 MB/s  real = 439.07, user = 0.01, sys = 17.55, 4% cpu
 25.0 MB/s  real = 859.03, user = 0.01, sys = 16.97, 1% cpu
 49.8 MB/s  real = 431.61, user = 0.01, sys = 16.68, 3% cpu

@ linux 3.1.10
 54.0 MB/s  real = 398.23, user = 0.01, sys = 17.36, 4% cpu
 29.4 MB/s  real = 731.47, user = 0.01, sys = 17.14, 2% cpu
 25.0 MB/s  real = 859.35, user = 0.01, sys = 14.51, 1% cpu

@ linux 3.2.2
 45.8 MB/s  real = 468.85, user = 0.01, sys = 17.11, 3% cpu
 44.8 MB/s  real = 478.92, user = 0.01, sys = 17.02, 3% cpu
 45.0 MB/s  real = 476.91, user = 0.01, sys = 16.14, 3% cpu


I have no idea why the I/O performance is that bad on any
kernel newer than 2.6.38.x, but I'm happy to test and/or
try various configurations as time permits ...

The detailed test results as well as the dmesg and config
of each kernel can be found here:
http://vserver.13thfloor.at/Stuff/SSD

many thanks in advance,
Herbert

here some technical information:

http://www.dell.com/us/dfb/p/latitude-e6400/pd
SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03)

http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec
Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q
supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le vendredi 27 janvier 2012 à 07:00 +0100, Herbert Poetzl a écrit :
> Dear Community!
> 
> Recently I decided to update the kernel on a Dell Laptop to
> a more recent version than 2.6.38.x, but experienced bad
> I/O performance with the new kernel, so I started to dig
> a little deeper and ended up with the following test:
> 
>   1) download and extract kernel (on 2.6.38.8)
>   2) make defconfig
>   3) make localmodconfig
>   4) make
>   5) make modules_install install
> 
> Then I booted each kernel in single user and ran the following
> test script:
> 
>   echo noop >/sys/class/block/sda/queue/scheduler
>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>         ionice -c0 nice -20 \
>         dd if=/dev/sda of=/dev/null bs=1M count=20480
> 
>   echo deadline >/sys/class/block/sda/queue/scheduler
>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>         ionice -c0 nice -20 \
>         dd if=/dev/sda of=/dev/null bs=1M count=20480
> 
>   echo cfq >/sys/class/block/sda/queue/scheduler
>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>         ionice -c0 nice -20 \
>         dd if=/dev/sda of=/dev/null bs=1M count=20480
> 
> 
> note that the Laptop is a relatively modern Latitude E6400
> with a Samsung 830 Series 256GB SSD
> 
> here are the surprising results:
> 
> @ linux 2.6.38.8
>   248 MB/s  real = 86.74, user = 0.01, sys = 21.65, 24% cpu
>   248 MB/s  real = 86.81, user = 0.02, sys = 21.75, 25% cpu
>   251 MB/s  real = 85.63, user = 0.01, sys = 22.24, 25% cpu
> 
> @ linux 2.6.39.4
>  49.0 MB/s  real = 438.79, user = 0.01, sys = 19.79, 4% cpu
>  25.7 MB/s  real = 836.70, user = 0.02, sys = 18.39, 2% cpu
>  27.7 MB/s  real = 776.53, user = 0.01, sys = 16.03, 2% cpu
> 
> @ linux 3.0.18
>  48.9 MB/s  real = 439.07, user = 0.01, sys = 17.55, 4% cpu
>  25.0 MB/s  real = 859.03, user = 0.01, sys = 16.97, 1% cpu
>  49.8 MB/s  real = 431.61, user = 0.01, sys = 16.68, 3% cpu
> 
> @ linux 3.1.10
>  54.0 MB/s  real = 398.23, user = 0.01, sys = 17.36, 4% cpu
>  29.4 MB/s  real = 731.47, user = 0.01, sys = 17.14, 2% cpu
>  25.0 MB/s  real = 859.35, user = 0.01, sys = 14.51, 1% cpu
> 
> @ linux 3.2.2
>  45.8 MB/s  real = 468.85, user = 0.01, sys = 17.11, 3% cpu
>  44.8 MB/s  real = 478.92, user = 0.01, sys = 17.02, 3% cpu
>  45.0 MB/s  real = 476.91, user = 0.01, sys = 16.14, 3% cpu
> 
> 
> I have no idea why the I/O performance is that bad on any
> kernel newer than 2.6.38.x, but I'm happy to test and/or
> try various configurations as time permits ...
> 
> The detailed test results as well as the dmesg and config
> of each kernel can be found here:
> http://vserver.13thfloor.at/Stuff/SSD
> 
> many thanks in advance,
> Herbert
> 
> here some technical information:
> 
> http://www.dell.com/us/dfb/p/latitude-e6400/pd
> SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03)
> 
> http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec
> Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q
> supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS
> 
> 


I confirm I just noticed this on a brand new SSD from Crucial (128 GB)
and a Dell E6400 too (3.5 years old model)

The Ubuntu upgrade from 11.04 to 11.10 switched kernel (from 2.6.38.X to
3.0.0.X) and raw SSD performance (hdparm -t /dev/sda) dropped from 200
MB/s to 140 MB/s or so.

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

Hi Herbert,

On Fri, Jan 27, 2012 at 07:00:34AM +0100, Herbert Poetzl wrote:
> 
> Dear Community!
> 
> Recently I decided to update the kernel on a Dell Laptop to
> a more recent version than 2.6.38.x, but experienced bad
> I/O performance with the new kernel, so I started to dig
> a little deeper and ended up with the following test:
> 
>   1) download and extract kernel (on 2.6.38.8)
>   2) make defconfig
>   3) make localmodconfig
>   4) make
>   5) make modules_install install
> 
> Then I booted each kernel in single user and ran the following
> test script:
> 
>   echo noop >/sys/class/block/sda/queue/scheduler
>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>         ionice -c0 nice -20 \
>         dd if=/dev/sda of=/dev/null bs=1M count=20480
> 
>   echo deadline >/sys/class/block/sda/queue/scheduler
>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>         ionice -c0 nice -20 \
>         dd if=/dev/sda of=/dev/null bs=1M count=20480
> 
>   echo cfq >/sys/class/block/sda/queue/scheduler
>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>         ionice -c0 nice -20 \
>         dd if=/dev/sda of=/dev/null bs=1M count=20480
> 
> 
> note that the Laptop is a relatively modern Latitude E6400
> with a Samsung 830 Series 256GB SSD
> 
> here are the surprising results:
> 
> @ linux 2.6.38.8
>   248 MB/s  real = 86.74, user = 0.01, sys = 21.65, 24% cpu
>   248 MB/s  real = 86.81, user = 0.02, sys = 21.75, 25% cpu
>   251 MB/s  real = 85.63, user = 0.01, sys = 22.24, 25% cpu
> 
> @ linux 2.6.39.4
>  49.0 MB/s  real = 438.79, user = 0.01, sys = 19.79, 4% cpu
>  25.7 MB/s  real = 836.70, user = 0.02, sys = 18.39, 2% cpu
>  27.7 MB/s  real = 776.53, user = 0.01, sys = 16.03, 2% cpu
> 
> @ linux 3.0.18
>  48.9 MB/s  real = 439.07, user = 0.01, sys = 17.55, 4% cpu
>  25.0 MB/s  real = 859.03, user = 0.01, sys = 16.97, 1% cpu
>  49.8 MB/s  real = 431.61, user = 0.01, sys = 16.68, 3% cpu
> 
> @ linux 3.1.10
>  54.0 MB/s  real = 398.23, user = 0.01, sys = 17.36, 4% cpu
>  29.4 MB/s  real = 731.47, user = 0.01, sys = 17.14, 2% cpu
>  25.0 MB/s  real = 859.35, user = 0.01, sys = 14.51, 1% cpu
> 
> @ linux 3.2.2
>  45.8 MB/s  real = 468.85, user = 0.01, sys = 17.11, 3% cpu
>  44.8 MB/s  real = 478.92, user = 0.01, sys = 17.02, 3% cpu
>  45.0 MB/s  real = 476.91, user = 0.01, sys = 16.14, 3% cpu

Thanks for the through tests! However I cannot reproduce the regressions,
but see slightly better performance in 3.2:

Linux lkp-nex04 3.2.0-rc7-shli+ #121 SMP Thu Jan 19 18:10:45 CST 2012 x86_64 x86_64 x86_64 GNU/Linux

21474836480 bytes (21 GB) copied, 90.9307 s, 236 MB/s real = 91.03, user = 0.01, sys = 25.80, 28% cpu
21474836480 bytes (21 GB) copied, 90.8864 s, 236 MB/s real = 90.90, user = 0.01, sys = 25.54, 28% cpu
21474836480 bytes (21 GB) copied, 93.4684 s, 230 MB/s real = 93.47, user = 0.02, sys = 25.12, 26% cpu


Linux lkp-nex04 2.6.38 #334 SMP Sat Jan 28 20:16:25 CST 2012 x86_64 GNU/Linux

21474836480 bytes (21 GB) copied, 96.2382 s, 223 MB/s real = 96.29, user = 0.01, sys = 57.64, 59% cpu
21474836480 bytes (21 GB) copied, 96.5796 s, 222 MB/s real = 96.60, user = 0.02, sys = 58.08, 60% cpu
21474836480 bytes (21 GB) copied, 95.486  s, 225 MB/s real = 95.51, user = 0.00, sys = 58.20, 60% cpu

My test box has 64GB memory and a dozen 80GB INTEL SSDSA2M080G2GN SSD drives attached to

87:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)

> I have no idea why the I/O performance is that bad on any
> kernel newer than 2.6.38.x, but I'm happy to test and/or
> try various configurations as time permits ...

Would you please create a filesystem and large file on sda and run the
tests on the file? There was some performance bug on reading the raw
/dev/sda device file..

> The detailed test results as well as the dmesg and config
> of each kernel can be found here:
> http://vserver.13thfloor.at/Stuff/SSD

It would be better for such files be attached in future.
(For one thing I get timeout when trying to open the link...)

Thanks,
Fengguang

> here some technical information:
> 
> http://www.dell.com/us/dfb/p/latitude-e6400/pd
> SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03)
> 
> http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec
> Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q
> supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit :

> Would you please create a filesystem and large file on sda and run the
> tests on the file? There was some performance bug on reading the raw
> /dev/sda device file..

Hmm... latest kernel has the performance bug right now.

Really if /dev/sda is slow, we are stuck.

FYI, I started a bisection.

Re: Bad SSD performance with recent kernels

[Herbert Poetzl]

[-- Attachment #1: Type: text/plain, Size: 5195 bytes --]

On Sat, Jan 28, 2012 at 08:51:08PM +0800, Wu Fengguang wrote:
> Hi Herbert,

> On Fri, Jan 27, 2012 at 07:00:34AM +0100, Herbert Poetzl wrote:

>> Dear Community!

>> Recently I decided to update the kernel on a Dell Laptop to
>> a more recent version than 2.6.38.x, but experienced bad
>> I/O performance with the new kernel, so I started to dig
>> a little deeper and ended up with the following test:

>>   1) download and extract kernel (on 2.6.38.8)
>>   2) make defconfig
>>   3) make localmodconfig
>>   4) make
>>   5) make modules_install install

>> Then I booted each kernel in single user and ran the following
>> test script:

>>   echo noop >/sys/class/block/sda/queue/scheduler
>>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>>         ionice -c0 nice -20 \
>>         dd if=/dev/sda of=/dev/null bs=1M count=20480

>>   echo deadline >/sys/class/block/sda/queue/scheduler
>>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>>         ionice -c0 nice -20 \
>>         dd if=/dev/sda of=/dev/null bs=1M count=20480

>>   echo cfq >/sys/class/block/sda/queue/scheduler
>>   for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
>>   /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
>>         ionice -c0 nice -20 \
>>         dd if=/dev/sda of=/dev/null bs=1M count=20480


>> note that the Laptop is a relatively modern Latitude E6400
>> with a Samsung 830 Series 256GB SSD

>> here are the surprising results:

>> @ linux 2.6.38.8
>>   248 MB/s  real = 86.74, user = 0.01, sys = 21.65, 24% cpu
>>   248 MB/s  real = 86.81, user = 0.02, sys = 21.75, 25% cpu
>>   251 MB/s  real = 85.63, user = 0.01, sys = 22.24, 25% cpu

>> @ linux 2.6.39.4
>>  49.0 MB/s  real = 438.79, user = 0.01, sys = 19.79, 4% cpu
>>  25.7 MB/s  real = 836.70, user = 0.02, sys = 18.39, 2% cpu
>>  27.7 MB/s  real = 776.53, user = 0.01, sys = 16.03, 2% cpu

>> @ linux 3.0.18
>>  48.9 MB/s  real = 439.07, user = 0.01, sys = 17.55, 4% cpu
>>  25.0 MB/s  real = 859.03, user = 0.01, sys = 16.97, 1% cpu
>>  49.8 MB/s  real = 431.61, user = 0.01, sys = 16.68, 3% cpu

>> @ linux 3.1.10
>>  54.0 MB/s  real = 398.23, user = 0.01, sys = 17.36, 4% cpu
>>  29.4 MB/s  real = 731.47, user = 0.01, sys = 17.14, 2% cpu
>>  25.0 MB/s  real = 859.35, user = 0.01, sys = 14.51, 1% cpu

>> @ linux 3.2.2
>>  45.8 MB/s  real = 468.85, user = 0.01, sys = 17.11, 3% cpu
>>  44.8 MB/s  real = 478.92, user = 0.01, sys = 17.02, 3% cpu
>>  45.0 MB/s  real = 476.91, user = 0.01, sys = 16.14, 3% cpu

> Thanks for the through tests! However I cannot reproduce the
> regressions, but see slightly better performance in 3.2:

> Linux lkp-nex04 3.2.0-rc7-shli+ #121 SMP Thu Jan 19 18:10:45 CST 2012 x86_64 x86_64 x86_64 GNU/Linux

> 21474836480 bytes (21 GB) copied, 90.9307 s, 236 MB/s real = 91.03, user = 0.01, sys = 25.80, 28% cpu
> 21474836480 bytes (21 GB) copied, 90.8864 s, 236 MB/s real = 90.90, user = 0.01, sys = 25.54, 28% cpu
> 21474836480 bytes (21 GB) copied, 93.4684 s, 230 MB/s real = 93.47, user = 0.02, sys = 25.12, 26% cpu


> Linux lkp-nex04 2.6.38 #334 SMP Sat Jan 28 20:16:25 CST 2012 x86_64 GNU/Linux

> 21474836480 bytes (21 GB) copied, 96.2382 s, 223 MB/s real = 96.29, user = 0.01, sys = 57.64, 59% cpu
> 21474836480 bytes (21 GB) copied, 96.5796 s, 222 MB/s real = 96.60, user = 0.02, sys = 58.08, 60% cpu
> 21474836480 bytes (21 GB) copied, 95.486  s, 225 MB/s real = 95.51, user = 0.00, sys = 58.20, 60% cpu

> My test box has 64GB memory and a dozen 80GB INTEL
> SSDSA2M080G2GN SSD drives attached to

so we have intel vs samsung here ...

> 87:00.0 Serial Attached SCSI controller: LSI Logic / Symbios
> Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)

... and sas vs sata

>> I have no idea why the I/O performance is that bad on any
>> kernel newer than 2.6.38.x, but I'm happy to test and/or
>> try various configurations as time permits ...

> Would you please create a filesystem and large file on sda and
> run the tests on the file? There was some performance bug on
> reading the raw /dev/sda device file..

will do tonight!

>> The detailed test results as well as the dmesg and config
>> of each kernel can be found here:
>> http://vserver.13thfloor.at/Stuff/SSD

> It would be better for such files be attached in future.
> (For one thing I get timeout when trying to open the link...)

attached the files as .txz

thanks,
Herbert

> Thanks,
> Fengguang

>> here some technical information:

>> http://www.dell.com/us/dfb/p/latitude-e6400/pd
>> SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03)

>> http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec
>> Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q
>> supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS


>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/

[-- Attachment #2: SSD.txz --]
[-- Type: application/octet-stream, Size: 57008 bytes --]

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Sat, Jan 28, 2012 at 02:33:31PM +0100, Eric Dumazet wrote:
> Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit :
> 
> > Would you please create a filesystem and large file on sda and run the
> > tests on the file? There was some performance bug on reading the raw
> > /dev/sda device file..
> 
> Hmm... latest kernel has the performance bug right now.
> 
> Really if /dev/sda is slow, we are stuck.

What's the block size? If it's < 4k, performance might be hurt.

        blockdev --getbsz /dev/sda

> FYI, I started a bisection.

Thank you! If the bisection would take much human time, it should be
easier to collect some blktrace data on reading /dev/sda for analyzes.

Thanks,
Fengguang

Re: Bad SSD performance with recent kernels

[Herbert Poetzl]

On Sun, Jan 29, 2012 at 01:59:17PM +0800, Wu Fengguang wrote:
> On Sat, Jan 28, 2012 at 02:33:31PM +0100, Eric Dumazet wrote:
>> Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit :

>>> Would you please create a filesystem and large file on sda
>>> and run the tests on the file? There was some performance bug
>>> on reading the raw /dev/sda device file..

as promised, I did the tests on a filesystem, created on
a partition of the disk, and here are the (IMHO quite
interesting) results:

kernel    -- write ---  ------------------read -----------------
          --- noop ---  --- noop ---  - deadline -  ---- cfs ---
          [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
----------------------------------------------------------------
2.6.38.8  268.76  49.6  169.20  11.3  169.17  11.3  167.89  11.4
2.6.39.4  269.73  50.3  162.03  10.9  161.58  10.9  161.64  11.0
3.0.18    269.17  42.0  161.87   9.9  161.36  10.0  161.68  10.1
3.1.10    271.62  43.1  161.91   9.9  161.68   9.9  161.25  10.1
3.2.2     270.95  42.6  162.36   9.9  162.63   9.9  162.65  10.1

so while the 'expected' performance should be somewhere around
300MB/s for read and write (raw disk access) we end up with
good write performance and roughly half the read performance
with 'dd bs=1M' on ext3

here the script I used:

mke2fs -j /dev/sda5
mount /dev/sda5 /media

/usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
	ionice -c0 nice -20 \
	dd if=/dev/zero of=/media/zero.data bs=1M count=19900

echo noop >/sys/class/block/sda/queue/scheduler
for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
/usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
	ionice -c0 nice -20 \
	dd if=/media/zero.data of=/dev/null bs=1M count=19900

echo deadline >/sys/class/block/sda/queue/scheduler
for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
/usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
	ionice -c0 nice -20 \
	dd if=/media/zero.data of=/dev/null bs=1M count=19900

echo cfq >/sys/class/block/sda/queue/scheduler
for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
/usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
	ionice -c0 nice -20 \
	dd if=/media/zero.data of=/dev/null bs=1M count=19900

>> Hmm... latest kernel has the performance bug right now.

>> Really if /dev/sda is slow, we are stuck.

> What's the block size? If it's < 4k, performance might be hurt.
>         blockdev --getbsz /dev/sda

4096

>> FYI, I started a bisection.

> Thank you! If the bisection would take much human time, it should be
> easier to collect some blktrace data on reading /dev/sda for analyzes.

will do some bonnie++ tests on the partition later today

HTH,
Herbert

> Thanks,
> Fengguang

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Sun, Jan 29, 2012 at 09:42:59AM +0100, Herbert Poetzl wrote:
> On Sun, Jan 29, 2012 at 01:59:17PM +0800, Wu Fengguang wrote:
> > On Sat, Jan 28, 2012 at 02:33:31PM +0100, Eric Dumazet wrote:
> >> Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit :
> 
> >>> Would you please create a filesystem and large file on sda
> >>> and run the tests on the file? There was some performance bug
> >>> on reading the raw /dev/sda device file..
> 
> as promised, I did the tests on a filesystem, created on
> a partition of the disk, and here are the (IMHO quite
> interesting) results:
> 
> kernel    -- write ---  ------------------read -----------------
>           --- noop ---  --- noop ---  - deadline -  ---- cfs ---
>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
> ----------------------------------------------------------------
> 2.6.38.8  268.76  49.6  169.20  11.3  169.17  11.3  167.89  11.4

Hmm, read performance drops between 2.6.38 and 2.6.39...

> 2.6.39.4  269.73  50.3  162.03  10.9  161.58  10.9  161.64  11.0
> 3.0.18    269.17  42.0  161.87   9.9  161.36  10.0  161.68  10.1

Between 3.0 and 3.1, the writeback chunk size is raised by commit
1a12d8bd7b2998b ("writeback: scale IO chunk size up to half device
bandwidth") which should be the main reason for the improved write
throughput.

> 3.1.10    271.62  43.1  161.91   9.9  161.68   9.9  161.25  10.1
> 3.2.2     270.95  42.6  162.36   9.9  162.63   9.9  162.65  10.1
> 
> so while the 'expected' performance should be somewhere around
> 300MB/s for read and write (raw disk access) we end up with
> good write performance and roughly half the read performance
> with 'dd bs=1M' on ext3

That could be explained by large write chunk size (>=4MB) and small
readahead size (128KB). Long time ago I collected some read
experiments on SSD and find it asks for 4MB readahead size to get
best performance: 

SSD 80G Intel x25-M SSDSA2M080 (reported by Li Shaohua)

        rasize  1st run         2nd run
        ----------------------------------
          4k    123 MB/s        122 MB/s
         16k    153 MB/s        153 MB/s
         32k    161 MB/s        162 MB/s
         64k    167 MB/s        168 MB/s
        128k    197 MB/s        197 MB/s
        256k    217 MB/s        217 MB/s
        512k    238 MB/s        234 MB/s
          1M    251 MB/s        248 MB/s
          2M    259 MB/s        257 MB/s
==>       4M    269 MB/s        264 MB/s
          8M    266 MB/s        266 MB/s

Note that ==> points to the readahead size that yields plateau throughput.

SSD 22G MARVELL SD88SA02 MP1F (reported by Jens Axboe)

        rasize  1st             2nd
        --------------------------------
          4k     41 MB/s         41 MB/s
         16k     85 MB/s         81 MB/s
         32k    102 MB/s        109 MB/s
         64k    125 MB/s        144 MB/s
        128k    183 MB/s        185 MB/s
        256k    216 MB/s        216 MB/s
        512k    216 MB/s        236 MB/s
        1024k   251 MB/s        252 MB/s
          2M    258 MB/s        258 MB/s
==>       4M    266 MB/s        266 MB/s
          8M    266 MB/s        266 MB/s

Thanks,
Fengguang

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le dimanche 29 janvier 2012 à 13:59 +0800, Wu Fengguang a écrit :

> What's the block size? If it's < 4k, performance might be hurt.
> 
>         blockdev --getbsz /dev/sda
> 

# blockdev --getbsz /dev/sda
4096



> > FYI, I started a bisection.
> 
> Thank you! If the bisection would take much human time, it should be
> easier to collect some blktrace data on reading /dev/sda for analyzes.
> 

Very strange, my bissection ended on following commit :

commit 805f6b5e1cbfedfb9b3d354013e7f4b13a79270f
Author: Tao Ma <boyu.mt@taobao.com>
Date:   Fri Mar 11 20:11:59 2011 +0100

    blktrace: Use rq->cmd_flags directly in blk_add_trace_rq.



This makes no sense.

hdparm uses 2MB block reads, so read_ahead (128KB) is too small for best
perf

# cat /sys/class/block/sda/queue/read_ahead_kb
128

# dd if=/dev/sda of=/dev/null bs=128k 
^C
63744+0 enregistrements lus
63743+0 enregistrements écrits
8354922496 octets (8,4 GB) copiés, 39,975 s, 209 MB/s

# hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads: 510 MB in  3.00 seconds = 169.75 MB/sec

# uname -a
Linux edumazet-laptop 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20
17:23:00 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Sun, Jan 29, 2012 at 11:03:00AM +0100, Eric Dumazet wrote:
> Le dimanche 29 janvier 2012 à 13:59 +0800, Wu Fengguang a écrit :
> 
> > What's the block size? If it's < 4k, performance might be hurt.
> > 
> >         blockdev --getbsz /dev/sda
> > 
> 
> # blockdev --getbsz /dev/sda
> 4096

OK, it's normal value.

> > > FYI, I started a bisection.
> > 
> > Thank you! If the bisection would take much human time, it should be
> > easier to collect some blktrace data on reading /dev/sda for analyzes.
> > 
> 
> Very strange, my bissection ended on following commit :
> 
> commit 805f6b5e1cbfedfb9b3d354013e7f4b13a79270f
> Author: Tao Ma <boyu.mt@taobao.com>
> Date:   Fri Mar 11 20:11:59 2011 +0100
> 
>     blktrace: Use rq->cmd_flags directly in blk_add_trace_rq.
> 
> 
> 
> This makes no sense.

Yup.

> hdparm uses 2MB block reads, so read_ahead (128KB) is too small for best
> perf

Right. In another thread, we are considering raising the default
readahead size to 1MB.

Note that as long as buffered read(2) is used, it makes almost no
difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
the 128kb readahead size will be used underneath to submit read IO.

> # cat /sys/class/block/sda/queue/read_ahead_kb
> 128
> 
> # dd if=/dev/sda of=/dev/null bs=128k 
> ^C
> 63744+0 enregistrements lus
> 63743+0 enregistrements écrits
> 8354922496 octets (8,4 GB) copiés, 39,975 s, 209 MB/s
> 
> # hdparm -t /dev/sda
> 
> /dev/sda:
>  Timing buffered disk reads: 510 MB in  3.00 seconds = 169.75 MB/sec

Interesting, hdparm gets lower throughput than dd.

> # uname -a
> Linux edumazet-laptop 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20
> 17:23:00 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Thanks,
Fengguang

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :


> Note that as long as buffered read(2) is used, it makes almost no
> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
> the 128kb readahead size will be used underneath to submit read IO.
> 

Hmm...

# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
32768+0 enregistrements lus
32768+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s


# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
2048+0 enregistrements lus
2048+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s

Re: Bad SSD performance with recent kernels

[Pádraig Brady]

On 01/29/2012 01:13 PM, Eric Dumazet wrote:
> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :
> 
> 
>> Note that as long as buffered read(2) is used, it makes almost no
>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
>> the 128kb readahead size will be used underneath to submit read IO.
>>
> 
> Hmm...
> 
> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> 32768+0 enregistrements lus
> 32768+0 enregistrements écrits
> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s
> 
> 
> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> 2048+0 enregistrements lus
> 2048+0 enregistrements écrits
> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s

Same here on 2.6.40.4-5.fc15.x86_64
Note the SSD is rated for 500MB/s but is on a SATA II port,
and so limited by that. So the 128k result below is
close to the limit on this system.

Hmm, I previously tested this SSD with kernel-2.6.38.6-26.rc1.fc15.src.rpm
and got 270MB/s. Testing now gives variable and lower results:

# echo 3 >/proc/sys/vm/drop_caches; hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   8388 MB in  2.00 seconds = 4200.73 MB/sec
 Timing buffered disk reads: 550 MB in  3.00 seconds = 183.19 MB/sec
# echo 3 >/proc/sys/vm/drop_caches; hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   8260 MB in  2.00 seconds = 4134.30 MB/sec
 Timing buffered disk reads: 680 MB in  3.00 seconds = 226.63 MB/sec
# echo 3 >/proc/sys/vm/drop_caches; hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   8426 MB in  2.00 seconds = 4217.87 MB/sec
 Timing buffered disk reads: 588 MB in  3.00 seconds = 195.96 MB/sec

Anyway testing different block sizes with dd:

# echo 3 >/proc/sys/vm/drop_caches; timeout -sINT 5 dd if=/dev/sdb of=/dev/null bs=2M
966787072 bytes (967 MB) copied, 5.00525 s, 193 MB/s
# echo 3 >/proc/sys/vm/drop_caches; timeout -sINT 5 dd if=/dev/sdb of=/dev/null bs=128k
1246494720 bytes (1.2 GB) copied, 4.99563 s, 250 MB/s

On a probably unrelated note, I've always noticed dd getting slower,
independent of disks, when the buffer size increases beyond 2M.

for i in $(seq 0 15); do
  size=$((16*1024**3)) #ensure this is big enough
  bs=$((1024*2**$i))
  printf "%8s=" $bs
  dd bs=$bs if=/dev/zero of=/dev/null count=$(($size/$bs)) 2>&1 |
  sed -n 's/.* \([0-9.]* [GM]B\/s\)/\1/p'
done

    1024=1.4 GB/s
    2048=2.6 GB/s
    4096=4.5 GB/s
    8192=6.7 GB/s
   16384=8.8 GB/s
   32768=9.4 GB/s
   65536=10.8 GB/s
  131072=11.5 GB/s
  262144=11.5 GB/s
  524288=11.3 GB/s
 1048576=11.3 GB/s
 2097152=10.6 GB/s
 4194304=6.5 GB/s
 8388608=5.9 GB/s
16777216=6.6 GB/s
33554432=6.6 GB/s

cheers,
Pádraig.

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :
> 
> 
> > Note that as long as buffered read(2) is used, it makes almost no
> > difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
> > the 128kb readahead size will be used underneath to submit read IO.
> > 
> 
> Hmm...
> 
> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> 32768+0 enregistrements lus
> 32768+0 enregistrements écrits
> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s
> 
> 
> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> 2048+0 enregistrements lus
> 2048+0 enregistrements écrits
> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s

Interesting. Here are my test results:

root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
2048+0 records in
2048+0 records out
4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s

Maybe the /dev/sda performance bug on your machine is sensitive to timing?

Thanks,
Fengguang

Re: Bad SSD performance with recent kernels

[Herbert Poetzl]

[-- Attachment #1: Type: text/plain, Size: 3331 bytes --]

On Mon, Jan 30, 2012 at 12:10:58AM +0800, Wu Fengguang wrote:
> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :

>>> Note that as long as buffered read(2) is used, it makes almost no
>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
>>> the 128kb readahead size will be used underneath to submit read IO.

>> Hmm...

>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>> 32768+0 enregistrements lus
>> 32768+0 enregistrements écrits
>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s


>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>> 2048+0 enregistrements lus
>> 2048+0 enregistrements écrits
>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s

> Interesting. Here are my test results:

> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> 32768+0 records in
> 32768+0 records out
> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> 2048+0 records in
> 2048+0 records out
> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s

> Maybe the /dev/sda performance bug on your machine is sensitive to timing?

here are some more confusing results from tests with dd and bonnie++, 
this time I focused on partition vs. loop vs. linear dm (of same partition)

kernel	  -------------- read --------------  -- write ---  all
	  -------- dd --------  -------- bonnie++ --------------
	  [MB/s]  real    %CPU  [MB/s]  %CPU  [MB/s]  %CPU  %CPU
direct
2.6.38.8  262.91   81.90  28.7	 72.30   6.0  248.53  52.0  15.9
2.6.39.4   36.09  595.17   3.1	 70.62   6.0  250.25  53.0  16.3
3.0.18     50.47  425.65   4.1	 70.00   5.0  251.70  44.0  13.9
3.1.10     27.28  787.32   2.0	 75.65   5.0  251.96  45.0  13.3
3.2.2      27.11  792.28   2.0	 76.89   6.0  250.38  44.0  13.3

loop
2.6.38.8  242.89   88.50  21.5	246.58  15.0  240.92  53.0  14.4
2.6.39.4  241.06   89.19  21.5	238.51  15.0  257.59  57.0  14.8
3.0.18	  261.44   82.23  18.8	256.66  15.0  255.17  48.0  12.6
3.1.10	  253.93   84.64  18.1	107.66   7.0  156.51  28.0  10.6
3.2.2	  262.58   81.82  19.8	110.54   7.0  212.01  40.0  11.6

linear
2.6.38.8  262.57   82.00  36.8	 72.46   6.0  243.25  53.0  16.5
2.6.39.4   25.45  843.93   2.3	 70.70   6.0  248.05  54.0  16.6
3.0.18	   55.45  387.43   5.6	 69.72   6.0  249.42  45.0  14.3
3.1.10	   36.62  586.50   3.3	 74.74   6.0  249.99  46.0  13.4
3.2.2	   28.28  759.26   2.3	 74.20   6.0  248.73  46.0  13.6


it seems that dd performance when using a loop device is unaffected
and even improves with the kernel version, the filesystem performance
OTOH degrades after 3.1 ...

in general, filesystem read performance is bad on everything but
a loop device ... judging from the results I'd conclude that there
are at least two different issues 

tests and test results are attached and can be found here:
http://vserver.13thfloor.at/Stuff/SSD/

I plan to do some more tests on the filesystem with -b and -D
tonight, please let me know if you want to see specific output
and/or have any tests I should run with each kernel ...

HTH,
Herbert

> Thanks,
> Fengguang

[-- Attachment #2: SSD.txz --]
[-- Type: application/octet-stream, Size: 56488 bytes --]

Re: Bad SSD performance with recent kernels

[Shaohua Li]

2012/1/30 Wu Fengguang <wfg@linux.intel.com>:
> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :
>>
>>
>> > Note that as long as buffered read(2) is used, it makes almost no
>> > difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
>> > the 128kb readahead size will be used underneath to submit read IO.
>> >
>>
>> Hmm...
>>
>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>> 32768+0 enregistrements lus
>> 32768+0 enregistrements écrits
>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s
>>
>>
>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>> 2048+0 enregistrements lus
>> 2048+0 enregistrements écrits
>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s
>
> Interesting. Here are my test results:
>
> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> 32768+0 records in
> 32768+0 records out
> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> 2048+0 records in
> 2048+0 records out
> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s
>
> Maybe the /dev/sda performance bug on your machine is sensitive to timing?
I got similar result:
128k: 224M/s
1M: 182M/s

1M block size is slow, I guess it's CPU related.

And as for the big regression with newer kernel than 2.6.38, please
check if idle=poll
helps. CPU idle dramatically impacts disk performance and even latest
cpuidle governor
doesn't help for some CPUs.

Thanks,
Shaohua

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le lundi 30 janvier 2012 à 11:17 +0800, Shaohua Li a écrit :

> 1M block size is slow, I guess it's CPU related.
> 
> And as for the big regression with newer kernel than 2.6.38, please
> check if idle=poll
> helps. CPU idle dramatically impacts disk performance and even latest
> cpuidle governor
> doesn't help for some CPUs.
> 

No change with idle=poll

# echo 3 >/proc/sys/vm/drop_caches ;perf record dd if=/dev/sda
of=/dev/null bs=2M count=2048
2048+0 enregistrements lus
2048+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 27,6942 s, 155 MB/s
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 0.884 MB perf.data (~38630 samples) ]

# perf report
Events: 22K
cycles                                                              
 40,14%  dd  [kernel.kallsyms]  [k] copy_user_generic_string
  3,12%  dd  [kernel.kallsyms]  [k] native_read_tsc
  3,00%  dd  [kernel.kallsyms]  [k] put_page
  2,98%  dd  [kernel.kallsyms]  [k] iowrite8
  2,04%  dd  [kernel.kallsyms]  [k] get_page_from_freelist
  1,87%  dd  [kernel.kallsyms]  [k] delay_tsc
  1,77%  dd  [kernel.kallsyms]  [k] kmem_cache_alloc
  1,69%  dd  [kernel.kallsyms]  [k] __mem_cgroup_commit_charge
  1,52%  dd  [kernel.kallsyms]  [k] submit_bh
  1,38%  dd  [kernel.kallsyms]  [k] ioread8
  1,32%  dd  [kernel.kallsyms]  [k] __rmqueue
  1,29%  dd  [kernel.kallsyms]  [k] block_read_full_page
  1,28%  dd  [kernel.kallsyms]  [k] __ticket_spin_lock
  1,23%  dd  [kernel.kallsyms]  [k] __slab_alloc
  1,15%  dd  [kernel.kallsyms]  [k] file_read_actor
  0,85%  dd  [kernel.kallsyms]  [k] generic_make_request.part.51
  0,84%  dd  [kernel.kallsyms]  [k] __alloc_pages_nodemask
  0,82%  dd  [kernel.kallsyms]  [k] create_empty_buffers
  0,81%  dd  [kernel.kallsyms]  [k] get_partial_node
  0,81%  dd  [kernel.kallsyms]  [k] lookup_page_cgroup
  0,78%  dd  [kernel.kallsyms]  [k] mem_cgroup_add_lru_list.part.45
  0,72%  dd  [kernel.kallsyms]  [k] ____pagevec_lru_add_fn

# echo 3 >/proc/sys/vm/drop_caches ;perf record dd if=/dev/sda
of=/dev/null bs=1M count=4096
4096+0 enregistrements lus
4096+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 25,8117 s, 166 MB/s
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 0.810 MB perf.data (~35398 samples) ]

# perf report
Events: 20K
cycles                                                              
 31,91%  dd  [kernel.kallsyms]  [k] copy_user_generic_string
  3,87%  dd  [kernel.kallsyms]  [k] native_read_tsc
  3,58%  dd  [kernel.kallsyms]  [k] iowrite8
  2,43%  dd  [kernel.kallsyms]  [k] put_page
  2,30%  dd  [kernel.kallsyms]  [k] delay_tsc
  2,09%  dd  [kernel.kallsyms]  [k] get_page_from_freelist
  1,96%  dd  [kernel.kallsyms]  [k] __mem_cgroup_commit_charge
  1,79%  dd  [kernel.kallsyms]  [k] submit_bh
  1,75%  dd  [kernel.kallsyms]  [k] kmem_cache_alloc
  1,74%  dd  [kernel.kallsyms]  [k] ioread8
  1,68%  dd  [kernel.kallsyms]  [k] block_read_full_page
  1,62%  dd  [kernel.kallsyms]  [k] __ticket_spin_lock
  1,38%  dd  [kernel.kallsyms]  [k] __rmqueue
  1,12%  dd  [kernel.kallsyms]  [k] create_empty_buffers
  1,08%  dd  [kernel.kallsyms]  [k] __slab_alloc
  1,07%  dd  [kernel.kallsyms]  [k] file_read_actor
  1,06%  dd  [kernel.kallsyms]  [k] generic_make_request.part.51
  0,94%  dd  [kernel.kallsyms]  [k] lookup_page_cgroup
  0,85%  dd  [kernel.kallsyms]  [k] __alloc_pages_nodemask
  0,85%  dd  [kernel.kallsyms]  [k] find_get_page
  0,83%  dd  [kernel.kallsyms]  [k] __lru_cache_add
  0,81%  dd  [kernel.kallsyms]  [k] add_to_page_cache_locked.part.29


# echo 3 >/proc/sys/vm/drop_caches ;perf record dd if=/dev/sda
of=/dev/null bs=128k count=32768
32768+0 enregistrements lus
32768+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 19,6924 s, 218 MB/s
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 0.961 MB perf.data (~41992 samples) ]

# perf report
 29,85%  dd  [kernel.kallsyms]  [k] copy_user_generic_string
  2,61%  dd  [kernel.kallsyms]  [k] __mem_cgroup_commit_charge
  2,53%  dd  [kernel.kallsyms]  [k] get_page_from_freelist
  2,27%  dd  [kernel.kallsyms]  [k] block_read_full_page
  1,99%  dd  [kernel.kallsyms]  [k] __ticket_spin_lock
  1,81%  dd  [kernel.kallsyms]  [k] kmem_cache_alloc
  1,65%  dd  [kernel.kallsyms]  [k] file_read_actor
  1,60%  dd  [kernel.kallsyms]  [k] __rmqueue
  1,58%  dd  [kernel.kallsyms]  [k] put_page
  1,51%  dd  [kernel.kallsyms]  [k] submit_bh
  1,37%  dd  [kernel.kallsyms]  [k] create_empty_buffers
  1,30%  dd  [kernel.kallsyms]  [k] generic_make_request.part.51
  1,09%  dd  [kernel.kallsyms]  [k] __alloc_pages_nodemask
  1,06%  dd  [kernel.kallsyms]  [k] do_generic_file_read.constprop.35
  1,01%  dd  [kernel.kallsyms]  [k] lookup_page_cgroup
  1,00%  dd  [kernel.kallsyms]  [k] add_to_page_cache_locked.part.29
  0,93%  dd  [kernel.kallsyms]  [k] find_get_page
  0,92%  dd  [kernel.kallsyms]  [k] blk_rq_map_sg
  0,92%  dd  [kernel.kallsyms]  [k] radix_tree_insert
  0,89%  dd  [kernel.kallsyms]  [k] alloc_pages_current
  0,88%  dd  [kernel.kallsyms]  [k] mem_cgroup_add_lru_list.part.45
  0,88%  dd  [kernel.kallsyms]  [k] radix_tree_lookup_element

Re: Bad SSD performance with recent kernels

[Shaohua Li]

2012/1/30 Eric Dumazet <eric.dumazet@gmail.com>:
> Le lundi 30 janvier 2012 à 11:17 +0800, Shaohua Li a écrit :
>
>> 1M block size is slow, I guess it's CPU related.
>>
>> And as for the big regression with newer kernel than 2.6.38, please
>> check if idle=poll
>> helps. CPU idle dramatically impacts disk performance and even latest
>> cpuidle governor
>> doesn't help for some CPUs.
>>
>
> No change with idle=poll
I mean the big regression might be related to CPU idle. because I
saw really bad performance with CPU idle with some CPUs.

The 1M block size performance is lower than 128K is not. but I still
guess it's CPU related (like copy). The test result is always like this
IIRC.

Re: Bad SSD performance with recent kernels

[Herbert Poetzl]

On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote:
> 2012/1/30 Wu Fengguang <wfg@linux.intel.com>:
>> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
>>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :


>>>> Note that as long as buffered read(2) is used, it makes almost no
>>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
>>>> the 128kb readahead size will be used underneath to submit read IO.


>>> Hmm...

>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>>> 32768+0 enregistrements lus
>>> 32768+0 enregistrements écrits
>>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s


>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>>> 2048+0 enregistrements lus
>>> 2048+0 enregistrements écrits
>>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s

>> Interesting. Here are my test results:

>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>> 32768+0 records in
>> 32768+0 records out
>> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>> 2048+0 records in
>> 2048+0 records out
>> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s

>> Maybe the /dev/sda performance bug on your machine is sensitive to timing?
> I got similar result:
> 128k: 224M/s
> 1M: 182M/s

> 1M block size is slow, I guess it's CPU related.

> And as for the big regression with newer kernel than 2.6.38,
> please check if idle=poll helps. CPU idle dramatically impacts
> disk performance and even latest cpuidle governor doesn't help
> for some CPUs.

here are the tests with idle=poll and after switching to 128k
(instead of 1M) blocksize (same amount of data transferred)

kernel    ------------ read /dev/sda -------------
          --- noop ---  - deadline -  ---- cfs ---
          [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
--------------------------------------------------
3.2.2      45.82   3.7   44.85   3.6   45.04   3.4
3.2.2i     45.59   2.3   51.78   2.6   46.03   2.2
3.2.2i128 250.24  20.9  252.68  21.3  250.00  21.6

kernel    -- write ---  ------------------read -----------------
          --- noop ---  --- noop ---  - deadline -  ---- cfs ---
          [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
----------------------------------------------------------------
3.2.2     270.95  42.6  162.36   9.9  162.63   9.9  162.65  10.1
3.2.2i    269.10  41.4  170.82   6.6  171.20   6.6  170.91   6.7
3.2.2i128 270.38  67.7  162.35  10.2  163.01  10.3  162.34  10.7

HTH,
Herbert

> Thanks,
> Shaohua

Re: Bad SSD performance with recent kernels

[Shaohua Li]

On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote:
> On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote:
> > 2012/1/30 Wu Fengguang <wfg@linux.intel.com>:
> >> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
> >>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :
> 
> 
> >>>> Note that as long as buffered read(2) is used, it makes almost no
> >>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
> >>>> the 128kb readahead size will be used underneath to submit read IO.
> 
> 
> >>> Hmm...
> 
> >>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> >>> 32768+0 enregistrements lus
> >>> 32768+0 enregistrements écrits
> >>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s
> 
> 
> >>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> >>> 2048+0 enregistrements lus
> >>> 2048+0 enregistrements écrits
> >>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s
> 
> >> Interesting. Here are my test results:
> 
> >> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> >> 32768+0 records in
> >> 32768+0 records out
> >> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
> >> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> >> 2048+0 records in
> >> 2048+0 records out
> >> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s
> 
> >> Maybe the /dev/sda performance bug on your machine is sensitive to timing?
> > I got similar result:
> > 128k: 224M/s
> > 1M: 182M/s
> 
> > 1M block size is slow, I guess it's CPU related.
> 
> > And as for the big regression with newer kernel than 2.6.38,
> > please check if idle=poll helps. CPU idle dramatically impacts
> > disk performance and even latest cpuidle governor doesn't help
> > for some CPUs.
> 
> here are the tests with idle=poll and after switching to 128k
> (instead of 1M) blocksize (same amount of data transferred)
> 
> kernel    ------------ read /dev/sda -------------
>           --- noop ---  - deadline -  ---- cfs ---
>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
> --------------------------------------------------
> 3.2.2      45.82   3.7   44.85   3.6   45.04   3.4
> 3.2.2i     45.59   2.3   51.78   2.6   46.03   2.2
> 3.2.2i128 250.24  20.9  252.68  21.3  250.00  21.6
> 
> kernel    -- write ---  ------------------read -----------------
>           --- noop ---  --- noop ---  - deadline -  ---- cfs ---
>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
> ----------------------------------------------------------------
> 3.2.2     270.95  42.6  162.36   9.9  162.63   9.9  162.65  10.1
> 3.2.2i    269.10  41.4  170.82   6.6  171.20   6.6  170.91   6.7
> 3.2.2i128 270.38  67.7  162.35  10.2  163.01  10.3  162.34  10.7
What's 3.2.2i and 3.2.2i128? does idle=poll help?

Re: Bad SSD performance with recent kernels

[Herbert Poetzl]

On Mon, Jan 30, 2012 at 03:22:38PM +0800, Shaohua Li wrote:
> On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote:
>> On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote:
>>> 2012/1/30 Wu Fengguang <wfg@linux.intel.com>:
>>>> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
>>>>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :

>>>>>> Note that as long as buffered read(2) is used, it makes almost no
>>>>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
>>>>>> the 128kb readahead size will be used underneath to submit read IO.

>>>>> Hmm...

>>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>>>>> 32768+0 enregistrements lus
>>>>> 32768+0 enregistrements écrits
>>>>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s


>>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>>>>> 2048+0 enregistrements lus
>>>>> 2048+0 enregistrements écrits
>>>>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s

>>>> Interesting. Here are my test results:

>>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>>>> 32768+0 records in
>>>> 32768+0 records out
>>>> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
>>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>>>> 2048+0 records in
>>>> 2048+0 records out
>>>> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s

>>>> Maybe the /dev/sda performance bug on your machine is sensitive to timing?
>>> I got similar result:
>>> 128k: 224M/s
>>> 1M: 182M/s

>>> 1M block size is slow, I guess it's CPU related.

>>> And as for the big regression with newer kernel than 2.6.38,
>>> please check if idle=poll helps. CPU idle dramatically impacts
>>> disk performance and even latest cpuidle governor doesn't help
>>> for some CPUs.

>> here are the tests with idle=poll and after switching to 128k
>> (instead of 1M) blocksize (same amount of data transferred)

>> kernel    ------------ read /dev/sda -------------
>>           --- noop ---  - deadline -  ---- cfs ---
>>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
>> --------------------------------------------------
>> 3.2.2      45.82   3.7   44.85   3.6   45.04   3.4
>> 3.2.2i     45.59   2.3   51.78   2.6   46.03   2.2
>> 3.2.2i128 250.24  20.9  252.68  21.3  250.00  21.6

>> kernel    -- write ---  ------------------read -----------------
>>           --- noop ---  --- noop ---  - deadline -  ---- cfs ---
>>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
>> ----------------------------------------------------------------
>> 3.2.2     270.95  42.6  162.36   9.9  162.63   9.9  162.65  10.1
>> 3.2.2i    269.10  41.4  170.82   6.6  171.20   6.6  170.91   6.7
>> 3.2.2i128 270.38  67.7  162.35  10.2  163.01  10.3  162.34  10.7

> What's 3.2.2i and 3.2.2i128? 

3.2.2 ...... kernel with default options (bs=1M)
3.2.2i ..... kernel with idle=poll (bs=1M)
3.2.2i128 .. kernel with idle=poll (bs=128k)

> does idle=poll help?

doesn't look like, at least to me ...

HTC,
Herbert

Re: Bad SSD performance with recent kernels

[Shaohua Li]

On Mon, 2012-01-30 at 08:36 +0100, Herbert Poetzl wrote:
> On Mon, Jan 30, 2012 at 03:22:38PM +0800, Shaohua Li wrote:
> > On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote:
> >> On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote:
> >>> 2012/1/30 Wu Fengguang <wfg@linux.intel.com>:
> >>>> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
> >>>>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :
> 
> >>>>>> Note that as long as buffered read(2) is used, it makes almost no
> >>>>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
> >>>>>> the 128kb readahead size will be used underneath to submit read IO.
> 
> >>>>> Hmm...
> 
> >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> >>>>> 32768+0 enregistrements lus
> >>>>> 32768+0 enregistrements écrits
> >>>>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s
> 
> 
> >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> >>>>> 2048+0 enregistrements lus
> >>>>> 2048+0 enregistrements écrits
> >>>>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s
> 
> >>>> Interesting. Here are my test results:
> 
> >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
> >>>> 32768+0 records in
> >>>> 32768+0 records out
> >>>> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
> >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
> >>>> 2048+0 records in
> >>>> 2048+0 records out
> >>>> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s
> 
> >>>> Maybe the /dev/sda performance bug on your machine is sensitive to timing?
> >>> I got similar result:
> >>> 128k: 224M/s
> >>> 1M: 182M/s
> 
> >>> 1M block size is slow, I guess it's CPU related.
> 
> >>> And as for the big regression with newer kernel than 2.6.38,
> >>> please check if idle=poll helps. CPU idle dramatically impacts
> >>> disk performance and even latest cpuidle governor doesn't help
> >>> for some CPUs.
> 
> >> here are the tests with idle=poll and after switching to 128k
> >> (instead of 1M) blocksize (same amount of data transferred)
> 
> >> kernel    ------------ read /dev/sda -------------
> >>           --- noop ---  - deadline -  ---- cfs ---
> >>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
> >> --------------------------------------------------
> >> 3.2.2      45.82   3.7   44.85   3.6   45.04   3.4
> >> 3.2.2i     45.59   2.3   51.78   2.6   46.03   2.2
> >> 3.2.2i128 250.24  20.9  252.68  21.3  250.00  21.6
> 
> >> kernel    -- write ---  ------------------read -----------------
> >>           --- noop ---  --- noop ---  - deadline -  ---- cfs ---
> >>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
> >> ----------------------------------------------------------------
> >> 3.2.2     270.95  42.6  162.36   9.9  162.63   9.9  162.65  10.1
> >> 3.2.2i    269.10  41.4  170.82   6.6  171.20   6.6  170.91   6.7
> >> 3.2.2i128 270.38  67.7  162.35  10.2  163.01  10.3  162.34  10.7
> 
> > What's 3.2.2i and 3.2.2i128? 
> 
> 3.2.2 ...... kernel with default options (bs=1M)
> 3.2.2i ..... kernel with idle=poll (bs=1M)
> 3.2.2i128 .. kernel with idle=poll (bs=128k)
> 
> > does idle=poll help?
> 
> doesn't look like, at least to me ...
what's your /sys/block/sdx/queue/max_sectors_kb? if you make it smaller,
does the performance increase? In my system, a smaller max_sectors_kb
makes bs=2M and bs=128k have similar performance, which makes me think
it's CPU doesn't catch up quickly after a request finishes.

Re: Bad SSD performance with recent kernels

[Shaohua Li]

2012/1/30 Shaohua Li <shaohua.li@intel.com>:
> On Mon, 2012-01-30 at 08:36 +0100, Herbert Poetzl wrote:
>> On Mon, Jan 30, 2012 at 03:22:38PM +0800, Shaohua Li wrote:
>> > On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote:
>> >> On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote:
>> >>> 2012/1/30 Wu Fengguang <wfg@linux.intel.com>:
>> >>>> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote:
>> >>>>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit :
>>
>> >>>>>> Note that as long as buffered read(2) is used, it makes almost no
>> >>>>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB":
>> >>>>>> the 128kb readahead size will be used underneath to submit read IO.
>>
>> >>>>> Hmm...
>>
>> >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>> >>>>> 32768+0 enregistrements lus
>> >>>>> 32768+0 enregistrements écrits
>> >>>>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s
>>
>>
>> >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>> >>>>> 2048+0 enregistrements lus
>> >>>>> 2048+0 enregistrements écrits
>> >>>>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s
>>
>> >>>> Interesting. Here are my test results:
>>
>> >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768
>> >>>> 32768+0 records in
>> >>>> 32768+0 records out
>> >>>> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s
>> >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048
>> >>>> 2048+0 records in
>> >>>> 2048+0 records out
>> >>>> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s
>>
>> >>>> Maybe the /dev/sda performance bug on your machine is sensitive to timing?
>> >>> I got similar result:
>> >>> 128k: 224M/s
>> >>> 1M: 182M/s
>>
>> >>> 1M block size is slow, I guess it's CPU related.
>>
>> >>> And as for the big regression with newer kernel than 2.6.38,
>> >>> please check if idle=poll helps. CPU idle dramatically impacts
>> >>> disk performance and even latest cpuidle governor doesn't help
>> >>> for some CPUs.
>>
>> >> here are the tests with idle=poll and after switching to 128k
>> >> (instead of 1M) blocksize (same amount of data transferred)
>>
>> >> kernel    ------------ read /dev/sda -------------
>> >>           --- noop ---  - deadline -  ---- cfs ---
>> >>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
>> >> --------------------------------------------------
>> >> 3.2.2      45.82   3.7   44.85   3.6   45.04   3.4
>> >> 3.2.2i     45.59   2.3   51.78   2.6   46.03   2.2
>> >> 3.2.2i128 250.24  20.9  252.68  21.3  250.00  21.6
>>
>> >> kernel    -- write ---  ------------------read -----------------
>> >>           --- noop ---  --- noop ---  - deadline -  ---- cfs ---
>> >>           [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
>> >> ----------------------------------------------------------------
>> >> 3.2.2     270.95  42.6  162.36   9.9  162.63   9.9  162.65  10.1
>> >> 3.2.2i    269.10  41.4  170.82   6.6  171.20   6.6  170.91   6.7
>> >> 3.2.2i128 270.38  67.7  162.35  10.2  163.01  10.3  162.34  10.7
>>
>> > What's 3.2.2i and 3.2.2i128?
>>
>> 3.2.2 ...... kernel with default options (bs=1M)
>> 3.2.2i ..... kernel with idle=poll (bs=1M)
>> 3.2.2i128 .. kernel with idle=poll (bs=128k)
>>
>> > does idle=poll help?
>>
>> doesn't look like, at least to me ...
> what's your /sys/block/sdx/queue/max_sectors_kb? if you make it smaller,
> does the performance increase? In my system, a smaller max_sectors_kb
> makes bs=2M and bs=128k have similar performance, which makes me think
> it's CPU doesn't catch up quickly after a request finishes.
Looks the 2.6.39 block plug introduces some latency here. deleting
blk_start_plug/blk_finish_plug in generic_file_aio_read seems
workaround
the issue. The plug seems not good for sequential IO, because readahead
code already has plug and has fine grained control.
On the other hand, ondemand_readahead seems not handle the case
that req_size is big well.

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Sun, Jan 29, 2012 at 09:15:43PM +0100, Herbert Poetzl wrote:
> On Mon, Jan 30, 2012 at 12:10:58AM +0800, Wu Fengguang wrote:

> > Maybe the /dev/sda performance bug on your machine is sensitive to timing?
> 
> here are some more confusing results from tests with dd and bonnie++, 
> this time I focused on partition vs. loop vs. linear dm (of same partition)
> 
> kernel	  -------------- read --------------  -- write ---  all
> 	  -------- dd --------  -------- bonnie++ --------------
> 	  [MB/s]  real    %CPU  [MB/s]  %CPU  [MB/s]  %CPU  %CPU
> direct
> 2.6.38.8  262.91   81.90  28.7	 72.30   6.0  248.53  52.0  15.9
> 2.6.39.4   36.09  595.17   3.1	 70.62   6.0  250.25  53.0  16.3
> 3.0.18     50.47  425.65   4.1	 70.00   5.0  251.70  44.0  13.9
> 3.1.10     27.28  787.32   2.0	 75.65   5.0  251.96  45.0  13.3
> 3.2.2      27.11  792.28   2.0	 76.89   6.0  250.38  44.0  13.3
> 
> loop
> 2.6.38.8  242.89   88.50  21.5	246.58  15.0  240.92  53.0  14.4
> 2.6.39.4  241.06   89.19  21.5	238.51  15.0  257.59  57.0  14.8
> 3.0.18	  261.44   82.23  18.8	256.66  15.0  255.17  48.0  12.6
> 3.1.10	  253.93   84.64  18.1	107.66   7.0  156.51  28.0  10.6
> 3.2.2	  262.58   81.82  19.8	110.54   7.0  212.01  40.0  11.6
> 
> linear
> 2.6.38.8  262.57   82.00  36.8	 72.46   6.0  243.25  53.0  16.5
> 2.6.39.4   25.45  843.93   2.3	 70.70   6.0  248.05  54.0  16.6
> 3.0.18	   55.45  387.43   5.6	 69.72   6.0  249.42  45.0  14.3
> 3.1.10	   36.62  586.50   3.3	 74.74   6.0  249.99  46.0  13.4
> 3.2.2	   28.28  759.26   2.3	 74.20   6.0  248.73  46.0  13.6
> 
> 
> it seems that dd performance when using a loop device is unaffected
> and even improves with the kernel version, the filesystem performance
> OTOH degrades after 3.1 ...
> 
> in general, filesystem read performance is bad on everything but
> a loop device ... judging from the results I'd conclude that there
> are at least two different issues 
> 
> tests and test results are attached and can be found here:
> http://vserver.13thfloor.at/Stuff/SSD/
> 
> I plan to do some more tests on the filesystem with -b and -D
> tonight, please let me know if you want to see specific output
> and/or have any tests I should run with each kernel ...

I agree with Shaohua that there may be timing/plug issues. There
happen to be some plug patches and (maybe correlated) big performance
drop between 2.6.38 and 2.6.39. The obvious way to move forward is to
get some blktrace data on simple dd + new buggy kernel and let's check
what's exactly going on.

# start a background dd read
blktrace /dev/sda -w 10
blkparse -t sda

Thanks,
Fengguang

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le lundi 30 janvier 2012 à 19:18 +0800, Wu Fengguang a écrit :

> I agree with Shaohua that there may be timing/plug issues. There
> happen to be some plug patches and (maybe correlated) big performance
> drop between 2.6.38 and 2.6.39. The obvious way to move forward is to
> get some blktrace data on simple dd + new buggy kernel and let's check
> what's exactly going on.
> 
> # start a background dd read
> blktrace /dev/sda -w 10
> blkparse -t sda
> 

(Another machine here at office, hence /dev/sdb)

echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=2M
count=2048
2048+0 enregistrements lus
2048+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 26,0278 s, 165 MB/s

# blktrace /dev/sdb -w 10
=== sdb ===
  CPU  0:                44766 events,     2099 KiB data
  CPU  1:               828895 events,    38855 KiB data
  Total:                873661 events (dropped 0),    40953 KiB data

blkparse result is huge (55Mo)

# blkparse -t sdb | tail -n 1000 

  8,16   1   770475     9.989714963  4580  Q   R 4379392 + 8 [dd]
  8,16   1   770476     9.989721470  4580  G   R 4379392 + 8 [dd]
  8,16   1   770477     9.989723546  4580  P   N [dd]
  8,16   1   770478     9.989726994  4580  Q   R 4379400 + 8 [dd]
  8,16   1   770479     9.989728858  4580  M   R 4379400 + 8 [dd]
  8,16   1   770480     9.989731893  4580  Q   R 4379408 + 8 [dd]
  8,16   1   770481     9.989732467  4580  M   R 4379408 + 8 [dd]
  8,16   1   770482     9.989734574  4580  Q   R 4379416 + 8 [dd]
  8,16   1   770483     9.989735108  4580  M   R 4379416 + 8 [dd]
  8,16   1   770484     9.989737220  4580  Q   R 4379424 + 8 [dd]
  8,16   1   770485     9.989737721  4580  M   R 4379424 + 8 [dd]
  8,16   1   770486     9.989739953  4580  Q   R 4379432 + 8 [dd]
  8,16   1   770487     9.989740450  4580  M   R 4379432 + 8 [dd]
  8,16   1   770488     9.989742577  4580  Q   R 4379440 + 8 [dd]
  8,16   1   770489     9.989743078  4580  M   R 4379440 + 8 [dd]
  8,16   1   770490     9.989745241  4580  Q   R 4379448 + 8 [dd]
  8,16   1   770491     9.989745742  4580  M   R 4379448 + 8 [dd]
  8,16   1   770492     9.989747953  4580  Q   R 4379456 + 8 [dd]
  8,16   1   770493     9.989748485  4580  M   R 4379456 + 8 [dd]
  8,16   1   770494     9.989750726  4580  Q   R 4379464 + 8 [dd]
  8,16   1   770495     9.989751225  4580  M   R 4379464 + 8 [dd]
  8,16   1   770496     9.989753390  4580  Q   R 4379472 + 8 [dd]
  8,16   1   770497     9.989753891  4580  M   R 4379472 + 8 [dd]
  8,16   1   770498     9.989755928  4580  Q   R 4379480 + 8 [dd]
  8,16   1   770499     9.989756429  4580  M   R 4379480 + 8 [dd]
  8,16   1   770500     9.989758470  4580  Q   R 4379488 + 8 [dd]
  8,16   1   770501     9.989758975  4580  M   R 4379488 + 8 [dd]
  8,16   1   770502     9.989765725  4580  Q   R 4379496 + 8 [dd]
  8,16   1   770503     9.989766222  4580  M   R 4379496 + 8 [dd]
  8,16   1   770504     9.989768362  4580  Q   R 4379504 + 8 [dd]
  8,16   1   770505     9.989768864  4580  M   R 4379504 + 8 [dd]
  8,16   1   770506     9.989771157  4580  Q   R 4379512 + 8 [dd]
  8,16   1   770507     9.989771662  4580  M   R 4379512 + 8 [dd]
  8,16   1   770508     9.989775151  4580  Q   R 4379520 + 8 [dd]
  8,16   1   770509     9.989775649  4580  M   R 4379520 + 8 [dd]
  8,16   1   770510     9.989781743  4580  Q   R 4379528 + 8 [dd]
  8,16   1   770511     9.989782248  4580  M   R 4379528 + 8 [dd]
  8,16   1   770512     9.989784532  4580  Q   R 4379536 + 8 [dd]
  8,16   1   770513     9.989785033  4580  M   R 4379536 + 8 [dd]
  8,16   1   770514     9.989787291  4580  Q   R 4379544 + 8 [dd]
  8,16   1   770515     9.989787792  4580  M   R 4379544 + 8 [dd]
  8,16   1   770516     9.989790041  4580  Q   R 4379552 + 8 [dd]
  8,16   1   770517     9.989790540  4580  M   R 4379552 + 8 [dd]
  8,16   1   770518     9.989792607  4580  Q   R 4379560 + 8 [dd]
  8,16   1   770519     9.989793104  4580  M   R 4379560 + 8 [dd]
  8,16   1   770520     9.989795693  4580  Q   R 4379568 + 8 [dd]
  8,16   1   770521     9.989796191  4580  M   R 4379568 + 8 [dd]
  8,16   1   770522     9.989798349  4580  Q   R 4379576 + 8 [dd]
  8,16   1   770523     9.989798847  4580  M   R 4379576 + 8 [dd]
  8,16   1   770524     9.989800877  4580  Q   R 4379584 + 8 [dd]
  8,16   1   770525     9.989801581  4580  M   R 4379584 + 8 [dd]
  8,16   1   770526     9.989803693  4580  Q   R 4379592 + 8 [dd]
  8,16   1   770527     9.989804221  4580  M   R 4379592 + 8 [dd]
  8,16   1   770528     9.989806399  4580  Q   R 4379600 + 8 [dd]
  8,16   1   770529     9.989806901  4580  M   R 4379600 + 8 [dd]
  8,16   1   770530     9.989812437  4580  Q   R 4379608 + 8 [dd]
  8,16   1   770531     9.989812938  4580  M   R 4379608 + 8 [dd]
  8,16   1   770532     9.989814981  4580  Q   R 4379616 + 8 [dd]
  8,16   1   770533     9.989815482  4580  M   R 4379616 + 8 [dd]
  8,16   1   770534     9.989818053  4580  Q   R 4379624 + 8 [dd]
  8,16   1   770535     9.989818551  4580  M   R 4379624 + 8 [dd]
  8,16   1   770536     9.989820795  4580  Q   R 4379632 + 8 [dd]
  8,16   1   770537     9.989821296  4580  M   R 4379632 + 8 [dd]
  8,16   1   770538     9.989823436  4580  Q   R 4379640 + 8 [dd]
  8,16   1   770539     9.989823937  4580  M   R 4379640 + 8 [dd]
  8,16   1   770540     9.989929502  4580  Q   R 4379648 + 8 [dd]
  8,16   1   770541     9.989930247  4580  M   R 4379648 + 8 [dd]
  8,16   1   770542     9.989933189  4580  Q   R 4379656 + 8 [dd]
  8,16   1   770543     9.989933705  4580  M   R 4379656 + 8 [dd]
  8,16   1   770544     9.989935909  4580  Q   R 4379664 + 8 [dd]
  8,16   1   770545     9.989936411  4580  M   R 4379664 + 8 [dd]
  8,16   1   770546     9.989938572  4580  Q   R 4379672 + 8 [dd]
  8,16   1   770547     9.989939072  4580  M   R 4379672 + 8 [dd]
  8,16   1   770548     9.989941272  4580  Q   R 4379680 + 8 [dd]
  8,16   1   770549     9.989941770  4580  M   R 4379680 + 8 [dd]
  8,16   1   770550     9.989943952  4580  Q   R 4379688 + 8 [dd]
  8,16   1   770551     9.989944450  4580  M   R 4379688 + 8 [dd]
  8,16   1   770552     9.989946991  4580  Q   R 4379696 + 8 [dd]
  8,16   1   770553     9.989947492  4580  M   R 4379696 + 8 [dd]
  8,16   1   770554     9.989949671  4580  Q   R 4379704 + 8 [dd]
  8,16   1   770555     9.989950172  4580  M   R 4379704 + 8 [dd]
  8,16   1   770556     9.989952330  4580  Q   R 4379712 + 8 [dd]
  8,16   1   770557     9.989952832  4580  M   R 4379712 + 8 [dd]
  8,16   1   770558     9.989958393  4580  Q   R 4379720 + 8 [dd]
  8,16   1   770559     9.989958891  4580  M   R 4379720 + 8 [dd]
  8,16   1   770560     9.989961097  4580  Q   R 4379728 + 8 [dd]
  8,16   1   770561     9.989961598  4580  M   R 4379728 + 8 [dd]
  8,16   1   770562     9.989963825  4580  Q   R 4379736 + 8 [dd]
  8,16   1   770563     9.989964331  4580  M   R 4379736 + 8 [dd]
  8,16   1   770564     9.989966623  4580  Q   R 4379744 + 8 [dd]
  8,16   1   770565     9.989967122  4580  M   R 4379744 + 8 [dd]
  8,16   1   770566     9.989969303  4580  Q   R 4379752 + 8 [dd]
  8,16   1   770567     9.989969800  4580  M   R 4379752 + 8 [dd]
  8,16   1   770568     9.989971886  4580  Q   R 4379760 + 8 [dd]
  8,16   1   770569     9.989972388  4580  M   R 4379760 + 8 [dd]
  8,16   1   770570     9.989974789  4580  Q   R 4379768 + 8 [dd]
  8,16   1   770571     9.989975290  4580  M   R 4379768 + 8 [dd]
  8,16   1   770572     9.989977480  4580  Q   R 4379776 + 8 [dd]
  8,16   1   770573     9.989977981  4580  M   R 4379776 + 8 [dd]
  8,16   1   770574     9.989981251  4580  Q   R 4379784 + 8 [dd]
  8,16   1   770575     9.989981749  4580  M   R 4379784 + 8 [dd]
  8,16   1   770576     9.989984018  4580  Q   R 4379792 + 8 [dd]
  8,16   1   770577     9.989984519  4580  M   R 4379792 + 8 [dd]
  8,16   1   770578     9.989986711  4580  Q   R 4379800 + 8 [dd]
  8,16   1   770579     9.989987222  4580  M   R 4379800 + 8 [dd]
  8,16   1   770580     9.989989414  4580  Q   R 4379808 + 8 [dd]
  8,16   1   770581     9.989989915  4580  M   R 4379808 + 8 [dd]
  8,16   1   770582     9.989992107  4580  Q   R 4379816 + 8 [dd]
  8,16   1   770583     9.989992617  4580  M   R 4379816 + 8 [dd]
  8,16   1   770584     9.989994776  4580  Q   R 4379824 + 8 [dd]
  8,16   1   770585     9.989995276  4580  M   R 4379824 + 8 [dd]
  8,16   1   770586     9.990000570  4580  Q   R 4379832 + 8 [dd]
  8,16   1   770587     9.990001076  4580  M   R 4379832 + 8 [dd]
  8,16   1   770588     9.990003264  4580  Q   R 4379840 + 8 [dd]
  8,16   1   770589     9.990003785  4580  M   R 4379840 + 8 [dd]
  8,16   1   770590     9.990005849  4580  Q   R 4379848 + 8 [dd]
  8,16   1   770591     9.990006354  4580  M   R 4379848 + 8 [dd]
  8,16   1   770592     9.990008472  4580  Q   R 4379856 + 8 [dd]
  8,16   1   770593     9.990008986  4580  M   R 4379856 + 8 [dd]
  8,16   1   770594     9.990011382  4580  Q   R 4379864 + 8 [dd]
  8,16   1   770595     9.990011884  4580  M   R 4379864 + 8 [dd]
  8,16   1   770596     9.990014066  4580  Q   R 4379872 + 8 [dd]
  8,16   1   770597     9.990014568  4580  M   R 4379872 + 8 [dd]
  8,16   1   770598     9.990016740  4580  Q   R 4379880 + 8 [dd]
  8,16   1   770599     9.990017237  4580  M   R 4379880 + 8 [dd]
  8,16   1   770600     9.990019491  4580  Q   R 4379888 + 8 [dd]
  8,16   1   770601     9.990020001  4580  M   R 4379888 + 8 [dd]
  8,16   1   770602     9.990022222  4580  Q   R 4379896 + 8 [dd]
  8,16   1   770603     9.990022723  4580  M   R 4379896 + 8 [dd]
  8,16   1   770604     9.990024981  4580  I   R 4379392 + 512 (  303511) [dd]
  8,16   1        0     9.990027734     0  m   N cfq4580S / insert_request
  8,16   1   770605     9.990031216  4580  U   N [dd] 1
  8,16   1        0     9.990033792     0  m   N cfq4580S / dispatch_insert
  8,16   1        0     9.990037429     0  m   N cfq4580S / dispatched a request
  8,16   1        0     9.990039146     0  m   N cfq4580S / activate rq, drv=1
  8,16   1   770606     9.990039807  4580  D   R 4379392 + 512 (   14826) [dd]
  8,16   1   770607     9.991083069     0  C   R 4379392 + 512 ( 1043262) [0]
  8,16   1        0     9.991155388     0  m   N cfq4580S / complete rqnoidle 0
  8,16   1        0     9.991158292     0  m   N cfq schedule dispatch
  8,16   1   770608     9.991321889  4580  Q   R 4379904 + 8 [dd]
  8,16   1   770609     9.991328302  4580  G   R 4379904 + 8 [dd]
  8,16   1   770610     9.991330287  4580  P   N [dd]
  8,16   1   770611     9.991333624  4580  Q   R 4379912 + 8 [dd]
  8,16   1   770612     9.991335576  4580  M   R 4379912 + 8 [dd]
  8,16   1   770613     9.991338586  4580  Q   R 4379920 + 8 [dd]
  8,16   1   770614     9.991339095  4580  M   R 4379920 + 8 [dd]
  8,16   1   770615     9.991341209  4580  Q   R 4379928 + 8 [dd]
  8,16   1   770616     9.991341909  4580  M   R 4379928 + 8 [dd]
  8,16   1   770617     9.991344208  4580  Q   R 4379936 + 8 [dd]
  8,16   1   770618     9.991344707  4580  M   R 4379936 + 8 [dd]
  8,16   1   770619     9.991351891  4580  Q   R 4379944 + 8 [dd]
  8,16   1   770620     9.991352393  4580  M   R 4379944 + 8 [dd]
  8,16   1   770621     9.991354635  4580  Q   R 4379952 + 8 [dd]
  8,16   1   770622     9.991355134  4580  M   R 4379952 + 8 [dd]
  8,16   1   770623     9.991357304  4580  Q   R 4379960 + 8 [dd]
  8,16   1   770624     9.991357809  4580  M   R 4379960 + 8 [dd]
  8,16   1   770625     9.991359960  4580  Q   R 4379968 + 8 [dd]
  8,16   1   770626     9.991360458  4580  M   R 4379968 + 8 [dd]
  8,16   1   770627     9.991362540  4580  Q   R 4379976 + 8 [dd]
  8,16   1   770628     9.991363044  4580  M   R 4379976 + 8 [dd]
  8,16   1   770629     9.991366526  4580  Q   R 4379984 + 8 [dd]
  8,16   1   770630     9.991367025  4580  M   R 4379984 + 8 [dd]
  8,16   1   770631     9.991369183  4580  Q   R 4379992 + 8 [dd]
  8,16   1   770632     9.991369681  4580  M   R 4379992 + 8 [dd]
  8,16   1   770633     9.991371837  4580  Q   R 4380000 + 8 [dd]
  8,16   1   770634     9.991372336  4580  M   R 4380000 + 8 [dd]
  8,16   1   770635     9.991374518  4580  Q   R 4380008 + 8 [dd]
  8,16   1   770636     9.991375031  4580  M   R 4380008 + 8 [dd]
  8,16   1   770637     9.991377249  4580  Q   R 4380016 + 8 [dd]
  8,16   1   770638     9.991377748  4580  M   R 4380016 + 8 [dd]
  8,16   1   770639     9.991379891  4580  Q   R 4380024 + 8 [dd]
  8,16   1   770640     9.991380390  4580  M   R 4380024 + 8 [dd]
  8,16   1   770641     9.991382585  4580  Q   R 4380032 + 8 [dd]
  8,16   1   770642     9.991383084  4580  M   R 4380032 + 8 [dd]
  8,16   1   770643     9.991385207  4580  Q   R 4380040 + 8 [dd]
  8,16   1   770644     9.991385704  4580  M   R 4380040 + 8 [dd]
  8,16   1   770645     9.991387897  4580  Q   R 4380048 + 8 [dd]
  8,16   1   770646     9.991388396  4580  M   R 4380048 + 8 [dd]
  8,16   1   770647     9.991393804  4580  Q   R 4380056 + 8 [dd]
  8,16   1   770648     9.991394303  4580  M   R 4380056 + 8 [dd]
  8,16   1   770649     9.991396440  4580  Q   R 4380064 + 8 [dd]
  8,16   1   770650     9.991396938  4580  M   R 4380064 + 8 [dd]
  8,16   1   770651     9.991398996  4580  Q   R 4380072 + 8 [dd]
  8,16   1   770652     9.991399500  4580  M   R 4380072 + 8 [dd]
  8,16   1   770653     9.991401608  4580  Q   R 4380080 + 8 [dd]
  8,16   1   770654     9.991402107  4580  M   R 4380080 + 8 [dd]
  8,16   1   770655     9.991404535  4580  Q   R 4380088 + 8 [dd]
  8,16   1   770656     9.991405034  4580  M   R 4380088 + 8 [dd]
  8,16   1   770657     9.991407171  4580  Q   R 4380096 + 8 [dd]
  8,16   1   770658     9.991407671  4580  M   R 4380096 + 8 [dd]
  8,16   1   770659     9.991409829  4580  Q   R 4380104 + 8 [dd]
  8,16   1   770660     9.991410339  4580  M   R 4380104 + 8 [dd]
  8,16   1   770661     9.991416346  4580  Q   R 4380112 + 8 [dd]
  8,16   1   770662     9.991416853  4580  M   R 4380112 + 8 [dd]
  8,16   1   770663     9.991419016  4580  Q   R 4380120 + 8 [dd]
  8,16   1   770664     9.991419515  4580  M   R 4380120 + 8 [dd]
  8,16   1   770665     9.991421984  4580  Q   R 4380128 + 8 [dd]
  8,16   1   770666     9.991422483  4580  M   R 4380128 + 8 [dd]
  8,16   1   770667     9.991424780  4580  Q   R 4380136 + 8 [dd]
  8,16   1   770668     9.991425284  4580  M   R 4380136 + 8 [dd]
  8,16   1   770669     9.991427479  4580  Q   R 4380144 + 8 [dd]
  8,16   1   770670     9.991427978  4580  M   R 4380144 + 8 [dd]
  8,16   1   770671     9.991430076  4580  Q   R 4380152 + 8 [dd]
  8,16   1   770672     9.991430575  4580  M   R 4380152 + 8 [dd]
  8,16   1   770673     9.991536122  4580  Q   R 4380160 + 8 [dd]
  8,16   1   770674     9.991536827  4580  M   R 4380160 + 8 [dd]
  8,16   1   770675     9.991543663  4580  Q   R 4380168 + 8 [dd]
  8,16   1   770676     9.991544181  4580  M   R 4380168 + 8 [dd]
  8,16   1   770677     9.991546458  4580  Q   R 4380176 + 8 [dd]
  8,16   1   770678     9.991546957  4580  M   R 4380176 + 8 [dd]
  8,16   1   770679     9.991549153  4580  Q   R 4380184 + 8 [dd]
  8,16   1   770680     9.991549652  4580  M   R 4380184 + 8 [dd]
  8,16   1   770681     9.991551902  4580  Q   R 4380192 + 8 [dd]
  8,16   1   770682     9.991552400  4580  M   R 4380192 + 8 [dd]
  8,16   1   770683     9.991554621  4580  Q   R 4380200 + 8 [dd]
  8,16   1   770684     9.991555113  4580  M   R 4380200 + 8 [dd]
  8,16   1   770685     9.991557325  4580  Q   R 4380208 + 8 [dd]
  8,16   1   770686     9.991557824  4580  M   R 4380208 + 8 [dd]
  8,16   1   770687     9.991560192  4580  Q   R 4380216 + 8 [dd]
  8,16   1   770688     9.991560732  4580  M   R 4380216 + 8 [dd]
  8,16   1   770689     9.991562806  4580  Q   R 4380224 + 8 [dd]
  8,16   1   770690     9.991563305  4580  M   R 4380224 + 8 [dd]
  8,16   1   770691     9.991565370  4580  Q   R 4380232 + 8 [dd]
  8,16   1   770692     9.991565874  4580  M   R 4380232 + 8 [dd]
  8,16   1   770693     9.991567969  4580  Q   R 4380240 + 8 [dd]
  8,16   1   770694     9.991568468  4580  M   R 4380240 + 8 [dd]
  8,16   1   770695     9.991570671  4580  Q   R 4380248 + 8 [dd]
  8,16   1   770696     9.991571170  4580  M   R 4380248 + 8 [dd]
  8,16   1   770697     9.991573285  4580  Q   R 4380256 + 8 [dd]
  8,16   1   770698     9.991573783  4580  M   R 4380256 + 8 [dd]
  8,16   1   770699     9.991575855  4580  Q   R 4380264 + 8 [dd]
  8,16   1   770700     9.991576358  4580  M   R 4380264 + 8 [dd]
  8,16   1   770701     9.991578551  4580  Q   R 4380272 + 8 [dd]
  8,16   1   770702     9.991579050  4580  M   R 4380272 + 8 [dd]
  8,16   1   770703     9.991584418  4580  Q   R 4380280 + 8 [dd]
  8,16   1   770704     9.991584916  4580  M   R 4380280 + 8 [dd]
  8,16   1   770705     9.991587004  4580  Q   R 4380288 + 8 [dd]
  8,16   1   770706     9.991587503  4580  M   R 4380288 + 8 [dd]
  8,16   1   770707     9.991589615  4580  Q   R 4380296 + 8 [dd]
  8,16   1   770708     9.991590128  4580  M   R 4380296 + 8 [dd]
  8,16   1   770709     9.991592390  4580  Q   R 4380304 + 8 [dd]
  8,16   1   770710     9.991592907  4580  M   R 4380304 + 8 [dd]
  8,16   1   770711     9.991594957  4580  Q   R 4380312 + 8 [dd]
  8,16   1   770712     9.991595456  4580  M   R 4380312 + 8 [dd]
  8,16   1   770713     9.991597599  4580  Q   R 4380320 + 8 [dd]
  8,16   1   770714     9.991598098  4580  M   R 4380320 + 8 [dd]
  8,16   1   770715     9.991600165  4580  Q   R 4380328 + 8 [dd]
  8,16   1   770716     9.991600669  4580  M   R 4380328 + 8 [dd]
  8,16   1   770717     9.991602915  4580  Q   R 4380336 + 8 [dd]
  8,16   1   770718     9.991603413  4580  M   R 4380336 + 8 [dd]
  8,16   1   770719     9.991605563  4580  Q   R 4380344 + 8 [dd]
  8,16   1   770720     9.991606061  4580  M   R 4380344 + 8 [dd]
  8,16   1   770721     9.991608110  4580  Q   R 4380352 + 8 [dd]
  8,16   1   770722     9.991608609  4580  M   R 4380352 + 8 [dd]
  8,16   1   770723     9.991610674  4580  Q   R 4380360 + 8 [dd]
  8,16   1   770724     9.991611178  4580  M   R 4380360 + 8 [dd]
  8,16   1   770725     9.991613249  4580  Q   R 4380368 + 8 [dd]
  8,16   1   770726     9.991613748  4580  M   R 4380368 + 8 [dd]
  8,16   1   770727     9.991616309  4580  Q   R 4380376 + 8 [dd]
  8,16   1   770728     9.991616808  4580  M   R 4380376 + 8 [dd]
  8,16   1   770729     9.991619009  4580  Q   R 4380384 + 8 [dd]
  8,16   1   770730     9.991619507  4580  M   R 4380384 + 8 [dd]
  8,16   1   770731     9.991624896  4580  Q   R 4380392 + 8 [dd]
  8,16   1   770732     9.991625401  4580  M   R 4380392 + 8 [dd]
  8,16   1   770733     9.991627620  4580  Q   R 4380400 + 8 [dd]
  8,16   1   770734     9.991628119  4580  M   R 4380400 + 8 [dd]
  8,16   1   770735     9.991630192  4580  Q   R 4380408 + 8 [dd]
  8,16   1   770736     9.991630690  4580  M   R 4380408 + 8 [dd]
  8,16   1   770737     9.991633001  4580  I   R 4379904 + 512 (  304699) [dd]
  8,16   1        0     9.991635738     0  m   N cfq4580S / insert_request
  8,16   1   770738     9.991638988  4580  U   N [dd] 1
  8,16   1        0     9.991641665     0  m   N cfq4580S / dispatch_insert
  8,16   1        0     9.991645149     0  m   N cfq4580S / dispatched a request
  8,16   1        0     9.991646825     0  m   N cfq4580S / activate rq, drv=1
  8,16   1   770739     9.991647434  4580  D   R 4379904 + 512 (   14433) [dd]
  8,16   1   770740     9.992693317     0  C   R 4379904 + 512 ( 1045883) [0]
  8,16   1        0     9.992766012     0  m   N cfq4580S / complete rqnoidle 0
  8,16   1        0     9.992768895     0  m   N cfq schedule dispatch
  8,16   1   770741     9.992932816  4580  Q   R 4380416 + 8 [dd]
  8,16   1   770742     9.992939152  4580  G   R 4380416 + 8 [dd]
  8,16   1   770743     9.992941241  4580  P   N [dd]
  8,16   1   770744     9.992944673  4580  Q   R 4380424 + 8 [dd]
  8,16   1   770745     9.992946529  4580  M   R 4380424 + 8 [dd]
  8,16   1   770746     9.992949525  4580  Q   R 4380432 + 8 [dd]
  8,16   1   770747     9.992950035  4580  M   R 4380432 + 8 [dd]
  8,16   1   770748     9.992952218  4580  Q   R 4380440 + 8 [dd]
  8,16   1   770749     9.992952765  4580  M   R 4380440 + 8 [dd]
  8,16   1   770750     9.992954867  4580  Q   R 4380448 + 8 [dd]
  8,16   1   770751     9.992955366  4580  M   R 4380448 + 8 [dd]
  8,16   1   770752     9.992957451  4580  Q   R 4380456 + 8 [dd]
  8,16   1   770753     9.992957956  4580  M   R 4380456 + 8 [dd]
  8,16   1   770754     9.992960073  4580  Q   R 4380464 + 8 [dd]
  8,16   1   770755     9.992960572  4580  M   R 4380464 + 8 [dd]
  8,16   1   770756     9.992962600  4580  Q   R 4380472 + 8 [dd]
  8,16   1   770757     9.992963104  4580  M   R 4380472 + 8 [dd]
  8,16   1   770758     9.992965247  4580  Q   R 4380480 + 8 [dd]
  8,16   1   770759     9.992965753  4580  M   R 4380480 + 8 [dd]
  8,16   1   770760     9.992967787  4580  Q   R 4380488 + 8 [dd]
  8,16   1   770761     9.992968284  4580  M   R 4380488 + 8 [dd]
  8,16   1   770762     9.992970602  4580  Q   R 4380496 + 8 [dd]
  8,16   1   770763     9.992971101  4580  M   R 4380496 + 8 [dd]
  8,16   1   770764     9.992978036  4580  Q   R 4380504 + 8 [dd]
  8,16   1   770765     9.992978575  4580  M   R 4380504 + 8 [dd]
  8,16   1   770766     9.992980800  4580  Q   R 4380512 + 8 [dd]
  8,16   1   770767     9.992981312  4580  M   R 4380512 + 8 [dd]
  8,16   1   770768     9.992983463  4580  Q   R 4380520 + 8 [dd]
  8,16   1   770769     9.992983976  4580  M   R 4380520 + 8 [dd]
  8,16   1   770770     9.992986141  4580  Q   R 4380528 + 8 [dd]
  8,16   1   770771     9.992986652  4580  M   R 4380528 + 8 [dd]
  8,16   1   770772     9.992988835  4580  Q   R 4380536 + 8 [dd]
  8,16   1   770773     9.992989338  4580  M   R 4380536 + 8 [dd]
  8,16   1   770774     9.992991462  4580  Q   R 4380544 + 8 [dd]
  8,16   1   770775     9.992991961  4580  M   R 4380544 + 8 [dd]
  8,16   1   770776     9.992994020  4580  Q   R 4380552 + 8 [dd]
  8,16   1   770777     9.992994519  4580  M   R 4380552 + 8 [dd]
  8,16   1   770778     9.992996632  4580  Q   R 4380560 + 8 [dd]
  8,16   1   770779     9.992997135  4580  M   R 4380560 + 8 [dd]
  8,16   1   770780     9.992999236  4580  Q   R 4380568 + 8 [dd]
  8,16   1   770781     9.992999742  4580  M   R 4380568 + 8 [dd]
  8,16   1   770782     9.993001890  4580  Q   R 4380576 + 8 [dd]
  8,16   1   770783     9.993002389  4580  M   R 4380576 + 8 [dd]
  8,16   1   770784     9.993004438  4580  Q   R 4380584 + 8 [dd]
  8,16   1   770785     9.993004937  4580  M   R 4380584 + 8 [dd]
  8,16   1   770786     9.993007255  4580  Q   R 4380592 + 8 [dd]
  8,16   1   770787     9.993007754  4580  M   R 4380592 + 8 [dd]
  8,16   1   770788     9.993009828  4580  Q   R 4380600 + 8 [dd]
  8,16   1   770789     9.993010351  4580  M   R 4380600 + 8 [dd]
  8,16   1   770790     9.993012459  4580  Q   R 4380608 + 8 [dd]
  8,16   1   770791     9.993012958  4580  M   R 4380608 + 8 [dd]
  8,16   1   770792     9.993019700  4580  Q   R 4380616 + 8 [dd]
  8,16   1   770793     9.993020199  4580  M   R 4380616 + 8 [dd]
  8,16   1   770794     9.993022401  4580  Q   R 4380624 + 8 [dd]
  8,16   1   770795     9.993022909  4580  M   R 4380624 + 8 [dd]
  8,16   1   770796     9.993024945  4580  Q   R 4380632 + 8 [dd]
  8,16   1   770797     9.993025449  4580  M   R 4380632 + 8 [dd]
  8,16   1   770798     9.993027501  4580  Q   R 4380640 + 8 [dd]
  8,16   1   770799     9.993027999  4580  M   R 4380640 + 8 [dd]
  8,16   1   770800     9.993030163  4580  Q   R 4380648 + 8 [dd]
  8,16   1   770801     9.993030666  4580  M   R 4380648 + 8 [dd]
  8,16   1   770802     9.993033040  4580  Q   R 4380656 + 8 [dd]
  8,16   1   770803     9.993033538  4580  M   R 4380656 + 8 [dd]
  8,16   1   770804     9.993035704  4580  Q   R 4380664 + 8 [dd]
  8,16   1   770805     9.993036205  4580  M   R 4380664 + 8 [dd]
  8,16   1   770806     9.993143012  4580  Q   R 4380672 + 8 [dd]
  8,16   1   770807     9.993143745  4580  M   R 4380672 + 8 [dd]
  8,16   1   770808     9.993147049  4580  Q   R 4380680 + 8 [dd]
  8,16   1   770809     9.993147550  4580  M   R 4380680 + 8 [dd]
  8,16   1   770810     9.993149983  4580  Q   R 4380688 + 8 [dd]
  8,16   1   770811     9.993150482  4580  M   R 4380688 + 8 [dd]
  8,16   1   770812     9.993156384  4580  Q   R 4380696 + 8 [dd]
  8,16   1   770813     9.993156894  4580  M   R 4380696 + 8 [dd]
  8,16   1   770814     9.993159096  4580  Q   R 4380704 + 8 [dd]
  8,16   1   770815     9.993159606  4580  M   R 4380704 + 8 [dd]
  8,16   1   770816     9.993161853  4580  Q   R 4380712 + 8 [dd]
  8,16   1   770817     9.993162351  4580  M   R 4380712 + 8 [dd]
  8,16   1   770818     9.993164628  4580  Q   R 4380720 + 8 [dd]
  8,16   1   770819     9.993165127  4580  M   R 4380720 + 8 [dd]
  8,16   1   770820     9.993170698  4580  Q   R 4380728 + 8 [dd]
  8,16   1   770821     9.993171200  4580  M   R 4380728 + 8 [dd]
  8,16   1   770822     9.993173392  4580  Q   R 4380736 + 8 [dd]
  8,16   1   770823     9.993173894  4580  M   R 4380736 + 8 [dd]
  8,16   1   770824     9.993176081  4580  Q   R 4380744 + 8 [dd]
  8,16   1   770825     9.993176579  4580  M   R 4380744 + 8 [dd]
  8,16   1   770826     9.993178841  4580  Q   R 4380752 + 8 [dd]
  8,16   1   770827     9.993179340  4580  M   R 4380752 + 8 [dd]
  8,16   1   770828     9.993181445  4580  Q   R 4380760 + 8 [dd]
  8,16   1   770829     9.993181949  4580  M   R 4380760 + 8 [dd]
  8,16   1   770830     9.993184121  4580  Q   R 4380768 + 8 [dd]
  8,16   1   770831     9.993184619  4580  M   R 4380768 + 8 [dd]
  8,16   1   770832     9.993186741  4580  Q   R 4380776 + 8 [dd]
  8,16   1   770833     9.993187239  4580  M   R 4380776 + 8 [dd]
  8,16   1   770834     9.993189324  4580  Q   R 4380784 + 8 [dd]
  8,16   1   770835     9.993189822  4580  M   R 4380784 + 8 [dd]
  8,16   1   770836     9.993191998  4580  Q   R 4380792 + 8 [dd]
  8,16   1   770837     9.993192502  4580  M   R 4380792 + 8 [dd]
  8,16   1   770838     9.993194798  4580  Q   R 4380800 + 8 [dd]
  8,16   1   770839     9.993195297  4580  M   R 4380800 + 8 [dd]
  8,16   1   770840     9.993197858  4580  Q   R 4380808 + 8 [dd]
  8,16   1   770841     9.993198358  4580  M   R 4380808 + 8 [dd]
  8,16   1   770842     9.993201427  4580  Q   R 4380816 + 8 [dd]
  8,16   1   770843     9.993201926  4580  M   R 4380816 + 8 [dd]
  8,16   1   770844     9.993204159  4580  Q   R 4380824 + 8 [dd]
  8,16   1   770845     9.993204658  4580  M   R 4380824 + 8 [dd]
  8,16   1   770846     9.993206840  4580  Q   R 4380832 + 8 [dd]
  8,16   1   770847     9.993207338  4580  M   R 4380832 + 8 [dd]
  8,16   1   770848     9.993212789  4580  Q   R 4380840 + 8 [dd]
  8,16   1   770849     9.993213321  4580  M   R 4380840 + 8 [dd]
  8,16   1   770850     9.993215482  4580  Q   R 4380848 + 8 [dd]
  8,16   1   770851     9.993215986  4580  M   R 4380848 + 8 [dd]
  8,16   1   770852     9.993218173  4580  Q   R 4380856 + 8 [dd]
  8,16   1   770853     9.993218673  4580  M   R 4380856 + 8 [dd]
  8,16   1   770854     9.993220839  4580  Q   R 4380864 + 8 [dd]
  8,16   1   770855     9.993221338  4580  M   R 4380864 + 8 [dd]
  8,16   1   770856     9.993223551  4580  Q   R 4380872 + 8 [dd]
  8,16   1   770857     9.993224051  4580  M   R 4380872 + 8 [dd]
  8,16   1   770858     9.993226133  4580  Q   R 4380880 + 8 [dd]
  8,16   1   770859     9.993226632  4580  M   R 4380880 + 8 [dd]
  8,16   1   770860     9.993228840  4580  Q   R 4380888 + 8 [dd]
  8,16   1   770861     9.993229342  4580  M   R 4380888 + 8 [dd]
  8,16   1   770862     9.993231460  4580  Q   R 4380896 + 8 [dd]
  8,16   1   770863     9.993231958  4580  M   R 4380896 + 8 [dd]
  8,16   1   770864     9.993234011  4580  Q   R 4380904 + 8 [dd]
  8,16   1   770865     9.993234513  4580  M   R 4380904 + 8 [dd]
  8,16   1   770866     9.993236661  4580  Q   R 4380912 + 8 [dd]
  8,16   1   770867     9.993237160  4580  M   R 4380912 + 8 [dd]
  8,16   1   770868     9.993239233  4580  Q   R 4380920 + 8 [dd]
  8,16   1   770869     9.993239736  4580  M   R 4380920 + 8 [dd]
  8,16   1   770870     9.993241912  4580  I   R 4380416 + 512 (  302760) [dd]
  8,16   1        0     9.993244448     0  m   N cfq4580S / insert_request
  8,16   1   770871     9.993247771  4580  U   N [dd] 1
  8,16   1        0     9.993250443     0  m   N cfq4580S / dispatch_insert
  8,16   1        0     9.993254093     0  m   N cfq4580S / dispatched a request
  8,16   1        0     9.993255838     0  m   N cfq4580S / activate rq, drv=1
  8,16   1   770872     9.993256451  4580  D   R 4380416 + 512 (   14539) [dd]
  8,16   1   770873     9.994299156     0  C   R 4380416 + 512 ( 1042705) [0]
  8,16   1        0     9.994371989     0  m   N cfq4580S / complete rqnoidle 0
  8,16   1        0     9.994375064     0  m   N cfq schedule dispatch
  8,16   1   770874     9.994536199  4580  Q   R 4380928 + 8 [dd]
  8,16   1   770875     9.994542473  4580  G   R 4380928 + 8 [dd]
  8,16   1   770876     9.994544517  4580  P   N [dd]
  8,16   1   770877     9.994548013  4580  Q   R 4380936 + 8 [dd]
  8,16   1   770878     9.994550069  4580  M   R 4380936 + 8 [dd]
  8,16   1   770879     9.994553102  4580  Q   R 4380944 + 8 [dd]
  8,16   1   770880     9.994553612  4580  M   R 4380944 + 8 [dd]
  8,16   1   770881     9.994560750  4580  Q   R 4380952 + 8 [dd]
  8,16   1   770882     9.994561294  4580  M   R 4380952 + 8 [dd]
  8,16   1   770883     9.994563660  4580  Q   R 4380960 + 8 [dd]
  8,16   1   770884     9.994564176  4580  M   R 4380960 + 8 [dd]
  8,16   1   770885     9.994566309  4580  Q   R 4380968 + 8 [dd]
  8,16   1   770886     9.994566819  4580  M   R 4380968 + 8 [dd]
  8,16   1   770887     9.994568963  4580  Q   R 4380976 + 8 [dd]
  8,16   1   770888     9.994569474  4580  M   R 4380976 + 8 [dd]
  8,16   1   770889     9.994571592  4580  Q   R 4380984 + 8 [dd]
  8,16   1   770890     9.994572104  4580  M   R 4380984 + 8 [dd]
  8,16   1   770891     9.994574265  4580  Q   R 4380992 + 8 [dd]
  8,16   1   770892     9.994574776  4580  M   R 4380992 + 8 [dd]
  8,16   1   770893     9.994576861  4580  Q   R 4381000 + 8 [dd]
  8,16   1   770894     9.994577360  4580  M   R 4381000 + 8 [dd]
  8,16   1   770895     9.994579582  4580  Q   R 4381008 + 8 [dd]
  8,16   1   770896     9.994580081  4580  M   R 4381008 + 8 [dd]
  8,16   1   770897     9.994582161  4580  Q   R 4381016 + 8 [dd]
  8,16   1   770898     9.994582661  4580  M   R 4381016 + 8 [dd]
  8,16   1   770899     9.994584717  4580  Q   R 4381024 + 8 [dd]
  8,16   1   770900     9.994585216  4580  M   R 4381024 + 8 [dd]
  8,16   1   770901     9.994587324  4580  Q   R 4381032 + 8 [dd]
  8,16   1   770902     9.994587822  4580  M   R 4381032 + 8 [dd]
  8,16   1   770903     9.994589915  4580  Q   R 4381040 + 8 [dd]
  8,16   1   770904     9.994590414  4580  M   R 4381040 + 8 [dd]
  8,16   1   770905     9.994594071  4580  Q   R 4381048 + 8 [dd]
  8,16   1   770906     9.994594575  4580  M   R 4381048 + 8 [dd]
  8,16   1   770907     9.994596896  4580  Q   R 4381056 + 8 [dd]
  8,16   1   770908     9.994597407  4580  M   R 4381056 + 8 [dd]
  8,16   1   770909     9.994602878  4580  Q   R 4381064 + 8 [dd]
  8,16   1   770910     9.994603394  4580  M   R 4381064 + 8 [dd]
  8,16   1   770911     9.994605653  4580  Q   R 4381072 + 8 [dd]
  8,16   1   770912     9.994606164  4580  M   R 4381072 + 8 [dd]
  8,16   1   770913     9.994608528  4580  Q   R 4381080 + 8 [dd]
  8,16   1   770914     9.994609040  4580  M   R 4381080 + 8 [dd]
  8,16   1   770915     9.994611239  4580  Q   R 4381088 + 8 [dd]
  8,16   1   770916     9.994611751  4580  M   R 4381088 + 8 [dd]
  8,16   1   770917     9.994613936  4580  Q   R 4381096 + 8 [dd]
  8,16   1   770918     9.994614447  4580  M   R 4381096 + 8 [dd]
  8,16   1   770919     9.994616718  4580  Q   R 4381104 + 8 [dd]
  8,16   1   770920     9.994617229  4580  M   R 4381104 + 8 [dd]
  8,16   1   770921     9.994619394  4580  Q   R 4381112 + 8 [dd]
  8,16   1   770922     9.994619905  4580  M   R 4381112 + 8 [dd]
  8,16   1   770923     9.994622152  4580  Q   R 4381120 + 8 [dd]
  8,16   1   770924     9.994622663  4580  M   R 4381120 + 8 [dd]
  8,16   1   770925     9.994624765  4580  Q   R 4381128 + 8 [dd]
  8,16   1   770926     9.994625294  4580  M   R 4381128 + 8 [dd]
  8,16   1   770927     9.994627643  4580  Q   R 4381136 + 8 [dd]
  8,16   1   770928     9.994628172  4580  M   R 4381136 + 8 [dd]
  8,16   1   770929     9.994630638  4580  Q   R 4381144 + 8 [dd]
  8,16   1   770930     9.994631141  4580  M   R 4381144 + 8 [dd]
  8,16   1   770931     9.994633374  4580  Q   R 4381152 + 8 [dd]
  8,16   1   770932     9.994633885  4580  M   R 4381152 + 8 [dd]
  8,16   1   770933     9.994636086  4580  Q   R 4381160 + 8 [dd]
  8,16   1   770934     9.994636592  4580  M   R 4381160 + 8 [dd]
  8,16   1   770935     9.994638779  4580  Q   R 4381168 + 8 [dd]
  8,16   1   770936     9.994639291  4580  M   R 4381168 + 8 [dd]
  8,16   1   770937     9.994644640  4580  Q   R 4381176 + 8 [dd]
  8,16   1   770938     9.994645153  4580  M   R 4381176 + 8 [dd]
  8,16   1   770939     9.994750984  4580  Q   R 4381184 + 8 [dd]
  8,16   1   770940     9.994751713  4580  M   R 4381184 + 8 [dd]
  8,16   1   770941     9.994754423  4580  Q   R 4381192 + 8 [dd]
  8,16   1   770942     9.994754936  4580  M   R 4381192 + 8 [dd]
  8,16   1   770943     9.994757220  4580  Q   R 4381200 + 8 [dd]
  8,16   1   770944     9.994757719  4580  M   R 4381200 + 8 [dd]
  8,16   1   770945     9.994759967  4580  Q   R 4381208 + 8 [dd]
  8,16   1   770946     9.994760506  4580  M   R 4381208 + 8 [dd]
  8,16   1   770947     9.994762743  4580  Q   R 4381216 + 8 [dd]
  8,16   1   770948     9.994763242  4580  M   R 4381216 + 8 [dd]
  8,16   1   770949     9.994765425  4580  Q   R 4381224 + 8 [dd]
  8,16   1   770950     9.994765922  4580  M   R 4381224 + 8 [dd]
  8,16   1   770951     9.994768201  4580  Q   R 4381232 + 8 [dd]
  8,16   1   770952     9.994768700  4580  M   R 4381232 + 8 [dd]
  8,16   1   770953     9.994770798  4580  Q   R 4381240 + 8 [dd]
  8,16   1   770954     9.994771298  4580  M   R 4381240 + 8 [dd]
  8,16   1   770955     9.994773393  4580  Q   R 4381248 + 8 [dd]
  8,16   1   770956     9.994773892  4580  M   R 4381248 + 8 [dd]
  8,16   1   770957     9.994776015  4580  Q   R 4381256 + 8 [dd]
  8,16   1   770958     9.994776513  4580  M   R 4381256 + 8 [dd]
  8,16   1   770959     9.994778608  4580  Q   R 4381264 + 8 [dd]
  8,16   1   770960     9.994779107  4580  M   R 4381264 + 8 [dd]
  8,16   1   770961     9.994781210  4580  Q   R 4381272 + 8 [dd]
  8,16   1   770962     9.994781710  4580  M   R 4381272 + 8 [dd]
  8,16   1   770963     9.994787731  4580  Q   R 4381280 + 8 [dd]
  8,16   1   770964     9.994788230  4580  M   R 4381280 + 8 [dd]
  8,16   1   770965     9.994793675  4580  Q   R 4381288 + 8 [dd]
  8,16   1   770966     9.994794174  4580  M   R 4381288 + 8 [dd]
  8,16   1   770967     9.994796307  4580  Q   R 4381296 + 8 [dd]
  8,16   1   770968     9.994796806  4580  M   R 4381296 + 8 [dd]
  8,16   1   770969     9.994799419  4580  Q   R 4381304 + 8 [dd]
  8,16   1   770970     9.994799919  4580  M   R 4381304 + 8 [dd]
  8,16   1   770971     9.994802209  4580  Q   R 4381312 + 8 [dd]
  8,16   1   770972     9.994802707  4580  M   R 4381312 + 8 [dd]
  8,16   1   770973     9.994805174  4580  Q   R 4381320 + 8 [dd]
  8,16   1   770974     9.994805672  4580  M   R 4381320 + 8 [dd]
  8,16   1   770975     9.994807936  4580  Q   R 4381328 + 8 [dd]
  8,16   1   770976     9.994808440  4580  M   R 4381328 + 8 [dd]
  8,16   1   770977     9.994810715  4580  Q   R 4381336 + 8 [dd]
  8,16   1   770978     9.994811242  4580  M   R 4381336 + 8 [dd]
  8,16   1   770979     9.994813464  4580  Q   R 4381344 + 8 [dd]
  8,16   1   770980     9.994813963  4580  M   R 4381344 + 8 [dd]
  8,16   1   770981     9.994816205  4580  Q   R 4381352 + 8 [dd]
  8,16   1   770982     9.994816776  4580  M   R 4381352 + 8 [dd]
  8,16   1   770983     9.994819127  4580  Q   R 4381360 + 8 [dd]
  8,16   1   770984     9.994819628  4580  M   R 4381360 + 8 [dd]
  8,16   1   770985     9.994821810  4580  Q   R 4381368 + 8 [dd]
  8,16   1   770986     9.994822315  4580  M   R 4381368 + 8 [dd]
  8,16   1   770987     9.994824377  4580  Q   R 4381376 + 8 [dd]
  8,16   1   770988     9.994824876  4580  M   R 4381376 + 8 [dd]
  8,16   1   770989     9.994827067  4580  Q   R 4381384 + 8 [dd]
  8,16   1   770990     9.994827565  4580  M   R 4381384 + 8 [dd]
  8,16   1   770991     9.994829760  4580  Q   R 4381392 + 8 [dd]
  8,16   1   770992     9.994830259  4580  M   R 4381392 + 8 [dd]
  8,16   1   770993     9.994835522  4580  Q   R 4381400 + 8 [dd]
  8,16   1   770994     9.994836021  4580  M   R 4381400 + 8 [dd]
  8,16   1   770995     9.994838181  4580  Q   R 4381408 + 8 [dd]
  8,16   1   770996     9.994838680  4580  M   R 4381408 + 8 [dd]
  8,16   1   770997     9.994840857  4580  Q   R 4381416 + 8 [dd]
  8,16   1   770998     9.994841369  4580  M   R 4381416 + 8 [dd]
  8,16   1   770999     9.994843455  4580  Q   R 4381424 + 8 [dd]
  8,16   1   771000     9.994843954  4580  M   R 4381424 + 8 [dd]
  8,16   1   771001     9.994846149  4580  Q   R 4381432 + 8 [dd]
  8,16   1   771002     9.994846662  4580  M   R 4381432 + 8 [dd]
  8,16   1   771003     9.994849336  4580  I   R 4380928 + 512 (  306863) [dd]
  8,16   1        0     9.994852010     0  m   N cfq4580S / insert_request
  8,16   1   771004     9.994855251  4580  U   N [dd] 1
  8,16   1        0     9.994857783     0  m   N cfq4580S / dispatch_insert
  8,16   1        0     9.994861277     0  m   N cfq4580S / dispatched a request
  8,16   1        0     9.994863066     0  m   N cfq4580S / activate rq, drv=1
  8,16   1   771005     9.994863680  4580  D   R 4380928 + 512 (   14344) [dd]
  8,16   1   771006     9.995909291     0  C   R 4380928 + 512 ( 1045611) [0]
  8,16   1        0     9.995983219     0  m   N cfq4580S / complete rqnoidle 0
  8,16   1        0     9.995986144     0  m   N cfq schedule dispatch
  8,16   1   771007     9.996149764  4580  Q   R 4381440 + 8 [dd]
  8,16   1   771008     9.996155872  4580  G   R 4381440 + 8 [dd]
  8,16   1   771009     9.996157994  4580  P   N [dd]
  8,16   1   771010     9.996161548  4580  Q   R 4381448 + 8 [dd]
  8,16   1   771011     9.996163428  4580  M   R 4381448 + 8 [dd]
  8,16   1   771012     9.996166608  4580  Q   R 4381456 + 8 [dd]
  8,16   1   771013     9.996167115  4580  M   R 4381456 + 8 [dd]
  8,16   1   771014     9.996169178  4580  Q   R 4381464 + 8 [dd]
  8,16   1   771015     9.996169681  4580  M   R 4381464 + 8 [dd]
  8,16   1   771016     9.996171779  4580  Q   R 4381472 + 8 [dd]
  8,16   1   771017     9.996172277  4580  M   R 4381472 + 8 [dd]
  8,16   1   771018     9.996174323  4580  Q   R 4381480 + 8 [dd]
  8,16   1   771019     9.996174822  4580  M   R 4381480 + 8 [dd]
  8,16   1   771020     9.996176942  4580  Q   R 4381488 + 8 [dd]
  8,16   1   771021     9.996177446  4580  M   R 4381488 + 8 [dd]
  8,16   1   771022     9.996179520  4580  Q   R 4381496 + 8 [dd]
  8,16   1   771023     9.996180022  4580  M   R 4381496 + 8 [dd]
  8,16   1   771024     9.996182075  4580  Q   R 4381504 + 8 [dd]
  8,16   1   771025     9.996182574  4580  M   R 4381504 + 8 [dd]
  8,16   1   771026     9.996189624  4580  Q   R 4381512 + 8 [dd]
  8,16   1   771027     9.996190124  4580  M   R 4381512 + 8 [dd]
  8,16   1   771028     9.996192318  4580  Q   R 4381520 + 8 [dd]
  8,16   1   771029     9.996192819  4580  M   R 4381520 + 8 [dd]
  8,16   1   771030     9.996194916  4580  Q   R 4381528 + 8 [dd]
  8,16   1   771031     9.996195415  4580  M   R 4381528 + 8 [dd]
  8,16   1   771032     9.996197792  4580  Q   R 4381536 + 8 [dd]
  8,16   1   771033     9.996198291  4580  M   R 4381536 + 8 [dd]
  8,16   1   771034     9.996200401  4580  Q   R 4381544 + 8 [dd]
  8,16   1   771035     9.996200899  4580  M   R 4381544 + 8 [dd]
  8,16   1   771036     9.996202991  4580  Q   R 4381552 + 8 [dd]
  8,16   1   771037     9.996203495  4580  M   R 4381552 + 8 [dd]
  8,16   1   771038     9.996205593  4580  Q   R 4381560 + 8 [dd]
  8,16   1   771039     9.996206095  4580  M   R 4381560 + 8 [dd]
  8,16   1   771040     9.996208179  4580  Q   R 4381568 + 8 [dd]
  8,16   1   771041     9.996208678  4580  M   R 4381568 + 8 [dd]
  8,16   1   771042     9.996210778  4580  Q   R 4381576 + 8 [dd]
  8,16   1   771043     9.996211285  4580  M   R 4381576 + 8 [dd]
  8,16   1   771044     9.996213428  4580  Q   R 4381584 + 8 [dd]
  8,16   1   771045     9.996213932  4580  M   R 4381584 + 8 [dd]
  8,16   1   771046     9.996217450  4580  Q   R 4381592 + 8 [dd]
  8,16   1   771047     9.996217952  4580  M   R 4381592 + 8 [dd]
  8,16   1   771048     9.996220131  4580  Q   R 4381600 + 8 [dd]
  8,16   1   771049     9.996220644  4580  M   R 4381600 + 8 [dd]
  8,16   1   771050     9.996222799  4580  Q   R 4381608 + 8 [dd]
  8,16   1   771051     9.996223302  4580  M   R 4381608 + 8 [dd]
  8,16   1   771052     9.996225537  4580  Q   R 4381616 + 8 [dd]
  8,16   1   771053     9.996226039  4580  M   R 4381616 + 8 [dd]
  8,16   1   771054     9.996231412  4580  Q   R 4381624 + 8 [dd]
  8,16   1   771055     9.996231912  4580  M   R 4381624 + 8 [dd]
  8,16   1   771056     9.996234006  4580  Q   R 4381632 + 8 [dd]
  8,16   1   771057     9.996234505  4580  M   R 4381632 + 8 [dd]
  8,16   1   771058     9.996236661  4580  Q   R 4381640 + 8 [dd]
  8,16   1   771059     9.996237160  4580  M   R 4381640 + 8 [dd]
  8,16   1   771060     9.996239325  4580  Q   R 4381648 + 8 [dd]
  8,16   1   771061     9.996239829  4580  M   R 4381648 + 8 [dd]
  8,16   1   771062     9.996242197  4580  Q   R 4381656 + 8 [dd]
  8,16   1   771063     9.996242700  4580  M   R 4381656 + 8 [dd]
  8,16   1   771064     9.996244888  4580  Q   R 4381664 + 8 [dd]
  8,16   1   771065     9.996245388  4580  M   R 4381664 + 8 [dd]
  8,16   1   771066     9.996247580  4580  Q   R 4381672 + 8 [dd]
  8,16   1   771067     9.996248079  4580  M   R 4381672 + 8 [dd]
  8,16   1   771068     9.996250234  4580  Q   R 4381680 + 8 [dd]
  8,16   1   771069     9.996250735  4580  M   R 4381680 + 8 [dd]
  8,16   1   771070     9.996252898  4580  Q   R 4381688 + 8 [dd]
  8,16   1   771071     9.996253388  4580  M   R 4381688 + 8 [dd]
  8,16   1   771072     9.996357598  4580  Q   R 4381696 + 8 [dd]
  8,16   1   771073     9.996358355  4580  M   R 4381696 + 8 [dd]
  8,16   1   771074     9.996360935  4580  Q   R 4381704 + 8 [dd]
  8,16   1   771075     9.996361434  4580  M   R 4381704 + 8 [dd]
  8,16   1   771076     9.996363662  4580  Q   R 4381712 + 8 [dd]
  8,16   1   771077     9.996364166  4580  M   R 4381712 + 8 [dd]
  8,16   1   771078     9.996366220  4580  Q   R 4381720 + 8 [dd]
  8,16   1   771079     9.996366731  4580  M   R 4381720 + 8 [dd]
  8,16   1   771080     9.996368930  4580  Q   R 4381728 + 8 [dd]
  8,16   1   771081     9.996369429  4580  M   R 4381728 + 8 [dd]
  8,16   1   771082     9.996375003  4580  Q   R 4381736 + 8 [dd]
  8,16   1   771083     9.996375513  4580  M   R 4381736 + 8 [dd]
  8,16   1   771084     9.996377664  4580  Q   R 4381744 + 8 [dd]
  8,16   1   771085     9.996378380  4580  M   R 4381744 + 8 [dd]
  8,16   1   771086     9.996380579  4580  Q   R 4381752 + 8 [dd]
  8,16   1   771087     9.996381098  4580  M   R 4381752 + 8 [dd]
  8,16   1   771088     9.996383260  4580  Q   R 4381760 + 8 [dd]
  8,16   1   771089     9.996383759  4580  M   R 4381760 + 8 [dd]
  8,16   1   771090     9.996385969  4580  Q   R 4381768 + 8 [dd]
  8,16   1   771091     9.996386469  4580  M   R 4381768 + 8 [dd]
  8,16   1   771092     9.996388658  4580  Q   R 4381776 + 8 [dd]
  8,16   1   771093     9.996389162  4580  M   R 4381776 + 8 [dd]
  8,16   1   771094     9.996391335  4580  Q   R 4381784 + 8 [dd]
  8,16   1   771095     9.996391837  4580  M   R 4381784 + 8 [dd]
  8,16   1   771096     9.996393920  4580  Q   R 4381792 + 8 [dd]
  8,16   1   771097     9.996394419  4580  M   R 4381792 + 8 [dd]
  8,16   1   771098     9.996396469  4580  Q   R 4381800 + 8 [dd]
  8,16   1   771099     9.996396967  4580  M   R 4381800 + 8 [dd]
  8,16   1   771100     9.996399181  4580  Q   R 4381808 + 8 [dd]
  8,16   1   771101     9.996399684  4580  M   R 4381808 + 8 [dd]
  8,16   1   771102     9.996402243  4580  Q   R 4381816 + 8 [dd]
  8,16   1   771103     9.996402751  4580  M   R 4381816 + 8 [dd]
  8,16   1   771104     9.996404942  4580  Q   R 4381824 + 8 [dd]
  8,16   1   771105     9.996405441  4580  M   R 4381824 + 8 [dd]
  8,16   1   771106     9.996407662  4580  Q   R 4381832 + 8 [dd]
  8,16   1   771107     9.996408160  4580  M   R 4381832 + 8 [dd]
  8,16   1   771108     9.996410409  4580  Q   R 4381840 + 8 [dd]
  8,16   1   771109     9.996410910  4580  M   R 4381840 + 8 [dd]
  8,16   1   771110     9.996416841  4580  Q   R 4381848 + 8 [dd]
  8,16   1   771111     9.996417341  4580  M   R 4381848 + 8 [dd]
  8,16   1   771112     9.996419551  4580  Q   R 4381856 + 8 [dd]
  8,16   1   771113     9.996420050  4580  M   R 4381856 + 8 [dd]
  8,16   1   771114     9.996425832  4580  Q   R 4381864 + 8 [dd]
  8,16   1   771115     9.996426331  4580  M   R 4381864 + 8 [dd]
  8,16   1   771116     9.996428600  4580  Q   R 4381872 + 8 [dd]
  8,16   1   771117     9.996429101  4580  M   R 4381872 + 8 [dd]
  8,16   1   771118     9.996431277  4580  Q   R 4381880 + 8 [dd]
  8,16   1   771119     9.996431779  4580  M   R 4381880 + 8 [dd]
  8,16   1   771120     9.996434026  4580  Q   R 4381888 + 8 [dd]
  8,16   1   771121     9.996434525  4580  M   R 4381888 + 8 [dd]
  8,16   1   771122     9.996436615  4580  Q   R 4381896 + 8 [dd]
  8,16   1   771123     9.996437114  4580  M   R 4381896 + 8 [dd]
  8,16   1   771124     9.996439353  4580  Q   R 4381904 + 8 [dd]
  8,16   1   771125     9.996439864  4580  M   R 4381904 + 8 [dd]
  8,16   1   771126     9.996441959  4580  Q   R 4381912 + 8 [dd]
  8,16   1   771127     9.996442460  4580  M   R 4381912 + 8 [dd]
  8,16   1   771128     9.996445027  4580  Q   R 4381920 + 8 [dd]
  8,16   1   771129     9.996445526  4580  M   R 4381920 + 8 [dd]
  8,16   1   771130     9.996447737  4580  Q   R 4381928 + 8 [dd]
  8,16   1   771131     9.996448252  4580  M   R 4381928 + 8 [dd]
  8,16   1   771132     9.996450536  4580  Q   R 4381936 + 8 [dd]
  8,16   1   771133     9.996451040  4580  M   R 4381936 + 8 [dd]
  8,16   1   771134     9.996453308  4580  Q   R 4381944 + 8 [dd]
  8,16   1   771135     9.996453811  4580  M   R 4381944 + 8 [dd]
  8,16   1   771136     9.996456417  4580  I   R 4381440 + 512 (  300545) [dd]
  8,16   1        0     9.996458987     0  m   N cfq4580S / insert_request
  8,16   1   771137     9.996462198  4580  U   N [dd] 1
  8,16   1        0     9.996464756     0  m   N cfq4580S / dispatch_insert
  8,16   1        0     9.996468170     0  m   N cfq4580S / dispatched a request
  8,16   1        0     9.996469836     0  m   N cfq4580S / activate rq, drv=1
  8,16   1   771138     9.996470460  4580  D   R 4381440 + 512 (   14043) [dd]
  8,16   1   771139     9.997514205     0  C   R 4381440 + 512 ( 1043745) [0]
  8,16   1        0     9.997587102     0  m   N cfq4580S / complete rqnoidle 0
  8,16   1        0     9.997589870     0  m   N cfq schedule dispatch
  8,16   1   771140     9.997751496  4580  Q   R 4381952 + 8 [dd]
  8,16   1   771141     9.997758630  4580  G   R 4381952 + 8 [dd]
  8,16   1   771142     9.997760814  4580  P   N [dd]
  8,16   1   771143     9.997769413  4580  Q   R 4381960 + 8 [dd]
  8,16   1   771144     9.997771473  4580  M   R 4381960 + 8 [dd]
  8,16   1   771145     9.997774552  4580  Q   R 4381968 + 8 [dd]
  8,16   1   771146     9.997775056  4580  M   R 4381968 + 8 [dd]
  8,16   1   771147     9.997777142  4580  Q   R 4381976 + 8 [dd]
  8,16   1   771148     9.997777717  4580  M   R 4381976 + 8 [dd]
  8,16   1   771149     9.997779832  4580  Q   R 4381984 + 8 [dd]
  8,16   1   771150     9.997780366  4580  M   R 4381984 + 8 [dd]
  8,16   1   771151     9.997782454  4580  Q   R 4381992 + 8 [dd]
  8,16   1   771152     9.997782953  4580  M   R 4381992 + 8 [dd]
  8,16   1   771153     9.997785271  4580  Q   R 4382000 + 8 [dd]
  8,16   1   771154     9.997785779  4580  M   R 4382000 + 8 [dd]
  8,16   1   771155     9.997787937  4580  Q   R 4382008 + 8 [dd]
  8,16   1   771156     9.997788434  4580  M   R 4382008 + 8 [dd]
  8,16   1   771157     9.997790588  4580  Q   R 4382016 + 8 [dd]
  8,16   1   771158     9.997791085  4580  M   R 4382016 + 8 [dd]
  8,16   1   771159     9.997793251  4580  Q   R 4382024 + 8 [dd]
  8,16   1   771160     9.997793954  4580  M   R 4382024 + 8 [dd]
  8,16   1   771161     9.997796144  4580  Q   R 4382032 + 8 [dd]
  8,16   1   771162     9.997796672  4580  M   R 4382032 + 8 [dd]
  8,16   1   771163     9.997798737  4580  Q   R 4382040 + 8 [dd]
  8,16   1   771164     9.997799238  4580  M   R 4382040 + 8 [dd]
  8,16   1   771165     9.997801277  4580  Q   R 4382048 + 8 [dd]
  8,16   1   771166     9.997801788  4580  M   R 4382048 + 8 [dd]
  8,16   1   771167     9.997803925  4580  Q   R 4382056 + 8 [dd]
  8,16   1   771168     9.997804439  4580  M   R 4382056 + 8 [dd]
  8,16   1   771169     9.997806598  4580  Q   R 4382064 + 8 [dd]
  8,16   1   771170     9.997807095  4580  M   R 4382064 + 8 [dd]
  8,16   1   771171     9.997812761  4580  Q   R 4382072 + 8 [dd]
  8,16   1   771172     9.997813262  4580  M   R 4382072 + 8 [dd]
  8,16   1   771173     9.997815335  4580  Q   R 4382080 + 8 [dd]
  8,16   1   771174     9.997815836  4580  M   R 4382080 + 8 [dd]
  8,16   1   771175     9.997817949  4580  Q   R 4382088 + 8 [dd]
  8,16   1   771176     9.997818448  4580  M   R 4382088 + 8 [dd]
  8,16   1   771177     9.997820558  4580  Q   R 4382096 + 8 [dd]
  8,16   1   771178     9.997821057  4580  M   R 4382096 + 8 [dd]
  8,16   1   771179     9.997824698  4580  Q   R 4382104 + 8 [dd]
  8,16   1   771180     9.997825197  4580  M   R 4382104 + 8 [dd]
  8,16   1   771181     9.997827376  4580  Q   R 4382112 + 8 [dd]
  8,16   1   771182     9.997827874  4580  M   R 4382112 + 8 [dd]
  8,16   1   771183     9.997830036  4580  Q   R 4382120 + 8 [dd]
  8,16   1   771184     9.997830538  4580  M   R 4382120 + 8 [dd]
  8,16   1   771185     9.997832781  4580  Q   R 4382128 + 8 [dd]
  8,16   1   771186     9.997833278  4580  M   R 4382128 + 8 [dd]
  8,16   1   771187     9.997835542  4580  Q   R 4382136 + 8 [dd]
  8,16   1   771188     9.997836044  4580  M   R 4382136 + 8 [dd]
  8,16   1   771189     9.997838090  4580  Q   R 4382144 + 8 [dd]
  8,16   1   771190     9.997838591  4580  M   R 4382144 + 8 [dd]
  8,16   1   771191     9.997840646  4580  Q   R 4382152 + 8 [dd]
  8,16   1   771192     9.997841147  4580  M   R 4382152 + 8 [dd]
  8,16   1   771193     9.997843299  4580  Q   R 4382160 + 8 [dd]
  8,16   1   771194     9.997843797  4580  M   R 4382160 + 8 [dd]
  8,16   1   771195     9.997845909  4580  Q   R 4382168 + 8 [dd]
  8,16   1   771196     9.997846408  4580  M   R 4382168 + 8 [dd]
  8,16   1   771197     9.997848578  4580  Q   R 4382176 + 8 [dd]
  8,16   1   771198     9.997849082  4580  M   R 4382176 + 8 [dd]
  8,16   1   771199     9.997854848  4580  Q   R 4382184 + 8 [dd]
  8,16   1   771200     9.997855350  4580  M   R 4382184 + 8 [dd]
  8,16   1   771201     9.997857436  4580  Q   R 4382192 + 8 [dd]
  8,16   1   771202     9.997857934  4580  M   R 4382192 + 8 [dd]
  8,16   1   771203     9.997860278  4580  Q   R 4382200 + 8 [dd]
  8,16   1   771204     9.997860779  4580  M   R 4382200 + 8 [dd]
  8,16   1   771205     9.997964321  4580  Q   R 4382208 + 8 [dd]
  8,16   1   771206     9.997965007  4580  M   R 4382208 + 8 [dd]
  8,16   1   771207     9.997967619  4580  Q   R 4382216 + 8 [dd]
  8,16   1   771208     9.997968161  4580  M   R 4382216 + 8 [dd]
  8,16   1   771209     9.997970459  4580  Q   R 4382224 + 8 [dd]
  8,16   1   771210     9.997970963  4580  M   R 4382224 + 8 [dd]
  8,16   1   771211     9.997973081  4580  Q   R 4382232 + 8 [dd]
  8,16   1   771212     9.997973580  4580  M   R 4382232 + 8 [dd]
  8,16   1   771213     9.997975787  4580  Q   R 4382240 + 8 [dd]
  8,16   1   771214     9.997976285  4580  M   R 4382240 + 8 [dd]
  8,16   1   771215     9.997978452  4580  Q   R 4382248 + 8 [dd]
  8,16   1   771216     9.997978953  4580  M   R 4382248 + 8 [dd]
  8,16   1   771217     9.997981249  4580  Q   R 4382256 + 8 [dd]
  8,16   1   771218     9.997981751  4580  M   R 4382256 + 8 [dd]
  8,16   1   771219     9.997983917  4580  Q   R 4382264 + 8 [dd]
  8,16   1   771220     9.997984418  4580  M   R 4382264 + 8 [dd]
  8,16   1   771221     9.997986611  4580  Q   R 4382272 + 8 [dd]
  8,16   1   771222     9.997987121  4580  M   R 4382272 + 8 [dd]
  8,16   1   771223     9.997989204  4580  Q   R 4382280 + 8 [dd]
  8,16   1   771224     9.997989705  4580  M   R 4382280 + 8 [dd]
  8,16   1   771225     9.997991904  4580  Q   R 4382288 + 8 [dd]
  8,16   1   771226     9.997992402  4580  M   R 4382288 + 8 [dd]
  8,16   1   771227     9.997998036  4580  Q   R 4382296 + 8 [dd]
  8,16   1   771228     9.997998579  4580  M   R 4382296 + 8 [dd]
  8,16   1   771229     9.998000634  4580  Q   R 4382304 + 8 [dd]
  8,16   1   771230     9.998001136  4580  M   R 4382304 + 8 [dd]
  8,16   1   771231     9.998003255  4580  Q   R 4382312 + 8 [dd]
  8,16   1   771232     9.998003774  4580  M   R 4382312 + 8 [dd]
  8,16   1   771233     9.998005975  4580  Q   R 4382320 + 8 [dd]
  8,16   1   771234     9.998006476  4580  M   R 4382320 + 8 [dd]
  8,16   1   771235     9.998008972  4580  Q   R 4382328 + 8 [dd]
  8,16   1   771236     9.998009529  4580  M   R 4382328 + 8 [dd]
  8,16   1   771237     9.998011779  4580  Q   R 4382336 + 8 [dd]
  8,16   1   771238     9.998012277  4580  M   R 4382336 + 8 [dd]
  8,16   1   771239     9.998014476  4580  Q   R 4382344 + 8 [dd]
  8,16   1   771240     9.998014974  4580  M   R 4382344 + 8 [dd]
  8,16   1   771241     9.998017401  4580  Q   R 4382352 + 8 [dd]
  8,16   1   771242     9.998017899  4580  M   R 4382352 + 8 [dd]
  8,16   1   771243     9.998020091  4580  Q   R 4382360 + 8 [dd]
  8,16   1   771244     9.998020588  4580  M   R 4382360 + 8 [dd]
  8,16   1   771245     9.998022782  4580  Q   R 4382368 + 8 [dd]
  8,16   1   771246     9.998023389  4580  M   R 4382368 + 8 [dd]
  8,16   1   771247     9.998025647  4580  Q   R 4382376 + 8 [dd]
  8,16   1   771248     9.998026180  4580  M   R 4382376 + 8 [dd]
  8,16   1   771249     9.998028480  4580  Q   R 4382384 + 8 [dd]
  8,16   1   771250     9.998029020  4580  M   R 4382384 + 8 [dd]
  8,16   1   771251     9.998031205  4580  Q   R 4382392 + 8 [dd]
  8,16   1   771252     9.998031706  4580  M   R 4382392 + 8 [dd]
  8,16   1   771253     9.998033865  4580  Q   R 4382400 + 8 [dd]
  8,16   1   771254     9.998034366  4580  M   R 4382400 + 8 [dd]
  8,16   1   771255     9.998040073  4580  Q   R 4382408 + 8 [dd]
  8,16   1   771256     9.998040571  4580  M   R 4382408 + 8 [dd]
  8,16   1   771257     9.998042647  4580  Q   R 4382416 + 8 [dd]
  8,16   1   771258     9.998043150  4580  M   R 4382416 + 8 [dd]
  8,16   1   771259     9.998045325  4580  Q   R 4382424 + 8 [dd]
  8,16   1   771260     9.998045826  4580  M   R 4382424 + 8 [dd]
  8,16   1   771261     9.998047931  4580  Q   R 4382432 + 8 [dd]
  8,16   1   771262     9.998048432  4580  M   R 4382432 + 8 [dd]
  8,16   1   771263     9.998050535  4580  Q   R 4382440 + 8 [dd]
  8,16   1   771264     9.998051038  4580  M   R 4382440 + 8 [dd]
  8,16   1   771265     9.998056985  4580  Q   R 4382448 + 8 [dd]
  8,16   1   771266     9.998057500  4580  M   R 4382448 + 8 [dd]
  8,16   1   771267     9.998059576  4580  Q   R 4382456 + 8 [dd]
  8,16   1   771268     9.998060077  4580  M   R 4382456 + 8 [dd]
  8,16   1   771269     9.998062341  4580  I   R 4381952 + 512 (  303711) [dd]
  8,16   1        0     9.998064881     0  m   N cfq4580S / insert_request
  8,16   1   771270     9.998068490  4580  U   N [dd] 1
  8,16   1        0     9.998071031     0  m   N cfq4580S / dispatch_insert
  8,16   1        0     9.998074991     0  m   N cfq4580S / dispatched a request
  8,16   1        0     9.998076645     0  m   N cfq4580S / activate rq, drv=1
  8,16   1   771271     9.998077269  4580  D   R 4381952 + 512 (   14928) [dd]
  8,16   1   771272     9.999120396     0  C   R 4381952 + 512 ( 1043127) [0]
  8,16   1        0     9.999192302     0  m   N cfq4580S / complete rqnoidle 0
  8,16   1        0     9.999195361     0  m   N cfq schedule dispatch
  8,16   1   771273     9.999356099  4580  Q   R 4382464 + 8 [dd]
  8,16   1   771274     9.999362719  4580  G   R 4382464 + 8 [dd]
  8,16   1   771275     9.999364795  4580  P   N [dd]
  8,16   1   771276     9.999368199  4580  Q   R 4382472 + 8 [dd]
  8,16   1   771277     9.999370072  4580  M   R 4382472 + 8 [dd]
  8,16   1   771278     9.999373166  4580  Q   R 4382480 + 8 [dd]
  8,16   1   771279     9.999373710  4580  M   R 4382480 + 8 [dd]
  8,16   1   771280     9.999375830  4580  Q   R 4382488 + 8 [dd]
  8,16   1   771281     9.999376334  4580  M   R 4382488 + 8 [dd]
  8,16   1   771282     9.999378433  4580  Q   R 4382496 + 8 [dd]
  8,16   1   771283     9.999378932  4580  M   R 4382496 + 8 [dd]
  8,16   1   771284     9.999380973  4580  Q   R 4382504 + 8 [dd]
  8,16   1   771285     9.999381477  4580  M   R 4382504 + 8 [dd]
  8,16   1   771286     9.999383602  4580  Q   R 4382512 + 8 [dd]
  8,16   1   771287     9.999384101  4580  M   R 4382512 + 8 [dd]
  8,16   1   771288     9.999391190  4580  Q   R 4382520 + 8 [dd]
  8,16   1   771289     9.999391690  4580  M   R 4382520 + 8 [dd]
  8,16   1   771290     9.999393850  4580  Q   R 4382528 + 8 [dd]
  8,16   1   771291     9.999394351  4580  M   R 4382528 + 8 [dd]
  8,16   1   771292     9.999396618  4580  Q   R 4382536 + 8 [dd]
  8,16   1   771293     9.999397190  4580  M   R 4382536 + 8 [dd]
  8,16   1   771294     9.999399323  4580  Q   R 4382544 + 8 [dd]
  8,16   1   771295     9.999399864  4580  M   R 4382544 + 8 [dd]
  8,16   1   771296     9.999401953  4580  Q   R 4382552 + 8 [dd]
  8,16   1   771297     9.999402472  4580  M   R 4382552 + 8 [dd]
  8,16   1   771298     9.999404563  4580  Q   R 4382560 + 8 [dd]
  8,16   1   771299     9.999405057  4580  M   R 4382560 + 8 [dd]
  8,16   1   771300     9.999407109  4580  Q   R 4382568 + 8 [dd]
  8,16   1   771301     9.999407611  4580  M   R 4382568 + 8 [dd]
  8,16   1   771302     9.999409677  4580  Q   R 4382576 + 8 [dd]
  8,16   1   771303     9.999410176  4580  M   R 4382576 + 8 [dd]
  8,16   1   771304     9.999412226  4580  Q   R 4382584 + 8 [dd]
  8,16   1   771305     9.999412734  4580  M   R 4382584 + 8 [dd]
  8,16   1   771306     9.999416340  4580  Q   R 4382592 + 8 [dd]
  8,16   1   771307     9.999416862  4580  M   R 4382592 + 8 [dd]
  8,16   1   771308     9.999419187  4580  Q   R 4382600 + 8 [dd]
  8,16   1   771309     9.999419706  4580  M   R 4382600 + 8 [dd]
  8,16   1   771310     9.999421977  4580  Q   R 4382608 + 8 [dd]
  8,16   1   771311     9.999422503  4580  M   R 4382608 + 8 [dd]
  8,16   1   771312     9.999424582  4580  Q   R 4382616 + 8 [dd]
  8,16   1   771313     9.999425120  4580  M   R 4382616 + 8 [dd]
  8,16   1   771314     9.999427292  4580  Q   R 4382624 + 8 [dd]
  8,16   1   771315     9.999427805  4580  M   R 4382624 + 8 [dd]
  8,16   1   771316     9.999433255  4580  Q   R 4382632 + 8 [dd]
  8,16   1   771317     9.999433766  4580  M   R 4382632 + 8 [dd]
  8,16   1   771318     9.999435901  4580  Q   R 4382640 + 8 [dd]
  8,16   1   771319     9.999436412  4580  M   R 4382640 + 8 [dd]
  8,16   1   771320     9.999438504  4580  Q   R 4382648 + 8 [dd]
  8,16   1   771321     9.999439243  4580  M   R 4382648 + 8 [dd]
  8,16   1   771322     9.999441418  4580  Q   R 4382656 + 8 [dd]
  8,16   1   771323     9.999441949  4580  M   R 4382656 + 8 [dd]
  8,16   1   771324     9.999444183  4580  Q   R 4382664 + 8 [dd]
  8,16   1   771325     9.999444698  4580  M   R 4382664 + 8 [dd]
  8,16   1   771326     9.999446894  4580  Q   R 4382672 + 8 [dd]
  8,16   1   771327     9.999447411  4580  M   R 4382672 + 8 [dd]
  8,16   1   771328     9.999449485  4580  Q   R 4382680 + 8 [dd]
  8,16   1   771329     9.999449997  4580  M   R 4382680 + 8 [dd]
  8,16   1   771330     9.999452055  4580  Q   R 4382688 + 8 [dd]
  8,16   1   771331     9.999452566  4580  M   R 4382688 + 8 [dd]
  8,16   1   771332     9.999454606  4580  Q   R 4382696 + 8 [dd]
  8,16   1   771333     9.999455154  4580  M   R 4382696 + 8 [dd]
  8,16   1   771334     9.999457363  4580  Q   R 4382704 + 8 [dd]
  8,16   1   771335     9.999457875  4580  M   R 4382704 + 8 [dd]
  8,16   1   771336     9.999460045  4580  Q   R 4382712 + 8 [dd]
  8,16   1   771337     9.999460561  4580  M   R 4382712 + 8 [dd]
  8,16   1   771338     9.999564615  4580  Q   R 4382720 + 8 [dd]
  8,16   1   771339     9.999565347  4580  M   R 4382720 + 8 [dd]
  8,16   1   771340     9.999568401  4580  Q   R 4382728 + 8 [dd]
  8,16   1   771341     9.999568908  4580  M   R 4382728 + 8 [dd]
  8,16   1   771342     9.999571276  4580  Q   R 4382736 + 8 [dd]
  8,16   1   771343     9.999571792  4580  M   R 4382736 + 8 [dd]
  8,16   1   771344     9.999577396  4580  Q   R 4382744 + 8 [dd]
  8,16   1   771345     9.999578003  4580  M   R 4382744 + 8 [dd]
  8,16   1   771346     9.999580231  4580  Q   R 4382752 + 8 [dd]
  8,16   1   771347     9.999580749  4580  M   R 4382752 + 8 [dd]
  8,16   1   771348     9.999583114  4580  Q   R 4382760 + 8 [dd]
  8,16   1   771349     9.999583625  4580  M   R 4382760 + 8 [dd]
  8,16   1   771350     9.999585875  4580  Q   R 4382768 + 8 [dd]
  8,16   1   771351     9.999586374  4580  M   R 4382768 + 8 [dd]
  8,16   1   771352     9.999588531  4580  Q   R 4382776 + 8 [dd]
  8,16   1   771353     9.999589032  4580  M   R 4382776 + 8 [dd]
  8,16   1   771354     9.999591214  4580  Q   R 4382784 + 8 [dd]
  8,16   1   771355     9.999591714  4580  M   R 4382784 + 8 [dd]
  8,16   1   771356     9.999593930  4580  Q   R 4382792 + 8 [dd]
  8,16   1   771357     9.999594442  4580  M   R 4382792 + 8 [dd]
  8,16   1   771358     9.999596696  4580  Q   R 4382800 + 8 [dd]
  8,16   1   771359     9.999597190  4580  M   R 4382800 + 8 [dd]
  8,16   1   771360     9.999599406  4580  Q   R 4382808 + 8 [dd]
  8,16   1   771361     9.999599907  4580  M   R 4382808 + 8 [dd]
  8,16   1   771362     9.999602166  4580  Q   R 4382816 + 8 [dd]
  8,16   1   771363     9.999602667  4580  M   R 4382816 + 8 [dd]
  8,16   1   771364     9.999604755  4580  Q   R 4382824 + 8 [dd]
  8,16   1   771365     9.999605259  4580  M   R 4382824 + 8 [dd]
  8,16   1   771366     9.999607486  4580  Q   R 4382832 + 8 [dd]
  8,16   1   771367     9.999607978  4580  M   R 4382832 + 8 [dd]
  8,16   1   771368     9.999610062  4580  Q   R 4382840 + 8 [dd]
  8,16   1   771369     9.999610564  4580  M   R 4382840 + 8 [dd]
  8,16   1   771370     9.999612914  4580  Q   R 4382848 + 8 [dd]
  8,16   1   771371     9.999613418  4580  M   R 4382848 + 8 [dd]
  8,16   1   771372     9.999618806  4580  Q   R 4382856 + 8 [dd]
  8,16   1   771373     9.999619308  4580  M   R 4382856 + 8 [dd]
  8,16   1   771374     9.999621579  4580  Q   R 4382864 + 8 [dd]
  8,16   1   771375     9.999622092  4580  M   R 4382864 + 8 [dd]
  8,16   1   771376     9.999624260  4580  Q   R 4382872 + 8 [dd]
  8,16   1   771377     9.999624779  4580  M   R 4382872 + 8 [dd]
  8,16   1   771378     9.999627506  4580  Q   R 4382880 + 8 [dd]
  8,16   1   771379     9.999628012  4580  M   R 4382880 + 8 [dd]
  8,16   1   771380     9.999630135  4580  Q   R 4382888 + 8 [dd]
  8,16   1   771381     9.999630635  4580  M   R 4382888 + 8 [dd]
  8,16   1   771382     9.999632876  4580  Q   R 4382896 + 8 [dd]
  8,16   1   771383     9.999633378  4580  M   R 4382896 + 8 [dd]
  8,16   1   771384     9.999635541  4580  Q   R 4382904 + 8 [dd]
  8,16   1   771385     9.999636043  4580  M   R 4382904 + 8 [dd]
  8,16   1   771386     9.999638232  4580  Q   R 4382912 + 8 [dd]
  8,16   1   771387     9.999638736  4580  M   R 4382912 + 8 [dd]
  8,16   1   771388     9.999640940  4580  Q   R 4382920 + 8 [dd]
  8,16   1   771389     9.999641440  4580  M   R 4382920 + 8 [dd]
  8,16   1   771390     9.999643644  4580  Q   R 4382928 + 8 [dd]
  8,16   1   771391     9.999644157  4580  M   R 4382928 + 8 [dd]
  8,16   1   771392     9.999646440  4580  Q   R 4382936 + 8 [dd]
  8,16   1   771393     9.999646960  4580  M   R 4382936 + 8 [dd]
  8,16   1   771394     9.999649193  4580  Q   R 4382944 + 8 [dd]
  8,16   1   771395     9.999649697  4580  M   R 4382944 + 8 [dd]
  8,16   1   771396     9.999651771  4580  Q   R 4382952 + 8 [dd]
  8,16   1   771397     9.999652272  4580  M   R 4382952 + 8 [dd]
  8,16   1   771398     9.999654476  4580  Q   R 4382960 + 8 [dd]
  8,16   1   771399     9.999654975  4580  M   R 4382960 + 8 [dd]
  8,16   1   771400     9.999660256  4580  Q   R 4382968 + 8 [dd]
  8,16   1   771401     9.999660760  4580  M   R 4382968 + 8 [dd]
  8,16   1   771402     9.999663197  4580  I   R 4382464 + 512 (  300478) [dd]
  8,16   1        0     9.999665743     0  m   N cfq4580S / insert_request
  8,16   1   771403     9.999669190  4580  U   N [dd] 1
  8,16   1        0     9.999672048     0  m   N cfq4580S / dispatch_insert
  8,16   1        0     9.999675568     0  m   N cfq4580S / dispatched a request
  8,16   1        0     9.999677281     0  m   N cfq4580S / activate rq, drv=1
  8,16   1   771404     9.999677892  4580  D   R 4382464 + 512 (   14695) [dd]
CPU0 (sdb):
 Reads Queued:       20608,    82432KiB	 Writes Queued:           0,        0KiB
 Read Dispatches:      322,    82432KiB	 Write Dispatches:        0,        0KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:       22,     5632KiB	 Writes Completed:        0,        0KiB
 Read Merges:        20286,    81144KiB	 Write Merges:            0,        0KiB
 Read depth:             1        	 Write depth:             0
 IO unplugs:           322        	 Timer unplugs:           0
CPU1 (sdb):
 Reads Queued:      371058,     1484MiB	 Writes Queued:           0,        0KiB
 Read Dispatches:     5798,     1484MiB	 Write Dispatches:        0,        0KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:     6097,     1560MiB	 Writes Completed:        0,        0KiB
 Read Merges:       365261,     1461MiB	 Write Merges:            0,        0KiB
 Read depth:             1        	 Write depth:             0
 IO unplugs:          5798        	 Timer unplugs:           0

Total (sdb):
 Reads Queued:      391666,     1566MiB	 Writes Queued:           0,        0KiB
 Read Dispatches:     6120,     1566MiB	 Write Dispatches:        0,        0KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:     6119,     1566MiB	 Writes Completed:        0,        0KiB
 Read Merges:       385547,     1542MiB	 Write Merges:            0,        0KiB
 IO unplugs:          6120        	 Timer unplugs:           0

Throughput (R/W): 156662KiB/s / 0KiB/s
Events (sdb): 851428 entries
Skips: 0 forward (0 -   0.0%)
Input file sdb.blktrace.0 added
Input file sdb.blktrace.1 added

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Mon, Jan 30, 2012 at 01:34:56PM +0100, Eric Dumazet wrote:
> Le lundi 30 janvier 2012 à 19:18 +0800, Wu Fengguang a écrit :
> 
> > I agree with Shaohua that there may be timing/plug issues. There
> > happen to be some plug patches and (maybe correlated) big performance
> > drop between 2.6.38 and 2.6.39. The obvious way to move forward is to
> > get some blktrace data on simple dd + new buggy kernel and let's check
> > what's exactly going on.
> > 
> > # start a background dd read
> > blktrace /dev/sda -w 10
> > blkparse -t sda
> > 
> 
> (Another machine here at office, hence /dev/sdb)
> 
> echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=2M
> count=2048
> 2048+0 enregistrements lus
> 2048+0 enregistrements écrits
> 4294967296 octets (4,3 GB) copiés, 26,0278 s, 165 MB/s
> 
> # blktrace /dev/sdb -w 10
> === sdb ===
>   CPU  0:                44766 events,     2099 KiB data
>   CPU  1:               828895 events,    38855 KiB data
>   Total:                873661 events (dropped 0),    40953 KiB data
> 
> blkparse result is huge (55Mo)
> 
> # blkparse -t sdb | tail -n 1000 

Thanks for the data!

It shows that

- IO size is 256KB (which is not a problem in itself)

- The dispatch/complete pattern is

        submit IO for range 1
        complete IO for range 1
        <dd busy, disk idle>
        submit IO for range 2
        complete IO for range 2

  So we have periods that no IO is in flight at all, which leads to
  under-utilized disk (which should show up in iostat as <100% disk util)

# grep '[DC]' blktrace

  8,16   1   770606     9.990039807  4580  D   R 4379392 + 512 (   14826) [dd]
  8,16   1   770607     9.991083069     0  C   R 4379392 + 512 ( 1043262) [0]

  8,16   1   770739     9.991647434  4580  D   R 4379904 + 512 (   14433) [dd]
  8,16   1   770740     9.992693317     0  C   R 4379904 + 512 ( 1045883) [0]

  8,16   1   770872     9.993256451  4580  D   R 4380416 + 512 (   14539) [dd]
  8,16   1   770873     9.994299156     0  C   R 4380416 + 512 ( 1042705) [0]

  8,16   1   771005     9.994863680  4580  D   R 4380928 + 512 (   14344) [dd]
  8,16   1   771006     9.995909291     0  C   R 4380928 + 512 ( 1045611) [0]

  8,16   1   771138     9.996470460  4580  D   R 4381440 + 512 (   14043) [dd]
  8,16   1   771139     9.997514205     0  C   R 4381440 + 512 ( 1043745) [0]

  8,16   1   771271     9.998077269  4580  D   R 4381952 + 512 (   14928) [dd]
  8,16   1   771272     9.999120396     0  C   R 4381952 + 512 ( 1043127) [0]

Thanks,
Fengguang

>   8,16   1   770475     9.989714963  4580  Q   R 4379392 + 8 [dd]
>   8,16   1   770476     9.989721470  4580  G   R 4379392 + 8 [dd]
>   8,16   1   770477     9.989723546  4580  P   N [dd]
>   8,16   1   770478     9.989726994  4580  Q   R 4379400 + 8 [dd]
>   8,16   1   770479     9.989728858  4580  M   R 4379400 + 8 [dd]
>   8,16   1   770480     9.989731893  4580  Q   R 4379408 + 8 [dd]
>   8,16   1   770481     9.989732467  4580  M   R 4379408 + 8 [dd]
>   8,16   1   770482     9.989734574  4580  Q   R 4379416 + 8 [dd]
>   8,16   1   770483     9.989735108  4580  M   R 4379416 + 8 [dd]
>   8,16   1   770484     9.989737220  4580  Q   R 4379424 + 8 [dd]
>   8,16   1   770485     9.989737721  4580  M   R 4379424 + 8 [dd]
>   8,16   1   770486     9.989739953  4580  Q   R 4379432 + 8 [dd]
>   8,16   1   770487     9.989740450  4580  M   R 4379432 + 8 [dd]
>   8,16   1   770488     9.989742577  4580  Q   R 4379440 + 8 [dd]
>   8,16   1   770489     9.989743078  4580  M   R 4379440 + 8 [dd]
>   8,16   1   770490     9.989745241  4580  Q   R 4379448 + 8 [dd]
>   8,16   1   770491     9.989745742  4580  M   R 4379448 + 8 [dd]
>   8,16   1   770492     9.989747953  4580  Q   R 4379456 + 8 [dd]
>   8,16   1   770493     9.989748485  4580  M   R 4379456 + 8 [dd]
>   8,16   1   770494     9.989750726  4580  Q   R 4379464 + 8 [dd]
>   8,16   1   770495     9.989751225  4580  M   R 4379464 + 8 [dd]
>   8,16   1   770496     9.989753390  4580  Q   R 4379472 + 8 [dd]
>   8,16   1   770497     9.989753891  4580  M   R 4379472 + 8 [dd]
>   8,16   1   770498     9.989755928  4580  Q   R 4379480 + 8 [dd]
>   8,16   1   770499     9.989756429  4580  M   R 4379480 + 8 [dd]
>   8,16   1   770500     9.989758470  4580  Q   R 4379488 + 8 [dd]
>   8,16   1   770501     9.989758975  4580  M   R 4379488 + 8 [dd]
>   8,16   1   770502     9.989765725  4580  Q   R 4379496 + 8 [dd]
>   8,16   1   770503     9.989766222  4580  M   R 4379496 + 8 [dd]
>   8,16   1   770504     9.989768362  4580  Q   R 4379504 + 8 [dd]
>   8,16   1   770505     9.989768864  4580  M   R 4379504 + 8 [dd]
>   8,16   1   770506     9.989771157  4580  Q   R 4379512 + 8 [dd]
>   8,16   1   770507     9.989771662  4580  M   R 4379512 + 8 [dd]
>   8,16   1   770508     9.989775151  4580  Q   R 4379520 + 8 [dd]
>   8,16   1   770509     9.989775649  4580  M   R 4379520 + 8 [dd]
>   8,16   1   770510     9.989781743  4580  Q   R 4379528 + 8 [dd]
>   8,16   1   770511     9.989782248  4580  M   R 4379528 + 8 [dd]
>   8,16   1   770512     9.989784532  4580  Q   R 4379536 + 8 [dd]
>   8,16   1   770513     9.989785033  4580  M   R 4379536 + 8 [dd]
>   8,16   1   770514     9.989787291  4580  Q   R 4379544 + 8 [dd]
>   8,16   1   770515     9.989787792  4580  M   R 4379544 + 8 [dd]
>   8,16   1   770516     9.989790041  4580  Q   R 4379552 + 8 [dd]
>   8,16   1   770517     9.989790540  4580  M   R 4379552 + 8 [dd]
>   8,16   1   770518     9.989792607  4580  Q   R 4379560 + 8 [dd]
>   8,16   1   770519     9.989793104  4580  M   R 4379560 + 8 [dd]
>   8,16   1   770520     9.989795693  4580  Q   R 4379568 + 8 [dd]
>   8,16   1   770521     9.989796191  4580  M   R 4379568 + 8 [dd]
>   8,16   1   770522     9.989798349  4580  Q   R 4379576 + 8 [dd]
>   8,16   1   770523     9.989798847  4580  M   R 4379576 + 8 [dd]
>   8,16   1   770524     9.989800877  4580  Q   R 4379584 + 8 [dd]
>   8,16   1   770525     9.989801581  4580  M   R 4379584 + 8 [dd]
>   8,16   1   770526     9.989803693  4580  Q   R 4379592 + 8 [dd]
>   8,16   1   770527     9.989804221  4580  M   R 4379592 + 8 [dd]
>   8,16   1   770528     9.989806399  4580  Q   R 4379600 + 8 [dd]
>   8,16   1   770529     9.989806901  4580  M   R 4379600 + 8 [dd]
>   8,16   1   770530     9.989812437  4580  Q   R 4379608 + 8 [dd]
>   8,16   1   770531     9.989812938  4580  M   R 4379608 + 8 [dd]
>   8,16   1   770532     9.989814981  4580  Q   R 4379616 + 8 [dd]
>   8,16   1   770533     9.989815482  4580  M   R 4379616 + 8 [dd]
>   8,16   1   770534     9.989818053  4580  Q   R 4379624 + 8 [dd]
>   8,16   1   770535     9.989818551  4580  M   R 4379624 + 8 [dd]
>   8,16   1   770536     9.989820795  4580  Q   R 4379632 + 8 [dd]
>   8,16   1   770537     9.989821296  4580  M   R 4379632 + 8 [dd]
>   8,16   1   770538     9.989823436  4580  Q   R 4379640 + 8 [dd]
>   8,16   1   770539     9.989823937  4580  M   R 4379640 + 8 [dd]
>   8,16   1   770540     9.989929502  4580  Q   R 4379648 + 8 [dd]
>   8,16   1   770541     9.989930247  4580  M   R 4379648 + 8 [dd]
>   8,16   1   770542     9.989933189  4580  Q   R 4379656 + 8 [dd]
>   8,16   1   770543     9.989933705  4580  M   R 4379656 + 8 [dd]
>   8,16   1   770544     9.989935909  4580  Q   R 4379664 + 8 [dd]
>   8,16   1   770545     9.989936411  4580  M   R 4379664 + 8 [dd]
>   8,16   1   770546     9.989938572  4580  Q   R 4379672 + 8 [dd]
>   8,16   1   770547     9.989939072  4580  M   R 4379672 + 8 [dd]
>   8,16   1   770548     9.989941272  4580  Q   R 4379680 + 8 [dd]
>   8,16   1   770549     9.989941770  4580  M   R 4379680 + 8 [dd]
>   8,16   1   770550     9.989943952  4580  Q   R 4379688 + 8 [dd]
>   8,16   1   770551     9.989944450  4580  M   R 4379688 + 8 [dd]
>   8,16   1   770552     9.989946991  4580  Q   R 4379696 + 8 [dd]
>   8,16   1   770553     9.989947492  4580  M   R 4379696 + 8 [dd]
>   8,16   1   770554     9.989949671  4580  Q   R 4379704 + 8 [dd]
>   8,16   1   770555     9.989950172  4580  M   R 4379704 + 8 [dd]
>   8,16   1   770556     9.989952330  4580  Q   R 4379712 + 8 [dd]
>   8,16   1   770557     9.989952832  4580  M   R 4379712 + 8 [dd]
>   8,16   1   770558     9.989958393  4580  Q   R 4379720 + 8 [dd]
>   8,16   1   770559     9.989958891  4580  M   R 4379720 + 8 [dd]
>   8,16   1   770560     9.989961097  4580  Q   R 4379728 + 8 [dd]
>   8,16   1   770561     9.989961598  4580  M   R 4379728 + 8 [dd]
>   8,16   1   770562     9.989963825  4580  Q   R 4379736 + 8 [dd]
>   8,16   1   770563     9.989964331  4580  M   R 4379736 + 8 [dd]
>   8,16   1   770564     9.989966623  4580  Q   R 4379744 + 8 [dd]
>   8,16   1   770565     9.989967122  4580  M   R 4379744 + 8 [dd]
>   8,16   1   770566     9.989969303  4580  Q   R 4379752 + 8 [dd]
>   8,16   1   770567     9.989969800  4580  M   R 4379752 + 8 [dd]
>   8,16   1   770568     9.989971886  4580  Q   R 4379760 + 8 [dd]
>   8,16   1   770569     9.989972388  4580  M   R 4379760 + 8 [dd]
>   8,16   1   770570     9.989974789  4580  Q   R 4379768 + 8 [dd]
>   8,16   1   770571     9.989975290  4580  M   R 4379768 + 8 [dd]
>   8,16   1   770572     9.989977480  4580  Q   R 4379776 + 8 [dd]
>   8,16   1   770573     9.989977981  4580  M   R 4379776 + 8 [dd]
>   8,16   1   770574     9.989981251  4580  Q   R 4379784 + 8 [dd]
>   8,16   1   770575     9.989981749  4580  M   R 4379784 + 8 [dd]
>   8,16   1   770576     9.989984018  4580  Q   R 4379792 + 8 [dd]
>   8,16   1   770577     9.989984519  4580  M   R 4379792 + 8 [dd]
>   8,16   1   770578     9.989986711  4580  Q   R 4379800 + 8 [dd]
>   8,16   1   770579     9.989987222  4580  M   R 4379800 + 8 [dd]
>   8,16   1   770580     9.989989414  4580  Q   R 4379808 + 8 [dd]
>   8,16   1   770581     9.989989915  4580  M   R 4379808 + 8 [dd]
>   8,16   1   770582     9.989992107  4580  Q   R 4379816 + 8 [dd]
>   8,16   1   770583     9.989992617  4580  M   R 4379816 + 8 [dd]
>   8,16   1   770584     9.989994776  4580  Q   R 4379824 + 8 [dd]
>   8,16   1   770585     9.989995276  4580  M   R 4379824 + 8 [dd]
>   8,16   1   770586     9.990000570  4580  Q   R 4379832 + 8 [dd]
>   8,16   1   770587     9.990001076  4580  M   R 4379832 + 8 [dd]
>   8,16   1   770588     9.990003264  4580  Q   R 4379840 + 8 [dd]
>   8,16   1   770589     9.990003785  4580  M   R 4379840 + 8 [dd]
>   8,16   1   770590     9.990005849  4580  Q   R 4379848 + 8 [dd]
>   8,16   1   770591     9.990006354  4580  M   R 4379848 + 8 [dd]
>   8,16   1   770592     9.990008472  4580  Q   R 4379856 + 8 [dd]
>   8,16   1   770593     9.990008986  4580  M   R 4379856 + 8 [dd]
>   8,16   1   770594     9.990011382  4580  Q   R 4379864 + 8 [dd]
>   8,16   1   770595     9.990011884  4580  M   R 4379864 + 8 [dd]
>   8,16   1   770596     9.990014066  4580  Q   R 4379872 + 8 [dd]
>   8,16   1   770597     9.990014568  4580  M   R 4379872 + 8 [dd]
>   8,16   1   770598     9.990016740  4580  Q   R 4379880 + 8 [dd]
>   8,16   1   770599     9.990017237  4580  M   R 4379880 + 8 [dd]
>   8,16   1   770600     9.990019491  4580  Q   R 4379888 + 8 [dd]
>   8,16   1   770601     9.990020001  4580  M   R 4379888 + 8 [dd]
>   8,16   1   770602     9.990022222  4580  Q   R 4379896 + 8 [dd]
>   8,16   1   770603     9.990022723  4580  M   R 4379896 + 8 [dd]
>   8,16   1   770604     9.990024981  4580  I   R 4379392 + 512 (  303511) [dd]
>   8,16   1        0     9.990027734     0  m   N cfq4580S / insert_request
>   8,16   1   770605     9.990031216  4580  U   N [dd] 1
>   8,16   1        0     9.990033792     0  m   N cfq4580S / dispatch_insert
>   8,16   1        0     9.990037429     0  m   N cfq4580S / dispatched a request
>   8,16   1        0     9.990039146     0  m   N cfq4580S / activate rq, drv=1
>   8,16   1   770606     9.990039807  4580  D   R 4379392 + 512 (   14826) [dd]
>   8,16   1   770607     9.991083069     0  C   R 4379392 + 512 ( 1043262) [0]
>   8,16   1        0     9.991155388     0  m   N cfq4580S / complete rqnoidle 0
>   8,16   1        0     9.991158292     0  m   N cfq schedule dispatch
>   8,16   1   770608     9.991321889  4580  Q   R 4379904 + 8 [dd]
>   8,16   1   770609     9.991328302  4580  G   R 4379904 + 8 [dd]
>   8,16   1   770610     9.991330287  4580  P   N [dd]
>   8,16   1   770611     9.991333624  4580  Q   R 4379912 + 8 [dd]
>   8,16   1   770612     9.991335576  4580  M   R 4379912 + 8 [dd]
>   8,16   1   770613     9.991338586  4580  Q   R 4379920 + 8 [dd]
>   8,16   1   770614     9.991339095  4580  M   R 4379920 + 8 [dd]
>   8,16   1   770615     9.991341209  4580  Q   R 4379928 + 8 [dd]
>   8,16   1   770616     9.991341909  4580  M   R 4379928 + 8 [dd]
>   8,16   1   770617     9.991344208  4580  Q   R 4379936 + 8 [dd]
>   8,16   1   770618     9.991344707  4580  M   R 4379936 + 8 [dd]
>   8,16   1   770619     9.991351891  4580  Q   R 4379944 + 8 [dd]
>   8,16   1   770620     9.991352393  4580  M   R 4379944 + 8 [dd]
>   8,16   1   770621     9.991354635  4580  Q   R 4379952 + 8 [dd]
>   8,16   1   770622     9.991355134  4580  M   R 4379952 + 8 [dd]
>   8,16   1   770623     9.991357304  4580  Q   R 4379960 + 8 [dd]
>   8,16   1   770624     9.991357809  4580  M   R 4379960 + 8 [dd]
>   8,16   1   770625     9.991359960  4580  Q   R 4379968 + 8 [dd]
>   8,16   1   770626     9.991360458  4580  M   R 4379968 + 8 [dd]
>   8,16   1   770627     9.991362540  4580  Q   R 4379976 + 8 [dd]
>   8,16   1   770628     9.991363044  4580  M   R 4379976 + 8 [dd]
>   8,16   1   770629     9.991366526  4580  Q   R 4379984 + 8 [dd]
>   8,16   1   770630     9.991367025  4580  M   R 4379984 + 8 [dd]
>   8,16   1   770631     9.991369183  4580  Q   R 4379992 + 8 [dd]
>   8,16   1   770632     9.991369681  4580  M   R 4379992 + 8 [dd]
>   8,16   1   770633     9.991371837  4580  Q   R 4380000 + 8 [dd]
>   8,16   1   770634     9.991372336  4580  M   R 4380000 + 8 [dd]
>   8,16   1   770635     9.991374518  4580  Q   R 4380008 + 8 [dd]
>   8,16   1   770636     9.991375031  4580  M   R 4380008 + 8 [dd]
>   8,16   1   770637     9.991377249  4580  Q   R 4380016 + 8 [dd]
>   8,16   1   770638     9.991377748  4580  M   R 4380016 + 8 [dd]
>   8,16   1   770639     9.991379891  4580  Q   R 4380024 + 8 [dd]
>   8,16   1   770640     9.991380390  4580  M   R 4380024 + 8 [dd]
>   8,16   1   770641     9.991382585  4580  Q   R 4380032 + 8 [dd]
>   8,16   1   770642     9.991383084  4580  M   R 4380032 + 8 [dd]
>   8,16   1   770643     9.991385207  4580  Q   R 4380040 + 8 [dd]
>   8,16   1   770644     9.991385704  4580  M   R 4380040 + 8 [dd]
>   8,16   1   770645     9.991387897  4580  Q   R 4380048 + 8 [dd]
>   8,16   1   770646     9.991388396  4580  M   R 4380048 + 8 [dd]
>   8,16   1   770647     9.991393804  4580  Q   R 4380056 + 8 [dd]
>   8,16   1   770648     9.991394303  4580  M   R 4380056 + 8 [dd]
>   8,16   1   770649     9.991396440  4580  Q   R 4380064 + 8 [dd]
>   8,16   1   770650     9.991396938  4580  M   R 4380064 + 8 [dd]
>   8,16   1   770651     9.991398996  4580  Q   R 4380072 + 8 [dd]
>   8,16   1   770652     9.991399500  4580  M   R 4380072 + 8 [dd]
>   8,16   1   770653     9.991401608  4580  Q   R 4380080 + 8 [dd]
>   8,16   1   770654     9.991402107  4580  M   R 4380080 + 8 [dd]
>   8,16   1   770655     9.991404535  4580  Q   R 4380088 + 8 [dd]
>   8,16   1   770656     9.991405034  4580  M   R 4380088 + 8 [dd]
>   8,16   1   770657     9.991407171  4580  Q   R 4380096 + 8 [dd]
>   8,16   1   770658     9.991407671  4580  M   R 4380096 + 8 [dd]
>   8,16   1   770659     9.991409829  4580  Q   R 4380104 + 8 [dd]
>   8,16   1   770660     9.991410339  4580  M   R 4380104 + 8 [dd]
>   8,16   1   770661     9.991416346  4580  Q   R 4380112 + 8 [dd]
>   8,16   1   770662     9.991416853  4580  M   R 4380112 + 8 [dd]
>   8,16   1   770663     9.991419016  4580  Q   R 4380120 + 8 [dd]
>   8,16   1   770664     9.991419515  4580  M   R 4380120 + 8 [dd]
>   8,16   1   770665     9.991421984  4580  Q   R 4380128 + 8 [dd]
>   8,16   1   770666     9.991422483  4580  M   R 4380128 + 8 [dd]
>   8,16   1   770667     9.991424780  4580  Q   R 4380136 + 8 [dd]
>   8,16   1   770668     9.991425284  4580  M   R 4380136 + 8 [dd]
>   8,16   1   770669     9.991427479  4580  Q   R 4380144 + 8 [dd]
>   8,16   1   770670     9.991427978  4580  M   R 4380144 + 8 [dd]
>   8,16   1   770671     9.991430076  4580  Q   R 4380152 + 8 [dd]
>   8,16   1   770672     9.991430575  4580  M   R 4380152 + 8 [dd]
>   8,16   1   770673     9.991536122  4580  Q   R 4380160 + 8 [dd]
>   8,16   1   770674     9.991536827  4580  M   R 4380160 + 8 [dd]
>   8,16   1   770675     9.991543663  4580  Q   R 4380168 + 8 [dd]
>   8,16   1   770676     9.991544181  4580  M   R 4380168 + 8 [dd]
>   8,16   1   770677     9.991546458  4580  Q   R 4380176 + 8 [dd]
>   8,16   1   770678     9.991546957  4580  M   R 4380176 + 8 [dd]
>   8,16   1   770679     9.991549153  4580  Q   R 4380184 + 8 [dd]
>   8,16   1   770680     9.991549652  4580  M   R 4380184 + 8 [dd]
>   8,16   1   770681     9.991551902  4580  Q   R 4380192 + 8 [dd]
>   8,16   1   770682     9.991552400  4580  M   R 4380192 + 8 [dd]
>   8,16   1   770683     9.991554621  4580  Q   R 4380200 + 8 [dd]
>   8,16   1   770684     9.991555113  4580  M   R 4380200 + 8 [dd]
>   8,16   1   770685     9.991557325  4580  Q   R 4380208 + 8 [dd]
>   8,16   1   770686     9.991557824  4580  M   R 4380208 + 8 [dd]
>   8,16   1   770687     9.991560192  4580  Q   R 4380216 + 8 [dd]
>   8,16   1   770688     9.991560732  4580  M   R 4380216 + 8 [dd]
>   8,16   1   770689     9.991562806  4580  Q   R 4380224 + 8 [dd]
>   8,16   1   770690     9.991563305  4580  M   R 4380224 + 8 [dd]
>   8,16   1   770691     9.991565370  4580  Q   R 4380232 + 8 [dd]
>   8,16   1   770692     9.991565874  4580  M   R 4380232 + 8 [dd]
>   8,16   1   770693     9.991567969  4580  Q   R 4380240 + 8 [dd]
>   8,16   1   770694     9.991568468  4580  M   R 4380240 + 8 [dd]
>   8,16   1   770695     9.991570671  4580  Q   R 4380248 + 8 [dd]
>   8,16   1   770696     9.991571170  4580  M   R 4380248 + 8 [dd]
>   8,16   1   770697     9.991573285  4580  Q   R 4380256 + 8 [dd]
>   8,16   1   770698     9.991573783  4580  M   R 4380256 + 8 [dd]
>   8,16   1   770699     9.991575855  4580  Q   R 4380264 + 8 [dd]
>   8,16   1   770700     9.991576358  4580  M   R 4380264 + 8 [dd]
>   8,16   1   770701     9.991578551  4580  Q   R 4380272 + 8 [dd]
>   8,16   1   770702     9.991579050  4580  M   R 4380272 + 8 [dd]
>   8,16   1   770703     9.991584418  4580  Q   R 4380280 + 8 [dd]
>   8,16   1   770704     9.991584916  4580  M   R 4380280 + 8 [dd]
>   8,16   1   770705     9.991587004  4580  Q   R 4380288 + 8 [dd]
>   8,16   1   770706     9.991587503  4580  M   R 4380288 + 8 [dd]
>   8,16   1   770707     9.991589615  4580  Q   R 4380296 + 8 [dd]
>   8,16   1   770708     9.991590128  4580  M   R 4380296 + 8 [dd]
>   8,16   1   770709     9.991592390  4580  Q   R 4380304 + 8 [dd]
>   8,16   1   770710     9.991592907  4580  M   R 4380304 + 8 [dd]
>   8,16   1   770711     9.991594957  4580  Q   R 4380312 + 8 [dd]
>   8,16   1   770712     9.991595456  4580  M   R 4380312 + 8 [dd]
>   8,16   1   770713     9.991597599  4580  Q   R 4380320 + 8 [dd]
>   8,16   1   770714     9.991598098  4580  M   R 4380320 + 8 [dd]
>   8,16   1   770715     9.991600165  4580  Q   R 4380328 + 8 [dd]
>   8,16   1   770716     9.991600669  4580  M   R 4380328 + 8 [dd]
>   8,16   1   770717     9.991602915  4580  Q   R 4380336 + 8 [dd]
>   8,16   1   770718     9.991603413  4580  M   R 4380336 + 8 [dd]
>   8,16   1   770719     9.991605563  4580  Q   R 4380344 + 8 [dd]
>   8,16   1   770720     9.991606061  4580  M   R 4380344 + 8 [dd]
>   8,16   1   770721     9.991608110  4580  Q   R 4380352 + 8 [dd]
>   8,16   1   770722     9.991608609  4580  M   R 4380352 + 8 [dd]
>   8,16   1   770723     9.991610674  4580  Q   R 4380360 + 8 [dd]
>   8,16   1   770724     9.991611178  4580  M   R 4380360 + 8 [dd]
>   8,16   1   770725     9.991613249  4580  Q   R 4380368 + 8 [dd]
>   8,16   1   770726     9.991613748  4580  M   R 4380368 + 8 [dd]
>   8,16   1   770727     9.991616309  4580  Q   R 4380376 + 8 [dd]
>   8,16   1   770728     9.991616808  4580  M   R 4380376 + 8 [dd]
>   8,16   1   770729     9.991619009  4580  Q   R 4380384 + 8 [dd]
>   8,16   1   770730     9.991619507  4580  M   R 4380384 + 8 [dd]
>   8,16   1   770731     9.991624896  4580  Q   R 4380392 + 8 [dd]
>   8,16   1   770732     9.991625401  4580  M   R 4380392 + 8 [dd]
>   8,16   1   770733     9.991627620  4580  Q   R 4380400 + 8 [dd]
>   8,16   1   770734     9.991628119  4580  M   R 4380400 + 8 [dd]
>   8,16   1   770735     9.991630192  4580  Q   R 4380408 + 8 [dd]
>   8,16   1   770736     9.991630690  4580  M   R 4380408 + 8 [dd]
>   8,16   1   770737     9.991633001  4580  I   R 4379904 + 512 (  304699) [dd]
>   8,16   1        0     9.991635738     0  m   N cfq4580S / insert_request
>   8,16   1   770738     9.991638988  4580  U   N [dd] 1
>   8,16   1        0     9.991641665     0  m   N cfq4580S / dispatch_insert
>   8,16   1        0     9.991645149     0  m   N cfq4580S / dispatched a request
>   8,16   1        0     9.991646825     0  m   N cfq4580S / activate rq, drv=1
>   8,16   1   770739     9.991647434  4580  D   R 4379904 + 512 (   14433) [dd]
>   8,16   1   770740     9.992693317     0  C   R 4379904 + 512 ( 1045883) [0]
>   8,16   1        0     9.992766012     0  m   N cfq4580S / complete rqnoidle 0
>   8,16   1        0     9.992768895     0  m   N cfq schedule dispatch
>   8,16   1   770741     9.992932816  4580  Q   R 4380416 + 8 [dd]
>   8,16   1   770742     9.992939152  4580  G   R 4380416 + 8 [dd]
>   8,16   1   770743     9.992941241  4580  P   N [dd]
>   8,16   1   770744     9.992944673  4580  Q   R 4380424 + 8 [dd]
>   8,16   1   770745     9.992946529  4580  M   R 4380424 + 8 [dd]
>   8,16   1   770746     9.992949525  4580  Q   R 4380432 + 8 [dd]
>   8,16   1   770747     9.992950035  4580  M   R 4380432 + 8 [dd]
>   8,16   1   770748     9.992952218  4580  Q   R 4380440 + 8 [dd]
>   8,16   1   770749     9.992952765  4580  M   R 4380440 + 8 [dd]
>   8,16   1   770750     9.992954867  4580  Q   R 4380448 + 8 [dd]
>   8,16   1   770751     9.992955366  4580  M   R 4380448 + 8 [dd]
>   8,16   1   770752     9.992957451  4580  Q   R 4380456 + 8 [dd]
>   8,16   1   770753     9.992957956  4580  M   R 4380456 + 8 [dd]
>   8,16   1   770754     9.992960073  4580  Q   R 4380464 + 8 [dd]
>   8,16   1   770755     9.992960572  4580  M   R 4380464 + 8 [dd]
>   8,16   1   770756     9.992962600  4580  Q   R 4380472 + 8 [dd]
>   8,16   1   770757     9.992963104  4580  M   R 4380472 + 8 [dd]
>   8,16   1   770758     9.992965247  4580  Q   R 4380480 + 8 [dd]
>   8,16   1   770759     9.992965753  4580  M   R 4380480 + 8 [dd]
>   8,16   1   770760     9.992967787  4580  Q   R 4380488 + 8 [dd]
>   8,16   1   770761     9.992968284  4580  M   R 4380488 + 8 [dd]
>   8,16   1   770762     9.992970602  4580  Q   R 4380496 + 8 [dd]
>   8,16   1   770763     9.992971101  4580  M   R 4380496 + 8 [dd]
>   8,16   1   770764     9.992978036  4580  Q   R 4380504 + 8 [dd]
>   8,16   1   770765     9.992978575  4580  M   R 4380504 + 8 [dd]
>   8,16   1   770766     9.992980800  4580  Q   R 4380512 + 8 [dd]
>   8,16   1   770767     9.992981312  4580  M   R 4380512 + 8 [dd]
>   8,16   1   770768     9.992983463  4580  Q   R 4380520 + 8 [dd]
>   8,16   1   770769     9.992983976  4580  M   R 4380520 + 8 [dd]
>   8,16   1   770770     9.992986141  4580  Q   R 4380528 + 8 [dd]
>   8,16   1   770771     9.992986652  4580  M   R 4380528 + 8 [dd]
>   8,16   1   770772     9.992988835  4580  Q   R 4380536 + 8 [dd]
>   8,16   1   770773     9.992989338  4580  M   R 4380536 + 8 [dd]
>   8,16   1   770774     9.992991462  4580  Q   R 4380544 + 8 [dd]
>   8,16   1   770775     9.992991961  4580  M   R 4380544 + 8 [dd]
>   8,16   1   770776     9.992994020  4580  Q   R 4380552 + 8 [dd]
>   8,16   1   770777     9.992994519  4580  M   R 4380552 + 8 [dd]
>   8,16   1   770778     9.992996632  4580  Q   R 4380560 + 8 [dd]
>   8,16   1   770779     9.992997135  4580  M   R 4380560 + 8 [dd]
>   8,16   1   770780     9.992999236  4580  Q   R 4380568 + 8 [dd]
>   8,16   1   770781     9.992999742  4580  M   R 4380568 + 8 [dd]
>   8,16   1   770782     9.993001890  4580  Q   R 4380576 + 8 [dd]
>   8,16   1   770783     9.993002389  4580  M   R 4380576 + 8 [dd]
>   8,16   1   770784     9.993004438  4580  Q   R 4380584 + 8 [dd]
>   8,16   1   770785     9.993004937  4580  M   R 4380584 + 8 [dd]
>   8,16   1   770786     9.993007255  4580  Q   R 4380592 + 8 [dd]
>   8,16   1   770787     9.993007754  4580  M   R 4380592 + 8 [dd]
>   8,16   1   770788     9.993009828  4580  Q   R 4380600 + 8 [dd]
>   8,16   1   770789     9.993010351  4580  M   R 4380600 + 8 [dd]
>   8,16   1   770790     9.993012459  4580  Q   R 4380608 + 8 [dd]
>   8,16   1   770791     9.993012958  4580  M   R 4380608 + 8 [dd]
>   8,16   1   770792     9.993019700  4580  Q   R 4380616 + 8 [dd]
>   8,16   1   770793     9.993020199  4580  M   R 4380616 + 8 [dd]
>   8,16   1   770794     9.993022401  4580  Q   R 4380624 + 8 [dd]
>   8,16   1   770795     9.993022909  4580  M   R 4380624 + 8 [dd]
>   8,16   1   770796     9.993024945  4580  Q   R 4380632 + 8 [dd]
>   8,16   1   770797     9.993025449  4580  M   R 4380632 + 8 [dd]
>   8,16   1   770798     9.993027501  4580  Q   R 4380640 + 8 [dd]
>   8,16   1   770799     9.993027999  4580  M   R 4380640 + 8 [dd]
>   8,16   1   770800     9.993030163  4580  Q   R 4380648 + 8 [dd]
>   8,16   1   770801     9.993030666  4580  M   R 4380648 + 8 [dd]
>   8,16   1   770802     9.993033040  4580  Q   R 4380656 + 8 [dd]
>   8,16   1   770803     9.993033538  4580  M   R 4380656 + 8 [dd]
>   8,16   1   770804     9.993035704  4580  Q   R 4380664 + 8 [dd]
>   8,16   1   770805     9.993036205  4580  M   R 4380664 + 8 [dd]
>   8,16   1   770806     9.993143012  4580  Q   R 4380672 + 8 [dd]
>   8,16   1   770807     9.993143745  4580  M   R 4380672 + 8 [dd]
>   8,16   1   770808     9.993147049  4580  Q   R 4380680 + 8 [dd]
>   8,16   1   770809     9.993147550  4580  M   R 4380680 + 8 [dd]
>   8,16   1   770810     9.993149983  4580  Q   R 4380688 + 8 [dd]
>   8,16   1   770811     9.993150482  4580  M   R 4380688 + 8 [dd]
>   8,16   1   770812     9.993156384  4580  Q   R 4380696 + 8 [dd]
>   8,16   1   770813     9.993156894  4580  M   R 4380696 + 8 [dd]
>   8,16   1   770814     9.993159096  4580  Q   R 4380704 + 8 [dd]
>   8,16   1   770815     9.993159606  4580  M   R 4380704 + 8 [dd]
>   8,16   1   770816     9.993161853  4580  Q   R 4380712 + 8 [dd]
>   8,16   1   770817     9.993162351  4580  M   R 4380712 + 8 [dd]
>   8,16   1   770818     9.993164628  4580  Q   R 4380720 + 8 [dd]
>   8,16   1   770819     9.993165127  4580  M   R 4380720 + 8 [dd]
>   8,16   1   770820     9.993170698  4580  Q   R 4380728 + 8 [dd]
>   8,16   1   770821     9.993171200  4580  M   R 4380728 + 8 [dd]
>   8,16   1   770822     9.993173392  4580  Q   R 4380736 + 8 [dd]
>   8,16   1   770823     9.993173894  4580  M   R 4380736 + 8 [dd]
>   8,16   1   770824     9.993176081  4580  Q   R 4380744 + 8 [dd]
>   8,16   1   770825     9.993176579  4580  M   R 4380744 + 8 [dd]
>   8,16   1   770826     9.993178841  4580  Q   R 4380752 + 8 [dd]
>   8,16   1   770827     9.993179340  4580  M   R 4380752 + 8 [dd]
>   8,16   1   770828     9.993181445  4580  Q   R 4380760 + 8 [dd]
>   8,16   1   770829     9.993181949  4580  M   R 4380760 + 8 [dd]
>   8,16   1   770830     9.993184121  4580  Q   R 4380768 + 8 [dd]
>   8,16   1   770831     9.993184619  4580  M   R 4380768 + 8 [dd]
>   8,16   1   770832     9.993186741  4580  Q   R 4380776 + 8 [dd]
>   8,16   1   770833     9.993187239  4580  M   R 4380776 + 8 [dd]
>   8,16   1   770834     9.993189324  4580  Q   R 4380784 + 8 [dd]
>   8,16   1   770835     9.993189822  4580  M   R 4380784 + 8 [dd]
>   8,16   1   770836     9.993191998  4580  Q   R 4380792 + 8 [dd]
>   8,16   1   770837     9.993192502  4580  M   R 4380792 + 8 [dd]
>   8,16   1   770838     9.993194798  4580  Q   R 4380800 + 8 [dd]
>   8,16   1   770839     9.993195297  4580  M   R 4380800 + 8 [dd]
>   8,16   1   770840     9.993197858  4580  Q   R 4380808 + 8 [dd]
>   8,16   1   770841     9.993198358  4580  M   R 4380808 + 8 [dd]
>   8,16   1   770842     9.993201427  4580  Q   R 4380816 + 8 [dd]
>   8,16   1   770843     9.993201926  4580  M   R 4380816 + 8 [dd]
>   8,16   1   770844     9.993204159  4580  Q   R 4380824 + 8 [dd]
>   8,16   1   770845     9.993204658  4580  M   R 4380824 + 8 [dd]
>   8,16   1   770846     9.993206840  4580  Q   R 4380832 + 8 [dd]
>   8,16   1   770847     9.993207338  4580  M   R 4380832 + 8 [dd]
>   8,16   1   770848     9.993212789  4580  Q   R 4380840 + 8 [dd]
>   8,16   1   770849     9.993213321  4580  M   R 4380840 + 8 [dd]
>   8,16   1   770850     9.993215482  4580  Q   R 4380848 + 8 [dd]
>   8,16   1   770851     9.993215986  4580  M   R 4380848 + 8 [dd]
>   8,16   1   770852     9.993218173  4580  Q   R 4380856 + 8 [dd]
>   8,16   1   770853     9.993218673  4580  M   R 4380856 + 8 [dd]
>   8,16   1   770854     9.993220839  4580  Q   R 4380864 + 8 [dd]
>   8,16   1   770855     9.993221338  4580  M   R 4380864 + 8 [dd]
>   8,16   1   770856     9.993223551  4580  Q   R 4380872 + 8 [dd]
>   8,16   1   770857     9.993224051  4580  M   R 4380872 + 8 [dd]
>   8,16   1   770858     9.993226133  4580  Q   R 4380880 + 8 [dd]
>   8,16   1   770859     9.993226632  4580  M   R 4380880 + 8 [dd]
>   8,16   1   770860     9.993228840  4580  Q   R 4380888 + 8 [dd]
>   8,16   1   770861     9.993229342  4580  M   R 4380888 + 8 [dd]
>   8,16   1   770862     9.993231460  4580  Q   R 4380896 + 8 [dd]
>   8,16   1   770863     9.993231958  4580  M   R 4380896 + 8 [dd]
>   8,16   1   770864     9.993234011  4580  Q   R 4380904 + 8 [dd]
>   8,16   1   770865     9.993234513  4580  M   R 4380904 + 8 [dd]
>   8,16   1   770866     9.993236661  4580  Q   R 4380912 + 8 [dd]
>   8,16   1   770867     9.993237160  4580  M   R 4380912 + 8 [dd]
>   8,16   1   770868     9.993239233  4580  Q   R 4380920 + 8 [dd]
>   8,16   1   770869     9.993239736  4580  M   R 4380920 + 8 [dd]
>   8,16   1   770870     9.993241912  4580  I   R 4380416 + 512 (  302760) [dd]
>   8,16   1        0     9.993244448     0  m   N cfq4580S / insert_request
>   8,16   1   770871     9.993247771  4580  U   N [dd] 1
>   8,16   1        0     9.993250443     0  m   N cfq4580S / dispatch_insert
>   8,16   1        0     9.993254093     0  m   N cfq4580S / dispatched a request
>   8,16   1        0     9.993255838     0  m   N cfq4580S / activate rq, drv=1
>   8,16   1   770872     9.993256451  4580  D   R 4380416 + 512 (   14539) [dd]
>   8,16   1   770873     9.994299156     0  C   R 4380416 + 512 ( 1042705) [0]
>   8,16   1        0     9.994371989     0  m   N cfq4580S / complete rqnoidle 0
>   8,16   1        0     9.994375064     0  m   N cfq schedule dispatch
>   8,16   1   770874     9.994536199  4580  Q   R 4380928 + 8 [dd]
>   8,16   1   770875     9.994542473  4580  G   R 4380928 + 8 [dd]
>   8,16   1   770876     9.994544517  4580  P   N [dd]
>   8,16   1   770877     9.994548013  4580  Q   R 4380936 + 8 [dd]
>   8,16   1   770878     9.994550069  4580  M   R 4380936 + 8 [dd]
>   8,16   1   770879     9.994553102  4580  Q   R 4380944 + 8 [dd]
>   8,16   1   770880     9.994553612  4580  M   R 4380944 + 8 [dd]
>   8,16   1   770881     9.994560750  4580  Q   R 4380952 + 8 [dd]
>   8,16   1   770882     9.994561294  4580  M   R 4380952 + 8 [dd]
>   8,16   1   770883     9.994563660  4580  Q   R 4380960 + 8 [dd]
>   8,16   1   770884     9.994564176  4580  M   R 4380960 + 8 [dd]
>   8,16   1   770885     9.994566309  4580  Q   R 4380968 + 8 [dd]
>   8,16   1   770886     9.994566819  4580  M   R 4380968 + 8 [dd]
>   8,16   1   770887     9.994568963  4580  Q   R 4380976 + 8 [dd]
>   8,16   1   770888     9.994569474  4580  M   R 4380976 + 8 [dd]
>   8,16   1   770889     9.994571592  4580  Q   R 4380984 + 8 [dd]
>   8,16   1   770890     9.994572104  4580  M   R 4380984 + 8 [dd]
>   8,16   1   770891     9.994574265  4580  Q   R 4380992 + 8 [dd]
>   8,16   1   770892     9.994574776  4580  M   R 4380992 + 8 [dd]
>   8,16   1   770893     9.994576861  4580  Q   R 4381000 + 8 [dd]
>   8,16   1   770894     9.994577360  4580  M   R 4381000 + 8 [dd]
>   8,16   1   770895     9.994579582  4580  Q   R 4381008 + 8 [dd]
>   8,16   1   770896     9.994580081  4580  M   R 4381008 + 8 [dd]
>   8,16   1   770897     9.994582161  4580  Q   R 4381016 + 8 [dd]
>   8,16   1   770898     9.994582661  4580  M   R 4381016 + 8 [dd]
>   8,16   1   770899     9.994584717  4580  Q   R 4381024 + 8 [dd]
>   8,16   1   770900     9.994585216  4580  M   R 4381024 + 8 [dd]
>   8,16   1   770901     9.994587324  4580  Q   R 4381032 + 8 [dd]
>   8,16   1   770902     9.994587822  4580  M   R 4381032 + 8 [dd]
>   8,16   1   770903     9.994589915  4580  Q   R 4381040 + 8 [dd]
>   8,16   1   770904     9.994590414  4580  M   R 4381040 + 8 [dd]
>   8,16   1   770905     9.994594071  4580  Q   R 4381048 + 8 [dd]
>   8,16   1   770906     9.994594575  4580  M   R 4381048 + 8 [dd]
>   8,16   1   770907     9.994596896  4580  Q   R 4381056 + 8 [dd]
>   8,16   1   770908     9.994597407  4580  M   R 4381056 + 8 [dd]
>   8,16   1   770909     9.994602878  4580  Q   R 4381064 + 8 [dd]
>   8,16   1   770910     9.994603394  4580  M   R 4381064 + 8 [dd]
>   8,16   1   770911     9.994605653  4580  Q   R 4381072 + 8 [dd]
>   8,16   1   770912     9.994606164  4580  M   R 4381072 + 8 [dd]
>   8,16   1   770913     9.994608528  4580  Q   R 4381080 + 8 [dd]
>   8,16   1   770914     9.994609040  4580  M   R 4381080 + 8 [dd]
>   8,16   1   770915     9.994611239  4580  Q   R 4381088 + 8 [dd]
>   8,16   1   770916     9.994611751  4580  M   R 4381088 + 8 [dd]
>   8,16   1   770917     9.994613936  4580  Q   R 4381096 + 8 [dd]
>   8,16   1   770918     9.994614447  4580  M   R 4381096 + 8 [dd]
>   8,16   1   770919     9.994616718  4580  Q   R 4381104 + 8 [dd]
>   8,16   1   770920     9.994617229  4580  M   R 4381104 + 8 [dd]
>   8,16   1   770921     9.994619394  4580  Q   R 4381112 + 8 [dd]
>   8,16   1   770922     9.994619905  4580  M   R 4381112 + 8 [dd]
>   8,16   1   770923     9.994622152  4580  Q   R 4381120 + 8 [dd]
>   8,16   1   770924     9.994622663  4580  M   R 4381120 + 8 [dd]
>   8,16   1   770925     9.994624765  4580  Q   R 4381128 + 8 [dd]
>   8,16   1   770926     9.994625294  4580  M   R 4381128 + 8 [dd]
>   8,16   1   770927     9.994627643  4580  Q   R 4381136 + 8 [dd]
>   8,16   1   770928     9.994628172  4580  M   R 4381136 + 8 [dd]
>   8,16   1   770929     9.994630638  4580  Q   R 4381144 + 8 [dd]
>   8,16   1   770930     9.994631141  4580  M   R 4381144 + 8 [dd]
>   8,16   1   770931     9.994633374  4580  Q   R 4381152 + 8 [dd]
>   8,16   1   770932     9.994633885  4580  M   R 4381152 + 8 [dd]
>   8,16   1   770933     9.994636086  4580  Q   R 4381160 + 8 [dd]
>   8,16   1   770934     9.994636592  4580  M   R 4381160 + 8 [dd]
>   8,16   1   770935     9.994638779  4580  Q   R 4381168 + 8 [dd]
>   8,16   1   770936     9.994639291  4580  M   R 4381168 + 8 [dd]
>   8,16   1   770937     9.994644640  4580  Q   R 4381176 + 8 [dd]
>   8,16   1   770938     9.994645153  4580  M   R 4381176 + 8 [dd]
>   8,16   1   770939     9.994750984  4580  Q   R 4381184 + 8 [dd]
>   8,16   1   770940     9.994751713  4580  M   R 4381184 + 8 [dd]
>   8,16   1   770941     9.994754423  4580  Q   R 4381192 + 8 [dd]
>   8,16   1   770942     9.994754936  4580  M   R 4381192 + 8 [dd]
>   8,16   1   770943     9.994757220  4580  Q   R 4381200 + 8 [dd]
>   8,16   1   770944     9.994757719  4580  M   R 4381200 + 8 [dd]
>   8,16   1   770945     9.994759967  4580  Q   R 4381208 + 8 [dd]
>   8,16   1   770946     9.994760506  4580  M   R 4381208 + 8 [dd]
>   8,16   1   770947     9.994762743  4580  Q   R 4381216 + 8 [dd]
>   8,16   1   770948     9.994763242  4580  M   R 4381216 + 8 [dd]
>   8,16   1   770949     9.994765425  4580  Q   R 4381224 + 8 [dd]
>   8,16   1   770950     9.994765922  4580  M   R 4381224 + 8 [dd]
>   8,16   1   770951     9.994768201  4580  Q   R 4381232 + 8 [dd]
>   8,16   1   770952     9.994768700  4580  M   R 4381232 + 8 [dd]
>   8,16   1   770953     9.994770798  4580  Q   R 4381240 + 8 [dd]
>   8,16   1   770954     9.994771298  4580  M   R 4381240 + 8 [dd]
>   8,16   1   770955     9.994773393  4580  Q   R 4381248 + 8 [dd]
>   8,16   1   770956     9.994773892  4580  M   R 4381248 + 8 [dd]
>   8,16   1   770957     9.994776015  4580  Q   R 4381256 + 8 [dd]
>   8,16   1   770958     9.994776513  4580  M   R 4381256 + 8 [dd]
>   8,16   1   770959     9.994778608  4580  Q   R 4381264 + 8 [dd]
>   8,16   1   770960     9.994779107  4580  M   R 4381264 + 8 [dd]
>   8,16   1   770961     9.994781210  4580  Q   R 4381272 + 8 [dd]
>   8,16   1   770962     9.994781710  4580  M   R 4381272 + 8 [dd]
>   8,16   1   770963     9.994787731  4580  Q   R 4381280 + 8 [dd]
>   8,16   1   770964     9.994788230  4580  M   R 4381280 + 8 [dd]
>   8,16   1   770965     9.994793675  4580  Q   R 4381288 + 8 [dd]
>   8,16   1   770966     9.994794174  4580  M   R 4381288 + 8 [dd]
>   8,16   1   770967     9.994796307  4580  Q   R 4381296 + 8 [dd]
>   8,16   1   770968     9.994796806  4580  M   R 4381296 + 8 [dd]
>   8,16   1   770969     9.994799419  4580  Q   R 4381304 + 8 [dd]
>   8,16   1   770970     9.994799919  4580  M   R 4381304 + 8 [dd]
>   8,16   1   770971     9.994802209  4580  Q   R 4381312 + 8 [dd]
>   8,16   1   770972     9.994802707  4580  M   R 4381312 + 8 [dd]
>   8,16   1   770973     9.994805174  4580  Q   R 4381320 + 8 [dd]
>   8,16   1   770974     9.994805672  4580  M   R 4381320 + 8 [dd]
>   8,16   1   770975     9.994807936  4580  Q   R 4381328 + 8 [dd]
>   8,16   1   770976     9.994808440  4580  M   R 4381328 + 8 [dd]
>   8,16   1   770977     9.994810715  4580  Q   R 4381336 + 8 [dd]
>   8,16   1   770978     9.994811242  4580  M   R 4381336 + 8 [dd]
>   8,16   1   770979     9.994813464  4580  Q   R 4381344 + 8 [dd]
>   8,16   1   770980     9.994813963  4580  M   R 4381344 + 8 [dd]
>   8,16   1   770981     9.994816205  4580  Q   R 4381352 + 8 [dd]
>   8,16   1   770982     9.994816776  4580  M   R 4381352 + 8 [dd]
>   8,16   1   770983     9.994819127  4580  Q   R 4381360 + 8 [dd]
>   8,16   1   770984     9.994819628  4580  M   R 4381360 + 8 [dd]
>   8,16   1   770985     9.994821810  4580  Q   R 4381368 + 8 [dd]
>   8,16   1   770986     9.994822315  4580  M   R 4381368 + 8 [dd]
>   8,16   1   770987     9.994824377  4580  Q   R 4381376 + 8 [dd]
>   8,16   1   770988     9.994824876  4580  M   R 4381376 + 8 [dd]
>   8,16   1   770989     9.994827067  4580  Q   R 4381384 + 8 [dd]
>   8,16   1   770990     9.994827565  4580  M   R 4381384 + 8 [dd]
>   8,16   1   770991     9.994829760  4580  Q   R 4381392 + 8 [dd]
>   8,16   1   770992     9.994830259  4580  M   R 4381392 + 8 [dd]
>   8,16   1   770993     9.994835522  4580  Q   R 4381400 + 8 [dd]
>   8,16   1   770994     9.994836021  4580  M   R 4381400 + 8 [dd]
>   8,16   1   770995     9.994838181  4580  Q   R 4381408 + 8 [dd]
>   8,16   1   770996     9.994838680  4580  M   R 4381408 + 8 [dd]
>   8,16   1   770997     9.994840857  4580  Q   R 4381416 + 8 [dd]
>   8,16   1   770998     9.994841369  4580  M   R 4381416 + 8 [dd]
>   8,16   1   770999     9.994843455  4580  Q   R 4381424 + 8 [dd]
>   8,16   1   771000     9.994843954  4580  M   R 4381424 + 8 [dd]
>   8,16   1   771001     9.994846149  4580  Q   R 4381432 + 8 [dd]
>   8,16   1   771002     9.994846662  4580  M   R 4381432 + 8 [dd]
>   8,16   1   771003     9.994849336  4580  I   R 4380928 + 512 (  306863) [dd]
>   8,16   1        0     9.994852010     0  m   N cfq4580S / insert_request
>   8,16   1   771004     9.994855251  4580  U   N [dd] 1
>   8,16   1        0     9.994857783     0  m   N cfq4580S / dispatch_insert
>   8,16   1        0     9.994861277     0  m   N cfq4580S / dispatched a request
>   8,16   1        0     9.994863066     0  m   N cfq4580S / activate rq, drv=1
>   8,16   1   771005     9.994863680  4580  D   R 4380928 + 512 (   14344) [dd]
>   8,16   1   771006     9.995909291     0  C   R 4380928 + 512 ( 1045611) [0]
>   8,16   1        0     9.995983219     0  m   N cfq4580S / complete rqnoidle 0
>   8,16   1        0     9.995986144     0  m   N cfq schedule dispatch
>   8,16   1   771007     9.996149764  4580  Q   R 4381440 + 8 [dd]
>   8,16   1   771008     9.996155872  4580  G   R 4381440 + 8 [dd]
>   8,16   1   771009     9.996157994  4580  P   N [dd]
>   8,16   1   771010     9.996161548  4580  Q   R 4381448 + 8 [dd]
>   8,16   1   771011     9.996163428  4580  M   R 4381448 + 8 [dd]
>   8,16   1   771012     9.996166608  4580  Q   R 4381456 + 8 [dd]
>   8,16   1   771013     9.996167115  4580  M   R 4381456 + 8 [dd]
>   8,16   1   771014     9.996169178  4580  Q   R 4381464 + 8 [dd]
>   8,16   1   771015     9.996169681  4580  M   R 4381464 + 8 [dd]
>   8,16   1   771016     9.996171779  4580  Q   R 4381472 + 8 [dd]
>   8,16   1   771017     9.996172277  4580  M   R 4381472 + 8 [dd]
>   8,16   1   771018     9.996174323  4580  Q   R 4381480 + 8 [dd]
>   8,16   1   771019     9.996174822  4580  M   R 4381480 + 8 [dd]
>   8,16   1   771020     9.996176942  4580  Q   R 4381488 + 8 [dd]
>   8,16   1   771021     9.996177446  4580  M   R 4381488 + 8 [dd]
>   8,16   1   771022     9.996179520  4580  Q   R 4381496 + 8 [dd]
>   8,16   1   771023     9.996180022  4580  M   R 4381496 + 8 [dd]
>   8,16   1   771024     9.996182075  4580  Q   R 4381504 + 8 [dd]
>   8,16   1   771025     9.996182574  4580  M   R 4381504 + 8 [dd]
>   8,16   1   771026     9.996189624  4580  Q   R 4381512 + 8 [dd]
>   8,16   1   771027     9.996190124  4580  M   R 4381512 + 8 [dd]
>   8,16   1   771028     9.996192318  4580  Q   R 4381520 + 8 [dd]
>   8,16   1   771029     9.996192819  4580  M   R 4381520 + 8 [dd]
>   8,16   1   771030     9.996194916  4580  Q   R 4381528 + 8 [dd]
>   8,16   1   771031     9.996195415  4580  M   R 4381528 + 8 [dd]
>   8,16   1   771032     9.996197792  4580  Q   R 4381536 + 8 [dd]
>   8,16   1   771033     9.996198291  4580  M   R 4381536 + 8 [dd]
>   8,16   1   771034     9.996200401  4580  Q   R 4381544 + 8 [dd]
>   8,16   1   771035     9.996200899  4580  M   R 4381544 + 8 [dd]
>   8,16   1   771036     9.996202991  4580  Q   R 4381552 + 8 [dd]
>   8,16   1   771037     9.996203495  4580  M   R 4381552 + 8 [dd]
>   8,16   1   771038     9.996205593  4580  Q   R 4381560 + 8 [dd]
>   8,16   1   771039     9.996206095  4580  M   R 4381560 + 8 [dd]
>   8,16   1   771040     9.996208179  4580  Q   R 4381568 + 8 [dd]
>   8,16   1   771041     9.996208678  4580  M   R 4381568 + 8 [dd]
>   8,16   1   771042     9.996210778  4580  Q   R 4381576 + 8 [dd]
>   8,16   1   771043     9.996211285  4580  M   R 4381576 + 8 [dd]
>   8,16   1   771044     9.996213428  4580  Q   R 4381584 + 8 [dd]
>   8,16   1   771045     9.996213932  4580  M   R 4381584 + 8 [dd]
>   8,16   1   771046     9.996217450  4580  Q   R 4381592 + 8 [dd]
>   8,16   1   771047     9.996217952  4580  M   R 4381592 + 8 [dd]
>   8,16   1   771048     9.996220131  4580  Q   R 4381600 + 8 [dd]
>   8,16   1   771049     9.996220644  4580  M   R 4381600 + 8 [dd]
>   8,16   1   771050     9.996222799  4580  Q   R 4381608 + 8 [dd]
>   8,16   1   771051     9.996223302  4580  M   R 4381608 + 8 [dd]
>   8,16   1   771052     9.996225537  4580  Q   R 4381616 + 8 [dd]
>   8,16   1   771053     9.996226039  4580  M   R 4381616 + 8 [dd]
>   8,16   1   771054     9.996231412  4580  Q   R 4381624 + 8 [dd]
>   8,16   1   771055     9.996231912  4580  M   R 4381624 + 8 [dd]
>   8,16   1   771056     9.996234006  4580  Q   R 4381632 + 8 [dd]
>   8,16   1   771057     9.996234505  4580  M   R 4381632 + 8 [dd]
>   8,16   1   771058     9.996236661  4580  Q   R 4381640 + 8 [dd]
>   8,16   1   771059     9.996237160  4580  M   R 4381640 + 8 [dd]
>   8,16   1   771060     9.996239325  4580  Q   R 4381648 + 8 [dd]
>   8,16   1   771061     9.996239829  4580  M   R 4381648 + 8 [dd]
>   8,16   1   771062     9.996242197  4580  Q   R 4381656 + 8 [dd]
>   8,16   1   771063     9.996242700  4580  M   R 4381656 + 8 [dd]
>   8,16   1   771064     9.996244888  4580  Q   R 4381664 + 8 [dd]
>   8,16   1   771065     9.996245388  4580  M   R 4381664 + 8 [dd]
>   8,16   1   771066     9.996247580  4580  Q   R 4381672 + 8 [dd]
>   8,16   1   771067     9.996248079  4580  M   R 4381672 + 8 [dd]
>   8,16   1   771068     9.996250234  4580  Q   R 4381680 + 8 [dd]
>   8,16   1   771069     9.996250735  4580  M   R 4381680 + 8 [dd]
>   8,16   1   771070     9.996252898  4580  Q   R 4381688 + 8 [dd]
>   8,16   1   771071     9.996253388  4580  M   R 4381688 + 8 [dd]
>   8,16   1   771072     9.996357598  4580  Q   R 4381696 + 8 [dd]
>   8,16   1   771073     9.996358355  4580  M   R 4381696 + 8 [dd]
>   8,16   1   771074     9.996360935  4580  Q   R 4381704 + 8 [dd]
>   8,16   1   771075     9.996361434  4580  M   R 4381704 + 8 [dd]
>   8,16   1   771076     9.996363662  4580  Q   R 4381712 + 8 [dd]
>   8,16   1   771077     9.996364166  4580  M   R 4381712 + 8 [dd]
>   8,16   1   771078     9.996366220  4580  Q   R 4381720 + 8 [dd]
>   8,16   1   771079     9.996366731  4580  M   R 4381720 + 8 [dd]
>   8,16   1   771080     9.996368930  4580  Q   R 4381728 + 8 [dd]
>   8,16   1   771081     9.996369429  4580  M   R 4381728 + 8 [dd]
>   8,16   1   771082     9.996375003  4580  Q   R 4381736 + 8 [dd]
>   8,16   1   771083     9.996375513  4580  M   R 4381736 + 8 [dd]
>   8,16   1   771084     9.996377664  4580  Q   R 4381744 + 8 [dd]
>   8,16   1   771085     9.996378380  4580  M   R 4381744 + 8 [dd]
>   8,16   1   771086     9.996380579  4580  Q   R 4381752 + 8 [dd]
>   8,16   1   771087     9.996381098  4580  M   R 4381752 + 8 [dd]
>   8,16   1   771088     9.996383260  4580  Q   R 4381760 + 8 [dd]
>   8,16   1   771089     9.996383759  4580  M   R 4381760 + 8 [dd]
>   8,16   1   771090     9.996385969  4580  Q   R 4381768 + 8 [dd]
>   8,16   1   771091     9.996386469  4580  M   R 4381768 + 8 [dd]
>   8,16   1   771092     9.996388658  4580  Q   R 4381776 + 8 [dd]
>   8,16   1   771093     9.996389162  4580  M   R 4381776 + 8 [dd]
>   8,16   1   771094     9.996391335  4580  Q   R 4381784 + 8 [dd]
>   8,16   1   771095     9.996391837  4580  M   R 4381784 + 8 [dd]
>   8,16   1   771096     9.996393920  4580  Q   R 4381792 + 8 [dd]
>   8,16   1   771097     9.996394419  4580  M   R 4381792 + 8 [dd]
>   8,16   1   771098     9.996396469  4580  Q   R 4381800 + 8 [dd]
>   8,16   1   771099     9.996396967  4580  M   R 4381800 + 8 [dd]
>   8,16   1   771100     9.996399181  4580  Q   R 4381808 + 8 [dd]
>   8,16   1   771101     9.996399684  4580  M   R 4381808 + 8 [dd]
>   8,16   1   771102     9.996402243  4580  Q   R 4381816 + 8 [dd]
>   8,16   1   771103     9.996402751  4580  M   R 4381816 + 8 [dd]
>   8,16   1   771104     9.996404942  4580  Q   R 4381824 + 8 [dd]
>   8,16   1   771105     9.996405441  4580  M   R 4381824 + 8 [dd]
>   8,16   1   771106     9.996407662  4580  Q   R 4381832 + 8 [dd]
>   8,16   1   771107     9.996408160  4580  M   R 4381832 + 8 [dd]
>   8,16   1   771108     9.996410409  4580  Q   R 4381840 + 8 [dd]
>   8,16   1   771109     9.996410910  4580  M   R 4381840 + 8 [dd]
>   8,16   1   771110     9.996416841  4580  Q   R 4381848 + 8 [dd]
>   8,16   1   771111     9.996417341  4580  M   R 4381848 + 8 [dd]
>   8,16   1   771112     9.996419551  4580  Q   R 4381856 + 8 [dd]
>   8,16   1   771113     9.996420050  4580  M   R 4381856 + 8 [dd]
>   8,16   1   771114     9.996425832  4580  Q   R 4381864 + 8 [dd]
>   8,16   1   771115     9.996426331  4580  M   R 4381864 + 8 [dd]
>   8,16   1   771116     9.996428600  4580  Q   R 4381872 + 8 [dd]
>   8,16   1   771117     9.996429101  4580  M   R 4381872 + 8 [dd]
>   8,16   1   771118     9.996431277  4580  Q   R 4381880 + 8 [dd]
>   8,16   1   771119     9.996431779  4580  M   R 4381880 + 8 [dd]
>   8,16   1   771120     9.996434026  4580  Q   R 4381888 + 8 [dd]
>   8,16   1   771121     9.996434525  4580  M   R 4381888 + 8 [dd]
>   8,16   1   771122     9.996436615  4580  Q   R 4381896 + 8 [dd]
>   8,16   1   771123     9.996437114  4580  M   R 4381896 + 8 [dd]
>   8,16   1   771124     9.996439353  4580  Q   R 4381904 + 8 [dd]
>   8,16   1   771125     9.996439864  4580  M   R 4381904 + 8 [dd]
>   8,16   1   771126     9.996441959  4580  Q   R 4381912 + 8 [dd]
>   8,16   1   771127     9.996442460  4580  M   R 4381912 + 8 [dd]
>   8,16   1   771128     9.996445027  4580  Q   R 4381920 + 8 [dd]
>   8,16   1   771129     9.996445526  4580  M   R 4381920 + 8 [dd]
>   8,16   1   771130     9.996447737  4580  Q   R 4381928 + 8 [dd]
>   8,16   1   771131     9.996448252  4580  M   R 4381928 + 8 [dd]
>   8,16   1   771132     9.996450536  4580  Q   R 4381936 + 8 [dd]
>   8,16   1   771133     9.996451040  4580  M   R 4381936 + 8 [dd]
>   8,16   1   771134     9.996453308  4580  Q   R 4381944 + 8 [dd]
>   8,16   1   771135     9.996453811  4580  M   R 4381944 + 8 [dd]
>   8,16   1   771136     9.996456417  4580  I   R 4381440 + 512 (  300545) [dd]
>   8,16   1        0     9.996458987     0  m   N cfq4580S / insert_request
>   8,16   1   771137     9.996462198  4580  U   N [dd] 1
>   8,16   1        0     9.996464756     0  m   N cfq4580S / dispatch_insert
>   8,16   1        0     9.996468170     0  m   N cfq4580S / dispatched a request
>   8,16   1        0     9.996469836     0  m   N cfq4580S / activate rq, drv=1
>   8,16   1   771138     9.996470460  4580  D   R 4381440 + 512 (   14043) [dd]
>   8,16   1   771139     9.997514205     0  C   R 4381440 + 512 ( 1043745) [0]
>   8,16   1        0     9.997587102     0  m   N cfq4580S / complete rqnoidle 0
>   8,16   1        0     9.997589870     0  m   N cfq schedule dispatch
>   8,16   1   771140     9.997751496  4580  Q   R 4381952 + 8 [dd]
>   8,16   1   771141     9.997758630  4580  G   R 4381952 + 8 [dd]
>   8,16   1   771142     9.997760814  4580  P   N [dd]
>   8,16   1   771143     9.997769413  4580  Q   R 4381960 + 8 [dd]
>   8,16   1   771144     9.997771473  4580  M   R 4381960 + 8 [dd]
>   8,16   1   771145     9.997774552  4580  Q   R 4381968 + 8 [dd]
>   8,16   1   771146     9.997775056  4580  M   R 4381968 + 8 [dd]
>   8,16   1   771147     9.997777142  4580  Q   R 4381976 + 8 [dd]
>   8,16   1   771148     9.997777717  4580  M   R 4381976 + 8 [dd]
>   8,16   1   771149     9.997779832  4580  Q   R 4381984 + 8 [dd]
>   8,16   1   771150     9.997780366  4580  M   R 4381984 + 8 [dd]
>   8,16   1   771151     9.997782454  4580  Q   R 4381992 + 8 [dd]
>   8,16   1   771152     9.997782953  4580  M   R 4381992 + 8 [dd]
>   8,16   1   771153     9.997785271  4580  Q   R 4382000 + 8 [dd]
>   8,16   1   771154     9.997785779  4580  M   R 4382000 + 8 [dd]
>   8,16   1   771155     9.997787937  4580  Q   R 4382008 + 8 [dd]
>   8,16   1   771156     9.997788434  4580  M   R 4382008 + 8 [dd]
>   8,16   1   771157     9.997790588  4580  Q   R 4382016 + 8 [dd]
>   8,16   1   771158     9.997791085  4580  M   R 4382016 + 8 [dd]
>   8,16   1   771159     9.997793251  4580  Q   R 4382024 + 8 [dd]
>   8,16   1   771160     9.997793954  4580  M   R 4382024 + 8 [dd]
>   8,16   1   771161     9.997796144  4580  Q   R 4382032 + 8 [dd]
>   8,16   1   771162     9.997796672  4580  M   R 4382032 + 8 [dd]
>   8,16   1   771163     9.997798737  4580  Q   R 4382040 + 8 [dd]
>   8,16   1   771164     9.997799238  4580  M   R 4382040 + 8 [dd]
>   8,16   1   771165     9.997801277  4580  Q   R 4382048 + 8 [dd]
>   8,16   1   771166     9.997801788  4580  M   R 4382048 + 8 [dd]
>   8,16   1   771167     9.997803925  4580  Q   R 4382056 + 8 [dd]
>   8,16   1   771168     9.997804439  4580  M   R 4382056 + 8 [dd]
>   8,16   1   771169     9.997806598  4580  Q   R 4382064 + 8 [dd]
>   8,16   1   771170     9.997807095  4580  M   R 4382064 + 8 [dd]
>   8,16   1   771171     9.997812761  4580  Q   R 4382072 + 8 [dd]
>   8,16   1   771172     9.997813262  4580  M   R 4382072 + 8 [dd]
>   8,16   1   771173     9.997815335  4580  Q   R 4382080 + 8 [dd]
>   8,16   1   771174     9.997815836  4580  M   R 4382080 + 8 [dd]
>   8,16   1   771175     9.997817949  4580  Q   R 4382088 + 8 [dd]
>   8,16   1   771176     9.997818448  4580  M   R 4382088 + 8 [dd]
>   8,16   1   771177     9.997820558  4580  Q   R 4382096 + 8 [dd]
>   8,16   1   771178     9.997821057  4580  M   R 4382096 + 8 [dd]
>   8,16   1   771179     9.997824698  4580  Q   R 4382104 + 8 [dd]
>   8,16   1   771180     9.997825197  4580  M   R 4382104 + 8 [dd]
>   8,16   1   771181     9.997827376  4580  Q   R 4382112 + 8 [dd]
>   8,16   1   771182     9.997827874  4580  M   R 4382112 + 8 [dd]
>   8,16   1   771183     9.997830036  4580  Q   R 4382120 + 8 [dd]
>   8,16   1   771184     9.997830538  4580  M   R 4382120 + 8 [dd]
>   8,16   1   771185     9.997832781  4580  Q   R 4382128 + 8 [dd]
>   8,16   1   771186     9.997833278  4580  M   R 4382128 + 8 [dd]
>   8,16   1   771187     9.997835542  4580  Q   R 4382136 + 8 [dd]
>   8,16   1   771188     9.997836044  4580  M   R 4382136 + 8 [dd]
>   8,16   1   771189     9.997838090  4580  Q   R 4382144 + 8 [dd]
>   8,16   1   771190     9.997838591  4580  M   R 4382144 + 8 [dd]
>   8,16   1   771191     9.997840646  4580  Q   R 4382152 + 8 [dd]
>   8,16   1   771192     9.997841147  4580  M   R 4382152 + 8 [dd]
>   8,16   1   771193     9.997843299  4580  Q   R 4382160 + 8 [dd]
>   8,16   1   771194     9.997843797  4580  M   R 4382160 + 8 [dd]
>   8,16   1   771195     9.997845909  4580  Q   R 4382168 + 8 [dd]
>   8,16   1   771196     9.997846408  4580  M   R 4382168 + 8 [dd]
>   8,16   1   771197     9.997848578  4580  Q   R 4382176 + 8 [dd]
>   8,16   1   771198     9.997849082  4580  M   R 4382176 + 8 [dd]
>   8,16   1   771199     9.997854848  4580  Q   R 4382184 + 8 [dd]
>   8,16   1   771200     9.997855350  4580  M   R 4382184 + 8 [dd]
>   8,16   1   771201     9.997857436  4580  Q   R 4382192 + 8 [dd]
>   8,16   1   771202     9.997857934  4580  M   R 4382192 + 8 [dd]
>   8,16   1   771203     9.997860278  4580  Q   R 4382200 + 8 [dd]
>   8,16   1   771204     9.997860779  4580  M   R 4382200 + 8 [dd]
>   8,16   1   771205     9.997964321  4580  Q   R 4382208 + 8 [dd]
>   8,16   1   771206     9.997965007  4580  M   R 4382208 + 8 [dd]
>   8,16   1   771207     9.997967619  4580  Q   R 4382216 + 8 [dd]
>   8,16   1   771208     9.997968161  4580  M   R 4382216 + 8 [dd]
>   8,16   1   771209     9.997970459  4580  Q   R 4382224 + 8 [dd]
>   8,16   1   771210     9.997970963  4580  M   R 4382224 + 8 [dd]
>   8,16   1   771211     9.997973081  4580  Q   R 4382232 + 8 [dd]
>   8,16   1   771212     9.997973580  4580  M   R 4382232 + 8 [dd]
>   8,16   1   771213     9.997975787  4580  Q   R 4382240 + 8 [dd]
>   8,16   1   771214     9.997976285  4580  M   R 4382240 + 8 [dd]
>   8,16   1   771215     9.997978452  4580  Q   R 4382248 + 8 [dd]
>   8,16   1   771216     9.997978953  4580  M   R 4382248 + 8 [dd]
>   8,16   1   771217     9.997981249  4580  Q   R 4382256 + 8 [dd]
>   8,16   1   771218     9.997981751  4580  M   R 4382256 + 8 [dd]
>   8,16   1   771219     9.997983917  4580  Q   R 4382264 + 8 [dd]
>   8,16   1   771220     9.997984418  4580  M   R 4382264 + 8 [dd]
>   8,16   1   771221     9.997986611  4580  Q   R 4382272 + 8 [dd]
>   8,16   1   771222     9.997987121  4580  M   R 4382272 + 8 [dd]
>   8,16   1   771223     9.997989204  4580  Q   R 4382280 + 8 [dd]
>   8,16   1   771224     9.997989705  4580  M   R 4382280 + 8 [dd]
>   8,16   1   771225     9.997991904  4580  Q   R 4382288 + 8 [dd]
>   8,16   1   771226     9.997992402  4580  M   R 4382288 + 8 [dd]
>   8,16   1   771227     9.997998036  4580  Q   R 4382296 + 8 [dd]
>   8,16   1   771228     9.997998579  4580  M   R 4382296 + 8 [dd]
>   8,16   1   771229     9.998000634  4580  Q   R 4382304 + 8 [dd]
>   8,16   1   771230     9.998001136  4580  M   R 4382304 + 8 [dd]
>   8,16   1   771231     9.998003255  4580  Q   R 4382312 + 8 [dd]
>   8,16   1   771232     9.998003774  4580  M   R 4382312 + 8 [dd]
>   8,16   1   771233     9.998005975  4580  Q   R 4382320 + 8 [dd]
>   8,16   1   771234     9.998006476  4580  M   R 4382320 + 8 [dd]
>   8,16   1   771235     9.998008972  4580  Q   R 4382328 + 8 [dd]
>   8,16   1   771236     9.998009529  4580  M   R 4382328 + 8 [dd]
>   8,16   1   771237     9.998011779  4580  Q   R 4382336 + 8 [dd]
>   8,16   1   771238     9.998012277  4580  M   R 4382336 + 8 [dd]
>   8,16   1   771239     9.998014476  4580  Q   R 4382344 + 8 [dd]
>   8,16   1   771240     9.998014974  4580  M   R 4382344 + 8 [dd]
>   8,16   1   771241     9.998017401  4580  Q   R 4382352 + 8 [dd]
>   8,16   1   771242     9.998017899  4580  M   R 4382352 + 8 [dd]
>   8,16   1   771243     9.998020091  4580  Q   R 4382360 + 8 [dd]
>   8,16   1   771244     9.998020588  4580  M   R 4382360 + 8 [dd]
>   8,16   1   771245     9.998022782  4580  Q   R 4382368 + 8 [dd]
>   8,16   1   771246     9.998023389  4580  M   R 4382368 + 8 [dd]
>   8,16   1   771247     9.998025647  4580  Q   R 4382376 + 8 [dd]
>   8,16   1   771248     9.998026180  4580  M   R 4382376 + 8 [dd]
>   8,16   1   771249     9.998028480  4580  Q   R 4382384 + 8 [dd]
>   8,16   1   771250     9.998029020  4580  M   R 4382384 + 8 [dd]
>   8,16   1   771251     9.998031205  4580  Q   R 4382392 + 8 [dd]
>   8,16   1   771252     9.998031706  4580  M   R 4382392 + 8 [dd]
>   8,16   1   771253     9.998033865  4580  Q   R 4382400 + 8 [dd]
>   8,16   1   771254     9.998034366  4580  M   R 4382400 + 8 [dd]
>   8,16   1   771255     9.998040073  4580  Q   R 4382408 + 8 [dd]
>   8,16   1   771256     9.998040571  4580  M   R 4382408 + 8 [dd]
>   8,16   1   771257     9.998042647  4580  Q   R 4382416 + 8 [dd]
>   8,16   1   771258     9.998043150  4580  M   R 4382416 + 8 [dd]
>   8,16   1   771259     9.998045325  4580  Q   R 4382424 + 8 [dd]
>   8,16   1   771260     9.998045826  4580  M   R 4382424 + 8 [dd]
>   8,16   1   771261     9.998047931  4580  Q   R 4382432 + 8 [dd]
>   8,16   1   771262     9.998048432  4580  M   R 4382432 + 8 [dd]
>   8,16   1   771263     9.998050535  4580  Q   R 4382440 + 8 [dd]
>   8,16   1   771264     9.998051038  4580  M   R 4382440 + 8 [dd]
>   8,16   1   771265     9.998056985  4580  Q   R 4382448 + 8 [dd]
>   8,16   1   771266     9.998057500  4580  M   R 4382448 + 8 [dd]
>   8,16   1   771267     9.998059576  4580  Q   R 4382456 + 8 [dd]
>   8,16   1   771268     9.998060077  4580  M   R 4382456 + 8 [dd]
>   8,16   1   771269     9.998062341  4580  I   R 4381952 + 512 (  303711) [dd]
>   8,16   1        0     9.998064881     0  m   N cfq4580S / insert_request
>   8,16   1   771270     9.998068490  4580  U   N [dd] 1
>   8,16   1        0     9.998071031     0  m   N cfq4580S / dispatch_insert
>   8,16   1        0     9.998074991     0  m   N cfq4580S / dispatched a request
>   8,16   1        0     9.998076645     0  m   N cfq4580S / activate rq, drv=1
>   8,16   1   771271     9.998077269  4580  D   R 4381952 + 512 (   14928) [dd]
>   8,16   1   771272     9.999120396     0  C   R 4381952 + 512 ( 1043127) [0]
>   8,16   1        0     9.999192302     0  m   N cfq4580S / complete rqnoidle 0
>   8,16   1        0     9.999195361     0  m   N cfq schedule dispatch
>   8,16   1   771273     9.999356099  4580  Q   R 4382464 + 8 [dd]
>   8,16   1   771274     9.999362719  4580  G   R 4382464 + 8 [dd]
>   8,16   1   771275     9.999364795  4580  P   N [dd]
>   8,16   1   771276     9.999368199  4580  Q   R 4382472 + 8 [dd]
>   8,16   1   771277     9.999370072  4580  M   R 4382472 + 8 [dd]
>   8,16   1   771278     9.999373166  4580  Q   R 4382480 + 8 [dd]
>   8,16   1   771279     9.999373710  4580  M   R 4382480 + 8 [dd]
>   8,16   1   771280     9.999375830  4580  Q   R 4382488 + 8 [dd]
>   8,16   1   771281     9.999376334  4580  M   R 4382488 + 8 [dd]
>   8,16   1   771282     9.999378433  4580  Q   R 4382496 + 8 [dd]
>   8,16   1   771283     9.999378932  4580  M   R 4382496 + 8 [dd]
>   8,16   1   771284     9.999380973  4580  Q   R 4382504 + 8 [dd]
>   8,16   1   771285     9.999381477  4580  M   R 4382504 + 8 [dd]
>   8,16   1   771286     9.999383602  4580  Q   R 4382512 + 8 [dd]
>   8,16   1   771287     9.999384101  4580  M   R 4382512 + 8 [dd]
>   8,16   1   771288     9.999391190  4580  Q   R 4382520 + 8 [dd]
>   8,16   1   771289     9.999391690  4580  M   R 4382520 + 8 [dd]
>   8,16   1   771290     9.999393850  4580  Q   R 4382528 + 8 [dd]
>   8,16   1   771291     9.999394351  4580  M   R 4382528 + 8 [dd]
>   8,16   1   771292     9.999396618  4580  Q   R 4382536 + 8 [dd]
>   8,16   1   771293     9.999397190  4580  M   R 4382536 + 8 [dd]
>   8,16   1   771294     9.999399323  4580  Q   R 4382544 + 8 [dd]
>   8,16   1   771295     9.999399864  4580  M   R 4382544 + 8 [dd]
>   8,16   1   771296     9.999401953  4580  Q   R 4382552 + 8 [dd]
>   8,16   1   771297     9.999402472  4580  M   R 4382552 + 8 [dd]
>   8,16   1   771298     9.999404563  4580  Q   R 4382560 + 8 [dd]
>   8,16   1   771299     9.999405057  4580  M   R 4382560 + 8 [dd]
>   8,16   1   771300     9.999407109  4580  Q   R 4382568 + 8 [dd]
>   8,16   1   771301     9.999407611  4580  M   R 4382568 + 8 [dd]
>   8,16   1   771302     9.999409677  4580  Q   R 4382576 + 8 [dd]
>   8,16   1   771303     9.999410176  4580  M   R 4382576 + 8 [dd]
>   8,16   1   771304     9.999412226  4580  Q   R 4382584 + 8 [dd]
>   8,16   1   771305     9.999412734  4580  M   R 4382584 + 8 [dd]
>   8,16   1   771306     9.999416340  4580  Q   R 4382592 + 8 [dd]
>   8,16   1   771307     9.999416862  4580  M   R 4382592 + 8 [dd]
>   8,16   1   771308     9.999419187  4580  Q   R 4382600 + 8 [dd]
>   8,16   1   771309     9.999419706  4580  M   R 4382600 + 8 [dd]
>   8,16   1   771310     9.999421977  4580  Q   R 4382608 + 8 [dd]
>   8,16   1   771311     9.999422503  4580  M   R 4382608 + 8 [dd]
>   8,16   1   771312     9.999424582  4580  Q   R 4382616 + 8 [dd]
>   8,16   1   771313     9.999425120  4580  M   R 4382616 + 8 [dd]
>   8,16   1   771314     9.999427292  4580  Q   R 4382624 + 8 [dd]
>   8,16   1   771315     9.999427805  4580  M   R 4382624 + 8 [dd]
>   8,16   1   771316     9.999433255  4580  Q   R 4382632 + 8 [dd]
>   8,16   1   771317     9.999433766  4580  M   R 4382632 + 8 [dd]
>   8,16   1   771318     9.999435901  4580  Q   R 4382640 + 8 [dd]
>   8,16   1   771319     9.999436412  4580  M   R 4382640 + 8 [dd]
>   8,16   1   771320     9.999438504  4580  Q   R 4382648 + 8 [dd]
>   8,16   1   771321     9.999439243  4580  M   R 4382648 + 8 [dd]
>   8,16   1   771322     9.999441418  4580  Q   R 4382656 + 8 [dd]
>   8,16   1   771323     9.999441949  4580  M   R 4382656 + 8 [dd]
>   8,16   1   771324     9.999444183  4580  Q   R 4382664 + 8 [dd]
>   8,16   1   771325     9.999444698  4580  M   R 4382664 + 8 [dd]
>   8,16   1   771326     9.999446894  4580  Q   R 4382672 + 8 [dd]
>   8,16   1   771327     9.999447411  4580  M   R 4382672 + 8 [dd]
>   8,16   1   771328     9.999449485  4580  Q   R 4382680 + 8 [dd]
>   8,16   1   771329     9.999449997  4580  M   R 4382680 + 8 [dd]
>   8,16   1   771330     9.999452055  4580  Q   R 4382688 + 8 [dd]
>   8,16   1   771331     9.999452566  4580  M   R 4382688 + 8 [dd]
>   8,16   1   771332     9.999454606  4580  Q   R 4382696 + 8 [dd]
>   8,16   1   771333     9.999455154  4580  M   R 4382696 + 8 [dd]
>   8,16   1   771334     9.999457363  4580  Q   R 4382704 + 8 [dd]
>   8,16   1   771335     9.999457875  4580  M   R 4382704 + 8 [dd]
>   8,16   1   771336     9.999460045  4580  Q   R 4382712 + 8 [dd]
>   8,16   1   771337     9.999460561  4580  M   R 4382712 + 8 [dd]
>   8,16   1   771338     9.999564615  4580  Q   R 4382720 + 8 [dd]
>   8,16   1   771339     9.999565347  4580  M   R 4382720 + 8 [dd]
>   8,16   1   771340     9.999568401  4580  Q   R 4382728 + 8 [dd]
>   8,16   1   771341     9.999568908  4580  M   R 4382728 + 8 [dd]
>   8,16   1   771342     9.999571276  4580  Q   R 4382736 + 8 [dd]
>   8,16   1   771343     9.999571792  4580  M   R 4382736 + 8 [dd]
>   8,16   1   771344     9.999577396  4580  Q   R 4382744 + 8 [dd]
>   8,16   1   771345     9.999578003  4580  M   R 4382744 + 8 [dd]
>   8,16   1   771346     9.999580231  4580  Q   R 4382752 + 8 [dd]
>   8,16   1   771347     9.999580749  4580  M   R 4382752 + 8 [dd]
>   8,16   1   771348     9.999583114  4580  Q   R 4382760 + 8 [dd]
>   8,16   1   771349     9.999583625  4580  M   R 4382760 + 8 [dd]
>   8,16   1   771350     9.999585875  4580  Q   R 4382768 + 8 [dd]
>   8,16   1   771351     9.999586374  4580  M   R 4382768 + 8 [dd]
>   8,16   1   771352     9.999588531  4580  Q   R 4382776 + 8 [dd]
>   8,16   1   771353     9.999589032  4580  M   R 4382776 + 8 [dd]
>   8,16   1   771354     9.999591214  4580  Q   R 4382784 + 8 [dd]
>   8,16   1   771355     9.999591714  4580  M   R 4382784 + 8 [dd]
>   8,16   1   771356     9.999593930  4580  Q   R 4382792 + 8 [dd]
>   8,16   1   771357     9.999594442  4580  M   R 4382792 + 8 [dd]
>   8,16   1   771358     9.999596696  4580  Q   R 4382800 + 8 [dd]
>   8,16   1   771359     9.999597190  4580  M   R 4382800 + 8 [dd]
>   8,16   1   771360     9.999599406  4580  Q   R 4382808 + 8 [dd]
>   8,16   1   771361     9.999599907  4580  M   R 4382808 + 8 [dd]
>   8,16   1   771362     9.999602166  4580  Q   R 4382816 + 8 [dd]
>   8,16   1   771363     9.999602667  4580  M   R 4382816 + 8 [dd]
>   8,16   1   771364     9.999604755  4580  Q   R 4382824 + 8 [dd]
>   8,16   1   771365     9.999605259  4580  M   R 4382824 + 8 [dd]
>   8,16   1   771366     9.999607486  4580  Q   R 4382832 + 8 [dd]
>   8,16   1   771367     9.999607978  4580  M   R 4382832 + 8 [dd]
>   8,16   1   771368     9.999610062  4580  Q   R 4382840 + 8 [dd]
>   8,16   1   771369     9.999610564  4580  M   R 4382840 + 8 [dd]
>   8,16   1   771370     9.999612914  4580  Q   R 4382848 + 8 [dd]
>   8,16   1   771371     9.999613418  4580  M   R 4382848 + 8 [dd]
>   8,16   1   771372     9.999618806  4580  Q   R 4382856 + 8 [dd]
>   8,16   1   771373     9.999619308  4580  M   R 4382856 + 8 [dd]
>   8,16   1   771374     9.999621579  4580  Q   R 4382864 + 8 [dd]
>   8,16   1   771375     9.999622092  4580  M   R 4382864 + 8 [dd]
>   8,16   1   771376     9.999624260  4580  Q   R 4382872 + 8 [dd]
>   8,16   1   771377     9.999624779  4580  M   R 4382872 + 8 [dd]
>   8,16   1   771378     9.999627506  4580  Q   R 4382880 + 8 [dd]
>   8,16   1   771379     9.999628012  4580  M   R 4382880 + 8 [dd]
>   8,16   1   771380     9.999630135  4580  Q   R 4382888 + 8 [dd]
>   8,16   1   771381     9.999630635  4580  M   R 4382888 + 8 [dd]
>   8,16   1   771382     9.999632876  4580  Q   R 4382896 + 8 [dd]
>   8,16   1   771383     9.999633378  4580  M   R 4382896 + 8 [dd]
>   8,16   1   771384     9.999635541  4580  Q   R 4382904 + 8 [dd]
>   8,16   1   771385     9.999636043  4580  M   R 4382904 + 8 [dd]
>   8,16   1   771386     9.999638232  4580  Q   R 4382912 + 8 [dd]
>   8,16   1   771387     9.999638736  4580  M   R 4382912 + 8 [dd]
>   8,16   1   771388     9.999640940  4580  Q   R 4382920 + 8 [dd]
>   8,16   1   771389     9.999641440  4580  M   R 4382920 + 8 [dd]
>   8,16   1   771390     9.999643644  4580  Q   R 4382928 + 8 [dd]
>   8,16   1   771391     9.999644157  4580  M   R 4382928 + 8 [dd]
>   8,16   1   771392     9.999646440  4580  Q   R 4382936 + 8 [dd]
>   8,16   1   771393     9.999646960  4580  M   R 4382936 + 8 [dd]
>   8,16   1   771394     9.999649193  4580  Q   R 4382944 + 8 [dd]
>   8,16   1   771395     9.999649697  4580  M   R 4382944 + 8 [dd]
>   8,16   1   771396     9.999651771  4580  Q   R 4382952 + 8 [dd]
>   8,16   1   771397     9.999652272  4580  M   R 4382952 + 8 [dd]
>   8,16   1   771398     9.999654476  4580  Q   R 4382960 + 8 [dd]
>   8,16   1   771399     9.999654975  4580  M   R 4382960 + 8 [dd]
>   8,16   1   771400     9.999660256  4580  Q   R 4382968 + 8 [dd]
>   8,16   1   771401     9.999660760  4580  M   R 4382968 + 8 [dd]
>   8,16   1   771402     9.999663197  4580  I   R 4382464 + 512 (  300478) [dd]
>   8,16   1        0     9.999665743     0  m   N cfq4580S / insert_request
>   8,16   1   771403     9.999669190  4580  U   N [dd] 1
>   8,16   1        0     9.999672048     0  m   N cfq4580S / dispatch_insert
>   8,16   1        0     9.999675568     0  m   N cfq4580S / dispatched a request
>   8,16   1        0     9.999677281     0  m   N cfq4580S / activate rq, drv=1
>   8,16   1   771404     9.999677892  4580  D   R 4382464 + 512 (   14695) [dd]
> CPU0 (sdb):
>  Reads Queued:       20608,    82432KiB	 Writes Queued:           0,        0KiB
>  Read Dispatches:      322,    82432KiB	 Write Dispatches:        0,        0KiB
>  Reads Requeued:         0		 Writes Requeued:         0
>  Reads Completed:       22,     5632KiB	 Writes Completed:        0,        0KiB
>  Read Merges:        20286,    81144KiB	 Write Merges:            0,        0KiB
>  Read depth:             1        	 Write depth:             0
>  IO unplugs:           322        	 Timer unplugs:           0
> CPU1 (sdb):
>  Reads Queued:      371058,     1484MiB	 Writes Queued:           0,        0KiB
>  Read Dispatches:     5798,     1484MiB	 Write Dispatches:        0,        0KiB
>  Reads Requeued:         0		 Writes Requeued:         0
>  Reads Completed:     6097,     1560MiB	 Writes Completed:        0,        0KiB
>  Read Merges:       365261,     1461MiB	 Write Merges:            0,        0KiB
>  Read depth:             1        	 Write depth:             0
>  IO unplugs:          5798        	 Timer unplugs:           0
> 
> Total (sdb):
>  Reads Queued:      391666,     1566MiB	 Writes Queued:           0,        0KiB
>  Read Dispatches:     6120,     1566MiB	 Write Dispatches:        0,        0KiB
>  Reads Requeued:         0		 Writes Requeued:         0
>  Reads Completed:     6119,     1566MiB	 Writes Completed:        0,        0KiB
>  Read Merges:       385547,     1542MiB	 Write Merges:            0,        0KiB
>  IO unplugs:          6120        	 Timer unplugs:           0
> 
> Throughput (R/W): 156662KiB/s / 0KiB/s
> Events (sdb): 851428 entries
> Skips: 0 forward (0 -   0.0%)
> Input file sdb.blktrace.0 added
> Input file sdb.blktrace.1 added
> 

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

> - IO size is 256KB (which is not a problem in itself)
> 
> - The dispatch/complete pattern is
> 
>         submit IO for range 1
>         complete IO for range 1
>         <dd busy, disk idle>
>         submit IO for range 2
>         complete IO for range 2
> 
>   So we have periods that no IO is in flight at all, which leads to
>   under-utilized disk (which should show up in iostat as <100% disk util)
> 
> # grep '[DC]' blktrace
> 
>   8,16   1   770606     9.990039807  4580  D   R 4379392 + 512 (   14826) [dd]
>   8,16   1   770607     9.991083069     0  C   R 4379392 + 512 ( 1043262) [0]
> 
>   8,16   1   770739     9.991647434  4580  D   R 4379904 + 512 (   14433) [dd]
>   8,16   1   770740     9.992693317     0  C   R 4379904 + 512 ( 1045883) [0]
> 
>   8,16   1   770872     9.993256451  4580  D   R 4380416 + 512 (   14539) [dd]
>   8,16   1   770873     9.994299156     0  C   R 4380416 + 512 ( 1042705) [0]
> 
>   8,16   1   771005     9.994863680  4580  D   R 4380928 + 512 (   14344) [dd]
>   8,16   1   771006     9.995909291     0  C   R 4380928 + 512 ( 1045611) [0]
> 
>   8,16   1   771138     9.996470460  4580  D   R 4381440 + 512 (   14043) [dd]
>   8,16   1   771139     9.997514205     0  C   R 4381440 + 512 ( 1043745) [0]
> 
>   8,16   1   771271     9.998077269  4580  D   R 4381952 + 512 (   14928) [dd]
>   8,16   1   771272     9.999120396     0  C   R 4381952 + 512 ( 1043127) [0]

Some better pattern for comparison:

  8,0    1    42940     0.990199812  4084  D   R 1432808 + 256 [dd]
  8,0    1    42941     0.990858326     0  C   R 1432552 + 256 [0]
  8,0    1    43009     0.991192107  4084  D   R 1433064 + 256 [dd]
  8,0    3    14691     0.991853319     0  C   R 1432808 + 256 [0]
  8,0    3    14759     0.992189451  4084  D   R 1433320 + 256 [dd]
  8,0    1    43010     0.994473159     0  C   R 1433064 + 256 [0]

Where the life time for requests 1432808, 1432552, 1433064, ...
are *interleaved* so that the disk always has something to do.

Thanks,
Fengguang

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:

> Looks the 2.6.39 block plug introduces some latency here. deleting
> blk_start_plug/blk_finish_plug in generic_file_aio_read seems
> workaround
> the issue. The plug seems not good for sequential IO, because readahead
> code already has plug and has fine grained control.

Why not remove the generic_file_aio_read() plug completely? It
actually prevents unplugging immediately after the readahead IO is
submitted and in turn stalls the IO pipeline as showed by Eric's
blktrace data.

Eric, will you test this patch? Thank you.

--- linux.orig/mm/filemap.c	2012-01-30 22:27:11.000000000 +0800
+++ linux/mm/filemap.c	2012-01-30 22:27:25.000000000 +0800
@@ -1414,15 +1414,12 @@ generic_file_aio_read(struct kiocb *iocb
 	unsigned long seg = 0;
 	size_t count;
 	loff_t *ppos = &iocb->ki_pos;
-	struct blk_plug plug;
 
 	count = 0;
 	retval = generic_segment_checks(iov, &nr_segs, &count, VERIFY_WRITE);
 	if (retval)
 		return retval;
 
-	blk_start_plug(&plug);
-
 	/* coalesce the iovecs and go direct-to-BIO for O_DIRECT */
 	if (filp->f_flags & O_DIRECT) {
 		loff_t size;
@@ -1495,7 +1492,6 @@ generic_file_aio_read(struct kiocb *iocb
 			break;
 	}
 out:
-	blk_finish_plug(&plug);
 	return retval;
 }
 EXPORT_SYMBOL(generic_file_aio_read);

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Sun, Jan 29, 2012 at 11:03:00AM +0100, Eric Dumazet wrote:

> Very strange, my bissection ended on following commit :
> 
> commit 805f6b5e1cbfedfb9b3d354013e7f4b13a79270f
> Author: Tao Ma <boyu.mt@taobao.com>
> Date:   Fri Mar 11 20:11:59 2011 +0100
> 
>     blktrace: Use rq->cmd_flags directly in blk_add_trace_rq.
> 
> This makes no sense.

Yeah, commit 55602dd66f535 ("fs: make generic file read/write
functions plug") looks like the real root cause and should probably
be reverted as a whole. Buffered read/write syscalls are not directly
tied to IO after all (readahead/writeback have the right plug points).

Thanks,
Fengguang

> hdparm uses 2MB block reads, so read_ahead (128KB) is too small for best
> perf
> 
> # cat /sys/class/block/sda/queue/read_ahead_kb
> 128
> 
> # dd if=/dev/sda of=/dev/null bs=128k 
> ^C
> 63744+0 enregistrements lus
> 63743+0 enregistrements écrits
> 8354922496 octets (8,4 GB) copiés, 39,975 s, 209 MB/s
> 
> # hdparm -t /dev/sda
> 
> /dev/sda:
>  Timing buffered disk reads: 510 MB in  3.00 seconds = 169.75 MB/sec
> 
> # uname -a
> Linux edumazet-laptop 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20
> 17:23:00 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> 
> 

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit :
> On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:
> 
> > Looks the 2.6.39 block plug introduces some latency here. deleting
> > blk_start_plug/blk_finish_plug in generic_file_aio_read seems
> > workaround
> > the issue. The plug seems not good for sequential IO, because readahead
> > code already has plug and has fine grained control.
> 
> Why not remove the generic_file_aio_read() plug completely? It
> actually prevents unplugging immediately after the readahead IO is
> submitted and in turn stalls the IO pipeline as showed by Eric's
> blktrace data.
> 
> Eric, will you test this patch? Thank you.
> 
> --- linux.orig/mm/filemap.c	2012-01-30 22:27:11.000000000 +0800
> +++ linux/mm/filemap.c	2012-01-30 22:27:25.000000000 +0800
> @@ -1414,15 +1414,12 @@ generic_file_aio_read(struct kiocb *iocb
>  	unsigned long seg = 0;
>  	size_t count;
>  	loff_t *ppos = &iocb->ki_pos;
> -	struct blk_plug plug;
>  
>  	count = 0;
>  	retval = generic_segment_checks(iov, &nr_segs, &count, VERIFY_WRITE);
>  	if (retval)
>  		return retval;
>  
> -	blk_start_plug(&plug);
> -
>  	/* coalesce the iovecs and go direct-to-BIO for O_DIRECT */
>  	if (filp->f_flags & O_DIRECT) {
>  		loff_t size;
> @@ -1495,7 +1492,6 @@ generic_file_aio_read(struct kiocb *iocb
>  			break;
>  	}
>  out:
> -	blk_finish_plug(&plug);
>  	return retval;
>  }
>  EXPORT_SYMBOL(generic_file_aio_read);


Excellent results

# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=2M count=2048
2048+0 enregistrements lus
2048+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 16,2309 s, 265 MB/s

# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=128k count=32768
32768+0 enregistrements lus
32768+0 enregistrements écrits
4294967296 octets (4,3 GB) copiés, 16,3153 s, 263 MB/s

# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads: 752 MB in  3.00 seconds = 250.62 MB/sec

Re: Bad SSD performance with recent kernels

[Vivek Goyal]

On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote:
> Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit :
> > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:
> > 
> > > Looks the 2.6.39 block plug introduces some latency here. deleting
> > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems
> > > workaround
> > > the issue. The plug seems not good for sequential IO, because readahead
> > > code already has plug and has fine grained control.
> > 
> > Why not remove the generic_file_aio_read() plug completely? It
> > actually prevents unplugging immediately after the readahead IO is
> > submitted and in turn stalls the IO pipeline as showed by Eric's
> > blktrace data.
> > 
> > Eric, will you test this patch? Thank you.

Can you please run the blktrace again with this patch applied. I am curious
to see how does traffic pattern look like now.

In your previous trace, there were so many small 8 sector requests which
were merged into 512 sector requests before dispatching to disk. (I am
not sure why those requests are not bigger. Shouldn't readahead logic
submit a bigger request?) Now with plug/unplug logic removed, I am assuming
we should be doing less merging and dispatching more smaller requests. May be
that is helping and cutting down on disk idling time.

In previous logs, 512 sector request seems to be taking around 1ms to
complete after dispatch. In between requests disk seems to be idle
for around .5 to .6 ms. Out of this .3 ms seems to be gone in just
coming up with new request after completion of previous one and another
.3ms seems to be consumed in merging the smaller IOs. So if we don't wait
for merging, it should keep disk busier for .3ms more which is 30% of time
it takes to complete 512 sector request. So theoritically it can give
30% boost for this workload. (Assuming request size will not impact the
disk throughput very severely).

Anyway, some blktrace data will shed some light..

Thanks
Vivek

Re: Bad SSD performance with recent kernels

[Shaohua Li]

On Mon, 2012-01-30 at 17:26 -0500, Vivek Goyal wrote:
> On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote:
> > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit :
> > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:
> > > 
> > > > Looks the 2.6.39 block plug introduces some latency here. deleting
> > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems
> > > > workaround
> > > > the issue. The plug seems not good for sequential IO, because readahead
> > > > code already has plug and has fine grained control.
> > > 
> > > Why not remove the generic_file_aio_read() plug completely? It
> > > actually prevents unplugging immediately after the readahead IO is
> > > submitted and in turn stalls the IO pipeline as showed by Eric's
> > > blktrace data.
> > > 
> > > Eric, will you test this patch? Thank you.
> 
> Can you please run the blktrace again with this patch applied. I am curious
> to see how does traffic pattern look like now.
> 
> In your previous trace, there were so many small 8 sector requests which
> were merged into 512 sector requests before dispatching to disk. (I am
> not sure why those requests are not bigger. Shouldn't readahead logic
> submit a bigger request?) Now with plug/unplug logic removed, I am assuming
> we should be doing less merging and dispatching more smaller requests. May be
> that is helping and cutting down on disk idling time.
> 
> In previous logs, 512 sector request seems to be taking around 1ms to
> complete after dispatch. In between requests disk seems to be idle
> for around .5 to .6 ms. Out of this .3 ms seems to be gone in just
> coming up with new request after completion of previous one and another
> .3ms seems to be consumed in merging the smaller IOs. So if we don't wait
> for merging, it should keep disk busier for .3ms more which is 30% of time
> it takes to complete 512 sector request. So theoritically it can give
> 30% boost for this workload. (Assuming request size will not impact the
> disk throughput very severely).
> 
> Anyway, some blktrace data will shed some light..
yep, I suspect plug merges big request too (iostat shows it too), that's
why I only think delete the plug in generic_file_aio_read as a
workaround. I still thought readahead has something to do here. I
observed the async readahead does readahead (A, A + 2M), and follows (A
+128k, A+2M), (A+256k, A+2M) ..., the later readahead doesn't work
because we already have (A, A+2M) in memory at that time. Anyway, I can
reproduce the issue, will play with it more today.

Re: Bad SSD performance with recent kernels

[Wu Fengguang]

On Tue, Jan 31, 2012 at 08:14:19AM +0800, Li, Shaohua wrote:
> On Mon, 2012-01-30 at 17:26 -0500, Vivek Goyal wrote:
> > On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote:
> > > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit :
> > > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:
> > > > 
> > > > > Looks the 2.6.39 block plug introduces some latency here. deleting
> > > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems
> > > > > workaround
> > > > > the issue. The plug seems not good for sequential IO, because readahead
> > > > > code already has plug and has fine grained control.
> > > > 
> > > > Why not remove the generic_file_aio_read() plug completely? It
> > > > actually prevents unplugging immediately after the readahead IO is
> > > > submitted and in turn stalls the IO pipeline as showed by Eric's
> > > > blktrace data.
> > > > 
> > > > Eric, will you test this patch? Thank you.
> > 
> > Can you please run the blktrace again with this patch applied. I am curious
> > to see how does traffic pattern look like now.
> > 
> > In your previous trace, there were so many small 8 sector requests which
> > were merged into 512 sector requests before dispatching to disk. (I am
> > not sure why those requests are not bigger. Shouldn't readahead logic
> > submit a bigger request?) Now with plug/unplug logic removed, I am assuming
> > we should be doing less merging and dispatching more smaller requests. May be
> > that is helping and cutting down on disk idling time.
> > 
> > In previous logs, 512 sector request seems to be taking around 1ms to
> > complete after dispatch. In between requests disk seems to be idle
> > for around .5 to .6 ms. Out of this .3 ms seems to be gone in just
> > coming up with new request after completion of previous one and another
> > .3ms seems to be consumed in merging the smaller IOs. So if we don't wait
> > for merging, it should keep disk busier for .3ms more which is 30% of time
> > it takes to complete 512 sector request. So theoritically it can give
> > 30% boost for this workload. (Assuming request size will not impact the
> > disk throughput very severely).
> > 
> > Anyway, some blktrace data will shed some light..
> yep, I suspect plug merges big request too (iostat shows it too), that's
> why I only think delete the plug in generic_file_aio_read as a
> workaround.

It's good to merge requests inside the same readahead window. However
I don't think readahead window A should be merged with B at the cost
of delaying A for some time, which will break the pipeline. If larger
IO is desirable, we can do so by increasing the readahead size.

> I still thought readahead has something to do here. I
> observed the async readahead does readahead (A, A + 2M), and follows (A
> +128k, A+2M), (A+256k, A+2M) ..., the later readahead doesn't work
> because we already have (A, A+2M) in memory at that time. Anyway, I can
> reproduce the issue, will play with it more today.

How do you observe that? I don't think that readahead pattern is
possible. However I do see such _read_ patterns.

Thanks,
Fengguang

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le 30 janvier 2012 23:26, Vivek Goyal <vgoyal@redhat.com> a écrit :
>
> Can you please run the blktrace again with this patch applied. I am curious
> to see how does traffic pattern look like now.
>
> In your previous trace, there were so many small 8 sector requests which
> were merged into 512 sector requests before dispatching to disk. (I am
> not sure why those requests are not bigger. Shouldn't readahead logic
> submit a bigger request?) Now with plug/unplug logic removed, I am assuming
> we should be doing less merging and dispatching more smaller requests. May be
> that is helping and cutting down on disk idling time.
>
> In previous logs, 512 sector request seems to be taking around 1ms to
> complete after dispatch. In between requests disk seems to be idle
> for around .5 to .6 ms. Out of this .3 ms seems to be gone in just
> coming up with new request after completion of previous one and another
> .3ms seems to be consumed in merging the smaller IOs. So if we don't wait
> for merging, it should keep disk busier for .3ms more which is 30% of time
> it takes to complete 512 sector request. So theoritically it can give
> 30% boost for this workload. (Assuming request size will not impact the
> disk throughput very severely).
>
> Anyway, some blktrace data will shed some light..
>
> Thanks
> Vivek


I'll do that when I have access to the machine again, in 6 hours or so :)

Thanks !

Re: Bad SSD performance with recent kernels

[Shaohua Li]

On Tue, 2012-01-31 at 09:07 +0800, Wu Fengguang wrote:
> On Tue, Jan 31, 2012 at 08:14:19AM +0800, Li, Shaohua wrote:
> > On Mon, 2012-01-30 at 17:26 -0500, Vivek Goyal wrote:
> > > On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote:
> > > > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit :
> > > > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:
> > > > > 
> > > > > > Looks the 2.6.39 block plug introduces some latency here. deleting
> > > > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems
> > > > > > workaround
> > > > > > the issue. The plug seems not good for sequential IO, because readahead
> > > > > > code already has plug and has fine grained control.
> > > > > 
> > > > > Why not remove the generic_file_aio_read() plug completely? It
> > > > > actually prevents unplugging immediately after the readahead IO is
> > > > > submitted and in turn stalls the IO pipeline as showed by Eric's
> > > > > blktrace data.
> > > > > 
> > > > > Eric, will you test this patch? Thank you.
> > > 
> > > Can you please run the blktrace again with this patch applied. I am curious
> > > to see how does traffic pattern look like now.
> > > 
> > > In your previous trace, there were so many small 8 sector requests which
> > > were merged into 512 sector requests before dispatching to disk. (I am
> > > not sure why those requests are not bigger. Shouldn't readahead logic
> > > submit a bigger request?) Now with plug/unplug logic removed, I am assuming
> > > we should be doing less merging and dispatching more smaller requests. May be
> > > that is helping and cutting down on disk idling time.
> > > 
> > > In previous logs, 512 sector request seems to be taking around 1ms to
> > > complete after dispatch. In between requests disk seems to be idle
> > > for around .5 to .6 ms. Out of this .3 ms seems to be gone in just
> > > coming up with new request after completion of previous one and another
> > > .3ms seems to be consumed in merging the smaller IOs. So if we don't wait
> > > for merging, it should keep disk busier for .3ms more which is 30% of time
> > > it takes to complete 512 sector request. So theoritically it can give
> > > 30% boost for this workload. (Assuming request size will not impact the
> > > disk throughput very severely).
> > > 
> > > Anyway, some blktrace data will shed some light..
> > yep, I suspect plug merges big request too (iostat shows it too), that's
> > why I only think delete the plug in generic_file_aio_read as a
> > workaround.
> 
> It's good to merge requests inside the same readahead window. However
> I don't think readahead window A should be merged with B at the cost
> of delaying A for some time, which will break the pipeline. If larger
> IO is desirable, we can do so by increasing the readahead size.
> 
> > I still thought readahead has something to do here. I
> > observed the async readahead does readahead (A, A + 2M), and follows (A
> > +128k, A+2M), (A+256k, A+2M) ..., the later readahead doesn't work
> > because we already have (A, A+2M) in memory at that time. Anyway, I can
> > reproduce the issue, will play with it more today.
> 
> How do you observe that? I don't think that readahead pattern is
> possible. However I do see such _read_ patterns.
Ok, after double checking the code and do some tracing, I'm now thinking
we should delete the plug code in generic_file_aio_read. I thought the
problem is:

T1: ra (A, A+128k), (A+128k, A+256k), submit the 256k because lock_page
T2: hit page A+128K, ra (A+256k, A+384). the range isn't submitted
because of plug and there isn't any lock_page till we hit page A+256k
because all pages from A to A+ 256k is in memory
T3: hit page A+256k, ra (A+384, A+ 512). because plug, the range isn't
submitted again.
T4: lock_page A+256, so (A+256, A+512) will be submitted. The task is
waitting for (A+256, A+512) finish

so the pipeline doesn't work.

Re: Bad SSD performance with recent kernels

[Herbert Poetzl]

On Mon, Jan 30, 2012 at 10:28:37PM +0800, Wu Fengguang wrote:
> On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:

>> Looks the 2.6.39 block plug introduces some latency here. deleting
>> blk_start_plug/blk_finish_plug in generic_file_aio_read seems
>> workaround
>> the issue. The plug seems not good for sequential IO, because readahead
>> code already has plug and has fine grained control.

> Why not remove the generic_file_aio_read() plug completely? It
> actually prevents unplugging immediately after the readahead IO is
> submitted and in turn stalls the IO pipeline as showed by Eric's
> blktrace data.

> Eric, will you test this patch? Thank you.

> --- linux.orig/mm/filemap.c	2012-01-30 22:27:11.000000000 +0800
> +++ linux/mm/filemap.c	2012-01-30 22:27:25.000000000 +0800
> @@ -1414,15 +1414,12 @@ generic_file_aio_read(struct kiocb *iocb
>  	unsigned long seg = 0;
>  	size_t count;
>  	loff_t *ppos = &iocb->ki_pos;
> -	struct blk_plug plug;

>  	count = 0;
>  	retval = generic_segment_checks(iov, &nr_segs, &count, VERIFY_WRITE);
>  	if (retval)
>  		return retval;

> -	blk_start_plug(&plug);
> -
>  	/* coalesce the iovecs and go direct-to-BIO for O_DIRECT */
>  	if (filp->f_flags & O_DIRECT) {
>  		loff_t size;
> @@ -1495,7 +1492,6 @@ generic_file_aio_read(struct kiocb *iocb
>  			break;
>  	}
>  out:
> -	blk_finish_plug(&plug);
>  	return retval;
>  }
>  EXPORT_SYMBOL(generic_file_aio_read);

I can confirm, this seems to fix the issue here as well:

kernel    ------------ read /dev/sda -------------
          --- noop ---  - deadline -  ---- cfs ---
          [MB/s]  %CPU  [MB/s]  %CPU  [MB/s]  %CPU
--------------------------------------------------
3.2.2      45.82   3.7   44.85   3.6   45.04   3.4
3.2.2i     45.59   2.3   51.78   2.6   46.03   2.2
3.2.2i128 250.24  20.9  252.68  21.3  250.00  21.6
3.2.2p    250.88  20.8  249.69  20.7  252.34  21.4
3.2.2p128 249.36  20.7  251.47  20.9  249.03  21.2

3.2.2 ... unmodified 3.2.2 kernel 1M
3.2.2i .. unmodified 3.2.2 kernel with idle=poll, 1M
3.2.2i128 unmodified 3.2.2 kernel with idle=poll, 128k 
3.2.2p .. unmodified 3.2.2 kernel with patch, 1M
3.2.2p128 unmodified 3.2.2 kernel with patch, 128k

thanks again,
Herbert

Re: Bad SSD performance with recent kernels

[Eric Dumazet]

Le lundi 30 janvier 2012 à 17:26 -0500, Vivek Goyal a écrit :
> On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote:
> > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit :
> > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote:
> > > 
> > > > Looks the 2.6.39 block plug introduces some latency here. deleting
> > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems
> > > > workaround
> > > > the issue. The plug seems not good for sequential IO, because readahead
> > > > code already has plug and has fine grained control.
> > > 
> > > Why not remove the generic_file_aio_read() plug completely? It
> > > actually prevents unplugging immediately after the readahead IO is
> > > submitted and in turn stalls the IO pipeline as showed by Eric's
> > > blktrace data.
> > > 
> > > Eric, will you test this patch? Thank you.
> 
> Can you please run the blktrace again with this patch applied. I am curious
> to see how does traffic pattern look like now.
> 
> In your previous trace, there were so many small 8 sector requests which
> were merged into 512 sector requests before dispatching to disk. (I am
> not sure why those requests are not bigger. Shouldn't readahead logic
> submit a bigger request?) Now with plug/unplug logic removed, I am assuming
> we should be doing less merging and dispatching more smaller requests. May be
> that is helping and cutting down on disk idling time.
> 
> In previous logs, 512 sector request seems to be taking around 1ms to
> complete after dispatch. In between requests disk seems to be idle
> for around .5 to .6 ms. Out of this .3 ms seems to be gone in just
> coming up with new request after completion of previous one and another
> .3ms seems to be consumed in merging the smaller IOs. So if we don't wait
> for merging, it should keep disk busier for .3ms more which is 30% of time
> it takes to complete 512 sector request. So theoritically it can give
> 30% boost for this workload. (Assuming request size will not impact the
> disk throughput very severely).
> 
> Anyway, some blktrace data will shed some light..

Here it is (last 1000 lines)

  8,16   1  1198126     9.992850217  2570  M   R 5716720 + 8 [dd]
  8,16   1  1198127     9.992851316  2570  Q   R 5716728 + 8 [dd]
  8,16   1  1198128     9.992851577  2570  M   R 5716728 + 8 [dd]
  8,16   1  1198129     9.992852592  2570  I   R 5716480 + 256 (   54889) [dd]
  8,16   1        0     9.992853926     0  m   N cfq2570 insert_request
  8,16   1  1198130     9.992855759  2570  U   N [dd] 1
  8,16   1        0     9.992856488     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.992857801     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.992858528     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198131     9.992858746  2570  D   R 5716480 + 256 (    6154) [dd]
  8,16   1  1198132     9.993162912     0  C   R 5716224 + 256 (  783118) [0]
  8,16   1        0     9.993195602     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198133     9.993294134  2570  Q   R 5716736 + 8 [dd]
  8,16   1  1198134     9.993299036  2570  G   R 5716736 + 8 [dd]
  8,16   1  1198135     9.993300482  2570  P   N [dd]
  8,16   1  1198136     9.993302456  2570  Q   R 5716744 + 8 [dd]
  8,16   1  1198137     9.993303374  2570  M   R 5716744 + 8 [dd]
  8,16   1  1198138     9.993305017  2570  Q   R 5716752 + 8 [dd]
  8,16   1  1198139     9.993305398  2570  M   R 5716752 + 8 [dd]
  8,16   1  1198140     9.993307106  2570  Q   R 5716760 + 8 [dd]
  8,16   1  1198141     9.993307428  2570  M   R 5716760 + 8 [dd]
  8,16   1  1198142     9.993308590  2570  Q   R 5716768 + 8 [dd]
  8,16   1  1198143     9.993308840  2570  M   R 5716768 + 8 [dd]
  8,16   1  1198144     9.993310021  2570  Q   R 5716776 + 8 [dd]
  8,16   1  1198145     9.993310276  2570  M   R 5716776 + 8 [dd]
  8,16   1  1198146     9.993311377  2570  Q   R 5716784 + 8 [dd]
  8,16   1  1198147     9.993311626  2570  M   R 5716784 + 8 [dd]
  8,16   1  1198148     9.993312717  2570  Q   R 5716792 + 8 [dd]
  8,16   1  1198149     9.993312967  2570  M   R 5716792 + 8 [dd]
  8,16   1  1198150     9.993314049  2570  Q   R 5716800 + 8 [dd]
  8,16   1  1198151     9.993314300  2570  M   R 5716800 + 8 [dd]
  8,16   1  1198152     9.993315494  2570  Q   R 5716808 + 8 [dd]
  8,16   1  1198153     9.993315748  2570  M   R 5716808 + 8 [dd]
  8,16   1  1198154     9.993316916  2570  Q   R 5716816 + 8 [dd]
  8,16   1  1198155     9.993317170  2570  M   R 5716816 + 8 [dd]
  8,16   1  1198156     9.993321418  2570  Q   R 5716824 + 8 [dd]
  8,16   1  1198157     9.993321668  2570  M   R 5716824 + 8 [dd]
  8,16   1  1198158     9.993322794  2570  Q   R 5716832 + 8 [dd]
  8,16   1  1198159     9.993323044  2570  M   R 5716832 + 8 [dd]
  8,16   1  1198160     9.993324133  2570  Q   R 5716840 + 8 [dd]
  8,16   1  1198161     9.993324388  2570  M   R 5716840 + 8 [dd]
  8,16   1  1198162     9.993325474  2570  Q   R 5716848 + 8 [dd]
  8,16   1  1198163     9.993325728  2570  M   R 5716848 + 8 [dd]
  8,16   1  1198164     9.993326816  2570  Q   R 5716856 + 8 [dd]
  8,16   1  1198165     9.993327066  2570  M   R 5716856 + 8 [dd]
  8,16   1  1198166     9.993328287  2570  Q   R 5716864 + 8 [dd]
  8,16   1  1198167     9.993328538  2570  M   R 5716864 + 8 [dd]
  8,16   1  1198168     9.993329630  2570  Q   R 5716872 + 8 [dd]
  8,16   1  1198169     9.993329884  2570  M   R 5716872 + 8 [dd]
  8,16   1  1198170     9.993331042  2570  Q   R 5716880 + 8 [dd]
  8,16   1  1198171     9.993331295  2570  M   R 5716880 + 8 [dd]
  8,16   1  1198172     9.993332426  2570  Q   R 5716888 + 8 [dd]
  8,16   1  1198173     9.993332676  2570  M   R 5716888 + 8 [dd]
  8,16   1  1198174     9.993333766  2570  Q   R 5716896 + 8 [dd]
  8,16   1  1198175     9.993334015  2570  M   R 5716896 + 8 [dd]
  8,16   1  1198176     9.993335359  2570  Q   R 5716904 + 8 [dd]
  8,16   1  1198177     9.993335611  2570  M   R 5716904 + 8 [dd]
  8,16   1  1198178     9.993336799  2570  Q   R 5716912 + 8 [dd]
  8,16   1  1198179     9.993337098  2570  M   R 5716912 + 8 [dd]
  8,16   1  1198180     9.993338295  2570  Q   R 5716920 + 8 [dd]
  8,16   1  1198181     9.993338545  2570  M   R 5716920 + 8 [dd]
  8,16   1  1198182     9.993339719  2570  Q   R 5716928 + 8 [dd]
  8,16   1  1198183     9.993339976  2570  M   R 5716928 + 8 [dd]
  8,16   1  1198184     9.993342740  2570  Q   R 5716936 + 8 [dd]
  8,16   1  1198185     9.993342995  2570  M   R 5716936 + 8 [dd]
  8,16   1  1198186     9.993344275  2570  Q   R 5716944 + 8 [dd]
  8,16   1  1198187     9.993344529  2570  M   R 5716944 + 8 [dd]
  8,16   1  1198188     9.993345637  2570  Q   R 5716952 + 8 [dd]
  8,16   1  1198189     9.993345886  2570  M   R 5716952 + 8 [dd]
  8,16   1  1198190     9.993346996  2570  Q   R 5716960 + 8 [dd]
  8,16   1  1198191     9.993347246  2570  M   R 5716960 + 8 [dd]
  8,16   1  1198192     9.993348322  2570  Q   R 5716968 + 8 [dd]
  8,16   1  1198193     9.993348722  2570  M   R 5716968 + 8 [dd]
  8,16   1  1198194     9.993349883  2570  Q   R 5716976 + 8 [dd]
  8,16   1  1198195     9.993350192  2570  M   R 5716976 + 8 [dd]
  8,16   1  1198196     9.993351388  2570  Q   R 5716984 + 8 [dd]
  8,16   1  1198197     9.993351638  2570  M   R 5716984 + 8 [dd]
  8,16   1  1198198     9.993352772  2570  I   R 5716736 + 256 (   53736) [dd]
  8,16   1        0     9.993354172     0  m   N cfq2570 insert_request
  8,16   1  1198199     9.993356100  2570  U   N [dd] 1
  8,16   1        0     9.993356925     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.993358274     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.993359003     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198200     9.993359283  2570  D   R 5716736 + 256 (    6511) [dd]
  8,16   1  1198201     9.993652117     0  C   R 5716480 + 256 (  793371) [0]
  8,16   1        0     9.993684544     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198202     9.993780104  2570  Q   R 5716992 + 8 [dd]
  8,16   1  1198203     9.993784668  2570  G   R 5716992 + 8 [dd]
  8,16   1  1198204     9.993786413  2570  P   N [dd]
  8,16   1  1198205     9.993788835  2570  Q   R 5717000 + 8 [dd]
  8,16   1  1198206     9.993789621  2570  M   R 5717000 + 8 [dd]
  8,16   1  1198207     9.993791535  2570  Q   R 5717008 + 8 [dd]
  8,16   1  1198208     9.993791785  2570  M   R 5717008 + 8 [dd]
  8,16   1  1198209     9.993792964  2570  Q   R 5717016 + 8 [dd]
  8,16   1  1198210     9.993793272  2570  M   R 5717016 + 8 [dd]
  8,16   1  1198211     9.993794443  2570  Q   R 5717024 + 8 [dd]
  8,16   1  1198212     9.993794759  2570  M   R 5717024 + 8 [dd]
  8,16   1  1198213     9.993795875  2570  Q   R 5717032 + 8 [dd]
  8,16   1  1198214     9.993796125  2570  M   R 5717032 + 8 [dd]
  8,16   1  1198215     9.993797251  2570  Q   R 5717040 + 8 [dd]
  8,16   1  1198216     9.993797499  2570  M   R 5717040 + 8 [dd]
  8,16   1  1198217     9.993801703  2570  Q   R 5717048 + 8 [dd]
  8,16   1  1198218     9.993802014  2570  M   R 5717048 + 8 [dd]
  8,16   1  1198219     9.993803167  2570  Q   R 5717056 + 8 [dd]
  8,16   1  1198220     9.993803422  2570  M   R 5717056 + 8 [dd]
  8,16   1  1198221     9.993804516  2570  Q   R 5717064 + 8 [dd]
  8,16   1  1198222     9.993804770  2570  M   R 5717064 + 8 [dd]
  8,16   1  1198223     9.993805960  2570  Q   R 5717072 + 8 [dd]
  8,16   1  1198224     9.993806211  2570  M   R 5717072 + 8 [dd]
  8,16   1  1198225     9.993807320  2570  Q   R 5717080 + 8 [dd]
  8,16   1  1198226     9.993807600  2570  M   R 5717080 + 8 [dd]
  8,16   1  1198227     9.993808914  2570  Q   R 5717088 + 8 [dd]
  8,16   1  1198228     9.993809176  2570  M   R 5717088 + 8 [dd]
  8,16   1  1198229     9.993810303  2570  Q   R 5717096 + 8 [dd]
  8,16   1  1198230     9.993810557  2570  M   R 5717096 + 8 [dd]
  8,16   1  1198231     9.993811689  2570  Q   R 5717104 + 8 [dd]
  8,16   1  1198232     9.993811941  2570  M   R 5717104 + 8 [dd]
  8,16   1  1198233     9.993813073  2570  Q   R 5717112 + 8 [dd]
  8,16   1  1198234     9.993813328  2570  M   R 5717112 + 8 [dd]
  8,16   1  1198235     9.993814430  2570  Q   R 5717120 + 8 [dd]
  8,16   1  1198236     9.993814685  2570  M   R 5717120 + 8 [dd]
  8,16   1  1198237     9.993815791  2570  Q   R 5717128 + 8 [dd]
  8,16   1  1198238     9.993816042  2570  M   R 5717128 + 8 [dd]
  8,16   1  1198239     9.993817225  2570  Q   R 5717136 + 8 [dd]
  8,16   1  1198240     9.993817492  2570  M   R 5717136 + 8 [dd]
  8,16   1  1198241     9.993818673  2570  Q   R 5717144 + 8 [dd]
  8,16   1  1198242     9.993818996  2570  M   R 5717144 + 8 [dd]
  8,16   1  1198243     9.993820108  2570  Q   R 5717152 + 8 [dd]
  8,16   1  1198244     9.993820426  2570  M   R 5717152 + 8 [dd]
  8,16   1  1198245     9.993823204  2570  Q   R 5717160 + 8 [dd]
  8,16   1  1198246     9.993823522  2570  M   R 5717160 + 8 [dd]
  8,16   1  1198247     9.993824646  2570  Q   R 5717168 + 8 [dd]
  8,16   1  1198248     9.993824912  2570  M   R 5717168 + 8 [dd]
  8,16   1  1198249     9.993826090  2570  Q   R 5717176 + 8 [dd]
  8,16   1  1198250     9.993826354  2570  M   R 5717176 + 8 [dd]
  8,16   1  1198251     9.993827454  2570  Q   R 5717184 + 8 [dd]
  8,16   1  1198252     9.993827709  2570  M   R 5717184 + 8 [dd]
  8,16   1  1198253     9.993828819  2570  Q   R 5717192 + 8 [dd]
  8,16   1  1198254     9.993829079  2570  M   R 5717192 + 8 [dd]
  8,16   1  1198255     9.993830271  2570  Q   R 5717200 + 8 [dd]
  8,16   1  1198256     9.993830537  2570  M   R 5717200 + 8 [dd]
  8,16   1  1198257     9.993831717  2570  Q   R 5717208 + 8 [dd]
  8,16   1  1198258     9.993831967  2570  M   R 5717208 + 8 [dd]
  8,16   1  1198259     9.993833624  2570  Q   R 5717216 + 8 [dd]
  8,16   1  1198260     9.993833887  2570  M   R 5717216 + 8 [dd]
  8,16   1  1198261     9.993835003  2570  Q   R 5717224 + 8 [dd]
  8,16   1  1198262     9.993835251  2570  M   R 5717224 + 8 [dd]
  8,16   1  1198263     9.993836361  2570  Q   R 5717232 + 8 [dd]
  8,16   1  1198264     9.993836609  2570  M   R 5717232 + 8 [dd]
  8,16   1  1198265     9.993837717  2570  Q   R 5717240 + 8 [dd]
  8,16   1  1198266     9.993837970  2570  M   R 5717240 + 8 [dd]
  8,16   1  1198267     9.993838830  2570  I   R 5716992 + 256 (   54162) [dd]
  8,16   1        0     9.993840146     0  m   N cfq2570 insert_request
  8,16   1  1198268     9.993841920  2570  U   N [dd] 1
  8,16   1        0     9.993842806     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.993844092     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.993844846     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198269     9.993845118  2570  D   R 5716992 + 256 (    6288) [dd]
  8,16   1  1198270     9.994141624     0  C   R 5716736 + 256 (  782341) [0]
  8,16   1        0     9.994174463     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198271     9.994266949  2570  Q   R 5717248 + 8 [dd]
  8,16   1  1198272     9.994271892  2570  G   R 5717248 + 8 [dd]
  8,16   1  1198273     9.994273520  2570  P   N [dd]
  8,16   1  1198274     9.994275563  2570  Q   R 5717256 + 8 [dd]
  8,16   1  1198275     9.994276389  2570  M   R 5717256 + 8 [dd]
  8,16   1  1198276     9.994278090  2570  Q   R 5717264 + 8 [dd]
  8,16   1  1198277     9.994278433  2570  M   R 5717264 + 8 [dd]
  8,16   1  1198278     9.994282797  2570  Q   R 5717272 + 8 [dd]
  8,16   1  1198279     9.994283057  2570  M   R 5717272 + 8 [dd]
  8,16   1  1198280     9.994284292  2570  Q   R 5717280 + 8 [dd]
  8,16   1  1198281     9.994284545  2570  M   R 5717280 + 8 [dd]
  8,16   1  1198282     9.994285642  2570  Q   R 5717288 + 8 [dd]
  8,16   1  1198283     9.994285903  2570  M   R 5717288 + 8 [dd]
  8,16   1  1198284     9.994287001  2570  Q   R 5717296 + 8 [dd]
  8,16   1  1198285     9.994287251  2570  M   R 5717296 + 8 [dd]
  8,16   1  1198286     9.994288341  2570  Q   R 5717304 + 8 [dd]
  8,16   1  1198287     9.994288593  2570  M   R 5717304 + 8 [dd]
  8,16   1  1198288     9.994289792  2570  Q   R 5717312 + 8 [dd]
  8,16   1  1198289     9.994290039  2570  M   R 5717312 + 8 [dd]
  8,16   1  1198290     9.994291131  2570  Q   R 5717320 + 8 [dd]
  8,16   1  1198291     9.994291381  2570  M   R 5717320 + 8 [dd]
  8,16   1  1198292     9.994292542  2570  Q   R 5717328 + 8 [dd]
  8,16   1  1198293     9.994292794  2570  M   R 5717328 + 8 [dd]
  8,16   1  1198294     9.994293886  2570  Q   R 5717336 + 8 [dd]
  8,16   1  1198295     9.994294136  2570  M   R 5717336 + 8 [dd]
  8,16   1  1198296     9.994295227  2570  Q   R 5717344 + 8 [dd]
  8,16   1  1198297     9.994295479  2570  M   R 5717344 + 8 [dd]
  8,16   1  1198298     9.994297088  2570  Q   R 5717352 + 8 [dd]
  8,16   1  1198299     9.994297338  2570  M   R 5717352 + 8 [dd]
  8,16   1  1198300     9.994298434  2570  Q   R 5717360 + 8 [dd]
  8,16   1  1198301     9.994298684  2570  M   R 5717360 + 8 [dd]
  8,16   1  1198302     9.994299804  2570  Q   R 5717368 + 8 [dd]
  8,16   1  1198303     9.994300063  2570  M   R 5717368 + 8 [dd]
  8,16   1  1198304     9.994301156  2570  Q   R 5717376 + 8 [dd]
  8,16   1  1198305     9.994301404  2570  M   R 5717376 + 8 [dd]
  8,16   1  1198306     9.994304103  2570  Q   R 5717384 + 8 [dd]
  8,16   1  1198307     9.994304352  2570  M   R 5717384 + 8 [dd]
  8,16   1  1198308     9.994305504  2570  Q   R 5717392 + 8 [dd]
  8,16   1  1198309     9.994305754  2570  M   R 5717392 + 8 [dd]
  8,16   1  1198310     9.994306851  2570  Q   R 5717400 + 8 [dd]
  8,16   1  1198311     9.994307101  2570  M   R 5717400 + 8 [dd]
  8,16   1  1198312     9.994308410  2570  Q   R 5717408 + 8 [dd]
  8,16   1  1198313     9.994308881  2570  M   R 5717408 + 8 [dd]
  8,16   1  1198314     9.994310006  2570  Q   R 5717416 + 8 [dd]
  8,16   1  1198315     9.994310270  2570  M   R 5717416 + 8 [dd]
  8,16   1  1198316     9.994311380  2570  Q   R 5717424 + 8 [dd]
  8,16   1  1198317     9.994311638  2570  M   R 5717424 + 8 [dd]
  8,16   1  1198318     9.994312743  2570  Q   R 5717432 + 8 [dd]
  8,16   1  1198319     9.994313050  2570  M   R 5717432 + 8 [dd]
  8,16   1  1198320     9.994314189  2570  Q   R 5717440 + 8 [dd]
  8,16   1  1198321     9.994314450  2570  M   R 5717440 + 8 [dd]
  8,16   1  1198322     9.994315584  2570  Q   R 5717448 + 8 [dd]
  8,16   1  1198323     9.994315833  2570  M   R 5717448 + 8 [dd]
  8,16   1  1198324     9.994316999  2570  Q   R 5717456 + 8 [dd]
  8,16   1  1198325     9.994317259  2570  M   R 5717456 + 8 [dd]
  8,16   1  1198326     9.994318364  2570  Q   R 5717464 + 8 [dd]
  8,16   1  1198327     9.994318612  2570  M   R 5717464 + 8 [dd]
  8,16   1  1198328     9.994319698  2570  Q   R 5717472 + 8 [dd]
  8,16   1  1198329     9.994319956  2570  M   R 5717472 + 8 [dd]
  8,16   1  1198330     9.994321048  2570  Q   R 5717480 + 8 [dd]
  8,16   1  1198331     9.994321298  2570  M   R 5717480 + 8 [dd]
  8,16   1  1198332     9.994322649  2570  Q   R 5717488 + 8 [dd]
  8,16   1  1198333     9.994322899  2570  M   R 5717488 + 8 [dd]
  8,16   1  1198334     9.994325719  2570  Q   R 5717496 + 8 [dd]
  8,16   1  1198335     9.994325970  2570  M   R 5717496 + 8 [dd]
  8,16   1  1198336     9.994326898  2570  I   R 5717248 + 256 (   55006) [dd]
  8,16   1        0     9.994328230     0  m   N cfq2570 insert_request
  8,16   1  1198337     9.994329907  2570  U   N [dd] 1
  8,16   1        0     9.994330886     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.994332096     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.994332828     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198338     9.994333050  2570  D   R 5717248 + 256 (    6152) [dd]
  8,16   1  1198339     9.994629829     0  C   R 5716992 + 256 (  784711) [0]
  8,16   1        0     9.994661464     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198340     9.994752149  2570  Q   R 5717504 + 8 [dd]
  8,16   1  1198341     9.994757116  2570  G   R 5717504 + 8 [dd]
  8,16   1  1198342     9.994758648  2570  P   N [dd]
  8,16   1  1198343     9.994760828  2570  Q   R 5717512 + 8 [dd]
  8,16   1  1198344     9.994761612  2570  M   R 5717512 + 8 [dd]
  8,16   1  1198345     9.994763312  2570  Q   R 5717520 + 8 [dd]
  8,16   1  1198346     9.994763571  2570  M   R 5717520 + 8 [dd]
  8,16   1  1198347     9.994764822  2570  Q   R 5717528 + 8 [dd]
  8,16   1  1198348     9.994765237  2570  M   R 5717528 + 8 [dd]
  8,16   1  1198349     9.994766430  2570  Q   R 5717536 + 8 [dd]
  8,16   1  1198350     9.994766680  2570  M   R 5717536 + 8 [dd]
  8,16   1  1198351     9.994767790  2570  Q   R 5717544 + 8 [dd]
  8,16   1  1198352     9.994768053  2570  M   R 5717544 + 8 [dd]
  8,16   1  1198353     9.994769187  2570  Q   R 5717552 + 8 [dd]
  8,16   1  1198354     9.994769438  2570  M   R 5717552 + 8 [dd]
  8,16   1  1198355     9.994770592  2570  Q   R 5717560 + 8 [dd]
  8,16   1  1198356     9.994770851  2570  M   R 5717560 + 8 [dd]
  8,16   1  1198357     9.994771968  2570  Q   R 5717568 + 8 [dd]
  8,16   1  1198358     9.994772218  2570  M   R 5717568 + 8 [dd]
  8,16   1  1198359     9.994773304  2570  Q   R 5717576 + 8 [dd]
  8,16   1  1198360     9.994773555  2570  M   R 5717576 + 8 [dd]
  8,16   1  1198361     9.994774719  2570  Q   R 5717584 + 8 [dd]
  8,16   1  1198362     9.994774982  2570  M   R 5717584 + 8 [dd]
  8,16   1  1198363     9.994776154  2570  Q   R 5717592 + 8 [dd]
  8,16   1  1198364     9.994776410  2570  M   R 5717592 + 8 [dd]
  8,16   1  1198365     9.994778214  2570  Q   R 5717600 + 8 [dd]
  8,16   1  1198366     9.994778471  2570  M   R 5717600 + 8 [dd]
  8,16   1  1198367     9.994783098  2570  Q   R 5717608 + 8 [dd]
  8,16   1  1198368     9.994783353  2570  M   R 5717608 + 8 [dd]
  8,16   1  1198369     9.994784517  2570  Q   R 5717616 + 8 [dd]
  8,16   1  1198370     9.994784769  2570  M   R 5717616 + 8 [dd]
  8,16   1  1198371     9.994785889  2570  Q   R 5717624 + 8 [dd]
  8,16   1  1198372     9.994786141  2570  M   R 5717624 + 8 [dd]
  8,16   1  1198373     9.994787248  2570  Q   R 5717632 + 8 [dd]
  8,16   1  1198374     9.994787498  2570  M   R 5717632 + 8 [dd]
  8,16   1  1198375     9.994788675  2570  Q   R 5717640 + 8 [dd]
  8,16   1  1198376     9.994788926  2570  M   R 5717640 + 8 [dd]
  8,16   1  1198377     9.994790174  2570  Q   R 5717648 + 8 [dd]
  8,16   1  1198378     9.994790429  2570  M   R 5717648 + 8 [dd]
  8,16   1  1198379     9.994791556  2570  Q   R 5717656 + 8 [dd]
  8,16   1  1198380     9.994791807  2570  M   R 5717656 + 8 [dd]
  8,16   1  1198381     9.994792970  2570  Q   R 5717664 + 8 [dd]
  8,16   1  1198382     9.994793220  2570  M   R 5717664 + 8 [dd]
  8,16   1  1198383     9.994794346  2570  Q   R 5717672 + 8 [dd]
  8,16   1  1198384     9.994794597  2570  M   R 5717672 + 8 [dd]
  8,16   1  1198385     9.994795687  2570  Q   R 5717680 + 8 [dd]
  8,16   1  1198386     9.994795938  2570  M   R 5717680 + 8 [dd]
  8,16   1  1198387     9.994797108  2570  Q   R 5717688 + 8 [dd]
  8,16   1  1198388     9.994797369  2570  M   R 5717688 + 8 [dd]
  8,16   1  1198389     9.994798465  2570  Q   R 5717696 + 8 [dd]
  8,16   1  1198390     9.994798715  2570  M   R 5717696 + 8 [dd]
  8,16   1  1198391     9.994799782  2570  Q   R 5717704 + 8 [dd]
  8,16   1  1198392     9.994800034  2570  M   R 5717704 + 8 [dd]
  8,16   1  1198393     9.994801267  2570  Q   R 5717712 + 8 [dd]
  8,16   1  1198394     9.994801518  2570  M   R 5717712 + 8 [dd]
  8,16   1  1198395     9.994804336  2570  Q   R 5717720 + 8 [dd]
  8,16   1  1198396     9.994804590  2570  M   R 5717720 + 8 [dd]
  8,16   1  1198397     9.994805730  2570  Q   R 5717728 + 8 [dd]
  8,16   1  1198398     9.994805980  2570  M   R 5717728 + 8 [dd]
  8,16   1  1198399     9.994807057  2570  Q   R 5717736 + 8 [dd]
  8,16   1  1198400     9.994807314  2570  M   R 5717736 + 8 [dd]
  8,16   1  1198401     9.994808391  2570  Q   R 5717744 + 8 [dd]
  8,16   1  1198402     9.994808642  2570  M   R 5717744 + 8 [dd]
  8,16   1  1198403     9.994809738  2570  Q   R 5717752 + 8 [dd]
  8,16   1  1198404     9.994809993  2570  M   R 5717752 + 8 [dd]
  8,16   1  1198405     9.994810844  2570  I   R 5717504 + 256 (   53728) [dd]
  8,16   1        0     9.994812228     0  m   N cfq2570 insert_request
  8,16   1  1198406     9.994813845  2570  U   N [dd] 1
  8,16   1        0     9.994814579     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.994815804     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.994816536     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198407     9.994816821  2570  D   R 5717504 + 256 (    5977) [dd]
  8,16   1  1198408     9.995120966     0  C   R 5717248 + 256 (  787916) [0]
  8,16   1        0     9.995152425     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198409     9.995588516    22  C   R 5717504 + 256 (  771695) [0]
  8,16   1        0     9.995604694     0  m   N cfq2570 complete rqnoidle 0
  8,16   1        0     9.995605304     0  m   N cfq schedule dispatch
  8,16   1        0     9.995610896     0  m   N cfq2570 slice expired t=0
  8,16   1        0     9.995613323     0  m   N cfq2570 sl_used=1 disp=14 charge=1 iops=0 sect=3584
  8,16   1        0     9.995613922     0  m   N cfq2570 del_from_rr
  8,16   1        0     9.995615317     0  m   N cfq workload slice:10
  8,16   1        0     9.995616276     0  m   N cfq2577 set_active wl_prio:0 wl_type:1
  8,16   1        0     9.995617803     0  m   N cfq2577 fifo=  (null)
  8,16   1        0     9.995618455     0  m   N cfq2577 dispatch_insert
  8,16   1        0     9.995620143     0  m   N cfq2577 dispatched a request
  8,16   1        0     9.995620870     0  m   N cfq2577 activate rq, drv=1
  8,16   1  1198410     9.995621255    22  D   R 127095467 + 8 ( 6830688) [kswapd0]
  8,16   1        0     9.995632325     0  m   N cfq2577 dispatch_insert
  8,16   1        0     9.995633106     0  m   N cfq2577 dispatched a request
  8,16   1        0     9.995633611     0  m   N cfq2577 activate rq, drv=2
  8,16   1  1198411     9.995633847    22  D   R 127100683 + 24 ( 6841571) [kswapd0]
  8,16   1  1198412     9.995862020     8  C   R 127095467 + 8 (  240765) [0]
  8,16   1        0     9.995866553     0  m   N cfq2577 complete rqnoidle 0
  8,16   1        0     9.995867340     0  m   N cfq2577 set_slice=10
  8,16   1  1198413     9.995909670  2577  C   R 127100683 + 24 (  275823) [0]
  8,16   1        0     9.995911822     0  m   N cfq2577 complete rqnoidle 0
  8,16   1        0     9.995912219     0  m   N cfq schedule dispatch
  8,16   1  1198414     9.995935586  2577  A   R 127109795 + 8 <- (8,21) 52503872
  8,16   1  1198415     9.995936258  2577  Q   R 127109795 + 8 [evolution]
  8,16   1  1198416     9.995940062  2577  G   R 127109795 + 8 [evolution]
  8,16   1  1198417     9.995941861  2577  P   N [evolution]
  8,16   1  1198418     9.995943395  2577  I   R 127109795 + 8 (    3333) [evolution]
  8,16   1        0     9.995944417     0  m   N cfq2577 insert_request
  8,16   1  1198419     9.995945545  2577  U   N [evolution] 1
  8,16   1        0     9.995946172     0  m   N cfq2577 dispatch_insert
  8,16   1        0     9.995946905     0  m   N cfq2577 dispatched a request
  8,16   1        0     9.995947455     0  m   N cfq2577 activate rq, drv=1
  8,16   1  1198420     9.995947618  2577  D   R 127109795 + 8 (    4223) [evolution]
  8,16   1  1198421     9.995960951  2570  Q   R 5717760 + 8 [dd]
  8,16   1  1198422     9.995961927  2570  G   R 5717760 + 8 [dd]
  8,16   1  1198423     9.995962173  2570  P   N [dd]
  8,16   1  1198424     9.995963514  2570  Q   R 5717768 + 8 [dd]
  8,16   1  1198425     9.995964687  2570  M   R 5717768 + 8 [dd]
  8,16   1  1198426     9.995966344  2570  Q   R 5717776 + 8 [dd]
  8,16   1  1198427     9.995966648  2570  M   R 5717776 + 8 [dd]
  8,16   1  1198428     9.995967874  2570  Q   R 5717784 + 8 [dd]
  8,16   1  1198429     9.995968126  2570  M   R 5717784 + 8 [dd]
  8,16   1  1198430     9.995969233  2570  Q   R 5717792 + 8 [dd]
  8,16   1  1198431     9.995969483  2570  M   R 5717792 + 8 [dd]
  8,16   1  1198432     9.995970602  2570  Q   R 5717800 + 8 [dd]
  8,16   1  1198433     9.995970852  2570  M   R 5717800 + 8 [dd]
  8,16   1  1198434     9.995971949  2570  Q   R 5717808 + 8 [dd]
  8,16   1  1198435     9.995972201  2570  M   R 5717808 + 8 [dd]
  8,16   1  1198436     9.995973299  2570  Q   R 5717816 + 8 [dd]
  8,16   1  1198437     9.995973559  2570  M   R 5717816 + 8 [dd]
  8,16   1  1198438     9.995974658  2570  Q   R 5717824 + 8 [dd]
  8,16   1  1198439     9.995974907  2570  M   R 5717824 + 8 [dd]
  8,16   1  1198440     9.995976012  2570  Q   R 5717832 + 8 [dd]
  8,16   1  1198441     9.995976263  2570  M   R 5717832 + 8 [dd]
  8,16   1  1198442     9.995977424  2570  Q   R 5717840 + 8 [dd]
  8,16   1  1198443     9.995977675  2570  M   R 5717840 + 8 [dd]
  8,16   1  1198444     9.995978776  2570  Q   R 5717848 + 8 [dd]
  8,16   1  1198445     9.995979026  2570  M   R 5717848 + 8 [dd]
  8,16   1  1198446     9.995982184  2570  Q   R 5717856 + 8 [dd]
  8,16   1  1198447     9.995982436  2570  M   R 5717856 + 8 [dd]
  8,16   1  1198448     9.995983606  2570  Q   R 5717864 + 8 [dd]
  8,16   1  1198449     9.995983856  2570  M   R 5717864 + 8 [dd]
  8,16   1  1198450     9.995984988  2570  Q   R 5717872 + 8 [dd]
  8,16   1  1198451     9.995985241  2570  M   R 5717872 + 8 [dd]
  8,16   1  1198452     9.995986350  2570  Q   R 5717880 + 8 [dd]
  8,16   1  1198453     9.995986600  2570  M   R 5717880 + 8 [dd]
  8,16   1  1198454     9.995987699  2570  Q   R 5717888 + 8 [dd]
  8,16   1  1198455     9.995987949  2570  M   R 5717888 + 8 [dd]
  8,16   1  1198456     9.995989053  2570  Q   R 5717896 + 8 [dd]
  8,16   1  1198457     9.995989303  2570  M   R 5717896 + 8 [dd]
  8,16   1  1198458     9.995991031  2570  Q   R 5717904 + 8 [dd]
  8,16   1  1198459     9.995991282  2570  M   R 5717904 + 8 [dd]
  8,16   1  1198460     9.995992402  2570  Q   R 5717912 + 8 [dd]
  8,16   1  1198461     9.995992665  2570  M   R 5717912 + 8 [dd]
  8,16   1  1198462     9.995993783  2570  Q   R 5717920 + 8 [dd]
  8,16   1  1198463     9.995994049  2570  M   R 5717920 + 8 [dd]
  8,16   1  1198464     9.995995194  2570  Q   R 5717928 + 8 [dd]
  8,16   1  1198465     9.995995453  2570  M   R 5717928 + 8 [dd]
  8,16   1  1198466     9.995996565  2570  Q   R 5717936 + 8 [dd]
  8,16   1  1198467     9.995996814  2570  M   R 5717936 + 8 [dd]
  8,16   1  1198468     9.995998022  2570  Q   R 5717944 + 8 [dd]
  8,16   1  1198469     9.995998269  2570  M   R 5717944 + 8 [dd]
  8,16   1  1198470     9.995999501  2570  Q   R 5717952 + 8 [dd]
  8,16   1  1198471     9.995999755  2570  M   R 5717952 + 8 [dd]
  8,16   1  1198472     9.996000869  2570  Q   R 5717960 + 8 [dd]
  8,16   1  1198473     9.996001119  2570  M   R 5717960 + 8 [dd]
  8,16   1  1198474     9.996003860  2570  Q   R 5717968 + 8 [dd]
  8,16   1  1198475     9.996004109  2570  M   R 5717968 + 8 [dd]
  8,16   1  1198476     9.996005214  2570  Q   R 5717976 + 8 [dd]
  8,16   1  1198477     9.996005464  2570  M   R 5717976 + 8 [dd]
  8,16   1  1198478     9.996006583  2570  Q   R 5717984 + 8 [dd]
  8,16   1  1198479     9.996006837  2570  M   R 5717984 + 8 [dd]
  8,16   1  1198480     9.996007952  2570  Q   R 5717992 + 8 [dd]
  8,16   1  1198481     9.996008205  2570  M   R 5717992 + 8 [dd]
  8,16   1  1198482     9.996009328  2570  Q   R 5718000 + 8 [dd]
  8,16   1  1198483     9.996009580  2570  M   R 5718000 + 8 [dd]
  8,16   1  1198484     9.996010703  2570  Q   R 5718008 + 8 [dd]
  8,16   1  1198485     9.996010953  2570  M   R 5718008 + 8 [dd]
  8,16   1  1198486     9.996011362  2570  I   R 5717760 + 256 (   49435) [dd]
  8,16   1        0     9.996011940     0  m   N cfq2570 insert_request
  8,16   1        0     9.996012410     0  m   N cfq2570 add_to_rr
  8,16   1  1198487     9.996013350  2570  U   N [dd] 1
  8,16   1  1198488     9.996076425  2570  Q   R 5718016 + 8 [dd]
  8,16   1  1198489     9.996076900  2570  M   R 5718016 + 8 [dd]
  8,16   1  1198490     9.996079343  2570  Q   R 5718024 + 8 [dd]
  8,16   1  1198491     9.996079716  2570  M   R 5718024 + 8 [dd]
  8,16   1  1198492     9.996081166  2570  Q   R 5718032 + 8 [dd]
  8,16   1  1198493     9.996081531  2570  M   R 5718032 + 8 [dd]
  8,16   1  1198494     9.996082875  2570  Q   R 5718040 + 8 [dd]
  8,16   1  1198495     9.996083254  2570  M   R 5718040 + 8 [dd]
  8,16   1  1198496     9.996084649  2570  Q   R 5718048 + 8 [dd]
  8,16   1  1198497     9.996085005  2570  M   R 5718048 + 8 [dd]
  8,16   1  1198498     9.996086309  2570  Q   R 5718056 + 8 [dd]
  8,16   1  1198499     9.996086676  2570  M   R 5718056 + 8 [dd]
  8,16   1  1198500     9.996087981  2570  Q   R 5718064 + 8 [dd]
  8,16   1  1198501     9.996088337  2570  M   R 5718064 + 8 [dd]
  8,16   1  1198502     9.996089538  2570  Q   R 5718072 + 8 [dd]
  8,16   1  1198503     9.996089903  2570  M   R 5718072 + 8 [dd]
  8,16   1  1198504     9.996092903  2570  Q   R 5718080 + 8 [dd]
  8,16   1  1198505     9.996093272  2570  M   R 5718080 + 8 [dd]
  8,16   1  1198506     9.996094544  2570  Q   R 5718088 + 8 [dd]
  8,16   1  1198507     9.996094978  2570  M   R 5718088 + 8 [dd]
  8,16   1  1198508     9.996096219  2570  Q   R 5718096 + 8 [dd]
  8,16   1  1198509     9.996096611  2570  M   R 5718096 + 8 [dd]
  8,16   1  1198510     9.996097834  2570  Q   R 5718104 + 8 [dd]
  8,16   1  1198511     9.996098216  2570  M   R 5718104 + 8 [dd]
  8,16   1  1198512     9.996099625  2570  Q   R 5718112 + 8 [dd]
  8,16   1  1198513     9.996099987  2570  M   R 5718112 + 8 [dd]
  8,16   1  1198514     9.996101339  2570  Q   R 5718120 + 8 [dd]
  8,16   1  1198515     9.996101707  2570  M   R 5718120 + 8 [dd]
  8,16   1  1198516     9.996102946  2570  Q   R 5718128 + 8 [dd]
  8,16   1  1198517     9.996103302  2570  M   R 5718128 + 8 [dd]
  8,16   1  1198518     9.996104534  2570  Q   R 5718136 + 8 [dd]
  8,16   1  1198519     9.996104906  2570  M   R 5718136 + 8 [dd]
  8,16   1  1198520     9.996106127  2570  Q   R 5718144 + 8 [dd]
  8,16   1  1198521     9.996106482  2570  M   R 5718144 + 8 [dd]
  8,16   1  1198522     9.996107704  2570  Q   R 5718152 + 8 [dd]
  8,16   1  1198523     9.996108068  2570  M   R 5718152 + 8 [dd]
  8,16   1  1198524     9.996109289  2570  Q   R 5718160 + 8 [dd]
  8,16   1  1198525     9.996109649  2570  M   R 5718160 + 8 [dd]
  8,16   1  1198526     9.996110859  2570  Q   R 5718168 + 8 [dd]
  8,16   1  1198527     9.996111227  2570  M   R 5718168 + 8 [dd]
  8,16   1  1198528     9.996112546  2570  Q   R 5718176 + 8 [dd]
  8,16   1  1198529     9.996112904  2570  M   R 5718176 + 8 [dd]
  8,16   1  1198530     9.996114125  2570  Q   R 5718184 + 8 [dd]
  8,16   1  1198531     9.996114494  2570  M   R 5718184 + 8 [dd]
  8,16   1  1198532     9.996117368  2570  Q   R 5718192 + 8 [dd]
  8,16   1  1198533     9.996117752  2570  M   R 5718192 + 8 [dd]
  8,16   1  1198534     9.996119006  2570  Q   R 5718200 + 8 [dd]
  8,16   1  1198535     9.996119376  2570  M   R 5718200 + 8 [dd]
  8,16   1  1198536     9.996120984  2570  Q   R 5718208 + 8 [dd]
  8,16   1  1198537     9.996121340  2570  M   R 5718208 + 8 [dd]
  8,16   1  1198538     9.996122560  2570  Q   R 5718216 + 8 [dd]
  8,16   1  1198539     9.996122928  2570  M   R 5718216 + 8 [dd]
  8,16   1  1198540     9.996124256  2570  Q   R 5718224 + 8 [dd]
  8,16   1  1198541     9.996124613  2570  M   R 5718224 + 8 [dd]
  8,16   1  1198542     9.996125840  2570  Q   R 5718232 + 8 [dd]
  8,16   1  1198543     9.996126205  2570  M   R 5718232 + 8 [dd]
  8,16   1  1198544     9.996127439  2570  Q   R 5718240 + 8 [dd]
  8,16   1  1198545     9.996127795  2570  M   R 5718240 + 8 [dd]
  8,16   1  1198546     9.996129008  2570  Q   R 5718248 + 8 [dd]
  8,16   1  1198547     9.996129368  2570  M   R 5718248 + 8 [dd]
  8,16   1  1198548     9.996130621  2570  Q   R 5718256 + 8 [dd]
  8,16   1  1198549     9.996130977  2570  M   R 5718256 + 8 [dd]
  8,16   1  1198550     9.996132180  2570  Q   R 5718264 + 8 [dd]
  8,16   1  1198551     9.996132629  2570  M   R 5718264 + 8 [dd]
  8,16   1  1198552     9.996172197     0  C   R 127109795 + 8 (  224579) [0]
  8,16   1        0     9.996175466     0  m   N cfq2577 complete rqnoidle 0
  8,16   1        0     9.996175960     0  m   N cfq schedule dispatch
  8,16   1        0     9.996178896     0  m   N cfq2577 slice expired t=0
  8,16   1        0     9.996180225     0  m   N cfq2577 sl_used=1 disp=3 charge=1 iops=0 sect=40
  8,16   1        0     9.996180604     0  m   N cfq2577 del_from_rr
  8,16   1        0     9.996181202     0  m   N cfq workload slice:10
  8,16   1        0     9.996181922     0  m   N cfq2570 set_active wl_prio:0 wl_type:2
  8,16   1        0     9.996182708     0  m   N cfq2570 fifo=  (null)
  8,16   1        0     9.996183077     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.996183707     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.996184185     0  m   N cfq2570 activate rq, drv=1
  8,16   1  1198553     9.996184373     0  D   R 5717760 + 512 (  173011) [swapper/1]
  8,16   1  1198554     9.996216977  2577  A   R 127110683 + 8 <- (8,21) 52504760
  8,16   1  1198555     9.996217210  2577  Q   R 127110683 + 8 [evolution]
  8,16   1  1198556     9.996218418  2577  G   R 127110683 + 8 [evolution]
  8,16   1  1198557     9.996219232  2577  P   N [evolution]
  8,16   1  1198558     9.996219820  2577  I   R 127110683 + 8 (    1402) [evolution]
  8,16   1        0     9.996220459     0  m   N cfq2577 insert_request
  8,16   1        0     9.996220927     0  m   N cfq2577 add_to_rr
  8,16   1  1198559     9.996221556  2577  U   N [evolution] 1
  8,16   1  1198560     9.997338818     0  C   R 5717760 + 512 ( 1154445) [0]
  8,16   1        0     9.997385039     0  m   N cfq2570 complete rqnoidle 0
  8,16   1        0     9.997386457     0  m   N cfq2570 set_slice=10
  8,16   1        0     9.997387543     0  m   N cfq schedule dispatch
  8,16   1        0     9.997396551     0  m   N cfq2570 slice expired t=0
  8,16   1        0     9.997398888     0  m   N cfq2570 sl_used=1 disp=1 charge=1 iops=0 sect=512
  8,16   1        0     9.997399487     0  m   N cfq2570 del_from_rr
  8,16   1        0     9.997400823     0  m   N cfq workload slice:10
  8,16   1        0     9.997401758     0  m   N cfq2577 set_active wl_prio:0 wl_type:1
  8,16   1        0     9.997403340     0  m   N cfq2577 fifo=  (null)
  8,16   1        0     9.997403945     0  m   N cfq2577 dispatch_insert
  8,16   1        0     9.997405661     0  m   N cfq2577 dispatched a request
  8,16   1        0     9.997406474     0  m   N cfq2577 activate rq, drv=1
  8,16   1  1198561     9.997406966     0  D   R 127110683 + 8 ( 1187146) [swapper/1]
  8,16   1  1198562     9.997512108  2570  Q   R 5718272 + 8 [dd]
  8,16   1  1198563     9.997516538  2570  G   R 5718272 + 8 [dd]
  8,16   1  1198564     9.997517923  2570  P   N [dd]
  8,16   1  1198565     9.997519958  2570  Q   R 5718280 + 8 [dd]
  8,16   1  1198566     9.997520911  2570  M   R 5718280 + 8 [dd]
  8,16   1  1198567     9.997522452  2570  Q   R 5718288 + 8 [dd]
  8,16   1  1198568     9.997522707  2570  M   R 5718288 + 8 [dd]
  8,16   1  1198569     9.997527866  2570  Q   R 5718296 + 8 [dd]
  8,16   1  1198570     9.997528118  2570  M   R 5718296 + 8 [dd]
  8,16   1  1198571     9.997529249  2570  Q   R 5718304 + 8 [dd]
  8,16   1  1198572     9.997529501  2570  M   R 5718304 + 8 [dd]
  8,16   1  1198573     9.997530720  2570  Q   R 5718312 + 8 [dd]
  8,16   1  1198574     9.997531017  2570  M   R 5718312 + 8 [dd]
  8,16   1  1198575     9.997532275  2570  Q   R 5718320 + 8 [dd]
  8,16   1  1198576     9.997532591  2570  M   R 5718320 + 8 [dd]
  8,16   1  1198577     9.997533778  2570  Q   R 5718328 + 8 [dd]
  8,16   1  1198578     9.997534031  2570  M   R 5718328 + 8 [dd]
  8,16   1  1198579     9.997535144  2570  Q   R 5718336 + 8 [dd]
  8,16   1  1198580     9.997535394  2570  M   R 5718336 + 8 [dd]
  8,16   1  1198581     9.997536544  2570  Q   R 5718344 + 8 [dd]
  8,16   1  1198582     9.997536795  2570  M   R 5718344 + 8 [dd]
  8,16   1  1198583     9.997537867  2570  Q   R 5718352 + 8 [dd]
  8,16   1  1198584     9.997538124  2570  M   R 5718352 + 8 [dd]
  8,16   1  1198585     9.997539210  2570  Q   R 5718360 + 8 [dd]
  8,16   1  1198586     9.997539468  2570  M   R 5718360 + 8 [dd]
  8,16   1  1198587     9.997540967  2570  Q   R 5718368 + 8 [dd]
  8,16   1  1198588     9.997541218  2570  M   R 5718368 + 8 [dd]
  8,16   1  1198589     9.997542384  2570  Q   R 5718376 + 8 [dd]
  8,16   1  1198590     9.997542636  2570  M   R 5718376 + 8 [dd]
  8,16   1  1198591     9.997543814  2570  Q   R 5718384 + 8 [dd]
  8,16   1  1198592     9.997544080  2570  M   R 5718384 + 8 [dd]
  8,16   1  1198593     9.997545150  2570  Q   R 5718392 + 8 [dd]
  8,16   1  1198594     9.997545401  2570  M   R 5718392 + 8 [dd]
  8,16   1  1198595     9.997546510  2570  Q   R 5718400 + 8 [dd]
  8,16   1  1198596     9.997546761  2570  M   R 5718400 + 8 [dd]
  8,16   1  1198597     9.997549903  2570  Q   R 5718408 + 8 [dd]
  8,16   1  1198598     9.997550193  2570  M   R 5718408 + 8 [dd]
  8,16   1  1198599     9.997551289  2570  Q   R 5718416 + 8 [dd]
  8,16   1  1198600     9.997551565  2570  M   R 5718416 + 8 [dd]
  8,16   1  1198601     9.997552842  2570  Q   R 5718424 + 8 [dd]
  8,16   1  1198602     9.997553106  2570  M   R 5718424 + 8 [dd]
  8,16   1  1198603     9.997554292  2570  Q   R 5718432 + 8 [dd]
  8,16   1  1198604     9.997554548  2570  M   R 5718432 + 8 [dd]
  8,16   1  1198605     9.997555761  2570  Q   R 5718440 + 8 [dd]
  8,16   1  1198606     9.997556013  2570  M   R 5718440 + 8 [dd]
  8,16   1  1198607     9.997557189  2570  Q   R 5718448 + 8 [dd]
  8,16   1  1198608     9.997557487  2570  M   R 5718448 + 8 [dd]
  8,16   1  1198609     9.997558670  2570  Q   R 5718456 + 8 [dd]
  8,16   1  1198610     9.997558931  2570  M   R 5718456 + 8 [dd]
  8,16   1  1198611     9.997560047  2570  Q   R 5718464 + 8 [dd]
  8,16   1  1198612     9.997560298  2570  M   R 5718464 + 8 [dd]
  8,16   1  1198613     9.997561453  2570  Q   R 5718472 + 8 [dd]
  8,16   1  1198614     9.997561705  2570  M   R 5718472 + 8 [dd]
  8,16   1  1198615     9.997562776  2570  Q   R 5718480 + 8 [dd]
  8,16   1  1198616     9.997563032  2570  M   R 5718480 + 8 [dd]
  8,16   1  1198617     9.997564186  2570  Q   R 5718488 + 8 [dd]
  8,16   1  1198618     9.997564435  2570  M   R 5718488 + 8 [dd]
  8,16   1  1198619     9.997565541  2570  Q   R 5718496 + 8 [dd]
  8,16   1  1198620     9.997565792  2570  M   R 5718496 + 8 [dd]
  8,16   1  1198621     9.997566979  2570  Q   R 5718504 + 8 [dd]
  8,16   1  1198622     9.997567231  2570  M   R 5718504 + 8 [dd]
  8,16   1  1198623     9.997568327  2570  Q   R 5718512 + 8 [dd]
  8,16   1  1198624     9.997568582  2570  M   R 5718512 + 8 [dd]
  8,16   1  1198625     9.997571309  2570  Q   R 5718520 + 8 [dd]
  8,16   1  1198626     9.997571563  2570  M   R 5718520 + 8 [dd]
  8,16   1  1198627     9.997572355  2570  I   R 5718272 + 256 (   55817) [dd]
  8,16   1        0     9.997573394     0  m   N cfq2570 insert_request
  8,16   1        0     9.997574036     0  m   N cfq2570 add_to_rr
  8,16   1  1198628     9.997575577  2570  U   N [dd] 1
  8,16   1  1198629     9.997613973  2570  C   R 127110683 + 8 (  207007) [0]
  8,16   1        0     9.997619361     0  m   N cfq2577 complete rqnoidle 0
  8,16   1        0     9.997619940     0  m   N cfq2577 set_slice=10
  8,16   1        0     9.997620243     0  m   N cfq schedule dispatch
  8,16   1        0     9.997623117     0  m   N cfq2577 slice expired t=0
  8,16   1        0     9.997624406     0  m   N cfq2577 sl_used=1 disp=1 charge=1 iops=0 sect=8
  8,16   1        0     9.997624813     0  m   N cfq2577 del_from_rr
  8,16   1        0     9.997625410     0  m   N cfq workload slice:10
  8,16   1        0     9.997626090     0  m   N cfq2570 set_active wl_prio:0 wl_type:2
  8,16   1        0     9.997626764     0  m   N cfq2570 fifo=  (null)
  8,16   1        0     9.997627143     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.997627812     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.997628296     0  m   N cfq2570 activate rq, drv=1
  8,16   1  1198630     9.997628489  2570  D   R 5718272 + 256 (   56134) [dd]
  8,16   1  1198631     9.997681693  2577  A   R 129160427 + 8 <- (8,21) 54554504
  8,16   1  1198632     9.997681939  2577  Q   R 129160427 + 8 [evolution]
  8,16   1  1198633     9.997683227  2577  G   R 129160427 + 8 [evolution]
  8,16   1  1198634     9.997683956  2577  P   N [evolution]
  8,16   1  1198635     9.997684797  2577  I   R 129160427 + 8 (    1570) [evolution]
  8,16   1        0     9.997685448     0  m   N cfq2577 insert_request
  8,16   1        0     9.997685951     0  m   N cfq2577 add_to_rr
  8,16   1  1198636     9.997686524  2577  U   N [evolution] 1
  8,16   1  1198637     9.997725992  2570  Q   R 5718528 + 8 [dd]
  8,16   1  1198638     9.997727282  2570  G   R 5718528 + 8 [dd]
  8,16   1  1198639     9.997727553  2570  P   N [dd]
  8,16   1  1198640     9.997729012  2570  Q   R 5718536 + 8 [dd]
  8,16   1  1198641     9.997729322  2570  M   R 5718536 + 8 [dd]
  8,16   1  1198642     9.997730493  2570  Q   R 5718544 + 8 [dd]
  8,16   1  1198643     9.997730813  2570  M   R 5718544 + 8 [dd]
  8,16   1  1198644     9.997731992  2570  Q   R 5718552 + 8 [dd]
  8,16   1  1198645     9.997732245  2570  M   R 5718552 + 8 [dd]
  8,16   1  1198646     9.997733320  2570  Q   R 5718560 + 8 [dd]
  8,16   1  1198647     9.997733578  2570  M   R 5718560 + 8 [dd]
  8,16   1  1198648     9.997734835  2570  Q   R 5718568 + 8 [dd]
  8,16   1  1198649     9.997735088  2570  M   R 5718568 + 8 [dd]
  8,16   1  1198650     9.997736172  2570  Q   R 5718576 + 8 [dd]
  8,16   1  1198651     9.997736425  2570  M   R 5718576 + 8 [dd]
  8,16   1  1198652     9.997737500  2570  Q   R 5718584 + 8 [dd]
  8,16   1  1198653     9.997737752  2570  M   R 5718584 + 8 [dd]
  8,16   1  1198654     9.997738832  2570  Q   R 5718592 + 8 [dd]
  8,16   1  1198655     9.997739092  2570  M   R 5718592 + 8 [dd]
  8,16   1  1198656     9.997740163  2570  Q   R 5718600 + 8 [dd]
  8,16   1  1198657     9.997740509  2570  M   R 5718600 + 8 [dd]
  8,16   1  1198658     9.997741600  2570  Q   R 5718608 + 8 [dd]
  8,16   1  1198659     9.997741918  2570  M   R 5718608 + 8 [dd]
  8,16   1  1198660     9.997743063  2570  Q   R 5718616 + 8 [dd]
  8,16   1  1198661     9.997743316  2570  M   R 5718616 + 8 [dd]
  8,16   1  1198662     9.997746451  2570  Q   R 5718624 + 8 [dd]
  8,16   1  1198663     9.997746707  2570  M   R 5718624 + 8 [dd]
  8,16   1  1198664     9.997747824  2570  Q   R 5718632 + 8 [dd]
  8,16   1  1198665     9.997748076  2570  M   R 5718632 + 8 [dd]
  8,16   1  1198666     9.997749317  2570  Q   R 5718640 + 8 [dd]
  8,16   1  1198667     9.997749567  2570  M   R 5718640 + 8 [dd]
  8,16   1  1198668     9.997750656  2570  Q   R 5718648 + 8 [dd]
  8,16   1  1198669     9.997750934  2570  M   R 5718648 + 8 [dd]
  8,16   1  1198670     9.997752046  2570  Q   R 5718656 + 8 [dd]
  8,16   1  1198671     9.997752301  2570  M   R 5718656 + 8 [dd]
  8,16   1  1198672     9.997753442  2570  Q   R 5718664 + 8 [dd]
  8,16   1  1198673     9.997753694  2570  M   R 5718664 + 8 [dd]
  8,16   1  1198674     9.997754778  2570  Q   R 5718672 + 8 [dd]
  8,16   1  1198675     9.997755031  2570  M   R 5718672 + 8 [dd]
  8,16   1  1198676     9.997756129  2570  Q   R 5718680 + 8 [dd]
  8,16   1  1198677     9.997756386  2570  M   R 5718680 + 8 [dd]
  8,16   1  1198678     9.997757480  2570  Q   R 5718688 + 8 [dd]
  8,16   1  1198679     9.997757735  2570  M   R 5718688 + 8 [dd]
  8,16   1  1198680     9.997758801  2570  Q   R 5718696 + 8 [dd]
  8,16   1  1198681     9.997759053  2570  M   R 5718696 + 8 [dd]
  8,16   1  1198682     9.997760136  2570  Q   R 5718704 + 8 [dd]
  8,16   1  1198683     9.997760389  2570  M   R 5718704 + 8 [dd]
  8,16   1  1198684     9.997761476  2570  Q   R 5718712 + 8 [dd]
  8,16   1  1198685     9.997761729  2570  M   R 5718712 + 8 [dd]
  8,16   1  1198686     9.997762839  2570  Q   R 5718720 + 8 [dd]
  8,16   1  1198687     9.997763103  2570  M   R 5718720 + 8 [dd]
  8,16   1  1198688     9.997764177  2570  Q   R 5718728 + 8 [dd]
  8,16   1  1198689     9.997764430  2570  M   R 5718728 + 8 [dd]
  8,16   1  1198690     9.997767140  2570  Q   R 5718736 + 8 [dd]
  8,16   1  1198691     9.997767392  2570  M   R 5718736 + 8 [dd]
  8,16   1  1198692     9.997768505  2570  Q   R 5718744 + 8 [dd]
  8,16   1  1198693     9.997768758  2570  M   R 5718744 + 8 [dd]
  8,16   1  1198694     9.997769907  2570  Q   R 5718752 + 8 [dd]
  8,16   1  1198695     9.997770165  2570  M   R 5718752 + 8 [dd]
  8,16   1  1198696     9.997771391  2570  Q   R 5718760 + 8 [dd]
  8,16   1  1198697     9.997771644  2570  M   R 5718760 + 8 [dd]
  8,16   1  1198698     9.997772715  2570  Q   R 5718768 + 8 [dd]
  8,16   1  1198699     9.997772968  2570  M   R 5718768 + 8 [dd]
  8,16   1  1198700     9.997774126  2570  Q   R 5718776 + 8 [dd]
  8,16   1  1198701     9.997774376  2570  M   R 5718776 + 8 [dd]
  8,16   1  1198702     9.997774818  2570  I   R 5718528 + 256 (   47536) [dd]
  8,16   1        0     9.997775685     0  m   N cfq2570 insert_request
  8,16   1  1198703     9.997776165  2570  U   N [dd] 1
  8,16   1        0     9.997776693     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.997777360     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.997777839     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198704     9.997778021  2570  D   R 5718528 + 256 (    3203) [dd]
  8,16   1  1198705     9.998145175     0  C   R 5718272 + 256 (  516686) [0]
  8,16   1        0     9.998178095     0  m   N cfq2570 complete rqnoidle 0
  8,16   1        0     9.998179391     0  m   N cfq2570 set_slice=10
  8,16   1  1198706     9.998277626  2570  Q   R 5718784 + 8 [dd]
  8,16   1  1198707     9.998282549  2570  G   R 5718784 + 8 [dd]
  8,16   1  1198708     9.998283909  2570  P   N [dd]
  8,16   1  1198709     9.998286466  2570  Q   R 5718792 + 8 [dd]
  8,16   1  1198710     9.998287233  2570  M   R 5718792 + 8 [dd]
  8,16   1  1198711     9.998288819  2570  Q   R 5718800 + 8 [dd]
  8,16   1  1198712     9.998289128  2570  M   R 5718800 + 8 [dd]
  8,16   1  1198713     9.998290463  2570  Q   R 5718808 + 8 [dd]
  8,16   1  1198714     9.998290782  2570  M   R 5718808 + 8 [dd]
  8,16   1  1198715     9.998291960  2570  Q   R 5718816 + 8 [dd]
  8,16   1  1198716     9.998292216  2570  M   R 5718816 + 8 [dd]
  8,16   1  1198717     9.998293452  2570  Q   R 5718824 + 8 [dd]
  8,16   1  1198718     9.998293706  2570  M   R 5718824 + 8 [dd]
  8,16   1  1198719     9.998294826  2570  Q   R 5718832 + 8 [dd]
  8,16   1  1198720     9.998295076  2570  M   R 5718832 + 8 [dd]
  8,16   1  1198721     9.998296188  2570  Q   R 5718840 + 8 [dd]
  8,16   1  1198722     9.998296440  2570  M   R 5718840 + 8 [dd]
  8,16   1  1198723     9.998301292  2570  Q   R 5718848 + 8 [dd]
  8,16   1  1198724     9.998301543  2570  M   R 5718848 + 8 [dd]
  8,16   1  1198725     9.998302671  2570  Q   R 5718856 + 8 [dd]
  8,16   1  1198726     9.998302921  2570  M   R 5718856 + 8 [dd]
  8,16   1  1198727     9.998304137  2570  Q   R 5718864 + 8 [dd]
  8,16   1  1198728     9.998304444  2570  M   R 5718864 + 8 [dd]
  8,16   1  1198729     9.998305612  2570  Q   R 5718872 + 8 [dd]
  8,16   1  1198730     9.998306023  2570  M   R 5718872 + 8 [dd]
  8,16   1  1198731     9.998307278  2570  Q   R 5718880 + 8 [dd]
  8,16   1  1198732     9.998307536  2570  M   R 5718880 + 8 [dd]
  8,16   1  1198733     9.998308749  2570  Q   R 5718888 + 8 [dd]
  8,16   1  1198734     9.998309159  2570  M   R 5718888 + 8 [dd]
  8,16   1  1198735     9.998310428  2570  Q   R 5718896 + 8 [dd]
  8,16   1  1198736     9.998310740  2570  M   R 5718896 + 8 [dd]
  8,16   1  1198737     9.998312007  2570  Q   R 5718904 + 8 [dd]
  8,16   1  1198738     9.998312254  2570  M   R 5718904 + 8 [dd]
  8,16   1  1198739     9.998313452  2570  Q   R 5718912 + 8 [dd]
  8,16   1  1198740     9.998313701  2570  M   R 5718912 + 8 [dd]
  8,16   1  1198741     9.998314969  2570  Q   R 5718920 + 8 [dd]
  8,16   1  1198742     9.998315219  2570  M   R 5718920 + 8 [dd]
  8,16   1  1198743     9.998316383  2570  Q   R 5718928 + 8 [dd]
  8,16   1  1198744     9.998316637  2570  M   R 5718928 + 8 [dd]
  8,16   1  1198745     9.998317744  2570  Q   R 5718936 + 8 [dd]
  8,16   1  1198746     9.998318003  2570  M   R 5718936 + 8 [dd]
  8,16   1  1198747     9.998319175  2570  Q   R 5718944 + 8 [dd]
  8,16   1  1198748     9.998319423  2570  M   R 5718944 + 8 [dd]
  8,16   1  1198749     9.998320514  2570  Q   R 5718952 + 8 [dd]
  8,16   1  1198750     9.998320763  2570  M   R 5718952 + 8 [dd]
  8,16   1  1198751     9.998323664  2570  Q   R 5718960 + 8 [dd]
  8,16   1  1198752     9.998323914  2570  M   R 5718960 + 8 [dd]
  8,16   1  1198753     9.998325013  2570  Q   R 5718968 + 8 [dd]
  8,16   1  1198754     9.998325262  2570  M   R 5718968 + 8 [dd]
  8,16   1  1198755     9.998326374  2570  Q   R 5718976 + 8 [dd]
  8,16   1  1198756     9.998326625  2570  M   R 5718976 + 8 [dd]
  8,16   1  1198757     9.998327712  2570  Q   R 5718984 + 8 [dd]
  8,16   1  1198758     9.998327962  2570  M   R 5718984 + 8 [dd]
  8,16   1  1198759     9.998329053  2570  Q   R 5718992 + 8 [dd]
  8,16   1  1198760     9.998329303  2570  M   R 5718992 + 8 [dd]
  8,16   1  1198761     9.998330389  2570  Q   R 5719000 + 8 [dd]
  8,16   1  1198762     9.998330637  2570  M   R 5719000 + 8 [dd]
  8,16   1  1198763     9.998331785  2570  Q   R 5719008 + 8 [dd]
  8,16   1  1198764     9.998332034  2570  M   R 5719008 + 8 [dd]
  8,16   1  1198765     9.998333123  2570  Q   R 5719016 + 8 [dd]
  8,16   1  1198766     9.998333379  2570  M   R 5719016 + 8 [dd]
  8,16   1  1198767     9.998334469  2570  Q   R 5719024 + 8 [dd]
  8,16   1  1198768     9.998334727  2570  M   R 5719024 + 8 [dd]
  8,16   1  1198769     9.998335818  2570  Q   R 5719032 + 8 [dd]
  8,16   1  1198770     9.998336080  2570  M   R 5719032 + 8 [dd]
  8,16   1  1198771     9.998337279  2570  I   R 5718784 + 256 (   54730) [dd]
  8,16   1        0     9.998338674     0  m   N cfq2570 insert_request
  8,16   1  1198772     9.998340424  2570  U   N [dd] 1
  8,16   1        0     9.998341194     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.998342443     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.998343187     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198773     9.998343485  2570  D   R 5718784 + 256 (    6206) [dd]
  8,16   1  1198774     9.998637335     0  C   R 5718528 + 256 (  859314) [0]
  8,16   1        0     9.998671549     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198775     9.998814006  2570  Q   R 5719040 + 8 [dd]
  8,16   1  1198776     9.998818725  2570  G   R 5719040 + 8 [dd]
  8,16   1  1198777     9.998820090  2570  P   N [dd]
  8,16   1  1198778     9.998822388  2570  Q   R 5719048 + 8 [dd]
  8,16   1  1198779     9.998823161  2570  M   R 5719048 + 8 [dd]
  8,16   1  1198780     9.998824965  2570  Q   R 5719056 + 8 [dd]
  8,16   1  1198781     9.998825233  2570  M   R 5719056 + 8 [dd]
  8,16   1  1198782     9.998826625  2570  Q   R 5719064 + 8 [dd]
  8,16   1  1198783     9.998826882  2570  M   R 5719064 + 8 [dd]
  8,16   1  1198784     9.998831436  2570  Q   R 5719072 + 8 [dd]
  8,16   1  1198785     9.998831701  2570  M   R 5719072 + 8 [dd]
  8,16   1  1198786     9.998832847  2570  Q   R 5719080 + 8 [dd]
  8,16   1  1198787     9.998833188  2570  M   R 5719080 + 8 [dd]
  8,16   1  1198788     9.998834360  2570  Q   R 5719088 + 8 [dd]
  8,16   1  1198789     9.998834619  2570  M   R 5719088 + 8 [dd]
  8,16   1  1198790     9.998835841  2570  Q   R 5719096 + 8 [dd]
  8,16   1  1198791     9.998836104  2570  M   R 5719096 + 8 [dd]
  8,16   1  1198792     9.998837283  2570  Q   R 5719104 + 8 [dd]
  8,16   1  1198793     9.998837538  2570  M   R 5719104 + 8 [dd]
  8,16   1  1198794     9.998838705  2570  Q   R 5719112 + 8 [dd]
  8,16   1  1198795     9.998838966  2570  M   R 5719112 + 8 [dd]
  8,16   1  1198796     9.998840129  2570  Q   R 5719120 + 8 [dd]
  8,16   1  1198797     9.998840383  2570  M   R 5719120 + 8 [dd]
  8,16   1  1198798     9.998841509  2570  Q   R 5719128 + 8 [dd]
  8,16   1  1198799     9.998841785  2570  M   R 5719128 + 8 [dd]
  8,16   1  1198800     9.998843055  2570  Q   R 5719136 + 8 [dd]
  8,16   1  1198801     9.998843306  2570  M   R 5719136 + 8 [dd]
  8,16   1  1198802     9.998844451  2570  Q   R 5719144 + 8 [dd]
  8,16   1  1198803     9.998844701  2570  M   R 5719144 + 8 [dd]
  8,16   1  1198804     9.998845793  2570  Q   R 5719152 + 8 [dd]
  8,16   1  1198805     9.998846046  2570  M   R 5719152 + 8 [dd]
  8,16   1  1198806     9.998847127  2570  Q   R 5719160 + 8 [dd]
  8,16   1  1198807     9.998847379  2570  M   R 5719160 + 8 [dd]
  8,16   1  1198808     9.998848468  2570  Q   R 5719168 + 8 [dd]
  8,16   1  1198809     9.998848724  2570  M   R 5719168 + 8 [dd]
  8,16   1  1198810     9.998849897  2570  Q   R 5719176 + 8 [dd]
  8,16   1  1198811     9.998850313  2570  M   R 5719176 + 8 [dd]
  8,16   1  1198812     9.998853178  2570  Q   R 5719184 + 8 [dd]
  8,16   1  1198813     9.998853436  2570  M   R 5719184 + 8 [dd]
  8,16   1  1198814     9.998854623  2570  Q   R 5719192 + 8 [dd]
  8,16   1  1198815     9.998854872  2570  M   R 5719192 + 8 [dd]
  8,16   1  1198816     9.998855971  2570  Q   R 5719200 + 8 [dd]
  8,16   1  1198817     9.998856224  2570  M   R 5719200 + 8 [dd]
  8,16   1  1198818     9.998857312  2570  Q   R 5719208 + 8 [dd]
  8,16   1  1198819     9.998857588  2570  M   R 5719208 + 8 [dd]
  8,16   1  1198820     9.998859026  2570  Q   R 5719216 + 8 [dd]
  8,16   1  1198821     9.998859281  2570  M   R 5719216 + 8 [dd]
  8,16   1  1198822     9.998860490  2570  Q   R 5719224 + 8 [dd]
  8,16   1  1198823     9.998860758  2570  M   R 5719224 + 8 [dd]
  8,16   1  1198824     9.998861935  2570  Q   R 5719232 + 8 [dd]
  8,16   1  1198825     9.998862185  2570  M   R 5719232 + 8 [dd]
  8,16   1  1198826     9.998863465  2570  Q   R 5719240 + 8 [dd]
  8,16   1  1198827     9.998863731  2570  M   R 5719240 + 8 [dd]
  8,16   1  1198828     9.998864923  2570  Q   R 5719248 + 8 [dd]
  8,16   1  1198829     9.998865180  2570  M   R 5719248 + 8 [dd]
  8,16   1  1198830     9.998866382  2570  Q   R 5719256 + 8 [dd]
  8,16   1  1198831     9.998866632  2570  M   R 5719256 + 8 [dd]
  8,16   1  1198832     9.998867814  2570  Q   R 5719264 + 8 [dd]
  8,16   1  1198833     9.998868069  2570  M   R 5719264 + 8 [dd]
  8,16   1  1198834     9.998869250  2570  Q   R 5719272 + 8 [dd]
  8,16   1  1198835     9.998869505  2570  M   R 5719272 + 8 [dd]
  8,16   1  1198836     9.998870701  2570  Q   R 5719280 + 8 [dd]
  8,16   1  1198837     9.998870955  2570  M   R 5719280 + 8 [dd]
  8,16   1  1198838     9.998872199  2570  Q   R 5719288 + 8 [dd]
  8,16   1  1198839     9.998872450  2570  M   R 5719288 + 8 [dd]
  8,16   1  1198840     9.998873325  2570  I   R 5719040 + 256 (   54600) [dd]
  8,16   1        0     9.998875079     0  m   N cfq2570 insert_request
  8,16   1  1198841     9.998876908  2570  U   N [dd] 1
  8,16   1        0     9.998877704     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.998879045     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.998879788     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198842     9.998880072  2570  D   R 5719040 + 256 (    6747) [dd]
  8,16   1  1198843     9.999128563     0  C   R 5718784 + 256 (  785078) [0]
  8,16   1        0     9.999161339     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198844     9.999263022  2570  Q   R 5719296 + 8 [dd]
  8,16   1  1198845     9.999268323  2570  G   R 5719296 + 8 [dd]
  8,16   1  1198846     9.999269923  2570  P   N [dd]
  8,16   1  1198847     9.999271887  2570  Q   R 5719304 + 8 [dd]
  8,16   1  1198848     9.999272680  2570  M   R 5719304 + 8 [dd]
  8,16   1  1198849     9.999274299  2570  Q   R 5719312 + 8 [dd]
  8,16   1  1198850     9.999274621  2570  M   R 5719312 + 8 [dd]
  8,16   1  1198851     9.999275954  2570  Q   R 5719320 + 8 [dd]
  8,16   1  1198852     9.999276219  2570  M   R 5719320 + 8 [dd]
  8,16   1  1198853     9.999277509  2570  Q   R 5719328 + 8 [dd]
  8,16   1  1198854     9.999277769  2570  M   R 5719328 + 8 [dd]
  8,16   1  1198855     9.999279405  2570  Q   R 5719336 + 8 [dd]
  8,16   1  1198856     9.999279656  2570  M   R 5719336 + 8 [dd]
  8,16   1  1198857     9.999280750  2570  Q   R 5719344 + 8 [dd]
  8,16   1  1198858     9.999281002  2570  M   R 5719344 + 8 [dd]
  8,16   1  1198859     9.999282113  2570  Q   R 5719352 + 8 [dd]
  8,16   1  1198860     9.999282364  2570  M   R 5719352 + 8 [dd]
  8,16   1  1198861     9.999283601  2570  Q   R 5719360 + 8 [dd]
  8,16   1  1198862     9.999283851  2570  M   R 5719360 + 8 [dd]
  8,16   1  1198863     9.999284938  2570  Q   R 5719368 + 8 [dd]
  8,16   1  1198864     9.999285193  2570  M   R 5719368 + 8 [dd]
  8,16   1  1198865     9.999286362  2570  Q   R 5719376 + 8 [dd]
  8,16   1  1198866     9.999286610  2570  M   R 5719376 + 8 [dd]
  8,16   1  1198867     9.999287703  2570  Q   R 5719384 + 8 [dd]
  8,16   1  1198868     9.999287954  2570  M   R 5719384 + 8 [dd]
  8,16   1  1198869     9.999289288  2570  Q   R 5719392 + 8 [dd]
  8,16   1  1198870     9.999289538  2570  M   R 5719392 + 8 [dd]
  8,16   1  1198871     9.999290703  2570  Q   R 5719400 + 8 [dd]
  8,16   1  1198872     9.999290953  2570  M   R 5719400 + 8 [dd]
  8,16   1  1198873     9.999293719  2570  Q   R 5719408 + 8 [dd]
  8,16   1  1198874     9.999293971  2570  M   R 5719408 + 8 [dd]
  8,16   1  1198875     9.999295079  2570  Q   R 5719416 + 8 [dd]
  8,16   1  1198876     9.999295333  2570  M   R 5719416 + 8 [dd]
  8,16   1  1198877     9.999296580  2570  Q   R 5719424 + 8 [dd]
  8,16   1  1198878     9.999296831  2570  M   R 5719424 + 8 [dd]
  8,16   1  1198879     9.999297922  2570  Q   R 5719432 + 8 [dd]
  8,16   1  1198880     9.999298173  2570  M   R 5719432 + 8 [dd]
  8,16   1  1198881     9.999299271  2570  Q   R 5719440 + 8 [dd]
  8,16   1  1198882     9.999299532  2570  M   R 5719440 + 8 [dd]
  8,16   1  1198883     9.999300619  2570  Q   R 5719448 + 8 [dd]
  8,16   1  1198884     9.999300871  2570  M   R 5719448 + 8 [dd]
  8,16   1  1198885     9.999302052  2570  Q   R 5719456 + 8 [dd]
  8,16   1  1198886     9.999302303  2570  M   R 5719456 + 8 [dd]
  8,16   1  1198887     9.999303397  2570  Q   R 5719464 + 8 [dd]
  8,16   1  1198888     9.999303647  2570  M   R 5719464 + 8 [dd]
  8,16   1  1198889     9.999304900  2570  Q   R 5719472 + 8 [dd]
  8,16   1  1198890     9.999305209  2570  M   R 5719472 + 8 [dd]
  8,16   1  1198891     9.999306412  2570  Q   R 5719480 + 8 [dd]
  8,16   1  1198892     9.999306675  2570  M   R 5719480 + 8 [dd]
  8,16   1  1198893     9.999307848  2570  Q   R 5719488 + 8 [dd]
  8,16   1  1198894     9.999308107  2570  M   R 5719488 + 8 [dd]
  8,16   1  1198895     9.999309220  2570  Q   R 5719496 + 8 [dd]
  8,16   1  1198896     9.999309471  2570  M   R 5719496 + 8 [dd]
  8,16   1  1198897     9.999310581  2570  Q   R 5719504 + 8 [dd]
  8,16   1  1198898     9.999310875  2570  M   R 5719504 + 8 [dd]
  8,16   1  1198899     9.999312017  2570  Q   R 5719512 + 8 [dd]
  8,16   1  1198900     9.999312272  2570  M   R 5719512 + 8 [dd]
  8,16   1  1198901     9.999315106  2570  Q   R 5719520 + 8 [dd]
  8,16   1  1198902     9.999315358  2570  M   R 5719520 + 8 [dd]
  8,16   1  1198903     9.999316495  2570  Q   R 5719528 + 8 [dd]
  8,16   1  1198904     9.999316746  2570  M   R 5719528 + 8 [dd]
  8,16   1  1198905     9.999317852  2570  Q   R 5719536 + 8 [dd]
  8,16   1  1198906     9.999318103  2570  M   R 5719536 + 8 [dd]
  8,16   1  1198907     9.999319271  2570  Q   R 5719544 + 8 [dd]
  8,16   1  1198908     9.999319523  2570  M   R 5719544 + 8 [dd]
  8,16   1  1198909     9.999320448  2570  I   R 5719296 + 256 (   52125) [dd]
  8,16   1        0     9.999321895     0  m   N cfq2570 insert_request
  8,16   1  1198910     9.999323661  2570  U   N [dd] 1
  8,16   1        0     9.999324400     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.999325762     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.999326490     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198911     9.999326769  2570  D   R 5719296 + 256 (    6321) [dd]
  8,16   1  1198912     9.999615787     0  C   R 5719040 + 256 (  735715) [0]
  8,16   1        0     9.999647801     0  m   N cfq2570 complete rqnoidle 0
  8,16   1  1198913     9.999800271  2570  Q   R 5719552 + 8 [dd]
  8,16   1  1198914     9.999805251  2570  G   R 5719552 + 8 [dd]
  8,16   1  1198915     9.999806646  2570  P   N [dd]
  8,16   1  1198916     9.999808904  2570  Q   R 5719560 + 8 [dd]
  8,16   1  1198917     9.999809719  2570  M   R 5719560 + 8 [dd]
  8,16   1  1198918     9.999811296  2570  Q   R 5719568 + 8 [dd]
  8,16   1  1198919     9.999811551  2570  M   R 5719568 + 8 [dd]
  8,16   1  1198920     9.999812779  2570  Q   R 5719576 + 8 [dd]
  8,16   1  1198921     9.999813096  2570  M   R 5719576 + 8 [dd]
  8,16   1  1198922     9.999814316  2570  Q   R 5719584 + 8 [dd]
  8,16   1  1198923     9.999814568  2570  M   R 5719584 + 8 [dd]
  8,16   1  1198924     9.999815662  2570  Q   R 5719592 + 8 [dd]
  8,16   1  1198925     9.999815914  2570  M   R 5719592 + 8 [dd]
  8,16   1  1198926     9.999817011  2570  Q   R 5719600 + 8 [dd]
  8,16   1  1198927     9.999817262  2570  M   R 5719600 + 8 [dd]
  8,16   1  1198928     9.999818366  2570  Q   R 5719608 + 8 [dd]
  8,16   1  1198929     9.999818628  2570  M   R 5719608 + 8 [dd]
  8,16   1  1198930     9.999819792  2570  Q   R 5719616 + 8 [dd]
  8,16   1  1198931     9.999820044  2570  M   R 5719616 + 8 [dd]
  8,16   1  1198932     9.999821139  2570  Q   R 5719624 + 8 [dd]
  8,16   1  1198933     9.999821389  2570  M   R 5719624 + 8 [dd]
  8,16   1  1198934     9.999825670  2570  Q   R 5719632 + 8 [dd]
  8,16   1  1198935     9.999825923  2570  M   R 5719632 + 8 [dd]
  8,16   1  1198936     9.999827154  2570  Q   R 5719640 + 8 [dd]
  8,16   1  1198937     9.999827405  2570  M   R 5719640 + 8 [dd]
  8,16   1  1198938     9.999828602  2570  Q   R 5719648 + 8 [dd]
  8,16   1  1198939     9.999828854  2570  M   R 5719648 + 8 [dd]
  8,16   1  1198940     9.999830028  2570  Q   R 5719656 + 8 [dd]
  8,16   1  1198941     9.999830355  2570  M   R 5719656 + 8 [dd]
  8,16   1  1198942     9.999831553  2570  Q   R 5719664 + 8 [dd]
  8,16   1  1198943     9.999831805  2570  M   R 5719664 + 8 [dd]
  8,16   1  1198944     9.999832905  2570  Q   R 5719672 + 8 [dd]
  8,16   1  1198945     9.999833157  2570  M   R 5719672 + 8 [dd]
  8,16   1  1198946     9.999834350  2570  Q   R 5719680 + 8 [dd]
  8,16   1  1198947     9.999834602  2570  M   R 5719680 + 8 [dd]
  8,16   1  1198948     9.999835704  2570  Q   R 5719688 + 8 [dd]
  8,16   1  1198949     9.999835957  2570  M   R 5719688 + 8 [dd]
  8,16   1  1198950     9.999837061  2570  Q   R 5719696 + 8 [dd]
  8,16   1  1198951     9.999837310  2570  M   R 5719696 + 8 [dd]
  8,16   1  1198952     9.999838418  2570  Q   R 5719704 + 8 [dd]
  8,16   1  1198953     9.999838674  2570  M   R 5719704 + 8 [dd]
  8,16   1  1198954     9.999839902  2570  Q   R 5719712 + 8 [dd]
  8,16   1  1198955     9.999840154  2570  M   R 5719712 + 8 [dd]
  8,16   1  1198956     9.999841286  2570  Q   R 5719720 + 8 [dd]
  8,16   1  1198957     9.999841537  2570  M   R 5719720 + 8 [dd]
  8,16   1  1198958     9.999842684  2570  Q   R 5719728 + 8 [dd]
  8,16   1  1198959     9.999842936  2570  M   R 5719728 + 8 [dd]
  8,16   1  1198960     9.999844037  2570  Q   R 5719736 + 8 [dd]
  8,16   1  1198961     9.999844298  2570  M   R 5719736 + 8 [dd]
  8,16   1  1198962     9.999847070  2570  Q   R 5719744 + 8 [dd]
  8,16   1  1198963     9.999847330  2570  M   R 5719744 + 8 [dd]
  8,16   1  1198964     9.999848422  2570  Q   R 5719752 + 8 [dd]
  8,16   1  1198965     9.999848672  2570  M   R 5719752 + 8 [dd]
  8,16   1  1198966     9.999850021  2570  Q   R 5719760 + 8 [dd]
  8,16   1  1198967     9.999850272  2570  M   R 5719760 + 8 [dd]
  8,16   1  1198968     9.999851440  2570  Q   R 5719768 + 8 [dd]
  8,16   1  1198969     9.999851714  2570  M   R 5719768 + 8 [dd]
  8,16   1  1198970     9.999852878  2570  Q   R 5719776 + 8 [dd]
  8,16   1  1198971     9.999853133  2570  M   R 5719776 + 8 [dd]
  8,16   1  1198972     9.999854239  2570  Q   R 5719784 + 8 [dd]
  8,16   1  1198973     9.999854495  2570  M   R 5719784 + 8 [dd]
  8,16   1  1198974     9.999855595  2570  Q   R 5719792 + 8 [dd]
  8,16   1  1198975     9.999855849  2570  M   R 5719792 + 8 [dd]
  8,16   1  1198976     9.999856947  2570  Q   R 5719800 + 8 [dd]
  8,16   1  1198977     9.999857198  2570  M   R 5719800 + 8 [dd]
  8,16   1  1198978     9.999858058  2570  I   R 5719552 + 256 (   52807) [dd]
  8,16   1        0     9.999859378     0  m   N cfq2570 insert_request
  8,16   1  1198979     9.999861062  2570  U   N [dd] 1
  8,16   1        0     9.999861854     0  m   N cfq2570 dispatch_insert
  8,16   1        0     9.999863078     0  m   N cfq2570 dispatched a request
  8,16   1        0     9.999863763     0  m   N cfq2570 activate rq, drv=2
  8,16   1  1198980     9.999864420  2570  D   R 5719552 + 256 (    6362) [dd]
CPU0 (sdb):
 Reads Queued:       60971,   247656KiB	 Writes Queued:           4,       16KiB
 Read Dispatches:     2015,   233044KiB	 Write Dispatches:        0,        0KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:      107,    13824KiB	 Writes Completed:        0,        0KiB
 Read Merges:        58809,   235564KiB	 Write Merges:            2,        8KiB
 Read depth:             8        	 Write depth:             1
 IO unplugs:          1955        	 Timer unplugs:           0
CPU1 (sdb):
 Reads Queued:      554679,     2223MiB	 Writes Queued:          13,       52KiB
 Read Dispatches:    17603,     2238MiB	 Write Dispatches:        5,       68KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:    19509,     2457MiB	 Writes Completed:        7,       68KiB
 Read Merges:       537222,     2149MiB	 Write Merges:           10,       40KiB
 Read depth:             8        	 Write depth:             1
 IO unplugs:         17408        	 Timer unplugs:           0

Total (sdb):
 Reads Queued:      615650,     2471MiB	 Writes Queued:          17,       68KiB
 Read Dispatches:    19618,     2471MiB	 Write Dispatches:        5,       68KiB
 Reads Requeued:         0		 Writes Requeued:         0
 Reads Completed:    19616,     2471MiB	 Writes Completed:        7,       68KiB
 Read Merges:       596031,     2385MiB	 Write Merges:           12,       48KiB
 IO unplugs:         19363        	 Timer unplugs:           0

Throughput (R/W): 247140KiB/s / 6KiB/s
Events (sdb): 1434007 entries
Skips: 0 forward (0 -   0.0%)
Input file sdb.blktrace.0 added
Input file sdb.blktrace.1 added