* Bad SSD performance with recent kernels @ 2012-01-27 6:00 Herbert Poetzl 2012-01-27 6:44 ` Eric Dumazet 2012-01-28 12:51 ` Wu Fengguang 0 siblings, 2 replies; 36+ messages in thread From: Herbert Poetzl @ 2012-01-27 6:00 UTC (permalink / raw) To: Linux Kernel ML Dear Community! Recently I decided to update the kernel on a Dell Laptop to a more recent version than 2.6.38.x, but experienced bad I/O performance with the new kernel, so I started to dig a little deeper and ended up with the following test: 1) download and extract kernel (on 2.6.38.8) 2) make defconfig 3) make localmodconfig 4) make 5) make modules_install install Then I booted each kernel in single user and ran the following test script: echo noop >/sys/class/block/sda/queue/scheduler for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ ionice -c0 nice -20 \ dd if=/dev/sda of=/dev/null bs=1M count=20480 echo deadline >/sys/class/block/sda/queue/scheduler for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ ionice -c0 nice -20 \ dd if=/dev/sda of=/dev/null bs=1M count=20480 echo cfq >/sys/class/block/sda/queue/scheduler for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ ionice -c0 nice -20 \ dd if=/dev/sda of=/dev/null bs=1M count=20480 note that the Laptop is a relatively modern Latitude E6400 with a Samsung 830 Series 256GB SSD here are the surprising results: @ linux 2.6.38.8 248 MB/s real = 86.74, user = 0.01, sys = 21.65, 24% cpu 248 MB/s real = 86.81, user = 0.02, sys = 21.75, 25% cpu 251 MB/s real = 85.63, user = 0.01, sys = 22.24, 25% cpu @ linux 2.6.39.4 49.0 MB/s real = 438.79, user = 0.01, sys = 19.79, 4% cpu 25.7 MB/s real = 836.70, user = 0.02, sys = 18.39, 2% cpu 27.7 MB/s real = 776.53, user = 0.01, sys = 16.03, 2% cpu @ linux 3.0.18 48.9 MB/s real = 439.07, user = 0.01, sys = 17.55, 4% cpu 25.0 MB/s real = 859.03, user = 0.01, sys = 16.97, 1% cpu 49.8 MB/s real = 431.61, user = 0.01, sys = 16.68, 3% cpu @ linux 3.1.10 54.0 MB/s real = 398.23, user = 0.01, sys = 17.36, 4% cpu 29.4 MB/s real = 731.47, user = 0.01, sys = 17.14, 2% cpu 25.0 MB/s real = 859.35, user = 0.01, sys = 14.51, 1% cpu @ linux 3.2.2 45.8 MB/s real = 468.85, user = 0.01, sys = 17.11, 3% cpu 44.8 MB/s real = 478.92, user = 0.01, sys = 17.02, 3% cpu 45.0 MB/s real = 476.91, user = 0.01, sys = 16.14, 3% cpu I have no idea why the I/O performance is that bad on any kernel newer than 2.6.38.x, but I'm happy to test and/or try various configurations as time permits ... The detailed test results as well as the dmesg and config of each kernel can be found here: http://vserver.13thfloor.at/Stuff/SSD many thanks in advance, Herbert here some technical information: http://www.dell.com/us/dfb/p/latitude-e6400/pd SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03) http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-27 6:00 Bad SSD performance with recent kernels Herbert Poetzl @ 2012-01-27 6:44 ` Eric Dumazet 2012-01-28 12:51 ` Wu Fengguang 1 sibling, 0 replies; 36+ messages in thread From: Eric Dumazet @ 2012-01-27 6:44 UTC (permalink / raw) To: Herbert Poetzl; +Cc: Linux Kernel ML Le vendredi 27 janvier 2012 à 07:00 +0100, Herbert Poetzl a écrit : > Dear Community! > > Recently I decided to update the kernel on a Dell Laptop to > a more recent version than 2.6.38.x, but experienced bad > I/O performance with the new kernel, so I started to dig > a little deeper and ended up with the following test: > > 1) download and extract kernel (on 2.6.38.8) > 2) make defconfig > 3) make localmodconfig > 4) make > 5) make modules_install install > > Then I booted each kernel in single user and ran the following > test script: > > echo noop >/sys/class/block/sda/queue/scheduler > for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done > /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ > ionice -c0 nice -20 \ > dd if=/dev/sda of=/dev/null bs=1M count=20480 > > echo deadline >/sys/class/block/sda/queue/scheduler > for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done > /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ > ionice -c0 nice -20 \ > dd if=/dev/sda of=/dev/null bs=1M count=20480 > > echo cfq >/sys/class/block/sda/queue/scheduler > for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done > /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ > ionice -c0 nice -20 \ > dd if=/dev/sda of=/dev/null bs=1M count=20480 > > > note that the Laptop is a relatively modern Latitude E6400 > with a Samsung 830 Series 256GB SSD > > here are the surprising results: > > @ linux 2.6.38.8 > 248 MB/s real = 86.74, user = 0.01, sys = 21.65, 24% cpu > 248 MB/s real = 86.81, user = 0.02, sys = 21.75, 25% cpu > 251 MB/s real = 85.63, user = 0.01, sys = 22.24, 25% cpu > > @ linux 2.6.39.4 > 49.0 MB/s real = 438.79, user = 0.01, sys = 19.79, 4% cpu > 25.7 MB/s real = 836.70, user = 0.02, sys = 18.39, 2% cpu > 27.7 MB/s real = 776.53, user = 0.01, sys = 16.03, 2% cpu > > @ linux 3.0.18 > 48.9 MB/s real = 439.07, user = 0.01, sys = 17.55, 4% cpu > 25.0 MB/s real = 859.03, user = 0.01, sys = 16.97, 1% cpu > 49.8 MB/s real = 431.61, user = 0.01, sys = 16.68, 3% cpu > > @ linux 3.1.10 > 54.0 MB/s real = 398.23, user = 0.01, sys = 17.36, 4% cpu > 29.4 MB/s real = 731.47, user = 0.01, sys = 17.14, 2% cpu > 25.0 MB/s real = 859.35, user = 0.01, sys = 14.51, 1% cpu > > @ linux 3.2.2 > 45.8 MB/s real = 468.85, user = 0.01, sys = 17.11, 3% cpu > 44.8 MB/s real = 478.92, user = 0.01, sys = 17.02, 3% cpu > 45.0 MB/s real = 476.91, user = 0.01, sys = 16.14, 3% cpu > > > I have no idea why the I/O performance is that bad on any > kernel newer than 2.6.38.x, but I'm happy to test and/or > try various configurations as time permits ... > > The detailed test results as well as the dmesg and config > of each kernel can be found here: > http://vserver.13thfloor.at/Stuff/SSD > > many thanks in advance, > Herbert > > here some technical information: > > http://www.dell.com/us/dfb/p/latitude-e6400/pd > SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03) > > http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec > Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q > supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS > > I confirm I just noticed this on a brand new SSD from Crucial (128 GB) and a Dell E6400 too (3.5 years old model) The Ubuntu upgrade from 11.04 to 11.10 switched kernel (from 2.6.38.X to 3.0.0.X) and raw SSD performance (hdparm -t /dev/sda) dropped from 200 MB/s to 140 MB/s or so. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-27 6:00 Bad SSD performance with recent kernels Herbert Poetzl 2012-01-27 6:44 ` Eric Dumazet @ 2012-01-28 12:51 ` Wu Fengguang 2012-01-28 13:33 ` Eric Dumazet 2012-01-28 17:01 ` Herbert Poetzl 1 sibling, 2 replies; 36+ messages in thread From: Wu Fengguang @ 2012-01-28 12:51 UTC (permalink / raw) To: Herbert Poetzl; +Cc: Andrew Morton, LKML, Jens Axboe, Tejun Heo Hi Herbert, On Fri, Jan 27, 2012 at 07:00:34AM +0100, Herbert Poetzl wrote: > > Dear Community! > > Recently I decided to update the kernel on a Dell Laptop to > a more recent version than 2.6.38.x, but experienced bad > I/O performance with the new kernel, so I started to dig > a little deeper and ended up with the following test: > > 1) download and extract kernel (on 2.6.38.8) > 2) make defconfig > 3) make localmodconfig > 4) make > 5) make modules_install install > > Then I booted each kernel in single user and ran the following > test script: > > echo noop >/sys/class/block/sda/queue/scheduler > for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done > /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ > ionice -c0 nice -20 \ > dd if=/dev/sda of=/dev/null bs=1M count=20480 > > echo deadline >/sys/class/block/sda/queue/scheduler > for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done > /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ > ionice -c0 nice -20 \ > dd if=/dev/sda of=/dev/null bs=1M count=20480 > > echo cfq >/sys/class/block/sda/queue/scheduler > for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done > /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ > ionice -c0 nice -20 \ > dd if=/dev/sda of=/dev/null bs=1M count=20480 > > > note that the Laptop is a relatively modern Latitude E6400 > with a Samsung 830 Series 256GB SSD > > here are the surprising results: > > @ linux 2.6.38.8 > 248 MB/s real = 86.74, user = 0.01, sys = 21.65, 24% cpu > 248 MB/s real = 86.81, user = 0.02, sys = 21.75, 25% cpu > 251 MB/s real = 85.63, user = 0.01, sys = 22.24, 25% cpu > > @ linux 2.6.39.4 > 49.0 MB/s real = 438.79, user = 0.01, sys = 19.79, 4% cpu > 25.7 MB/s real = 836.70, user = 0.02, sys = 18.39, 2% cpu > 27.7 MB/s real = 776.53, user = 0.01, sys = 16.03, 2% cpu > > @ linux 3.0.18 > 48.9 MB/s real = 439.07, user = 0.01, sys = 17.55, 4% cpu > 25.0 MB/s real = 859.03, user = 0.01, sys = 16.97, 1% cpu > 49.8 MB/s real = 431.61, user = 0.01, sys = 16.68, 3% cpu > > @ linux 3.1.10 > 54.0 MB/s real = 398.23, user = 0.01, sys = 17.36, 4% cpu > 29.4 MB/s real = 731.47, user = 0.01, sys = 17.14, 2% cpu > 25.0 MB/s real = 859.35, user = 0.01, sys = 14.51, 1% cpu > > @ linux 3.2.2 > 45.8 MB/s real = 468.85, user = 0.01, sys = 17.11, 3% cpu > 44.8 MB/s real = 478.92, user = 0.01, sys = 17.02, 3% cpu > 45.0 MB/s real = 476.91, user = 0.01, sys = 16.14, 3% cpu Thanks for the through tests! However I cannot reproduce the regressions, but see slightly better performance in 3.2: Linux lkp-nex04 3.2.0-rc7-shli+ #121 SMP Thu Jan 19 18:10:45 CST 2012 x86_64 x86_64 x86_64 GNU/Linux 21474836480 bytes (21 GB) copied, 90.9307 s, 236 MB/s real = 91.03, user = 0.01, sys = 25.80, 28% cpu 21474836480 bytes (21 GB) copied, 90.8864 s, 236 MB/s real = 90.90, user = 0.01, sys = 25.54, 28% cpu 21474836480 bytes (21 GB) copied, 93.4684 s, 230 MB/s real = 93.47, user = 0.02, sys = 25.12, 26% cpu Linux lkp-nex04 2.6.38 #334 SMP Sat Jan 28 20:16:25 CST 2012 x86_64 GNU/Linux 21474836480 bytes (21 GB) copied, 96.2382 s, 223 MB/s real = 96.29, user = 0.01, sys = 57.64, 59% cpu 21474836480 bytes (21 GB) copied, 96.5796 s, 222 MB/s real = 96.60, user = 0.02, sys = 58.08, 60% cpu 21474836480 bytes (21 GB) copied, 95.486 s, 225 MB/s real = 95.51, user = 0.00, sys = 58.20, 60% cpu My test box has 64GB memory and a dozen 80GB INTEL SSDSA2M080G2GN SSD drives attached to 87:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02) > I have no idea why the I/O performance is that bad on any > kernel newer than 2.6.38.x, but I'm happy to test and/or > try various configurations as time permits ... Would you please create a filesystem and large file on sda and run the tests on the file? There was some performance bug on reading the raw /dev/sda device file.. > The detailed test results as well as the dmesg and config > of each kernel can be found here: > http://vserver.13thfloor.at/Stuff/SSD It would be better for such files be attached in future. (For one thing I get timeout when trying to open the link...) Thanks, Fengguang > here some technical information: > > http://www.dell.com/us/dfb/p/latitude-e6400/pd > SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03) > > http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec > Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q > supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS > > > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-28 12:51 ` Wu Fengguang @ 2012-01-28 13:33 ` Eric Dumazet 2012-01-29 5:59 ` Wu Fengguang 2012-01-28 17:01 ` Herbert Poetzl 1 sibling, 1 reply; 36+ messages in thread From: Eric Dumazet @ 2012-01-28 13:33 UTC (permalink / raw) To: Wu Fengguang; +Cc: Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit : > Would you please create a filesystem and large file on sda and run the > tests on the file? There was some performance bug on reading the raw > /dev/sda device file.. Hmm... latest kernel has the performance bug right now. Really if /dev/sda is slow, we are stuck. FYI, I started a bisection. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-28 13:33 ` Eric Dumazet @ 2012-01-29 5:59 ` Wu Fengguang 2012-01-29 8:42 ` Herbert Poetzl 2012-01-29 10:03 ` Eric Dumazet 0 siblings, 2 replies; 36+ messages in thread From: Wu Fengguang @ 2012-01-29 5:59 UTC (permalink / raw) To: Eric Dumazet; +Cc: Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Sat, Jan 28, 2012 at 02:33:31PM +0100, Eric Dumazet wrote: > Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit : > > > Would you please create a filesystem and large file on sda and run the > > tests on the file? There was some performance bug on reading the raw > > /dev/sda device file.. > > Hmm... latest kernel has the performance bug right now. > > Really if /dev/sda is slow, we are stuck. What's the block size? If it's < 4k, performance might be hurt. blockdev --getbsz /dev/sda > FYI, I started a bisection. Thank you! If the bisection would take much human time, it should be easier to collect some blktrace data on reading /dev/sda for analyzes. Thanks, Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 5:59 ` Wu Fengguang @ 2012-01-29 8:42 ` Herbert Poetzl 2012-01-29 9:28 ` Wu Fengguang 2012-01-29 10:03 ` Eric Dumazet 1 sibling, 1 reply; 36+ messages in thread From: Herbert Poetzl @ 2012-01-29 8:42 UTC (permalink / raw) To: Wu Fengguang; +Cc: Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Sun, Jan 29, 2012 at 01:59:17PM +0800, Wu Fengguang wrote: > On Sat, Jan 28, 2012 at 02:33:31PM +0100, Eric Dumazet wrote: >> Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit : >>> Would you please create a filesystem and large file on sda >>> and run the tests on the file? There was some performance bug >>> on reading the raw /dev/sda device file.. as promised, I did the tests on a filesystem, created on a partition of the disk, and here are the (IMHO quite interesting) results: kernel -- write --- ------------------read ----------------- --- noop --- --- noop --- - deadline - ---- cfs --- [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU ---------------------------------------------------------------- 2.6.38.8 268.76 49.6 169.20 11.3 169.17 11.3 167.89 11.4 2.6.39.4 269.73 50.3 162.03 10.9 161.58 10.9 161.64 11.0 3.0.18 269.17 42.0 161.87 9.9 161.36 10.0 161.68 10.1 3.1.10 271.62 43.1 161.91 9.9 161.68 9.9 161.25 10.1 3.2.2 270.95 42.6 162.36 9.9 162.63 9.9 162.65 10.1 so while the 'expected' performance should be somewhere around 300MB/s for read and write (raw disk access) we end up with good write performance and roughly half the read performance with 'dd bs=1M' on ext3 here the script I used: mke2fs -j /dev/sda5 mount /dev/sda5 /media /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ ionice -c0 nice -20 \ dd if=/dev/zero of=/media/zero.data bs=1M count=19900 echo noop >/sys/class/block/sda/queue/scheduler for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ ionice -c0 nice -20 \ dd if=/media/zero.data of=/dev/null bs=1M count=19900 echo deadline >/sys/class/block/sda/queue/scheduler for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ ionice -c0 nice -20 \ dd if=/media/zero.data of=/dev/null bs=1M count=19900 echo cfq >/sys/class/block/sda/queue/scheduler for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ ionice -c0 nice -20 \ dd if=/media/zero.data of=/dev/null bs=1M count=19900 >> Hmm... latest kernel has the performance bug right now. >> Really if /dev/sda is slow, we are stuck. > What's the block size? If it's < 4k, performance might be hurt. > blockdev --getbsz /dev/sda 4096 >> FYI, I started a bisection. > Thank you! If the bisection would take much human time, it should be > easier to collect some blktrace data on reading /dev/sda for analyzes. will do some bonnie++ tests on the partition later today HTH, Herbert > Thanks, > Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 8:42 ` Herbert Poetzl @ 2012-01-29 9:28 ` Wu Fengguang 0 siblings, 0 replies; 36+ messages in thread From: Wu Fengguang @ 2012-01-29 9:28 UTC (permalink / raw) To: Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Sun, Jan 29, 2012 at 09:42:59AM +0100, Herbert Poetzl wrote: > On Sun, Jan 29, 2012 at 01:59:17PM +0800, Wu Fengguang wrote: > > On Sat, Jan 28, 2012 at 02:33:31PM +0100, Eric Dumazet wrote: > >> Le samedi 28 janvier 2012 à 20:51 +0800, Wu Fengguang a écrit : > > >>> Would you please create a filesystem and large file on sda > >>> and run the tests on the file? There was some performance bug > >>> on reading the raw /dev/sda device file.. > > as promised, I did the tests on a filesystem, created on > a partition of the disk, and here are the (IMHO quite > interesting) results: > > kernel -- write --- ------------------read ----------------- > --- noop --- --- noop --- - deadline - ---- cfs --- > [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU > ---------------------------------------------------------------- > 2.6.38.8 268.76 49.6 169.20 11.3 169.17 11.3 167.89 11.4 Hmm, read performance drops between 2.6.38 and 2.6.39... > 2.6.39.4 269.73 50.3 162.03 10.9 161.58 10.9 161.64 11.0 > 3.0.18 269.17 42.0 161.87 9.9 161.36 10.0 161.68 10.1 Between 3.0 and 3.1, the writeback chunk size is raised by commit 1a12d8bd7b2998b ("writeback: scale IO chunk size up to half device bandwidth") which should be the main reason for the improved write throughput. > 3.1.10 271.62 43.1 161.91 9.9 161.68 9.9 161.25 10.1 > 3.2.2 270.95 42.6 162.36 9.9 162.63 9.9 162.65 10.1 > > so while the 'expected' performance should be somewhere around > 300MB/s for read and write (raw disk access) we end up with > good write performance and roughly half the read performance > with 'dd bs=1M' on ext3 That could be explained by large write chunk size (>=4MB) and small readahead size (128KB). Long time ago I collected some read experiments on SSD and find it asks for 4MB readahead size to get best performance: SSD 80G Intel x25-M SSDSA2M080 (reported by Li Shaohua) rasize 1st run 2nd run ---------------------------------- 4k 123 MB/s 122 MB/s 16k 153 MB/s 153 MB/s 32k 161 MB/s 162 MB/s 64k 167 MB/s 168 MB/s 128k 197 MB/s 197 MB/s 256k 217 MB/s 217 MB/s 512k 238 MB/s 234 MB/s 1M 251 MB/s 248 MB/s 2M 259 MB/s 257 MB/s ==> 4M 269 MB/s 264 MB/s 8M 266 MB/s 266 MB/s Note that ==> points to the readahead size that yields plateau throughput. SSD 22G MARVELL SD88SA02 MP1F (reported by Jens Axboe) rasize 1st 2nd -------------------------------- 4k 41 MB/s 41 MB/s 16k 85 MB/s 81 MB/s 32k 102 MB/s 109 MB/s 64k 125 MB/s 144 MB/s 128k 183 MB/s 185 MB/s 256k 216 MB/s 216 MB/s 512k 216 MB/s 236 MB/s 1024k 251 MB/s 252 MB/s 2M 258 MB/s 258 MB/s ==> 4M 266 MB/s 266 MB/s 8M 266 MB/s 266 MB/s Thanks, Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 5:59 ` Wu Fengguang 2012-01-29 8:42 ` Herbert Poetzl @ 2012-01-29 10:03 ` Eric Dumazet 2012-01-29 11:16 ` Wu Fengguang 2012-01-30 14:48 ` Wu Fengguang 1 sibling, 2 replies; 36+ messages in thread From: Eric Dumazet @ 2012-01-29 10:03 UTC (permalink / raw) To: Wu Fengguang; +Cc: Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo Le dimanche 29 janvier 2012 à 13:59 +0800, Wu Fengguang a écrit : > What's the block size? If it's < 4k, performance might be hurt. > > blockdev --getbsz /dev/sda > # blockdev --getbsz /dev/sda 4096 > > FYI, I started a bisection. > > Thank you! If the bisection would take much human time, it should be > easier to collect some blktrace data on reading /dev/sda for analyzes. > Very strange, my bissection ended on following commit : commit 805f6b5e1cbfedfb9b3d354013e7f4b13a79270f Author: Tao Ma <boyu.mt@taobao.com> Date: Fri Mar 11 20:11:59 2011 +0100 blktrace: Use rq->cmd_flags directly in blk_add_trace_rq. This makes no sense. hdparm uses 2MB block reads, so read_ahead (128KB) is too small for best perf # cat /sys/class/block/sda/queue/read_ahead_kb 128 # dd if=/dev/sda of=/dev/null bs=128k ^C 63744+0 enregistrements lus 63743+0 enregistrements écrits 8354922496 octets (8,4 GB) copiés, 39,975 s, 209 MB/s # hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 510 MB in 3.00 seconds = 169.75 MB/sec # uname -a Linux edumazet-laptop 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20 17:23:00 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 10:03 ` Eric Dumazet @ 2012-01-29 11:16 ` Wu Fengguang 2012-01-29 13:13 ` Eric Dumazet 2012-01-30 14:48 ` Wu Fengguang 1 sibling, 1 reply; 36+ messages in thread From: Wu Fengguang @ 2012-01-29 11:16 UTC (permalink / raw) To: Eric Dumazet; +Cc: Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Sun, Jan 29, 2012 at 11:03:00AM +0100, Eric Dumazet wrote: > Le dimanche 29 janvier 2012 à 13:59 +0800, Wu Fengguang a écrit : > > > What's the block size? If it's < 4k, performance might be hurt. > > > > blockdev --getbsz /dev/sda > > > > # blockdev --getbsz /dev/sda > 4096 OK, it's normal value. > > > FYI, I started a bisection. > > > > Thank you! If the bisection would take much human time, it should be > > easier to collect some blktrace data on reading /dev/sda for analyzes. > > > > Very strange, my bissection ended on following commit : > > commit 805f6b5e1cbfedfb9b3d354013e7f4b13a79270f > Author: Tao Ma <boyu.mt@taobao.com> > Date: Fri Mar 11 20:11:59 2011 +0100 > > blktrace: Use rq->cmd_flags directly in blk_add_trace_rq. > > > > This makes no sense. Yup. > hdparm uses 2MB block reads, so read_ahead (128KB) is too small for best > perf Right. In another thread, we are considering raising the default readahead size to 1MB. Note that as long as buffered read(2) is used, it makes almost no difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": the 128kb readahead size will be used underneath to submit read IO. > # cat /sys/class/block/sda/queue/read_ahead_kb > 128 > > # dd if=/dev/sda of=/dev/null bs=128k > ^C > 63744+0 enregistrements lus > 63743+0 enregistrements écrits > 8354922496 octets (8,4 GB) copiés, 39,975 s, 209 MB/s > > # hdparm -t /dev/sda > > /dev/sda: > Timing buffered disk reads: 510 MB in 3.00 seconds = 169.75 MB/sec Interesting, hdparm gets lower throughput than dd. > # uname -a > Linux edumazet-laptop 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20 > 17:23:00 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Thanks, Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 11:16 ` Wu Fengguang @ 2012-01-29 13:13 ` Eric Dumazet 2012-01-29 15:52 ` Pádraig Brady 2012-01-29 16:10 ` Wu Fengguang 0 siblings, 2 replies; 36+ messages in thread From: Eric Dumazet @ 2012-01-29 13:13 UTC (permalink / raw) To: Wu Fengguang; +Cc: Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : > Note that as long as buffered read(2) is used, it makes almost no > difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": > the 128kb readahead size will be used underneath to submit read IO. > Hmm... # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 32768+0 enregistrements lus 32768+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 2048+0 enregistrements lus 2048+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 13:13 ` Eric Dumazet @ 2012-01-29 15:52 ` Pádraig Brady 2012-01-29 16:10 ` Wu Fengguang 1 sibling, 0 replies; 36+ messages in thread From: Pádraig Brady @ 2012-01-29 15:52 UTC (permalink / raw) To: Eric Dumazet Cc: Wu Fengguang, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On 01/29/2012 01:13 PM, Eric Dumazet wrote: > Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : > > >> Note that as long as buffered read(2) is used, it makes almost no >> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": >> the 128kb readahead size will be used underneath to submit read IO. >> > > Hmm... > > # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > 32768+0 enregistrements lus > 32768+0 enregistrements écrits > 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s > > > # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > 2048+0 enregistrements lus > 2048+0 enregistrements écrits > 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s Same here on 2.6.40.4-5.fc15.x86_64 Note the SSD is rated for 500MB/s but is on a SATA II port, and so limited by that. So the 128k result below is close to the limit on this system. Hmm, I previously tested this SSD with kernel-2.6.38.6-26.rc1.fc15.src.rpm and got 270MB/s. Testing now gives variable and lower results: # echo 3 >/proc/sys/vm/drop_caches; hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 8388 MB in 2.00 seconds = 4200.73 MB/sec Timing buffered disk reads: 550 MB in 3.00 seconds = 183.19 MB/sec # echo 3 >/proc/sys/vm/drop_caches; hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 8260 MB in 2.00 seconds = 4134.30 MB/sec Timing buffered disk reads: 680 MB in 3.00 seconds = 226.63 MB/sec # echo 3 >/proc/sys/vm/drop_caches; hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 8426 MB in 2.00 seconds = 4217.87 MB/sec Timing buffered disk reads: 588 MB in 3.00 seconds = 195.96 MB/sec Anyway testing different block sizes with dd: # echo 3 >/proc/sys/vm/drop_caches; timeout -sINT 5 dd if=/dev/sdb of=/dev/null bs=2M 966787072 bytes (967 MB) copied, 5.00525 s, 193 MB/s # echo 3 >/proc/sys/vm/drop_caches; timeout -sINT 5 dd if=/dev/sdb of=/dev/null bs=128k 1246494720 bytes (1.2 GB) copied, 4.99563 s, 250 MB/s On a probably unrelated note, I've always noticed dd getting slower, independent of disks, when the buffer size increases beyond 2M. for i in $(seq 0 15); do size=$((16*1024**3)) #ensure this is big enough bs=$((1024*2**$i)) printf "%8s=" $bs dd bs=$bs if=/dev/zero of=/dev/null count=$(($size/$bs)) 2>&1 | sed -n 's/.* \([0-9.]* [GM]B\/s\)/\1/p' done 1024=1.4 GB/s 2048=2.6 GB/s 4096=4.5 GB/s 8192=6.7 GB/s 16384=8.8 GB/s 32768=9.4 GB/s 65536=10.8 GB/s 131072=11.5 GB/s 262144=11.5 GB/s 524288=11.3 GB/s 1048576=11.3 GB/s 2097152=10.6 GB/s 4194304=6.5 GB/s 8388608=5.9 GB/s 16777216=6.6 GB/s 33554432=6.6 GB/s cheers, Pádraig. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 13:13 ` Eric Dumazet 2012-01-29 15:52 ` Pádraig Brady @ 2012-01-29 16:10 ` Wu Fengguang 2012-01-29 20:15 ` Herbert Poetzl 2012-01-30 3:17 ` Shaohua Li 1 sibling, 2 replies; 36+ messages in thread From: Wu Fengguang @ 2012-01-29 16:10 UTC (permalink / raw) To: Eric Dumazet; +Cc: Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: > Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : > > > > Note that as long as buffered read(2) is used, it makes almost no > > difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": > > the 128kb readahead size will be used underneath to submit read IO. > > > > Hmm... > > # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > 32768+0 enregistrements lus > 32768+0 enregistrements écrits > 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s > > > # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > 2048+0 enregistrements lus > 2048+0 enregistrements écrits > 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s Interesting. Here are my test results: root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 32768+0 records in 32768+0 records out 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 2048+0 records in 2048+0 records out 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s Maybe the /dev/sda performance bug on your machine is sensitive to timing? Thanks, Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 16:10 ` Wu Fengguang @ 2012-01-29 20:15 ` Herbert Poetzl 2012-01-30 11:18 ` Wu Fengguang 2012-01-30 3:17 ` Shaohua Li 1 sibling, 1 reply; 36+ messages in thread From: Herbert Poetzl @ 2012-01-29 20:15 UTC (permalink / raw) To: Wu Fengguang; +Cc: Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo [-- Attachment #1: Type: text/plain, Size: 3331 bytes --] On Mon, Jan 30, 2012 at 12:10:58AM +0800, Wu Fengguang wrote: > On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: >> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : >>> Note that as long as buffered read(2) is used, it makes almost no >>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": >>> the 128kb readahead size will be used underneath to submit read IO. >> Hmm... >> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >> 32768+0 enregistrements lus >> 32768+0 enregistrements écrits >> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s >> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >> 2048+0 enregistrements lus >> 2048+0 enregistrements écrits >> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s > Interesting. Here are my test results: > root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > 32768+0 records in > 32768+0 records out > 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s > root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > 2048+0 records in > 2048+0 records out > 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s > Maybe the /dev/sda performance bug on your machine is sensitive to timing? here are some more confusing results from tests with dd and bonnie++, this time I focused on partition vs. loop vs. linear dm (of same partition) kernel -------------- read -------------- -- write --- all -------- dd -------- -------- bonnie++ -------------- [MB/s] real %CPU [MB/s] %CPU [MB/s] %CPU %CPU direct 2.6.38.8 262.91 81.90 28.7 72.30 6.0 248.53 52.0 15.9 2.6.39.4 36.09 595.17 3.1 70.62 6.0 250.25 53.0 16.3 3.0.18 50.47 425.65 4.1 70.00 5.0 251.70 44.0 13.9 3.1.10 27.28 787.32 2.0 75.65 5.0 251.96 45.0 13.3 3.2.2 27.11 792.28 2.0 76.89 6.0 250.38 44.0 13.3 loop 2.6.38.8 242.89 88.50 21.5 246.58 15.0 240.92 53.0 14.4 2.6.39.4 241.06 89.19 21.5 238.51 15.0 257.59 57.0 14.8 3.0.18 261.44 82.23 18.8 256.66 15.0 255.17 48.0 12.6 3.1.10 253.93 84.64 18.1 107.66 7.0 156.51 28.0 10.6 3.2.2 262.58 81.82 19.8 110.54 7.0 212.01 40.0 11.6 linear 2.6.38.8 262.57 82.00 36.8 72.46 6.0 243.25 53.0 16.5 2.6.39.4 25.45 843.93 2.3 70.70 6.0 248.05 54.0 16.6 3.0.18 55.45 387.43 5.6 69.72 6.0 249.42 45.0 14.3 3.1.10 36.62 586.50 3.3 74.74 6.0 249.99 46.0 13.4 3.2.2 28.28 759.26 2.3 74.20 6.0 248.73 46.0 13.6 it seems that dd performance when using a loop device is unaffected and even improves with the kernel version, the filesystem performance OTOH degrades after 3.1 ... in general, filesystem read performance is bad on everything but a loop device ... judging from the results I'd conclude that there are at least two different issues tests and test results are attached and can be found here: http://vserver.13thfloor.at/Stuff/SSD/ I plan to do some more tests on the filesystem with -b and -D tonight, please let me know if you want to see specific output and/or have any tests I should run with each kernel ... HTH, Herbert > Thanks, > Fengguang [-- Attachment #2: SSD.txz --] [-- Type: application/octet-stream, Size: 56488 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 20:15 ` Herbert Poetzl @ 2012-01-30 11:18 ` Wu Fengguang 2012-01-30 12:34 ` Eric Dumazet 0 siblings, 1 reply; 36+ messages in thread From: Wu Fengguang @ 2012-01-30 11:18 UTC (permalink / raw) To: Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo Cc: Li Shaohua, Herbert Poetzl On Sun, Jan 29, 2012 at 09:15:43PM +0100, Herbert Poetzl wrote: > On Mon, Jan 30, 2012 at 12:10:58AM +0800, Wu Fengguang wrote: > > Maybe the /dev/sda performance bug on your machine is sensitive to timing? > > here are some more confusing results from tests with dd and bonnie++, > this time I focused on partition vs. loop vs. linear dm (of same partition) > > kernel -------------- read -------------- -- write --- all > -------- dd -------- -------- bonnie++ -------------- > [MB/s] real %CPU [MB/s] %CPU [MB/s] %CPU %CPU > direct > 2.6.38.8 262.91 81.90 28.7 72.30 6.0 248.53 52.0 15.9 > 2.6.39.4 36.09 595.17 3.1 70.62 6.0 250.25 53.0 16.3 > 3.0.18 50.47 425.65 4.1 70.00 5.0 251.70 44.0 13.9 > 3.1.10 27.28 787.32 2.0 75.65 5.0 251.96 45.0 13.3 > 3.2.2 27.11 792.28 2.0 76.89 6.0 250.38 44.0 13.3 > > loop > 2.6.38.8 242.89 88.50 21.5 246.58 15.0 240.92 53.0 14.4 > 2.6.39.4 241.06 89.19 21.5 238.51 15.0 257.59 57.0 14.8 > 3.0.18 261.44 82.23 18.8 256.66 15.0 255.17 48.0 12.6 > 3.1.10 253.93 84.64 18.1 107.66 7.0 156.51 28.0 10.6 > 3.2.2 262.58 81.82 19.8 110.54 7.0 212.01 40.0 11.6 > > linear > 2.6.38.8 262.57 82.00 36.8 72.46 6.0 243.25 53.0 16.5 > 2.6.39.4 25.45 843.93 2.3 70.70 6.0 248.05 54.0 16.6 > 3.0.18 55.45 387.43 5.6 69.72 6.0 249.42 45.0 14.3 > 3.1.10 36.62 586.50 3.3 74.74 6.0 249.99 46.0 13.4 > 3.2.2 28.28 759.26 2.3 74.20 6.0 248.73 46.0 13.6 > > > it seems that dd performance when using a loop device is unaffected > and even improves with the kernel version, the filesystem performance > OTOH degrades after 3.1 ... > > in general, filesystem read performance is bad on everything but > a loop device ... judging from the results I'd conclude that there > are at least two different issues > > tests and test results are attached and can be found here: > http://vserver.13thfloor.at/Stuff/SSD/ > > I plan to do some more tests on the filesystem with -b and -D > tonight, please let me know if you want to see specific output > and/or have any tests I should run with each kernel ... I agree with Shaohua that there may be timing/plug issues. There happen to be some plug patches and (maybe correlated) big performance drop between 2.6.38 and 2.6.39. The obvious way to move forward is to get some blktrace data on simple dd + new buggy kernel and let's check what's exactly going on. # start a background dd read blktrace /dev/sda -w 10 blkparse -t sda Thanks, Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 11:18 ` Wu Fengguang @ 2012-01-30 12:34 ` Eric Dumazet 2012-01-30 14:01 ` Wu Fengguang 0 siblings, 1 reply; 36+ messages in thread From: Eric Dumazet @ 2012-01-30 12:34 UTC (permalink / raw) To: Wu Fengguang Cc: Andrew Morton, LKML, Jens Axboe, Tejun Heo, Li Shaohua, Herbert Poetzl Le lundi 30 janvier 2012 à 19:18 +0800, Wu Fengguang a écrit : > I agree with Shaohua that there may be timing/plug issues. There > happen to be some plug patches and (maybe correlated) big performance > drop between 2.6.38 and 2.6.39. The obvious way to move forward is to > get some blktrace data on simple dd + new buggy kernel and let's check > what's exactly going on. > > # start a background dd read > blktrace /dev/sda -w 10 > blkparse -t sda > (Another machine here at office, hence /dev/sdb) echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=2M count=2048 2048+0 enregistrements lus 2048+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 26,0278 s, 165 MB/s # blktrace /dev/sdb -w 10 === sdb === CPU 0: 44766 events, 2099 KiB data CPU 1: 828895 events, 38855 KiB data Total: 873661 events (dropped 0), 40953 KiB data blkparse result is huge (55Mo) # blkparse -t sdb | tail -n 1000 8,16 1 770475 9.989714963 4580 Q R 4379392 + 8 [dd] 8,16 1 770476 9.989721470 4580 G R 4379392 + 8 [dd] 8,16 1 770477 9.989723546 4580 P N [dd] 8,16 1 770478 9.989726994 4580 Q R 4379400 + 8 [dd] 8,16 1 770479 9.989728858 4580 M R 4379400 + 8 [dd] 8,16 1 770480 9.989731893 4580 Q R 4379408 + 8 [dd] 8,16 1 770481 9.989732467 4580 M R 4379408 + 8 [dd] 8,16 1 770482 9.989734574 4580 Q R 4379416 + 8 [dd] 8,16 1 770483 9.989735108 4580 M R 4379416 + 8 [dd] 8,16 1 770484 9.989737220 4580 Q R 4379424 + 8 [dd] 8,16 1 770485 9.989737721 4580 M R 4379424 + 8 [dd] 8,16 1 770486 9.989739953 4580 Q R 4379432 + 8 [dd] 8,16 1 770487 9.989740450 4580 M R 4379432 + 8 [dd] 8,16 1 770488 9.989742577 4580 Q R 4379440 + 8 [dd] 8,16 1 770489 9.989743078 4580 M R 4379440 + 8 [dd] 8,16 1 770490 9.989745241 4580 Q R 4379448 + 8 [dd] 8,16 1 770491 9.989745742 4580 M R 4379448 + 8 [dd] 8,16 1 770492 9.989747953 4580 Q R 4379456 + 8 [dd] 8,16 1 770493 9.989748485 4580 M R 4379456 + 8 [dd] 8,16 1 770494 9.989750726 4580 Q R 4379464 + 8 [dd] 8,16 1 770495 9.989751225 4580 M R 4379464 + 8 [dd] 8,16 1 770496 9.989753390 4580 Q R 4379472 + 8 [dd] 8,16 1 770497 9.989753891 4580 M R 4379472 + 8 [dd] 8,16 1 770498 9.989755928 4580 Q R 4379480 + 8 [dd] 8,16 1 770499 9.989756429 4580 M R 4379480 + 8 [dd] 8,16 1 770500 9.989758470 4580 Q R 4379488 + 8 [dd] 8,16 1 770501 9.989758975 4580 M R 4379488 + 8 [dd] 8,16 1 770502 9.989765725 4580 Q R 4379496 + 8 [dd] 8,16 1 770503 9.989766222 4580 M R 4379496 + 8 [dd] 8,16 1 770504 9.989768362 4580 Q R 4379504 + 8 [dd] 8,16 1 770505 9.989768864 4580 M R 4379504 + 8 [dd] 8,16 1 770506 9.989771157 4580 Q R 4379512 + 8 [dd] 8,16 1 770507 9.989771662 4580 M R 4379512 + 8 [dd] 8,16 1 770508 9.989775151 4580 Q R 4379520 + 8 [dd] 8,16 1 770509 9.989775649 4580 M R 4379520 + 8 [dd] 8,16 1 770510 9.989781743 4580 Q R 4379528 + 8 [dd] 8,16 1 770511 9.989782248 4580 M R 4379528 + 8 [dd] 8,16 1 770512 9.989784532 4580 Q R 4379536 + 8 [dd] 8,16 1 770513 9.989785033 4580 M R 4379536 + 8 [dd] 8,16 1 770514 9.989787291 4580 Q R 4379544 + 8 [dd] 8,16 1 770515 9.989787792 4580 M R 4379544 + 8 [dd] 8,16 1 770516 9.989790041 4580 Q R 4379552 + 8 [dd] 8,16 1 770517 9.989790540 4580 M R 4379552 + 8 [dd] 8,16 1 770518 9.989792607 4580 Q R 4379560 + 8 [dd] 8,16 1 770519 9.989793104 4580 M R 4379560 + 8 [dd] 8,16 1 770520 9.989795693 4580 Q R 4379568 + 8 [dd] 8,16 1 770521 9.989796191 4580 M R 4379568 + 8 [dd] 8,16 1 770522 9.989798349 4580 Q R 4379576 + 8 [dd] 8,16 1 770523 9.989798847 4580 M R 4379576 + 8 [dd] 8,16 1 770524 9.989800877 4580 Q R 4379584 + 8 [dd] 8,16 1 770525 9.989801581 4580 M R 4379584 + 8 [dd] 8,16 1 770526 9.989803693 4580 Q R 4379592 + 8 [dd] 8,16 1 770527 9.989804221 4580 M R 4379592 + 8 [dd] 8,16 1 770528 9.989806399 4580 Q R 4379600 + 8 [dd] 8,16 1 770529 9.989806901 4580 M R 4379600 + 8 [dd] 8,16 1 770530 9.989812437 4580 Q R 4379608 + 8 [dd] 8,16 1 770531 9.989812938 4580 M R 4379608 + 8 [dd] 8,16 1 770532 9.989814981 4580 Q R 4379616 + 8 [dd] 8,16 1 770533 9.989815482 4580 M R 4379616 + 8 [dd] 8,16 1 770534 9.989818053 4580 Q R 4379624 + 8 [dd] 8,16 1 770535 9.989818551 4580 M R 4379624 + 8 [dd] 8,16 1 770536 9.989820795 4580 Q R 4379632 + 8 [dd] 8,16 1 770537 9.989821296 4580 M R 4379632 + 8 [dd] 8,16 1 770538 9.989823436 4580 Q R 4379640 + 8 [dd] 8,16 1 770539 9.989823937 4580 M R 4379640 + 8 [dd] 8,16 1 770540 9.989929502 4580 Q R 4379648 + 8 [dd] 8,16 1 770541 9.989930247 4580 M R 4379648 + 8 [dd] 8,16 1 770542 9.989933189 4580 Q R 4379656 + 8 [dd] 8,16 1 770543 9.989933705 4580 M R 4379656 + 8 [dd] 8,16 1 770544 9.989935909 4580 Q R 4379664 + 8 [dd] 8,16 1 770545 9.989936411 4580 M R 4379664 + 8 [dd] 8,16 1 770546 9.989938572 4580 Q R 4379672 + 8 [dd] 8,16 1 770547 9.989939072 4580 M R 4379672 + 8 [dd] 8,16 1 770548 9.989941272 4580 Q R 4379680 + 8 [dd] 8,16 1 770549 9.989941770 4580 M R 4379680 + 8 [dd] 8,16 1 770550 9.989943952 4580 Q R 4379688 + 8 [dd] 8,16 1 770551 9.989944450 4580 M R 4379688 + 8 [dd] 8,16 1 770552 9.989946991 4580 Q R 4379696 + 8 [dd] 8,16 1 770553 9.989947492 4580 M R 4379696 + 8 [dd] 8,16 1 770554 9.989949671 4580 Q R 4379704 + 8 [dd] 8,16 1 770555 9.989950172 4580 M R 4379704 + 8 [dd] 8,16 1 770556 9.989952330 4580 Q R 4379712 + 8 [dd] 8,16 1 770557 9.989952832 4580 M R 4379712 + 8 [dd] 8,16 1 770558 9.989958393 4580 Q R 4379720 + 8 [dd] 8,16 1 770559 9.989958891 4580 M R 4379720 + 8 [dd] 8,16 1 770560 9.989961097 4580 Q R 4379728 + 8 [dd] 8,16 1 770561 9.989961598 4580 M R 4379728 + 8 [dd] 8,16 1 770562 9.989963825 4580 Q R 4379736 + 8 [dd] 8,16 1 770563 9.989964331 4580 M R 4379736 + 8 [dd] 8,16 1 770564 9.989966623 4580 Q R 4379744 + 8 [dd] 8,16 1 770565 9.989967122 4580 M R 4379744 + 8 [dd] 8,16 1 770566 9.989969303 4580 Q R 4379752 + 8 [dd] 8,16 1 770567 9.989969800 4580 M R 4379752 + 8 [dd] 8,16 1 770568 9.989971886 4580 Q R 4379760 + 8 [dd] 8,16 1 770569 9.989972388 4580 M R 4379760 + 8 [dd] 8,16 1 770570 9.989974789 4580 Q R 4379768 + 8 [dd] 8,16 1 770571 9.989975290 4580 M R 4379768 + 8 [dd] 8,16 1 770572 9.989977480 4580 Q R 4379776 + 8 [dd] 8,16 1 770573 9.989977981 4580 M R 4379776 + 8 [dd] 8,16 1 770574 9.989981251 4580 Q R 4379784 + 8 [dd] 8,16 1 770575 9.989981749 4580 M R 4379784 + 8 [dd] 8,16 1 770576 9.989984018 4580 Q R 4379792 + 8 [dd] 8,16 1 770577 9.989984519 4580 M R 4379792 + 8 [dd] 8,16 1 770578 9.989986711 4580 Q R 4379800 + 8 [dd] 8,16 1 770579 9.989987222 4580 M R 4379800 + 8 [dd] 8,16 1 770580 9.989989414 4580 Q R 4379808 + 8 [dd] 8,16 1 770581 9.989989915 4580 M R 4379808 + 8 [dd] 8,16 1 770582 9.989992107 4580 Q R 4379816 + 8 [dd] 8,16 1 770583 9.989992617 4580 M R 4379816 + 8 [dd] 8,16 1 770584 9.989994776 4580 Q R 4379824 + 8 [dd] 8,16 1 770585 9.989995276 4580 M R 4379824 + 8 [dd] 8,16 1 770586 9.990000570 4580 Q R 4379832 + 8 [dd] 8,16 1 770587 9.990001076 4580 M R 4379832 + 8 [dd] 8,16 1 770588 9.990003264 4580 Q R 4379840 + 8 [dd] 8,16 1 770589 9.990003785 4580 M R 4379840 + 8 [dd] 8,16 1 770590 9.990005849 4580 Q R 4379848 + 8 [dd] 8,16 1 770591 9.990006354 4580 M R 4379848 + 8 [dd] 8,16 1 770592 9.990008472 4580 Q R 4379856 + 8 [dd] 8,16 1 770593 9.990008986 4580 M R 4379856 + 8 [dd] 8,16 1 770594 9.990011382 4580 Q R 4379864 + 8 [dd] 8,16 1 770595 9.990011884 4580 M R 4379864 + 8 [dd] 8,16 1 770596 9.990014066 4580 Q R 4379872 + 8 [dd] 8,16 1 770597 9.990014568 4580 M R 4379872 + 8 [dd] 8,16 1 770598 9.990016740 4580 Q R 4379880 + 8 [dd] 8,16 1 770599 9.990017237 4580 M R 4379880 + 8 [dd] 8,16 1 770600 9.990019491 4580 Q R 4379888 + 8 [dd] 8,16 1 770601 9.990020001 4580 M R 4379888 + 8 [dd] 8,16 1 770602 9.990022222 4580 Q R 4379896 + 8 [dd] 8,16 1 770603 9.990022723 4580 M R 4379896 + 8 [dd] 8,16 1 770604 9.990024981 4580 I R 4379392 + 512 ( 303511) [dd] 8,16 1 0 9.990027734 0 m N cfq4580S / insert_request 8,16 1 770605 9.990031216 4580 U N [dd] 1 8,16 1 0 9.990033792 0 m N cfq4580S / dispatch_insert 8,16 1 0 9.990037429 0 m N cfq4580S / dispatched a request 8,16 1 0 9.990039146 0 m N cfq4580S / activate rq, drv=1 8,16 1 770606 9.990039807 4580 D R 4379392 + 512 ( 14826) [dd] 8,16 1 770607 9.991083069 0 C R 4379392 + 512 ( 1043262) [0] 8,16 1 0 9.991155388 0 m N cfq4580S / complete rqnoidle 0 8,16 1 0 9.991158292 0 m N cfq schedule dispatch 8,16 1 770608 9.991321889 4580 Q R 4379904 + 8 [dd] 8,16 1 770609 9.991328302 4580 G R 4379904 + 8 [dd] 8,16 1 770610 9.991330287 4580 P N [dd] 8,16 1 770611 9.991333624 4580 Q R 4379912 + 8 [dd] 8,16 1 770612 9.991335576 4580 M R 4379912 + 8 [dd] 8,16 1 770613 9.991338586 4580 Q R 4379920 + 8 [dd] 8,16 1 770614 9.991339095 4580 M R 4379920 + 8 [dd] 8,16 1 770615 9.991341209 4580 Q R 4379928 + 8 [dd] 8,16 1 770616 9.991341909 4580 M R 4379928 + 8 [dd] 8,16 1 770617 9.991344208 4580 Q R 4379936 + 8 [dd] 8,16 1 770618 9.991344707 4580 M R 4379936 + 8 [dd] 8,16 1 770619 9.991351891 4580 Q R 4379944 + 8 [dd] 8,16 1 770620 9.991352393 4580 M R 4379944 + 8 [dd] 8,16 1 770621 9.991354635 4580 Q R 4379952 + 8 [dd] 8,16 1 770622 9.991355134 4580 M R 4379952 + 8 [dd] 8,16 1 770623 9.991357304 4580 Q R 4379960 + 8 [dd] 8,16 1 770624 9.991357809 4580 M R 4379960 + 8 [dd] 8,16 1 770625 9.991359960 4580 Q R 4379968 + 8 [dd] 8,16 1 770626 9.991360458 4580 M R 4379968 + 8 [dd] 8,16 1 770627 9.991362540 4580 Q R 4379976 + 8 [dd] 8,16 1 770628 9.991363044 4580 M R 4379976 + 8 [dd] 8,16 1 770629 9.991366526 4580 Q R 4379984 + 8 [dd] 8,16 1 770630 9.991367025 4580 M R 4379984 + 8 [dd] 8,16 1 770631 9.991369183 4580 Q R 4379992 + 8 [dd] 8,16 1 770632 9.991369681 4580 M R 4379992 + 8 [dd] 8,16 1 770633 9.991371837 4580 Q R 4380000 + 8 [dd] 8,16 1 770634 9.991372336 4580 M R 4380000 + 8 [dd] 8,16 1 770635 9.991374518 4580 Q R 4380008 + 8 [dd] 8,16 1 770636 9.991375031 4580 M R 4380008 + 8 [dd] 8,16 1 770637 9.991377249 4580 Q R 4380016 + 8 [dd] 8,16 1 770638 9.991377748 4580 M R 4380016 + 8 [dd] 8,16 1 770639 9.991379891 4580 Q R 4380024 + 8 [dd] 8,16 1 770640 9.991380390 4580 M R 4380024 + 8 [dd] 8,16 1 770641 9.991382585 4580 Q R 4380032 + 8 [dd] 8,16 1 770642 9.991383084 4580 M R 4380032 + 8 [dd] 8,16 1 770643 9.991385207 4580 Q R 4380040 + 8 [dd] 8,16 1 770644 9.991385704 4580 M R 4380040 + 8 [dd] 8,16 1 770645 9.991387897 4580 Q R 4380048 + 8 [dd] 8,16 1 770646 9.991388396 4580 M R 4380048 + 8 [dd] 8,16 1 770647 9.991393804 4580 Q R 4380056 + 8 [dd] 8,16 1 770648 9.991394303 4580 M R 4380056 + 8 [dd] 8,16 1 770649 9.991396440 4580 Q R 4380064 + 8 [dd] 8,16 1 770650 9.991396938 4580 M R 4380064 + 8 [dd] 8,16 1 770651 9.991398996 4580 Q R 4380072 + 8 [dd] 8,16 1 770652 9.991399500 4580 M R 4380072 + 8 [dd] 8,16 1 770653 9.991401608 4580 Q R 4380080 + 8 [dd] 8,16 1 770654 9.991402107 4580 M R 4380080 + 8 [dd] 8,16 1 770655 9.991404535 4580 Q R 4380088 + 8 [dd] 8,16 1 770656 9.991405034 4580 M R 4380088 + 8 [dd] 8,16 1 770657 9.991407171 4580 Q R 4380096 + 8 [dd] 8,16 1 770658 9.991407671 4580 M R 4380096 + 8 [dd] 8,16 1 770659 9.991409829 4580 Q R 4380104 + 8 [dd] 8,16 1 770660 9.991410339 4580 M R 4380104 + 8 [dd] 8,16 1 770661 9.991416346 4580 Q R 4380112 + 8 [dd] 8,16 1 770662 9.991416853 4580 M R 4380112 + 8 [dd] 8,16 1 770663 9.991419016 4580 Q R 4380120 + 8 [dd] 8,16 1 770664 9.991419515 4580 M R 4380120 + 8 [dd] 8,16 1 770665 9.991421984 4580 Q R 4380128 + 8 [dd] 8,16 1 770666 9.991422483 4580 M R 4380128 + 8 [dd] 8,16 1 770667 9.991424780 4580 Q R 4380136 + 8 [dd] 8,16 1 770668 9.991425284 4580 M R 4380136 + 8 [dd] 8,16 1 770669 9.991427479 4580 Q R 4380144 + 8 [dd] 8,16 1 770670 9.991427978 4580 M R 4380144 + 8 [dd] 8,16 1 770671 9.991430076 4580 Q R 4380152 + 8 [dd] 8,16 1 770672 9.991430575 4580 M R 4380152 + 8 [dd] 8,16 1 770673 9.991536122 4580 Q R 4380160 + 8 [dd] 8,16 1 770674 9.991536827 4580 M R 4380160 + 8 [dd] 8,16 1 770675 9.991543663 4580 Q R 4380168 + 8 [dd] 8,16 1 770676 9.991544181 4580 M R 4380168 + 8 [dd] 8,16 1 770677 9.991546458 4580 Q R 4380176 + 8 [dd] 8,16 1 770678 9.991546957 4580 M R 4380176 + 8 [dd] 8,16 1 770679 9.991549153 4580 Q R 4380184 + 8 [dd] 8,16 1 770680 9.991549652 4580 M R 4380184 + 8 [dd] 8,16 1 770681 9.991551902 4580 Q R 4380192 + 8 [dd] 8,16 1 770682 9.991552400 4580 M R 4380192 + 8 [dd] 8,16 1 770683 9.991554621 4580 Q R 4380200 + 8 [dd] 8,16 1 770684 9.991555113 4580 M R 4380200 + 8 [dd] 8,16 1 770685 9.991557325 4580 Q R 4380208 + 8 [dd] 8,16 1 770686 9.991557824 4580 M R 4380208 + 8 [dd] 8,16 1 770687 9.991560192 4580 Q R 4380216 + 8 [dd] 8,16 1 770688 9.991560732 4580 M R 4380216 + 8 [dd] 8,16 1 770689 9.991562806 4580 Q R 4380224 + 8 [dd] 8,16 1 770690 9.991563305 4580 M R 4380224 + 8 [dd] 8,16 1 770691 9.991565370 4580 Q R 4380232 + 8 [dd] 8,16 1 770692 9.991565874 4580 M R 4380232 + 8 [dd] 8,16 1 770693 9.991567969 4580 Q R 4380240 + 8 [dd] 8,16 1 770694 9.991568468 4580 M R 4380240 + 8 [dd] 8,16 1 770695 9.991570671 4580 Q R 4380248 + 8 [dd] 8,16 1 770696 9.991571170 4580 M R 4380248 + 8 [dd] 8,16 1 770697 9.991573285 4580 Q R 4380256 + 8 [dd] 8,16 1 770698 9.991573783 4580 M R 4380256 + 8 [dd] 8,16 1 770699 9.991575855 4580 Q R 4380264 + 8 [dd] 8,16 1 770700 9.991576358 4580 M R 4380264 + 8 [dd] 8,16 1 770701 9.991578551 4580 Q R 4380272 + 8 [dd] 8,16 1 770702 9.991579050 4580 M R 4380272 + 8 [dd] 8,16 1 770703 9.991584418 4580 Q R 4380280 + 8 [dd] 8,16 1 770704 9.991584916 4580 M R 4380280 + 8 [dd] 8,16 1 770705 9.991587004 4580 Q R 4380288 + 8 [dd] 8,16 1 770706 9.991587503 4580 M R 4380288 + 8 [dd] 8,16 1 770707 9.991589615 4580 Q R 4380296 + 8 [dd] 8,16 1 770708 9.991590128 4580 M R 4380296 + 8 [dd] 8,16 1 770709 9.991592390 4580 Q R 4380304 + 8 [dd] 8,16 1 770710 9.991592907 4580 M R 4380304 + 8 [dd] 8,16 1 770711 9.991594957 4580 Q R 4380312 + 8 [dd] 8,16 1 770712 9.991595456 4580 M R 4380312 + 8 [dd] 8,16 1 770713 9.991597599 4580 Q R 4380320 + 8 [dd] 8,16 1 770714 9.991598098 4580 M R 4380320 + 8 [dd] 8,16 1 770715 9.991600165 4580 Q R 4380328 + 8 [dd] 8,16 1 770716 9.991600669 4580 M R 4380328 + 8 [dd] 8,16 1 770717 9.991602915 4580 Q R 4380336 + 8 [dd] 8,16 1 770718 9.991603413 4580 M R 4380336 + 8 [dd] 8,16 1 770719 9.991605563 4580 Q R 4380344 + 8 [dd] 8,16 1 770720 9.991606061 4580 M R 4380344 + 8 [dd] 8,16 1 770721 9.991608110 4580 Q R 4380352 + 8 [dd] 8,16 1 770722 9.991608609 4580 M R 4380352 + 8 [dd] 8,16 1 770723 9.991610674 4580 Q R 4380360 + 8 [dd] 8,16 1 770724 9.991611178 4580 M R 4380360 + 8 [dd] 8,16 1 770725 9.991613249 4580 Q R 4380368 + 8 [dd] 8,16 1 770726 9.991613748 4580 M R 4380368 + 8 [dd] 8,16 1 770727 9.991616309 4580 Q R 4380376 + 8 [dd] 8,16 1 770728 9.991616808 4580 M R 4380376 + 8 [dd] 8,16 1 770729 9.991619009 4580 Q R 4380384 + 8 [dd] 8,16 1 770730 9.991619507 4580 M R 4380384 + 8 [dd] 8,16 1 770731 9.991624896 4580 Q R 4380392 + 8 [dd] 8,16 1 770732 9.991625401 4580 M R 4380392 + 8 [dd] 8,16 1 770733 9.991627620 4580 Q R 4380400 + 8 [dd] 8,16 1 770734 9.991628119 4580 M R 4380400 + 8 [dd] 8,16 1 770735 9.991630192 4580 Q R 4380408 + 8 [dd] 8,16 1 770736 9.991630690 4580 M R 4380408 + 8 [dd] 8,16 1 770737 9.991633001 4580 I R 4379904 + 512 ( 304699) [dd] 8,16 1 0 9.991635738 0 m N cfq4580S / insert_request 8,16 1 770738 9.991638988 4580 U N [dd] 1 8,16 1 0 9.991641665 0 m N cfq4580S / dispatch_insert 8,16 1 0 9.991645149 0 m N cfq4580S / dispatched a request 8,16 1 0 9.991646825 0 m N cfq4580S / activate rq, drv=1 8,16 1 770739 9.991647434 4580 D R 4379904 + 512 ( 14433) [dd] 8,16 1 770740 9.992693317 0 C R 4379904 + 512 ( 1045883) [0] 8,16 1 0 9.992766012 0 m N cfq4580S / complete rqnoidle 0 8,16 1 0 9.992768895 0 m N cfq schedule dispatch 8,16 1 770741 9.992932816 4580 Q R 4380416 + 8 [dd] 8,16 1 770742 9.992939152 4580 G R 4380416 + 8 [dd] 8,16 1 770743 9.992941241 4580 P N [dd] 8,16 1 770744 9.992944673 4580 Q R 4380424 + 8 [dd] 8,16 1 770745 9.992946529 4580 M R 4380424 + 8 [dd] 8,16 1 770746 9.992949525 4580 Q R 4380432 + 8 [dd] 8,16 1 770747 9.992950035 4580 M R 4380432 + 8 [dd] 8,16 1 770748 9.992952218 4580 Q R 4380440 + 8 [dd] 8,16 1 770749 9.992952765 4580 M R 4380440 + 8 [dd] 8,16 1 770750 9.992954867 4580 Q R 4380448 + 8 [dd] 8,16 1 770751 9.992955366 4580 M R 4380448 + 8 [dd] 8,16 1 770752 9.992957451 4580 Q R 4380456 + 8 [dd] 8,16 1 770753 9.992957956 4580 M R 4380456 + 8 [dd] 8,16 1 770754 9.992960073 4580 Q R 4380464 + 8 [dd] 8,16 1 770755 9.992960572 4580 M R 4380464 + 8 [dd] 8,16 1 770756 9.992962600 4580 Q R 4380472 + 8 [dd] 8,16 1 770757 9.992963104 4580 M R 4380472 + 8 [dd] 8,16 1 770758 9.992965247 4580 Q R 4380480 + 8 [dd] 8,16 1 770759 9.992965753 4580 M R 4380480 + 8 [dd] 8,16 1 770760 9.992967787 4580 Q R 4380488 + 8 [dd] 8,16 1 770761 9.992968284 4580 M R 4380488 + 8 [dd] 8,16 1 770762 9.992970602 4580 Q R 4380496 + 8 [dd] 8,16 1 770763 9.992971101 4580 M R 4380496 + 8 [dd] 8,16 1 770764 9.992978036 4580 Q R 4380504 + 8 [dd] 8,16 1 770765 9.992978575 4580 M R 4380504 + 8 [dd] 8,16 1 770766 9.992980800 4580 Q R 4380512 + 8 [dd] 8,16 1 770767 9.992981312 4580 M R 4380512 + 8 [dd] 8,16 1 770768 9.992983463 4580 Q R 4380520 + 8 [dd] 8,16 1 770769 9.992983976 4580 M R 4380520 + 8 [dd] 8,16 1 770770 9.992986141 4580 Q R 4380528 + 8 [dd] 8,16 1 770771 9.992986652 4580 M R 4380528 + 8 [dd] 8,16 1 770772 9.992988835 4580 Q R 4380536 + 8 [dd] 8,16 1 770773 9.992989338 4580 M R 4380536 + 8 [dd] 8,16 1 770774 9.992991462 4580 Q R 4380544 + 8 [dd] 8,16 1 770775 9.992991961 4580 M R 4380544 + 8 [dd] 8,16 1 770776 9.992994020 4580 Q R 4380552 + 8 [dd] 8,16 1 770777 9.992994519 4580 M R 4380552 + 8 [dd] 8,16 1 770778 9.992996632 4580 Q R 4380560 + 8 [dd] 8,16 1 770779 9.992997135 4580 M R 4380560 + 8 [dd] 8,16 1 770780 9.992999236 4580 Q R 4380568 + 8 [dd] 8,16 1 770781 9.992999742 4580 M R 4380568 + 8 [dd] 8,16 1 770782 9.993001890 4580 Q R 4380576 + 8 [dd] 8,16 1 770783 9.993002389 4580 M R 4380576 + 8 [dd] 8,16 1 770784 9.993004438 4580 Q R 4380584 + 8 [dd] 8,16 1 770785 9.993004937 4580 M R 4380584 + 8 [dd] 8,16 1 770786 9.993007255 4580 Q R 4380592 + 8 [dd] 8,16 1 770787 9.993007754 4580 M R 4380592 + 8 [dd] 8,16 1 770788 9.993009828 4580 Q R 4380600 + 8 [dd] 8,16 1 770789 9.993010351 4580 M R 4380600 + 8 [dd] 8,16 1 770790 9.993012459 4580 Q R 4380608 + 8 [dd] 8,16 1 770791 9.993012958 4580 M R 4380608 + 8 [dd] 8,16 1 770792 9.993019700 4580 Q R 4380616 + 8 [dd] 8,16 1 770793 9.993020199 4580 M R 4380616 + 8 [dd] 8,16 1 770794 9.993022401 4580 Q R 4380624 + 8 [dd] 8,16 1 770795 9.993022909 4580 M R 4380624 + 8 [dd] 8,16 1 770796 9.993024945 4580 Q R 4380632 + 8 [dd] 8,16 1 770797 9.993025449 4580 M R 4380632 + 8 [dd] 8,16 1 770798 9.993027501 4580 Q R 4380640 + 8 [dd] 8,16 1 770799 9.993027999 4580 M R 4380640 + 8 [dd] 8,16 1 770800 9.993030163 4580 Q R 4380648 + 8 [dd] 8,16 1 770801 9.993030666 4580 M R 4380648 + 8 [dd] 8,16 1 770802 9.993033040 4580 Q R 4380656 + 8 [dd] 8,16 1 770803 9.993033538 4580 M R 4380656 + 8 [dd] 8,16 1 770804 9.993035704 4580 Q R 4380664 + 8 [dd] 8,16 1 770805 9.993036205 4580 M R 4380664 + 8 [dd] 8,16 1 770806 9.993143012 4580 Q R 4380672 + 8 [dd] 8,16 1 770807 9.993143745 4580 M R 4380672 + 8 [dd] 8,16 1 770808 9.993147049 4580 Q R 4380680 + 8 [dd] 8,16 1 770809 9.993147550 4580 M R 4380680 + 8 [dd] 8,16 1 770810 9.993149983 4580 Q R 4380688 + 8 [dd] 8,16 1 770811 9.993150482 4580 M R 4380688 + 8 [dd] 8,16 1 770812 9.993156384 4580 Q R 4380696 + 8 [dd] 8,16 1 770813 9.993156894 4580 M R 4380696 + 8 [dd] 8,16 1 770814 9.993159096 4580 Q R 4380704 + 8 [dd] 8,16 1 770815 9.993159606 4580 M R 4380704 + 8 [dd] 8,16 1 770816 9.993161853 4580 Q R 4380712 + 8 [dd] 8,16 1 770817 9.993162351 4580 M R 4380712 + 8 [dd] 8,16 1 770818 9.993164628 4580 Q R 4380720 + 8 [dd] 8,16 1 770819 9.993165127 4580 M R 4380720 + 8 [dd] 8,16 1 770820 9.993170698 4580 Q R 4380728 + 8 [dd] 8,16 1 770821 9.993171200 4580 M R 4380728 + 8 [dd] 8,16 1 770822 9.993173392 4580 Q R 4380736 + 8 [dd] 8,16 1 770823 9.993173894 4580 M R 4380736 + 8 [dd] 8,16 1 770824 9.993176081 4580 Q R 4380744 + 8 [dd] 8,16 1 770825 9.993176579 4580 M R 4380744 + 8 [dd] 8,16 1 770826 9.993178841 4580 Q R 4380752 + 8 [dd] 8,16 1 770827 9.993179340 4580 M R 4380752 + 8 [dd] 8,16 1 770828 9.993181445 4580 Q R 4380760 + 8 [dd] 8,16 1 770829 9.993181949 4580 M R 4380760 + 8 [dd] 8,16 1 770830 9.993184121 4580 Q R 4380768 + 8 [dd] 8,16 1 770831 9.993184619 4580 M R 4380768 + 8 [dd] 8,16 1 770832 9.993186741 4580 Q R 4380776 + 8 [dd] 8,16 1 770833 9.993187239 4580 M R 4380776 + 8 [dd] 8,16 1 770834 9.993189324 4580 Q R 4380784 + 8 [dd] 8,16 1 770835 9.993189822 4580 M R 4380784 + 8 [dd] 8,16 1 770836 9.993191998 4580 Q R 4380792 + 8 [dd] 8,16 1 770837 9.993192502 4580 M R 4380792 + 8 [dd] 8,16 1 770838 9.993194798 4580 Q R 4380800 + 8 [dd] 8,16 1 770839 9.993195297 4580 M R 4380800 + 8 [dd] 8,16 1 770840 9.993197858 4580 Q R 4380808 + 8 [dd] 8,16 1 770841 9.993198358 4580 M R 4380808 + 8 [dd] 8,16 1 770842 9.993201427 4580 Q R 4380816 + 8 [dd] 8,16 1 770843 9.993201926 4580 M R 4380816 + 8 [dd] 8,16 1 770844 9.993204159 4580 Q R 4380824 + 8 [dd] 8,16 1 770845 9.993204658 4580 M R 4380824 + 8 [dd] 8,16 1 770846 9.993206840 4580 Q R 4380832 + 8 [dd] 8,16 1 770847 9.993207338 4580 M R 4380832 + 8 [dd] 8,16 1 770848 9.993212789 4580 Q R 4380840 + 8 [dd] 8,16 1 770849 9.993213321 4580 M R 4380840 + 8 [dd] 8,16 1 770850 9.993215482 4580 Q R 4380848 + 8 [dd] 8,16 1 770851 9.993215986 4580 M R 4380848 + 8 [dd] 8,16 1 770852 9.993218173 4580 Q R 4380856 + 8 [dd] 8,16 1 770853 9.993218673 4580 M R 4380856 + 8 [dd] 8,16 1 770854 9.993220839 4580 Q R 4380864 + 8 [dd] 8,16 1 770855 9.993221338 4580 M R 4380864 + 8 [dd] 8,16 1 770856 9.993223551 4580 Q R 4380872 + 8 [dd] 8,16 1 770857 9.993224051 4580 M R 4380872 + 8 [dd] 8,16 1 770858 9.993226133 4580 Q R 4380880 + 8 [dd] 8,16 1 770859 9.993226632 4580 M R 4380880 + 8 [dd] 8,16 1 770860 9.993228840 4580 Q R 4380888 + 8 [dd] 8,16 1 770861 9.993229342 4580 M R 4380888 + 8 [dd] 8,16 1 770862 9.993231460 4580 Q R 4380896 + 8 [dd] 8,16 1 770863 9.993231958 4580 M R 4380896 + 8 [dd] 8,16 1 770864 9.993234011 4580 Q R 4380904 + 8 [dd] 8,16 1 770865 9.993234513 4580 M R 4380904 + 8 [dd] 8,16 1 770866 9.993236661 4580 Q R 4380912 + 8 [dd] 8,16 1 770867 9.993237160 4580 M R 4380912 + 8 [dd] 8,16 1 770868 9.993239233 4580 Q R 4380920 + 8 [dd] 8,16 1 770869 9.993239736 4580 M R 4380920 + 8 [dd] 8,16 1 770870 9.993241912 4580 I R 4380416 + 512 ( 302760) [dd] 8,16 1 0 9.993244448 0 m N cfq4580S / insert_request 8,16 1 770871 9.993247771 4580 U N [dd] 1 8,16 1 0 9.993250443 0 m N cfq4580S / dispatch_insert 8,16 1 0 9.993254093 0 m N cfq4580S / dispatched a request 8,16 1 0 9.993255838 0 m N cfq4580S / activate rq, drv=1 8,16 1 770872 9.993256451 4580 D R 4380416 + 512 ( 14539) [dd] 8,16 1 770873 9.994299156 0 C R 4380416 + 512 ( 1042705) [0] 8,16 1 0 9.994371989 0 m N cfq4580S / complete rqnoidle 0 8,16 1 0 9.994375064 0 m N cfq schedule dispatch 8,16 1 770874 9.994536199 4580 Q R 4380928 + 8 [dd] 8,16 1 770875 9.994542473 4580 G R 4380928 + 8 [dd] 8,16 1 770876 9.994544517 4580 P N [dd] 8,16 1 770877 9.994548013 4580 Q R 4380936 + 8 [dd] 8,16 1 770878 9.994550069 4580 M R 4380936 + 8 [dd] 8,16 1 770879 9.994553102 4580 Q R 4380944 + 8 [dd] 8,16 1 770880 9.994553612 4580 M R 4380944 + 8 [dd] 8,16 1 770881 9.994560750 4580 Q R 4380952 + 8 [dd] 8,16 1 770882 9.994561294 4580 M R 4380952 + 8 [dd] 8,16 1 770883 9.994563660 4580 Q R 4380960 + 8 [dd] 8,16 1 770884 9.994564176 4580 M R 4380960 + 8 [dd] 8,16 1 770885 9.994566309 4580 Q R 4380968 + 8 [dd] 8,16 1 770886 9.994566819 4580 M R 4380968 + 8 [dd] 8,16 1 770887 9.994568963 4580 Q R 4380976 + 8 [dd] 8,16 1 770888 9.994569474 4580 M R 4380976 + 8 [dd] 8,16 1 770889 9.994571592 4580 Q R 4380984 + 8 [dd] 8,16 1 770890 9.994572104 4580 M R 4380984 + 8 [dd] 8,16 1 770891 9.994574265 4580 Q R 4380992 + 8 [dd] 8,16 1 770892 9.994574776 4580 M R 4380992 + 8 [dd] 8,16 1 770893 9.994576861 4580 Q R 4381000 + 8 [dd] 8,16 1 770894 9.994577360 4580 M R 4381000 + 8 [dd] 8,16 1 770895 9.994579582 4580 Q R 4381008 + 8 [dd] 8,16 1 770896 9.994580081 4580 M R 4381008 + 8 [dd] 8,16 1 770897 9.994582161 4580 Q R 4381016 + 8 [dd] 8,16 1 770898 9.994582661 4580 M R 4381016 + 8 [dd] 8,16 1 770899 9.994584717 4580 Q R 4381024 + 8 [dd] 8,16 1 770900 9.994585216 4580 M R 4381024 + 8 [dd] 8,16 1 770901 9.994587324 4580 Q R 4381032 + 8 [dd] 8,16 1 770902 9.994587822 4580 M R 4381032 + 8 [dd] 8,16 1 770903 9.994589915 4580 Q R 4381040 + 8 [dd] 8,16 1 770904 9.994590414 4580 M R 4381040 + 8 [dd] 8,16 1 770905 9.994594071 4580 Q R 4381048 + 8 [dd] 8,16 1 770906 9.994594575 4580 M R 4381048 + 8 [dd] 8,16 1 770907 9.994596896 4580 Q R 4381056 + 8 [dd] 8,16 1 770908 9.994597407 4580 M R 4381056 + 8 [dd] 8,16 1 770909 9.994602878 4580 Q R 4381064 + 8 [dd] 8,16 1 770910 9.994603394 4580 M R 4381064 + 8 [dd] 8,16 1 770911 9.994605653 4580 Q R 4381072 + 8 [dd] 8,16 1 770912 9.994606164 4580 M R 4381072 + 8 [dd] 8,16 1 770913 9.994608528 4580 Q R 4381080 + 8 [dd] 8,16 1 770914 9.994609040 4580 M R 4381080 + 8 [dd] 8,16 1 770915 9.994611239 4580 Q R 4381088 + 8 [dd] 8,16 1 770916 9.994611751 4580 M R 4381088 + 8 [dd] 8,16 1 770917 9.994613936 4580 Q R 4381096 + 8 [dd] 8,16 1 770918 9.994614447 4580 M R 4381096 + 8 [dd] 8,16 1 770919 9.994616718 4580 Q R 4381104 + 8 [dd] 8,16 1 770920 9.994617229 4580 M R 4381104 + 8 [dd] 8,16 1 770921 9.994619394 4580 Q R 4381112 + 8 [dd] 8,16 1 770922 9.994619905 4580 M R 4381112 + 8 [dd] 8,16 1 770923 9.994622152 4580 Q R 4381120 + 8 [dd] 8,16 1 770924 9.994622663 4580 M R 4381120 + 8 [dd] 8,16 1 770925 9.994624765 4580 Q R 4381128 + 8 [dd] 8,16 1 770926 9.994625294 4580 M R 4381128 + 8 [dd] 8,16 1 770927 9.994627643 4580 Q R 4381136 + 8 [dd] 8,16 1 770928 9.994628172 4580 M R 4381136 + 8 [dd] 8,16 1 770929 9.994630638 4580 Q R 4381144 + 8 [dd] 8,16 1 770930 9.994631141 4580 M R 4381144 + 8 [dd] 8,16 1 770931 9.994633374 4580 Q R 4381152 + 8 [dd] 8,16 1 770932 9.994633885 4580 M R 4381152 + 8 [dd] 8,16 1 770933 9.994636086 4580 Q R 4381160 + 8 [dd] 8,16 1 770934 9.994636592 4580 M R 4381160 + 8 [dd] 8,16 1 770935 9.994638779 4580 Q R 4381168 + 8 [dd] 8,16 1 770936 9.994639291 4580 M R 4381168 + 8 [dd] 8,16 1 770937 9.994644640 4580 Q R 4381176 + 8 [dd] 8,16 1 770938 9.994645153 4580 M R 4381176 + 8 [dd] 8,16 1 770939 9.994750984 4580 Q R 4381184 + 8 [dd] 8,16 1 770940 9.994751713 4580 M R 4381184 + 8 [dd] 8,16 1 770941 9.994754423 4580 Q R 4381192 + 8 [dd] 8,16 1 770942 9.994754936 4580 M R 4381192 + 8 [dd] 8,16 1 770943 9.994757220 4580 Q R 4381200 + 8 [dd] 8,16 1 770944 9.994757719 4580 M R 4381200 + 8 [dd] 8,16 1 770945 9.994759967 4580 Q R 4381208 + 8 [dd] 8,16 1 770946 9.994760506 4580 M R 4381208 + 8 [dd] 8,16 1 770947 9.994762743 4580 Q R 4381216 + 8 [dd] 8,16 1 770948 9.994763242 4580 M R 4381216 + 8 [dd] 8,16 1 770949 9.994765425 4580 Q R 4381224 + 8 [dd] 8,16 1 770950 9.994765922 4580 M R 4381224 + 8 [dd] 8,16 1 770951 9.994768201 4580 Q R 4381232 + 8 [dd] 8,16 1 770952 9.994768700 4580 M R 4381232 + 8 [dd] 8,16 1 770953 9.994770798 4580 Q R 4381240 + 8 [dd] 8,16 1 770954 9.994771298 4580 M R 4381240 + 8 [dd] 8,16 1 770955 9.994773393 4580 Q R 4381248 + 8 [dd] 8,16 1 770956 9.994773892 4580 M R 4381248 + 8 [dd] 8,16 1 770957 9.994776015 4580 Q R 4381256 + 8 [dd] 8,16 1 770958 9.994776513 4580 M R 4381256 + 8 [dd] 8,16 1 770959 9.994778608 4580 Q R 4381264 + 8 [dd] 8,16 1 770960 9.994779107 4580 M R 4381264 + 8 [dd] 8,16 1 770961 9.994781210 4580 Q R 4381272 + 8 [dd] 8,16 1 770962 9.994781710 4580 M R 4381272 + 8 [dd] 8,16 1 770963 9.994787731 4580 Q R 4381280 + 8 [dd] 8,16 1 770964 9.994788230 4580 M R 4381280 + 8 [dd] 8,16 1 770965 9.994793675 4580 Q R 4381288 + 8 [dd] 8,16 1 770966 9.994794174 4580 M R 4381288 + 8 [dd] 8,16 1 770967 9.994796307 4580 Q R 4381296 + 8 [dd] 8,16 1 770968 9.994796806 4580 M R 4381296 + 8 [dd] 8,16 1 770969 9.994799419 4580 Q R 4381304 + 8 [dd] 8,16 1 770970 9.994799919 4580 M R 4381304 + 8 [dd] 8,16 1 770971 9.994802209 4580 Q R 4381312 + 8 [dd] 8,16 1 770972 9.994802707 4580 M R 4381312 + 8 [dd] 8,16 1 770973 9.994805174 4580 Q R 4381320 + 8 [dd] 8,16 1 770974 9.994805672 4580 M R 4381320 + 8 [dd] 8,16 1 770975 9.994807936 4580 Q R 4381328 + 8 [dd] 8,16 1 770976 9.994808440 4580 M R 4381328 + 8 [dd] 8,16 1 770977 9.994810715 4580 Q R 4381336 + 8 [dd] 8,16 1 770978 9.994811242 4580 M R 4381336 + 8 [dd] 8,16 1 770979 9.994813464 4580 Q R 4381344 + 8 [dd] 8,16 1 770980 9.994813963 4580 M R 4381344 + 8 [dd] 8,16 1 770981 9.994816205 4580 Q R 4381352 + 8 [dd] 8,16 1 770982 9.994816776 4580 M R 4381352 + 8 [dd] 8,16 1 770983 9.994819127 4580 Q R 4381360 + 8 [dd] 8,16 1 770984 9.994819628 4580 M R 4381360 + 8 [dd] 8,16 1 770985 9.994821810 4580 Q R 4381368 + 8 [dd] 8,16 1 770986 9.994822315 4580 M R 4381368 + 8 [dd] 8,16 1 770987 9.994824377 4580 Q R 4381376 + 8 [dd] 8,16 1 770988 9.994824876 4580 M R 4381376 + 8 [dd] 8,16 1 770989 9.994827067 4580 Q R 4381384 + 8 [dd] 8,16 1 770990 9.994827565 4580 M R 4381384 + 8 [dd] 8,16 1 770991 9.994829760 4580 Q R 4381392 + 8 [dd] 8,16 1 770992 9.994830259 4580 M R 4381392 + 8 [dd] 8,16 1 770993 9.994835522 4580 Q R 4381400 + 8 [dd] 8,16 1 770994 9.994836021 4580 M R 4381400 + 8 [dd] 8,16 1 770995 9.994838181 4580 Q R 4381408 + 8 [dd] 8,16 1 770996 9.994838680 4580 M R 4381408 + 8 [dd] 8,16 1 770997 9.994840857 4580 Q R 4381416 + 8 [dd] 8,16 1 770998 9.994841369 4580 M R 4381416 + 8 [dd] 8,16 1 770999 9.994843455 4580 Q R 4381424 + 8 [dd] 8,16 1 771000 9.994843954 4580 M R 4381424 + 8 [dd] 8,16 1 771001 9.994846149 4580 Q R 4381432 + 8 [dd] 8,16 1 771002 9.994846662 4580 M R 4381432 + 8 [dd] 8,16 1 771003 9.994849336 4580 I R 4380928 + 512 ( 306863) [dd] 8,16 1 0 9.994852010 0 m N cfq4580S / insert_request 8,16 1 771004 9.994855251 4580 U N [dd] 1 8,16 1 0 9.994857783 0 m N cfq4580S / dispatch_insert 8,16 1 0 9.994861277 0 m N cfq4580S / dispatched a request 8,16 1 0 9.994863066 0 m N cfq4580S / activate rq, drv=1 8,16 1 771005 9.994863680 4580 D R 4380928 + 512 ( 14344) [dd] 8,16 1 771006 9.995909291 0 C R 4380928 + 512 ( 1045611) [0] 8,16 1 0 9.995983219 0 m N cfq4580S / complete rqnoidle 0 8,16 1 0 9.995986144 0 m N cfq schedule dispatch 8,16 1 771007 9.996149764 4580 Q R 4381440 + 8 [dd] 8,16 1 771008 9.996155872 4580 G R 4381440 + 8 [dd] 8,16 1 771009 9.996157994 4580 P N [dd] 8,16 1 771010 9.996161548 4580 Q R 4381448 + 8 [dd] 8,16 1 771011 9.996163428 4580 M R 4381448 + 8 [dd] 8,16 1 771012 9.996166608 4580 Q R 4381456 + 8 [dd] 8,16 1 771013 9.996167115 4580 M R 4381456 + 8 [dd] 8,16 1 771014 9.996169178 4580 Q R 4381464 + 8 [dd] 8,16 1 771015 9.996169681 4580 M R 4381464 + 8 [dd] 8,16 1 771016 9.996171779 4580 Q R 4381472 + 8 [dd] 8,16 1 771017 9.996172277 4580 M R 4381472 + 8 [dd] 8,16 1 771018 9.996174323 4580 Q R 4381480 + 8 [dd] 8,16 1 771019 9.996174822 4580 M R 4381480 + 8 [dd] 8,16 1 771020 9.996176942 4580 Q R 4381488 + 8 [dd] 8,16 1 771021 9.996177446 4580 M R 4381488 + 8 [dd] 8,16 1 771022 9.996179520 4580 Q R 4381496 + 8 [dd] 8,16 1 771023 9.996180022 4580 M R 4381496 + 8 [dd] 8,16 1 771024 9.996182075 4580 Q R 4381504 + 8 [dd] 8,16 1 771025 9.996182574 4580 M R 4381504 + 8 [dd] 8,16 1 771026 9.996189624 4580 Q R 4381512 + 8 [dd] 8,16 1 771027 9.996190124 4580 M R 4381512 + 8 [dd] 8,16 1 771028 9.996192318 4580 Q R 4381520 + 8 [dd] 8,16 1 771029 9.996192819 4580 M R 4381520 + 8 [dd] 8,16 1 771030 9.996194916 4580 Q R 4381528 + 8 [dd] 8,16 1 771031 9.996195415 4580 M R 4381528 + 8 [dd] 8,16 1 771032 9.996197792 4580 Q R 4381536 + 8 [dd] 8,16 1 771033 9.996198291 4580 M R 4381536 + 8 [dd] 8,16 1 771034 9.996200401 4580 Q R 4381544 + 8 [dd] 8,16 1 771035 9.996200899 4580 M R 4381544 + 8 [dd] 8,16 1 771036 9.996202991 4580 Q R 4381552 + 8 [dd] 8,16 1 771037 9.996203495 4580 M R 4381552 + 8 [dd] 8,16 1 771038 9.996205593 4580 Q R 4381560 + 8 [dd] 8,16 1 771039 9.996206095 4580 M R 4381560 + 8 [dd] 8,16 1 771040 9.996208179 4580 Q R 4381568 + 8 [dd] 8,16 1 771041 9.996208678 4580 M R 4381568 + 8 [dd] 8,16 1 771042 9.996210778 4580 Q R 4381576 + 8 [dd] 8,16 1 771043 9.996211285 4580 M R 4381576 + 8 [dd] 8,16 1 771044 9.996213428 4580 Q R 4381584 + 8 [dd] 8,16 1 771045 9.996213932 4580 M R 4381584 + 8 [dd] 8,16 1 771046 9.996217450 4580 Q R 4381592 + 8 [dd] 8,16 1 771047 9.996217952 4580 M R 4381592 + 8 [dd] 8,16 1 771048 9.996220131 4580 Q R 4381600 + 8 [dd] 8,16 1 771049 9.996220644 4580 M R 4381600 + 8 [dd] 8,16 1 771050 9.996222799 4580 Q R 4381608 + 8 [dd] 8,16 1 771051 9.996223302 4580 M R 4381608 + 8 [dd] 8,16 1 771052 9.996225537 4580 Q R 4381616 + 8 [dd] 8,16 1 771053 9.996226039 4580 M R 4381616 + 8 [dd] 8,16 1 771054 9.996231412 4580 Q R 4381624 + 8 [dd] 8,16 1 771055 9.996231912 4580 M R 4381624 + 8 [dd] 8,16 1 771056 9.996234006 4580 Q R 4381632 + 8 [dd] 8,16 1 771057 9.996234505 4580 M R 4381632 + 8 [dd] 8,16 1 771058 9.996236661 4580 Q R 4381640 + 8 [dd] 8,16 1 771059 9.996237160 4580 M R 4381640 + 8 [dd] 8,16 1 771060 9.996239325 4580 Q R 4381648 + 8 [dd] 8,16 1 771061 9.996239829 4580 M R 4381648 + 8 [dd] 8,16 1 771062 9.996242197 4580 Q R 4381656 + 8 [dd] 8,16 1 771063 9.996242700 4580 M R 4381656 + 8 [dd] 8,16 1 771064 9.996244888 4580 Q R 4381664 + 8 [dd] 8,16 1 771065 9.996245388 4580 M R 4381664 + 8 [dd] 8,16 1 771066 9.996247580 4580 Q R 4381672 + 8 [dd] 8,16 1 771067 9.996248079 4580 M R 4381672 + 8 [dd] 8,16 1 771068 9.996250234 4580 Q R 4381680 + 8 [dd] 8,16 1 771069 9.996250735 4580 M R 4381680 + 8 [dd] 8,16 1 771070 9.996252898 4580 Q R 4381688 + 8 [dd] 8,16 1 771071 9.996253388 4580 M R 4381688 + 8 [dd] 8,16 1 771072 9.996357598 4580 Q R 4381696 + 8 [dd] 8,16 1 771073 9.996358355 4580 M R 4381696 + 8 [dd] 8,16 1 771074 9.996360935 4580 Q R 4381704 + 8 [dd] 8,16 1 771075 9.996361434 4580 M R 4381704 + 8 [dd] 8,16 1 771076 9.996363662 4580 Q R 4381712 + 8 [dd] 8,16 1 771077 9.996364166 4580 M R 4381712 + 8 [dd] 8,16 1 771078 9.996366220 4580 Q R 4381720 + 8 [dd] 8,16 1 771079 9.996366731 4580 M R 4381720 + 8 [dd] 8,16 1 771080 9.996368930 4580 Q R 4381728 + 8 [dd] 8,16 1 771081 9.996369429 4580 M R 4381728 + 8 [dd] 8,16 1 771082 9.996375003 4580 Q R 4381736 + 8 [dd] 8,16 1 771083 9.996375513 4580 M R 4381736 + 8 [dd] 8,16 1 771084 9.996377664 4580 Q R 4381744 + 8 [dd] 8,16 1 771085 9.996378380 4580 M R 4381744 + 8 [dd] 8,16 1 771086 9.996380579 4580 Q R 4381752 + 8 [dd] 8,16 1 771087 9.996381098 4580 M R 4381752 + 8 [dd] 8,16 1 771088 9.996383260 4580 Q R 4381760 + 8 [dd] 8,16 1 771089 9.996383759 4580 M R 4381760 + 8 [dd] 8,16 1 771090 9.996385969 4580 Q R 4381768 + 8 [dd] 8,16 1 771091 9.996386469 4580 M R 4381768 + 8 [dd] 8,16 1 771092 9.996388658 4580 Q R 4381776 + 8 [dd] 8,16 1 771093 9.996389162 4580 M R 4381776 + 8 [dd] 8,16 1 771094 9.996391335 4580 Q R 4381784 + 8 [dd] 8,16 1 771095 9.996391837 4580 M R 4381784 + 8 [dd] 8,16 1 771096 9.996393920 4580 Q R 4381792 + 8 [dd] 8,16 1 771097 9.996394419 4580 M R 4381792 + 8 [dd] 8,16 1 771098 9.996396469 4580 Q R 4381800 + 8 [dd] 8,16 1 771099 9.996396967 4580 M R 4381800 + 8 [dd] 8,16 1 771100 9.996399181 4580 Q R 4381808 + 8 [dd] 8,16 1 771101 9.996399684 4580 M R 4381808 + 8 [dd] 8,16 1 771102 9.996402243 4580 Q R 4381816 + 8 [dd] 8,16 1 771103 9.996402751 4580 M R 4381816 + 8 [dd] 8,16 1 771104 9.996404942 4580 Q R 4381824 + 8 [dd] 8,16 1 771105 9.996405441 4580 M R 4381824 + 8 [dd] 8,16 1 771106 9.996407662 4580 Q R 4381832 + 8 [dd] 8,16 1 771107 9.996408160 4580 M R 4381832 + 8 [dd] 8,16 1 771108 9.996410409 4580 Q R 4381840 + 8 [dd] 8,16 1 771109 9.996410910 4580 M R 4381840 + 8 [dd] 8,16 1 771110 9.996416841 4580 Q R 4381848 + 8 [dd] 8,16 1 771111 9.996417341 4580 M R 4381848 + 8 [dd] 8,16 1 771112 9.996419551 4580 Q R 4381856 + 8 [dd] 8,16 1 771113 9.996420050 4580 M R 4381856 + 8 [dd] 8,16 1 771114 9.996425832 4580 Q R 4381864 + 8 [dd] 8,16 1 771115 9.996426331 4580 M R 4381864 + 8 [dd] 8,16 1 771116 9.996428600 4580 Q R 4381872 + 8 [dd] 8,16 1 771117 9.996429101 4580 M R 4381872 + 8 [dd] 8,16 1 771118 9.996431277 4580 Q R 4381880 + 8 [dd] 8,16 1 771119 9.996431779 4580 M R 4381880 + 8 [dd] 8,16 1 771120 9.996434026 4580 Q R 4381888 + 8 [dd] 8,16 1 771121 9.996434525 4580 M R 4381888 + 8 [dd] 8,16 1 771122 9.996436615 4580 Q R 4381896 + 8 [dd] 8,16 1 771123 9.996437114 4580 M R 4381896 + 8 [dd] 8,16 1 771124 9.996439353 4580 Q R 4381904 + 8 [dd] 8,16 1 771125 9.996439864 4580 M R 4381904 + 8 [dd] 8,16 1 771126 9.996441959 4580 Q R 4381912 + 8 [dd] 8,16 1 771127 9.996442460 4580 M R 4381912 + 8 [dd] 8,16 1 771128 9.996445027 4580 Q R 4381920 + 8 [dd] 8,16 1 771129 9.996445526 4580 M R 4381920 + 8 [dd] 8,16 1 771130 9.996447737 4580 Q R 4381928 + 8 [dd] 8,16 1 771131 9.996448252 4580 M R 4381928 + 8 [dd] 8,16 1 771132 9.996450536 4580 Q R 4381936 + 8 [dd] 8,16 1 771133 9.996451040 4580 M R 4381936 + 8 [dd] 8,16 1 771134 9.996453308 4580 Q R 4381944 + 8 [dd] 8,16 1 771135 9.996453811 4580 M R 4381944 + 8 [dd] 8,16 1 771136 9.996456417 4580 I R 4381440 + 512 ( 300545) [dd] 8,16 1 0 9.996458987 0 m N cfq4580S / insert_request 8,16 1 771137 9.996462198 4580 U N [dd] 1 8,16 1 0 9.996464756 0 m N cfq4580S / dispatch_insert 8,16 1 0 9.996468170 0 m N cfq4580S / dispatched a request 8,16 1 0 9.996469836 0 m N cfq4580S / activate rq, drv=1 8,16 1 771138 9.996470460 4580 D R 4381440 + 512 ( 14043) [dd] 8,16 1 771139 9.997514205 0 C R 4381440 + 512 ( 1043745) [0] 8,16 1 0 9.997587102 0 m N cfq4580S / complete rqnoidle 0 8,16 1 0 9.997589870 0 m N cfq schedule dispatch 8,16 1 771140 9.997751496 4580 Q R 4381952 + 8 [dd] 8,16 1 771141 9.997758630 4580 G R 4381952 + 8 [dd] 8,16 1 771142 9.997760814 4580 P N [dd] 8,16 1 771143 9.997769413 4580 Q R 4381960 + 8 [dd] 8,16 1 771144 9.997771473 4580 M R 4381960 + 8 [dd] 8,16 1 771145 9.997774552 4580 Q R 4381968 + 8 [dd] 8,16 1 771146 9.997775056 4580 M R 4381968 + 8 [dd] 8,16 1 771147 9.997777142 4580 Q R 4381976 + 8 [dd] 8,16 1 771148 9.997777717 4580 M R 4381976 + 8 [dd] 8,16 1 771149 9.997779832 4580 Q R 4381984 + 8 [dd] 8,16 1 771150 9.997780366 4580 M R 4381984 + 8 [dd] 8,16 1 771151 9.997782454 4580 Q R 4381992 + 8 [dd] 8,16 1 771152 9.997782953 4580 M R 4381992 + 8 [dd] 8,16 1 771153 9.997785271 4580 Q R 4382000 + 8 [dd] 8,16 1 771154 9.997785779 4580 M R 4382000 + 8 [dd] 8,16 1 771155 9.997787937 4580 Q R 4382008 + 8 [dd] 8,16 1 771156 9.997788434 4580 M R 4382008 + 8 [dd] 8,16 1 771157 9.997790588 4580 Q R 4382016 + 8 [dd] 8,16 1 771158 9.997791085 4580 M R 4382016 + 8 [dd] 8,16 1 771159 9.997793251 4580 Q R 4382024 + 8 [dd] 8,16 1 771160 9.997793954 4580 M R 4382024 + 8 [dd] 8,16 1 771161 9.997796144 4580 Q R 4382032 + 8 [dd] 8,16 1 771162 9.997796672 4580 M R 4382032 + 8 [dd] 8,16 1 771163 9.997798737 4580 Q R 4382040 + 8 [dd] 8,16 1 771164 9.997799238 4580 M R 4382040 + 8 [dd] 8,16 1 771165 9.997801277 4580 Q R 4382048 + 8 [dd] 8,16 1 771166 9.997801788 4580 M R 4382048 + 8 [dd] 8,16 1 771167 9.997803925 4580 Q R 4382056 + 8 [dd] 8,16 1 771168 9.997804439 4580 M R 4382056 + 8 [dd] 8,16 1 771169 9.997806598 4580 Q R 4382064 + 8 [dd] 8,16 1 771170 9.997807095 4580 M R 4382064 + 8 [dd] 8,16 1 771171 9.997812761 4580 Q R 4382072 + 8 [dd] 8,16 1 771172 9.997813262 4580 M R 4382072 + 8 [dd] 8,16 1 771173 9.997815335 4580 Q R 4382080 + 8 [dd] 8,16 1 771174 9.997815836 4580 M R 4382080 + 8 [dd] 8,16 1 771175 9.997817949 4580 Q R 4382088 + 8 [dd] 8,16 1 771176 9.997818448 4580 M R 4382088 + 8 [dd] 8,16 1 771177 9.997820558 4580 Q R 4382096 + 8 [dd] 8,16 1 771178 9.997821057 4580 M R 4382096 + 8 [dd] 8,16 1 771179 9.997824698 4580 Q R 4382104 + 8 [dd] 8,16 1 771180 9.997825197 4580 M R 4382104 + 8 [dd] 8,16 1 771181 9.997827376 4580 Q R 4382112 + 8 [dd] 8,16 1 771182 9.997827874 4580 M R 4382112 + 8 [dd] 8,16 1 771183 9.997830036 4580 Q R 4382120 + 8 [dd] 8,16 1 771184 9.997830538 4580 M R 4382120 + 8 [dd] 8,16 1 771185 9.997832781 4580 Q R 4382128 + 8 [dd] 8,16 1 771186 9.997833278 4580 M R 4382128 + 8 [dd] 8,16 1 771187 9.997835542 4580 Q R 4382136 + 8 [dd] 8,16 1 771188 9.997836044 4580 M R 4382136 + 8 [dd] 8,16 1 771189 9.997838090 4580 Q R 4382144 + 8 [dd] 8,16 1 771190 9.997838591 4580 M R 4382144 + 8 [dd] 8,16 1 771191 9.997840646 4580 Q R 4382152 + 8 [dd] 8,16 1 771192 9.997841147 4580 M R 4382152 + 8 [dd] 8,16 1 771193 9.997843299 4580 Q R 4382160 + 8 [dd] 8,16 1 771194 9.997843797 4580 M R 4382160 + 8 [dd] 8,16 1 771195 9.997845909 4580 Q R 4382168 + 8 [dd] 8,16 1 771196 9.997846408 4580 M R 4382168 + 8 [dd] 8,16 1 771197 9.997848578 4580 Q R 4382176 + 8 [dd] 8,16 1 771198 9.997849082 4580 M R 4382176 + 8 [dd] 8,16 1 771199 9.997854848 4580 Q R 4382184 + 8 [dd] 8,16 1 771200 9.997855350 4580 M R 4382184 + 8 [dd] 8,16 1 771201 9.997857436 4580 Q R 4382192 + 8 [dd] 8,16 1 771202 9.997857934 4580 M R 4382192 + 8 [dd] 8,16 1 771203 9.997860278 4580 Q R 4382200 + 8 [dd] 8,16 1 771204 9.997860779 4580 M R 4382200 + 8 [dd] 8,16 1 771205 9.997964321 4580 Q R 4382208 + 8 [dd] 8,16 1 771206 9.997965007 4580 M R 4382208 + 8 [dd] 8,16 1 771207 9.997967619 4580 Q R 4382216 + 8 [dd] 8,16 1 771208 9.997968161 4580 M R 4382216 + 8 [dd] 8,16 1 771209 9.997970459 4580 Q R 4382224 + 8 [dd] 8,16 1 771210 9.997970963 4580 M R 4382224 + 8 [dd] 8,16 1 771211 9.997973081 4580 Q R 4382232 + 8 [dd] 8,16 1 771212 9.997973580 4580 M R 4382232 + 8 [dd] 8,16 1 771213 9.997975787 4580 Q R 4382240 + 8 [dd] 8,16 1 771214 9.997976285 4580 M R 4382240 + 8 [dd] 8,16 1 771215 9.997978452 4580 Q R 4382248 + 8 [dd] 8,16 1 771216 9.997978953 4580 M R 4382248 + 8 [dd] 8,16 1 771217 9.997981249 4580 Q R 4382256 + 8 [dd] 8,16 1 771218 9.997981751 4580 M R 4382256 + 8 [dd] 8,16 1 771219 9.997983917 4580 Q R 4382264 + 8 [dd] 8,16 1 771220 9.997984418 4580 M R 4382264 + 8 [dd] 8,16 1 771221 9.997986611 4580 Q R 4382272 + 8 [dd] 8,16 1 771222 9.997987121 4580 M R 4382272 + 8 [dd] 8,16 1 771223 9.997989204 4580 Q R 4382280 + 8 [dd] 8,16 1 771224 9.997989705 4580 M R 4382280 + 8 [dd] 8,16 1 771225 9.997991904 4580 Q R 4382288 + 8 [dd] 8,16 1 771226 9.997992402 4580 M R 4382288 + 8 [dd] 8,16 1 771227 9.997998036 4580 Q R 4382296 + 8 [dd] 8,16 1 771228 9.997998579 4580 M R 4382296 + 8 [dd] 8,16 1 771229 9.998000634 4580 Q R 4382304 + 8 [dd] 8,16 1 771230 9.998001136 4580 M R 4382304 + 8 [dd] 8,16 1 771231 9.998003255 4580 Q R 4382312 + 8 [dd] 8,16 1 771232 9.998003774 4580 M R 4382312 + 8 [dd] 8,16 1 771233 9.998005975 4580 Q R 4382320 + 8 [dd] 8,16 1 771234 9.998006476 4580 M R 4382320 + 8 [dd] 8,16 1 771235 9.998008972 4580 Q R 4382328 + 8 [dd] 8,16 1 771236 9.998009529 4580 M R 4382328 + 8 [dd] 8,16 1 771237 9.998011779 4580 Q R 4382336 + 8 [dd] 8,16 1 771238 9.998012277 4580 M R 4382336 + 8 [dd] 8,16 1 771239 9.998014476 4580 Q R 4382344 + 8 [dd] 8,16 1 771240 9.998014974 4580 M R 4382344 + 8 [dd] 8,16 1 771241 9.998017401 4580 Q R 4382352 + 8 [dd] 8,16 1 771242 9.998017899 4580 M R 4382352 + 8 [dd] 8,16 1 771243 9.998020091 4580 Q R 4382360 + 8 [dd] 8,16 1 771244 9.998020588 4580 M R 4382360 + 8 [dd] 8,16 1 771245 9.998022782 4580 Q R 4382368 + 8 [dd] 8,16 1 771246 9.998023389 4580 M R 4382368 + 8 [dd] 8,16 1 771247 9.998025647 4580 Q R 4382376 + 8 [dd] 8,16 1 771248 9.998026180 4580 M R 4382376 + 8 [dd] 8,16 1 771249 9.998028480 4580 Q R 4382384 + 8 [dd] 8,16 1 771250 9.998029020 4580 M R 4382384 + 8 [dd] 8,16 1 771251 9.998031205 4580 Q R 4382392 + 8 [dd] 8,16 1 771252 9.998031706 4580 M R 4382392 + 8 [dd] 8,16 1 771253 9.998033865 4580 Q R 4382400 + 8 [dd] 8,16 1 771254 9.998034366 4580 M R 4382400 + 8 [dd] 8,16 1 771255 9.998040073 4580 Q R 4382408 + 8 [dd] 8,16 1 771256 9.998040571 4580 M R 4382408 + 8 [dd] 8,16 1 771257 9.998042647 4580 Q R 4382416 + 8 [dd] 8,16 1 771258 9.998043150 4580 M R 4382416 + 8 [dd] 8,16 1 771259 9.998045325 4580 Q R 4382424 + 8 [dd] 8,16 1 771260 9.998045826 4580 M R 4382424 + 8 [dd] 8,16 1 771261 9.998047931 4580 Q R 4382432 + 8 [dd] 8,16 1 771262 9.998048432 4580 M R 4382432 + 8 [dd] 8,16 1 771263 9.998050535 4580 Q R 4382440 + 8 [dd] 8,16 1 771264 9.998051038 4580 M R 4382440 + 8 [dd] 8,16 1 771265 9.998056985 4580 Q R 4382448 + 8 [dd] 8,16 1 771266 9.998057500 4580 M R 4382448 + 8 [dd] 8,16 1 771267 9.998059576 4580 Q R 4382456 + 8 [dd] 8,16 1 771268 9.998060077 4580 M R 4382456 + 8 [dd] 8,16 1 771269 9.998062341 4580 I R 4381952 + 512 ( 303711) [dd] 8,16 1 0 9.998064881 0 m N cfq4580S / insert_request 8,16 1 771270 9.998068490 4580 U N [dd] 1 8,16 1 0 9.998071031 0 m N cfq4580S / dispatch_insert 8,16 1 0 9.998074991 0 m N cfq4580S / dispatched a request 8,16 1 0 9.998076645 0 m N cfq4580S / activate rq, drv=1 8,16 1 771271 9.998077269 4580 D R 4381952 + 512 ( 14928) [dd] 8,16 1 771272 9.999120396 0 C R 4381952 + 512 ( 1043127) [0] 8,16 1 0 9.999192302 0 m N cfq4580S / complete rqnoidle 0 8,16 1 0 9.999195361 0 m N cfq schedule dispatch 8,16 1 771273 9.999356099 4580 Q R 4382464 + 8 [dd] 8,16 1 771274 9.999362719 4580 G R 4382464 + 8 [dd] 8,16 1 771275 9.999364795 4580 P N [dd] 8,16 1 771276 9.999368199 4580 Q R 4382472 + 8 [dd] 8,16 1 771277 9.999370072 4580 M R 4382472 + 8 [dd] 8,16 1 771278 9.999373166 4580 Q R 4382480 + 8 [dd] 8,16 1 771279 9.999373710 4580 M R 4382480 + 8 [dd] 8,16 1 771280 9.999375830 4580 Q R 4382488 + 8 [dd] 8,16 1 771281 9.999376334 4580 M R 4382488 + 8 [dd] 8,16 1 771282 9.999378433 4580 Q R 4382496 + 8 [dd] 8,16 1 771283 9.999378932 4580 M R 4382496 + 8 [dd] 8,16 1 771284 9.999380973 4580 Q R 4382504 + 8 [dd] 8,16 1 771285 9.999381477 4580 M R 4382504 + 8 [dd] 8,16 1 771286 9.999383602 4580 Q R 4382512 + 8 [dd] 8,16 1 771287 9.999384101 4580 M R 4382512 + 8 [dd] 8,16 1 771288 9.999391190 4580 Q R 4382520 + 8 [dd] 8,16 1 771289 9.999391690 4580 M R 4382520 + 8 [dd] 8,16 1 771290 9.999393850 4580 Q R 4382528 + 8 [dd] 8,16 1 771291 9.999394351 4580 M R 4382528 + 8 [dd] 8,16 1 771292 9.999396618 4580 Q R 4382536 + 8 [dd] 8,16 1 771293 9.999397190 4580 M R 4382536 + 8 [dd] 8,16 1 771294 9.999399323 4580 Q R 4382544 + 8 [dd] 8,16 1 771295 9.999399864 4580 M R 4382544 + 8 [dd] 8,16 1 771296 9.999401953 4580 Q R 4382552 + 8 [dd] 8,16 1 771297 9.999402472 4580 M R 4382552 + 8 [dd] 8,16 1 771298 9.999404563 4580 Q R 4382560 + 8 [dd] 8,16 1 771299 9.999405057 4580 M R 4382560 + 8 [dd] 8,16 1 771300 9.999407109 4580 Q R 4382568 + 8 [dd] 8,16 1 771301 9.999407611 4580 M R 4382568 + 8 [dd] 8,16 1 771302 9.999409677 4580 Q R 4382576 + 8 [dd] 8,16 1 771303 9.999410176 4580 M R 4382576 + 8 [dd] 8,16 1 771304 9.999412226 4580 Q R 4382584 + 8 [dd] 8,16 1 771305 9.999412734 4580 M R 4382584 + 8 [dd] 8,16 1 771306 9.999416340 4580 Q R 4382592 + 8 [dd] 8,16 1 771307 9.999416862 4580 M R 4382592 + 8 [dd] 8,16 1 771308 9.999419187 4580 Q R 4382600 + 8 [dd] 8,16 1 771309 9.999419706 4580 M R 4382600 + 8 [dd] 8,16 1 771310 9.999421977 4580 Q R 4382608 + 8 [dd] 8,16 1 771311 9.999422503 4580 M R 4382608 + 8 [dd] 8,16 1 771312 9.999424582 4580 Q R 4382616 + 8 [dd] 8,16 1 771313 9.999425120 4580 M R 4382616 + 8 [dd] 8,16 1 771314 9.999427292 4580 Q R 4382624 + 8 [dd] 8,16 1 771315 9.999427805 4580 M R 4382624 + 8 [dd] 8,16 1 771316 9.999433255 4580 Q R 4382632 + 8 [dd] 8,16 1 771317 9.999433766 4580 M R 4382632 + 8 [dd] 8,16 1 771318 9.999435901 4580 Q R 4382640 + 8 [dd] 8,16 1 771319 9.999436412 4580 M R 4382640 + 8 [dd] 8,16 1 771320 9.999438504 4580 Q R 4382648 + 8 [dd] 8,16 1 771321 9.999439243 4580 M R 4382648 + 8 [dd] 8,16 1 771322 9.999441418 4580 Q R 4382656 + 8 [dd] 8,16 1 771323 9.999441949 4580 M R 4382656 + 8 [dd] 8,16 1 771324 9.999444183 4580 Q R 4382664 + 8 [dd] 8,16 1 771325 9.999444698 4580 M R 4382664 + 8 [dd] 8,16 1 771326 9.999446894 4580 Q R 4382672 + 8 [dd] 8,16 1 771327 9.999447411 4580 M R 4382672 + 8 [dd] 8,16 1 771328 9.999449485 4580 Q R 4382680 + 8 [dd] 8,16 1 771329 9.999449997 4580 M R 4382680 + 8 [dd] 8,16 1 771330 9.999452055 4580 Q R 4382688 + 8 [dd] 8,16 1 771331 9.999452566 4580 M R 4382688 + 8 [dd] 8,16 1 771332 9.999454606 4580 Q R 4382696 + 8 [dd] 8,16 1 771333 9.999455154 4580 M R 4382696 + 8 [dd] 8,16 1 771334 9.999457363 4580 Q R 4382704 + 8 [dd] 8,16 1 771335 9.999457875 4580 M R 4382704 + 8 [dd] 8,16 1 771336 9.999460045 4580 Q R 4382712 + 8 [dd] 8,16 1 771337 9.999460561 4580 M R 4382712 + 8 [dd] 8,16 1 771338 9.999564615 4580 Q R 4382720 + 8 [dd] 8,16 1 771339 9.999565347 4580 M R 4382720 + 8 [dd] 8,16 1 771340 9.999568401 4580 Q R 4382728 + 8 [dd] 8,16 1 771341 9.999568908 4580 M R 4382728 + 8 [dd] 8,16 1 771342 9.999571276 4580 Q R 4382736 + 8 [dd] 8,16 1 771343 9.999571792 4580 M R 4382736 + 8 [dd] 8,16 1 771344 9.999577396 4580 Q R 4382744 + 8 [dd] 8,16 1 771345 9.999578003 4580 M R 4382744 + 8 [dd] 8,16 1 771346 9.999580231 4580 Q R 4382752 + 8 [dd] 8,16 1 771347 9.999580749 4580 M R 4382752 + 8 [dd] 8,16 1 771348 9.999583114 4580 Q R 4382760 + 8 [dd] 8,16 1 771349 9.999583625 4580 M R 4382760 + 8 [dd] 8,16 1 771350 9.999585875 4580 Q R 4382768 + 8 [dd] 8,16 1 771351 9.999586374 4580 M R 4382768 + 8 [dd] 8,16 1 771352 9.999588531 4580 Q R 4382776 + 8 [dd] 8,16 1 771353 9.999589032 4580 M R 4382776 + 8 [dd] 8,16 1 771354 9.999591214 4580 Q R 4382784 + 8 [dd] 8,16 1 771355 9.999591714 4580 M R 4382784 + 8 [dd] 8,16 1 771356 9.999593930 4580 Q R 4382792 + 8 [dd] 8,16 1 771357 9.999594442 4580 M R 4382792 + 8 [dd] 8,16 1 771358 9.999596696 4580 Q R 4382800 + 8 [dd] 8,16 1 771359 9.999597190 4580 M R 4382800 + 8 [dd] 8,16 1 771360 9.999599406 4580 Q R 4382808 + 8 [dd] 8,16 1 771361 9.999599907 4580 M R 4382808 + 8 [dd] 8,16 1 771362 9.999602166 4580 Q R 4382816 + 8 [dd] 8,16 1 771363 9.999602667 4580 M R 4382816 + 8 [dd] 8,16 1 771364 9.999604755 4580 Q R 4382824 + 8 [dd] 8,16 1 771365 9.999605259 4580 M R 4382824 + 8 [dd] 8,16 1 771366 9.999607486 4580 Q R 4382832 + 8 [dd] 8,16 1 771367 9.999607978 4580 M R 4382832 + 8 [dd] 8,16 1 771368 9.999610062 4580 Q R 4382840 + 8 [dd] 8,16 1 771369 9.999610564 4580 M R 4382840 + 8 [dd] 8,16 1 771370 9.999612914 4580 Q R 4382848 + 8 [dd] 8,16 1 771371 9.999613418 4580 M R 4382848 + 8 [dd] 8,16 1 771372 9.999618806 4580 Q R 4382856 + 8 [dd] 8,16 1 771373 9.999619308 4580 M R 4382856 + 8 [dd] 8,16 1 771374 9.999621579 4580 Q R 4382864 + 8 [dd] 8,16 1 771375 9.999622092 4580 M R 4382864 + 8 [dd] 8,16 1 771376 9.999624260 4580 Q R 4382872 + 8 [dd] 8,16 1 771377 9.999624779 4580 M R 4382872 + 8 [dd] 8,16 1 771378 9.999627506 4580 Q R 4382880 + 8 [dd] 8,16 1 771379 9.999628012 4580 M R 4382880 + 8 [dd] 8,16 1 771380 9.999630135 4580 Q R 4382888 + 8 [dd] 8,16 1 771381 9.999630635 4580 M R 4382888 + 8 [dd] 8,16 1 771382 9.999632876 4580 Q R 4382896 + 8 [dd] 8,16 1 771383 9.999633378 4580 M R 4382896 + 8 [dd] 8,16 1 771384 9.999635541 4580 Q R 4382904 + 8 [dd] 8,16 1 771385 9.999636043 4580 M R 4382904 + 8 [dd] 8,16 1 771386 9.999638232 4580 Q R 4382912 + 8 [dd] 8,16 1 771387 9.999638736 4580 M R 4382912 + 8 [dd] 8,16 1 771388 9.999640940 4580 Q R 4382920 + 8 [dd] 8,16 1 771389 9.999641440 4580 M R 4382920 + 8 [dd] 8,16 1 771390 9.999643644 4580 Q R 4382928 + 8 [dd] 8,16 1 771391 9.999644157 4580 M R 4382928 + 8 [dd] 8,16 1 771392 9.999646440 4580 Q R 4382936 + 8 [dd] 8,16 1 771393 9.999646960 4580 M R 4382936 + 8 [dd] 8,16 1 771394 9.999649193 4580 Q R 4382944 + 8 [dd] 8,16 1 771395 9.999649697 4580 M R 4382944 + 8 [dd] 8,16 1 771396 9.999651771 4580 Q R 4382952 + 8 [dd] 8,16 1 771397 9.999652272 4580 M R 4382952 + 8 [dd] 8,16 1 771398 9.999654476 4580 Q R 4382960 + 8 [dd] 8,16 1 771399 9.999654975 4580 M R 4382960 + 8 [dd] 8,16 1 771400 9.999660256 4580 Q R 4382968 + 8 [dd] 8,16 1 771401 9.999660760 4580 M R 4382968 + 8 [dd] 8,16 1 771402 9.999663197 4580 I R 4382464 + 512 ( 300478) [dd] 8,16 1 0 9.999665743 0 m N cfq4580S / insert_request 8,16 1 771403 9.999669190 4580 U N [dd] 1 8,16 1 0 9.999672048 0 m N cfq4580S / dispatch_insert 8,16 1 0 9.999675568 0 m N cfq4580S / dispatched a request 8,16 1 0 9.999677281 0 m N cfq4580S / activate rq, drv=1 8,16 1 771404 9.999677892 4580 D R 4382464 + 512 ( 14695) [dd] CPU0 (sdb): Reads Queued: 20608, 82432KiB Writes Queued: 0, 0KiB Read Dispatches: 322, 82432KiB Write Dispatches: 0, 0KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 22, 5632KiB Writes Completed: 0, 0KiB Read Merges: 20286, 81144KiB Write Merges: 0, 0KiB Read depth: 1 Write depth: 0 IO unplugs: 322 Timer unplugs: 0 CPU1 (sdb): Reads Queued: 371058, 1484MiB Writes Queued: 0, 0KiB Read Dispatches: 5798, 1484MiB Write Dispatches: 0, 0KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 6097, 1560MiB Writes Completed: 0, 0KiB Read Merges: 365261, 1461MiB Write Merges: 0, 0KiB Read depth: 1 Write depth: 0 IO unplugs: 5798 Timer unplugs: 0 Total (sdb): Reads Queued: 391666, 1566MiB Writes Queued: 0, 0KiB Read Dispatches: 6120, 1566MiB Write Dispatches: 0, 0KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 6119, 1566MiB Writes Completed: 0, 0KiB Read Merges: 385547, 1542MiB Write Merges: 0, 0KiB IO unplugs: 6120 Timer unplugs: 0 Throughput (R/W): 156662KiB/s / 0KiB/s Events (sdb): 851428 entries Skips: 0 forward (0 - 0.0%) Input file sdb.blktrace.0 added Input file sdb.blktrace.1 added ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 12:34 ` Eric Dumazet @ 2012-01-30 14:01 ` Wu Fengguang 2012-01-30 14:05 ` Wu Fengguang 0 siblings, 1 reply; 36+ messages in thread From: Wu Fengguang @ 2012-01-30 14:01 UTC (permalink / raw) To: Eric Dumazet Cc: Andrew Morton, LKML, Jens Axboe, Tejun Heo, Li Shaohua, Herbert Poetzl On Mon, Jan 30, 2012 at 01:34:56PM +0100, Eric Dumazet wrote: > Le lundi 30 janvier 2012 à 19:18 +0800, Wu Fengguang a écrit : > > > I agree with Shaohua that there may be timing/plug issues. There > > happen to be some plug patches and (maybe correlated) big performance > > drop between 2.6.38 and 2.6.39. The obvious way to move forward is to > > get some blktrace data on simple dd + new buggy kernel and let's check > > what's exactly going on. > > > > # start a background dd read > > blktrace /dev/sda -w 10 > > blkparse -t sda > > > > (Another machine here at office, hence /dev/sdb) > > echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=2M > count=2048 > 2048+0 enregistrements lus > 2048+0 enregistrements écrits > 4294967296 octets (4,3 GB) copiés, 26,0278 s, 165 MB/s > > # blktrace /dev/sdb -w 10 > === sdb === > CPU 0: 44766 events, 2099 KiB data > CPU 1: 828895 events, 38855 KiB data > Total: 873661 events (dropped 0), 40953 KiB data > > blkparse result is huge (55Mo) > > # blkparse -t sdb | tail -n 1000 Thanks for the data! It shows that - IO size is 256KB (which is not a problem in itself) - The dispatch/complete pattern is submit IO for range 1 complete IO for range 1 <dd busy, disk idle> submit IO for range 2 complete IO for range 2 So we have periods that no IO is in flight at all, which leads to under-utilized disk (which should show up in iostat as <100% disk util) # grep '[DC]' blktrace 8,16 1 770606 9.990039807 4580 D R 4379392 + 512 ( 14826) [dd] 8,16 1 770607 9.991083069 0 C R 4379392 + 512 ( 1043262) [0] 8,16 1 770739 9.991647434 4580 D R 4379904 + 512 ( 14433) [dd] 8,16 1 770740 9.992693317 0 C R 4379904 + 512 ( 1045883) [0] 8,16 1 770872 9.993256451 4580 D R 4380416 + 512 ( 14539) [dd] 8,16 1 770873 9.994299156 0 C R 4380416 + 512 ( 1042705) [0] 8,16 1 771005 9.994863680 4580 D R 4380928 + 512 ( 14344) [dd] 8,16 1 771006 9.995909291 0 C R 4380928 + 512 ( 1045611) [0] 8,16 1 771138 9.996470460 4580 D R 4381440 + 512 ( 14043) [dd] 8,16 1 771139 9.997514205 0 C R 4381440 + 512 ( 1043745) [0] 8,16 1 771271 9.998077269 4580 D R 4381952 + 512 ( 14928) [dd] 8,16 1 771272 9.999120396 0 C R 4381952 + 512 ( 1043127) [0] Thanks, Fengguang > 8,16 1 770475 9.989714963 4580 Q R 4379392 + 8 [dd] > 8,16 1 770476 9.989721470 4580 G R 4379392 + 8 [dd] > 8,16 1 770477 9.989723546 4580 P N [dd] > 8,16 1 770478 9.989726994 4580 Q R 4379400 + 8 [dd] > 8,16 1 770479 9.989728858 4580 M R 4379400 + 8 [dd] > 8,16 1 770480 9.989731893 4580 Q R 4379408 + 8 [dd] > 8,16 1 770481 9.989732467 4580 M R 4379408 + 8 [dd] > 8,16 1 770482 9.989734574 4580 Q R 4379416 + 8 [dd] > 8,16 1 770483 9.989735108 4580 M R 4379416 + 8 [dd] > 8,16 1 770484 9.989737220 4580 Q R 4379424 + 8 [dd] > 8,16 1 770485 9.989737721 4580 M R 4379424 + 8 [dd] > 8,16 1 770486 9.989739953 4580 Q R 4379432 + 8 [dd] > 8,16 1 770487 9.989740450 4580 M R 4379432 + 8 [dd] > 8,16 1 770488 9.989742577 4580 Q R 4379440 + 8 [dd] > 8,16 1 770489 9.989743078 4580 M R 4379440 + 8 [dd] > 8,16 1 770490 9.989745241 4580 Q R 4379448 + 8 [dd] > 8,16 1 770491 9.989745742 4580 M R 4379448 + 8 [dd] > 8,16 1 770492 9.989747953 4580 Q R 4379456 + 8 [dd] > 8,16 1 770493 9.989748485 4580 M R 4379456 + 8 [dd] > 8,16 1 770494 9.989750726 4580 Q R 4379464 + 8 [dd] > 8,16 1 770495 9.989751225 4580 M R 4379464 + 8 [dd] > 8,16 1 770496 9.989753390 4580 Q R 4379472 + 8 [dd] > 8,16 1 770497 9.989753891 4580 M R 4379472 + 8 [dd] > 8,16 1 770498 9.989755928 4580 Q R 4379480 + 8 [dd] > 8,16 1 770499 9.989756429 4580 M R 4379480 + 8 [dd] > 8,16 1 770500 9.989758470 4580 Q R 4379488 + 8 [dd] > 8,16 1 770501 9.989758975 4580 M R 4379488 + 8 [dd] > 8,16 1 770502 9.989765725 4580 Q R 4379496 + 8 [dd] > 8,16 1 770503 9.989766222 4580 M R 4379496 + 8 [dd] > 8,16 1 770504 9.989768362 4580 Q R 4379504 + 8 [dd] > 8,16 1 770505 9.989768864 4580 M R 4379504 + 8 [dd] > 8,16 1 770506 9.989771157 4580 Q R 4379512 + 8 [dd] > 8,16 1 770507 9.989771662 4580 M R 4379512 + 8 [dd] > 8,16 1 770508 9.989775151 4580 Q R 4379520 + 8 [dd] > 8,16 1 770509 9.989775649 4580 M R 4379520 + 8 [dd] > 8,16 1 770510 9.989781743 4580 Q R 4379528 + 8 [dd] > 8,16 1 770511 9.989782248 4580 M R 4379528 + 8 [dd] > 8,16 1 770512 9.989784532 4580 Q R 4379536 + 8 [dd] > 8,16 1 770513 9.989785033 4580 M R 4379536 + 8 [dd] > 8,16 1 770514 9.989787291 4580 Q R 4379544 + 8 [dd] > 8,16 1 770515 9.989787792 4580 M R 4379544 + 8 [dd] > 8,16 1 770516 9.989790041 4580 Q R 4379552 + 8 [dd] > 8,16 1 770517 9.989790540 4580 M R 4379552 + 8 [dd] > 8,16 1 770518 9.989792607 4580 Q R 4379560 + 8 [dd] > 8,16 1 770519 9.989793104 4580 M R 4379560 + 8 [dd] > 8,16 1 770520 9.989795693 4580 Q R 4379568 + 8 [dd] > 8,16 1 770521 9.989796191 4580 M R 4379568 + 8 [dd] > 8,16 1 770522 9.989798349 4580 Q R 4379576 + 8 [dd] > 8,16 1 770523 9.989798847 4580 M R 4379576 + 8 [dd] > 8,16 1 770524 9.989800877 4580 Q R 4379584 + 8 [dd] > 8,16 1 770525 9.989801581 4580 M R 4379584 + 8 [dd] > 8,16 1 770526 9.989803693 4580 Q R 4379592 + 8 [dd] > 8,16 1 770527 9.989804221 4580 M R 4379592 + 8 [dd] > 8,16 1 770528 9.989806399 4580 Q R 4379600 + 8 [dd] > 8,16 1 770529 9.989806901 4580 M R 4379600 + 8 [dd] > 8,16 1 770530 9.989812437 4580 Q R 4379608 + 8 [dd] > 8,16 1 770531 9.989812938 4580 M R 4379608 + 8 [dd] > 8,16 1 770532 9.989814981 4580 Q R 4379616 + 8 [dd] > 8,16 1 770533 9.989815482 4580 M R 4379616 + 8 [dd] > 8,16 1 770534 9.989818053 4580 Q R 4379624 + 8 [dd] > 8,16 1 770535 9.989818551 4580 M R 4379624 + 8 [dd] > 8,16 1 770536 9.989820795 4580 Q R 4379632 + 8 [dd] > 8,16 1 770537 9.989821296 4580 M R 4379632 + 8 [dd] > 8,16 1 770538 9.989823436 4580 Q R 4379640 + 8 [dd] > 8,16 1 770539 9.989823937 4580 M R 4379640 + 8 [dd] > 8,16 1 770540 9.989929502 4580 Q R 4379648 + 8 [dd] > 8,16 1 770541 9.989930247 4580 M R 4379648 + 8 [dd] > 8,16 1 770542 9.989933189 4580 Q R 4379656 + 8 [dd] > 8,16 1 770543 9.989933705 4580 M R 4379656 + 8 [dd] > 8,16 1 770544 9.989935909 4580 Q R 4379664 + 8 [dd] > 8,16 1 770545 9.989936411 4580 M R 4379664 + 8 [dd] > 8,16 1 770546 9.989938572 4580 Q R 4379672 + 8 [dd] > 8,16 1 770547 9.989939072 4580 M R 4379672 + 8 [dd] > 8,16 1 770548 9.989941272 4580 Q R 4379680 + 8 [dd] > 8,16 1 770549 9.989941770 4580 M R 4379680 + 8 [dd] > 8,16 1 770550 9.989943952 4580 Q R 4379688 + 8 [dd] > 8,16 1 770551 9.989944450 4580 M R 4379688 + 8 [dd] > 8,16 1 770552 9.989946991 4580 Q R 4379696 + 8 [dd] > 8,16 1 770553 9.989947492 4580 M R 4379696 + 8 [dd] > 8,16 1 770554 9.989949671 4580 Q R 4379704 + 8 [dd] > 8,16 1 770555 9.989950172 4580 M R 4379704 + 8 [dd] > 8,16 1 770556 9.989952330 4580 Q R 4379712 + 8 [dd] > 8,16 1 770557 9.989952832 4580 M R 4379712 + 8 [dd] > 8,16 1 770558 9.989958393 4580 Q R 4379720 + 8 [dd] > 8,16 1 770559 9.989958891 4580 M R 4379720 + 8 [dd] > 8,16 1 770560 9.989961097 4580 Q R 4379728 + 8 [dd] > 8,16 1 770561 9.989961598 4580 M R 4379728 + 8 [dd] > 8,16 1 770562 9.989963825 4580 Q R 4379736 + 8 [dd] > 8,16 1 770563 9.989964331 4580 M R 4379736 + 8 [dd] > 8,16 1 770564 9.989966623 4580 Q R 4379744 + 8 [dd] > 8,16 1 770565 9.989967122 4580 M R 4379744 + 8 [dd] > 8,16 1 770566 9.989969303 4580 Q R 4379752 + 8 [dd] > 8,16 1 770567 9.989969800 4580 M R 4379752 + 8 [dd] > 8,16 1 770568 9.989971886 4580 Q R 4379760 + 8 [dd] > 8,16 1 770569 9.989972388 4580 M R 4379760 + 8 [dd] > 8,16 1 770570 9.989974789 4580 Q R 4379768 + 8 [dd] > 8,16 1 770571 9.989975290 4580 M R 4379768 + 8 [dd] > 8,16 1 770572 9.989977480 4580 Q R 4379776 + 8 [dd] > 8,16 1 770573 9.989977981 4580 M R 4379776 + 8 [dd] > 8,16 1 770574 9.989981251 4580 Q R 4379784 + 8 [dd] > 8,16 1 770575 9.989981749 4580 M R 4379784 + 8 [dd] > 8,16 1 770576 9.989984018 4580 Q R 4379792 + 8 [dd] > 8,16 1 770577 9.989984519 4580 M R 4379792 + 8 [dd] > 8,16 1 770578 9.989986711 4580 Q R 4379800 + 8 [dd] > 8,16 1 770579 9.989987222 4580 M R 4379800 + 8 [dd] > 8,16 1 770580 9.989989414 4580 Q R 4379808 + 8 [dd] > 8,16 1 770581 9.989989915 4580 M R 4379808 + 8 [dd] > 8,16 1 770582 9.989992107 4580 Q R 4379816 + 8 [dd] > 8,16 1 770583 9.989992617 4580 M R 4379816 + 8 [dd] > 8,16 1 770584 9.989994776 4580 Q R 4379824 + 8 [dd] > 8,16 1 770585 9.989995276 4580 M R 4379824 + 8 [dd] > 8,16 1 770586 9.990000570 4580 Q R 4379832 + 8 [dd] > 8,16 1 770587 9.990001076 4580 M R 4379832 + 8 [dd] > 8,16 1 770588 9.990003264 4580 Q R 4379840 + 8 [dd] > 8,16 1 770589 9.990003785 4580 M R 4379840 + 8 [dd] > 8,16 1 770590 9.990005849 4580 Q R 4379848 + 8 [dd] > 8,16 1 770591 9.990006354 4580 M R 4379848 + 8 [dd] > 8,16 1 770592 9.990008472 4580 Q R 4379856 + 8 [dd] > 8,16 1 770593 9.990008986 4580 M R 4379856 + 8 [dd] > 8,16 1 770594 9.990011382 4580 Q R 4379864 + 8 [dd] > 8,16 1 770595 9.990011884 4580 M R 4379864 + 8 [dd] > 8,16 1 770596 9.990014066 4580 Q R 4379872 + 8 [dd] > 8,16 1 770597 9.990014568 4580 M R 4379872 + 8 [dd] > 8,16 1 770598 9.990016740 4580 Q R 4379880 + 8 [dd] > 8,16 1 770599 9.990017237 4580 M R 4379880 + 8 [dd] > 8,16 1 770600 9.990019491 4580 Q R 4379888 + 8 [dd] > 8,16 1 770601 9.990020001 4580 M R 4379888 + 8 [dd] > 8,16 1 770602 9.990022222 4580 Q R 4379896 + 8 [dd] > 8,16 1 770603 9.990022723 4580 M R 4379896 + 8 [dd] > 8,16 1 770604 9.990024981 4580 I R 4379392 + 512 ( 303511) [dd] > 8,16 1 0 9.990027734 0 m N cfq4580S / insert_request > 8,16 1 770605 9.990031216 4580 U N [dd] 1 > 8,16 1 0 9.990033792 0 m N cfq4580S / dispatch_insert > 8,16 1 0 9.990037429 0 m N cfq4580S / dispatched a request > 8,16 1 0 9.990039146 0 m N cfq4580S / activate rq, drv=1 > 8,16 1 770606 9.990039807 4580 D R 4379392 + 512 ( 14826) [dd] > 8,16 1 770607 9.991083069 0 C R 4379392 + 512 ( 1043262) [0] > 8,16 1 0 9.991155388 0 m N cfq4580S / complete rqnoidle 0 > 8,16 1 0 9.991158292 0 m N cfq schedule dispatch > 8,16 1 770608 9.991321889 4580 Q R 4379904 + 8 [dd] > 8,16 1 770609 9.991328302 4580 G R 4379904 + 8 [dd] > 8,16 1 770610 9.991330287 4580 P N [dd] > 8,16 1 770611 9.991333624 4580 Q R 4379912 + 8 [dd] > 8,16 1 770612 9.991335576 4580 M R 4379912 + 8 [dd] > 8,16 1 770613 9.991338586 4580 Q R 4379920 + 8 [dd] > 8,16 1 770614 9.991339095 4580 M R 4379920 + 8 [dd] > 8,16 1 770615 9.991341209 4580 Q R 4379928 + 8 [dd] > 8,16 1 770616 9.991341909 4580 M R 4379928 + 8 [dd] > 8,16 1 770617 9.991344208 4580 Q R 4379936 + 8 [dd] > 8,16 1 770618 9.991344707 4580 M R 4379936 + 8 [dd] > 8,16 1 770619 9.991351891 4580 Q R 4379944 + 8 [dd] > 8,16 1 770620 9.991352393 4580 M R 4379944 + 8 [dd] > 8,16 1 770621 9.991354635 4580 Q R 4379952 + 8 [dd] > 8,16 1 770622 9.991355134 4580 M R 4379952 + 8 [dd] > 8,16 1 770623 9.991357304 4580 Q R 4379960 + 8 [dd] > 8,16 1 770624 9.991357809 4580 M R 4379960 + 8 [dd] > 8,16 1 770625 9.991359960 4580 Q R 4379968 + 8 [dd] > 8,16 1 770626 9.991360458 4580 M R 4379968 + 8 [dd] > 8,16 1 770627 9.991362540 4580 Q R 4379976 + 8 [dd] > 8,16 1 770628 9.991363044 4580 M R 4379976 + 8 [dd] > 8,16 1 770629 9.991366526 4580 Q R 4379984 + 8 [dd] > 8,16 1 770630 9.991367025 4580 M R 4379984 + 8 [dd] > 8,16 1 770631 9.991369183 4580 Q R 4379992 + 8 [dd] > 8,16 1 770632 9.991369681 4580 M R 4379992 + 8 [dd] > 8,16 1 770633 9.991371837 4580 Q R 4380000 + 8 [dd] > 8,16 1 770634 9.991372336 4580 M R 4380000 + 8 [dd] > 8,16 1 770635 9.991374518 4580 Q R 4380008 + 8 [dd] > 8,16 1 770636 9.991375031 4580 M R 4380008 + 8 [dd] > 8,16 1 770637 9.991377249 4580 Q R 4380016 + 8 [dd] > 8,16 1 770638 9.991377748 4580 M R 4380016 + 8 [dd] > 8,16 1 770639 9.991379891 4580 Q R 4380024 + 8 [dd] > 8,16 1 770640 9.991380390 4580 M R 4380024 + 8 [dd] > 8,16 1 770641 9.991382585 4580 Q R 4380032 + 8 [dd] > 8,16 1 770642 9.991383084 4580 M R 4380032 + 8 [dd] > 8,16 1 770643 9.991385207 4580 Q R 4380040 + 8 [dd] > 8,16 1 770644 9.991385704 4580 M R 4380040 + 8 [dd] > 8,16 1 770645 9.991387897 4580 Q R 4380048 + 8 [dd] > 8,16 1 770646 9.991388396 4580 M R 4380048 + 8 [dd] > 8,16 1 770647 9.991393804 4580 Q R 4380056 + 8 [dd] > 8,16 1 770648 9.991394303 4580 M R 4380056 + 8 [dd] > 8,16 1 770649 9.991396440 4580 Q R 4380064 + 8 [dd] > 8,16 1 770650 9.991396938 4580 M R 4380064 + 8 [dd] > 8,16 1 770651 9.991398996 4580 Q R 4380072 + 8 [dd] > 8,16 1 770652 9.991399500 4580 M R 4380072 + 8 [dd] > 8,16 1 770653 9.991401608 4580 Q R 4380080 + 8 [dd] > 8,16 1 770654 9.991402107 4580 M R 4380080 + 8 [dd] > 8,16 1 770655 9.991404535 4580 Q R 4380088 + 8 [dd] > 8,16 1 770656 9.991405034 4580 M R 4380088 + 8 [dd] > 8,16 1 770657 9.991407171 4580 Q R 4380096 + 8 [dd] > 8,16 1 770658 9.991407671 4580 M R 4380096 + 8 [dd] > 8,16 1 770659 9.991409829 4580 Q R 4380104 + 8 [dd] > 8,16 1 770660 9.991410339 4580 M R 4380104 + 8 [dd] > 8,16 1 770661 9.991416346 4580 Q R 4380112 + 8 [dd] > 8,16 1 770662 9.991416853 4580 M R 4380112 + 8 [dd] > 8,16 1 770663 9.991419016 4580 Q R 4380120 + 8 [dd] > 8,16 1 770664 9.991419515 4580 M R 4380120 + 8 [dd] > 8,16 1 770665 9.991421984 4580 Q R 4380128 + 8 [dd] > 8,16 1 770666 9.991422483 4580 M R 4380128 + 8 [dd] > 8,16 1 770667 9.991424780 4580 Q R 4380136 + 8 [dd] > 8,16 1 770668 9.991425284 4580 M R 4380136 + 8 [dd] > 8,16 1 770669 9.991427479 4580 Q R 4380144 + 8 [dd] > 8,16 1 770670 9.991427978 4580 M R 4380144 + 8 [dd] > 8,16 1 770671 9.991430076 4580 Q R 4380152 + 8 [dd] > 8,16 1 770672 9.991430575 4580 M R 4380152 + 8 [dd] > 8,16 1 770673 9.991536122 4580 Q R 4380160 + 8 [dd] > 8,16 1 770674 9.991536827 4580 M R 4380160 + 8 [dd] > 8,16 1 770675 9.991543663 4580 Q R 4380168 + 8 [dd] > 8,16 1 770676 9.991544181 4580 M R 4380168 + 8 [dd] > 8,16 1 770677 9.991546458 4580 Q R 4380176 + 8 [dd] > 8,16 1 770678 9.991546957 4580 M R 4380176 + 8 [dd] > 8,16 1 770679 9.991549153 4580 Q R 4380184 + 8 [dd] > 8,16 1 770680 9.991549652 4580 M R 4380184 + 8 [dd] > 8,16 1 770681 9.991551902 4580 Q R 4380192 + 8 [dd] > 8,16 1 770682 9.991552400 4580 M R 4380192 + 8 [dd] > 8,16 1 770683 9.991554621 4580 Q R 4380200 + 8 [dd] > 8,16 1 770684 9.991555113 4580 M R 4380200 + 8 [dd] > 8,16 1 770685 9.991557325 4580 Q R 4380208 + 8 [dd] > 8,16 1 770686 9.991557824 4580 M R 4380208 + 8 [dd] > 8,16 1 770687 9.991560192 4580 Q R 4380216 + 8 [dd] > 8,16 1 770688 9.991560732 4580 M R 4380216 + 8 [dd] > 8,16 1 770689 9.991562806 4580 Q R 4380224 + 8 [dd] > 8,16 1 770690 9.991563305 4580 M R 4380224 + 8 [dd] > 8,16 1 770691 9.991565370 4580 Q R 4380232 + 8 [dd] > 8,16 1 770692 9.991565874 4580 M R 4380232 + 8 [dd] > 8,16 1 770693 9.991567969 4580 Q R 4380240 + 8 [dd] > 8,16 1 770694 9.991568468 4580 M R 4380240 + 8 [dd] > 8,16 1 770695 9.991570671 4580 Q R 4380248 + 8 [dd] > 8,16 1 770696 9.991571170 4580 M R 4380248 + 8 [dd] > 8,16 1 770697 9.991573285 4580 Q R 4380256 + 8 [dd] > 8,16 1 770698 9.991573783 4580 M R 4380256 + 8 [dd] > 8,16 1 770699 9.991575855 4580 Q R 4380264 + 8 [dd] > 8,16 1 770700 9.991576358 4580 M R 4380264 + 8 [dd] > 8,16 1 770701 9.991578551 4580 Q R 4380272 + 8 [dd] > 8,16 1 770702 9.991579050 4580 M R 4380272 + 8 [dd] > 8,16 1 770703 9.991584418 4580 Q R 4380280 + 8 [dd] > 8,16 1 770704 9.991584916 4580 M R 4380280 + 8 [dd] > 8,16 1 770705 9.991587004 4580 Q R 4380288 + 8 [dd] > 8,16 1 770706 9.991587503 4580 M R 4380288 + 8 [dd] > 8,16 1 770707 9.991589615 4580 Q R 4380296 + 8 [dd] > 8,16 1 770708 9.991590128 4580 M R 4380296 + 8 [dd] > 8,16 1 770709 9.991592390 4580 Q R 4380304 + 8 [dd] > 8,16 1 770710 9.991592907 4580 M R 4380304 + 8 [dd] > 8,16 1 770711 9.991594957 4580 Q R 4380312 + 8 [dd] > 8,16 1 770712 9.991595456 4580 M R 4380312 + 8 [dd] > 8,16 1 770713 9.991597599 4580 Q R 4380320 + 8 [dd] > 8,16 1 770714 9.991598098 4580 M R 4380320 + 8 [dd] > 8,16 1 770715 9.991600165 4580 Q R 4380328 + 8 [dd] > 8,16 1 770716 9.991600669 4580 M R 4380328 + 8 [dd] > 8,16 1 770717 9.991602915 4580 Q R 4380336 + 8 [dd] > 8,16 1 770718 9.991603413 4580 M R 4380336 + 8 [dd] > 8,16 1 770719 9.991605563 4580 Q R 4380344 + 8 [dd] > 8,16 1 770720 9.991606061 4580 M R 4380344 + 8 [dd] > 8,16 1 770721 9.991608110 4580 Q R 4380352 + 8 [dd] > 8,16 1 770722 9.991608609 4580 M R 4380352 + 8 [dd] > 8,16 1 770723 9.991610674 4580 Q R 4380360 + 8 [dd] > 8,16 1 770724 9.991611178 4580 M R 4380360 + 8 [dd] > 8,16 1 770725 9.991613249 4580 Q R 4380368 + 8 [dd] > 8,16 1 770726 9.991613748 4580 M R 4380368 + 8 [dd] > 8,16 1 770727 9.991616309 4580 Q R 4380376 + 8 [dd] > 8,16 1 770728 9.991616808 4580 M R 4380376 + 8 [dd] > 8,16 1 770729 9.991619009 4580 Q R 4380384 + 8 [dd] > 8,16 1 770730 9.991619507 4580 M R 4380384 + 8 [dd] > 8,16 1 770731 9.991624896 4580 Q R 4380392 + 8 [dd] > 8,16 1 770732 9.991625401 4580 M R 4380392 + 8 [dd] > 8,16 1 770733 9.991627620 4580 Q R 4380400 + 8 [dd] > 8,16 1 770734 9.991628119 4580 M R 4380400 + 8 [dd] > 8,16 1 770735 9.991630192 4580 Q R 4380408 + 8 [dd] > 8,16 1 770736 9.991630690 4580 M R 4380408 + 8 [dd] > 8,16 1 770737 9.991633001 4580 I R 4379904 + 512 ( 304699) [dd] > 8,16 1 0 9.991635738 0 m N cfq4580S / insert_request > 8,16 1 770738 9.991638988 4580 U N [dd] 1 > 8,16 1 0 9.991641665 0 m N cfq4580S / dispatch_insert > 8,16 1 0 9.991645149 0 m N cfq4580S / dispatched a request > 8,16 1 0 9.991646825 0 m N cfq4580S / activate rq, drv=1 > 8,16 1 770739 9.991647434 4580 D R 4379904 + 512 ( 14433) [dd] > 8,16 1 770740 9.992693317 0 C R 4379904 + 512 ( 1045883) [0] > 8,16 1 0 9.992766012 0 m N cfq4580S / complete rqnoidle 0 > 8,16 1 0 9.992768895 0 m N cfq schedule dispatch > 8,16 1 770741 9.992932816 4580 Q R 4380416 + 8 [dd] > 8,16 1 770742 9.992939152 4580 G R 4380416 + 8 [dd] > 8,16 1 770743 9.992941241 4580 P N [dd] > 8,16 1 770744 9.992944673 4580 Q R 4380424 + 8 [dd] > 8,16 1 770745 9.992946529 4580 M R 4380424 + 8 [dd] > 8,16 1 770746 9.992949525 4580 Q R 4380432 + 8 [dd] > 8,16 1 770747 9.992950035 4580 M R 4380432 + 8 [dd] > 8,16 1 770748 9.992952218 4580 Q R 4380440 + 8 [dd] > 8,16 1 770749 9.992952765 4580 M R 4380440 + 8 [dd] > 8,16 1 770750 9.992954867 4580 Q R 4380448 + 8 [dd] > 8,16 1 770751 9.992955366 4580 M R 4380448 + 8 [dd] > 8,16 1 770752 9.992957451 4580 Q R 4380456 + 8 [dd] > 8,16 1 770753 9.992957956 4580 M R 4380456 + 8 [dd] > 8,16 1 770754 9.992960073 4580 Q R 4380464 + 8 [dd] > 8,16 1 770755 9.992960572 4580 M R 4380464 + 8 [dd] > 8,16 1 770756 9.992962600 4580 Q R 4380472 + 8 [dd] > 8,16 1 770757 9.992963104 4580 M R 4380472 + 8 [dd] > 8,16 1 770758 9.992965247 4580 Q R 4380480 + 8 [dd] > 8,16 1 770759 9.992965753 4580 M R 4380480 + 8 [dd] > 8,16 1 770760 9.992967787 4580 Q R 4380488 + 8 [dd] > 8,16 1 770761 9.992968284 4580 M R 4380488 + 8 [dd] > 8,16 1 770762 9.992970602 4580 Q R 4380496 + 8 [dd] > 8,16 1 770763 9.992971101 4580 M R 4380496 + 8 [dd] > 8,16 1 770764 9.992978036 4580 Q R 4380504 + 8 [dd] > 8,16 1 770765 9.992978575 4580 M R 4380504 + 8 [dd] > 8,16 1 770766 9.992980800 4580 Q R 4380512 + 8 [dd] > 8,16 1 770767 9.992981312 4580 M R 4380512 + 8 [dd] > 8,16 1 770768 9.992983463 4580 Q R 4380520 + 8 [dd] > 8,16 1 770769 9.992983976 4580 M R 4380520 + 8 [dd] > 8,16 1 770770 9.992986141 4580 Q R 4380528 + 8 [dd] > 8,16 1 770771 9.992986652 4580 M R 4380528 + 8 [dd] > 8,16 1 770772 9.992988835 4580 Q R 4380536 + 8 [dd] > 8,16 1 770773 9.992989338 4580 M R 4380536 + 8 [dd] > 8,16 1 770774 9.992991462 4580 Q R 4380544 + 8 [dd] > 8,16 1 770775 9.992991961 4580 M R 4380544 + 8 [dd] > 8,16 1 770776 9.992994020 4580 Q R 4380552 + 8 [dd] > 8,16 1 770777 9.992994519 4580 M R 4380552 + 8 [dd] > 8,16 1 770778 9.992996632 4580 Q R 4380560 + 8 [dd] > 8,16 1 770779 9.992997135 4580 M R 4380560 + 8 [dd] > 8,16 1 770780 9.992999236 4580 Q R 4380568 + 8 [dd] > 8,16 1 770781 9.992999742 4580 M R 4380568 + 8 [dd] > 8,16 1 770782 9.993001890 4580 Q R 4380576 + 8 [dd] > 8,16 1 770783 9.993002389 4580 M R 4380576 + 8 [dd] > 8,16 1 770784 9.993004438 4580 Q R 4380584 + 8 [dd] > 8,16 1 770785 9.993004937 4580 M R 4380584 + 8 [dd] > 8,16 1 770786 9.993007255 4580 Q R 4380592 + 8 [dd] > 8,16 1 770787 9.993007754 4580 M R 4380592 + 8 [dd] > 8,16 1 770788 9.993009828 4580 Q R 4380600 + 8 [dd] > 8,16 1 770789 9.993010351 4580 M R 4380600 + 8 [dd] > 8,16 1 770790 9.993012459 4580 Q R 4380608 + 8 [dd] > 8,16 1 770791 9.993012958 4580 M R 4380608 + 8 [dd] > 8,16 1 770792 9.993019700 4580 Q R 4380616 + 8 [dd] > 8,16 1 770793 9.993020199 4580 M R 4380616 + 8 [dd] > 8,16 1 770794 9.993022401 4580 Q R 4380624 + 8 [dd] > 8,16 1 770795 9.993022909 4580 M R 4380624 + 8 [dd] > 8,16 1 770796 9.993024945 4580 Q R 4380632 + 8 [dd] > 8,16 1 770797 9.993025449 4580 M R 4380632 + 8 [dd] > 8,16 1 770798 9.993027501 4580 Q R 4380640 + 8 [dd] > 8,16 1 770799 9.993027999 4580 M R 4380640 + 8 [dd] > 8,16 1 770800 9.993030163 4580 Q R 4380648 + 8 [dd] > 8,16 1 770801 9.993030666 4580 M R 4380648 + 8 [dd] > 8,16 1 770802 9.993033040 4580 Q R 4380656 + 8 [dd] > 8,16 1 770803 9.993033538 4580 M R 4380656 + 8 [dd] > 8,16 1 770804 9.993035704 4580 Q R 4380664 + 8 [dd] > 8,16 1 770805 9.993036205 4580 M R 4380664 + 8 [dd] > 8,16 1 770806 9.993143012 4580 Q R 4380672 + 8 [dd] > 8,16 1 770807 9.993143745 4580 M R 4380672 + 8 [dd] > 8,16 1 770808 9.993147049 4580 Q R 4380680 + 8 [dd] > 8,16 1 770809 9.993147550 4580 M R 4380680 + 8 [dd] > 8,16 1 770810 9.993149983 4580 Q R 4380688 + 8 [dd] > 8,16 1 770811 9.993150482 4580 M R 4380688 + 8 [dd] > 8,16 1 770812 9.993156384 4580 Q R 4380696 + 8 [dd] > 8,16 1 770813 9.993156894 4580 M R 4380696 + 8 [dd] > 8,16 1 770814 9.993159096 4580 Q R 4380704 + 8 [dd] > 8,16 1 770815 9.993159606 4580 M R 4380704 + 8 [dd] > 8,16 1 770816 9.993161853 4580 Q R 4380712 + 8 [dd] > 8,16 1 770817 9.993162351 4580 M R 4380712 + 8 [dd] > 8,16 1 770818 9.993164628 4580 Q R 4380720 + 8 [dd] > 8,16 1 770819 9.993165127 4580 M R 4380720 + 8 [dd] > 8,16 1 770820 9.993170698 4580 Q R 4380728 + 8 [dd] > 8,16 1 770821 9.993171200 4580 M R 4380728 + 8 [dd] > 8,16 1 770822 9.993173392 4580 Q R 4380736 + 8 [dd] > 8,16 1 770823 9.993173894 4580 M R 4380736 + 8 [dd] > 8,16 1 770824 9.993176081 4580 Q R 4380744 + 8 [dd] > 8,16 1 770825 9.993176579 4580 M R 4380744 + 8 [dd] > 8,16 1 770826 9.993178841 4580 Q R 4380752 + 8 [dd] > 8,16 1 770827 9.993179340 4580 M R 4380752 + 8 [dd] > 8,16 1 770828 9.993181445 4580 Q R 4380760 + 8 [dd] > 8,16 1 770829 9.993181949 4580 M R 4380760 + 8 [dd] > 8,16 1 770830 9.993184121 4580 Q R 4380768 + 8 [dd] > 8,16 1 770831 9.993184619 4580 M R 4380768 + 8 [dd] > 8,16 1 770832 9.993186741 4580 Q R 4380776 + 8 [dd] > 8,16 1 770833 9.993187239 4580 M R 4380776 + 8 [dd] > 8,16 1 770834 9.993189324 4580 Q R 4380784 + 8 [dd] > 8,16 1 770835 9.993189822 4580 M R 4380784 + 8 [dd] > 8,16 1 770836 9.993191998 4580 Q R 4380792 + 8 [dd] > 8,16 1 770837 9.993192502 4580 M R 4380792 + 8 [dd] > 8,16 1 770838 9.993194798 4580 Q R 4380800 + 8 [dd] > 8,16 1 770839 9.993195297 4580 M R 4380800 + 8 [dd] > 8,16 1 770840 9.993197858 4580 Q R 4380808 + 8 [dd] > 8,16 1 770841 9.993198358 4580 M R 4380808 + 8 [dd] > 8,16 1 770842 9.993201427 4580 Q R 4380816 + 8 [dd] > 8,16 1 770843 9.993201926 4580 M R 4380816 + 8 [dd] > 8,16 1 770844 9.993204159 4580 Q R 4380824 + 8 [dd] > 8,16 1 770845 9.993204658 4580 M R 4380824 + 8 [dd] > 8,16 1 770846 9.993206840 4580 Q R 4380832 + 8 [dd] > 8,16 1 770847 9.993207338 4580 M R 4380832 + 8 [dd] > 8,16 1 770848 9.993212789 4580 Q R 4380840 + 8 [dd] > 8,16 1 770849 9.993213321 4580 M R 4380840 + 8 [dd] > 8,16 1 770850 9.993215482 4580 Q R 4380848 + 8 [dd] > 8,16 1 770851 9.993215986 4580 M R 4380848 + 8 [dd] > 8,16 1 770852 9.993218173 4580 Q R 4380856 + 8 [dd] > 8,16 1 770853 9.993218673 4580 M R 4380856 + 8 [dd] > 8,16 1 770854 9.993220839 4580 Q R 4380864 + 8 [dd] > 8,16 1 770855 9.993221338 4580 M R 4380864 + 8 [dd] > 8,16 1 770856 9.993223551 4580 Q R 4380872 + 8 [dd] > 8,16 1 770857 9.993224051 4580 M R 4380872 + 8 [dd] > 8,16 1 770858 9.993226133 4580 Q R 4380880 + 8 [dd] > 8,16 1 770859 9.993226632 4580 M R 4380880 + 8 [dd] > 8,16 1 770860 9.993228840 4580 Q R 4380888 + 8 [dd] > 8,16 1 770861 9.993229342 4580 M R 4380888 + 8 [dd] > 8,16 1 770862 9.993231460 4580 Q R 4380896 + 8 [dd] > 8,16 1 770863 9.993231958 4580 M R 4380896 + 8 [dd] > 8,16 1 770864 9.993234011 4580 Q R 4380904 + 8 [dd] > 8,16 1 770865 9.993234513 4580 M R 4380904 + 8 [dd] > 8,16 1 770866 9.993236661 4580 Q R 4380912 + 8 [dd] > 8,16 1 770867 9.993237160 4580 M R 4380912 + 8 [dd] > 8,16 1 770868 9.993239233 4580 Q R 4380920 + 8 [dd] > 8,16 1 770869 9.993239736 4580 M R 4380920 + 8 [dd] > 8,16 1 770870 9.993241912 4580 I R 4380416 + 512 ( 302760) [dd] > 8,16 1 0 9.993244448 0 m N cfq4580S / insert_request > 8,16 1 770871 9.993247771 4580 U N [dd] 1 > 8,16 1 0 9.993250443 0 m N cfq4580S / dispatch_insert > 8,16 1 0 9.993254093 0 m N cfq4580S / dispatched a request > 8,16 1 0 9.993255838 0 m N cfq4580S / activate rq, drv=1 > 8,16 1 770872 9.993256451 4580 D R 4380416 + 512 ( 14539) [dd] > 8,16 1 770873 9.994299156 0 C R 4380416 + 512 ( 1042705) [0] > 8,16 1 0 9.994371989 0 m N cfq4580S / complete rqnoidle 0 > 8,16 1 0 9.994375064 0 m N cfq schedule dispatch > 8,16 1 770874 9.994536199 4580 Q R 4380928 + 8 [dd] > 8,16 1 770875 9.994542473 4580 G R 4380928 + 8 [dd] > 8,16 1 770876 9.994544517 4580 P N [dd] > 8,16 1 770877 9.994548013 4580 Q R 4380936 + 8 [dd] > 8,16 1 770878 9.994550069 4580 M R 4380936 + 8 [dd] > 8,16 1 770879 9.994553102 4580 Q R 4380944 + 8 [dd] > 8,16 1 770880 9.994553612 4580 M R 4380944 + 8 [dd] > 8,16 1 770881 9.994560750 4580 Q R 4380952 + 8 [dd] > 8,16 1 770882 9.994561294 4580 M R 4380952 + 8 [dd] > 8,16 1 770883 9.994563660 4580 Q R 4380960 + 8 [dd] > 8,16 1 770884 9.994564176 4580 M R 4380960 + 8 [dd] > 8,16 1 770885 9.994566309 4580 Q R 4380968 + 8 [dd] > 8,16 1 770886 9.994566819 4580 M R 4380968 + 8 [dd] > 8,16 1 770887 9.994568963 4580 Q R 4380976 + 8 [dd] > 8,16 1 770888 9.994569474 4580 M R 4380976 + 8 [dd] > 8,16 1 770889 9.994571592 4580 Q R 4380984 + 8 [dd] > 8,16 1 770890 9.994572104 4580 M R 4380984 + 8 [dd] > 8,16 1 770891 9.994574265 4580 Q R 4380992 + 8 [dd] > 8,16 1 770892 9.994574776 4580 M R 4380992 + 8 [dd] > 8,16 1 770893 9.994576861 4580 Q R 4381000 + 8 [dd] > 8,16 1 770894 9.994577360 4580 M R 4381000 + 8 [dd] > 8,16 1 770895 9.994579582 4580 Q R 4381008 + 8 [dd] > 8,16 1 770896 9.994580081 4580 M R 4381008 + 8 [dd] > 8,16 1 770897 9.994582161 4580 Q R 4381016 + 8 [dd] > 8,16 1 770898 9.994582661 4580 M R 4381016 + 8 [dd] > 8,16 1 770899 9.994584717 4580 Q R 4381024 + 8 [dd] > 8,16 1 770900 9.994585216 4580 M R 4381024 + 8 [dd] > 8,16 1 770901 9.994587324 4580 Q R 4381032 + 8 [dd] > 8,16 1 770902 9.994587822 4580 M R 4381032 + 8 [dd] > 8,16 1 770903 9.994589915 4580 Q R 4381040 + 8 [dd] > 8,16 1 770904 9.994590414 4580 M R 4381040 + 8 [dd] > 8,16 1 770905 9.994594071 4580 Q R 4381048 + 8 [dd] > 8,16 1 770906 9.994594575 4580 M R 4381048 + 8 [dd] > 8,16 1 770907 9.994596896 4580 Q R 4381056 + 8 [dd] > 8,16 1 770908 9.994597407 4580 M R 4381056 + 8 [dd] > 8,16 1 770909 9.994602878 4580 Q R 4381064 + 8 [dd] > 8,16 1 770910 9.994603394 4580 M R 4381064 + 8 [dd] > 8,16 1 770911 9.994605653 4580 Q R 4381072 + 8 [dd] > 8,16 1 770912 9.994606164 4580 M R 4381072 + 8 [dd] > 8,16 1 770913 9.994608528 4580 Q R 4381080 + 8 [dd] > 8,16 1 770914 9.994609040 4580 M R 4381080 + 8 [dd] > 8,16 1 770915 9.994611239 4580 Q R 4381088 + 8 [dd] > 8,16 1 770916 9.994611751 4580 M R 4381088 + 8 [dd] > 8,16 1 770917 9.994613936 4580 Q R 4381096 + 8 [dd] > 8,16 1 770918 9.994614447 4580 M R 4381096 + 8 [dd] > 8,16 1 770919 9.994616718 4580 Q R 4381104 + 8 [dd] > 8,16 1 770920 9.994617229 4580 M R 4381104 + 8 [dd] > 8,16 1 770921 9.994619394 4580 Q R 4381112 + 8 [dd] > 8,16 1 770922 9.994619905 4580 M R 4381112 + 8 [dd] > 8,16 1 770923 9.994622152 4580 Q R 4381120 + 8 [dd] > 8,16 1 770924 9.994622663 4580 M R 4381120 + 8 [dd] > 8,16 1 770925 9.994624765 4580 Q R 4381128 + 8 [dd] > 8,16 1 770926 9.994625294 4580 M R 4381128 + 8 [dd] > 8,16 1 770927 9.994627643 4580 Q R 4381136 + 8 [dd] > 8,16 1 770928 9.994628172 4580 M R 4381136 + 8 [dd] > 8,16 1 770929 9.994630638 4580 Q R 4381144 + 8 [dd] > 8,16 1 770930 9.994631141 4580 M R 4381144 + 8 [dd] > 8,16 1 770931 9.994633374 4580 Q R 4381152 + 8 [dd] > 8,16 1 770932 9.994633885 4580 M R 4381152 + 8 [dd] > 8,16 1 770933 9.994636086 4580 Q R 4381160 + 8 [dd] > 8,16 1 770934 9.994636592 4580 M R 4381160 + 8 [dd] > 8,16 1 770935 9.994638779 4580 Q R 4381168 + 8 [dd] > 8,16 1 770936 9.994639291 4580 M R 4381168 + 8 [dd] > 8,16 1 770937 9.994644640 4580 Q R 4381176 + 8 [dd] > 8,16 1 770938 9.994645153 4580 M R 4381176 + 8 [dd] > 8,16 1 770939 9.994750984 4580 Q R 4381184 + 8 [dd] > 8,16 1 770940 9.994751713 4580 M R 4381184 + 8 [dd] > 8,16 1 770941 9.994754423 4580 Q R 4381192 + 8 [dd] > 8,16 1 770942 9.994754936 4580 M R 4381192 + 8 [dd] > 8,16 1 770943 9.994757220 4580 Q R 4381200 + 8 [dd] > 8,16 1 770944 9.994757719 4580 M R 4381200 + 8 [dd] > 8,16 1 770945 9.994759967 4580 Q R 4381208 + 8 [dd] > 8,16 1 770946 9.994760506 4580 M R 4381208 + 8 [dd] > 8,16 1 770947 9.994762743 4580 Q R 4381216 + 8 [dd] > 8,16 1 770948 9.994763242 4580 M R 4381216 + 8 [dd] > 8,16 1 770949 9.994765425 4580 Q R 4381224 + 8 [dd] > 8,16 1 770950 9.994765922 4580 M R 4381224 + 8 [dd] > 8,16 1 770951 9.994768201 4580 Q R 4381232 + 8 [dd] > 8,16 1 770952 9.994768700 4580 M R 4381232 + 8 [dd] > 8,16 1 770953 9.994770798 4580 Q R 4381240 + 8 [dd] > 8,16 1 770954 9.994771298 4580 M R 4381240 + 8 [dd] > 8,16 1 770955 9.994773393 4580 Q R 4381248 + 8 [dd] > 8,16 1 770956 9.994773892 4580 M R 4381248 + 8 [dd] > 8,16 1 770957 9.994776015 4580 Q R 4381256 + 8 [dd] > 8,16 1 770958 9.994776513 4580 M R 4381256 + 8 [dd] > 8,16 1 770959 9.994778608 4580 Q R 4381264 + 8 [dd] > 8,16 1 770960 9.994779107 4580 M R 4381264 + 8 [dd] > 8,16 1 770961 9.994781210 4580 Q R 4381272 + 8 [dd] > 8,16 1 770962 9.994781710 4580 M R 4381272 + 8 [dd] > 8,16 1 770963 9.994787731 4580 Q R 4381280 + 8 [dd] > 8,16 1 770964 9.994788230 4580 M R 4381280 + 8 [dd] > 8,16 1 770965 9.994793675 4580 Q R 4381288 + 8 [dd] > 8,16 1 770966 9.994794174 4580 M R 4381288 + 8 [dd] > 8,16 1 770967 9.994796307 4580 Q R 4381296 + 8 [dd] > 8,16 1 770968 9.994796806 4580 M R 4381296 + 8 [dd] > 8,16 1 770969 9.994799419 4580 Q R 4381304 + 8 [dd] > 8,16 1 770970 9.994799919 4580 M R 4381304 + 8 [dd] > 8,16 1 770971 9.994802209 4580 Q R 4381312 + 8 [dd] > 8,16 1 770972 9.994802707 4580 M R 4381312 + 8 [dd] > 8,16 1 770973 9.994805174 4580 Q R 4381320 + 8 [dd] > 8,16 1 770974 9.994805672 4580 M R 4381320 + 8 [dd] > 8,16 1 770975 9.994807936 4580 Q R 4381328 + 8 [dd] > 8,16 1 770976 9.994808440 4580 M R 4381328 + 8 [dd] > 8,16 1 770977 9.994810715 4580 Q R 4381336 + 8 [dd] > 8,16 1 770978 9.994811242 4580 M R 4381336 + 8 [dd] > 8,16 1 770979 9.994813464 4580 Q R 4381344 + 8 [dd] > 8,16 1 770980 9.994813963 4580 M R 4381344 + 8 [dd] > 8,16 1 770981 9.994816205 4580 Q R 4381352 + 8 [dd] > 8,16 1 770982 9.994816776 4580 M R 4381352 + 8 [dd] > 8,16 1 770983 9.994819127 4580 Q R 4381360 + 8 [dd] > 8,16 1 770984 9.994819628 4580 M R 4381360 + 8 [dd] > 8,16 1 770985 9.994821810 4580 Q R 4381368 + 8 [dd] > 8,16 1 770986 9.994822315 4580 M R 4381368 + 8 [dd] > 8,16 1 770987 9.994824377 4580 Q R 4381376 + 8 [dd] > 8,16 1 770988 9.994824876 4580 M R 4381376 + 8 [dd] > 8,16 1 770989 9.994827067 4580 Q R 4381384 + 8 [dd] > 8,16 1 770990 9.994827565 4580 M R 4381384 + 8 [dd] > 8,16 1 770991 9.994829760 4580 Q R 4381392 + 8 [dd] > 8,16 1 770992 9.994830259 4580 M R 4381392 + 8 [dd] > 8,16 1 770993 9.994835522 4580 Q R 4381400 + 8 [dd] > 8,16 1 770994 9.994836021 4580 M R 4381400 + 8 [dd] > 8,16 1 770995 9.994838181 4580 Q R 4381408 + 8 [dd] > 8,16 1 770996 9.994838680 4580 M R 4381408 + 8 [dd] > 8,16 1 770997 9.994840857 4580 Q R 4381416 + 8 [dd] > 8,16 1 770998 9.994841369 4580 M R 4381416 + 8 [dd] > 8,16 1 770999 9.994843455 4580 Q R 4381424 + 8 [dd] > 8,16 1 771000 9.994843954 4580 M R 4381424 + 8 [dd] > 8,16 1 771001 9.994846149 4580 Q R 4381432 + 8 [dd] > 8,16 1 771002 9.994846662 4580 M R 4381432 + 8 [dd] > 8,16 1 771003 9.994849336 4580 I R 4380928 + 512 ( 306863) [dd] > 8,16 1 0 9.994852010 0 m N cfq4580S / insert_request > 8,16 1 771004 9.994855251 4580 U N [dd] 1 > 8,16 1 0 9.994857783 0 m N cfq4580S / dispatch_insert > 8,16 1 0 9.994861277 0 m N cfq4580S / dispatched a request > 8,16 1 0 9.994863066 0 m N cfq4580S / activate rq, drv=1 > 8,16 1 771005 9.994863680 4580 D R 4380928 + 512 ( 14344) [dd] > 8,16 1 771006 9.995909291 0 C R 4380928 + 512 ( 1045611) [0] > 8,16 1 0 9.995983219 0 m N cfq4580S / complete rqnoidle 0 > 8,16 1 0 9.995986144 0 m N cfq schedule dispatch > 8,16 1 771007 9.996149764 4580 Q R 4381440 + 8 [dd] > 8,16 1 771008 9.996155872 4580 G R 4381440 + 8 [dd] > 8,16 1 771009 9.996157994 4580 P N [dd] > 8,16 1 771010 9.996161548 4580 Q R 4381448 + 8 [dd] > 8,16 1 771011 9.996163428 4580 M R 4381448 + 8 [dd] > 8,16 1 771012 9.996166608 4580 Q R 4381456 + 8 [dd] > 8,16 1 771013 9.996167115 4580 M R 4381456 + 8 [dd] > 8,16 1 771014 9.996169178 4580 Q R 4381464 + 8 [dd] > 8,16 1 771015 9.996169681 4580 M R 4381464 + 8 [dd] > 8,16 1 771016 9.996171779 4580 Q R 4381472 + 8 [dd] > 8,16 1 771017 9.996172277 4580 M R 4381472 + 8 [dd] > 8,16 1 771018 9.996174323 4580 Q R 4381480 + 8 [dd] > 8,16 1 771019 9.996174822 4580 M R 4381480 + 8 [dd] > 8,16 1 771020 9.996176942 4580 Q R 4381488 + 8 [dd] > 8,16 1 771021 9.996177446 4580 M R 4381488 + 8 [dd] > 8,16 1 771022 9.996179520 4580 Q R 4381496 + 8 [dd] > 8,16 1 771023 9.996180022 4580 M R 4381496 + 8 [dd] > 8,16 1 771024 9.996182075 4580 Q R 4381504 + 8 [dd] > 8,16 1 771025 9.996182574 4580 M R 4381504 + 8 [dd] > 8,16 1 771026 9.996189624 4580 Q R 4381512 + 8 [dd] > 8,16 1 771027 9.996190124 4580 M R 4381512 + 8 [dd] > 8,16 1 771028 9.996192318 4580 Q R 4381520 + 8 [dd] > 8,16 1 771029 9.996192819 4580 M R 4381520 + 8 [dd] > 8,16 1 771030 9.996194916 4580 Q R 4381528 + 8 [dd] > 8,16 1 771031 9.996195415 4580 M R 4381528 + 8 [dd] > 8,16 1 771032 9.996197792 4580 Q R 4381536 + 8 [dd] > 8,16 1 771033 9.996198291 4580 M R 4381536 + 8 [dd] > 8,16 1 771034 9.996200401 4580 Q R 4381544 + 8 [dd] > 8,16 1 771035 9.996200899 4580 M R 4381544 + 8 [dd] > 8,16 1 771036 9.996202991 4580 Q R 4381552 + 8 [dd] > 8,16 1 771037 9.996203495 4580 M R 4381552 + 8 [dd] > 8,16 1 771038 9.996205593 4580 Q R 4381560 + 8 [dd] > 8,16 1 771039 9.996206095 4580 M R 4381560 + 8 [dd] > 8,16 1 771040 9.996208179 4580 Q R 4381568 + 8 [dd] > 8,16 1 771041 9.996208678 4580 M R 4381568 + 8 [dd] > 8,16 1 771042 9.996210778 4580 Q R 4381576 + 8 [dd] > 8,16 1 771043 9.996211285 4580 M R 4381576 + 8 [dd] > 8,16 1 771044 9.996213428 4580 Q R 4381584 + 8 [dd] > 8,16 1 771045 9.996213932 4580 M R 4381584 + 8 [dd] > 8,16 1 771046 9.996217450 4580 Q R 4381592 + 8 [dd] > 8,16 1 771047 9.996217952 4580 M R 4381592 + 8 [dd] > 8,16 1 771048 9.996220131 4580 Q R 4381600 + 8 [dd] > 8,16 1 771049 9.996220644 4580 M R 4381600 + 8 [dd] > 8,16 1 771050 9.996222799 4580 Q R 4381608 + 8 [dd] > 8,16 1 771051 9.996223302 4580 M R 4381608 + 8 [dd] > 8,16 1 771052 9.996225537 4580 Q R 4381616 + 8 [dd] > 8,16 1 771053 9.996226039 4580 M R 4381616 + 8 [dd] > 8,16 1 771054 9.996231412 4580 Q R 4381624 + 8 [dd] > 8,16 1 771055 9.996231912 4580 M R 4381624 + 8 [dd] > 8,16 1 771056 9.996234006 4580 Q R 4381632 + 8 [dd] > 8,16 1 771057 9.996234505 4580 M R 4381632 + 8 [dd] > 8,16 1 771058 9.996236661 4580 Q R 4381640 + 8 [dd] > 8,16 1 771059 9.996237160 4580 M R 4381640 + 8 [dd] > 8,16 1 771060 9.996239325 4580 Q R 4381648 + 8 [dd] > 8,16 1 771061 9.996239829 4580 M R 4381648 + 8 [dd] > 8,16 1 771062 9.996242197 4580 Q R 4381656 + 8 [dd] > 8,16 1 771063 9.996242700 4580 M R 4381656 + 8 [dd] > 8,16 1 771064 9.996244888 4580 Q R 4381664 + 8 [dd] > 8,16 1 771065 9.996245388 4580 M R 4381664 + 8 [dd] > 8,16 1 771066 9.996247580 4580 Q R 4381672 + 8 [dd] > 8,16 1 771067 9.996248079 4580 M R 4381672 + 8 [dd] > 8,16 1 771068 9.996250234 4580 Q R 4381680 + 8 [dd] > 8,16 1 771069 9.996250735 4580 M R 4381680 + 8 [dd] > 8,16 1 771070 9.996252898 4580 Q R 4381688 + 8 [dd] > 8,16 1 771071 9.996253388 4580 M R 4381688 + 8 [dd] > 8,16 1 771072 9.996357598 4580 Q R 4381696 + 8 [dd] > 8,16 1 771073 9.996358355 4580 M R 4381696 + 8 [dd] > 8,16 1 771074 9.996360935 4580 Q R 4381704 + 8 [dd] > 8,16 1 771075 9.996361434 4580 M R 4381704 + 8 [dd] > 8,16 1 771076 9.996363662 4580 Q R 4381712 + 8 [dd] > 8,16 1 771077 9.996364166 4580 M R 4381712 + 8 [dd] > 8,16 1 771078 9.996366220 4580 Q R 4381720 + 8 [dd] > 8,16 1 771079 9.996366731 4580 M R 4381720 + 8 [dd] > 8,16 1 771080 9.996368930 4580 Q R 4381728 + 8 [dd] > 8,16 1 771081 9.996369429 4580 M R 4381728 + 8 [dd] > 8,16 1 771082 9.996375003 4580 Q R 4381736 + 8 [dd] > 8,16 1 771083 9.996375513 4580 M R 4381736 + 8 [dd] > 8,16 1 771084 9.996377664 4580 Q R 4381744 + 8 [dd] > 8,16 1 771085 9.996378380 4580 M R 4381744 + 8 [dd] > 8,16 1 771086 9.996380579 4580 Q R 4381752 + 8 [dd] > 8,16 1 771087 9.996381098 4580 M R 4381752 + 8 [dd] > 8,16 1 771088 9.996383260 4580 Q R 4381760 + 8 [dd] > 8,16 1 771089 9.996383759 4580 M R 4381760 + 8 [dd] > 8,16 1 771090 9.996385969 4580 Q R 4381768 + 8 [dd] > 8,16 1 771091 9.996386469 4580 M R 4381768 + 8 [dd] > 8,16 1 771092 9.996388658 4580 Q R 4381776 + 8 [dd] > 8,16 1 771093 9.996389162 4580 M R 4381776 + 8 [dd] > 8,16 1 771094 9.996391335 4580 Q R 4381784 + 8 [dd] > 8,16 1 771095 9.996391837 4580 M R 4381784 + 8 [dd] > 8,16 1 771096 9.996393920 4580 Q R 4381792 + 8 [dd] > 8,16 1 771097 9.996394419 4580 M R 4381792 + 8 [dd] > 8,16 1 771098 9.996396469 4580 Q R 4381800 + 8 [dd] > 8,16 1 771099 9.996396967 4580 M R 4381800 + 8 [dd] > 8,16 1 771100 9.996399181 4580 Q R 4381808 + 8 [dd] > 8,16 1 771101 9.996399684 4580 M R 4381808 + 8 [dd] > 8,16 1 771102 9.996402243 4580 Q R 4381816 + 8 [dd] > 8,16 1 771103 9.996402751 4580 M R 4381816 + 8 [dd] > 8,16 1 771104 9.996404942 4580 Q R 4381824 + 8 [dd] > 8,16 1 771105 9.996405441 4580 M R 4381824 + 8 [dd] > 8,16 1 771106 9.996407662 4580 Q R 4381832 + 8 [dd] > 8,16 1 771107 9.996408160 4580 M R 4381832 + 8 [dd] > 8,16 1 771108 9.996410409 4580 Q R 4381840 + 8 [dd] > 8,16 1 771109 9.996410910 4580 M R 4381840 + 8 [dd] > 8,16 1 771110 9.996416841 4580 Q R 4381848 + 8 [dd] > 8,16 1 771111 9.996417341 4580 M R 4381848 + 8 [dd] > 8,16 1 771112 9.996419551 4580 Q R 4381856 + 8 [dd] > 8,16 1 771113 9.996420050 4580 M R 4381856 + 8 [dd] > 8,16 1 771114 9.996425832 4580 Q R 4381864 + 8 [dd] > 8,16 1 771115 9.996426331 4580 M R 4381864 + 8 [dd] > 8,16 1 771116 9.996428600 4580 Q R 4381872 + 8 [dd] > 8,16 1 771117 9.996429101 4580 M R 4381872 + 8 [dd] > 8,16 1 771118 9.996431277 4580 Q R 4381880 + 8 [dd] > 8,16 1 771119 9.996431779 4580 M R 4381880 + 8 [dd] > 8,16 1 771120 9.996434026 4580 Q R 4381888 + 8 [dd] > 8,16 1 771121 9.996434525 4580 M R 4381888 + 8 [dd] > 8,16 1 771122 9.996436615 4580 Q R 4381896 + 8 [dd] > 8,16 1 771123 9.996437114 4580 M R 4381896 + 8 [dd] > 8,16 1 771124 9.996439353 4580 Q R 4381904 + 8 [dd] > 8,16 1 771125 9.996439864 4580 M R 4381904 + 8 [dd] > 8,16 1 771126 9.996441959 4580 Q R 4381912 + 8 [dd] > 8,16 1 771127 9.996442460 4580 M R 4381912 + 8 [dd] > 8,16 1 771128 9.996445027 4580 Q R 4381920 + 8 [dd] > 8,16 1 771129 9.996445526 4580 M R 4381920 + 8 [dd] > 8,16 1 771130 9.996447737 4580 Q R 4381928 + 8 [dd] > 8,16 1 771131 9.996448252 4580 M R 4381928 + 8 [dd] > 8,16 1 771132 9.996450536 4580 Q R 4381936 + 8 [dd] > 8,16 1 771133 9.996451040 4580 M R 4381936 + 8 [dd] > 8,16 1 771134 9.996453308 4580 Q R 4381944 + 8 [dd] > 8,16 1 771135 9.996453811 4580 M R 4381944 + 8 [dd] > 8,16 1 771136 9.996456417 4580 I R 4381440 + 512 ( 300545) [dd] > 8,16 1 0 9.996458987 0 m N cfq4580S / insert_request > 8,16 1 771137 9.996462198 4580 U N [dd] 1 > 8,16 1 0 9.996464756 0 m N cfq4580S / dispatch_insert > 8,16 1 0 9.996468170 0 m N cfq4580S / dispatched a request > 8,16 1 0 9.996469836 0 m N cfq4580S / activate rq, drv=1 > 8,16 1 771138 9.996470460 4580 D R 4381440 + 512 ( 14043) [dd] > 8,16 1 771139 9.997514205 0 C R 4381440 + 512 ( 1043745) [0] > 8,16 1 0 9.997587102 0 m N cfq4580S / complete rqnoidle 0 > 8,16 1 0 9.997589870 0 m N cfq schedule dispatch > 8,16 1 771140 9.997751496 4580 Q R 4381952 + 8 [dd] > 8,16 1 771141 9.997758630 4580 G R 4381952 + 8 [dd] > 8,16 1 771142 9.997760814 4580 P N [dd] > 8,16 1 771143 9.997769413 4580 Q R 4381960 + 8 [dd] > 8,16 1 771144 9.997771473 4580 M R 4381960 + 8 [dd] > 8,16 1 771145 9.997774552 4580 Q R 4381968 + 8 [dd] > 8,16 1 771146 9.997775056 4580 M R 4381968 + 8 [dd] > 8,16 1 771147 9.997777142 4580 Q R 4381976 + 8 [dd] > 8,16 1 771148 9.997777717 4580 M R 4381976 + 8 [dd] > 8,16 1 771149 9.997779832 4580 Q R 4381984 + 8 [dd] > 8,16 1 771150 9.997780366 4580 M R 4381984 + 8 [dd] > 8,16 1 771151 9.997782454 4580 Q R 4381992 + 8 [dd] > 8,16 1 771152 9.997782953 4580 M R 4381992 + 8 [dd] > 8,16 1 771153 9.997785271 4580 Q R 4382000 + 8 [dd] > 8,16 1 771154 9.997785779 4580 M R 4382000 + 8 [dd] > 8,16 1 771155 9.997787937 4580 Q R 4382008 + 8 [dd] > 8,16 1 771156 9.997788434 4580 M R 4382008 + 8 [dd] > 8,16 1 771157 9.997790588 4580 Q R 4382016 + 8 [dd] > 8,16 1 771158 9.997791085 4580 M R 4382016 + 8 [dd] > 8,16 1 771159 9.997793251 4580 Q R 4382024 + 8 [dd] > 8,16 1 771160 9.997793954 4580 M R 4382024 + 8 [dd] > 8,16 1 771161 9.997796144 4580 Q R 4382032 + 8 [dd] > 8,16 1 771162 9.997796672 4580 M R 4382032 + 8 [dd] > 8,16 1 771163 9.997798737 4580 Q R 4382040 + 8 [dd] > 8,16 1 771164 9.997799238 4580 M R 4382040 + 8 [dd] > 8,16 1 771165 9.997801277 4580 Q R 4382048 + 8 [dd] > 8,16 1 771166 9.997801788 4580 M R 4382048 + 8 [dd] > 8,16 1 771167 9.997803925 4580 Q R 4382056 + 8 [dd] > 8,16 1 771168 9.997804439 4580 M R 4382056 + 8 [dd] > 8,16 1 771169 9.997806598 4580 Q R 4382064 + 8 [dd] > 8,16 1 771170 9.997807095 4580 M R 4382064 + 8 [dd] > 8,16 1 771171 9.997812761 4580 Q R 4382072 + 8 [dd] > 8,16 1 771172 9.997813262 4580 M R 4382072 + 8 [dd] > 8,16 1 771173 9.997815335 4580 Q R 4382080 + 8 [dd] > 8,16 1 771174 9.997815836 4580 M R 4382080 + 8 [dd] > 8,16 1 771175 9.997817949 4580 Q R 4382088 + 8 [dd] > 8,16 1 771176 9.997818448 4580 M R 4382088 + 8 [dd] > 8,16 1 771177 9.997820558 4580 Q R 4382096 + 8 [dd] > 8,16 1 771178 9.997821057 4580 M R 4382096 + 8 [dd] > 8,16 1 771179 9.997824698 4580 Q R 4382104 + 8 [dd] > 8,16 1 771180 9.997825197 4580 M R 4382104 + 8 [dd] > 8,16 1 771181 9.997827376 4580 Q R 4382112 + 8 [dd] > 8,16 1 771182 9.997827874 4580 M R 4382112 + 8 [dd] > 8,16 1 771183 9.997830036 4580 Q R 4382120 + 8 [dd] > 8,16 1 771184 9.997830538 4580 M R 4382120 + 8 [dd] > 8,16 1 771185 9.997832781 4580 Q R 4382128 + 8 [dd] > 8,16 1 771186 9.997833278 4580 M R 4382128 + 8 [dd] > 8,16 1 771187 9.997835542 4580 Q R 4382136 + 8 [dd] > 8,16 1 771188 9.997836044 4580 M R 4382136 + 8 [dd] > 8,16 1 771189 9.997838090 4580 Q R 4382144 + 8 [dd] > 8,16 1 771190 9.997838591 4580 M R 4382144 + 8 [dd] > 8,16 1 771191 9.997840646 4580 Q R 4382152 + 8 [dd] > 8,16 1 771192 9.997841147 4580 M R 4382152 + 8 [dd] > 8,16 1 771193 9.997843299 4580 Q R 4382160 + 8 [dd] > 8,16 1 771194 9.997843797 4580 M R 4382160 + 8 [dd] > 8,16 1 771195 9.997845909 4580 Q R 4382168 + 8 [dd] > 8,16 1 771196 9.997846408 4580 M R 4382168 + 8 [dd] > 8,16 1 771197 9.997848578 4580 Q R 4382176 + 8 [dd] > 8,16 1 771198 9.997849082 4580 M R 4382176 + 8 [dd] > 8,16 1 771199 9.997854848 4580 Q R 4382184 + 8 [dd] > 8,16 1 771200 9.997855350 4580 M R 4382184 + 8 [dd] > 8,16 1 771201 9.997857436 4580 Q R 4382192 + 8 [dd] > 8,16 1 771202 9.997857934 4580 M R 4382192 + 8 [dd] > 8,16 1 771203 9.997860278 4580 Q R 4382200 + 8 [dd] > 8,16 1 771204 9.997860779 4580 M R 4382200 + 8 [dd] > 8,16 1 771205 9.997964321 4580 Q R 4382208 + 8 [dd] > 8,16 1 771206 9.997965007 4580 M R 4382208 + 8 [dd] > 8,16 1 771207 9.997967619 4580 Q R 4382216 + 8 [dd] > 8,16 1 771208 9.997968161 4580 M R 4382216 + 8 [dd] > 8,16 1 771209 9.997970459 4580 Q R 4382224 + 8 [dd] > 8,16 1 771210 9.997970963 4580 M R 4382224 + 8 [dd] > 8,16 1 771211 9.997973081 4580 Q R 4382232 + 8 [dd] > 8,16 1 771212 9.997973580 4580 M R 4382232 + 8 [dd] > 8,16 1 771213 9.997975787 4580 Q R 4382240 + 8 [dd] > 8,16 1 771214 9.997976285 4580 M R 4382240 + 8 [dd] > 8,16 1 771215 9.997978452 4580 Q R 4382248 + 8 [dd] > 8,16 1 771216 9.997978953 4580 M R 4382248 + 8 [dd] > 8,16 1 771217 9.997981249 4580 Q R 4382256 + 8 [dd] > 8,16 1 771218 9.997981751 4580 M R 4382256 + 8 [dd] > 8,16 1 771219 9.997983917 4580 Q R 4382264 + 8 [dd] > 8,16 1 771220 9.997984418 4580 M R 4382264 + 8 [dd] > 8,16 1 771221 9.997986611 4580 Q R 4382272 + 8 [dd] > 8,16 1 771222 9.997987121 4580 M R 4382272 + 8 [dd] > 8,16 1 771223 9.997989204 4580 Q R 4382280 + 8 [dd] > 8,16 1 771224 9.997989705 4580 M R 4382280 + 8 [dd] > 8,16 1 771225 9.997991904 4580 Q R 4382288 + 8 [dd] > 8,16 1 771226 9.997992402 4580 M R 4382288 + 8 [dd] > 8,16 1 771227 9.997998036 4580 Q R 4382296 + 8 [dd] > 8,16 1 771228 9.997998579 4580 M R 4382296 + 8 [dd] > 8,16 1 771229 9.998000634 4580 Q R 4382304 + 8 [dd] > 8,16 1 771230 9.998001136 4580 M R 4382304 + 8 [dd] > 8,16 1 771231 9.998003255 4580 Q R 4382312 + 8 [dd] > 8,16 1 771232 9.998003774 4580 M R 4382312 + 8 [dd] > 8,16 1 771233 9.998005975 4580 Q R 4382320 + 8 [dd] > 8,16 1 771234 9.998006476 4580 M R 4382320 + 8 [dd] > 8,16 1 771235 9.998008972 4580 Q R 4382328 + 8 [dd] > 8,16 1 771236 9.998009529 4580 M R 4382328 + 8 [dd] > 8,16 1 771237 9.998011779 4580 Q R 4382336 + 8 [dd] > 8,16 1 771238 9.998012277 4580 M R 4382336 + 8 [dd] > 8,16 1 771239 9.998014476 4580 Q R 4382344 + 8 [dd] > 8,16 1 771240 9.998014974 4580 M R 4382344 + 8 [dd] > 8,16 1 771241 9.998017401 4580 Q R 4382352 + 8 [dd] > 8,16 1 771242 9.998017899 4580 M R 4382352 + 8 [dd] > 8,16 1 771243 9.998020091 4580 Q R 4382360 + 8 [dd] > 8,16 1 771244 9.998020588 4580 M R 4382360 + 8 [dd] > 8,16 1 771245 9.998022782 4580 Q R 4382368 + 8 [dd] > 8,16 1 771246 9.998023389 4580 M R 4382368 + 8 [dd] > 8,16 1 771247 9.998025647 4580 Q R 4382376 + 8 [dd] > 8,16 1 771248 9.998026180 4580 M R 4382376 + 8 [dd] > 8,16 1 771249 9.998028480 4580 Q R 4382384 + 8 [dd] > 8,16 1 771250 9.998029020 4580 M R 4382384 + 8 [dd] > 8,16 1 771251 9.998031205 4580 Q R 4382392 + 8 [dd] > 8,16 1 771252 9.998031706 4580 M R 4382392 + 8 [dd] > 8,16 1 771253 9.998033865 4580 Q R 4382400 + 8 [dd] > 8,16 1 771254 9.998034366 4580 M R 4382400 + 8 [dd] > 8,16 1 771255 9.998040073 4580 Q R 4382408 + 8 [dd] > 8,16 1 771256 9.998040571 4580 M R 4382408 + 8 [dd] > 8,16 1 771257 9.998042647 4580 Q R 4382416 + 8 [dd] > 8,16 1 771258 9.998043150 4580 M R 4382416 + 8 [dd] > 8,16 1 771259 9.998045325 4580 Q R 4382424 + 8 [dd] > 8,16 1 771260 9.998045826 4580 M R 4382424 + 8 [dd] > 8,16 1 771261 9.998047931 4580 Q R 4382432 + 8 [dd] > 8,16 1 771262 9.998048432 4580 M R 4382432 + 8 [dd] > 8,16 1 771263 9.998050535 4580 Q R 4382440 + 8 [dd] > 8,16 1 771264 9.998051038 4580 M R 4382440 + 8 [dd] > 8,16 1 771265 9.998056985 4580 Q R 4382448 + 8 [dd] > 8,16 1 771266 9.998057500 4580 M R 4382448 + 8 [dd] > 8,16 1 771267 9.998059576 4580 Q R 4382456 + 8 [dd] > 8,16 1 771268 9.998060077 4580 M R 4382456 + 8 [dd] > 8,16 1 771269 9.998062341 4580 I R 4381952 + 512 ( 303711) [dd] > 8,16 1 0 9.998064881 0 m N cfq4580S / insert_request > 8,16 1 771270 9.998068490 4580 U N [dd] 1 > 8,16 1 0 9.998071031 0 m N cfq4580S / dispatch_insert > 8,16 1 0 9.998074991 0 m N cfq4580S / dispatched a request > 8,16 1 0 9.998076645 0 m N cfq4580S / activate rq, drv=1 > 8,16 1 771271 9.998077269 4580 D R 4381952 + 512 ( 14928) [dd] > 8,16 1 771272 9.999120396 0 C R 4381952 + 512 ( 1043127) [0] > 8,16 1 0 9.999192302 0 m N cfq4580S / complete rqnoidle 0 > 8,16 1 0 9.999195361 0 m N cfq schedule dispatch > 8,16 1 771273 9.999356099 4580 Q R 4382464 + 8 [dd] > 8,16 1 771274 9.999362719 4580 G R 4382464 + 8 [dd] > 8,16 1 771275 9.999364795 4580 P N [dd] > 8,16 1 771276 9.999368199 4580 Q R 4382472 + 8 [dd] > 8,16 1 771277 9.999370072 4580 M R 4382472 + 8 [dd] > 8,16 1 771278 9.999373166 4580 Q R 4382480 + 8 [dd] > 8,16 1 771279 9.999373710 4580 M R 4382480 + 8 [dd] > 8,16 1 771280 9.999375830 4580 Q R 4382488 + 8 [dd] > 8,16 1 771281 9.999376334 4580 M R 4382488 + 8 [dd] > 8,16 1 771282 9.999378433 4580 Q R 4382496 + 8 [dd] > 8,16 1 771283 9.999378932 4580 M R 4382496 + 8 [dd] > 8,16 1 771284 9.999380973 4580 Q R 4382504 + 8 [dd] > 8,16 1 771285 9.999381477 4580 M R 4382504 + 8 [dd] > 8,16 1 771286 9.999383602 4580 Q R 4382512 + 8 [dd] > 8,16 1 771287 9.999384101 4580 M R 4382512 + 8 [dd] > 8,16 1 771288 9.999391190 4580 Q R 4382520 + 8 [dd] > 8,16 1 771289 9.999391690 4580 M R 4382520 + 8 [dd] > 8,16 1 771290 9.999393850 4580 Q R 4382528 + 8 [dd] > 8,16 1 771291 9.999394351 4580 M R 4382528 + 8 [dd] > 8,16 1 771292 9.999396618 4580 Q R 4382536 + 8 [dd] > 8,16 1 771293 9.999397190 4580 M R 4382536 + 8 [dd] > 8,16 1 771294 9.999399323 4580 Q R 4382544 + 8 [dd] > 8,16 1 771295 9.999399864 4580 M R 4382544 + 8 [dd] > 8,16 1 771296 9.999401953 4580 Q R 4382552 + 8 [dd] > 8,16 1 771297 9.999402472 4580 M R 4382552 + 8 [dd] > 8,16 1 771298 9.999404563 4580 Q R 4382560 + 8 [dd] > 8,16 1 771299 9.999405057 4580 M R 4382560 + 8 [dd] > 8,16 1 771300 9.999407109 4580 Q R 4382568 + 8 [dd] > 8,16 1 771301 9.999407611 4580 M R 4382568 + 8 [dd] > 8,16 1 771302 9.999409677 4580 Q R 4382576 + 8 [dd] > 8,16 1 771303 9.999410176 4580 M R 4382576 + 8 [dd] > 8,16 1 771304 9.999412226 4580 Q R 4382584 + 8 [dd] > 8,16 1 771305 9.999412734 4580 M R 4382584 + 8 [dd] > 8,16 1 771306 9.999416340 4580 Q R 4382592 + 8 [dd] > 8,16 1 771307 9.999416862 4580 M R 4382592 + 8 [dd] > 8,16 1 771308 9.999419187 4580 Q R 4382600 + 8 [dd] > 8,16 1 771309 9.999419706 4580 M R 4382600 + 8 [dd] > 8,16 1 771310 9.999421977 4580 Q R 4382608 + 8 [dd] > 8,16 1 771311 9.999422503 4580 M R 4382608 + 8 [dd] > 8,16 1 771312 9.999424582 4580 Q R 4382616 + 8 [dd] > 8,16 1 771313 9.999425120 4580 M R 4382616 + 8 [dd] > 8,16 1 771314 9.999427292 4580 Q R 4382624 + 8 [dd] > 8,16 1 771315 9.999427805 4580 M R 4382624 + 8 [dd] > 8,16 1 771316 9.999433255 4580 Q R 4382632 + 8 [dd] > 8,16 1 771317 9.999433766 4580 M R 4382632 + 8 [dd] > 8,16 1 771318 9.999435901 4580 Q R 4382640 + 8 [dd] > 8,16 1 771319 9.999436412 4580 M R 4382640 + 8 [dd] > 8,16 1 771320 9.999438504 4580 Q R 4382648 + 8 [dd] > 8,16 1 771321 9.999439243 4580 M R 4382648 + 8 [dd] > 8,16 1 771322 9.999441418 4580 Q R 4382656 + 8 [dd] > 8,16 1 771323 9.999441949 4580 M R 4382656 + 8 [dd] > 8,16 1 771324 9.999444183 4580 Q R 4382664 + 8 [dd] > 8,16 1 771325 9.999444698 4580 M R 4382664 + 8 [dd] > 8,16 1 771326 9.999446894 4580 Q R 4382672 + 8 [dd] > 8,16 1 771327 9.999447411 4580 M R 4382672 + 8 [dd] > 8,16 1 771328 9.999449485 4580 Q R 4382680 + 8 [dd] > 8,16 1 771329 9.999449997 4580 M R 4382680 + 8 [dd] > 8,16 1 771330 9.999452055 4580 Q R 4382688 + 8 [dd] > 8,16 1 771331 9.999452566 4580 M R 4382688 + 8 [dd] > 8,16 1 771332 9.999454606 4580 Q R 4382696 + 8 [dd] > 8,16 1 771333 9.999455154 4580 M R 4382696 + 8 [dd] > 8,16 1 771334 9.999457363 4580 Q R 4382704 + 8 [dd] > 8,16 1 771335 9.999457875 4580 M R 4382704 + 8 [dd] > 8,16 1 771336 9.999460045 4580 Q R 4382712 + 8 [dd] > 8,16 1 771337 9.999460561 4580 M R 4382712 + 8 [dd] > 8,16 1 771338 9.999564615 4580 Q R 4382720 + 8 [dd] > 8,16 1 771339 9.999565347 4580 M R 4382720 + 8 [dd] > 8,16 1 771340 9.999568401 4580 Q R 4382728 + 8 [dd] > 8,16 1 771341 9.999568908 4580 M R 4382728 + 8 [dd] > 8,16 1 771342 9.999571276 4580 Q R 4382736 + 8 [dd] > 8,16 1 771343 9.999571792 4580 M R 4382736 + 8 [dd] > 8,16 1 771344 9.999577396 4580 Q R 4382744 + 8 [dd] > 8,16 1 771345 9.999578003 4580 M R 4382744 + 8 [dd] > 8,16 1 771346 9.999580231 4580 Q R 4382752 + 8 [dd] > 8,16 1 771347 9.999580749 4580 M R 4382752 + 8 [dd] > 8,16 1 771348 9.999583114 4580 Q R 4382760 + 8 [dd] > 8,16 1 771349 9.999583625 4580 M R 4382760 + 8 [dd] > 8,16 1 771350 9.999585875 4580 Q R 4382768 + 8 [dd] > 8,16 1 771351 9.999586374 4580 M R 4382768 + 8 [dd] > 8,16 1 771352 9.999588531 4580 Q R 4382776 + 8 [dd] > 8,16 1 771353 9.999589032 4580 M R 4382776 + 8 [dd] > 8,16 1 771354 9.999591214 4580 Q R 4382784 + 8 [dd] > 8,16 1 771355 9.999591714 4580 M R 4382784 + 8 [dd] > 8,16 1 771356 9.999593930 4580 Q R 4382792 + 8 [dd] > 8,16 1 771357 9.999594442 4580 M R 4382792 + 8 [dd] > 8,16 1 771358 9.999596696 4580 Q R 4382800 + 8 [dd] > 8,16 1 771359 9.999597190 4580 M R 4382800 + 8 [dd] > 8,16 1 771360 9.999599406 4580 Q R 4382808 + 8 [dd] > 8,16 1 771361 9.999599907 4580 M R 4382808 + 8 [dd] > 8,16 1 771362 9.999602166 4580 Q R 4382816 + 8 [dd] > 8,16 1 771363 9.999602667 4580 M R 4382816 + 8 [dd] > 8,16 1 771364 9.999604755 4580 Q R 4382824 + 8 [dd] > 8,16 1 771365 9.999605259 4580 M R 4382824 + 8 [dd] > 8,16 1 771366 9.999607486 4580 Q R 4382832 + 8 [dd] > 8,16 1 771367 9.999607978 4580 M R 4382832 + 8 [dd] > 8,16 1 771368 9.999610062 4580 Q R 4382840 + 8 [dd] > 8,16 1 771369 9.999610564 4580 M R 4382840 + 8 [dd] > 8,16 1 771370 9.999612914 4580 Q R 4382848 + 8 [dd] > 8,16 1 771371 9.999613418 4580 M R 4382848 + 8 [dd] > 8,16 1 771372 9.999618806 4580 Q R 4382856 + 8 [dd] > 8,16 1 771373 9.999619308 4580 M R 4382856 + 8 [dd] > 8,16 1 771374 9.999621579 4580 Q R 4382864 + 8 [dd] > 8,16 1 771375 9.999622092 4580 M R 4382864 + 8 [dd] > 8,16 1 771376 9.999624260 4580 Q R 4382872 + 8 [dd] > 8,16 1 771377 9.999624779 4580 M R 4382872 + 8 [dd] > 8,16 1 771378 9.999627506 4580 Q R 4382880 + 8 [dd] > 8,16 1 771379 9.999628012 4580 M R 4382880 + 8 [dd] > 8,16 1 771380 9.999630135 4580 Q R 4382888 + 8 [dd] > 8,16 1 771381 9.999630635 4580 M R 4382888 + 8 [dd] > 8,16 1 771382 9.999632876 4580 Q R 4382896 + 8 [dd] > 8,16 1 771383 9.999633378 4580 M R 4382896 + 8 [dd] > 8,16 1 771384 9.999635541 4580 Q R 4382904 + 8 [dd] > 8,16 1 771385 9.999636043 4580 M R 4382904 + 8 [dd] > 8,16 1 771386 9.999638232 4580 Q R 4382912 + 8 [dd] > 8,16 1 771387 9.999638736 4580 M R 4382912 + 8 [dd] > 8,16 1 771388 9.999640940 4580 Q R 4382920 + 8 [dd] > 8,16 1 771389 9.999641440 4580 M R 4382920 + 8 [dd] > 8,16 1 771390 9.999643644 4580 Q R 4382928 + 8 [dd] > 8,16 1 771391 9.999644157 4580 M R 4382928 + 8 [dd] > 8,16 1 771392 9.999646440 4580 Q R 4382936 + 8 [dd] > 8,16 1 771393 9.999646960 4580 M R 4382936 + 8 [dd] > 8,16 1 771394 9.999649193 4580 Q R 4382944 + 8 [dd] > 8,16 1 771395 9.999649697 4580 M R 4382944 + 8 [dd] > 8,16 1 771396 9.999651771 4580 Q R 4382952 + 8 [dd] > 8,16 1 771397 9.999652272 4580 M R 4382952 + 8 [dd] > 8,16 1 771398 9.999654476 4580 Q R 4382960 + 8 [dd] > 8,16 1 771399 9.999654975 4580 M R 4382960 + 8 [dd] > 8,16 1 771400 9.999660256 4580 Q R 4382968 + 8 [dd] > 8,16 1 771401 9.999660760 4580 M R 4382968 + 8 [dd] > 8,16 1 771402 9.999663197 4580 I R 4382464 + 512 ( 300478) [dd] > 8,16 1 0 9.999665743 0 m N cfq4580S / insert_request > 8,16 1 771403 9.999669190 4580 U N [dd] 1 > 8,16 1 0 9.999672048 0 m N cfq4580S / dispatch_insert > 8,16 1 0 9.999675568 0 m N cfq4580S / dispatched a request > 8,16 1 0 9.999677281 0 m N cfq4580S / activate rq, drv=1 > 8,16 1 771404 9.999677892 4580 D R 4382464 + 512 ( 14695) [dd] > CPU0 (sdb): > Reads Queued: 20608, 82432KiB Writes Queued: 0, 0KiB > Read Dispatches: 322, 82432KiB Write Dispatches: 0, 0KiB > Reads Requeued: 0 Writes Requeued: 0 > Reads Completed: 22, 5632KiB Writes Completed: 0, 0KiB > Read Merges: 20286, 81144KiB Write Merges: 0, 0KiB > Read depth: 1 Write depth: 0 > IO unplugs: 322 Timer unplugs: 0 > CPU1 (sdb): > Reads Queued: 371058, 1484MiB Writes Queued: 0, 0KiB > Read Dispatches: 5798, 1484MiB Write Dispatches: 0, 0KiB > Reads Requeued: 0 Writes Requeued: 0 > Reads Completed: 6097, 1560MiB Writes Completed: 0, 0KiB > Read Merges: 365261, 1461MiB Write Merges: 0, 0KiB > Read depth: 1 Write depth: 0 > IO unplugs: 5798 Timer unplugs: 0 > > Total (sdb): > Reads Queued: 391666, 1566MiB Writes Queued: 0, 0KiB > Read Dispatches: 6120, 1566MiB Write Dispatches: 0, 0KiB > Reads Requeued: 0 Writes Requeued: 0 > Reads Completed: 6119, 1566MiB Writes Completed: 0, 0KiB > Read Merges: 385547, 1542MiB Write Merges: 0, 0KiB > IO unplugs: 6120 Timer unplugs: 0 > > Throughput (R/W): 156662KiB/s / 0KiB/s > Events (sdb): 851428 entries > Skips: 0 forward (0 - 0.0%) > Input file sdb.blktrace.0 added > Input file sdb.blktrace.1 added > ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 14:01 ` Wu Fengguang @ 2012-01-30 14:05 ` Wu Fengguang 0 siblings, 0 replies; 36+ messages in thread From: Wu Fengguang @ 2012-01-30 14:05 UTC (permalink / raw) To: Eric Dumazet Cc: Andrew Morton, LKML, Jens Axboe, Tejun Heo, Li Shaohua, Herbert Poetzl > - IO size is 256KB (which is not a problem in itself) > > - The dispatch/complete pattern is > > submit IO for range 1 > complete IO for range 1 > <dd busy, disk idle> > submit IO for range 2 > complete IO for range 2 > > So we have periods that no IO is in flight at all, which leads to > under-utilized disk (which should show up in iostat as <100% disk util) > > # grep '[DC]' blktrace > > 8,16 1 770606 9.990039807 4580 D R 4379392 + 512 ( 14826) [dd] > 8,16 1 770607 9.991083069 0 C R 4379392 + 512 ( 1043262) [0] > > 8,16 1 770739 9.991647434 4580 D R 4379904 + 512 ( 14433) [dd] > 8,16 1 770740 9.992693317 0 C R 4379904 + 512 ( 1045883) [0] > > 8,16 1 770872 9.993256451 4580 D R 4380416 + 512 ( 14539) [dd] > 8,16 1 770873 9.994299156 0 C R 4380416 + 512 ( 1042705) [0] > > 8,16 1 771005 9.994863680 4580 D R 4380928 + 512 ( 14344) [dd] > 8,16 1 771006 9.995909291 0 C R 4380928 + 512 ( 1045611) [0] > > 8,16 1 771138 9.996470460 4580 D R 4381440 + 512 ( 14043) [dd] > 8,16 1 771139 9.997514205 0 C R 4381440 + 512 ( 1043745) [0] > > 8,16 1 771271 9.998077269 4580 D R 4381952 + 512 ( 14928) [dd] > 8,16 1 771272 9.999120396 0 C R 4381952 + 512 ( 1043127) [0] Some better pattern for comparison: 8,0 1 42940 0.990199812 4084 D R 1432808 + 256 [dd] 8,0 1 42941 0.990858326 0 C R 1432552 + 256 [0] 8,0 1 43009 0.991192107 4084 D R 1433064 + 256 [dd] 8,0 3 14691 0.991853319 0 C R 1432808 + 256 [0] 8,0 3 14759 0.992189451 4084 D R 1433320 + 256 [dd] 8,0 1 43010 0.994473159 0 C R 1433064 + 256 [0] Where the life time for requests 1432808, 1432552, 1433064, ... are *interleaved* so that the disk always has something to do. Thanks, Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 16:10 ` Wu Fengguang 2012-01-29 20:15 ` Herbert Poetzl @ 2012-01-30 3:17 ` Shaohua Li 2012-01-30 5:31 ` Eric Dumazet 2012-01-30 7:13 ` Herbert Poetzl 1 sibling, 2 replies; 36+ messages in thread From: Shaohua Li @ 2012-01-30 3:17 UTC (permalink / raw) To: Wu Fengguang Cc: Eric Dumazet, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo 2012/1/30 Wu Fengguang <wfg@linux.intel.com>: > On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: >> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : >> >> >> > Note that as long as buffered read(2) is used, it makes almost no >> > difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": >> > the 128kb readahead size will be used underneath to submit read IO. >> > >> >> Hmm... >> >> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >> 32768+0 enregistrements lus >> 32768+0 enregistrements écrits >> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s >> >> >> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >> 2048+0 enregistrements lus >> 2048+0 enregistrements écrits >> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s > > Interesting. Here are my test results: > > root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > 32768+0 records in > 32768+0 records out > 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s > root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > 2048+0 records in > 2048+0 records out > 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s > > Maybe the /dev/sda performance bug on your machine is sensitive to timing? I got similar result: 128k: 224M/s 1M: 182M/s 1M block size is slow, I guess it's CPU related. And as for the big regression with newer kernel than 2.6.38, please check if idle=poll helps. CPU idle dramatically impacts disk performance and even latest cpuidle governor doesn't help for some CPUs. Thanks, Shaohua ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 3:17 ` Shaohua Li @ 2012-01-30 5:31 ` Eric Dumazet 2012-01-30 5:45 ` Shaohua Li 2012-01-30 7:13 ` Herbert Poetzl 1 sibling, 1 reply; 36+ messages in thread From: Eric Dumazet @ 2012-01-30 5:31 UTC (permalink / raw) To: Shaohua Li Cc: Wu Fengguang, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo Le lundi 30 janvier 2012 à 11:17 +0800, Shaohua Li a écrit : > 1M block size is slow, I guess it's CPU related. > > And as for the big regression with newer kernel than 2.6.38, please > check if idle=poll > helps. CPU idle dramatically impacts disk performance and even latest > cpuidle governor > doesn't help for some CPUs. > No change with idle=poll # echo 3 >/proc/sys/vm/drop_caches ;perf record dd if=/dev/sda of=/dev/null bs=2M count=2048 2048+0 enregistrements lus 2048+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 27,6942 s, 155 MB/s [ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.884 MB perf.data (~38630 samples) ] # perf report Events: 22K cycles 40,14% dd [kernel.kallsyms] [k] copy_user_generic_string 3,12% dd [kernel.kallsyms] [k] native_read_tsc 3,00% dd [kernel.kallsyms] [k] put_page 2,98% dd [kernel.kallsyms] [k] iowrite8 2,04% dd [kernel.kallsyms] [k] get_page_from_freelist 1,87% dd [kernel.kallsyms] [k] delay_tsc 1,77% dd [kernel.kallsyms] [k] kmem_cache_alloc 1,69% dd [kernel.kallsyms] [k] __mem_cgroup_commit_charge 1,52% dd [kernel.kallsyms] [k] submit_bh 1,38% dd [kernel.kallsyms] [k] ioread8 1,32% dd [kernel.kallsyms] [k] __rmqueue 1,29% dd [kernel.kallsyms] [k] block_read_full_page 1,28% dd [kernel.kallsyms] [k] __ticket_spin_lock 1,23% dd [kernel.kallsyms] [k] __slab_alloc 1,15% dd [kernel.kallsyms] [k] file_read_actor 0,85% dd [kernel.kallsyms] [k] generic_make_request.part.51 0,84% dd [kernel.kallsyms] [k] __alloc_pages_nodemask 0,82% dd [kernel.kallsyms] [k] create_empty_buffers 0,81% dd [kernel.kallsyms] [k] get_partial_node 0,81% dd [kernel.kallsyms] [k] lookup_page_cgroup 0,78% dd [kernel.kallsyms] [k] mem_cgroup_add_lru_list.part.45 0,72% dd [kernel.kallsyms] [k] ____pagevec_lru_add_fn # echo 3 >/proc/sys/vm/drop_caches ;perf record dd if=/dev/sda of=/dev/null bs=1M count=4096 4096+0 enregistrements lus 4096+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 25,8117 s, 166 MB/s [ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.810 MB perf.data (~35398 samples) ] # perf report Events: 20K cycles 31,91% dd [kernel.kallsyms] [k] copy_user_generic_string 3,87% dd [kernel.kallsyms] [k] native_read_tsc 3,58% dd [kernel.kallsyms] [k] iowrite8 2,43% dd [kernel.kallsyms] [k] put_page 2,30% dd [kernel.kallsyms] [k] delay_tsc 2,09% dd [kernel.kallsyms] [k] get_page_from_freelist 1,96% dd [kernel.kallsyms] [k] __mem_cgroup_commit_charge 1,79% dd [kernel.kallsyms] [k] submit_bh 1,75% dd [kernel.kallsyms] [k] kmem_cache_alloc 1,74% dd [kernel.kallsyms] [k] ioread8 1,68% dd [kernel.kallsyms] [k] block_read_full_page 1,62% dd [kernel.kallsyms] [k] __ticket_spin_lock 1,38% dd [kernel.kallsyms] [k] __rmqueue 1,12% dd [kernel.kallsyms] [k] create_empty_buffers 1,08% dd [kernel.kallsyms] [k] __slab_alloc 1,07% dd [kernel.kallsyms] [k] file_read_actor 1,06% dd [kernel.kallsyms] [k] generic_make_request.part.51 0,94% dd [kernel.kallsyms] [k] lookup_page_cgroup 0,85% dd [kernel.kallsyms] [k] __alloc_pages_nodemask 0,85% dd [kernel.kallsyms] [k] find_get_page 0,83% dd [kernel.kallsyms] [k] __lru_cache_add 0,81% dd [kernel.kallsyms] [k] add_to_page_cache_locked.part.29 # echo 3 >/proc/sys/vm/drop_caches ;perf record dd if=/dev/sda of=/dev/null bs=128k count=32768 32768+0 enregistrements lus 32768+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 19,6924 s, 218 MB/s [ perf record: Woken up 3 times to write data ] [ perf record: Captured and wrote 0.961 MB perf.data (~41992 samples) ] # perf report 29,85% dd [kernel.kallsyms] [k] copy_user_generic_string 2,61% dd [kernel.kallsyms] [k] __mem_cgroup_commit_charge 2,53% dd [kernel.kallsyms] [k] get_page_from_freelist 2,27% dd [kernel.kallsyms] [k] block_read_full_page 1,99% dd [kernel.kallsyms] [k] __ticket_spin_lock 1,81% dd [kernel.kallsyms] [k] kmem_cache_alloc 1,65% dd [kernel.kallsyms] [k] file_read_actor 1,60% dd [kernel.kallsyms] [k] __rmqueue 1,58% dd [kernel.kallsyms] [k] put_page 1,51% dd [kernel.kallsyms] [k] submit_bh 1,37% dd [kernel.kallsyms] [k] create_empty_buffers 1,30% dd [kernel.kallsyms] [k] generic_make_request.part.51 1,09% dd [kernel.kallsyms] [k] __alloc_pages_nodemask 1,06% dd [kernel.kallsyms] [k] do_generic_file_read.constprop.35 1,01% dd [kernel.kallsyms] [k] lookup_page_cgroup 1,00% dd [kernel.kallsyms] [k] add_to_page_cache_locked.part.29 0,93% dd [kernel.kallsyms] [k] find_get_page 0,92% dd [kernel.kallsyms] [k] blk_rq_map_sg 0,92% dd [kernel.kallsyms] [k] radix_tree_insert 0,89% dd [kernel.kallsyms] [k] alloc_pages_current 0,88% dd [kernel.kallsyms] [k] mem_cgroup_add_lru_list.part.45 0,88% dd [kernel.kallsyms] [k] radix_tree_lookup_element ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 5:31 ` Eric Dumazet @ 2012-01-30 5:45 ` Shaohua Li 0 siblings, 0 replies; 36+ messages in thread From: Shaohua Li @ 2012-01-30 5:45 UTC (permalink / raw) To: Eric Dumazet Cc: Wu Fengguang, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo 2012/1/30 Eric Dumazet <eric.dumazet@gmail.com>: > Le lundi 30 janvier 2012 à 11:17 +0800, Shaohua Li a écrit : > >> 1M block size is slow, I guess it's CPU related. >> >> And as for the big regression with newer kernel than 2.6.38, please >> check if idle=poll >> helps. CPU idle dramatically impacts disk performance and even latest >> cpuidle governor >> doesn't help for some CPUs. >> > > No change with idle=poll I mean the big regression might be related to CPU idle. because I saw really bad performance with CPU idle with some CPUs. The 1M block size performance is lower than 128K is not. but I still guess it's CPU related (like copy). The test result is always like this IIRC. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 3:17 ` Shaohua Li 2012-01-30 5:31 ` Eric Dumazet @ 2012-01-30 7:13 ` Herbert Poetzl 2012-01-30 7:22 ` Shaohua Li 1 sibling, 1 reply; 36+ messages in thread From: Herbert Poetzl @ 2012-01-30 7:13 UTC (permalink / raw) To: Shaohua Li Cc: Wu Fengguang, Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote: > 2012/1/30 Wu Fengguang <wfg@linux.intel.com>: >> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: >>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : >>>> Note that as long as buffered read(2) is used, it makes almost no >>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": >>>> the 128kb readahead size will be used underneath to submit read IO. >>> Hmm... >>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >>> 32768+0 enregistrements lus >>> 32768+0 enregistrements écrits >>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s >>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >>> 2048+0 enregistrements lus >>> 2048+0 enregistrements écrits >>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s >> Interesting. Here are my test results: >> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >> 32768+0 records in >> 32768+0 records out >> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s >> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >> 2048+0 records in >> 2048+0 records out >> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s >> Maybe the /dev/sda performance bug on your machine is sensitive to timing? > I got similar result: > 128k: 224M/s > 1M: 182M/s > 1M block size is slow, I guess it's CPU related. > And as for the big regression with newer kernel than 2.6.38, > please check if idle=poll helps. CPU idle dramatically impacts > disk performance and even latest cpuidle governor doesn't help > for some CPUs. here are the tests with idle=poll and after switching to 128k (instead of 1M) blocksize (same amount of data transferred) kernel ------------ read /dev/sda ------------- --- noop --- - deadline - ---- cfs --- [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU -------------------------------------------------- 3.2.2 45.82 3.7 44.85 3.6 45.04 3.4 3.2.2i 45.59 2.3 51.78 2.6 46.03 2.2 3.2.2i128 250.24 20.9 252.68 21.3 250.00 21.6 kernel -- write --- ------------------read ----------------- --- noop --- --- noop --- - deadline - ---- cfs --- [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU ---------------------------------------------------------------- 3.2.2 270.95 42.6 162.36 9.9 162.63 9.9 162.65 10.1 3.2.2i 269.10 41.4 170.82 6.6 171.20 6.6 170.91 6.7 3.2.2i128 270.38 67.7 162.35 10.2 163.01 10.3 162.34 10.7 HTH, Herbert > Thanks, > Shaohua ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 7:13 ` Herbert Poetzl @ 2012-01-30 7:22 ` Shaohua Li 2012-01-30 7:36 ` Herbert Poetzl 0 siblings, 1 reply; 36+ messages in thread From: Shaohua Li @ 2012-01-30 7:22 UTC (permalink / raw) To: Herbert Poetzl Cc: Wu Fengguang, Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote: > On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote: > > 2012/1/30 Wu Fengguang <wfg@linux.intel.com>: > >> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: > >>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : > > > >>>> Note that as long as buffered read(2) is used, it makes almost no > >>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": > >>>> the 128kb readahead size will be used underneath to submit read IO. > > > >>> Hmm... > > >>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > >>> 32768+0 enregistrements lus > >>> 32768+0 enregistrements écrits > >>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s > > > >>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > >>> 2048+0 enregistrements lus > >>> 2048+0 enregistrements écrits > >>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s > > >> Interesting. Here are my test results: > > >> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > >> 32768+0 records in > >> 32768+0 records out > >> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s > >> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > >> 2048+0 records in > >> 2048+0 records out > >> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s > > >> Maybe the /dev/sda performance bug on your machine is sensitive to timing? > > I got similar result: > > 128k: 224M/s > > 1M: 182M/s > > > 1M block size is slow, I guess it's CPU related. > > > And as for the big regression with newer kernel than 2.6.38, > > please check if idle=poll helps. CPU idle dramatically impacts > > disk performance and even latest cpuidle governor doesn't help > > for some CPUs. > > here are the tests with idle=poll and after switching to 128k > (instead of 1M) blocksize (same amount of data transferred) > > kernel ------------ read /dev/sda ------------- > --- noop --- - deadline - ---- cfs --- > [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU > -------------------------------------------------- > 3.2.2 45.82 3.7 44.85 3.6 45.04 3.4 > 3.2.2i 45.59 2.3 51.78 2.6 46.03 2.2 > 3.2.2i128 250.24 20.9 252.68 21.3 250.00 21.6 > > kernel -- write --- ------------------read ----------------- > --- noop --- --- noop --- - deadline - ---- cfs --- > [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU > ---------------------------------------------------------------- > 3.2.2 270.95 42.6 162.36 9.9 162.63 9.9 162.65 10.1 > 3.2.2i 269.10 41.4 170.82 6.6 171.20 6.6 170.91 6.7 > 3.2.2i128 270.38 67.7 162.35 10.2 163.01 10.3 162.34 10.7 What's 3.2.2i and 3.2.2i128? does idle=poll help? ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 7:22 ` Shaohua Li @ 2012-01-30 7:36 ` Herbert Poetzl 2012-01-30 8:12 ` Shaohua Li 0 siblings, 1 reply; 36+ messages in thread From: Herbert Poetzl @ 2012-01-30 7:36 UTC (permalink / raw) To: Shaohua Li Cc: Wu Fengguang, Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, Jan 30, 2012 at 03:22:38PM +0800, Shaohua Li wrote: > On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote: >> On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote: >>> 2012/1/30 Wu Fengguang <wfg@linux.intel.com>: >>>> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: >>>>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : >>>>>> Note that as long as buffered read(2) is used, it makes almost no >>>>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": >>>>>> the 128kb readahead size will be used underneath to submit read IO. >>>>> Hmm... >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >>>>> 32768+0 enregistrements lus >>>>> 32768+0 enregistrements écrits >>>>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >>>>> 2048+0 enregistrements lus >>>>> 2048+0 enregistrements écrits >>>>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s >>>> Interesting. Here are my test results: >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >>>> 32768+0 records in >>>> 32768+0 records out >>>> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >>>> 2048+0 records in >>>> 2048+0 records out >>>> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s >>>> Maybe the /dev/sda performance bug on your machine is sensitive to timing? >>> I got similar result: >>> 128k: 224M/s >>> 1M: 182M/s >>> 1M block size is slow, I guess it's CPU related. >>> And as for the big regression with newer kernel than 2.6.38, >>> please check if idle=poll helps. CPU idle dramatically impacts >>> disk performance and even latest cpuidle governor doesn't help >>> for some CPUs. >> here are the tests with idle=poll and after switching to 128k >> (instead of 1M) blocksize (same amount of data transferred) >> kernel ------------ read /dev/sda ------------- >> --- noop --- - deadline - ---- cfs --- >> [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU >> -------------------------------------------------- >> 3.2.2 45.82 3.7 44.85 3.6 45.04 3.4 >> 3.2.2i 45.59 2.3 51.78 2.6 46.03 2.2 >> 3.2.2i128 250.24 20.9 252.68 21.3 250.00 21.6 >> kernel -- write --- ------------------read ----------------- >> --- noop --- --- noop --- - deadline - ---- cfs --- >> [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU >> ---------------------------------------------------------------- >> 3.2.2 270.95 42.6 162.36 9.9 162.63 9.9 162.65 10.1 >> 3.2.2i 269.10 41.4 170.82 6.6 171.20 6.6 170.91 6.7 >> 3.2.2i128 270.38 67.7 162.35 10.2 163.01 10.3 162.34 10.7 > What's 3.2.2i and 3.2.2i128? 3.2.2 ...... kernel with default options (bs=1M) 3.2.2i ..... kernel with idle=poll (bs=1M) 3.2.2i128 .. kernel with idle=poll (bs=128k) > does idle=poll help? doesn't look like, at least to me ... HTC, Herbert ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 7:36 ` Herbert Poetzl @ 2012-01-30 8:12 ` Shaohua Li 2012-01-30 10:31 ` Shaohua Li 0 siblings, 1 reply; 36+ messages in thread From: Shaohua Li @ 2012-01-30 8:12 UTC (permalink / raw) To: Herbert Poetzl Cc: Wu Fengguang, Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, 2012-01-30 at 08:36 +0100, Herbert Poetzl wrote: > On Mon, Jan 30, 2012 at 03:22:38PM +0800, Shaohua Li wrote: > > On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote: > >> On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote: > >>> 2012/1/30 Wu Fengguang <wfg@linux.intel.com>: > >>>> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: > >>>>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : > > >>>>>> Note that as long as buffered read(2) is used, it makes almost no > >>>>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": > >>>>>> the 128kb readahead size will be used underneath to submit read IO. > > >>>>> Hmm... > > >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > >>>>> 32768+0 enregistrements lus > >>>>> 32768+0 enregistrements écrits > >>>>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s > > > >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > >>>>> 2048+0 enregistrements lus > >>>>> 2048+0 enregistrements écrits > >>>>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s > > >>>> Interesting. Here are my test results: > > >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 > >>>> 32768+0 records in > >>>> 32768+0 records out > >>>> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s > >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 > >>>> 2048+0 records in > >>>> 2048+0 records out > >>>> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s > > >>>> Maybe the /dev/sda performance bug on your machine is sensitive to timing? > >>> I got similar result: > >>> 128k: 224M/s > >>> 1M: 182M/s > > >>> 1M block size is slow, I guess it's CPU related. > > >>> And as for the big regression with newer kernel than 2.6.38, > >>> please check if idle=poll helps. CPU idle dramatically impacts > >>> disk performance and even latest cpuidle governor doesn't help > >>> for some CPUs. > > >> here are the tests with idle=poll and after switching to 128k > >> (instead of 1M) blocksize (same amount of data transferred) > > >> kernel ------------ read /dev/sda ------------- > >> --- noop --- - deadline - ---- cfs --- > >> [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU > >> -------------------------------------------------- > >> 3.2.2 45.82 3.7 44.85 3.6 45.04 3.4 > >> 3.2.2i 45.59 2.3 51.78 2.6 46.03 2.2 > >> 3.2.2i128 250.24 20.9 252.68 21.3 250.00 21.6 > > >> kernel -- write --- ------------------read ----------------- > >> --- noop --- --- noop --- - deadline - ---- cfs --- > >> [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU > >> ---------------------------------------------------------------- > >> 3.2.2 270.95 42.6 162.36 9.9 162.63 9.9 162.65 10.1 > >> 3.2.2i 269.10 41.4 170.82 6.6 171.20 6.6 170.91 6.7 > >> 3.2.2i128 270.38 67.7 162.35 10.2 163.01 10.3 162.34 10.7 > > > What's 3.2.2i and 3.2.2i128? > > 3.2.2 ...... kernel with default options (bs=1M) > 3.2.2i ..... kernel with idle=poll (bs=1M) > 3.2.2i128 .. kernel with idle=poll (bs=128k) > > > does idle=poll help? > > doesn't look like, at least to me ... what's your /sys/block/sdx/queue/max_sectors_kb? if you make it smaller, does the performance increase? In my system, a smaller max_sectors_kb makes bs=2M and bs=128k have similar performance, which makes me think it's CPU doesn't catch up quickly after a request finishes. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 8:12 ` Shaohua Li @ 2012-01-30 10:31 ` Shaohua Li 2012-01-30 14:28 ` Wu Fengguang 0 siblings, 1 reply; 36+ messages in thread From: Shaohua Li @ 2012-01-30 10:31 UTC (permalink / raw) To: Herbert Poetzl Cc: Wu Fengguang, Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo 2012/1/30 Shaohua Li <shaohua.li@intel.com>: > On Mon, 2012-01-30 at 08:36 +0100, Herbert Poetzl wrote: >> On Mon, Jan 30, 2012 at 03:22:38PM +0800, Shaohua Li wrote: >> > On Mon, 2012-01-30 at 08:13 +0100, Herbert Poetzl wrote: >> >> On Mon, Jan 30, 2012 at 11:17:38AM +0800, Shaohua Li wrote: >> >>> 2012/1/30 Wu Fengguang <wfg@linux.intel.com>: >> >>>> On Sun, Jan 29, 2012 at 02:13:51PM +0100, Eric Dumazet wrote: >> >>>>> Le dimanche 29 janvier 2012 à 19:16 +0800, Wu Fengguang a écrit : >> >> >>>>>> Note that as long as buffered read(2) is used, it makes almost no >> >>>>>> difference (well, at least for now) to do "dd bs=128k" or "dd bs=2MB": >> >>>>>> the 128kb readahead size will be used underneath to submit read IO. >> >> >>>>> Hmm... >> >> >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >> >>>>> 32768+0 enregistrements lus >> >>>>> 32768+0 enregistrements écrits >> >>>>> 4294967296 octets (4,3 GB) copiés, 20,7718 s, 207 MB/s >> >> >> >>>>> # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >> >>>>> 2048+0 enregistrements lus >> >>>>> 2048+0 enregistrements écrits >> >>>>> 4294967296 octets (4,3 GB) copiés, 27,7824 s, 155 MB/s >> >> >>>> Interesting. Here are my test results: >> >> >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=128k count=32768 >> >>>> 32768+0 records in >> >>>> 32768+0 records out >> >>>> 4294967296 bytes (4.3 GB) copied, 19.0121 s, 226 MB/s >> >>>> root@lkp-nex04 /home/wfg# echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sda of=/dev/null bs=2M count=2048 >> >>>> 2048+0 records in >> >>>> 2048+0 records out >> >>>> 4294967296 bytes (4.3 GB) copied, 19.0214 s, 226 MB/s >> >> >>>> Maybe the /dev/sda performance bug on your machine is sensitive to timing? >> >>> I got similar result: >> >>> 128k: 224M/s >> >>> 1M: 182M/s >> >> >>> 1M block size is slow, I guess it's CPU related. >> >> >>> And as for the big regression with newer kernel than 2.6.38, >> >>> please check if idle=poll helps. CPU idle dramatically impacts >> >>> disk performance and even latest cpuidle governor doesn't help >> >>> for some CPUs. >> >> >> here are the tests with idle=poll and after switching to 128k >> >> (instead of 1M) blocksize (same amount of data transferred) >> >> >> kernel ------------ read /dev/sda ------------- >> >> --- noop --- - deadline - ---- cfs --- >> >> [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU >> >> -------------------------------------------------- >> >> 3.2.2 45.82 3.7 44.85 3.6 45.04 3.4 >> >> 3.2.2i 45.59 2.3 51.78 2.6 46.03 2.2 >> >> 3.2.2i128 250.24 20.9 252.68 21.3 250.00 21.6 >> >> >> kernel -- write --- ------------------read ----------------- >> >> --- noop --- --- noop --- - deadline - ---- cfs --- >> >> [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU >> >> ---------------------------------------------------------------- >> >> 3.2.2 270.95 42.6 162.36 9.9 162.63 9.9 162.65 10.1 >> >> 3.2.2i 269.10 41.4 170.82 6.6 171.20 6.6 170.91 6.7 >> >> 3.2.2i128 270.38 67.7 162.35 10.2 163.01 10.3 162.34 10.7 >> >> > What's 3.2.2i and 3.2.2i128? >> >> 3.2.2 ...... kernel with default options (bs=1M) >> 3.2.2i ..... kernel with idle=poll (bs=1M) >> 3.2.2i128 .. kernel with idle=poll (bs=128k) >> >> > does idle=poll help? >> >> doesn't look like, at least to me ... > what's your /sys/block/sdx/queue/max_sectors_kb? if you make it smaller, > does the performance increase? In my system, a smaller max_sectors_kb > makes bs=2M and bs=128k have similar performance, which makes me think > it's CPU doesn't catch up quickly after a request finishes. Looks the 2.6.39 block plug introduces some latency here. deleting blk_start_plug/blk_finish_plug in generic_file_aio_read seems workaround the issue. The plug seems not good for sequential IO, because readahead code already has plug and has fine grained control. On the other hand, ondemand_readahead seems not handle the case that req_size is big well. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 10:31 ` Shaohua Li @ 2012-01-30 14:28 ` Wu Fengguang 2012-01-30 14:51 ` Eric Dumazet 2012-01-31 6:36 ` Herbert Poetzl 0 siblings, 2 replies; 36+ messages in thread From: Wu Fengguang @ 2012-01-30 14:28 UTC (permalink / raw) To: Shaohua Li Cc: Herbert Poetzl, Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: > Looks the 2.6.39 block plug introduces some latency here. deleting > blk_start_plug/blk_finish_plug in generic_file_aio_read seems > workaround > the issue. The plug seems not good for sequential IO, because readahead > code already has plug and has fine grained control. Why not remove the generic_file_aio_read() plug completely? It actually prevents unplugging immediately after the readahead IO is submitted and in turn stalls the IO pipeline as showed by Eric's blktrace data. Eric, will you test this patch? Thank you. --- linux.orig/mm/filemap.c 2012-01-30 22:27:11.000000000 +0800 +++ linux/mm/filemap.c 2012-01-30 22:27:25.000000000 +0800 @@ -1414,15 +1414,12 @@ generic_file_aio_read(struct kiocb *iocb unsigned long seg = 0; size_t count; loff_t *ppos = &iocb->ki_pos; - struct blk_plug plug; count = 0; retval = generic_segment_checks(iov, &nr_segs, &count, VERIFY_WRITE); if (retval) return retval; - blk_start_plug(&plug); - /* coalesce the iovecs and go direct-to-BIO for O_DIRECT */ if (filp->f_flags & O_DIRECT) { loff_t size; @@ -1495,7 +1492,6 @@ generic_file_aio_read(struct kiocb *iocb break; } out: - blk_finish_plug(&plug); return retval; } EXPORT_SYMBOL(generic_file_aio_read); ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 14:28 ` Wu Fengguang @ 2012-01-30 14:51 ` Eric Dumazet 2012-01-30 22:26 ` Vivek Goyal 2012-01-31 6:36 ` Herbert Poetzl 1 sibling, 1 reply; 36+ messages in thread From: Eric Dumazet @ 2012-01-30 14:51 UTC (permalink / raw) To: Wu Fengguang Cc: Shaohua Li, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit : > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: > > > Looks the 2.6.39 block plug introduces some latency here. deleting > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems > > workaround > > the issue. The plug seems not good for sequential IO, because readahead > > code already has plug and has fine grained control. > > Why not remove the generic_file_aio_read() plug completely? It > actually prevents unplugging immediately after the readahead IO is > submitted and in turn stalls the IO pipeline as showed by Eric's > blktrace data. > > Eric, will you test this patch? Thank you. > > --- linux.orig/mm/filemap.c 2012-01-30 22:27:11.000000000 +0800 > +++ linux/mm/filemap.c 2012-01-30 22:27:25.000000000 +0800 > @@ -1414,15 +1414,12 @@ generic_file_aio_read(struct kiocb *iocb > unsigned long seg = 0; > size_t count; > loff_t *ppos = &iocb->ki_pos; > - struct blk_plug plug; > > count = 0; > retval = generic_segment_checks(iov, &nr_segs, &count, VERIFY_WRITE); > if (retval) > return retval; > > - blk_start_plug(&plug); > - > /* coalesce the iovecs and go direct-to-BIO for O_DIRECT */ > if (filp->f_flags & O_DIRECT) { > loff_t size; > @@ -1495,7 +1492,6 @@ generic_file_aio_read(struct kiocb *iocb > break; > } > out: > - blk_finish_plug(&plug); > return retval; > } > EXPORT_SYMBOL(generic_file_aio_read); Excellent results # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=2M count=2048 2048+0 enregistrements lus 2048+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 16,2309 s, 265 MB/s # echo 3 >/proc/sys/vm/drop_caches ;dd if=/dev/sdb of=/dev/null bs=128k count=32768 32768+0 enregistrements lus 32768+0 enregistrements écrits 4294967296 octets (4,3 GB) copiés, 16,3153 s, 263 MB/s # hdparm -t /dev/sdb /dev/sdb: Timing buffered disk reads: 752 MB in 3.00 seconds = 250.62 MB/sec ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 14:51 ` Eric Dumazet @ 2012-01-30 22:26 ` Vivek Goyal 2012-01-31 0:14 ` Shaohua Li ` (2 more replies) 0 siblings, 3 replies; 36+ messages in thread From: Vivek Goyal @ 2012-01-30 22:26 UTC (permalink / raw) To: Eric Dumazet Cc: Wu Fengguang, Shaohua Li, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote: > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit : > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: > > > > > Looks the 2.6.39 block plug introduces some latency here. deleting > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems > > > workaround > > > the issue. The plug seems not good for sequential IO, because readahead > > > code already has plug and has fine grained control. > > > > Why not remove the generic_file_aio_read() plug completely? It > > actually prevents unplugging immediately after the readahead IO is > > submitted and in turn stalls the IO pipeline as showed by Eric's > > blktrace data. > > > > Eric, will you test this patch? Thank you. Can you please run the blktrace again with this patch applied. I am curious to see how does traffic pattern look like now. In your previous trace, there were so many small 8 sector requests which were merged into 512 sector requests before dispatching to disk. (I am not sure why those requests are not bigger. Shouldn't readahead logic submit a bigger request?) Now with plug/unplug logic removed, I am assuming we should be doing less merging and dispatching more smaller requests. May be that is helping and cutting down on disk idling time. In previous logs, 512 sector request seems to be taking around 1ms to complete after dispatch. In between requests disk seems to be idle for around .5 to .6 ms. Out of this .3 ms seems to be gone in just coming up with new request after completion of previous one and another .3ms seems to be consumed in merging the smaller IOs. So if we don't wait for merging, it should keep disk busier for .3ms more which is 30% of time it takes to complete 512 sector request. So theoritically it can give 30% boost for this workload. (Assuming request size will not impact the disk throughput very severely). Anyway, some blktrace data will shed some light.. Thanks Vivek ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 22:26 ` Vivek Goyal @ 2012-01-31 0:14 ` Shaohua Li 2012-01-31 1:07 ` Wu Fengguang 2012-01-31 2:17 ` Eric Dumazet 2012-01-31 8:46 ` Eric Dumazet 2 siblings, 1 reply; 36+ messages in thread From: Shaohua Li @ 2012-01-31 0:14 UTC (permalink / raw) To: Vivek Goyal Cc: Eric Dumazet, Wu Fengguang, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, 2012-01-30 at 17:26 -0500, Vivek Goyal wrote: > On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote: > > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit : > > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: > > > > > > > Looks the 2.6.39 block plug introduces some latency here. deleting > > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems > > > > workaround > > > > the issue. The plug seems not good for sequential IO, because readahead > > > > code already has plug and has fine grained control. > > > > > > Why not remove the generic_file_aio_read() plug completely? It > > > actually prevents unplugging immediately after the readahead IO is > > > submitted and in turn stalls the IO pipeline as showed by Eric's > > > blktrace data. > > > > > > Eric, will you test this patch? Thank you. > > Can you please run the blktrace again with this patch applied. I am curious > to see how does traffic pattern look like now. > > In your previous trace, there were so many small 8 sector requests which > were merged into 512 sector requests before dispatching to disk. (I am > not sure why those requests are not bigger. Shouldn't readahead logic > submit a bigger request?) Now with plug/unplug logic removed, I am assuming > we should be doing less merging and dispatching more smaller requests. May be > that is helping and cutting down on disk idling time. > > In previous logs, 512 sector request seems to be taking around 1ms to > complete after dispatch. In between requests disk seems to be idle > for around .5 to .6 ms. Out of this .3 ms seems to be gone in just > coming up with new request after completion of previous one and another > .3ms seems to be consumed in merging the smaller IOs. So if we don't wait > for merging, it should keep disk busier for .3ms more which is 30% of time > it takes to complete 512 sector request. So theoritically it can give > 30% boost for this workload. (Assuming request size will not impact the > disk throughput very severely). > > Anyway, some blktrace data will shed some light.. yep, I suspect plug merges big request too (iostat shows it too), that's why I only think delete the plug in generic_file_aio_read as a workaround. I still thought readahead has something to do here. I observed the async readahead does readahead (A, A + 2M), and follows (A +128k, A+2M), (A+256k, A+2M) ..., the later readahead doesn't work because we already have (A, A+2M) in memory at that time. Anyway, I can reproduce the issue, will play with it more today. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-31 0:14 ` Shaohua Li @ 2012-01-31 1:07 ` Wu Fengguang 2012-01-31 3:00 ` Shaohua Li 0 siblings, 1 reply; 36+ messages in thread From: Wu Fengguang @ 2012-01-31 1:07 UTC (permalink / raw) To: Shaohua Li Cc: Vivek Goyal, Eric Dumazet, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Tue, Jan 31, 2012 at 08:14:19AM +0800, Li, Shaohua wrote: > On Mon, 2012-01-30 at 17:26 -0500, Vivek Goyal wrote: > > On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote: > > > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit : > > > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: > > > > > > > > > Looks the 2.6.39 block plug introduces some latency here. deleting > > > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems > > > > > workaround > > > > > the issue. The plug seems not good for sequential IO, because readahead > > > > > code already has plug and has fine grained control. > > > > > > > > Why not remove the generic_file_aio_read() plug completely? It > > > > actually prevents unplugging immediately after the readahead IO is > > > > submitted and in turn stalls the IO pipeline as showed by Eric's > > > > blktrace data. > > > > > > > > Eric, will you test this patch? Thank you. > > > > Can you please run the blktrace again with this patch applied. I am curious > > to see how does traffic pattern look like now. > > > > In your previous trace, there were so many small 8 sector requests which > > were merged into 512 sector requests before dispatching to disk. (I am > > not sure why those requests are not bigger. Shouldn't readahead logic > > submit a bigger request?) Now with plug/unplug logic removed, I am assuming > > we should be doing less merging and dispatching more smaller requests. May be > > that is helping and cutting down on disk idling time. > > > > In previous logs, 512 sector request seems to be taking around 1ms to > > complete after dispatch. In between requests disk seems to be idle > > for around .5 to .6 ms. Out of this .3 ms seems to be gone in just > > coming up with new request after completion of previous one and another > > .3ms seems to be consumed in merging the smaller IOs. So if we don't wait > > for merging, it should keep disk busier for .3ms more which is 30% of time > > it takes to complete 512 sector request. So theoritically it can give > > 30% boost for this workload. (Assuming request size will not impact the > > disk throughput very severely). > > > > Anyway, some blktrace data will shed some light.. > yep, I suspect plug merges big request too (iostat shows it too), that's > why I only think delete the plug in generic_file_aio_read as a > workaround. It's good to merge requests inside the same readahead window. However I don't think readahead window A should be merged with B at the cost of delaying A for some time, which will break the pipeline. If larger IO is desirable, we can do so by increasing the readahead size. > I still thought readahead has something to do here. I > observed the async readahead does readahead (A, A + 2M), and follows (A > +128k, A+2M), (A+256k, A+2M) ..., the later readahead doesn't work > because we already have (A, A+2M) in memory at that time. Anyway, I can > reproduce the issue, will play with it more today. How do you observe that? I don't think that readahead pattern is possible. However I do see such _read_ patterns. Thanks, Fengguang ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-31 1:07 ` Wu Fengguang @ 2012-01-31 3:00 ` Shaohua Li 0 siblings, 0 replies; 36+ messages in thread From: Shaohua Li @ 2012-01-31 3:00 UTC (permalink / raw) To: Wu Fengguang Cc: Vivek Goyal, Eric Dumazet, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Tue, 2012-01-31 at 09:07 +0800, Wu Fengguang wrote: > On Tue, Jan 31, 2012 at 08:14:19AM +0800, Li, Shaohua wrote: > > On Mon, 2012-01-30 at 17:26 -0500, Vivek Goyal wrote: > > > On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote: > > > > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit : > > > > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: > > > > > > > > > > > Looks the 2.6.39 block plug introduces some latency here. deleting > > > > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems > > > > > > workaround > > > > > > the issue. The plug seems not good for sequential IO, because readahead > > > > > > code already has plug and has fine grained control. > > > > > > > > > > Why not remove the generic_file_aio_read() plug completely? It > > > > > actually prevents unplugging immediately after the readahead IO is > > > > > submitted and in turn stalls the IO pipeline as showed by Eric's > > > > > blktrace data. > > > > > > > > > > Eric, will you test this patch? Thank you. > > > > > > Can you please run the blktrace again with this patch applied. I am curious > > > to see how does traffic pattern look like now. > > > > > > In your previous trace, there were so many small 8 sector requests which > > > were merged into 512 sector requests before dispatching to disk. (I am > > > not sure why those requests are not bigger. Shouldn't readahead logic > > > submit a bigger request?) Now with plug/unplug logic removed, I am assuming > > > we should be doing less merging and dispatching more smaller requests. May be > > > that is helping and cutting down on disk idling time. > > > > > > In previous logs, 512 sector request seems to be taking around 1ms to > > > complete after dispatch. In between requests disk seems to be idle > > > for around .5 to .6 ms. Out of this .3 ms seems to be gone in just > > > coming up with new request after completion of previous one and another > > > .3ms seems to be consumed in merging the smaller IOs. So if we don't wait > > > for merging, it should keep disk busier for .3ms more which is 30% of time > > > it takes to complete 512 sector request. So theoritically it can give > > > 30% boost for this workload. (Assuming request size will not impact the > > > disk throughput very severely). > > > > > > Anyway, some blktrace data will shed some light.. > > yep, I suspect plug merges big request too (iostat shows it too), that's > > why I only think delete the plug in generic_file_aio_read as a > > workaround. > > It's good to merge requests inside the same readahead window. However > I don't think readahead window A should be merged with B at the cost > of delaying A for some time, which will break the pipeline. If larger > IO is desirable, we can do so by increasing the readahead size. > > > I still thought readahead has something to do here. I > > observed the async readahead does readahead (A, A + 2M), and follows (A > > +128k, A+2M), (A+256k, A+2M) ..., the later readahead doesn't work > > because we already have (A, A+2M) in memory at that time. Anyway, I can > > reproduce the issue, will play with it more today. > > How do you observe that? I don't think that readahead pattern is > possible. However I do see such _read_ patterns. Ok, after double checking the code and do some tracing, I'm now thinking we should delete the plug code in generic_file_aio_read. I thought the problem is: T1: ra (A, A+128k), (A+128k, A+256k), submit the 256k because lock_page T2: hit page A+128K, ra (A+256k, A+384). the range isn't submitted because of plug and there isn't any lock_page till we hit page A+256k because all pages from A to A+ 256k is in memory T3: hit page A+256k, ra (A+384, A+ 512). because plug, the range isn't submitted again. T4: lock_page A+256, so (A+256, A+512) will be submitted. The task is waitting for (A+256, A+512) finish so the pipeline doesn't work. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 22:26 ` Vivek Goyal 2012-01-31 0:14 ` Shaohua Li @ 2012-01-31 2:17 ` Eric Dumazet 2012-01-31 8:46 ` Eric Dumazet 2 siblings, 0 replies; 36+ messages in thread From: Eric Dumazet @ 2012-01-31 2:17 UTC (permalink / raw) To: Vivek Goyal Cc: Wu Fengguang, Shaohua Li, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo Le 30 janvier 2012 23:26, Vivek Goyal <vgoyal@redhat.com> a écrit : > > Can you please run the blktrace again with this patch applied. I am curious > to see how does traffic pattern look like now. > > In your previous trace, there were so many small 8 sector requests which > were merged into 512 sector requests before dispatching to disk. (I am > not sure why those requests are not bigger. Shouldn't readahead logic > submit a bigger request?) Now with plug/unplug logic removed, I am assuming > we should be doing less merging and dispatching more smaller requests. May be > that is helping and cutting down on disk idling time. > > In previous logs, 512 sector request seems to be taking around 1ms to > complete after dispatch. In between requests disk seems to be idle > for around .5 to .6 ms. Out of this .3 ms seems to be gone in just > coming up with new request after completion of previous one and another > .3ms seems to be consumed in merging the smaller IOs. So if we don't wait > for merging, it should keep disk busier for .3ms more which is 30% of time > it takes to complete 512 sector request. So theoritically it can give > 30% boost for this workload. (Assuming request size will not impact the > disk throughput very severely). > > Anyway, some blktrace data will shed some light.. > > Thanks > Vivek I'll do that when I have access to the machine again, in 6 hours or so :) Thanks ! ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 22:26 ` Vivek Goyal 2012-01-31 0:14 ` Shaohua Li 2012-01-31 2:17 ` Eric Dumazet @ 2012-01-31 8:46 ` Eric Dumazet 2 siblings, 0 replies; 36+ messages in thread From: Eric Dumazet @ 2012-01-31 8:46 UTC (permalink / raw) To: Vivek Goyal Cc: Wu Fengguang, Shaohua Li, Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo Le lundi 30 janvier 2012 à 17:26 -0500, Vivek Goyal a écrit : > On Mon, Jan 30, 2012 at 03:51:49PM +0100, Eric Dumazet wrote: > > Le lundi 30 janvier 2012 à 22:28 +0800, Wu Fengguang a écrit : > > > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: > > > > > > > Looks the 2.6.39 block plug introduces some latency here. deleting > > > > blk_start_plug/blk_finish_plug in generic_file_aio_read seems > > > > workaround > > > > the issue. The plug seems not good for sequential IO, because readahead > > > > code already has plug and has fine grained control. > > > > > > Why not remove the generic_file_aio_read() plug completely? It > > > actually prevents unplugging immediately after the readahead IO is > > > submitted and in turn stalls the IO pipeline as showed by Eric's > > > blktrace data. > > > > > > Eric, will you test this patch? Thank you. > > Can you please run the blktrace again with this patch applied. I am curious > to see how does traffic pattern look like now. > > In your previous trace, there were so many small 8 sector requests which > were merged into 512 sector requests before dispatching to disk. (I am > not sure why those requests are not bigger. Shouldn't readahead logic > submit a bigger request?) Now with plug/unplug logic removed, I am assuming > we should be doing less merging and dispatching more smaller requests. May be > that is helping and cutting down on disk idling time. > > In previous logs, 512 sector request seems to be taking around 1ms to > complete after dispatch. In between requests disk seems to be idle > for around .5 to .6 ms. Out of this .3 ms seems to be gone in just > coming up with new request after completion of previous one and another > .3ms seems to be consumed in merging the smaller IOs. So if we don't wait > for merging, it should keep disk busier for .3ms more which is 30% of time > it takes to complete 512 sector request. So theoritically it can give > 30% boost for this workload. (Assuming request size will not impact the > disk throughput very severely). > > Anyway, some blktrace data will shed some light.. Here it is (last 1000 lines) 8,16 1 1198126 9.992850217 2570 M R 5716720 + 8 [dd] 8,16 1 1198127 9.992851316 2570 Q R 5716728 + 8 [dd] 8,16 1 1198128 9.992851577 2570 M R 5716728 + 8 [dd] 8,16 1 1198129 9.992852592 2570 I R 5716480 + 256 ( 54889) [dd] 8,16 1 0 9.992853926 0 m N cfq2570 insert_request 8,16 1 1198130 9.992855759 2570 U N [dd] 1 8,16 1 0 9.992856488 0 m N cfq2570 dispatch_insert 8,16 1 0 9.992857801 0 m N cfq2570 dispatched a request 8,16 1 0 9.992858528 0 m N cfq2570 activate rq, drv=2 8,16 1 1198131 9.992858746 2570 D R 5716480 + 256 ( 6154) [dd] 8,16 1 1198132 9.993162912 0 C R 5716224 + 256 ( 783118) [0] 8,16 1 0 9.993195602 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198133 9.993294134 2570 Q R 5716736 + 8 [dd] 8,16 1 1198134 9.993299036 2570 G R 5716736 + 8 [dd] 8,16 1 1198135 9.993300482 2570 P N [dd] 8,16 1 1198136 9.993302456 2570 Q R 5716744 + 8 [dd] 8,16 1 1198137 9.993303374 2570 M R 5716744 + 8 [dd] 8,16 1 1198138 9.993305017 2570 Q R 5716752 + 8 [dd] 8,16 1 1198139 9.993305398 2570 M R 5716752 + 8 [dd] 8,16 1 1198140 9.993307106 2570 Q R 5716760 + 8 [dd] 8,16 1 1198141 9.993307428 2570 M R 5716760 + 8 [dd] 8,16 1 1198142 9.993308590 2570 Q R 5716768 + 8 [dd] 8,16 1 1198143 9.993308840 2570 M R 5716768 + 8 [dd] 8,16 1 1198144 9.993310021 2570 Q R 5716776 + 8 [dd] 8,16 1 1198145 9.993310276 2570 M R 5716776 + 8 [dd] 8,16 1 1198146 9.993311377 2570 Q R 5716784 + 8 [dd] 8,16 1 1198147 9.993311626 2570 M R 5716784 + 8 [dd] 8,16 1 1198148 9.993312717 2570 Q R 5716792 + 8 [dd] 8,16 1 1198149 9.993312967 2570 M R 5716792 + 8 [dd] 8,16 1 1198150 9.993314049 2570 Q R 5716800 + 8 [dd] 8,16 1 1198151 9.993314300 2570 M R 5716800 + 8 [dd] 8,16 1 1198152 9.993315494 2570 Q R 5716808 + 8 [dd] 8,16 1 1198153 9.993315748 2570 M R 5716808 + 8 [dd] 8,16 1 1198154 9.993316916 2570 Q R 5716816 + 8 [dd] 8,16 1 1198155 9.993317170 2570 M R 5716816 + 8 [dd] 8,16 1 1198156 9.993321418 2570 Q R 5716824 + 8 [dd] 8,16 1 1198157 9.993321668 2570 M R 5716824 + 8 [dd] 8,16 1 1198158 9.993322794 2570 Q R 5716832 + 8 [dd] 8,16 1 1198159 9.993323044 2570 M R 5716832 + 8 [dd] 8,16 1 1198160 9.993324133 2570 Q R 5716840 + 8 [dd] 8,16 1 1198161 9.993324388 2570 M R 5716840 + 8 [dd] 8,16 1 1198162 9.993325474 2570 Q R 5716848 + 8 [dd] 8,16 1 1198163 9.993325728 2570 M R 5716848 + 8 [dd] 8,16 1 1198164 9.993326816 2570 Q R 5716856 + 8 [dd] 8,16 1 1198165 9.993327066 2570 M R 5716856 + 8 [dd] 8,16 1 1198166 9.993328287 2570 Q R 5716864 + 8 [dd] 8,16 1 1198167 9.993328538 2570 M R 5716864 + 8 [dd] 8,16 1 1198168 9.993329630 2570 Q R 5716872 + 8 [dd] 8,16 1 1198169 9.993329884 2570 M R 5716872 + 8 [dd] 8,16 1 1198170 9.993331042 2570 Q R 5716880 + 8 [dd] 8,16 1 1198171 9.993331295 2570 M R 5716880 + 8 [dd] 8,16 1 1198172 9.993332426 2570 Q R 5716888 + 8 [dd] 8,16 1 1198173 9.993332676 2570 M R 5716888 + 8 [dd] 8,16 1 1198174 9.993333766 2570 Q R 5716896 + 8 [dd] 8,16 1 1198175 9.993334015 2570 M R 5716896 + 8 [dd] 8,16 1 1198176 9.993335359 2570 Q R 5716904 + 8 [dd] 8,16 1 1198177 9.993335611 2570 M R 5716904 + 8 [dd] 8,16 1 1198178 9.993336799 2570 Q R 5716912 + 8 [dd] 8,16 1 1198179 9.993337098 2570 M R 5716912 + 8 [dd] 8,16 1 1198180 9.993338295 2570 Q R 5716920 + 8 [dd] 8,16 1 1198181 9.993338545 2570 M R 5716920 + 8 [dd] 8,16 1 1198182 9.993339719 2570 Q R 5716928 + 8 [dd] 8,16 1 1198183 9.993339976 2570 M R 5716928 + 8 [dd] 8,16 1 1198184 9.993342740 2570 Q R 5716936 + 8 [dd] 8,16 1 1198185 9.993342995 2570 M R 5716936 + 8 [dd] 8,16 1 1198186 9.993344275 2570 Q R 5716944 + 8 [dd] 8,16 1 1198187 9.993344529 2570 M R 5716944 + 8 [dd] 8,16 1 1198188 9.993345637 2570 Q R 5716952 + 8 [dd] 8,16 1 1198189 9.993345886 2570 M R 5716952 + 8 [dd] 8,16 1 1198190 9.993346996 2570 Q R 5716960 + 8 [dd] 8,16 1 1198191 9.993347246 2570 M R 5716960 + 8 [dd] 8,16 1 1198192 9.993348322 2570 Q R 5716968 + 8 [dd] 8,16 1 1198193 9.993348722 2570 M R 5716968 + 8 [dd] 8,16 1 1198194 9.993349883 2570 Q R 5716976 + 8 [dd] 8,16 1 1198195 9.993350192 2570 M R 5716976 + 8 [dd] 8,16 1 1198196 9.993351388 2570 Q R 5716984 + 8 [dd] 8,16 1 1198197 9.993351638 2570 M R 5716984 + 8 [dd] 8,16 1 1198198 9.993352772 2570 I R 5716736 + 256 ( 53736) [dd] 8,16 1 0 9.993354172 0 m N cfq2570 insert_request 8,16 1 1198199 9.993356100 2570 U N [dd] 1 8,16 1 0 9.993356925 0 m N cfq2570 dispatch_insert 8,16 1 0 9.993358274 0 m N cfq2570 dispatched a request 8,16 1 0 9.993359003 0 m N cfq2570 activate rq, drv=2 8,16 1 1198200 9.993359283 2570 D R 5716736 + 256 ( 6511) [dd] 8,16 1 1198201 9.993652117 0 C R 5716480 + 256 ( 793371) [0] 8,16 1 0 9.993684544 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198202 9.993780104 2570 Q R 5716992 + 8 [dd] 8,16 1 1198203 9.993784668 2570 G R 5716992 + 8 [dd] 8,16 1 1198204 9.993786413 2570 P N [dd] 8,16 1 1198205 9.993788835 2570 Q R 5717000 + 8 [dd] 8,16 1 1198206 9.993789621 2570 M R 5717000 + 8 [dd] 8,16 1 1198207 9.993791535 2570 Q R 5717008 + 8 [dd] 8,16 1 1198208 9.993791785 2570 M R 5717008 + 8 [dd] 8,16 1 1198209 9.993792964 2570 Q R 5717016 + 8 [dd] 8,16 1 1198210 9.993793272 2570 M R 5717016 + 8 [dd] 8,16 1 1198211 9.993794443 2570 Q R 5717024 + 8 [dd] 8,16 1 1198212 9.993794759 2570 M R 5717024 + 8 [dd] 8,16 1 1198213 9.993795875 2570 Q R 5717032 + 8 [dd] 8,16 1 1198214 9.993796125 2570 M R 5717032 + 8 [dd] 8,16 1 1198215 9.993797251 2570 Q R 5717040 + 8 [dd] 8,16 1 1198216 9.993797499 2570 M R 5717040 + 8 [dd] 8,16 1 1198217 9.993801703 2570 Q R 5717048 + 8 [dd] 8,16 1 1198218 9.993802014 2570 M R 5717048 + 8 [dd] 8,16 1 1198219 9.993803167 2570 Q R 5717056 + 8 [dd] 8,16 1 1198220 9.993803422 2570 M R 5717056 + 8 [dd] 8,16 1 1198221 9.993804516 2570 Q R 5717064 + 8 [dd] 8,16 1 1198222 9.993804770 2570 M R 5717064 + 8 [dd] 8,16 1 1198223 9.993805960 2570 Q R 5717072 + 8 [dd] 8,16 1 1198224 9.993806211 2570 M R 5717072 + 8 [dd] 8,16 1 1198225 9.993807320 2570 Q R 5717080 + 8 [dd] 8,16 1 1198226 9.993807600 2570 M R 5717080 + 8 [dd] 8,16 1 1198227 9.993808914 2570 Q R 5717088 + 8 [dd] 8,16 1 1198228 9.993809176 2570 M R 5717088 + 8 [dd] 8,16 1 1198229 9.993810303 2570 Q R 5717096 + 8 [dd] 8,16 1 1198230 9.993810557 2570 M R 5717096 + 8 [dd] 8,16 1 1198231 9.993811689 2570 Q R 5717104 + 8 [dd] 8,16 1 1198232 9.993811941 2570 M R 5717104 + 8 [dd] 8,16 1 1198233 9.993813073 2570 Q R 5717112 + 8 [dd] 8,16 1 1198234 9.993813328 2570 M R 5717112 + 8 [dd] 8,16 1 1198235 9.993814430 2570 Q R 5717120 + 8 [dd] 8,16 1 1198236 9.993814685 2570 M R 5717120 + 8 [dd] 8,16 1 1198237 9.993815791 2570 Q R 5717128 + 8 [dd] 8,16 1 1198238 9.993816042 2570 M R 5717128 + 8 [dd] 8,16 1 1198239 9.993817225 2570 Q R 5717136 + 8 [dd] 8,16 1 1198240 9.993817492 2570 M R 5717136 + 8 [dd] 8,16 1 1198241 9.993818673 2570 Q R 5717144 + 8 [dd] 8,16 1 1198242 9.993818996 2570 M R 5717144 + 8 [dd] 8,16 1 1198243 9.993820108 2570 Q R 5717152 + 8 [dd] 8,16 1 1198244 9.993820426 2570 M R 5717152 + 8 [dd] 8,16 1 1198245 9.993823204 2570 Q R 5717160 + 8 [dd] 8,16 1 1198246 9.993823522 2570 M R 5717160 + 8 [dd] 8,16 1 1198247 9.993824646 2570 Q R 5717168 + 8 [dd] 8,16 1 1198248 9.993824912 2570 M R 5717168 + 8 [dd] 8,16 1 1198249 9.993826090 2570 Q R 5717176 + 8 [dd] 8,16 1 1198250 9.993826354 2570 M R 5717176 + 8 [dd] 8,16 1 1198251 9.993827454 2570 Q R 5717184 + 8 [dd] 8,16 1 1198252 9.993827709 2570 M R 5717184 + 8 [dd] 8,16 1 1198253 9.993828819 2570 Q R 5717192 + 8 [dd] 8,16 1 1198254 9.993829079 2570 M R 5717192 + 8 [dd] 8,16 1 1198255 9.993830271 2570 Q R 5717200 + 8 [dd] 8,16 1 1198256 9.993830537 2570 M R 5717200 + 8 [dd] 8,16 1 1198257 9.993831717 2570 Q R 5717208 + 8 [dd] 8,16 1 1198258 9.993831967 2570 M R 5717208 + 8 [dd] 8,16 1 1198259 9.993833624 2570 Q R 5717216 + 8 [dd] 8,16 1 1198260 9.993833887 2570 M R 5717216 + 8 [dd] 8,16 1 1198261 9.993835003 2570 Q R 5717224 + 8 [dd] 8,16 1 1198262 9.993835251 2570 M R 5717224 + 8 [dd] 8,16 1 1198263 9.993836361 2570 Q R 5717232 + 8 [dd] 8,16 1 1198264 9.993836609 2570 M R 5717232 + 8 [dd] 8,16 1 1198265 9.993837717 2570 Q R 5717240 + 8 [dd] 8,16 1 1198266 9.993837970 2570 M R 5717240 + 8 [dd] 8,16 1 1198267 9.993838830 2570 I R 5716992 + 256 ( 54162) [dd] 8,16 1 0 9.993840146 0 m N cfq2570 insert_request 8,16 1 1198268 9.993841920 2570 U N [dd] 1 8,16 1 0 9.993842806 0 m N cfq2570 dispatch_insert 8,16 1 0 9.993844092 0 m N cfq2570 dispatched a request 8,16 1 0 9.993844846 0 m N cfq2570 activate rq, drv=2 8,16 1 1198269 9.993845118 2570 D R 5716992 + 256 ( 6288) [dd] 8,16 1 1198270 9.994141624 0 C R 5716736 + 256 ( 782341) [0] 8,16 1 0 9.994174463 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198271 9.994266949 2570 Q R 5717248 + 8 [dd] 8,16 1 1198272 9.994271892 2570 G R 5717248 + 8 [dd] 8,16 1 1198273 9.994273520 2570 P N [dd] 8,16 1 1198274 9.994275563 2570 Q R 5717256 + 8 [dd] 8,16 1 1198275 9.994276389 2570 M R 5717256 + 8 [dd] 8,16 1 1198276 9.994278090 2570 Q R 5717264 + 8 [dd] 8,16 1 1198277 9.994278433 2570 M R 5717264 + 8 [dd] 8,16 1 1198278 9.994282797 2570 Q R 5717272 + 8 [dd] 8,16 1 1198279 9.994283057 2570 M R 5717272 + 8 [dd] 8,16 1 1198280 9.994284292 2570 Q R 5717280 + 8 [dd] 8,16 1 1198281 9.994284545 2570 M R 5717280 + 8 [dd] 8,16 1 1198282 9.994285642 2570 Q R 5717288 + 8 [dd] 8,16 1 1198283 9.994285903 2570 M R 5717288 + 8 [dd] 8,16 1 1198284 9.994287001 2570 Q R 5717296 + 8 [dd] 8,16 1 1198285 9.994287251 2570 M R 5717296 + 8 [dd] 8,16 1 1198286 9.994288341 2570 Q R 5717304 + 8 [dd] 8,16 1 1198287 9.994288593 2570 M R 5717304 + 8 [dd] 8,16 1 1198288 9.994289792 2570 Q R 5717312 + 8 [dd] 8,16 1 1198289 9.994290039 2570 M R 5717312 + 8 [dd] 8,16 1 1198290 9.994291131 2570 Q R 5717320 + 8 [dd] 8,16 1 1198291 9.994291381 2570 M R 5717320 + 8 [dd] 8,16 1 1198292 9.994292542 2570 Q R 5717328 + 8 [dd] 8,16 1 1198293 9.994292794 2570 M R 5717328 + 8 [dd] 8,16 1 1198294 9.994293886 2570 Q R 5717336 + 8 [dd] 8,16 1 1198295 9.994294136 2570 M R 5717336 + 8 [dd] 8,16 1 1198296 9.994295227 2570 Q R 5717344 + 8 [dd] 8,16 1 1198297 9.994295479 2570 M R 5717344 + 8 [dd] 8,16 1 1198298 9.994297088 2570 Q R 5717352 + 8 [dd] 8,16 1 1198299 9.994297338 2570 M R 5717352 + 8 [dd] 8,16 1 1198300 9.994298434 2570 Q R 5717360 + 8 [dd] 8,16 1 1198301 9.994298684 2570 M R 5717360 + 8 [dd] 8,16 1 1198302 9.994299804 2570 Q R 5717368 + 8 [dd] 8,16 1 1198303 9.994300063 2570 M R 5717368 + 8 [dd] 8,16 1 1198304 9.994301156 2570 Q R 5717376 + 8 [dd] 8,16 1 1198305 9.994301404 2570 M R 5717376 + 8 [dd] 8,16 1 1198306 9.994304103 2570 Q R 5717384 + 8 [dd] 8,16 1 1198307 9.994304352 2570 M R 5717384 + 8 [dd] 8,16 1 1198308 9.994305504 2570 Q R 5717392 + 8 [dd] 8,16 1 1198309 9.994305754 2570 M R 5717392 + 8 [dd] 8,16 1 1198310 9.994306851 2570 Q R 5717400 + 8 [dd] 8,16 1 1198311 9.994307101 2570 M R 5717400 + 8 [dd] 8,16 1 1198312 9.994308410 2570 Q R 5717408 + 8 [dd] 8,16 1 1198313 9.994308881 2570 M R 5717408 + 8 [dd] 8,16 1 1198314 9.994310006 2570 Q R 5717416 + 8 [dd] 8,16 1 1198315 9.994310270 2570 M R 5717416 + 8 [dd] 8,16 1 1198316 9.994311380 2570 Q R 5717424 + 8 [dd] 8,16 1 1198317 9.994311638 2570 M R 5717424 + 8 [dd] 8,16 1 1198318 9.994312743 2570 Q R 5717432 + 8 [dd] 8,16 1 1198319 9.994313050 2570 M R 5717432 + 8 [dd] 8,16 1 1198320 9.994314189 2570 Q R 5717440 + 8 [dd] 8,16 1 1198321 9.994314450 2570 M R 5717440 + 8 [dd] 8,16 1 1198322 9.994315584 2570 Q R 5717448 + 8 [dd] 8,16 1 1198323 9.994315833 2570 M R 5717448 + 8 [dd] 8,16 1 1198324 9.994316999 2570 Q R 5717456 + 8 [dd] 8,16 1 1198325 9.994317259 2570 M R 5717456 + 8 [dd] 8,16 1 1198326 9.994318364 2570 Q R 5717464 + 8 [dd] 8,16 1 1198327 9.994318612 2570 M R 5717464 + 8 [dd] 8,16 1 1198328 9.994319698 2570 Q R 5717472 + 8 [dd] 8,16 1 1198329 9.994319956 2570 M R 5717472 + 8 [dd] 8,16 1 1198330 9.994321048 2570 Q R 5717480 + 8 [dd] 8,16 1 1198331 9.994321298 2570 M R 5717480 + 8 [dd] 8,16 1 1198332 9.994322649 2570 Q R 5717488 + 8 [dd] 8,16 1 1198333 9.994322899 2570 M R 5717488 + 8 [dd] 8,16 1 1198334 9.994325719 2570 Q R 5717496 + 8 [dd] 8,16 1 1198335 9.994325970 2570 M R 5717496 + 8 [dd] 8,16 1 1198336 9.994326898 2570 I R 5717248 + 256 ( 55006) [dd] 8,16 1 0 9.994328230 0 m N cfq2570 insert_request 8,16 1 1198337 9.994329907 2570 U N [dd] 1 8,16 1 0 9.994330886 0 m N cfq2570 dispatch_insert 8,16 1 0 9.994332096 0 m N cfq2570 dispatched a request 8,16 1 0 9.994332828 0 m N cfq2570 activate rq, drv=2 8,16 1 1198338 9.994333050 2570 D R 5717248 + 256 ( 6152) [dd] 8,16 1 1198339 9.994629829 0 C R 5716992 + 256 ( 784711) [0] 8,16 1 0 9.994661464 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198340 9.994752149 2570 Q R 5717504 + 8 [dd] 8,16 1 1198341 9.994757116 2570 G R 5717504 + 8 [dd] 8,16 1 1198342 9.994758648 2570 P N [dd] 8,16 1 1198343 9.994760828 2570 Q R 5717512 + 8 [dd] 8,16 1 1198344 9.994761612 2570 M R 5717512 + 8 [dd] 8,16 1 1198345 9.994763312 2570 Q R 5717520 + 8 [dd] 8,16 1 1198346 9.994763571 2570 M R 5717520 + 8 [dd] 8,16 1 1198347 9.994764822 2570 Q R 5717528 + 8 [dd] 8,16 1 1198348 9.994765237 2570 M R 5717528 + 8 [dd] 8,16 1 1198349 9.994766430 2570 Q R 5717536 + 8 [dd] 8,16 1 1198350 9.994766680 2570 M R 5717536 + 8 [dd] 8,16 1 1198351 9.994767790 2570 Q R 5717544 + 8 [dd] 8,16 1 1198352 9.994768053 2570 M R 5717544 + 8 [dd] 8,16 1 1198353 9.994769187 2570 Q R 5717552 + 8 [dd] 8,16 1 1198354 9.994769438 2570 M R 5717552 + 8 [dd] 8,16 1 1198355 9.994770592 2570 Q R 5717560 + 8 [dd] 8,16 1 1198356 9.994770851 2570 M R 5717560 + 8 [dd] 8,16 1 1198357 9.994771968 2570 Q R 5717568 + 8 [dd] 8,16 1 1198358 9.994772218 2570 M R 5717568 + 8 [dd] 8,16 1 1198359 9.994773304 2570 Q R 5717576 + 8 [dd] 8,16 1 1198360 9.994773555 2570 M R 5717576 + 8 [dd] 8,16 1 1198361 9.994774719 2570 Q R 5717584 + 8 [dd] 8,16 1 1198362 9.994774982 2570 M R 5717584 + 8 [dd] 8,16 1 1198363 9.994776154 2570 Q R 5717592 + 8 [dd] 8,16 1 1198364 9.994776410 2570 M R 5717592 + 8 [dd] 8,16 1 1198365 9.994778214 2570 Q R 5717600 + 8 [dd] 8,16 1 1198366 9.994778471 2570 M R 5717600 + 8 [dd] 8,16 1 1198367 9.994783098 2570 Q R 5717608 + 8 [dd] 8,16 1 1198368 9.994783353 2570 M R 5717608 + 8 [dd] 8,16 1 1198369 9.994784517 2570 Q R 5717616 + 8 [dd] 8,16 1 1198370 9.994784769 2570 M R 5717616 + 8 [dd] 8,16 1 1198371 9.994785889 2570 Q R 5717624 + 8 [dd] 8,16 1 1198372 9.994786141 2570 M R 5717624 + 8 [dd] 8,16 1 1198373 9.994787248 2570 Q R 5717632 + 8 [dd] 8,16 1 1198374 9.994787498 2570 M R 5717632 + 8 [dd] 8,16 1 1198375 9.994788675 2570 Q R 5717640 + 8 [dd] 8,16 1 1198376 9.994788926 2570 M R 5717640 + 8 [dd] 8,16 1 1198377 9.994790174 2570 Q R 5717648 + 8 [dd] 8,16 1 1198378 9.994790429 2570 M R 5717648 + 8 [dd] 8,16 1 1198379 9.994791556 2570 Q R 5717656 + 8 [dd] 8,16 1 1198380 9.994791807 2570 M R 5717656 + 8 [dd] 8,16 1 1198381 9.994792970 2570 Q R 5717664 + 8 [dd] 8,16 1 1198382 9.994793220 2570 M R 5717664 + 8 [dd] 8,16 1 1198383 9.994794346 2570 Q R 5717672 + 8 [dd] 8,16 1 1198384 9.994794597 2570 M R 5717672 + 8 [dd] 8,16 1 1198385 9.994795687 2570 Q R 5717680 + 8 [dd] 8,16 1 1198386 9.994795938 2570 M R 5717680 + 8 [dd] 8,16 1 1198387 9.994797108 2570 Q R 5717688 + 8 [dd] 8,16 1 1198388 9.994797369 2570 M R 5717688 + 8 [dd] 8,16 1 1198389 9.994798465 2570 Q R 5717696 + 8 [dd] 8,16 1 1198390 9.994798715 2570 M R 5717696 + 8 [dd] 8,16 1 1198391 9.994799782 2570 Q R 5717704 + 8 [dd] 8,16 1 1198392 9.994800034 2570 M R 5717704 + 8 [dd] 8,16 1 1198393 9.994801267 2570 Q R 5717712 + 8 [dd] 8,16 1 1198394 9.994801518 2570 M R 5717712 + 8 [dd] 8,16 1 1198395 9.994804336 2570 Q R 5717720 + 8 [dd] 8,16 1 1198396 9.994804590 2570 M R 5717720 + 8 [dd] 8,16 1 1198397 9.994805730 2570 Q R 5717728 + 8 [dd] 8,16 1 1198398 9.994805980 2570 M R 5717728 + 8 [dd] 8,16 1 1198399 9.994807057 2570 Q R 5717736 + 8 [dd] 8,16 1 1198400 9.994807314 2570 M R 5717736 + 8 [dd] 8,16 1 1198401 9.994808391 2570 Q R 5717744 + 8 [dd] 8,16 1 1198402 9.994808642 2570 M R 5717744 + 8 [dd] 8,16 1 1198403 9.994809738 2570 Q R 5717752 + 8 [dd] 8,16 1 1198404 9.994809993 2570 M R 5717752 + 8 [dd] 8,16 1 1198405 9.994810844 2570 I R 5717504 + 256 ( 53728) [dd] 8,16 1 0 9.994812228 0 m N cfq2570 insert_request 8,16 1 1198406 9.994813845 2570 U N [dd] 1 8,16 1 0 9.994814579 0 m N cfq2570 dispatch_insert 8,16 1 0 9.994815804 0 m N cfq2570 dispatched a request 8,16 1 0 9.994816536 0 m N cfq2570 activate rq, drv=2 8,16 1 1198407 9.994816821 2570 D R 5717504 + 256 ( 5977) [dd] 8,16 1 1198408 9.995120966 0 C R 5717248 + 256 ( 787916) [0] 8,16 1 0 9.995152425 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198409 9.995588516 22 C R 5717504 + 256 ( 771695) [0] 8,16 1 0 9.995604694 0 m N cfq2570 complete rqnoidle 0 8,16 1 0 9.995605304 0 m N cfq schedule dispatch 8,16 1 0 9.995610896 0 m N cfq2570 slice expired t=0 8,16 1 0 9.995613323 0 m N cfq2570 sl_used=1 disp=14 charge=1 iops=0 sect=3584 8,16 1 0 9.995613922 0 m N cfq2570 del_from_rr 8,16 1 0 9.995615317 0 m N cfq workload slice:10 8,16 1 0 9.995616276 0 m N cfq2577 set_active wl_prio:0 wl_type:1 8,16 1 0 9.995617803 0 m N cfq2577 fifo= (null) 8,16 1 0 9.995618455 0 m N cfq2577 dispatch_insert 8,16 1 0 9.995620143 0 m N cfq2577 dispatched a request 8,16 1 0 9.995620870 0 m N cfq2577 activate rq, drv=1 8,16 1 1198410 9.995621255 22 D R 127095467 + 8 ( 6830688) [kswapd0] 8,16 1 0 9.995632325 0 m N cfq2577 dispatch_insert 8,16 1 0 9.995633106 0 m N cfq2577 dispatched a request 8,16 1 0 9.995633611 0 m N cfq2577 activate rq, drv=2 8,16 1 1198411 9.995633847 22 D R 127100683 + 24 ( 6841571) [kswapd0] 8,16 1 1198412 9.995862020 8 C R 127095467 + 8 ( 240765) [0] 8,16 1 0 9.995866553 0 m N cfq2577 complete rqnoidle 0 8,16 1 0 9.995867340 0 m N cfq2577 set_slice=10 8,16 1 1198413 9.995909670 2577 C R 127100683 + 24 ( 275823) [0] 8,16 1 0 9.995911822 0 m N cfq2577 complete rqnoidle 0 8,16 1 0 9.995912219 0 m N cfq schedule dispatch 8,16 1 1198414 9.995935586 2577 A R 127109795 + 8 <- (8,21) 52503872 8,16 1 1198415 9.995936258 2577 Q R 127109795 + 8 [evolution] 8,16 1 1198416 9.995940062 2577 G R 127109795 + 8 [evolution] 8,16 1 1198417 9.995941861 2577 P N [evolution] 8,16 1 1198418 9.995943395 2577 I R 127109795 + 8 ( 3333) [evolution] 8,16 1 0 9.995944417 0 m N cfq2577 insert_request 8,16 1 1198419 9.995945545 2577 U N [evolution] 1 8,16 1 0 9.995946172 0 m N cfq2577 dispatch_insert 8,16 1 0 9.995946905 0 m N cfq2577 dispatched a request 8,16 1 0 9.995947455 0 m N cfq2577 activate rq, drv=1 8,16 1 1198420 9.995947618 2577 D R 127109795 + 8 ( 4223) [evolution] 8,16 1 1198421 9.995960951 2570 Q R 5717760 + 8 [dd] 8,16 1 1198422 9.995961927 2570 G R 5717760 + 8 [dd] 8,16 1 1198423 9.995962173 2570 P N [dd] 8,16 1 1198424 9.995963514 2570 Q R 5717768 + 8 [dd] 8,16 1 1198425 9.995964687 2570 M R 5717768 + 8 [dd] 8,16 1 1198426 9.995966344 2570 Q R 5717776 + 8 [dd] 8,16 1 1198427 9.995966648 2570 M R 5717776 + 8 [dd] 8,16 1 1198428 9.995967874 2570 Q R 5717784 + 8 [dd] 8,16 1 1198429 9.995968126 2570 M R 5717784 + 8 [dd] 8,16 1 1198430 9.995969233 2570 Q R 5717792 + 8 [dd] 8,16 1 1198431 9.995969483 2570 M R 5717792 + 8 [dd] 8,16 1 1198432 9.995970602 2570 Q R 5717800 + 8 [dd] 8,16 1 1198433 9.995970852 2570 M R 5717800 + 8 [dd] 8,16 1 1198434 9.995971949 2570 Q R 5717808 + 8 [dd] 8,16 1 1198435 9.995972201 2570 M R 5717808 + 8 [dd] 8,16 1 1198436 9.995973299 2570 Q R 5717816 + 8 [dd] 8,16 1 1198437 9.995973559 2570 M R 5717816 + 8 [dd] 8,16 1 1198438 9.995974658 2570 Q R 5717824 + 8 [dd] 8,16 1 1198439 9.995974907 2570 M R 5717824 + 8 [dd] 8,16 1 1198440 9.995976012 2570 Q R 5717832 + 8 [dd] 8,16 1 1198441 9.995976263 2570 M R 5717832 + 8 [dd] 8,16 1 1198442 9.995977424 2570 Q R 5717840 + 8 [dd] 8,16 1 1198443 9.995977675 2570 M R 5717840 + 8 [dd] 8,16 1 1198444 9.995978776 2570 Q R 5717848 + 8 [dd] 8,16 1 1198445 9.995979026 2570 M R 5717848 + 8 [dd] 8,16 1 1198446 9.995982184 2570 Q R 5717856 + 8 [dd] 8,16 1 1198447 9.995982436 2570 M R 5717856 + 8 [dd] 8,16 1 1198448 9.995983606 2570 Q R 5717864 + 8 [dd] 8,16 1 1198449 9.995983856 2570 M R 5717864 + 8 [dd] 8,16 1 1198450 9.995984988 2570 Q R 5717872 + 8 [dd] 8,16 1 1198451 9.995985241 2570 M R 5717872 + 8 [dd] 8,16 1 1198452 9.995986350 2570 Q R 5717880 + 8 [dd] 8,16 1 1198453 9.995986600 2570 M R 5717880 + 8 [dd] 8,16 1 1198454 9.995987699 2570 Q R 5717888 + 8 [dd] 8,16 1 1198455 9.995987949 2570 M R 5717888 + 8 [dd] 8,16 1 1198456 9.995989053 2570 Q R 5717896 + 8 [dd] 8,16 1 1198457 9.995989303 2570 M R 5717896 + 8 [dd] 8,16 1 1198458 9.995991031 2570 Q R 5717904 + 8 [dd] 8,16 1 1198459 9.995991282 2570 M R 5717904 + 8 [dd] 8,16 1 1198460 9.995992402 2570 Q R 5717912 + 8 [dd] 8,16 1 1198461 9.995992665 2570 M R 5717912 + 8 [dd] 8,16 1 1198462 9.995993783 2570 Q R 5717920 + 8 [dd] 8,16 1 1198463 9.995994049 2570 M R 5717920 + 8 [dd] 8,16 1 1198464 9.995995194 2570 Q R 5717928 + 8 [dd] 8,16 1 1198465 9.995995453 2570 M R 5717928 + 8 [dd] 8,16 1 1198466 9.995996565 2570 Q R 5717936 + 8 [dd] 8,16 1 1198467 9.995996814 2570 M R 5717936 + 8 [dd] 8,16 1 1198468 9.995998022 2570 Q R 5717944 + 8 [dd] 8,16 1 1198469 9.995998269 2570 M R 5717944 + 8 [dd] 8,16 1 1198470 9.995999501 2570 Q R 5717952 + 8 [dd] 8,16 1 1198471 9.995999755 2570 M R 5717952 + 8 [dd] 8,16 1 1198472 9.996000869 2570 Q R 5717960 + 8 [dd] 8,16 1 1198473 9.996001119 2570 M R 5717960 + 8 [dd] 8,16 1 1198474 9.996003860 2570 Q R 5717968 + 8 [dd] 8,16 1 1198475 9.996004109 2570 M R 5717968 + 8 [dd] 8,16 1 1198476 9.996005214 2570 Q R 5717976 + 8 [dd] 8,16 1 1198477 9.996005464 2570 M R 5717976 + 8 [dd] 8,16 1 1198478 9.996006583 2570 Q R 5717984 + 8 [dd] 8,16 1 1198479 9.996006837 2570 M R 5717984 + 8 [dd] 8,16 1 1198480 9.996007952 2570 Q R 5717992 + 8 [dd] 8,16 1 1198481 9.996008205 2570 M R 5717992 + 8 [dd] 8,16 1 1198482 9.996009328 2570 Q R 5718000 + 8 [dd] 8,16 1 1198483 9.996009580 2570 M R 5718000 + 8 [dd] 8,16 1 1198484 9.996010703 2570 Q R 5718008 + 8 [dd] 8,16 1 1198485 9.996010953 2570 M R 5718008 + 8 [dd] 8,16 1 1198486 9.996011362 2570 I R 5717760 + 256 ( 49435) [dd] 8,16 1 0 9.996011940 0 m N cfq2570 insert_request 8,16 1 0 9.996012410 0 m N cfq2570 add_to_rr 8,16 1 1198487 9.996013350 2570 U N [dd] 1 8,16 1 1198488 9.996076425 2570 Q R 5718016 + 8 [dd] 8,16 1 1198489 9.996076900 2570 M R 5718016 + 8 [dd] 8,16 1 1198490 9.996079343 2570 Q R 5718024 + 8 [dd] 8,16 1 1198491 9.996079716 2570 M R 5718024 + 8 [dd] 8,16 1 1198492 9.996081166 2570 Q R 5718032 + 8 [dd] 8,16 1 1198493 9.996081531 2570 M R 5718032 + 8 [dd] 8,16 1 1198494 9.996082875 2570 Q R 5718040 + 8 [dd] 8,16 1 1198495 9.996083254 2570 M R 5718040 + 8 [dd] 8,16 1 1198496 9.996084649 2570 Q R 5718048 + 8 [dd] 8,16 1 1198497 9.996085005 2570 M R 5718048 + 8 [dd] 8,16 1 1198498 9.996086309 2570 Q R 5718056 + 8 [dd] 8,16 1 1198499 9.996086676 2570 M R 5718056 + 8 [dd] 8,16 1 1198500 9.996087981 2570 Q R 5718064 + 8 [dd] 8,16 1 1198501 9.996088337 2570 M R 5718064 + 8 [dd] 8,16 1 1198502 9.996089538 2570 Q R 5718072 + 8 [dd] 8,16 1 1198503 9.996089903 2570 M R 5718072 + 8 [dd] 8,16 1 1198504 9.996092903 2570 Q R 5718080 + 8 [dd] 8,16 1 1198505 9.996093272 2570 M R 5718080 + 8 [dd] 8,16 1 1198506 9.996094544 2570 Q R 5718088 + 8 [dd] 8,16 1 1198507 9.996094978 2570 M R 5718088 + 8 [dd] 8,16 1 1198508 9.996096219 2570 Q R 5718096 + 8 [dd] 8,16 1 1198509 9.996096611 2570 M R 5718096 + 8 [dd] 8,16 1 1198510 9.996097834 2570 Q R 5718104 + 8 [dd] 8,16 1 1198511 9.996098216 2570 M R 5718104 + 8 [dd] 8,16 1 1198512 9.996099625 2570 Q R 5718112 + 8 [dd] 8,16 1 1198513 9.996099987 2570 M R 5718112 + 8 [dd] 8,16 1 1198514 9.996101339 2570 Q R 5718120 + 8 [dd] 8,16 1 1198515 9.996101707 2570 M R 5718120 + 8 [dd] 8,16 1 1198516 9.996102946 2570 Q R 5718128 + 8 [dd] 8,16 1 1198517 9.996103302 2570 M R 5718128 + 8 [dd] 8,16 1 1198518 9.996104534 2570 Q R 5718136 + 8 [dd] 8,16 1 1198519 9.996104906 2570 M R 5718136 + 8 [dd] 8,16 1 1198520 9.996106127 2570 Q R 5718144 + 8 [dd] 8,16 1 1198521 9.996106482 2570 M R 5718144 + 8 [dd] 8,16 1 1198522 9.996107704 2570 Q R 5718152 + 8 [dd] 8,16 1 1198523 9.996108068 2570 M R 5718152 + 8 [dd] 8,16 1 1198524 9.996109289 2570 Q R 5718160 + 8 [dd] 8,16 1 1198525 9.996109649 2570 M R 5718160 + 8 [dd] 8,16 1 1198526 9.996110859 2570 Q R 5718168 + 8 [dd] 8,16 1 1198527 9.996111227 2570 M R 5718168 + 8 [dd] 8,16 1 1198528 9.996112546 2570 Q R 5718176 + 8 [dd] 8,16 1 1198529 9.996112904 2570 M R 5718176 + 8 [dd] 8,16 1 1198530 9.996114125 2570 Q R 5718184 + 8 [dd] 8,16 1 1198531 9.996114494 2570 M R 5718184 + 8 [dd] 8,16 1 1198532 9.996117368 2570 Q R 5718192 + 8 [dd] 8,16 1 1198533 9.996117752 2570 M R 5718192 + 8 [dd] 8,16 1 1198534 9.996119006 2570 Q R 5718200 + 8 [dd] 8,16 1 1198535 9.996119376 2570 M R 5718200 + 8 [dd] 8,16 1 1198536 9.996120984 2570 Q R 5718208 + 8 [dd] 8,16 1 1198537 9.996121340 2570 M R 5718208 + 8 [dd] 8,16 1 1198538 9.996122560 2570 Q R 5718216 + 8 [dd] 8,16 1 1198539 9.996122928 2570 M R 5718216 + 8 [dd] 8,16 1 1198540 9.996124256 2570 Q R 5718224 + 8 [dd] 8,16 1 1198541 9.996124613 2570 M R 5718224 + 8 [dd] 8,16 1 1198542 9.996125840 2570 Q R 5718232 + 8 [dd] 8,16 1 1198543 9.996126205 2570 M R 5718232 + 8 [dd] 8,16 1 1198544 9.996127439 2570 Q R 5718240 + 8 [dd] 8,16 1 1198545 9.996127795 2570 M R 5718240 + 8 [dd] 8,16 1 1198546 9.996129008 2570 Q R 5718248 + 8 [dd] 8,16 1 1198547 9.996129368 2570 M R 5718248 + 8 [dd] 8,16 1 1198548 9.996130621 2570 Q R 5718256 + 8 [dd] 8,16 1 1198549 9.996130977 2570 M R 5718256 + 8 [dd] 8,16 1 1198550 9.996132180 2570 Q R 5718264 + 8 [dd] 8,16 1 1198551 9.996132629 2570 M R 5718264 + 8 [dd] 8,16 1 1198552 9.996172197 0 C R 127109795 + 8 ( 224579) [0] 8,16 1 0 9.996175466 0 m N cfq2577 complete rqnoidle 0 8,16 1 0 9.996175960 0 m N cfq schedule dispatch 8,16 1 0 9.996178896 0 m N cfq2577 slice expired t=0 8,16 1 0 9.996180225 0 m N cfq2577 sl_used=1 disp=3 charge=1 iops=0 sect=40 8,16 1 0 9.996180604 0 m N cfq2577 del_from_rr 8,16 1 0 9.996181202 0 m N cfq workload slice:10 8,16 1 0 9.996181922 0 m N cfq2570 set_active wl_prio:0 wl_type:2 8,16 1 0 9.996182708 0 m N cfq2570 fifo= (null) 8,16 1 0 9.996183077 0 m N cfq2570 dispatch_insert 8,16 1 0 9.996183707 0 m N cfq2570 dispatched a request 8,16 1 0 9.996184185 0 m N cfq2570 activate rq, drv=1 8,16 1 1198553 9.996184373 0 D R 5717760 + 512 ( 173011) [swapper/1] 8,16 1 1198554 9.996216977 2577 A R 127110683 + 8 <- (8,21) 52504760 8,16 1 1198555 9.996217210 2577 Q R 127110683 + 8 [evolution] 8,16 1 1198556 9.996218418 2577 G R 127110683 + 8 [evolution] 8,16 1 1198557 9.996219232 2577 P N [evolution] 8,16 1 1198558 9.996219820 2577 I R 127110683 + 8 ( 1402) [evolution] 8,16 1 0 9.996220459 0 m N cfq2577 insert_request 8,16 1 0 9.996220927 0 m N cfq2577 add_to_rr 8,16 1 1198559 9.996221556 2577 U N [evolution] 1 8,16 1 1198560 9.997338818 0 C R 5717760 + 512 ( 1154445) [0] 8,16 1 0 9.997385039 0 m N cfq2570 complete rqnoidle 0 8,16 1 0 9.997386457 0 m N cfq2570 set_slice=10 8,16 1 0 9.997387543 0 m N cfq schedule dispatch 8,16 1 0 9.997396551 0 m N cfq2570 slice expired t=0 8,16 1 0 9.997398888 0 m N cfq2570 sl_used=1 disp=1 charge=1 iops=0 sect=512 8,16 1 0 9.997399487 0 m N cfq2570 del_from_rr 8,16 1 0 9.997400823 0 m N cfq workload slice:10 8,16 1 0 9.997401758 0 m N cfq2577 set_active wl_prio:0 wl_type:1 8,16 1 0 9.997403340 0 m N cfq2577 fifo= (null) 8,16 1 0 9.997403945 0 m N cfq2577 dispatch_insert 8,16 1 0 9.997405661 0 m N cfq2577 dispatched a request 8,16 1 0 9.997406474 0 m N cfq2577 activate rq, drv=1 8,16 1 1198561 9.997406966 0 D R 127110683 + 8 ( 1187146) [swapper/1] 8,16 1 1198562 9.997512108 2570 Q R 5718272 + 8 [dd] 8,16 1 1198563 9.997516538 2570 G R 5718272 + 8 [dd] 8,16 1 1198564 9.997517923 2570 P N [dd] 8,16 1 1198565 9.997519958 2570 Q R 5718280 + 8 [dd] 8,16 1 1198566 9.997520911 2570 M R 5718280 + 8 [dd] 8,16 1 1198567 9.997522452 2570 Q R 5718288 + 8 [dd] 8,16 1 1198568 9.997522707 2570 M R 5718288 + 8 [dd] 8,16 1 1198569 9.997527866 2570 Q R 5718296 + 8 [dd] 8,16 1 1198570 9.997528118 2570 M R 5718296 + 8 [dd] 8,16 1 1198571 9.997529249 2570 Q R 5718304 + 8 [dd] 8,16 1 1198572 9.997529501 2570 M R 5718304 + 8 [dd] 8,16 1 1198573 9.997530720 2570 Q R 5718312 + 8 [dd] 8,16 1 1198574 9.997531017 2570 M R 5718312 + 8 [dd] 8,16 1 1198575 9.997532275 2570 Q R 5718320 + 8 [dd] 8,16 1 1198576 9.997532591 2570 M R 5718320 + 8 [dd] 8,16 1 1198577 9.997533778 2570 Q R 5718328 + 8 [dd] 8,16 1 1198578 9.997534031 2570 M R 5718328 + 8 [dd] 8,16 1 1198579 9.997535144 2570 Q R 5718336 + 8 [dd] 8,16 1 1198580 9.997535394 2570 M R 5718336 + 8 [dd] 8,16 1 1198581 9.997536544 2570 Q R 5718344 + 8 [dd] 8,16 1 1198582 9.997536795 2570 M R 5718344 + 8 [dd] 8,16 1 1198583 9.997537867 2570 Q R 5718352 + 8 [dd] 8,16 1 1198584 9.997538124 2570 M R 5718352 + 8 [dd] 8,16 1 1198585 9.997539210 2570 Q R 5718360 + 8 [dd] 8,16 1 1198586 9.997539468 2570 M R 5718360 + 8 [dd] 8,16 1 1198587 9.997540967 2570 Q R 5718368 + 8 [dd] 8,16 1 1198588 9.997541218 2570 M R 5718368 + 8 [dd] 8,16 1 1198589 9.997542384 2570 Q R 5718376 + 8 [dd] 8,16 1 1198590 9.997542636 2570 M R 5718376 + 8 [dd] 8,16 1 1198591 9.997543814 2570 Q R 5718384 + 8 [dd] 8,16 1 1198592 9.997544080 2570 M R 5718384 + 8 [dd] 8,16 1 1198593 9.997545150 2570 Q R 5718392 + 8 [dd] 8,16 1 1198594 9.997545401 2570 M R 5718392 + 8 [dd] 8,16 1 1198595 9.997546510 2570 Q R 5718400 + 8 [dd] 8,16 1 1198596 9.997546761 2570 M R 5718400 + 8 [dd] 8,16 1 1198597 9.997549903 2570 Q R 5718408 + 8 [dd] 8,16 1 1198598 9.997550193 2570 M R 5718408 + 8 [dd] 8,16 1 1198599 9.997551289 2570 Q R 5718416 + 8 [dd] 8,16 1 1198600 9.997551565 2570 M R 5718416 + 8 [dd] 8,16 1 1198601 9.997552842 2570 Q R 5718424 + 8 [dd] 8,16 1 1198602 9.997553106 2570 M R 5718424 + 8 [dd] 8,16 1 1198603 9.997554292 2570 Q R 5718432 + 8 [dd] 8,16 1 1198604 9.997554548 2570 M R 5718432 + 8 [dd] 8,16 1 1198605 9.997555761 2570 Q R 5718440 + 8 [dd] 8,16 1 1198606 9.997556013 2570 M R 5718440 + 8 [dd] 8,16 1 1198607 9.997557189 2570 Q R 5718448 + 8 [dd] 8,16 1 1198608 9.997557487 2570 M R 5718448 + 8 [dd] 8,16 1 1198609 9.997558670 2570 Q R 5718456 + 8 [dd] 8,16 1 1198610 9.997558931 2570 M R 5718456 + 8 [dd] 8,16 1 1198611 9.997560047 2570 Q R 5718464 + 8 [dd] 8,16 1 1198612 9.997560298 2570 M R 5718464 + 8 [dd] 8,16 1 1198613 9.997561453 2570 Q R 5718472 + 8 [dd] 8,16 1 1198614 9.997561705 2570 M R 5718472 + 8 [dd] 8,16 1 1198615 9.997562776 2570 Q R 5718480 + 8 [dd] 8,16 1 1198616 9.997563032 2570 M R 5718480 + 8 [dd] 8,16 1 1198617 9.997564186 2570 Q R 5718488 + 8 [dd] 8,16 1 1198618 9.997564435 2570 M R 5718488 + 8 [dd] 8,16 1 1198619 9.997565541 2570 Q R 5718496 + 8 [dd] 8,16 1 1198620 9.997565792 2570 M R 5718496 + 8 [dd] 8,16 1 1198621 9.997566979 2570 Q R 5718504 + 8 [dd] 8,16 1 1198622 9.997567231 2570 M R 5718504 + 8 [dd] 8,16 1 1198623 9.997568327 2570 Q R 5718512 + 8 [dd] 8,16 1 1198624 9.997568582 2570 M R 5718512 + 8 [dd] 8,16 1 1198625 9.997571309 2570 Q R 5718520 + 8 [dd] 8,16 1 1198626 9.997571563 2570 M R 5718520 + 8 [dd] 8,16 1 1198627 9.997572355 2570 I R 5718272 + 256 ( 55817) [dd] 8,16 1 0 9.997573394 0 m N cfq2570 insert_request 8,16 1 0 9.997574036 0 m N cfq2570 add_to_rr 8,16 1 1198628 9.997575577 2570 U N [dd] 1 8,16 1 1198629 9.997613973 2570 C R 127110683 + 8 ( 207007) [0] 8,16 1 0 9.997619361 0 m N cfq2577 complete rqnoidle 0 8,16 1 0 9.997619940 0 m N cfq2577 set_slice=10 8,16 1 0 9.997620243 0 m N cfq schedule dispatch 8,16 1 0 9.997623117 0 m N cfq2577 slice expired t=0 8,16 1 0 9.997624406 0 m N cfq2577 sl_used=1 disp=1 charge=1 iops=0 sect=8 8,16 1 0 9.997624813 0 m N cfq2577 del_from_rr 8,16 1 0 9.997625410 0 m N cfq workload slice:10 8,16 1 0 9.997626090 0 m N cfq2570 set_active wl_prio:0 wl_type:2 8,16 1 0 9.997626764 0 m N cfq2570 fifo= (null) 8,16 1 0 9.997627143 0 m N cfq2570 dispatch_insert 8,16 1 0 9.997627812 0 m N cfq2570 dispatched a request 8,16 1 0 9.997628296 0 m N cfq2570 activate rq, drv=1 8,16 1 1198630 9.997628489 2570 D R 5718272 + 256 ( 56134) [dd] 8,16 1 1198631 9.997681693 2577 A R 129160427 + 8 <- (8,21) 54554504 8,16 1 1198632 9.997681939 2577 Q R 129160427 + 8 [evolution] 8,16 1 1198633 9.997683227 2577 G R 129160427 + 8 [evolution] 8,16 1 1198634 9.997683956 2577 P N [evolution] 8,16 1 1198635 9.997684797 2577 I R 129160427 + 8 ( 1570) [evolution] 8,16 1 0 9.997685448 0 m N cfq2577 insert_request 8,16 1 0 9.997685951 0 m N cfq2577 add_to_rr 8,16 1 1198636 9.997686524 2577 U N [evolution] 1 8,16 1 1198637 9.997725992 2570 Q R 5718528 + 8 [dd] 8,16 1 1198638 9.997727282 2570 G R 5718528 + 8 [dd] 8,16 1 1198639 9.997727553 2570 P N [dd] 8,16 1 1198640 9.997729012 2570 Q R 5718536 + 8 [dd] 8,16 1 1198641 9.997729322 2570 M R 5718536 + 8 [dd] 8,16 1 1198642 9.997730493 2570 Q R 5718544 + 8 [dd] 8,16 1 1198643 9.997730813 2570 M R 5718544 + 8 [dd] 8,16 1 1198644 9.997731992 2570 Q R 5718552 + 8 [dd] 8,16 1 1198645 9.997732245 2570 M R 5718552 + 8 [dd] 8,16 1 1198646 9.997733320 2570 Q R 5718560 + 8 [dd] 8,16 1 1198647 9.997733578 2570 M R 5718560 + 8 [dd] 8,16 1 1198648 9.997734835 2570 Q R 5718568 + 8 [dd] 8,16 1 1198649 9.997735088 2570 M R 5718568 + 8 [dd] 8,16 1 1198650 9.997736172 2570 Q R 5718576 + 8 [dd] 8,16 1 1198651 9.997736425 2570 M R 5718576 + 8 [dd] 8,16 1 1198652 9.997737500 2570 Q R 5718584 + 8 [dd] 8,16 1 1198653 9.997737752 2570 M R 5718584 + 8 [dd] 8,16 1 1198654 9.997738832 2570 Q R 5718592 + 8 [dd] 8,16 1 1198655 9.997739092 2570 M R 5718592 + 8 [dd] 8,16 1 1198656 9.997740163 2570 Q R 5718600 + 8 [dd] 8,16 1 1198657 9.997740509 2570 M R 5718600 + 8 [dd] 8,16 1 1198658 9.997741600 2570 Q R 5718608 + 8 [dd] 8,16 1 1198659 9.997741918 2570 M R 5718608 + 8 [dd] 8,16 1 1198660 9.997743063 2570 Q R 5718616 + 8 [dd] 8,16 1 1198661 9.997743316 2570 M R 5718616 + 8 [dd] 8,16 1 1198662 9.997746451 2570 Q R 5718624 + 8 [dd] 8,16 1 1198663 9.997746707 2570 M R 5718624 + 8 [dd] 8,16 1 1198664 9.997747824 2570 Q R 5718632 + 8 [dd] 8,16 1 1198665 9.997748076 2570 M R 5718632 + 8 [dd] 8,16 1 1198666 9.997749317 2570 Q R 5718640 + 8 [dd] 8,16 1 1198667 9.997749567 2570 M R 5718640 + 8 [dd] 8,16 1 1198668 9.997750656 2570 Q R 5718648 + 8 [dd] 8,16 1 1198669 9.997750934 2570 M R 5718648 + 8 [dd] 8,16 1 1198670 9.997752046 2570 Q R 5718656 + 8 [dd] 8,16 1 1198671 9.997752301 2570 M R 5718656 + 8 [dd] 8,16 1 1198672 9.997753442 2570 Q R 5718664 + 8 [dd] 8,16 1 1198673 9.997753694 2570 M R 5718664 + 8 [dd] 8,16 1 1198674 9.997754778 2570 Q R 5718672 + 8 [dd] 8,16 1 1198675 9.997755031 2570 M R 5718672 + 8 [dd] 8,16 1 1198676 9.997756129 2570 Q R 5718680 + 8 [dd] 8,16 1 1198677 9.997756386 2570 M R 5718680 + 8 [dd] 8,16 1 1198678 9.997757480 2570 Q R 5718688 + 8 [dd] 8,16 1 1198679 9.997757735 2570 M R 5718688 + 8 [dd] 8,16 1 1198680 9.997758801 2570 Q R 5718696 + 8 [dd] 8,16 1 1198681 9.997759053 2570 M R 5718696 + 8 [dd] 8,16 1 1198682 9.997760136 2570 Q R 5718704 + 8 [dd] 8,16 1 1198683 9.997760389 2570 M R 5718704 + 8 [dd] 8,16 1 1198684 9.997761476 2570 Q R 5718712 + 8 [dd] 8,16 1 1198685 9.997761729 2570 M R 5718712 + 8 [dd] 8,16 1 1198686 9.997762839 2570 Q R 5718720 + 8 [dd] 8,16 1 1198687 9.997763103 2570 M R 5718720 + 8 [dd] 8,16 1 1198688 9.997764177 2570 Q R 5718728 + 8 [dd] 8,16 1 1198689 9.997764430 2570 M R 5718728 + 8 [dd] 8,16 1 1198690 9.997767140 2570 Q R 5718736 + 8 [dd] 8,16 1 1198691 9.997767392 2570 M R 5718736 + 8 [dd] 8,16 1 1198692 9.997768505 2570 Q R 5718744 + 8 [dd] 8,16 1 1198693 9.997768758 2570 M R 5718744 + 8 [dd] 8,16 1 1198694 9.997769907 2570 Q R 5718752 + 8 [dd] 8,16 1 1198695 9.997770165 2570 M R 5718752 + 8 [dd] 8,16 1 1198696 9.997771391 2570 Q R 5718760 + 8 [dd] 8,16 1 1198697 9.997771644 2570 M R 5718760 + 8 [dd] 8,16 1 1198698 9.997772715 2570 Q R 5718768 + 8 [dd] 8,16 1 1198699 9.997772968 2570 M R 5718768 + 8 [dd] 8,16 1 1198700 9.997774126 2570 Q R 5718776 + 8 [dd] 8,16 1 1198701 9.997774376 2570 M R 5718776 + 8 [dd] 8,16 1 1198702 9.997774818 2570 I R 5718528 + 256 ( 47536) [dd] 8,16 1 0 9.997775685 0 m N cfq2570 insert_request 8,16 1 1198703 9.997776165 2570 U N [dd] 1 8,16 1 0 9.997776693 0 m N cfq2570 dispatch_insert 8,16 1 0 9.997777360 0 m N cfq2570 dispatched a request 8,16 1 0 9.997777839 0 m N cfq2570 activate rq, drv=2 8,16 1 1198704 9.997778021 2570 D R 5718528 + 256 ( 3203) [dd] 8,16 1 1198705 9.998145175 0 C R 5718272 + 256 ( 516686) [0] 8,16 1 0 9.998178095 0 m N cfq2570 complete rqnoidle 0 8,16 1 0 9.998179391 0 m N cfq2570 set_slice=10 8,16 1 1198706 9.998277626 2570 Q R 5718784 + 8 [dd] 8,16 1 1198707 9.998282549 2570 G R 5718784 + 8 [dd] 8,16 1 1198708 9.998283909 2570 P N [dd] 8,16 1 1198709 9.998286466 2570 Q R 5718792 + 8 [dd] 8,16 1 1198710 9.998287233 2570 M R 5718792 + 8 [dd] 8,16 1 1198711 9.998288819 2570 Q R 5718800 + 8 [dd] 8,16 1 1198712 9.998289128 2570 M R 5718800 + 8 [dd] 8,16 1 1198713 9.998290463 2570 Q R 5718808 + 8 [dd] 8,16 1 1198714 9.998290782 2570 M R 5718808 + 8 [dd] 8,16 1 1198715 9.998291960 2570 Q R 5718816 + 8 [dd] 8,16 1 1198716 9.998292216 2570 M R 5718816 + 8 [dd] 8,16 1 1198717 9.998293452 2570 Q R 5718824 + 8 [dd] 8,16 1 1198718 9.998293706 2570 M R 5718824 + 8 [dd] 8,16 1 1198719 9.998294826 2570 Q R 5718832 + 8 [dd] 8,16 1 1198720 9.998295076 2570 M R 5718832 + 8 [dd] 8,16 1 1198721 9.998296188 2570 Q R 5718840 + 8 [dd] 8,16 1 1198722 9.998296440 2570 M R 5718840 + 8 [dd] 8,16 1 1198723 9.998301292 2570 Q R 5718848 + 8 [dd] 8,16 1 1198724 9.998301543 2570 M R 5718848 + 8 [dd] 8,16 1 1198725 9.998302671 2570 Q R 5718856 + 8 [dd] 8,16 1 1198726 9.998302921 2570 M R 5718856 + 8 [dd] 8,16 1 1198727 9.998304137 2570 Q R 5718864 + 8 [dd] 8,16 1 1198728 9.998304444 2570 M R 5718864 + 8 [dd] 8,16 1 1198729 9.998305612 2570 Q R 5718872 + 8 [dd] 8,16 1 1198730 9.998306023 2570 M R 5718872 + 8 [dd] 8,16 1 1198731 9.998307278 2570 Q R 5718880 + 8 [dd] 8,16 1 1198732 9.998307536 2570 M R 5718880 + 8 [dd] 8,16 1 1198733 9.998308749 2570 Q R 5718888 + 8 [dd] 8,16 1 1198734 9.998309159 2570 M R 5718888 + 8 [dd] 8,16 1 1198735 9.998310428 2570 Q R 5718896 + 8 [dd] 8,16 1 1198736 9.998310740 2570 M R 5718896 + 8 [dd] 8,16 1 1198737 9.998312007 2570 Q R 5718904 + 8 [dd] 8,16 1 1198738 9.998312254 2570 M R 5718904 + 8 [dd] 8,16 1 1198739 9.998313452 2570 Q R 5718912 + 8 [dd] 8,16 1 1198740 9.998313701 2570 M R 5718912 + 8 [dd] 8,16 1 1198741 9.998314969 2570 Q R 5718920 + 8 [dd] 8,16 1 1198742 9.998315219 2570 M R 5718920 + 8 [dd] 8,16 1 1198743 9.998316383 2570 Q R 5718928 + 8 [dd] 8,16 1 1198744 9.998316637 2570 M R 5718928 + 8 [dd] 8,16 1 1198745 9.998317744 2570 Q R 5718936 + 8 [dd] 8,16 1 1198746 9.998318003 2570 M R 5718936 + 8 [dd] 8,16 1 1198747 9.998319175 2570 Q R 5718944 + 8 [dd] 8,16 1 1198748 9.998319423 2570 M R 5718944 + 8 [dd] 8,16 1 1198749 9.998320514 2570 Q R 5718952 + 8 [dd] 8,16 1 1198750 9.998320763 2570 M R 5718952 + 8 [dd] 8,16 1 1198751 9.998323664 2570 Q R 5718960 + 8 [dd] 8,16 1 1198752 9.998323914 2570 M R 5718960 + 8 [dd] 8,16 1 1198753 9.998325013 2570 Q R 5718968 + 8 [dd] 8,16 1 1198754 9.998325262 2570 M R 5718968 + 8 [dd] 8,16 1 1198755 9.998326374 2570 Q R 5718976 + 8 [dd] 8,16 1 1198756 9.998326625 2570 M R 5718976 + 8 [dd] 8,16 1 1198757 9.998327712 2570 Q R 5718984 + 8 [dd] 8,16 1 1198758 9.998327962 2570 M R 5718984 + 8 [dd] 8,16 1 1198759 9.998329053 2570 Q R 5718992 + 8 [dd] 8,16 1 1198760 9.998329303 2570 M R 5718992 + 8 [dd] 8,16 1 1198761 9.998330389 2570 Q R 5719000 + 8 [dd] 8,16 1 1198762 9.998330637 2570 M R 5719000 + 8 [dd] 8,16 1 1198763 9.998331785 2570 Q R 5719008 + 8 [dd] 8,16 1 1198764 9.998332034 2570 M R 5719008 + 8 [dd] 8,16 1 1198765 9.998333123 2570 Q R 5719016 + 8 [dd] 8,16 1 1198766 9.998333379 2570 M R 5719016 + 8 [dd] 8,16 1 1198767 9.998334469 2570 Q R 5719024 + 8 [dd] 8,16 1 1198768 9.998334727 2570 M R 5719024 + 8 [dd] 8,16 1 1198769 9.998335818 2570 Q R 5719032 + 8 [dd] 8,16 1 1198770 9.998336080 2570 M R 5719032 + 8 [dd] 8,16 1 1198771 9.998337279 2570 I R 5718784 + 256 ( 54730) [dd] 8,16 1 0 9.998338674 0 m N cfq2570 insert_request 8,16 1 1198772 9.998340424 2570 U N [dd] 1 8,16 1 0 9.998341194 0 m N cfq2570 dispatch_insert 8,16 1 0 9.998342443 0 m N cfq2570 dispatched a request 8,16 1 0 9.998343187 0 m N cfq2570 activate rq, drv=2 8,16 1 1198773 9.998343485 2570 D R 5718784 + 256 ( 6206) [dd] 8,16 1 1198774 9.998637335 0 C R 5718528 + 256 ( 859314) [0] 8,16 1 0 9.998671549 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198775 9.998814006 2570 Q R 5719040 + 8 [dd] 8,16 1 1198776 9.998818725 2570 G R 5719040 + 8 [dd] 8,16 1 1198777 9.998820090 2570 P N [dd] 8,16 1 1198778 9.998822388 2570 Q R 5719048 + 8 [dd] 8,16 1 1198779 9.998823161 2570 M R 5719048 + 8 [dd] 8,16 1 1198780 9.998824965 2570 Q R 5719056 + 8 [dd] 8,16 1 1198781 9.998825233 2570 M R 5719056 + 8 [dd] 8,16 1 1198782 9.998826625 2570 Q R 5719064 + 8 [dd] 8,16 1 1198783 9.998826882 2570 M R 5719064 + 8 [dd] 8,16 1 1198784 9.998831436 2570 Q R 5719072 + 8 [dd] 8,16 1 1198785 9.998831701 2570 M R 5719072 + 8 [dd] 8,16 1 1198786 9.998832847 2570 Q R 5719080 + 8 [dd] 8,16 1 1198787 9.998833188 2570 M R 5719080 + 8 [dd] 8,16 1 1198788 9.998834360 2570 Q R 5719088 + 8 [dd] 8,16 1 1198789 9.998834619 2570 M R 5719088 + 8 [dd] 8,16 1 1198790 9.998835841 2570 Q R 5719096 + 8 [dd] 8,16 1 1198791 9.998836104 2570 M R 5719096 + 8 [dd] 8,16 1 1198792 9.998837283 2570 Q R 5719104 + 8 [dd] 8,16 1 1198793 9.998837538 2570 M R 5719104 + 8 [dd] 8,16 1 1198794 9.998838705 2570 Q R 5719112 + 8 [dd] 8,16 1 1198795 9.998838966 2570 M R 5719112 + 8 [dd] 8,16 1 1198796 9.998840129 2570 Q R 5719120 + 8 [dd] 8,16 1 1198797 9.998840383 2570 M R 5719120 + 8 [dd] 8,16 1 1198798 9.998841509 2570 Q R 5719128 + 8 [dd] 8,16 1 1198799 9.998841785 2570 M R 5719128 + 8 [dd] 8,16 1 1198800 9.998843055 2570 Q R 5719136 + 8 [dd] 8,16 1 1198801 9.998843306 2570 M R 5719136 + 8 [dd] 8,16 1 1198802 9.998844451 2570 Q R 5719144 + 8 [dd] 8,16 1 1198803 9.998844701 2570 M R 5719144 + 8 [dd] 8,16 1 1198804 9.998845793 2570 Q R 5719152 + 8 [dd] 8,16 1 1198805 9.998846046 2570 M R 5719152 + 8 [dd] 8,16 1 1198806 9.998847127 2570 Q R 5719160 + 8 [dd] 8,16 1 1198807 9.998847379 2570 M R 5719160 + 8 [dd] 8,16 1 1198808 9.998848468 2570 Q R 5719168 + 8 [dd] 8,16 1 1198809 9.998848724 2570 M R 5719168 + 8 [dd] 8,16 1 1198810 9.998849897 2570 Q R 5719176 + 8 [dd] 8,16 1 1198811 9.998850313 2570 M R 5719176 + 8 [dd] 8,16 1 1198812 9.998853178 2570 Q R 5719184 + 8 [dd] 8,16 1 1198813 9.998853436 2570 M R 5719184 + 8 [dd] 8,16 1 1198814 9.998854623 2570 Q R 5719192 + 8 [dd] 8,16 1 1198815 9.998854872 2570 M R 5719192 + 8 [dd] 8,16 1 1198816 9.998855971 2570 Q R 5719200 + 8 [dd] 8,16 1 1198817 9.998856224 2570 M R 5719200 + 8 [dd] 8,16 1 1198818 9.998857312 2570 Q R 5719208 + 8 [dd] 8,16 1 1198819 9.998857588 2570 M R 5719208 + 8 [dd] 8,16 1 1198820 9.998859026 2570 Q R 5719216 + 8 [dd] 8,16 1 1198821 9.998859281 2570 M R 5719216 + 8 [dd] 8,16 1 1198822 9.998860490 2570 Q R 5719224 + 8 [dd] 8,16 1 1198823 9.998860758 2570 M R 5719224 + 8 [dd] 8,16 1 1198824 9.998861935 2570 Q R 5719232 + 8 [dd] 8,16 1 1198825 9.998862185 2570 M R 5719232 + 8 [dd] 8,16 1 1198826 9.998863465 2570 Q R 5719240 + 8 [dd] 8,16 1 1198827 9.998863731 2570 M R 5719240 + 8 [dd] 8,16 1 1198828 9.998864923 2570 Q R 5719248 + 8 [dd] 8,16 1 1198829 9.998865180 2570 M R 5719248 + 8 [dd] 8,16 1 1198830 9.998866382 2570 Q R 5719256 + 8 [dd] 8,16 1 1198831 9.998866632 2570 M R 5719256 + 8 [dd] 8,16 1 1198832 9.998867814 2570 Q R 5719264 + 8 [dd] 8,16 1 1198833 9.998868069 2570 M R 5719264 + 8 [dd] 8,16 1 1198834 9.998869250 2570 Q R 5719272 + 8 [dd] 8,16 1 1198835 9.998869505 2570 M R 5719272 + 8 [dd] 8,16 1 1198836 9.998870701 2570 Q R 5719280 + 8 [dd] 8,16 1 1198837 9.998870955 2570 M R 5719280 + 8 [dd] 8,16 1 1198838 9.998872199 2570 Q R 5719288 + 8 [dd] 8,16 1 1198839 9.998872450 2570 M R 5719288 + 8 [dd] 8,16 1 1198840 9.998873325 2570 I R 5719040 + 256 ( 54600) [dd] 8,16 1 0 9.998875079 0 m N cfq2570 insert_request 8,16 1 1198841 9.998876908 2570 U N [dd] 1 8,16 1 0 9.998877704 0 m N cfq2570 dispatch_insert 8,16 1 0 9.998879045 0 m N cfq2570 dispatched a request 8,16 1 0 9.998879788 0 m N cfq2570 activate rq, drv=2 8,16 1 1198842 9.998880072 2570 D R 5719040 + 256 ( 6747) [dd] 8,16 1 1198843 9.999128563 0 C R 5718784 + 256 ( 785078) [0] 8,16 1 0 9.999161339 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198844 9.999263022 2570 Q R 5719296 + 8 [dd] 8,16 1 1198845 9.999268323 2570 G R 5719296 + 8 [dd] 8,16 1 1198846 9.999269923 2570 P N [dd] 8,16 1 1198847 9.999271887 2570 Q R 5719304 + 8 [dd] 8,16 1 1198848 9.999272680 2570 M R 5719304 + 8 [dd] 8,16 1 1198849 9.999274299 2570 Q R 5719312 + 8 [dd] 8,16 1 1198850 9.999274621 2570 M R 5719312 + 8 [dd] 8,16 1 1198851 9.999275954 2570 Q R 5719320 + 8 [dd] 8,16 1 1198852 9.999276219 2570 M R 5719320 + 8 [dd] 8,16 1 1198853 9.999277509 2570 Q R 5719328 + 8 [dd] 8,16 1 1198854 9.999277769 2570 M R 5719328 + 8 [dd] 8,16 1 1198855 9.999279405 2570 Q R 5719336 + 8 [dd] 8,16 1 1198856 9.999279656 2570 M R 5719336 + 8 [dd] 8,16 1 1198857 9.999280750 2570 Q R 5719344 + 8 [dd] 8,16 1 1198858 9.999281002 2570 M R 5719344 + 8 [dd] 8,16 1 1198859 9.999282113 2570 Q R 5719352 + 8 [dd] 8,16 1 1198860 9.999282364 2570 M R 5719352 + 8 [dd] 8,16 1 1198861 9.999283601 2570 Q R 5719360 + 8 [dd] 8,16 1 1198862 9.999283851 2570 M R 5719360 + 8 [dd] 8,16 1 1198863 9.999284938 2570 Q R 5719368 + 8 [dd] 8,16 1 1198864 9.999285193 2570 M R 5719368 + 8 [dd] 8,16 1 1198865 9.999286362 2570 Q R 5719376 + 8 [dd] 8,16 1 1198866 9.999286610 2570 M R 5719376 + 8 [dd] 8,16 1 1198867 9.999287703 2570 Q R 5719384 + 8 [dd] 8,16 1 1198868 9.999287954 2570 M R 5719384 + 8 [dd] 8,16 1 1198869 9.999289288 2570 Q R 5719392 + 8 [dd] 8,16 1 1198870 9.999289538 2570 M R 5719392 + 8 [dd] 8,16 1 1198871 9.999290703 2570 Q R 5719400 + 8 [dd] 8,16 1 1198872 9.999290953 2570 M R 5719400 + 8 [dd] 8,16 1 1198873 9.999293719 2570 Q R 5719408 + 8 [dd] 8,16 1 1198874 9.999293971 2570 M R 5719408 + 8 [dd] 8,16 1 1198875 9.999295079 2570 Q R 5719416 + 8 [dd] 8,16 1 1198876 9.999295333 2570 M R 5719416 + 8 [dd] 8,16 1 1198877 9.999296580 2570 Q R 5719424 + 8 [dd] 8,16 1 1198878 9.999296831 2570 M R 5719424 + 8 [dd] 8,16 1 1198879 9.999297922 2570 Q R 5719432 + 8 [dd] 8,16 1 1198880 9.999298173 2570 M R 5719432 + 8 [dd] 8,16 1 1198881 9.999299271 2570 Q R 5719440 + 8 [dd] 8,16 1 1198882 9.999299532 2570 M R 5719440 + 8 [dd] 8,16 1 1198883 9.999300619 2570 Q R 5719448 + 8 [dd] 8,16 1 1198884 9.999300871 2570 M R 5719448 + 8 [dd] 8,16 1 1198885 9.999302052 2570 Q R 5719456 + 8 [dd] 8,16 1 1198886 9.999302303 2570 M R 5719456 + 8 [dd] 8,16 1 1198887 9.999303397 2570 Q R 5719464 + 8 [dd] 8,16 1 1198888 9.999303647 2570 M R 5719464 + 8 [dd] 8,16 1 1198889 9.999304900 2570 Q R 5719472 + 8 [dd] 8,16 1 1198890 9.999305209 2570 M R 5719472 + 8 [dd] 8,16 1 1198891 9.999306412 2570 Q R 5719480 + 8 [dd] 8,16 1 1198892 9.999306675 2570 M R 5719480 + 8 [dd] 8,16 1 1198893 9.999307848 2570 Q R 5719488 + 8 [dd] 8,16 1 1198894 9.999308107 2570 M R 5719488 + 8 [dd] 8,16 1 1198895 9.999309220 2570 Q R 5719496 + 8 [dd] 8,16 1 1198896 9.999309471 2570 M R 5719496 + 8 [dd] 8,16 1 1198897 9.999310581 2570 Q R 5719504 + 8 [dd] 8,16 1 1198898 9.999310875 2570 M R 5719504 + 8 [dd] 8,16 1 1198899 9.999312017 2570 Q R 5719512 + 8 [dd] 8,16 1 1198900 9.999312272 2570 M R 5719512 + 8 [dd] 8,16 1 1198901 9.999315106 2570 Q R 5719520 + 8 [dd] 8,16 1 1198902 9.999315358 2570 M R 5719520 + 8 [dd] 8,16 1 1198903 9.999316495 2570 Q R 5719528 + 8 [dd] 8,16 1 1198904 9.999316746 2570 M R 5719528 + 8 [dd] 8,16 1 1198905 9.999317852 2570 Q R 5719536 + 8 [dd] 8,16 1 1198906 9.999318103 2570 M R 5719536 + 8 [dd] 8,16 1 1198907 9.999319271 2570 Q R 5719544 + 8 [dd] 8,16 1 1198908 9.999319523 2570 M R 5719544 + 8 [dd] 8,16 1 1198909 9.999320448 2570 I R 5719296 + 256 ( 52125) [dd] 8,16 1 0 9.999321895 0 m N cfq2570 insert_request 8,16 1 1198910 9.999323661 2570 U N [dd] 1 8,16 1 0 9.999324400 0 m N cfq2570 dispatch_insert 8,16 1 0 9.999325762 0 m N cfq2570 dispatched a request 8,16 1 0 9.999326490 0 m N cfq2570 activate rq, drv=2 8,16 1 1198911 9.999326769 2570 D R 5719296 + 256 ( 6321) [dd] 8,16 1 1198912 9.999615787 0 C R 5719040 + 256 ( 735715) [0] 8,16 1 0 9.999647801 0 m N cfq2570 complete rqnoidle 0 8,16 1 1198913 9.999800271 2570 Q R 5719552 + 8 [dd] 8,16 1 1198914 9.999805251 2570 G R 5719552 + 8 [dd] 8,16 1 1198915 9.999806646 2570 P N [dd] 8,16 1 1198916 9.999808904 2570 Q R 5719560 + 8 [dd] 8,16 1 1198917 9.999809719 2570 M R 5719560 + 8 [dd] 8,16 1 1198918 9.999811296 2570 Q R 5719568 + 8 [dd] 8,16 1 1198919 9.999811551 2570 M R 5719568 + 8 [dd] 8,16 1 1198920 9.999812779 2570 Q R 5719576 + 8 [dd] 8,16 1 1198921 9.999813096 2570 M R 5719576 + 8 [dd] 8,16 1 1198922 9.999814316 2570 Q R 5719584 + 8 [dd] 8,16 1 1198923 9.999814568 2570 M R 5719584 + 8 [dd] 8,16 1 1198924 9.999815662 2570 Q R 5719592 + 8 [dd] 8,16 1 1198925 9.999815914 2570 M R 5719592 + 8 [dd] 8,16 1 1198926 9.999817011 2570 Q R 5719600 + 8 [dd] 8,16 1 1198927 9.999817262 2570 M R 5719600 + 8 [dd] 8,16 1 1198928 9.999818366 2570 Q R 5719608 + 8 [dd] 8,16 1 1198929 9.999818628 2570 M R 5719608 + 8 [dd] 8,16 1 1198930 9.999819792 2570 Q R 5719616 + 8 [dd] 8,16 1 1198931 9.999820044 2570 M R 5719616 + 8 [dd] 8,16 1 1198932 9.999821139 2570 Q R 5719624 + 8 [dd] 8,16 1 1198933 9.999821389 2570 M R 5719624 + 8 [dd] 8,16 1 1198934 9.999825670 2570 Q R 5719632 + 8 [dd] 8,16 1 1198935 9.999825923 2570 M R 5719632 + 8 [dd] 8,16 1 1198936 9.999827154 2570 Q R 5719640 + 8 [dd] 8,16 1 1198937 9.999827405 2570 M R 5719640 + 8 [dd] 8,16 1 1198938 9.999828602 2570 Q R 5719648 + 8 [dd] 8,16 1 1198939 9.999828854 2570 M R 5719648 + 8 [dd] 8,16 1 1198940 9.999830028 2570 Q R 5719656 + 8 [dd] 8,16 1 1198941 9.999830355 2570 M R 5719656 + 8 [dd] 8,16 1 1198942 9.999831553 2570 Q R 5719664 + 8 [dd] 8,16 1 1198943 9.999831805 2570 M R 5719664 + 8 [dd] 8,16 1 1198944 9.999832905 2570 Q R 5719672 + 8 [dd] 8,16 1 1198945 9.999833157 2570 M R 5719672 + 8 [dd] 8,16 1 1198946 9.999834350 2570 Q R 5719680 + 8 [dd] 8,16 1 1198947 9.999834602 2570 M R 5719680 + 8 [dd] 8,16 1 1198948 9.999835704 2570 Q R 5719688 + 8 [dd] 8,16 1 1198949 9.999835957 2570 M R 5719688 + 8 [dd] 8,16 1 1198950 9.999837061 2570 Q R 5719696 + 8 [dd] 8,16 1 1198951 9.999837310 2570 M R 5719696 + 8 [dd] 8,16 1 1198952 9.999838418 2570 Q R 5719704 + 8 [dd] 8,16 1 1198953 9.999838674 2570 M R 5719704 + 8 [dd] 8,16 1 1198954 9.999839902 2570 Q R 5719712 + 8 [dd] 8,16 1 1198955 9.999840154 2570 M R 5719712 + 8 [dd] 8,16 1 1198956 9.999841286 2570 Q R 5719720 + 8 [dd] 8,16 1 1198957 9.999841537 2570 M R 5719720 + 8 [dd] 8,16 1 1198958 9.999842684 2570 Q R 5719728 + 8 [dd] 8,16 1 1198959 9.999842936 2570 M R 5719728 + 8 [dd] 8,16 1 1198960 9.999844037 2570 Q R 5719736 + 8 [dd] 8,16 1 1198961 9.999844298 2570 M R 5719736 + 8 [dd] 8,16 1 1198962 9.999847070 2570 Q R 5719744 + 8 [dd] 8,16 1 1198963 9.999847330 2570 M R 5719744 + 8 [dd] 8,16 1 1198964 9.999848422 2570 Q R 5719752 + 8 [dd] 8,16 1 1198965 9.999848672 2570 M R 5719752 + 8 [dd] 8,16 1 1198966 9.999850021 2570 Q R 5719760 + 8 [dd] 8,16 1 1198967 9.999850272 2570 M R 5719760 + 8 [dd] 8,16 1 1198968 9.999851440 2570 Q R 5719768 + 8 [dd] 8,16 1 1198969 9.999851714 2570 M R 5719768 + 8 [dd] 8,16 1 1198970 9.999852878 2570 Q R 5719776 + 8 [dd] 8,16 1 1198971 9.999853133 2570 M R 5719776 + 8 [dd] 8,16 1 1198972 9.999854239 2570 Q R 5719784 + 8 [dd] 8,16 1 1198973 9.999854495 2570 M R 5719784 + 8 [dd] 8,16 1 1198974 9.999855595 2570 Q R 5719792 + 8 [dd] 8,16 1 1198975 9.999855849 2570 M R 5719792 + 8 [dd] 8,16 1 1198976 9.999856947 2570 Q R 5719800 + 8 [dd] 8,16 1 1198977 9.999857198 2570 M R 5719800 + 8 [dd] 8,16 1 1198978 9.999858058 2570 I R 5719552 + 256 ( 52807) [dd] 8,16 1 0 9.999859378 0 m N cfq2570 insert_request 8,16 1 1198979 9.999861062 2570 U N [dd] 1 8,16 1 0 9.999861854 0 m N cfq2570 dispatch_insert 8,16 1 0 9.999863078 0 m N cfq2570 dispatched a request 8,16 1 0 9.999863763 0 m N cfq2570 activate rq, drv=2 8,16 1 1198980 9.999864420 2570 D R 5719552 + 256 ( 6362) [dd] CPU0 (sdb): Reads Queued: 60971, 247656KiB Writes Queued: 4, 16KiB Read Dispatches: 2015, 233044KiB Write Dispatches: 0, 0KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 107, 13824KiB Writes Completed: 0, 0KiB Read Merges: 58809, 235564KiB Write Merges: 2, 8KiB Read depth: 8 Write depth: 1 IO unplugs: 1955 Timer unplugs: 0 CPU1 (sdb): Reads Queued: 554679, 2223MiB Writes Queued: 13, 52KiB Read Dispatches: 17603, 2238MiB Write Dispatches: 5, 68KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 19509, 2457MiB Writes Completed: 7, 68KiB Read Merges: 537222, 2149MiB Write Merges: 10, 40KiB Read depth: 8 Write depth: 1 IO unplugs: 17408 Timer unplugs: 0 Total (sdb): Reads Queued: 615650, 2471MiB Writes Queued: 17, 68KiB Read Dispatches: 19618, 2471MiB Write Dispatches: 5, 68KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 19616, 2471MiB Writes Completed: 7, 68KiB Read Merges: 596031, 2385MiB Write Merges: 12, 48KiB IO unplugs: 19363 Timer unplugs: 0 Throughput (R/W): 247140KiB/s / 6KiB/s Events (sdb): 1434007 entries Skips: 0 forward (0 - 0.0%) Input file sdb.blktrace.0 added Input file sdb.blktrace.1 added ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-30 14:28 ` Wu Fengguang 2012-01-30 14:51 ` Eric Dumazet @ 2012-01-31 6:36 ` Herbert Poetzl 1 sibling, 0 replies; 36+ messages in thread From: Herbert Poetzl @ 2012-01-31 6:36 UTC (permalink / raw) To: Wu Fengguang Cc: Shaohua Li, Eric Dumazet, Andrew Morton, LKML, Jens Axboe, Tejun Heo On Mon, Jan 30, 2012 at 10:28:37PM +0800, Wu Fengguang wrote: > On Mon, Jan 30, 2012 at 06:31:34PM +0800, Li, Shaohua wrote: >> Looks the 2.6.39 block plug introduces some latency here. deleting >> blk_start_plug/blk_finish_plug in generic_file_aio_read seems >> workaround >> the issue. The plug seems not good for sequential IO, because readahead >> code already has plug and has fine grained control. > Why not remove the generic_file_aio_read() plug completely? It > actually prevents unplugging immediately after the readahead IO is > submitted and in turn stalls the IO pipeline as showed by Eric's > blktrace data. > Eric, will you test this patch? Thank you. > --- linux.orig/mm/filemap.c 2012-01-30 22:27:11.000000000 +0800 > +++ linux/mm/filemap.c 2012-01-30 22:27:25.000000000 +0800 > @@ -1414,15 +1414,12 @@ generic_file_aio_read(struct kiocb *iocb > unsigned long seg = 0; > size_t count; > loff_t *ppos = &iocb->ki_pos; > - struct blk_plug plug; > count = 0; > retval = generic_segment_checks(iov, &nr_segs, &count, VERIFY_WRITE); > if (retval) > return retval; > - blk_start_plug(&plug); > - > /* coalesce the iovecs and go direct-to-BIO for O_DIRECT */ > if (filp->f_flags & O_DIRECT) { > loff_t size; > @@ -1495,7 +1492,6 @@ generic_file_aio_read(struct kiocb *iocb > break; > } > out: > - blk_finish_plug(&plug); > return retval; > } > EXPORT_SYMBOL(generic_file_aio_read); I can confirm, this seems to fix the issue here as well: kernel ------------ read /dev/sda ------------- --- noop --- - deadline - ---- cfs --- [MB/s] %CPU [MB/s] %CPU [MB/s] %CPU -------------------------------------------------- 3.2.2 45.82 3.7 44.85 3.6 45.04 3.4 3.2.2i 45.59 2.3 51.78 2.6 46.03 2.2 3.2.2i128 250.24 20.9 252.68 21.3 250.00 21.6 3.2.2p 250.88 20.8 249.69 20.7 252.34 21.4 3.2.2p128 249.36 20.7 251.47 20.9 249.03 21.2 3.2.2 ... unmodified 3.2.2 kernel 1M 3.2.2i .. unmodified 3.2.2 kernel with idle=poll, 1M 3.2.2i128 unmodified 3.2.2 kernel with idle=poll, 128k 3.2.2p .. unmodified 3.2.2 kernel with patch, 1M 3.2.2p128 unmodified 3.2.2 kernel with patch, 128k thanks again, Herbert ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-29 10:03 ` Eric Dumazet 2012-01-29 11:16 ` Wu Fengguang @ 2012-01-30 14:48 ` Wu Fengguang 1 sibling, 0 replies; 36+ messages in thread From: Wu Fengguang @ 2012-01-30 14:48 UTC (permalink / raw) To: Eric Dumazet Cc: Herbert Poetzl, Andrew Morton, LKML, Jens Axboe, Tejun Heo, Li Shaohua On Sun, Jan 29, 2012 at 11:03:00AM +0100, Eric Dumazet wrote: > Very strange, my bissection ended on following commit : > > commit 805f6b5e1cbfedfb9b3d354013e7f4b13a79270f > Author: Tao Ma <boyu.mt@taobao.com> > Date: Fri Mar 11 20:11:59 2011 +0100 > > blktrace: Use rq->cmd_flags directly in blk_add_trace_rq. > > This makes no sense. Yeah, commit 55602dd66f535 ("fs: make generic file read/write functions plug") looks like the real root cause and should probably be reverted as a whole. Buffered read/write syscalls are not directly tied to IO after all (readahead/writeback have the right plug points). Thanks, Fengguang > hdparm uses 2MB block reads, so read_ahead (128KB) is too small for best > perf > > # cat /sys/class/block/sda/queue/read_ahead_kb > 128 > > # dd if=/dev/sda of=/dev/null bs=128k > ^C > 63744+0 enregistrements lus > 63743+0 enregistrements écrits > 8354922496 octets (8,4 GB) copiés, 39,975 s, 209 MB/s > > # hdparm -t /dev/sda > > /dev/sda: > Timing buffered disk reads: 510 MB in 3.00 seconds = 169.75 MB/sec > > # uname -a > Linux edumazet-laptop 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20 > 17:23:00 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux > > ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: Bad SSD performance with recent kernels 2012-01-28 12:51 ` Wu Fengguang 2012-01-28 13:33 ` Eric Dumazet @ 2012-01-28 17:01 ` Herbert Poetzl 1 sibling, 0 replies; 36+ messages in thread From: Herbert Poetzl @ 2012-01-28 17:01 UTC (permalink / raw) To: Wu Fengguang; +Cc: Andrew Morton, LKML, Jens Axboe, Tejun Heo [-- Attachment #1: Type: text/plain, Size: 5195 bytes --] On Sat, Jan 28, 2012 at 08:51:08PM +0800, Wu Fengguang wrote: > Hi Herbert, > On Fri, Jan 27, 2012 at 07:00:34AM +0100, Herbert Poetzl wrote: >> Dear Community! >> Recently I decided to update the kernel on a Dell Laptop to >> a more recent version than 2.6.38.x, but experienced bad >> I/O performance with the new kernel, so I started to dig >> a little deeper and ended up with the following test: >> 1) download and extract kernel (on 2.6.38.8) >> 2) make defconfig >> 3) make localmodconfig >> 4) make >> 5) make modules_install install >> Then I booted each kernel in single user and ran the following >> test script: >> echo noop >/sys/class/block/sda/queue/scheduler >> for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done >> /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ >> ionice -c0 nice -20 \ >> dd if=/dev/sda of=/dev/null bs=1M count=20480 >> echo deadline >/sys/class/block/sda/queue/scheduler >> for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done >> /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ >> ionice -c0 nice -20 \ >> dd if=/dev/sda of=/dev/null bs=1M count=20480 >> echo cfq >/sys/class/block/sda/queue/scheduler >> for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done >> /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \ >> ionice -c0 nice -20 \ >> dd if=/dev/sda of=/dev/null bs=1M count=20480 >> note that the Laptop is a relatively modern Latitude E6400 >> with a Samsung 830 Series 256GB SSD >> here are the surprising results: >> @ linux 2.6.38.8 >> 248 MB/s real = 86.74, user = 0.01, sys = 21.65, 24% cpu >> 248 MB/s real = 86.81, user = 0.02, sys = 21.75, 25% cpu >> 251 MB/s real = 85.63, user = 0.01, sys = 22.24, 25% cpu >> @ linux 2.6.39.4 >> 49.0 MB/s real = 438.79, user = 0.01, sys = 19.79, 4% cpu >> 25.7 MB/s real = 836.70, user = 0.02, sys = 18.39, 2% cpu >> 27.7 MB/s real = 776.53, user = 0.01, sys = 16.03, 2% cpu >> @ linux 3.0.18 >> 48.9 MB/s real = 439.07, user = 0.01, sys = 17.55, 4% cpu >> 25.0 MB/s real = 859.03, user = 0.01, sys = 16.97, 1% cpu >> 49.8 MB/s real = 431.61, user = 0.01, sys = 16.68, 3% cpu >> @ linux 3.1.10 >> 54.0 MB/s real = 398.23, user = 0.01, sys = 17.36, 4% cpu >> 29.4 MB/s real = 731.47, user = 0.01, sys = 17.14, 2% cpu >> 25.0 MB/s real = 859.35, user = 0.01, sys = 14.51, 1% cpu >> @ linux 3.2.2 >> 45.8 MB/s real = 468.85, user = 0.01, sys = 17.11, 3% cpu >> 44.8 MB/s real = 478.92, user = 0.01, sys = 17.02, 3% cpu >> 45.0 MB/s real = 476.91, user = 0.01, sys = 16.14, 3% cpu > Thanks for the through tests! However I cannot reproduce the > regressions, but see slightly better performance in 3.2: > Linux lkp-nex04 3.2.0-rc7-shli+ #121 SMP Thu Jan 19 18:10:45 CST 2012 x86_64 x86_64 x86_64 GNU/Linux > 21474836480 bytes (21 GB) copied, 90.9307 s, 236 MB/s real = 91.03, user = 0.01, sys = 25.80, 28% cpu > 21474836480 bytes (21 GB) copied, 90.8864 s, 236 MB/s real = 90.90, user = 0.01, sys = 25.54, 28% cpu > 21474836480 bytes (21 GB) copied, 93.4684 s, 230 MB/s real = 93.47, user = 0.02, sys = 25.12, 26% cpu > Linux lkp-nex04 2.6.38 #334 SMP Sat Jan 28 20:16:25 CST 2012 x86_64 GNU/Linux > 21474836480 bytes (21 GB) copied, 96.2382 s, 223 MB/s real = 96.29, user = 0.01, sys = 57.64, 59% cpu > 21474836480 bytes (21 GB) copied, 96.5796 s, 222 MB/s real = 96.60, user = 0.02, sys = 58.08, 60% cpu > 21474836480 bytes (21 GB) copied, 95.486 s, 225 MB/s real = 95.51, user = 0.00, sys = 58.20, 60% cpu > My test box has 64GB memory and a dozen 80GB INTEL > SSDSA2M080G2GN SSD drives attached to so we have intel vs samsung here ... > 87:00.0 Serial Attached SCSI controller: LSI Logic / Symbios > Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02) ... and sas vs sata >> I have no idea why the I/O performance is that bad on any >> kernel newer than 2.6.38.x, but I'm happy to test and/or >> try various configurations as time permits ... > Would you please create a filesystem and large file on sda and > run the tests on the file? There was some performance bug on > reading the raw /dev/sda device file.. will do tonight! >> The detailed test results as well as the dmesg and config >> of each kernel can be found here: >> http://vserver.13thfloor.at/Stuff/SSD > It would be better for such files be attached in future. > (For one thing I get timeout when trying to open the link...) attached the files as .txz thanks, Herbert > Thanks, > Fengguang >> here some technical information: >> http://www.dell.com/us/dfb/p/latitude-e6400/pd >> SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03) >> http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec >> Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q >> supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ [-- Attachment #2: SSD.txz --] [-- Type: application/octet-stream, Size: 57008 bytes --] ^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2012-01-31 8:46 UTC | newest] Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-01-27 6:00 Bad SSD performance with recent kernels Herbert Poetzl 2012-01-27 6:44 ` Eric Dumazet 2012-01-28 12:51 ` Wu Fengguang 2012-01-28 13:33 ` Eric Dumazet 2012-01-29 5:59 ` Wu Fengguang 2012-01-29 8:42 ` Herbert Poetzl 2012-01-29 9:28 ` Wu Fengguang 2012-01-29 10:03 ` Eric Dumazet 2012-01-29 11:16 ` Wu Fengguang 2012-01-29 13:13 ` Eric Dumazet 2012-01-29 15:52 ` Pádraig Brady 2012-01-29 16:10 ` Wu Fengguang 2012-01-29 20:15 ` Herbert Poetzl 2012-01-30 11:18 ` Wu Fengguang 2012-01-30 12:34 ` Eric Dumazet 2012-01-30 14:01 ` Wu Fengguang 2012-01-30 14:05 ` Wu Fengguang 2012-01-30 3:17 ` Shaohua Li 2012-01-30 5:31 ` Eric Dumazet 2012-01-30 5:45 ` Shaohua Li 2012-01-30 7:13 ` Herbert Poetzl 2012-01-30 7:22 ` Shaohua Li 2012-01-30 7:36 ` Herbert Poetzl 2012-01-30 8:12 ` Shaohua Li 2012-01-30 10:31 ` Shaohua Li 2012-01-30 14:28 ` Wu Fengguang 2012-01-30 14:51 ` Eric Dumazet 2012-01-30 22:26 ` Vivek Goyal 2012-01-31 0:14 ` Shaohua Li 2012-01-31 1:07 ` Wu Fengguang 2012-01-31 3:00 ` Shaohua Li 2012-01-31 2:17 ` Eric Dumazet 2012-01-31 8:46 ` Eric Dumazet 2012-01-31 6:36 ` Herbert Poetzl 2012-01-30 14:48 ` Wu Fengguang 2012-01-28 17:01 ` Herbert Poetzl
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).