Bad performance when reads and writes on same LUN

From: "Rettl, Matthias" <Matthias.Rettl@netapp.com>
To: Jens Axboe <axboe@kernel.dk>
Cc: "Neto,
	Antonio Jose Rodrigues" <Antonio.Jose.Rodrigues.Neto@netapp.com>,
	fio@vger.kernel.org
Subject: Bad performance when reads and writes on same LUN
Date: Fri, 2 Sep 2011 16:32:19 +0100	[thread overview]
Message-ID: <78421E8B2231BE40AEDFFDC5CD3820750D607110@LDCMVEXC1-PRD.hq.netapp.com> (raw)

Hi fio users and developers,

I am working on a setup for a customer with a very specific FIO
configuration:
We have to use 8 LUNs (from a very performant storage system, by the
way), doing three readers and one writer per LUN.

So we use 8 fio config files, each of them look the same, except the
directory= entry:

[global] 
directory=/n1-dm-0
rw=randread  
ioengine=libaio  
iodepth=4 
size=1024m  
invalidate=1 
direct=0 
runtime=60  
time_based  

[Reader-1] 

[Reader-2] 

[Reader-3] 

[Writer]
rw=randwrite
ioengine=libaio
iodepth=32
direct=1

When running this configurations (8 fio's in parallel) I receive these
results (grep'ed iops from the 8 out-files):
  read : io=128520KB, bw=2141.1KB/s, iops=535 , runt= 60001msec
  read : io=127048KB, bw=2117.5KB/s, iops=529 , runt= 60001msec
  read : io=126816KB, bw=2113.6KB/s, iops=528 , runt= 60001msec
  write: io=3811.8MB, bw=65052KB/s, iops=16263 , runt= 60001msec
  read : io=123704KB, bw=2061.7KB/s, iops=515 , runt= 60004msec
  read : io=123656KB, bw=2060.9KB/s, iops=515 , runt= 60002msec
  read : io=122924KB, bw=2048.7KB/s, iops=512 , runt= 60004msec
  write: io=3761.8MB, bw=64187KB/s, iops=16046 , runt= 60002msec
  read : io=127636KB, bw=2127.3KB/s, iops=531 , runt= 60001msec
  read : io=126440KB, bw=2107.4KB/s, iops=526 , runt= 60001msec
  read : io=125612KB, bw=2093.6KB/s, iops=523 , runt= 60001msec
  write: io=3832.4MB, bw=65403KB/s, iops=16350 , runt= 60002msec
  read : io=125344KB, bw=2089.4KB/s, iops=522 , runt= 60001msec
  read : io=125284KB, bw=2088.4KB/s, iops=522 , runt= 60001msec
  read : io=125080KB, bw=2084.7KB/s, iops=521 , runt= 60001msec
  write: io=3784.8MB, bw=64592KB/s, iops=16147 , runt= 60001msec
  read : io=127656KB, bw=2127.6KB/s, iops=531 , runt= 60001msec
  read : io=127144KB, bw=2119.4KB/s, iops=529 , runt= 60001msec
  read : io=126248KB, bw=2104.1KB/s, iops=526 , runt= 60001msec
  write: io=3828.9MB, bw=65343KB/s, iops=16335 , runt= 60002msec
  read : io=124236KB, bw=2070.6KB/s, iops=517 , runt= 60001msec
  read : io=123908KB, bw=2065.2KB/s, iops=516 , runt= 60001msec
  read : io=123352KB, bw=2055.9KB/s, iops=513 , runt= 60001msec
  write: io=3764.1MB, bw=64253KB/s, iops=16063 , runt= 60001msec
  read : io=127784KB, bw=2129.8KB/s, iops=532 , runt= 60001msec
  read : io=127276KB, bw=2121.3KB/s, iops=530 , runt= 60001msec
  read : io=127240KB, bw=2120.7KB/s, iops=530 , runt= 60001msec
  write: io=3839.5MB, bw=65524KB/s, iops=16380 , runt= 60002msec
  read : io=124864KB, bw=2081.4KB/s, iops=520 , runt= 60001msec
  read : io=124008KB, bw=2066.8KB/s, iops=516 , runt= 60002msec
  read : io=124068KB, bw=2067.8KB/s, iops=516 , runt= 60001msec
  write: io=3748.9MB, bw=63979KB/s, iops=15994 , runt= 60001msec

As you can see, read-performance is very bad! Writes are fine, but reads
are not acceptable!
By the way, for reads the parameter "direct=0" is set and I can see that
the reads are not even hitting the storage system!

I have then tried to remove the writers from the fio config files, and
the read-results are as we would expect it:

  read : io=1522.1MB, bw=25977KB/s, iops=6494 , runt= 60001msec
  read : io=1526.3MB, bw=26047KB/s, iops=6511 , runt= 60000msec
  read : io=1526.9MB, bw=26058KB/s, iops=6514 , runt= 60001msec
  read : io=990.98MB, bw=16912KB/s, iops=4227 , runt= 60001msec
  read : io=990.85MB, bw=16910KB/s, iops=4227 , runt= 60001msec
  read : io=1107.5MB, bw=18900KB/s, iops=4724 , runt= 60001msec
  read : io=1006.8MB, bw=17181KB/s, iops=4295 , runt= 60001msec
  read : io=1011.2MB, bw=17256KB/s, iops=4314 , runt= 60001msec
  read : io=1046.2MB, bw=17854KB/s, iops=4463 , runt= 60001msec
  read : io=987.33MB, bw=16850KB/s, iops=4212 , runt= 60001msec
  read : io=991.72MB, bw=16925KB/s, iops=4231 , runt= 60000msec
  read : io=1102.4MB, bw=18813KB/s, iops=4703 , runt= 60001msec
  read : io=1014.8MB, bw=17318KB/s, iops=4329 , runt= 60001msec
  read : io=1012.1MB, bw=17287KB/s, iops=4321 , runt= 60001msec
  read : io=1128.7MB, bw=19252KB/s, iops=4813 , runt= 60001msec
  read : io=1186.1MB, bw=20257KB/s, iops=5064 , runt= 60001msec
  read : io=996.34MB, bw=17004KB/s, iops=4250 , runt= 60001msec
  read : io=1051.6MB, bw=17945KB/s, iops=4486 , runt= 60001msec
  read : io=1012.9MB, bw=17286KB/s, iops=4321 , runt= 60001msec
  read : io=1014.8MB, bw=17318KB/s, iops=4329 , runt= 60001msec
  read : io=1125.5MB, bw=19208KB/s, iops=4801 , runt= 60001msec
  read : io=1125.3MB, bw=19204KB/s, iops=4800 , runt= 60001msec
  read : io=1005.2MB, bw=17155KB/s, iops=4288 , runt= 60001msec
  read : io=1016.7MB, bw=17351KB/s, iops=4337 , runt= 60001msec

We are on a RHEL 5.5 Box, 8 Gb FC. With another tool I receive about
180k 4k direct (!) random-read IOPS (with 64 threads to 4 LUNs).

We have also tried to use separate read- and write config files. But as
soon as we run the tests against the same LUN, read-performance drops!

Any feedback is highly appreciated!

Cheers,
Matt