fio is being killed by the oom-killer after fio verify runs for some time ~13 hours

* fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
@ 2013-01-10 20:57 brian arb
  2013-01-10 21:09 ` Chris Worley
  2013-01-11  9:10 ` Jens Axboe
  0 siblings, 2 replies; 13+ messages in thread
From: brian arb @ 2013-01-10 20:57 UTC (permalink / raw)
  To: fio

Seems fio is being killed by the oom-killer after fio verify runs for
some time ~13 hours. What parameters can I tweak or how can I run my
test differently so the test will be completed with out interruption?

Any help is appericated

My System has 16GB of memory, 12 x 3TB Drives and one 60GB SSD

Disk information
cat /sys/bus/scsi/devices/*/model | sort | uniq -c
     12 Hitachi HUA72303
      1 KINGSTON SVP200S

fio --version
fio 1.50

uname -a
Linux myhostname 2.6.39-t4.el5 #1 SMP Tue Jan 31 13:18:29 PST 2012
x86_64 x86_64 x86_64 GNU/Linux

the script:
#!/bin/bash
set -e

IO_DEPTH=1
BLOCK_SIZE=4096
IO_ENGINE=libaio
READ_WRITE=write
DIRECT=1
VERIFY=crc32c-intel
BUFFERED=0
VERIFY_FATAL=1

/usr/bin/fio \
	--minimal \
	--readwrite=${READ_WRITE} \
	--blocksize=${BLOCK_SIZE} \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdj \
	--new_group \
	--filename=/dev/sdj \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdm \
	--new_group \
	--filename=/dev/sdm \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdd \
	--new_group \
	--filename=/dev/sdd \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sde \
	--new_group \
	--filename=/dev/sde \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdf \
	--new_group \
	--filename=/dev/sdf \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdg \
	--new_group \
	--filename=/dev/sdg \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sda \
	--new_group \
	--filename=/dev/sda \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdb \
	--new_group \
	--filename=/dev/sdb \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdc \
	--new_group \
	--filename=/dev/sdc \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdl \
	--new_group \
	--filename=/dev/sdl \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdk \
	--new_group \
	--filename=/dev/sdk \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdh \
	--new_group \
	--filename=/dev/sdh \
	--ioengine=${IO_ENGINE} \
	--direct=${DIRECT} \
	--buffered=${BUFFERED} \
	--iodepth=${IO_DEPTH} \
	--verify=${VERIFY} \
	--verify_fatal=${VERIFY_FATAL} \
	--name=verify_sdi \
	--new_group \
	--filename=/dev/sdi

^ permalink raw reply	[flat|nested] 13+ messages in thread