* fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
@ 2013-01-10 20:57 brian arb
2013-01-10 21:09 ` Chris Worley
2013-01-11 9:10 ` Jens Axboe
0 siblings, 2 replies; 13+ messages in thread
From: brian arb @ 2013-01-10 20:57 UTC (permalink / raw)
To: fio
Seems fio is being killed by the oom-killer after fio verify runs for
some time ~13 hours. What parameters can I tweak or how can I run my
test differently so the test will be completed with out interruption?
Any help is appericated
My System has 16GB of memory, 12 x 3TB Drives and one 60GB SSD
Disk information
cat /sys/bus/scsi/devices/*/model | sort | uniq -c
12 Hitachi HUA72303
1 KINGSTON SVP200S
fio --version
fio 1.50
uname -a
Linux myhostname 2.6.39-t4.el5 #1 SMP Tue Jan 31 13:18:29 PST 2012
x86_64 x86_64 x86_64 GNU/Linux
the script:
#!/bin/bash
set -e
IO_DEPTH=1
BLOCK_SIZE=4096
IO_ENGINE=libaio
READ_WRITE=write
DIRECT=1
VERIFY=crc32c-intel
BUFFERED=0
VERIFY_FATAL=1
/usr/bin/fio \
--minimal \
--readwrite=${READ_WRITE} \
--blocksize=${BLOCK_SIZE} \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdj \
--new_group \
--filename=/dev/sdj \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdm \
--new_group \
--filename=/dev/sdm \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdd \
--new_group \
--filename=/dev/sdd \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sde \
--new_group \
--filename=/dev/sde \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdf \
--new_group \
--filename=/dev/sdf \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdg \
--new_group \
--filename=/dev/sdg \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sda \
--new_group \
--filename=/dev/sda \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdb \
--new_group \
--filename=/dev/sdb \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdc \
--new_group \
--filename=/dev/sdc \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdl \
--new_group \
--filename=/dev/sdl \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdk \
--new_group \
--filename=/dev/sdk \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdh \
--new_group \
--filename=/dev/sdh \
--ioengine=${IO_ENGINE} \
--direct=${DIRECT} \
--buffered=${BUFFERED} \
--iodepth=${IO_DEPTH} \
--verify=${VERIFY} \
--verify_fatal=${VERIFY_FATAL} \
--name=verify_sdi \
--new_group \
--filename=/dev/sdi
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-10 20:57 fio is being killed by the oom-killer after fio verify runs for some time ~13 hours brian arb
@ 2013-01-10 21:09 ` Chris Worley
2013-01-11 9:10 ` Jens Axboe
1 sibling, 0 replies; 13+ messages in thread
From: Chris Worley @ 2013-01-10 21:09 UTC (permalink / raw)
To: brian arb; +Cc: fio
Try norandommap ... that gives me an issue sometimes if I don't have
enough memory.
On Thu, Jan 10, 2013 at 1:57 PM, brian arb <brianjamesarb@gmail.com> wrote:
> Seems fio is being killed by the oom-killer after fio verify runs for
> some time ~13 hours. What parameters can I tweak or how can I run my
> test differently so the test will be completed with out interruption?
>
> Any help is appericated
>
> My System has 16GB of memory, 12 x 3TB Drives and one 60GB SSD
>
> Disk information
> cat /sys/bus/scsi/devices/*/model | sort | uniq -c
> 12 Hitachi HUA72303
> 1 KINGSTON SVP200S
>
> fio --version
> fio 1.50
>
> uname -a
> Linux myhostname 2.6.39-t4.el5 #1 SMP Tue Jan 31 13:18:29 PST 2012
> x86_64 x86_64 x86_64 GNU/Linux
>
>
> the script:
> #!/bin/bash
> set -e
>
> IO_DEPTH=1
> BLOCK_SIZE=4096
> IO_ENGINE=libaio
> READ_WRITE=write
> DIRECT=1
> VERIFY=crc32c-intel
> BUFFERED=0
> VERIFY_FATAL=1
>
> /usr/bin/fio \
> --minimal \
> --readwrite=${READ_WRITE} \
> --blocksize=${BLOCK_SIZE} \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdj \
> --new_group \
> --filename=/dev/sdj \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdm \
> --new_group \
> --filename=/dev/sdm \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdd \
> --new_group \
> --filename=/dev/sdd \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sde \
> --new_group \
> --filename=/dev/sde \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdf \
> --new_group \
> --filename=/dev/sdf \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdg \
> --new_group \
> --filename=/dev/sdg \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sda \
> --new_group \
> --filename=/dev/sda \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdb \
> --new_group \
> --filename=/dev/sdb \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdc \
> --new_group \
> --filename=/dev/sdc \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdl \
> --new_group \
> --filename=/dev/sdl \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdk \
> --new_group \
> --filename=/dev/sdk \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdh \
> --new_group \
> --filename=/dev/sdh \
> --ioengine=${IO_ENGINE} \
> --direct=${DIRECT} \
> --buffered=${BUFFERED} \
> --iodepth=${IO_DEPTH} \
> --verify=${VERIFY} \
> --verify_fatal=${VERIFY_FATAL} \
> --name=verify_sdi \
> --new_group \
> --filename=/dev/sdi
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-10 20:57 fio is being killed by the oom-killer after fio verify runs for some time ~13 hours brian arb
2013-01-10 21:09 ` Chris Worley
@ 2013-01-11 9:10 ` Jens Axboe
2013-01-12 8:51 ` Jens Axboe
1 sibling, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2013-01-11 9:10 UTC (permalink / raw)
To: brian arb; +Cc: fio
On Thu, Jan 10 2013, brian arb wrote:
> Seems fio is being killed by the oom-killer after fio verify runs for
> some time ~13 hours. What parameters can I tweak or how can I run my
> test differently so the test will be completed with out interruption?
You are probably running into OOM issues since each completed write will
log some meta data to help verify that later. The easiest fix for you
would be to verify continously, setting a backlog of how old data can
get before being verified. See verify_backlog and verify_async for that.
You should also upgrade your fio. Fio uses a random map for tracking
what has been written. It's static memory, so it wont cause your OOM
during runtime, but it will gobble up some memory when you start. If you
upgrade to 2.0.13 and use random_distribution=lfsr, then that memory
consumption will go away.
There's room for a bit of improvement on fio for verification. Since IO
buffer contents and offsets etc are fully randomized with specific
seeding, it is possible to verify what has been written without storing
this meta data. Basically verify can just re-create the contents for
verification, instead of storing a checksum of it. That will cost some
CPU, but it will get you more predictable (and much lower) memory
consumption numbers. I will look into that. But as a starter, the above
suggestions should help you out.
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-11 9:10 ` Jens Axboe
@ 2013-01-12 8:51 ` Jens Axboe
2013-01-18 17:25 ` brian arb
0 siblings, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2013-01-12 8:51 UTC (permalink / raw)
To: brian arb; +Cc: fio
On 2013-01-11 10:10, Jens Axboe wrote:
> On Thu, Jan 10 2013, brian arb wrote:
>> Seems fio is being killed by the oom-killer after fio verify runs for
>> some time ~13 hours. What parameters can I tweak or how can I run my
>> test differently so the test will be completed with out interruption?
>
> You are probably running into OOM issues since each completed write will
> log some meta data to help verify that later. The easiest fix for you
> would be to verify continously, setting a backlog of how old data can
> get before being verified. See verify_backlog and verify_async for that.
>
> You should also upgrade your fio. Fio uses a random map for tracking
> what has been written. It's static memory, so it wont cause your OOM
> during runtime, but it will gobble up some memory when you start. If you
> upgrade to 2.0.13 and use random_distribution=lfsr, then that memory
> consumption will go away.
>
> There's room for a bit of improvement on fio for verification. Since IO
> buffer contents and offsets etc are fully randomized with specific
> seeding, it is possible to verify what has been written without storing
> this meta data. Basically verify can just re-create the contents for
> verification, instead of storing a checksum of it. That will cost some
> CPU, but it will get you more predictable (and much lower) memory
> consumption numbers. I will look into that. But as a starter, the above
> suggestions should help you out.
I committed the first part of this. Now I just need to double check that
we re-seed properly, then we can dump the meta data storage for the
"normal" verify workload (that has both a write and a read phase).
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-12 8:51 ` Jens Axboe
@ 2013-01-18 17:25 ` brian arb
2013-01-18 20:40 ` Jens Axboe
0 siblings, 1 reply; 13+ messages in thread
From: brian arb @ 2013-01-18 17:25 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]
adding the option --verify_backlog=1024 resolved my issue.
On a side note how would one determine what the optimum number is for this
option?
On Sat, Jan 12, 2013 at 3:51 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2013-01-11 10:10, Jens Axboe wrote:
> > On Thu, Jan 10 2013, brian arb wrote:
> >> Seems fio is being killed by the oom-killer after fio verify runs for
> >> some time ~13 hours. What parameters can I tweak or how can I run my
> >> test differently so the test will be completed with out interruption?
> >
> > You are probably running into OOM issues since each completed write will
> > log some meta data to help verify that later. The easiest fix for you
> > would be to verify continously, setting a backlog of how old data can
> > get before being verified. See verify_backlog and verify_async for that.
> >
> > You should also upgrade your fio. Fio uses a random map for tracking
> > what has been written. It's static memory, so it wont cause your OOM
> > during runtime, but it will gobble up some memory when you start. If you
> > upgrade to 2.0.13 and use random_distribution=lfsr, then that memory
> > consumption will go away.
> >
> > There's room for a bit of improvement on fio for verification. Since IO
> > buffer contents and offsets etc are fully randomized with specific
> > seeding, it is possible to verify what has been written without storing
> > this meta data. Basically verify can just re-create the contents for
> > verification, instead of storing a checksum of it. That will cost some
> > CPU, but it will get you more predictable (and much lower) memory
> > consumption numbers. I will look into that. But as a starter, the above
> > suggestions should help you out.
>
> I committed the first part of this. Now I just need to double check that
> we re-seed properly, then we can dump the meta data storage for the
> "normal" verify workload (that has both a write and a read phase).
>
> --
> Jens Axboe
>
>
[-- Attachment #2: Type: text/html, Size: 2534 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-18 17:25 ` brian arb
@ 2013-01-18 20:40 ` Jens Axboe
2013-01-21 21:29 ` Jens Axboe
0 siblings, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2013-01-18 20:40 UTC (permalink / raw)
To: brian arb; +Cc: fio
On Fri, Jan 18 2013, brian arb wrote:
> adding the option --verify_backlog=1024 resolved my issue.
> On a side note how would one determine what the optimum number is for this
> option?
It's easier if you know the internals... The version you are running
will store some metadata for each block written, for verification
purposes. Standard install is 80 bytes per block. So if you have a 2TB
disk and use 4K IO sizes, the memory required runs into the gigabytes.
Quite unfortunate, really, will be fixed up shortly.
In any case, with a backlog of just 1024, you wont use more than 80KB
memory for verifies. So you could easily size it larger. I'd improve the
option documentation, but I think it's better to just improve fio to not
require the stored meta data to be able to verify. It will be in the
2.0.14 release.
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-18 20:40 ` Jens Axboe
@ 2013-01-21 21:29 ` Jens Axboe
2013-01-23 18:21 ` Jens Axboe
0 siblings, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2013-01-21 21:29 UTC (permalink / raw)
To: brian arb; +Cc: fio
On Fri, Jan 18 2013, Jens Axboe wrote:
> On Fri, Jan 18 2013, brian arb wrote:
> > adding the option --verify_backlog=1024 resolved my issue.
> > On a side note how would one determine what the optimum number is for this
> > option?
>
> It's easier if you know the internals... The version you are running
> will store some metadata for each block written, for verification
> purposes. Standard install is 80 bytes per block. So if you have a 2TB
> disk and use 4K IO sizes, the memory required runs into the gigabytes.
> Quite unfortunate, really, will be fixed up shortly.
>
> In any case, with a backlog of just 1024, you wont use more than 80KB
> memory for verifies. So you could easily size it larger. I'd improve the
> option documentation, but I think it's better to just improve fio to not
> require the stored meta data to be able to verify. It will be in the
> 2.0.14 release.
Current -git has it implemented. If you do your normal verify workload
but add --experimental_verify=1 to the options, it wont track meta data
needed for verify, but rather just re-wind the various attributes to
replay the workload for read and verify instead. Just tested it on one
box here, bringing the needed memory down for a job from 25G to 1M. The
worst part about the verify memory consumption was the slowly growing
job, meaning that it could have the OOM consequences that you described
for long running jobs. No allocations are taking place for IO when
experimental_verify is set now.
Side note, this doesn't work for mixed write/trim workloads for now. We
need a bit of extra tracking for that.
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-21 21:29 ` Jens Axboe
@ 2013-01-23 18:21 ` Jens Axboe
2013-01-23 18:52 ` brian arb
2013-01-25 20:03 ` brian arb
0 siblings, 2 replies; 13+ messages in thread
From: Jens Axboe @ 2013-01-23 18:21 UTC (permalink / raw)
To: brian arb; +Cc: fio
On Mon, Jan 21 2013, Jens Axboe wrote:
> On Fri, Jan 18 2013, Jens Axboe wrote:
> > On Fri, Jan 18 2013, brian arb wrote:
> > > adding the option --verify_backlog=1024 resolved my issue.
> > > On a side note how would one determine what the optimum number is for this
> > > option?
> >
> > It's easier if you know the internals... The version you are running
> > will store some metadata for each block written, for verification
> > purposes. Standard install is 80 bytes per block. So if you have a 2TB
> > disk and use 4K IO sizes, the memory required runs into the gigabytes.
> > Quite unfortunate, really, will be fixed up shortly.
> >
> > In any case, with a backlog of just 1024, you wont use more than 80KB
> > memory for verifies. So you could easily size it larger. I'd improve the
> > option documentation, but I think it's better to just improve fio to not
> > require the stored meta data to be able to verify. It will be in the
> > 2.0.14 release.
>
> Current -git has it implemented. If you do your normal verify workload
> but add --experimental_verify=1 to the options, it wont track meta data
> needed for verify, but rather just re-wind the various attributes to
> replay the workload for read and verify instead. Just tested it on one
> box here, bringing the needed memory down for a job from 25G to 1M. The
> worst part about the verify memory consumption was the slowly growing
> job, meaning that it could have the OOM consequences that you described
> for long running jobs. No allocations are taking place for IO when
> experimental_verify is set now.
>
> Side note, this doesn't work for mixed write/trim workloads for now. We
> need a bit of extra tracking for that.
Final bits are in now, works for both trim and mixed workloads etc. So
to recap, testing now with minimal mem pressure does not need
verify_backlog= being set. Suggested new parameters:
--experimental_verify=1 --random_generator=lfsr
if you could try that and report back if things ran successfully, I
would much appreciate it.
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-23 18:21 ` Jens Axboe
@ 2013-01-23 18:52 ` brian arb
2013-01-25 20:03 ` brian arb
1 sibling, 0 replies; 13+ messages in thread
From: brian arb @ 2013-01-23 18:52 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]
The changes you made are much appreciated! I'll test it out and give you
some feedback soon.
On Wed, Jan 23, 2013 at 1:21 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On Mon, Jan 21 2013, Jens Axboe wrote:
> > On Fri, Jan 18 2013, Jens Axboe wrote:
> > > On Fri, Jan 18 2013, brian arb wrote:
> > > > adding the option --verify_backlog=1024 resolved my issue.
> > > > On a side note how would one determine what the optimum number is
> for this
> > > > option?
> > >
> > > It's easier if you know the internals... The version you are running
> > > will store some metadata for each block written, for verification
> > > purposes. Standard install is 80 bytes per block. So if you have a 2TB
> > > disk and use 4K IO sizes, the memory required runs into the gigabytes.
> > > Quite unfortunate, really, will be fixed up shortly.
> > >
> > > In any case, with a backlog of just 1024, you wont use more than 80KB
> > > memory for verifies. So you could easily size it larger. I'd improve
> the
> > > option documentation, but I think it's better to just improve fio to
> not
> > > require the stored meta data to be able to verify. It will be in the
> > > 2.0.14 release.
> >
> > Current -git has it implemented. If you do your normal verify workload
> > but add --experimental_verify=1 to the options, it wont track meta data
> > needed for verify, but rather just re-wind the various attributes to
> > replay the workload for read and verify instead. Just tested it on one
> > box here, bringing the needed memory down for a job from 25G to 1M. The
> > worst part about the verify memory consumption was the slowly growing
> > job, meaning that it could have the OOM consequences that you described
> > for long running jobs. No allocations are taking place for IO when
> > experimental_verify is set now.
> >
> > Side note, this doesn't work for mixed write/trim workloads for now. We
> > need a bit of extra tracking for that.
>
> Final bits are in now, works for both trim and mixed workloads etc. So
> to recap, testing now with minimal mem pressure does not need
> verify_backlog= being set. Suggested new parameters:
>
> --experimental_verify=1 --random_generator=lfsr
>
> if you could try that and report back if things ran successfully, I
> would much appreciate it.
>
> --
> Jens Axboe
>
>
[-- Attachment #2: Type: text/html, Size: 2941 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-23 18:21 ` Jens Axboe
2013-01-23 18:52 ` brian arb
@ 2013-01-25 20:03 ` brian arb
2013-01-25 20:06 ` brian arb
2013-01-25 21:22 ` Jens Axboe
1 sibling, 2 replies; 13+ messages in thread
From: brian arb @ 2013-01-25 20:03 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
[-- Attachment #1: Type: text/plain, Size: 34567 bytes --]
Hi Jen,
It appears every thing ran as expected.
fio version: fio-2.0.13-29-ga240e
list of the disks being tested:
Smart Array P410 in Slot 1 (sn: PACCRID122806HJ)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 2 TB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 2 TB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 2 TB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 2 TB, OK)
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 2 TB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 2 TB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 2 TB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 2 TB, OK)
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 2 TB, OK)
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 2 TB, OK)
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 2 TB, OK)
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 2 TB, OK)
physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SATA, 500 GB, OK)
Expander 250 (WWID: 50014380221F43A0, Port: 1I, Box: 1)
Enclosure SEP (Vendor ID HP, Model DL18xG6BP) 248 (WWID:
50014380221F43B3, Port: 1I, Box: 1)
SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 249 (WWID: 5001438021D95EBF)
output from pmap:
14446: /usr/bin/fio --minimal --readwrite=write --blocksize=65536
--ioengine=libaio --direct=1 --buffered=0 --iodepth=64
--verify=crc32c-intel --experimental_verify=1 --random_generator=lfsr
--verify_fatal=1 --name=verify_sda --new_group --filename=/dev/sda
--ioengine=libaio --direct=1 --buffered=0 --iodepth=64
--verify=crc32c-intel --experimental_verify=1 --random_generator=lfsr
--verify_fatal=1 --name=verify_sdb --new_group --filename=/dev/sdb
--ioengine=libaio --direct=1 --buffered=0 --iodepth=64 --verify=crc3
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 368 228 228 r-x-- fio
000000000065b000 620 212 212 rw--- fio
00000000006f6000 52 36 36 rw--- [ anon ]
000000000231c000 140 28 28 rw--- [ anon ]
00007f1b40000000 132 4 4 rw--- [ anon ]
00007f1b40021000 65404 0 0 ----- [ anon ]
00007f1b46c52000 4 4 4 rw-s- zero (deleted)
00007f1b46c53000 4 4 4 rw-s- zero (deleted)
00007f1b46c54000 4 4 4 rw-s- zero (deleted)
00007f1b46c55000 4 4 4 rw-s- zero (deleted)
00007f1b46c56000 4 4 4 rw-s- zero (deleted)
00007f1b46c57000 4 4 4 rw-s- zero (deleted)
00007f1b46c58000 4 4 4 rw-s- zero (deleted)
00007f1b46c59000 4 4 4 rw-s- zero (deleted)
00007f1b46c5a000 4 4 4 rw-s- zero (deleted)
00007f1b46c5b000 4 4 4 rw-s- zero (deleted)
00007f1b46c5c000 4 4 4 rw-s- zero (deleted)
00007f1b46c5d000 4 4 4 rw-s- zero (deleted)
00007f1b46c5e000 4 4 4 rw-s- zero (deleted)
00007f1b46c5f000 4 0 0 ----- [ anon ]
00007f1b46c60000 10240 8 8 rw--- [ anon ]
00007f1b47660000 4 4 4 rw-s- zero (deleted)
00007f1b47661000 4 4 4 rw-s- zero (deleted)
00007f1b47662000 4 4 4 rw-s- zero (deleted)
00007f1b47663000 4 4 4 rw-s- zero (deleted)
00007f1b47664000 4 4 4 rw-s- zero (deleted)
00007f1b47665000 4 4 4 rw-s- zero (deleted)
00007f1b47666000 4 4 4 rw-s- zero (deleted)
00007f1b47667000 4 4 4 rw-s- zero (deleted)
00007f1b47668000 4 4 4 rw-s- zero (deleted)
00007f1b47669000 4 4 4 rw-s- zero (deleted)
00007f1b4766a000 4 4 4 rw-s- zero (deleted)
00007f1b4766b000 4 4 4 rw-s- zero (deleted)
00007f1b4766c000 4 4 4 rw-s- zero (deleted)
00007f1b4766d000 4 4 4 rw-s- zero (deleted)
00007f1b4766e000 4 4 4 rw-s- zero (deleted)
00007f1b4766f000 4 4 4 rw-s- zero (deleted)
00007f1b47670000 4 4 4 rw-s- zero (deleted)
00007f1b47671000 4 4 4 rw-s- zero (deleted)
00007f1b47672000 4 4 4 rw-s- zero (deleted)
00007f1b47673000 40636 40636 40636 rw-s- [ shmid=0x8000 ]
00007f1b49e22000 96836 44 44 r---- locale-archive
00007f1b4fcb3000 8228 8228 8228 rw-s- zero (deleted)
00007f1b504bc000 1560 484 484 r-x-- libc-2.12.so
00007f1b50642000 2048 0 0 ----- libc-2.12.so
00007f1b50842000 16 16 16 r---- libc-2.12.so
00007f1b50846000 4 4 4 rw--- libc-2.12.so
00007f1b50847000 20 16 16 rw--- [ anon ]
00007f1b5084c000 8 8 8 r-x-- libdl-2.12.so
00007f1b5084e000 2048 0 0 ----- libdl-2.12.so
00007f1b50a4e000 4 4 4 r---- libdl-2.12.so
00007f1b50a4f000 4 4 4 rw--- libdl-2.12.so
00007f1b50a50000 92 72 72 r-x-- libpthread-2.12.so
00007f1b50a67000 2044 0 0 ----- libpthread-2.12.so
00007f1b50c66000 4 4 4 r---- libpthread-2.12.so
00007f1b50c67000 4 4 4 rw--- libpthread-2.12.so
00007f1b50c68000 16 4 4 rw--- [ anon ]
00007f1b50c6c000 4 4 4 r-x-- libaio.so.1.0.1
00007f1b50c6d000 2044 0 0 ----- libaio.so.1.0.1
00007f1b50e6c000 4 4 4 rw--- libaio.so.1.0.1
00007f1b50e6d000 28 20 20 r-x-- librt-2.12.so
00007f1b50e74000 2044 0 0 ----- librt-2.12.so
00007f1b51073000 4 4 4 r---- librt-2.12.so
00007f1b51074000 4 4 4 rw--- librt-2.12.so
00007f1b51075000 524 20 20 r-x-- libm-2.12.so
00007f1b510f8000 2044 0 0 ----- libm-2.12.so
00007f1b512f7000 4 4 4 r---- libm-2.12.so
00007f1b512f8000 4 4 4 rw--- libm-2.12.so
00007f1b512f9000 128 104 104 r-x-- ld-2.12.so
00007f1b5150a000 20 20 20 rw--- [ anon ]
00007f1b51514000 4 4 4 rw-s- zero (deleted)
00007f1b51515000 4 4 4 rw-s- zero (deleted)
00007f1b51516000 4 4 4 rw-s- zero (deleted)
00007f1b51517000 4 4 4 rw--- [ anon ]
00007f1b51518000 4 4 4 r---- ld-2.12.so
00007f1b51519000 4 4 4 rw--- ld-2.12.so
00007f1b5151a000 4 4 4 rw--- [ anon ]
00007fffd4274000 88 44 44 rw--- [ stack ]
00007fffd43e1000 4 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------ ------ ------
total kB 237640 50436 50432
output from fio verify
3;fio-2.0.13-29-ga240e;verify_sda;0;0;1953481792;58213;909;33557145;9;1313;17.672265;5.400529;49708;384569;70326.106834;4792.280663;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=98816;99.99%=226304;0%=0;0%=0;0%=0;49724;384588;70344.020321;4792.253522;21844;76032;100.000000%;58263.147853;1536.756066;1953481792;48583;759;40208998;8;193342;16.621460;54.585787;2590;898206;84266.675226;12340.921878;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;2602;898221;84283.509061;12341.091703;11367;76774;100.000000%;48639.330189;3396.240803;2.062426%;1.691247%;39002830;0;34;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdb;1;0;1953481792;58209;909;33559583;8;1191;17.634260;5.419100;47915;382327;70331.219869;4796.001299;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=224256;0%=0;0%=0;0%=0;47932;382342;70349.105374;4795.970304;114;75136;100.000000%;58258.029230;1564.296175;1953481792;48582;759;40209235;10;492827;16.618412;100.721241;2300;899477;84266.926933;12367.231676;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=126464;99.90%=140288;99.95%=148480;99.99%=171008;0%=0;0%=0;0%=0;18224;899501;84283.760454;12367.659971;7037;57740;100.000000%;48639.631794;3390.481446;2.063188%;1.690176%;39026554;0;34;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.00%;0.01%;0.08%;95.00%;4.93%;0.01%;0.01%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdc;2;0;1953481792;58200;909;33564970;9;1109;17.337755;5.431915;49159;402048;70342.675698;4809.463793;1.00%=63232;5.00%=65280;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=226304;0%=0;0%=0;0%=0;49173;402063;70360.255419;4809.431452;19887;71808;100.000000%;58249.798011;1540.670248;1953481792;48583;759;40209078;9;90631;16.740572;24.303533;2367;898737;84266.358976;12351.046614;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;14330;898756;84283.301867;12351.110450;8359;57344;100.000000%;48638.592854;3393.572120;2.077428%;1.681800%;39018977;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.00%;0.01%;0.07%;94.98%;4.94%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdd;3;0;1953481792;58229;909;33548169;7;2168;17.352509;5.492676;79;370409;70307.440261;4435.330440;1.00%=63232;5.00%=65280;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=95744;99.99%=115200;0%=0;0%=0;0%=0;95;370423;70325.030463;4435.297186;6400;68096;100.000000%;58276.911278;1426.510561;1953481792;48582;759;40209545;7;990032;16.947586;184.093766;11052;900116;84264.486095;12338.751846;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=171008;0%=0;0%=0;0%=0;11067;1207740;84281.639314;12340.628960;52;81615;100.000000%;48639.807740;3392.226600;2.089567%;1.687281%;39037195;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.00%;0.00%;0.01%;0.08%;95.00%;4.92%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sde;4;0;1953481792;58225;909;33550400;8;1226;17.680698;5.410698;44075;377981;70311.899465;4462.694203;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=96768;99.99%=114176;0%=0;0%=0;0%=0;44091;378005;70329.838934;4462.661766;5248;71040;100.000000%;58273.685362;1452.672189;1953481792;48583;759;40209090;8;184495;16.673269;53.138988;2608;901185;84266.751764;12345.172786;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;2628;901201;84283.644250;12345.285941;10009;73846;100.000000%;48638.473913;3382.385295;2.060859%;1.701799%;39015083;0;34;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.07%;95.00%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdf;5;0;1953481792;58227;909;33549257;6;1153;17.361354;5.453471;49913;393167;70309.734170;4791.041354;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=226304;0%=0;0%=0;0%=0;49956;393183;70327.343517;4791.011392;8000;73088;100.000000%;58276.925354;1530.754345;1953481792;48582;759;40209218;6;510403;16.625422;93.286531;2222;898950;84266.162859;12354.171344;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;3470;898966;84282.994116;12354.427408;124;57625;100.000000%;48639.946695;3388.723319;2.063268%;1.676591%;39018401;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.00%;0.07%;94.98%;4.95%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdg;6;0;1953481792;58224;909;33550659;7;1219;17.412055;5.480904;49713;479638;70312.687923;4504.142853;1.00%=63232;5.00%=65280;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=96768;99.99%=120320;0%=0;0%=0;0%=0;49736;479653;70330.336462;4504.108164;104;72704;100.000000%;58273.251251;1466.466080;1953481792;48583;759;40209132;8;179968;16.833696;53.754999;5712;898160;84266.220461;12361.028188;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;5726;898179;84283.261595;12361.175717;16270;65867;100.000000%;48638.900038;3386.045236;2.080009%;1.697632%;39030679;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.07%;94.98%;4.95%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdh;7;0;1953481792;58250;910;33535635;8;915;17.368223;5.477037;49229;376997;70281.200737;4393.947699;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=95744;99.99%=113152;0%=0;0%=0;0%=0;49246;377013;70298.807154;4393.917514;118;63616;100.000000%;58299.234045;1424.283266;1953481792;48583;759;40209117;8;170611;17.161256;49.275710;3980;898487;84265.675675;12366.635093;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=142336;99.95%=150528;99.99%=173056;0%=0;0%=0;0%=0;4115;898504;84283.040704;12366.803605;16051;67415;100.000000%;48638.914125;3390.404216;2.086662%;1.708763%;39038001;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdi;8;0;1953481792;58250;910;33535747;8;1205;17.352585;5.488223;66;413704;70281.435748;4436.976307;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=95744;99.99%=110080;0%=0;0%=0;0%=0;80;413718;70299.031916;4436.946360;18778;62850;100.000000%;58297.264240;1411.060135;1953481792;48581;759;40210352;9;3465413;16.879511;627.508387;2497;897992;84261.988459;12353.828840;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=171008;0%=0;0%=0;0%=0;2513;3474758;84279.072127;12369.112211;18;114816;100.000000%;48642.517717;3421.598011;2.069417%;1.687665%;39016573;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdj;9;0;1953481792;58230;909;33547423;8;1676;17.349636;5.497407;48268;413442;70305.847829;4780.591827;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=224256;0%=0;0%=0;0%=0;48282;413462;70323.445638;4780.560400;13187;70272;100.000000%;58279.772865;1524.744456;1953481792;48583;759;40209086;8;169301;16.394425;51.211199;4337;898104;84267.262943;12351.073130;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;4473;898122;84283.875714;12351.155235;16189;66909;100.000000%;48639.774896;3412.293098;2.053515%;1.673543%;39004115;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.08%;94.99%;4.94%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdk;10;0;1953481792;58230;909;33547633;8;1065;17.375935;5.473649;49717;377847;70306.300846;4754.462148;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=222208;0%=0;0%=0;0%=0;49745;377861;70323.930470;4754.431842;18941;68453;100.000000%;58279.390915;1521.354035;1953481792;48583;759;40209116;8;181780;16.663474;53.301287;5923;897651;84266.618342;12347.927369;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;5937;897670;84283.502254;12348.072401;11042;66666;100.000000%;48641.405652;3399.696123;2.073710%;1.685050%;39022210;0;35;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.07%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdl;11;0;1953481792;58235;909;33544533;9;1111;17.367272;5.469736;50574;393494;70299.842990;4770.431257;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=97792;99.99%=222208;0%=0;0%=0;0%=0;50595;393508;70317.463931;4770.399378;8062;66048;100.000000%;58284.618446;1524.809714;1953481792;48583;759;40209139;5;201341;16.744767;59.224761;3944;893266;84266.339004;12356.284041;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=142336;99.95%=150528;99.99%=173056;0%=0;0%=0;0%=0;3998;893285;84283.300454;12356.456648;14209;73051;100.000000%;48639.899408;3401.158855;2.088608%;1.678533%;39022507;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdm;12;0;488353792;23065;360;21172858;8;322;15.511666;6.209519;40203;710447;177548.702516;17731.998701;1.00%=132096;5.00%=148480;10.00%=156672;20.00%=164864;30.00%=168960;40.00%=173056;50.00%=177152;60.00%=181248;70.00%=185344;80.00%=191488;90.00%=199680;95.00%=205824;99.00%=222208;99.50%=230400;99.90%=248832;99.95%=254976;99.99%=272384;0%=0;0%=0;0%=0;40222;710462;177564.431215;17731.898606;6688;34263;100.000000%;23116.149610;1949.123920;488353792;45835;716;10654562;6;1554804;17.600941;577.093065;9470;1618227;89319.169274;13929.576671;1.00%=64256;5.00%=71168;10.00%=74240;20.00%=78336;30.00%=81408;40.00%=84480;50.00%=88576;60.00%=90624;70.00%=94720;80.00%=99840;90.00%=107008;95.00%=113152;99.00%=126464;99.50%=130560;99.90%=140288;99.95%=144384;99.99%=156672;0%=0;0%=0;0%=0;9490;1642267;89336.985224;13941.641992;10711;54784;100.000000%;45897.103713;3189.870097;1.236685%;0.911208%;3888761;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.00%;0.05%;40.68%;59.22%;0.04%;0.01%;0.01%;0.01%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
On Wed, Jan 23, 2013 at 1:21 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On Mon, Jan 21 2013, Jens Axboe wrote:
> > On Fri, Jan 18 2013, Jens Axboe wrote:
> > > On Fri, Jan 18 2013, brian arb wrote:
> > > > adding the option --verify_backlog=1024 resolved my issue.
> > > > On a side note how would one determine what the optimum number is
> for this
> > > > option?
> > >
> > > It's easier if you know the internals... The version you are running
> > > will store some metadata for each block written, for verification
> > > purposes. Standard install is 80 bytes per block. So if you have a 2TB
> > > disk and use 4K IO sizes, the memory required runs into the gigabytes.
> > > Quite unfortunate, really, will be fixed up shortly.
> > >
> > > In any case, with a backlog of just 1024, you wont use more than 80KB
> > > memory for verifies. So you could easily size it larger. I'd improve
> the
> > > option documentation, but I think it's better to just improve fio to
> not
> > > require the stored meta data to be able to verify. It will be in the
> > > 2.0.14 release.
> >
> > Current -git has it implemented. If you do your normal verify workload
> > but add --experimental_verify=1 to the options, it wont track meta data
> > needed for verify, but rather just re-wind the various attributes to
> > replay the workload for read and verify instead. Just tested it on one
> > box here, bringing the needed memory down for a job from 25G to 1M. The
> > worst part about the verify memory consumption was the slowly growing
> > job, meaning that it could have the OOM consequences that you described
> > for long running jobs. No allocations are taking place for IO when
> > experimental_verify is set now.
> >
> > Side note, this doesn't work for mixed write/trim workloads for now. We
> > need a bit of extra tracking for that.
>
> Final bits are in now, works for both trim and mixed workloads etc. So
> to recap, testing now with minimal mem pressure does not need
> verify_backlog= being set. Suggested new parameters:
>
> --experimental_verify=1 --random_generator=lfsr
>
> if you could try that and report back if things ran successfully, I
> would much appreciate it.
>
> --
> Jens Axboe
>
>
[-- Attachment #2: Type: text/html, Size: 37235 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-25 20:03 ` brian arb
@ 2013-01-25 20:06 ` brian arb
2013-01-25 21:22 ` Jens Axboe
1 sibling, 0 replies; 13+ messages in thread
From: brian arb @ 2013-01-25 20:06 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Hi Jen,
It appears every thing ran as expected.
fio version: fio-2.0.13-29-ga240e
list of the disks being tested:
Smart Array P410 in Slot 1 (sn: PACCRID122806HJ)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 2 TB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 2 TB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 2 TB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 2 TB, OK)
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 2 TB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 2 TB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 2 TB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 2 TB, OK)
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 2 TB, OK)
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 2 TB, OK)
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 2 TB, OK)
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 2 TB, OK)
physicaldrive 1I:1:13 (port 1I:box 1:bay 13, SATA, 500 GB, OK)
Expander 250 (WWID: 50014380221F43A0, Port: 1I, Box: 1)
Enclosure SEP (Vendor ID HP, Model DL18xG6BP) 248 (WWID:
50014380221F43B3, Port: 1I, Box: 1)
SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 249 (WWID: 5001438021D95EBF)
output from pmap:
14446: /usr/bin/fio --minimal --readwrite=write --blocksize=65536
--ioengine=libaio --direct=1 --buffered=0 --iodepth=64 --verify=crc32c-intel
--experimental_verify=1 --random_generator=lfsr --verify_fatal=1
--name=verify_sda --new_group --filename=/dev/sda --ioengine=libaio
--direct=1 --buffered=0 --iodepth=64 --verify=crc32c-intel
--experimental_verify=1 --random_generator=lfsr --verify_fatal=1
--name=verify_sdb --new_group --filename=/dev/sdb --ioengine=libaio
--direct=1 --buffered=0 --iodepth=64 --verify=crc3
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 368 228 228 r-x-- fio
000000000065b000 620 212 212 rw--- fio
00000000006f6000 52 36 36 rw--- [ anon ]
000000000231c000 140 28 28 rw--- [ anon ]
00007f1b40000000 132 4 4 rw--- [ anon ]
00007f1b40021000 65404 0 0 ----- [ anon ]
00007f1b46c52000 4 4 4 rw-s- zero (deleted)
00007f1b46c53000 4 4 4 rw-s- zero (deleted)
00007f1b46c54000 4 4 4 rw-s- zero (deleted)
00007f1b46c55000 4 4 4 rw-s- zero (deleted)
00007f1b46c56000 4 4 4 rw-s- zero (deleted)
00007f1b46c57000 4 4 4 rw-s- zero (deleted)
00007f1b46c58000 4 4 4 rw-s- zero (deleted)
00007f1b46c59000 4 4 4 rw-s- zero (deleted)
00007f1b46c5a000 4 4 4 rw-s- zero (deleted)
00007f1b46c5b000 4 4 4 rw-s- zero (deleted)
00007f1b46c5c000 4 4 4 rw-s- zero (deleted)
00007f1b46c5d000 4 4 4 rw-s- zero (deleted)
00007f1b46c5e000 4 4 4 rw-s- zero (deleted)
00007f1b46c5f000 4 0 0 ----- [ anon ]
00007f1b46c60000 10240 8 8 rw--- [ anon ]
00007f1b47660000 4 4 4 rw-s- zero (deleted)
00007f1b47661000 4 4 4 rw-s- zero (deleted)
00007f1b47662000 4 4 4 rw-s- zero (deleted)
00007f1b47663000 4 4 4 rw-s- zero (deleted)
00007f1b47664000 4 4 4 rw-s- zero (deleted)
00007f1b47665000 4 4 4 rw-s- zero (deleted)
00007f1b47666000 4 4 4 rw-s- zero (deleted)
00007f1b47667000 4 4 4 rw-s- zero (deleted)
00007f1b47668000 4 4 4 rw-s- zero (deleted)
00007f1b47669000 4 4 4 rw-s- zero (deleted)
00007f1b4766a000 4 4 4 rw-s- zero (deleted)
00007f1b4766b000 4 4 4 rw-s- zero (deleted)
00007f1b4766c000 4 4 4 rw-s- zero (deleted)
00007f1b4766d000 4 4 4 rw-s- zero (deleted)
00007f1b4766e000 4 4 4 rw-s- zero (deleted)
00007f1b4766f000 4 4 4 rw-s- zero (deleted)
00007f1b47670000 4 4 4 rw-s- zero (deleted)
00007f1b47671000 4 4 4 rw-s- zero (deleted)
00007f1b47672000 4 4 4 rw-s- zero (deleted)
00007f1b47673000 40636 40636 40636 rw-s- [ shmid=0x8000 ]
00007f1b49e22000 96836 44 44 r---- locale-archive
00007f1b4fcb3000 8228 8228 8228 rw-s- zero (deleted)
00007f1b504bc000 1560 484 484 r-x-- libc-2.12.so
00007f1b50642000 2048 0 0 ----- libc-2.12.so
00007f1b50842000 16 16 16 r---- libc-2.12.so
00007f1b50846000 4 4 4 rw--- libc-2.12.so
00007f1b50847000 20 16 16 rw--- [ anon ]
00007f1b5084c000 8 8 8 r-x-- libdl-2.12.so
00007f1b5084e000 2048 0 0 ----- libdl-2.12.so
00007f1b50a4e000 4 4 4 r---- libdl-2.12.so
00007f1b50a4f000 4 4 4 rw--- libdl-2.12.so
00007f1b50a50000 92 72 72 r-x-- libpthread-2.12.so
00007f1b50a67000 2044 0 0 ----- libpthread-2.12.so
00007f1b50c66000 4 4 4 r---- libpthread-2.12.so
00007f1b50c67000 4 4 4 rw--- libpthread-2.12.so
00007f1b50c68000 16 4 4 rw--- [ anon ]
00007f1b50c6c000 4 4 4 r-x-- libaio.so.1.0.1
00007f1b50c6d000 2044 0 0 ----- libaio.so.1.0.1
00007f1b50e6c000 4 4 4 rw--- libaio.so.1.0.1
00007f1b50e6d000 28 20 20 r-x-- librt-2.12.so
00007f1b50e74000 2044 0 0 ----- librt-2.12.so
00007f1b51073000 4 4 4 r---- librt-2.12.so
00007f1b51074000 4 4 4 rw--- librt-2.12.so
00007f1b51075000 524 20 20 r-x-- libm-2.12.so
00007f1b510f8000 2044 0 0 ----- libm-2.12.so
00007f1b512f7000 4 4 4 r---- libm-2.12.so
00007f1b512f8000 4 4 4 rw--- libm-2.12.so
00007f1b512f9000 128 104 104 r-x-- ld-2.12.so
00007f1b5150a000 20 20 20 rw--- [ anon ]
00007f1b51514000 4 4 4 rw-s- zero (deleted)
00007f1b51515000 4 4 4 rw-s- zero (deleted)
00007f1b51516000 4 4 4 rw-s- zero (deleted)
00007f1b51517000 4 4 4 rw--- [ anon ]
00007f1b51518000 4 4 4 r---- ld-2.12.so
00007f1b51519000 4 4 4 rw--- ld-2.12.so
00007f1b5151a000 4 4 4 rw--- [ anon ]
00007fffd4274000 88 44 44 rw--- [ stack ]
00007fffd43e1000 4 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------ ------ ------
total kB 237640 50436 50432
output from fio verify
3;fio-2.0.13-29-ga240e;verify_sda;0;0;1953481792;58213;909;33557145;9;1313;17.672265;5.400529;49708;384569;70326.106834;4792.280663;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=98816;99.99%=226304;0%=0;0%=0;0%=0;49724;384588;70344.020321;4792.253522;21844;76032;100.000000%;58263.147853;1536.756066;1953481792;48583;759;40208998;8;193342;16.621460;54.585787;2590;898206;84266.675226;12340.921878;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;2602;898221;84283.509061;12341.091703;11367;76774;100.000000%;48639.330189;3396.240803;2.062426%;1.691247%;39002830;0;34;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdb;1;0;1953481792;58209;909;33559583;8;1191;17.634260;5.419100;47915;382327;70331.219869;4796.001299;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=224256;0%=0;0%=0;0%=0;47932;382342;70349.105374;4795.970304;114;75136;100.000000%;58258.029230;1564.296175;1953481792;48582;759;40209235;10;492827;16.618412;100.721241;2300;899477;84266.926933;12367.231676;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=126464;99.90%=140288;99.95%=148480;99.99%=171008;0%=0;0%=0;0%=0;18224;899501;84283.760454;12367.659971;7037;57740;100.000000%;48639.631794;3390.481446;2.063188%;1.690176%;39026554;0;34;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.00%;0.01%;0.08%;95.00%;4.93%;0.01%;0.01%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdc;2;0;1953481792;58200;909;33564970;9;1109;17.337755;5.431915;49159;402048;70342.675698;4809.463793;1.00%=63232;5.00%=65280;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=226304;0%=0;0%=0;0%=0;49173;402063;70360.255419;4809.431452;19887;71808;100.000000%;58249.798011;1540.670248;1953481792;48583;759;40209078;9;90631;16.740572;24.303533;2367;898737;84266.358976;12351.046614;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;14330;898756;84283.301867;12351.110450;8359;57344;100.000000%;48638.592854;3393.572120;2.077428%;1.681800%;39018977;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.00%;0.01%;0.07%;94.98%;4.94%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdd;3;0;1953481792;58229;909;33548169;7;2168;17.352509;5.492676;79;370409;70307.440261;4435.330440;1.00%=63232;5.00%=65280;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=95744;99.99%=115200;0%=0;0%=0;0%=0;95;370423;70325.030463;4435.297186;6400;68096;100.000000%;58276.911278;1426.510561;1953481792;48582;759;40209545;7;990032;16.947586;184.093766;11052;900116;84264.486095;12338.751846;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=171008;0%=0;0%=0;0%=0;11067;1207740;84281.639314;12340.628960;52;81615;100.000000%;48639.807740;3392.226600;2.089567%;1.687281%;39037195;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.00%;0.00%;0.01%;0.08%;95.00%;4.92%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sde;4;0;1953481792;58225;909;33550400;8;1226;17.680698;5.410698;44075;377981;70311.899465;4462.694203;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=96768;99.99%=114176;0%=0;0%=0;0%=0;44091;378005;70329.838934;4462.661766;5248;71040;100.000000%;58273.685362;1452.672189;1953481792;48583;759;40209090;8;184495;16.673269;53.138988;2608;901185;84266.751764;12345.172786;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;2628;901201;84283.644250;12345.285941;10009;73846;100.000000%;48638.473913;3382.385295;2.060859%;1.701799%;39015083;0;34;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.07%;95.00%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdf;5;0;1953481792;58227;909;33549257;6;1153;17.361354;5.453471;49913;393167;70309.734170;4791.041354;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=226304;0%=0;0%=0;0%=0;49956;393183;70327.343517;4791.011392;8000;73088;100.000000%;58276.925354;1530.754345;1953481792;48582;759;40209218;6;510403;16.625422;93.286531;2222;898950;84266.162859;12354.171344;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;3470;898966;84282.994116;12354.427408;124;57625;100.000000%;48639.946695;3388.723319;2.063268%;1.676591%;39018401;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.00%;0.07%;94.98%;4.95%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdg;6;0;1953481792;58224;909;33550659;7;1219;17.412055;5.480904;49713;479638;70312.687923;4504.142853;1.00%=63232;5.00%=65280;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=96768;99.99%=120320;0%=0;0%=0;0%=0;49736;479653;70330.336462;4504.108164;104;72704;100.000000%;58273.251251;1466.466080;1953481792;48583;759;40209132;8;179968;16.833696;53.754999;5712;898160;84266.220461;12361.028188;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;5726;898179;84283.261595;12361.175717;16270;65867;100.000000%;48638.900038;3386.045236;2.080009%;1.697632%;39030679;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.07%;94.98%;4.95%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdh;7;0;1953481792;58250;910;33535635;8;915;17.368223;5.477037;49229;376997;70281.200737;4393.947699;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=95744;99.99%=113152;0%=0;0%=0;0%=0;49246;377013;70298.807154;4393.917514;118;63616;100.000000%;58299.234045;1424.283266;1953481792;48583;759;40209117;8;170611;17.161256;49.275710;3980;898487;84265.675675;12366.635093;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=142336;99.95%=150528;99.99%=173056;0%=0;0%=0;0%=0;4115;898504;84283.040704;12366.803605;16051;67415;100.000000%;48638.914125;3390.404216;2.086662%;1.708763%;39038001;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdi;8;0;1953481792;58250;910;33535747;8;1205;17.352585;5.488223;66;413704;70281.435748;4436.976307;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=69120;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=95744;99.99%=110080;0%=0;0%=0;0%=0;80;413718;70299.031916;4436.946360;18778;62850;100.000000%;58297.264240;1411.060135;1953481792;48581;759;40210352;9;3465413;16.879511;627.508387;2497;897992;84261.988459;12353.828840;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=171008;0%=0;0%=0;0%=0;2513;3474758;84279.072127;12369.112211;18;114816;100.000000%;48642.517717;3421.598011;2.069417%;1.687665%;39016573;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdj;9;0;1953481792;58230;909;33547423;8;1676;17.349636;5.497407;48268;413442;70305.847829;4780.591827;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=224256;0%=0;0%=0;0%=0;48282;413462;70323.445638;4780.560400;13187;70272;100.000000%;58279.772865;1524.744456;1953481792;48583;759;40209086;8;169301;16.394425;51.211199;4337;898104;84267.262943;12351.073130;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;4473;898122;84283.875714;12351.155235;16189;66909;100.000000%;48639.774896;3412.293098;2.053515%;1.673543%;39004115;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.08%;94.99%;4.94%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdk;10;0;1953481792;58230;909;33547633;8;1065;17.375935;5.473649;49717;377847;70306.300846;4754.462148;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=93696;99.95%=97792;99.99%=222208;0%=0;0%=0;0%=0;49745;377861;70323.930470;4754.431842;18941;68453;100.000000%;58279.390915;1521.354035;1953481792;48583;759;40209116;8;181780;16.663474;53.301287;5923;897651;84266.618342;12347.927369;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=140288;99.95%=148480;99.99%=173056;0%=0;0%=0;0%=0;5937;897670;84283.502254;12348.072401;11042;66666;100.000000%;48641.405652;3399.696123;2.073710%;1.685050%;39022210;0;35;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.07%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdl;11;0;1953481792;58235;909;33544533;9;1111;17.367272;5.469736;50574;393494;70299.842990;4770.431257;1.00%=63232;5.00%=64768;10.00%=66048;20.00%=67072;30.00%=68096;40.00%=68096;50.00%=69120;60.00%=70144;70.00%=71168;80.00%=73216;90.00%=76288;95.00%=78336;99.00%=84480;99.50%=86528;99.90%=92672;99.95%=97792;99.99%=222208;0%=0;0%=0;0%=0;50595;393508;70317.463931;4770.399378;8062;66048;100.000000%;58284.618446;1524.809714;1953481792;48583;759;40209139;5;201341;16.744767;59.224761;3944;893266;84266.339004;12356.284041;1.00%=60160;5.00%=67072;10.00%=70144;20.00%=74240;30.00%=78336;40.00%=80384;50.00%=82432;60.00%=85504;70.00%=89600;80.00%=92672;90.00%=99840;95.00%=107008;99.00%=120320;99.50%=127488;99.90%=142336;99.95%=150528;99.99%=173056;0%=0;0%=0;0%=0;3998;893285;84283.300454;12356.456648;14209;73051;100.000000%;48639.899408;3401.158855;2.088608%;1.678533%;39022507;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.01%;0.01%;0.08%;94.99%;4.93%;0.01%;0.00%;0.01%;0.00%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
3;fio-2.0.13-29-ga240e;verify_sdm;12;0;488353792;23065;360;21172858;8;322;15.511666;6.209519;40203;710447;177548.702516;17731.998701;1.00%=132096;5.00%=148480;10.00%=156672;20.00%=164864;30.00%=168960;40.00%=173056;50.00%=177152;60.00%=181248;70.00%=185344;80.00%=191488;90.00%=199680;95.00%=205824;99.00%=222208;99.50%=230400;99.90%=248832;99.95%=254976;99.99%=272384;0%=0;0%=0;0%=0;40222;710462;177564.431215;17731.898606;6688;34263;100.000000%;23116.149610;1949.123920;488353792;45835;716;10654562;6;1554804;17.600941;577.093065;9470;1618227;89319.169274;13929.576671;1.00%=64256;5.00%=71168;10.00%=74240;20.00%=78336;30.00%=81408;40.00%=84480;50.00%=88576;60.00%=90624;70.00%=94720;80.00%=99840;90.00%=107008;95.00%=113152;99.00%=126464;99.50%=130560;99.90%=140288;99.95%=144384;99.99%=156672;0%=0;0%=0;0%=0;9490;1642267;89336.985224;13941.641992;10711;54784;100.000000%;45897.103713;3189.870097;1.236685%;0.911208%;3888761;0;33;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.00%;0.05%;40.68%;59.22%;0.04%;0.01%;0.01%;0.01%;0.00%;sda;30523251;30523153;0;0;2146666549;2571023519;422696987;100.00%;sdb;30523241;30523153;0;0;2146839630;2570915919;422759353;100.00%;sdc;30522992;30523153;0;0;2147198097;2570678149;422879491;100.00%;sdd;30522559;30523146;0;7;2146125316;2570991207;422121924;100.00%;sde;30523017;30523153;19;0;2146228816;2570856413;422089298;100.00%;sdf;30523060;30523124;14;29;2146184609;2570690055;421880299;100.00%;sdg;30522977;30523153;0;0;2146272625;2571004594;422279538;100.00%;sdh;30523053;30523153;0;0;2145309964;2570883739;421199014;100.00%;sdi;30522017;30523153;0;0;2145327127;2570631579;420962805;100.00%;sdj;30523031;30523153;0;0;2146066232;2570832836;421901932;100.00%;sdk;30523031;30523153;0;0;2146083305;2570866245;421952323;100.00%;sdl;30523125;30523097;0;56;2145889182;2570848211;421744411;100.00%;sdm;7630578;7630472;0;56;1354634644;681206659;2035841519;100.00%
>
> On Wed, Jan 23, 2013 at 1:21 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On Mon, Jan 21 2013, Jens Axboe wrote:
>> > On Fri, Jan 18 2013, Jens Axboe wrote:
>> > > On Fri, Jan 18 2013, brian arb wrote:
>> > > > adding the option --verify_backlog=1024 resolved my issue.
>> > > > On a side note how would one determine what the optimum number is
>> > > > for this
>> > > > option?
>> > >
>> > > It's easier if you know the internals... The version you are running
>> > > will store some metadata for each block written, for verification
>> > > purposes. Standard install is 80 bytes per block. So if you have a
>> > > 2TB
>> > > disk and use 4K IO sizes, the memory required runs into the
>> > > gigabytes.
>> > > Quite unfortunate, really, will be fixed up shortly.
>> > >
>> > > In any case, with a backlog of just 1024, you wont use more than 80KB
>> > > memory for verifies. So you could easily size it larger. I'd improve
>> > > the
>> > > option documentation, but I think it's better to just improve fio to
>> > > not
>> > > require the stored meta data to be able to verify. It will be in the
>> > > 2.0.14 release.
>> >
>> > Current -git has it implemented. If you do your normal verify workload
>> > but add --experimental_verify=1 to the options, it wont track meta data
>> > needed for verify, but rather just re-wind the various attributes to
>> > replay the workload for read and verify instead. Just tested it on one
>> > box here, bringing the needed memory down for a job from 25G to 1M. The
>> > worst part about the verify memory consumption was the slowly growing
>> > job, meaning that it could have the OOM consequences that you described
>> > for long running jobs. No allocations are taking place for IO when
>> > experimental_verify is set now.
>> >
>> > Side note, this doesn't work for mixed write/trim workloads for now. We
>> > need a bit of extra tracking for that.
>>
>> Final bits are in now, works for both trim and mixed workloads etc. So
>> to recap, testing now with minimal mem pressure does not need
>> verify_backlog= being set. Suggested new parameters:
>>
>> --experimental_verify=1 --random_generator=lfsr
>>
>> if you could try that and report back if things ran successfully, I
>> would much appreciate it.
>>
>> --
>> Jens Axboe
>>
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-25 20:03 ` brian arb
2013-01-25 20:06 ` brian arb
@ 2013-01-25 21:22 ` Jens Axboe
2013-01-25 21:28 ` brian arb
1 sibling, 1 reply; 13+ messages in thread
From: Jens Axboe @ 2013-01-25 21:22 UTC (permalink / raw)
To: brian arb; +Cc: fio
On Fri, Jan 25 2013, brian arb wrote:
> Hi Jen,
> It appears every thing ran as expected.
Great! I take it that means it:
1) Ran to completion
2) Completed the verify successfully
Correct?
BTW, fio also supports a json output format. It's more robust than the
csv format, so both easier to parse by a computer reliably, and readable
by humans as well.
--
Jens Axboe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
2013-01-25 21:22 ` Jens Axboe
@ 2013-01-25 21:28 ` brian arb
0 siblings, 0 replies; 13+ messages in thread
From: brian arb @ 2013-01-25 21:28 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Yes Sir,
It ran to completion with out incident and completed the verification
successfully too.
On Fri, Jan 25, 2013 at 4:22 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On Fri, Jan 25 2013, brian arb wrote:
>> Hi Jen,
>> It appears every thing ran as expected.
>
> Great! I take it that means it:
>
> 1) Ran to completion
> 2) Completed the verify successfully
>
> Correct?
>
> BTW, fio also supports a json output format. It's more robust than the
> csv format, so both easier to parse by a computer reliably, and readable
> by humans as well.
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2013-01-25 21:28 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-01-10 20:57 fio is being killed by the oom-killer after fio verify runs for some time ~13 hours brian arb
2013-01-10 21:09 ` Chris Worley
2013-01-11 9:10 ` Jens Axboe
2013-01-12 8:51 ` Jens Axboe
2013-01-18 17:25 ` brian arb
2013-01-18 20:40 ` Jens Axboe
2013-01-21 21:29 ` Jens Axboe
2013-01-23 18:21 ` Jens Axboe
2013-01-23 18:52 ` brian arb
2013-01-25 20:03 ` brian arb
2013-01-25 20:06 ` brian arb
2013-01-25 21:22 ` Jens Axboe
2013-01-25 21:28 ` brian arb
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.