during fio scenario with verify=meta some jobs fails

[Samuel Shapiro <samuel.sh79@gmail.com>]

Hi All,

I'd recently started using fio for NAS FS tests, and it's really great
tool, as for loading system,
so for finding various protocols bugs. Thank You :)
One of the scenarios, I'm using is "fsx", that is slightly modified to
run with 16 jobs:

; This job file works pretty works similarly to running fsx-linux
; with -r 4096 -w 4096 -Z -N 500000
[global]
verify=meta
verify_dump=1

[fsx-file-job1]
directory=/mnt/FIO-some-mountpoint-to-NAS-01
ioengine=libaio
iodepth=256
rw=randrw
size=256k
bs=4k
norandommap
direct=1
loops=500000

[fsx-file-job2]
directory=/mnt/FIO-some-mountpoint-to-NAS-02
ioengine=libaio
iodepth=256
rw=randrw
size=256k
bs=4k
norandommap
direct=1
loops=500000
.
.
.
[fsx-file-job16]
directory=/mnt/FIO-some-mountpoint-to-NAS-03
etc....

After several minutes if running scenario, jobs starting to fail on
"verify meta", generating
".state" files.
When I tried to run the same scenario with verify=md5 , there were no
more verify failures reported.

So I have couple of questions:
1. What is the possible reason verify=meta can fail?
2. Should I suspect my File System for data corruption or there is
some known fio issue?
3. How can I parse ".state" file so I could compare its data to
original file on storage?

Thank You,
Samuel