Fio verify crc32 fault

Fio verify crc32 fault

[Peter Sabaini]

Hey,

I've got a fio verify run on an NVMe that gives me a crc32 verification fault, however it's not reproducible -- when rerunning on the same NVMe it comes back fine. 

Also I'm getting a funny "error=Invalid or incomplete multibyte or wide character" error reported, see below.

Any hints for this?

Many thanks,
peter.

Test output

Running command: sudo -n fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fio_test_verify --bs=4M --iodepth=64 --size=95% --verify=crc32c-intel --runtime=7200s --filename=/dev/nvme0n1 --readwrite=randwrite

fio_test_verify: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=64
fio-2.2.10
Starting 1 process
crc32c: verify failed at file /dev/nvme0n1 offset 480499466240, length 4194304
       Expected CRC: 5afe4d21
       Received CRC: db867b6c

fio_test_verify: (groupid=0, jobs=1): err=84 (file:io_u.c:1842, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=5437: Thu Nov 26 12:22:11 2020
  read : io=320324MB, bw=2662.2MB/s, iops=665, runt=120287msec
  write: io=1769.9GB, bw=1800.3MB/s, iops=450, runt=1006702msec
  cpu          : usr=54.13%, sys=4.95%, ctx=815675, majf=0, minf=63136
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=80081/w=453085/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=320324MB, aggrb=2662.2MB/s, minb=2662.2MB/s, maxb=2662.2MB/s, mint=120287msec, maxt=120287msec
  WRITE: io=1769.9GB, aggrb=1800.3MB/s, minb=1800.3MB/s, maxb=1800.3MB/s, mint=1006702msec, maxt=1006702msec

Disk stats (read/write):
  nvme0n1: ios=2565084/14498835, merge=5/0, ticks=122248144/1021942068, in_queue=1142848760, util=100.00%

--------------------------------------------------------------------------------

Re: Fio verify crc32 fault

[Sitsofe Wheeler]

Hi,

On Thu, 3 Dec 2020 at 17:27, Peter Sabaini <peter@sabaini.at> wrote:
>
> Hey,
>
> I've got a fio verify run on an NVMe that gives me a crc32 verification fault, however it's not reproducible -- when rerunning on the same NVMe it comes back fine.
>
> Also I'm getting a funny "error=Invalid or incomplete multibyte or wide character" error reported, see below.
>
> Any hints for this?
>
> Many thanks,
> peter.
>
> Test output
>
> Running command: sudo -n fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fio_test_verify --bs=4M --iodepth=64 --size=95% --verify=crc32c-intel --runtime=7200s --filename=/dev/nvme0n1 --readwrite=randwrite
>
> fio_test_verify: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=64
> fio-2.2.10

^^^ Your fio is VERY old... Does the same thing happen with a recent
fio (see https://github.com/axboe/fio/releases for what we're up to)?

-- 
Sitsofe | http://sucs.org/~sits/

RE: Fio verify crc32 fault

[Jeff Furlong]

After you update to the latest fio, also consider adding serialize_overlap=1 as a parameter (HOWTO has background there).  With large random writes, there is some probability (increasing with smaller device size and increasing with higher QD) that two inflight IO's conflict in range/time.  Suppose the first write is acknowledged by the device while the second conflicting write is already sent.  By the time fio reads the first command, the second write will change the data, causing verification errors.

Regards,
Jeff

-----Original Message-----
From: Sitsofe Wheeler <sitsofe@gmail.com> 
Sent: Friday, December 4, 2020 6:42 AM
To: Peter Sabaini <peter@sabaini.at>
Cc: fio@vger.kernel.org
Subject: Re: Fio verify crc32 fault

CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.


Hi,

On Thu, 3 Dec 2020 at 17:27, Peter Sabaini <peter@sabaini.at> wrote:
>
> Hey,
>
> I've got a fio verify run on an NVMe that gives me a crc32 verification fault, however it's not reproducible -- when rerunning on the same NVMe it comes back fine.
>
> Also I'm getting a funny "error=Invalid or incomplete multibyte or wide character" error reported, see below.
>
> Any hints for this?
>
> Many thanks,
> peter.
>
> Test output
>
> Running command: sudo -n fio --randrepeat=1 --ioengine=libaio 
> --direct=1 --gtod_reduce=1 --name=fio_test_verify --bs=4M --iodepth=64 
> --size=95% --verify=crc32c-intel --runtime=7200s 
> --filename=/dev/nvme0n1 --readwrite=randwrite
>
> fio_test_verify: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, 
> ioengine=libaio, iodepth=64
> fio-2.2.10

^^^ Your fio is VERY old... Does the same thing happen with a recent fio (see https://github.com/axboe/fio/releases for what we're up to)?

--
Sitsofe | http://sucs.org/~sits/

Re: Fio verify crc32 fault

[Peter Sabaini]

Thanks both of you. I'll try reproducing this with a newer fio (it wasn't easy to repro to begin with though).

However reading the commit[0] and issue for serialize_overlap it seems not unlikely I'm indeed hitting this. 

[0] https://github.com/axboe/fio/commit/997b5680d139ce82c2034ba3a0d602cfd778b89b


On 04.12.20 18:01, Jeff Furlong wrote:
> After you update to the latest fio, also consider adding serialize_overlap=1 as a parameter (HOWTO has background there).  With large random writes, there is some probability (increasing with smaller device size and increasing with higher QD) that two inflight IO's conflict in range/time.  Suppose the first write is acknowledged by the device while the second conflicting write is already sent.  By the time fio reads the first command, the second write will change the data, causing verification errors.
> 
> Regards,
> Jeff
> 
> -----Original Message-----
> From: Sitsofe Wheeler <sitsofe@gmail.com> 
> Sent: Friday, December 4, 2020 6:42 AM
> To: Peter Sabaini <peter@sabaini.at>
> Cc: fio@vger.kernel.org
> Subject: Re: Fio verify crc32 fault
> 
> CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.
> 
> 
> Hi,
> 
> On Thu, 3 Dec 2020 at 17:27, Peter Sabaini <peter@sabaini.at> wrote:
>>
>> Hey,
>>
>> I've got a fio verify run on an NVMe that gives me a crc32 verification fault, however it's not reproducible -- when rerunning on the same NVMe it comes back fine.
>>
>> Also I'm getting a funny "error=Invalid or incomplete multibyte or wide character" error reported, see below.
>>
>> Any hints for this?
>>
>> Many thanks,
>> peter.
>>
>> Test output
>>
>> Running command: sudo -n fio --randrepeat=1 --ioengine=libaio 
>> --direct=1 --gtod_reduce=1 --name=fio_test_verify --bs=4M --iodepth=64 
>> --size=95% --verify=crc32c-intel --runtime=7200s 
>> --filename=/dev/nvme0n1 --readwrite=randwrite
>>
>> fio_test_verify: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, 
>> ioengine=libaio, iodepth=64
>> fio-2.2.10
> 
> ^^^ Your fio is VERY old... Does the same thing happen with a recent fio (see https://github.com/axboe/fio/releases for what we're up to)?
> 
> --
> Sitsofe | http://sucs.org/~sits/
>