Fwd: Failures noticed in FIO run

Fwd: Failures noticed in FIO run

[Gnana Sekhar]

Hi,

Occasionally stepping into couple of failures with FIO with arguments
captured below
So wanted to get opinion on if I am missing something in parameters

Failure 1:
The arguments passed are: sudo fio --thread --direct=1 --minimal
--ioengine=libaio --numjobs=1 --filename=/dev/nvme0n1 -o
/tmp/nvme0n1_temp.log --name=bs16384_rwwrite_qd256 --buffer_pattern=0
--iodepth=256 --size=409600 --percentage_random=0 --bs=256k
--offset=625181037 --rw=write

Error in FIO run: fio: io_u error on file /dev/nvme0n1: Invalid
argument: write offset=625181037, buflen=262144

Error in FIO run: fio: io_u error on file /dev/nvme0n1: Invalid
argument: write offset=625443181, buflen=262144



Failure 2:

Write followed by verify:
sudo fio --thread --direct=1 --minimal --ioengine=libaio --numjobs=1
--filename=/dev/nvme0n1 -o /tmp/nvme0n1_temp.log
--name=bs16384_rwwrite_qd256 --buffer_pattern=1 --iodepth=256
--size=40960 --percentage_random=0 --bs=16384 --rw=write


sudo fio --thread --direct=1 --minimal --ioengine=libaio --numjobs=1
--filename=/dev/nvme0n1 -o /tmp/nvme0n1_temp.log
--name=bs16384_rwverify_qd256 --buffer_pattern=1 --iodepth=256
--size=40960 --percentage_random=0 --bs=16384 --rw=read
--verify=pattern --verify_pattern=1

Error in starting FIO: fio: got pattern '00', wanted '01'. Bad bits 1
fio: bad pattern block offset 2048

pattern: verify failed at file /dev/nvme0n1 offset 0, length 16843009
fio: verify type mismatch (257 media, 18 given)
fio: got pattern '00', wanted '01'. Bad bits 1
fio: bad pattern block offset 2048
pattern: verify failed at file /dev/nvme0n1 offset 16384, length 16843009
fio: verify type mismatch (257 media, 18 given)
fio: got pattern '00', wanted '01'. Bad bits 1
fio: bad pattern block offset 2048
pattern: verify failed at file /dev/nvme0n1 offset 32768, length 16843009
fio: verify type mismatch (257 media, 18 given)

Regards,
Gnana

Re: Failures noticed in FIO run

[Gnana Sekhar]

Thanks for your input sitsofe. Appreciate it

Experimented few loops and ran into the below failures even when I
didn't pass in the write offset to FIO. The block size is 4k on the
drive

2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - The arguments
passed are: sudo fio --thread --direct=1 --minimal --ioengine=libaio
--numjobs=1 --filename=/dev/nvme0n1 -o /tmp/nvme0n1_temp.log
--name=bs16384_rwwrite_qd256 --buffer_pattern=1193046 --iodepth=256
--size=100% --percentage_random=0 --bs=256k --ba=4k --rw=write
2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - Error in FIO
run: fio: io_u error on file /dev/nvme0n1: Input/output error: write
offset=27160215552, buflen=262144

2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - Error in FIO
run: fio: io_u error on file /dev/nvme0n1: Input/output error: write
offset=27205828608,


Similarly if I pass offsets that aren't aligned with file system
underneath, is there a way to inform fio to round it up

Regards,
Gnana

On Tue, Feb 6, 2018 at 6:31 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 6 February 2018 at 05:15, Gnana Sekhar <kgsgnana2020@gmail.com> wrote:
>>
>> Occasionally stepping into couple of failures with FIO with arguments
>> captured below
>> So wanted to get opinion on if I am missing something in parameters
>>
>> Failure 1:
>> The arguments passed are: sudo fio --thread --direct=1 --minimal
>> --ioengine=libaio --numjobs=1 --filename=/dev/nvme0n1 -o
>> /tmp/nvme0n1_temp.log --name=bs16384_rwwrite_qd256 --buffer_pattern=0
>> --iodepth=256 --size=409600 --percentage_random=0 --bs=256k
>> --offset=625181037 --rw=write
>>
>> Error in FIO run: fio: io_u error on file /dev/nvme0n1: Invalid
>> argument: write offset=625181037, buflen=262144
>>
>> Error in FIO run: fio: io_u error on file /dev/nvme0n1: Invalid
>> argument: write offset=625443181, buflen=262144
>
> Requesting direct/unbuffered access comes with conditions on some
> platforms. In the case of Linux direct=1 is mapped to O_DIRECT and on
> Linux to use O_DIRECT you often have to send I/O whose size and offset
> are aligned to the underlying filesystem/device
> (logical) block size (this is a simplified generalisation but is kind of all you
> need to know for fio, see the NOTES section for O_DIRECT in the
> open(2) man page for gory details - https://linux.die.net/man/2/open
> ). I don't know what block size your NVMe device
> exposes but I'm going to guess it's at least 4k. So let's see if your
> job might violate the alignment rule assuming this:
>
> --bs=256k - this is ok because 262144 is a multiple of 4096.
> --offset=625181037 - we've got a problem. 625181037 is NOT a multiple
> of 4096 and yet you are asking for I/O to start at that byte offset.
> This violates the O_DIRECT constraint (you will only send down well
> aligned I/O) so it's likely every I/O of this job will fail for being
> badly aligned. Thus the problem occurred because you requested I/O to
> start at an unaligned offset causing all I/Os to be unaligned.
>
>> Failure 2:
>>
>> Write followed by verify:
>> sudo fio --thread --direct=1 --minimal --ioengine=libaio --numjobs=1
>> --filename=/dev/nvme0n1 -o /tmp/nvme0n1_temp.log
>> --name=bs16384_rwwrite_qd256 --buffer_pattern=1 --iodepth=256
>> --size=40960 --percentage_random=0 --bs=16384 --rw=write
>>
>>
>> sudo fio --thread --direct=1 --minimal --ioengine=libaio --numjobs=1
>> --filename=/dev/nvme0n1 -o /tmp/nvme0n1_temp.log
>> --name=bs16384_rwverify_qd256 --buffer_pattern=1 --iodepth=256
>> --size=40960 --percentage_random=0 --bs=16384 --rw=read
>> --verify=pattern --verify_pattern=1
>>
>> Error in starting FIO: fio: got pattern '00', wanted '01'. Bad bits 1
>> fio: bad pattern block offset 2048
>>
>> pattern: verify failed at file /dev/nvme0n1 offset 0, length 16843009
>> fio: verify type mismatch (257 media, 18 given)
>> fio: got pattern '00', wanted '01'. Bad bits 1
>> fio: bad pattern block offset 2048
>> pattern: verify failed at file /dev/nvme0n1 offset 16384, length 16843009
>> fio: verify type mismatch (257 media, 18 given)
>> fio: got pattern '00', wanted '01'. Bad bits 1
>> fio: bad pattern block offset 2048
>> pattern: verify failed at file /dev/nvme0n1 offset 32768, length 16843009
>> fio: verify type mismatch (257 media, 18 given)
>
> In the above ignore "length 16843009" - that's just your pattern
> (0x01010101) being interpreted as a 32 bit unsigned int. Also ignore
> "verify type mismatch (257 media, 18 given)" that's your pattern
> (0x01010) being interpreted as a 16 bit unsigned int. Perhaps when
> doing a headerless verify the length should be fudged to be the block
> size and the media type check skipped? However it's a bit strange as
> to why it would fail in the first place.
>
> I just ran
> ./fio --thread --direct=1 --ioengine=libaio --numjobs=1
> --filename=/tmp/fio.tmp --name=bs16384_rwwrite_qd256
> --buffer_pattern=1 --iodepth=256 --size=40960 --percentage_random=0
> --bs=16384 --rw=write
> ./fio --thread --direct=1 --ioengine=libaio --numjobs=1
> --filename=/tmp/fio.tmp --name=bs16384_rwverify_qd256
> --buffer_pattern=1 --iodepth=256 --size=40960 --percentage_random=0
> --bs=16384 --rw=read --verify=pattern --verify_pattern=1
>
> (where /tmp was part of an XFS filesystem) and don't get an error. Are
> you sure the fio job you ran before bs16384_rwverify_qd256 had
> buffer_pattern=1 - I notice in Failure 1 you were using
> buffer_pattern=0...
>
> You might find it easier to construct your jobs like this:
> fio --direct=1 --ioengine=libaio --name=write_pattern
> --filename=/dev/nvme0n1 --size=32k --bs=16k --rw=write
> --verify=pattern --verify_pattern=1 --do_verify=0
> fio --direct=1 --ioengine=libaio --name=verify_pattern
> --filename=/dev/nvme0n1 --size=32k --bs=16k --rw=read --verify=pattern
> --verify_pattern=1
>
> --
> Sitsofe | http://sucs.org/~sits/

RE: Failures noticed in FIO run

[Jeff Furlong]

> Similarly if I pass offsets that aren't aligned with file system underneath, is there a way to inform fio to round it up
You may want to look at the offset_align parameter.

Regards,
Jeff

Re: Failures noticed in FIO run

[Sitsofe Wheeler]

On 7 February 2018 at 01:39, Jeff Furlong <jeff.furlong@wdc.com> wrote:

>> Similarly if I pass offsets that aren't aligned with file system underneath, is there a way to inform fio to round it up
> You may want to look at the offset_align parameter.

If you are specifying an absolute number (i.e. it's not a percentage)
as the offset then I don't think offset_align will fix it up for you -
you asked explicitly for that offset:

"offset_align=int If set to non-zero value, the byte offset generated
by a percentage offset is aligned upwards to this value."

-- 
Sitsofe | http://sucs.org/~sits/

RE: Failures noticed in FIO run

[Jeff Furlong]

> If you are specifying an absolute number (i.e. it's not a percentage) as the offset then I 
>don't think offset_align will fix it up for you - you asked explicitly for that offset:
Right.  If you specify the exact byte in offset, you get it.  If you specify 20% and offset_align, then the 20% computes some potentially misaligned byte that then gets aligned to whatever offset you specify.

Regards,
Jeff

Re: Failures noticed in FIO run

[Sitsofe Wheeler]

On 7 February 2018 at 00:19, Gnana Sekhar <kgsgnana2020@gmail.com> wrote:
>
> Experimented few loops and ran into the below failures even when I
> didn't pass in the write offset to FIO. The block size is 4k on the
> drive
>
> 2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - The arguments
> passed are: sudo fio --thread --direct=1 --minimal --ioengine=libaio
> --numjobs=1 --filename=/dev/nvme0n1 -o /tmp/nvme0n1_temp.log
> --name=bs16384_rwwrite_qd256 --buffer_pattern=1193046 --iodepth=256
> --size=100% --percentage_random=0 --bs=256k --ba=4k --rw=write
> 2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - Error in FIO
> run: fio: io_u error on file /dev/nvme0n1: Input/output error: write
> offset=27160215552, buflen=262144
>
> 2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - Error in FIO
> run: fio: io_u error on file /dev/nvme0n1: Input/output error: write
> offset=27205828608,

(note that blockalign/ba is only really for random I/O)

This looks strange. What does
blockdev --getsize64  /dev/nvme0n1
report and is there anything interesting in the tail end of dmesg just
after you trigger this issue?

> Similarly if I pass offsets that aren't aligned with file system
> underneath, is there a way to inform fio to round it up

Jeff answered this one in another reply.

-- 
Sitsofe | http://sucs.org/~sits/

Re: Failures noticed in FIO run

[Gnana Sekhar]

Thanks Sitsofe and Jeff

As suggested removed the --ba parameter for sequential writes


Below is the drive information
blockdev --getsize64  /dev/nvme0n1

3840755982336

fdisk -l

Disk /dev/nvme0n1: 3840.8 GB, 3840755982336 bytes, 7501476528 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I had to change the sector size from 4K to 512 and now I didn't see
the failure mentioned below. Will loop through and update


I was using offset and sizeToWrite parameters when I wanted to fill
certain number of bytes from certain start location.
Currently I am using "sizeToWrite" to specify the bytes to write and
"offset" from the byte to start
But from the suggestions it seems I have to chose the offset such that
its divisible by blocksize of filesystem. I am using blockdev --getbsz
/dev/nvme0n1 to determine the blocksize
Let me know if you guys think that should solve the problem or you
come across any other optimal way to do the same without using offset
and sizeToWrite

Also which of the below do you think I should use to get the LBA size on device
blockdev --getbsz /dev/nvme0n1 or blockdev --getss /dev/nvme0n1


Regards,
Gnana

On Wed, Feb 7, 2018 at 9:12 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 7 February 2018 at 00:19, Gnana Sekhar <kgsgnana2020@gmail.com> wrote:
>>
>> Experimented few loops and ran into the below failures even when I
>> didn't pass in the write offset to FIO. The block size is 4k on the
>> drive
>>
>> 2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - The arguments
>> passed are: sudo fio --thread --direct=1 --minimal --ioengine=libaio
>> --numjobs=1 --filename=/dev/nvme0n1 -o /tmp/nvme0n1_temp.log
>> --name=bs16384_rwwrite_qd256 --buffer_pattern=1193046 --iodepth=256
>> --size=100% --percentage_random=0 --bs=256k --ba=4k --rw=write
>> 2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - Error in FIO
>> run: fio: io_u error on file /dev/nvme0n1: Input/output error: write
>> offset=27160215552, buflen=262144
>>
>> 2018-02-05 07:20:53,033-07 ERROR    l_IoActionsLinux - Error in FIO
>> run: fio: io_u error on file /dev/nvme0n1: Input/output error: write
>> offset=27205828608,
>
> (note that blockalign/ba is only really for random I/O)
>
> This looks strange. What does
> blockdev --getsize64  /dev/nvme0n1
> report and is there anything interesting in the tail end of dmesg just
> after you trigger this issue?
>
>> Similarly if I pass offsets that aren't aligned with file system
>> underneath, is there a way to inform fio to round it up
>
> Jeff answered this one in another reply.
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: Failures noticed in FIO run

[Sitsofe Wheeler]

On 9 February 2018 at 04:21, Gnana Sekhar <kgsgnana2020@gmail.com> wrote:
>
> Below is the drive information
> blockdev --getsize64  /dev/nvme0n1
>
> 3840755982336
>
> fdisk -l
>
> Disk /dev/nvme0n1: 3840.8 GB, 3840755982336 bytes, 7501476528 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
>
> I had to change the sector size from 4K to 512 and now I didn't see
> the failure mentioned below. Will loop through and update

Seems strange that the problem would go away just due to shrinking
fio's blocksize given that 4k is a multiple of 512. You never said if
dmesg showed any clues though...

> I was using offset and sizeToWrite parameters when I wanted to fill
> certain number of bytes from certain start location.
> Currently I am using "sizeToWrite" to specify the bytes to write and
> "offset" from the byte to start
> But from the suggestions it seems I have to chose the offset such that
> its divisible by blocksize of filesystem. I am using blockdev --getbsz

Note in the examples you gave you were not doing I/O to a file in a
filesystem - you were just writing directly to a disk so I'm going to
assume you just mean disk where you have written filesystem :-) But
yes, you will have to ensure an absolute offset is a multiple of the
disk's block size when doing direct IO to a disk on Linux.

> Let me know if you guys think that should solve the problem or you
> come across any other optimal way to do the same without using offset
> and sizeToWrite

Only passing in an aligned offset should solve the problem and I'm not
aware of anther way.

> Also which of the below do you think I should use to get the LBA size on device
> blockdev --getbsz /dev/nvme0n1 or blockdev --getss /dev/nvme0n1

Either will do but if I were choosing I'd prefer the number from
getbsz. From what I can see getbsz corresponds to the physical block
size (which is closer to the minimum block size the "disk" actually
has to use behind the scenes) and getss corresponds to the logical
block size (the minimum block size your "disk" can accept even if it
means it means faking it and doing extra work).

-- 
Sitsofe | http://sucs.org/~sits/

RE: Failures noticed in FIO run

[Jeff Furlong]

> Also which of the below do you think I should use to get the LBA size on device blockdev --getbsz /dev/nvme0n1 or blockdev --getss /dev/nvme0n1

I believe getbsz will always return the physical block size, while getss will return the logical block size.  Most SSDs have a 4096B physical block size, and some support emulating a 512B block size to the host, while still internally maintaining a 4096B physical block size.  So if you formatted your namespace 1 for 4096B or 512B, in both cases, getbsz returns 4096B.  Only if you formatted your namespace 1 to 512B will getss return 512B.  And if you formatted for metadata or protection information, it will be more difficult (you might want to look at a nvme command with nvme-cli instead of using these legacy ioctls).

Regards,
Jeff