get err 5

get err 5

[آرش خاوری راد]

[-- Attachment #1: Type: text/plain, Size: 3635 bytes --]

Hi i got this error when test FIO sequential write every time, in
sequential read, random read, random write not got this.

my Linux distro is CentOS Linux release 7.5.1804 (Core)
and my kernel version is 3.10.0-327.10.1.el7.x86_64

My output is:

job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=libaio, iodepth=16
...
fio-3.1
Starting 16 processes

job1: (groupid=0, jobs=16): err= 5 (file:io_u.c:1756, func=io_u error,
error=Input/output error): pid=10664: Fri Sep  4 00:03:23 2015
  write: IOPS=319, BW=1279MiB/s (1341MB/s)(2559GiB/2048690msec)
    slat (usec): min=227, max=101577k, avg=16698.60, stdev=395259.01
    clat (msec): min=71, max=102013, avg=302.24, stdev=1524.49
     lat (msec): min=72, max=102014, avg=318.94, stdev=1574.94
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  161], 10.00th=[  180], 20.00th=[  209],
     | 30.00th=[  230], 40.00th=[  247], 50.00th=[  264], 60.00th=[  284],
     | 70.00th=[  305], 80.00th=[  330], 90.00th=[  376], 95.00th=[  430],
     | 99.00th=[  550], 99.50th=[  592], 99.90th=[  726], 99.95th=[  810],
     | 99.99th=[17113]
   bw (  KiB/s): min= 8308, max=403830, per=13.68%, avg=179223.36,
stdev=54500.84, samples=15737
   iops        : min=    2, max=   98, avg=43.32, stdev=13.35, samples=15737
  lat (msec)   : 100=0.01%, 250=27.42%, 500=36.58%, 750=1.31%, 1000=0.03%
  lat (msec)   : 2000=0.01%, >=2000=0.02%
  cpu          : usr=16.84%, sys=4.74%, ctx=199315, majf=0, minf=117471
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=1221.9%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued rwt: total=0,655023,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1279MiB/s (1341MB/s), 1279MiB/s-1279MiB/s (1341MB/s-1341MB/s),
io=2559GiB (2748GB), run=2048690-2048690msec

Disk stats (read/write):
  sdb: ios=87/12938655, merge=0/0, ticks=62/1728507436,
in_queue=1728692357, util=100.00%
  sdc: ios=43/12688086, merge=0/0, ticks=12/1680848678,
in_queue=1680921305, util=100.00%
  sdd: ios=43/12793381, merge=0/0, ticks=9/1726297058, in_queue=1726441852,
util=100.00%
  sde: ios=34/12824943, merge=0/0, ticks=9/1726084912, in_queue=1726194592,
util=100.00%
  sdf: ios=0/1816141, merge=0/0, ticks=0/288898653, in_queue=300747577,
util=100.00%
  sdg: ios=0/10970577, merge=0/0, ticks=0/1759675137, in_queue=1759784380,
util=100.00%

My parameter is :


[global]
readwrite=write
blocksize=4m
direct=1
#numjobs=1
iodepth=16
ioengine=libaio
#filename=/dev/sdx
refill_buffers
time_based
runtime=600
ramp_time=10800
group_reporting

[job1]
numjobs=3
filename=/dev/sdb


[job2]
numjobs=2
filename=/dev/sdc


[job3]
numjobs=3
filename=/dev/sdd


[job4]
numjobs=3
filename=/dev/sde


[job5]
numjobs=2
filename=/dev/sdf


[job6]
numjobs=3
filename=/dev/sdg


.




Why get err 5?
thanks

[-- Attachment #2: Type: text/html, Size: 4293 bytes --]

Re: get err 5

[Sitsofe Wheeler]

On 1 August 2018 at 06:31, آرش خاوری راد <arash.khavary@gmail.com> wrote:
> Hi i got this error when test FIO sequential write every time, in sequential
> read, random read, random write not got this.
>
> my Linux distro is CentOS Linux release 7.5.1804 (Core)
> and my kernel version is 3.10.0-327.10.1.el7.x86_64
>
> My output is:
>
> job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
> 4096KiB-4096KiB, ioengine=libaio, iodepth=16

[snip]

>
> My parameter is :
>
>
> [global]
> readwrite=write
> blocksize=4m
> direct=1

[snip]

>
> [job1]
> numjobs=3
> filename=/dev/sdb
>
>
> [job2]
> numjobs=2
> filename=/dev/sdc
>
>
> [job3]
> numjobs=3
> filename=/dev/sdd
>
>
> [job4]
> numjobs=3
> filename=/dev/sde
>
>
> [job5]
> numjobs=2
> filename=/dev/sdf
>
>
> [job6]
> numjobs=3
> filename=/dev/sdg

[snip]

> Why get err 5?
> thanks

So you have 6 separate  jobs and the jobs specify numjobs. If we add
all the numjobs together we get 16. You specify a giant blocksize (4
megabytes) which is strangely big unless you are doing unless you're
doing extreme NVMe testing. You also specify an iodepth of 16. So 16 *
16 * 4 = 1024 megabytes of I/O buffers - are you sure you're not
causing yourself to hit memory issues?

-- 
Sitsofe | http://sucs.org/~sits/

Re: get err 5

[Alireza Haghdoost]

[-- Attachment #1: Type: text/plain, Size: 4385 bytes --]

Your jobs step on the foot of each other. You can’t write with multiple
process on the same LBA of a LUN on the same time. That is why you get
error 5 because one thread is writting an LBA and another thread modify if
at the same time. In this case either one can get unknown result.

On Wed, Aug 1, 2018 at 12:32 AM آرش خاوری راد <arash.khavary@gmail.com>
wrote:

> Hi i got this error when test FIO sequential write every time, in
> sequential read, random read, random write not got this.
>
> my Linux distro is CentOS Linux release 7.5.1804 (Core)
> and my kernel version is 3.10.0-327.10.1.el7.x86_64
>
> My output is:
>
> job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
> 4096KiB-4096KiB, ioengine=libaio, iodepth=16
> ...
> job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
> 4096KiB-4096KiB, ioengine=libaio, iodepth=16
> ...
> job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
> 4096KiB-4096KiB, ioengine=libaio, iodepth=16
> ...
> job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
> 4096KiB-4096KiB, ioengine=libaio, iodepth=16
> ...
> job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
> 4096KiB-4096KiB, ioengine=libaio, iodepth=16
> ...
> job1: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
> 4096KiB-4096KiB, ioengine=libaio, iodepth=16
> ...
> fio-3.1
> Starting 16 processes
>
> job1: (groupid=0, jobs=16): err= 5 (file:io_u.c:1756, func=io_u error,
> error=Input/output error): pid=10664: Fri Sep  4 00:03:23 2015
>   write: IOPS=319, BW=1279MiB/s (1341MB/s)(2559GiB/2048690msec)
>     slat (usec): min=227, max=101577k, avg=16698.60, stdev=395259.01
>     clat (msec): min=71, max=102013, avg=302.24, stdev=1524.49
>      lat (msec): min=72, max=102014, avg=318.94, stdev=1574.94
>     clat percentiles (msec):
>      |  1.00th=[  136],  5.00th=[  161], 10.00th=[  180], 20.00th=[  209],
>      | 30.00th=[  230], 40.00th=[  247], 50.00th=[  264], 60.00th=[  284],
>      | 70.00th=[  305], 80.00th=[  330], 90.00th=[  376], 95.00th=[  430],
>      | 99.00th=[  550], 99.50th=[  592], 99.90th=[  726], 99.95th=[  810],
>      | 99.99th=[17113]
>    bw (  KiB/s): min= 8308, max=403830, per=13.68%, avg=179223.36,
> stdev=54500.84, samples=15737
>    iops        : min=    2, max=   98, avg=43.32, stdev=13.35,
> samples=15737
>   lat (msec)   : 100=0.01%, 250=27.42%, 500=36.58%, 750=1.31%, 1000=0.03%
>   lat (msec)   : 2000=0.01%, >=2000=0.02%
>   cpu          : usr=16.84%, sys=4.74%, ctx=199315, majf=0, minf=117471
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=1221.9%, 32=0.0%,
> >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
> >=64=0.0%
>      issued rwt: total=0,655023,0, short=0,0,0, dropped=0,0,0
>      latency   : target=0, window=0, percentile=100.00%, depth=16
>
> Run status group 0 (all jobs):
>   WRITE: bw=1279MiB/s (1341MB/s), 1279MiB/s-1279MiB/s (1341MB/s-1341MB/s),
> io=2559GiB (2748GB), run=2048690-2048690msec
>
> Disk stats (read/write):
>   sdb: ios=87/12938655, merge=0/0, ticks=62/1728507436,
> in_queue=1728692357, util=100.00%
>   sdc: ios=43/12688086, merge=0/0, ticks=12/1680848678,
> in_queue=1680921305, util=100.00%
>   sdd: ios=43/12793381, merge=0/0, ticks=9/1726297058,
> in_queue=1726441852, util=100.00%
>   sde: ios=34/12824943, merge=0/0, ticks=9/1726084912,
> in_queue=1726194592, util=100.00%
>   sdf: ios=0/1816141, merge=0/0, ticks=0/288898653, in_queue=300747577,
> util=100.00%
>   sdg: ios=0/10970577, merge=0/0, ticks=0/1759675137, in_queue=1759784380,
> util=100.00%
>
> My parameter is :
>
>
> [global]
> readwrite=write
> blocksize=4m
> direct=1
> #numjobs=1
> iodepth=16
> ioengine=libaio
> #filename=/dev/sdx
> refill_buffers
> time_based
> runtime=600
> ramp_time=10800
> group_reporting
>
> [job1]
> numjobs=3
> filename=/dev/sdb
>
>
> [job2]
> numjobs=2
> filename=/dev/sdc
>
>
> [job3]
> numjobs=3
> filename=/dev/sdd
>
>
> [job4]
> numjobs=3
> filename=/dev/sde
>
>
> [job5]
> numjobs=2
> filename=/dev/sdf
>
>
> [job6]
> numjobs=3
> filename=/dev/sdg
>
>
> .
>
>
>
>
> Why get err 5?
> thanks
>

[-- Attachment #2: Type: text/html, Size: 4965 bytes --]

Re: get err 5

[Sitsofe Wheeler]

On 1 August 2018 at 07:39, Alireza Haghdoost <haghdoost@gmail.com> wrote:
> Your jobs step on the foot of each other. You can’t write with multiple
> process on the same LBA of a LUN on the same time. That is why you get error
> 5 because one thread is writting an LBA and another thread modify if at the
> same time. In this case either one can get unknown result.

I agree with part of this statement (unknown result in the LBA) but
I'm doubtful it would cause an error message of "I/O error". Disks
definitely do accept simultaneous write I/Os against the same LBA but
the problem becomes that in most cases the end result as to which data
the LBA will contain is undefined. It's not illegal (in the sense that
nothing prevents and nothing will error) it's just bad from a data
integrity perspective (and since this wasn't a verify job...).

-- 
Sitsofe | http://sucs.org/~sits/

Re: get err 5

[Alireza Haghdoost]

[-- Attachment #1: Type: text/plain, Size: 1568 bytes --]

On Wed, Aug 1, 2018 at 1:51 AM Sitsofe Wheeler <sitsofe@gmail.com> wrote:

> On 1 August 2018 at 07:39, Alireza Haghdoost <haghdoost@gmail.com> wrote:
> > Your jobs step on the foot of each other. You can’t write with multiple
> > process on the same LBA of a LUN on the same time. That is why you get
> error
> > 5 because one thread is writting an LBA and another thread modify if at
> the
> > same time. In this case either one can get unknown result.
>
> I agree with part of this statement (unknown result in the LBA) but
> I'm doubtful it would cause an error message of "I/O error". Disks
> definitely do accept simultaneous write I/Os against the same LBA but
> the problem becomes that in most cases the end result as to which data
> the LBA will contain is undefined. It's not illegal (in the sense that
> nothing prevents and nothing will error) it's just bad from a data
> integrity perspective (and since this wasn't a verify job...).



> I agree that it is not is not a felony two write on the same lba with
multiple threads.

Not sure what kind of storage device is used in this experience but some
storage devices *do* Lock a sector/chunk while it is being written. It
means the second in-inflight write hitting the same LBA/chunk will wait for
the first one.
Do the math to find out in this job file how many of these big in-flight
write the system can handle before it times-out the last one in the queue.

Now, does it cause err 5? I don’t know for sure. However. This is yet
another reason that might cause it.

[-- Attachment #2: Type: text/html, Size: 2243 bytes --]

Re: get err 5

[Sitsofe Wheeler]

On 7 August 2018 at 04:57, آرش خاوری راد <arash.khavary@gmail.com> wrote:
>
> Unfortunately my problem not solved, I change Blocksize but got arror again
>
> fio: io_u error on file /dev/sdc: Input/output error: write
> offset=1677721600, buflen=4194304

Is there anything suspicious in dmesg? If you cut the number of
sections down to one does it still happen? Is it alway against the
same disk?

-- 
Sitsofe | http://sucs.org/~sits/

Re: get err 5

[آرش خاوری راد]

No every time error on 2 or 3 different disk, and its not follow the
pattern or same disk maybe error on 2 disk maybe 3.


in dmesg just log:
[94930.551379] qla2xxx [0000:02:00.0]-801c:13: Abort command issued
nexus=13:0:1 --  1 2002.
ج91373.328470] ICMPv6 checksum failed [fe80::2046:fd22:d246:7474 > ff02::1]


to answer you question about If you cut the number of sections down to
one does it still happen? i not understand whats your meaning please
told whats i changed ?


2018-08-07 5:28 GMT+00:00 Sitsofe Wheeler <sitsofe@gmail.com>:
>
> On 7 August 2018 at 04:57, آرش خاوری راد <arash.khavary@gmail.com> wrote:
> >
> > Unfortunately my problem not solved, I change Blocksize but got arror again
> >
> > fio: io_u error on file /dev/sdc: Input/output error: write
> > offset=1677721600, buflen=4194304
>
> Is there anything suspicious in dmesg? If you cut the number of
> sections down to one does it still happen? Is it alway against the
> same disk?
>
> --
> Sitsofe | http://sucs.org/~sits/




-- 
با عرض احترام خاوری راد - آرش

Re: get err 5

[آرش خاوری راد]

No every time error on 2 or 3 different disk, and its not follow the
pattern or same disk maybe error on 2 disk maybe 3.


in dmesg just log:
[94930.551379] qla2xxx [0000:02:00.0]-801c:13: Abort command issued
nexus=13:0:1 --  1 2002.
ج91373.328470] ICMPv6 checksum failed [fe80::2046:fd22:d246:7474 > ff02::1]


to answer you question about If you cut the number of sections down to
one does it still happen? i not understand whats your meaning please
told whats i changed ?

2018-08-07 5:28 GMT+00:00 Sitsofe Wheeler <sitsofe@gmail.com>:
> On 7 August 2018 at 04:57, آرش خاوری راد <arash.khavary@gmail.com> wrote:
>>
>> Unfortunately my problem not solved, I change Blocksize but got arror again
>>
>> fio: io_u error on file /dev/sdc: Input/output error: write
>> offset=1677721600, buflen=4194304
>
> Is there anything suspicious in dmesg? If you cut the number of
> sections down to one does it still happen? Is it alway against the
> same disk?
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: get err 5

[Michael Green]

The abort command below can only come come from the SCSI layer and therefore is unlikely to be a FIO issue.
Are you able to perform simple dd read/write commands to the device? Are you able to partition or format it?

Michael

> On Aug 7, 2018, at 2:26 AM, ⁨آرش خاوری راد⁩ <⁨arash.khavary@gmail.com⁩> wrote:
> 
> No every time error on 2 or 3 different disk, and its not follow the
> pattern or same disk maybe error on 2 disk maybe 3.
> 
> 
> in dmesg just log:
> [94930.551379] qla2xxx [0000:02:00.0]-801c:13: Abort command issued
> nexus=13:0:1 --  1 2002.
> ج91373.328470] ICMPv6 checksum failed [fe80::2046:fd22:d246:7474 > ff02::1]
> 

Re: get err 5

[آرش خاوری راد]

all be clear just sequential write got an error

2018-08-07 19:12 GMT+04:30 Michael Green <mishagreen@gmail.com>:
> The abort command below can only come come from the SCSI layer and therefore is unlikely to be a FIO issue.
> Are you able to perform simple dd read/write commands to the device? Are you able to partition or format it?
>
> Michael
>
>> On Aug 7, 2018, at 2:26 AM, ⁨آرش خاوری راد⁩ <⁨arash.khavary@gmail.com⁩> wrote:
>>
>> No every time error on 2 or 3 different disk, and its not follow the
>> pattern or same disk maybe error on 2 disk maybe 3.
>>
>>
>> in dmesg just log:
>> [94930.551379] qla2xxx [0000:02:00.0]-801c:13: Abort command issued
>> nexus=13:0:1 --  1 2002.
>> ج91373.328470] ICMPv6 checksum failed [fe80::2046:fd22:d246:7474 > ff02::1]
>>
>



-- 
با عرض احترام خاوری راد - آرش

Re: get err 5

[Sitsofe Wheeler]

(I've just replied to your first message - I think you sent this
multiple times...)

On 7 August 2018 at 06:36, آرش خاوری راد <arash.khavary@gmail.com> wrote:
> No, every time error on 2 or 3 different disk, and its not follow the
> pattern or same disk maybe error on 2 disk maybe 3.
>
>
> in dmesg just log:
> `[94930.551379] qla2xxx [0000:02:00.0]-801c:13: Abort command issued
> nexus=13:0:1 --  1 2002.`

If there's always the above it looks like your disk controller is
aborting commands. If this turns out to be the cause then you'll need
to investigate why your controller is getting unhappy (which is beyond
what I can help you with) - maybe something can't handle the load?

> to answer you question about " If you cut the number of sections down to one
> does it still happen?" i not understand whats your meaning please told whats
> i changed ?

Make your job file as small as possible (e.g. remove job6 and see if
the problem still happens, then remove job5 and see if it still
happens etc).

-- 
Sitsofe | http://sucs.org/~sits/

Re: get err 5

[آرش خاوری راد]

I check in deep in terminal when FIO is running i got this error:

fio: io_u error on file /dev/sdg: Input/output error: write
offset=241038262272, buflen=4194304
fio: io_u error on file /dev/sdg: Input/output error: write
offset=241042456576, buflen=4194304
fio: io_u error on file /dev/sdg: Input/output error: write
offset=246167896064, buflen=4194304
fio: io_u error on file /dev/sdg: Input/output error: write
offset=246172090368, buflen=4194304



2018-08-07 5:28 GMT+00:00 Sitsofe Wheeler <sitsofe@gmail.com>:
> On 7 August 2018 at 04:57, آرش خاوری راد <arash.khavary@gmail.com> wrote:
>>
>> Unfortunately my problem not solved, I change Blocksize but got arror again
>>
>> fio: io_u error on file /dev/sdc: Input/output error: write
>> offset=1677721600, buflen=4194304
>
> Is there anything suspicious in dmesg? If you cut the number of
> sections down to one does it still happen? Is it alway against the
> same disk?
>
> --
> Sitsofe | http://sucs.org/~sits/



-- 
با عرض احترام خاوری راد - آرش

Re: get err 5

[Sitsofe Wheeler]

You mentioned that you're getting messages like this:

> [94930.551379] qla2xxx [0000:02:00.0]-801c:13: Abort command issued nexus=13:0:1 --  1 2002.

As previously mentioned if your disk controller is aborting commands
you need to find out why
- it's not really a userspace generated issue. Until you know why
that's happening all that will happen is that the abort will bubble
all the
way back up to whatever sent the I/O as an error.

‪On Tue, 14 Aug 2018 at 05:23, ‫آرش خاوری راد‬‎
<arash.khavary@gmail.com> wrote:‬
>
> I check in deep in terminal when FIO is running i got this error:
>
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=241038262272, buflen=4194304
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=241042456576, buflen=4194304
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=246167896064, buflen=4194304
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=246172090368, buflen=4194304
>
>
>
> 2018-08-07 5:28 GMT+00:00 Sitsofe Wheeler <sitsofe@gmail.com>:
> > On 7 August 2018 at 04:57, آرش خاوری راد <arash.khavary@gmail.com> wrote:
> >>
> >> Unfortunately my problem not solved, I change Blocksize but got arror again
> >>
> >> fio: io_u error on file /dev/sdc: Input/output error: write
> >> offset=1677721600, buflen=4194304
> >
> > Is there anything suspicious in dmesg? If you cut the number of
> > sections down to one does it still happen? Is it alway against the
> > same disk?
> >
> > --
> > Sitsofe | http://sucs.org/~sits/
>
>
>
> --
> با عرض احترام خاوری راد - آرش



--
Sitsofe | http://sucs.org/~sits/

Re: get err 5

[Alireza Haghdoost]

‪On Mon, Aug 13, 2018 at 11:23 PM ‫آرش خاوری راد‬‎
<arash.khavary@gmail.com> wrote:‬
>
> I check in deep in terminal when FIO is running i got this error:
>
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=241038262272, buflen=4194304
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=241042456576, buflen=4194304
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=246167896064, buflen=4194304
> fio: io_u error on file /dev/sdg: Input/output error: write
> offset=246172090368, buflen=4194304
>

Remove all numjobs=* from your job file and see what happen.