4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[smitha sunder]

Hello,

If I format the drive with 512+8 LBAF run fio, I don’t see any issue :

# nvme list
Node             SN                   Model
        Namespace Usage                      Format           FW Rev
---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1     160401RICHC01        PM1733V4TLC
        1           1.92  TB /   1.92  TB    512   B +  8 B   TEMC8881
/dev/nvme1n1     160401RICHC01        PM1733V4TLC
        1           1.92  TB /   1.92  TB    512   B +  8 B   TEMC8881
[root@smitha-Z390-hawk tmp]# fio --filename=/dev/nvme0n1 --size=1M
--rw=write --ioengine=libaio --direct=1 --verify=crc32 --verify_dump=1
--name=test --bs=512
test: (g=0): rw=write, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B,
ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=32631: Wed Jan  8 09:40:32 2020
   read: IOPS=18.0k, BW=9481KiB/s (9709kB/s)(1024KiB/108msec)
 <snip>…<snip>
  write: IOPS=27.3k, BW=13.3MiB/s (13.0MB/s)(1024KiB/75msec)

Run status group 0 (all jobs):
   READ: bw=9481KiB/s (9709kB/s), 9481KiB/s-9481KiB/s
(9709kB/s-9709kB/s), io=1024KiB (1049kB), run=108-108msec
  WRITE: bw=13.3MiB/s (13.0MB/s), 13.3MiB/s-13.3MiB/s
(13.0MB/s-13.0MB/s), io=1024KiB (1049kB), run=75-75msec

Disk stats (read/write):
  nvme0n1: ios=1288/2048, merge=0/0, ticks=67/66, in_queue=0, util=58.63%

How ever if I format the drive with LBAF type 4 that is supported by
the drive, then I get the following error :

# nvme format /dev/nvme1 --namespace-id=1 --lbaf=4   --pi=1 --ms=0
Success formatting namespace:1
#  nvme list
Node             SN                   Model
        Namespace Usage                      Format           FW Rev
---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1     160401RICHC01        PM1733V4TLC
        1           1.92  TB /   1.92  TB      4 KiB + 64 B   TEMC8881
/dev/nvme1n1     160401RICHC01        PM1733V4TLC
        1           1.92  TB /   1.92  TB      4 KiB + 64 B   TEMC8881


# fio --filename=/dev/nvme0n1 --size=1M --rw=write --ioengine=libaio
--direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte
or wide character: write offset=4096, buflen=4096
fio: pid=32709, err=84/file:io_u.c:1747, func=io_u error,
error=Invalid or incomplete multibyte or wide character

test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error,
error=Invalid or incomplete multibyte or wide character): pid=32709:
Wed Jan  8 09:42:12 2020
  write: IOPS=2000, BW=4000KiB/s (4096kB/s)(4096B/1msec)
    slat (nsec): min=12142, max=27097, avg=19619.50, stdev=10574.78
    clat (nsec): min=24389, max=24389, avg=24389.00, stdev= 0.00
     lat (nsec): min=52811, max=52811, avg=52811.00, stdev= 0.00
  lat (usec)   : 50=50.00%
  cpu          : usr=0.00%, sys=0.00%, ctx=2, majf=0, minf=22

     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4000KiB/s (4096kB/s), 4000KiB/s-4000KiB/s
(4096kB/s-4096kB/s), io=4096B (4096B), run=1-1msec

Disk stats (read/write):
  nvme0n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Am I missing something?
Thanks for the help !

Smitha

Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[Sitsofe Wheeler]

On Wed, 8 Jan 2020 at 19:19, smitha sunder <sundersmitha@gmail.com> wrote:
>
> How ever if I format the drive with LBAF type 4 that is supported by
> the drive, then I get the following error :
>
> # nvme format /dev/nvme1 --namespace-id=1 --lbaf=4   --pi=1 --ms=0
> Success formatting namespace:1
> #  nvme list
> Node             SN                   Model
>         Namespace Usage                      Format           FW Rev
> ---------------- --------------------
> ---------------------------------------- ---------
> -------------------------- ---------------- --------
> /dev/nvme0n1     160401RICHC01        PM1733V4TLC
>         1           1.92  TB /   1.92  TB      4 KiB + 64 B   TEMC8881
> /dev/nvme1n1     160401RICHC01        PM1733V4TLC
>         1           1.92  TB /   1.92  TB      4 KiB + 64 B   TEMC8881
>
>
> # fio --filename=/dev/nvme0n1 --size=1M --rw=write --ioengine=libaio
> --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096
> test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.7
> Starting 1 process
> fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte
> or wide character: write offset=4096, buflen=4096
> fio: pid=32709, err=84/file:io_u.c:1747, func=io_u error,
> error=Invalid or incomplete multibyte or wide character

[...]

> Am I missing something?
> Thanks for the help !

All fio is saying is that it didn't get back the data that it wrote
down on the second block but I don't know what using "LBAF type 4"
might do to your disk (you are saying it changes the logical block
size but does it change anything else?).  Do you don't have anything
else that might be writing to that the device too and might have
interfered with the data? What happens if you add --offset=1M?

-- 
Sitsofe | http://sucs.org/~sits/

Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[smitha sunder]

On Wed, Jan 8, 2020 at 11:41 PM Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>
> On Wed, 8 Jan 2020 at 19:19, smitha sunder <sundersmitha@gmail.com> wrote:
> >
> > How ever if I format the drive with LBAF type 4 that is supported by
> > the drive, then I get the following error :
> >
> > # nvme format /dev/nvme1 --namespace-id=1 --lbaf=4   --pi=1 --ms=0
> > Success formatting namespace:1
> > #  nvme list
> > Node             SN                   Model
> >         Namespace Usage                      Format           FW Rev
> > ---------------- --------------------
> > ---------------------------------------- ---------
> > -------------------------- ---------------- --------
> > /dev/nvme0n1     160401RICHC01        PM1733V4TLC
> >         1           1.92  TB /   1.92  TB      4 KiB + 64 B   TEMC8881
> > /dev/nvme1n1     160401RICHC01        PM1733V4TLC
> >         1           1.92  TB /   1.92  TB      4 KiB + 64 B   TEMC8881
> >
> >
> > # fio --filename=/dev/nvme0n1 --size=1M --rw=write --ioengine=libaio
> > --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096
> > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > 4096B-4096B, ioengine=libaio, iodepth=1
> > fio-3.7
> > Starting 1 process
> > fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte
> > or wide character: write offset=4096, buflen=4096
> > fio: pid=32709, err=84/file:io_u.c:1747, func=io_u error,
> > error=Invalid or incomplete multibyte or wide character
>
> [...]
>
> > Am I missing something?
> > Thanks for the help !
>
> All fio is saying is that it didn't get back the data that it wrote
> down on the second block but I don't know what using "LBAF type 4"
> might do to your disk (you are saying it changes the logical block
> size but does it change anything else?).  Do you don't have anything
> else that might be writing to that the device too and might have
> interfered with the data? What happens if you add --offset=1M?
>
> --
> Sitsofe | http://sucs.org/~sits/

Thanks Sitsofe for your reply and your time.

The drive supports various LBA formats :

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes -
Relative Performance: 0x1 Better
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes -
Relative Performance: 0x2 Good
LBA Format  2 : Metadata Size: 0   bytes - Data Size: 4096 bytes -
Relative Performance: 0 Best
LBA Format  3 : Metadata Size: 8   bytes - Data Size: 4096 bytes -
Relative Performance: 0x2 Good (in use)
LBA Format  4 : Metadata Size: 64  bytes - Data Size: 4096 bytes -
Relative Performance: 0x2 Good

So LBAF 4 is 4K formmated with 64B metadata

Nothing else is running on the device.
The error is seen even if I issue offset=1M or 50% etc.

# fio --filename=/dev/nvme0n1  --rw=write --ioengine=libaio --direct=1
--verify=crc32 --verify_dump=1 --name=test --bs=4096 --offset=1M
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte
or wide character: write offset=1048576, buflen=4096
test: No I/O performed by libaio, perhaps try --debug=io option for details?
fio: pid=3392, err=84/file:io_u.c:1747, func=io_u error, error=Invalid
or incomplete multibyte or wide character

test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error,
error=Invalid or incomplete multibyte or wide character): pid=3392:
Thu Jan  9 09:28:24 2020
  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=21
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

Disk stats (read/write):
  nvme0n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

If I format the drive to 1KiB+8B metadata then fio runs fine :
# nvme list
Node             SN                   Model
        Namespace Usage                      Format           FW Rev
---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1     160401RICHC01        PM1733V4TLC
        1           1.92  TB /   1.92  TB      4 KiB +  8 B   TEMC8881
/dev/nvme1n1     160401RICHC01        PM1733V4TLC
        1           1.92  TB /   1.92  TB      4 KiB +  8 B   TEMC8881
# fio --filename=/dev/nvme0n1  --rw=write --ioengine=libaio --direct=1
--verify=crc32 --verify_dump=1 --name=test --bs=4096 --offset=1M
--size=1G
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [V(1)][100.0%][r=43.5MiB/s,w=0KiB/s][r=11.1k,w=0
IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3486: Thu Jan  9 09:30:19 2020
   read: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(1024MiB/23485msec)
    slat (nsec): min=1496, max=5603.7k, avg=1745.06, stdev=11027.69
    clat (usec): min=35, max=517, avg=78.56, stdev=13.52
     lat (usec): min=37, max=5672, avg=80.33, stdev=17.43
    clat percentiles (usec):
     |  1.00th=[   63],  5.00th=[   73], 10.00th=[   73], 20.00th=[   73],
     | 30.00th=[   74], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   89], 80.00th=[   89], 90.00th=[   90], 95.00th=[   90],
     | 99.00th=[   90], 99.50th=[   91], 99.90th=[  347], 99.95th=[  375],
     | 99.99th=[  400]
  write: IOPS=27.2k, BW=106MiB/s (112MB/s)(1024MiB/9625msec)
    slat (nsec): min=11232, max=68549, avg=12131.36, stdev=469.35
    clat (usec): min=18, max=372, avg=24.25, stdev= 4.91
     lat (usec): min=29, max=384, avg=36.41, stdev= 4.95
    clat percentiles (nsec):
     |  1.00th=[20352],  5.00th=[20352], 10.00th=[20608], 20.00th=[21120],
     | 30.00th=[22144], 40.00th=[22144], 50.00th=[22400], 60.00th=[22656],
     | 70.00th=[23168], 80.00th=[24448], 90.00th=[33536], 95.00th=[34048],
     | 99.00th=[36096], 99.50th=[38656], 99.90th=[48384], 99.95th=[54016],
     | 99.99th=[74240]
   bw (  KiB/s): min=28447, max=114880, per=96.25%, avg=104854.35,
stdev=18105.13, samples=20
   iops        : min= 7111, max=28720, avg=26213.65, stdev=4526.47, samples=20
  lat (usec)   : 20=0.01%, 50=50.05%, 100=49.86%, 250=0.01%, 500=0.07%
  lat (usec)   : 750=0.01%
  cpu          : usr=17.96%, sys=3.42%, ctx=524342, majf=0, minf=6158
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s
(45.7MB/s-45.7MB/s), io=1024MiB (1074MB), run=23485-23485msec
  WRITE: bw=106MiB/s (112MB/s), 106MiB/s-106MiB/s (112MB/s-112MB/s),
io=1024MiB (1074MB), run=9625-9625msec

Disk stats (read/write):
  nvme0n1: ios=262222/262144, merge=0/0, ticks=19871/6270, in_queue=0,
util=99.61%

Thanks
smitha

RE: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[Elliott, Robert (Servers)]

That errno is EILSEQ, which the linux block layer generates for a few
reasons related to the T10 protection information (using the internal
return value of BLK_STS_PROTECTION).

Look at your kernel log (dmesg command) for these two possible reasons:
    pr_err("%s: ref tag error at location %llu (rcvd %u)\n",
    pr_err("%s: guard tag error at sector %llu (rcvd %04x, want %04x)\n",

Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[smitha sunder]

On Thu, Jan 9, 2020 at 4:45 PM Elliott, Robert (Servers)
<elliott@hpe.com> wrote:
>
> That errno is EILSEQ, which the linux block layer generates for a few
> reasons related to the T10 protection information (using the internal
> return value of BLK_STS_PROTECTION).
>
> Look at your kernel log (dmesg command) for these two possible reasons:
>     pr_err("%s: ref tag error at location %llu (rcvd %u)\n",
>     pr_err("%s: guard tag error at sector %llu (rcvd %04x, want %04x)\n",
>
>

I don't see a REF TAG error: I did format the drive with pi=1 option :
# nvme format /dev/nvme1 --namespace-id=1 --lbaf=4 --pi=1 --ms=0 -f

kern  :info  : [Thu Jan  9 09:28:44 2020] nvme nvme0: rescanning namespaces.
kern  :info  : [Thu Jan  9 09:28:44 2020] nvme nvme1: rescanning namespaces.
kern  :info  : [Thu Jan  9 16:53:15 2020] nvme nvme0: rescanning namespaces.
kern  :info  : [Thu Jan  9 16:53:15 2020] nvme nvme1: rescanning namespaces.
kern  :info  : [Thu Jan  9 16:53:24 2020] nvme nvme0: rescanning namespaces.
kern  :info  : [Thu Jan  9 16:53:24 2020] nvme nvme1: rescanning namespaces.
kern  :warn  : [Thu Jan  9 16:53:37 2020] print_req_error: 4 callbacks
suppressed
kern  :err   : [Thu Jan  9 16:53:37 2020] blk_update_request:
protection error, dev nvme0n1, sector 8 op 0x0:(READ) flags 0x10000
phys_seg 1 prio class 0
kern  :err   : [Thu Jan  9 16:53:37 2020] Buffer I/O error on dev
nvme0n1, logical block 1, async page read
kern  :err   : [Thu Jan  9 16:53:37 2020] Dev nvme0n1: unable to read
RDB block 8
kern  :info  : [Thu Jan  9 16:53:37 2020]  nvme0n1: unable to read
partition table
kern  :err   : [Thu Jan  9 16:53:37 2020] blk_update_request:
protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags
0x90700 phys_seg 1 prio class 0
kern  :err   : [Thu Jan  9 16:53:37 2020] blk_update_request:
protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags
0x10000 phys_seg 1 prio class 0
kern  :err   : [Thu Jan  9 16:53:37 2020] Buffer I/O error on dev
nvme0n1, logical block 468843584, async page read
kern  :err   : [Thu Jan  9 16:53:37 2020] blk_update_request:
protection error, dev nvme0n1, sector 2048 op 0x1:(WRITE) flags
0x18800 phys_seg 1 prio class 0
kern  :err   : [Thu Jan  9 16:53:37 2020] blk_update_request:
protection error, dev nvme0n1, sector 8 op 0x0:(READ) flags 0x10000
phys_seg 1 prio class 0
kern  :err   : [Thu Jan  9 16:53:37 2020] Buffer I/O error on dev
nvme0n1, logical block 1, async page read
kern  :err   : [Thu Jan  9 16:53:37 2020] Dev nvme0n1: unable to read
RDB block 8
kern  :info  : [Thu Jan  9 16:53:37 2020]  nvme0n1: unable to read
partition table
kern  :err   : [Thu Jan  9 16:53:37 2020] blk_update_request:
protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags
0x90700 phys_seg 1 prio class 0
kern  :err   : [Thu Jan  9 16:53:37 2020] blk_update_request:
protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags
0x10000 phys_seg 1 prio class 0
kern  :err   : [Thu Jan  9 16:53:37 2020] Buffer I/O error on dev
nvme0n1, logical block 468843584, async page read

Also if its REF tag error, shouldn't I have seen the same issue with
4K+ 8B meta data? fio works fine with 4K+8 format.

Thanks
smitha

RE: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[Elliott, Robert (Servers)]

> Also if its REF tag error, shouldn't I have seen the same issue with
> 4K+ 8B meta data? fio works fine with 4K+8 format.

When PI was originally added to SCSI, the only PI size defined was
8 bytes. I don't know if the kernel supports 64 bytes, and what
it puts there if it does.

It's possible that reading a block before it's been written
could trigger PI errors, since the PI content is uninitialized.
I don't think fio ever does that (even with a verify thread)
- the --debug=io option will show all the IOs that it
generates. Check if the kernel log reports problems while
running some other application (e.g., dd).

What are all those reads to LBA 3750748672? Something is looking
for partition tables?

You're using direct=1 so the page cache shouldn't be doing 
prefetching. 

Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[Sitsofe Wheeler]

On Fri, 10 Jan 2020 at 06:35, Elliott, Robert (Servers) <elliott@hpe.com> wrote:
>
> What are all those reads to LBA 3750748672? Something is looking
> for partition tables?
>
> You're using direct=1 so the page cache shouldn't be doing
> prefetching.

If you scribble over the start of a disk that can trigger udev rules
to rescan partitions and maybe that is checking for a backup GPT...

-- 
Sitsofe | http://sucs.org/~sits/

Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[smitha sunder]

On Mon, Jan 13, 2020 at 12:04 AM Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>
> On Fri, 10 Jan 2020 at 06:35, Elliott, Robert (Servers) <elliott@hpe.com> wrote:
> >
> > What are all those reads to LBA 3750748672? Something is looking
> > for partition tables?
> >
> > You're using direct=1 so the page cache shouldn't be doing
> > prefetching.
>
> If you scribble over the start of a disk that can trigger udev rules
> to rescan partitions and maybe that is checking for a backup GPT...
>
> --
> Sitsofe | http://sucs.org/~sits/


If I issue "nvme write" that works :

#  echo "hello" | nvme write -s 0 -c 0 -y 64 -z 4096 --prinfo=0 /dev/nvme0n1
write: Success

# nvme read /dev/nvme0n1 -s 0  --prinfo=0 -z 4096  -y 64
hello
read: Success


fio fails :

#  fio --filename=/dev/nvme0n1  --rw=write --ioengine=libaio
--direct=1  --do_verify=0  --name=test --bs=4096 --number_ios=1
--offset=100M --debug=io
fio: set debug option io
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
io       20252 drop page cache /dev/nvme0n1
io       20252 fill: io_u 0xb4cc80:
off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
io       20252 prep: io_u 0xb4cc80:
off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
io       20252 prep: io_u 0xb4cc80: ret=0
io       20252 queue: io_u 0xb4cc80:
off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
io       20252 calling ->commit(), depth 1
io       20252 io_u_queued_complete: min=1
io       20252 getevents: 1
io       20252 complete: io_u 0xb4cc80:
off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte
or wide character: write offset=104857600, buflen=4096
io       20252 io_u_queued_complete: min=0
io       20252 getevents: 0
test: No I/O performed by libaio, perhaps try --debug=io option for details?
fio: pid=20252, err=84/file:io_u.c:1747, func=io_u error,
error=Invalid or incomplete multibyte or wide character
io       20252 close ioengine libaio
io       20252 free ioengine libaio

test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error,
error=Invalid or incomplete multibyte or wide character): pid=20252:
Fri Jan 17 13:13:52 2020
  cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs)

Thanks
smitha

Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[Matthew Eaton]

On Fri, Jan 17, 2020 at 1:18 PM smitha sunder <sundersmitha@gmail.com> wrote:
>
> On Mon, Jan 13, 2020 at 12:04 AM Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> >
> > On Fri, 10 Jan 2020 at 06:35, Elliott, Robert (Servers) <elliott@hpe.com> wrote:
> > >
> > > What are all those reads to LBA 3750748672? Something is looking
> > > for partition tables?
> > >
> > > You're using direct=1 so the page cache shouldn't be doing
> > > prefetching.
> >
> > If you scribble over the start of a disk that can trigger udev rules
> > to rescan partitions and maybe that is checking for a backup GPT...
> >
> > --
> > Sitsofe | http://sucs.org/~sits/
>
>
> If I issue "nvme write" that works :
>
> #  echo "hello" | nvme write -s 0 -c 0 -y 64 -z 4096 --prinfo=0 /dev/nvme0n1
> write: Success
>
> # nvme read /dev/nvme0n1 -s 0  --prinfo=0 -z 4096  -y 64
> hello
> read: Success
>
>
> fio fails :
>
> #  fio --filename=/dev/nvme0n1  --rw=write --ioengine=libaio
> --direct=1  --do_verify=0  --name=test --bs=4096 --number_ios=1
> --offset=100M --debug=io
> fio: set debug option io
> test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.7
> Starting 1 process
> io       20252 drop page cache /dev/nvme0n1
> io       20252 fill: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> io       20252 prep: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> io       20252 prep: io_u 0xb4cc80: ret=0
> io       20252 queue: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> io       20252 calling ->commit(), depth 1
> io       20252 io_u_queued_complete: min=1
> io       20252 getevents: 1
> io       20252 complete: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte
> or wide character: write offset=104857600, buflen=4096
> io       20252 io_u_queued_complete: min=0
> io       20252 getevents: 0
> test: No I/O performed by libaio, perhaps try --debug=io option for details?
> fio: pid=20252, err=84/file:io_u.c:1747, func=io_u error,
> error=Invalid or incomplete multibyte or wide character
> io       20252 close ioengine libaio
> io       20252 free ioengine libaio
>
> test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error,
> error=Invalid or incomplete multibyte or wide character): pid=20252:
> Fri Jan 17 13:13:52 2020
>   cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=24
>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
>      latency   : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs)
>
> Thanks
> smitha

Maybe try the latest version of fio or try changing ioengine to
something other than libaio?

Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096

[Sitsofe Wheeler]

On Fri, 17 Jan 2020 at 21:17, smitha sunder <sundersmitha@gmail.com> wrote:
>
> If I issue "nvme write" that works :
>
> #  echo "hello" | nvme write -s 0 -c 0 -y 64 -z 4096 --prinfo=0 /dev/nvme0n1
> write: Success
>
> # nvme read /dev/nvme0n1 -s 0  --prinfo=0 -z 4096  -y 64
> hello
> read: Success
>
>
> fio fails :
>
> #  fio --filename=/dev/nvme0n1  --rw=write --ioengine=libaio
> --direct=1  --do_verify=0  --name=test --bs=4096 --number_ios=1
> --offset=100M --debug=io
> fio: set debug option io
> test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=libaio, iodepth=1
> fio-3.7
> Starting 1 process
> io       20252 drop page cache /dev/nvme0n1
> io       20252 fill: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> io       20252 prep: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> io       20252 prep: io_u 0xb4cc80: ret=0
> io       20252 queue: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> io       20252 calling ->commit(), depth 1
> io       20252 io_u_queued_complete: min=1
> io       20252 getevents: 1
> io       20252 complete: io_u 0xb4cc80:
> off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1
> fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte
> or wide character: write offset=104857600, buflen=4096
> io       20252 io_u_queued_complete: min=0
> io       20252 getevents: 0
> test: No I/O performed by libaio, perhaps try --debug=io option for details?
> fio: pid=20252, err=84/file:io_u.c:1747, func=io_u error,
> error=Invalid or incomplete multibyte or wide character
> io       20252 close ioengine libaio
> io       20252 free ioengine libaio
>
> test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error,
> error=Invalid or incomplete multibyte or wide character): pid=20252:
> Fri Jan 17 13:13:52 2020
>   cpu          : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=24
>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
>      latency   : target=0, window=0, percentile=100.00%, depth=1
>
> Run status group 0 (all jobs)

Your nvme write is presumably crafting an entire NVMe command and
getting that sent to the device rather than going through the block
layer and letting that create NVMe commands so it's not really a fair
comparison (but it's another data point). I'd guess you're going to
still get kernel errors even using something like dd (but with the
appropriate direct flag with it and the same block size as fio)...

Generally speaking, if you're seeing kernel errors while doing block
level I/O that's indicative of something happening below fio and
you're better off talking to the folks on the linux-block list
(http://vger.kernel.org/vger-lists.html#linux-block ) to learn more...

--
Sitsofe | http://sucs.org/~sits/