* 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 @ 2020-01-08 17:51 smitha sunder 2020-01-09 7:40 ` Sitsofe Wheeler 0 siblings, 1 reply; 10+ messages in thread From: smitha sunder @ 2020-01-08 17:51 UTC (permalink / raw) To: fio Hello, If I format the drive with 512+8 LBAF run fio, I don’t see any issue : # nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 160401RICHC01 PM1733V4TLC 1 1.92 TB / 1.92 TB 512 B + 8 B TEMC8881 /dev/nvme1n1 160401RICHC01 PM1733V4TLC 1 1.92 TB / 1.92 TB 512 B + 8 B TEMC8881 [root@smitha-Z390-hawk tmp]# fio --filename=/dev/nvme0n1 --size=1M --rw=write --ioengine=libaio --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=512 test: (g=0): rw=write, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=32631: Wed Jan 8 09:40:32 2020 read: IOPS=18.0k, BW=9481KiB/s (9709kB/s)(1024KiB/108msec) <snip>…<snip> write: IOPS=27.3k, BW=13.3MiB/s (13.0MB/s)(1024KiB/75msec) Run status group 0 (all jobs): READ: bw=9481KiB/s (9709kB/s), 9481KiB/s-9481KiB/s (9709kB/s-9709kB/s), io=1024KiB (1049kB), run=108-108msec WRITE: bw=13.3MiB/s (13.0MB/s), 13.3MiB/s-13.3MiB/s (13.0MB/s-13.0MB/s), io=1024KiB (1049kB), run=75-75msec Disk stats (read/write): nvme0n1: ios=1288/2048, merge=0/0, ticks=67/66, in_queue=0, util=58.63% How ever if I format the drive with LBAF type 4 that is supported by the drive, then I get the following error : # nvme format /dev/nvme1 --namespace-id=1 --lbaf=4 --pi=1 --ms=0 Success formatting namespace:1 # nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 160401RICHC01 PM1733V4TLC 1 1.92 TB / 1.92 TB 4 KiB + 64 B TEMC8881 /dev/nvme1n1 160401RICHC01 PM1733V4TLC 1 1.92 TB / 1.92 TB 4 KiB + 64 B TEMC8881 # fio --filename=/dev/nvme0n1 --size=1M --rw=write --ioengine=libaio --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 fio: pid=32709, err=84/file:io_u.c:1747, func=io_u error, error=Invalid or incomplete multibyte or wide character test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error, error=Invalid or incomplete multibyte or wide character): pid=32709: Wed Jan 8 09:42:12 2020 write: IOPS=2000, BW=4000KiB/s (4096kB/s)(4096B/1msec) slat (nsec): min=12142, max=27097, avg=19619.50, stdev=10574.78 clat (nsec): min=24389, max=24389, avg=24389.00, stdev= 0.00 lat (nsec): min=52811, max=52811, avg=52811.00, stdev= 0.00 lat (usec) : 50=50.00% cpu : usr=0.00%, sys=0.00%, ctx=2, majf=0, minf=22 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=4000KiB/s (4096kB/s), 4000KiB/s-4000KiB/s (4096kB/s-4096kB/s), io=4096B (4096B), run=1-1msec Disk stats (read/write): nvme0n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% Am I missing something? Thanks for the help ! Smitha ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-08 17:51 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 smitha sunder @ 2020-01-09 7:40 ` Sitsofe Wheeler 2020-01-09 17:30 ` smitha sunder 0 siblings, 1 reply; 10+ messages in thread From: Sitsofe Wheeler @ 2020-01-09 7:40 UTC (permalink / raw) To: smitha sunder; +Cc: fio On Wed, 8 Jan 2020 at 19:19, smitha sunder <sundersmitha@gmail.com> wrote: > > How ever if I format the drive with LBAF type 4 that is supported by > the drive, then I get the following error : > > # nvme format /dev/nvme1 --namespace-id=1 --lbaf=4 --pi=1 --ms=0 > Success formatting namespace:1 > # nvme list > Node SN Model > Namespace Usage Format FW Rev > ---------------- -------------------- > ---------------------------------------- --------- > -------------------------- ---------------- -------- > /dev/nvme0n1 160401RICHC01 PM1733V4TLC > 1 1.92 TB / 1.92 TB 4 KiB + 64 B TEMC8881 > /dev/nvme1n1 160401RICHC01 PM1733V4TLC > 1 1.92 TB / 1.92 TB 4 KiB + 64 B TEMC8881 > > > # fio --filename=/dev/nvme0n1 --size=1M --rw=write --ioengine=libaio > --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096 > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > 4096B-4096B, ioengine=libaio, iodepth=1 > fio-3.7 > Starting 1 process > fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte > or wide character: write offset=4096, buflen=4096 > fio: pid=32709, err=84/file:io_u.c:1747, func=io_u error, > error=Invalid or incomplete multibyte or wide character [...] > Am I missing something? > Thanks for the help ! All fio is saying is that it didn't get back the data that it wrote down on the second block but I don't know what using "LBAF type 4" might do to your disk (you are saying it changes the logical block size but does it change anything else?). Do you don't have anything else that might be writing to that the device too and might have interfered with the data? What happens if you add --offset=1M? -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-09 7:40 ` Sitsofe Wheeler @ 2020-01-09 17:30 ` smitha sunder 2020-01-10 0:45 ` Elliott, Robert (Servers) 0 siblings, 1 reply; 10+ messages in thread From: smitha sunder @ 2020-01-09 17:30 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: fio On Wed, Jan 8, 2020 at 11:41 PM Sitsofe Wheeler <sitsofe@gmail.com> wrote: > > On Wed, 8 Jan 2020 at 19:19, smitha sunder <sundersmitha@gmail.com> wrote: > > > > How ever if I format the drive with LBAF type 4 that is supported by > > the drive, then I get the following error : > > > > # nvme format /dev/nvme1 --namespace-id=1 --lbaf=4 --pi=1 --ms=0 > > Success formatting namespace:1 > > # nvme list > > Node SN Model > > Namespace Usage Format FW Rev > > ---------------- -------------------- > > ---------------------------------------- --------- > > -------------------------- ---------------- -------- > > /dev/nvme0n1 160401RICHC01 PM1733V4TLC > > 1 1.92 TB / 1.92 TB 4 KiB + 64 B TEMC8881 > > /dev/nvme1n1 160401RICHC01 PM1733V4TLC > > 1 1.92 TB / 1.92 TB 4 KiB + 64 B TEMC8881 > > > > > > # fio --filename=/dev/nvme0n1 --size=1M --rw=write --ioengine=libaio > > --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096 > > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > > 4096B-4096B, ioengine=libaio, iodepth=1 > > fio-3.7 > > Starting 1 process > > fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte > > or wide character: write offset=4096, buflen=4096 > > fio: pid=32709, err=84/file:io_u.c:1747, func=io_u error, > > error=Invalid or incomplete multibyte or wide character > > [...] > > > Am I missing something? > > Thanks for the help ! > > All fio is saying is that it didn't get back the data that it wrote > down on the second block but I don't know what using "LBAF type 4" > might do to your disk (you are saying it changes the logical block > size but does it change anything else?). Do you don't have anything > else that might be writing to that the device too and might have > interfered with the data? What happens if you add --offset=1M? > > -- > Sitsofe | http://sucs.org/~sits/ Thanks Sitsofe for your reply and your time. The drive supports various LBA formats : LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x1 Better LBA Format 1 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good LBA Format 2 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best LBA Format 3 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good (in use) LBA Format 4 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good So LBAF 4 is 4K formmated with 64B metadata Nothing else is running on the device. The error is seen even if I issue offset=1M or 50% etc. # fio --filename=/dev/nvme0n1 --rw=write --ioengine=libaio --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096 --offset=1M test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte or wide character: write offset=1048576, buflen=4096 test: No I/O performed by libaio, perhaps try --debug=io option for details? fio: pid=3392, err=84/file:io_u.c:1747, func=io_u error, error=Invalid or incomplete multibyte or wide character test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error, error=Invalid or incomplete multibyte or wide character): pid=3392: Thu Jan 9 09:28:24 2020 cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=21 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): Disk stats (read/write): nvme0n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% If I format the drive to 1KiB+8B metadata then fio runs fine : # nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 160401RICHC01 PM1733V4TLC 1 1.92 TB / 1.92 TB 4 KiB + 8 B TEMC8881 /dev/nvme1n1 160401RICHC01 PM1733V4TLC 1 1.92 TB / 1.92 TB 4 KiB + 8 B TEMC8881 # fio --filename=/dev/nvme0n1 --rw=write --ioengine=libaio --direct=1 --verify=crc32 --verify_dump=1 --name=test --bs=4096 --offset=1M --size=1G test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [V(1)][100.0%][r=43.5MiB/s,w=0KiB/s][r=11.1k,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=3486: Thu Jan 9 09:30:19 2020 read: IOPS=11.2k, BW=43.6MiB/s (45.7MB/s)(1024MiB/23485msec) slat (nsec): min=1496, max=5603.7k, avg=1745.06, stdev=11027.69 clat (usec): min=35, max=517, avg=78.56, stdev=13.52 lat (usec): min=37, max=5672, avg=80.33, stdev=17.43 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 74], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 75], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 90], 95.00th=[ 90], | 99.00th=[ 90], 99.50th=[ 91], 99.90th=[ 347], 99.95th=[ 375], | 99.99th=[ 400] write: IOPS=27.2k, BW=106MiB/s (112MB/s)(1024MiB/9625msec) slat (nsec): min=11232, max=68549, avg=12131.36, stdev=469.35 clat (usec): min=18, max=372, avg=24.25, stdev= 4.91 lat (usec): min=29, max=384, avg=36.41, stdev= 4.95 clat percentiles (nsec): | 1.00th=[20352], 5.00th=[20352], 10.00th=[20608], 20.00th=[21120], | 30.00th=[22144], 40.00th=[22144], 50.00th=[22400], 60.00th=[22656], | 70.00th=[23168], 80.00th=[24448], 90.00th=[33536], 95.00th=[34048], | 99.00th=[36096], 99.50th=[38656], 99.90th=[48384], 99.95th=[54016], | 99.99th=[74240] bw ( KiB/s): min=28447, max=114880, per=96.25%, avg=104854.35, stdev=18105.13, samples=20 iops : min= 7111, max=28720, avg=26213.65, stdev=4526.47, samples=20 lat (usec) : 20=0.01%, 50=50.05%, 100=49.86%, 250=0.01%, 500=0.07% lat (usec) : 750=0.01% cpu : usr=17.96%, sys=3.42%, ctx=524342, majf=0, minf=6158 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=262144,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=1024MiB (1074MB), run=23485-23485msec WRITE: bw=106MiB/s (112MB/s), 106MiB/s-106MiB/s (112MB/s-112MB/s), io=1024MiB (1074MB), run=9625-9625msec Disk stats (read/write): nvme0n1: ios=262222/262144, merge=0/0, ticks=19871/6270, in_queue=0, util=99.61% Thanks smitha ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-09 17:30 ` smitha sunder @ 2020-01-10 0:45 ` Elliott, Robert (Servers) 2020-01-10 0:56 ` smitha sunder 0 siblings, 1 reply; 10+ messages in thread From: Elliott, Robert (Servers) @ 2020-01-10 0:45 UTC (permalink / raw) To: smitha sunder, Sitsofe Wheeler; +Cc: fio That errno is EILSEQ, which the linux block layer generates for a few reasons related to the T10 protection information (using the internal return value of BLK_STS_PROTECTION). Look at your kernel log (dmesg command) for these two possible reasons: pr_err("%s: ref tag error at location %llu (rcvd %u)\n", pr_err("%s: guard tag error at sector %llu (rcvd %04x, want %04x)\n", ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-10 0:45 ` Elliott, Robert (Servers) @ 2020-01-10 0:56 ` smitha sunder 2020-01-10 6:35 ` Elliott, Robert (Servers) 0 siblings, 1 reply; 10+ messages in thread From: smitha sunder @ 2020-01-10 0:56 UTC (permalink / raw) To: Elliott, Robert (Servers); +Cc: Sitsofe Wheeler, fio On Thu, Jan 9, 2020 at 4:45 PM Elliott, Robert (Servers) <elliott@hpe.com> wrote: > > That errno is EILSEQ, which the linux block layer generates for a few > reasons related to the T10 protection information (using the internal > return value of BLK_STS_PROTECTION). > > Look at your kernel log (dmesg command) for these two possible reasons: > pr_err("%s: ref tag error at location %llu (rcvd %u)\n", > pr_err("%s: guard tag error at sector %llu (rcvd %04x, want %04x)\n", > > I don't see a REF TAG error: I did format the drive with pi=1 option : # nvme format /dev/nvme1 --namespace-id=1 --lbaf=4 --pi=1 --ms=0 -f kern :info : [Thu Jan 9 09:28:44 2020] nvme nvme0: rescanning namespaces. kern :info : [Thu Jan 9 09:28:44 2020] nvme nvme1: rescanning namespaces. kern :info : [Thu Jan 9 16:53:15 2020] nvme nvme0: rescanning namespaces. kern :info : [Thu Jan 9 16:53:15 2020] nvme nvme1: rescanning namespaces. kern :info : [Thu Jan 9 16:53:24 2020] nvme nvme0: rescanning namespaces. kern :info : [Thu Jan 9 16:53:24 2020] nvme nvme1: rescanning namespaces. kern :warn : [Thu Jan 9 16:53:37 2020] print_req_error: 4 callbacks suppressed kern :err : [Thu Jan 9 16:53:37 2020] blk_update_request: protection error, dev nvme0n1, sector 8 op 0x0:(READ) flags 0x10000 phys_seg 1 prio class 0 kern :err : [Thu Jan 9 16:53:37 2020] Buffer I/O error on dev nvme0n1, logical block 1, async page read kern :err : [Thu Jan 9 16:53:37 2020] Dev nvme0n1: unable to read RDB block 8 kern :info : [Thu Jan 9 16:53:37 2020] nvme0n1: unable to read partition table kern :err : [Thu Jan 9 16:53:37 2020] blk_update_request: protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags 0x90700 phys_seg 1 prio class 0 kern :err : [Thu Jan 9 16:53:37 2020] blk_update_request: protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags 0x10000 phys_seg 1 prio class 0 kern :err : [Thu Jan 9 16:53:37 2020] Buffer I/O error on dev nvme0n1, logical block 468843584, async page read kern :err : [Thu Jan 9 16:53:37 2020] blk_update_request: protection error, dev nvme0n1, sector 2048 op 0x1:(WRITE) flags 0x18800 phys_seg 1 prio class 0 kern :err : [Thu Jan 9 16:53:37 2020] blk_update_request: protection error, dev nvme0n1, sector 8 op 0x0:(READ) flags 0x10000 phys_seg 1 prio class 0 kern :err : [Thu Jan 9 16:53:37 2020] Buffer I/O error on dev nvme0n1, logical block 1, async page read kern :err : [Thu Jan 9 16:53:37 2020] Dev nvme0n1: unable to read RDB block 8 kern :info : [Thu Jan 9 16:53:37 2020] nvme0n1: unable to read partition table kern :err : [Thu Jan 9 16:53:37 2020] blk_update_request: protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags 0x90700 phys_seg 1 prio class 0 kern :err : [Thu Jan 9 16:53:37 2020] blk_update_request: protection error, dev nvme0n1, sector 3750748672 op 0x0:(READ) flags 0x10000 phys_seg 1 prio class 0 kern :err : [Thu Jan 9 16:53:37 2020] Buffer I/O error on dev nvme0n1, logical block 468843584, async page read Also if its REF tag error, shouldn't I have seen the same issue with 4K+ 8B meta data? fio works fine with 4K+8 format. Thanks smitha ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-10 0:56 ` smitha sunder @ 2020-01-10 6:35 ` Elliott, Robert (Servers) 2020-01-13 8:04 ` Sitsofe Wheeler 0 siblings, 1 reply; 10+ messages in thread From: Elliott, Robert (Servers) @ 2020-01-10 6:35 UTC (permalink / raw) To: smitha sunder; +Cc: Sitsofe Wheeler, fio > Also if its REF tag error, shouldn't I have seen the same issue with > 4K+ 8B meta data? fio works fine with 4K+8 format. When PI was originally added to SCSI, the only PI size defined was 8 bytes. I don't know if the kernel supports 64 bytes, and what it puts there if it does. It's possible that reading a block before it's been written could trigger PI errors, since the PI content is uninitialized. I don't think fio ever does that (even with a verify thread) - the --debug=io option will show all the IOs that it generates. Check if the kernel log reports problems while running some other application (e.g., dd). What are all those reads to LBA 3750748672? Something is looking for partition tables? You're using direct=1 so the page cache shouldn't be doing prefetching. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-10 6:35 ` Elliott, Robert (Servers) @ 2020-01-13 8:04 ` Sitsofe Wheeler 2020-01-17 21:17 ` smitha sunder 0 siblings, 1 reply; 10+ messages in thread From: Sitsofe Wheeler @ 2020-01-13 8:04 UTC (permalink / raw) To: Elliott, Robert (Servers); +Cc: smitha sunder, fio On Fri, 10 Jan 2020 at 06:35, Elliott, Robert (Servers) <elliott@hpe.com> wrote: > > What are all those reads to LBA 3750748672? Something is looking > for partition tables? > > You're using direct=1 so the page cache shouldn't be doing > prefetching. If you scribble over the start of a disk that can trigger udev rules to rescan partitions and maybe that is checking for a backup GPT... -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-13 8:04 ` Sitsofe Wheeler @ 2020-01-17 21:17 ` smitha sunder 2020-01-17 22:15 ` Matthew Eaton 2020-01-18 15:48 ` Sitsofe Wheeler 0 siblings, 2 replies; 10+ messages in thread From: smitha sunder @ 2020-01-17 21:17 UTC (permalink / raw) To: Sitsofe Wheeler; +Cc: Elliott, Robert (Servers), fio On Mon, Jan 13, 2020 at 12:04 AM Sitsofe Wheeler <sitsofe@gmail.com> wrote: > > On Fri, 10 Jan 2020 at 06:35, Elliott, Robert (Servers) <elliott@hpe.com> wrote: > > > > What are all those reads to LBA 3750748672? Something is looking > > for partition tables? > > > > You're using direct=1 so the page cache shouldn't be doing > > prefetching. > > If you scribble over the start of a disk that can trigger udev rules > to rescan partitions and maybe that is checking for a backup GPT... > > -- > Sitsofe | http://sucs.org/~sits/ If I issue "nvme write" that works : # echo "hello" | nvme write -s 0 -c 0 -y 64 -z 4096 --prinfo=0 /dev/nvme0n1 write: Success # nvme read /dev/nvme0n1 -s 0 --prinfo=0 -z 4096 -y 64 hello read: Success fio fails : # fio --filename=/dev/nvme0n1 --rw=write --ioengine=libaio --direct=1 --do_verify=0 --name=test --bs=4096 --number_ios=1 --offset=100M --debug=io fio: set debug option io test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process io 20252 drop page cache /dev/nvme0n1 io 20252 fill: io_u 0xb4cc80: off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 io 20252 prep: io_u 0xb4cc80: off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 io 20252 prep: io_u 0xb4cc80: ret=0 io 20252 queue: io_u 0xb4cc80: off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 io 20252 calling ->commit(), depth 1 io 20252 io_u_queued_complete: min=1 io 20252 getevents: 1 io 20252 complete: io_u 0xb4cc80: off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte or wide character: write offset=104857600, buflen=4096 io 20252 io_u_queued_complete: min=0 io 20252 getevents: 0 test: No I/O performed by libaio, perhaps try --debug=io option for details? fio: pid=20252, err=84/file:io_u.c:1747, func=io_u error, error=Invalid or incomplete multibyte or wide character io 20252 close ioengine libaio io 20252 free ioengine libaio test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error, error=Invalid or incomplete multibyte or wide character): pid=20252: Fri Jan 17 13:13:52 2020 cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=24 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs) Thanks smitha ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-17 21:17 ` smitha sunder @ 2020-01-17 22:15 ` Matthew Eaton 2020-01-18 15:48 ` Sitsofe Wheeler 1 sibling, 0 replies; 10+ messages in thread From: Matthew Eaton @ 2020-01-17 22:15 UTC (permalink / raw) To: smitha sunder; +Cc: Sitsofe Wheeler, Elliott, Robert (Servers), fio On Fri, Jan 17, 2020 at 1:18 PM smitha sunder <sundersmitha@gmail.com> wrote: > > On Mon, Jan 13, 2020 at 12:04 AM Sitsofe Wheeler <sitsofe@gmail.com> wrote: > > > > On Fri, 10 Jan 2020 at 06:35, Elliott, Robert (Servers) <elliott@hpe.com> wrote: > > > > > > What are all those reads to LBA 3750748672? Something is looking > > > for partition tables? > > > > > > You're using direct=1 so the page cache shouldn't be doing > > > prefetching. > > > > If you scribble over the start of a disk that can trigger udev rules > > to rescan partitions and maybe that is checking for a backup GPT... > > > > -- > > Sitsofe | http://sucs.org/~sits/ > > > If I issue "nvme write" that works : > > # echo "hello" | nvme write -s 0 -c 0 -y 64 -z 4096 --prinfo=0 /dev/nvme0n1 > write: Success > > # nvme read /dev/nvme0n1 -s 0 --prinfo=0 -z 4096 -y 64 > hello > read: Success > > > fio fails : > > # fio --filename=/dev/nvme0n1 --rw=write --ioengine=libaio > --direct=1 --do_verify=0 --name=test --bs=4096 --number_ios=1 > --offset=100M --debug=io > fio: set debug option io > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > 4096B-4096B, ioengine=libaio, iodepth=1 > fio-3.7 > Starting 1 process > io 20252 drop page cache /dev/nvme0n1 > io 20252 fill: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > io 20252 prep: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > io 20252 prep: io_u 0xb4cc80: ret=0 > io 20252 queue: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > io 20252 calling ->commit(), depth 1 > io 20252 io_u_queued_complete: min=1 > io 20252 getevents: 1 > io 20252 complete: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte > or wide character: write offset=104857600, buflen=4096 > io 20252 io_u_queued_complete: min=0 > io 20252 getevents: 0 > test: No I/O performed by libaio, perhaps try --debug=io option for details? > fio: pid=20252, err=84/file:io_u.c:1747, func=io_u error, > error=Invalid or incomplete multibyte or wide character > io 20252 close ioengine libaio > io 20252 free ioengine libaio > > test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error, > error=Invalid or incomplete multibyte or wide character): pid=20252: > Fri Jan 17 13:13:52 2020 > cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=24 > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=1 > > Run status group 0 (all jobs) > > Thanks > smitha Maybe try the latest version of fio or try changing ioengine to something other than libaio? ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 2020-01-17 21:17 ` smitha sunder 2020-01-17 22:15 ` Matthew Eaton @ 2020-01-18 15:48 ` Sitsofe Wheeler 1 sibling, 0 replies; 10+ messages in thread From: Sitsofe Wheeler @ 2020-01-18 15:48 UTC (permalink / raw) To: smitha sunder; +Cc: Elliott, Robert (Servers), fio On Fri, 17 Jan 2020 at 21:17, smitha sunder <sundersmitha@gmail.com> wrote: > > If I issue "nvme write" that works : > > # echo "hello" | nvme write -s 0 -c 0 -y 64 -z 4096 --prinfo=0 /dev/nvme0n1 > write: Success > > # nvme read /dev/nvme0n1 -s 0 --prinfo=0 -z 4096 -y 64 > hello > read: Success > > > fio fails : > > # fio --filename=/dev/nvme0n1 --rw=write --ioengine=libaio > --direct=1 --do_verify=0 --name=test --bs=4096 --number_ios=1 > --offset=100M --debug=io > fio: set debug option io > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > 4096B-4096B, ioengine=libaio, iodepth=1 > fio-3.7 > Starting 1 process > io 20252 drop page cache /dev/nvme0n1 > io 20252 fill: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > io 20252 prep: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > io 20252 prep: io_u 0xb4cc80: ret=0 > io 20252 queue: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > io 20252 calling ->commit(), depth 1 > io 20252 io_u_queued_complete: min=1 > io 20252 getevents: 1 > io 20252 complete: io_u 0xb4cc80: > off=0x6400000,len=0x1000,ddir=1,file=/dev/nvme0n1 > fio: io_u error on file /dev/nvme0n1: Invalid or incomplete multibyte > or wide character: write offset=104857600, buflen=4096 > io 20252 io_u_queued_complete: min=0 > io 20252 getevents: 0 > test: No I/O performed by libaio, perhaps try --debug=io option for details? > fio: pid=20252, err=84/file:io_u.c:1747, func=io_u error, > error=Invalid or incomplete multibyte or wide character > io 20252 close ioengine libaio > io 20252 free ioengine libaio > > test: (groupid=0, jobs=1): err=84 (file:io_u.c:1747, func=io_u error, > error=Invalid or incomplete multibyte or wide character): pid=20252: > Fri Jan 17 13:13:52 2020 > cpu : usr=0.00%, sys=0.00%, ctx=1, majf=0, minf=24 > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=1 > > Run status group 0 (all jobs) Your nvme write is presumably crafting an entire NVMe command and getting that sent to the device rather than going through the block layer and letting that create NVMe commands so it's not really a fair comparison (but it's another data point). I'd guess you're going to still get kernel errors even using something like dd (but with the appropriate direct flag with it and the same block size as fio)... Generally speaking, if you're seeing kernel errors while doing block level I/O that's indicative of something happening below fio and you're better off talking to the folks on the linux-block list (http://vger.kernel.org/vger-lists.html#linux-block ) to learn more... -- Sitsofe | http://sucs.org/~sits/ ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-01-18 15:48 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-08 17:51 4KiB + 64 B formatted drive : Invalid or incomplete multibyte or wide character: write offset=4096, buflen=4096 smitha sunder 2020-01-09 7:40 ` Sitsofe Wheeler 2020-01-09 17:30 ` smitha sunder 2020-01-10 0:45 ` Elliott, Robert (Servers) 2020-01-10 0:56 ` smitha sunder 2020-01-10 6:35 ` Elliott, Robert (Servers) 2020-01-13 8:04 ` Sitsofe Wheeler 2020-01-17 21:17 ` smitha sunder 2020-01-17 22:15 ` Matthew Eaton 2020-01-18 15:48 ` Sitsofe Wheeler
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.