unaligned IO and starting LBA

unaligned IO and starting LBA

[smitha sunder]

Hello,



I have a 512 byte formatted nvme drive and I want to send an unaligned
IO and here’s the fio command line I am using , fio version fio-3.7

# fio --name=job1 --filename=/dev/nvme0n1  --rw=randwrite
--ioengine=libaio  --number_ios=1 --debug=io --bs_unaligned=1
--bs=1029

Starting 1 process

io       24015 drop page cache /dev/nvme0n1
io       24015 fill: io_u 0x8c9bc0:
off=0x1af7af9cec,len=0x405,ddir=1,file=/dev/nvme0n1
io       24015 prep: io_u 0x8c9bc0:
off=0x1af7af9cec,len=0x405,ddir=1,file=/dev/nvme0n1
io       24015 prep: io_u 0x8c9bc0: ret=0
io       24015 queue: io_u 0x8c9bc0:
off=0x1af7af9cec,len=0x405,ddir=1,file=/dev/nvme0n1
io       24015 calling ->commit(), depth 1
io       24015 io_u_queued_complete: min=1
io       24015 getevents: 1
io       24015 complete: io_u 0x8c9bc0:
off=0x1af7af9cec,len=0x405,ddir=1,file=/dev/nvme0n1
io       24015 close ioengine libaio
io       24015 free ioengine libaio



Node             SN                   Model
        Namespace Usage                      Format           FW Rev

---------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------

/dev/nvme0n1     160401RICHC01        PM1733V4TLC
        1           0.00   B /   1.92  TB    512   B +  0 B   NQ52
/dev/nvme1n1     160401RICHC01        PM1733V4TLC
        1           0.00   B /   1.92  TB    512   B +  0 B   NQ52



Shouldn’t I see D7BD7CE as the starting LBA for the write based on the
above offset (0x1af7af9cec) ?



From the PCI trace I see two writes with D7BD7C8 and D7BD7D0 as LBAs,
I am not sure why that is, any help is greatly appreciated :

; Write; Good; 11.345_5; 0; D7BD7C8; 1000; B008B; 0000; xxxx:A824;
00:10.336_991_850_0; 49.864_0; PCIe Expert Target(1:1:2);
; Write; Good; 13.761_5; 0; D7BD7D0; 1000; B008C; 0000; xxxx:A824;
00:10.336_995_406_0; 3.556_0; PCIe Expert Target(1:1:2);
; Identify; Good; 96359.046_9; 0;; 1000; 12; 0000; xxxx:A824;
00:10.433_423_218_0; 96427.812_0; PCIe Expert Target(1:1:2;
; Identify; Good; 41.635_5; 0;; 1000; 13; 0000; xxxx:A824;
00:10.439_182_147_0; 5758.929_0; PCIe Expert Target(1:1:2);

Thanks
smitha

RE: unaligned IO and starting LBA

[Elliott, Robert (Servers)]

> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of smitha sunder
> Sent: Tuesday, October 15, 2019 4:37 PM
> To: fio <fio@vger.kernel.org>
> Subject: unaligned IO and starting LBA
> 
> Hello,
> 
> 
> 
> I have a 512 byte formatted nvme drive and I want to send an unaligned
> IO and here’s the fio command line I am using , fio version fio-3.7
> 
> # fio --name=job1 --filename=/dev/nvme0n1  --rw=randwrite
> --ioengine=libaio  --number_ios=1 --debug=io --bs_unaligned=1
> --bs=1029
... 
> Shouldn’t I see D7BD7CE as the starting LBA for the write based on the
> above offset (0x1af7af9cec) ?

Without direct=1 you're exercising the OS page cache logic,
not directly accessing the drive.

Re: unaligned IO and starting LBA

[smitha sunder]

Hi Robert,

Since I want to use blocksize_unaligned=1 , I had to get rid of direct=1 :

Per man page :
blocksize_unaligned, bs_unaligned
If set, any size in blocksize_range may be used. This typically won't
work with direct I/O, as that normally requires sector alignment.

# fio --name=job1 --filename=/dev/nvme0n1  --rw=randwrite
--ioengine=libaio  --number_ios=1 --debug=io --bs_unaligned=1
--bs=1029 --direct=1
fio: set debug option io
fio: bs_unaligned may not work with raw io

Thanks
smitha

On Wed, Oct 16, 2019 at 8:12 AM Elliott, Robert (Servers)
<elliott@hpe.com> wrote:
>
>
>
> > -----Original Message-----
> > From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of smitha sunder
> > Sent: Tuesday, October 15, 2019 4:37 PM
> > To: fio <fio@vger.kernel.org>
> > Subject: unaligned IO and starting LBA
> >
> > Hello,
> >
> >
> >
> > I have a 512 byte formatted nvme drive and I want to send an unaligned
> > IO and here’s the fio command line I am using , fio version fio-3.7
> >
> > # fio --name=job1 --filename=/dev/nvme0n1  --rw=randwrite
> > --ioengine=libaio  --number_ios=1 --debug=io --bs_unaligned=1
> > --bs=1029
> ...
> > Shouldn’t I see D7BD7CE as the starting LBA for the write based on the
> > above offset (0x1af7af9cec) ?
>
> Without direct=1 you're exercising the OS page cache logic,
> not directly accessing the drive.
>

Re: unaligned IO and starting LBA

[Sitsofe Wheeler]

On Thu, 17 Oct 2019 at 13:49, smitha sunder <sundersmitha@gmail.com> wrote:
>
> Hi Robert,
>
> Since I want to use blocksize_unaligned=1 , I had to get rid of direct=1 :
>
> Per man page :
> blocksize_unaligned, bs_unaligned
> If set, any size in blocksize_range may be used. This typically won't
> work with direct I/O, as that normally requires sector alignment.
>
> # fio --name=job1 --filename=/dev/nvme0n1  --rw=randwrite
> --ioengine=libaio  --number_ios=1 --debug=io --bs_unaligned=1
> --bs=1029 --direct=1
> fio: set debug option io
> fio: bs_unaligned may not work with raw io

> On Wed, Oct 16, 2019 at 8:12 AM Elliott, Robert (Servers)
> <elliott@hpe.com> wrote:
> >
> > > -----Original Message-----
> > > From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of smitha sunder
> > > Sent: Tuesday, October 15, 2019 4:37 PM
> > > To: fio <fio@vger.kernel.org>
> > > Subject: unaligned IO and starting LBA
> > >
> > > Hello,
> > >
> > > I have a 512 byte formatted nvme drive and I want to send an unaligned
> > > IO and here’s the fio command line I am using , fio version fio-3.7
> > >
> > > # fio --name=job1 --filename=/dev/nvme0n1  --rw=randwrite
> > > --ioengine=libaio  --number_ios=1 --debug=io --bs_unaligned=1
> > > --bs=1029

A blocksize of 1029 is neither a multiple of 4096 nor 512! You simply
can't send I/O that is not aligned (and is not a multiple) of the
device's logical block size down to a device so something will have to
"fix" things for you.

> > > Shouldn’t I see D7BD7CE as the starting LBA for the write based on the
> > > above offset (0x1af7af9cec) ?
> >
> > Without direct=1 you're exercising the OS page cache logic,
> > not directly accessing the drive.

Robert's right - when you have buffering going on the linux kernel
will quietly do a read-modify-write (two reads assuming it didn't
already have those blocks in memory, a modify, then two writes)
because you straddled a block.

-- 
Sitsofe | http://sucs.org/~sits/