Re: CPUs, threads, and speed

[Andrey Kuzmin <andrey.v.kuzmin@gmail.com>]

On Thu, Jan 16, 2020 at 9:31 PM Jared Walton <jawalking@gmail.com> wrote:
>
> Not sure if this will help, but I use the following to prep multiple
> 4TB drives at the same time in a little over an hour.

You seem to be preconditioning with sequential writes only, and
further doing so
with essentially single write frontier.

That doesn't stress FTL maps enough and doesn't trigger any substantial garbage
collection since SSD is intelligent enough to spot sequential write
workload with
128K sequential (re)writes.

So what you're doing is only good for bandwidth measurements, and if
this steady
state is applied to random IOPS profiling, you'd be getting highly
inflated results.

Regards,
Andrey

> Is it inelegant, yes, but it works for me.
>
> globalFIOParameters="--offset=0 --ioengine=libaio --invalidate=1
> --group_reporting --direct=1 --thread --refill_buffers --norandommap
> --randrepeat=0 --allow_mounted_write=1 --output-format=json,normal"
>
> # Drives should be FOB or LLF'd (if it's good to do that)
> # LLF logic
>
> # 128k Pre-Condition
> # Write to entire disk
> for i in `ls -1 /dev/nvme*n1`
> do
>     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> print $1 }')
> ./fio --name=PreconditionPass1of3 --filename=${i} --iodepth=$iodepth
> --bs=128k --rw=write --size=${size} --fill_device=1
> $globalFIOParameters &
> done
> wait
>
> # Read entire disk
> for i in `ls -1 /dev/nvme*n1`
> do
>     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> print $1 }')
> ./fio --name=PreconditionPass2of3 --filename=${i} --iodepth=$iodepth
> --bs=128k --rw=read --size=${size} --fill_device=1
> $globalFIOParameters &
> done
> wait
>
> # Write to entire disk one last time
> for i in `ls -1 /dev/nvme*n1`
> do
>     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> print $1 }')
> ./fio --name=PreconditionPass3of3 --filename=${i} --iodepth=$iodepth
> --bs=128k --rw=write --size=${size} --fill_device=1
> $globalFIOParameters &
> done
> wait
>
>
> # Check 128k steady-state
> for i in `ls -1 /dev/nvme*n1`
> do
> ./fio --name=SteadyState --filename=${i} --iodepth=16 --numjobs=16
> --bs=4k --rw=read --ss_dur=1800 --ss=iops_slope:0.3% --runtime=24h
> $globalFIOParameters &
> done
> wait
>
> On Thu, Jan 16, 2020 at 9:13 AM Mauricio Tavares <raubvogel@gmail.com> wrote:
> >
> > On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > >
> > > On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > > > <joseph.r.gruher@intel.com> wrote:
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > > > > Mauricio Tavares
> > > > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > > > To: fio@vger.kernel.org
> > > > > > > > > Subject: CPUs, threads, and speed
> > > > > > > > >
> > > > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > > > _4KRandom_NVMe.ini)
> > > > > > > > >
> > > > > > > > > [global]
> > > > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > > ioengine=libaio
> > > > > > > > > direct=1
> > > > > > > > > bs=4k
> > > > > > > > > rw=randwrite
> > > > > > > > > iodepth=4
> > > > > > > > > numjobs=32
> > > > > > > > > buffered=0
> > > > > > > > > size=100%
> > > > > > > > > loops=2
> > > > > > > > > randrepeat=0
> > > > > > > > > norandommap
> > > > > > > > > refill_buffers
> > > > > > > > >
> > > > > > > > > [job1]
> > > > > > > > >
> > > > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > > > up?
> > > > > > > >
> > > > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > > > >
> > > > > > > I believe that preload here means what in SSD world is called drive
> > > > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > > > where it gives you the true performance in production over months of
> > > > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > > > >
> > > > > > > > A sequential workload with larger blocks will be faster,
> > > > > > >
> > > > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > > > populate FTL translation tables like random writes do.
> > > > > > >
> > > > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > > > worth of random writes. At today speeds, that should take just a
> > > > > > > couple of hours.
> > > > > > >
> > > > > >       When you say 2xcapacity worth of random writes, do you mean just
> > > > > > setting size=200%?
> > > > >
> > > > > Right.
> > > > >
> > > >       Then I wonder what I am doing wrong now. I changed the config file to
> > > >
> > > > [root@testbox tests]# cat preload.conf
> > > > [global]
> > > > name=4k random write 4 ios in the queue in 32 queues
> > > > ioengine=libaio
> > > > direct=1
> > > > bs=4k
> > > > rw=randwrite
> > > > iodepth=4
> > > > numjobs=32
> > > > buffered=0
> > > > size=200%
> > > > loops=2
> > > > random_generator=tausworthe64
> > > > thread=1
> > > >
> > > > [job1]
> > > > filename=/dev/nvme0n1
> > > > [root@testbox tests]#
> > > >
> > > > but when I run it, now it spits out much larger eta times:
> > > >
> > > > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > > > 16580099d:14h:55m:27s]]
> > >
> > >  Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> > > drive capacities here.
> > >
> > > Also, using 32 threads doesn't improve anything. 2 (and even one)
> > > threads with qd=128 will push the drive
> > > to its limits.
> > >
> >      Update: so I redid the config file a bit to pass some of the
> > arguments from command line, and cut down number of jobs and loops.
> > And I ran it again, this time sequential write to the drive I have not
> > touched to see how fast it was going to go. My eta is still
> > astronomical:
> >
> > [root@testbox tests]# cat preload_fio.conf
> > [global]
> > name=4k random
> > ioengine=${ioengine}
> > direct=1
> > bs=${bs_size}
> > rw=${iotype}
> > iodepth=4
> > numjobs=1
> > buffered=0
> > size=200%
> > loops=1
> >
> > [job1]
> > filename=${devicename}
> > [root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
> > iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
> > job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
> > 128KiB-128KiB, ioengine=libaio, iodepth=4
> > fio-3.17-68-g3f1e
> > Starting 1 process
> > Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]
> >
> > > Regards,
> > > Andrey
> > > >
> > > > Compare with what I was getting with size=100%
> > > >
> > > >  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> > > >
> > > > > Regards,
> > > > > Andrey
> > > > > >
> > > > > > > Regards,
> > > > > > > Andrey
> > > > > > >
> > > > > > > > like:
> > > > > > > >
> > > > > > > > [global]
> > > > > > > > ioengine=libaio
> > > > > > > > thread=1
> > > > > > > > direct=1
> > > > > > > > bs=128k
> > > > > > > > rw=write
> > > > > > > > numjobs=1
> > > > > > > > iodepth=128
> > > > > > > > size=100%
> > > > > > > > loops=2
> > > > > > > > [job00]
> > > > > > > > filename=/dev/nvme0n1
> > > > > > > >
> > > > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > > > >
> > > > > > > > -Joe