CPUs, threads, and speed

CPUs, threads, and speed

[Mauricio Tavares]

Let's say I have a config file to preload drive that looks like this
(stolen from https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill_4KRandom_NVMe.ini)

[global]
name=4k random write 4 ios in the queue in 32 queues
filename=/dev/nvme0n1
ioengine=libaio
direct=1
bs=4k
rw=randwrite
iodepth=4
numjobs=32
buffered=0
size=100%
loops=2
randrepeat=0
norandommap
refill_buffers

[job1]

That is taking a ton of time, like days to go. Is there anything I can
do to speed it up? For instance, what is the default value for
cpus_allowed (or cpumask)[2]? Is it all CPUs? If not what would I gain
by throwing more cpus at the problem?

I also read[2] by default fio uses fork. What would I get by going to threads?

[2] https://fio.readthedocs.io/en/latest/fio_doc.html#threads-processes-and-job-synchronization

RE: CPUs, threads, and speed

[Gruher, Joseph R]

> -----Original Message-----
> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> Mauricio Tavares
> Sent: Wednesday, January 15, 2020 7:51 AM
> To: fio@vger.kernel.org
> Subject: CPUs, threads, and speed
> 
> Let's say I have a config file to preload drive that looks like this (stolen from
> https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> _4KRandom_NVMe.ini)
> 
> [global]
> name=4k random write 4 ios in the queue in 32 queues
> filename=/dev/nvme0n1
> ioengine=libaio
> direct=1
> bs=4k
> rw=randwrite
> iodepth=4
> numjobs=32
> buffered=0
> size=100%
> loops=2
> randrepeat=0
> norandommap
> refill_buffers
> 
> [job1]
> 
> That is taking a ton of time, like days to go. Is there anything I can do to speed it
> up? 

When you say preload, do you just want to write in the full capacity of the drive?  A sequential workload with larger blocks will be faster, like:

[global]
ioengine=libaio
thread=1
direct=1
bs=128k
rw=write
numjobs=1
iodepth=128
size=100%
loops=2
[job00]
filename=/dev/nvme0n1

Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.

-Joe

Re: CPUs, threads, and speed

[Andrey Kuzmin]

On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
<joseph.r.gruher@intel.com> wrote:
>
> > -----Original Message-----
> > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > Mauricio Tavares
> > Sent: Wednesday, January 15, 2020 7:51 AM
> > To: fio@vger.kernel.org
> > Subject: CPUs, threads, and speed
> >
> > Let's say I have a config file to preload drive that looks like this (stolen from
> > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > _4KRandom_NVMe.ini)
> >
> > [global]
> > name=4k random write 4 ios in the queue in 32 queues
> > filename=/dev/nvme0n1
> > ioengine=libaio
> > direct=1
> > bs=4k
> > rw=randwrite
> > iodepth=4
> > numjobs=32
> > buffered=0
> > size=100%
> > loops=2
> > randrepeat=0
> > norandommap
> > refill_buffers
> >
> > [job1]
> >
> > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > up?
>
> When you say preload, do you just want to write in the full capacity of the drive?

I believe that preload here means what in SSD world is called drive
preconditioning. It means bringing a fresh drive into steady mode
where it gives you the true performance in production over months of
use rather than the unrealistic fresh drive random write IOPS.

> A sequential workload with larger blocks will be faster,

No, you cannot get the job done by sequential writes since it doesn't
populate FTL translation tables like random writes do.

As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
worth of random writes. At today speeds, that should take just a
couple of hours.

Regards,
Andrey

> like:
>
> [global]
> ioengine=libaio
> thread=1
> direct=1
> bs=128k
> rw=write
> numjobs=1
> iodepth=128
> size=100%
> loops=2
> [job00]
> filename=/dev/nvme0n1
>
> Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
>
> -Joe

Re: CPUs, threads, and speed

[Mauricio Tavares]

On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
>
> On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> <joseph.r.gruher@intel.com> wrote:
> >
> > > -----Original Message-----
> > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > Mauricio Tavares
> > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > To: fio@vger.kernel.org
> > > Subject: CPUs, threads, and speed
> > >
> > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > _4KRandom_NVMe.ini)
> > >
> > > [global]
> > > name=4k random write 4 ios in the queue in 32 queues
> > > filename=/dev/nvme0n1
> > > ioengine=libaio
> > > direct=1
> > > bs=4k
> > > rw=randwrite
> > > iodepth=4
> > > numjobs=32
> > > buffered=0
> > > size=100%
> > > loops=2
> > > randrepeat=0
> > > norandommap
> > > refill_buffers
> > >
> > > [job1]
> > >
> > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > up?
> >
> > When you say preload, do you just want to write in the full capacity of the drive?
>
> I believe that preload here means what in SSD world is called drive
> preconditioning. It means bringing a fresh drive into steady mode
> where it gives you the true performance in production over months of
> use rather than the unrealistic fresh drive random write IOPS.
>
> > A sequential workload with larger blocks will be faster,
>
> No, you cannot get the job done by sequential writes since it doesn't
> populate FTL translation tables like random writes do.
>
> As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> worth of random writes. At today speeds, that should take just a
> couple of hours.
>
      When you say 2xcapacity worth of random writes, do you mean just
setting size=200%?

> Regards,
> Andrey
>
> > like:
> >
> > [global]
> > ioengine=libaio
> > thread=1
> > direct=1
> > bs=128k
> > rw=write
> > numjobs=1
> > iodepth=128
> > size=100%
> > loops=2
> > [job00]
> > filename=/dev/nvme0n1
> >
> > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> >
> > -Joe

Re: CPUs, threads, and speed

[Andrey Kuzmin]

On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
>
> On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> >
> > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > <joseph.r.gruher@intel.com> wrote:
> > >
> > > > -----Original Message-----
> > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > Mauricio Tavares
> > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > To: fio@vger.kernel.org
> > > > Subject: CPUs, threads, and speed
> > > >
> > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > _4KRandom_NVMe.ini)
> > > >
> > > > [global]
> > > > name=4k random write 4 ios in the queue in 32 queues
> > > > filename=/dev/nvme0n1
> > > > ioengine=libaio
> > > > direct=1
> > > > bs=4k
> > > > rw=randwrite
> > > > iodepth=4
> > > > numjobs=32
> > > > buffered=0
> > > > size=100%
> > > > loops=2
> > > > randrepeat=0
> > > > norandommap
> > > > refill_buffers
> > > >
> > > > [job1]
> > > >
> > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > up?
> > >
> > > When you say preload, do you just want to write in the full capacity of the drive?
> >
> > I believe that preload here means what in SSD world is called drive
> > preconditioning. It means bringing a fresh drive into steady mode
> > where it gives you the true performance in production over months of
> > use rather than the unrealistic fresh drive random write IOPS.
> >
> > > A sequential workload with larger blocks will be faster,
> >
> > No, you cannot get the job done by sequential writes since it doesn't
> > populate FTL translation tables like random writes do.
> >
> > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > worth of random writes. At today speeds, that should take just a
> > couple of hours.
> >
>       When you say 2xcapacity worth of random writes, do you mean just
> setting size=200%?

Right.

Regards,
Andrey
>
> > Regards,
> > Andrey
> >
> > > like:
> > >
> > > [global]
> > > ioengine=libaio
> > > thread=1
> > > direct=1
> > > bs=128k
> > > rw=write
> > > numjobs=1
> > > iodepth=128
> > > size=100%
> > > loops=2
> > > [job00]
> > > filename=/dev/nvme0n1
> > >
> > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > >
> > > -Joe

Re: CPUs, threads, and speed

[Mauricio Tavares]

On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
>
> On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> >
> > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > >
> > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > <joseph.r.gruher@intel.com> wrote:
> > > >
> > > > > -----Original Message-----
> > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > Mauricio Tavares
> > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > To: fio@vger.kernel.org
> > > > > Subject: CPUs, threads, and speed
> > > > >
> > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > _4KRandom_NVMe.ini)
> > > > >
> > > > > [global]
> > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > filename=/dev/nvme0n1
> > > > > ioengine=libaio
> > > > > direct=1
> > > > > bs=4k
> > > > > rw=randwrite
> > > > > iodepth=4
> > > > > numjobs=32
> > > > > buffered=0
> > > > > size=100%
> > > > > loops=2
> > > > > randrepeat=0
> > > > > norandommap
> > > > > refill_buffers
> > > > >
> > > > > [job1]
> > > > >
> > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > up?
> > > >
> > > > When you say preload, do you just want to write in the full capacity of the drive?
> > >
> > > I believe that preload here means what in SSD world is called drive
> > > preconditioning. It means bringing a fresh drive into steady mode
> > > where it gives you the true performance in production over months of
> > > use rather than the unrealistic fresh drive random write IOPS.
> > >
> > > > A sequential workload with larger blocks will be faster,
> > >
> > > No, you cannot get the job done by sequential writes since it doesn't
> > > populate FTL translation tables like random writes do.
> > >
> > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > worth of random writes. At today speeds, that should take just a
> > > couple of hours.
> > >
> >       When you say 2xcapacity worth of random writes, do you mean just
> > setting size=200%?
>
> Right.
>
      Then I wonder what I am doing wrong now. I changed the config file to

[root@testbox tests]# cat preload.conf
[global]
name=4k random write 4 ios in the queue in 32 queues
ioengine=libaio
direct=1
bs=4k
rw=randwrite
iodepth=4
numjobs=32
buffered=0
size=200%
loops=2
random_generator=tausworthe64
thread=1

[job1]
filename=/dev/nvme0n1
[root@testbox tests]#

but when I run it, now it spits out much larger eta times:

Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
16580099d:14h:55m:27s]]

Compare with what I was getting with size=100%

 Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]

> Regards,
> Andrey
> >
> > > Regards,
> > > Andrey
> > >
> > > > like:
> > > >
> > > > [global]
> > > > ioengine=libaio
> > > > thread=1
> > > > direct=1
> > > > bs=128k
> > > > rw=write
> > > > numjobs=1
> > > > iodepth=128
> > > > size=100%
> > > > loops=2
> > > > [job00]
> > > > filename=/dev/nvme0n1
> > > >
> > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > >
> > > > -Joe

Re: CPUs, threads, and speed

[Andrey Kuzmin]

On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
>
> On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> >
> > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > >
> > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > <joseph.r.gruher@intel.com> wrote:
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > Mauricio Tavares
> > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > To: fio@vger.kernel.org
> > > > > > Subject: CPUs, threads, and speed
> > > > > >
> > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > _4KRandom_NVMe.ini)
> > > > > >
> > > > > > [global]
> > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > filename=/dev/nvme0n1
> > > > > > ioengine=libaio
> > > > > > direct=1
> > > > > > bs=4k
> > > > > > rw=randwrite
> > > > > > iodepth=4
> > > > > > numjobs=32
> > > > > > buffered=0
> > > > > > size=100%
> > > > > > loops=2
> > > > > > randrepeat=0
> > > > > > norandommap
> > > > > > refill_buffers
> > > > > >
> > > > > > [job1]
> > > > > >
> > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > up?
> > > > >
> > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > >
> > > > I believe that preload here means what in SSD world is called drive
> > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > where it gives you the true performance in production over months of
> > > > use rather than the unrealistic fresh drive random write IOPS.
> > > >
> > > > > A sequential workload with larger blocks will be faster,
> > > >
> > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > populate FTL translation tables like random writes do.
> > > >
> > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > worth of random writes. At today speeds, that should take just a
> > > > couple of hours.
> > > >
> > >       When you say 2xcapacity worth of random writes, do you mean just
> > > setting size=200%?
> >
> > Right.
> >
>       Then I wonder what I am doing wrong now. I changed the config file to
>
> [root@testbox tests]# cat preload.conf
> [global]
> name=4k random write 4 ios in the queue in 32 queues
> ioengine=libaio
> direct=1
> bs=4k
> rw=randwrite
> iodepth=4
> numjobs=32
> buffered=0
> size=200%
> loops=2
> random_generator=tausworthe64
> thread=1
>
> [job1]
> filename=/dev/nvme0n1
> [root@testbox tests]#
>
> but when I run it, now it spits out much larger eta times:
>
> Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> 16580099d:14h:55m:27s]]

 Size is set on per thread basis, so you're doing 32x200%x2 loops=128
drive capacities here.

Also, using 32 threads doesn't improve anything. 2 (and even one)
threads with qd=128 will push the drive
to its limits.

Regards,
Andrey
>
> Compare with what I was getting with size=100%
>
>  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
>
> > Regards,
> > Andrey
> > >
> > > > Regards,
> > > > Andrey
> > > >
> > > > > like:
> > > > >
> > > > > [global]
> > > > > ioengine=libaio
> > > > > thread=1
> > > > > direct=1
> > > > > bs=128k
> > > > > rw=write
> > > > > numjobs=1
> > > > > iodepth=128
> > > > > size=100%
> > > > > loops=2
> > > > > [job00]
> > > > > filename=/dev/nvme0n1
> > > > >
> > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > >
> > > > > -Joe

Re: CPUs, threads, and speed

[Mauricio Tavares]

On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
>
> On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> >
> > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > >
> > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > <joseph.r.gruher@intel.com> wrote:
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > > Mauricio Tavares
> > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > To: fio@vger.kernel.org
> > > > > > > Subject: CPUs, threads, and speed
> > > > > > >
> > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > _4KRandom_NVMe.ini)
> > > > > > >
> > > > > > > [global]
> > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > filename=/dev/nvme0n1
> > > > > > > ioengine=libaio
> > > > > > > direct=1
> > > > > > > bs=4k
> > > > > > > rw=randwrite
> > > > > > > iodepth=4
> > > > > > > numjobs=32
> > > > > > > buffered=0
> > > > > > > size=100%
> > > > > > > loops=2
> > > > > > > randrepeat=0
> > > > > > > norandommap
> > > > > > > refill_buffers
> > > > > > >
> > > > > > > [job1]
> > > > > > >
> > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > up?
> > > > > >
> > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > >
> > > > > I believe that preload here means what in SSD world is called drive
> > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > where it gives you the true performance in production over months of
> > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > >
> > > > > > A sequential workload with larger blocks will be faster,
> > > > >
> > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > populate FTL translation tables like random writes do.
> > > > >
> > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > worth of random writes. At today speeds, that should take just a
> > > > > couple of hours.
> > > > >
> > > >       When you say 2xcapacity worth of random writes, do you mean just
> > > > setting size=200%?
> > >
> > > Right.
> > >
> >       Then I wonder what I am doing wrong now. I changed the config file to
> >
> > [root@testbox tests]# cat preload.conf
> > [global]
> > name=4k random write 4 ios in the queue in 32 queues
> > ioengine=libaio
> > direct=1
> > bs=4k
> > rw=randwrite
> > iodepth=4
> > numjobs=32
> > buffered=0
> > size=200%
> > loops=2
> > random_generator=tausworthe64
> > thread=1
> >
> > [job1]
> > filename=/dev/nvme0n1
> > [root@testbox tests]#
> >
> > but when I run it, now it spits out much larger eta times:
> >
> > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > 16580099d:14h:55m:27s]]
>
>  Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> drive capacities here.
>
> Also, using 32 threads doesn't improve anything. 2 (and even one)
> threads with qd=128 will push the drive
> to its limits.
>
     Update: so I redid the config file a bit to pass some of the
arguments from command line, and cut down number of jobs and loops.
And I ran it again, this time sequential write to the drive I have not
touched to see how fast it was going to go. My eta is still
astronomical:

[root@testbox tests]# cat preload_fio.conf
[global]
name=4k random
ioengine=${ioengine}
direct=1
bs=${bs_size}
rw=${iotype}
iodepth=4
numjobs=1
buffered=0
size=200%
loops=1

[job1]
filename=${devicename}
[root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
128KiB-128KiB, ioengine=libaio, iodepth=4
fio-3.17-68-g3f1e
Starting 1 process
Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]

> Regards,
> Andrey
> >
> > Compare with what I was getting with size=100%
> >
> >  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> >
> > > Regards,
> > > Andrey
> > > >
> > > > > Regards,
> > > > > Andrey
> > > > >
> > > > > > like:
> > > > > >
> > > > > > [global]
> > > > > > ioengine=libaio
> > > > > > thread=1
> > > > > > direct=1
> > > > > > bs=128k
> > > > > > rw=write
> > > > > > numjobs=1
> > > > > > iodepth=128
> > > > > > size=100%
> > > > > > loops=2
> > > > > > [job00]
> > > > > > filename=/dev/nvme0n1
> > > > > >
> > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > >
> > > > > > -Joe

Re: CPUs, threads, and speed

[Andrey Kuzmin]

On Thu, Jan 16, 2020 at 7:13 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
>
> On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> >
> > On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > >
> > > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > > <joseph.r.gruher@intel.com> wrote:
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > > > Mauricio Tavares
> > > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > > To: fio@vger.kernel.org
> > > > > > > > Subject: CPUs, threads, and speed
> > > > > > > >
> > > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > > _4KRandom_NVMe.ini)
> > > > > > > >
> > > > > > > > [global]
> > > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > > filename=/dev/nvme0n1
> > > > > > > > ioengine=libaio
> > > > > > > > direct=1
> > > > > > > > bs=4k
> > > > > > > > rw=randwrite
> > > > > > > > iodepth=4
> > > > > > > > numjobs=32
> > > > > > > > buffered=0
> > > > > > > > size=100%
> > > > > > > > loops=2
> > > > > > > > randrepeat=0
> > > > > > > > norandommap
> > > > > > > > refill_buffers
> > > > > > > >
> > > > > > > > [job1]
> > > > > > > >
> > > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > > up?
> > > > > > >
> > > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > > >
> > > > > > I believe that preload here means what in SSD world is called drive
> > > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > > where it gives you the true performance in production over months of
> > > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > > >
> > > > > > > A sequential workload with larger blocks will be faster,
> > > > > >
> > > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > > populate FTL translation tables like random writes do.
> > > > > >
> > > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > > worth of random writes. At today speeds, that should take just a
> > > > > > couple of hours.
> > > > > >
> > > > >       When you say 2xcapacity worth of random writes, do you mean just
> > > > > setting size=200%?
> > > >
> > > > Right.
> > > >
> > >       Then I wonder what I am doing wrong now. I changed the config file to
> > >
> > > [root@testbox tests]# cat preload.conf
> > > [global]
> > > name=4k random write 4 ios in the queue in 32 queues
> > > ioengine=libaio
> > > direct=1
> > > bs=4k
> > > rw=randwrite
> > > iodepth=4
> > > numjobs=32
> > > buffered=0
> > > size=200%
> > > loops=2
> > > random_generator=tausworthe64
> > > thread=1
> > >
> > > [job1]
> > > filename=/dev/nvme0n1
> > > [root@testbox tests]#
> > >
> > > but when I run it, now it spits out much larger eta times:
> > >
> > > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > > 16580099d:14h:55m:27s]]
> >
> >  Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> > drive capacities here.
> >
> > Also, using 32 threads doesn't improve anything. 2 (and even one)
> > threads with qd=128 will push the drive
> > to its limits.
> >
>      Update: so I redid the config file a bit to pass some of the
> arguments from command line, and cut down number of jobs and loops.
> And I ran it again, this time sequential write to the drive I have not
> touched to see how fast it was going to go. My eta is still
> astronomical:
>
> [root@testbox tests]# cat preload_fio.conf
> [global]
> name=4k random
> ioengine=${ioengine}
> direct=1
> bs=${bs_size}
> rw=${iotype}
> iodepth=4
> numjobs=1
> buffered=0
> size=200%
> loops=1
>
> [job1]
> filename=${devicename}
> [root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
> iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
> job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
> 128KiB-128KiB, ioengine=libaio, iodepth=4
> fio-3.17-68-g3f1e
> Starting 1 process
> Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]

At almost 2GBps, your run against 4TB drive should take about an hor.
I'm not sure if fio properly reports ETA for size-based jobs at all.
You should check with Jens.

Regards,
Andrey

>
> > Regards,
> > Andrey
> > >
> > > Compare with what I was getting with size=100%
> > >
> > >  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> > >
> > > > Regards,
> > > > Andrey
> > > > >
> > > > > > Regards,
> > > > > > Andrey
> > > > > >
> > > > > > > like:
> > > > > > >
> > > > > > > [global]
> > > > > > > ioengine=libaio
> > > > > > > thread=1
> > > > > > > direct=1
> > > > > > > bs=128k
> > > > > > > rw=write
> > > > > > > numjobs=1
> > > > > > > iodepth=128
> > > > > > > size=100%
> > > > > > > loops=2
> > > > > > > [job00]
> > > > > > > filename=/dev/nvme0n1
> > > > > > >
> > > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > > >
> > > > > > > -Joe

Re: CPUs, threads, and speed

[Jared Walton]

Not sure if this will help, but I use the following to prep multiple
4TB drives at the same time in a little over an hour. Is it inelegant,
yes, but it works for me.

globalFIOParameters="--offset=0 --ioengine=libaio --invalidate=1
--group_reporting --direct=1 --thread --refill_buffers --norandommap
--randrepeat=0 --allow_mounted_write=1 --output-format=json,normal"

# Drives should be FOB or LLF'd (if it's good to do that)
# LLF logic

# 128k Pre-Condition
# Write to entire disk
for i in `ls -1 /dev/nvme*n1`
do
    size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
print $1 }')
./fio --name=PreconditionPass1of3 --filename=${i} --iodepth=$iodepth
--bs=128k --rw=write --size=${size} --fill_device=1
$globalFIOParameters &
done
wait

# Read entire disk
for i in `ls -1 /dev/nvme*n1`
do
    size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
print $1 }')
./fio --name=PreconditionPass2of3 --filename=${i} --iodepth=$iodepth
--bs=128k --rw=read --size=${size} --fill_device=1
$globalFIOParameters &
done
wait

# Write to entire disk one last time
for i in `ls -1 /dev/nvme*n1`
do
    size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
print $1 }')
./fio --name=PreconditionPass3of3 --filename=${i} --iodepth=$iodepth
--bs=128k --rw=write --size=${size} --fill_device=1
$globalFIOParameters &
done
wait


# Check 128k steady-state
for i in `ls -1 /dev/nvme*n1`
do
./fio --name=SteadyState --filename=${i} --iodepth=16 --numjobs=16
--bs=4k --rw=read --ss_dur=1800 --ss=iops_slope:0.3% --runtime=24h
$globalFIOParameters &
done
wait

On Thu, Jan 16, 2020 at 9:13 AM Mauricio Tavares <raubvogel@gmail.com> wrote:
>
> On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> >
> > On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > >
> > > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > > <joseph.r.gruher@intel.com> wrote:
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > > > Mauricio Tavares
> > > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > > To: fio@vger.kernel.org
> > > > > > > > Subject: CPUs, threads, and speed
> > > > > > > >
> > > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > > _4KRandom_NVMe.ini)
> > > > > > > >
> > > > > > > > [global]
> > > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > > filename=/dev/nvme0n1
> > > > > > > > ioengine=libaio
> > > > > > > > direct=1
> > > > > > > > bs=4k
> > > > > > > > rw=randwrite
> > > > > > > > iodepth=4
> > > > > > > > numjobs=32
> > > > > > > > buffered=0
> > > > > > > > size=100%
> > > > > > > > loops=2
> > > > > > > > randrepeat=0
> > > > > > > > norandommap
> > > > > > > > refill_buffers
> > > > > > > >
> > > > > > > > [job1]
> > > > > > > >
> > > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > > up?
> > > > > > >
> > > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > > >
> > > > > > I believe that preload here means what in SSD world is called drive
> > > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > > where it gives you the true performance in production over months of
> > > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > > >
> > > > > > > A sequential workload with larger blocks will be faster,
> > > > > >
> > > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > > populate FTL translation tables like random writes do.
> > > > > >
> > > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > > worth of random writes. At today speeds, that should take just a
> > > > > > couple of hours.
> > > > > >
> > > > >       When you say 2xcapacity worth of random writes, do you mean just
> > > > > setting size=200%?
> > > >
> > > > Right.
> > > >
> > >       Then I wonder what I am doing wrong now. I changed the config file to
> > >
> > > [root@testbox tests]# cat preload.conf
> > > [global]
> > > name=4k random write 4 ios in the queue in 32 queues
> > > ioengine=libaio
> > > direct=1
> > > bs=4k
> > > rw=randwrite
> > > iodepth=4
> > > numjobs=32
> > > buffered=0
> > > size=200%
> > > loops=2
> > > random_generator=tausworthe64
> > > thread=1
> > >
> > > [job1]
> > > filename=/dev/nvme0n1
> > > [root@testbox tests]#
> > >
> > > but when I run it, now it spits out much larger eta times:
> > >
> > > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > > 16580099d:14h:55m:27s]]
> >
> >  Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> > drive capacities here.
> >
> > Also, using 32 threads doesn't improve anything. 2 (and even one)
> > threads with qd=128 will push the drive
> > to its limits.
> >
>      Update: so I redid the config file a bit to pass some of the
> arguments from command line, and cut down number of jobs and loops.
> And I ran it again, this time sequential write to the drive I have not
> touched to see how fast it was going to go. My eta is still
> astronomical:
>
> [root@testbox tests]# cat preload_fio.conf
> [global]
> name=4k random
> ioengine=${ioengine}
> direct=1
> bs=${bs_size}
> rw=${iotype}
> iodepth=4
> numjobs=1
> buffered=0
> size=200%
> loops=1
>
> [job1]
> filename=${devicename}
> [root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
> iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
> job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
> 128KiB-128KiB, ioengine=libaio, iodepth=4
> fio-3.17-68-g3f1e
> Starting 1 process
> Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]
>
> > Regards,
> > Andrey
> > >
> > > Compare with what I was getting with size=100%
> > >
> > >  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> > >
> > > > Regards,
> > > > Andrey
> > > > >
> > > > > > Regards,
> > > > > > Andrey
> > > > > >
> > > > > > > like:
> > > > > > >
> > > > > > > [global]
> > > > > > > ioengine=libaio
> > > > > > > thread=1
> > > > > > > direct=1
> > > > > > > bs=128k
> > > > > > > rw=write
> > > > > > > numjobs=1
> > > > > > > iodepth=128
> > > > > > > size=100%
> > > > > > > loops=2
> > > > > > > [job00]
> > > > > > > filename=/dev/nvme0n1
> > > > > > >
> > > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > > >
> > > > > > > -Joe

Re: CPUs, threads, and speed

[Andrey Kuzmin]

On Thu, Jan 16, 2020 at 9:31 PM Jared Walton <jawalking@gmail.com> wrote:
>
> Not sure if this will help, but I use the following to prep multiple
> 4TB drives at the same time in a little over an hour.

You seem to be preconditioning with sequential writes only, and
further doing so
with essentially single write frontier.

That doesn't stress FTL maps enough and doesn't trigger any substantial garbage
collection since SSD is intelligent enough to spot sequential write
workload with
128K sequential (re)writes.

So what you're doing is only good for bandwidth measurements, and if
this steady
state is applied to random IOPS profiling, you'd be getting highly
inflated results.

Regards,
Andrey

> Is it inelegant, yes, but it works for me.
>
> globalFIOParameters="--offset=0 --ioengine=libaio --invalidate=1
> --group_reporting --direct=1 --thread --refill_buffers --norandommap
> --randrepeat=0 --allow_mounted_write=1 --output-format=json,normal"
>
> # Drives should be FOB or LLF'd (if it's good to do that)
> # LLF logic
>
> # 128k Pre-Condition
> # Write to entire disk
> for i in `ls -1 /dev/nvme*n1`
> do
>     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> print $1 }')
> ./fio --name=PreconditionPass1of3 --filename=${i} --iodepth=$iodepth
> --bs=128k --rw=write --size=${size} --fill_device=1
> $globalFIOParameters &
> done
> wait
>
> # Read entire disk
> for i in `ls -1 /dev/nvme*n1`
> do
>     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> print $1 }')
> ./fio --name=PreconditionPass2of3 --filename=${i} --iodepth=$iodepth
> --bs=128k --rw=read --size=${size} --fill_device=1
> $globalFIOParameters &
> done
> wait
>
> # Write to entire disk one last time
> for i in `ls -1 /dev/nvme*n1`
> do
>     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> print $1 }')
> ./fio --name=PreconditionPass3of3 --filename=${i} --iodepth=$iodepth
> --bs=128k --rw=write --size=${size} --fill_device=1
> $globalFIOParameters &
> done
> wait
>
>
> # Check 128k steady-state
> for i in `ls -1 /dev/nvme*n1`
> do
> ./fio --name=SteadyState --filename=${i} --iodepth=16 --numjobs=16
> --bs=4k --rw=read --ss_dur=1800 --ss=iops_slope:0.3% --runtime=24h
> $globalFIOParameters &
> done
> wait
>
> On Thu, Jan 16, 2020 at 9:13 AM Mauricio Tavares <raubvogel@gmail.com> wrote:
> >
> > On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > >
> > > On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > > > <joseph.r.gruher@intel.com> wrote:
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > > > > Mauricio Tavares
> > > > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > > > To: fio@vger.kernel.org
> > > > > > > > > Subject: CPUs, threads, and speed
> > > > > > > > >
> > > > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > > > _4KRandom_NVMe.ini)
> > > > > > > > >
> > > > > > > > > [global]
> > > > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > > ioengine=libaio
> > > > > > > > > direct=1
> > > > > > > > > bs=4k
> > > > > > > > > rw=randwrite
> > > > > > > > > iodepth=4
> > > > > > > > > numjobs=32
> > > > > > > > > buffered=0
> > > > > > > > > size=100%
> > > > > > > > > loops=2
> > > > > > > > > randrepeat=0
> > > > > > > > > norandommap
> > > > > > > > > refill_buffers
> > > > > > > > >
> > > > > > > > > [job1]
> > > > > > > > >
> > > > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > > > up?
> > > > > > > >
> > > > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > > > >
> > > > > > > I believe that preload here means what in SSD world is called drive
> > > > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > > > where it gives you the true performance in production over months of
> > > > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > > > >
> > > > > > > > A sequential workload with larger blocks will be faster,
> > > > > > >
> > > > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > > > populate FTL translation tables like random writes do.
> > > > > > >
> > > > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > > > worth of random writes. At today speeds, that should take just a
> > > > > > > couple of hours.
> > > > > > >
> > > > > >       When you say 2xcapacity worth of random writes, do you mean just
> > > > > > setting size=200%?
> > > > >
> > > > > Right.
> > > > >
> > > >       Then I wonder what I am doing wrong now. I changed the config file to
> > > >
> > > > [root@testbox tests]# cat preload.conf
> > > > [global]
> > > > name=4k random write 4 ios in the queue in 32 queues
> > > > ioengine=libaio
> > > > direct=1
> > > > bs=4k
> > > > rw=randwrite
> > > > iodepth=4
> > > > numjobs=32
> > > > buffered=0
> > > > size=200%
> > > > loops=2
> > > > random_generator=tausworthe64
> > > > thread=1
> > > >
> > > > [job1]
> > > > filename=/dev/nvme0n1
> > > > [root@testbox tests]#
> > > >
> > > > but when I run it, now it spits out much larger eta times:
> > > >
> > > > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > > > 16580099d:14h:55m:27s]]
> > >
> > >  Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> > > drive capacities here.
> > >
> > > Also, using 32 threads doesn't improve anything. 2 (and even one)
> > > threads with qd=128 will push the drive
> > > to its limits.
> > >
> >      Update: so I redid the config file a bit to pass some of the
> > arguments from command line, and cut down number of jobs and loops.
> > And I ran it again, this time sequential write to the drive I have not
> > touched to see how fast it was going to go. My eta is still
> > astronomical:
> >
> > [root@testbox tests]# cat preload_fio.conf
> > [global]
> > name=4k random
> > ioengine=${ioengine}
> > direct=1
> > bs=${bs_size}
> > rw=${iotype}
> > iodepth=4
> > numjobs=1
> > buffered=0
> > size=200%
> > loops=1
> >
> > [job1]
> > filename=${devicename}
> > [root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
> > iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
> > job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
> > 128KiB-128KiB, ioengine=libaio, iodepth=4
> > fio-3.17-68-g3f1e
> > Starting 1 process
> > Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]
> >
> > > Regards,
> > > Andrey
> > > >
> > > > Compare with what I was getting with size=100%
> > > >
> > > >  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> > > >
> > > > > Regards,
> > > > > Andrey
> > > > > >
> > > > > > > Regards,
> > > > > > > Andrey
> > > > > > >
> > > > > > > > like:
> > > > > > > >
> > > > > > > > [global]
> > > > > > > > ioengine=libaio
> > > > > > > > thread=1
> > > > > > > > direct=1
> > > > > > > > bs=128k
> > > > > > > > rw=write
> > > > > > > > numjobs=1
> > > > > > > > iodepth=128
> > > > > > > > size=100%
> > > > > > > > loops=2
> > > > > > > > [job00]
> > > > > > > > filename=/dev/nvme0n1
> > > > > > > >
> > > > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > > > >
> > > > > > > > -Joe

Re: CPUs, threads, and speed

[Jared Walton]

Correct, I pre-condition for IOPS testing by utilizing the the last if
block, only using randwrite. which will run random writes for about
45min, until a steady state is achieved.

On Thu, Jan 16, 2020 at 11:40 AM Andrey Kuzmin
<andrey.v.kuzmin@gmail.com> wrote:
>
> On Thu, Jan 16, 2020 at 9:31 PM Jared Walton <jawalking@gmail.com> wrote:
> >
> > Not sure if this will help, but I use the following to prep multiple
> > 4TB drives at the same time in a little over an hour.
>
> You seem to be preconditioning with sequential writes only, and
> further doing so
> with essentially single write frontier.
>
> That doesn't stress FTL maps enough and doesn't trigger any substantial garbage
> collection since SSD is intelligent enough to spot sequential write
> workload with
> 128K sequential (re)writes.
>
> So what you're doing is only good for bandwidth measurements, and if
> this steady
> state is applied to random IOPS profiling, you'd be getting highly
> inflated results.
>
> Regards,
> Andrey
>
> > Is it inelegant, yes, but it works for me.
> >
> > globalFIOParameters="--offset=0 --ioengine=libaio --invalidate=1
> > --group_reporting --direct=1 --thread --refill_buffers --norandommap
> > --randrepeat=0 --allow_mounted_write=1 --output-format=json,normal"
> >
> > # Drives should be FOB or LLF'd (if it's good to do that)
> > # LLF logic
> >
> > # 128k Pre-Condition
> > # Write to entire disk
> > for i in `ls -1 /dev/nvme*n1`
> > do
> >     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > print $1 }')
> > ./fio --name=PreconditionPass1of3 --filename=${i} --iodepth=$iodepth
> > --bs=128k --rw=write --size=${size} --fill_device=1
> > $globalFIOParameters &
> > done
> > wait
> >
> > # Read entire disk
> > for i in `ls -1 /dev/nvme*n1`
> > do
> >     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > print $1 }')
> > ./fio --name=PreconditionPass2of3 --filename=${i} --iodepth=$iodepth
> > --bs=128k --rw=read --size=${size} --fill_device=1
> > $globalFIOParameters &
> > done
> > wait
> >
> > # Write to entire disk one last time
> > for i in `ls -1 /dev/nvme*n1`
> > do
> >     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > print $1 }')
> > ./fio --name=PreconditionPass3of3 --filename=${i} --iodepth=$iodepth
> > --bs=128k --rw=write --size=${size} --fill_device=1
> > $globalFIOParameters &
> > done
> > wait
> >
> >
> > # Check 128k steady-state
> > for i in `ls -1 /dev/nvme*n1`
> > do
> > ./fio --name=SteadyState --filename=${i} --iodepth=16 --numjobs=16
> > --bs=4k --rw=read --ss_dur=1800 --ss=iops_slope:0.3% --runtime=24h
> > $globalFIOParameters &
> > done
> > wait
> >
> > On Thu, Jan 16, 2020 at 9:13 AM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > >
> > > On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > > > > <joseph.r.gruher@intel.com> wrote:
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > > > > > Mauricio Tavares
> > > > > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > > > > To: fio@vger.kernel.org
> > > > > > > > > > Subject: CPUs, threads, and speed
> > > > > > > > > >
> > > > > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > > > > _4KRandom_NVMe.ini)
> > > > > > > > > >
> > > > > > > > > > [global]
> > > > > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > > > ioengine=libaio
> > > > > > > > > > direct=1
> > > > > > > > > > bs=4k
> > > > > > > > > > rw=randwrite
> > > > > > > > > > iodepth=4
> > > > > > > > > > numjobs=32
> > > > > > > > > > buffered=0
> > > > > > > > > > size=100%
> > > > > > > > > > loops=2
> > > > > > > > > > randrepeat=0
> > > > > > > > > > norandommap
> > > > > > > > > > refill_buffers
> > > > > > > > > >
> > > > > > > > > > [job1]
> > > > > > > > > >
> > > > > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > > > > up?
> > > > > > > > >
> > > > > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > > > > >
> > > > > > > > I believe that preload here means what in SSD world is called drive
> > > > > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > > > > where it gives you the true performance in production over months of
> > > > > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > > > > >
> > > > > > > > > A sequential workload with larger blocks will be faster,
> > > > > > > >
> > > > > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > > > > populate FTL translation tables like random writes do.
> > > > > > > >
> > > > > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > > > > worth of random writes. At today speeds, that should take just a
> > > > > > > > couple of hours.
> > > > > > > >
> > > > > > >       When you say 2xcapacity worth of random writes, do you mean just
> > > > > > > setting size=200%?
> > > > > >
> > > > > > Right.
> > > > > >
> > > > >       Then I wonder what I am doing wrong now. I changed the config file to
> > > > >
> > > > > [root@testbox tests]# cat preload.conf
> > > > > [global]
> > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > ioengine=libaio
> > > > > direct=1
> > > > > bs=4k
> > > > > rw=randwrite
> > > > > iodepth=4
> > > > > numjobs=32
> > > > > buffered=0
> > > > > size=200%
> > > > > loops=2
> > > > > random_generator=tausworthe64
> > > > > thread=1
> > > > >
> > > > > [job1]
> > > > > filename=/dev/nvme0n1
> > > > > [root@testbox tests]#
> > > > >
> > > > > but when I run it, now it spits out much larger eta times:
> > > > >
> > > > > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > > > > 16580099d:14h:55m:27s]]
> > > >
> > > >  Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> > > > drive capacities here.
> > > >
> > > > Also, using 32 threads doesn't improve anything. 2 (and even one)
> > > > threads with qd=128 will push the drive
> > > > to its limits.
> > > >
> > >      Update: so I redid the config file a bit to pass some of the
> > > arguments from command line, and cut down number of jobs and loops.
> > > And I ran it again, this time sequential write to the drive I have not
> > > touched to see how fast it was going to go. My eta is still
> > > astronomical:
> > >
> > > [root@testbox tests]# cat preload_fio.conf
> > > [global]
> > > name=4k random
> > > ioengine=${ioengine}
> > > direct=1
> > > bs=${bs_size}
> > > rw=${iotype}
> > > iodepth=4
> > > numjobs=1
> > > buffered=0
> > > size=200%
> > > loops=1
> > >
> > > [job1]
> > > filename=${devicename}
> > > [root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
> > > iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
> > > job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
> > > 128KiB-128KiB, ioengine=libaio, iodepth=4
> > > fio-3.17-68-g3f1e
> > > Starting 1 process
> > > Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]
> > >
> > > > Regards,
> > > > Andrey
> > > > >
> > > > > Compare with what I was getting with size=100%
> > > > >
> > > > >  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> > > > >
> > > > > > Regards,
> > > > > > Andrey
> > > > > > >
> > > > > > > > Regards,
> > > > > > > > Andrey
> > > > > > > >
> > > > > > > > > like:
> > > > > > > > >
> > > > > > > > > [global]
> > > > > > > > > ioengine=libaio
> > > > > > > > > thread=1
> > > > > > > > > direct=1
> > > > > > > > > bs=128k
> > > > > > > > > rw=write
> > > > > > > > > numjobs=1
> > > > > > > > > iodepth=128
> > > > > > > > > size=100%
> > > > > > > > > loops=2
> > > > > > > > > [job00]
> > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > >
> > > > > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > > > > >
> > > > > > > > > -Joe

Re: CPUs, threads, and speed

[Matthew Eaton]

On Thu, Jan 16, 2020 at 11:04 AM Jared Walton <jawalking@gmail.com> wrote:
>
> Correct, I pre-condition for IOPS testing by utilizing the the last if
> block, only using randwrite. which will run random writes for about
> 45min, until a steady state is achieved.
>
> On Thu, Jan 16, 2020 at 11:40 AM Andrey Kuzmin
> <andrey.v.kuzmin@gmail.com> wrote:
> >
> > On Thu, Jan 16, 2020 at 9:31 PM Jared Walton <jawalking@gmail.com> wrote:
> > >
> > > Not sure if this will help, but I use the following to prep multiple
> > > 4TB drives at the same time in a little over an hour.
> >
> > You seem to be preconditioning with sequential writes only, and
> > further doing so
> > with essentially single write frontier.
> >
> > That doesn't stress FTL maps enough and doesn't trigger any substantial garbage
> > collection since SSD is intelligent enough to spot sequential write
> > workload with
> > 128K sequential (re)writes.
> >
> > So what you're doing is only good for bandwidth measurements, and if
> > this steady
> > state is applied to random IOPS profiling, you'd be getting highly
> > inflated results.
> >
> > Regards,
> > Andrey
> >
> > > Is it inelegant, yes, but it works for me.
> > >
> > > globalFIOParameters="--offset=0 --ioengine=libaio --invalidate=1
> > > --group_reporting --direct=1 --thread --refill_buffers --norandommap
> > > --randrepeat=0 --allow_mounted_write=1 --output-format=json,normal"
> > >
> > > # Drives should be FOB or LLF'd (if it's good to do that)
> > > # LLF logic
> > >
> > > # 128k Pre-Condition
> > > # Write to entire disk
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > >     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > > print $1 }')
> > > ./fio --name=PreconditionPass1of3 --filename=${i} --iodepth=$iodepth
> > > --bs=128k --rw=write --size=${size} --fill_device=1
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > > # Read entire disk
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > >     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > > print $1 }')
> > > ./fio --name=PreconditionPass2of3 --filename=${i} --iodepth=$iodepth
> > > --bs=128k --rw=read --size=${size} --fill_device=1
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > > # Write to entire disk one last time
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > >     size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > > print $1 }')
> > > ./fio --name=PreconditionPass3of3 --filename=${i} --iodepth=$iodepth
> > > --bs=128k --rw=write --size=${size} --fill_device=1
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > >
> > > # Check 128k steady-state
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > > ./fio --name=SteadyState --filename=${i} --iodepth=16 --numjobs=16
> > > --bs=4k --rw=read --ss_dur=1800 --ss=iops_slope:0.3% --runtime=24h
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > > On Thu, Jan 16, 2020 at 9:13 AM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > >
> > > > On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > > > > > <joseph.r.gruher@intel.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > > > > > > > > > Mauricio Tavares
> > > > > > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > > > > > To: fio@vger.kernel.org
> > > > > > > > > > > Subject: CPUs, threads, and speed
> > > > > > > > > > >
> > > > > > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > > > > > _4KRandom_NVMe.ini)
> > > > > > > > > > >
> > > > > > > > > > > [global]
> > > > > > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > > > > ioengine=libaio
> > > > > > > > > > > direct=1
> > > > > > > > > > > bs=4k
> > > > > > > > > > > rw=randwrite
> > > > > > > > > > > iodepth=4
> > > > > > > > > > > numjobs=32
> > > > > > > > > > > buffered=0
> > > > > > > > > > > size=100%
> > > > > > > > > > > loops=2
> > > > > > > > > > > randrepeat=0
> > > > > > > > > > > norandommap
> > > > > > > > > > > refill_buffers
> > > > > > > > > > >
> > > > > > > > > > > [job1]
> > > > > > > > > > >
> > > > > > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > > > > > up?
> > > > > > > > > >
> > > > > > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > > > > > >
> > > > > > > > > I believe that preload here means what in SSD world is called drive
> > > > > > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > > > > > where it gives you the true performance in production over months of
> > > > > > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > > > > > >
> > > > > > > > > > A sequential workload with larger blocks will be faster,
> > > > > > > > >
> > > > > > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > > > > > populate FTL translation tables like random writes do.
> > > > > > > > >
> > > > > > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > > > > > worth of random writes. At today speeds, that should take just a
> > > > > > > > > couple of hours.
> > > > > > > > >
> > > > > > > >       When you say 2xcapacity worth of random writes, do you mean just
> > > > > > > > setting size=200%?
> > > > > > >
> > > > > > > Right.
> > > > > > >
> > > > > >       Then I wonder what I am doing wrong now. I changed the config file to
> > > > > >
> > > > > > [root@testbox tests]# cat preload.conf
> > > > > > [global]
> > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > ioengine=libaio
> > > > > > direct=1
> > > > > > bs=4k
> > > > > > rw=randwrite
> > > > > > iodepth=4
> > > > > > numjobs=32
> > > > > > buffered=0
> > > > > > size=200%
> > > > > > loops=2
> > > > > > random_generator=tausworthe64
> > > > > > thread=1
> > > > > >
> > > > > > [job1]
> > > > > > filename=/dev/nvme0n1
> > > > > > [root@testbox tests]#
> > > > > >
> > > > > > but when I run it, now it spits out much larger eta times:
> > > > > >
> > > > > > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > > > > > 16580099d:14h:55m:27s]]
> > > > >
> > > > >  Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> > > > > drive capacities here.
> > > > >
> > > > > Also, using 32 threads doesn't improve anything. 2 (and even one)
> > > > > threads with qd=128 will push the drive
> > > > > to its limits.
> > > > >
> > > >      Update: so I redid the config file a bit to pass some of the
> > > > arguments from command line, and cut down number of jobs and loops.
> > > > And I ran it again, this time sequential write to the drive I have not
> > > > touched to see how fast it was going to go. My eta is still
> > > > astronomical:
> > > >
> > > > [root@testbox tests]# cat preload_fio.conf
> > > > [global]
> > > > name=4k random
> > > > ioengine=${ioengine}
> > > > direct=1
> > > > bs=${bs_size}
> > > > rw=${iotype}
> > > > iodepth=4
> > > > numjobs=1
> > > > buffered=0
> > > > size=200%
> > > > loops=1
> > > >
> > > > [job1]
> > > > filename=${devicename}
> > > > [root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
> > > > iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
> > > > job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
> > > > 128KiB-128KiB, ioengine=libaio, iodepth=4
> > > > fio-3.17-68-g3f1e
> > > > Starting 1 process
> > > > Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]
> > > >
> > > > > Regards,
> > > > > Andrey
> > > > > >
> > > > > > Compare with what I was getting with size=100%
> > > > > >
> > > > > >  Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> > > > > >
> > > > > > > Regards,
> > > > > > > Andrey
> > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > > > > > like:
> > > > > > > > > >
> > > > > > > > > > [global]
> > > > > > > > > > ioengine=libaio
> > > > > > > > > > thread=1
> > > > > > > > > > direct=1
> > > > > > > > > > bs=128k
> > > > > > > > > > rw=write
> > > > > > > > > > numjobs=1
> > > > > > > > > > iodepth=128
> > > > > > > > > > size=100%
> > > > > > > > > > loops=2
> > > > > > > > > > [job00]
> > > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > > >
> > > > > > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > > > > > >
> > > > > > > > > > -Joe

I have pretty much standardized on two sequential drive writes and
four random drive writes to get to steady state. It may be overkill
but has worked well for me since we test a variety of SSDs and some
reach steady state faster than others.

Re: CPUs, threads, and speed

[Mauricio Tavares]

On Fri, Jan 17, 2020 at 5:08 PM Matthew Eaton <m.eaton82@gmail.com> wrote:
>
> On Thu, Jan 16, 2020 at 11:04 AM Jared Walton <jawalking@gmail.com> wrote:
> >
> > Correct, I pre-condition for IOPS testing by utilizing the the last if
> > block, only using randwrite. which will run random writes for about
> > 45min, until a steady state is achieved.
> >
[...]
>
> I have pretty much standardized on two sequential drive writes and
> four random drive writes to get to steady state. It may be overkill
> but has worked well for me since we test a variety of SSDs and some
> reach steady state faster than others.

Thank you for all the replies and to Jared for the script, which leads
me to one related question: even though it used steady state as
criteria to consider drive preconditioned, am I correct to assume that
steady state does not necessarily have any relationship with the
steady state where I want to take the measurements when actually
running the tests?

Re: CPUs, threads, and speed

[Kudryavtsev, Andrey O]

I can clarify that as I posted the original script on github. 
Sequential preconditioning is mandatory for bandwidth test. Random 4k preconditioning is for everything else. For all mixed scenarios data has to be randomized, so, that puts highest pressure on the drive (and internally WAF in case of NAND SSD). That makes all following benchmarks fair.  
Norandommap - I agree in general, but the FIO overhead of tracking LBAs impacts the performance and extends pre-fill time. 

-- 
Andrey Kudryavtsev, 
SSD Solution Architect
Intel Corp. 


On 1/15/20, 9:29 AM, "fio-owner@vger.kernel.org on behalf of Gruher, Joseph R" <fio-owner@vger.kernel.org on behalf of joseph.r.gruher@intel.com> wrote:

    > -----Original Message-----
    > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
    > Mauricio Tavares
    > Sent: Wednesday, January 15, 2020 7:51 AM
    > To: fio@vger.kernel.org
    > Subject: CPUs, threads, and speed
    > 
    > Let's say I have a config file to preload drive that looks like this (stolen from
    > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
    > _4KRandom_NVMe.ini)
    > 
    > [global]
    > name=4k random write 4 ios in the queue in 32 queues
    > filename=/dev/nvme0n1
    > ioengine=libaio
    > direct=1
    > bs=4k
    > rw=randwrite
    > iodepth=4
    > numjobs=32
    > buffered=0
    > size=100%
    > loops=2
    > randrepeat=0
    > norandommap
    > refill_buffers
    > 
    > [job1]
    > 
    > That is taking a ton of time, like days to go. Is there anything I can do to speed it
    > up? 
    
    When you say preload, do you just want to write in the full capacity of the drive?  A sequential workload with larger blocks will be faster, like:
    
    [global]
    ioengine=libaio
    thread=1
    direct=1
    bs=128k
    rw=write
    numjobs=1
    iodepth=128
    size=100%
    loops=2
    [job00]
    filename=/dev/nvme0n1
    
    Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
    
    -Joe
    

RE: CPUs, threads, and speed

[Elliott, Robert (Servers)]

> -----Original Message-----
> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> Mauricio Tavares
> Sent: Wednesday, January 15, 2020 9:51 AM
> Subject: CPUs, threads, and speed
> 
...
> [global]
> name=4k random write 4 ios in the queue in 32 queues
> filename=/dev/nvme0n1
> ioengine=libaio
> direct=1
> bs=4k
> rw=randwrite
> iodepth=4
> numjobs=32
> buffered=0
> size=100%
> loops=2
> randrepeat=0
> norandommap
> refill_buffers
> 
> [job1]
> 
> That is taking a ton of time, like days to go. Is there anything I can
> do to speed it up? For instance, what is the default value for
> cpus_allowed (or cpumask)[2]? Is it all CPUs? If not what would I gain
> by throwing more cpus at the problem?
> 
> I also read[2] by default fio uses fork. What would I get by going to
> threads?

> Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]

77 kIOPs for random writes isn't bad - check your drive data sheet.
If the drive is 1 TB, it should take 
    1 TB / (77k * 4 KiB) = 3170 s = 52.8 minutes 
to write the whole drive.

Best practice is to use all CPU cores, lock threads to cores, and
be NUMA aware. If the device is attached to physical CPU 0 and that CPU
has 12 cores known to linux as 0-11 (per "lscpu" or "numactl --hardware"),
try:
  iodepth=16
  numjobs=12
  cpus_allowed=0-11
  cpus_allowed_policy=split

Based on these:
  numjobs=32, size=100%, loops=2
fio will run each job for that many bytes, so a 1 TB drive will result 
in IOs for 64 TB rather than 1 TB. That could easily result in the
multi-day estimate.

Other nits:
* thread - threading might be slightly more efficient than 
  spawning full processes
* gtod_reduce=1 - precision latency measurements don't matter for this
* refill_buffers - presuming you don't care about the data contents,
  don't include this. zero_buffers is the simplest/fastest, unless you're
  concerned that the device might do compression or zero detection
* norandommap - if you want it to hit each LBA a precise number
  of times, you can't include this; fio won't remember what it's 
  done. There is a lot of overhead in keeping track, though.

Re: CPUs, threads, and speed

[Mauricio Tavares]

On Wed, Jan 15, 2020 at 4:33 PM Elliott, Robert (Servers)
<elliott@hpe.com> wrote:
>
>
>
> > -----Original Message-----
> > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > Mauricio Tavares
> > Sent: Wednesday, January 15, 2020 9:51 AM
> > Subject: CPUs, threads, and speed
> >
> ...
> > [global]
> > name=4k random write 4 ios in the queue in 32 queues
> > filename=/dev/nvme0n1
> > ioengine=libaio
> > direct=1
> > bs=4k
> > rw=randwrite
> > iodepth=4
> > numjobs=32
> > buffered=0
> > size=100%
> > loops=2
> > randrepeat=0
> > norandommap
> > refill_buffers
> >
> > [job1]
> >
> > That is taking a ton of time, like days to go. Is there anything I can
> > do to speed it up? For instance, what is the default value for
> > cpus_allowed (or cpumask)[2]? Is it all CPUs? If not what would I gain
> > by throwing more cpus at the problem?
> >
> > I also read[2] by default fio uses fork. What would I get by going to
> > threads?
>
> > Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
>
> 77 kIOPs for random writes isn't bad - check your drive data sheet.
> If the drive is 1 TB, it should take
>     1 TB / (77k * 4 KiB) = 3170 s = 52.8 minutes
> to write the whole drive.
>
      Since the drive is 4TB, we are talking about 3.5h to complete
the task, right?

> Best practice is to use all CPU cores, lock threads to cores, and
> be NUMA aware. If the device is attached to physical CPU 0 and that CPU
> has 12 cores known to linux as 0-11 (per "lscpu" or "numactl --hardware"),

I have two CPUs with 16 cores each; I thought that meant numjobs=32.
If Iw as wrong, I learned something new!

> try:
>   iodepth=16
>   numjobs=12
>   cpus_allowed=0-11
>   cpus_allowed_policy=split
>
> Based on these:
>   numjobs=32, size=100%, loops=2
> fio will run each job for that many bytes, so a 1 TB drive will result
> in IOs for 64 TB rather than 1 TB. That could easily result in the
> multi-day estimate.
>
      Let's see if I understand this: your 64TB number came from 32*1TB*1*2?

> Other nits:
> * thread - threading might be slightly more efficient than
>   spawning full processes
> * gtod_reduce=1 - precision latency measurements don't matter for this
> * refill_buffers - presuming you don't care about the data contents,
>   don't include this. zero_buffers is the simplest/fastest, unless you're
>   concerned that the device might do compression or zero detection
> * norandommap - if you want it to hit each LBA a precise number
>   of times, you can't include this; fio won't remember what it's
>   done. There is a lot of overhead in keeping track, though.
>

Re: CPUs, threads, and speed

[Ming Lei]

On Thu, Jan 16, 2020 at 8:15 AM Mauricio Tavares <raubvogel@gmail.com> wrote:
>
> On Wed, Jan 15, 2020 at 4:33 PM Elliott, Robert (Servers)
> <elliott@hpe.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> > > Mauricio Tavares
> > > Sent: Wednesday, January 15, 2020 9:51 AM
> > > Subject: CPUs, threads, and speed
> > >
> > ...
> > > [global]
> > > name=4k random write 4 ios in the queue in 32 queues
> > > filename=/dev/nvme0n1
> > > ioengine=libaio
> > > direct=1
> > > bs=4k
> > > rw=randwrite
> > > iodepth=4
> > > numjobs=32
> > > buffered=0
> > > size=100%
> > > loops=2
> > > randrepeat=0
> > > norandommap
> > > refill_buffers
> > >
> > > [job1]
> > >
> > > That is taking a ton of time, like days to go. Is there anything I can
> > > do to speed it up? For instance, what is the default value for
> > > cpus_allowed (or cpumask)[2]? Is it all CPUs? If not what would I gain
> > > by throwing more cpus at the problem?
> > >
> > > I also read[2] by default fio uses fork. What would I get by going to
> > > threads?
> >
> > > Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> >
> > 77 kIOPs for random writes isn't bad - check your drive data sheet.
> > If the drive is 1 TB, it should take
> >     1 TB / (77k * 4 KiB) = 3170 s = 52.8 minutes
> > to write the whole drive.
> >
>       Since the drive is 4TB, we are talking about 3.5h to complete
> the task, right?
>
> > Best practice is to use all CPU cores, lock threads to cores, and
> > be NUMA aware. If the device is attached to physical CPU 0 and that CPU
> > has 12 cores known to linux as 0-11 (per "lscpu" or "numactl --hardware"),
>
> I have two CPUs with 16 cores each; I thought that meant numjobs=32.
> If Iw as wrong, I learned something new!

Not sure 32 jobs is good, which will write the drive 32 times, and each job
runs the random write on the whole drive. Does that preconditioning need to
run random write over the drive so many times?

Thanks,
Ming Lei