Scalability issue with multiple NVMe Devices with one core

Scalability issue with multiple NVMe Devices with one core

[Roy Shterman]

Hi,

scenario is when running traffic over 1 NVMe Device with 1 core I'm 
getting X IOPS and Y% core utilization.

In my perception, when adding more NVMe Devices I should see some 
linearity of the above results, but I'm getting only a small improvement 
in IOPS and still not getting 100% (or closer ) in CPU utilization.

Any suggestions?

Thanks,

Roy

Scalability issue with multiple NVMe Devices with one core

[Roy Shterman]

On 10/13/2016 5:18 PM, Keith Busch wrote:
> On Thu, Oct 13, 2016@08:44:37AM +0300, Roy Shterman wrote:
>> scenario is when running traffic over 1 NVMe Device with 1 core I'm getting
>> X IOPS and Y% core utilization.
>>
>> In my perception, when adding more NVMe Devices I should see some linearity
>> of the above results, but I'm getting only a small improvement in IOPS and
>> still not getting 100% (or closer ) in CPU utilization.
>>
>> Any suggestions?
> How are you generating IO?

fio --group_reporting --rw=randread --bs=4k --numjobs=1 --ramp_time=30 
--iodepth=1 --runtime=300 --direct=1 --time_based --loops=1 
--ioengine=libaio --invalidate=1 --randrepeat=1 --norandommap --exitall 
--name task_nvme0n1 --filename=/dev/nvme0n1

Scalability issue with multiple NVMe Devices with one core

[Roy Shterman]

On 10/13/2016 5:18 PM, Keith Busch wrote:
> On Thu, Oct 13, 2016@08:44:37AM +0300, Roy Shterman wrote:
>> scenario is when running traffic over 1 NVMe Device with 1 core I'm getting
>> X IOPS and Y% core utilization.
>>
>> In my perception, when adding more NVMe Devices I should see some linearity
>> of the above results, but I'm getting only a small improvement in IOPS and
>> still not getting 100% (or closer ) in CPU utilization.
>>
>> Any suggestions?
> How are you generating IO?
Sorry, last response I sent was with IODEPTH=1, I'm changing the IODEPTH 
to 8-32.

Scalability issue with multiple NVMe Devices with one core

[Keith Busch]

On Thu, Oct 13, 2016@08:44:37AM +0300, Roy Shterman wrote:
> scenario is when running traffic over 1 NVMe Device with 1 core I'm getting
> X IOPS and Y% core utilization.
> 
> In my perception, when adding more NVMe Devices I should see some linearity
> of the above results, but I'm getting only a small improvement in IOPS and
> still not getting 100% (or closer ) in CPU utilization.
> 
> Any suggestions?

How are you generating IO?

Scalability issue with multiple NVMe Devices with one core

[Keith Busch]

On Thu, Oct 13, 2016@05:11:58PM +0300, Roy Shterman wrote:
> On 10/13/2016 5:18 PM, Keith Busch wrote:
> > On Thu, Oct 13, 2016@08:44:37AM +0300, Roy Shterman wrote:
> > > scenario is when running traffic over 1 NVMe Device with 1 core I'm getting
> > > X IOPS and Y% core utilization.
> > > 
> > > In my perception, when adding more NVMe Devices I should see some linearity
> > > of the above results, but I'm getting only a small improvement in IOPS and
> > > still not getting 100% (or closer ) in CPU utilization.
> > > 
> > > Any suggestions?
> > How are you generating IO?
> 
> fio --group_reporting --rw=randread --bs=4k --numjobs=1 --ramp_time=30
> --iodepth=1 --runtime=300 --direct=1 --time_based --loops=1
> --ioengine=libaio --invalidate=1 --randrepeat=1 --norandommap --exitall
> --name task_nvme0n1 --filename=/dev/nvme0n1

And if you append "--name task_nvme1n1 --filename=/dev/nvme1n1" to this
command, you are not observing a meaningful IOPS improvement?

Scalability issue with multiple NVMe Devices with one core

[Keith Busch]

On Thu, Oct 13, 2016@02:48:34PM +0000, Roy Shterman wrote:
>> And if you append "--name task_nvme1n1 --filename=/dev/nvme1n1" to this
>> command, you are not observing a
>> meaningful IOPS improvement?
> 
> I appended only --filename=/dev/nvme0n1 without the new name, I'm not sure if it makes any difference but I can check.

If you don't add --name, fio isn't going to start another job, so you're
not accessing the devices in parallel and shouldn't expect an IOPS
improvement.