* Re: [SPDK] SPDK performance over 4*10G nic cards
@ 2018-03-28 6:36 Lego Lin
0 siblings, 0 replies; 6+ messages in thread
From: Lego Lin @ 2018-03-28 6:36 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3302 bytes --]
Hi, Vishal
Thanks for quickly response. I will try your suggestion.
Yes, per-iSCSI LUN has 1 FIO job with 1024 qdepth.
We reserve vcore 1, 2, 3 of socket 0 for handling nic card IRQ and RPS.
I will check whether nic card connect to CPU 0.
Thanks
On Wed, Mar 28, 2018 at 2:21 PM, Verma, Vishal4 <vishal4.verma(a)intel.com>
wrote:
> Hello Lin,
>
>
>
> Thanks for detailed configuration regarding your setup. When you say FIO
> queue depth = 1024, is that per iSCSI LUN? Have you tried running with
> numjobs=4 (more jobs per device) and iodepth=32 per job, ensuring ~128 per
> LUN.
>
> If it is 1024 iodepth per LUN then it maybe too much I/O request
> submission at a given time and in turn could be causing I/O requests queue
> for some time before being serviced by the target.
>
>
>
> Also, are you making sure to run SPDK iSCSI worked cores (using reactor
> mask) on the same socket where the 10G NIC is plugged in?
>
>
>
> Thanks,
>
> Vishal
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Lego Lin
> *Sent:* Tuesday, March 27, 2018 11:05 PM
> *To:* spdk(a)lists.01.org
> *Subject:* [SPDK] SPDK performance over 4*10G nic cards
>
>
>
> Hi @all:
>
>
>
> Our team has setup a platform to measure SPDK performance. After an
> evaluation, we found that SPDK read performance is not as good as write.
> Following are the platform information
>
>
>
> Server: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
>
> DRAM: 128G
>
> nic card: 2*10G on board + 4*10G cards
>
> NVMe SSD: Intel P4501 * 10
>
> Client: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
>
> DRAM: 128G
>
> nic card: 2*10G on board + 4*10G cards
>
> NVMe SSD: Intel P4501 * 10
>
> Software version: SPDK version 17.10
>
> Network setting:
>
> Server IP: 10.10.63.25, 10G ipaddr: 192.168.100 ~ 103.25
>
> Client IP: 10.10.63.61, 10G ipaddr: 192.168.100 ~ 103.61
>
> MTU 9000 for jumbo frames.
>
>
>
> We test following backend devices
>
> 1. SPDK null bdev * 16
>
> 2. SPDK malloc bdev * 16
>
> 3. Intel P4501 bdev * 8 (We divide each P4501 into 2 partitions for get
> enough bdev).
>
>
>
> We will generate 16 target devices and use fio with 16 jobs, each job R/W
> 1 target devices.
>
>
>
> FIO: qdepth = 1024
>
> iSCSI initiator block device nr_requests = 1024, nomerge, noop scheduler
>
> SPDK iSCSI target qdepth = 256
>
>
>
> following is the performance data:
>
> Null backend
>
> Write: 1169 KIOPS
>
> Read: 1008 KIOPS
>
>
>
> Malloc backend
>
> Write: 1168 KIOPS
>
> Read: 975 KIOPS
>
>
>
> NVMe backend
>
> Write: 1069 KIOPS
>
> Read: 799 KIOPS
>
>
>
> Related configuration as following:
>
> ReactorMask 0x3FF003F0 (16 vcores)
>
> MinConnectionsPerCore 1
>
>
>
> Can someone help that whether any other parameters for trying so that
> overall read performance can hit 1M.
>
>
>
> I already try increase HUGHMEM and different MinConnectionIdleInterval
> and it won't works.
>
>
>
> thanks
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9737 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [SPDK] SPDK performance over 4*10G nic cards
@ 2018-03-30 1:25 Yang, Ziye
0 siblings, 0 replies; 6+ messages in thread
From: Yang, Ziye @ 2018-03-30 1:25 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5261 bytes --]
Hi Lego,
Thanks for your update. Maybe, we can open this configuration to users.
Best Regards
Ziye Yang
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lego Lin
Sent: Friday, March 30, 2018 9:21 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK performance over 4*10G nic cards
Hi, Yang and Vishal
Thanks for providing comment about performance issue. Now we can gain 1M read performance with Null / Malloc / NVMe SSD (Intel P4501) over 4*10G nic cards.
The key point to improve performance is enlarge SPDK internal resources pool that suggested by Yang.
Following is my current parameters:
#define SPDK_BDEV_IO_POOL_SIZE (64 * 1024* 16)
#define SPDK_BDEV_IO_CACHE_SIZE 512
#define BUF_SMALL_POOL_SIZE 1024*16
#define BUF_LARGE_POOL_SIZE 1024*2
Current performance is
Null backend * 16 devs : Read: 1180 (FIO QD=64), Random Read: 1182 (FIO QD=64)
Malloc backend * 16 devs : Read : 1191 (FIO QD=64), Random Read: 1191 (FIO QD=64)
NVMe backend * 16 devs (Intel P4501 * 8) : Read : 1171 (FIO QD = 64), Random Read: 1182 (FIO QD=64)
Other parameters for reference
SPDK Reactor mask: 0x3FF003F0
HughMem = 16G
iSCSI MinConnectionsPerCore 1
iSCSI MaxQueueDepth 256
SCSI device qdepth 256
initator side parameters:
disk scheduler: noop
nomerges: 2
nr_requests: 128
queue_depth 128
Thanks
On Wed, Mar 28, 2018 at 2:37 PM, Yang, Ziye <ziye.yang(a)intel.com<mailto:ziye.yang(a)intel.com>> wrote:
Hi,
In SPDK bdev, I have the following parameters to configure the buf: (located in lib/bdev/bdev.c)
#define BUF_SMALL_POOL_SIZE 8192
#define BUF_LARGE_POOL_SIZE 1024
For NULL and Malloc, you will not need the buffer for read (zero copy for malloc). But for NVMe devices, for read, you need to pre-allocate the buffer from the pool for each thread, however the total number of buffer is limited. Though each thread has its own related channel, and there will be the buffer list. But definitely, there is contention for the buffer, and this will decrease the buffer. The easiest way is to enlarge those value to see whether there are some performance improvements.
Best Regards
Ziye Yang
From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Verma, Vishal4
Sent: Wednesday, March 28, 2018 2:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK performance over 4*10G nic cards
Hello Lin,
Thanks for detailed configuration regarding your setup. When you say FIO queue depth = 1024, is that per iSCSI LUN? Have you tried running with numjobs=4 (more jobs per device) and iodepth=32 per job, ensuring ~128 per LUN.
If it is 1024 iodepth per LUN then it maybe too much I/O request submission at a given time and in turn could be causing I/O requests queue for some time before being serviced by the target.
Also, are you making sure to run SPDK iSCSI worked cores (using reactor mask) on the same socket where the 10G NIC is plugged in?
Thanks,
Vishal
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lego Lin
Sent: Tuesday, March 27, 2018 11:05 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK performance over 4*10G nic cards
Hi @all:
Our team has setup a platform to measure SPDK performance. After an evaluation, we found that SPDK read performance is not as good as write. Following are the platform information
Server: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Client: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Software version: SPDK version 17.10
Network setting:
Server IP: 10.10.63.25, 10G ipaddr: 192.168.100 ~ 103.25
Client IP: 10.10.63.61, 10G ipaddr: 192.168.100 ~ 103.61
MTU 9000 for jumbo frames.
We test following backend devices
1. SPDK null bdev * 16
2. SPDK malloc bdev * 16
3. Intel P4501 bdev * 8 (We divide each P4501 into 2 partitions for get enough bdev).
We will generate 16 target devices and use fio with 16 jobs, each job R/W 1 target devices.
FIO: qdepth = 1024
iSCSI initiator block device nr_requests = 1024, nomerge, noop scheduler
SPDK iSCSI target qdepth = 256
following is the performance data:
Null backend
Write: 1169 KIOPS
Read: 1008 KIOPS
Malloc backend
Write: 1168 KIOPS
Read: 975 KIOPS
NVMe backend
Write: 1069 KIOPS
Read: 799 KIOPS
Related configuration as following:
ReactorMask 0x3FF003F0 (16 vcores)
MinConnectionsPerCore 1
Can someone help that whether any other parameters for trying so that overall read performance can hit 1M.
I already try increase HUGHMEM and different MinConnectionIdleInterval and it won't works.
thanks
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 21911 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [SPDK] SPDK performance over 4*10G nic cards
@ 2018-03-30 1:21 Lego Lin
0 siblings, 0 replies; 6+ messages in thread
From: Lego Lin @ 2018-03-30 1:21 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5230 bytes --]
Hi, Yang and Vishal
Thanks for providing comment about performance issue. Now we can gain 1M
read performance with Null / Malloc / NVMe SSD (Intel P4501) over 4*10G nic
cards.
The key point to improve performance is enlarge SPDK internal resources
pool that suggested by Yang.
Following is my current parameters:
#define SPDK_BDEV_IO_POOL_SIZE (64 * 1024* 16)
#define SPDK_BDEV_IO_CACHE_SIZE 512
#define BUF_SMALL_POOL_SIZE 1024*16
#define BUF_LARGE_POOL_SIZE 1024*2
Current performance is
Null backend * 16 devs : Read: 1180 (FIO QD=64), Random Read: 1182 (FIO
QD=64)
Malloc backend * 16 devs : Read : 1191 (FIO QD=64), Random Read: 1191 (FIO
QD=64)
NVMe backend * 16 devs (Intel P4501 * 8) : Read : 1171 (FIO QD = 64),
Random Read: 1182 (FIO QD=64)
Other parameters for reference
SPDK Reactor mask: 0x3FF003F0
HughMem = 16G
iSCSI MinConnectionsPerCore 1
iSCSI MaxQueueDepth 256
SCSI device qdepth 256
initator side parameters:
disk scheduler: noop
nomerges: 2
nr_requests: 128
queue_depth 128
Thanks
On Wed, Mar 28, 2018 at 2:37 PM, Yang, Ziye <ziye.yang(a)intel.com> wrote:
> Hi,
>
>
>
> In SPDK bdev, I have the following parameters to configure the buf:
> (located in lib/bdev/bdev.c)
>
>
>
> #define BUF_SMALL_POOL_SIZE 8192
>
> #define BUF_LARGE_POOL_SIZE 1024
>
>
>
> For NULL and Malloc, you will not need the buffer for read (zero copy for
> malloc). But for NVMe devices, for read, you need to pre-allocate the
> buffer from the pool for each thread, however the total number of buffer is
> limited. Though each thread has its own related channel, and there will be
> the buffer list. But definitely, there is contention for the buffer, and
> this will decrease the buffer. The easiest way is to enlarge those value to
> see whether there are some performance improvements.
>
>
>
> Best Regards
>
> Ziye Yang
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Verma,
> Vishal4
> *Sent:* Wednesday, March 28, 2018 2:22 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] SPDK performance over 4*10G nic cards
>
>
>
> Hello Lin,
>
>
>
> Thanks for detailed configuration regarding your setup. When you say FIO
> queue depth = 1024, is that per iSCSI LUN? Have you tried running with
> numjobs=4 (more jobs per device) and iodepth=32 per job, ensuring ~128 per
> LUN.
>
> If it is 1024 iodepth per LUN then it maybe too much I/O request
> submission at a given time and in turn could be causing I/O requests queue
> for some time before being serviced by the target.
>
>
>
> Also, are you making sure to run SPDK iSCSI worked cores (using reactor
> mask) on the same socket where the 10G NIC is plugged in?
>
>
>
> Thanks,
>
> Vishal
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>]
> *On Behalf Of *Lego Lin
> *Sent:* Tuesday, March 27, 2018 11:05 PM
> *To:* spdk(a)lists.01.org
> *Subject:* [SPDK] SPDK performance over 4*10G nic cards
>
>
>
> Hi @all:
>
>
>
> Our team has setup a platform to measure SPDK performance. After an
> evaluation, we found that SPDK read performance is not as good as write.
> Following are the platform information
>
>
>
> Server: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
>
> DRAM: 128G
>
> nic card: 2*10G on board + 4*10G cards
>
> NVMe SSD: Intel P4501 * 10
>
> Client: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
>
> DRAM: 128G
>
> nic card: 2*10G on board + 4*10G cards
>
> NVMe SSD: Intel P4501 * 10
>
> Software version: SPDK version 17.10
>
> Network setting:
>
> Server IP: 10.10.63.25, 10G ipaddr: 192.168.100 ~ 103.25
>
> Client IP: 10.10.63.61, 10G ipaddr: 192.168.100 ~ 103.61
>
> MTU 9000 for jumbo frames.
>
>
>
> We test following backend devices
>
> 1. SPDK null bdev * 16
>
> 2. SPDK malloc bdev * 16
>
> 3. Intel P4501 bdev * 8 (We divide each P4501 into 2 partitions for get
> enough bdev).
>
>
>
> We will generate 16 target devices and use fio with 16 jobs, each job R/W
> 1 target devices.
>
>
>
> FIO: qdepth = 1024
>
> iSCSI initiator block device nr_requests = 1024, nomerge, noop scheduler
>
> SPDK iSCSI target qdepth = 256
>
>
>
> following is the performance data:
>
> Null backend
>
> Write: 1169 KIOPS
>
> Read: 1008 KIOPS
>
>
>
> Malloc backend
>
> Write: 1168 KIOPS
>
> Read: 975 KIOPS
>
>
>
> NVMe backend
>
> Write: 1069 KIOPS
>
> Read: 799 KIOPS
>
>
>
> Related configuration as following:
>
> ReactorMask 0x3FF003F0 (16 vcores)
>
> MinConnectionsPerCore 1
>
>
>
> Can someone help that whether any other parameters for trying so that
> overall read performance can hit 1M.
>
>
>
> I already try increase HUGHMEM and different MinConnectionIdleInterval
> and it won't works.
>
>
>
> thanks
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13810 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [SPDK] SPDK performance over 4*10G nic cards
@ 2018-03-28 6:37 Yang, Ziye
0 siblings, 0 replies; 6+ messages in thread
From: Yang, Ziye @ 2018-03-28 6:37 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3520 bytes --]
Hi,
In SPDK bdev, I have the following parameters to configure the buf: (located in lib/bdev/bdev.c)
#define BUF_SMALL_POOL_SIZE 8192
#define BUF_LARGE_POOL_SIZE 1024
For NULL and Malloc, you will not need the buffer for read (zero copy for malloc). But for NVMe devices, for read, you need to pre-allocate the buffer from the pool for each thread, however the total number of buffer is limited. Though each thread has its own related channel, and there will be the buffer list. But definitely, there is contention for the buffer, and this will decrease the buffer. The easiest way is to enlarge those value to see whether there are some performance improvements.
Best Regards
Ziye Yang
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Verma, Vishal4
Sent: Wednesday, March 28, 2018 2:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK performance over 4*10G nic cards
Hello Lin,
Thanks for detailed configuration regarding your setup. When you say FIO queue depth = 1024, is that per iSCSI LUN? Have you tried running with numjobs=4 (more jobs per device) and iodepth=32 per job, ensuring ~128 per LUN.
If it is 1024 iodepth per LUN then it maybe too much I/O request submission at a given time and in turn could be causing I/O requests queue for some time before being serviced by the target.
Also, are you making sure to run SPDK iSCSI worked cores (using reactor mask) on the same socket where the 10G NIC is plugged in?
Thanks,
Vishal
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lego Lin
Sent: Tuesday, March 27, 2018 11:05 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK performance over 4*10G nic cards
Hi @all:
Our team has setup a platform to measure SPDK performance. After an evaluation, we found that SPDK read performance is not as good as write. Following are the platform information
Server: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Client: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Software version: SPDK version 17.10
Network setting:
Server IP: 10.10.63.25, 10G ipaddr: 192.168.100 ~ 103.25
Client IP: 10.10.63.61, 10G ipaddr: 192.168.100 ~ 103.61
MTU 9000 for jumbo frames.
We test following backend devices
1. SPDK null bdev * 16
2. SPDK malloc bdev * 16
3. Intel P4501 bdev * 8 (We divide each P4501 into 2 partitions for get enough bdev).
We will generate 16 target devices and use fio with 16 jobs, each job R/W 1 target devices.
FIO: qdepth = 1024
iSCSI initiator block device nr_requests = 1024, nomerge, noop scheduler
SPDK iSCSI target qdepth = 256
following is the performance data:
Null backend
Write: 1169 KIOPS
Read: 1008 KIOPS
Malloc backend
Write: 1168 KIOPS
Read: 975 KIOPS
NVMe backend
Write: 1069 KIOPS
Read: 799 KIOPS
Related configuration as following:
ReactorMask 0x3FF003F0 (16 vcores)
MinConnectionsPerCore 1
Can someone help that whether any other parameters for trying so that overall read performance can hit 1M.
I already try increase HUGHMEM and different MinConnectionIdleInterval and it won't works.
thanks
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 12236 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [SPDK] SPDK performance over 4*10G nic cards
@ 2018-03-28 6:21 Verma, Vishal4
0 siblings, 0 replies; 6+ messages in thread
From: Verma, Vishal4 @ 2018-03-28 6:21 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2479 bytes --]
Hello Lin,
Thanks for detailed configuration regarding your setup. When you say FIO queue depth = 1024, is that per iSCSI LUN? Have you tried running with numjobs=4 (more jobs per device) and iodepth=32 per job, ensuring ~128 per LUN.
If it is 1024 iodepth per LUN then it maybe too much I/O request submission at a given time and in turn could be causing I/O requests queue for some time before being serviced by the target.
Also, are you making sure to run SPDK iSCSI worked cores (using reactor mask) on the same socket where the 10G NIC is plugged in?
Thanks,
Vishal
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lego Lin
Sent: Tuesday, March 27, 2018 11:05 PM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK performance over 4*10G nic cards
Hi @all:
Our team has setup a platform to measure SPDK performance. After an evaluation, we found that SPDK read performance is not as good as write. Following are the platform information
Server: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Client: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Software version: SPDK version 17.10
Network setting:
Server IP: 10.10.63.25, 10G ipaddr: 192.168.100 ~ 103.25
Client IP: 10.10.63.61, 10G ipaddr: 192.168.100 ~ 103.61
MTU 9000 for jumbo frames.
We test following backend devices
1. SPDK null bdev * 16
2. SPDK malloc bdev * 16
3. Intel P4501 bdev * 8 (We divide each P4501 into 2 partitions for get enough bdev).
We will generate 16 target devices and use fio with 16 jobs, each job R/W 1 target devices.
FIO: qdepth = 1024
iSCSI initiator block device nr_requests = 1024, nomerge, noop scheduler
SPDK iSCSI target qdepth = 256
following is the performance data:
Null backend
Write: 1169 KIOPS
Read: 1008 KIOPS
Malloc backend
Write: 1168 KIOPS
Read: 975 KIOPS
NVMe backend
Write: 1069 KIOPS
Read: 799 KIOPS
Related configuration as following:
ReactorMask 0x3FF003F0 (16 vcores)
MinConnectionsPerCore 1
Can someone help that whether any other parameters for trying so that overall read performance can hit 1M.
I already try increase HUGHMEM and different MinConnectionIdleInterval and it won't works.
thanks
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8557 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* [SPDK] SPDK performance over 4*10G nic cards
@ 2018-03-28 6:04 Lego Lin
0 siblings, 0 replies; 6+ messages in thread
From: Lego Lin @ 2018-03-28 6:04 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 1705 bytes --]
Hi @all:
Our team has setup a platform to measure SPDK performance. After an
evaluation, we found that SPDK read performance is not as good as write.
Following are the platform information
Server: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Client: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Software version: SPDK version 17.10
Network setting:
Server IP: 10.10.63.25, 10G ipaddr: 192.168.100 ~ 103.25
Client IP: 10.10.63.61, 10G ipaddr: 192.168.100 ~ 103.61
MTU 9000 for jumbo frames.
We test following backend devices
1. SPDK null bdev * 16
2. SPDK malloc bdev * 16
3. Intel P4501 bdev * 8 (We divide each P4501 into 2 partitions for get
enough bdev).
We will generate 16 target devices and use fio with 16 jobs, each job R/W 1
target devices.
FIO: qdepth = 1024
iSCSI initiator block device nr_requests = 1024, nomerge, noop scheduler
SPDK iSCSI target qdepth = 256
following is the performance data:
Null backend
Write: 1169 KIOPS
Read: 1008 KIOPS
Malloc backend
Write: 1168 KIOPS
Read: 975 KIOPS
NVMe backend
Write: 1069 KIOPS
Read: 799 KIOPS
Related configuration as following:
ReactorMask 0x3FF003F0 (16 vcores)
MinConnectionsPerCore 1
Can someone help that whether any other parameters for trying so that
overall read performance can hit 1M.
I already try increase HUGHMEM and different MinConnectionIdleInterval and
it won't works.
thanks
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2251 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-03-30 1:25 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-28 6:36 [SPDK] SPDK performance over 4*10G nic cards Lego Lin
-- strict thread matches above, loose matches on Subject: below --
2018-03-30 1:25 Yang, Ziye
2018-03-30 1:21 Lego Lin
2018-03-28 6:37 Yang, Ziye
2018-03-28 6:21 Verma, Vishal4
2018-03-28 6:04 Lego Lin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.