Re: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01

Re: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01

[Lego Lin]

[-- Attachment #1: Type: text/plain, Size: 3019 bytes --]

Hi, Ziye

Many thanks for your quick response. I will try it.
Thank you

On Fri, Jun 14, 2019 at 9:33 AM Yang, Ziye <ziye.yang(a)intel.com> wrote:

> Hi Lego,
>
> Better to upgrade to 19.04 or 19.01.1. The issue you reported is similar
> to the issue reported by others through the following issue:
>
> https://github.com/spdk/spdk/issues/699
>
> I think that you observe the similar issue. So would you please read the
> link I provided.
>
> Thanks.
>
>
>
>
> Best Regards
> Ziye Yang
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lego Lin
> Sent: Friday, June 14, 2019 9:29 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01
>
> Hi, all:
>
> Our team build up our storage system base on SPDK. Recenly, we try to
> upgrade SPDK to 19.01, but we got performance degradation when upgrade to
> SPDK 19.01.
> I have compared our the reactor implementation in lib\event\reactor.c code
> base with different SPDK version (18.04, 18.07, 18.10, 19.01). The
> implementation is close to 18.04, a little bit different with 18.07, 18.10,
> but big difference with 19.01.
>
> The test environment as following:
> 1. hugemem size = 64G
> 2. Network environment: 4*10G and connect nic card directly without switch
> 3. HW info: 2.2GHz with 2 CPU socket with total 40 vcore 4. total memory:
> 256G 5. CPU assignment for SPDK: total vcore for SPDK, 8 vcore of CPU
> socket 0 for SPDK iSCSI 6. Use SPDK null block device to evaluate
> performance 7. We got the code base (include dpdk) at 2018/01 8. Create 16
> targets and 16 null block devices, mini connections per core = 2, each
> physical connections has 4 iSCSI connections. The max QD of iSCSI is 128.
> 9. FIO test pattern: T1(BS=4K, 100%Randwrite, QD=32), T2(BS=4K,
> 100%Randread, QD=32), T3(BS=4K, 70%RandRead30%RandWrite, QD=32)
>
> Our code base
> T1 (100%RandWrite): 1170 KIOPS
> T2 (100%RandWrite): 1189 KIOPS
> T3 (100%RandWrite): 941 KIOPS (Read), 403 KIOPS (Write)
>
> Merge reactor.c from 18.07 to our code base
> T3 (100%RandWrite): 948 KIOPS (Read), 406 KIOPS (Write)
>
> Merge reactor.c from 18.10 to our code base
> T3 (100%RandWrite): 933 KIOPS (Read), 400 KIOPS (Write)
>
> Our code base upgrade to SPDK 19.01
> T1 (100%RandWrite): 1150 KIOPS
> T2 (100%RandWrite): 1187 KIOPS
> T3 (100%RandWrite): 826 KIOPS (Read), 354 KIOPS (Write)
>
> The RWMix performance has obvious degrade.
> Can anyone give me some idea about possible root cause for RWMix
> performance tunning?
> Is it related to DPDK version or SPDK new reactor implementation?
>
> Thank you.
> Lego
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01

[Lego Lin]

[-- Attachment #1: Type: text/plain, Size: 3849 bytes --]

Hi, Ziye and Feng:

I have found the performance degradation root cause. SPDK still keep good
performance of RWMix at 19.01 version. I found the performance degrade
cause by our application. In our application, we create 8 POSIX thread and
run at CPU vcore 22 ~ 29. There is a bug that our threads will busy and
don't know why it affect SPDK iSCSI performance (VCORE 4 ~ 9, 20, 21) with
null block device.
After fix the bug, our thread won't busy and SPDK iSCSI RWMix performance
with null block device back to normal.
(NOTE: all these vcores locate at CPU socket 0, I still feel wired that
performance got affected)

Thank you for supporting.



On Fri, Jun 14, 2019 at 9:34 AM Lego Lin <lego.lin(a)gmail.com> wrote:

> Hi, Ziye
>
> Many thanks for your quick response. I will try it.
> Thank you
>
> On Fri, Jun 14, 2019 at 9:33 AM Yang, Ziye <ziye.yang(a)intel.com> wrote:
>
>> Hi Lego,
>>
>> Better to upgrade to 19.04 or 19.01.1. The issue you reported is similar
>> to the issue reported by others through the following issue:
>>
>> https://github.com/spdk/spdk/issues/699
>>
>> I think that you observe the similar issue. So would you please read the
>> link I provided.
>>
>> Thanks.
>>
>>
>>
>>
>> Best Regards
>> Ziye Yang
>>
>> -----Original Message-----
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lego Lin
>> Sent: Friday, June 14, 2019 9:29 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01
>>
>> Hi, all:
>>
>> Our team build up our storage system base on SPDK. Recenly, we try to
>> upgrade SPDK to 19.01, but we got performance degradation when upgrade to
>> SPDK 19.01.
>> I have compared our the reactor implementation in lib\event\reactor.c
>> code base with different SPDK version (18.04, 18.07, 18.10, 19.01). The
>> implementation is close to 18.04, a little bit different with 18.07, 18.10,
>> but big difference with 19.01.
>>
>> The test environment as following:
>> 1. hugemem size = 64G
>> 2. Network environment: 4*10G and connect nic card directly without
>> switch 3. HW info: 2.2GHz with 2 CPU socket with total 40 vcore 4. total
>> memory: 256G 5. CPU assignment for SPDK: total vcore for SPDK, 8 vcore of
>> CPU socket 0 for SPDK iSCSI 6. Use SPDK null block device to evaluate
>> performance 7. We got the code base (include dpdk) at 2018/01 8. Create 16
>> targets and 16 null block devices, mini connections per core = 2, each
>> physical connections has 4 iSCSI connections. The max QD of iSCSI is 128.
>> 9. FIO test pattern: T1(BS=4K, 100%Randwrite, QD=32), T2(BS=4K,
>> 100%Randread, QD=32), T3(BS=4K, 70%RandRead30%RandWrite, QD=32)
>>
>> Our code base
>> T1 (100%RandWrite): 1170 KIOPS
>> T2 (100%RandWrite): 1189 KIOPS
>> T3 (100%RandWrite): 941 KIOPS (Read), 403 KIOPS (Write)
>>
>> Merge reactor.c from 18.07 to our code base
>> T3 (100%RandWrite): 948 KIOPS (Read), 406 KIOPS (Write)
>>
>> Merge reactor.c from 18.10 to our code base
>> T3 (100%RandWrite): 933 KIOPS (Read), 400 KIOPS (Write)
>>
>> Our code base upgrade to SPDK 19.01
>> T1 (100%RandWrite): 1150 KIOPS
>> T2 (100%RandWrite): 1187 KIOPS
>> T3 (100%RandWrite): 826 KIOPS (Read), 354 KIOPS (Write)
>>
>> The RWMix performance has obvious degrade.
>> Can anyone give me some idea about possible root cause for RWMix
>> performance tunning?
>> Is it related to DPDK version or SPDK new reactor implementation?
>>
>> Thank you.
>> Lego
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
>

Re: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01

[Yang, Ziye]

[-- Attachment #1: Type: text/plain, Size: 2578 bytes --]

Hi Lego,

Better to upgrade to 19.04 or 19.01.1. The issue you reported is similar to the issue reported by others through the following issue:

https://github.com/spdk/spdk/issues/699

I think that you observe the similar issue. So would you please read the link I provided.

Thanks.




Best Regards
Ziye Yang 

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lego Lin
Sent: Friday, June 14, 2019 9:29 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01

Hi, all:

Our team build up our storage system base on SPDK. Recenly, we try to upgrade SPDK to 19.01, but we got performance degradation when upgrade to SPDK 19.01.
I have compared our the reactor implementation in lib\event\reactor.c code base with different SPDK version (18.04, 18.07, 18.10, 19.01). The implementation is close to 18.04, a little bit different with 18.07, 18.10, but big difference with 19.01.

The test environment as following:
1. hugemem size = 64G
2. Network environment: 4*10G and connect nic card directly without switch 3. HW info: 2.2GHz with 2 CPU socket with total 40 vcore 4. total memory: 256G 5. CPU assignment for SPDK: total vcore for SPDK, 8 vcore of CPU socket 0 for SPDK iSCSI 6. Use SPDK null block device to evaluate performance 7. We got the code base (include dpdk) at 2018/01 8. Create 16 targets and 16 null block devices, mini connections per core = 2, each physical connections has 4 iSCSI connections. The max QD of iSCSI is 128.
9. FIO test pattern: T1(BS=4K, 100%Randwrite, QD=32), T2(BS=4K, 100%Randread, QD=32), T3(BS=4K, 70%RandRead30%RandWrite, QD=32)

Our code base
T1 (100%RandWrite): 1170 KIOPS
T2 (100%RandWrite): 1189 KIOPS
T3 (100%RandWrite): 941 KIOPS (Read), 403 KIOPS (Write)

Merge reactor.c from 18.07 to our code base
T3 (100%RandWrite): 948 KIOPS (Read), 406 KIOPS (Write)

Merge reactor.c from 18.10 to our code base
T3 (100%RandWrite): 933 KIOPS (Read), 400 KIOPS (Write)

Our code base upgrade to SPDK 19.01
T1 (100%RandWrite): 1150 KIOPS
T2 (100%RandWrite): 1187 KIOPS
T3 (100%RandWrite): 826 KIOPS (Read), 354 KIOPS (Write)

The RWMix performance has obvious degrade.
Can anyone give me some idea about possible root cause for RWMix performance tunning?
Is it related to DPDK version or SPDK new reactor implementation?

Thank you.
Lego
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

[SPDK] Performance degrade of RWMix when upgrade SPDK to 19.01

[Lego Lin]

[-- Attachment #1: Type: text/plain, Size: 1857 bytes --]

Hi, all:

Our team build up our storage system base on SPDK. Recenly, we try to
upgrade SPDK to 19.01, but we
got performance degradation when upgrade to SPDK 19.01.
I have compared our the reactor implementation in lib\event\reactor.c code
base with different SPDK version (18.04, 18.07, 18.10, 19.01). The
implementation is close to 18.04, a little bit different with 18.07, 18.10,
but big difference with 19.01.

The test environment as following:
1. hugemem size = 64G
2. Network environment: 4*10G and connect nic card directly without switch
3. HW info: 2.2GHz with 2 CPU socket with total 40 vcore
4. total memory: 256G
5. CPU assignment for SPDK: total vcore for SPDK, 8 vcore of CPU socket 0
for SPDK iSCSI
6. Use SPDK null block device to evaluate performance
7. We got the code base (include dpdk) at 2018/01
8. Create 16 targets and 16 null block devices, mini connections per core =
2, each physical connections has 4 iSCSI connections. The max QD of iSCSI
is 128.
9. FIO test pattern: T1(BS=4K, 100%Randwrite, QD=32), T2(BS=4K,
100%Randread, QD=32), T3(BS=4K, 70%RandRead30%RandWrite, QD=32)

Our code base
T1 (100%RandWrite): 1170 KIOPS
T2 (100%RandWrite): 1189 KIOPS
T3 (100%RandWrite): 941 KIOPS (Read), 403 KIOPS (Write)

Merge reactor.c from 18.07 to our code base
T3 (100%RandWrite): 948 KIOPS (Read), 406 KIOPS (Write)

Merge reactor.c from 18.10 to our code base
T3 (100%RandWrite): 933 KIOPS (Read), 400 KIOPS (Write)

Our code base upgrade to SPDK 19.01
T1 (100%RandWrite): 1150 KIOPS
T2 (100%RandWrite): 1187 KIOPS
T3 (100%RandWrite): 826 KIOPS (Read), 354 KIOPS (Write)

The RWMix performance has obvious degrade.
Can anyone give me some idea about possible root cause for RWMix
performance tunning?
Is it related to DPDK version or SPDK new reactor implementation?

Thank you.
Lego