Re: [SPDK] How to utilize multiple threads?

Re: [SPDK] How to utilize multiple threads?

[@ 2018-07-31 10:46 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

Hi Li,

Sorry, some help may be needed not by me.
I’ll add it this time.

Thanks
Shuhei
2018年7月31日(火) 19:34 Li Feng <fengli(a)smartx.com>:

> Hi shuuhei,
>
> Thank you.
>
> I don't think I have permission to add an item on this board.
>
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1842 bytes --]

Re: [SPDK] How to utilize multiple threads?

[Li Feng]

[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]

Hi Jim,

Thank you.
This is my Trello ID: vonnyfly

Thanks,

Feng Li



手机：13401157876

Web：http://www.smartx.com/

地址：北京市海淀区科学院南路2号融科资讯中心C座北楼712

2018-07-31 22:55 GMT+08:00 Harris, James R <james.r.harris(a)intel.com>:

> Hi Feng,
>
>
>
> If you let me know your Trello ID, I can add you to the SPDK Trello “team”
> – then you will be able to add to or edit the SPDK Trello boards.
>
>
>
> Thanks,
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Li Feng <
> fengli(a)smartx.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, July 31, 2018 at 3:34 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] How to utilize multiple threads?
>
>
>
> Hi shuuhei,
>
>
>
> Thank you.
>
>
>
> I don't think I have permission to add an item on this board.
>
>
>
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
>
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>

-- 
The SmartX email address is only for business purpose. Any sent message 
that is not related to the business is not authorized or permitted by 
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8624 bytes --]

Re: [SPDK] How to utilize multiple threads?

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

Hi Feng,

If you let me know your Trello ID, I can add you to the SPDK Trello “team” – then you will be able to add to or edit the SPDK Trello boards.

Thanks,

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Li Feng <fengli(a)smartx.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, July 31, 2018 at 3:34 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] How to utilize multiple threads?

Hi shuuhei,

Thank you.

I don't think I have permission to add an item on this board.

The SmartX email address is only for business purpose. Any sent message that is not related to the business is not authorized or permitted by SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7348 bytes --]

Re: [SPDK] How to utilize multiple threads?

[Li Feng]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

Hi shuuhei,

Thank you.

I don't think I have permission to add an item on this board.

-- 
The SmartX email address is only for business purpose. Any sent message 
that is not related to the business is not authorized or permitted by 
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1173 bytes --]

Re: [SPDK] How to utilize multiple threads?

[@ 2018-07-31 9:23 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 6576 bytes --]

Hi Li,


As long as I know, we don't have any activity in SCSI persistent reservation yet.

Adding the item into https://trello.com/b/P5xBO7UR/things-to-do may be a good small start.

[https://s3.amazonaws.com/trello/images/og/trello-icon.png?v=2013-08-15]<https://trello.com/b/P5xBO7UR/things-to-do>

Things To Do - Trello<https://trello.com/b/P5xBO7UR/things-to-do>
trello.com
Trello is the visual collaboration platform that gives teams perspective on projects. Use Trello to collaborate, communicate and coordinate on all of your projects.

Thanks,
Shuhei


________________________________
差出人: SPDK <spdk-bounces(a)lists.01.org> が Li Feng <fengli(a)smartx.com> の代理で送信
送信日�: 2018年7月31日 16:22:24
宛先: Storage Performance Development Kit
件名: [!]Re: [SPDK] How to utilize multiple threads?

Thank you, shuuhei and Ziye.

Your response are so fast.
The spdk_iscsi_conn_migration will schedule a reactor to handle the new connection.
I have understood now.

As I know, the openiscsi initiator only uses one session/connection when connecting to one target, even though it has multiple luns.
So it will process IOs on the same core when they belong to the same target.


Another question, would you have a plan to support SCSI PR(persistent reservation)?






Thanks,

Feng Li



[https://docs.google.com/a/smartx.com/uc?id=0B6v_9z-4gTszRGFCWVFMSGlsMVE&export=download]

手机：13401157876

Web：http://www.smartx.com/<https://clicktime.symantec.com/a/1/NIaW0nHCIx-Su56w_ey9I4SlonPBj2x5JgN0UTEHECc=?d=a0dtS7vXF_8XU5lRc9TqVjuR4RPAHbYy6IvSPmMZ7pTBiZTXbP5TXZbdIzvwmoka9tVOBtp0siOmXn-jtuEjyEQIUSOTOSG5eMy8nB-130BOAAJRXVpEM9cp4AkBhh_ug0rnmnq0mNbh6rKdmKkPGk95jvhwjblau-wkVRo48lcDl2Q3TkbrRY3TkjNIEQnCGGTpjB9g2KFhXv9zME38o51fHMyfj1Os1NlIDFffQ3ihkjvvxwm1QkBRSsAKtC6j-u1w1t1tJ11LuUTpFRmzvqNoTcgOSc7m26Qs75d-hXxsISgIwXQooAnXI18uGtH6WQqc5Jz4OB24cFdjmqWvJy2aB-0MrwXD77gNKeh5HGm98AC4yE-1RDqVWQnpMYq4FMLym8Ilr-dStVgFa5I9wW6EvTdOOwup60dB&u=http%3A%2F%2Fwww.smartx.com%2F>

地址：北京市海淀区科学院南路2号融科资讯中心C座北楼712

2018-07-31 14:50 GMT+08:00 Yang, Ziye <ziye.yang(a)intel.com<mailto:ziye.yang(a)intel.com>>:

Hi Li,



The acceptor is running on core 0 on default,  but when the connection is in ffp phase, it will be transferred to other reactors for connection handling, see the code in spdk_iscsi_conn_migration in lib/iscsi/iscsi.c. According to the code, you can see it is also related with the portal group info (cpu mask).





Thanks.







Best Regards

Ziye Yang



From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of 松本周平 / MATSUMOTO，SHUUHEI
Sent: Tuesday, July 31, 2018 2:06 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] How to utilize multiple threads?



Hi Li,



But the acceptor is only running on the core 0, right?

Yes.



But if what you see is in acceptor.c, it may be the acceptor dedicated to check start of communication by initiator.



Regardless of this, you are correct and each iSCSI connection is handled by single CPU core because SPDK is highly efficient and lockless.

By increasing number of iSCSI connections, all CPU cores will be used.



The following by Ziye Yang is not maintained now but it will be helpful if you read.

https://review.gerrithub.io/#/c/spdk/spdk/+/392733/18/doc/iscsi_pg.md<https://clicktime.symantec.com/a/1/JbD4EzJNwwqhHPk2VL091L4Xdlq-0n9mN1AKLUcmI3M=?d=a0dtS7vXF_8XU5lRc9TqVjuR4RPAHbYy6IvSPmMZ7pTBiZTXbP5TXZbdIzvwmoka9tVOBtp0siOmXn-jtuEjyEQIUSOTOSG5eMy8nB-130BOAAJRXVpEM9cp4AkBhh_ug0rnmnq0mNbh6rKdmKkPGk95jvhwjblau-wkVRo48lcDl2Q3TkbrRY3TkjNIEQnCGGTpjB9g2KFhXv9zME38o51fHMyfj1Os1NlIDFffQ3ihkjvvxwm1QkBRSsAKtC6j-u1w1t1tJ11LuUTpFRmzvqNoTcgOSc7m26Qs75d-hXxsISgIwXQooAnXI18uGtH6WQqc5Jz4OB24cFdjmqWvJy2aB-0MrwXD77gNKeh5HGm98AC4yE-1RDqVWQnpMYq4FMLym8Ilr-dStVgFa5I9wW6EvTdOOwup60dB&u=https%3A%2F%2Freview.gerrithub.io%2F%23%2Fc%2Fspdk%2Fspdk%2F%2B%2F392733%2F18%2Fdoc%2Fiscsi_pg.md>



Thanks,

Shuhei

________________________________

差出人: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> が Li Feng <fengli(a)smartx.com<mailto:fengli(a)smartx.com>> の代理で送信
送信日�: 2018年7月31日 14:32
宛先: Storage Performance Development Kit
件名: [!][SPDK] How to utilize multiple threads?



Hi experts,



When I'm digging into SPDK iscsi source codes, I have some doubt about the threads/reactors.





Look at this example.



Start iscsi target on core 0-7:

```

app/iscsi_tgt/iscsi_tgt -L all -d 1 -m 0xff

```

I can see 8 threads which are polling.



But the acceptor is only running on the core 0, right?



This means all IOs are processed on core 0, no matter how many bdevs are connecting ?





I can see sending NOP IN is a poller running on all reactors. But this is not about read/write.



So how to utilize the other threads to maximum the iops and throughput?







Thanks in advance.





The SmartX email address is only for business purpose. Any sent message that is not related to the business is not authorized or permitted by SmartX.

本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.



_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk<https://clicktime.symantec.com/a/1/psu51-xE-fz6Ay-Fx9aVMUrxShFG7Xa-cYDSRqDm3bA=?d=a0dtS7vXF_8XU5lRc9TqVjuR4RPAHbYy6IvSPmMZ7pTBiZTXbP5TXZbdIzvwmoka9tVOBtp0siOmXn-jtuEjyEQIUSOTOSG5eMy8nB-130BOAAJRXVpEM9cp4AkBhh_ug0rnmnq0mNbh6rKdmKkPGk95jvhwjblau-wkVRo48lcDl2Q3TkbrRY3TkjNIEQnCGGTpjB9g2KFhXv9zME38o51fHMyfj1Os1NlIDFffQ3ihkjvvxwm1QkBRSsAKtC6j-u1w1t1tJ11LuUTpFRmzvqNoTcgOSc7m26Qs75d-hXxsISgIwXQooAnXI18uGtH6WQqc5Jz4OB24cFdjmqWvJy2aB-0MrwXD77gNKeh5HGm98AC4yE-1RDqVWQnpMYq4FMLym8Ilr-dStVgFa5I9wW6EvTdOOwup60dB&u=https%3A%2F%2Flists.01.org%2Fmailman%2Flistinfo%2Fspdk>



The SmartX email address is only for business purpose. Any sent message that is not related to the business is not authorized or permitted by SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 24846 bytes --]

Re: [SPDK] How to utilize multiple threads?

[Li Feng]

[-- Attachment #1: Type: text/plain, Size: 3993 bytes --]

Thank you, shuuhei and Ziye.

Your response are so fast.
The spdk_iscsi_conn_migration will schedule a reactor to handle the new
connection.
I have understood now.

As I know, the openiscsi initiator only uses one session/connection when
connecting to one target, even though it has multiple luns.
So it will process IOs on the same core when they belong to the same
target.


Another question, would you have a plan to support SCSI PR(persistent
reservation)?





Thanks,

Feng Li



手机：13401157876

Web：http://www.smartx.com/

地址：北京市海淀区科学院南路2号融科资讯中心C座北楼712

2018-07-31 14:50 GMT+08:00 Yang, Ziye <ziye.yang(a)intel.com>:

> Hi Li,
>
>
>
> The acceptor is running on core 0 on default,  but when the connection is
> in ffp phase, it will be transferred to other reactors for connection
> handling, see the code in spdk_iscsi_conn_migration in lib/iscsi/iscsi.c.
> According to the code, you can see it is also related with the portal group
> info (cpu mask).
>
>
>
>
>
> Thanks.
>
>
>
>
>
>
>
> Best Regards
>
> Ziye Yang
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *松本周平 /
> MATSUMOTO，SHUUHEI
> *Sent:* Tuesday, July 31, 2018 2:06 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] How to utilize multiple threads?
>
>
>
> Hi Li,
>
>
>
> But the acceptor is only running on the core 0, right?
>
> Yes.
>
>
>
> But if what you see is in acceptor.c, it may be the acceptor dedicated to
> check start of communication by initiator.
>
>
>
> Regardless of this, you are correct and each iSCSI connection is handled
> by single CPU core because SPDK is highly efficient and lockless.
>
> By increasing number of iSCSI connections, all CPU cores will be used.
>
>
>
> The following by Ziye Yang is not maintained now but it will be helpful if
> you read.
>
> https://review.gerrithub.io/#/c/spdk/spdk/+/392733/18/doc/iscsi_pg.md
>
>
>
> Thanks,
>
> Shuhei
> ------------------------------
>
> *差出人**:* SPDK <spdk-bounces(a)lists.01.org> が Li Feng <fengli(a)smartx.com>
> の代理で送信
> *送信日時**:* 2018年7月31日 14:32
> *宛先**:* Storage Performance Development Kit
> *件名**:* [!][SPDK] How to utilize multiple threads?
>
>
>
> Hi experts,
>
>
>
> When I'm digging into SPDK iscsi source codes, I have some doubt about the
> threads/reactors.
>
>
>
>
>
> Look at this example.
>
>
>
> Start iscsi target on core 0-7:
>
> ```
>
> app/iscsi_tgt/iscsi_tgt -L all -d 1 -m 0xff
>
> ```
>
> I can see 8 threads which are polling.
>
>
>
> But the acceptor is only running on the core 0, right?
>
>
>
> This means all IOs are processed on core 0, no matter how many bdevs are
> connecting ?
>
>
>
>
>
> I can see sending NOP IN is a poller running on all reactors. But this is
> not about read/write.
>
>
>
> So how to utilize the other threads to maximum the iops and throughput?
>
>
>
>
>
>
>
> Thanks in advance.
>
>
>
>
>
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
>
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>

-- 
The SmartX email address is only for business purpose. Any sent message 
that is not related to the business is not authorized or permitted by 
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 18652 bytes --]

Re: [SPDK] How to utilize multiple threads?

[Yang, Ziye]

[-- Attachment #1: Type: text/plain, Size: 2519 bytes --]

Hi Li,

The acceptor is running on core 0 on default,  but when the connection is in ffp phase, it will be transferred to other reactors for connection handling, see the code in spdk_iscsi_conn_migration in lib/iscsi/iscsi.c. According to the code, you can see it is also related with the portal group info (cpu mask).


Thanks.



Best Regards
Ziye Yang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of 松本周平 / MATSUMOTO，SHUUHEI
Sent: Tuesday, July 31, 2018 2:06 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] How to utilize multiple threads?


Hi Li,



But the acceptor is only running on the core 0, right?

Yes.



But if what you see is in acceptor.c, it may be the acceptor dedicated to check start of communication by initiator.



Regardless of this, you are correct and each iSCSI connection is handled by single CPU core because SPDK is highly efficient and lockless.

By increasing number of iSCSI connections, all CPU cores will be used.



The following by Ziye Yang is not maintained now but it will be helpful if you read.

https://review.gerrithub.io/#/c/spdk/spdk/+/392733/18/doc/iscsi_pg.md


Thanks,
Shuhei
________________________________
差出人: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> が Li Feng <fengli(a)smartx.com<mailto:fengli(a)smartx.com>> の代理で送信
送信日�: 2018年7月31日 14:32
宛先: Storage Performance Development Kit
件名: [!][SPDK] How to utilize multiple threads?

Hi experts,

When I'm digging into SPDK iscsi source codes, I have some doubt about the threads/reactors.


Look at this example.

Start iscsi target on core 0-7:
```
app/iscsi_tgt/iscsi_tgt -L all -d 1 -m 0xff
```
I can see 8 threads which are polling.

But the acceptor is only running on the core 0, right?

This means all IOs are processed on core 0, no matter how many bdevs are connecting ?


I can see sending NOP IN is a poller running on all reactors. But this is not about read/write.

So how to utilize the other threads to maximum the iops and throughput?



Thanks in advance.


The SmartX email address is only for business purpose. Any sent message that is not related to the business is not authorized or permitted by SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 15373 bytes --]

Re: [SPDK] How to utilize multiple threads?

[@ 2018-07-31 6:06 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]

Hi Li,


But the acceptor is only running on the core 0, right?

Yes.


But if what you see is in acceptor.c, it may be the acceptor dedicated to check start of communication by initiator.


Regardless of this, you are correct and each iSCSI connection is handled by single CPU core because SPDK is highly efficient and lockless.

By increasing number of iSCSI connections, all CPU cores will be used.


The following by Ziye Yang is not maintained now but it will be helpful if you read.

https://review.gerrithub.io/#/c/spdk/spdk/+/392733/18/doc/iscsi_pg.md


Thanks,
Shuhei

________________________________
差出人: SPDK <spdk-bounces(a)lists.01.org> が Li Feng <fengli(a)smartx.com> の代理で送信
送信日�: 2018年7月31日 14:32
宛先: Storage Performance Development Kit
件名: [!][SPDK] How to utilize multiple threads?

Hi experts,

When I'm digging into SPDK iscsi source codes, I have some doubt about the threads/reactors.


Look at this example.

Start iscsi target on core 0-7:
```
app/iscsi_tgt/iscsi_tgt -L all -d 1 -m 0xff
```
I can see 8 threads which are polling.

But the acceptor is only running on the core 0, right?

This means all IOs are processed on core 0, no matter how many bdevs are connecting ?


I can see sending NOP IN is a poller running on all reactors. But this is not about read/write.

So how to utilize the other threads to maximum the iops and throughput?



Thanks in advance.


The SmartX email address is only for business purpose. Any sent message that is not related to the business is not authorized or permitted by SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4605 bytes --]

[SPDK] How to utilize multiple threads?

[Li Feng]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Hi experts,

When I'm digging into SPDK iscsi source codes, I have some doubt about the
threads/reactors.


Look at this example.

Start iscsi target on core 0-7:
```
app/iscsi_tgt/iscsi_tgt -L all -d 1 -m 0xff
```
I can see 8 threads which are polling.

But the acceptor is only running on the core 0, right?

This means all IOs are processed on core 0, no matter how many bdevs are
connecting ?


I can see sending NOP IN is a poller running on all reactors. But this is
not about read/write.

So how to utilize the other threads to maximum the iops and throughput?



Thanks in advance.

-- 
The SmartX email address is only for business purpose. Any sent message 
that is not related to the business is not authorized or permitted by 
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1825 bytes --]