[SPDK] Re: 答复: Re: 答复: Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

[-- Attachment #1: Type: text/plain, Size: 6749 bytes --]

Hi,

Basically all runtime operations should be executed via spdk_thread_poll()

During runtime, all one-time operations are requested by
spdk_thread_send_msg() and executed via spdk_thread_poll().
All periodic operations are executed as pollers. Each poller is registered
via spdk_poller_register(), and spdk_poller_register() is called in a
message handler, and the message is requested by spdk_thread_send_msg().

You read that at the first time call spdk_thread_send_msg in
spdk_app_start, the local thread is not a spdk thread.

This means the first message call is to execute the start function of the
first SPDK thread.
We don't specify any start function or main function when creating a SPDK
thread.
After the first SPDK thread processes the first sent message, it creates
another SPDK thread or a poller.
Hence the first SPDK thread can be executed via message or poller in
spdk_thread_poll().


If your g_master_thread does not use a reactor, you have to call
spdk_set_thread() by yourself before calling spdk_thread_send_msg().

Please take a look at example/nvmf/nvmf.c

Thanks,
Shuhei

On Thu, Sep 17, 2020 at 11:07 AM Sunshihao (Euler OS) <sunshihao(a)huawei.com>
wrote:

> Dear maintainer Jim,
>
> We have modified spdk for some requirement, our io workflow is :
>
> Frist we do init wok:
> Spdk Init : spdk_env_init-> spdk_reactors_init->
> spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we
> have modify code to make the g_masterThread never reactor run, just return
> to do io submit job, so we execte every msg send to g_masterThread)
>
> And then, construct io resource:
> Io resource construct: get a spdk_thread point thread_1->
> spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to
> thread_n to get io channel)
>
> After finish that, submit io to thread_1
> Submit io: spdk_thread_send_msg(thread_n, LaunchIo,  io);
> LaunchIo will call bdev submit io function given by spdk;
> Io is a struct have all args needed by bdev submit fuction.
>
> We do init work one time and do io resource construct and submit work many
> times for io read/write.
>
> I have read the bdevperf code, and at the frist time call
> spdk_thread_send_msg in spdk_app_start, the local thread is not a
> spdk_thread, we get msg form pool; after that, in the reactor_run to call
> spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache.
>
> In our submit work, we only get the spdk_thread  thread_1 to send msg; I
> set the local thread to thread_1 and create two or more thread to do submit
> work; so, it cause the problem when get msg.
>
> Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n,
> LaunchIo,  io);  the problem has been solved.
>
> Above is what i used spdk to submit io, maybe I used wrong way somewhere,
> please give me some advice.
>
> Thank you!
>
>
> -----邮件原件-----
> 发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
> 发送时间: 2020年9月17日 1:33
> 收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
> 主题: [SPDK] Re: 答复: Re: thread->msg_cache is not safe when multiple threads
> send messages to same spdk_thread
>
> Hi,
>
> Can you describe more about your application and when and where it submits
> IO?  Applications using the SPDK application framework (i.e.
> spdk_app_start()) typically do not need to create their own spdk_threads -
> they submit IO from the spdk_threads created by the reactors.
>
> Thanks,
>
> Jim
>
>
> On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:
>
>     Dear maintainer Jim, thank you for your reply.
>
>     After reading your email, I know I have used the wrong way to
> submiting io.
>
>     When I create a thread to submit io, I don't create spdk_thread
> structure for the thread, this structure only be created in
> spdk_reactors_start api for every lcore;
>     Must I create spdk_thread structure for every submit thread?
>
>
>     Please give me some advice, thank you!
>
>
>     -----邮件原件-----
>     发件人: Harris, James R [mailto:james.r.harris(a)intel.com]
>     发送时间: 2020年9月16日 3:32
>     收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
>     主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads
> send messages to same spdk_thread
>
>     Hi,
>
>     local_thread gets its value from _get_thread().  _get_thread() returns
> the thread-local spdk_thread structure for the calling thread. So if
> multiple threads call spdk_thread_send_msg() in parallel, they will each
> use their own thread-local spdk_thread structure and no lock or
> synchronization is needed.
>
>     -Jim
>
>     On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com>
> wrote:
>
>         hello，maintainer：
>         when i use spdk api spdk_thread_send_msg to send io read/write
> msg，i found a bug：
>         multiple threads send msg to one spdk_thread，they may get msg form
> local_thread->msg_cache，there is not a lock or cas to
>         ensure the safety of the linked list。
>
>                  msg = NULL;
>
>                 if (local_thread != NULL) {
>                         if (local_thread->msg_cache_count > 0) {
>                                         msg =
> SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
>                                         assert(msg != NULL);
>
> SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
>                                         local_thread->msg_cache_count--;
>                         }
>                 }
>
>         i don't konw whether is my wrong way to use spdk_thread_send_msg
> or the list is not safe，so please give me some advice。
>         Thank you！
>         _______________________________________________
>         SPDK mailing list -- spdk(a)lists.01.org
>         To unsubscribe send an email to spdk-leave(a)lists.01.org
>
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>