Hi, Basically all runtime operations should be executed via spdk_thread_poll() During runtime, all one-time operations are requested by spdk_thread_send_msg() and executed via spdk_thread_poll(). All periodic operations are executed as pollers. Each poller is registered via spdk_poller_register(), and spdk_poller_register() is called in a message handler, and the message is requested by spdk_thread_send_msg(). You read that at the first time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk thread. This means the first message call is to execute the start function of the first SPDK thread. We don't specify any start function or main function when creating a SPDK thread. After the first SPDK thread processes the first sent message, it creates another SPDK thread or a poller. Hence the first SPDK thread can be executed via message or poller in spdk_thread_poll(). If your g_master_thread does not use a reactor, you have to call spdk_set_thread() by yourself before calling spdk_thread_send_msg(). Please take a look at example/nvmf/nvmf.c Thanks, Shuhei On Thu, Sep 17, 2020 at 11:07 AM Sunshihao (Euler OS) wrote: > Dear maintainer Jim, > > We have modified spdk for some requirement, our io workflow is : > > Frist we do init wok: > Spdk Init : spdk_env_init-> spdk_reactors_init-> > spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we > have modify code to make the g_masterThread never reactor run, just return > to do io submit job, so we execte every msg send to g_masterThread) > > And then, construct io resource: > Io resource construct: get a spdk_thread point thread_1-> > spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to > thread_n to get io channel) > > After finish that, submit io to thread_1 > Submit io: spdk_thread_send_msg(thread_n, LaunchIo, io); > LaunchIo will call bdev submit io function given by spdk; > Io is a struct have all args needed by bdev submit fuction. > > We do init work one time and do io resource construct and submit work many > times for io read/write. > > I have read the bdevperf code, and at the frist time call > spdk_thread_send_msg in spdk_app_start, the local thread is not a > spdk_thread, we get msg form pool; after that, in the reactor_run to call > spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache. > > In our submit work, we only get the spdk_thread thread_1 to send msg; I > set the local thread to thread_1 and create two or more thread to do submit > work; so, it cause the problem when get msg. > > Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n, > LaunchIo, io); the problem has been solved. > > Above is what i used spdk to submit io, maybe I used wrong way somewhere, > please give me some advice. > > Thank you! > > > -----邮件原件----- > 发件人: Harris, James R [mailto:james.r.harris(a)intel.com] > 发送时间: 2020年9月17日 1:33 > 收件人: Storage Performance Development Kit > 主题: [SPDK] Re: 答复: Re: thread->msg_cache is not safe when multiple threads > send messages to same spdk_thread > > Hi, > > Can you describe more about your application and when and where it submits > IO? Applications using the SPDK application framework (i.e. > spdk_app_start()) typically do not need to create their own spdk_threads - > they submit IO from the spdk_threads created by the reactors. > > Thanks, > > Jim > > > On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" wrote: > > Dear maintainer Jim, thank you for your reply. > > After reading your email, I know I have used the wrong way to > submiting io. > > When I create a thread to submit io, I don't create spdk_thread > structure for the thread, this structure only be created in > spdk_reactors_start api for every lcore; > Must I create spdk_thread structure for every submit thread? > > > Please give me some advice, thank you! > > > -----邮件原件----- > 发件人: Harris, James R [mailto:james.r.harris(a)intel.com] > 发送时间: 2020年9月16日 3:32 > 收件人: Storage Performance Development Kit > 主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads > send messages to same spdk_thread > > Hi, > > local_thread gets its value from _get_thread(). _get_thread() returns > the thread-local spdk_thread structure for the calling thread. So if > multiple threads call spdk_thread_send_msg() in parallel, they will each > use their own thread-local spdk_thread structure and no lock or > synchronization is needed. > > -Jim > > On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" > wrote: > > hello，maintainer： > when i use spdk api spdk_thread_send_msg to send io read/write > msg，i found a bug： > multiple threads send msg to one spdk_thread，they may get msg form > local_thread->msg_cache，there is not a lock or cas to > ensure the safety of the linked list。 > > msg = NULL; > > if (local_thread != NULL) { > if (local_thread->msg_cache_count > 0) { > msg = > SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */ > assert(msg != NULL); > > SLIST_REMOVE_HEAD(&local_thread->msg_cache, link); > local_thread->msg_cache_count--; > } > } > > i don't konw whether is my wrong way to use spdk_thread_send_msg > or the list is not safe，so please give me some advice。 > Thank you！ > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org >