From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8014898091651064520==" MIME-Version: 1.0 From: =?utf-8?q?=E6=9D=BE=E6=9C=AC=E5=91=A8=E5=B9=B3_=3Cshuheimatsumoto_at_gmai?= =?utf-8?q?l=2Ecom=3E?= Subject: =?utf-8?q?=5BSPDK=5D_Re=3A_=E7=AD=94=E5=A4=8D=3A_Re=3A_=E7=AD=94=E5=A4=8D?= =?utf-8?q?=3A_Re=3A_thread-=3Emsg=5Fcache_is_not_safe_when_multiple_threa?= =?utf-8?q?ds_send_messages_to_same_spdk=5Fthread?= Date: Fri, 18 Sep 2020 06:36:31 +0900 Message-ID: In-Reply-To: 944b683bedea4146ac4c82e7d863db59@huawei.com List-ID: To: spdk@lists.01.org --===============8014898091651064520== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, Basically all runtime operations should be executed via spdk_thread_poll() During runtime, all one-time operations are requested by spdk_thread_send_msg() and executed via spdk_thread_poll(). All periodic operations are executed as pollers. Each poller is registered via spdk_poller_register(), and spdk_poller_register() is called in a message handler, and the message is requested by spdk_thread_send_msg(). You read that at the first time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk thread. This means the first message call is to execute the start function of the first SPDK thread. We don't specify any start function or main function when creating a SPDK thread. After the first SPDK thread processes the first sent message, it creates another SPDK thread or a poller. Hence the first SPDK thread can be executed via message or poller in spdk_thread_poll(). If your g_master_thread does not use a reactor, you have to call spdk_set_thread() by yourself before calling spdk_thread_send_msg(). Please take a look at example/nvmf/nvmf.c Thanks, Shuhei On Thu, Sep 17, 2020 at 11:07 AM Sunshihao (Euler OS) wrote: > Dear maintainer Jim, > > We have modified spdk for some requirement, our io workflow is : > > Frist we do init wok: > Spdk Init : spdk_env_init-> spdk_reactors_init-> > spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we > have modify code to make the g_masterThread never reactor run, just return > to do io submit job, so we execte every msg send to g_masterThread) > > And then, construct io resource: > Io resource construct: get a spdk_thread point thread_1-> > spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to > thread_n to get io channel) > > After finish that, submit io to thread_1 > Submit io: spdk_thread_send_msg(thread_n, LaunchIo, io); > LaunchIo will call bdev submit io function given by spdk; > Io is a struct have all args needed by bdev submit fuction. > > We do init work one time and do io resource construct and submit work many > times for io read/write. > > I have read the bdevperf code, and at the frist time call > spdk_thread_send_msg in spdk_app_start, the local thread is not a > spdk_thread, we get msg form pool; after that, in the reactor_run to call > spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cach= e. > > In our submit work, we only get the spdk_thread thread_1 to send msg; I > set the local thread to thread_1 and create two or more thread to do subm= it > work; so, it cause the problem when get msg. > > Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n, > LaunchIo, io); the problem has been solved. > > Above is what i used spdk to submit io, maybe I used wrong way somewhere, > please give me some advice. > > Thank you! > > > -----=E9=82=AE=E4=BB=B6=E5=8E=9F=E4=BB=B6----- > =E5=8F=91=E4=BB=B6=E4=BA=BA: Harris, James R [mailto:james.r.harris(a)int= el.com] > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2020=E5=B9=B49=E6=9C=8817=E6=97=A5 = 1:33 > =E6=94=B6=E4=BB=B6=E4=BA=BA: Storage Performance Development Kit > =E4=B8=BB=E9=A2=98: [SPDK] Re: =E7=AD=94=E5=A4=8D: Re: thread->msg_cache = is not safe when multiple threads > send messages to same spdk_thread > > Hi, > > Can you describe more about your application and when and where it submits > IO? Applications using the SPDK application framework (i.e. > spdk_app_start()) typically do not need to create their own spdk_threads - > they submit IO from the spdk_threads created by the reactors. > > Thanks, > > Jim > > > =EF=BB=BFOn 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" wrote: > > Dear maintainer Jim, thank you for your reply. > > After reading your email, I know I have used the wrong way to > submiting io. > > When I create a thread to submit io, I don't create spdk_thread > structure for the thread, this structure only be created in > spdk_reactors_start api for every lcore; > Must I create spdk_thread structure for every submit thread? > > > Please give me some advice, thank you! > > > -----=E9=82=AE=E4=BB=B6=E5=8E=9F=E4=BB=B6----- > =E5=8F=91=E4=BB=B6=E4=BA=BA: Harris, James R [mailto:james.r.harris(a= )intel.com] > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2020=E5=B9=B49=E6=9C=8816=E6=97= =A5 3:32 > =E6=94=B6=E4=BB=B6=E4=BA=BA: Storage Performance Development Kit > =E4=B8=BB=E9=A2=98: [SPDK] Re: thread->msg_cache is not safe when mul= tiple threads > send messages to same spdk_thread > > Hi, > > local_thread gets its value from _get_thread(). _get_thread() returns > the thread-local spdk_thread structure for the calling thread. So if > multiple threads call spdk_thread_send_msg() in parallel, they will each > use their own thread-local spdk_thread structure and no lock or > synchronization is needed. > > -Jim > > On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" > wrote: > > hello=EF=BC=8Cmaintainer=EF=BC=9A > when i use spdk api spdk_thread_send_msg to send io read/write > msg=EF=BC=8Ci found a bug=EF=BC=9A > multiple threads send msg to one spdk_thread=EF=BC=8Cthey may get= msg form > local_thread->msg_cache=EF=BC=8Cthere is not a lock or cas to > ensure the safety of the linked list=E3=80=82 > > msg =3D NULL; > > if (local_thread !=3D NULL) { > if (local_thread->msg_cache_count > 0) { > msg =3D > SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */ > assert(msg !=3D NULL); > > SLIST_REMOVE_HEAD(&local_thread->msg_cache, link); > local_thread->msg_cache_count--; > } > } > > i don't konw whether is my wrong way to use spdk_thread_send_msg > or the list is not safe=EF=BC=8Cso please give me some advice=E3=80=82 > Thank you=EF=BC=81 > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > --===============8014898091651064520==--