From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8806659859278822934==" MIME-Version: 1.0 From: Luse, Paul E Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf Date: Sun, 09 Sep 2018 13:41:54 +0000 Message-ID: <82C9F782B054C94B9FC04A331649C77AAD5367B5@fmsmsx104.amr.corp.intel.com> In-Reply-To: FBE7E039FA50BF47A673AD0BD3CD56A8461E462B@HASMSX105.ger.corp.intel.com List-ID: To: spdk@lists.01.org --===============8806659859278822934== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Darek, I can close on this from my end, thanks though! -Paul -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dar= iusz Sent: Saturday, September 8, 2018 4:10 PM To: Storage Performance Development Kit Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdev= perf Apparently the socket id =3D=3D -1 you were getting did not mean unitialize= d. It was implicitly set to SOCKET_ID_ANY by rte_thread_set_affinity() in s= pdk_unaffinitize_thread(). Since DPDK allows SOCKET_ID_ANY as an possible s= ocket id value, then it must be the vdev or the crypto stuff that's broken.= Do you mind sharing the exact error log? I don't have the setup to reprodu= ce this myself. D. -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E Sent: Friday, September 7, 2018 5:07 PM To: Storage Performance Development Kit Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdev= perf For sure, I'll wait for Derek's response on what he was planning on doing t= omorrow and pick anything up from there... Thx Paul -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R Sent: Friday, September 7, 2018 8:05 AM To: Storage Performance Development Kit Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdev= perf I like Darek=E2=80=99s suggestions. I do think this is a DPDK bug though. = Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul =E2= =80=93 can you dig into this further? -Jim On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" wrote: That's really good to know, thanks! I'm hardcoded to 0 now which is fi= ne for the remainder of testing I think, let me know how it goes with get_m= empolicy = Thx, Paul = -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk,= Dariusz Sent: Friday, September 7, 2018 7:31 AM To: Storage Performance Development Kit Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or = bdevperf = I wasn't registered on the mailing list with this email before, so I re= gistered just now and am resending this message. D. = -----Original Message----- From: Stojaczyk, Dariusz = Sent: Friday, September 7, 2018 4:04 PM To: Storage Performance Development Kit Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf = HI Paul, = Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstract= ion for multi-threading. It uses standard pthreads for most of its work and= that's why you're seeing socket id 4294967295 (unsigned -1). This value i= s simply uninitialized. To get socket id in a portable way, you might want = to use get_mempolicy. In fact, we should probably use it somewhere within o= ur SPDK APIs. I'll look into adding it tomorrow. D. = -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul= E Sent: Friday, September 7, 2018 3:27 PM To: Storage Performance Development Kit Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdev= perf = = In the crypto patch when I init the aesni virtual device I've been pass= ing in NULL for optional args and it's worked fine. When running w/fio how= ever, the aesni driver took the null args to mean "socket any" which should= be fine but for whatever reason its arg parser doesn't like that value. = So this works in bdevio/bdevperf but does not in FIO = rc =3D rte_vdev_init(AESNI_MB, NULL); = Hardcoding the only arg I care about, socket_id, because it's the one t= hat the aesni driver doesn't like not being specified works in fio as well = as our apps: = rc =3D rte_vdev_init(AESNI_MB, "socket_id=3D0"); = So using spdk calls I tried this: = snprintf(socket_id, 32, "socket_id=3D%d", spdk_env_get_socket_id(spdk_env_get_current_core())); rc =3D rte_vdev_init(AESNI_MB, socket_id); = and once again woks with bdevio and fails with fio. Looking at the dif= ferences in the code here what I see: = running with bdevio: = =E2=94=82173 rte_lcore_to_socket_id(unsigned lcore_id) = = =E2=94=82 =E2=94=82174 { = = =E2=94=82 >=E2=94=82175 return lcore_config[lcore_id].socket_id; = = =E2=94=82 =E2=94=82176 } = (gdb) p lcore_config[0] $1 =3D {detected =3D 1, thread_id =3D 0, pipe_master2slave =3D {0, 0}, = pipe_slave2master =3D {0, 0}, f =3D 0x0, arg =3D 0x0, ret =3D 0, state = =3D WAIT, socket_id =3D 0, core_id =3D 0, core_index =3D 0, cpuset =3D {__bits = =3D {1, 0 }}, core_role =3D 0 '\000'} = and running with fio: = $1 =3D {detected =3D 1, thread_id =3D 0, pipe_master2slave =3D {0, 0}, = pipe_slave2master =3D {0, 0}, f =3D 0x0, arg =3D 0x0, ret =3D 0, state = =3D WAIT, socket_id =3D 4294967295, core_id =3D 0, core_index =3D 0, cpuset =3D= {__bits =3D {18446744073709551615, 16777215, 0 }}, core_role =3D 0 '\000'} = Notice the difference. Any idea why running with fio plugin vs bdevio = or bdevperf would cause this?? = Thanks! Paul = = = = = = = = _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============8806659859278822934==--