From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8462820642601290377==" MIME-Version: 1.0 From: Howell, Seth Subject: Re: [SPDK] A problem with SPDK 19.01 NVMeoF/RDMA target Date: Wed, 27 Feb 2019 15:04:43 +0000 Message-ID: In-Reply-To: F5C3CA8F-4A52-467B-8756-1D9BDB36FA19@oracle.com List-ID: To: spdk@lists.01.org --===============8462820642601290377== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Lance, The data_wr tracker was not added until 19.01 and was not backported to 18.= 10.1 so this issue won't occur there. Thanks, Seth -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lance Hartmann= ORACLE Sent: Tuesday, February 26, 2019 4:45 PM To: Storage Performance Development Kit Subject: Re: [SPDK] A problem with SPDK 19.01 NVMeoF/RDMA target Hi Seth, > On Feb 7, 2019, at 12:18 PM, Howell, Seth wrote: > = > Hi Sasha, Valeriy, > = > With the help of Valeriy's logs I was able to get to the bottom of this. = The root cause is that for NVMe-oF requests that don't transfer any data, s= uch as keep_alive, we were not properly resetting the value of rdma_req->nu= m_outstanding_data_wr between uses of that structure. All data carrying ope= rations properly reset this value in spdk_nvmf_rdma_req_parse_sgl. = > = > My local repro steps look like this for anyone interested. > = > Start the SPDK target, > Submit a full queue depth worth of Smart log requests (sequentially is fi= ne). A smaller number also works, but takes much longer. > Wait for a while (This assumes you have keep alive enabled). Keep alive r= equests will reuse the rdma_req objects slowly incrementing the curr_send_d= epth on the admin qpair. > Eventually the admin qpair will be unable to submit I/O. > = > I was able to fix the issue locally with the following patch. https://rev= iew.gerrithub.io/#/c/spdk/spdk/+/443811/. Valeriy, please let me know if ap= plying this patch also fixes it for you ( I am pretty sure that it will). Does this issue present only in 19.01, or can it also occur in 18.10.1? -- Lance Hartmann _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============8462820642601290377==--