From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Ananyev, Konstantin" Subject: Re: rte_kni_tx_burst() hangs because of no freedescriptors Date: Tue, 25 Oct 2016 09:20:41 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772583F0CC64A@irsmsx105.ger.corp.intel.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: "dev@dpdk.org" To: "Zhang, Helin" , yingzhi Return-path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id BB8B0591F for ; Tue, 25 Oct 2016 11:20:45 +0200 (CEST) In-Reply-To: Content-Language: en-US List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Helin, >=20 > From: yingzhi [mailto:kaitoy@qq.com] > Sent: Monday, October 24, 2016 6:39 PM > To: Zhang, Helin > Cc: dev@dpdk.org > Subject: Re: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freede= scriptors >=20 > Hi Helin, >=20 > Thanks for your response, to answer your questions: > 1. we send only one packet each time calling rte_kni_tx_burst(), which me= ans the last argument is 1. > 2. it returns 0 because the free mbuf function inside tx_burst will not f= ree any mbuf: >=20 > if (txq->nb_tx_free < txq->tx_free_thresh) > ixgbe_tx_free_bufs(txq); >=20 > after this operation, the txq->nb_tx_free is not increased and keeps to b= e "0" eventually. >=20 > I did some tests today, I commented out this section of ixgbe_rxtx_vec_co= mmon.h -> ixgbe_tx_free_bufs >=20 > status =3D txq->tx_ring[txq->tx_next_dd].wb.status; > if (!(status & IXGBE_ADVTXD_STAT_DD)) > return 0; >=20 > After ignoring DD bit check, our app runs for about 6 hours without issue= . I suspect there is something wrong in my program set the DD bit > somewhere. One of the possible cause currently I suspect is as far as I k= now, rte_pktmbuf_free(mbuf.array[k]) will free the mbuf of the > packet and any fragmented packets following by it. But in our application= such as below code snippet: >=20 > auto nb_tx =3D rte_eth_tx_burst(port, queue, mbuf.array, (uint16_= t) nb_rx); > if (unlikely(nb_tx < nb_rx)) { > for (unsigned k =3D nb_tx; k < nb_rx; k++) { > rte_pktmbuf_free(mbuf.array[k]); > } > } > [Zhang, Helin] it seems above code piece has memory leak, if the buffer i= s chained. After all memory leaked, then the issue comes. Please > try to check if this is the root cause! >=20 > In this case if there are fragmented packets and failed transmission, may= cause a mbuf be freed multiple times. Yes rte_pktmbuf_free() will free all chained segments for that packet. The code above seems correct to me (if of course .you don't try to free these packets again somewhere later). = =20 Can you clarify where do you think is a memory leak here? Konstantin >=20 > Above is just my suspect, need to do some tests later today or tomorrow. >=20 > Thanks > ------------------ Original ------------------ > From: "Zhang, Helin";; > Date: Mon, Oct 24, 2016 11:33 AM > To: "yingzhi"; > Cc: "dev@dpdk.org"; > Subject: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedescr= iptors >=20 > Hi Yingzhi >=20 > Thank you for the reporting! The description is not so clear at least for= me. > Please help to narrown down the issue by youself. > How many packets would it have for calling TX function? > Why it would return 0 after calling TX function? No memory? Or return fro= m else? Have you found anything? >=20 > Regards, > Helin >=20 > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of yingzhi > > Sent: Sunday, October 23, 2016 9:30 PM > > To: users; dev@dpdk.org > > Subject: [dpdk-dev] rte_kni_tx_burst() hangs because of no free descrip= tors > > > > - > > Hi Experts, > > > > Background: > > > > We are using DPDK to develop a LoadBalancer following below logic: When > > a new packet is received: > > 1. if the dst_addr is management IP, forward to KNI. 2. if the dst_add= r is in > > VIP list, select backend and forward(modify dst mac address). 3. otherw= ise > > drop the packet. > > > > At this stage, we use one single thread for KNI forwarding and another = for > > VIP forwarding(forward to eth). > > > > DPDK version: 16.07 > > NIC: 82599ES 10-Gigabit SFI/SFP+ Network Connection > > Linux: 14.04.1-Ubuntu x64 > > > > Promblem description: > > > > The program runs correctly for sometime(around 2 hours for 400Mb traffi= c). > > But it it will hang. When problem happens, rte_eth_tx_burst() will not = able to > > send out any packets(always returns 0). We tracked into that function a= nd > > noticed it is actually calling ixgbe driver's ixgbe_xmit_pkts_vec() fun= ction in > > our environment, because we use default tx queue configuration, after > > printing some info, we found if the free function works fine: > > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 31 > > > > it will trigger free and make 32 more free descriptors: > > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 62 > > > > but when something going wrong, it will no longer free anything: > > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 0 tx_rs_thresh: 32, > > tx_free_thresh: 32, nb_tx_free: 0 > > > > It may related with the DD flag of the descriptor but we are not quite = sure. > > > > Our program logic: > > > > create two mbuf pools on socket 0, one for rx_queue and one for kni. (a= ll > > lcore threads runs on socket0) > > > > init kni interface with rte_kni_alloc() > > > > > > init one NIC interface with > > rte_eth_dev_configure(); rte_eth_rx_queue_setup(); > > rte_eth_tx_queue_setup(); rte_eth_dev_start(); > > > > > > > > in the eth main loop: (code is simplified) > > while(1) { n =3D rte_eth_rx_burst(packets); for (i =3D 0; i < = n; ++i) { if > > (SEND_TO_KNI) { m =3D rte_kni_tx_burst(packets[i]); = if (m !=3D 1)) > > { rte_pktmbuf_free(packets[i]); } }= if (SEND_TO_ETH) > > { // after modify the packet m =3D rte_eth_tx_b= urst(packets[i]); > > if (m !=3D 1)) { rte_pktmbuf_free(packets[i]); = } } // > > otherwise drop the packet rte_pktmbuf_free(packets[i]); } } > > > > > > Please advise if I'm using DPDK in a wrong way. Sorry if I missed somet= hing > > basic, I'm new to DPDK. > > > > Thanks in advance > > Best regards