From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhang, Helin" Subject: Re: rte_kni_tx_burst() hangs because of no freedescriptors Date: Tue, 25 Oct 2016 01:05:26 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: "dev@dpdk.org" To: yingzhi Return-path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 288332C09 for ; Tue, 25 Oct 2016 03:05:31 +0200 (CEST) In-Reply-To: Content-Language: en-US List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: yingzhi [mailto:kaitoy@qq.com] Sent: Monday, October 24, 2016 6:39 PM To: Zhang, Helin Cc: dev@dpdk.org Subject: Re: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedesc= riptors Hi Helin, Thanks for your response, to answer your questions: 1. we send only one packet each time calling rte_kni_tx_burst(), which mean= s the last argument is 1. 2. it returns 0 because the free mbuf function inside tx_burst will not fre= e any mbuf: if (txq->nb_tx_free < txq->tx_free_thresh) ixgbe_tx_free_bufs(txq); after this operation, the txq->nb_tx_free is not increased and keeps to be = "0" eventually. I did some tests today, I commented out this section of ixgbe_rxtx_vec_comm= on.h -> ixgbe_tx_free_bufs status =3D txq->tx_ring[txq->tx_next_dd].wb.status; if (!(status & IXGBE_ADVTXD_STAT_DD)) return 0; After ignoring DD bit check, our app runs for about 6 hours without issue. = I suspect there is something wrong in my program set the DD bit somewhere. = One of the possible cause currently I suspect is as far as I know, rte_pktm= buf_free(mbuf.array[k]) will free the mbuf of the packet and any fragmented= packets following by it. But in our application such as below code snippet= : auto nb_tx =3D rte_eth_tx_burst(port, queue, mbuf.array, (uint16_t)= nb_rx); if (unlikely(nb_tx < nb_rx)) { for (unsigned k =3D nb_tx; k < nb_rx; k++) { rte_pktmbuf_free(mbuf.array[k]); } } [Zhang, Helin] it seems above code piece has memory leak, if the buffer is = chained. After all memory leaked, then the issue comes. Please try to check= if this is the root cause! In this case if there are fragmented packets and failed transmission, may c= ause a mbuf be freed multiple times. Above is just my suspect, need to do some tests later today or tomorrow. Thanks ------------------ Original ------------------ From: "Zhang, Helin";; Date: Mon, Oct 24, 2016 11:33 AM To: "yingzhi"; Cc: "dev@dpdk.org"; Subject: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedescrip= tors Hi Yingzhi Thank you for the reporting! The description is not so clear at least for m= e. Please help to narrown down the issue by youself. How many packets would it have for calling TX function? Why it would return 0 after calling TX function? No memory? Or return from = else? Have you found anything? Regards, Helin > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of yingzhi > Sent: Sunday, October 23, 2016 9:30 PM > To: users; dev@dpdk.org > Subject: [dpdk-dev] rte_kni_tx_burst() hangs because of no free descripto= rs > > - > Hi Experts, > > Background: > > We are using DPDK to develop a LoadBalancer following below logic: When > a new packet is received: > 1. if the dst_addr is management IP, forward to KNI. 2. if the dst_addr = is in > VIP list, select backend and forward(modify dst mac address). 3. otherwis= e > drop the packet. > > At this stage, we use one single thread for KNI forwarding and another fo= r > VIP forwarding(forward to eth). > > DPDK version: 16.07 > NIC: 82599ES 10-Gigabit SFI/SFP+ Network Connection > Linux: 14.04.1-Ubuntu x64 > > Promblem description: > > The program runs correctly for sometime(around 2 hours for 400Mb traffic)= . > But it it will hang. When problem happens, rte_eth_tx_burst() will not ab= le to > send out any packets(always returns 0). We tracked into that function and > noticed it is actually calling ixgbe driver's ixgbe_xmit_pkts_vec() funct= ion in > our environment, because we use default tx queue configuration, after > printing some info, we found if the free function works fine: > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 31 > > it will trigger free and make 32 more free descriptors: > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 62 > > but when something going wrong, it will no longer free anything: > tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 0 tx_rs_thresh: 32, > tx_free_thresh: 32, nb_tx_free: 0 > > It may related with the DD flag of the descriptor but we are not quite su= re. > > Our program logic: > > create two mbuf pools on socket 0, one for rx_queue and one for kni. (all > lcore threads runs on socket0) > > init kni interface with rte_kni_alloc() > > > init one NIC interface with > rte_eth_dev_configure(); rte_eth_rx_queue_setup(); > rte_eth_tx_queue_setup(); rte_eth_dev_start(); > > > > in the eth main loop: (code is simplified) > while(1) { n =3D rte_eth_rx_burst(packets); for (i =3D 0; i < n;= ++i) { if > (SEND_TO_KNI) { m =3D rte_kni_tx_burst(packets[i]); = if (m !=3D 1)) > { rte_pktmbuf_free(packets[i]); } } = if (SEND_TO_ETH) > { // after modify the packet m =3D rte_eth_tx_bur= st(packets[i]); > if (m !=3D 1)) { rte_pktmbuf_free(packets[i]); = } } // > otherwise drop the packet rte_pktmbuf_free(packets[i]); } } > > > Please advise if I'm using DPDK in a wrong way. Sorry if I missed somethi= ng > basic, I'm new to DPDK. > > Thanks in advance > Best regards