From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sander Eikelenboom Subject: Re: xennet: skb rides the rocket: 20 slots Date: Tue, 8 Jan 2013 21:55:03 +0100 Message-ID: <323202711.20130108215503@eikelenboom.it> References: <72958707.20130104172854@eikelenboom.it> <1357556115.7989.13.camel@zakaz.uk.xensource.com> <50EB8091.90705@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <50EB8091.90705@oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: ANNIE LI Cc: Konrad Rzeszutek Wilk , Ian Campbell , xen-devel List-Id: xen-devel@lists.xenproject.org Tuesday, January 8, 2013, 3:12:33 AM, you wrote: > On 2013-1-7 18:55, Ian Campbell wrote: >> On Fri, 2013-01-04 at 16:28 +0000, Sander Eikelenboom wrote: >>> Hi Ian, >> I'm not actually the netfront maintainer. Adding Konrad. Annie too since >> IIRC she was at one point looking at various buffer coalescing schemes. >> >>> Today i fired up an old VM with a bittorrent client, trying to >>> download some torrents. >> How old? Which kernel? >> >>> I seem to be hitting the unlikely case of "xennet: skb rides the >>> rocket: xx slots" and this results in some dropped packets in domU, I >>> don't see any warnings in dom0. > If debug is enabled, following netback code should print out messages in > netbk_count_requests. > if (unlikely(frags >= MAX_SKB_FRAGS)) { > netdev_dbg(vif->dev, "Too many frags\n"); > return -frags; > } I have added some rate limited warns in this function. However none seems to be triggered while the pv-guest reports the "skb rides the rocket" .. diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c index f2d6b78..2f02da9 100644 --- a/drivers/net/xen-netback/netback.c +++ b/drivers/net/xen-netback/netback.c @@ -902,11 +902,13 @@ static int netbk_count_requests(struct xenvif *vif, do { if (frags >= work_to_do) { netdev_dbg(vif->dev, "Need more frags\n"); + net_warn_ratelimited("%s: Need more frags:%d, work_to_do:%d \n",vif->dev->name, frags, work_to_do); return -frags; } if (unlikely(frags >= MAX_SKB_FRAGS)) { netdev_dbg(vif->dev, "Too many frags\n"); + net_warn_ratelimited("%s: Too many frags:%d, MAX_SKB_FRAGS:%d \n",vif->dev->name, frags, MAX_SKB_FRAGS); return -frags; } @@ -914,6 +916,7 @@ static int netbk_count_requests(struct xenvif *vif, sizeof(*txp)); if (txp->size > first->size) { netdev_dbg(vif->dev, "Frags galore\n"); + net_warn_ratelimited("%s: Frags galore:%d, txp->size:%d first->size:%d\n",vif->dev->name, frags, txp->size , first->size); return -frags; } @@ -923,6 +926,7 @@ static int netbk_count_requests(struct xenvif *vif, if (unlikely((txp->offset + txp->size) > PAGE_SIZE)) { netdev_dbg(vif->dev, "txp->offset: %x, size: %u\n", txp->offset, txp->size); + net_warn_ratelimited("%s: Hmm:%d, (txp->offset + txp->size):%d PAGE_SIZE:%d\n",vif->dev->name, frags, (txp->offset + txp->size) ,PAGE_SIZE); return -frags; } } while ((txp++)->flags & XEN_NETTXF_more_data); >>> >>> I have added some extra info, but i don't have enough knowledge if >>> this could/should be prevented from happening ? >> MAX_SKB_FRAGS has never, AFAIK, been as big as 19 or 20 so I'm not sure >> this can have ever worked. >> >> These SKBs seem to be pretty big (not quite 64KB), seemingly most of the >> data is contained in a smallish number of frags, which suggests compound >> pages and therefore a reasonably modern kernel? >> >> This probably relates somewhat to the issues described in the >> "netchannel vs MAX_SKB_FRAGS" thread last year, or at least the solution >> to that would necessarily involve fixing this issue. > If netback complains about "Too many frags", then it should be > MAX_SKB_FRAGS limitation in netback results in dropping packets in > netfront. It is possible that other netfronts(windows?) also hit this. > Thanks > Annie >> >> In the meantime you could try disabling sg and/or various offloads for >> that domain's vif. >> >>> [16798.629141] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> [16800.575182] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> [16801.589166] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800 >>> [16803.279039] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:3 page_size:4096 prot:0800 >>> [16809.973268] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> [16811.420048] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:6 page_size:4096 prot:0800 >>> [16814.872686] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800 >>> [16815.359099] xennet: skb rides the rocket: 21 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:20 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800 >>> [16825.851906] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> [16828.295083] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800 >>> [16837.386684] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> [16838.609683] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> [16841.783729] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:6 page_size:4096 prot:0800 >>> [16843.841678] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> [16849.847614] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:3690 skb_headlen:54 skb->len:64294, skb->data_len:64240 skb->truesize:65008 nr_frags:4 page_size:4096 prot:0800 >>> [16853.787128] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800 >>> >>> # ifconfig >>> eth0 Link encap:Ethernet HWaddr 00:16:3e:c4:20:46 >>> inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0 >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:9090214 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:5902090 errors:0 dropped:304 overruns:0 carrier:0 >>> collisions:0 txqueuelen:1000 >>> RX bytes:19757934770 (18.4 GiB) TX bytes:16855238200 (15.6 GiB) >>> Interrupt:25 >>> >>> >>> >>> >>> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c >>> index c26e28b..50ac403 100644 >>> --- a/drivers/net/xen-netfront.c >>> +++ b/drivers/net/xen-netfront.c >>> @@ -552,7 +552,11 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev) >>> xennet_count_skb_frag_slots(skb); >>> if (unlikely(slots> MAX_SKB_FRAGS + 1)) { >>> net_alert_ratelimited( >>> - "xennet: skb rides the rocket: %d slots\n", slots); >>> + "xennet: skb rides the rocket: %d slots MAX_SKB_FRAGS: %d div_roundup:%d xennet_count_skb_frag_slots:%d offset:%d skb_headlen:%d skb->len:%d, skb->data_len:%d skb->truesize:%d nr_frags:%d page_size:%d prot:%04x \n >>> + slots, MAX_SKB_FRAGS,DIV_ROUND_UP(offset + len, PAGE_SIZE), >>> + xennet_count_skb_frag_slots(skb), offset, len, skb->len, >>> + skb->data_len, skb->truesize, skb_shinfo(skb)->nr_frags, >>> + PAGE_SIZE, ntohs(skb->protocol)); >>> goto drop; >>> } >>> >>