Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom <linux@eikelenboom.it>]

Wednesday, March 12, 2014, 4:04:35 PM, you wrote:

> On Wed, Mar 12, 2014 at 04:01:56PM +0100, Sander Eikelenboom wrote:
>> 
>> Wednesday, March 12, 2014, 3:59:15 PM, you wrote:
>> 
>> > On Wed, Mar 12, 2014 at 03:49:46PM +0100, Sander Eikelenboom wrote:
>> >> 
>> >> Wednesday, March 12, 2014, 3:48:26 PM, you wrote:
>> >> 
>> >> > On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
>> >> > [...]
>> >> >> > From the look of the code, 0 looks correct to me. Netback won't complain
>> >> >> > about "bad gref" because it has no idea what a "good gref" looks like.
>> >> >> > Only Xen has the knowledge whether a gref is legit. All netback sees is
>> >> >> > the hypercall fails and it tries to push corresponding response to
>> >> >> > netfront. But you can probably safely assume a gref larger than several
>> >> >> > thousands be bad.
>> >> >> 
>> >> >> OK it's indeed netback setting the status to -1:
>> >> >> 
>> >> >> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> >> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
>> >> >> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
>> >> >> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> >> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
>> >> >> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
>> >> >> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
>> >> >> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
>> >> >> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> >> 
>> >> >> Now to find out why ..
>> >> >> 
>> >> 
>> >> > -1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.
>> >> 
>> >> grant copy fails ->> netback sets response status to -1.
>> >> 
>> >> > Assuming the traffic volumn is quite high both way (TX and RX), probably
>> >> > worth checking if RX ring is overflowed. That is, if consumer pointer(s)
>> >> > advance before producer pointer(s).
>> >> 
>> >> In what function ?
>> >> 
>> 
>> > That should be in RX path. Look for "RING_GET_REQUESTS(&vif->rx". That's
>> > the macro to get a slot in RX ring; second argument is consumer index.
>> > You can see if it advances before producer index.
>> 
>> grep -r -i -n "RING_GET_REQUESTS" drivers/net/xen-netback/* delivers 0.0 results ..
>> 

> Sorry, remove the trailing "S". Actually you only need to look at netback.c.

What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
npo->meta_prod
vif->rx.sring->req_prod
vif->pending_prod

to name a few ..
Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?


>> > Wei.
>> 
>> >> > Wei.
>> >> 
>>