Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

From: Sander Eikelenboom <linux@eikelenboom.it>
To: Wei Liu <wei.liu2@citrix.com>
Cc: annie li <annie.li@oracle.com>,
	Paul Durrant <Paul.Durrant@citrix.com>,
	Zoltan Kiss <zoltan.kiss@citrix.com>,
	xen-devel@lists.xen.org
Subject: Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles
Date: Fri, 7 Mar 2014 12:55:18 +0100	[thread overview]
Message-ID: <1554992598.20140307125518@eikelenboom.it> (raw)
In-Reply-To: <20140307111929.GL19620@zion.uk.xensource.com>

Friday, March 7, 2014, 12:19:29 PM, you wrote:

> On Fri, Mar 07, 2014 at 11:33:21AM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> >> >> 
>> >> >> > My suggestion is, if you have a working base line, you can try to setup
>> >> >> > different frontend / backend combination to help narrow down the
>> >> >> > problem.
>> >> >> 
>> >> >> Will see what i can do after the weekend
>> >> >> 
>> A small update
>> 
>> I tried reverting the latest netback / netfront patches .. but to no avail ..
>> Also tried if i could trigger it somehow by using netperf and generating a lot
>> of frags (as that would make it more easily reproduceable).
>> But that was also to no avail .. it seems to only trigger sometimes with my
>> specific workload.
>> 
>> So i took a flight forward by trying out Zoltan's series v6
>> (since it also had changes to the way the network code uses the granttables),
>> got that running overnight applying the same workload as before and
>> i haven't triggered anything yet .. looking good so far :-)
>> 

> Thanks for letting us know. If there's any update don't hesitate to post
> to xen-devel.

*sigh* .. it seems posting to xen-devel triggers things ;-)
back to square one again:

Guest kernel:
Mar  7 11:45:29 backup kernel: [49954.928062] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928081] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928086] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928092] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928096] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928101] net eth0: Need more slots
Mar  7 11:45:29 backup kernel: [49954.928196] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928202] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928206] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928210] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397350] net_ratelimit: 14 callbacks suppressed
Mar  7 11:50:42 backup kernel: [50267.397366] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397372] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397377] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397381] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397386] net eth0: rx->offset: 0, size: 4294967295

Xen:
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 20316163
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 6684675
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 13238275
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 20054019
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 7471105
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 107085839
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 107085839
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:50:42] grant_table.c:1857:d0v4 Bad grant reference 4325379

Will be testing 3.13 vanilla .. see how that works out and if there is a baseline somewhere.

> Wei.