Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[annie li <annie.li@oracle.com>]

On 2014/2/20 19:18, Sander Eikelenboom wrote:
> Thursday, February 20, 2014, 10:49:58 AM, you wrote:
>
>
>> On 2014/2/19 5:25, Sander Eikelenboom wrote:
>>> Hi All,
>>>
>>> I'm currently having some network troubles with Xen and recent linux kernels.
>>>
>>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>>>     I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>>>
>>>     In the guest:
>>>     [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>>>     [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>>>     [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>>>     [57539.859610] net eth0: Need more slots
>>>     [58157.675939] net eth0: Need more slots
>>>     [58725.344712] net eth0: Need more slots
>>>     [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>>>     [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>>>     [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>>>     [61815.849225] net eth0: Need more slots
>> This issue is familiar... and I thought it get fixed.
>>   From original analysis for similar issue I hit before, the root cause
>> is netback still creates response when the ring is full. I remember
>> larger MTU can trigger this issue before, what is the MTU size?
> In dom0 both for the physical nics and the guest vif's MTU=1500
> In domU the eth0 also has MTU=1500.
>
> So it's not jumbo frames .. just everywhere the same plain defaults ..
>
> With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
> I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
> But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.
>
> This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.

The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
Probably the response overlaps the request and grantcopy return error 
when using wrong grant reference, Netback returns resp->status with 
||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend.
Would it be possible to print log in xenvif_rx_action of netback to see 
whether something wrong with max slots and used slots?

Thanks
Annie

>
> Will keep you posted when it triggers again with the extra info in the warn.
>
> --
> Sander
>
>
>
>> Thanks
>> Annie
>>>     Xen reports:
>>>     (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>>>     (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>>>     (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>>>     (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>>>     (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>>>     (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>>>     (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>>>     (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>>>     (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>>>     (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>     (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
>>>
>>>
>>>
>>> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>>>     - i can ping the guests from dom0
>>>     - i can ping dom0 from the guests
>>>     - But i can't ssh or access things by http
>>>     - I don't see any relevant error messages ...
>>>     - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>>>       (that previously worked fine)
>>>
>>> --
>>>
>>> Sander
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>
>