Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Hi All,

I'm currently having some network troubles with Xen and recent linux kernels.

- When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
  I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html

  In the guest:
  [57539.859584] net eth0: rx->offset: 0, size: 4294967295
  [57539.859599] net eth0: rx->offset: 0, size: 4294967295
  [57539.859605] net eth0: rx->offset: 0, size: 4294967295
  [57539.859610] net eth0: Need more slots
  [58157.675939] net eth0: Need more slots
  [58725.344712] net eth0: Need more slots
  [61815.849180] net eth0: rx->offset: 0, size: 4294967295
  [61815.849205] net eth0: rx->offset: 0, size: 4294967295
  [61815.849216] net eth0: rx->offset: 0, size: 4294967295
  [61815.849225] net eth0: Need more slots

  Xen reports:
  (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
  (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
  (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
  (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
  (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
  (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
  (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
  (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
  (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
  (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
  (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
  (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
  (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
  (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
  (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
  (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
  (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
  (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
  (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
  (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
  (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377



Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
  - i can ping the guests from dom0
  - i can ping dom0 from the guests
  - But i can't ssh or access things by http
  - I don't see any relevant error messages ...
  - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
    (that previously worked fine)

--

Sander

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Wei Liu]

On Tue, Feb 18, 2014 at 10:25:13PM +0100, Sander Eikelenboom wrote:
> Hi All,
> 
> I'm currently having some network troubles with Xen and recent linux kernels.
> 
> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>   I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
> 

I *think* that issue should've been fixed -- not with that patch but with
a different one. *sigh*

>   In the guest:
>   [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>   [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>   [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>   [57539.859610] net eth0: Need more slots
>   [58157.675939] net eth0: Need more slots
>   [58725.344712] net eth0: Need more slots
>   [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>   [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>   [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>   [61815.849225] net eth0: Need more slots
> 
>   Xen reports:
>   (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>   (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>   (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>   (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>   (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>   (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>   (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>   (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>   (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>   (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>   (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>   (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>   (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>   (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>   (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>   (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>   (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>   (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>   (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>   (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>   (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
> 

Judging from the log above I presume it happens after DomU has been
running for quite a while? Could you elaborate on the exact steps to
reproduce?

Does it happen when you use 3.13 as backend?

> 
> 
> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>   - i can ping the guests from dom0
>   - i can ping dom0 from the guests
>   - But i can't ssh or access things by http
>   - I don't see any relevant error messages ...
>   - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>     (that previously worked fine)

I think I am able to reproduce this. I'm looking into it now. Luckily
there isn't that many changesets between 3.13 and 3.14.


Wei.

> 
> --
> 
> Sander

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Wednesday, February 19, 2014, 12:28:58 PM, you wrote:

> On Tue, Feb 18, 2014 at 10:25:13PM +0100, Sander Eikelenboom wrote:
>> Hi All,
>> 
>> I'm currently having some network troubles with Xen and recent linux kernels.
>> 
>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>>   I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>> 

> I *think* that issue should've been fixed -- not with that patch but with
> a different one. *sigh*

>>   In the guest:
>>   [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>>   [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>>   [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>>   [57539.859610] net eth0: Need more slots
>>   [58157.675939] net eth0: Need more slots
>>   [58725.344712] net eth0: Need more slots
>>   [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>>   [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>>   [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>>   [61815.849225] net eth0: Need more slots
>> 
>>   Xen reports:
>>   (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>>   (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>>   (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>>   (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>>   (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>>   (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>>   (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>>   (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>>   (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>>   (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>>   (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>>   (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>>   (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>>   (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>>   (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>>   (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>>   (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>>   (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>>   (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>   (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>   (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
>> 

> Judging from the log above I presume it happens after DomU has been
> running for quite a while? Could you elaborate on the exact steps to
> reproduce?

> Does it happen when you use 3.13 as backend?

yes i happens when it runs for a while (could be due to the fragment packet problems that seems
to be a nasty problem that reoccurs from time to time.
But i just noticed in the specific VM that caused this i had TSO switched off (to see if it would effect the case below which it didn't)
so it could be due to that. I will see if i can figure that out and report back.


>> 
>> 
>> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>>   - i can ping the guests from dom0
>>   - i can ping dom0 from the guests
>>   - But i can't ssh or access things by http
>>   - I don't see any relevant error messages ...
>>   - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>>     (that previously worked fine)

> I think I am able to reproduce this. I'm looking into it now. Luckily
> there isn't that many changesets between 3.13 and 3.14.

>From what i can remember it's somehwere from early in the merge window, but i didn't get to report it due quite some other bugs
in different subsystems this mergewindow.

> Wei.

>> 
>> --
>> 
>> Sander

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[annie li]

On 2014/2/19 5:25, Sander Eikelenboom wrote:
> Hi All,
>
> I'm currently having some network troubles with Xen and recent linux kernels.
>
> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>    I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>
>    In the guest:
>    [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>    [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>    [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>    [57539.859610] net eth0: Need more slots
>    [58157.675939] net eth0: Need more slots
>    [58725.344712] net eth0: Need more slots
>    [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>    [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>    [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>    [61815.849225] net eth0: Need more slots

This issue is familiar... and I thought it get fixed.
 From original analysis for similar issue I hit before, the root cause 
is netback still creates response when the ring is full. I remember 
larger MTU can trigger this issue before, what is the MTU size?

Thanks
Annie
>
>    Xen reports:
>    (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>    (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>    (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>    (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>    (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>    (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>    (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>    (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>    (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>    (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>    (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>    (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>    (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>    (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>    (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>    (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>    (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>    (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>    (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>    (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>    (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
>
>
>
> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>    - i can ping the guests from dom0
>    - i can ping dom0 from the guests
>    - But i can't ssh or access things by http
>    - I don't see any relevant error messages ...
>    - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>      (that previously worked fine)
>
> --
>
> Sander
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Thursday, February 20, 2014, 10:49:58 AM, you wrote:


> On 2014/2/19 5:25, Sander Eikelenboom wrote:
>> Hi All,
>>
>> I'm currently having some network troubles with Xen and recent linux kernels.
>>
>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>>    I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>>
>>    In the guest:
>>    [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>>    [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>>    [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>>    [57539.859610] net eth0: Need more slots
>>    [58157.675939] net eth0: Need more slots
>>    [58725.344712] net eth0: Need more slots
>>    [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>>    [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>>    [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>>    [61815.849225] net eth0: Need more slots

> This issue is familiar... and I thought it get fixed.
>  From original analysis for similar issue I hit before, the root cause 
> is netback still creates response when the ring is full. I remember 
> larger MTU can trigger this issue before, what is the MTU size?

In dom0 both for the physical nics and the guest vif's MTU=1500
In domU the eth0 also has MTU=1500.

So it's not jumbo frames .. just everywhere the same plain defaults ..

With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.

This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.

Will keep you posted when it triggers again with the extra info in the warn.

--
Sander



> Thanks
> Annie
>>
>>    Xen reports:
>>    (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>>    (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>>    (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>>    (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>>    (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>>    (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>>    (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>>    (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>>    (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>>    (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>>    (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>>    (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>>    (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>>    (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>>    (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>>    (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>>    (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>>    (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>>    (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>    (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>    (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
>>
>>
>>
>> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>>    - i can ping the guests from dom0
>>    - i can ping dom0 from the guests
>>    - But i can't ssh or access things by http
>>    - I don't see any relevant error messages ...
>>    - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>>      (that previously worked fine)
>>
>> --
>>
>> Sander
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[annie li]

On 2014/2/20 19:18, Sander Eikelenboom wrote:
> Thursday, February 20, 2014, 10:49:58 AM, you wrote:
>
>
>> On 2014/2/19 5:25, Sander Eikelenboom wrote:
>>> Hi All,
>>>
>>> I'm currently having some network troubles with Xen and recent linux kernels.
>>>
>>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>>>     I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>>>
>>>     In the guest:
>>>     [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>>>     [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>>>     [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>>>     [57539.859610] net eth0: Need more slots
>>>     [58157.675939] net eth0: Need more slots
>>>     [58725.344712] net eth0: Need more slots
>>>     [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>>>     [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>>>     [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>>>     [61815.849225] net eth0: Need more slots
>> This issue is familiar... and I thought it get fixed.
>>   From original analysis for similar issue I hit before, the root cause
>> is netback still creates response when the ring is full. I remember
>> larger MTU can trigger this issue before, what is the MTU size?
> In dom0 both for the physical nics and the guest vif's MTU=1500
> In domU the eth0 also has MTU=1500.
>
> So it's not jumbo frames .. just everywhere the same plain defaults ..
>
> With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
> I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
> But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.
>
> This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.

The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
Probably the response overlaps the request and grantcopy return error 
when using wrong grant reference, Netback returns resp->status with 
||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend.
Would it be possible to print log in xenvif_rx_action of netback to see 
whether something wrong with max slots and used slots?

Thanks
Annie

>
> Will keep you posted when it triggers again with the extra info in the warn.
>
> --
> Sander
>
>
>
>> Thanks
>> Annie
>>>     Xen reports:
>>>     (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>>>     (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>>>     (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>>>     (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>>>     (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>>>     (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>>>     (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>>>     (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>>>     (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>>>     (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>     (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
>>>
>>>
>>>
>>> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>>>     - i can ping the guests from dom0
>>>     - i can ping dom0 from the guests
>>>     - But i can't ssh or access things by http
>>>     - I don't see any relevant error messages ...
>>>     - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>>>       (that previously worked fine)
>>>
>>> --
>>>
>>> Sander
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>
>

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Friday, February 21, 2014, 7:32:08 AM, you wrote:


> On 2014/2/20 19:18, Sander Eikelenboom wrote:
>> Thursday, February 20, 2014, 10:49:58 AM, you wrote:
>>
>>
>>> On 2014/2/19 5:25, Sander Eikelenboom wrote:
>>>> Hi All,
>>>>
>>>> I'm currently having some network troubles with Xen and recent linux kernels.
>>>>
>>>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>>>>     I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>>>>
>>>>     In the guest:
>>>>     [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>>>>     [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>>>>     [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>>>>     [57539.859610] net eth0: Need more slots
>>>>     [58157.675939] net eth0: Need more slots
>>>>     [58725.344712] net eth0: Need more slots
>>>>     [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>>>>     [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>>>>     [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>>>>     [61815.849225] net eth0: Need more slots
>>> This issue is familiar... and I thought it get fixed.
>>>   From original analysis for similar issue I hit before, the root cause
>>> is netback still creates response when the ring is full. I remember
>>> larger MTU can trigger this issue before, what is the MTU size?
>> In dom0 both for the physical nics and the guest vif's MTU=1500
>> In domU the eth0 also has MTU=1500.
>>
>> So it's not jumbo frames .. just everywhere the same plain defaults ..
>>
>> With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
>> I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
>> But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.
>>
>> This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.

> The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
> Probably the response overlaps the request and grantcopy return error 
> when using wrong grant reference, Netback returns resp->status with 
> ||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend.
> Would it be possible to print log in xenvif_rx_action of netback to see 
> whether something wrong with max slots and used slots?

> Thanks
> Annie

Looking more closely it are perhaps 2 different issues ... the bad grant references do not happen
at the same time as the netfront messages in the guest.

I added some debugpatches to the kernel netback, netfront and xen granttable code (see below)
One of the things was to simplify the code for the debug key to print the granttables, the present code
takes too long to execute and brings down the box due to stalls and NMI's. So it now only prints
the nr of entries per domain.


Issue 1: grant_table.c:1858:d0 Bad grant reference

After running the box for just one night (with 15 VM's) i get these mentions of "Bad grant reference".
The maptrack also seems to increase quite fast and the number of entries seem to have gone up quite fast as well.

Most domains have just one disk(blkfront/blkback) and one nic, a few have a second disk.
The blk drivers use persistent grants so i would assume it would reuse those and not increase it (by much).

Domain 1 seems to have increased it's nr_grant_entries from 2048 to 3072 somewhere this night.
Domain 7 is the domain that happens to give the netfront messages.

I also don't get why it is reporting the "Bad grant reference" for domain 0, which seems to have 0 active entries ..
Also is this amount of grant entries "normal" ? or could it be a leak somewhere ?

(XEN) [2014-02-26 00:00:38] grant_table.c:1250:d1 Expanding dom (1) grant table from (4) to (5) frames.
(XEN) [2014-02-26 00:00:38] grant_table.c:1250:d1 Expanding dom (1) grant table from (5) to (6) frames.
(XEN) [2014-02-26 00:00:38] grant_table.c:290:d0 Increased maptrack size to 13/256 frames
(XEN) [2014-02-26 00:01:13] grant_table.c:290:d0 Increased maptrack size to 14/256 frames
(XEN) [2014-02-26 04:02:55] grant_table.c:1858:d0 Bad grant reference 4325377 | 2048 | 1 | 0
(XEN) [2014-02-26 04:15:33] grant_table.c:290:d0 Increased maptrack size to 15/256 frames
(XEN) [2014-02-26 04:15:53] grant_table.c:290:d0 Increased maptrack size to 16/256 frames
(XEN) [2014-02-26 04:15:56] grant_table.c:290:d0 Increased maptrack size to 17/256 frames
(XEN) [2014-02-26 04:15:56] grant_table.c:290:d0 Increased maptrack size to 18/256 frames
(XEN) [2014-02-26 04:15:57] grant_table.c:290:d0 Increased maptrack size to 19/256 frames
(XEN) [2014-02-26 04:15:57] grant_table.c:290:d0 Increased maptrack size to 20/256 frames
(XEN) [2014-02-26 04:15:59] grant_table.c:290:d0 Increased maptrack size to 21/256 frames
(XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 22/256 frames
(XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 23/256 frames
(XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 24/256 frames
(XEN) [2014-02-26 04:16:10] grant_table.c:290:d0 Increased maptrack size to 25/256 frames
(XEN) [2014-02-26 04:16:10] grant_table.c:290:d0 Increased maptrack size to 26/256 frames
(XEN) [2014-02-26 04:16:17] grant_table.c:290:d0 Increased maptrack size to 27/256 frames
(XEN) [2014-02-26 04:16:20] grant_table.c:290:d0 Increased maptrack size to 28/256 frames
(XEN) [2014-02-26 04:16:56] grant_table.c:290:d0 Increased maptrack size to 29/256 frames
(XEN) [2014-02-26 05:15:04] grant_table.c:290:d0 Increased maptrack size to 30/256 frames
(XEN) [2014-02-26 05:15:05] grant_table.c:290:d0 Increased maptrack size to 31/256 frames
(XEN) [2014-02-26 05:21:15] grant_table.c:1858:d0 Bad grant reference 107085839 | 2048 | 1 | 0
(XEN) [2014-02-26 05:29:47] grant_table.c:1858:d0 Bad grant reference 268435460 | 2048 | 1 | 0
(XEN) [2014-02-26 07:53:20] gnttab_usage_print_all [ key 'g' pressed
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    0 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    0 active entries: 0
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 (v1) nr_grant_entries: 3072
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 active entries: 2117
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 active entries: 1061
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 active entries: 1045
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 active entries: 1060
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 active entries: 709
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 (v1) nr_grant_entries: 2048
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 (v1)
(XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 active entries: 163
(XEN) [2014-02-26 07:53:20] gnttab_usage_print_all ] done
(XEN) [2014-02-26 07:55:09] grant_table.c:1858:d0 Bad grant reference 4325377 | 2048 | 1 | 0
(XEN) [2014-02-26 08:37:16] grant_table.c:1858:d0 Bad grant reference 268435460 | 2048 | 1 | 0



Issue 2: net eth0: rx->offset: 0, size: xxxxxxxxxx

In the guest (domain 7):

Feb 26 08:55:09 backup kernel: [39258.090375] net eth0: rx->offset: 0, size: 4294967295
Feb 26 08:55:09 backup kernel: [39258.090392] net eth0: me here .. cons:15177803 slots:1 rp:15177807 max:18 err:0 rx->id:74 rx->offset:0 size:4294967295 ref:533
Feb 26 08:55:09 backup kernel: [39258.090401] net eth0: rx->offset: 0, size: 4294967295
Feb 26 08:55:09 backup kernel: [39258.090406] net eth0: me here .. cons:15177803 slots:2 rp:15177807 max:18 err:-22 rx->id:76 rx->offset:0 size:4294967295 ref:686
Feb 26 08:55:09 backup kernel: [39258.090415] net eth0: rx->offset: 0, size: 4294967295
Feb 26 08:55:09 backup kernel: [39258.090420] net eth0: me here .. cons:15177803 slots:3 rp:15177807 max:18 err:-22 rx->id:77 rx->offset:0 size:4294967295 ref:571

In dom0 i don't see any specific netback warnings related to this domain at this specific times, the printk's i added do trigger quite some times but these are probably not
errorneous, but they seem to only occur on the vif of domain 7 (probably the only domain that is swamping the network by doing rsync and webdavs and causes some fragmented packets)

Feb 26 08:53:20 serveerstertje kernel: [39324.917255] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15101115 cons:15101112 j:8
Feb 26 08:53:56 serveerstertje kernel: [39361.001436] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15127649 cons:15127648 j:13
Feb 26 08:54:00 serveerstertje kernel: [39364.725613] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15130263 cons:15130261 j:2
Feb 26 08:54:04 serveerstertje kernel: [39368.739504] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15133143 cons:15133141 j:0
Feb 26 08:54:20 serveerstertje kernel: [39384.665044] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15144113 cons:15144112 j:0
Feb 26 08:54:29 serveerstertje kernel: [39393.569871] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15150203 cons:15150200 j:0
Feb 26 08:54:40 serveerstertje kernel: [39404.586566] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15157706 cons:15157704 j:12
Feb 26 08:54:56 serveerstertje kernel: [39420.759769] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15168839 cons:15168835 j:0
Feb 26 08:54:56 serveerstertje kernel: [39421.001372] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15169002 cons:15168999 j:8
Feb 26 08:55:00 serveerstertje kernel: [39424.515073] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15171450 cons:15171447 j:0
Feb 26 08:55:10 serveerstertje kernel: [39435.154510] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15178773 cons:15178770 j:1
Feb 26 08:56:19 serveerstertje kernel: [39504.195908] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15227444 cons:15227444 j:0
Feb 26 08:57:39 serveerstertje kernel: [39583.799392] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15283346 cons:15283344 j:8
Feb 26 08:57:55 serveerstertje kernel: [39599.517673] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15293937 cons:15293935 j:0
Feb 26 08:58:07 serveerstertje kernel: [39612.156622] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15302891 cons:15302889 j:19
Feb 26 08:58:07 serveerstertje kernel: [39612.400907] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15303034 cons:15303033 j:0
Feb 26 08:58:18 serveerstertje kernel: [39623.439383] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15310915 cons:15310911 j:0
Feb 26 08:58:39 serveerstertje kernel: [39643.521808] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15324769 cons:15324766 j:1

Feb 26 09:27:07 serveerstertje kernel: [41351.622501] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16502932 cons:16502932 j:8
Feb 26 09:27:19 serveerstertje kernel: [41363.541003] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:16510837 cons:16510834 j:7
Feb 26 09:27:23 serveerstertje kernel: [41368.133306] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16513940 cons:16513937 j:0
Feb 26 09:27:43 serveerstertje kernel: [41388.025147] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16527870 cons:16527868 j:0
Feb 26 09:27:47 serveerstertje kernel: [41391.530802] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:16530437 cons:16530437 j:7
Feb 26 09:27:51 serveerstertje kernel: [41395.521166] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533320 cons:16533317 j:6
Feb 26 09:27:51 serveerstertje kernel: [41395.767066] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533469 cons:16533469 j:0
Feb 26 09:27:51 serveerstertje kernel: [41395.802319] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:1 GSO:0 vif->rx_last_skb_slots:0 nr_frags:0 prod:16533533 cons:16533533 j:24
Feb 26 09:27:51 serveerstertje kernel: [41395.837456] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:1 GSO:0 vif->rx_last_skb_slots:0 nr_frags:0 prod:16533534 cons:16533534 j:1
Feb 26 09:27:51 serveerstertje kernel: [41395.872587] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533597 cons:16533596 j:25
Feb 26 09:27:51 serveerstertje kernel: [41396.192784] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533833 cons:16533832 j:3
Feb 26 09:27:51 serveerstertje kernel: [41396.235611] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533890 cons:16533890 j:30
Feb 26 09:27:51 serveerstertje kernel: [41396.271047] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533898 cons:16533896 j:3


--
Sander





>>
>> Will keep you posted when it triggers again with the extra info in the warn.
>>
>> --
>> Sander
>>
>>
>>
>>> Thanks
>>> Annie
>>>>     Xen reports:
>>>>     (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>>>>     (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>>>>     (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>>>>     (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>>>>     (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>>>>     (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>>>>     (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>>>>     (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>>>>     (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>>>>     (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>>     (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
>>>>
>>>>
>>>>
>>>> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>>>>     - i can ping the guests from dom0
>>>>     - i can ping dom0 from the guests
>>>>     - But i can't ssh or access things by http
>>>>     - I don't see any relevant error messages ...
>>>>     - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>>>>       (that previously worked fine)
>>>>
>>>> --
>>>>
>>>> Sander
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xen.org
>>>> http://lists.xen.org/xen-devel
>>
>>



diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 4fc46eb..4d720b4 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1667,6 +1667,8 @@ skip_vfb:
                 b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_STD;
             } else if (!strcmp(buf, "cirrus")) {
                 b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_CIRRUS;
+            } else if (!strcmp(buf, "none")) {
+                b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_NONE;
             } else {
                 fprintf(stderr, "Unknown vga \"%s\" specified\n", buf);
                 exit(1);
diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index 107b000..ab56927 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -265,9 +265,10 @@ get_maptrack_handle(
     while ( unlikely((handle = __get_maptrack_handle(lgt)) == -1) )
     {
         nr_frames = nr_maptrack_frames(lgt);
-        if ( nr_frames >= max_nr_maptrack_frames() )
+        if ( nr_frames >= max_nr_maptrack_frames() ){
+                 gdprintk(XENLOG_INFO, "Already at max maptrack size: %u/%u frames\n",nr_frames, max_nr_maptrack_frames());
             break;
-
+       }
         new_mt = alloc_xenheap_page();
         if ( !new_mt )
             break;
@@ -285,8 +286,8 @@ get_maptrack_handle(
         smp_wmb();
         lgt->maptrack_limit      = new_mt_limit;

-        gdprintk(XENLOG_INFO, "Increased maptrack size to %u frames\n",
-                 nr_frames + 1);
+        gdprintk(XENLOG_INFO, "Increased maptrack size to %u/%u frames\n",
+                 nr_frames + 1, max_nr_maptrack_frames());
     }

     spin_unlock(&lgt->lock);
@@ -1854,7 +1855,7 @@ __acquire_grant_for_copy(

     if ( unlikely(gref >= nr_grant_entries(rgt)) )
         PIN_FAIL(unlock_out, GNTST_bad_gntref,
-                 "Bad grant reference %ld\n", gref);
+                 "Bad grant reference %ld | %d | %d | %d \n", gref, nr_grant_entries(rgt), rgt->gt_version, ldom);

     act = &active_entry(rgt, gref);
     shah = shared_entry_header(rgt, gref);
@@ -2830,15 +2831,19 @@ static void gnttab_usage_print(struct domain *rd)
     int first = 1;
     grant_ref_t ref;
     struct grant_table *gt = rd->grant_table;
-
+    unsigned int active=0;
+/*
     printk("      -------- active --------       -------- shared --------\n");
     printk("[ref] localdom mfn      pin          localdom gmfn     flags\n");
-
+*/
     spin_lock(&gt->lock);

     if ( gt->gt_version == 0 )
         goto out;

+    printk("grant-table for remote domain:%5d (v%d) nr_grant_entries: %d\n",
+                   rd->domain_id, gt->gt_version, nr_grant_entries(gt));
+
     for ( ref = 0; ref != nr_grant_entries(gt); ref++ )
     {
         struct active_grant_entry *act;
@@ -2875,19 +2880,22 @@ static void gnttab_usage_print(struct domain *rd)
                    rd->domain_id, gt->gt_version);
             first = 0;
         }
-
+        active++;
         /*      [ddd]    ddddd 0xXXXXXX 0xXXXXXXXX      ddddd 0xXXXXXX 0xXX */
-        printk("[%3d]    %5d 0x%06lx 0x%08x      %5d 0x%06"PRIx64" 0x%02x\n",
-               ref, act->domid, act->frame, act->pin,
-               sha->domid, frame, status);
+        /* printk("[%3d]    %5d 0x%06lx 0x%08x      %5d 0x%06"PRIx64" 0x%02x\n", ref, act->domid, act->frame, act->pin, sha->domid, frame, status); */
     }

  out:
     spin_unlock(&gt->lock);

+    printk("grant-table for remote domain:%5d active entries: %d\n",
+                   rd->domain_id, active);
+/*
     if ( first )
         printk("grant-table for remote domain:%5d ... "
                "no active grant table entries\n", rd->domain_id);
+*/
+
 }

 static void gnttab_usage_print_all(unsigned char key)






diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index e5284bc..6d93358 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -482,20 +482,23 @@ static void xenvif_rx_action(struct xenvif *vif)
                .meta  = vif->meta,
        };

+       int j=0;
+
        skb_queue_head_init(&rxq);

        while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
                RING_IDX max_slots_needed;
                int i;
+               int nr_frags;

                /* We need a cheap worse case estimate for the number of
                 * slots we'll use.
                 */

                max_slots_needed = DIV_ROUND_UP(offset_in_page(skb->data) +
-                                               skb_headlen(skb),
-                                               PAGE_SIZE);
-               for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+                                               skb_headlen(skb), PAGE_SIZE);
+               nr_frags = skb_shinfo(skb)->nr_frags;
+               for (i = 0; i < nr_frags; i++) {
                        unsigned int size;
                        size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
                        max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
@@ -508,6 +511,9 @@ static void xenvif_rx_action(struct xenvif *vif)
                if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
                        skb_queue_head(&vif->rx_queue, skb);
                        need_to_notify = true;
+                       if (net_ratelimit())
+                               netdev_err(vif->dev, "!?!?!?! skb may not fit .. bail out now max_slots_needed:%d GSO:%d vif->rx_last_skb_slots:%d nr_frags:%d prod:%d cons:%d j:%d\n",
+                                       max_slots_needed, (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 || skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6) ? 1 : 0, vif->rx_last_skb_slots, nr_frags,vif->rx.sring->req_prod,vif->rx.req_cons,j);
                        vif->rx_last_skb_slots = max_slots_needed;
                        break;
                } else
@@ -518,6 +524,7 @@ static void xenvif_rx_action(struct xenvif *vif)
                BUG_ON(sco->meta_slots_used > max_slots_needed);

                __skb_queue_tail(&rxq, skb);
+               j++;
        }

        BUG_ON(npo.meta_prod > ARRAY_SIZE(vif->meta));
@@ -541,7 +548,7 @@ static void xenvif_rx_action(struct xenvif *vif)
                        resp->offset = vif->meta[npo.meta_cons].gso_size;
                        resp->id = vif->meta[npo.meta_cons].id;
                        resp->status = sco->meta_slots_used;
-
+
                        npo.meta_cons++;
                        sco->meta_slots_used--;
                }
@@ -705,7 +712,7 @@ static int xenvif_count_requests(struct xenvif *vif,
                 */
                if (!drop_err && slots >= XEN_NETBK_LEGACY_SLOTS_MAX) {
                        if (net_ratelimit())
-                               netdev_dbg(vif->dev,
+                               netdev_err(vif->dev,
                                           "Too many slots (%d) exceeding limit (%d), dropping packet\n",
                                           slots, XEN_NETBK_LEGACY_SLOTS_MAX);
                        drop_err = -E2BIG;
@@ -728,7 +735,7 @@ static int xenvif_count_requests(struct xenvif *vif,
                 */
                if (!drop_err && txp->size > first->size) {
                        if (net_ratelimit())
-                               netdev_dbg(vif->dev,
+                               netdev_err(vif->dev,
                                           "Invalid tx request, slot size %u > remaining size %u\n",
                                           txp->size, first->size);
                        drop_err = -EIO;



diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index f9daa9e..67d5221 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -753,6 +753,7 @@ static int xennet_get_responses(struct netfront_info *np,
                        if (net_ratelimit())
                                dev_warn(dev, "rx->offset: %x, size: %u\n",
                                         rx->offset, rx->status);
+                               dev_warn(dev, "me here .. cons:%d slots:%d rp:%d max:%d err:%d rx->id:%d rx->offset:%x size:%u ref:%ld\n",cons,slots,rp,max,err,rx->id, rx->offset, rx->status, ref);
                        xennet_move_rx_slot(np, skb, ref);
                        err = -EINVAL;
                        goto next;
@@ -784,7 +785,7 @@ next:

                if (cons + slots == rp) {
                        if (net_ratelimit())
-                               dev_warn(dev, "Need more slots\n");
+                               dev_warn(dev, "Need more slots cons:%d slots:%d rp:%d max:%d err:%d rx-id:%d rx->offset:%x size:%u ref:%ld\n",cons,slots,rp,max,err,rx->id, rx->offset, rx->status, ref);
                        err = -ENOENT;
                        break;
                }
@@ -803,7 +804,6 @@ next:

        if (unlikely(err))
                np->rx.rsp_cons = cons + slots;
-
        return err;
 }

@@ -907,6 +907,7 @@ static int handle_incoming_queue(struct net_device *dev,

                /* Ethernet work: Delayed to here as it peeks the header. */
                skb->protocol = eth_type_trans(skb, dev);
+               skb_reset_network_header(skb);

                if (checksum_setup(dev, skb)) {
                        kfree_skb(skb);

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Wednesday, February 26, 2014, 10:14:42 AM, you wrote:


> Friday, February 21, 2014, 7:32:08 AM, you wrote:


>> On 2014/2/20 19:18, Sander Eikelenboom wrote:
>>> Thursday, February 20, 2014, 10:49:58 AM, you wrote:
>>>
>>>
>>>> On 2014/2/19 5:25, Sander Eikelenboom wrote:
>>>>> Hi All,
>>>>>
>>>>> I'm currently having some network troubles with Xen and recent linux kernels.
>>>>>
>>>>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>>>>>     I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>>>>>
>>>>>     In the guest:
>>>>>     [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>>>>>     [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>>>>>     [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>>>>>     [57539.859610] net eth0: Need more slots
>>>>>     [58157.675939] net eth0: Need more slots
>>>>>     [58725.344712] net eth0: Need more slots
>>>>>     [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>>>>>     [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>>>>>     [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>>>>>     [61815.849225] net eth0: Need more slots
>>>> This issue is familiar... and I thought it get fixed.
>>>>   From original analysis for similar issue I hit before, the root cause
>>>> is netback still creates response when the ring is full. I remember
>>>> larger MTU can trigger this issue before, what is the MTU size?
>>> In dom0 both for the physical nics and the guest vif's MTU=1500
>>> In domU the eth0 also has MTU=1500.
>>>
>>> So it's not jumbo frames .. just everywhere the same plain defaults ..
>>>
>>> With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
>>> I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
>>> But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.
>>>
>>> This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.

>> The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
>> Probably the response overlaps the request and grantcopy return error 
>> when using wrong grant reference, Netback returns resp->status with 
>> ||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend.
>> Would it be possible to print log in xenvif_rx_action of netback to see 
>> whether something wrong with max slots and used slots?

>> Thanks
>> Annie

> Looking more closely it are perhaps 2 different issues ... the bad grant references do not happen
> at the same time as the netfront messages in the guest.

> I added some debugpatches to the kernel netback, netfront and xen granttable code (see below)
> One of the things was to simplify the code for the debug key to print the granttables, the present code
> takes too long to execute and brings down the box due to stalls and NMI's. So it now only prints
> the nr of entries per domain.


> Issue 1: grant_table.c:1858:d0 Bad grant reference

> After running the box for just one night (with 15 VM's) i get these mentions of "Bad grant reference".
> The maptrack also seems to increase quite fast and the number of entries seem to have gone up quite fast as well.

> Most domains have just one disk(blkfront/blkback) and one nic, a few have a second disk.
> The blk drivers use persistent grants so i would assume it would reuse those and not increase it (by much).

> Domain 1 seems to have increased it's nr_grant_entries from 2048 to 3072 somewhere this night.
> Domain 7 is the domain that happens to give the netfront messages.

> I also don't get why it is reporting the "Bad grant reference" for domain 0, which seems to have 0 active entries ..
> Also is this amount of grant entries "normal" ? or could it be a leak somewhere ?

> (XEN) [2014-02-26 00:00:38] grant_table.c:1250:d1 Expanding dom (1) grant table from (4) to (5) frames.
> (XEN) [2014-02-26 00:00:38] grant_table.c:1250:d1 Expanding dom (1) grant table from (5) to (6) frames.
> (XEN) [2014-02-26 00:00:38] grant_table.c:290:d0 Increased maptrack size to 13/256 frames
> (XEN) [2014-02-26 00:01:13] grant_table.c:290:d0 Increased maptrack size to 14/256 frames
> (XEN) [2014-02-26 04:02:55] grant_table.c:1858:d0 Bad grant reference 4325377 | 2048 | 1 | 0
> (XEN) [2014-02-26 04:15:33] grant_table.c:290:d0 Increased maptrack size to 15/256 frames
> (XEN) [2014-02-26 04:15:53] grant_table.c:290:d0 Increased maptrack size to 16/256 frames
> (XEN) [2014-02-26 04:15:56] grant_table.c:290:d0 Increased maptrack size to 17/256 frames
> (XEN) [2014-02-26 04:15:56] grant_table.c:290:d0 Increased maptrack size to 18/256 frames
> (XEN) [2014-02-26 04:15:57] grant_table.c:290:d0 Increased maptrack size to 19/256 frames
> (XEN) [2014-02-26 04:15:57] grant_table.c:290:d0 Increased maptrack size to 20/256 frames
> (XEN) [2014-02-26 04:15:59] grant_table.c:290:d0 Increased maptrack size to 21/256 frames
> (XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 22/256 frames
> (XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 23/256 frames
> (XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 24/256 frames
> (XEN) [2014-02-26 04:16:10] grant_table.c:290:d0 Increased maptrack size to 25/256 frames
> (XEN) [2014-02-26 04:16:10] grant_table.c:290:d0 Increased maptrack size to 26/256 frames
> (XEN) [2014-02-26 04:16:17] grant_table.c:290:d0 Increased maptrack size to 27/256 frames
> (XEN) [2014-02-26 04:16:20] grant_table.c:290:d0 Increased maptrack size to 28/256 frames
> (XEN) [2014-02-26 04:16:56] grant_table.c:290:d0 Increased maptrack size to 29/256 frames
> (XEN) [2014-02-26 05:15:04] grant_table.c:290:d0 Increased maptrack size to 30/256 frames
> (XEN) [2014-02-26 05:15:05] grant_table.c:290:d0 Increased maptrack size to 31/256 frames
> (XEN) [2014-02-26 05:21:15] grant_table.c:1858:d0 Bad grant reference 107085839 | 2048 | 1 | 0
> (XEN) [2014-02-26 05:29:47] grant_table.c:1858:d0 Bad grant reference 268435460 | 2048 | 1 | 0
> (XEN) [2014-02-26 07:53:20] gnttab_usage_print_all [ key 'g' pressed
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    0 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    0 active entries: 0
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 (v1) nr_grant_entries: 3072
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 active entries: 2117
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 active entries: 1061
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 active entries: 1045
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 active entries: 1060
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 active entries: 709
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 (v1) nr_grant_entries: 2048
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 (v1)
> (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 active entries: 163
> (XEN) [2014-02-26 07:53:20] gnttab_usage_print_all ] done
> (XEN) [2014-02-26 07:55:09] grant_table.c:1858:d0 Bad grant reference 4325377 | 2048 | 1 | 0
> (XEN) [2014-02-26 08:37:16] grant_table.c:1858:d0 Bad grant reference 268435460 | 2048 | 1 | 0



> Issue 2: net eth0: rx->offset: 0, size: xxxxxxxxxx

> In the guest (domain 7):

> Feb 26 08:55:09 backup kernel: [39258.090375] net eth0: rx->offset: 0, size: 4294967295
> Feb 26 08:55:09 backup kernel: [39258.090392] net eth0: me here .. cons:15177803 slots:1 rp:15177807 max:18 err:0 rx->id:74 rx->offset:0 size:4294967295 ref:533
> Feb 26 08:55:09 backup kernel: [39258.090401] net eth0: rx->offset: 0, size: 4294967295
> Feb 26 08:55:09 backup kernel: [39258.090406] net eth0: me here .. cons:15177803 slots:2 rp:15177807 max:18 err:-22 rx->id:76 rx->offset:0 size:4294967295 ref:686
> Feb 26 08:55:09 backup kernel: [39258.090415] net eth0: rx->offset: 0, size: 4294967295
> Feb 26 08:55:09 backup kernel: [39258.090420] net eth0: me here .. cons:15177803 slots:3 rp:15177807 max:18 err:-22 rx->id:77 rx->offset:0 size:4294967295 ref:571

> In dom0 i don't see any specific netback warnings related to this domain at this specific times, the printk's i added do trigger quite some times but these are probably not
> errorneous, but they seem to only occur on the vif of domain 7 (probably the only domain that is swamping the network by doing rsync and webdavs and causes some fragmented packets)

Another addition ... the guest doesn't shutdown anymore on "xl shutdown" .. it just does .. erhmm nothing .. (tried multiple times)
After that i ssh'ed into the guest and did a "halt -p" ... the guest shutted down .. but the guest remained in xl list in blocked state ..
Doing a "xl console" shows:

[30024.559656] net eth0: me here .. cons:8713451 slots:1 rp:8713462 max:18 err:0 rx->id:234 rx->offset:0 size:4294967295 ref:-131941395332550
[30024.559666] net eth0: rx->offset: 0, size: 4294967295
[30024.559671] net eth0: me here .. cons:8713451 slots:2 rp:8713462 max:18 err:-22 rx->id:236 rx->offset:0 size:4294967295 ref:-131941395332504
[30024.559680] net eth0: rx->offset: 0, size: 4294967295
[30024.559686] net eth0: me here .. cons:8713451 slots:3 rp:8713462 max:18 err:-22 rx->id:1 rx->offset:0 size:4294967295 ref:-131941395332390
[30536.665135] net eth0: Need more slots cons:9088533 slots:6 rp:9088539 max:17 err:0 rx-id:26 rx->offset:0 size:0 ref:687
[39258.090375] net eth0: rx->offset: 0, size: 4294967295
[39258.090392] net eth0: me here .. cons:15177803 slots:1 rp:15177807 max:18 err:0 rx->id:74 rx->offset:0 size:4294967295 ref:533
[39258.090401] net eth0: rx->offset: 0, size: 4294967295
[39258.090406] net eth0: me here .. cons:15177803 slots:2 rp:15177807 max:18 err:-22 rx->id:76 rx->offset:0 size:4294967295 ref:686
[39258.090415] net eth0: rx->offset: 0, size: 4294967295
[39258.090420] net eth0: me here .. cons:15177803 slots:3 rp:15177807 max:18 err:-22 rx->id:77 rx->offset:0 size:4294967295 ref:571
INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal
[info] Using makefile-style concurrent boot in runlevel 0.
Stopping openntpd: ntpd.
[ ok ] Stopping mail-transfer-agent: nullmailer.
[ ok ] Stopping web server: apache2 ... waiting .
[ ok ] Asking all remaining processes to terminate...done.
[ ok ] All processes ended within 2 seconds...done.
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Deconfiguring network interfaces...done.
[ ok ] Deactivating swap...done.
[65015.958259] EXT4-fs (xvda1): re-mounted. Opts: (null)
[info] Will now halt.
[65018.166546] vif vif-0: 5 starting transaction
[65160.490419] INFO: task halt:4846 blocked for more than 120 seconds.
[65160.490464]       Not tainted 3.14.0-rc4-20140225-vanilla-nfnbdebug2+ #1
[65160.490485] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[65160.490510] halt            D ffff88001d6cfc38     0  4846   4838 0x00000000
[65280.490470] INFO: task halt:4846 blocked for more than 120 seconds.
[65280.490517]       Not tainted 3.14.0-rc4-20140225-vanilla-nfnbdebug2+ #1
[65280.490540] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[65280.490564] halt            D ffff88001d6cfc38     0  4846   4838 0x00000000


Especially the  "[65018.166546] vif vif-0: 5 starting transaction" after the halt surprises me ..

--
Sander

> Feb 26 08:53:20 serveerstertje kernel: [39324.917255] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15101115 cons:15101112 j:8
> Feb 26 08:53:56 serveerstertje kernel: [39361.001436] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15127649 cons:15127648 j:13
> Feb 26 08:54:00 serveerstertje kernel: [39364.725613] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15130263 cons:15130261 j:2
> Feb 26 08:54:04 serveerstertje kernel: [39368.739504] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15133143 cons:15133141 j:0
> Feb 26 08:54:20 serveerstertje kernel: [39384.665044] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15144113 cons:15144112 j:0
> Feb 26 08:54:29 serveerstertje kernel: [39393.569871] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15150203 cons:15150200 j:0
> Feb 26 08:54:40 serveerstertje kernel: [39404.586566] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15157706 cons:15157704 j:12
> Feb 26 08:54:56 serveerstertje kernel: [39420.759769] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15168839 cons:15168835 j:0
> Feb 26 08:54:56 serveerstertje kernel: [39421.001372] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15169002 cons:15168999 j:8
> Feb 26 08:55:00 serveerstertje kernel: [39424.515073] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15171450 cons:15171447 j:0
> Feb 26 08:55:10 serveerstertje kernel: [39435.154510] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15178773 cons:15178770 j:1
> Feb 26 08:56:19 serveerstertje kernel: [39504.195908] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15227444 cons:15227444 j:0
> Feb 26 08:57:39 serveerstertje kernel: [39583.799392] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15283346 cons:15283344 j:8
> Feb 26 08:57:55 serveerstertje kernel: [39599.517673] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15293937 cons:15293935 j:0
> Feb 26 08:58:07 serveerstertje kernel: [39612.156622] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15302891 cons:15302889 j:19
> Feb 26 08:58:07 serveerstertje kernel: [39612.400907] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15303034 cons:15303033 j:0
> Feb 26 08:58:18 serveerstertje kernel: [39623.439383] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15310915 cons:15310911 j:0
> Feb 26 08:58:39 serveerstertje kernel: [39643.521808] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15324769 cons:15324766 j:1

> Feb 26 09:27:07 serveerstertje kernel: [41351.622501] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16502932 cons:16502932 j:8
> Feb 26 09:27:19 serveerstertje kernel: [41363.541003] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:16510837 cons:16510834 j:7
> Feb 26 09:27:23 serveerstertje kernel: [41368.133306] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16513940 cons:16513937 j:0
> Feb 26 09:27:43 serveerstertje kernel: [41388.025147] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16527870 cons:16527868 j:0
> Feb 26 09:27:47 serveerstertje kernel: [41391.530802] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:16530437 cons:16530437 j:7
> Feb 26 09:27:51 serveerstertje kernel: [41395.521166] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533320 cons:16533317 j:6
> Feb 26 09:27:51 serveerstertje kernel: [41395.767066] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533469 cons:16533469 j:0
> Feb 26 09:27:51 serveerstertje kernel: [41395.802319] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:1 GSO:0 vif->rx_last_skb_slots:0 nr_frags:0 prod:16533533 cons:16533533 j:24
> Feb 26 09:27:51 serveerstertje kernel: [41395.837456] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:1 GSO:0 vif->rx_last_skb_slots:0 nr_frags:0 prod:16533534 cons:16533534 j:1
> Feb 26 09:27:51 serveerstertje kernel: [41395.872587] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533597 cons:16533596 j:25
> Feb 26 09:27:51 serveerstertje kernel: [41396.192784] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533833 cons:16533832 j:3
> Feb 26 09:27:51 serveerstertje kernel: [41396.235611] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533890 cons:16533890 j:30
> Feb 26 09:27:51 serveerstertje kernel: [41396.271047] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533898 cons:16533896 j:3


> --
> Sander





>>>
>>> Will keep you posted when it triggers again with the extra info in the warn.
>>>
>>> --
>>> Sander
>>>
>>>
>>>
>>>> Thanks
>>>> Annie
>>>>>     Xen reports:
>>>>>     (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
>>>>>     (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
>>>>>     (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
>>>>>     (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
>>>>>     (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
>>>>>     (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
>>>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
>>>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
>>>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
>>>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
>>>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
>>>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
>>>>>     (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
>>>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
>>>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
>>>>>     (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
>>>>>     (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
>>>>>     (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
>>>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
>>>>>     (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
>>>>>
>>>>>
>>>>>
>>>>> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
>>>>>     - i can ping the guests from dom0
>>>>>     - i can ping dom0 from the guests
>>>>>     - But i can't ssh or access things by http
>>>>>     - I don't see any relevant error messages ...
>>>>>     - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
>>>>>       (that previously worked fine)
>>>>>
>>>>> --
>>>>>
>>>>> Sander
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xen.org
>>>>> http://lists.xen.org/xen-devel
>>>
>>>



> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
> index 4fc46eb..4d720b4 100644
> --- a/tools/libxl/xl_cmdimpl.c
> +++ b/tools/libxl/xl_cmdimpl.c
> @@ -1667,6 +1667,8 @@ skip_vfb:
>                  b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_STD;
>              } else if (!strcmp(buf, "cirrus")) {
>                  b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_CIRRUS;
> +            } else if (!strcmp(buf, "none")) {
> +                b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_NONE;
>              } else {
>                  fprintf(stderr, "Unknown vga \"%s\" specified\n", buf);
>                  exit(1);
> diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
> index 107b000..ab56927 100644
> --- a/xen/common/grant_table.c
> +++ b/xen/common/grant_table.c
> @@ -265,9 +265,10 @@ get_maptrack_handle(
>      while ( unlikely((handle = __get_maptrack_handle(lgt)) == -1) )
>      {
>          nr_frames = nr_maptrack_frames(lgt);
> -        if ( nr_frames >= max_nr_maptrack_frames() )
> +        if ( nr_frames >= max_nr_maptrack_frames() ){
> +                 gdprintk(XENLOG_INFO, "Already at max maptrack size: %u/%u frames\n",nr_frames, max_nr_maptrack_frames());
>              break;
> -
> +       }
>          new_mt = alloc_xenheap_page();
>          if ( !new_mt )
>              break;
> @@ -285,8 +286,8 @@ get_maptrack_handle(
>          smp_wmb();
>          lgt->maptrack_limit      = new_mt_limit;

> -        gdprintk(XENLOG_INFO, "Increased maptrack size to %u frames\n",
> -                 nr_frames + 1);
> +        gdprintk(XENLOG_INFO, "Increased maptrack size to %u/%u frames\n",
> +                 nr_frames + 1, max_nr_maptrack_frames());
>      }

>      spin_unlock(&lgt->lock);
> @@ -1854,7 +1855,7 @@ __acquire_grant_for_copy(

>      if ( unlikely(gref >= nr_grant_entries(rgt)) )
>          PIN_FAIL(unlock_out, GNTST_bad_gntref,
> -                 "Bad grant reference %ld\n", gref);
> +                 "Bad grant reference %ld | %d | %d | %d \n", gref, nr_grant_entries(rgt), rgt->gt_version, ldom);

>      act = &active_entry(rgt, gref);
>      shah = shared_entry_header(rgt, gref);
> @@ -2830,15 +2831,19 @@ static void gnttab_usage_print(struct domain *rd)
>      int first = 1;
>      grant_ref_t ref;
>      struct grant_table *gt = rd->grant_table;
> -
> +    unsigned int active=0;
> +/*
>      printk("      -------- active --------       -------- shared --------\n");
>      printk("[ref] localdom mfn      pin          localdom gmfn     flags\n");
> -
> +*/
>      spin_lock(&gt->lock);

>      if ( gt->gt_version == 0 )
>          goto out;

> +    printk("grant-table for remote domain:%5d (v%d) nr_grant_entries: %d\n",
> +                   rd->domain_id, gt->gt_version, nr_grant_entries(gt));
> +
>      for ( ref = 0; ref != nr_grant_entries(gt); ref++ )
>      {
>          struct active_grant_entry *act;
> @@ -2875,19 +2880,22 @@ static void gnttab_usage_print(struct domain *rd)
>                     rd->domain_id, gt->gt_version);
>              first = 0;
>          }
> -
> +        active++;
>          /*      [ddd]    ddddd 0xXXXXXX 0xXXXXXXXX      ddddd 0xXXXXXX 0xXX */
> -        printk("[%3d]    %5d 0x%06lx 0x%08x      %5d 0x%06"PRIx64" 0x%02x\n",
> -               ref, act->domid, act->frame, act->pin,
> -               sha->domid, frame, status);
> +        /* printk("[%3d]    %5d 0x%06lx 0x%08x      %5d 0x%06"PRIx64" 0x%02x\n", ref, act->domid, act->frame, act->pin, sha->domid, frame, status); */
>      }

>   out:
>      spin_unlock(&gt->lock);

> +    printk("grant-table for remote domain:%5d active entries: %d\n",
> +                   rd->domain_id, active);
> +/*
>      if ( first )
>          printk("grant-table for remote domain:%5d ... "
>                 "no active grant table entries\n", rd->domain_id);
> +*/
> +
>  }

>  static void gnttab_usage_print_all(unsigned char key)






> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> index e5284bc..6d93358 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -482,20 +482,23 @@ static void xenvif_rx_action(struct xenvif *vif)
>                 .meta  = vif->meta,
>         };

> +       int j=0;
> +
>         skb_queue_head_init(&rxq);

>         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
>                 RING_IDX max_slots_needed;
>                 int i;
> +               int nr_frags;

>                 /* We need a cheap worse case estimate for the number of
>                  * slots we'll use.
>                  */

>                 max_slots_needed = DIV_ROUND_UP(offset_in_page(skb->data) +
> -                                               skb_headlen(skb),
> -                                               PAGE_SIZE);
> -               for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> +                                               skb_headlen(skb), PAGE_SIZE);
> +               nr_frags = skb_shinfo(skb)->nr_frags;
> +               for (i = 0; i < nr_frags; i++) {
>                         unsigned int size;
>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> @@ -508,6 +511,9 @@ static void xenvif_rx_action(struct xenvif *vif)
>                 if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
>                         skb_queue_head(&vif->rx_queue, skb);
>                         need_to_notify = true;
> +                       if (net_ratelimit())
> +                               netdev_err(vif->dev, "!?!?!?! skb may not fit .. bail out now max_slots_needed:%d GSO:%d vif->rx_last_skb_slots:%d nr_frags:%d prod:%d cons:%d j:%d\n",
> +                                       max_slots_needed, (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 || skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6) ? 1 : 0, vif->rx_last_skb_slots, nr_frags,vif->rx.sring->req_prod,vif->rx.req_cons,j);
>                         vif->rx_last_skb_slots = max_slots_needed;
>                         break;
>                 } else
> @@ -518,6 +524,7 @@ static void xenvif_rx_action(struct xenvif *vif)
>                 BUG_ON(sco->meta_slots_used > max_slots_needed);

>                 __skb_queue_tail(&rxq, skb);
> +               j++;
>         }

>         BUG_ON(npo.meta_prod > ARRAY_SIZE(vif->meta));
> @@ -541,7 +548,7 @@ static void xenvif_rx_action(struct xenvif *vif)
>                         resp->offset = vif->meta[npo.meta_cons].gso_size;
>                         resp->id = vif->meta[npo.meta_cons].id;
>                         resp->status = sco->meta_slots_used;
> -
> +
>                         npo.meta_cons++;
>                         sco->meta_slots_used--;
>                 }
> @@ -705,7 +712,7 @@ static int xenvif_count_requests(struct xenvif *vif,
>                  */
>                 if (!drop_err && slots >= XEN_NETBK_LEGACY_SLOTS_MAX) {
>                         if (net_ratelimit())
> -                               netdev_dbg(vif->dev,
> +                               netdev_err(vif->dev,
>                                            "Too many slots (%d) exceeding limit (%d), dropping packet\n",
>                                            slots, XEN_NETBK_LEGACY_SLOTS_MAX);
>                         drop_err = -E2BIG;
> @@ -728,7 +735,7 @@ static int xenvif_count_requests(struct xenvif *vif,
>                  */
>                 if (!drop_err && txp->size > first->size) {
>                         if (net_ratelimit())
> -                               netdev_dbg(vif->dev,
> +                               netdev_err(vif->dev,
>                                            "Invalid tx request, slot size %u > remaining size %u\n",
>                                            txp->size, first->size);
>                         drop_err = -EIO;



> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index f9daa9e..67d5221 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -753,6 +753,7 @@ static int xennet_get_responses(struct netfront_info *np,
>                         if (net_ratelimit())
>                                 dev_warn(dev, "rx->offset: %x, size: %u\n",
>                                          rx->offset, rx->status);
> +                               dev_warn(dev, "me here .. cons:%d slots:%d rp:%d max:%d err:%d rx->id:%d rx->offset:%x size:%u ref:%ld\n",cons,slots,rp,max,err,rx->id, rx->offset, rx->status, ref);
>                         xennet_move_rx_slot(np, skb, ref);
>                         err = -EINVAL;
>                         goto next;
> @@ -784,7 +785,7 @@ next:

>                 if (cons + slots == rp) {
>                         if (net_ratelimit())
> -                               dev_warn(dev, "Need more slots\n");
> +                               dev_warn(dev, "Need more slots cons:%d slots:%d rp:%d max:%d err:%d rx-id:%d rx->offset:%x size:%u ref:%ld\n",cons,slots,rp,max,err,rx->id, rx->offset, rx->status, ref);
>                         err = -ENOENT;
>                         break;
>                 }
> @@ -803,7 +804,6 @@ next:

>         if (unlikely(err))
>                 np->rx.rsp_cons = cons + slots;
> -
>         return err;
>  }

> @@ -907,6 +907,7 @@ static int handle_incoming_queue(struct net_device *dev,

>                 /* Ethernet work: Delayed to here as it peeks the header. */
>                 skb->protocol = eth_type_trans(skb, dev);
> +               skb_reset_network_header(skb);

>                 if (checksum_setup(dev, skb)) {
>                         kfree_skb(skb);

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Wei Liu]

On Wed, Feb 26, 2014 at 04:11:23PM +0100, Sander Eikelenboom wrote:
> 
> Wednesday, February 26, 2014, 10:14:42 AM, you wrote:
> 
> 
> > Friday, February 21, 2014, 7:32:08 AM, you wrote:
> 
> 
> >> On 2014/2/20 19:18, Sander Eikelenboom wrote:
> >>> Thursday, February 20, 2014, 10:49:58 AM, you wrote:
> >>>
> >>>
> >>>> On 2014/2/19 5:25, Sander Eikelenboom wrote:
> >>>>> Hi All,
> >>>>>
> >>>>> I'm currently having some network troubles with Xen and recent linux kernels.
> >>>>>
> >>>>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
> >>>>>     I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
> >>>>>
> >>>>>     In the guest:
> >>>>>     [57539.859584] net eth0: rx->offset: 0, size: 4294967295
> >>>>>     [57539.859599] net eth0: rx->offset: 0, size: 4294967295
> >>>>>     [57539.859605] net eth0: rx->offset: 0, size: 4294967295
> >>>>>     [57539.859610] net eth0: Need more slots
> >>>>>     [58157.675939] net eth0: Need more slots
> >>>>>     [58725.344712] net eth0: Need more slots
> >>>>>     [61815.849180] net eth0: rx->offset: 0, size: 4294967295
> >>>>>     [61815.849205] net eth0: rx->offset: 0, size: 4294967295
> >>>>>     [61815.849216] net eth0: rx->offset: 0, size: 4294967295
> >>>>>     [61815.849225] net eth0: Need more slots
> >>>> This issue is familiar... and I thought it get fixed.
> >>>>   From original analysis for similar issue I hit before, the root cause
> >>>> is netback still creates response when the ring is full. I remember
> >>>> larger MTU can trigger this issue before, what is the MTU size?
> >>> In dom0 both for the physical nics and the guest vif's MTU=1500
> >>> In domU the eth0 also has MTU=1500.
> >>>
> >>> So it's not jumbo frames .. just everywhere the same plain defaults ..
> >>>
> >>> With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
> >>> I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
> >>> But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.
> >>>
> >>> This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.
> 
> >> The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
> >> Probably the response overlaps the request and grantcopy return error 
> >> when using wrong grant reference, Netback returns resp->status with 
> >> ||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend.
> >> Would it be possible to print log in xenvif_rx_action of netback to see 
> >> whether something wrong with max slots and used slots?
> 
> >> Thanks
> >> Annie
> 
> > Looking more closely it are perhaps 2 different issues ... the bad grant references do not happen
> > at the same time as the netfront messages in the guest.
> 
> > I added some debugpatches to the kernel netback, netfront and xen granttable code (see below)
> > One of the things was to simplify the code for the debug key to print the granttables, the present code
> > takes too long to execute and brings down the box due to stalls and NMI's. So it now only prints
> > the nr of entries per domain.
> 
> 
> > Issue 1: grant_table.c:1858:d0 Bad grant reference
> 
> > After running the box for just one night (with 15 VM's) i get these mentions of "Bad grant reference".
> > The maptrack also seems to increase quite fast and the number of entries seem to have gone up quite fast as well.
> 
> > Most domains have just one disk(blkfront/blkback) and one nic, a few have a second disk.
> > The blk drivers use persistent grants so i would assume it would reuse those and not increase it (by much).
> 

As far as I can tell netfront has a pool of grant references and it
will BUG_ON() if there's no grefs in the pool when you request one.
Since your DomU didn't crash so I suspect the book-keeping is still
intact.

> > Domain 1 seems to have increased it's nr_grant_entries from 2048 to 3072 somewhere this night.
> > Domain 7 is the domain that happens to give the netfront messages.
> 
> > I also don't get why it is reporting the "Bad grant reference" for domain 0, which seems to have 0 active entries ..
> > Also is this amount of grant entries "normal" ? or could it be a leak somewhere ?
> 

I suppose Dom0 expanding its maptrack is normal. I see as well when I
increase the number of domains. But if it keeps increasing while the
number of DomUs stay the same then it is not normal.

Presumably you only have netfront and blkfront to use grant table and
your workload as described below invovled both so it would be hard to
tell which one is faulty.

There's no immediate functional changes regarding slot counting in this
dev cycle for network driver. But there's some changes to blkfront/back
which seem interesting (memory related).

My suggestion is, if you have a working base line, you can try to setup
different frontend / backend combination to help narrow down the
problem.

Wei.

> > (XEN) [2014-02-26 00:00:38] grant_table.c:1250:d1 Expanding dom (1) grant table from (4) to (5) frames.
> > (XEN) [2014-02-26 00:00:38] grant_table.c:1250:d1 Expanding dom (1) grant table from (5) to (6) frames.
> > (XEN) [2014-02-26 00:00:38] grant_table.c:290:d0 Increased maptrack size to 13/256 frames
> > (XEN) [2014-02-26 00:01:13] grant_table.c:290:d0 Increased maptrack size to 14/256 frames
> > (XEN) [2014-02-26 04:02:55] grant_table.c:1858:d0 Bad grant reference 4325377 | 2048 | 1 | 0
> > (XEN) [2014-02-26 04:15:33] grant_table.c:290:d0 Increased maptrack size to 15/256 frames
> > (XEN) [2014-02-26 04:15:53] grant_table.c:290:d0 Increased maptrack size to 16/256 frames
> > (XEN) [2014-02-26 04:15:56] grant_table.c:290:d0 Increased maptrack size to 17/256 frames
> > (XEN) [2014-02-26 04:15:56] grant_table.c:290:d0 Increased maptrack size to 18/256 frames
> > (XEN) [2014-02-26 04:15:57] grant_table.c:290:d0 Increased maptrack size to 19/256 frames
> > (XEN) [2014-02-26 04:15:57] grant_table.c:290:d0 Increased maptrack size to 20/256 frames
> > (XEN) [2014-02-26 04:15:59] grant_table.c:290:d0 Increased maptrack size to 21/256 frames
> > (XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 22/256 frames
> > (XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 23/256 frames
> > (XEN) [2014-02-26 04:16:00] grant_table.c:290:d0 Increased maptrack size to 24/256 frames
> > (XEN) [2014-02-26 04:16:10] grant_table.c:290:d0 Increased maptrack size to 25/256 frames
> > (XEN) [2014-02-26 04:16:10] grant_table.c:290:d0 Increased maptrack size to 26/256 frames
> > (XEN) [2014-02-26 04:16:17] grant_table.c:290:d0 Increased maptrack size to 27/256 frames
> > (XEN) [2014-02-26 04:16:20] grant_table.c:290:d0 Increased maptrack size to 28/256 frames
> > (XEN) [2014-02-26 04:16:56] grant_table.c:290:d0 Increased maptrack size to 29/256 frames
> > (XEN) [2014-02-26 05:15:04] grant_table.c:290:d0 Increased maptrack size to 30/256 frames
> > (XEN) [2014-02-26 05:15:05] grant_table.c:290:d0 Increased maptrack size to 31/256 frames
> > (XEN) [2014-02-26 05:21:15] grant_table.c:1858:d0 Bad grant reference 107085839 | 2048 | 1 | 0
> > (XEN) [2014-02-26 05:29:47] grant_table.c:1858:d0 Bad grant reference 268435460 | 2048 | 1 | 0
> > (XEN) [2014-02-26 07:53:20] gnttab_usage_print_all [ key 'g' pressed
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    0 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    0 active entries: 0
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 (v1) nr_grant_entries: 3072
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    1 active entries: 2117
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    2 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    3 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    4 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    5 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    6 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    7 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    8 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:    9 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   10 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   11 active entries: 1061
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   12 active entries: 1045
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   13 active entries: 1060
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   14 active entries: 709
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 (v1) nr_grant_entries: 2048
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 (v1)
> > (XEN) [2014-02-26 07:53:20] grant-table for remote domain:   15 active entries: 163
> > (XEN) [2014-02-26 07:53:20] gnttab_usage_print_all ] done
> > (XEN) [2014-02-26 07:55:09] grant_table.c:1858:d0 Bad grant reference 4325377 | 2048 | 1 | 0
> > (XEN) [2014-02-26 08:37:16] grant_table.c:1858:d0 Bad grant reference 268435460 | 2048 | 1 | 0
> 
> 
> 
> > Issue 2: net eth0: rx->offset: 0, size: xxxxxxxxxx
> 
> > In the guest (domain 7):
> 
> > Feb 26 08:55:09 backup kernel: [39258.090375] net eth0: rx->offset: 0, size: 4294967295
> > Feb 26 08:55:09 backup kernel: [39258.090392] net eth0: me here .. cons:15177803 slots:1 rp:15177807 max:18 err:0 rx->id:74 rx->offset:0 size:4294967295 ref:533
> > Feb 26 08:55:09 backup kernel: [39258.090401] net eth0: rx->offset: 0, size: 4294967295
> > Feb 26 08:55:09 backup kernel: [39258.090406] net eth0: me here .. cons:15177803 slots:2 rp:15177807 max:18 err:-22 rx->id:76 rx->offset:0 size:4294967295 ref:686
> > Feb 26 08:55:09 backup kernel: [39258.090415] net eth0: rx->offset: 0, size: 4294967295
> > Feb 26 08:55:09 backup kernel: [39258.090420] net eth0: me here .. cons:15177803 slots:3 rp:15177807 max:18 err:-22 rx->id:77 rx->offset:0 size:4294967295 ref:571
> 
> > In dom0 i don't see any specific netback warnings related to this domain at this specific times, the printk's i added do trigger quite some times but these are probably not
> > errorneous, but they seem to only occur on the vif of domain 7 (probably the only domain that is swamping the network by doing rsync and webdavs and causes some fragmented packets)
> 
> Another addition ... the guest doesn't shutdown anymore on "xl shutdown" .. it just does .. erhmm nothing .. (tried multiple times)
> After that i ssh'ed into the guest and did a "halt -p" ... the guest shutted down .. but the guest remained in xl list in blocked state ..
> Doing a "xl console" shows:
> 
> [30024.559656] net eth0: me here .. cons:8713451 slots:1 rp:8713462 max:18 err:0 rx->id:234 rx->offset:0 size:4294967295 ref:-131941395332550
> [30024.559666] net eth0: rx->offset: 0, size: 4294967295
> [30024.559671] net eth0: me here .. cons:8713451 slots:2 rp:8713462 max:18 err:-22 rx->id:236 rx->offset:0 size:4294967295 ref:-131941395332504
> [30024.559680] net eth0: rx->offset: 0, size: 4294967295
> [30024.559686] net eth0: me here .. cons:8713451 slots:3 rp:8713462 max:18 err:-22 rx->id:1 rx->offset:0 size:4294967295 ref:-131941395332390
> [30536.665135] net eth0: Need more slots cons:9088533 slots:6 rp:9088539 max:17 err:0 rx-id:26 rx->offset:0 size:0 ref:687
> [39258.090375] net eth0: rx->offset: 0, size: 4294967295
> [39258.090392] net eth0: me here .. cons:15177803 slots:1 rp:15177807 max:18 err:0 rx->id:74 rx->offset:0 size:4294967295 ref:533
> [39258.090401] net eth0: rx->offset: 0, size: 4294967295
> [39258.090406] net eth0: me here .. cons:15177803 slots:2 rp:15177807 max:18 err:-22 rx->id:76 rx->offset:0 size:4294967295 ref:686
> [39258.090415] net eth0: rx->offset: 0, size: 4294967295
> [39258.090420] net eth0: me here .. cons:15177803 slots:3 rp:15177807 max:18 err:-22 rx->id:77 rx->offset:0 size:4294967295 ref:571
> INIT: Switching to runlevel: 0
> INIT: Sending processes the TERM signal
> [info] Using makefile-style concurrent boot in runlevel 0.
> Stopping openntpd: ntpd.
> [ ok ] Stopping mail-transfer-agent: nullmailer.
> [ ok ] Stopping web server: apache2 ... waiting .
> [ ok ] Asking all remaining processes to terminate...done.
> [ ok ] All processes ended within 2 seconds...done.
> [ ok ] Stopping enhanced syslogd: rsyslogd.
> [ ok ] Deconfiguring network interfaces...done.
> [ ok ] Deactivating swap...done.
> [65015.958259] EXT4-fs (xvda1): re-mounted. Opts: (null)
> [info] Will now halt.
> [65018.166546] vif vif-0: 5 starting transaction
> [65160.490419] INFO: task halt:4846 blocked for more than 120 seconds.
> [65160.490464]       Not tainted 3.14.0-rc4-20140225-vanilla-nfnbdebug2+ #1
> [65160.490485] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [65160.490510] halt            D ffff88001d6cfc38     0  4846   4838 0x00000000
> [65280.490470] INFO: task halt:4846 blocked for more than 120 seconds.
> [65280.490517]       Not tainted 3.14.0-rc4-20140225-vanilla-nfnbdebug2+ #1
> [65280.490540] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [65280.490564] halt            D ffff88001d6cfc38     0  4846   4838 0x00000000
> 
> 
> Especially the  "[65018.166546] vif vif-0: 5 starting transaction" after the halt surprises me ..
> 
> --
> Sander
> 
> > Feb 26 08:53:20 serveerstertje kernel: [39324.917255] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15101115 cons:15101112 j:8
> > Feb 26 08:53:56 serveerstertje kernel: [39361.001436] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15127649 cons:15127648 j:13
> > Feb 26 08:54:00 serveerstertje kernel: [39364.725613] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15130263 cons:15130261 j:2
> > Feb 26 08:54:04 serveerstertje kernel: [39368.739504] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15133143 cons:15133141 j:0
> > Feb 26 08:54:20 serveerstertje kernel: [39384.665044] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15144113 cons:15144112 j:0
> > Feb 26 08:54:29 serveerstertje kernel: [39393.569871] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15150203 cons:15150200 j:0
> > Feb 26 08:54:40 serveerstertje kernel: [39404.586566] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15157706 cons:15157704 j:12
> > Feb 26 08:54:56 serveerstertje kernel: [39420.759769] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15168839 cons:15168835 j:0
> > Feb 26 08:54:56 serveerstertje kernel: [39421.001372] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15169002 cons:15168999 j:8
> > Feb 26 08:55:00 serveerstertje kernel: [39424.515073] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15171450 cons:15171447 j:0
> > Feb 26 08:55:10 serveerstertje kernel: [39435.154510] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15178773 cons:15178770 j:1
> > Feb 26 08:56:19 serveerstertje kernel: [39504.195908] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15227444 cons:15227444 j:0
> > Feb 26 08:57:39 serveerstertje kernel: [39583.799392] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15283346 cons:15283344 j:8
> > Feb 26 08:57:55 serveerstertje kernel: [39599.517673] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:4 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15293937 cons:15293935 j:0
> > Feb 26 08:58:07 serveerstertje kernel: [39612.156622] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15302891 cons:15302889 j:19
> > Feb 26 08:58:07 serveerstertje kernel: [39612.400907] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15303034 cons:15303033 j:0
> > Feb 26 08:58:18 serveerstertje kernel: [39623.439383] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:15310915 cons:15310911 j:0
> > Feb 26 08:58:39 serveerstertje kernel: [39643.521808] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:6 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:15324769 cons:15324766 j:1
> 
> > Feb 26 09:27:07 serveerstertje kernel: [41351.622501] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16502932 cons:16502932 j:8
> > Feb 26 09:27:19 serveerstertje kernel: [41363.541003] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:16510837 cons:16510834 j:7
> > Feb 26 09:27:23 serveerstertje kernel: [41368.133306] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16513940 cons:16513937 j:0
> > Feb 26 09:27:43 serveerstertje kernel: [41388.025147] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16527870 cons:16527868 j:0
> > Feb 26 09:27:47 serveerstertje kernel: [41391.530802] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:2 prod:16530437 cons:16530437 j:7
> > Feb 26 09:27:51 serveerstertje kernel: [41395.521166] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:5 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533320 cons:16533317 j:6
> > Feb 26 09:27:51 serveerstertje kernel: [41395.767066] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533469 cons:16533469 j:0
> > Feb 26 09:27:51 serveerstertje kernel: [41395.802319] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:1 GSO:0 vif->rx_last_skb_slots:0 nr_frags:0 prod:16533533 cons:16533533 j:24
> > Feb 26 09:27:51 serveerstertje kernel: [41395.837456] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:1 GSO:0 vif->rx_last_skb_slots:0 nr_frags:0 prod:16533534 cons:16533534 j:1
> > Feb 26 09:27:51 serveerstertje kernel: [41395.872587] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533597 cons:16533596 j:25
> > Feb 26 09:27:51 serveerstertje kernel: [41396.192784] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533833 cons:16533832 j:3
> > Feb 26 09:27:51 serveerstertje kernel: [41396.235611] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533890 cons:16533890 j:30
> > Feb 26 09:27:51 serveerstertje kernel: [41396.271047] vif vif-7-0 vif7.0: !?!?!?! skb may not fit .. bail out now max_slots_needed:3 GSO:1 vif->rx_last_skb_slots:0 nr_frags:1 prod:16533898 cons:16533896 j:3
> 
> 
> > --
> > Sander
> 
> 
> 
> 
> 
> >>>
> >>> Will keep you posted when it triggers again with the extra info in the warn.
> >>>
> >>> --
> >>> Sander
> >>>
> >>>
> >>>
> >>>> Thanks
> >>>> Annie
> >>>>>     Xen reports:
> >>>>>     (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 19791875
> >>>>>     (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 268435460
> >>>>>     (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 14 frames
> >>>>>     (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 15 frames
> >>>>>     (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 16 frames
> >>>>>     (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 17 frames
> >>>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 18 frames
> >>>>>     (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 19 frames
> >>>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 20 frames
> >>>>>     (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 21 frames
> >>>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 22 frames
> >>>>>     (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 23 frames
> >>>>>     (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 24 frames
> >>>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 25 frames
> >>>>>     (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 26 frames
> >>>>>     (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 27 frames
> >>>>>     (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 28 frames
> >>>>>     (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 29 frames
> >>>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
> >>>>>     (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 268435460
> >>>>>     (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 4325377
> >>>>>
> >>>>>
> >>>>>
> >>>>> Another issue with networking is when running both dom0 and domU's with a 3.14-rc3 kernel:
> >>>>>     - i can ping the guests from dom0
> >>>>>     - i can ping dom0 from the guests
> >>>>>     - But i can't ssh or access things by http
> >>>>>     - I don't see any relevant error messages ...
> >>>>>     - This is with the same system and kernel config as with the 3.14 and 3.13 combination above
> >>>>>       (that previously worked fine)
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Sander
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Xen-devel mailing list
> >>>>> Xen-devel@lists.xen.org
> >>>>> http://lists.xen.org/xen-devel
> >>>
> >>>
> 
> 
> 
> > diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
> > index 4fc46eb..4d720b4 100644
> > --- a/tools/libxl/xl_cmdimpl.c
> > +++ b/tools/libxl/xl_cmdimpl.c
> > @@ -1667,6 +1667,8 @@ skip_vfb:
> >                  b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_STD;
> >              } else if (!strcmp(buf, "cirrus")) {
> >                  b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_CIRRUS;
> > +            } else if (!strcmp(buf, "none")) {
> > +                b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_NONE;
> >              } else {
> >                  fprintf(stderr, "Unknown vga \"%s\" specified\n", buf);
> >                  exit(1);
> > diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
> > index 107b000..ab56927 100644
> > --- a/xen/common/grant_table.c
> > +++ b/xen/common/grant_table.c
> > @@ -265,9 +265,10 @@ get_maptrack_handle(
> >      while ( unlikely((handle = __get_maptrack_handle(lgt)) == -1) )
> >      {
> >          nr_frames = nr_maptrack_frames(lgt);
> > -        if ( nr_frames >= max_nr_maptrack_frames() )
> > +        if ( nr_frames >= max_nr_maptrack_frames() ){
> > +                 gdprintk(XENLOG_INFO, "Already at max maptrack size: %u/%u frames\n",nr_frames, max_nr_maptrack_frames());
> >              break;
> > -
> > +       }
> >          new_mt = alloc_xenheap_page();
> >          if ( !new_mt )
> >              break;
> > @@ -285,8 +286,8 @@ get_maptrack_handle(
> >          smp_wmb();
> >          lgt->maptrack_limit      = new_mt_limit;
> 
> > -        gdprintk(XENLOG_INFO, "Increased maptrack size to %u frames\n",
> > -                 nr_frames + 1);
> > +        gdprintk(XENLOG_INFO, "Increased maptrack size to %u/%u frames\n",
> > +                 nr_frames + 1, max_nr_maptrack_frames());
> >      }
> 
> >      spin_unlock(&lgt->lock);
> > @@ -1854,7 +1855,7 @@ __acquire_grant_for_copy(
> 
> >      if ( unlikely(gref >= nr_grant_entries(rgt)) )
> >          PIN_FAIL(unlock_out, GNTST_bad_gntref,
> > -                 "Bad grant reference %ld\n", gref);
> > +                 "Bad grant reference %ld | %d | %d | %d \n", gref, nr_grant_entries(rgt), rgt->gt_version, ldom);
> 
> >      act = &active_entry(rgt, gref);
> >      shah = shared_entry_header(rgt, gref);
> > @@ -2830,15 +2831,19 @@ static void gnttab_usage_print(struct domain *rd)
> >      int first = 1;
> >      grant_ref_t ref;
> >      struct grant_table *gt = rd->grant_table;
> > -
> > +    unsigned int active=0;
> > +/*
> >      printk("      -------- active --------       -------- shared --------\n");
> >      printk("[ref] localdom mfn      pin          localdom gmfn     flags\n");
> > -
> > +*/
> >      spin_lock(&gt->lock);
> 
> >      if ( gt->gt_version == 0 )
> >          goto out;
> 
> > +    printk("grant-table for remote domain:%5d (v%d) nr_grant_entries: %d\n",
> > +                   rd->domain_id, gt->gt_version, nr_grant_entries(gt));
> > +
> >      for ( ref = 0; ref != nr_grant_entries(gt); ref++ )
> >      {
> >          struct active_grant_entry *act;
> > @@ -2875,19 +2880,22 @@ static void gnttab_usage_print(struct domain *rd)
> >                     rd->domain_id, gt->gt_version);
> >              first = 0;
> >          }
> > -
> > +        active++;
> >          /*      [ddd]    ddddd 0xXXXXXX 0xXXXXXXXX      ddddd 0xXXXXXX 0xXX */
> > -        printk("[%3d]    %5d 0x%06lx 0x%08x      %5d 0x%06"PRIx64" 0x%02x\n",
> > -               ref, act->domid, act->frame, act->pin,
> > -               sha->domid, frame, status);
> > +        /* printk("[%3d]    %5d 0x%06lx 0x%08x      %5d 0x%06"PRIx64" 0x%02x\n", ref, act->domid, act->frame, act->pin, sha->domid, frame, status); */
> >      }
> 
> >   out:
> >      spin_unlock(&gt->lock);
> 
> > +    printk("grant-table for remote domain:%5d active entries: %d\n",
> > +                   rd->domain_id, active);
> > +/*
> >      if ( first )
> >          printk("grant-table for remote domain:%5d ... "
> >                 "no active grant table entries\n", rd->domain_id);
> > +*/
> > +
> >  }
> 
> >  static void gnttab_usage_print_all(unsigned char key)
> 
> 
> 
> 
> 
> 
> > diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> > index e5284bc..6d93358 100644
> > --- a/drivers/net/xen-netback/netback.c
> > +++ b/drivers/net/xen-netback/netback.c
> > @@ -482,20 +482,23 @@ static void xenvif_rx_action(struct xenvif *vif)
> >                 .meta  = vif->meta,
> >         };
> 
> > +       int j=0;
> > +
> >         skb_queue_head_init(&rxq);
> 
> >         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
> >                 RING_IDX max_slots_needed;
> >                 int i;
> > +               int nr_frags;
> 
> >                 /* We need a cheap worse case estimate for the number of
> >                  * slots we'll use.
> >                  */
> 
> >                 max_slots_needed = DIV_ROUND_UP(offset_in_page(skb->data) +
> > -                                               skb_headlen(skb),
> > -                                               PAGE_SIZE);
> > -               for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> > +                                               skb_headlen(skb), PAGE_SIZE);
> > +               nr_frags = skb_shinfo(skb)->nr_frags;
> > +               for (i = 0; i < nr_frags; i++) {
> >                         unsigned int size;
> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> > @@ -508,6 +511,9 @@ static void xenvif_rx_action(struct xenvif *vif)
> >                 if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
> >                         skb_queue_head(&vif->rx_queue, skb);
> >                         need_to_notify = true;
> > +                       if (net_ratelimit())
> > +                               netdev_err(vif->dev, "!?!?!?! skb may not fit .. bail out now max_slots_needed:%d GSO:%d vif->rx_last_skb_slots:%d nr_frags:%d prod:%d cons:%d j:%d\n",
> > +                                       max_slots_needed, (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 || skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6) ? 1 : 0, vif->rx_last_skb_slots, nr_frags,vif->rx.sring->req_prod,vif->rx.req_cons,j);
> >                         vif->rx_last_skb_slots = max_slots_needed;
> >                         break;
> >                 } else
> > @@ -518,6 +524,7 @@ static void xenvif_rx_action(struct xenvif *vif)
> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> 
> >                 __skb_queue_tail(&rxq, skb);
> > +               j++;
> >         }
> 
> >         BUG_ON(npo.meta_prod > ARRAY_SIZE(vif->meta));
> > @@ -541,7 +548,7 @@ static void xenvif_rx_action(struct xenvif *vif)
> >                         resp->offset = vif->meta[npo.meta_cons].gso_size;
> >                         resp->id = vif->meta[npo.meta_cons].id;
> >                         resp->status = sco->meta_slots_used;
> > -
> > +
> >                         npo.meta_cons++;
> >                         sco->meta_slots_used--;
> >                 }
> > @@ -705,7 +712,7 @@ static int xenvif_count_requests(struct xenvif *vif,
> >                  */
> >                 if (!drop_err && slots >= XEN_NETBK_LEGACY_SLOTS_MAX) {
> >                         if (net_ratelimit())
> > -                               netdev_dbg(vif->dev,
> > +                               netdev_err(vif->dev,
> >                                            "Too many slots (%d) exceeding limit (%d), dropping packet\n",
> >                                            slots, XEN_NETBK_LEGACY_SLOTS_MAX);
> >                         drop_err = -E2BIG;
> > @@ -728,7 +735,7 @@ static int xenvif_count_requests(struct xenvif *vif,
> >                  */
> >                 if (!drop_err && txp->size > first->size) {
> >                         if (net_ratelimit())
> > -                               netdev_dbg(vif->dev,
> > +                               netdev_err(vif->dev,
> >                                            "Invalid tx request, slot size %u > remaining size %u\n",
> >                                            txp->size, first->size);
> >                         drop_err = -EIO;
> 
> 
> 
> > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> > index f9daa9e..67d5221 100644
> > --- a/drivers/net/xen-netfront.c
> > +++ b/drivers/net/xen-netfront.c
> > @@ -753,6 +753,7 @@ static int xennet_get_responses(struct netfront_info *np,
> >                         if (net_ratelimit())
> >                                 dev_warn(dev, "rx->offset: %x, size: %u\n",
> >                                          rx->offset, rx->status);
> > +                               dev_warn(dev, "me here .. cons:%d slots:%d rp:%d max:%d err:%d rx->id:%d rx->offset:%x size:%u ref:%ld\n",cons,slots,rp,max,err,rx->id, rx->offset, rx->status, ref);
> >                         xennet_move_rx_slot(np, skb, ref);
> >                         err = -EINVAL;
> >                         goto next;
> > @@ -784,7 +785,7 @@ next:
> 
> >                 if (cons + slots == rp) {
> >                         if (net_ratelimit())
> > -                               dev_warn(dev, "Need more slots\n");
> > +                               dev_warn(dev, "Need more slots cons:%d slots:%d rp:%d max:%d err:%d rx-id:%d rx->offset:%x size:%u ref:%ld\n",cons,slots,rp,max,err,rx->id, rx->offset, rx->status, ref);
> >                         err = -ENOENT;
> >                         break;
> >                 }
> > @@ -803,7 +804,6 @@ next:
> 
> >         if (unlikely(err))
> >                 np->rx.rsp_cons = cons + slots;
> > -
> >         return err;
> >  }
> 
> > @@ -907,6 +907,7 @@ static int handle_incoming_queue(struct net_device *dev,
> 
> >                 /* Ethernet work: Delayed to here as it peeks the header. */
> >                 skb->protocol = eth_type_trans(skb, dev);
> > +               skb_reset_network_header(skb);
> 
> >                 if (checksum_setup(dev, skb)) {
> >                         kfree_skb(skb);
> 
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Thursday, February 27, 2014, 3:18:12 PM, you wrote:

> On Wed, Feb 26, 2014 at 04:11:23PM +0100, Sander Eikelenboom wrote:
>> 
>> Wednesday, February 26, 2014, 10:14:42 AM, you wrote:
>> 
>> 
>> > Friday, February 21, 2014, 7:32:08 AM, you wrote:
>> 
>> 
>> >> On 2014/2/20 19:18, Sander Eikelenboom wrote:
>> >>> Thursday, February 20, 2014, 10:49:58 AM, you wrote:
>> >>>
>> >>>
>> >>>> On 2014/2/19 5:25, Sander Eikelenboom wrote:
>> >>>>> Hi All,
>> >>>>>
>> >>>>> I'm currently having some network troubles with Xen and recent linux kernels.
>> >>>>>
>> >>>>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>> >>>>>     I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>> >>>>>
>> >>>>>     In the guest:
>> >>>>>     [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>> >>>>>     [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>> >>>>>     [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>> >>>>>     [57539.859610] net eth0: Need more slots
>> >>>>>     [58157.675939] net eth0: Need more slots
>> >>>>>     [58725.344712] net eth0: Need more slots
>> >>>>>     [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>> >>>>>     [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>> >>>>>     [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>> >>>>>     [61815.849225] net eth0: Need more slots
>> >>>> This issue is familiar... and I thought it get fixed.
>> >>>>   From original analysis for similar issue I hit before, the root cause
>> >>>> is netback still creates response when the ring is full. I remember
>> >>>> larger MTU can trigger this issue before, what is the MTU size?
>> >>> In dom0 both for the physical nics and the guest vif's MTU=1500
>> >>> In domU the eth0 also has MTU=1500.
>> >>>
>> >>> So it's not jumbo frames .. just everywhere the same plain defaults ..
>> >>>
>> >>> With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
>> >>> I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
>> >>> But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.
>> >>>
>> >>> This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.
>> 
>> >> The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
>> >> Probably the response overlaps the request and grantcopy return error 
>> >> when using wrong grant reference, Netback returns resp->status with 
>> >> ||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend.
>> >> Would it be possible to print log in xenvif_rx_action of netback to see 
>> >> whether something wrong with max slots and used slots?
>> 
>> >> Thanks
>> >> Annie
>> 
>> > Looking more closely it are perhaps 2 different issues ... the bad grant references do not happen
>> > at the same time as the netfront messages in the guest.
>> 
>> > I added some debugpatches to the kernel netback, netfront and xen granttable code (see below)
>> > One of the things was to simplify the code for the debug key to print the granttables, the present code
>> > takes too long to execute and brings down the box due to stalls and NMI's. So it now only prints
>> > the nr of entries per domain.
>> 
>> 
>> > Issue 1: grant_table.c:1858:d0 Bad grant reference
>> 
>> > After running the box for just one night (with 15 VM's) i get these mentions of "Bad grant reference".
>> > The maptrack also seems to increase quite fast and the number of entries seem to have gone up quite fast as well.
>> 
>> > Most domains have just one disk(blkfront/blkback) and one nic, a few have a second disk.
>> > The blk drivers use persistent grants so i would assume it would reuse those and not increase it (by much).
>> 

> As far as I can tell netfront has a pool of grant references and it
> will BUG_ON() if there's no grefs in the pool when you request one.
> Since your DomU didn't crash so I suspect the book-keeping is still
> intact.

>> > Domain 1 seems to have increased it's nr_grant_entries from 2048 to 3072 somewhere this night.
>> > Domain 7 is the domain that happens to give the netfront messages.
>> 
>> > I also don't get why it is reporting the "Bad grant reference" for domain 0, which seems to have 0 active entries ..
>> > Also is this amount of grant entries "normal" ? or could it be a leak somewhere ?
>> 

> I suppose Dom0 expanding its maptrack is normal. I see as well when I
> increase the number of domains. But if it keeps increasing while the
> number of DomUs stay the same then it is not normal.

It keeps increasing (without (re)starting domains) although eventually it looks like it is settling at a round a maptrack size of 31/256 frames.


> Presumably you only have netfront and blkfront to use grant table and
> your workload as described below invovled both so it would be hard to
> tell which one is faulty.

> There's no immediate functional changes regarding slot counting in this
> dev cycle for network driver. But there's some changes to blkfront/back
> which seem interesting (memory related).

Hmm all the times i get a "Bad grant reference" are related to that one specific guest.
And it's not doing much blkback/front I/O (it's providing webdav and rsync to network based storage (glusterfs))

Added some more printk's:

@@ -2072,7 +2076,11 @@ __gnttab_copy(
                                       &s_frame, &s_pg,
                                       &source_off, &source_len, 1);
         if ( rc != GNTST_okay )
-            goto error_out;
+            PIN_FAIL(error_out, GNTST_general_error,
+                     "?!?!? src_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
+                     current->domain->domain_id, op->source.domid, op->dest.domid);
+
+
         have_s_grant = 1;
         if ( op->source.offset < source_off ||
              op->len > source_len )
@@ -2096,7 +2104,11 @@ __gnttab_copy(
                                       current->domain->domain_id, 0,
                                       &d_frame, &d_pg, &dest_off, &dest_len, 1);
         if ( rc != GNTST_okay )
-            goto error_out;
+            PIN_FAIL(error_out, GNTST_general_error,
+                     "?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
+                     current->domain->domain_id, op->source.domid, op->dest.domid);
+
+
         have_d_grant = 1;


this comes out:

(XEN) [2014-02-27 02:34:37] grant_table.c:2109:d0 ?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:0 src_dom_id:32752 dest_dom_id:7


> My suggestion is, if you have a working base line, you can try to setup
> different frontend / backend combination to help narrow down the
> problem.

Will see what i can do after the weekend

> Wei.

<snip>

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Wei Liu]

On Thu, Feb 27, 2014 at 03:43:51PM +0100, Sander Eikelenboom wrote:
[...]
> 
> > As far as I can tell netfront has a pool of grant references and it
> > will BUG_ON() if there's no grefs in the pool when you request one.
> > Since your DomU didn't crash so I suspect the book-keeping is still
> > intact.
> 
> >> > Domain 1 seems to have increased it's nr_grant_entries from 2048 to 3072 somewhere this night.
> >> > Domain 7 is the domain that happens to give the netfront messages.
> >> 
> >> > I also don't get why it is reporting the "Bad grant reference" for domain 0, which seems to have 0 active entries ..
> >> > Also is this amount of grant entries "normal" ? or could it be a leak somewhere ?
> >> 
> 
> > I suppose Dom0 expanding its maptrack is normal. I see as well when I
> > increase the number of domains. But if it keeps increasing while the
> > number of DomUs stay the same then it is not normal.
> 
> It keeps increasing (without (re)starting domains) although eventually it looks like it is settling at a round a maptrack size of 31/256 frames.
> 

Then I guess that's reasonable. You have 15 DomUs after all...

> 
> > Presumably you only have netfront and blkfront to use grant table and
> > your workload as described below invovled both so it would be hard to
> > tell which one is faulty.
> 
> > There's no immediate functional changes regarding slot counting in this
> > dev cycle for network driver. But there's some changes to blkfront/back
> > which seem interesting (memory related).
> 
> Hmm all the times i get a "Bad grant reference" are related to that one specific guest.
> And it's not doing much blkback/front I/O (it's providing webdav and rsync to network based storage (glusterfs))
> 

OK. I misunderstood that you were rsync'ing from / to your VM disk.

What does webdav do anyway? Does it have a specific traffic pattern?

> Added some more printk's:
> 
> @@ -2072,7 +2076,11 @@ __gnttab_copy(
>                                        &s_frame, &s_pg,
>                                        &source_off, &source_len, 1);
>          if ( rc != GNTST_okay )
> -            goto error_out;
> +            PIN_FAIL(error_out, GNTST_general_error,
> +                     "?!?!? src_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
> +
> +
>          have_s_grant = 1;
>          if ( op->source.offset < source_off ||
>               op->len > source_len )
> @@ -2096,7 +2104,11 @@ __gnttab_copy(
>                                        current->domain->domain_id, 0,
>                                        &d_frame, &d_pg, &dest_off, &dest_len, 1);
>          if ( rc != GNTST_okay )
> -            goto error_out;
> +            PIN_FAIL(error_out, GNTST_general_error,
> +                     "?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
> +
> +
>          have_d_grant = 1;
> 
> 
> this comes out:
> 
> (XEN) [2014-02-27 02:34:37] grant_table.c:2109:d0 ?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:0 src_dom_id:32752 dest_dom_id:7
> 

If it fails in gnttab_copy then I very much suspects this is a network
driver problem as persistent grant in blk driver doesn't use grant
copy.

> 
> > My suggestion is, if you have a working base line, you can try to setup
> > different frontend / backend combination to help narrow down the
> > problem.
> 
> Will see what i can do after the weekend
> 

Thanks

> > Wei.
> 
> <snip>
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Thursday, February 27, 2014, 4:15:39 PM, you wrote:

> On Thu, Feb 27, 2014 at 03:43:51PM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> > As far as I can tell netfront has a pool of grant references and it
>> > will BUG_ON() if there's no grefs in the pool when you request one.
>> > Since your DomU didn't crash so I suspect the book-keeping is still
>> > intact.
>> 
>> >> > Domain 1 seems to have increased it's nr_grant_entries from 2048 to 3072 somewhere this night.
>> >> > Domain 7 is the domain that happens to give the netfront messages.
>> >> 
>> >> > I also don't get why it is reporting the "Bad grant reference" for domain 0, which seems to have 0 active entries ..
>> >> > Also is this amount of grant entries "normal" ? or could it be a leak somewhere ?
>> >> 
>> 
>> > I suppose Dom0 expanding its maptrack is normal. I see as well when I
>> > increase the number of domains. But if it keeps increasing while the
>> > number of DomUs stay the same then it is not normal.
>> 
>> It keeps increasing (without (re)starting domains) although eventually it looks like it is settling at a round a maptrack size of 31/256 frames.
>> 

> Then I guess that's reasonable. You have 15 DomUs after all...

>> 
>> > Presumably you only have netfront and blkfront to use grant table and
>> > your workload as described below invovled both so it would be hard to
>> > tell which one is faulty.
>> 
>> > There's no immediate functional changes regarding slot counting in this
>> > dev cycle for network driver. But there's some changes to blkfront/back
>> > which seem interesting (memory related).
>> 
>> Hmm all the times i get a "Bad grant reference" are related to that one specific guest.
>> And it's not doing much blkback/front I/O (it's providing webdav and rsync to network based storage (glusterfs))
>> 

> OK. I misunderstood that you were rsync'ing from / to your VM disk.

> What does webdav do anyway? Does it have a specific traffic pattern?

The VM is a webdav store .. and the storage for that is network based (at the moment glusterfs in dom0).
Remote backup solutions use this to store backups with duplicity.

Besides that in the guest runs a rsync script that syncs the storage with a remote location.

So the webdav network traffic from outside to the vm causes about the same amount of traffic from the vm to dom0,
so yes .. that gets stretched and tested ;-)

>> Added some more printk's:
>> 
>> @@ -2072,7 +2076,11 @@ __gnttab_copy(
>>                                        &s_frame, &s_pg,
>>                                        &source_off, &source_len, 1);
>>          if ( rc != GNTST_okay )
>> -            goto error_out;
>> +            PIN_FAIL(error_out, GNTST_general_error,
>> +                     "?!?!? src_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
>> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
>> +
>> +
>>          have_s_grant = 1;
>>          if ( op->source.offset < source_off ||
>>               op->len > source_len )
>> @@ -2096,7 +2104,11 @@ __gnttab_copy(
>>                                        current->domain->domain_id, 0,
>>                                        &d_frame, &d_pg, &dest_off, &dest_len, 1);
>>          if ( rc != GNTST_okay )
>> -            goto error_out;
>> +            PIN_FAIL(error_out, GNTST_general_error,
>> +                     "?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
>> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
>> +
>> +
>>          have_d_grant = 1;
>> 
>> 
>> this comes out:
>> 
>> (XEN) [2014-02-27 02:34:37] grant_table.c:2109:d0 ?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:0 src_dom_id:32752 dest_dom_id:7
>> 

> If it fails in gnttab_copy then I very much suspects this is a network
> driver problem as persistent grant in blk driver doesn't use grant
> copy.

Does the dest_gref or src_is_gref by any chance give some sort of direction ?

>> 
>> > My suggestion is, if you have a working base line, you can try to setup
>> > different frontend / backend combination to help narrow down the
>> > problem.
>> 
>> Will see what i can do after the weekend
>> 

> Thanks

>> > Wei.
>> 
>> <snip>
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Wei Liu]

On Thu, Feb 27, 2014 at 04:26:55PM +0100, Sander Eikelenboom wrote:
[...]
> >> Added some more printk's:
> >> 
> >> @@ -2072,7 +2076,11 @@ __gnttab_copy(
> >>                                        &s_frame, &s_pg,
> >>                                        &source_off, &source_len, 1);
> >>          if ( rc != GNTST_okay )
> >> -            goto error_out;
> >> +            PIN_FAIL(error_out, GNTST_general_error,
> >> +                     "?!?!? src_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
> >> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
> >> +
> >> +
> >>          have_s_grant = 1;
> >>          if ( op->source.offset < source_off ||
> >>               op->len > source_len )
> >> @@ -2096,7 +2104,11 @@ __gnttab_copy(
> >>                                        current->domain->domain_id, 0,
> >>                                        &d_frame, &d_pg, &dest_off, &dest_len, 1);
> >>          if ( rc != GNTST_okay )
> >> -            goto error_out;
> >> +            PIN_FAIL(error_out, GNTST_general_error,
> >> +                     "?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
> >> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
> >> +
> >> +
> >>          have_d_grant = 1;
> >> 
> >> 
> >> this comes out:
> >> 
> >> (XEN) [2014-02-27 02:34:37] grant_table.c:2109:d0 ?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:0 src_dom_id:32752 dest_dom_id:7
> >> 
> 
> > If it fails in gnttab_copy then I very much suspects this is a network
> > driver problem as persistent grant in blk driver doesn't use grant
> > copy.
> 
> Does the dest_gref or src_is_gref by any chance give some sort of direction ?
> 

Yes, there's indication. For network driver, dest_is_gref means DomU RX
path, src_is_gref means DomU TX path.

In the particular error message you mentioned, it means that this
happens in DomU's RX path, but it would not give us clear idea what had
happened. As the ring is skewed any way it's not surprised to see a
garbage gref in hypervisor. 

Wei.

> >> 
> >> > My suggestion is, if you have a working base line, you can try to setup
> >> > different frontend / backend combination to help narrow down the
> >> > problem.
> >> 
> >> Will see what i can do after the weekend
> >> 
> 
> > Thanks
> 
> >> > Wei.
> >> 
> >> <snip>
> >> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Thursday, February 27, 2014, 4:57:26 PM, you wrote:

> On Thu, Feb 27, 2014 at 04:26:55PM +0100, Sander Eikelenboom wrote:
> [...]
>> >> Added some more printk's:
>> >> 
>> >> @@ -2072,7 +2076,11 @@ __gnttab_copy(
>> >>                                        &s_frame, &s_pg,
>> >>                                        &source_off, &source_len, 1);
>> >>          if ( rc != GNTST_okay )
>> >> -            goto error_out;
>> >> +            PIN_FAIL(error_out, GNTST_general_error,
>> >> +                     "?!?!? src_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
>> >> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
>> >> +
>> >> +
>> >>          have_s_grant = 1;
>> >>          if ( op->source.offset < source_off ||
>> >>               op->len > source_len )
>> >> @@ -2096,7 +2104,11 @@ __gnttab_copy(
>> >>                                        current->domain->domain_id, 0,
>> >>                                        &d_frame, &d_pg, &dest_off, &dest_len, 1);
>> >>          if ( rc != GNTST_okay )
>> >> -            goto error_out;
>> >> +            PIN_FAIL(error_out, GNTST_general_error,
>> >> +                     "?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:%d src_dom_id:%d dest_dom_id:%d\n",
>> >> +                     current->domain->domain_id, op->source.domid, op->dest.domid);
>> >> +
>> >> +
>> >>          have_d_grant = 1;
>> >> 
>> >> 
>> >> this comes out:
>> >> 
>> >> (XEN) [2014-02-27 02:34:37] grant_table.c:2109:d0 ?!?!? dest_is_gref: aquire grant for copy failed current_dom_id:0 src_dom_id:32752 dest_dom_id:7
>> >> 
>> 
>> > If it fails in gnttab_copy then I very much suspects this is a network
>> > driver problem as persistent grant in blk driver doesn't use grant
>> > copy.
>> 
>> Does the dest_gref or src_is_gref by any chance give some sort of direction ?
>> 

> Yes, there's indication. For network driver, dest_is_gref means DomU RX
> path, src_is_gref means DomU TX path.

> In the particular error message you mentioned, it means that this
> happens in DomU's RX path, but it would not give us clear idea what had
> happened. As the ring is skewed any way it's not surprised to see a
> garbage gref in hypervisor. 

> Wei.

>> >> 
>> >> > My suggestion is, if you have a working base line, you can try to setup
>> >> > different frontend / backend combination to help narrow down the
>> >> > problem.
>> >> 
>> >> Will see what i can do after the weekend
>> >> 
A small update

I tried reverting the latest netback / netfront patches .. but to no avail ..
Also tried if i could trigger it somehow by using netperf and generating a lot
of frags (as that would make it more easily reproduceable).
But that was also to no avail .. it seems to only trigger sometimes with my
specific workload.

So i took a flight forward by trying out Zoltan's series v6
(since it also had changes to the way the network code uses the granttables),
got that running overnight applying the same workload as before and
i haven't triggered anything yet .. looking good so far :-)

--
Sander



>> 
>> > Thanks
>> 
>> >> > Wei.
>> >> 
>> >> <snip>
>> >> 
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Wei Liu]

On Fri, Mar 07, 2014 at 11:33:21AM +0100, Sander Eikelenboom wrote:
[...]
> 
> >> >> 
> >> >> > My suggestion is, if you have a working base line, you can try to setup
> >> >> > different frontend / backend combination to help narrow down the
> >> >> > problem.
> >> >> 
> >> >> Will see what i can do after the weekend
> >> >> 
> A small update
> 
> I tried reverting the latest netback / netfront patches .. but to no avail ..
> Also tried if i could trigger it somehow by using netperf and generating a lot
> of frags (as that would make it more easily reproduceable).
> But that was also to no avail .. it seems to only trigger sometimes with my
> specific workload.
> 
> So i took a flight forward by trying out Zoltan's series v6
> (since it also had changes to the way the network code uses the granttables),
> got that running overnight applying the same workload as before and
> i haven't triggered anything yet .. looking good so far :-)
> 

Thanks for letting us know. If there's any update don't hesitate to post
to xen-devel.

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Sander Eikelenboom]

Friday, March 7, 2014, 12:19:29 PM, you wrote:

> On Fri, Mar 07, 2014 at 11:33:21AM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> >> >> 
>> >> >> > My suggestion is, if you have a working base line, you can try to setup
>> >> >> > different frontend / backend combination to help narrow down the
>> >> >> > problem.
>> >> >> 
>> >> >> Will see what i can do after the weekend
>> >> >> 
>> A small update
>> 
>> I tried reverting the latest netback / netfront patches .. but to no avail ..
>> Also tried if i could trigger it somehow by using netperf and generating a lot
>> of frags (as that would make it more easily reproduceable).
>> But that was also to no avail .. it seems to only trigger sometimes with my
>> specific workload.
>> 
>> So i took a flight forward by trying out Zoltan's series v6
>> (since it also had changes to the way the network code uses the granttables),
>> got that running overnight applying the same workload as before and
>> i haven't triggered anything yet .. looking good so far :-)
>> 

> Thanks for letting us know. If there's any update don't hesitate to post
> to xen-devel.

*sigh* .. it seems posting to xen-devel triggers things ;-)
back to square one again:

Guest kernel:
Mar  7 11:45:29 backup kernel: [49954.928062] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928081] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928086] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928092] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928096] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928101] net eth0: Need more slots
Mar  7 11:45:29 backup kernel: [49954.928196] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928202] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928206] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:45:29 backup kernel: [49954.928210] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397350] net_ratelimit: 14 callbacks suppressed
Mar  7 11:50:42 backup kernel: [50267.397366] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397372] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397377] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397381] net eth0: rx->offset: 0, size: 4294967295
Mar  7 11:50:42 backup kernel: [50267.397386] net eth0: rx->offset: 0, size: 4294967295

Xen:
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 20316163
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 6684675
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 13238275
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 20054019
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 7471105
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 107085839
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 107085839
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
(XEN) [2014-03-07 10:50:42] grant_table.c:1857:d0v4 Bad grant reference 4325379

Will be testing 3.13 vanilla .. see how that works out and if there is a baseline somewhere.

> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Friday, March 7, 2014, 12:55:18 PM, you wrote:

> Friday, March 7, 2014, 12:19:29 PM, you wrote:

>> On Fri, Mar 07, 2014 at 11:33:21AM +0100, Sander Eikelenboom wrote:
>> [...]
>>> 
>>> >> >> 
>>> >> >> > My suggestion is, if you have a working base line, you can try to setup
>>> >> >> > different frontend / backend combination to help narrow down the
>>> >> >> > problem.
>>> >> >> 
>>> >> >> Will see what i can do after the weekend
>>> >> >> 
>>> A small update
>>> 
>>> I tried reverting the latest netback / netfront patches .. but to no avail ..
>>> Also tried if i could trigger it somehow by using netperf and generating a lot
>>> of frags (as that would make it more easily reproduceable).
>>> But that was also to no avail .. it seems to only trigger sometimes with my
>>> specific workload.
>>> 
>>> So i took a flight forward by trying out Zoltan's series v6
>>> (since it also had changes to the way the network code uses the granttables),
>>> got that running overnight applying the same workload as before and
>>> i haven't triggered anything yet .. looking good so far :-)
>>> 

>> Thanks for letting us know. If there's any update don't hesitate to post
>> to xen-devel.

> *sigh* .. it seems posting to xen-devel triggers things ;-)
> back to square one again:

> Guest kernel:
> Mar  7 11:45:29 backup kernel: [49954.928062] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928081] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928086] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928092] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928096] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928101] net eth0: Need more slots
> Mar  7 11:45:29 backup kernel: [49954.928196] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928202] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928206] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:45:29 backup kernel: [49954.928210] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:50:42 backup kernel: [50267.397350] net_ratelimit: 14 callbacks suppressed
> Mar  7 11:50:42 backup kernel: [50267.397366] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:50:42 backup kernel: [50267.397372] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:50:42 backup kernel: [50267.397377] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:50:42 backup kernel: [50267.397381] net eth0: rx->offset: 0, size: 4294967295
> Mar  7 11:50:42 backup kernel: [50267.397386] net eth0: rx->offset: 0, size: 4294967295

> Xen:
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 20316163
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 6684675
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 13238275
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 20054019
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 3538945
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 7471105
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 4325377
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 107085839
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 107085839
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
> (XEN) [2014-03-07 10:45:29] grant_table.c:1857:d0v3 Bad grant reference 268435460
> (XEN) [2014-03-07 10:50:42] grant_table.c:1857:d0v4 Bad grant reference 4325379

> Will be testing 3.13 vanilla .. see how that works out and if there is a baseline somewhere.

>> Wei.

Hi Paul,

It seems a commit by you: "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
is the first that gives the Bad grant references.
It seems later patches partly prevent or mask the issue, so it is less easy to trigger it.
With only this commit applied i can trigger it quite fast.

This is the result of:
- First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
- Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
- Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
- After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
- After that started to apply the netback changes for 3.14 and that failed after the commit stated above.

So i'm quite confident i'm reporting the right thing now :-)
If you would like me to run debug patches on top of this commit, don't hesitate to send them !

--
Sander

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Tue, Mar 11, 2014 at 12:00:26AM +0100, Sander Eikelenboom wrote:
[...]
> 
> >> Wei.
> 
> Hi Paul,
> 
> It seems a commit by you: "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
> is the first that gives the Bad grant references.
> It seems later patches partly prevent or mask the issue, so it is less easy to trigger it.
> With only this commit applied i can trigger it quite fast.
> 
> This is the result of:
> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
> - After that started to apply the netback changes for 3.14 and that failed after the commit stated above.
> 
> So i'm quite confident i'm reporting the right thing now :-)
> If you would like me to run debug patches on top of this commit, don't hesitate to send them !
> 

Hmm.... I just looked at the commit, something that's obvious wrong is
the use of gso_type to determine whether an extra slot is required,
which is fixed by Annie yesterday. Annie fixed that for netback.c but
missed interface.c.

This can probably fix the problem you're seeing. I will submit a proper
patch if you confirm that...

>From e560333a95f29886bc33352bdf6c0d09c227e833 Mon Sep 17 00:00:00 2001
From: Wei Liu <wei.liu2@citrix.com>
Date: Tue, 11 Mar 2014 10:15:25 +0000
Subject: [PATCH] xen-netback: fix calculation of extra slot

In 5bd076708 ("Xen-netback: Fix issue caused by using gso_type wrongly")
we use skb_is_gso to determine if we need an extra slot to accommodate
the SKB. There's similar error in interface.c as well. Fix that using
skb_is_gso.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 drivers/net/xen-netback/interface.c |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
index 7669d49..301cc03 100644
--- a/drivers/net/xen-netback/interface.c
+++ b/drivers/net/xen-netback/interface.c
@@ -132,8 +132,7 @@ static int xenvif_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	/* If the skb is GSO then we'll also need an extra slot for the
 	 * metadata.
 	 */
-	if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
-	    skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
+	if (skb_is_gso(skb))
 		min_slots_needed++;
 
 	/* If the skb can't possibly fit in the remaining slots
-- 
1.7.10.4




> --
> Sander
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 11, 2014, 11:19:48 AM, you wrote:

> On Tue, Mar 11, 2014 at 12:00:26AM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> >> Wei.
>> 
>> Hi Paul,
>> 
>> It seems a commit by you: "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>> is the first that gives the Bad grant references.
>> It seems later patches partly prevent or mask the issue, so it is less easy to trigger it.
>> With only this commit applied i can trigger it quite fast.
>> 
>> This is the result of:
>> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
>> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
>> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
>> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
>> - After that started to apply the netback changes for 3.14 and that failed after the commit stated above.
>> 
>> So i'm quite confident i'm reporting the right thing now :-)
>> If you would like me to run debug patches on top of this commit, don't hesitate to send them !
>> 

> Hmm.... I just looked at the commit, something that's obvious wrong is
> the use of gso_type to determine whether an extra slot is required,
> which is fixed by Annie yesterday. Annie fixed that for netback.c but
> missed interface.c.

Ah i saw and tried that patch as well .. but on it's own it didn't help.
Will try with both patches ..


> This can probably fix the problem you're seeing. I will submit a proper
> patch if you confirm that...

> From e560333a95f29886bc33352bdf6c0d09c227e833 Mon Sep 17 00:00:00 2001
> From: Wei Liu <wei.liu2@citrix.com>
> Date: Tue, 11 Mar 2014 10:15:25 +0000
> Subject: [PATCH] xen-netback: fix calculation of extra slot

> In 5bd076708 ("Xen-netback: Fix issue caused by using gso_type wrongly")
> we use skb_is_gso to determine if we need an extra slot to accommodate
> the SKB. There's similar error in interface.c as well. Fix that using
> skb_is_gso.

> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  drivers/net/xen-netback/interface.c |    3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)

> diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
> index 7669d49..301cc03 100644
> --- a/drivers/net/xen-netback/interface.c
> +++ b/drivers/net/xen-netback/interface.c
> @@ -132,8 +132,7 @@ static int xenvif_start_xmit(struct sk_buff *skb, struct net_device *dev)
>         /* If the skb is GSO then we'll also need an extra slot for the
>          * metadata.
>          */
> -       if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
> -           skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
> +       if (skb_is_gso(skb))
>                 min_slots_needed++;
>  
>         /* If the skb can't possibly fit in the remaining slots

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 11, 2014, 11:19:48 AM, you wrote:

> On Tue, Mar 11, 2014 at 12:00:26AM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> >> Wei.
>> 
>> Hi Paul,
>> 
>> It seems a commit by you: "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>> is the first that gives the Bad grant references.
>> It seems later patches partly prevent or mask the issue, so it is less easy to trigger it.
>> With only this commit applied i can trigger it quite fast.
>> 
>> This is the result of:
>> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
>> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
>> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
>> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
>> - After that started to apply the netback changes for 3.14 and that failed after the commit stated above.
>> 
>> So i'm quite confident i'm reporting the right thing now :-)
>> If you would like me to run debug patches on top of this commit, don't hesitate to send them !
>> 

> Hmm.... I just looked at the commit, something that's obvious wrong is
> the use of gso_type to determine whether an extra slot is required,
> which is fixed by Annie yesterday. Annie fixed that for netback.c but
> missed interface.c.

> This can probably fix the problem you're seeing. I will submit a proper
> patch if you confirm that...

Although the patch seems correct in it's own right .. it doesn't seem to fix
the issue when using 3.13.6 as a base and ..
  - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
  - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
  - applying annie's v2 patch
  - applying your patch
as dom0 and using a 3.14-rc5 as domU kernel.

Unfortunately i'm still getting the Bad grant references ..

> From e560333a95f29886bc33352bdf6c0d09c227e833 Mon Sep 17 00:00:00 2001
> From: Wei Liu <wei.liu2@citrix.com>
> Date: Tue, 11 Mar 2014 10:15:25 +0000
> Subject: [PATCH] xen-netback: fix calculation of extra slot

> In 5bd076708 ("Xen-netback: Fix issue caused by using gso_type wrongly")
> we use skb_is_gso to determine if we need an extra slot to accommodate
> the SKB. There's similar error in interface.c as well. Fix that using
> skb_is_gso.

> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  drivers/net/xen-netback/interface.c |    3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)

> diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
> index 7669d49..301cc03 100644
> --- a/drivers/net/xen-netback/interface.c
> +++ b/drivers/net/xen-netback/interface.c
> @@ -132,8 +132,7 @@ static int xenvif_start_xmit(struct sk_buff *skb, struct net_device *dev)
>         /* If the skb is GSO then we'll also need an extra slot for the
>          * metadata.
>          */
> -       if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
> -           skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
> +       if (skb_is_gso(skb))
>                 min_slots_needed++;
>  
>         /* If the skb can't possibly fit in the remaining slots

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Tue, Mar 11, 2014 at 01:31:42PM +0100, Sander Eikelenboom wrote:
> 
> Tuesday, March 11, 2014, 11:19:48 AM, you wrote:
> 
> > On Tue, Mar 11, 2014 at 12:00:26AM +0100, Sander Eikelenboom wrote:
> > [...]
> >> 
> >> >> Wei.
> >> 
> >> Hi Paul,
> >> 
> >> It seems a commit by you: "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
> >> is the first that gives the Bad grant references.
> >> It seems later patches partly prevent or mask the issue, so it is less easy to trigger it.
> >> With only this commit applied i can trigger it quite fast.
> >> 
> >> This is the result of:
> >> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
> >> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
> >> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
> >> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
> >> - After that started to apply the netback changes for 3.14 and that failed after the commit stated above.
> >> 
> >> So i'm quite confident i'm reporting the right thing now :-)
> >> If you would like me to run debug patches on top of this commit, don't hesitate to send them !
> >> 
> 
> > Hmm.... I just looked at the commit, something that's obvious wrong is
> > the use of gso_type to determine whether an extra slot is required,
> > which is fixed by Annie yesterday. Annie fixed that for netback.c but
> > missed interface.c.
> 
> > This can probably fix the problem you're seeing. I will submit a proper
> > patch if you confirm that...
> 
> Although the patch seems correct in it's own right .. it doesn't seem to fix

I will submit that patch anyway...

> the issue when using 3.13.6 as a base and ..
>   - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
>   - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>   - applying annie's v2 patch
>   - applying your patch
> as dom0 and using a 3.14-rc5 as domU kernel.
> 
> Unfortunately i'm still getting the Bad grant references ..
> 

:-( That's bad news.

I guess you always have the same DomU kernel when testing? That means we
can narrow down the bug to netback only.

Paul, do you have any idea what might go wrong?

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Tue, Mar 11, 2014 at 12:38:07PM +0000, Wei Liu wrote:
> On Tue, Mar 11, 2014 at 01:31:42PM +0100, Sander Eikelenboom wrote:
> > 
> > Tuesday, March 11, 2014, 11:19:48 AM, you wrote:
> > 
> > > On Tue, Mar 11, 2014 at 12:00:26AM +0100, Sander Eikelenboom wrote:
> > > [...]
> > >> 
> > >> >> Wei.
> > >> 
> > >> Hi Paul,
> > >> 
> > >> It seems a commit by you: "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
> > >> is the first that gives the Bad grant references.
> > >> It seems later patches partly prevent or mask the issue, so it is less easy to trigger it.
> > >> With only this commit applied i can trigger it quite fast.
> > >> 
> > >> This is the result of:
> > >> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
> > >> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
> > >> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
> > >> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
> > >> - After that started to apply the netback changes for 3.14 and that failed after the commit stated above.
> > >> 
> > >> So i'm quite confident i'm reporting the right thing now :-)
> > >> If you would like me to run debug patches on top of this commit, don't hesitate to send them !
> > >> 
> > 
> > > Hmm.... I just looked at the commit, something that's obvious wrong is
> > > the use of gso_type to determine whether an extra slot is required,
> > > which is fixed by Annie yesterday. Annie fixed that for netback.c but
> > > missed interface.c.
> > 
> > > This can probably fix the problem you're seeing. I will submit a proper
> > > patch if you confirm that...
> > 
> > Although the patch seems correct in it's own right .. it doesn't seem to fix
> 
> I will submit that patch anyway...
> 
> > the issue when using 3.13.6 as a base and ..
> >   - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
> >   - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
> >   - applying annie's v2 patch
> >   - applying your patch
> > as dom0 and using a 3.14-rc5 as domU kernel.
> > 
> > Unfortunately i'm still getting the Bad grant references ..
> > 
> 
> :-( That's bad news.
> 
> I guess you always have the same DomU kernel when testing? That means we

And apparently you're! Ignore this. I hit "send" too fast!

> can narrow down the bug to netback only.
> 
> Paul, do you have any idea what might go wrong?
> 
> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 11, 2014, 1:38:07 PM, you wrote:

> On Tue, Mar 11, 2014 at 01:31:42PM +0100, Sander Eikelenboom wrote:
>> 
>> Tuesday, March 11, 2014, 11:19:48 AM, you wrote:
>> 
>> > On Tue, Mar 11, 2014 at 12:00:26AM +0100, Sander Eikelenboom wrote:
>> > [...]
>> >> 
>> >> >> Wei.
>> >> 
>> >> Hi Paul,
>> >> 
>> >> It seems a commit by you: "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>> >> is the first that gives the Bad grant references.
>> >> It seems later patches partly prevent or mask the issue, so it is less easy to trigger it.
>> >> With only this commit applied i can trigger it quite fast.
>> >> 
>> >> This is the result of:
>> >> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
>> >> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
>> >> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
>> >> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
>> >> - After that started to apply the netback changes for 3.14 and that failed after the commit stated above.
>> >> 
>> >> So i'm quite confident i'm reporting the right thing now :-)
>> >> If you would like me to run debug patches on top of this commit, don't hesitate to send them !
>> >> 
>> 
>> > Hmm.... I just looked at the commit, something that's obvious wrong is
>> > the use of gso_type to determine whether an extra slot is required,
>> > which is fixed by Annie yesterday. Annie fixed that for netback.c but
>> > missed interface.c.
>> 
>> > This can probably fix the problem you're seeing. I will submit a proper
>> > patch if you confirm that...
>> 
>> Although the patch seems correct in it's own right .. it doesn't seem to fix

> I will submit that patch anyway...

>> the issue when using 3.13.6 as a base and ..
>>   - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
>>   - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>>   - applying annie's v2 patch
>>   - applying your patch
>> as dom0 and using a 3.14-rc5 as domU kernel.
>> 
>> Unfortunately i'm still getting the Bad grant references ..
>> 

> :-( That's bad news.

> I guess you always have the same DomU kernel when testing? That means we
> can narrow down the bug to netback only.

Yes my previous tests (from my previous mail):

- First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
- Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
- Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
- After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
- After that started to apply the netback changes for 3.14 and that failed after the commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control".

Also seem to indicate just that, although it could also be something in this netback commit that triggers a latent bug in netfront, can't rule that one out completly.

But the trigger is in that commit &&
annie's and your patch seem to have no effect at all( on this issue) &&
later commits in 3.14 do seems to mask it / make it less likely to trigger, but do not fix it.

> Paul, do you have any idea what might go wrong?

> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Tue, Mar 11, 2014 at 02:00:41PM +0100, Sander Eikelenboom wrote:
[...]
> >> the issue when using 3.13.6 as a base and ..
> >>   - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
> >>   - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
> >>   - applying annie's v2 patch
> >>   - applying your patch
> >> as dom0 and using a 3.14-rc5 as domU kernel.
> >> 
> >> Unfortunately i'm still getting the Bad grant references ..
> >> 
> 
> > :-( That's bad news.
> 
> > I guess you always have the same DomU kernel when testing? That means we
> > can narrow down the bug to netback only.
> 
> Yes my previous tests (from my previous mail):
> 
> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
> - After that started to apply the netback changes for 3.14 and that failed after the commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control".
> 
> Also seem to indicate just that, although it could also be something in this netback commit that triggers a latent bug in netfront, can't rule that one out completly.
> 
> But the trigger is in that commit &&
> annie's and your patch seem to have no effect at all( on this issue) &&
> later commits in 3.14 do seems to mask it / make it less likely to trigger, but do not fix it.
> 

Unfortunately I've stared at the same piece of code for some time but
don't have immediate clue. Later commits don't look suspecious either.

I also looked at netfront code, but there's no slot couting change
between 3.13 and 3.14.

Do you have some straight setup instructions so that I can try to
reproduce.

Wei.

> > Paul, do you have any idea what might go wrong?
> 
> > Wei.
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 11, 2014, 4:36:16 PM, you wrote:

> On Tue, Mar 11, 2014 at 02:00:41PM +0100, Sander Eikelenboom wrote:
> [...]
>> >> the issue when using 3.13.6 as a base and ..
>> >>   - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
>> >>   - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>> >>   - applying annie's v2 patch
>> >>   - applying your patch
>> >> as dom0 and using a 3.14-rc5 as domU kernel.
>> >> 
>> >> Unfortunately i'm still getting the Bad grant references ..
>> >> 
>> 
>> > :-( That's bad news.
>> 
>> > I guess you always have the same DomU kernel when testing? That means we
>> > can narrow down the bug to netback only.
>> 
>> Yes my previous tests (from my previous mail):
>> 
>> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
>> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
>> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
>> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
>> - After that started to apply the netback changes for 3.14 and that failed after the commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control".
>> 
>> Also seem to indicate just that, although it could also be something in this netback commit that triggers a latent bug in netfront, can't rule that one out completly.
>> 
>> But the trigger is in that commit &&
>> annie's and your patch seem to have no effect at all( on this issue) &&
>> later commits in 3.14 do seems to mask it / make it less likely to trigger, but do not fix it.
>> 

> Unfortunately I've stared at the same piece of code for some time but
> don't have immediate clue. Later commits don't look suspecious either.

> I also looked at netfront code, but there's no slot couting change
> between 3.13 and 3.14.

> Do you have some straight setup instructions so that I can try to
> reproduce.

Not really .. since it is more easily triggerable without the later commits i will see if i can trigger it now by using
netperf.

My feeling says it is something like a off by one error (since it doesn't happen on all packets .. )
and that it was already in the code before but just wasn't triggered ..

Which makes it hard to spot .. especially since it seems quite difficult to trace were that bad grant reference comes from
(and if it is the cause .. or the effect).

Will see if i can reliably trigger it now with netperf and report back.

> Wei.

>> > Paul, do you have any idea what might go wrong?
>> 
>> > Wei.
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

[-- Attachment #1: Type: text/plain, Size: 11947 bytes --]


Tuesday, March 11, 2014, 4:36:16 PM, you wrote:

> On Tue, Mar 11, 2014 at 02:00:41PM +0100, Sander Eikelenboom wrote:
> [...]
>> >> the issue when using 3.13.6 as a base and ..
>> >>   - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
>> >>   - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>> >>   - applying annie's v2 patch
>> >>   - applying your patch
>> >> as dom0 and using a 3.14-rc5 as domU kernel.
>> >> 
>> >> Unfortunately i'm still getting the Bad grant references ..
>> >> 
>> 
>> > :-( That's bad news.
>> 
>> > I guess you always have the same DomU kernel when testing? That means we
>> > can narrow down the bug to netback only.
>> 
>> Yes my previous tests (from my previous mail):
>> 
>> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
>> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
>> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
>> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
>> - After that started to apply the netback changes for 3.14 and that failed after the commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control".
>> 
>> Also seem to indicate just that, although it could also be something in this netback commit that triggers a latent bug in netfront, can't rule that one out completly.
>> 
>> But the trigger is in that commit &&
>> annie's and your patch seem to have no effect at all( on this issue) &&
>> later commits in 3.14 do seems to mask it / make it less likely to trigger, but do not fix it.
>> 

> Unfortunately I've stared at the same piece of code for some time but
> don't have immediate clue. Later commits don't look suspecious either.

> I also looked at netfront code, but there's no slot couting change
> between 3.13 and 3.14.

> Do you have some straight setup instructions so that I can try to
> reproduce.

Ok you asked for it .. so here we go .. ;-) :

- Xen-unstable
- DomU:
     - PV guest
     - Debian Wheezy
     - Kernel: 3.14-rc5 (.config see dom0 .config)
     - Running netserver
- Dom0:
     - Debian Wheezy
     - Kernel: 3.13.6 + additional patches (see attached git log and .config)
     - 2 physical NIC's
     - Autoballooning is prevented by using dom0_mem=1536M,max:1536M for xen and mem=1536M for dom0 kernel stanzas in grub
     - vcpu 0 is pinned on pcpu 0 and exclusively for dom0

- Networking:
     - Routed bridge
     - eth0 = internet
     - eth1 = lan 172.16.x.x
     - xen_bridge = bridge for VM's 192.168.1.x
     - iptables NAT and routing  
     - attached dom0 and domU ifconfig output
     - attached ethtool -k output for the bridge, vif and guest eth0

Triggering workload:
     - Well that's were the problem is :-)
     - The Guest has a normal disk and swap (phy/lvm) and
       shared storage with glusterfs (glusterfs server is on dom0)
     - The Guest exposes this storage via webdavs 

     What triggers it is:
     - The Guest runs it's rsync of the shared storage to a remote client on the internet
       So this causes traffic from Dom0 to domU (reading storage) ..
       back from domU to dom0 and via iptables NAT on to eth0 (actual rsync)
       or vice versa when syncing the other way around
     - At the same time do a backup from a windows machine from the lan to the webdavs server
       So this causes traffic from eth1 to domU (webdav) .. 
       back from domU to dom0 (writing to the shared storage) 

So in essence it is doing quite some netback && netfront stress testing :-)

It seems to only trigger when doing both the rsync and the webdav simultaneous.

I tried my best to emulate any of this with netperf (and multiple instances),
i tried with various (odd) packet sizes and the packet / byte rates transmitted
are higher then with the workload above ... but it doesn't seem to trigger with netpref

So i don't think it will be easy to replicate ...

Perhaps running through the available logging again .. and try to answer some questions ... this is just with one guest running kernels as before only added debugging to netfront and xen (diffs attached):

Mar 12 02:00:44 backup kernel: [  496.840646] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.840665] net eth0: cons:1346005 slots:1 rp:1346013 max:18 err:0 rx->id:212 rx->offset:0 size:4294967295 ref:572 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.840680] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.840687] net eth0: cons:1346005 slots:2 rp:1346013 max:18 err:-22 rx->id:214 rx->offset:0 size:4294967295 ref:657 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.840701] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.840712] net eth0: cons:1346005 slots:3 rp:1346013 max:18 err:-22 rx->id:215 rx->offset:0 size:4294967295 ref:667 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.840733] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.840740] net eth0: cons:1346005 slots:4 rp:1346013 max:18 err:-22 rx->id:216 rx->offset:0 size:4294967295 ref:716 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.840757] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.840764] net eth0: cons:1346005 slots:5 rp:1346013 max:18 err:-22 rx->id:217 rx->offset:0 size:4294967295 ref:755 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.840778] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.840784] net eth0: cons:1346005 slots:6 rp:1346013 max:18 err:-22 rx->id:218 rx->offset:0 size:4294967295 ref:592 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.840801] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.840807] net eth0: cons:1346005 slots:7 rp:1346013 max:18 err:-22 rx->id:219 rx->offset:0 size:4294967295 ref:633 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.841043] net eth0: rx->offset: 0, size: 4294967295

Mar 12 02:00:44 backup kernel: [  496.841052] net eth0: cons:1346025 slots:1 rp:1346038 max:18 err:0 rx->id:232 rx->offset:0 size:4294967295 ref:-131941395332491 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:13 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.841067] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.841074] net eth0: cons:1346025 slots:2 rp:1346038 max:18 err:-22 rx->id:234 rx->offset:0 size:4294967295 ref:-131941395332579 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
Mar 12 02:00:44 backup kernel: [  496.841092] net eth0: rx->offset: 0, size: 4294967295
Mar 12 02:00:44 backup kernel: [  496.841101] net eth0: cons:1346025 slots:3 rp:1346038 max:18 err:-22 rx->id:235 rx->offset:0 size:4294967295 ref:-131941395332408 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0


(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478146 source_off:0 source_len:4096 op->source.offset:0 op->len:1168
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:0 op->len:2476
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478282 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:1634 op->len:1620
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497609 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1

- Sometimes (but not always) netfront also spits out:
  dev_warn(dev, "Invalid extra type: %d\n", extra->type);
  where the extra type seems a random value (seen 196, 31 ..)
- Sometimes (but not always) netfront also spits out:
  dev_warn(dev, "Need more slots\n");
- Sometimes (but not always) netfront also spits out:
  dev_warn(dev, "Missing extra info\n");


First question that comes to my mind:
- Are the warnings netfront spits out the cause of xen reporting the bad grant reference ?
  Or
  Are the Bad grant references Xen is reporting .. causing netfront to spit out the warnings ?
- Why is that "if (rx->flags & XEN_NETRXF_extra_info) {}" part in xen-netfront.c doing there and changing cons midway ?
  The commit message from f942dc2552b8bfdee607be867b12a8971bb9cd85 that introduced the if says:

    One major change from xen.git is that the guest transmit path (i.e. what
    looks like receive to netback) has been significantly reworked to remove
    the dependency on the out of tree PageForeign page flag (a core kernel
    patch which enables a per page destructor callback on the final
    put_page). This page flag was used in order to implement a grant map
    based transmit path (where guest pages are mapped directly into SKB
    frags). Instead this version of netback uses grant copy operations into
    regular memory belonging to the backend domain. Reinstating the grant
    map functionality is something which I would like to revisit in the
    future.

  It *is* using grant copy now .. so should this part have been removed ?
  And
  Could Paul's commit that seems to be the first to trigger these events affect this ?

--
Sander



> Wei.

>> > Paul, do you have any idea what might go wrong?
>> 
>> > Wei.
>> 

[-- Attachment #2: ethtool-domu-eth0.txt --]
[-- Type: text/plain, Size: 1269 bytes --]

Features for eth0:
rx-checksumming: on [fixed]
tx-checksumming: on
	tx-checksum-ipv4: on [fixed]
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: on
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

[-- Attachment #3: ethtool-vif.txt --]
[-- Type: text/plain, Size: 1263 bytes --]

Features for vif1.0:
rx-checksumming: on [fixed]
tx-checksumming: on
	tx-checksum-ipv4: on
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: on
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

[-- Attachment #4: ethtool-xenbridge.txt --]
[-- Type: text/plain, Size: 1263 bytes --]

Features for xen_bridge:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [requested on]
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [requested on]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [requested on]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [requested on]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-ipip-segmentation: on
tx-sit-segmentation: on
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

[-- Attachment #5: grant_table_debug.diff --]
[-- Type: application/octet-stream, Size: 2362 bytes --]

diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index 107b000..337e17a 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -1852,10 +1852,9 @@ __acquire_grant_for_copy(
         PIN_FAIL(unlock_out, GNTST_general_error,
                  "remote grant table not ready\n");
 
-    if ( unlikely(gref >= nr_grant_entries(rgt)) )
-        PIN_FAIL(unlock_out, GNTST_bad_gntref,
-                 "Bad grant reference %ld\n", gref);
-
+    if ( unlikely(gref >= nr_grant_entries(rgt)) ){
+        PIN_FAIL(unlock_out, GNTST_bad_gntref, "Bad grant reference %ld gt_version:%d ldom:%d readonly:%d allow_transitive:%d\n", gref, rgt->gt_version, ldom, readonly, allow_transitive);
+    }
     act = &active_entry(rgt, gref);
     shah = shared_entry_header(rgt, gref);
     if ( rgt->gt_version == 1 )
@@ -2071,8 +2070,10 @@ __gnttab_copy(
                                       current->domain->domain_id, 1,
                                       &s_frame, &s_pg,
                                       &source_off, &source_len, 1);
-        if ( rc != GNTST_okay )
-            goto error_out;
+        if ( rc != GNTST_okay ){
+            gdprintk(XENLOG_WARNING, "acquire_grant_for_copy failed .. src_is_gref rc:%d source.domid:%d dest.domid:%d s_frame:%ld source_off:%u source_len:%u op->source.offset:%d op->len:%d\n", rc, op->source.domid, op->dest.domid, s_frame, source_off, source_len, op->source.offset, op->len);
+	    goto error_out;
+	}
         have_s_grant = 1;
         if ( op->source.offset < source_off ||
              op->len > source_len )
@@ -2095,8 +2096,10 @@ __gnttab_copy(
         rc = __acquire_grant_for_copy(dd, op->dest.u.ref,
                                       current->domain->domain_id, 0,
                                       &d_frame, &d_pg, &dest_off, &dest_len, 1);
-        if ( rc != GNTST_okay )
-            goto error_out;
+        if ( rc != GNTST_okay ){
+            gdprintk(XENLOG_WARNING, "acquire_grant_for_copy failed .. dest_is_gref rc:%d source.domid:%d dest.domid:%d s_frame:%ld source_off:%u source_len:%u op->source.offset:%d op->len:%d\n", rc, op->source.domid, op->dest.domid, s_frame, dest_off, dest_len, op->dest.offset, op->len);
+	    goto error_out;
+	}
         have_d_grant = 1;
         if ( op->dest.offset < dest_off ||
              op->len > dest_len )

[-- Attachment #6: ifconfig-domu.txt --]
[-- Type: text/plain, Size: 792 bytes --]

eth0      Link encap:Ethernet  HWaddr 00:16:3e:3e:3c:b9  
          inet addr:192.168.1.9  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20127144 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13416829 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:11115605663 (10.3 GiB)  TX bytes:10954554508 (10.2 GiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


[-- Attachment #7: kernel-dom0-dotconfig.txt --]
[-- Type: text/plain, Size: 93690 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 3.13.6-20140311-xen314-netback-ca2f09-annie-wei Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_FHANDLE is not set
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_DOMAIN=y
# CONFIG_IRQ_DOMAIN_DEBUG is not set
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
CONFIG_RCU_STALL_COMMON=y
# CONFIG_RCU_USER_QS is not set
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_RCU_NOCB_CPU is not set
CONFIG_IKCONFIG=y
# CONFIG_IKCONFIG_PROC is not set
CONFIG_LOG_BUF_SHIFT=18
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_SUPPORTS_INT128=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
# CONFIG_NUMA_BALANCING is not set
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_FREEZER=y
# CONFIG_CGROUP_DEVICE is not set
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
# CONFIG_MEMCG is not set
# CONFIG_CGROUP_HUGETLB is not set
# CONFIG_CGROUP_PERF is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_CFS_BANDWIDTH is not set
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
# CONFIG_USER_NS is not set
CONFIG_PID_NS=y
CONFIG_NET_NS=y
# CONFIG_UIDGID_STRICT_TYPE_CHECKS is not set
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
# CONFIG_EXPERT is not set
CONFIG_UID16=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_PCI_QUIRKS=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
CONFIG_SLUB_CPU_PARTIAL=y
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
# CONFIG_KPROBES is not set
CONFIG_JUMP_LABEL=y
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
# CONFIG_SYSTEM_TRUSTED_KEYRING is not set
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
# CONFIG_BLK_DEV_BSGLIB is not set
CONFIG_BLK_DEV_INTEGRITY=y
# CONFIG_BLK_DEV_THROTTLING is not set
# CONFIG_BLK_CMDLINE_PARSER is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_X2APIC=y
# CONFIG_X86_MPPARSE is not set
# CONFIG_X86_EXTENDED_PLATFORM is not set
# CONFIG_X86_INTEL_LPSS is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_DEBUG=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_DEBUG_FS=y
CONFIG_XEN_PVH=y
# CONFIG_KVM_GUEST is not set
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
CONFIG_PARAVIRT_CLOCK=y
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
CONFIG_X86_THERMAL_VECTOR=y
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_MICROCODE_INTEL_EARLY is not set
# CONFIG_MICROCODE_AMD_EARLY is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=8
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
# CONFIG_MOVABLE_NODE is not set
# CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_NEED_BOUNCE_POOL=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_MEMORY_FAILURE is not set
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
# CONFIG_CLEANCACHE is not set
# CONFIG_FRONTSWAP is not set
# CONFIG_CMA is not set
# CONFIG_ZBUD is not set
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_X86_SMAP=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
CONFIG_HZ_300=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y

#
# Power management and ACPI options
#
# CONFIG_SUSPEND is not set
CONFIG_HIBERNATE_CALLBACKS=y
# CONFIG_HIBERNATION is not set
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
# CONFIG_PM_RUNTIME is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_ADVANCED_DEBUG is not set
CONFIG_PM_SLEEP_DEBUG=y
# CONFIG_PM_TRACE_RTC is not set
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
CONFIG_ACPI=y
CONFIG_ACPI_PROCFS=y
# CONFIG_ACPI_EC_DEBUGFS is not set
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=y
CONFIG_ACPI_FAN=y
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
CONFIG_ACPI_CUSTOM_DSDT_FILE=""
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_INITRD_TABLE_OVERRIDE=y
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set
CONFIG_ACPI_HED=y
# CONFIG_ACPI_CUSTOM_METHOD is not set
# CONFIG_ACPI_APEI is not set
# CONFIG_ACPI_EXTLOG is not set
# CONFIG_SFI is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_COMMON=y
# CONFIG_CPU_FREQ_STAT is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_GOV_POWERSAVE is not set
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set

#
# x86 CPU frequency scaling drivers
#
# CONFIG_X86_INTEL_PSTATE is not set
CONFIG_X86_PCC_CPUFREQ=y
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_ACPI_CPUFREQ_CPB=y
# CONFIG_X86_POWERNOW_K8 is not set
# CONFIG_X86_AMD_FREQ_SENSITIVITY is not set
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set

#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_MULTIPLE_DRIVERS is not set
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
# CONFIG_INTEL_IDLE is not set

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_XEN=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEAER=y
CONFIG_PCIE_ECRC=y
CONFIG_PCIEAER_INJECT=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEBUG=y
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
CONFIG_PCI_MSI=y
CONFIG_PCI_DEBUG=y
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
CONFIG_PCI_STUB=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y

#
# PCI host controller drivers
#
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
# CONFIG_PCCARD is not set
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=y
CONFIG_HOTPLUG_PCI_CPCI=y
# CONFIG_HOTPLUG_PCI_CPCI_ZT5550 is not set
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=y
CONFIG_HOTPLUG_PCI_SHPC=y
# CONFIG_RAPIDIO is not set
# CONFIG_X86_SYSFB is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_SCRIPT=y
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
CONFIG_COREDUMP=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
# CONFIG_X86_X32 is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_DIAG is not set
CONFIG_UNIX=y
# CONFIG_UNIX_DIAG is not set
# CONFIG_XFRM_USER is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_IP_FIB_TRIE_STATS is not set
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
# CONFIG_NET_IP_TUNNEL is not set
# CONFIG_IP_MROUTE is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
# CONFIG_INET_DIAG is not set
CONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
CONFIG_TCP_CONG_CUBIC=y
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not set
# CONFIG_TCP_CONG_HSTCP is not set
# CONFIG_TCP_CONG_HYBLA is not set
# CONFIG_TCP_CONG_VEGAS is not set
# CONFIG_TCP_CONG_SCALABLE is not set
# CONFIG_TCP_CONG_LP is not set
# CONFIG_TCP_CONG_VENO is not set
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_ACCT=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
# CONFIG_NF_CONNTRACK_TIMEOUT is not set
CONFIG_NF_CONNTRACK_TIMESTAMP=y
# CONFIG_NF_CT_PROTO_DCCP is not set
CONFIG_NF_CT_PROTO_GRE=y
# CONFIG_NF_CT_PROTO_SCTP is not set
# CONFIG_NF_CT_PROTO_UDPLITE is not set
# CONFIG_NF_CONNTRACK_AMANDA is not set
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_CONNTRACK_H323=y
CONFIG_NF_CONNTRACK_IRC=y
# CONFIG_NF_CONNTRACK_NETBIOS_NS is not set
# CONFIG_NF_CONNTRACK_SNMP is not set
CONFIG_NF_CONNTRACK_PPTP=y
# CONFIG_NF_CONNTRACK_SANE is not set
CONFIG_NF_CONNTRACK_SIP=y
# CONFIG_NF_CONNTRACK_TFTP is not set
CONFIG_NF_CT_NETLINK=y
# CONFIG_NF_CT_NETLINK_TIMEOUT is not set
# CONFIG_NETFILTER_NETLINK_QUEUE_CT is not set
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
# CONFIG_NF_NAT_AMANDA is not set
CONFIG_NF_NAT_FTP=y
CONFIG_NF_NAT_IRC=y
CONFIG_NF_NAT_SIP=y
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_TABLES is not set
CONFIG_NETFILTER_XTABLES=y

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=y
CONFIG_NETFILTER_XT_CONNMARK=y
# CONFIG_NETFILTER_XT_SET is not set

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=y
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=y
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
# CONFIG_NETFILTER_XT_TARGET_CT is not set
CONFIG_NETFILTER_XT_TARGET_DSCP=y
CONFIG_NETFILTER_XT_TARGET_HL=y
# CONFIG_NETFILTER_XT_TARGET_HMARK is not set
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=y
CONFIG_NETFILTER_XT_TARGET_LOG=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NETMAP=y
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
# CONFIG_NETFILTER_XT_TARGET_NOTRACK is not set
CONFIG_NETFILTER_XT_TARGET_RATEEST=y
CONFIG_NETFILTER_XT_TARGET_REDIRECT=y
CONFIG_NETFILTER_XT_TARGET_TEE=y
# CONFIG_NETFILTER_XT_TARGET_TPROXY is not set
# CONFIG_NETFILTER_XT_TARGET_TRACE is not set
CONFIG_NETFILTER_XT_TARGET_SECMARK=y
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
# CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=y
CONFIG_NETFILTER_XT_MATCH_COMMENT=y
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=y
# CONFIG_NETFILTER_XT_MATCH_CONNLABEL is not set
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=y
CONFIG_NETFILTER_XT_MATCH_CONNMARK=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_CPU=y
CONFIG_NETFILTER_XT_MATCH_DCCP=y
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=y
CONFIG_NETFILTER_XT_MATCH_DSCP=y
CONFIG_NETFILTER_XT_MATCH_ECN=y
CONFIG_NETFILTER_XT_MATCH_ESP=y
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y
CONFIG_NETFILTER_XT_MATCH_HELPER=y
CONFIG_NETFILTER_XT_MATCH_HL=y
CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
CONFIG_NETFILTER_XT_MATCH_IPVS=y
CONFIG_NETFILTER_XT_MATCH_LENGTH=y
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_NFACCT=y
CONFIG_NETFILTER_XT_MATCH_OSF=y
CONFIG_NETFILTER_XT_MATCH_OWNER=y
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=y
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
CONFIG_NETFILTER_XT_MATCH_QUOTA=y
CONFIG_NETFILTER_XT_MATCH_RATEEST=y
CONFIG_NETFILTER_XT_MATCH_REALM=y
CONFIG_NETFILTER_XT_MATCH_RECENT=y
# CONFIG_NETFILTER_XT_MATCH_SCTP is not set
# CONFIG_NETFILTER_XT_MATCH_SOCKET is not set
CONFIG_NETFILTER_XT_MATCH_STATE=y
CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
CONFIG_NETFILTER_XT_MATCH_STRING=y
CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
CONFIG_NETFILTER_XT_MATCH_TIME=y
CONFIG_NETFILTER_XT_MATCH_U32=y
CONFIG_IP_SET=y
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=y
CONFIG_IP_SET_BITMAP_IPMAC=y
CONFIG_IP_SET_BITMAP_PORT=y
CONFIG_IP_SET_HASH_IP=y
CONFIG_IP_SET_HASH_IPPORT=y
CONFIG_IP_SET_HASH_IPPORTIP=y
CONFIG_IP_SET_HASH_IPPORTNET=y
CONFIG_IP_SET_HASH_NETPORTNET=y
CONFIG_IP_SET_HASH_NET=y
CONFIG_IP_SET_HASH_NETNET=y
CONFIG_IP_SET_HASH_NETPORT=y
CONFIG_IP_SET_HASH_NETIFACE=y
CONFIG_IP_SET_LIST_SET=y
CONFIG_IP_VS=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
# CONFIG_IP_VS_PROTO_TCP is not set
# CONFIG_IP_VS_PROTO_UDP is not set
# CONFIG_IP_VS_PROTO_AH_ESP is not set
# CONFIG_IP_VS_PROTO_ESP is not set
# CONFIG_IP_VS_PROTO_AH is not set
# CONFIG_IP_VS_PROTO_SCTP is not set

#
# IPVS scheduler
#
# CONFIG_IP_VS_RR is not set
# CONFIG_IP_VS_WRR is not set
# CONFIG_IP_VS_LC is not set
# CONFIG_IP_VS_WLC is not set
# CONFIG_IP_VS_LBLC is not set
# CONFIG_IP_VS_LBLCR is not set
# CONFIG_IP_VS_DH is not set
# CONFIG_IP_VS_SH is not set
# CONFIG_IP_VS_SED is not set
# CONFIG_IP_VS_NQ is not set

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS application helper
#
CONFIG_IP_VS_NFCT=y

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_AH=y
CONFIG_IP_NF_MATCH_ECN=y
# CONFIG_IP_NF_MATCH_RPFILTER is not set
CONFIG_IP_NF_MATCH_TTL=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
# CONFIG_IP_NF_TARGET_SYNPROXY is not set
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_NF_NAT_IPV4=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_NF_NAT_PROTO_GRE=y
CONFIG_NF_NAT_PPTP=y
CONFIG_NF_NAT_H323=y
CONFIG_IP_NF_MANGLE=y
# CONFIG_IP_NF_TARGET_CLUSTERIP is not set
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_TTL is not set
CONFIG_IP_NF_RAW=y
# CONFIG_IP_NF_ARPTABLES is not set
CONFIG_BRIDGE_NF_EBTABLES=y
# CONFIG_BRIDGE_EBT_BROUTE is not set
# CONFIG_BRIDGE_EBT_T_FILTER is not set
# CONFIG_BRIDGE_EBT_T_NAT is not set
# CONFIG_BRIDGE_EBT_802_3 is not set
# CONFIG_BRIDGE_EBT_AMONG is not set
# CONFIG_BRIDGE_EBT_ARP is not set
# CONFIG_BRIDGE_EBT_IP is not set
# CONFIG_BRIDGE_EBT_LIMIT is not set
# CONFIG_BRIDGE_EBT_MARK is not set
# CONFIG_BRIDGE_EBT_PKTTYPE is not set
# CONFIG_BRIDGE_EBT_STP is not set
# CONFIG_BRIDGE_EBT_VLAN is not set
# CONFIG_BRIDGE_EBT_ARPREPLY is not set
# CONFIG_BRIDGE_EBT_DNAT is not set
# CONFIG_BRIDGE_EBT_MARK_T is not set
# CONFIG_BRIDGE_EBT_REDIRECT is not set
# CONFIG_BRIDGE_EBT_SNAT is not set
# CONFIG_BRIDGE_EBT_LOG is not set
# CONFIG_BRIDGE_EBT_ULOG is not set
# CONFIG_BRIDGE_EBT_NFLOG is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
CONFIG_STP=y
CONFIG_BRIDGE=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_HAVE_NET_DSA=y
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
# CONFIG_NET_SCH_CBQ is not set
# CONFIG_NET_SCH_HTB is not set
# CONFIG_NET_SCH_HFSC is not set
# CONFIG_NET_SCH_PRIO is not set
# CONFIG_NET_SCH_MULTIQ is not set
# CONFIG_NET_SCH_RED is not set
# CONFIG_NET_SCH_SFB is not set
# CONFIG_NET_SCH_SFQ is not set
# CONFIG_NET_SCH_TEQL is not set
# CONFIG_NET_SCH_TBF is not set
# CONFIG_NET_SCH_GRED is not set
# CONFIG_NET_SCH_DSMARK is not set
# CONFIG_NET_SCH_NETEM is not set
# CONFIG_NET_SCH_DRR is not set
# CONFIG_NET_SCH_MQPRIO is not set
# CONFIG_NET_SCH_CHOKE is not set
# CONFIG_NET_SCH_QFQ is not set
# CONFIG_NET_SCH_CODEL is not set
# CONFIG_NET_SCH_FQ_CODEL is not set
# CONFIG_NET_SCH_FQ is not set
# CONFIG_NET_SCH_INGRESS is not set
# CONFIG_NET_SCH_PLUG is not set

#
# Classification
#
CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
# CONFIG_NET_CLS_TCINDEX is not set
# CONFIG_NET_CLS_ROUTE4 is not set
# CONFIG_NET_CLS_FW is not set
# CONFIG_NET_CLS_U32 is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_CLS_FLOW is not set
# CONFIG_NET_CLS_CGROUP is not set
# CONFIG_NET_CLS_BPF is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
# CONFIG_NET_EMATCH_CMP is not set
# CONFIG_NET_EMATCH_NBYTE is not set
# CONFIG_NET_EMATCH_U32 is not set
# CONFIG_NET_EMATCH_META is not set
# CONFIG_NET_EMATCH_TEXT is not set
# CONFIG_NET_EMATCH_IPSET is not set
CONFIG_NET_CLS_ACT=y
# CONFIG_NET_ACT_POLICE is not set
# CONFIG_NET_ACT_GACT is not set
# CONFIG_NET_ACT_MIRRED is not set
# CONFIG_NET_ACT_IPT is not set
# CONFIG_NET_ACT_NAT is not set
# CONFIG_NET_ACT_PEDIT is not set
# CONFIG_NET_ACT_SIMP is not set
# CONFIG_NET_ACT_SKBEDIT is not set
# CONFIG_NET_ACT_CSUM is not set
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set
# CONFIG_DNS_RESOLVER is not set
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
# CONFIG_VSOCKETS is not set
# CONFIG_NETLINK_MMAP is not set
# CONFIG_NETLINK_DIAG is not set
# CONFIG_NET_MPLS_GSO is not set
# CONFIG_HSR is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
# CONFIG_NETPRIO_CGROUP is not set
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
# CONFIG_BPF_JIT is not set
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
CONFIG_BT=y
CONFIG_BT_RFCOMM=y
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=y
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=y

#
# Bluetooth device drivers
#
CONFIG_BT_HCIBTUSB=y
CONFIG_BT_HCIUART=y
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_ATH3K=y
CONFIG_BT_HCIUART_LL=y
CONFIG_BT_HCIUART_3WIRE=y
CONFIG_BT_HCIBCM203X=y
CONFIG_BT_HCIBPA10X=y
CONFIG_BT_HCIBFUSB=y
CONFIG_BT_HCIVHCI=y
CONFIG_BT_MRVL=y
CONFIG_BT_ATH3K=y
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
CONFIG_CEPH_LIB=y
# CONFIG_CEPH_LIB_PRETTYDEBUG is not set
# CONFIG_CEPH_LIB_USE_DNS_RESOLVER is not set
# CONFIG_NFC is not set
CONFIG_HAVE_BPF_JIT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# CONFIG_STANDALONE is not set
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_DEBUG_DRIVER is not set
CONFIG_DEBUG_DEVRES=y
CONFIG_SYS_HYPERVISOR=y
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_DMA_SHARED_BUFFER=y

#
# Bus devices
#
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_NULL_BLK is not set
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_DEV_PCIESSD_MTIP32XX is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_NVME is not set
# CONFIG_BLK_DEV_SKD is not set
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
# CONFIG_BLK_DEV_HD is not set
# CONFIG_BLK_DEV_RBD is not set
# CONFIG_BLK_DEV_RSXX is not set

#
# Misc devices
#
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ATMEL_SSC is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_APDS9802ALS is not set
# CONFIG_ISL29003 is not set
# CONFIG_ISL29020 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_SENSORS_BH1780 is not set
# CONFIG_SENSORS_BH1770 is not set
# CONFIG_SENSORS_APDS990X is not set
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
# CONFIG_VMWARE_BALLOON is not set
# CONFIG_BMP085_I2C is not set
# CONFIG_PCH_PHUB is not set
# CONFIG_USB_SWITCH_FSA9480 is not set
# CONFIG_SRAM is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_CB710_CORE is not set

#
# Texas Instruments shared transport line discipline
#
# CONFIG_SENSORS_LIS3_I2C is not set

#
# Altera FPGA firmware download module
#
CONFIG_ALTERA_STAPL=y
# CONFIG_INTEL_MEI is not set
# CONFIG_INTEL_MEI_ME is not set
# CONFIG_VMWARE_VMCI is not set

#
# Intel MIC Host Driver
#
# CONFIG_INTEL_MIC_HOST is not set

#
# Intel MIC Card Driver
#
# CONFIG_INTEL_MIC_CARD is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
# CONFIG_SCSI_LOWLEVEL is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
# CONFIG_SATA_ZPODD is not set
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
CONFIG_SATA_AHCI_PLATFORM=y
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
# CONFIG_SATA_SIL24 is not set
# CONFIG_ATA_SFF is not set
CONFIG_MD=y
# CONFIG_BLK_DEV_MD is not set
CONFIG_BCACHE=y
# CONFIG_BCACHE_DEBUG is not set
# CONFIG_BCACHE_CLOSURES_DEBUG is not set
CONFIG_BLK_DEV_DM_BUILTIN=y
CONFIG_BLK_DEV_DM=y
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=y
CONFIG_DM_BIO_PRISON=y
CONFIG_DM_PERSISTENT_DATA=y
CONFIG_DM_CRYPT=y
CONFIG_DM_SNAPSHOT=y
# CONFIG_DM_THIN_PROVISIONING is not set
CONFIG_DM_CACHE=y
CONFIG_DM_CACHE_MQ=y
CONFIG_DM_CACHE_CLEANER=y
CONFIG_DM_MIRROR=y
# CONFIG_DM_LOG_USERSPACE is not set
# CONFIG_DM_RAID is not set
CONFIG_DM_ZERO=y
# CONFIG_DM_MULTIPATH is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
# CONFIG_DM_FLAKEY is not set
# CONFIG_DM_VERITY is not set
# CONFIG_DM_SWITCH is not set
# CONFIG_TARGET_CORE is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_FIREWIRE_NOSY is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y
# CONFIG_BONDING is not set
# CONFIG_DUMMY is not set
# CONFIG_EQUALIZER is not set
# CONFIG_NET_FC is not set
# CONFIG_IFB is not set
# CONFIG_NET_TEAM is not set
# CONFIG_MACVLAN is not set
# CONFIG_VXLAN is not set
CONFIG_NETCONSOLE=y
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=y
CONFIG_VETH=y
# CONFIG_NLMON is not set
# CONFIG_ARCNET is not set

#
# CAIF transport drivers
#

#
# Distributed Switch Architecture drivers
#
# CONFIG_NET_DSA_MV88E6XXX is not set
# CONFIG_NET_DSA_MV88E6060 is not set
# CONFIG_NET_DSA_MV88E6XXX_NEED_PPU is not set
# CONFIG_NET_DSA_MV88E6131 is not set
# CONFIG_NET_DSA_MV88E6123_61_65 is not set
CONFIG_ETHERNET=y
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
# CONFIG_NET_VENDOR_ALTEON is not set
# CONFIG_NET_VENDOR_AMD is not set
CONFIG_NET_VENDOR_ARC=y
# CONFIG_NET_VENDOR_ATHEROS is not set
CONFIG_NET_CADENCE=y
# CONFIG_ARM_AT91_ETHER is not set
# CONFIG_MACB is not set
# CONFIG_NET_VENDOR_BROADCOM is not set
# CONFIG_NET_VENDOR_BROCADE is not set
# CONFIG_NET_CALXEDA_XGMAC is not set
# CONFIG_NET_VENDOR_CHELSIO is not set
# CONFIG_NET_VENDOR_CISCO is not set
# CONFIG_DNET is not set
# CONFIG_NET_VENDOR_DEC is not set
# CONFIG_NET_VENDOR_DLINK is not set
# CONFIG_NET_VENDOR_EMULEX is not set
# CONFIG_NET_VENDOR_EXAR is not set
# CONFIG_NET_VENDOR_HP is not set
CONFIG_NET_VENDOR_INTEL=y
# CONFIG_E100 is not set
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_IGB=y
CONFIG_IGB_HWMON=y
CONFIG_IGBVF=y
# CONFIG_IXGB is not set
# CONFIG_IXGBE is not set
# CONFIG_IXGBEVF is not set
# CONFIG_I40E is not set
CONFIG_NET_VENDOR_I825XX=y
# CONFIG_IP1000 is not set
# CONFIG_JME is not set
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MELLANOX is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_MYRI is not set
# CONFIG_FEALNX is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
# CONFIG_NET_VENDOR_NVIDIA is not set
# CONFIG_NET_VENDOR_OKI is not set
# CONFIG_ETHOC is not set
# CONFIG_NET_PACKET_ENGINE is not set
# CONFIG_NET_VENDOR_QLOGIC is not set
CONFIG_NET_VENDOR_REALTEK=y
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
CONFIG_R8169=y
# CONFIG_SH_ETH is not set
# CONFIG_NET_VENDOR_RDC is not set
CONFIG_NET_VENDOR_SEEQ=y
CONFIG_NET_VENDOR_SILAN=y
# CONFIG_SC92031 is not set
# CONFIG_NET_VENDOR_SIS is not set
# CONFIG_SFC is not set
# CONFIG_NET_VENDOR_SMSC is not set
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SUN is not set
# CONFIG_NET_VENDOR_TEHUTI is not set
# CONFIG_NET_VENDOR_TI is not set
# CONFIG_NET_VENDOR_VIA is not set
CONFIG_NET_VENDOR_WIZNET=y
# CONFIG_WIZNET_W5100 is not set
# CONFIG_WIZNET_W5300 is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_NET_SB1000 is not set
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
# CONFIG_AT803X_PHY is not set
# CONFIG_AMD_PHY is not set
# CONFIG_MARVELL_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_BCM87XX_PHY is not set
# CONFIG_ICPLUS_PHY is not set
CONFIG_REALTEK_PHY=y
# CONFIG_NATIONAL_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_RTL8152 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_IPHETH is not set
# CONFIG_WLAN is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=y
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_CYAPA is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
# CONFIG_MOUSE_SYNAPTICS_USB is not set
# CONFIG_INPUT_JOYSTICK is not set
CONFIG_INPUT_TABLET=y
# CONFIG_TABLET_USB_ACECAD is not set
# CONFIG_TABLET_USB_AIPTEK is not set
# CONFIG_TABLET_USB_GTCO is not set
# CONFIG_TABLET_USB_HANWANG is not set
# CONFIG_TABLET_USB_KBTAB is not set
# CONFIG_TABLET_USB_WACOM is not set
CONFIG_INPUT_TOUCHSCREEN=y
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_ATMEL_MXT is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_CYTTSP_CORE is not set
# CONFIG_TOUCHSCREEN_CYTTSP4_CORE is not set
# CONFIG_TOUCHSCREEN_DYNAPRO is not set
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
# CONFIG_TOUCHSCREEN_ILI210X is not set
# CONFIG_TOUCHSCREEN_GUNZE is not set
# CONFIG_TOUCHSCREEN_ELO is not set
# CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
# CONFIG_TOUCHSCREEN_WACOM_I2C is not set
# CONFIG_TOUCHSCREEN_MAX11801 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
# CONFIG_TOUCHSCREEN_MMS114 is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_INEXIO is not set
# CONFIG_TOUCHSCREEN_MK712 is not set
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_EDT_FT5X06 is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
# CONFIG_TOUCHSCREEN_PIXCIR is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC_SERIO is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
# CONFIG_TOUCHSCREEN_ST1232 is not set
# CONFIG_TOUCHSCREEN_SUR40 is not set
# CONFIG_TOUCHSCREEN_TPS6507X is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_BMA150 is not set
# CONFIG_INPUT_PCSPKR is not set
# CONFIG_INPUT_MMA8450 is not set
# CONFIG_INPUT_MPU3050 is not set
# CONFIG_INPUT_APANEL is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_KXTJ9 is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_IMS_PCU is not set
# CONFIG_INPUT_CMA3000 is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_SERIO_ARC_PS2 is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_ROCKETPORT is not set
# CONFIG_CYCLADES is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_SYNCLINK is not set
# CONFIG_SYNCLINKMP is not set
# CONFIG_SYNCLINK_GT is not set
# CONFIG_NOZOMI is not set
# CONFIG_ISI is not set
# CONFIG_N_HDLC is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
# CONFIG_DEVKMEM is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_RSA=y
# CONFIG_SERIAL_8250_DW is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MFD_HSU is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_PCH_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=y
CONFIG_HW_RANDOM_INTEL=y
CONFIG_HW_RANDOM_AMD=y
CONFIG_HW_RANDOM_VIA=y
# CONFIG_NVRAM is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
# CONFIG_HPET_MMAP is not set
CONFIG_HANGCHECK_TIMER=y
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
# CONFIG_I2C_CHARDEV is not set
CONFIG_I2C_MUX=y

#
# Multiplexer I2C Chip support
#
# CONFIG_I2C_MUX_PCA9541 is not set
# CONFIG_I2C_MUX_PCA954x is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=y
# CONFIG_I2C_AMD756_S4882 is not set
CONFIG_I2C_AMD8111=y
CONFIG_I2C_I801=y
CONFIG_I2C_ISCH=y
# CONFIG_I2C_ISMT is not set
CONFIG_I2C_PIIX4=y
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set

#
# ACPI drivers
#
CONFIG_I2C_SCMI=y

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_EG20T is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_PXA_PCI is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_SPI is not set
# CONFIG_HSI is not set

#
# PPS support
#
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
# CONFIG_PPS_CLIENT_LDISC is not set
# CONFIG_PPS_CLIENT_GPIO is not set

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=y

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# CONFIG_PTP_1588_CLOCK_PCH is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2781 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_SBS is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_MAX8903 is not set
# CONFIG_CHARGER_LP8727 is not set
# CONFIG_CHARGER_BQ2415X is not set
# CONFIG_CHARGER_SMB347 is not set
# CONFIG_POWER_RESET is not set
# CONFIG_POWER_AVS is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7410 is not set
# CONFIG_SENSORS_ADT7411 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_ASC7621 is not set
# CONFIG_SENSORS_K8TEMP is not set
CONFIG_SENSORS_K10TEMP=y
CONFIG_SENSORS_FAM15H_POWER=y
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS620 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
CONFIG_SENSORS_F71805F=y
CONFIG_SENSORS_F71882FG=y
CONFIG_SENSORS_F75375S=y
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_G762 is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_HIH6130 is not set
# CONFIG_SENSORS_HTU21 is not set
# CONFIG_SENSORS_CORETEMP is not set
CONFIG_SENSORS_IT87=y
CONFIG_SENSORS_JC42=y
# CONFIG_SENSORS_LINEAGE is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM73 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4151 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LTC4261 is not set
# CONFIG_SENSORS_LM95234 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_LM95245 is not set
# CONFIG_SENSORS_MAX16065 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX1668 is not set
# CONFIG_SENSORS_MAX197 is not set
# CONFIG_SENSORS_MAX6639 is not set
# CONFIG_SENSORS_MAX6642 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_MAX6697 is not set
# CONFIG_SENSORS_MCP3021 is not set
# CONFIG_SENSORS_NCT6775 is not set
# CONFIG_SENSORS_NTC_THERMISTOR is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_PMBUS is not set
# CONFIG_SENSORS_SHT21 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_EMC1403 is not set
# CONFIG_SENSORS_EMC2103 is not set
# CONFIG_SENSORS_EMC6W201 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_SCH56XX_COMMON is not set
# CONFIG_SENSORS_SCH5627 is not set
# CONFIG_SENSORS_SCH5636 is not set
# CONFIG_SENSORS_ADS1015 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_AMC6821 is not set
# CONFIG_SENSORS_INA209 is not set
# CONFIG_SENSORS_INA2XX is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_TMP421 is not set
# CONFIG_SENSORS_VIA_CPUTEMP is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83795 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_APPLESMC is not set

#
# ACPI drivers
#
CONFIG_SENSORS_ACPI_POWER=y
# CONFIG_SENSORS_ATK0110 is not set
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
# CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set
# CONFIG_THERMAL_GOV_FAIR_SHARE is not set
CONFIG_THERMAL_GOV_STEP_WISE=y
# CONFIG_THERMAL_GOV_USER_SPACE is not set
# CONFIG_CPU_THERMAL is not set
# CONFIG_THERMAL_EMULATION is not set
# CONFIG_INTEL_POWERCLAMP is not set
# CONFIG_X86_PKG_TEMP_THERMAL is not set

#
# Texas Instruments thermal drivers
#
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
# CONFIG_SOFT_WATCHDOG is not set
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_F71808E_WDT is not set
CONFIG_SP5100_TCO=y
# CONFIG_SC520_WDT is not set
# CONFIG_SBC_FITPC2_WATCHDOG is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
# CONFIG_I6300ESB_WDT is not set
# CONFIG_IE6XX_WDT is not set
# CONFIG_ITCO_WDT is not set
# CONFIG_IT8712F_WDT is not set
# CONFIG_IT87_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
# CONFIG_NV_TCO is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC_SCH311X_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
# CONFIG_VIA_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83697HF_WDT is not set
# CONFIG_W83697UG_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
CONFIG_XEN_WDT=y

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y

#
# Broadcom specific AMBA
#
# CONFIG_BCMA is not set

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
# CONFIG_MFD_CS5535 is not set
# CONFIG_MFD_AS3711 is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_CROS_EC is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
# CONFIG_MFD_DA9063 is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_LPC_ICH is not set
CONFIG_LPC_SCH=y
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_88PM800 is not set
# CONFIG_MFD_88PM805 is not set
# CONFIG_MFD_88PM860X is not set
# CONFIG_MFD_MAX77686 is not set
# CONFIG_MFD_MAX77693 is not set
# CONFIG_MFD_MAX8907 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_VIPERBOARD is not set
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_RTSX_PCI is not set
# CONFIG_MFD_RC5T583 is not set
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_MFD_SMSC is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_STMPE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_PALMAS is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS65090 is not set
# CONFIG_MFD_TPS65217 is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS80031 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_TWL6040_CORE is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TC3589X is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_MFD_VX855 is not set
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# CONFIG_REGULATOR is not set
CONFIG_MEDIA_SUPPORT=y

#
# Multimedia core support
#
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_ANALOG_TV_SUPPORT=y
CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y
CONFIG_MEDIA_RADIO_SUPPORT=y
CONFIG_MEDIA_RC_SUPPORT=y
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=y
CONFIG_VIDEO_V4L2=y
CONFIG_VIDEO_ADV_DEBUG=y
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_TUNER=y
CONFIG_VIDEOBUF_GEN=y
CONFIG_VIDEOBUF_DMA_SG=y
# CONFIG_VIDEO_V4L2_INT_DEVICE is not set
CONFIG_DVB_CORE=y
CONFIG_DVB_NET=y
# CONFIG_TTPCI_EEPROM is not set
CONFIG_DVB_MAX_ADAPTERS=8
# CONFIG_DVB_DYNAMIC_MINORS is not set

#
# Media drivers
#
CONFIG_RC_CORE=y
CONFIG_RC_MAP=y
CONFIG_RC_DECODERS=y
CONFIG_LIRC=y
CONFIG_IR_LIRC_CODEC=y
CONFIG_IR_NEC_DECODER=y
CONFIG_IR_RC5_DECODER=y
CONFIG_IR_RC6_DECODER=y
CONFIG_IR_JVC_DECODER=y
CONFIG_IR_SONY_DECODER=y
CONFIG_IR_RC5_SZ_DECODER=y
CONFIG_IR_SANYO_DECODER=y
CONFIG_IR_MCE_KBD_DECODER=y
# CONFIG_RC_DEVICES is not set
CONFIG_MEDIA_USB_SUPPORT=y

#
# Webcam devices
#
# CONFIG_USB_VIDEO_CLASS is not set
# CONFIG_USB_GSPCA is not set
# CONFIG_USB_PWC is not set
# CONFIG_VIDEO_CPIA2 is not set
# CONFIG_USB_ZR364XX is not set
# CONFIG_USB_STKWEBCAM is not set
# CONFIG_USB_S2255 is not set
# CONFIG_USB_SN9C102 is not set
# CONFIG_VIDEO_USBTV is not set

#
# Analog TV USB devices
#
CONFIG_VIDEO_PVRUSB2=y
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
# CONFIG_VIDEO_HDPVR is not set
# CONFIG_VIDEO_TLG2300 is not set
# CONFIG_VIDEO_USBVISION is not set
# CONFIG_VIDEO_STK1160_COMMON is not set

#
# Analog/digital TV USB devices
#
# CONFIG_VIDEO_AU0828 is not set
# CONFIG_VIDEO_CX231XX is not set
# CONFIG_VIDEO_TM6000 is not set

#
# Digital TV USB devices
#
# CONFIG_DVB_USB is not set
# CONFIG_DVB_USB_V2 is not set
# CONFIG_DVB_TTUSB_BUDGET is not set
# CONFIG_DVB_TTUSB_DEC is not set
# CONFIG_SMS_USB_DRV is not set
# CONFIG_DVB_B2C2_FLEXCOP_USB is not set

#
# Webcam, TV (analog/digital) USB devices
#
# CONFIG_VIDEO_EM28XX is not set
CONFIG_MEDIA_PCI_SUPPORT=y

#
# Media capture support
#

#
# Media capture/analog TV support
#
# CONFIG_VIDEO_IVTV is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_MXB is not set

#
# Media capture/analog/hybrid TV support
#
# CONFIG_VIDEO_CX18 is not set
# CONFIG_VIDEO_CX23885 is not set
CONFIG_VIDEO_CX25821=y
# CONFIG_VIDEO_CX25821_ALSA is not set
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_BT848 is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_SAA7164 is not set

#
# Media digital TV PCI Adapters
#
# CONFIG_DVB_AV7110 is not set
# CONFIG_DVB_BUDGET_CORE is not set
# CONFIG_DVB_B2C2_FLEXCOP_PCI is not set
# CONFIG_DVB_PLUTO2 is not set
# CONFIG_DVB_DM1105 is not set
# CONFIG_DVB_PT1 is not set
# CONFIG_MANTIS_CORE is not set
# CONFIG_DVB_NGENE is not set
# CONFIG_DVB_DDBRIDGE is not set
# CONFIG_V4L_PLATFORM_DRIVERS is not set
# CONFIG_V4L_MEM2MEM_DRIVERS is not set
# CONFIG_V4L_TEST_DRIVERS is not set

#
# Supported MMC/SDIO adapters
#
# CONFIG_RADIO_ADAPTERS is not set
CONFIG_VIDEO_CX2341X=y
CONFIG_VIDEO_BTCX=y
CONFIG_VIDEO_TVEEPROM=y
# CONFIG_CYPRESS_FIRMWARE is not set

#
# Media ancillary drivers (tuners, sensors, i2c, frontends)
#
CONFIG_MEDIA_SUBDRV_AUTOSELECT=y
CONFIG_MEDIA_ATTACH=y
CONFIG_VIDEO_IR_I2C=y

#
# Audio decoders, processors and mixers
#
CONFIG_VIDEO_MSP3400=y
CONFIG_VIDEO_CS53L32A=y
CONFIG_VIDEO_WM8775=y

#
# RDS decoders
#

#
# Video decoders
#
CONFIG_VIDEO_SAA711X=y

#
# Video and audio decoders
#
CONFIG_VIDEO_CX25840=y

#
# Video encoders
#

#
# Camera sensor devices
#

#
# Flash devices
#

#
# Video improvement chips
#

#
# Miscellaneous helper chips
#

#
# Sensors used on soc_camera driver
#
CONFIG_MEDIA_TUNER=y
CONFIG_MEDIA_TUNER_SIMPLE=y
CONFIG_MEDIA_TUNER_TDA8290=y
CONFIG_MEDIA_TUNER_TDA827X=y
CONFIG_MEDIA_TUNER_TDA18271=y
CONFIG_MEDIA_TUNER_TDA9887=y
CONFIG_MEDIA_TUNER_TEA5761=y
CONFIG_MEDIA_TUNER_TEA5767=y
CONFIG_MEDIA_TUNER_MT20XX=y
CONFIG_MEDIA_TUNER_XC2028=y
CONFIG_MEDIA_TUNER_XC5000=y
CONFIG_MEDIA_TUNER_XC4000=y
CONFIG_MEDIA_TUNER_MC44S803=y

#
# Multistandard (satellite) frontends
#

#
# Multistandard (cable + terrestrial) frontends
#

#
# DVB-S (satellite) frontends
#

#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_TDA10048=y

#
# DVB-C (cable) frontends
#

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_LGDT330X=y
CONFIG_DVB_S5H1409=y
CONFIG_DVB_S5H1411=y

#
# ISDB-T (terrestrial) frontends
#

#
# Digital terrestrial only tuners/PLL
#

#
# SEC control devices for DVB-S
#

#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=y
CONFIG_DRM_KMS_HELPER=y
CONFIG_DRM_KMS_FB_HELPER=y
# CONFIG_DRM_LOAD_EDID_FIRMWARE is not set
CONFIG_DRM_TTM=y

#
# I2C encoder or helper chips
#
# CONFIG_DRM_I2C_CH7006 is not set
# CONFIG_DRM_I2C_SIL164 is not set
# CONFIG_DRM_I2C_NXP_TDA998X is not set
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
CONFIG_DRM_RADEON=y
# CONFIG_DRM_RADEON_UMS is not set
# CONFIG_DRM_NOUVEAU is not set
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
# CONFIG_DRM_VMWGFX is not set
# CONFIG_DRM_GMA500 is not set
# CONFIG_DRM_UDL is not set
# CONFIG_DRM_AST is not set
# CONFIG_DRM_MGAG200 is not set
CONFIG_DRM_CIRRUS_QEMU=y
CONFIG_DRM_QXL=y
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=y
CONFIG_HDMI=y
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
# CONFIG_FB_DDC is not set
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_IMAGEBLIT=y
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=y
CONFIG_FB_DEFERRED_IO=y
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I740 is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_TMIO is not set
# CONFIG_FB_SMSCUFX is not set
CONFIG_FB_UDL=y
# CONFIG_FB_GOLDFISH is not set
# CONFIG_FB_VIRTUAL is not set
CONFIG_XEN_FBDEV_FRONTEND=y
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
# CONFIG_FB_AUO_K190X is not set
# CONFIG_FB_SIMPLE is not set
# CONFIG_EXYNOS_VIDEO is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=y
# CONFIG_BACKLIGHT_APPLE is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_ADP8870 is not set
# CONFIG_BACKLIGHT_LM3630A is not set
# CONFIG_BACKLIGHT_LM3639 is not set
# CONFIG_BACKLIGHT_LP855X is not set
# CONFIG_BACKLIGHT_LV5207LP is not set
# CONFIG_BACKLIGHT_BD6107 is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_SEQUENCER=y
CONFIG_SND_SEQ_DUMMY=y
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=y
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_MAX_CARDS=32
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=y
CONFIG_SND_OPL3_LIB_SEQ=y
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
# CONFIG_SND_EMU10K1_SEQ is not set
CONFIG_SND_MPU401_UART=y
CONFIG_SND_OPL3_LIB=y
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_ALOOP is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ASIHPI is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
CONFIG_SND_CMIPCI=y
CONFIG_SND_OXYGEN_LIB=y
CONFIG_SND_OXYGEN=y
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_CTXFI is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=y
CONFIG_SND_HDA_PREALLOC_SIZE=64
CONFIG_SND_HDA_HWDEP=y
# CONFIG_SND_HDA_RECONFIG is not set
# CONFIG_SND_HDA_INPUT_BEEP is not set
# CONFIG_SND_HDA_INPUT_JACK is not set
# CONFIG_SND_HDA_PATCH_LOADER is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CA0132=y
# CONFIG_SND_HDA_CODEC_CA0132_DSP is not set
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_LOLA is not set
# CONFIG_SND_LX6464ES is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=y
CONFIG_SND_USB_UA101=y
CONFIG_SND_USB_USX2Y=y
CONFIG_SND_USB_CAIAQ=y
CONFIG_SND_USB_CAIAQ_INPUT=y
# CONFIG_SND_USB_US122L is not set
CONFIG_SND_USB_6FIRE=y
# CONFIG_SND_USB_HIFACE is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set

#
# HID support
#
CONFIG_HID=y
# CONFIG_HID_BATTERY_STRENGTH is not set
CONFIG_HIDRAW=y
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACRUX is not set
CONFIG_HID_APPLE=y
# CONFIG_HID_APPLEIR is not set
# CONFIG_HID_AUREAL is not set
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_PRODIKEYS is not set
CONFIG_HID_CYPRESS=y
# CONFIG_HID_DRAGONRISE is not set
# CONFIG_HID_EMS_FF is not set
# CONFIG_HID_ELECOM is not set
# CONFIG_HID_ELO is not set
CONFIG_HID_EZKEY=y
# CONFIG_HID_HOLTEK is not set
# CONFIG_HID_HUION is not set
# CONFIG_HID_KEYTOUCH is not set
# CONFIG_HID_KYE is not set
# CONFIG_HID_UCLOGIC is not set
# CONFIG_HID_WALTOP is not set
# CONFIG_HID_GYRATION is not set
# CONFIG_HID_ICADE is not set
# CONFIG_HID_TWINHAN is not set
CONFIG_HID_KENSINGTON=y
# CONFIG_HID_LCPOWER is not set
# CONFIG_HID_LENOVO_TPKBD is not set
CONFIG_HID_LOGITECH=y
# CONFIG_HID_LOGITECH_DJ is not set
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_LOGIWHEELS_FF is not set
# CONFIG_HID_MAGICMOUSE is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
# CONFIG_HID_MULTITOUCH is not set
# CONFIG_HID_NTRIG is not set
# CONFIG_HID_ORTEK is not set
# CONFIG_HID_PANTHERLORD is not set
# CONFIG_HID_PETALYNX is not set
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_PRIMAX is not set
# CONFIG_HID_ROCCAT is not set
# CONFIG_HID_SAITEK is not set
# CONFIG_HID_SAMSUNG is not set
# CONFIG_HID_SONY is not set
# CONFIG_HID_SPEEDLINK is not set
# CONFIG_HID_STEELSERIES is not set
# CONFIG_HID_SUNPLUS is not set
# CONFIG_HID_GREENASIA is not set
# CONFIG_HID_SMARTJOYPLUS is not set
# CONFIG_HID_TIVO is not set
# CONFIG_HID_TOPSEED is not set
# CONFIG_HID_THINGM is not set
# CONFIG_HID_THRUSTMASTER is not set
# CONFIG_HID_WACOM is not set
# CONFIG_HID_WIIMOTE is not set
# CONFIG_HID_XINMO is not set
# CONFIG_HID_ZEROPLUS is not set
# CONFIG_HID_ZYDACRON is not set
# CONFIG_HID_SENSOR_HUB is not set

#
# USB HID support
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# I2C HID support
#
# CONFIG_I2C_HID is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
CONFIG_USB_DEFAULT_PERSIST=y
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
# CONFIG_USB_EHCI_HCD_PLATFORM is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
# CONFIG_USB_FUSBH200_HCD is not set
# CONFIG_USB_FOTG210_HCD is not set
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PCI=y
# CONFIG_USB_OHCI_HCD_PLATFORM is not set
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_HCD_TEST_MODE is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=y
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_STORAGE_ENE_UB6250 is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_DWC3 is not set
# CONFIG_USB_CHIPIDEA is not set

#
# USB port drivers
#
CONFIG_USB_SERIAL=y
# CONFIG_USB_SERIAL_CONSOLE is not set
# CONFIG_USB_SERIAL_GENERIC is not set
# CONFIG_USB_SERIAL_SIMPLE is not set
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
CONFIG_USB_SERIAL_CP210X=y
CONFIG_USB_SERIAL_CYPRESS_M8=y
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_F81232 is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_METRO is not set
CONFIG_USB_SERIAL_MOS7720=y
CONFIG_USB_SERIAL_MOS7840=y
# CONFIG_USB_SERIAL_NAVMAN is not set
# CONFIG_USB_SERIAL_PL2303 is not set
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QCAUX is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_SYMBOL is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OPTION is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_XSENS_MT is not set
# CONFIG_USB_SERIAL_WISHBONE is not set
# CONFIG_USB_SERIAL_ZTE is not set
# CONFIG_USB_SERIAL_SSU100 is not set
# CONFIG_USB_SERIAL_QT2 is not set
# CONFIG_USB_SERIAL_DEBUG is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_EHSET_TEST_FIXTURE is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_YUREX is not set
# CONFIG_USB_EZUSB_FX2 is not set
# CONFIG_USB_HSIC_USB3503 is not set

#
# USB Physical Layer drivers
#
# CONFIG_USB_PHY is not set
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_SAMSUNG_USB2PHY is not set
# CONFIG_SAMSUNG_USB3PHY is not set
# CONFIG_USB_ISP1301 is not set
# CONFIG_USB_RCAR_PHY is not set
# CONFIG_USB_GADGET is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
# CONFIG_LEDS_LM3530 is not set
# CONFIG_LEDS_LM3642 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_LP5521 is not set
# CONFIG_LEDS_LP5523 is not set
# CONFIG_LEDS_LP5562 is not set
# CONFIG_LEDS_LP8501 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_PCA963X is not set
# CONFIG_LEDS_PCA9685 is not set
# CONFIG_LEDS_BD2802 is not set
# CONFIG_LEDS_INTEL_SS4200 is not set
# CONFIG_LEDS_TCA6507 is not set
# CONFIG_LEDS_LM355x is not set
# CONFIG_LEDS_OT200 is not set
# CONFIG_LEDS_BLINKM is not set

#
# LED Triggers
#
# CONFIG_LEDS_TRIGGERS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_SYSTOHC=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_DS3232 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
# CONFIG_RTC_DRV_RS5C372 is not set
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_ISL12022 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF2127 is not set
# CONFIG_RTC_DRV_PCF8523 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_BQ32K is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set
# CONFIG_RTC_DRV_EM3027 is not set
# CONFIG_RTC_DRV_RV3029C2 is not set

#
# SPI RTC drivers
#

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set
# CONFIG_RTC_DRV_V3020 is not set
# CONFIG_RTC_DRV_DS2404 is not set

#
# on-CPU RTC drivers
#
# CONFIG_RTC_DRV_MOXART is not set

#
# HID Sensor RTC drivers
#
# CONFIG_RTC_DRV_HID_SENSOR_TIME is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_VFIO is not set
# CONFIG_VIRT_DRIVERS is not set

#
# Virtio drivers
#
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
# CONFIG_HYPERV is not set

#
# Xen driver support
#
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_ACPI_PROCESSOR=y
# CONFIG_XEN_MCE_LOG is not set
CONFIG_XEN_HAVE_PVMMU=y
# CONFIG_STAGING is not set
# CONFIG_X86_PLATFORM_DEVICES is not set
# CONFIG_CHROME_PLATFORMS is not set

#
# Hardware Spinlock drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# CONFIG_MAILBOX is not set
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_STATS=y
CONFIG_DMAR_TABLE=y
# CONFIG_INTEL_IOMMU is not set
CONFIG_IRQ_REMAP=y

#
# Remoteproc drivers
#
# CONFIG_STE_MODEM_RPROC is not set

#
# Rpmsg drivers
#
# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_NTB is not set
# CONFIG_VME_BUS is not set
# CONFIG_PWM is not set
# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set
# CONFIG_FMC is not set

#
# PHY Subsystem
#
CONFIG_GENERIC_PHY=y
# CONFIG_PHY_EXYNOS_MIPI_VIDEO is not set
# CONFIG_POWERCAP is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=y
# CONFIG_ISCSI_IBFT_FIND is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
# CONFIG_EXT2_FS is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
# CONFIG_EXT4_FS_POSIX_ACL is not set
# CONFIG_EXT4_FS_SECURITY is not set
CONFIG_EXT4_DEBUG=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=y
CONFIG_JBD2_DEBUG=y
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
CONFIG_GFS2_FS=y
CONFIG_BTRFS_FS=y
CONFIG_BTRFS_FS_POSIX_ACL=y
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
# CONFIG_BTRFS_ASSERT is not set
# CONFIG_NILFS2_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=y
CONFIG_FUSE_FS=y
# CONFIG_CUSE is not set
CONFIG_GENERIC_ACL=y

#
# Caches
#
CONFIG_FSCACHE=y
CONFIG_FSCACHE_STATS=y
CONFIG_FSCACHE_HISTOGRAM=y
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
# CONFIG_CACHEFILES is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=y
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
# CONFIG_CONFIGFS_FS is not set
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NETWORK_FILESYSTEMS=y
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
CONFIG_CEPH_FS=y
# CONFIG_CEPH_FSCACHE is not set
CONFIG_CIFS=y
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
# CONFIG_CIFS_UPCALL is not set
# CONFIG_CIFS_XATTR is not set
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_SMB2 is not set
# CONFIG_CIFS_FSCACHE is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_MAC_ROMAN is not set
# CONFIG_NLS_MAC_CELTIC is not set
# CONFIG_NLS_MAC_CENTEURO is not set
# CONFIG_NLS_MAC_CROATIAN is not set
# CONFIG_NLS_MAC_CYRILLIC is not set
# CONFIG_NLS_MAC_GAELIC is not set
# CONFIG_NLS_MAC_GREEK is not set
# CONFIG_NLS_MAC_ICELAND is not set
# CONFIG_NLS_MAC_INUIT is not set
# CONFIG_NLS_MAC_ROMANIAN is not set
# CONFIG_NLS_MAC_TURKISH is not set
CONFIG_NLS_UTF8=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=7
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_DYNAMIC_DEBUG is not set

#
# Compile-time checks and compiler options
#
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_DEBUG_KERNEL=y

#
# Memory Debugging
#
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=400
# CONFIG_DEBUG_KMEMLEAK_TEST is not set
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_DEBUG_SHIRQ=y

#
# Debug Lockups and Hangs
#
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_HARDLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
# CONFIG_SCHED_DEBUG is not set
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_PI_LIST=y
# CONFIG_RT_MUTEX_TESTER is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_LOCKDEP=y
# CONFIG_LOCK_STAT is not set
CONFIG_DEBUG_LOCKDEP=y
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_TRACE_IRQFLAGS=y
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_WRITECOUNT=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_SG=y
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
# CONFIG_PROVE_RCU is not set
CONFIG_SPARSE_RCU_POINTER=y
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_INFO=y
# CONFIG_RCU_TRACE is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS=y
# CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_FTRACE_SYSCALLS is not set
# CONFIG_TRACER_SNAPSHOT is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_UPROBE_EVENT is not set
# CONFIG_PROBE_EVENTS is not set
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
# CONFIG_FUNCTION_PROFILER is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_RING_BUFFER_STARTUP_TEST is not set

#
# Runtime Testing
#
# CONFIG_LKDTM is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_RBTREE_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
# CONFIG_PERCPU_TEST is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_TEST_STRING_HELPERS is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
CONFIG_DMA_API_DEBUG=y
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_SET_MODULE_RONX is not set
# CONFIG_DEBUG_NX_TEST is not set
CONFIG_DOUBLEFAULT=y
# CONFIG_DEBUG_TLBFLUSH is not set
CONFIG_IOMMU_DEBUG=y
# CONFIG_IOMMU_STRESS is not set
# CONFIG_IOMMU_LEAK is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set
# CONFIG_DEBUG_NMI_SELFTEST is not set
# CONFIG_X86_DEBUG_STATIC_CPU_HAS is not set

#
# Security options
#
CONFIG_KEYS=y
# CONFIG_PERSISTENT_KEYRINGS is not set
# CONFIG_BIG_KEYS is not set
# CONFIG_ENCRYPTED_KEYS is not set
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_XOR_BLOCKS=y
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_USER is not set
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=y
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_PCRYPT is not set
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=y
CONFIG_CRYPTO_AUTHENC=y
# CONFIG_CRYPTO_TEST is not set
CONFIG_CRYPTO_ABLK_HELPER=y
CONFIG_CRYPTO_GLUE_HELPER_X86=y

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=y
# CONFIG_CRYPTO_PCBC is not set
CONFIG_CRYPTO_XTS=y

#
# Hash modes
#
CONFIG_CRYPTO_CMAC=y
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_VMAC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=y
# CONFIG_CRYPTO_CRC32 is not set
# CONFIG_CRYPTO_CRC32_PCLMUL is not set
CONFIG_CRYPTO_CRCT10DIF=y
# CONFIG_CRYPTO_CRCT10DIF_PCLMUL is not set
# CONFIG_CRYPTO_GHASH is not set
CONFIG_CRYPTO_MD4=y
CONFIG_CRYPTO_MD5=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=y
CONFIG_CRYPTO_SHA256_SSSE3=y
CONFIG_CRYPTO_SHA512_SSSE3=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=y
CONFIG_CRYPTO_AES_NI_INTEL=y
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_ARC4=y
CONFIG_CRYPTO_BLOWFISH=y
CONFIG_CRYPTO_BLOWFISH_COMMON=y
CONFIG_CRYPTO_BLOWFISH_X86_64=y
# CONFIG_CRYPTO_CAMELLIA is not set
CONFIG_CRYPTO_CAMELLIA_X86_64=y
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=y
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=y
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST5_AVX_X86_64 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_CAST6_AVX_X86_64 is not set
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_SERPENT=y
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=y
CONFIG_CRYPTO_SERPENT_AVX_X86_64=y
CONFIG_CRYPTO_SERPENT_AVX2_X86_64=y
# CONFIG_CRYPTO_TEA is not set
CONFIG_CRYPTO_TWOFISH=y
CONFIG_CRYPTO_TWOFISH_COMMON=y
CONFIG_CRYPTO_TWOFISH_X86_64=y
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=y
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=y

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_ZLIB=y
CONFIG_CRYPTO_LZO=y
# CONFIG_CRYPTO_LZ4 is not set
# CONFIG_CRYPTO_LZ4HC is not set

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=y
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
# CONFIG_CRYPTO_HW is not set
# CONFIG_ASYMMETRIC_KEY_TYPE is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=y
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
# CONFIG_CRC_CCITT is not set
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_CRC32_SELFTEST=y
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
# CONFIG_CRC8 is not set
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_DECOMPRESS_LZ4=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=y
CONFIG_TEXTSEARCH_BM=y
CONFIG_TEXTSEARCH_FSM=y
CONFIG_ASSOCIATIVE_ARRAY=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
CONFIG_AVERAGE=y
# CONFIG_CORDIC is not set
# CONFIG_DDR is not set
CONFIG_FONT_SUPPORT=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y

[-- Attachment #8: kernel-dom0-git-log1.txt --]
[-- Type: text/plain, Size: 6218 bytes --]

commit 127d061dc224962acca9ea55cff74aef1668ae30
Author: Paul Durrant <Paul.Durrant@citrix.com>
Date:   Fri Dec 6 16:36:07 2013 +0000

    xen-netback: improve guest-receive-side flow control
    
    The way that flow control works without this patch is that, in start_xmit()
    the code uses xenvif_count_skb_slots() to predict how many slots
    xenvif_gop_skb() will consume and then adds this to a 'req_cons_peek'
    counter which it then uses to determine if the shared ring has that amount
    of space available by checking whether 'req_prod' has passed that value.
    If the ring doesn't have space the tx queue is stopped.
    xenvif_gop_skb() will then consume slots and update 'req_cons' and issue
    responses, updating 'rsp_prod' as it goes. The frontend will consume those
    responses and post new requests, by updating req_prod. So, req_prod chases
    req_cons which chases rsp_prod, and can never exceed that value. Thus if
    xenvif_count_skb_slots() ever returns a number of slots greater than
    xenvif_gop_skb() uses, req_cons_peek will get to a value that req_prod cannot
    possibly achieve (since it's limited by the 'real' req_cons) and, if this
    happens enough times, req_cons_peek gets more than a ring size ahead of
    req_cons and the tx queue then remains stopped forever waiting for an
    unachievable amount of space to become available in the ring.
    
    Having two routines trying to calculate the same value is always going to be
    fragile, so this patch does away with that. All we essentially need to do is
    make sure that we have 'enough stuff' on our internal queue without letting
    it build up uncontrollably. So start_xmit() makes a cheap optimistic check
    of how much space is needed for an skb and only turns the queue off if that
    is unachievable. net_rx_action() is the place where we could do with an
    accurate predicition but, since that has proven tricky to calculate, a cheap
    worse-case (but not too bad) estimate is all we really need since the only
    thing we *must* prevent is xenvif_gop_skb() consuming more slots than are
    available.
    
    Without this patch I can trivially stall netback permanently by just doing
    a large guest to guest file copy between two Windows Server 2008R2 VMs on a
    single host.
    
    Patch tested with frontends in:
    - Windows Server 2008R2
    - CentOS 6.0
    - Debian Squeeze
    - Debian Wheezy
    - SLES11
    
    Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
    Cc: Wei Liu <wei.liu2@citrix.com>
    Cc: Ian Campbell <ian.campbell@citrix.com>
    Cc: David Vrabel <david.vrabel@citrix.com>
    Cc: Annie Li <annie.li@oracle.com>
    Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
    Acked-by: Wei Liu <wei.liu2@citrix.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

commit b5744778373de94dcc7089be0bbf7d78210da30e
Merge: ca3f97e 7693dec
Author: Sander Eikelenboom <linux@eikelenboom.it>
Date:   Mon Mar 10 18:43:37 2014 +0100

    Merge tag 'stable/for-linus-3.14-rc5-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into HEAD
    
    Bug:
     - Fix compile dependency on Xen ARM to have MMU.

commit ca3f97e643c9017510f2823a47795affaf32f0e0
Merge: d685a88 d8320b2
Author: Sander Eikelenboom <linux@eikelenboom.it>
Date:   Mon Mar 10 18:42:51 2014 +0100

    Merge tag 'stable/for-linus-3.14-rc2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into HEAD
    
    Bug-fix:
     - Fix ARM and Xen FIFO not working.
     - Remove more Xen ia64 vestigates.
     - Fix UAPI missing Xen files.

commit d685a88845a6f82cb8bbb714ddf7647c1f2ee158
Merge: 0ac97c3 afca501
Author: Sander Eikelenboom <linux@eikelenboom.it>
Date:   Mon Mar 10 18:42:08 2014 +0100

    Merge tag 'stable/for-linus-3.14-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into HEAD
    
    Bug-fixes:
     - Revert "xen/grant-table: Avoid m2p_override during mapping" as it broke Xen ARM build.
     - Fix CR4 not being set on AP processors in Xen PVH mode.

commit 0ac97c365be141cc6fa70b685bbce6890ab4f99c
Merge: 3c4aac9 f93576e
Author: Sander Eikelenboom <linux@eikelenboom.it>
Date:   Mon Mar 10 18:38:44 2014 +0100

    Merge tag 'stable/for-linus-3.14-rc0-late-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into HEAD
    
    Bug-fixes:
     - Xen ARM couldn't use the new FIFO events
     - Xen ARM couldn't use the SWIOTLB if compiled as 32-bit with 64-bit PCIe devices.
     - Grant table were doing needless M2P operations.
     - Ratchet down the self-balloon code so it won't OOM.
     - Fix misplaced kfree in Xen PVH error code paths.

commit 3c4aac97ca0b8f8c67f3c62719d880ef3147fe78
Merge: 404df65 c9f6e99
Author: Sander Eikelenboom <linux@eikelenboom.it>
Date:   Mon Mar 10 18:31:38 2014 +0100

    Merge tag 'stable/for-linus-3.14-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into HEAD
    
    Features:
     - FIFO event channels. Key advantages: support for over 100,000 events (2^17),
       16 different event priorities, improved fairness in event latency through
       the use of FIFOs.
     - Xen PVH support. "It’s a fully PV kernel mode, running with paravirtualized
       disk and network, paravirtualized interrupts and timers, no emulated devices
       of any kind (and thus no qemu), no BIOS or legacy boot — but instead of
       requiring PV MMU, it uses the HVM hardware extensions to virtualize the
       pagetables, as well as system calls and other privileged operations."
       (from "The Paravirtualization Spectrum, Part 2: From poles to a spectrum")
    Bug-fixes:
     - Fixes in balloon driver (refactor and make it work under ARM)
     - Allow xenfb to be used in HVM guests.
     - Allow xen_platform_pci=0 to work properly.
     - Refactors in event channels.
    
    Conflicts:
    	arch/arm/include/asm/xen/page.h
    	include/xen/platform_pci.h

commit 404df65d0480f6da2b768f6c9b5259436b1de10f
Author: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date:   Thu Mar 6 22:07:02 2014 -0800

    Linux 3.13.6

[-- Attachment #9: netfront-debug.diff --]
[-- Type: application/octet-stream, Size: 1478 bytes --]

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index e30d800..5d23266 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -741,18 +741,26 @@ static int xennet_get_responses(struct netfront_info *np,
 	int slots = 1;
 	int err = 0;
 	unsigned long ret;
-
+        int cons_changed = 0;
+	int cons_before = 0;
+	int xennet_get_extras_err = 0;
+	
 	if (rx->flags & XEN_NETRXF_extra_info) {
 		err = xennet_get_extras(np, extras, rp);
+                xennet_get_extras_err = err;
+		cons_before = cons;
 		cons = np->rx.rsp_cons;
+		cons_changed++;
 	}
 
 	for (;;) {
 		if (unlikely(rx->status < 0 ||
 			     rx->offset + rx->status > PAGE_SIZE)) {
-			if (net_ratelimit())
+			if (net_ratelimit()){
 				dev_warn(dev, "rx->offset: %x, size: %u\n",
 					 rx->offset, rx->status);
+				dev_warn(dev, "cons:%d slots:%d rp:%d max:%d err:%d rx->id:%d rx->offset:%x size:%u ref:%ld pagesize:%d skb_ipsummed:%d is_gso:%d gso_size:%d gso_type:%d gso_segs:%d RING_HAS_UNCONSUMED_RESPONSES:%d cons_changed:%d cons_before:%d xennet_get_extras_err:%d\n", cons, slots, rp, max, err, rx->id, rx->offset, rx->status, ref, PAGE_SIZE, skb->ip_summed, skb_is_gso(skb) ? 1 : 0, skb_shinfo(skb)->gso_size, skb_shinfo(skb)->gso_type, skb_shinfo(skb)->gso_segs, RING_HAS_UNCONSUMED_RESPONSES(&np->rx), cons_changed, cons_before, xennet_get_extras_err);
+                        }
 			xennet_move_rx_slot(np, skb, ref);
 			err = -EINVAL;
 			goto next;

[-- Attachment #10: ifconfig-dom0.txt --]
[-- Type: text/plain, Size: 2435 bytes --]

eth0      Link encap:Ethernet  HWaddr 40:61:86:f4:67:d9  
          inet addr:xxx.xxx.xxx.xxx  Bcast:xxx.xxx.xxx.xxx  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:17987 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14735 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:15696068 (14.9 MiB)  TX bytes:1544835 (1.4 MiB)

eth1      Link encap:Ethernet  HWaddr 40:61:86:f4:67:d8  
          inet addr:172.16.1.1  Bcast:172.16.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11360 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11169 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1012835 (989.0 KiB)  TX bytes:15151725 (14.4 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:90 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:7920 (7.7 KiB)  TX bytes:7920 (7.7 KiB)

vif1.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4358682 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6538948 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:3497735292 (3.2 GiB)  TX bytes:3702631634 (3.4 GiB)

vpn_bridge Link encap:Ethernet  HWaddr b6:36:82:32:c1:4b  
          inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

xen_bridge Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4358682 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6539894 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3497735292 (3.2 GiB)  TX bytes:3702671366 (3.4 GiB)


[-- Attachment #11: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 2:42:43 AM, you wrote:


> Tuesday, March 11, 2014, 4:36:16 PM, you wrote:

>> On Tue, Mar 11, 2014 at 02:00:41PM +0100, Sander Eikelenboom wrote:
>> [...]
>>> >> the issue when using 3.13.6 as a base and ..
>>> >>   - pull all 3.14 patches from the  git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git tree
>>> >>   - apply paul's commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control"
>>> >>   - applying annie's v2 patch
>>> >>   - applying your patch
>>> >> as dom0 and using a 3.14-rc5 as domU kernel.
>>> >> 
>>> >> Unfortunately i'm still getting the Bad grant references ..
>>> >> 
>>> 
>>> > :-( That's bad news.
>>> 
>>> > I guess you always have the same DomU kernel when testing? That means we
>>> > can narrow down the bug to netback only.
>>> 
>>> Yes my previous tests (from my previous mail):
>>> 
>>> - First testing a baseline that worked o.k. for several days (3.13.6 for both dom0 and domU)
>>> - Testing domU 3.14-rc5 and dom0 3.13.6, this worked ok.
>>> - Testing dom0 3.14-rc5 and domU 3.13.6, this failed.
>>> - After that took 3.13.6 as base and first applied all the general xen related patches for the dom0 kernel, that works ok.
>>> - After that started to apply the netback changes for 3.14 and that failed after the commit "ca2f09f2b2c6c25047cfc545d057c4edfcfe561c xen-netback: improve guest-receive-side flow control".
>>> 
>>> Also seem to indicate just that, although it could also be something in this netback commit that triggers a latent bug in netfront, can't rule that one out completly.
>>> 
>>> But the trigger is in that commit &&
>>> annie's and your patch seem to have no effect at all( on this issue) &&
>>> later commits in 3.14 do seems to mask it / make it less likely to trigger, but do not fix it.
>>> 

>> Unfortunately I've stared at the same piece of code for some time but
>> don't have immediate clue. Later commits don't look suspecious either.

>> I also looked at netfront code, but there's no slot couting change
>> between 3.13 and 3.14.

>> Do you have some straight setup instructions so that I can try to
>> reproduce.

> Ok you asked for it .. so here we go .. ;-) :

> - Xen-unstable
> - DomU:
>      - PV guest
>      - Debian Wheezy
>      - Kernel: 3.14-rc5 (.config see dom0 .config)
>      - Running netserver
> - Dom0:
>      - Debian Wheezy
>      - Kernel: 3.13.6 + additional patches (see attached git log and .config)
>      - 2 physical NIC's
>      - Autoballooning is prevented by using dom0_mem=1536M,max:1536M for xen and mem=1536M for dom0 kernel stanzas in grub
>      - vcpu 0 is pinned on pcpu 0 and exclusively for dom0

> - Networking:
>      - Routed bridge
>      - eth0 = internet
>      - eth1 = lan 172.16.x.x
>      - xen_bridge = bridge for VM's 192.168.1.x
>      - iptables NAT and routing  
>      - attached dom0 and domU ifconfig output
>      - attached ethtool -k output for the bridge, vif and guest eth0

> Triggering workload:
>      - Well that's were the problem is :-)
>      - The Guest has a normal disk and swap (phy/lvm) and
>        shared storage with glusterfs (glusterfs server is on dom0)
>      - The Guest exposes this storage via webdavs 

>      What triggers it is:
>      - The Guest runs it's rsync of the shared storage to a remote client on the internet
>        So this causes traffic from Dom0 to domU (reading storage) ..
>        back from domU to dom0 and via iptables NAT on to eth0 (actual rsync)
>        or vice versa when syncing the other way around
>      - At the same time do a backup from a windows machine from the lan to the webdavs server
>        So this causes traffic from eth1 to domU (webdav) .. 
>        back from domU to dom0 (writing to the shared storage) 

> So in essence it is doing quite some netback && netfront stress testing :-)

> It seems to only trigger when doing both the rsync and the webdav simultaneous.

> I tried my best to emulate any of this with netperf (and multiple instances),
> i tried with various (odd) packet sizes and the packet / byte rates transmitted
> are higher then with the workload above ... but it doesn't seem to trigger with netpref

> So i don't think it will be easy to replicate ...

> Perhaps running through the available logging again .. and try to answer some questions ... this is just with one guest running kernels as before only added debugging to netfront and xen (diffs attached):

> Mar 12 02:00:44 backup kernel: [  496.840646] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840665] net eth0: cons:1346005 slots:1 rp:1346013 max:18 err:0 rx->id:212 rx->offset:0 size:4294967295 ref:572 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840680] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840687] net eth0: cons:1346005 slots:2 rp:1346013 max:18 err:-22 rx->id:214 rx->offset:0 size:4294967295 ref:657 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840701] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840712] net eth0: cons:1346005 slots:3 rp:1346013 max:18 err:-22 rx->id:215 rx->offset:0 size:4294967295 ref:667 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840733] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840740] net eth0: cons:1346005 slots:4 rp:1346013 max:18 err:-22 rx->id:216 rx->offset:0 size:4294967295 ref:716 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840757] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840764] net eth0: cons:1346005 slots:5 rp:1346013 max:18 err:-22 rx->id:217 rx->offset:0 size:4294967295 ref:755 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840778] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840784] net eth0: cons:1346005 slots:6 rp:1346013 max:18 err:-22 rx->id:218 rx->offset:0 size:4294967295 ref:592 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840801] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840807] net eth0: cons:1346005 slots:7 rp:1346013 max:18 err:-22 rx->id:219 rx->offset:0 size:4294967295 ref:633 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.841043] net eth0: rx->offset: 0, size: 4294967295

> Mar 12 02:00:44 backup kernel: [  496.841052] net eth0: cons:1346025 slots:1 rp:1346038 max:18 err:0 rx->id:232 rx->offset:0 size:4294967295 ref:-131941395332491 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:13 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.841067] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.841074] net eth0: cons:1346025 slots:2 rp:1346038 max:18 err:-22 rx->id:234 rx->offset:0 size:4294967295 ref:-131941395332579 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.841092] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.841101] net eth0: cons:1346025 slots:3 rp:1346038 max:18 err:-22 rx->id:235 rx->offset:0 size:4294967295 ref:-131941395332408 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0


> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478146 source_off:0 source_len:4096 op->source.offset:0 op->len:1168
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:0 op->len:2476
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478282 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:1634 op->len:1620
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497609 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1

Hrrmm sorry seems i was incomplete in copying and pasting the accompanying xl dmesg part:

(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478146 source_off:0 source_len:4096 op->source.offset:0 op->len:1168
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:0 op->len:2476
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478282 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:1634 op->len:1620
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497609 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489800 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489799 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 194904079 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489798 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 224788480 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489796 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489795 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489794 source_off:0 source_len:4096 op->source.offset:0 op->len:2892
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478282 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489794 source_off:0 source_len:4096 op->source.offset:1634 op->len:1204
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5489793 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5490944 source_off:0 source_len:4096 op->source.offset:0 op->len:3268



> - Sometimes (but not always) netfront also spits out:
>   dev_warn(dev, "Invalid extra type: %d\n", extra->type);
>   where the extra type seems a random value (seen 196, 31 ..)
> - Sometimes (but not always) netfront also spits out:
>   dev_warn(dev, "Need more slots\n");
> - Sometimes (but not always) netfront also spits out:
>   dev_warn(dev, "Missing extra info\n");


> First question that comes to my mind:
> - Are the warnings netfront spits out the cause of xen reporting the bad grant reference ?
>   Or
>   Are the Bad grant references Xen is reporting .. causing netfront to spit out the warnings ?
> - Why is that "if (rx->flags & XEN_NETRXF_extra_info) {}" part in xen-netfront.c doing there and changing cons midway ?
>   The commit message from f942dc2552b8bfdee607be867b12a8971bb9cd85 that introduced the if says:

>     One major change from xen.git is that the guest transmit path (i.e. what
>     looks like receive to netback) has been significantly reworked to remove
>     the dependency on the out of tree PageForeign page flag (a core kernel
>     patch which enables a per page destructor callback on the final
>     put_page). This page flag was used in order to implement a grant map
>     based transmit path (where guest pages are mapped directly into SKB
>     frags). Instead this version of netback uses grant copy operations into
>     regular memory belonging to the backend domain. Reinstating the grant
>     map functionality is something which I would like to revisit in the
>     future.

>   It *is* using grant copy now .. so should this part have been removed ?
>   And
>   Could Paul's commit that seems to be the first to trigger these events affect this ?

> --
> Sander



>> Wei.

>>> > Paul, do you have any idea what might go wrong?
>>> 
>>> > Wei.
>>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 12, 2014 at 02:42:43AM +0100, Sander Eikelenboom wrote:
[...]
> Ok you asked for it .. so here we go .. ;-) :
> 
> - Xen-unstable
> - DomU:
>      - PV guest
>      - Debian Wheezy
>      - Kernel: 3.14-rc5 (.config see dom0 .config)
>      - Running netserver
> - Dom0:
>      - Debian Wheezy
>      - Kernel: 3.13.6 + additional patches (see attached git log and .config)
>      - 2 physical NIC's
>      - Autoballooning is prevented by using dom0_mem=1536M,max:1536M for xen and mem=1536M for dom0 kernel stanzas in grub
>      - vcpu 0 is pinned on pcpu 0 and exclusively for dom0
> 
> - Networking:
>      - Routed bridge
>      - eth0 = internet
>      - eth1 = lan 172.16.x.x
>      - xen_bridge = bridge for VM's 192.168.1.x
>      - iptables NAT and routing  
>      - attached dom0 and domU ifconfig output
>      - attached ethtool -k output for the bridge, vif and guest eth0
> 
> Triggering workload:
>      - Well that's were the problem is :-)
>      - The Guest has a normal disk and swap (phy/lvm) and
>        shared storage with glusterfs (glusterfs server is on dom0)
>      - The Guest exposes this storage via webdavs 
> 
>      What triggers it is:
>      - The Guest runs it's rsync of the shared storage to a remote client on the internet
>        So this causes traffic from Dom0 to domU (reading storage) ..
>        back from domU to dom0 and via iptables NAT on to eth0 (actual rsync)
>        or vice versa when syncing the other way around
>      - At the same time do a backup from a windows machine from the lan to the webdavs server
>        So this causes traffic from eth1 to domU (webdav) .. 
>        back from domU to dom0 (writing to the shared storage) 
> 
> So in essence it is doing quite some netback && netfront stress testing :-)
> 
> It seems to only trigger when doing both the rsync and the webdav simultaneous.
> 
> I tried my best to emulate any of this with netperf (and multiple instances),
> i tried with various (odd) packet sizes and the packet / byte rates transmitted
> are higher then with the workload above ... but it doesn't seem to trigger with netpref
> 
> So i don't think it will be easy to replicate ...
> 

Sorry, this is probably not something I can setup. *sigh*

> Perhaps running through the available logging again .. and try to answer some questions ... this is just with one guest running kernels as before only added debugging to netfront and xen (diffs attached):
> 
> Mar 12 02:00:44 backup kernel: [  496.840646] net eth0: rx->offset: 0, size: 4294967295

This "size" here is actually "status", status=-1, it's an error...

> Mar 12 02:00:44 backup kernel: [  496.840665] net eth0: cons:1346005 slots:1 rp:1346013 max:18 err:0 rx->id:212 rx->offset:0 size:4294967295 ref:572 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840680] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840687] net eth0: cons:1346005 slots:2 rp:1346013 max:18 err:-22 rx->id:214 rx->offset:0 size:4294967295 ref:657 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840701] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840712] net eth0: cons:1346005 slots:3 rp:1346013 max:18 err:-22 rx->id:215 rx->offset:0 size:4294967295 ref:667 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840733] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840740] net eth0: cons:1346005 slots:4 rp:1346013 max:18 err:-22 rx->id:216 rx->offset:0 size:4294967295 ref:716 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840757] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840764] net eth0: cons:1346005 slots:5 rp:1346013 max:18 err:-22 rx->id:217 rx->offset:0 size:4294967295 ref:755 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840778] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840784] net eth0: cons:1346005 slots:6 rp:1346013 max:18 err:-22 rx->id:218 rx->offset:0 size:4294967295 ref:592 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.840801] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.840807] net eth0: cons:1346005 slots:7 rp:1346013 max:18 err:-22 rx->id:219 rx->offset:0 size:4294967295 ref:633 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.841043] net eth0: rx->offset: 0, size: 4294967295
> 
> Mar 12 02:00:44 backup kernel: [  496.841052] net eth0: cons:1346025 slots:1 rp:1346038 max:18 err:0 rx->id:232 rx->offset:0 size:4294967295 ref:-131941395332491 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:13 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.841067] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.841074] net eth0: cons:1346025 slots:2 rp:1346038 max:18 err:-22 rx->id:234 rx->offset:0 size:4294967295 ref:-131941395332579 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
> Mar 12 02:00:44 backup kernel: [  496.841092] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 02:00:44 backup kernel: [  496.841101] net eth0: cons:1346025 slots:3 rp:1346038 max:18 err:-22 rx->id:235 rx->offset:0 size:4294967295 ref:-131941395332408 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
> 
> 
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478146 source_off:0 source_len:4096 op->source.offset:0 op->len:1168

rc = -3 when acquiring grant. This means grant reference is unrecognised
or inappropriate. The grant reference is 4325377 which is obviously too
large for a sensible grant ref. It's just garbage.

> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:0 op->len:2476
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478282 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:1634 op->len:1620
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497609 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> 
> - Sometimes (but not always) netfront also spits out:
>   dev_warn(dev, "Invalid extra type: %d\n", extra->type);
>   where the extra type seems a random value (seen 196, 31 ..)
> - Sometimes (but not always) netfront also spits out:
>   dev_warn(dev, "Need more slots\n");
> - Sometimes (but not always) netfront also spits out:
>   dev_warn(dev, "Missing extra info\n");
> 

That's because garbage is pushed to frontend. It's trying to decode some
random values.

> 
> First question that comes to my mind:
> - Are the warnings netfront spits out the cause of xen reporting the bad grant reference ?
>   Or
>   Are the Bad grant references Xen is reporting .. causing netfront to spit out the warnings ?

No. They are not directly connected.

0. backend breaks down a skb into slots.
1. backend gets a request from the rx ring (a slot with gref provided by
   frontend), until all slots in skb are handled.
2. backend grant-copies data to that slot (with gref).
3. backend pushes a response to frontend, whose content depends on the
   result of previous step.
4. frontend handles that response.

So the grant copy error in hypervisor you're seeing is in 2. The
complaint from frontend is in 4, when backend pushes a response with
error status in it. The bug is most likely in 0, when the calculation
goes wrong - either in the breakdown process, or in the macro that
returns usable slots (this can lead to backend thinks there's enough
slots while there's not).

> - Why is that "if (rx->flags & XEN_NETRXF_extra_info) {}" part in xen-netfront.c doing there and changing cons midway ?
>   The commit message from f942dc2552b8bfdee607be867b12a8971bb9cd85 that introduced the if says:
> 
>     One major change from xen.git is that the guest transmit path (i.e. what
>     looks like receive to netback) has been significantly reworked to remove
>     the dependency on the out of tree PageForeign page flag (a core kernel
>     patch which enables a per page destructor callback on the final
>     put_page). This page flag was used in order to implement a grant map
>     based transmit path (where guest pages are mapped directly into SKB
>     frags). Instead this version of netback uses grant copy operations into
>     regular memory belonging to the backend domain. Reinstating the grant
>     map functionality is something which I would like to revisit in the
>     future.
> 
>   It *is* using grant copy now .. so should this part have been removed ?
>   And
>   Could Paul's commit that seems to be the first to trigger these events affect this ?
> 

This depicts guest *transmit* path, but the issue you're seeing is in
guest *receive* path. So that's totally different thing.

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 12:35:22 PM, you wrote:

> On Wed, Mar 12, 2014 at 02:42:43AM +0100, Sander Eikelenboom wrote:
> [...]
>> Ok you asked for it .. so here we go .. ;-) :
>> 
>> - Xen-unstable
>> - DomU:
>>      - PV guest
>>      - Debian Wheezy
>>      - Kernel: 3.14-rc5 (.config see dom0 .config)
>>      - Running netserver
>> - Dom0:
>>      - Debian Wheezy
>>      - Kernel: 3.13.6 + additional patches (see attached git log and .config)
>>      - 2 physical NIC's
>>      - Autoballooning is prevented by using dom0_mem=1536M,max:1536M for xen and mem=1536M for dom0 kernel stanzas in grub
>>      - vcpu 0 is pinned on pcpu 0 and exclusively for dom0
>> 
>> - Networking:
>>      - Routed bridge
>>      - eth0 = internet
>>      - eth1 = lan 172.16.x.x
>>      - xen_bridge = bridge for VM's 192.168.1.x
>>      - iptables NAT and routing  
>>      - attached dom0 and domU ifconfig output
>>      - attached ethtool -k output for the bridge, vif and guest eth0
>> 
>> Triggering workload:
>>      - Well that's were the problem is :-)
>>      - The Guest has a normal disk and swap (phy/lvm) and
>>        shared storage with glusterfs (glusterfs server is on dom0)
>>      - The Guest exposes this storage via webdavs 
>> 
>>      What triggers it is:
>>      - The Guest runs it's rsync of the shared storage to a remote client on the internet
>>        So this causes traffic from Dom0 to domU (reading storage) ..
>>        back from domU to dom0 and via iptables NAT on to eth0 (actual rsync)
>>        or vice versa when syncing the other way around
>>      - At the same time do a backup from a windows machine from the lan to the webdavs server
>>        So this causes traffic from eth1 to domU (webdav) .. 
>>        back from domU to dom0 (writing to the shared storage) 
>> 
>> So in essence it is doing quite some netback && netfront stress testing :-)
>> 
>> It seems to only trigger when doing both the rsync and the webdav simultaneous.
>> 
>> I tried my best to emulate any of this with netperf (and multiple instances),
>> i tried with various (odd) packet sizes and the packet / byte rates transmitted
>> are higher then with the workload above ... but it doesn't seem to trigger with netpref
>> 
>> So i don't think it will be easy to replicate ...
>> 

> Sorry, this is probably not something I can setup. *sigh*

>> Perhaps running through the available logging again .. and try to answer some questions ... this is just with one guest running kernels as before only added debugging to netfront and xen (diffs attached):
>> 
>> Mar 12 02:00:44 backup kernel: [  496.840646] net eth0: rx->offset: 0, size: 4294967295

> This "size" here is actually "status", status=-1, it's an error...

>> Mar 12 02:00:44 backup kernel: [  496.840665] net eth0: cons:1346005 slots:1 rp:1346013 max:18 err:0 rx->id:212 rx->offset:0 size:4294967295 ref:572 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.840680] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.840687] net eth0: cons:1346005 slots:2 rp:1346013 max:18 err:-22 rx->id:214 rx->offset:0 size:4294967295 ref:657 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.840701] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.840712] net eth0: cons:1346005 slots:3 rp:1346013 max:18 err:-22 rx->id:215 rx->offset:0 size:4294967295 ref:667 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.840733] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.840740] net eth0: cons:1346005 slots:4 rp:1346013 max:18 err:-22 rx->id:216 rx->offset:0 size:4294967295 ref:716 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.840757] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.840764] net eth0: cons:1346005 slots:5 rp:1346013 max:18 err:-22 rx->id:217 rx->offset:0 size:4294967295 ref:755 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.840778] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.840784] net eth0: cons:1346005 slots:6 rp:1346013 max:18 err:-22 rx->id:218 rx->offset:0 size:4294967295 ref:592 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.840801] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.840807] net eth0: cons:1346005 slots:7 rp:1346013 max:18 err:-22 rx->id:219 rx->offset:0 size:4294967295 ref:633 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:1346004 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.841043] net eth0: rx->offset: 0, size: 4294967295
>> 
>> Mar 12 02:00:44 backup kernel: [  496.841052] net eth0: cons:1346025 slots:1 rp:1346038 max:18 err:0 rx->id:232 rx->offset:0 size:4294967295 ref:-131941395332491 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:13 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.841067] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.841074] net eth0: cons:1346025 slots:2 rp:1346038 max:18 err:-22 rx->id:234 rx->offset:0 size:4294967295 ref:-131941395332579 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
>> Mar 12 02:00:44 backup kernel: [  496.841092] net eth0: rx->offset: 0, size: 4294967295
>> Mar 12 02:00:44 backup kernel: [  496.841101] net eth0: cons:1346025 slots:3 rp:1346038 max:18 err:-22 rx->id:235 rx->offset:0 size:4294967295 ref:-131941395332408 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:29 cons_changed:1 cons_before:1346024 xennet_get_extras_err:0
>> 
>> 
>> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
>> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478146 source_off:0 source_len:4096 op->source.offset:0 op->len:1168

> rc = -3 when acquiring grant. This means grant reference is unrecognised
> or inappropriate. The grant reference is 4325377 which is obviously too
> large for a sensible grant ref. It's just garbage.

>> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
>> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:0 op->len:2476
>> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
>> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5478282 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
>> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
>> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497610 source_off:0 source_len:4096 op->source.offset:1634 op->len:1620
>> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
>> (XEN) [2014-03-12 01:00:44] grant_table.c:2100:d0v2 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:1 s_frame:5497609 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
>> (XEN) [2014-03-12 01:00:44] grant_table.c:1856:d0v2 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
>> 
>> - Sometimes (but not always) netfront also spits out:
>>   dev_warn(dev, "Invalid extra type: %d\n", extra->type);
>>   where the extra type seems a random value (seen 196, 31 ..)
>> - Sometimes (but not always) netfront also spits out:
>>   dev_warn(dev, "Need more slots\n");
>> - Sometimes (but not always) netfront also spits out:
>>   dev_warn(dev, "Missing extra info\n");
>> 

> That's because garbage is pushed to frontend. It's trying to decode some
> random values.

>> 
>> First question that comes to my mind:
>> - Are the warnings netfront spits out the cause of xen reporting the bad grant reference ?
>>   Or
>>   Are the Bad grant references Xen is reporting .. causing netfront to spit out the warnings ?

> No. They are not directly connected.

> 0. backend breaks down a skb into slots.
> 1. backend gets a request from the rx ring (a slot with gref provided by
>    frontend), until all slots in skb are handled.
> 2. backend grant-copies data to that slot (with gref).
> 3. backend pushes a response to frontend, whose content depends on the
>    result of previous step.
> 4. frontend handles that response.

> So the grant copy error in hypervisor you're seeing is in 2. The
> complaint from frontend is in 4, when backend pushes a response with
> error status in it. The bug is most likely in 0, when the calculation
> goes wrong - either in the breakdown process, or in the macro that
> returns usable slots (this can lead to backend thinks there's enough
> slots while there's not).

Ok so perhaps a debug patch which does more sanity checking there ?
Because i don't see any errors/warnings from netback in dom0.

>> - Why is that "if (rx->flags & XEN_NETRXF_extra_info) {}" part in xen-netfront.c doing there and changing cons midway ?
>>   The commit message from f942dc2552b8bfdee607be867b12a8971bb9cd85 that introduced the if says:
>> 
>>     One major change from xen.git is that the guest transmit path (i.e. what
>>     looks like receive to netback) has been significantly reworked to remove
>>     the dependency on the out of tree PageForeign page flag (a core kernel
>>     patch which enables a per page destructor callback on the final
>>     put_page). This page flag was used in order to implement a grant map
>>     based transmit path (where guest pages are mapped directly into SKB
>>     frags). Instead this version of netback uses grant copy operations into
>>     regular memory belonging to the backend domain. Reinstating the grant
>>     map functionality is something which I would like to revisit in the
>>     future.
>> 
>>   It *is* using grant copy now .. so should this part have been removed ?
>>   And
>>   Could Paul's commit that seems to be the first to trigger these events affect this ?
>> 

> This depicts guest *transmit* path, but the issue you're seeing is in
> guest *receive* path. So that's totally different thing.

Errr yes well if the (rx->flags & XEN_NETRXF_extra_info) {}) should only be doing something on the transmit path ..
why does it seem to result to true on my issue ?  (see cons_changed=1 in the debub output.. and you see cons being cons + 1 after that)


> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 12, 2014 at 12:45:16PM +0100, Sander Eikelenboom wrote:
[...]
> >> 
> >> - Sometimes (but not always) netfront also spits out:
> >>   dev_warn(dev, "Invalid extra type: %d\n", extra->type);
> >>   where the extra type seems a random value (seen 196, 31 ..)
> >> - Sometimes (but not always) netfront also spits out:
> >>   dev_warn(dev, "Need more slots\n");
> >> - Sometimes (but not always) netfront also spits out:
> >>   dev_warn(dev, "Missing extra info\n");
> >> 
> 
> > That's because garbage is pushed to frontend. It's trying to decode some
> > random values.
> 
> >> 
> >> First question that comes to my mind:
> >> - Are the warnings netfront spits out the cause of xen reporting the bad grant reference ?
> >>   Or
> >>   Are the Bad grant references Xen is reporting .. causing netfront to spit out the warnings ?
> 
> > No. They are not directly connected.
> 
> > 0. backend breaks down a skb into slots.
> > 1. backend gets a request from the rx ring (a slot with gref provided by
> >    frontend), until all slots in skb are handled.
> > 2. backend grant-copies data to that slot (with gref).
> > 3. backend pushes a response to frontend, whose content depends on the
> >    result of previous step.
> > 4. frontend handles that response.
> 
> > So the grant copy error in hypervisor you're seeing is in 2. The
> > complaint from frontend is in 4, when backend pushes a response with
> > error status in it. The bug is most likely in 0, when the calculation
> > goes wrong - either in the breakdown process, or in the macro that
> > returns usable slots (this can lead to backend thinks there's enough
> > slots while there's not).
> 
> Ok so perhaps a debug patch which does more sanity checking there ?
> Because i don't see any errors/warnings from netback in dom0.
> 

>From the look of the code, 0 looks correct to me. Netback won't complain
about "bad gref" because it has no idea what a "good gref" looks like.
Only Xen has the knowledge whether a gref is legit. All netback sees is
the hypercall fails and it tries to push corresponding response to
netfront. But you can probably safely assume a gref larger than several
thousands be bad.

> >> - Why is that "if (rx->flags & XEN_NETRXF_extra_info) {}" part in xen-netfront.c doing there and changing cons midway ?
> >>   The commit message from f942dc2552b8bfdee607be867b12a8971bb9cd85 that introduced the if says:
> >> 
> >>     One major change from xen.git is that the guest transmit path (i.e. what
> >>     looks like receive to netback) has been significantly reworked to remove
> >>     the dependency on the out of tree PageForeign page flag (a core kernel
> >>     patch which enables a per page destructor callback on the final
> >>     put_page). This page flag was used in order to implement a grant map
> >>     based transmit path (where guest pages are mapped directly into SKB
> >>     frags). Instead this version of netback uses grant copy operations into
> >>     regular memory belonging to the backend domain. Reinstating the grant
> >>     map functionality is something which I would like to revisit in the
> >>     future.
> >> 
> >>   It *is* using grant copy now .. so should this part have been removed ?
> >>   And
> >>   Could Paul's commit that seems to be the first to trigger these events affect this ?
> >> 
> 
> > This depicts guest *transmit* path, but the issue you're seeing is in
> > guest *receive* path. So that's totally different thing.
> 
> Errr yes well if the (rx->flags & XEN_NETRXF_extra_info) {}) should only be doing something on the transmit path ..
> why does it seem to result to true on my issue ?  (see cons_changed=1 in the debub output.. and you see cons being cons + 1 after that)

First thing is the response goes wild and probably contains random value
so it's possible the flag is "set".

For a legit response, if you see that flag it means there's an "extra
info" slot in the response. That code snippet is to extract that extra
info slot (could be several in a row, that's why there's a loop in
xennet_get_extras). The request / response format is documented in
include/xen/interface/io/netif.h -- it's only the tx one, but rx one is
more or less the same.

Wei.

> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 1:04:44 PM, you wrote:

> On Wed, Mar 12, 2014 at 12:45:16PM +0100, Sander Eikelenboom wrote:
> [...]
>> >> 
>> >> - Sometimes (but not always) netfront also spits out:
>> >>   dev_warn(dev, "Invalid extra type: %d\n", extra->type);
>> >>   where the extra type seems a random value (seen 196, 31 ..)
>> >> - Sometimes (but not always) netfront also spits out:
>> >>   dev_warn(dev, "Need more slots\n");
>> >> - Sometimes (but not always) netfront also spits out:
>> >>   dev_warn(dev, "Missing extra info\n");
>> >> 
>> 
>> > That's because garbage is pushed to frontend. It's trying to decode some
>> > random values.
>> 
>> >> 
>> >> First question that comes to my mind:
>> >> - Are the warnings netfront spits out the cause of xen reporting the bad grant reference ?
>> >>   Or
>> >>   Are the Bad grant references Xen is reporting .. causing netfront to spit out the warnings ?
>> 
>> > No. They are not directly connected.
>> 
>> > 0. backend breaks down a skb into slots.
>> > 1. backend gets a request from the rx ring (a slot with gref provided by
>> >    frontend), until all slots in skb are handled.
>> > 2. backend grant-copies data to that slot (with gref).
>> > 3. backend pushes a response to frontend, whose content depends on the
>> >    result of previous step.
>> > 4. frontend handles that response.
>> 
>> > So the grant copy error in hypervisor you're seeing is in 2. The
>> > complaint from frontend is in 4, when backend pushes a response with
>> > error status in it. The bug is most likely in 0, when the calculation
>> > goes wrong - either in the breakdown process, or in the macro that
>> > returns usable slots (this can lead to backend thinks there's enough
>> > slots while there's not).
>> 
>> Ok so perhaps a debug patch which does more sanity checking there ?
>> Because i don't see any errors/warnings from netback in dom0.
>> 

> From the look of the code, 0 looks correct to me. Netback won't complain
> about "bad gref" because it has no idea what a "good gref" looks like.
> Only Xen has the knowledge whether a gref is legit. All netback sees is
> the hypercall fails and it tries to push corresponding response to
> netfront. But you can probably safely assume a gref larger than several
> thousands be bad.

OK it's indeed netback setting the status to -1:

[  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
[ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
[ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
[ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
[ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
[ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
[ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
[ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
[ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
[ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
[ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
[ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
[ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
[ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958

Now to find out why ..

>> >> - Why is that "if (rx->flags & XEN_NETRXF_extra_info) {}" part in xen-netfront.c doing there and changing cons midway ?
>> >>   The commit message from f942dc2552b8bfdee607be867b12a8971bb9cd85 that introduced the if says:
>> >> 
>> >>     One major change from xen.git is that the guest transmit path (i.e. what
>> >>     looks like receive to netback) has been significantly reworked to remove
>> >>     the dependency on the out of tree PageForeign page flag (a core kernel
>> >>     patch which enables a per page destructor callback on the final
>> >>     put_page). This page flag was used in order to implement a grant map
>> >>     based transmit path (where guest pages are mapped directly into SKB
>> >>     frags). Instead this version of netback uses grant copy operations into
>> >>     regular memory belonging to the backend domain. Reinstating the grant
>> >>     map functionality is something which I would like to revisit in the
>> >>     future.
>> >> 
>> >>   It *is* using grant copy now .. so should this part have been removed ?
>> >>   And
>> >>   Could Paul's commit that seems to be the first to trigger these events affect this ?
>> >> 
>> 
>> > This depicts guest *transmit* path, but the issue you're seeing is in
>> > guest *receive* path. So that's totally different thing.
>> 
>> Errr yes well if the (rx->flags & XEN_NETRXF_extra_info) {}) should only be doing something on the transmit path ..
>> why does it seem to result to true on my issue ?  (see cons_changed=1 in the debub output.. and you see cons being cons + 1 after that)

> First thing is the response goes wild and probably contains random value
> so it's possible the flag is "set".

> For a legit response, if you see that flag it means there's an "extra
> info" slot in the response. That code snippet is to extract that extra
> info slot (could be several in a row, that's why there's a loop in
> xennet_get_extras). The request / response format is documented in
> include/xen/interface/io/netif.h -- it's only the tx one, but rx one is
> more or less the same.

> Wei.

>> 
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
[...]
> > From the look of the code, 0 looks correct to me. Netback won't complain
> > about "bad gref" because it has no idea what a "good gref" looks like.
> > Only Xen has the knowledge whether a gref is legit. All netback sees is
> > the hypercall fails and it tries to push corresponding response to
> > netfront. But you can probably safely assume a gref larger than several
> > thousands be bad.
> 
> OK it's indeed netback setting the status to -1:
> 
> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> 
> Now to find out why ..
> 

-1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.

grant copy fails -> netback sets response status to -1.

Assuming the traffic volumn is quite high both way (TX and RX), probably
worth checking if RX ring is overflowed. That is, if consumer pointer(s)
advance before producer pointer(s).

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 3:48:26 PM, you wrote:

> On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
> [...]
>> > From the look of the code, 0 looks correct to me. Netback won't complain
>> > about "bad gref" because it has no idea what a "good gref" looks like.
>> > Only Xen has the knowledge whether a gref is legit. All netback sees is
>> > the hypercall fails and it tries to push corresponding response to
>> > netfront. But you can probably safely assume a gref larger than several
>> > thousands be bad.
>> 
>> OK it's indeed netback setting the status to -1:
>> 
>> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
>> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
>> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
>> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
>> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
>> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
>> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
>> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
>> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
>> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
>> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
>> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> 
>> Now to find out why ..
>> 

> -1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.

grant copy fails ->> netback sets response status to -1.

> Assuming the traffic volumn is quite high both way (TX and RX), probably
> worth checking if RX ring is overflowed. That is, if consumer pointer(s)
> advance before producer pointer(s).

In what function ?

> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 12, 2014 at 03:49:46PM +0100, Sander Eikelenboom wrote:
> 
> Wednesday, March 12, 2014, 3:48:26 PM, you wrote:
> 
> > On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
> > [...]
> >> > From the look of the code, 0 looks correct to me. Netback won't complain
> >> > about "bad gref" because it has no idea what a "good gref" looks like.
> >> > Only Xen has the knowledge whether a gref is legit. All netback sees is
> >> > the hypercall fails and it tries to push corresponding response to
> >> > netfront. But you can probably safely assume a gref larger than several
> >> > thousands be bad.
> >> 
> >> OK it's indeed netback setting the status to -1:
> >> 
> >> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> >> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
> >> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
> >> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> >> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
> >> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
> >> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
> >> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
> >> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
> >> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
> >> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
> >> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
> >> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
> >> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> >> 
> >> Now to find out why ..
> >> 
> 
> > -1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.
> 
> grant copy fails ->> netback sets response status to -1.
> 
> > Assuming the traffic volumn is quite high both way (TX and RX), probably
> > worth checking if RX ring is overflowed. That is, if consumer pointer(s)
> > advance before producer pointer(s).
> 
> In what function ?
> 

That should be in RX path. Look for "RING_GET_REQUESTS(&vif->rx". That's
the macro to get a slot in RX ring; second argument is consumer index.
You can see if it advances before producer index.

Wei.

> > Wei.
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 3:59:15 PM, you wrote:

> On Wed, Mar 12, 2014 at 03:49:46PM +0100, Sander Eikelenboom wrote:
>> 
>> Wednesday, March 12, 2014, 3:48:26 PM, you wrote:
>> 
>> > On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
>> > [...]
>> >> > From the look of the code, 0 looks correct to me. Netback won't complain
>> >> > about "bad gref" because it has no idea what a "good gref" looks like.
>> >> > Only Xen has the knowledge whether a gref is legit. All netback sees is
>> >> > the hypercall fails and it tries to push corresponding response to
>> >> > netfront. But you can probably safely assume a gref larger than several
>> >> > thousands be bad.
>> >> 
>> >> OK it's indeed netback setting the status to -1:
>> >> 
>> >> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
>> >> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
>> >> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
>> >> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
>> >> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
>> >> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
>> >> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> 
>> >> Now to find out why ..
>> >> 
>> 
>> > -1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.
>> 
>> grant copy fails ->> netback sets response status to -1.
>> 
>> > Assuming the traffic volumn is quite high both way (TX and RX), probably
>> > worth checking if RX ring is overflowed. That is, if consumer pointer(s)
>> > advance before producer pointer(s).
>> 
>> In what function ?
>> 

> That should be in RX path. Look for "RING_GET_REQUESTS(&vif->rx". That's
> the macro to get a slot in RX ring; second argument is consumer index.
> You can see if it advances before producer index.

grep -r -i -n "RING_GET_REQUESTS" drivers/net/xen-netback/* delivers 0.0 results ..

> Wei.

>> > Wei.
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 12, 2014 at 04:01:56PM +0100, Sander Eikelenboom wrote:
> 
> Wednesday, March 12, 2014, 3:59:15 PM, you wrote:
> 
> > On Wed, Mar 12, 2014 at 03:49:46PM +0100, Sander Eikelenboom wrote:
> >> 
> >> Wednesday, March 12, 2014, 3:48:26 PM, you wrote:
> >> 
> >> > On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
> >> > [...]
> >> >> > From the look of the code, 0 looks correct to me. Netback won't complain
> >> >> > about "bad gref" because it has no idea what a "good gref" looks like.
> >> >> > Only Xen has the knowledge whether a gref is legit. All netback sees is
> >> >> > the hypercall fails and it tries to push corresponding response to
> >> >> > netfront. But you can probably safely assume a gref larger than several
> >> >> > thousands be bad.
> >> >> 
> >> >> OK it's indeed netback setting the status to -1:
> >> >> 
> >> >> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> >> >> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
> >> >> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
> >> >> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> >> >> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
> >> >> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
> >> >> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
> >> >> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
> >> >> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
> >> >> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
> >> >> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
> >> >> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
> >> >> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
> >> >> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
> >> >> 
> >> >> Now to find out why ..
> >> >> 
> >> 
> >> > -1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.
> >> 
> >> grant copy fails ->> netback sets response status to -1.
> >> 
> >> > Assuming the traffic volumn is quite high both way (TX and RX), probably
> >> > worth checking if RX ring is overflowed. That is, if consumer pointer(s)
> >> > advance before producer pointer(s).
> >> 
> >> In what function ?
> >> 
> 
> > That should be in RX path. Look for "RING_GET_REQUESTS(&vif->rx". That's
> > the macro to get a slot in RX ring; second argument is consumer index.
> > You can see if it advances before producer index.
> 
> grep -r -i -n "RING_GET_REQUESTS" drivers/net/xen-netback/* delivers 0.0 results ..
> 

Sorry, remove the trailing "S". Actually you only need to look at netback.c.

> > Wei.
> 
> >> > Wei.
> >> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 4:04:35 PM, you wrote:

> On Wed, Mar 12, 2014 at 04:01:56PM +0100, Sander Eikelenboom wrote:
>> 
>> Wednesday, March 12, 2014, 3:59:15 PM, you wrote:
>> 
>> > On Wed, Mar 12, 2014 at 03:49:46PM +0100, Sander Eikelenboom wrote:
>> >> 
>> >> Wednesday, March 12, 2014, 3:48:26 PM, you wrote:
>> >> 
>> >> > On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
>> >> > [...]
>> >> >> > From the look of the code, 0 looks correct to me. Netback won't complain
>> >> >> > about "bad gref" because it has no idea what a "good gref" looks like.
>> >> >> > Only Xen has the knowledge whether a gref is legit. All netback sees is
>> >> >> > the hypercall fails and it tries to push corresponding response to
>> >> >> > netfront. But you can probably safely assume a gref larger than several
>> >> >> > thousands be bad.
>> >> >> 
>> >> >> OK it's indeed netback setting the status to -1:
>> >> >> 
>> >> >> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> >> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
>> >> >> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
>> >> >> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> >> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
>> >> >> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
>> >> >> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
>> >> >> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
>> >> >> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
>> >> >> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> >> 
>> >> >> Now to find out why ..
>> >> >> 
>> >> 
>> >> > -1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.
>> >> 
>> >> grant copy fails ->> netback sets response status to -1.
>> >> 
>> >> > Assuming the traffic volumn is quite high both way (TX and RX), probably
>> >> > worth checking if RX ring is overflowed. That is, if consumer pointer(s)
>> >> > advance before producer pointer(s).
>> >> 
>> >> In what function ?
>> >> 
>> 
>> > That should be in RX path. Look for "RING_GET_REQUESTS(&vif->rx". That's
>> > the macro to get a slot in RX ring; second argument is consumer index.
>> > You can see if it advances before producer index.
>> 
>> grep -r -i -n "RING_GET_REQUESTS" drivers/net/xen-netback/* delivers 0.0 results ..
>> 

> Sorry, remove the trailing "S". Actually you only need to look at netback.c.

What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
npo->meta_prod
vif->rx.sring->req_prod
vif->pending_prod

to name a few ..
Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?


>> > Wei.
>> 
>> >> > Wei.
>> >> 
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 12, 2014 at 04:20:03PM +0100, Sander Eikelenboom wrote:
[...]
> 
> > Sorry, remove the trailing "S". Actually you only need to look at netback.c.
> 
> What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
> npo->meta_prod
> vif->rx.sring->req_prod
> vif->pending_prod
> 
> to name a few ..
> Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?
> 

There are two macros you can use

RING_REQUEST_CONS_OVERFLOW and RING_REQUEST_PROD_OVERFLOW.

> 
> >> > Wei.
> >> 
> >> >> > Wei.
> >> >> 
> >> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

[-- Attachment #1: Type: text/plain, Size: 21561 bytes --]


Wednesday, March 12, 2014, 4:45:01 PM, you wrote:

> On Wed, Mar 12, 2014 at 04:20:03PM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> > Sorry, remove the trailing "S". Actually you only need to look at netback.c.
>> 
>> What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
>> npo->meta_prod
>> vif->rx.sring->req_prod
>> vif->pending_prod
>> 
>> to name a few ..
>> Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?
>> 

> There are two macros you can use

> RING_REQUEST_CONS_OVERFLOW and RING_REQUEST_PROD_OVERFLOW.

Ah i already produced my own .. diff to netback is attached ..

Netback:
Mar 12 17:41:26 serveerstertje kernel: [  464.778614] vif vif-7-0 vif7.0: ?!? npo->meta_prod:37 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431007
Mar 12 17:41:26 serveerstertje kernel: [  464.786203] vif vif-7-0 vif7.0: ?!? npo->meta_prod:38 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431008
Mar 12 17:41:26 serveerstertje kernel: [  464.793749] vif vif-7-0 vif7.0: Bad status -3 from copy to DOM7.
Mar 12 17:41:26 serveerstertje kernel: [  464.801116] vif vif-7-0 vif7.0: ?!? xenvif_check_gop status err? status:-1 i:7 nr_meta_slots:8 copy_op->status:-3 npo->copy_cons:38
Mar 12 17:41:26 serveerstertje kernel: [  464.808636] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1 ip_summed:1 is_gso:1 skb_shinfo(skb)->gso_type:1634 vif->meta[npo.meta_cons].gso_type:3 nr_frags:1 npo.copy_prod:39, npo.meta_cons:30
Mar 12 17:41:26 serveerstertje kernel: [  464.824158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
Mar 12 17:41:26 serveerstertje kernel: [  464.832237] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
Mar 12 17:41:26 serveerstertje kernel: [  464.840146] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
Mar 12 17:41:26 serveerstertje kernel: [  464.847854] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
Mar 12 17:41:26 serveerstertje kernel: [  464.855596] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
Mar 12 17:41:26 serveerstertje kernel: [  464.863244] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096

Netfront:
Mar 12 17:41:26 backup kernel: [  284.941405] net eth0: rx->offset: 0, size: 4294967295
Mar 12 17:41:26 backup kernel: [  284.941427] net eth0: cons:430999 slots:1 rp:431008 max:18 err:0 rx->id:150 rx->offset:0 size:4294967295 ref:-131941395332386 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
Mar 12 17:41:26 backup kernel: [  284.941443] net eth0: rx->offset: 0, size: 4294967295
Mar 12 17:41:26 backup kernel: [  284.941450] net eth0: cons:430999 slots:2 rp:431008 max:18 err:-22 rx->id:152 rx->offset:0 size:4294967295 ref:-131941395332397 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
Mar 12 17:41:26 backup kernel: [  284.941464] net eth0: rx->offset: 0, size: 4294967295
Mar 12 17:41:26 backup kernel: [  284.941471] net eth0: cons:430999 slots:3 rp:431008 max:18 err:-22 rx->id:153 rx->offset:0 size:4294967295 ref:-131941395332400 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
Mar 12 17:41:26 backup kernel: [  284.941485] net eth0: rx->offset: 0, size: 4294967295
Mar 12 17:41:26 backup kernel: [  284.941492] net eth0: cons:430999 slots:4 rp:431008 max:18 err:-22 rx->id:154 rx->offset:0 size:4294967295 ref:-131941395332383 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
Mar 12 17:41:26 backup kernel: [  284.941506] net eth0: rx->offset: 0, size: 4294967295
Mar 12 17:41:26 backup kernel: [  284.941512] net eth0: cons:430999 slots:5 rp:431008 max:18 err:-22 rx->id:155 rx->offset:0 size:4294967295 ref:-131941395332594 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
Mar 12 17:41:26 backup kernel: [  284.941527] net eth0: rx->offset: 0, size: 4294967295
Mar 12 17:41:26 backup kernel: [  284.941557] net eth0: cons:430999 slots:6 rp:431008 max:18 err:-22 rx->id:156 rx->offset:0 size:4294967295 ref:-131941395332360 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
Mar 12 17:41:26 backup kernel: [  284.941572] net eth0: rx->offset: 0, size: 4294967295
Mar 12 17:41:26 backup kernel: [  284.941579] net eth0: cons:430999 slots:7 rp:431008 max:18 err:-22 rx->id:157 rx->offset:0 size:4294967295 ref:-131941395332552 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
Mar 12 17:41:26 backup kernel: [  284.941743] net eth0: Invalid extra type: 171
Mar 12 17:41:26 backup kernel: [  284.941750] net eth0: Invalid extra type: 174
Mar 12 17:41:26 backup kernel: [  284.941754] net eth0: Invalid extra type: 176

Xen:
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v5 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v5 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5459370 source_off:0 source_len:4096 op->source.offset:0 op->len:1168
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v5 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v5 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5467551 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:1325235656 source_len:4294935301 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 254279695 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:1325235656 source_len:4294935301 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 227672068 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 95682560 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:0 source_len:4096 op->source.offset:0 op->len:64
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:64 op->len:2
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 194904075 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5467552 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5459370 source_off:0 source_len:4096 op->source.offset:66 op->len:2928
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5459369 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457736 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457735 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457734 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457733 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457732 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457731 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457730 source_off:0 source_len:4096 op->source.offset:0 op->len:3152
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5461696 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457730 source_off:0 source_len:4096 op->source.offset:1634 op->len:944
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457729 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457728 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457727 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457726 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457725 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457724 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457723 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457722 source_off:0 source_len:4096 op->source.offset:0 op->len:3568
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 194904079 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 200409092 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 182321152 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:1514
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 237502479 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 53084160 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455318 source_off:0 source_len:4096 op->source.offset:0 op->len:66
(XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
(XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455318 source_off:0 source_len:4096 op->source.offset:0 op->len:66


>> 
>> >> > Wei.
>> >> 
>> >> >> > Wei.
>> >> >> 
>> >> 
>> 

[-- Attachment #2: netback_debug.diff --]
[-- Type: application/octet-stream, Size: 5777 bytes --]

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 98de8ab..66a7f9e 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -215,6 +215,7 @@ static struct xenvif_rx_meta *get_next_rx_buffer(struct xenvif *vif,
 	struct xenvif_rx_meta *meta;
 	struct xen_netif_rx_request *req;
 
+	if((vif->rx.req_cons + 1 > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d", npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons + 1);}
 	req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
 
 	meta = npo->meta + npo->meta_prod++;
@@ -356,6 +357,7 @@ static int xenvif_gop_skb(struct sk_buff *skb,
 
 	/* Set up a GSO prefix descriptor, if necessary */
 	if ((1 << gso_type) & vif->gso_prefix_mask) {
+        	if((vif->rx.req_cons + 1 > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d", npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons + 1);}
 		req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
 		meta = npo->meta + npo->meta_prod++;
 		meta->gso_type = gso_type;
@@ -363,7 +365,7 @@ static int xenvif_gop_skb(struct sk_buff *skb,
 		meta->size = 0;
 		meta->id = req->id;
 	}
-
+       	if((vif->rx.req_cons + 1 > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d", npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons + 1);}
 	req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
 	meta = npo->meta + npo->meta_prod++;
 
@@ -380,6 +382,7 @@ static int xenvif_gop_skb(struct sk_buff *skb,
 	npo->copy_off = 0;
 	npo->copy_gref = req->gref;
 
+
 	data = skb->data;
 	while (data < skb_tail_pointer(skb)) {
 		unsigned int offset = offset_in_page(data);
@@ -420,11 +423,14 @@ static int xenvif_check_gop(struct xenvif *vif, int nr_meta_slots,
 	for (i = 0; i < nr_meta_slots; i++) {
 		copy_op = npo->copy + npo->copy_cons++;
 		if (copy_op->status != GNTST_okay) {
-			netdev_dbg(vif->dev,
+                    if (net_ratelimit()){
+			netdev_warn(vif->dev,
 				   "Bad status %d from copy to DOM%d.\n",
 				   copy_op->status, vif->domid);
 			status = XEN_NETIF_RSP_ERROR;
-		}
+                        netdev_warn(vif->dev, "?!? %s status err? status:%d i:%d nr_meta_slots:%d copy_op->status:%d npo->copy_cons:%d  \n", __FUNCTION__,status, i, nr_meta_slots, copy_op->status, npo->copy_cons );
+		    }
+       		}
 	}
 
 	return status;
@@ -451,8 +457,14 @@ static void xenvif_add_frag_responses(struct xenvif *vif, int status,
 			flags = XEN_NETRXF_more_data;
 
 		offset = 0;
+
+                if(status > 10000 || status < 0){
+                    if (net_ratelimit())
+                        netdev_warn(vif->dev, "?!? %s status err? status:%d i:%d nr_meta_slots:%d flags:%d size:%d\n", __FUNCTION__,status, i, nr_meta_slots, flags, meta[i].size);
+                }
 		make_rx_response(vif, meta[i].id, status, offset,
 				 meta[i].size, flags);
+	
 	}
 }
 
@@ -514,6 +526,9 @@ static void xenvif_rx_action(struct xenvif *vif)
 
 		sco = (struct skb_cb_overlay *)skb->cb;
 		sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
+
+
+
 		BUG_ON(sco->meta_slots_used > max_slots_needed);
 
 		__skb_queue_tail(&rxq, skb);
@@ -551,6 +566,8 @@ static void xenvif_rx_action(struct xenvif *vif)
 
 		status = xenvif_check_gop(vif, sco->meta_slots_used, &npo);
 
+
+
 		if (sco->meta_slots_used == 1)
 			flags = 0;
 		else
@@ -563,6 +580,12 @@ static void xenvif_rx_action(struct xenvif *vif)
 			flags |= XEN_NETRXF_data_validated;
 
 		offset = 0;
+
+                if(status > 10000 || status < 0){
+                    if (net_ratelimit())
+                        netdev_warn(vif->dev, "?!? %s status err? status:%d meta_slots_used:%d flags:%d size:%d ip_summed:%d is_gso:%d skb_shinfo(skb)->gso_type:%d vif->meta[npo.meta_cons].gso_type:%d nr_frags:%d npo.copy_prod:%d, npo.meta_cons:%d\n", __FUNCTION__,status, sco->meta_slots_used, flags, skb_is_gso(skb) , skb_shinfo(skb)->gso_type, vif->meta[npo.meta_cons].gso_type, vif->meta[npo.meta_cons].size, skb->ip_summed, skb_shinfo(skb)->nr_frags , npo.copy_prod, npo.meta_cons);
+                }
+		
 		resp = make_rx_response(vif, vif->meta[npo.meta_cons].id,
 					status, offset,
 					vif->meta[npo.meta_cons].size,
@@ -713,7 +736,6 @@ static int xenvif_count_requests(struct xenvif *vif,
 
 		if (drop_err)
 			txp = &dropped_tx;
-
 		memcpy(txp, RING_GET_REQUEST(&vif->tx, cons + slots),
 		       sizeof(*txp));
 
@@ -823,6 +845,12 @@ static struct gnttab_copy *xenvif_get_requests(struct xenvif *vif,
 			gop->dest.offset = dst_offset;
 			gop->dest.u.gmfn = virt_to_mfn(page_address(page));
 
+                        if(txp->gref > 10000 || txp->gref <=0){
+	                     if (net_ratelimit())
+                                 netdev_warn(vif->dev, "?!? %s ref out of bounds? gref:%d srcdomid:%d dstdomid:%d slot:%d \n",__FUNCTION__, txp->gref, vif->domid, DOMID_SELF, slot);
+                        }
+	
+
 			if (dst_offset + txp->size > PAGE_SIZE) {
 				/* This page can only merge a portion
 				 * of tx request. Do not increment any
@@ -1000,7 +1028,6 @@ static int xenvif_get_extras(struct xenvif *vif,
 			xenvif_fatal_tx_err(vif);
 			return -EBADR;
 		}
-
 		memcpy(&extra, RING_GET_REQUEST(&vif->tx, cons),
 		       sizeof(extra));
 		if (unlikely(!extra.type ||
@@ -1393,6 +1420,7 @@ static unsigned xenvif_tx_build_gops(struct xenvif *vif, int budget)
 			break;
 
 		idx = vif->tx.req_cons;
+
 		rmb(); /* Ensure that we see the request before we copy it. */
 		memcpy(&txreq, RING_GET_REQUEST(&vif->tx, idx), sizeof(txreq));
 

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 5:47:29 PM, you wrote:


> Wednesday, March 12, 2014, 4:45:01 PM, you wrote:

>> On Wed, Mar 12, 2014 at 04:20:03PM +0100, Sander Eikelenboom wrote:
>> [...]
>>> 
>>> > Sorry, remove the trailing "S". Actually you only need to look at netback.c.
>>> 
>>> What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
>>> npo->meta_prod
>>> vif->rx.sring->req_prod
>>> vif->pending_prod
>>> 
>>> to name a few ..
>>> Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?
>>> 

>> There are two macros you can use

>> RING_REQUEST_CONS_OVERFLOW and RING_REQUEST_PROD_OVERFLOW.

> Ah i already produced my own .. diff to netback is attached ..

> Netback:
> Mar 12 17:41:26 serveerstertje kernel: [  464.778614] vif vif-7-0 vif7.0: ?!? npo->meta_prod:37 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431007
> Mar 12 17:41:26 serveerstertje kernel: [  464.786203] vif vif-7-0 vif7.0: ?!? npo->meta_prod:38 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431008
> Mar 12 17:41:26 serveerstertje kernel: [  464.793749] vif vif-7-0 vif7.0: Bad status -3 from copy to DOM7.
> Mar 12 17:41:26 serveerstertje kernel: [  464.801116] vif vif-7-0 vif7.0: ?!? xenvif_check_gop status err? status:-1 i:7 nr_meta_slots:8 copy_op->status:-3 npo->copy_cons:38
> Mar 12 17:41:26 serveerstertje kernel: [  464.808636] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1 ip_summed:1 is_gso:1 skb_shinfo(skb)->gso_type:1634 vif->meta[npo.meta_cons].gso_type:3 nr_frags:1 npo.copy_prod:39, npo.meta_cons:30
> Mar 12 17:41:26 serveerstertje kernel: [  464.824158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
> Mar 12 17:41:26 serveerstertje kernel: [  464.832237] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
> Mar 12 17:41:26 serveerstertje kernel: [  464.840146] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
> Mar 12 17:41:26 serveerstertje kernel: [  464.847854] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
> Mar 12 17:41:26 serveerstertje kernel: [  464.855596] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
> Mar 12 17:41:26 serveerstertje kernel: [  464.863244] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096

> Netfront:
> Mar 12 17:41:26 backup kernel: [  284.941405] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 17:41:26 backup kernel: [  284.941427] net eth0: cons:430999 slots:1 rp:431008 max:18 err:0 rx->id:150 rx->offset:0 size:4294967295 ref:-131941395332386 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
> Mar 12 17:41:26 backup kernel: [  284.941443] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 17:41:26 backup kernel: [  284.941450] net eth0: cons:430999 slots:2 rp:431008 max:18 err:-22 rx->id:152 rx->offset:0 size:4294967295 ref:-131941395332397 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
> Mar 12 17:41:26 backup kernel: [  284.941464] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 17:41:26 backup kernel: [  284.941471] net eth0: cons:430999 slots:3 rp:431008 max:18 err:-22 rx->id:153 rx->offset:0 size:4294967295 ref:-131941395332400 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
> Mar 12 17:41:26 backup kernel: [  284.941485] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 17:41:26 backup kernel: [  284.941492] net eth0: cons:430999 slots:4 rp:431008 max:18 err:-22 rx->id:154 rx->offset:0 size:4294967295 ref:-131941395332383 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
> Mar 12 17:41:26 backup kernel: [  284.941506] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 17:41:26 backup kernel: [  284.941512] net eth0: cons:430999 slots:5 rp:431008 max:18 err:-22 rx->id:155 rx->offset:0 size:4294967295 ref:-131941395332594 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
> Mar 12 17:41:26 backup kernel: [  284.941527] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 17:41:26 backup kernel: [  284.941557] net eth0: cons:430999 slots:6 rp:431008 max:18 err:-22 rx->id:156 rx->offset:0 size:4294967295 ref:-131941395332360 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
> Mar 12 17:41:26 backup kernel: [  284.941572] net eth0: rx->offset: 0, size: 4294967295
> Mar 12 17:41:26 backup kernel: [  284.941579] net eth0: cons:430999 slots:7 rp:431008 max:18 err:-22 rx->id:157 rx->offset:0 size:4294967295 ref:-131941395332552 pagesize:4096 skb_ipsummed:0 is_gso:0 gso_size:0 gso_type:0 gso_segs:0 RING_HAS_UNCONSUMED_RESPONSES:9 cons_changed:1 cons_before:430998 xennet_get_extras_err:0
> Mar 12 17:41:26 backup kernel: [  284.941743] net eth0: Invalid extra type: 171
> Mar 12 17:41:26 backup kernel: [  284.941750] net eth0: Invalid extra type: 174
> Mar 12 17:41:26 backup kernel: [  284.941754] net eth0: Invalid extra type: 176

> Xen:
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v5 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v5 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5459370 source_off:0 source_len:4096 op->source.offset:0 op->len:1168
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v5 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v5 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5467551 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:1325235656 source_len:4294935301 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 254279695 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:1325235656 source_len:4294935301 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 227672068 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 95682560 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456674 source_off:0 source_len:4096 op->source.offset:0 op->len:64
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:64 op->len:2
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 194904075 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5467552 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5459370 source_off:0 source_len:4096 op->source.offset:66 op->len:2928
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5459369 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457736 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457735 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457734 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457733 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457732 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457731 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457730 source_off:0 source_len:4096 op->source.offset:0 op->len:3152
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5461696 source_off:0 source_len:4096 op->source.offset:0 op->len:1634
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457730 source_off:0 source_len:4096 op->source.offset:1634 op->len:944
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457729 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457728 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457727 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457726 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457725 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457724 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457723 source_off:0 source_len:4096 op->source.offset:0 op->len:4096
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5457722 source_off:0 source_len:4096 op->source.offset:0 op->len:3568
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 194904079 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5456673 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 200409092 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 182321152 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:1514
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 237502479 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455320 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 53084160 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 19791875 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455319 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325379 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455318 source_off:0 source_len:4096 op->source.offset:0 op->len:66
> (XEN) [2014-03-12 16:41:26] grant_table.c:1856:d0v1 Bad grant reference 4325377 gt_version:1 ldom:0 readonly:0 allow_transitive:1
> (XEN) [2014-03-12 16:41:26] grant_table.c:2100:d0v1 acquire_grant_for_copy failed .. dest_is_gref rc:-3 source.domid:32752 dest.domid:7 s_frame:5455318 source_off:0 source_len:4096 op->source.offset:0 op->len:66

>>>
>>> >> > Wei.
>>> >> 
>>> >> >> > Wei.
>>> >> >> 
>>> >> 
>>> 

Hi Wei,

Do you think this supplies enough debug info for you ?

--
Sander

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 12, 2014 at 05:47:29PM +0100, Sander Eikelenboom wrote:
> 
> Wednesday, March 12, 2014, 4:45:01 PM, you wrote:
> 
> > On Wed, Mar 12, 2014 at 04:20:03PM +0100, Sander Eikelenboom wrote:
> > [...]
> >> 
> >> > Sorry, remove the trailing "S". Actually you only need to look at netback.c.
> >> 
> >> What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
> >> npo->meta_prod
> >> vif->rx.sring->req_prod
> >> vif->pending_prod
> >> 
> >> to name a few ..
> >> Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?
> >> 
> 
> > There are two macros you can use
> 
> > RING_REQUEST_CONS_OVERFLOW and RING_REQUEST_PROD_OVERFLOW.
> 
> Ah i already produced my own .. diff to netback is attached ..
> 
> Netback:
> Mar 12 17:41:26 serveerstertje kernel: [  464.778614] vif vif-7-0 vif7.0: ?!? npo->meta_prod:37 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431007
> Mar 12 17:41:26 serveerstertje kernel: [  464.786203] vif vif-7-0 vif7.0: ?!? npo->meta_prod:38 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431008

req_prod < req_cons, so there's an overflow. I'm actually curious how
this could happen.

Back to the code, before netback enqueues SKB to its internal queue, it
will check if there's enough room in the ring. Before Paul's changeset,
it checks against a static number (the possible maximum slots that can
be consumed by an SKB). Paul's changeset made it check against the
actual slots the incoming SKB consumes. See
interface.c:xenvif_start_xmit.  Another interesting site would be when
the SKB is broken down later on in internal queue. See
netback.c:xenvif_rx_action. The routine to break down an SKB is
xenvif_gop_skb.

Although they look alright to me, but you might want to instrument them
a bit more to see what triggers that overflow. It's a bit frustrating,
but a bug that cannot be easily reproduced is indeed extremely hard to
fix.

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

[-- Attachment #1: Type: text/plain, Size: 7303 bytes --]


Monday, March 17, 2014, 11:35:24 AM, you wrote:

> On Wed, Mar 12, 2014 at 05:47:29PM +0100, Sander Eikelenboom wrote:
>> 
>> Wednesday, March 12, 2014, 4:45:01 PM, you wrote:
>> 
>> > On Wed, Mar 12, 2014 at 04:20:03PM +0100, Sander Eikelenboom wrote:
>> > [...]
>> >> 
>> >> > Sorry, remove the trailing "S". Actually you only need to look at netback.c.
>> >> 
>> >> What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
>> >> npo->meta_prod
>> >> vif->rx.sring->req_prod
>> >> vif->pending_prod
>> >> 
>> >> to name a few ..
>> >> Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?
>> >> 
>> 
>> > There are two macros you can use
>> 
>> > RING_REQUEST_CONS_OVERFLOW and RING_REQUEST_PROD_OVERFLOW.
>> 
>> Ah i already produced my own .. diff to netback is attached ..
>> 
>> Netback:
>> Mar 12 17:41:26 serveerstertje kernel: [  464.778614] vif vif-7-0 vif7.0: ?!? npo->meta_prod:37 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431007
>> Mar 12 17:41:26 serveerstertje kernel: [  464.786203] vif vif-7-0 vif7.0: ?!? npo->meta_prod:38 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431008

> req_prod < req_cons, so there's an overflow. I'm actually curious how
> this could happen.

> Back to the code, before netback enqueues SKB to its internal queue, it
> will check if there's enough room in the ring. Before Paul's changeset,
> it checks against a static number (the possible maximum slots that can
> be consumed by an SKB). Paul's changeset made it check against the
> actual slots the incoming SKB consumes. See
> interface.c:xenvif_start_xmit.  Another interesting site would be when
> the SKB is broken down later on in internal queue. See
> netback.c:xenvif_rx_action. The routine to break down an SKB is
> xenvif_gop_skb.

> Although they look alright to me, but you might want to instrument them
> a bit more to see what triggers that overflow. It's a bit frustrating,
> but a bug that cannot be easily reproduced is indeed extremely hard to
> fix.

[  554.166914] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:7 vif->rx.sring->req_prod:301750 vif->rx.req_cons:301751
[  571.143788] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823
[  571.150762] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 req->gref:19791875 req->id:22
[  571.164691] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 npo->copy_gref:19791875
[  571.179474] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 gso_type:1 gso_size:1448 nr_frags:1 req->gref:576 req->id:14
[  571.194844] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0
[  571.211215] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325379 req->id:23
[  571.228471] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 npo->meta_prod:39 old_meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325379 req->id:23
[  571.246412] vif vif-7-0 vif7.0: Bad status -3 from copy to DOM7.
[  571.255531] vif vif-7-0 vif7.0: ?!? xenvif_check_gop status err? status:-1 i:7 nr_meta_slots:8 copy_op->status:-3 npo->copy_cons:38
[  571.264804] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1 ip_summed:1 is_gso:1 skb_shinfo(skb)->gso_type:1634 vif->meta[npo.meta_cons].gso_type:3 nr_frags:1 npo.copy_prod:39, npo.meta_cons:30

It seems in 'get_next_rx_buffer' you can get the situation that cons > prod, but the first time somehow the code after the actual RING_GET_REQUEST isn't ran (why?!? queue stop ?) .. and it doesn't lead to an immediate problem ..
The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...

So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:

[  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222
[  403.409624] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:191612 vif->rx.req_cons:191609
[  403.658975] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:192860 vif->rx.req_cons:192860
[  406.417504] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:204870 vif->rx.req_cons:204869
[  408.156112] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:212862 vif->rx.req_cons:212862
[  408.407375] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:213613 vif->rx.req_cons:213609
[  408.899250] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:215352 vif->rx.req_cons:215352
[  413.655552] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:237250 vif->rx.req_cons:237249
[  417.649867] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:252580 vif->rx.req_cons:252580
[  418.400194] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:255828 vif->rx.req_cons:255826
[  419.147431] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:258745 vif->rx.req_cons:258746
[  419.149966] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:258745 vif->rx.req_cons:258745
[  419.153145] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:258745 vif->rx.req_cons:258746 req->gref:19791875 req->id:185
[  419.159518] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:258745 vif->rx.req_cons:258746 npo->copy_gref:19791875
[  419.166980] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:258745 vif->rx.req_cons:258746 gso_type:1 gso_size:1448 nr_frags:1 req->gref:686 req->id:177
[  419.175597] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:38 vif->rx.sring->req_prod:258745 vif->rx.req_cons:258747 gso_type:0 gso_size:0 nr_frags:0
[  419.185284] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:38 vif->rx.sring->req_prod:258745 vif->rx.req_cons:258747 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325379 req->id:186
[  422.717242] net_ratelimit: 158 callbacks suppressed

Hope that this time it does shed some light :-)

diff to netback is attached.

--
Sander

> Wei.

[-- Attachment #2: netback_debug.diff --]
[-- Type: application/octet-stream, Size: 17211 bytes --]

diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
index b9de31e..1809728 100644
--- a/drivers/net/xen-netback/interface.c
+++ b/drivers/net/xen-netback/interface.c
@@ -122,9 +122,13 @@ static int xenvif_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	BUG_ON(skb->dev != dev);
 
 	/* Drop the packet if vif is not ready */
-	if (vif->task == NULL || !xenvif_schedulable(vif))
+	if (vif->task == NULL || !xenvif_schedulable(vif)){
+                if (net_ratelimit())
+                        netdev_warn(vif->dev, "?!? %s dropping packet vif not ready !  min_slots_needed:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d\n", __FUNCTION__, min_slots_needed, vif->rx.sring->req_prod, vif->rx.req_cons);
+	
 		goto drop;
-
+	}
+	
 	/* At best we'll need one slot for the header and one for each
 	 * frag.
 	 */
@@ -133,16 +137,19 @@ static int xenvif_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	/* If the skb is GSO then we'll also need an extra slot for the
 	 * metadata.
 	 */
-	if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
-	    skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
+	if (skb_is_gso(skb))
 		min_slots_needed++;
 
 	/* If the skb can't possibly fit in the remaining slots
 	 * then turn off the queue to give the ring a chance to
 	 * drain.
 	 */
-	if (!xenvif_rx_ring_slots_available(vif, min_slots_needed))
+	if (!xenvif_rx_ring_slots_available(vif, min_slots_needed)){
+                if (net_ratelimit())
+                        netdev_warn(vif->dev, "?!? %s stopping queue !  min_slots_needed:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d\n", __FUNCTION__, min_slots_needed, vif->rx.sring->req_prod, vif->rx.req_cons);
+
 		xenvif_stop_queue(vif);
+	}
 
 	skb_queue_tail(&vif->rx_queue, skb);
 	xenvif_kick_thread(vif);
diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 98de8ab..2f86a63 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -157,6 +157,8 @@ bool xenvif_rx_ring_slots_available(struct xenvif *vif, int needed)
 		mb();
 	} while (vif->rx.sring->req_prod != prod);
 
+       	if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s slotsavailable: false vif->rx.sring->req_prod:%d vif->rx.req_cons:%d vif->rx.sring->req_event:%d needed:%d",__FUNCTION__, vif->rx.sring->req_prod, vif->rx.req_cons, vif->rx.sring->req_event, needed);}
+
 	return false;
 }
 
@@ -215,7 +217,9 @@ static struct xenvif_rx_meta *get_next_rx_buffer(struct xenvif *vif,
 	struct xenvif_rx_meta *meta;
 	struct xen_netif_rx_request *req;
 
+	if((vif->rx.req_cons + 1 > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s before req npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d", __FUNCTION__, npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons + 1);}
 	req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
+	if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s after req npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d req->gref:%d req->id:%d", __FUNCTION__, npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons, req->gref, req->id);}
 
 	meta = npo->meta + npo->meta_prod++;
 	meta->gso_type = XEN_NETIF_GSO_TYPE_NONE;
@@ -241,7 +245,7 @@ static void xenvif_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
 	struct gnttab_copy *copy_gop;
 	struct xenvif_rx_meta *meta;
 	unsigned long bytes;
-	int gso_type;
+	int gso_type = XEN_NETIF_GSO_TYPE_NONE;
 
 	/* Data must not cross a page boundary. */
 	BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
@@ -267,8 +271,10 @@ static void xenvif_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
 			 * buffer.
 			 */
 			BUG_ON(*head);
-
+			if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s Me here 1  npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d npo->copy_gref:%d", __FUNCTION__,npo->meta_prod, vif->rx.sring->req_prod, vif->rx.req_cons, npo->copy_gref);}
 			meta = get_next_rx_buffer(vif, npo);
+			if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s Me here 2  npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d npo->copy_gref:%d", __FUNCTION__,npo->meta_prod, vif->rx.sring->req_prod, vif->rx.req_cons, npo->copy_gref);}
+
 		}
 
 		if (npo->copy_off + bytes > MAX_BUFFER_OFFSET)
@@ -300,16 +306,17 @@ static void xenvif_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
 		}
 
 		/* Leave a gap for the GSO descriptor. */
-		if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4)
-			gso_type = XEN_NETIF_GSO_TYPE_TCPV4;
-		else if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
-			gso_type = XEN_NETIF_GSO_TYPE_TCPV6;
-		else
-			gso_type = XEN_NETIF_GSO_TYPE_NONE;
+		if (skb_is_gso(skb)) {
+			if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4)
+				gso_type = XEN_NETIF_GSO_TYPE_TCPV4;
+			else if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
+				gso_type = XEN_NETIF_GSO_TYPE_TCPV6;
+		}
 
-		if (*head && ((1 << gso_type) & vif->gso_mask))
+		if (*head && ((1 << gso_type) & vif->gso_mask)){
 			vif->rx.req_cons++;
-
+			if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s  Me here 3  npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d gso_type:%d npo->copy_gref:%d", __FUNCTION__,npo->meta_prod, vif->rx.sring->req_prod, vif->rx.req_cons, skb_shinfo(skb)->gso_type, npo->copy_gref);}	
+		}
 		*head = 0; /* There must be something in this buffer now. */
 
 	}
@@ -339,37 +346,37 @@ static int xenvif_gop_skb(struct sk_buff *skb,
 	int head = 1;
 	int old_meta_prod;
 	int gso_type;
-	int gso_size;
 
 	old_meta_prod = npo->meta_prod;
 
-	if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4) {
-		gso_type = XEN_NETIF_GSO_TYPE_TCPV4;
-		gso_size = skb_shinfo(skb)->gso_size;
-	} else if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6) {
-		gso_type = XEN_NETIF_GSO_TYPE_TCPV6;
-		gso_size = skb_shinfo(skb)->gso_size;
-	} else {
-		gso_type = XEN_NETIF_GSO_TYPE_NONE;
-		gso_size = 0;
+	gso_type = XEN_NETIF_GSO_TYPE_NONE;
+	if (skb_is_gso(skb)) {
+		if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4)
+			gso_type = XEN_NETIF_GSO_TYPE_TCPV4;
+		else if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
+			gso_type = XEN_NETIF_GSO_TYPE_TCPV6;
 	}
 
 	/* Set up a GSO prefix descriptor, if necessary */
 	if ((1 << gso_type) & vif->gso_prefix_mask) {
+        	if((vif->rx.req_cons + 1 > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s Me here 1 before req npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d gso_type:%d gso_size:%d nr_frags:%d", __FUNCTION__,npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons + 1, skb_shinfo(skb)->gso_type, skb_shinfo(skb)->gso_size, nr_frags);}
 		req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
+        	if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s Me here 1 after req npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d gso_type:%d gso_size:%d nr_frags:%d req->gref:%d req->id:%d", __FUNCTION__,npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons , skb_shinfo(skb)->gso_type, skb_shinfo(skb)->gso_size, nr_frags, req->gref, req->id);}
 		meta = npo->meta + npo->meta_prod++;
 		meta->gso_type = gso_type;
-		meta->gso_size = gso_size;
+		meta->gso_size = skb_shinfo(skb)->gso_size;
 		meta->size = 0;
 		meta->id = req->id;
 	}
-
+       	if((vif->rx.req_cons + 1 > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s Me here 2 before req npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d gso_type:%d gso_size:%d nr_frags:%d", __FUNCTION__,npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons + 1, skb_shinfo(skb)->gso_type, skb_shinfo(skb)->gso_size, nr_frags);}
 	req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
+       	if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s Me here 2 after req npo->meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d gso_type:%d gso_size:%d nr_frags:%d req->gref:%d req->id:%d", __FUNCTION__,npo->meta_prod , vif->rx.sring->req_prod, vif->rx.req_cons, skb_shinfo(skb)->gso_type, skb_shinfo(skb)->gso_size, nr_frags, req->gref, req->id);}
+
 	meta = npo->meta + npo->meta_prod++;
 
 	if ((1 << gso_type) & vif->gso_mask) {
 		meta->gso_type = gso_type;
-		meta->gso_size = gso_size;
+		meta->gso_size = skb_shinfo(skb)->gso_size;
 	} else {
 		meta->gso_type = XEN_NETIF_GSO_TYPE_NONE;
 		meta->gso_size = 0;
@@ -380,6 +387,7 @@ static int xenvif_gop_skb(struct sk_buff *skb,
 	npo->copy_off = 0;
 	npo->copy_gref = req->gref;
 
+
 	data = skb->data;
 	while (data < skb_tail_pointer(skb)) {
 		unsigned int offset = offset_in_page(data);
@@ -401,6 +409,8 @@ static int xenvif_gop_skb(struct sk_buff *skb,
 				     &head);
 	}
 
+       	if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s Me here 3 npo->meta_prod:%d old_meta_prod:%d vif->rx.sring->req_prod:%d vif->rx.req_cons:%d gso_type:%d gso_size:%d nr_frags:%d req->gref:%d req->id:%d", __FUNCTION__,npo->meta_prod , old_meta_prod,vif->rx.sring->req_prod, vif->rx.req_cons, skb_shinfo(skb)->gso_type, skb_shinfo(skb)->gso_size, nr_frags, req->gref, req->id);}
+
 	return npo->meta_prod - old_meta_prod;
 }
 
@@ -420,11 +430,14 @@ static int xenvif_check_gop(struct xenvif *vif, int nr_meta_slots,
 	for (i = 0; i < nr_meta_slots; i++) {
 		copy_op = npo->copy + npo->copy_cons++;
 		if (copy_op->status != GNTST_okay) {
-			netdev_dbg(vif->dev,
+                    if (net_ratelimit()){
+			netdev_warn(vif->dev,
 				   "Bad status %d from copy to DOM%d.\n",
 				   copy_op->status, vif->domid);
 			status = XEN_NETIF_RSP_ERROR;
-		}
+                        netdev_warn(vif->dev, "?!? %s status err? status:%d i:%d nr_meta_slots:%d copy_op->status:%d npo->copy_cons:%d  \n", __FUNCTION__,status, i, nr_meta_slots, copy_op->status, npo->copy_cons );
+		    }
+       		}
 	}
 
 	return status;
@@ -451,8 +464,14 @@ static void xenvif_add_frag_responses(struct xenvif *vif, int status,
 			flags = XEN_NETRXF_more_data;
 
 		offset = 0;
+
+                if(status > 10000 || status < 0){
+                    if (net_ratelimit())
+                        netdev_warn(vif->dev, "?!? %s status err? status:%d i:%d nr_meta_slots:%d flags:%d size:%d\n", __FUNCTION__,status, i, nr_meta_slots, flags, meta[i].size);
+                }
 		make_rx_response(vif, meta[i].id, status, offset,
 				 meta[i].size, flags);
+	
 	}
 }
 
@@ -501,12 +520,15 @@ static void xenvif_rx_action(struct xenvif *vif)
 			size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
 			max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
 		}
-		if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
-		    skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
+		if (skb_is_gso(skb) &&
+		   (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
+		    skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6))
 			max_slots_needed++;
 
 		/* If the skb may not fit then bail out now */
 		if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
+
+
 			skb_queue_head(&vif->rx_queue, skb);
 			need_to_notify = 1;
 			break;
@@ -514,6 +536,9 @@ static void xenvif_rx_action(struct xenvif *vif)
 
 		sco = (struct skb_cb_overlay *)skb->cb;
 		sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
+
+
+
 		BUG_ON(sco->meta_slots_used > max_slots_needed);
 
 		__skb_queue_tail(&rxq, skb);
@@ -521,8 +546,10 @@ static void xenvif_rx_action(struct xenvif *vif)
 
 	BUG_ON(npo.meta_prod > ARRAY_SIZE(vif->meta));
 
-	if (!npo.copy_prod)
+	if (!npo.copy_prod){
+	        if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s go to done ..  vif->rx.sring->req_prod:%d vif->rx.req_cons:%d", __FUNCTION__, vif->rx.sring->req_prod, vif->rx.req_cons);}
 		goto done;
+	}
 
 	BUG_ON(npo.copy_prod > MAX_GRANT_COPY_OPS);
 	gnttab_batch_copy(vif->grant_copy_op, npo.copy_prod);
@@ -551,6 +578,8 @@ static void xenvif_rx_action(struct xenvif *vif)
 
 		status = xenvif_check_gop(vif, sco->meta_slots_used, &npo);
 
+
+
 		if (sco->meta_slots_used == 1)
 			flags = 0;
 		else
@@ -563,6 +592,12 @@ static void xenvif_rx_action(struct xenvif *vif)
 			flags |= XEN_NETRXF_data_validated;
 
 		offset = 0;
+
+                if(status > 10000 || status < 0){
+                    if (net_ratelimit())
+                        netdev_warn(vif->dev, "?!? %s status err? status:%d meta_slots_used:%d flags:%d size:%d ip_summed:%d is_gso:%d skb_shinfo(skb)->gso_type:%d vif->meta[npo.meta_cons].gso_type:%d nr_frags:%d npo.copy_prod:%d, npo.meta_cons:%d\n", __FUNCTION__,status, sco->meta_slots_used, flags, skb_is_gso(skb) , skb_shinfo(skb)->gso_type, vif->meta[npo.meta_cons].gso_type, vif->meta[npo.meta_cons].size, skb->ip_summed, skb_shinfo(skb)->nr_frags , npo.copy_prod, npo.meta_cons);
+                }
+		
 		resp = make_rx_response(vif, vif->meta[npo.meta_cons].id,
 					status, offset,
 					vif->meta[npo.meta_cons].size,
@@ -600,8 +635,10 @@ static void xenvif_rx_action(struct xenvif *vif)
 	}
 
 done:
-	if (need_to_notify)
+	if (need_to_notify){
+	        if((vif->rx.req_cons > vif->rx.sring->req_prod) && net_ratelimit()){netdev_warn(vif->dev, "?!? %s need to notify ..  vif->rx.sring->req_prod:%d vif->rx.req_cons:%d", __FUNCTION__, vif->rx.sring->req_prod, vif->rx.req_cons);}
 		notify_remote_via_irq(vif->rx_irq);
+	}
 }
 
 void xenvif_check_rx_xenvif(struct xenvif *vif)
@@ -705,7 +742,7 @@ static int xenvif_count_requests(struct xenvif *vif,
 		 */
 		if (!drop_err && slots >= XEN_NETBK_LEGACY_SLOTS_MAX) {
 			if (net_ratelimit())
-				netdev_dbg(vif->dev,
+				netdev_warn(vif->dev,
 					   "Too many slots (%d) exceeding limit (%d), dropping packet\n",
 					   slots, XEN_NETBK_LEGACY_SLOTS_MAX);
 			drop_err = -E2BIG;
@@ -713,7 +750,6 @@ static int xenvif_count_requests(struct xenvif *vif,
 
 		if (drop_err)
 			txp = &dropped_tx;
-
 		memcpy(txp, RING_GET_REQUEST(&vif->tx, cons + slots),
 		       sizeof(*txp));
 
@@ -728,7 +764,7 @@ static int xenvif_count_requests(struct xenvif *vif,
 		 */
 		if (!drop_err && txp->size > first->size) {
 			if (net_ratelimit())
-				netdev_dbg(vif->dev,
+				netdev_warn(vif->dev,
 					   "Invalid tx request, slot size %u > remaining size %u\n",
 					   txp->size, first->size);
 			drop_err = -EIO;
@@ -823,6 +859,12 @@ static struct gnttab_copy *xenvif_get_requests(struct xenvif *vif,
 			gop->dest.offset = dst_offset;
 			gop->dest.u.gmfn = virt_to_mfn(page_address(page));
 
+                        if(txp->gref > 10000 || txp->gref <=0){
+	                     if (net_ratelimit())
+                                 netdev_warn(vif->dev, "?!? %s ref out of bounds? gref:%d srcdomid:%d dstdomid:%d slot:%d nr_slots:%d\n",__FUNCTION__, txp->gref, vif->domid, DOMID_SELF, slot, nr_slots);
+                        }
+	
+
 			if (dst_offset + txp->size > PAGE_SIZE) {
 				/* This page can only merge a portion
 				 * of tx request. Do not increment any
@@ -1000,7 +1042,6 @@ static int xenvif_get_extras(struct xenvif *vif,
 			xenvif_fatal_tx_err(vif);
 			return -EBADR;
 		}
-
 		memcpy(&extra, RING_GET_REQUEST(&vif->tx, cons),
 		       sizeof(extra));
 		if (unlikely(!extra.type ||
@@ -1393,6 +1434,7 @@ static unsigned xenvif_tx_build_gops(struct xenvif *vif, int budget)
 			break;
 
 		idx = vif->tx.req_cons;
+
 		rmb(); /* Ensure that we see the request before we copy it. */
 		memcpy(&txreq, RING_GET_REQUEST(&vif->tx, idx), sizeof(txreq));
 
@@ -1422,8 +1464,9 @@ static unsigned xenvif_tx_build_gops(struct xenvif *vif, int budget)
 		idx += ret;
 
 		if (unlikely(txreq.size < ETH_HLEN)) {
-			netdev_dbg(vif->dev,
-				   "Bad packet size: %d\n", txreq.size);
+			if (net_ratelimit())
+				netdev_warn(vif->dev,
+					   "Bad packet size: %d\n", txreq.size);
 			xenvif_tx_err(vif, &txreq, idx);
 			break;
 		}
@@ -1448,8 +1491,9 @@ static unsigned xenvif_tx_build_gops(struct xenvif *vif, int budget)
 		skb = alloc_skb(data_len + NET_SKB_PAD + NET_IP_ALIGN,
 				GFP_ATOMIC | __GFP_NOWARN);
 		if (unlikely(skb == NULL)) {
-			netdev_dbg(vif->dev,
-				   "Can't allocate a skb in start_xmit.\n");
+			if (net_ratelimit())
+				netdev_warn(vif->dev,
+					   "Can't allocate a skb in start_xmit.\n");
 			xenvif_tx_err(vif, &txreq, idx);
 			break;
 		}
@@ -1544,7 +1588,8 @@ static int xenvif_tx_submit(struct xenvif *vif)
 
 		/* Check the remap error code. */
 		if (unlikely(xenvif_tx_check_gop(vif, skb, &gop))) {
-			netdev_dbg(vif->dev, "netback grant failed.\n");
+			if (net_ratelimit())
+				netdev_warn(vif->dev, "netback grant failed.\n");
 			skb_shinfo(skb)->nr_frags = 0;
 			kfree_skb(skb);
 			continue;
@@ -1581,8 +1626,9 @@ static int xenvif_tx_submit(struct xenvif *vif)
 		skb_reset_network_header(skb);
 
 		if (checksum_setup(vif, skb)) {
-			netdev_dbg(vif->dev,
-				   "Can't setup checksum in net_tx_action\n");
+			if (net_ratelimit())
+				netdev_warn(vif->dev,
+					   "Can't setup checksum in net_tx_action\n");
 			kfree_skb(skb);
 			continue;
 		}

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Mon, Mar 17, 2014 at 11:33:18PM +0100, Sander Eikelenboom wrote:
> 
> Monday, March 17, 2014, 11:35:24 AM, you wrote:
> 
> > On Wed, Mar 12, 2014 at 05:47:29PM +0100, Sander Eikelenboom wrote:
> >> 
> >> Wednesday, March 12, 2014, 4:45:01 PM, you wrote:
> >> 
> >> > On Wed, Mar 12, 2014 at 04:20:03PM +0100, Sander Eikelenboom wrote:
> >> > [...]
> >> >> 
> >> >> > Sorry, remove the trailing "S". Actually you only need to look at netback.c.
> >> >> 
> >> >> What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
> >> >> npo->meta_prod
> >> >> vif->rx.sring->req_prod
> >> >> vif->pending_prod
> >> >> 
> >> >> to name a few ..
> >> >> Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?
> >> >> 
> >> 
> >> > There are two macros you can use
> >> 
> >> > RING_REQUEST_CONS_OVERFLOW and RING_REQUEST_PROD_OVERFLOW.
> >> 
> >> Ah i already produced my own .. diff to netback is attached ..
> >> 
> >> Netback:
> >> Mar 12 17:41:26 serveerstertje kernel: [  464.778614] vif vif-7-0 vif7.0: ?!? npo->meta_prod:37 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431007
> >> Mar 12 17:41:26 serveerstertje kernel: [  464.786203] vif vif-7-0 vif7.0: ?!? npo->meta_prod:38 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431008
> 
> > req_prod < req_cons, so there's an overflow. I'm actually curious how
> > this could happen.
> 
> > Back to the code, before netback enqueues SKB to its internal queue, it
> > will check if there's enough room in the ring. Before Paul's changeset,
> > it checks against a static number (the possible maximum slots that can
> > be consumed by an SKB). Paul's changeset made it check against the
> > actual slots the incoming SKB consumes. See
> > interface.c:xenvif_start_xmit.  Another interesting site would be when
> > the SKB is broken down later on in internal queue. See
> > netback.c:xenvif_rx_action. The routine to break down an SKB is
> > xenvif_gop_skb.
> 
> > Although they look alright to me, but you might want to instrument them
> > a bit more to see what triggers that overflow. It's a bit frustrating,
> > but a bug that cannot be easily reproduced is indeed extremely hard to
> > fix.
> 
> [  554.166914] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:7 vif->rx.sring->req_prod:301750 vif->rx.req_cons:301751
> [  571.143788] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823
> [  571.150762] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 req->gref:19791875 req->id:22
> [  571.164691] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 npo->copy_gref:19791875
> [  571.179474] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 gso_type:1 gso_size:1448 nr_frags:1 req->gref:576 req->id:14
> [  571.194844] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0
> [  571.211215] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325379 req->id:23
> [  571.228471] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 npo->meta_prod:39 old_meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325379 req->id:23
> [  571.246412] vif vif-7-0 vif7.0: Bad status -3 from copy to DOM7.
> [  571.255531] vif vif-7-0 vif7.0: ?!? xenvif_check_gop status err? status:-1 i:7 nr_meta_slots:8 copy_op->status:-3 npo->copy_cons:38
> [  571.264804] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1 ip_summed:1 is_gso:1 skb_shinfo(skb)->gso_type:1634 vif->meta[npo.meta_cons].gso_type:3 nr_frags:1 npo.copy_prod:39, npo.meta_cons:30
> 
> It seems in 'get_next_rx_buffer' you can get the situation that cons > prod, but the first time somehow the code after the actual RING_GET_REQUEST isn't ran (why?!? queue stop ?) .. and it doesn't lead to an immediate problem ..

You've rate limited the output. get_next_rx_buffer won't sleep
whatsoever. Stopping the queue won't affect a running RX kthread. You
can think of the start_xmit and kthread run in parallel.

get_next_rx_buffer doens't check if the ring is overflowed. It's the
caller's fault. It might be worth checking start_new_rx_buffer

> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
> 
> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
> 
> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222

I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
req_cons < needed, the queue is stopeed.

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 18, 2014, 1:04:01 PM, you wrote:

> On Mon, Mar 17, 2014 at 11:33:18PM +0100, Sander Eikelenboom wrote:
>> 
>> Monday, March 17, 2014, 11:35:24 AM, you wrote:
>> 
>> > On Wed, Mar 12, 2014 at 05:47:29PM +0100, Sander Eikelenboom wrote:
>> >> 
>> >> Wednesday, March 12, 2014, 4:45:01 PM, you wrote:
>> >> 
>> >> > On Wed, Mar 12, 2014 at 04:20:03PM +0100, Sander Eikelenboom wrote:
>> >> > [...]
>> >> >> 
>> >> >> > Sorry, remove the trailing "S". Actually you only need to look at netback.c.
>> >> >> 
>> >> >> What producer index to compare with .. there are quite some RING_GET_REQUESTS .. and i see:
>> >> >> npo->meta_prod
>> >> >> vif->rx.sring->req_prod
>> >> >> vif->pending_prod
>> >> >> 
>> >> >> to name a few ..
>> >> >> Any particular RING_GET_REQUESTS call and particular producer index you are interested in ?
>> >> >> 
>> >> 
>> >> > There are two macros you can use
>> >> 
>> >> > RING_REQUEST_CONS_OVERFLOW and RING_REQUEST_PROD_OVERFLOW.
>> >> 
>> >> Ah i already produced my own .. diff to netback is attached ..
>> >> 
>> >> Netback:
>> >> Mar 12 17:41:26 serveerstertje kernel: [  464.778614] vif vif-7-0 vif7.0: ?!? npo->meta_prod:37 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431007
>> >> Mar 12 17:41:26 serveerstertje kernel: [  464.786203] vif vif-7-0 vif7.0: ?!? npo->meta_prod:38 vif->rx.sring->req_prod:431006 vif->rx.req_cons:431008
>> 
>> > req_prod < req_cons, so there's an overflow. I'm actually curious how
>> > this could happen.
>> 
>> > Back to the code, before netback enqueues SKB to its internal queue, it
>> > will check if there's enough room in the ring. Before Paul's changeset,
>> > it checks against a static number (the possible maximum slots that can
>> > be consumed by an SKB). Paul's changeset made it check against the
>> > actual slots the incoming SKB consumes. See
>> > interface.c:xenvif_start_xmit.  Another interesting site would be when
>> > the SKB is broken down later on in internal queue. See
>> > netback.c:xenvif_rx_action. The routine to break down an SKB is
>> > xenvif_gop_skb.
>> 
>> > Although they look alright to me, but you might want to instrument them
>> > a bit more to see what triggers that overflow. It's a bit frustrating,
>> > but a bug that cannot be easily reproduced is indeed extremely hard to
>> > fix.
>> 
>> [  554.166914] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:7 vif->rx.sring->req_prod:301750 vif->rx.req_cons:301751
>> [  571.143788] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823
>> [  571.150762] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 req->gref:19791875 req->id:22
>> [  571.164691] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 npo->copy_gref:19791875
>> [  571.179474] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364823 gso_type:1 gso_size:1448 nr_frags:1 req->gref:576 req->id:14
>> [  571.194844] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0
>> [  571.211215] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325379 req->id:23
>> [  571.228471] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 npo->meta_prod:39 old_meta_prod:38 vif->rx.sring->req_prod:364822 vif->rx.req_cons:364824 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325379 req->id:23
>> [  571.246412] vif vif-7-0 vif7.0: Bad status -3 from copy to DOM7.
>> [  571.255531] vif vif-7-0 vif7.0: ?!? xenvif_check_gop status err? status:-1 i:7 nr_meta_slots:8 copy_op->status:-3 npo->copy_cons:38
>> [  571.264804] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1 ip_summed:1 is_gso:1 skb_shinfo(skb)->gso_type:1634 vif->meta[npo.meta_cons].gso_type:3 nr_frags:1 npo.copy_prod:39, npo.meta_cons:30
>> 
>> It seems in 'get_next_rx_buffer' you can get the situation that cons > prod, but the first time somehow the code after the actual RING_GET_REQUEST isn't ran (why?!? queue stop ?) .. and it doesn't lead to an immediate problem ..

> You've rate limited the output. get_next_rx_buffer won't sleep
> whatsoever. Stopping the queue won't affect a running RX kthread. You
> can think of the start_xmit and kthread run in parallel.

> get_next_rx_buffer doens't check if the ring is overflowed. It's the
> caller's fault. It might be worth checking start_new_rx_buffer

Added even more warns ...

[  297.885969] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:21764 vif->rx.req_cons:21762
[  298.760555] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:22488 vif->rx.req_cons:22486

[  306.376176] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
[  306.376556] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
[  306.391863] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 req->gref:4325377 req->id:153

[  306.407599] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:640 size:640 i:4
[  306.423913] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377 npo->copy_off:640  MAX_BUFFER_OFFSET:4096 bytes:640 size:0 i:5


[  306.440941] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:31 old_meta_prod:25 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 gso_type:1 gso_size:1448 nr_frags:1 req->gref:638 req->id:147
[  306.458334] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0
[  306.476097] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154
[  306.494462] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 before npo->meta_prod:32 old_meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154 j:0
[  306.513424] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here start   npo->meta_prod:32 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 npo->copy_gref:4325377 npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:0 size:66 i:0
[  311.390883] net_ratelimit: 317 callbacks suppressed
[  311.400901] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:32386 vif->rx.req_cons:32322

- So in this case we are in the 3rd iteration of the loop in xenvif_gop_frag_copy ...
- Xenvif_start_xmit stop the queue since it has detected it needs one more slot which is unavailable at that time.
- The current rx thread however doesn't know and doesn't check  (neither in the loop in xenvif_gop_frag_copy nor in get_next_rx_buffer that the ring if full) .. while prod == cons now .. consumes another one ..
- That ring request leads to the bad grant references reported by the hypervisor

(XEN) [2014-03-18 15:02:58.928] grant_table.c:1857:d0v2 Bad grant reference 4325377

So should there be a check added there ... or should the callers  "xenvif_gop_frag_copy" and the caller of that one "xenvif_gop_skb" already have anticipated that what the were about
to do wasn't going to fit anyway ?

And of course .. how made Paul's change trigger this ?


>> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
>> 
>> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
>> 
>> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222

> I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
> req_cons < needed, the queue is stopeed.

> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Tue, Mar 18, 2014 at 04:21:27PM +0100, Sander Eikelenboom wrote:
[...]
> 
> Added even more warns ...
> 
> [  297.885969] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:21764 vif->rx.req_cons:21762
> [  298.760555] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:22488 vif->rx.req_cons:22486
> 
> [  306.376176] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
> [  306.376556] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
> [  306.391863] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 req->gref:4325377 req->id:153
> 
> [  306.407599] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:640 size:640 i:4
> [  306.423913] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377 npo->copy_off:640  MAX_BUFFER_OFFSET:4096 bytes:640 size:0 i:5
> 
> 
> [  306.440941] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:31 old_meta_prod:25 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 gso_type:1 gso_size:1448 nr_frags:1 req->gref:638 req->id:147
> [  306.458334] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0
> [  306.476097] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154
> [  306.494462] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 before npo->meta_prod:32 old_meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154 j:0
> [  306.513424] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here start   npo->meta_prod:32 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 npo->copy_gref:4325377 npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:0 size:66 i:0
> [  311.390883] net_ratelimit: 317 callbacks suppressed
> [  311.400901] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:32386 vif->rx.req_cons:32322
> 
> - So in this case we are in the 3rd iteration of the loop in xenvif_gop_frag_copy ...
> - Xenvif_start_xmit stop the queue since it has detected it needs one more slot which is unavailable at that time.

Yes.

> - The current rx thread however doesn't know and doesn't check  (neither in the loop in xenvif_gop_frag_copy nor in get_next_rx_buffer that the ring if full) .. while prod == cons now .. consumes another one ..

It does check -- but not in xenvif_gop_frag_copy -- see
xenvif_rx_action, which calls xenvif_rx_ring_slots_available before
queueing skb to break down. That is, when you call xenvif_gop_skb there
should be enough room to accommodate that SKB.

> - That ring request leads to the bad grant references reported by the hypervisor
> 
> (XEN) [2014-03-18 15:02:58.928] grant_table.c:1857:d0v2 Bad grant reference 4325377
> 
> So should there be a check added there ... or should the callers  "xenvif_gop_frag_copy" and the caller of that one "xenvif_gop_skb" already have anticipated that what the were about
> to do wasn't going to fit anyway ?
> 

No, see above.

> And of course .. how made Paul's change trigger this ?
> 

Before Paul's change, we always reserve very large room for an incoming
SKB. After Paul's change, we only reserve just enough room. Probably
some extra room prevents this bug being triggered.

Wei.

> 
> >> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
> >> 
> >> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
> >> 
> >> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222
> 
> > I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
> > req_cons < needed, the queue is stopeed.
> 
> > Wei.
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 18, 2014, 5:04:12 PM, you wrote:

> On Tue, Mar 18, 2014 at 04:21:27PM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> Added even more warns ...
>> 
>> [  297.885969] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:21764 vif->rx.req_cons:21762
>> [  298.760555] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:22488 vif->rx.req_cons:22486
>> 
>> [  306.376176] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
>> [  306.376556] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
>> [  306.391863] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 req->gref:4325377 req->id:153
>> 
>> [  306.407599] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:640 size:640 i:4
>> [  306.423913] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377 npo->copy_off:640  MAX_BUFFER_OFFSET:4096 bytes:640 size:0 i:5
>> 
>> 
>> [  306.440941] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:31 old_meta_prod:25 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 gso_type:1 gso_size:1448 nr_frags:1 req->gref:638 req->id:147
>> [  306.458334] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0
>> [  306.476097] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154
>> [  306.494462] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 before npo->meta_prod:32 old_meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154 j:0
>> [  306.513424] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here start   npo->meta_prod:32 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 npo->copy_gref:4325377 npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:0 size:66 i:0
>> [  311.390883] net_ratelimit: 317 callbacks suppressed
>> [  311.400901] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:32386 vif->rx.req_cons:32322
>> 
>> - So in this case we are in the 3rd iteration of the loop in xenvif_gop_frag_copy ...
>> - Xenvif_start_xmit stop the queue since it has detected it needs one more slot which is unavailable at that time.

> Yes.

>> - The current rx thread however doesn't know and doesn't check  (neither in the loop in xenvif_gop_frag_copy nor in get_next_rx_buffer that the ring if full) .. while prod == cons now .. consumes another one ..

> It does check -- but not in xenvif_gop_frag_copy -- see
> xenvif_rx_action, which calls xenvif_rx_ring_slots_available before
> queueing skb to break down. That is, when you call xenvif_gop_skb there
> should be enough room to accommodate that SKB.

>> - That ring request leads to the bad grant references reported by the hypervisor
>> 
>> (XEN) [2014-03-18 15:02:58.928] grant_table.c:1857:d0v2 Bad grant reference 4325377
>> 
>> So should there be a check added there ... or should the callers  "xenvif_gop_frag_copy" and the caller of that one "xenvif_gop_skb" already have anticipated that what the were about
>> to do wasn't going to fit anyway ?
>> 

> No, see above.

>> And of course .. how made Paul's change trigger this ?
>> 

> Before Paul's change, we always reserve very large room for an incoming
> SKB. After Paul's change, we only reserve just enough room. Probably
> some extra room prevents this bug being triggered.

[  599.970745] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:504174
[  599.972487] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:506388
[  600.044322] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 req->gref:165543936 req->id:19 vif->rx.sring->req_event:506388
[  600.081167] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:6 vif->rx.sring->req_event:506388 estimated_slots_needed:8
[  600.118268] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:7 vif->rx.sring->req_event:506388 estimated_slots_needed:8
[  600.155378] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8 i(frag): 0
[  600.192438] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8
[  600.229395] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 sco->meta_slots_used:8 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:8 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:6 min_slots_needed:3
[  600.266518] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 1 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388                         max_upped_gso:1 skb_is_gso(skb):0 max_slots_needed:1 j:4 is_gso:0 nr_frags:0 firstpart:1 secondpart:0 min_slots_needed:1

It seems to estimate 8 slots and need 8 slots ... however .. shouldn't the queue have been stopped before getting here ..


> Wei.

>> 
>> >> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
>> >> 
>> >> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
>> >> 
>> >> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222
>> 
>> > I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
>> > req_cons < needed, the queue is stopeed.

So it seems .. most of the time .. but if i look at the calculation of "min_slots_needed" in this function it seems completely different from the one in
xenvif_rx_action for max_slots_needed .. though both seem to be used for the same thing .. to calcultate how many slots the brokendown SKB would need to fit in ..
So i added the calculation method from xenvif_start_xmit to xenvif_rx_action .. in the error case you see min_slots_needed was 3 .. but max_slots_needed and max_slots_used both were 8.

The main difference between these calculation methods seems to be that min_slots_needed doesn't take the PAGE_SIZE into account to see how many slots are needed for the frags.

So Paul .. why was the "xenvif_count_skb_slots(vif, skb)" function dropped and replaced by two seperate and different calculations ?

--
Sander

>> 
>> > Wei.
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 18, 2014, 9:14:02 PM, you wrote:


> Tuesday, March 18, 2014, 5:04:12 PM, you wrote:

>> On Tue, Mar 18, 2014 at 04:21:27PM +0100, Sander Eikelenboom wrote:
>> [...]
>>> 
>>> Added even more warns ...
>>> 
>>> [  297.885969] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:21764 vif->rx.req_cons:21762
>>> [  298.760555] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:22488 vif->rx.req_cons:22486
>>> 
>>> [  306.376176] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
>>> [  306.376556] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
>>> [  306.391863] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 req->gref:4325377 req->id:153
>>> 
>>> [  306.407599] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:640 size:640 i:4
>>> [  306.423913] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377 npo->copy_off:640  MAX_BUFFER_OFFSET:4096 bytes:640 size:0 i:5
>>> 
>>> 
>>> [  306.440941] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:31 old_meta_prod:25 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 gso_type:1 gso_size:1448 nr_frags:1 req->gref:638 req->id:147
>>> [  306.458334] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0
>>> [  306.476097] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154
>>> [  306.494462] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 before npo->meta_prod:32 old_meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154 j:0
>>> [  306.513424] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here start   npo->meta_prod:32 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 npo->copy_gref:4325377 npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:0 size:66 i:0
>>> [  311.390883] net_ratelimit: 317 callbacks suppressed
>>> [  311.400901] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:32386 vif->rx.req_cons:32322
>>> 
>>> - So in this case we are in the 3rd iteration of the loop in xenvif_gop_frag_copy ...
>>> - Xenvif_start_xmit stop the queue since it has detected it needs one more slot which is unavailable at that time.

>> Yes.

>>> - The current rx thread however doesn't know and doesn't check  (neither in the loop in xenvif_gop_frag_copy nor in get_next_rx_buffer that the ring if full) .. while prod == cons now .. consumes another one ..

>> It does check -- but not in xenvif_gop_frag_copy -- see
>> xenvif_rx_action, which calls xenvif_rx_ring_slots_available before
>> queueing skb to break down. That is, when you call xenvif_gop_skb there
>> should be enough room to accommodate that SKB.

>>> - That ring request leads to the bad grant references reported by the hypervisor
>>> 
>>> (XEN) [2014-03-18 15:02:58.928] grant_table.c:1857:d0v2 Bad grant reference 4325377
>>> 
>>> So should there be a check added there ... or should the callers  "xenvif_gop_frag_copy" and the caller of that one "xenvif_gop_skb" already have anticipated that what the were about
>>> to do wasn't going to fit anyway ?
>>> 

>> No, see above.

>>> And of course .. how made Paul's change trigger this ?
>>> 

>> Before Paul's change, we always reserve very large room for an incoming
>> SKB. After Paul's change, we only reserve just enough room. Probably
>> some extra room prevents this bug being triggered.

> [  599.970745] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:504174
> [  599.972487] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:506388
> [  600.044322] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 req->gref:165543936 req->id:19 vif->rx.sring->req_event:506388
> [  600.081167] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:6 vif->rx.sring->req_event:506388 estimated_slots_needed:8
> [  600.118268] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:7 vif->rx.sring->req_event:506388 estimated_slots_needed:8
> [  600.155378] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8 i(frag): 0
> [  600.192438] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8
> [  600.229395] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 sco->meta_slots_used:8 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:8 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:6 min_slots_needed:3
> [  600.266518] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 1 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388                         max_upped_gso:1 skb_is_gso(skb):0 max_slots_needed:1 j:4 is_gso:0 nr_frags:0 firstpart:1 secondpart:0 min_slots_needed:1

> It seems to estimate 8 slots and need 8 slots ... however .. shouldn't the queue have been stopped before getting here ..

*hrmm please just ignore* .. got to get some sleep i guess ..


>> Wei.

>>> 
>>> >> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
>>> >> 
>>> >> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
>>> >> 
>>> >> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222
>>> 
>>> > I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
>>> > req_cons < needed, the queue is stopeed.

> So it seems .. most of the time .. but if i look at the calculation of "min_slots_needed" in this function it seems completely different from the one in
> xenvif_rx_action for max_slots_needed .. though both seem to be used for the same thing .. to calcultate how many slots the brokendown SKB would need to fit in ..
> So i added the calculation method from xenvif_start_xmit to xenvif_rx_action .. in the error case you see min_slots_needed was 3 .. but max_slots_needed and max_slots_used both were 8.

> The main difference between these calculation methods seems to be that min_slots_needed doesn't take the PAGE_SIZE into account to see how many slots are needed for the frags.

> So Paul .. why was the "xenvif_count_skb_slots(vif, skb)" function dropped and replaced by two seperate and different calculations ?

> --
> Sander

>>> 
>>> > Wei.
>>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Hello Sander,

Tuesday, March 18, 2014, 10:18:59 PM, you wrote:




> Tuesday, March 18, 2014, 9:14:02 PM, you wrote:


>> Tuesday, March 18, 2014, 5:04:12 PM, you wrote:

>>> On Tue, Mar 18, 2014 at 04:21:27PM +0100, Sander Eikelenboom wrote:
>>> [...]
>>>> 
>>>> Added even more warns ...
>>>> 
>>>> [  297.885969] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:21764 vif->rx.req_cons:21762
>>>> [  298.760555] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:22488 vif->rx.req_cons:22486
>>>> 
>>>> [  306.376176] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
>>>> [  306.376556] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28313
>>>> [  306.391863] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:30 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 req->gref:4325377 req->id:153
>>>> 
>>>> [  306.407599] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:640 size:640 i:4
>>>> [  306.423913] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 npo->copy_gref:4325377 npo->copy_off:640  MAX_BUFFER_OFFSET:4096 bytes:640 size:0 i:5
>>>> 
>>>> 
>>>> [  306.440941] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:31 old_meta_prod:25 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28314 gso_type:1 gso_size:1448 nr_frags:1 req->gref:638 req->id:147
>>>> [  306.458334] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 before req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0
>>>> [  306.476097] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 2 after req npo->meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154
>>>> [  306.494462] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 3 before npo->meta_prod:32 old_meta_prod:31 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 gso_type:0 gso_size:0 nr_frags:0 req->gref:4325377 req->id:154 j:0
>>>> [  306.513424] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here start   npo->meta_prod:32 vif->rx.sring->req_prod:28313 vif->rx.req_cons:28315 npo->copy_gref:4325377 npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:0 size:66 i:0
>>>> [  311.390883] net_ratelimit: 317 callbacks suppressed
>>>> [  311.400901] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:3 vif->rx.sring->req_prod:32386 vif->rx.req_cons:32322
>>>> 
>>>> - So in this case we are in the 3rd iteration of the loop in xenvif_gop_frag_copy ...
>>>> - Xenvif_start_xmit stop the queue since it has detected it needs one more slot which is unavailable at that time.

>>> Yes.

>>>> - The current rx thread however doesn't know and doesn't check  (neither in the loop in xenvif_gop_frag_copy nor in get_next_rx_buffer that the ring if full) .. while prod == cons now .. consumes another one ..

>>> It does check -- but not in xenvif_gop_frag_copy -- see
>>> xenvif_rx_action, which calls xenvif_rx_ring_slots_available before
>>> queueing skb to break down. That is, when you call xenvif_gop_skb there
>>> should be enough room to accommodate that SKB.

>>>> - That ring request leads to the bad grant references reported by the hypervisor
>>>> 
>>>> (XEN) [2014-03-18 15:02:58.928] grant_table.c:1857:d0v2 Bad grant reference 4325377
>>>> 
>>>> So should there be a check added there ... or should the callers  "xenvif_gop_frag_copy" and the caller of that one "xenvif_gop_skb" already have anticipated that what the were about
>>>> to do wasn't going to fit anyway ?
>>>> 

>>> No, see above.

>>>> And of course .. how made Paul's change trigger this ?
>>>> 

>>> Before Paul's change, we always reserve very large room for an incoming
>>> SKB. After Paul's change, we only reserve just enough room. Probably
>>> some extra room prevents this bug being triggered.

>> [  599.970745] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:504174
>> [  599.972487] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:506388
>> [  600.044322] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 req->gref:165543936 req->id:19 vif->rx.sring->req_event:506388
>> [  600.081167] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:6 vif->rx.sring->req_event:506388 estimated_slots_needed:8
>> [  600.118268] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:7 vif->rx.sring->req_event:506388 estimated_slots_needed:8
>> [  600.155378] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8 i(frag): 0
>> [  600.192438] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8
>> [  600.229395] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 sco->meta_slots_used:8 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:8 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:6 min_slots_needed:3
>> [  600.266518] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 1 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388                         max_upped_gso:1 skb_is_gso(skb):0 max_slots_needed:1 j:4 is_gso:0 nr_frags:0 firstpart:1 secondpart:0 min_slots_needed:1

>> It seems to estimate 8 slots and need 8 slots ... however .. shouldn't the queue have been stopped before getting here ..

> *hrmm please just ignore* .. got to get some sleep i guess ..

Just went the empirical way .. and unconditionally upped the calculated "max_slots_needed" by one in "xenvif_rx_action"  just before checking the "xenvif_rx_ring_slots_available",
this has prevented all non-fatal and fatal occurrences of "cons > prod" so far. I will leave my tests running overnight, see if it survives the pounding.

>From other net drivers i see different calculations .. seems it is kind of voodoo to determine the value .. one of which (drivers/net/ethernet/marvell/sky2.c)
seems to unconditionally reserve a slot for both GSO and CKSUM. Don't know if that makes any sense though:

/* This is the worst case number of transmit list elements for a single skb:
   VLAN:GSO + CKSUM + Data + skb_frags * DMA */
#define MAX_SKB_TX_LE   (2 + (sizeof(dma_addr_t)/sizeof(u32))*(MAX_SKB_FRAGS+1))



>>> Wei.

>>>> 
>>>> >> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
>>>> >> 
>>>> >> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
>>>> >> 
>>>> >> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222
>>>> 
>>>> > I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
>>>> > req_cons < needed, the queue is stopeed.

>> So it seems .. most of the time .. but if i look at the calculation of "min_slots_needed" in this function it seems completely different from the one in
>> xenvif_rx_action for max_slots_needed .. though both seem to be used for the same thing .. to calcultate how many slots the brokendown SKB would need to fit in ..
>> So i added the calculation method from xenvif_start_xmit to xenvif_rx_action .. in the error case you see min_slots_needed was 3 .. but max_slots_needed and max_slots_used both were 8.

>> The main difference between these calculation methods seems to be that min_slots_needed doesn't take the PAGE_SIZE into account to see how many slots are needed for the frags.

>> So Paul .. why was the "xenvif_count_skb_slots(vif, skb)" function dropped and replaced by two seperate and different calculations ?

>> --
>> Sander

>>>> 
>>>> > Wei.
>>>> 







-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Wed, Mar 19, 2014 at 12:11:04AM +0100, Sander Eikelenboom wrote:
[...]
> 
> >>> Before Paul's change, we always reserve very large room for an incoming
> >>> SKB. After Paul's change, we only reserve just enough room. Probably
> >>> some extra room prevents this bug being triggered.
> 
> >> [  599.970745] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:504174
> >> [  599.972487] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:506388
> >> [  600.044322] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 req->gref:165543936 req->id:19 vif->rx.sring->req_event:506388
> >> [  600.081167] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:6 vif->rx.sring->req_event:506388 estimated_slots_needed:8
> >> [  600.118268] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:7 vif->rx.sring->req_event:506388 estimated_slots_needed:8
> >> [  600.155378] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8 i(frag): 0
> >> [  600.192438] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8
> >> [  600.229395] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 sco->meta_slots_used:8 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:8 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:6 min_slots_needed:3
> >> [  600.266518] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 1 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388                         max_upped_gso:1 skb_is_gso(skb):0 max_slots_needed:1 j:4 is_gso:0 nr_frags:0 firstpart:1 secondpart:0 min_slots_needed:1
> 
> >> It seems to estimate 8 slots and need 8 slots ... however .. shouldn't the queue have been stopped before getting here ..
> 
> > *hrmm please just ignore* .. got to get some sleep i guess ..
> 
> Just went the empirical way .. and unconditionally upped the calculated "max_slots_needed" by one in "xenvif_rx_action"  just before checking the "xenvif_rx_ring_slots_available",
> this has prevented all non-fatal and fatal occurrences of "cons > prod" so far. I will leave my tests running overnight, see if it survives the pounding.
> 
> >From other net drivers i see different calculations .. seems it is kind of voodoo to determine the value .. one of which (drivers/net/ethernet/marvell/sky2.c)
> seems to unconditionally reserve a slot for both GSO and CKSUM. Don't know if that makes any sense though:
> 
> /* This is the worst case number of transmit list elements for a single skb:
>    VLAN:GSO + CKSUM + Data + skb_frags * DMA */
> #define MAX_SKB_TX_LE   (2 + (sizeof(dma_addr_t)/sizeof(u32))*(MAX_SKB_FRAGS+1))
> 

Xen network "wire" format doesn't have a CKSUM metadata type though.

> 
> 
> >>> Wei.
> 
> >>>> 
> >>>> >> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
> >>>> >> 
> >>>> >> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
> >>>> >> 
> >>>> >> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222
> >>>> 
> >>>> > I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
> >>>> > req_cons < needed, the queue is stopeed.
> 
> >> So it seems .. most of the time .. but if i look at the calculation of "min_slots_needed" in this function it seems completely different from the one in
> >> xenvif_rx_action for max_slots_needed .. though both seem to be used for the same thing .. to calcultate how many slots the brokendown SKB would need to fit in ..
> >> So i added the calculation method from xenvif_start_xmit to xenvif_rx_action .. in the error case you see min_slots_needed was 3 .. but max_slots_needed and max_slots_used both were 8.
> 

Those are different estimations. max_slots_needed estimates the worst
case while min_slots_needed estimates the best case. When
min_slots_needed is met, netback puts the SKB into its internal queue.
xenvif_rx_action then will dequeue that packet, check max_slots_needed,
if met, break SKB down.

What I would expect is, even if they yield different values, checking
the ring availability is enough before breaking SKB down.

In your above case, max_slots_needed was met and SKB broken down. And as
you said in your empirical test, bumping max_slots_needed by 1 prevented
issues, then we might have problem calculating max_slots_needed. If you
can work out what went wrong that can be very useful.

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 19, 2014, 12:35:32 PM, you wrote:

> On Wed, Mar 19, 2014 at 12:11:04AM +0100, Sander Eikelenboom wrote:
> [...]
>> 
>> >>> Before Paul's change, we always reserve very large room for an incoming
>> >>> SKB. After Paul's change, we only reserve just enough room. Probably
>> >>> some extra room prevents this bug being triggered.
>> 
>> >> [  599.970745] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:504174
>> >> [  599.972487] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506387 vif->rx.sring->req_event:506388
>> >> [  600.044322] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:37 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 req->gref:165543936 req->id:19 vif->rx.sring->req_event:506388
>> >> [  600.081167] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:6 vif->rx.sring->req_event:506388 estimated_slots_needed:8
>> >> [  600.118268] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:38 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 npo->copy_gref:165543936 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:7 vif->rx.sring->req_event:506388 estimated_slots_needed:8
>> >> [  600.155378] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8 i(frag): 0
>> >> [  600.192438] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:38 old_meta_prod:30 vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 gso_type:1 gso_size:1448 nr_frags:1 req->gref:570 req->id:11 estimated_slots_needed:8
>> >> [  600.229395] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388 sco->meta_slots_used:8 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:8 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:6 min_slots_needed:3
>> >> [  600.266518] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 1 ..  vif->rx.sring->req_prod:506387 vif->rx.req_cons:506388                         max_upped_gso:1 skb_is_gso(skb):0 max_slots_needed:1 j:4 is_gso:0 nr_frags:0 firstpart:1 secondpart:0 min_slots_needed:1
>> 
>> >> It seems to estimate 8 slots and need 8 slots ... however .. shouldn't the queue have been stopped before getting here ..
>> 
>> > *hrmm please just ignore* .. got to get some sleep i guess ..
>> 
>> Just went the empirical way .. and unconditionally upped the calculated "max_slots_needed" by one in "xenvif_rx_action"  just before checking the "xenvif_rx_ring_slots_available",
>> this has prevented all non-fatal and fatal occurrences of "cons > prod" so far. I will leave my tests running overnight, see if it survives the pounding.
>> 
>> >From other net drivers i see different calculations .. seems it is kind of voodoo to determine the value .. one of which (drivers/net/ethernet/marvell/sky2.c)
>> seems to unconditionally reserve a slot for both GSO and CKSUM. Don't know if that makes any sense though:
>> 
>> /* This is the worst case number of transmit list elements for a single skb:
>>    VLAN:GSO + CKSUM + Data + skb_frags * DMA */
>> #define MAX_SKB_TX_LE   (2 + (sizeof(dma_addr_t)/sizeof(u32))*(MAX_SKB_FRAGS+1))
>> 

> Xen network "wire" format doesn't have a CKSUM metadata type though.

>> 
>> 
>> >>> Wei.
>> 
>> >>>> 
>> >>>> >> The second time it does get to the code after the RING_GET_REQUEST in 'get_next_rx_buffer' and that leads to mayhem ...
>> >>>> >> 
>> >>>> >> So added a netdev_warn to xenvif_start_xmit for the "stop queue" case .. unfortunately it now triggers net_ratelimit at the end:
>> >>>> >> 
>> >>>> >> [  402.909693] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:7 vif->rx.sring->req_prod:189228 vif->rx.req_cons:189222
>> >>>> 
>> >>>> > I think xenvif_rx_ring_slots_available is doing its job. If req_prod -
>> >>>> > req_cons < needed, the queue is stopeed.
>> 
>> >> So it seems .. most of the time .. but if i look at the calculation of "min_slots_needed" in this function it seems completely different from the one in
>> >> xenvif_rx_action for max_slots_needed .. though both seem to be used for the same thing .. to calcultate how many slots the brokendown SKB would need to fit in ..
>> >> So i added the calculation method from xenvif_start_xmit to xenvif_rx_action .. in the error case you see min_slots_needed was 3 .. but max_slots_needed and max_slots_used both were 8.
>> 

> Those are different estimations. max_slots_needed estimates the worst
> case while min_slots_needed estimates the best case. When
> min_slots_needed is met, netback puts the SKB into its internal queue.
> xenvif_rx_action then will dequeue that packet, check max_slots_needed,
> if met, break SKB down.

I realised that, though if you don't count the cost of calculation .. the best would be
to have min_slots and max_slots the same .. and being correct. But cost of the calculation count ..
on the other hand min_slots seems to be far too low in this case. Is the nr_frags the best thing to check for ?
Wouldn't something like skbsize / PAGE_SIZE be a better estimate ?


> What I would expect is, even if they yield different values, checking
> the ring availability is enough before breaking SKB down.

> In your above case, max_slots_needed was met and SKB broken down. And as
> you said in your empirical test, bumping max_slots_needed by 1 prevented
> issues, then we might have problem calculating max_slots_needed. If you
> can work out what went wrong that can be very useful.

OK ran the whole night with "max_slots_needed+1" and i saw no triggering of both:
    - the non-fatal oddity (only a temporary cons > prod ..)
    - the fatal case (leading to the grantref error)

So it seems we are off by one .. somewhere


At first the non-fatal oddity .. this one i really can't grasp:

[  567.880282] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:562972 vif->rx.req_cons:562969 vif->rx.sring->req_event:562973
[  569.893801] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:571188 vif->rx.req_cons:571187 vif->rx.sring->req_event:571189
[  571.892419] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:577141 vif->rx.req_cons:577138 vif->rx.sring->req_event:577142
[  575.451383] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:6 vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 vif->rx.sring->req_event:589615, reserved_slots_left:0
[  575.487721] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:6 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590604 req->gref:569 req->id:11 vif->rx.sring->req_event:589615 reserved_slots_left:-1
[  575.524427] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590660 vif->rx.sring->req_event:590661
[  576.508063] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:16 vif->rx.sring->req_prod:594343 vif->rx.req_cons:594343 vif->rx.sring->req_event:591619, reserved_slots_left:0
[  576.544708] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:16 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594344 req->gref:668 req->id:167 vif->rx.sring->req_event:591619 reserved_slots_left:-1
[  576.581595] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594389 vif->rx.sring->req_event:594390
[  577.325826] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:597248 vif->rx.req_cons:597247 vif->rx.sring->req_event:597249
[  577.576973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:598146 vif->rx.req_cons:598143 vif->rx.sring->req_event:598147

What i don't get is:
- i do warn just before the RING_GET_REQUEST on vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 .. so cons == prod
- i warn because  req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++); is going to add one to cons .. leading to a situation that should not exist ..
- however in the warn just after this RING_GET_REQUEST .. prod is somehow increased a lot .. probably the "next_rx_buffer" aspect of things .. so cons is much smaller than prod again ..
- How everi don't see anything in the RING_GET_REQUEST macro fiddling with vif->rx.sring->req_prod ..
- so who has been fiddling with vif->rx.sring->req_prod in the mean time .. my very limited c skills fail to see how this works :-)


Then the fatal case ...

[ 1108.566486] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1843278 vif->rx.req_cons:1843278 vif->rx.sring->req_event:1843279
[ 1109.775757] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1847779 vif->rx.req_cons:1847776 vif->rx.sring->req_event:1847780
[ 1110.041661] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1848795 vif->rx.req_cons:1848794 vif->rx.sring->req_event:1848796
[ 1110.301778] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1849755 vif->rx.req_cons:1849754 vif->rx.sring->req_event:1849756
[ 1111.565412] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1853482, reserved_slots_left:0
[ 1111.565973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1854292
[ 1111.638425] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 req->gref:4325377 req->id:83 vif->rx.sring->req_event:1854292 reserved_slots_left:-1
[ 1111.675042] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:5 vif->rx.sring->req_event:1854292 estimated_slots_needed:7 reserved_slots_left:-1
[ 1111.711833] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:6 vif->rx.sring->req_event:1854292 gso_gaps:0 estimated_slots_needed:7 reserved_slots_left:-1
[ 1111.766836] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 i(frag):0 j(data):1 reserved_slots_left:-1
[ 1111.803702] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:6
[ 1111.858602] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 sco->meta_slots_used:7 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:7 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:5 reserved_slots_left:-1

Ok so if i try to pull this apart (from bottom to top):

- it starts out with "xenvif_rx_action" and calculated we would need:
      1 slot for the  DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE);
      5 slots for 1 frag
      1 slot for GSO
      So 7 slots in total .. 

- it checks  if there are enough slots available if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) and aparently there are .. so it continues

- We end up in "xenvif_gop_skb" processing .. and everything seems to go well ..
      - In the start of this function we do one unconditional RING_GET_REQUEST .. and potentially one conditional (if gso)
          - Question: in "xenvif_rx_action" we only reserved ONE slot for GSO stuff .. so this could cost TWO slots .. is that right ?
          - But in this case we only use one ... see "used in funcstart: 0 + 1"

      - Now we loop 1 time through the  "while (data < skb_tail_pointer(skb))" loop ... since j(data)=1
          - and call "xenvif_gop_frag_copy" from inside this loop here .. which potentially calls .. get_next_rx_buffer
          - If i understand it correctly the number of slots this loop should take at a maximum should be what we calculated before with the
            DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE) ? (1 slot in this case)
          - Nothing is logged yet because cons is still smaller than prod at this time, but from the aftermath we see this was "used_dataloop:1"
          - So that seems correct .. 

      - Now we are at the loop through "nr_frags"  and loop the 1 frag we have .. 
                  - This all goes well until the sixth iteration (i from 0..5) .. and call "xenvif_gop_frag_copy" from inside this loop ...
                  - "xenvif_gop_frag_copy" calls "get_next_rx_buffer" .. cons becomes larger than prod ..and the "miracle" from the non-fatal case of prod suddenly becoming larger does not happen

      - So the calculation of how many slots are required for the frags in "xenvif_rx_action" does not match wath the code in "xenvif_gop_frag_copy" seems to need,
        I don't see why .. it's rounding up .. should behave correct if the left over happens to be 0 size ..



> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

[...]
> > Those are different estimations. max_slots_needed estimates the worst
> > case while min_slots_needed estimates the best case. When
> > min_slots_needed is met, netback puts the SKB into its internal queue.
> > xenvif_rx_action then will dequeue that packet, check max_slots_needed,
> > if met, break SKB down.
> 
> I realised that, though if you don't count the cost of calculation .. the best would be
> to have min_slots and max_slots the same .. and being correct. But cost of the calculation count ..
> on the other hand min_slots seems to be far too low in this case. Is the nr_frags the best thing to check for ?
> Wouldn't something like skbsize / PAGE_SIZE be a better estimate ?
> 

It doesn't matter. It won't cause breakage at that point.

> 
> > What I would expect is, even if they yield different values, checking
> > the ring availability is enough before breaking SKB down.
> 
> > In your above case, max_slots_needed was met and SKB broken down. And as
> > you said in your empirical test, bumping max_slots_needed by 1 prevented
> > issues, then we might have problem calculating max_slots_needed. If you
> > can work out what went wrong that can be very useful.
> 
> OK ran the whole night with "max_slots_needed+1" and i saw no triggering of both:
>     - the non-fatal oddity (only a temporary cons > prod ..)
>     - the fatal case (leading to the grantref error)
> 
> So it seems we are off by one .. somewhere
> 
> 
> At first the non-fatal oddity .. this one i really can't grasp:
> 
> [  567.880282] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:562972 vif->rx.req_cons:562969 vif->rx.sring->req_event:562973
> [  569.893801] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:571188 vif->rx.req_cons:571187 vif->rx.sring->req_event:571189
> [  571.892419] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:577141 vif->rx.req_cons:577138 vif->rx.sring->req_event:577142
> [  575.451383] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:6 vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 vif->rx.sring->req_event:589615, reserved_slots_left:0
> [  575.487721] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:6 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590604 req->gref:569 req->id:11 vif->rx.sring->req_event:589615 reserved_slots_left:-1
> [  575.524427] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590660 vif->rx.sring->req_event:590661
> [  576.508063] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:16 vif->rx.sring->req_prod:594343 vif->rx.req_cons:594343 vif->rx.sring->req_event:591619, reserved_slots_left:0
> [  576.544708] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:16 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594344 req->gref:668 req->id:167 vif->rx.sring->req_event:591619 reserved_slots_left:-1
> [  576.581595] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594389 vif->rx.sring->req_event:594390
> [  577.325826] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:597248 vif->rx.req_cons:597247 vif->rx.sring->req_event:597249
> [  577.576973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:598146 vif->rx.req_cons:598143 vif->rx.sring->req_event:598147
> 
> What i don't get is:
> - i do warn just before the RING_GET_REQUEST on vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 .. so cons == prod
> - i warn because  req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++); is going to add one to cons .. leading to a situation that should not exist ..
> - however in the warn just after this RING_GET_REQUEST .. prod is somehow increased a lot .. probably the "next_rx_buffer" aspect of things .. so cons is much smaller than prod again ..

Frontend is also accessing the ring. So it's normal that some index
changed.

> - How everi don't see anything in the RING_GET_REQUEST macro fiddling with vif->rx.sring->req_prod ..
> - so who has been fiddling with vif->rx.sring->req_prod in the mean time .. my very limited c skills fail to see how this works :-)
> 
> 
> Then the fatal case ...
> 
> [ 1108.566486] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1843278 vif->rx.req_cons:1843278 vif->rx.sring->req_event:1843279
> [ 1109.775757] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1847779 vif->rx.req_cons:1847776 vif->rx.sring->req_event:1847780
> [ 1110.041661] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1848795 vif->rx.req_cons:1848794 vif->rx.sring->req_event:1848796
> [ 1110.301778] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1849755 vif->rx.req_cons:1849754 vif->rx.sring->req_event:1849756
> [ 1111.565412] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1853482, reserved_slots_left:0
> [ 1111.565973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1854292
> [ 1111.638425] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 req->gref:4325377 req->id:83 vif->rx.sring->req_event:1854292 reserved_slots_left:-1
> [ 1111.675042] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:5 vif->rx.sring->req_event:1854292 estimated_slots_needed:7 reserved_slots_left:-1
> [ 1111.711833] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:6 vif->rx.sring->req_event:1854292 gso_gaps:0 estimated_slots_needed:7 reserved_slots_left:-1
> [ 1111.766836] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 i(frag):0 j(data):1 reserved_slots_left:-1
> [ 1111.803702] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:6
> [ 1111.858602] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 sco->meta_slots_used:7 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:7 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:5 reserved_slots_left:-1
> 
> Ok so if i try to pull this apart (from bottom to top):
> 
> - it starts out with "xenvif_rx_action" and calculated we would need:
>       1 slot for the  DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE);
>       5 slots for 1 frag
>       1 slot for GSO
>       So 7 slots in total .. 
> 
> - it checks  if there are enough slots available if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) and aparently there are .. so it continues
> 
> - We end up in "xenvif_gop_skb" processing .. and everything seems to go well ..
>       - In the start of this function we do one unconditional RING_GET_REQUEST .. and potentially one conditional (if gso)
>           - Question: in "xenvif_rx_action" we only reserved ONE slot for GSO stuff .. so this could cost TWO slots .. is that right ?

The slot reserved for GSO is for meta data, so 1 is enough.

>           - But in this case we only use one ... see "used in funcstart: 0 + 1"
> 
>       - Now we loop 1 time through the  "while (data < skb_tail_pointer(skb))" loop ... since j(data)=1
>           - and call "xenvif_gop_frag_copy" from inside this loop here .. which potentially calls .. get_next_rx_buffer
>           - If i understand it correctly the number of slots this loop should take at a maximum should be what we calculated before with the
>             DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE) ? (1 slot in this case)
>           - Nothing is logged yet because cons is still smaller than prod at this time, but from the aftermath we see this was "used_dataloop:1"
>           - So that seems correct .. 
> 
>       - Now we are at the loop through "nr_frags"  and loop the 1 frag we have .. 
>                   - This all goes well until the sixth iteration (i from 0..5) .. and call "xenvif_gop_frag_copy" from inside this loop ...

Isn't there only one frag?

>                   - "xenvif_gop_frag_copy" calls "get_next_rx_buffer" .. cons becomes larger than prod ..and the "miracle" from the non-fatal case of prod suddenly becoming larger does not happen
> 
>       - So the calculation of how many slots are required for the frags in "xenvif_rx_action" does not match wath the code in "xenvif_gop_frag_copy" seems to need,
>         I don't see why .. it's rounding up .. should behave correct if the left over happens to be 0 size ..
> 
> 
> 
> > Wei.
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Friday, March 21, 2014, 5:49:58 PM, you wrote:

> [...]
>> > Those are different estimations. max_slots_needed estimates the worst
>> > case while min_slots_needed estimates the best case. When
>> > min_slots_needed is met, netback puts the SKB into its internal queue.
>> > xenvif_rx_action then will dequeue that packet, check max_slots_needed,
>> > if met, break SKB down.
>> 
>> I realised that, though if you don't count the cost of calculation .. the best would be
>> to have min_slots and max_slots the same .. and being correct. But cost of the calculation count ..
>> on the other hand min_slots seems to be far too low in this case. Is the nr_frags the best thing to check for ?
>> Wouldn't something like skbsize / PAGE_SIZE be a better estimate ?
>> 

> It doesn't matter. It won't cause breakage at that point.

That was more from a performance point of view ... if the estimate can be that much off
it would do the more costly check .. and still bail out. So the better the first estimation
(at about the same cost of course) the better.


>> 
>> > What I would expect is, even if they yield different values, checking
>> > the ring availability is enough before breaking SKB down.
>> 
>> > In your above case, max_slots_needed was met and SKB broken down. And as
>> > you said in your empirical test, bumping max_slots_needed by 1 prevented
>> > issues, then we might have problem calculating max_slots_needed. If you
>> > can work out what went wrong that can be very useful.
>> 
>> OK ran the whole night with "max_slots_needed+1" and i saw no triggering of both:
>>     - the non-fatal oddity (only a temporary cons > prod ..)
>>     - the fatal case (leading to the grantref error)
>> 
>> So it seems we are off by one .. somewhere
>> 
>> 
>> At first the non-fatal oddity .. this one i really can't grasp:
>> 
>> [  567.880282] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:562972 vif->rx.req_cons:562969 vif->rx.sring->req_event:562973
>> [  569.893801] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:571188 vif->rx.req_cons:571187 vif->rx.sring->req_event:571189
>> [  571.892419] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:577141 vif->rx.req_cons:577138 vif->rx.sring->req_event:577142
>> [  575.451383] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:6 vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 vif->rx.sring->req_event:589615, reserved_slots_left:0
>> [  575.487721] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:6 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590604 req->gref:569 req->id:11 vif->rx.sring->req_event:589615 reserved_slots_left:-1
>> [  575.524427] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590660 vif->rx.sring->req_event:590661
>> [  576.508063] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:16 vif->rx.sring->req_prod:594343 vif->rx.req_cons:594343 vif->rx.sring->req_event:591619, reserved_slots_left:0
>> [  576.544708] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:16 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594344 req->gref:668 req->id:167 vif->rx.sring->req_event:591619 reserved_slots_left:-1
>> [  576.581595] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594389 vif->rx.sring->req_event:594390
>> [  577.325826] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:597248 vif->rx.req_cons:597247 vif->rx.sring->req_event:597249
>> [  577.576973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:598146 vif->rx.req_cons:598143 vif->rx.sring->req_event:598147
>> 
>> What i don't get is:
>> - i do warn just before the RING_GET_REQUEST on vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 .. so cons == prod
>> - i warn because  req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++); is going to add one to cons .. leading to a situation that should not exist ..
>> - however in the warn just after this RING_GET_REQUEST .. prod is somehow increased a lot .. probably the "next_rx_buffer" aspect of things .. so cons is much smaller than prod again ..

> Frontend is also accessing the ring. So it's normal that some index
> changed.

Ah ok .. so here the frontend is "saving us" by putting more stuff on the ring so we don't run dry ..

>> - How everi don't see anything in the RING_GET_REQUEST macro fiddling with vif->rx.sring->req_prod ..
>> - so who has been fiddling with vif->rx.sring->req_prod in the mean time .. my very limited c skills fail to see how this works :-)
>> 
>> 
>> Then the fatal case ...
>> 
>> [ 1108.566486] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1843278 vif->rx.req_cons:1843278 vif->rx.sring->req_event:1843279
>> [ 1109.775757] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1847779 vif->rx.req_cons:1847776 vif->rx.sring->req_event:1847780
>> [ 1110.041661] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1848795 vif->rx.req_cons:1848794 vif->rx.sring->req_event:1848796
>> [ 1110.301778] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1849755 vif->rx.req_cons:1849754 vif->rx.sring->req_event:1849756
>> [ 1111.565412] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1853482, reserved_slots_left:0
>> [ 1111.565973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1854292
>> [ 1111.638425] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 req->gref:4325377 req->id:83 vif->rx.sring->req_event:1854292 reserved_slots_left:-1
>> [ 1111.675042] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:5 vif->rx.sring->req_event:1854292 estimated_slots_needed:7 reserved_slots_left:-1
>> [ 1111.711833] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:6 vif->rx.sring->req_event:1854292 gso_gaps:0 estimated_slots_needed:7 reserved_slots_left:-1
>> [ 1111.766836] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 i(frag):0 j(data):1 reserved_slots_left:-1
>> [ 1111.803702] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:6
>> [ 1111.858602] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 sco->meta_slots_used:7 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:7 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:5 reserved_slots_left:-1
>> 
>> Ok so if i try to pull this apart (from bottom to top):
>> 
>> - it starts out with "xenvif_rx_action" and calculated we would need:
>>       1 slot for the  DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE);
>>       5 slots for 1 frag
>>       1 slot for GSO
>>       So 7 slots in total .. 
>> 
>> - it checks  if there are enough slots available if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) and aparently there are .. so it continues
>> 
>> - We end up in "xenvif_gop_skb" processing .. and everything seems to go well ..
>>       - In the start of this function we do one unconditional RING_GET_REQUEST .. and potentially one conditional (if gso)
>>           - Question: in "xenvif_rx_action" we only reserved ONE slot for GSO stuff .. so this could cost TWO slots .. is that right ?

> The slot reserved for GSO is for meta data, so 1 is enough.

>>           - But in this case we only use one ... see "used in funcstart: 0 + 1"
>> 
>>       - Now we loop 1 time through the  "while (data < skb_tail_pointer(skb))" loop ... since j(data)=1
>>           - and call "xenvif_gop_frag_copy" from inside this loop here .. which potentially calls .. get_next_rx_buffer
>>           - If i understand it correctly the number of slots this loop should take at a maximum should be what we calculated before with the
>>             DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE) ? (1 slot in this case)
>>           - Nothing is logged yet because cons is still smaller than prod at this time, but from the aftermath we see this was "used_dataloop:1"
>>           - So that seems correct .. 
>> 
>>       - Now we are at the loop through "nr_frags"  and loop the 1 frag we have .. 
>>                   - This all goes well until the sixth iteration (i from 0..5) .. and call "xenvif_gop_frag_copy" from inside this loop ...

> Isn't there only one frag?

Yes there is only one frag .. but it seems to be much larger than PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller bits .. hence the calculation in xenvif_rx_action determining the slots needed by doing:

                for (i = 0; i < nr_frags; i++) {
                        unsigned int size;
                        size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
                        max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
                }

But the code in xenvif_gop_frag_copy .. seems to be needing one more slot (from the emperical test) .. and calling "get_next_rx_buffer" seems involved in that ..


>>                   - "xenvif_gop_frag_copy" calls "get_next_rx_buffer" .. cons becomes larger than prod ..and the "miracle" from the non-fatal case of prod suddenly becoming larger does not happen
>> 
>>       - So the calculation of how many slots are required for the frags in "xenvif_rx_action" does not match wath the code in "xenvif_gop_frag_copy" seems to need,
>>         I don't see why .. it's rounding up .. should behave correct if the left over happens to be 0 size ..
>> 
>> 
>> 
>> > Wei.
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Friday, March 21, 2014, 6:27:27 PM, you wrote:


> Friday, March 21, 2014, 5:49:58 PM, you wrote:

>> [...]
>>> > Those are different estimations. max_slots_needed estimates the worst
>>> > case while min_slots_needed estimates the best case. When
>>> > min_slots_needed is met, netback puts the SKB into its internal queue.
>>> > xenvif_rx_action then will dequeue that packet, check max_slots_needed,
>>> > if met, break SKB down.
>>> 
>>> I realised that, though if you don't count the cost of calculation .. the best would be
>>> to have min_slots and max_slots the same .. and being correct. But cost of the calculation count ..
>>> on the other hand min_slots seems to be far too low in this case. Is the nr_frags the best thing to check for ?
>>> Wouldn't something like skbsize / PAGE_SIZE be a better estimate ?
>>> 

>> It doesn't matter. It won't cause breakage at that point.

> That was more from a performance point of view ... if the estimate can be that much off
> it would do the more costly check .. and still bail out. So the better the first estimation
> (at about the same cost of course) the better.


>>> 
>>> > What I would expect is, even if they yield different values, checking
>>> > the ring availability is enough before breaking SKB down.
>>> 
>>> > In your above case, max_slots_needed was met and SKB broken down. And as
>>> > you said in your empirical test, bumping max_slots_needed by 1 prevented
>>> > issues, then we might have problem calculating max_slots_needed. If you
>>> > can work out what went wrong that can be very useful.
>>> 
>>> OK ran the whole night with "max_slots_needed+1" and i saw no triggering of both:
>>>     - the non-fatal oddity (only a temporary cons > prod ..)
>>>     - the fatal case (leading to the grantref error)
>>> 
>>> So it seems we are off by one .. somewhere
>>> 
>>> 
>>> At first the non-fatal oddity .. this one i really can't grasp:
>>> 
>>> [  567.880282] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:562972 vif->rx.req_cons:562969 vif->rx.sring->req_event:562973
>>> [  569.893801] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:571188 vif->rx.req_cons:571187 vif->rx.sring->req_event:571189
>>> [  571.892419] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:577141 vif->rx.req_cons:577138 vif->rx.sring->req_event:577142
>>> [  575.451383] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:6 vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 vif->rx.sring->req_event:589615, reserved_slots_left:0
>>> [  575.487721] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:6 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590604 req->gref:569 req->id:11 vif->rx.sring->req_event:589615 reserved_slots_left:-1
>>> [  575.524427] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590660 vif->rx.sring->req_event:590661
>>> [  576.508063] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:16 vif->rx.sring->req_prod:594343 vif->rx.req_cons:594343 vif->rx.sring->req_event:591619, reserved_slots_left:0
>>> [  576.544708] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:16 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594344 req->gref:668 req->id:167 vif->rx.sring->req_event:591619 reserved_slots_left:-1
>>> [  576.581595] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594389 vif->rx.sring->req_event:594390
>>> [  577.325826] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:597248 vif->rx.req_cons:597247 vif->rx.sring->req_event:597249
>>> [  577.576973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:598146 vif->rx.req_cons:598143 vif->rx.sring->req_event:598147
>>> 
>>> What i don't get is:
>>> - i do warn just before the RING_GET_REQUEST on vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 .. so cons == prod
>>> - i warn because  req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++); is going to add one to cons .. leading to a situation that should not exist ..
>>> - however in the warn just after this RING_GET_REQUEST .. prod is somehow increased a lot .. probably the "next_rx_buffer" aspect of things .. so cons is much smaller than prod again ..

>> Frontend is also accessing the ring. So it's normal that some index
>> changed.

> Ah ok .. so here the frontend is "saving us" by putting more stuff on the ring so we don't run dry ..

>>> - How everi don't see anything in the RING_GET_REQUEST macro fiddling with vif->rx.sring->req_prod ..
>>> - so who has been fiddling with vif->rx.sring->req_prod in the mean time .. my very limited c skills fail to see how this works :-)
>>> 
>>> 
>>> Then the fatal case ...
>>> 
>>> [ 1108.566486] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1843278 vif->rx.req_cons:1843278 vif->rx.sring->req_event:1843279
>>> [ 1109.775757] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1847779 vif->rx.req_cons:1847776 vif->rx.sring->req_event:1847780
>>> [ 1110.041661] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1848795 vif->rx.req_cons:1848794 vif->rx.sring->req_event:1848796
>>> [ 1110.301778] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1849755 vif->rx.req_cons:1849754 vif->rx.sring->req_event:1849756
>>> [ 1111.565412] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1853482, reserved_slots_left:0
>>> [ 1111.565973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1854292
>>> [ 1111.638425] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 req->gref:4325377 req->id:83 vif->rx.sring->req_event:1854292 reserved_slots_left:-1
>>> [ 1111.675042] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:5 vif->rx.sring->req_event:1854292 estimated_slots_needed:7 reserved_slots_left:-1
>>> [ 1111.711833] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:6 vif->rx.sring->req_event:1854292 gso_gaps:0 estimated_slots_needed:7 reserved_slots_left:-1
>>> [ 1111.766836] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 i(frag):0 j(data):1 reserved_slots_left:-1
>>> [ 1111.803702] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:6
>>> [ 1111.858602] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 sco->meta_slots_used:7 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:7 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:5 reserved_slots_left:-1
>>> 
>>> Ok so if i try to pull this apart (from bottom to top):
>>> 
>>> - it starts out with "xenvif_rx_action" and calculated we would need:
>>>       1 slot for the  DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE);
>>>       5 slots for 1 frag
>>>       1 slot for GSO
>>>       So 7 slots in total .. 
>>> 
>>> - it checks  if there are enough slots available if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) and aparently there are .. so it continues
>>> 
>>> - We end up in "xenvif_gop_skb" processing .. and everything seems to go well ..
>>>       - In the start of this function we do one unconditional RING_GET_REQUEST .. and potentially one conditional (if gso)
>>>           - Question: in "xenvif_rx_action" we only reserved ONE slot for GSO stuff .. so this could cost TWO slots .. is that right ?

>> The slot reserved for GSO is for meta data, so 1 is enough.

>>>           - But in this case we only use one ... see "used in funcstart: 0 + 1"
>>> 
>>>       - Now we loop 1 time through the  "while (data < skb_tail_pointer(skb))" loop ... since j(data)=1
>>>           - and call "xenvif_gop_frag_copy" from inside this loop here .. which potentially calls .. get_next_rx_buffer
>>>           - If i understand it correctly the number of slots this loop should take at a maximum should be what we calculated before with the
>>>             DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE) ? (1 slot in this case)
>>>           - Nothing is logged yet because cons is still smaller than prod at this time, but from the aftermath we see this was "used_dataloop:1"
>>>           - So that seems correct .. 
>>> 
>>>       - Now we are at the loop through "nr_frags"  and loop the 1 frag we have .. 
>>>                   - This all goes well until the sixth iteration (i from 0..5) .. and call "xenvif_gop_frag_copy" from inside this loop ...

>> Isn't there only one frag?

> Yes there is only one frag .. but it seems to be much larger than PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller bits .. hence the calculation in xenvif_rx_action determining the slots needed by doing:

>                 for (i = 0; i < nr_frags; i++) {
>                         unsigned int size;
>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>                 }

> But the code in xenvif_gop_frag_copy .. seems to be needing one more slot (from the emperical test) .. and calling "get_next_rx_buffer" seems involved in that ..

Hmm looked again .. and it seems this is it .. when your frags are large enough you have the chance of running into this.

Problem is .. i don't see an easy fix, the "one more slot" of the empirical test doesn't seem to be the worst case either (i think):

- In my case the packets that hit this only have 1 frag, but i could have had more frags.
  I also think you can't rule out the possibility of doing the "get_next_rx_buffer" for multiple subsequent frags from one packet,
  so in the worst (and perhaps even from a single frag since it's looped over a split of it in what seems PAGE_SIZE pieces.)
  - So an exact calculation of how much slots we are going to need for hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems unfeasible.
  - A worst case gamble seems impossible either .. if you take multiple frags * multiple times the "get_next_rx_buffer" ... you would probably be back at just
    setting the needed_slots to MAX_SKB_FRAGS.

- Other thing would be checking for the available slots before doing the  "get_next_rx_buffer" .. how ever .. we don't account for how many slots we still need to
  just process the remaining frags.

- Just remove the whole "get_next_rx_buffer".. just tested it but it seems the "get_next_rx_buffer" is necessary ..  when i make start_new_rx_buffer always return false
  i hit the bug below :S

So the questions are ... is the "get_next_rx_buffer" part really necessary ?
   - if not, what is the benefit of the "get_next_rx_buffer" and does that outweigh the additional cost of accounting
     of needed_slots for the frags that have yet to be processed ?
   - if yes, erhmmm what would be the best acceptable solution .. returning to using MAX_SKB_FRAGS ?

--
Sander


[  235.655964] BUG: unable to handle kernel paging request at ffffc9001223b018
[  235.660380] IP: [<ffffffff8177d30e>] xenvif_gop_frag_copy+0x13e/0x580
[  235.664887] PGD 59a2a067 PUD 59a2b067 PMD 55c38067 PTE 0
[  235.669236] Oops: 0002 [#1] SMP
[  235.673611] Modules linked in:
[  235.677936] CPU: 0 PID: 10518 Comm: vif9.0 Not tainted 3.13.6-20140322-nbdebug30+ #1
[  235.682377] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS V1.8B1 09/13/2010
[  235.686711] task: ffff88004dc40000 ti: ffff88004b74a000 task.ti: ffff88004b74a000
[  235.691185] RIP: e030:[<ffffffff8177d30e>]  [<ffffffff8177d30e>] xenvif_gop_frag_copy+0x13e/0x580
[  235.695869] RSP: e02b:ffff88004b74bb28  EFLAGS: 00010203
[  235.700613] RAX: ffff880080000000 RBX: ffff88004b74bd40 RCX: 000000000126d940
[  235.705522] RDX: ffffc9001223aff8 RSI: ffff880000000000 RDI: 0000000000049b65
[  235.710488] RBP: ffff88004b74bc08 R08: 000000000000012c R09: 00000000000000ae
[  235.715515] R10: ffff88004b74bd2c R11: 0000000000000001 R12: 0000000000000000
[  235.720570] R13: 000000000000099e R14: 0000000000000662 R15: ffff88004a6d0940
[  235.725688] FS:  00007f4cbed78700(0000) GS:ffff88005f600000(0000) knlGS:0000000000000000
[  235.730961] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  235.736061] CR2: ffffc9001223b018 CR3: 00000000572b1000 CR4: 0000000000000660
[  235.741247] Stack:
[  235.746320]  ffff880000000000 ffffffff810fa475 ffff88004dc40000 ffff88004dc40000
[  235.751848]  ffffffff81b4ca4d 00000000000002dc 0000000000000000 ffff88004b74be18
[  235.757195]  ffff88004b74bb88 ffffffff810f6251 ffffffff822ee9a0 00000000000002dc
[  235.762802] Call Trace:
[  235.768086]  [<ffffffff810fa475>] ? lock_acquire+0xe5/0x150
[  235.773386]  [<ffffffff81b4ca4d>] ? _raw_spin_unlock_irqrestore+0x6d/0x90
[  235.778783]  [<ffffffff810f6251>] ? trace_hardirqs_on_caller+0x101/0x240
[  235.784252]  [<ffffffff810f639d>] ? trace_hardirqs_on+0xd/0x10
[  235.789726]  [<ffffffff8177f32e>] xenvif_rx_action+0x86e/0x1670
[  235.795167]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
[  235.800560]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
[  235.805782]  [<ffffffff817810e0>] xenvif_kthread+0x40/0x190
[  235.811064]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
[  235.816442]  [<ffffffff817810a0>] ? xenvif_stop_queue+0x60/0x60
[  235.821888]  [<ffffffff810d4f24>] kthread+0xe4/0x100
[  235.827688]  [<ffffffff81b4cc10>] ? _raw_spin_unlock_irq+0x30/0x50
[  235.833925]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
[  235.839890]  [<ffffffff81b4de3c>] ret_from_fork+0x7c/0xb0
[  235.845920]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
[  235.852174] Code: 89 c2 83 c0 01 48 8d 0c 92 48 8b 53 10 89 03 48 b8 00 00 00 80 00 88 ff ff 48 8d 14 ca 48 b9 00 00 00 00 00 16 00 00 48 03 4d c0 <66> 44 89 62 20 66 c7 42 08 f0 7f 66 c7 42 22 02 00 48 c1 f9 06
[  235.865664] RIP  [<ffffffff8177d30e>] xenvif_gop_frag_copy+0x13e/0x580
[  235.872106]  RSP <ffff88004b74bb28>
[  235.878358] CR2: ffffc9001223b018
[  235.885099] ---[ end trace 5971b65cc816997b ]---




>>>                   - "xenvif_gop_frag_copy" calls "get_next_rx_buffer" .. cons becomes larger than prod ..and the "miracle" from the non-fatal case of prod suddenly becoming larger does not happen
>>> 
>>>       - So the calculation of how many slots are required for the frags in "xenvif_rx_action" does not match wath the code in "xenvif_gop_frag_copy" seems to need,
>>>         I don't see why .. it's rounding up .. should behave correct if the left over happens to be 0 size ..
>>> 
>>> 
>>> 
>>> > Wei.
>>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Ping ?


Saturday, March 22, 2014, 7:28:34 PM, you wrote:


> Friday, March 21, 2014, 6:27:27 PM, you wrote:


>> Friday, March 21, 2014, 5:49:58 PM, you wrote:

>>> [...]
>>>> > Those are different estimations. max_slots_needed estimates the worst
>>>> > case while min_slots_needed estimates the best case. When
>>>> > min_slots_needed is met, netback puts the SKB into its internal queue.
>>>> > xenvif_rx_action then will dequeue that packet, check max_slots_needed,
>>>> > if met, break SKB down.
>>>> 
>>>> I realised that, though if you don't count the cost of calculation .. the best would be
>>>> to have min_slots and max_slots the same .. and being correct. But cost of the calculation count ..
>>>> on the other hand min_slots seems to be far too low in this case. Is the nr_frags the best thing to check for ?
>>>> Wouldn't something like skbsize / PAGE_SIZE be a better estimate ?
>>>> 

>>> It doesn't matter. It won't cause breakage at that point.

>> That was more from a performance point of view ... if the estimate can be that much off
>> it would do the more costly check .. and still bail out. So the better the first estimation
>> (at about the same cost of course) the better.


>>>> 
>>>> > What I would expect is, even if they yield different values, checking
>>>> > the ring availability is enough before breaking SKB down.
>>>> 
>>>> > In your above case, max_slots_needed was met and SKB broken down. And as
>>>> > you said in your empirical test, bumping max_slots_needed by 1 prevented
>>>> > issues, then we might have problem calculating max_slots_needed. If you
>>>> > can work out what went wrong that can be very useful.
>>>> 
>>>> OK ran the whole night with "max_slots_needed+1" and i saw no triggering of both:
>>>>     - the non-fatal oddity (only a temporary cons > prod ..)
>>>>     - the fatal case (leading to the grantref error)
>>>> 
>>>> So it seems we are off by one .. somewhere
>>>> 
>>>> 
>>>> At first the non-fatal oddity .. this one i really can't grasp:
>>>> 
>>>> [  567.880282] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:562972 vif->rx.req_cons:562969 vif->rx.sring->req_event:562973
>>>> [  569.893801] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:571188 vif->rx.req_cons:571187 vif->rx.sring->req_event:571189
>>>> [  571.892419] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:577141 vif->rx.req_cons:577138 vif->rx.sring->req_event:577142
>>>> [  575.451383] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:6 vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 vif->rx.sring->req_event:589615, reserved_slots_left:0
>>>> [  575.487721] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:6 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590604 req->gref:569 req->id:11 vif->rx.sring->req_event:589615 reserved_slots_left:-1
>>>> [  575.524427] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:590660 vif->rx.req_cons:590660 vif->rx.sring->req_event:590661
>>>> [  576.508063] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:16 vif->rx.sring->req_prod:594343 vif->rx.req_cons:594343 vif->rx.sring->req_event:591619, reserved_slots_left:0
>>>> [  576.544708] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:16 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594344 req->gref:668 req->id:167 vif->rx.sring->req_event:591619 reserved_slots_left:-1
>>>> [  576.581595] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:594389 vif->rx.req_cons:594389 vif->rx.sring->req_event:594390
>>>> [  577.325826] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:2 vif->rx.sring->req_prod:597248 vif->rx.req_cons:597247 vif->rx.sring->req_event:597249
>>>> [  577.576973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:598146 vif->rx.req_cons:598143 vif->rx.sring->req_event:598147
>>>> 
>>>> What i don't get is:
>>>> - i do warn just before the RING_GET_REQUEST on vif->rx.sring->req_prod:590603 vif->rx.req_cons:590603 .. so cons == prod
>>>> - i warn because  req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++); is going to add one to cons .. leading to a situation that should not exist ..
>>>> - however in the warn just after this RING_GET_REQUEST .. prod is somehow increased a lot .. probably the "next_rx_buffer" aspect of things .. so cons is much smaller than prod again ..

>>> Frontend is also accessing the ring. So it's normal that some index
>>> changed.

>> Ah ok .. so here the frontend is "saving us" by putting more stuff on the ring so we don't run dry ..

>>>> - How everi don't see anything in the RING_GET_REQUEST macro fiddling with vif->rx.sring->req_prod ..
>>>> - so who has been fiddling with vif->rx.sring->req_prod in the mean time .. my very limited c skills fail to see how this works :-)
>>>> 
>>>> 
>>>> Then the fatal case ...
>>>> 
>>>> [ 1108.566486] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1843278 vif->rx.req_cons:1843278 vif->rx.sring->req_event:1843279
>>>> [ 1109.775757] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1847779 vif->rx.req_cons:1847776 vif->rx.sring->req_event:1847780
>>>> [ 1110.041661] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1848795 vif->rx.req_cons:1848794 vif->rx.sring->req_event:1848796
>>>> [ 1110.301778] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:4 vif->rx.sring->req_prod:1849755 vif->rx.req_cons:1849754 vif->rx.sring->req_event:1849756
>>>> [ 1111.565412] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1853482, reserved_slots_left:0
>>>> [ 1111.565973] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854291 vif->rx.sring->req_event:1854292
>>>> [ 1111.638425] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:35 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 req->gref:4325377 req->id:83 vif->rx.sring->req_event:1854292 reserved_slots_left:-1
>>>> [ 1111.675042] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:1168 size:1168 i:5 vif->rx.sring->req_event:1854292 estimated_slots_needed:7 reserved_slots_left:-1
>>>> [ 1111.711833] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:36 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 npo->copy_gref:4325377 npo->copy_off:1168  MAX_BUFFER_OFFSET:4096 bytes:1168 size:0 i:6 vif->rx.sring->req_event:1854292 gso_gaps:0 estimated_slots_needed:7 reserved_slots_left:-1
>>>> [ 1111.766836] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 i(frag):0 j(data):1 reserved_slots_left:-1
>>>> [ 1111.803702] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:36 old_meta_prod:29 vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:674 req->id:76 estimated_slots_needed:7 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:6
>>>> [ 1111.858602] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:1854291 vif->rx.req_cons:1854292 sco->meta_slots_used:7 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:7 j:3 is_gso:1 nr_frags:1 firstpart:1 secondpart:5 reserved_slots_left:-1
>>>> 
>>>> Ok so if i try to pull this apart (from bottom to top):
>>>> 
>>>> - it starts out with "xenvif_rx_action" and calculated we would need:
>>>>       1 slot for the  DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE);
>>>>       5 slots for 1 frag
>>>>       1 slot for GSO
>>>>       So 7 slots in total .. 
>>>> 
>>>> - it checks  if there are enough slots available if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) and aparently there are .. so it continues
>>>> 
>>>> - We end up in "xenvif_gop_skb" processing .. and everything seems to go well ..
>>>>       - In the start of this function we do one unconditional RING_GET_REQUEST .. and potentially one conditional (if gso)
>>>>           - Question: in "xenvif_rx_action" we only reserved ONE slot for GSO stuff .. so this could cost TWO slots .. is that right ?

>>> The slot reserved for GSO is for meta data, so 1 is enough.

>>>>           - But in this case we only use one ... see "used in funcstart: 0 + 1"
>>>> 
>>>>       - Now we loop 1 time through the  "while (data < skb_tail_pointer(skb))" loop ... since j(data)=1
>>>>           - and call "xenvif_gop_frag_copy" from inside this loop here .. which potentially calls .. get_next_rx_buffer
>>>>           - If i understand it correctly the number of slots this loop should take at a maximum should be what we calculated before with the
>>>>             DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE) ? (1 slot in this case)
>>>>           - Nothing is logged yet because cons is still smaller than prod at this time, but from the aftermath we see this was "used_dataloop:1"
>>>>           - So that seems correct .. 
>>>> 
>>>>       - Now we are at the loop through "nr_frags"  and loop the 1 frag we have .. 
>>>>                   - This all goes well until the sixth iteration (i from 0..5) .. and call "xenvif_gop_frag_copy" from inside this loop ...

>>> Isn't there only one frag?

>> Yes there is only one frag .. but it seems to be much larger than PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller bits .. hence the calculation in xenvif_rx_action determining the slots needed by doing:

>>                 for (i = 0; i < nr_frags; i++) {
>>                         unsigned int size;
>>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>>                 }

>> But the code in xenvif_gop_frag_copy .. seems to be needing one more slot (from the emperical test) .. and calling "get_next_rx_buffer" seems involved in that ..

> Hmm looked again .. and it seems this is it .. when your frags are large enough you have the chance of running into this.

> Problem is .. i don't see an easy fix, the "one more slot" of the empirical test doesn't seem to be the worst case either (i think):

> - In my case the packets that hit this only have 1 frag, but i could have had more frags.
>   I also think you can't rule out the possibility of doing the "get_next_rx_buffer" for multiple subsequent frags from one packet,
>   so in the worst (and perhaps even from a single frag since it's looped over a split of it in what seems PAGE_SIZE pieces.)
>   - So an exact calculation of how much slots we are going to need for hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems unfeasible.
>   - A worst case gamble seems impossible either .. if you take multiple frags * multiple times the "get_next_rx_buffer" ... you would probably be back at just
>     setting the needed_slots to MAX_SKB_FRAGS.

> - Other thing would be checking for the available slots before doing the  "get_next_rx_buffer" .. how ever .. we don't account for how many slots we still need to
>   just process the remaining frags.

> - Just remove the whole "get_next_rx_buffer".. just tested it but it seems the "get_next_rx_buffer" is necessary ..  when i make start_new_rx_buffer always return false
>   i hit the bug below :S

> So the questions are ... is the "get_next_rx_buffer" part really necessary ?
>    - if not, what is the benefit of the "get_next_rx_buffer" and does that outweigh the additional cost of accounting
>      of needed_slots for the frags that have yet to be processed ?
>    - if yes, erhmmm what would be the best acceptable solution .. returning to using MAX_SKB_FRAGS ?

> --
> Sander


> [  235.655964] BUG: unable to handle kernel paging request at ffffc9001223b018
> [  235.660380] IP: [<ffffffff8177d30e>] xenvif_gop_frag_copy+0x13e/0x580
> [  235.664887] PGD 59a2a067 PUD 59a2b067 PMD 55c38067 PTE 0
> [  235.669236] Oops: 0002 [#1] SMP
> [  235.673611] Modules linked in:
> [  235.677936] CPU: 0 PID: 10518 Comm: vif9.0 Not tainted 3.13.6-20140322-nbdebug30+ #1
> [  235.682377] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS V1.8B1 09/13/2010
> [  235.686711] task: ffff88004dc40000 ti: ffff88004b74a000 task.ti: ffff88004b74a000
> [  235.691185] RIP: e030:[<ffffffff8177d30e>]  [<ffffffff8177d30e>] xenvif_gop_frag_copy+0x13e/0x580
> [  235.695869] RSP: e02b:ffff88004b74bb28  EFLAGS: 00010203
> [  235.700613] RAX: ffff880080000000 RBX: ffff88004b74bd40 RCX: 000000000126d940
> [  235.705522] RDX: ffffc9001223aff8 RSI: ffff880000000000 RDI: 0000000000049b65
> [  235.710488] RBP: ffff88004b74bc08 R08: 000000000000012c R09: 00000000000000ae
> [  235.715515] R10: ffff88004b74bd2c R11: 0000000000000001 R12: 0000000000000000
> [  235.720570] R13: 000000000000099e R14: 0000000000000662 R15: ffff88004a6d0940
> [  235.725688] FS:  00007f4cbed78700(0000) GS:ffff88005f600000(0000) knlGS:0000000000000000
> [  235.730961] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  235.736061] CR2: ffffc9001223b018 CR3: 00000000572b1000 CR4: 0000000000000660
> [  235.741247] Stack:
> [  235.746320]  ffff880000000000 ffffffff810fa475 ffff88004dc40000 ffff88004dc40000
> [  235.751848]  ffffffff81b4ca4d 00000000000002dc 0000000000000000 ffff88004b74be18
> [  235.757195]  ffff88004b74bb88 ffffffff810f6251 ffffffff822ee9a0 00000000000002dc
> [  235.762802] Call Trace:
> [  235.768086]  [<ffffffff810fa475>] ? lock_acquire+0xe5/0x150
> [  235.773386]  [<ffffffff81b4ca4d>] ? _raw_spin_unlock_irqrestore+0x6d/0x90
> [  235.778783]  [<ffffffff810f6251>] ? trace_hardirqs_on_caller+0x101/0x240
> [  235.784252]  [<ffffffff810f639d>] ? trace_hardirqs_on+0xd/0x10
> [  235.789726]  [<ffffffff8177f32e>] xenvif_rx_action+0x86e/0x1670
> [  235.795167]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
> [  235.800560]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
> [  235.805782]  [<ffffffff817810e0>] xenvif_kthread+0x40/0x190
> [  235.811064]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
> [  235.816442]  [<ffffffff817810a0>] ? xenvif_stop_queue+0x60/0x60
> [  235.821888]  [<ffffffff810d4f24>] kthread+0xe4/0x100
> [  235.827688]  [<ffffffff81b4cc10>] ? _raw_spin_unlock_irq+0x30/0x50
> [  235.833925]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
> [  235.839890]  [<ffffffff81b4de3c>] ret_from_fork+0x7c/0xb0
> [  235.845920]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
> [  235.852174] Code: 89 c2 83 c0 01 48 8d 0c 92 48 8b 53 10 89 03 48 b8 00 00 00 80 00 88 ff ff 48 8d 14 ca 48 b9 00 00 00 00 00 16 00 00 48 03 4d c0 <66> 44 89 62 20 66 c7 42 08 f0 7f 66 c7 42 22 02 00 48 c1 f9 06
> [  235.865664] RIP  [<ffffffff8177d30e>] xenvif_gop_frag_copy+0x13e/0x580
> [  235.872106]  RSP <ffff88004b74bb28>
> [  235.878358] CR2: ffffc9001223b018
> [  235.885099] ---[ end trace 5971b65cc816997b ]---




>>>>                   - "xenvif_gop_frag_copy" calls "get_next_rx_buffer" .. cons becomes larger than prod ..and the "miracle" from the non-fatal case of prod suddenly becoming larger does not happen
>>>> 
>>>>       - So the calculation of how many slots are required for the frags in "xenvif_rx_action" does not match wath the code in "xenvif_gop_frag_copy" seems to need,
>>>>         I don't see why .. it's rounding up .. should behave correct if the left over happens to be 0 size ..
>>>> 
>>>> 
>>>> 
>>>> > Wei.
>>>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Wei Liu]

On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
[...]
> > Yes there is only one frag .. but it seems to be much larger than PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller bits .. hence the calculation in xenvif_rx_action determining the slots needed by doing:
> 
> >                 for (i = 0; i < nr_frags; i++) {
> >                         unsigned int size;
> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >                 }
> 
> > But the code in xenvif_gop_frag_copy .. seems to be needing one more slot (from the emperical test) .. and calling "get_next_rx_buffer" seems involved in that ..
> 
> Hmm looked again .. and it seems this is it .. when your frags are large enough you have the chance of running into this.
> 

get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
problem with that implementation?

> Problem is .. i don't see an easy fix, the "one more slot" of the empirical test doesn't seem to be the worst case either (i think):
> 
> - In my case the packets that hit this only have 1 frag, but i could have had more frags.
>   I also think you can't rule out the possibility of doing the "get_next_rx_buffer" for multiple subsequent frags from one packet,
>   so in the worst (and perhaps even from a single frag since it's looped over a split of it in what seems PAGE_SIZE pieces.)
>   - So an exact calculation of how much slots we are going to need for hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems unfeasible.
>   - A worst case gamble seems impossible either .. if you take multiple frags * multiple times the "get_next_rx_buffer" ... you would probably be back at just
>     setting the needed_slots to MAX_SKB_FRAGS.
> 
> - Other thing would be checking for the available slots before doing the  "get_next_rx_buffer" .. how ever .. we don't account for how many slots we still need to
>   just process the remaining frags.
> 

We've done a worst case estimation for whole SKB (linear area + all
frags) already, at the very first beginning. That's what
max_slots_needed is for.

> - Just remove the whole "get_next_rx_buffer".. just tested it but it seems the "get_next_rx_buffer" is necessary ..  when i make start_new_rx_buffer always return false
>   i hit the bug below :S
> 
> So the questions are ... is the "get_next_rx_buffer" part really necessary ?
>    - if not, what is the benefit of the "get_next_rx_buffer" and does that outweigh the additional cost of accounting
>      of needed_slots for the frags that have yet to be processed ?
>    - if yes, erhmmm what would be the best acceptable solution .. returning to using MAX_SKB_FRAGS ?
> 

I think you need to answer several questions first:
1. is max_slots_needed actually large enough to cover whole SKB?
2. is the test of ring slot availability acurate?
3. is the number of ring slots consumed larger than max_slots_needed? (I
   guess the answer is yes)
4. which step in the break-down procedure causes backend to overrun the
   ring?

It doesn't matter how many frags your SKB has and how big a frag is. If
every step is correct then you're fine. The code you're looking at
(start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
carefully examined.

Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Tuesday, March 25, 2014, 4:15:39 PM, you wrote:

> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
> [...]
>> > Yes there is only one frag .. but it seems to be much larger than PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller bits .. hence the calculation in xenvif_rx_action determining the slots needed by doing:
>> 
>> >                 for (i = 0; i < nr_frags; i++) {
>> >                         unsigned int size;
>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>> >                 }
>> 
>> > But the code in xenvif_gop_frag_copy .. seems to be needing one more slot (from the emperical test) .. and calling "get_next_rx_buffer" seems involved in that ..
>> 
>> Hmm looked again .. and it seems this is it .. when your frags are large enough you have the chance of running into this.
>> 

> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
> problem with that implementation?
In general no, but "get_next_rx_buffer" up's cons .. and the calculations done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
don't count in this possibility. So it's not the guarding of "start_new_rx_buffer" that is at fault. It's the ones early in "xenvif_rx_action".
The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a calculated value that should be a "slim fit".

The problem is in determining upfront in "xenvif_rx_action" when and how often the "get_next_rx_buffer" path will be taken.
Unless there are other non direct restrictions (from a size point of view) it can be called multiple times per frag per skb.

>> Problem is .. i don't see an easy fix, the "one more slot" of the empirical test doesn't seem to be the worst case either (i think):
>> 
>> - In my case the packets that hit this only have 1 frag, but i could have had more frags.
>>   I also think you can't rule out the possibility of doing the "get_next_rx_buffer" for multiple subsequent frags from one packet,
>>   so in the worst (and perhaps even from a single frag since it's looped over a split of it in what seems PAGE_SIZE pieces.)
>>   - So an exact calculation of how much slots we are going to need for hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems unfeasible.
>>   - A worst case gamble seems impossible either .. if you take multiple frags * multiple times the "get_next_rx_buffer" ... you would probably be back at just
>>     setting the needed_slots to MAX_SKB_FRAGS.
>> 
>> - Other thing would be checking for the available slots before doing the  "get_next_rx_buffer" .. how ever .. we don't account for how many slots we still need to
>>   just process the remaining frags.
>> 

> We've done a worst case estimation for whole SKB (linear area + all
> frags) already, at the very first beginning. That's what
> max_slots_needed is for.

>> - Just remove the whole "get_next_rx_buffer".. just tested it but it seems the "get_next_rx_buffer" is necessary ..  when i make start_new_rx_buffer always return false
>>   i hit the bug below :S
>> 
>> So the questions are ... is the "get_next_rx_buffer" part really necessary ?
>>    - if not, what is the benefit of the "get_next_rx_buffer" and does that outweigh the additional cost of accounting
>>      of needed_slots for the frags that have yet to be processed ?
>>    - if yes, erhmmm what would be the best acceptable solution .. returning to using MAX_SKB_FRAGS ?
>> 

> I think you need to answer several questions first:
> 1. is max_slots_needed actually large enough to cover whole SKB?
        No it's not if, you end up calling "get_next_rx_buffer" one or multiple times when processing the SKB
        you have the chance of overrunning (or be saved because prod gets upped before you overrun).

> 2. is the test of ring slot availability acurate?
        Seems to be.

> 3. is the number of ring slots consumed larger than max_slots_needed? (I
>    guess the answer is yes)
        Yes that was the whole point.

> 4. which step in the break-down procedure causes backend to overrun the
>    ring?
        The "get_next_rx_buffer" call .. that is not accounted for (which can be called
        multiple times per frag (unless some other none obvious size restriction limits this
        to one time per frag or one time per SKB).
        In my errorneous case it is called one time, but it would be nice if there would be some theoretical proof
        instead of just some emperical test.


> It doesn't matter how many frags your SKB has and how big a frag is. If
> every step is correct then you're fine. The code you're looking at
> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
> carefully examined.

Well it seems it only calls "get_next_rx_buffer" in some special cases .. (and that also what i'm seeing because it doesn't happen
continously.

Question is shouldn't this max_needed_slots calc be reverted to what it was before 3.14 and take the time in 3.15 to figure out a
the theoretical max (if it can be calculated upfront) .. or another way to guarantee the code is able to process the whole SKB  ?

> Wei.

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Paul,

You have been awfully silent for this whole thread while this is a regression caused by a patch of you
(ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier in this thread).

The commit messages states:
    "net_rx_action() is the place where we could do with an accurate predicition but,
    since that has proven tricky to calculate, a cheap worse-case (but not too bad)
    estimate is all we really need since the only thing we *must* prevent is xenvif_gop_skb()
    consuming more slots than are available."

Your "worst-case" calculation stated in the commit message is clearly not the worst case,
since it doesn't take calls to "get_next_rx_buffer" into account.

Problem is that a worst case calculation would probably be reverting to the old calculation,
and the problems this patch was trying to solve would reappear, but introducing new regressions
isn't very useful either. And since it seems such a tricky and fragile thing to determine, it would
probably be best to be split into a distinct function with a comment to explain the rationale used.

Since this doesn't seem to progress very fast .. CC'ed some more folks .. you never know ..

--
Sander


Tuesday, March 25, 2014, 4:29:42 PM, you wrote:


> Tuesday, March 25, 2014, 4:15:39 PM, you wrote:

>> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
>> [...]
>>> > Yes there is only one frag .. but it seems to be much larger than PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller bits .. hence the calculation in xenvif_rx_action determining the slots needed by doing:
>>> 
>>> >                 for (i = 0; i < nr_frags; i++) {
>>> >                         unsigned int size;
>>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>>> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>>> >                 }
>>> 
>>> > But the code in xenvif_gop_frag_copy .. seems to be needing one more slot (from the emperical test) .. and calling "get_next_rx_buffer" seems involved in that ..
>>> 
>>> Hmm looked again .. and it seems this is it .. when your frags are large enough you have the chance of running into this.
>>> 

>> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
>> problem with that implementation?
> In general no, but "get_next_rx_buffer" up's cons .. and the calculations done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
> don't count in this possibility. So it's not the guarding of "start_new_rx_buffer" that is at fault. It's the ones early in "xenvif_rx_action".
> The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a calculated value that should be a "slim fit".

> The problem is in determining upfront in "xenvif_rx_action" when and how often the "get_next_rx_buffer" path will be taken.
> Unless there are other non direct restrictions (from a size point of view) it can be called multiple times per frag per skb.

>>> Problem is .. i don't see an easy fix, the "one more slot" of the empirical test doesn't seem to be the worst case either (i think):
>>> 
>>> - In my case the packets that hit this only have 1 frag, but i could have had more frags.
>>>   I also think you can't rule out the possibility of doing the "get_next_rx_buffer" for multiple subsequent frags from one packet,
>>>   so in the worst (and perhaps even from a single frag since it's looped over a split of it in what seems PAGE_SIZE pieces.)
>>>   - So an exact calculation of how much slots we are going to need for hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems unfeasible.
>>>   - A worst case gamble seems impossible either .. if you take multiple frags * multiple times the "get_next_rx_buffer" ... you would probably be back at just
>>>     setting the needed_slots to MAX_SKB_FRAGS.
>>> 
>>> - Other thing would be checking for the available slots before doing the  "get_next_rx_buffer" .. how ever .. we don't account for how many slots we still need to
>>>   just process the remaining frags.
>>> 

>> We've done a worst case estimation for whole SKB (linear area + all
>> frags) already, at the very first beginning. That's what
>> max_slots_needed is for.

>>> - Just remove the whole "get_next_rx_buffer".. just tested it but it seems the "get_next_rx_buffer" is necessary ..  when i make start_new_rx_buffer always return false
>>>   i hit the bug below :S
>>> 
>>> So the questions are ... is the "get_next_rx_buffer" part really necessary ?
>>>    - if not, what is the benefit of the "get_next_rx_buffer" and does that outweigh the additional cost of accounting
>>>      of needed_slots for the frags that have yet to be processed ?
>>>    - if yes, erhmmm what would be the best acceptable solution .. returning to using MAX_SKB_FRAGS ?
>>> 

>> I think you need to answer several questions first:
>> 1. is max_slots_needed actually large enough to cover whole SKB?
>         No it's not if, you end up calling "get_next_rx_buffer" one or multiple times when processing the SKB
>         you have the chance of overrunning (or be saved because prod gets upped before you overrun).

>> 2. is the test of ring slot availability acurate?
>         Seems to be.

>> 3. is the number of ring slots consumed larger than max_slots_needed? (I
>>    guess the answer is yes)
>         Yes that was the whole point.

>> 4. which step in the break-down procedure causes backend to overrun the
>>    ring?
>         The "get_next_rx_buffer" call .. that is not accounted for (which can be called
>         multiple times per frag (unless some other none obvious size restriction limits this
>         to one time per frag or one time per SKB).
>         In my errorneous case it is called one time, but it would be nice if there would be some theoretical proof
>         instead of just some emperical test.


>> It doesn't matter how many frags your SKB has and how big a frag is. If
>> every step is correct then you're fine. The code you're looking at
>> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
>> carefully examined.

> Well it seems it only calls "get_next_rx_buffer" in some special cases .. (and that also what i'm seeing because it doesn't happen
> continously.

> Question is shouldn't this max_needed_slots calc be reverted to what it was before 3.14 and take the time in 3.15 to figure out a
> the theoretical max (if it can be calculated upfront) .. or another way to guarantee the code is able to process the whole SKB  ?

>> Wei.

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 11:11
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> Paul,
> 
> You have been awfully silent for this whole thread while this is a regression
> caused by a patch of you
> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier in
> this thread).
> 

Sorry, I've been distracted...

> The commit messages states:
>     "net_rx_action() is the place where we could do with an accurate
> predicition but,
>     since that has proven tricky to calculate, a cheap worse-case (but not too
> bad)
>     estimate is all we really need since the only thing we *must* prevent is
> xenvif_gop_skb()
>     consuming more slots than are available."
> 
> Your "worst-case" calculation stated in the commit message is clearly not the
> worst case,
> since it doesn't take calls to "get_next_rx_buffer" into account.
> 

It should be taking into account the behaviour of start_new_rx_buffer(), which should be true if a slot is full or a frag will overflow the current slot and doesn't require splitting. The code in net_rx_action() makes the assumption that each frag will require as many slots as its size requires, i.e. it assumes no packing of multiple frags into a single slot, so it should be a worst case. Did I miss something in that logic?

  Paul

> Problem is that a worst case calculation would probably be reverting to the
> old calculation,
> and the problems this patch was trying to solve would reappear, but
> introducing new regressions
> isn't very useful either. And since it seems such a tricky and fragile thing to
> determine, it would
> probably be best to be split into a distinct function with a comment to explain
> the rationale used.
> 
> Since this doesn't seem to progress very fast .. CC'ed some more folks .. you
> never know ..
> 
> --
> Sander
> 
> 
> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> 
> 
> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> 
> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
> >> [...]
> >>> > Yes there is only one frag .. but it seems to be much larger than
> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller
> bits .. hence the calculation in xenvif_rx_action determining the slots needed
> by doing:
> >>>
> >>> >                 for (i = 0; i < nr_frags; i++) {
> >>> >                         unsigned int size;
> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >>> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >>> >                 }
> >>>
> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
> seems involved in that ..
> >>>
> >>> Hmm looked again .. and it seems this is it .. when your frags are large
> enough you have the chance of running into this.
> >>>
> 
> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
> >> problem with that implementation?
> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
> > don't count in this possibility. So it's not the guarding of
> "start_new_rx_buffer" that is at fault. It's the ones early in
> "xenvif_rx_action".
> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
> calculated value that should be a "slim fit".
> 
> > The problem is in determining upfront in "xenvif_rx_action" when and how
> often the "get_next_rx_buffer" path will be taken.
> > Unless there are other non direct restrictions (from a size point of view) it
> can be called multiple times per frag per skb.
> 
> >>> Problem is .. i don't see an easy fix, the "one more slot" of the empirical
> test doesn't seem to be the worst case either (i think):
> >>>
> >>> - In my case the packets that hit this only have 1 frag, but i could have
> had more frags.
> >>>   I also think you can't rule out the possibility of doing the
> "get_next_rx_buffer" for multiple subsequent frags from one packet,
> >>>   so in the worst (and perhaps even from a single frag since it's looped
> over a split of it in what seems PAGE_SIZE pieces.)
> >>>   - So an exact calculation of how much slots we are going to need for
> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
> unfeasible.
> >>>   - A worst case gamble seems impossible either .. if you take multiple
> frags * multiple times the "get_next_rx_buffer" ... you would probably be
> back at just
> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >>>
> >>> - Other thing would be checking for the available slots before doing the
> "get_next_rx_buffer" .. how ever .. we don't account for how many slots we
> still need to
> >>>   just process the remaining frags.
> >>>
> 
> >> We've done a worst case estimation for whole SKB (linear area + all
> >> frags) already, at the very first beginning. That's what
> >> max_slots_needed is for.
> 
> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
> seems the "get_next_rx_buffer" is necessary ..  when i make
> start_new_rx_buffer always return false
> >>>   i hit the bug below :S
> >>>
> >>> So the questions are ... is the "get_next_rx_buffer" part really necessary
> ?
> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does that
> outweigh the additional cost of accounting
> >>>      of needed_slots for the frags that have yet to be processed ?
> >>>    - if yes, erhmmm what would be the best acceptable solution ..
> returning to using MAX_SKB_FRAGS ?
> >>>
> 
> >> I think you need to answer several questions first:
> >> 1. is max_slots_needed actually large enough to cover whole SKB?
> >         No it's not if, you end up calling "get_next_rx_buffer" one or multiple
> times when processing the SKB
> >         you have the chance of overrunning (or be saved because prod gets
> upped before you overrun).
> 
> >> 2. is the test of ring slot availability acurate?
> >         Seems to be.
> 
> >> 3. is the number of ring slots consumed larger than max_slots_needed? (I
> >>    guess the answer is yes)
> >         Yes that was the whole point.
> 
> >> 4. which step in the break-down procedure causes backend to overrun
> the
> >>    ring?
> >         The "get_next_rx_buffer" call .. that is not accounted for (which can be
> called
> >         multiple times per frag (unless some other none obvious size
> restriction limits this
> >         to one time per frag or one time per SKB).
> >         In my errorneous case it is called one time, but it would be nice if there
> would be some theoretical proof
> >         instead of just some emperical test.
> 
> 
> >> It doesn't matter how many frags your SKB has and how big a frag is. If
> >> every step is correct then you're fine. The code you're looking at
> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
> >> carefully examined.
> 
> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
> (and that also what i'm seeing because it doesn't happen
> > continously.
> 
> > Question is shouldn't this max_needed_slots calc be reverted to what it
> was before 3.14 and take the time in 3.15 to figure out a
> > the theoretical max (if it can be calculated upfront) .. or another way to
> guarantee the code is able to process the whole SKB  ?
> 
> >> Wei.
> 
> 
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 3:44:42 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 11:11
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> Paul,
>> 
>> You have been awfully silent for this whole thread while this is a regression
>> caused by a patch of you
>> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier in
>> this thread).
>> 

> Sorry, I've been distracted...

>> The commit messages states:
>>     "net_rx_action() is the place where we could do with an accurate
>> predicition but,
>>     since that has proven tricky to calculate, a cheap worse-case (but not too
>> bad)
>>     estimate is all we really need since the only thing we *must* prevent is
>> xenvif_gop_skb()
>>     consuming more slots than are available."
>> 
>> Your "worst-case" calculation stated in the commit message is clearly not the
>> worst case,
>> since it doesn't take calls to "get_next_rx_buffer" into account.
>> 

> It should be taking into account the behaviour of start_new_rx_buffer(), which should be true if a slot is full or a frag will overflow the current slot and doesn't require splitting.
> The code in net_rx_action() makes the assumption that each frag will require as many slots as its size requires, i.e. it assumes no packing of multiple frags into a single slot, so it should be a worst case.
> Did I miss something in that logic?

Yes.
In "xenvif_gop_skb()" this loop:

        for (i = 0; i < nr_frags; i++) {
                xenvif_gop_frag_copy(vif, skb, npo,
                                     skb_frag_page(&skb_shinfo(skb)->frags[i]),
                                     skb_frag_size(&skb_shinfo(skb)->frags[i]),
                                     skb_shinfo(skb)->frags[i].page_offset,
                                     &head);
        }

Is capable of using up (at least) 1 slot more than is anticipated for in "net_rx_action()"  by this code:

                for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
                        unsigned int size;
                        size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
                        max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
                }

And this happens when it calls "get_next_rx_buffer()" from "xenvif_gop_frag_copy()" where it's breaking down the frag.

Ultimately this results in bad grant reference warnings (and packets marked as "errors" in the interface statistics).

In my case it always seems to be a skb with 1 frag which is broken down in 5 or 6 pieces ..

So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring with 1 slot, but i'm not sure if that's not coincedence
since in the code there seem to be no explicit limitation on how often this code path is taken. So perhaps it's implicitly limited
since packets and frags can't be arbitrarily large in comparison with the page_size but that's not something i'm capable of figuring out :-)



>   Paul

>> Problem is that a worst case calculation would probably be reverting to the
>> old calculation,
>> and the problems this patch was trying to solve would reappear, but
>> introducing new regressions
>> isn't very useful either. And since it seems such a tricky and fragile thing to
>> determine, it would
>> probably be best to be split into a distinct function with a comment to explain
>> the rationale used.
>> 
>> Since this doesn't seem to progress very fast .. CC'ed some more folks .. you
>> never know ..
>> 
>> --
>> Sander
>> 
>> 
>> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
>> 
>> 
>> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
>> 
>> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
>> >> [...]
>> >>> > Yes there is only one frag .. but it seems to be much larger than
>> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller
>> bits .. hence the calculation in xenvif_rx_action determining the slots needed
>> by doing:
>> >>>
>> >>> >                 for (i = 0; i < nr_frags; i++) {
>> >>> >                         unsigned int size;
>> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >>> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>> >>> >                 }
>> >>>
>> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
>> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
>> seems involved in that ..
>> >>>
>> >>> Hmm looked again .. and it seems this is it .. when your frags are large
>> enough you have the chance of running into this.
>> >>>
>> 
>> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
>> >> problem with that implementation?
>> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
>> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
>> > don't count in this possibility. So it's not the guarding of
>> "start_new_rx_buffer" that is at fault. It's the ones early in
>> "xenvif_rx_action".
>> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
>> calculated value that should be a "slim fit".
>> 
>> > The problem is in determining upfront in "xenvif_rx_action" when and how
>> often the "get_next_rx_buffer" path will be taken.
>> > Unless there are other non direct restrictions (from a size point of view) it
>> can be called multiple times per frag per skb.
>> 
>> >>> Problem is .. i don't see an easy fix, the "one more slot" of the empirical
>> test doesn't seem to be the worst case either (i think):
>> >>>
>> >>> - In my case the packets that hit this only have 1 frag, but i could have
>> had more frags.
>> >>>   I also think you can't rule out the possibility of doing the
>> "get_next_rx_buffer" for multiple subsequent frags from one packet,
>> >>>   so in the worst (and perhaps even from a single frag since it's looped
>> over a split of it in what seems PAGE_SIZE pieces.)
>> >>>   - So an exact calculation of how much slots we are going to need for
>> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
>> unfeasible.
>> >>>   - A worst case gamble seems impossible either .. if you take multiple
>> frags * multiple times the "get_next_rx_buffer" ... you would probably be
>> back at just
>> >>>     setting the needed_slots to MAX_SKB_FRAGS.
>> >>>
>> >>> - Other thing would be checking for the available slots before doing the
>> "get_next_rx_buffer" .. how ever .. we don't account for how many slots we
>> still need to
>> >>>   just process the remaining frags.
>> >>>
>> 
>> >> We've done a worst case estimation for whole SKB (linear area + all
>> >> frags) already, at the very first beginning. That's what
>> >> max_slots_needed is for.
>> 
>> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
>> seems the "get_next_rx_buffer" is necessary ..  when i make
>> start_new_rx_buffer always return false
>> >>>   i hit the bug below :S
>> >>>
>> >>> So the questions are ... is the "get_next_rx_buffer" part really necessary
>> ?
>> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does that
>> outweigh the additional cost of accounting
>> >>>      of needed_slots for the frags that have yet to be processed ?
>> >>>    - if yes, erhmmm what would be the best acceptable solution ..
>> returning to using MAX_SKB_FRAGS ?
>> >>>
>> 
>> >> I think you need to answer several questions first:
>> >> 1. is max_slots_needed actually large enough to cover whole SKB?
>> >         No it's not if, you end up calling "get_next_rx_buffer" one or multiple
>> times when processing the SKB
>> >         you have the chance of overrunning (or be saved because prod gets
>> upped before you overrun).
>> 
>> >> 2. is the test of ring slot availability acurate?
>> >         Seems to be.
>> 
>> >> 3. is the number of ring slots consumed larger than max_slots_needed? (I
>> >>    guess the answer is yes)
>> >         Yes that was the whole point.
>> 
>> >> 4. which step in the break-down procedure causes backend to overrun
>> the
>> >>    ring?
>> >         The "get_next_rx_buffer" call .. that is not accounted for (which can be
>> called
>> >         multiple times per frag (unless some other none obvious size
>> restriction limits this
>> >         to one time per frag or one time per SKB).
>> >         In my errorneous case it is called one time, but it would be nice if there
>> would be some theoretical proof
>> >         instead of just some emperical test.
>> 
>> 
>> >> It doesn't matter how many frags your SKB has and how big a frag is. If
>> >> every step is correct then you're fine. The code you're looking at
>> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
>> >> carefully examined.
>> 
>> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
>> (and that also what i'm seeing because it doesn't happen
>> > continously.
>> 
>> > Question is shouldn't this max_needed_slots calc be reverted to what it
>> was before 3.14 and take the time in 3.15 to figure out a
>> > the theoretical max (if it can be calculated upfront) .. or another way to
>> guarantee the code is able to process the whole SKB  ?
>> 
>> >> Wei.
>> 
>> 
>> 
>> 

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 3:44:42 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 11:11
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> Paul,
>> 
>> You have been awfully silent for this whole thread while this is a regression
>> caused by a patch of you
>> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier in
>> this thread).
>> 

> Sorry, I've been distracted...

>> The commit messages states:
>>     "net_rx_action() is the place where we could do with an accurate
>> predicition but,
>>     since that has proven tricky to calculate, a cheap worse-case (but not too
>> bad)
>>     estimate is all we really need since the only thing we *must* prevent is
>> xenvif_gop_skb()
>>     consuming more slots than are available."
>> 
>> Your "worst-case" calculation stated in the commit message is clearly not the
>> worst case,
>> since it doesn't take calls to "get_next_rx_buffer" into account.
>> 

> It should be taking into account the behaviour of start_new_rx_buffer(), which should be true if a slot is full or a frag will overflow the current slot and doesn't require splitting.
> The code in net_rx_action() makes the assumption that each frag will require as many slots as its size requires, i.e. it assumes no packing of multiple frags into a single slot, so it should be a worst case.
> Did I miss something in that logic?

Yes.
In "xenvif_gop_skb()" this loop:

        for (i = 0; i < nr_frags; i++) {
                xenvif_gop_frag_copy(vif, skb, npo,
                                     skb_frag_page(&skb_shinfo(skb)->frags[i]),
                                     skb_frag_size(&skb_shinfo(skb)->frags[i]),
                                     skb_shinfo(skb)->frags[i].page_offset,
                                     &head);
        }

Is capable of using up (at least) 1 slot more than is anticipated for in "net_rx_action()"  by this code:

                for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
                        unsigned int size;
                        size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
                        max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
                }

And this happens when it calls "get_next_rx_buffer()" from "xenvif_gop_frag_copy()" where it's breaking down the frag.

Ultimately this results in bad grant reference warnings (and packets marked as "errors" in the interface statistics).

In my case it always seems to be a skb with 1 frag which is broken down in 5 or 6 pieces ..

So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring with 1 slot, but i'm not sure if that's not coincedence
since in the code there seem to be no explicit limitation on how often this code path is taken. So perhaps it's implicitly limited
since packets and frags can't be arbitrarily large in comparison with the page_size but that's not something i'm capable of figuring out :-)



>   Paul

>> Problem is that a worst case calculation would probably be reverting to the
>> old calculation,
>> and the problems this patch was trying to solve would reappear, but
>> introducing new regressions
>> isn't very useful either. And since it seems such a tricky and fragile thing to
>> determine, it would
>> probably be best to be split into a distinct function with a comment to explain
>> the rationale used.
>> 
>> Since this doesn't seem to progress very fast .. CC'ed some more folks .. you
>> never know ..
>> 
>> --
>> Sander
>> 
>> 
>> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
>> 
>> 
>> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
>> 
>> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
>> >> [...]
>> >>> > Yes there is only one frag .. but it seems to be much larger than
>> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller
>> bits .. hence the calculation in xenvif_rx_action determining the slots needed
>> by doing:
>> >>>
>> >>> >                 for (i = 0; i < nr_frags; i++) {
>> >>> >                         unsigned int size;
>> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >>> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>> >>> >                 }
>> >>>
>> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
>> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
>> seems involved in that ..
>> >>>
>> >>> Hmm looked again .. and it seems this is it .. when your frags are large
>> enough you have the chance of running into this.
>> >>>
>> 
>> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
>> >> problem with that implementation?
>> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
>> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
>> > don't count in this possibility. So it's not the guarding of
>> "start_new_rx_buffer" that is at fault. It's the ones early in
>> "xenvif_rx_action".
>> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
>> calculated value that should be a "slim fit".
>> 
>> > The problem is in determining upfront in "xenvif_rx_action" when and how
>> often the "get_next_rx_buffer" path will be taken.
>> > Unless there are other non direct restrictions (from a size point of view) it
>> can be called multiple times per frag per skb.
>> 
>> >>> Problem is .. i don't see an easy fix, the "one more slot" of the empirical
>> test doesn't seem to be the worst case either (i think):
>> >>>
>> >>> - In my case the packets that hit this only have 1 frag, but i could have
>> had more frags.
>> >>>   I also think you can't rule out the possibility of doing the
>> "get_next_rx_buffer" for multiple subsequent frags from one packet,
>> >>>   so in the worst (and perhaps even from a single frag since it's looped
>> over a split of it in what seems PAGE_SIZE pieces.)
>> >>>   - So an exact calculation of how much slots we are going to need for
>> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
>> unfeasible.
>> >>>   - A worst case gamble seems impossible either .. if you take multiple
>> frags * multiple times the "get_next_rx_buffer" ... you would probably be
>> back at just
>> >>>     setting the needed_slots to MAX_SKB_FRAGS.
>> >>>
>> >>> - Other thing would be checking for the available slots before doing the
>> "get_next_rx_buffer" .. how ever .. we don't account for how many slots we
>> still need to
>> >>>   just process the remaining frags.
>> >>>
>> 
>> >> We've done a worst case estimation for whole SKB (linear area + all
>> >> frags) already, at the very first beginning. That's what
>> >> max_slots_needed is for.
>> 
>> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
>> seems the "get_next_rx_buffer" is necessary ..  when i make
>> start_new_rx_buffer always return false
>> >>>   i hit the bug below :S
>> >>>
>> >>> So the questions are ... is the "get_next_rx_buffer" part really necessary
>> ?
>> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does that
>> outweigh the additional cost of accounting
>> >>>      of needed_slots for the frags that have yet to be processed ?
>> >>>    - if yes, erhmmm what would be the best acceptable solution ..
>> returning to using MAX_SKB_FRAGS ?
>> >>>
>> 
>> >> I think you need to answer several questions first:
>> >> 1. is max_slots_needed actually large enough to cover whole SKB?
>> >         No it's not if, you end up calling "get_next_rx_buffer" one or multiple
>> times when processing the SKB
>> >         you have the chance of overrunning (or be saved because prod gets
>> upped before you overrun).
>> 
>> >> 2. is the test of ring slot availability acurate?
>> >         Seems to be.
>> 
>> >> 3. is the number of ring slots consumed larger than max_slots_needed? (I
>> >>    guess the answer is yes)
>> >         Yes that was the whole point.
>> 
>> >> 4. which step in the break-down procedure causes backend to overrun
>> the
>> >>    ring?
>> >         The "get_next_rx_buffer" call .. that is not accounted for (which can be
>> called
>> >         multiple times per frag (unless some other none obvious size
>> restriction limits this
>> >         to one time per frag or one time per SKB).
>> >         In my errorneous case it is called one time, but it would be nice if there
>> would be some theoretical proof
>> >         instead of just some emperical test.
>> 
>> 
>> >> It doesn't matter how many frags your SKB has and how big a frag is. If
>> >> every step is correct then you're fine. The code you're looking at
>> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
>> >> carefully examined.
>> 
>> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
>> (and that also what i'm seeing because it doesn't happen
>> > continously.
>> 
>> > Question is shouldn't this max_needed_slots calc be reverted to what it
>> was before 3.14 and take the time in 3.15 to figure out a
>> > the theoretical max (if it can be calculated upfront) .. or another way to
>> guarantee the code is able to process the whole SKB  ?
>> 
>> >> Wei.
>> 
>> 
>> 
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 15:23
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 11:11
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >> Paul,
> >>
> >> You have been awfully silent for this whole thread while this is a
> regression
> >> caused by a patch of you
> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier
> in
> >> this thread).
> >>
> 
> > Sorry, I've been distracted...
> 
> >> The commit messages states:
> >>     "net_rx_action() is the place where we could do with an accurate
> >> predicition but,
> >>     since that has proven tricky to calculate, a cheap worse-case (but not
> too
> >> bad)
> >>     estimate is all we really need since the only thing we *must* prevent is
> >> xenvif_gop_skb()
> >>     consuming more slots than are available."
> >>
> >> Your "worst-case" calculation stated in the commit message is clearly not
> the
> >> worst case,
> >> since it doesn't take calls to "get_next_rx_buffer" into account.
> >>
> 
> > It should be taking into account the behaviour of start_new_rx_buffer(),
> which should be true if a slot is full or a frag will overflow the current slot and
> doesn't require splitting.
> > The code in net_rx_action() makes the assumption that each frag will
> require as many slots as its size requires, i.e. it assumes no packing of
> multiple frags into a single slot, so it should be a worst case.
> > Did I miss something in that logic?
> 
> Yes.
> In "xenvif_gop_skb()" this loop:
> 
>         for (i = 0; i < nr_frags; i++) {
>                 xenvif_gop_frag_copy(vif, skb, npo,
>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
>                                      skb_shinfo(skb)->frags[i].page_offset,
>                                      &head);
>         }
> 
> Is capable of using up (at least) 1 slot more than is anticipated for in
> "net_rx_action()"  by this code:
> 
>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
>                         unsigned int size;
>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>                 }
> 
> And this happens when it calls "get_next_rx_buffer()" from
> "xenvif_gop_frag_copy()" where it's breaking down the frag.
> 

The function that determines whether to consume another slot is start_new_rx_buffer() and for each frag I don't see why this would return true more than DIV_ROUND_UP(size, PAGE_SIZE) times. It may be called more times than that since the code in xenvif_gop_frag_copy() must also allow for the offset of the frag but should not return true in all cases. So, I still cannot see why a frag would ever consume more than DIV_ROUND_UP(size, PAGE_SIZE) slots.

  Paul

> Ultimately this results in bad grant reference warnings (and packets marked
> as "errors" in the interface statistics).
> 
> In my case it always seems to be a skb with 1 frag which is broken down in 5
> or 6 pieces ..
> 
> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring with
> 1 slot, but i'm not sure if that's not coincedence
> since in the code there seem to be no explicit limitation on how often this
> code path is taken. So perhaps it's implicitly limited
> since packets and frags can't be arbitrarily large in comparison with the
> page_size but that's not something i'm capable of figuring out :-)
> 
> 
> 
> >   Paul
> 
> >> Problem is that a worst case calculation would probably be reverting to
> the
> >> old calculation,
> >> and the problems this patch was trying to solve would reappear, but
> >> introducing new regressions
> >> isn't very useful either. And since it seems such a tricky and fragile thing to
> >> determine, it would
> >> probably be best to be split into a distinct function with a comment to
> explain
> >> the rationale used.
> >>
> >> Since this doesn't seem to progress very fast .. CC'ed some more folks ..
> you
> >> never know ..
> >>
> >> --
> >> Sander
> >>
> >>
> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> >>
> >>
> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> >>
> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
> >> >> [...]
> >> >>> > Yes there is only one frag .. but it seems to be much larger than
> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
> smaller
> >> bits .. hence the calculation in xenvif_rx_action determining the slots
> needed
> >> by doing:
> >> >>>
> >> >>> >                 for (i = 0; i < nr_frags; i++) {
> >> >>> >                         unsigned int size;
> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
> PAGE_SIZE);
> >> >>> >                 }
> >> >>>
> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
> >> seems involved in that ..
> >> >>>
> >> >>> Hmm looked again .. and it seems this is it .. when your frags are large
> >> enough you have the chance of running into this.
> >> >>>
> >>
> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see
> any
> >> >> problem with that implementation?
> >> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
> >> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
> >> > don't count in this possibility. So it's not the guarding of
> >> "start_new_rx_buffer" that is at fault. It's the ones early in
> >> "xenvif_rx_action".
> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
> >> calculated value that should be a "slim fit".
> >>
> >> > The problem is in determining upfront in "xenvif_rx_action" when and
> how
> >> often the "get_next_rx_buffer" path will be taken.
> >> > Unless there are other non direct restrictions (from a size point of view)
> it
> >> can be called multiple times per frag per skb.
> >>
> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
> empirical
> >> test doesn't seem to be the worst case either (i think):
> >> >>>
> >> >>> - In my case the packets that hit this only have 1 frag, but i could have
> >> had more frags.
> >> >>>   I also think you can't rule out the possibility of doing the
> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
> >> >>>   so in the worst (and perhaps even from a single frag since it's looped
> >> over a split of it in what seems PAGE_SIZE pieces.)
> >> >>>   - So an exact calculation of how much slots we are going to need for
> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
> >> unfeasible.
> >> >>>   - A worst case gamble seems impossible either .. if you take multiple
> >> frags * multiple times the "get_next_rx_buffer" ... you would probably be
> >> back at just
> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >> >>>
> >> >>> - Other thing would be checking for the available slots before doing
> the
> >> "get_next_rx_buffer" .. how ever .. we don't account for how many slots
> we
> >> still need to
> >> >>>   just process the remaining frags.
> >> >>>
> >>
> >> >> We've done a worst case estimation for whole SKB (linear area + all
> >> >> frags) already, at the very first beginning. That's what
> >> >> max_slots_needed is for.
> >>
> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
> >> seems the "get_next_rx_buffer" is necessary ..  when i make
> >> start_new_rx_buffer always return false
> >> >>>   i hit the bug below :S
> >> >>>
> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
> necessary
> >> ?
> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does
> that
> >> outweigh the additional cost of accounting
> >> >>>      of needed_slots for the frags that have yet to be processed ?
> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
> >> returning to using MAX_SKB_FRAGS ?
> >> >>>
> >>
> >> >> I think you need to answer several questions first:
> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
> multiple
> >> times when processing the SKB
> >> >         you have the chance of overrunning (or be saved because prod gets
> >> upped before you overrun).
> >>
> >> >> 2. is the test of ring slot availability acurate?
> >> >         Seems to be.
> >>
> >> >> 3. is the number of ring slots consumed larger than
> max_slots_needed? (I
> >> >>    guess the answer is yes)
> >> >         Yes that was the whole point.
> >>
> >> >> 4. which step in the break-down procedure causes backend to overrun
> >> the
> >> >>    ring?
> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which can
> be
> >> called
> >> >         multiple times per frag (unless some other none obvious size
> >> restriction limits this
> >> >         to one time per frag or one time per SKB).
> >> >         In my errorneous case it is called one time, but it would be nice if
> there
> >> would be some theoretical proof
> >> >         instead of just some emperical test.
> >>
> >>
> >> >> It doesn't matter how many frags your SKB has and how big a frag is. If
> >> >> every step is correct then you're fine. The code you're looking at
> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
> >> >> carefully examined.
> >>
> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
> >> (and that also what i'm seeing because it doesn't happen
> >> > continously.
> >>
> >> > Question is shouldn't this max_needed_slots calc be reverted to what it
> >> was before 3.14 and take the time in 3.15 to figure out a
> >> > the theoretical max (if it can be calculated upfront) .. or another way to
> >> guarantee the code is able to process the whole SKB  ?
> >>
> >> >> Wei.
> >>
> >>
> >>
> >>
> 
> 

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 15:23
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 11:11
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >> Paul,
> >>
> >> You have been awfully silent for this whole thread while this is a
> regression
> >> caused by a patch of you
> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier
> in
> >> this thread).
> >>
> 
> > Sorry, I've been distracted...
> 
> >> The commit messages states:
> >>     "net_rx_action() is the place where we could do with an accurate
> >> predicition but,
> >>     since that has proven tricky to calculate, a cheap worse-case (but not
> too
> >> bad)
> >>     estimate is all we really need since the only thing we *must* prevent is
> >> xenvif_gop_skb()
> >>     consuming more slots than are available."
> >>
> >> Your "worst-case" calculation stated in the commit message is clearly not
> the
> >> worst case,
> >> since it doesn't take calls to "get_next_rx_buffer" into account.
> >>
> 
> > It should be taking into account the behaviour of start_new_rx_buffer(),
> which should be true if a slot is full or a frag will overflow the current slot and
> doesn't require splitting.
> > The code in net_rx_action() makes the assumption that each frag will
> require as many slots as its size requires, i.e. it assumes no packing of
> multiple frags into a single slot, so it should be a worst case.
> > Did I miss something in that logic?
> 
> Yes.
> In "xenvif_gop_skb()" this loop:
> 
>         for (i = 0; i < nr_frags; i++) {
>                 xenvif_gop_frag_copy(vif, skb, npo,
>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
>                                      skb_shinfo(skb)->frags[i].page_offset,
>                                      &head);
>         }
> 
> Is capable of using up (at least) 1 slot more than is anticipated for in
> "net_rx_action()"  by this code:
> 
>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
>                         unsigned int size;
>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>                 }
> 
> And this happens when it calls "get_next_rx_buffer()" from
> "xenvif_gop_frag_copy()" where it's breaking down the frag.
> 

The function that determines whether to consume another slot is start_new_rx_buffer() and for each frag I don't see why this would return true more than DIV_ROUND_UP(size, PAGE_SIZE) times. It may be called more times than that since the code in xenvif_gop_frag_copy() must also allow for the offset of the frag but should not return true in all cases. So, I still cannot see why a frag would ever consume more than DIV_ROUND_UP(size, PAGE_SIZE) slots.

  Paul

> Ultimately this results in bad grant reference warnings (and packets marked
> as "errors" in the interface statistics).
> 
> In my case it always seems to be a skb with 1 frag which is broken down in 5
> or 6 pieces ..
> 
> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring with
> 1 slot, but i'm not sure if that's not coincedence
> since in the code there seem to be no explicit limitation on how often this
> code path is taken. So perhaps it's implicitly limited
> since packets and frags can't be arbitrarily large in comparison with the
> page_size but that's not something i'm capable of figuring out :-)
> 
> 
> 
> >   Paul
> 
> >> Problem is that a worst case calculation would probably be reverting to
> the
> >> old calculation,
> >> and the problems this patch was trying to solve would reappear, but
> >> introducing new regressions
> >> isn't very useful either. And since it seems such a tricky and fragile thing to
> >> determine, it would
> >> probably be best to be split into a distinct function with a comment to
> explain
> >> the rationale used.
> >>
> >> Since this doesn't seem to progress very fast .. CC'ed some more folks ..
> you
> >> never know ..
> >>
> >> --
> >> Sander
> >>
> >>
> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> >>
> >>
> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> >>
> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
> >> >> [...]
> >> >>> > Yes there is only one frag .. but it seems to be much larger than
> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
> smaller
> >> bits .. hence the calculation in xenvif_rx_action determining the slots
> needed
> >> by doing:
> >> >>>
> >> >>> >                 for (i = 0; i < nr_frags; i++) {
> >> >>> >                         unsigned int size;
> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
> PAGE_SIZE);
> >> >>> >                 }
> >> >>>
> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
> >> seems involved in that ..
> >> >>>
> >> >>> Hmm looked again .. and it seems this is it .. when your frags are large
> >> enough you have the chance of running into this.
> >> >>>
> >>
> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see
> any
> >> >> problem with that implementation?
> >> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
> >> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
> >> > don't count in this possibility. So it's not the guarding of
> >> "start_new_rx_buffer" that is at fault. It's the ones early in
> >> "xenvif_rx_action".
> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
> >> calculated value that should be a "slim fit".
> >>
> >> > The problem is in determining upfront in "xenvif_rx_action" when and
> how
> >> often the "get_next_rx_buffer" path will be taken.
> >> > Unless there are other non direct restrictions (from a size point of view)
> it
> >> can be called multiple times per frag per skb.
> >>
> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
> empirical
> >> test doesn't seem to be the worst case either (i think):
> >> >>>
> >> >>> - In my case the packets that hit this only have 1 frag, but i could have
> >> had more frags.
> >> >>>   I also think you can't rule out the possibility of doing the
> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
> >> >>>   so in the worst (and perhaps even from a single frag since it's looped
> >> over a split of it in what seems PAGE_SIZE pieces.)
> >> >>>   - So an exact calculation of how much slots we are going to need for
> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
> >> unfeasible.
> >> >>>   - A worst case gamble seems impossible either .. if you take multiple
> >> frags * multiple times the "get_next_rx_buffer" ... you would probably be
> >> back at just
> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >> >>>
> >> >>> - Other thing would be checking for the available slots before doing
> the
> >> "get_next_rx_buffer" .. how ever .. we don't account for how many slots
> we
> >> still need to
> >> >>>   just process the remaining frags.
> >> >>>
> >>
> >> >> We've done a worst case estimation for whole SKB (linear area + all
> >> >> frags) already, at the very first beginning. That's what
> >> >> max_slots_needed is for.
> >>
> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
> >> seems the "get_next_rx_buffer" is necessary ..  when i make
> >> start_new_rx_buffer always return false
> >> >>>   i hit the bug below :S
> >> >>>
> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
> necessary
> >> ?
> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does
> that
> >> outweigh the additional cost of accounting
> >> >>>      of needed_slots for the frags that have yet to be processed ?
> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
> >> returning to using MAX_SKB_FRAGS ?
> >> >>>
> >>
> >> >> I think you need to answer several questions first:
> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
> multiple
> >> times when processing the SKB
> >> >         you have the chance of overrunning (or be saved because prod gets
> >> upped before you overrun).
> >>
> >> >> 2. is the test of ring slot availability acurate?
> >> >         Seems to be.
> >>
> >> >> 3. is the number of ring slots consumed larger than
> max_slots_needed? (I
> >> >>    guess the answer is yes)
> >> >         Yes that was the whole point.
> >>
> >> >> 4. which step in the break-down procedure causes backend to overrun
> >> the
> >> >>    ring?
> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which can
> be
> >> called
> >> >         multiple times per frag (unless some other none obvious size
> >> restriction limits this
> >> >         to one time per frag or one time per SKB).
> >> >         In my errorneous case it is called one time, but it would be nice if
> there
> >> would be some theoretical proof
> >> >         instead of just some emperical test.
> >>
> >>
> >> >> It doesn't matter how many frags your SKB has and how big a frag is. If
> >> >> every step is correct then you're fine. The code you're looking at
> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
> >> >> carefully examined.
> >>
> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
> >> (and that also what i'm seeing because it doesn't happen
> >> > continously.
> >>
> >> > Question is shouldn't this max_needed_slots calc be reverted to what it
> >> was before 3.14 and take the time in 3.15 to figure out a
> >> > the theoretical max (if it can be calculated upfront) .. or another way to
> >> guarantee the code is able to process the whole SKB  ?
> >>
> >> >> Wei.
> >>
> >>
> >>
> >>
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 4:50:30 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 15:23
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 11:11
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >> Paul,
>> >>
>> >> You have been awfully silent for this whole thread while this is a
>> regression
>> >> caused by a patch of you
>> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier
>> in
>> >> this thread).
>> >>
>> 
>> > Sorry, I've been distracted...
>> 
>> >> The commit messages states:
>> >>     "net_rx_action() is the place where we could do with an accurate
>> >> predicition but,
>> >>     since that has proven tricky to calculate, a cheap worse-case (but not
>> too
>> >> bad)
>> >>     estimate is all we really need since the only thing we *must* prevent is
>> >> xenvif_gop_skb()
>> >>     consuming more slots than are available."
>> >>
>> >> Your "worst-case" calculation stated in the commit message is clearly not
>> the
>> >> worst case,
>> >> since it doesn't take calls to "get_next_rx_buffer" into account.
>> >>
>> 
>> > It should be taking into account the behaviour of start_new_rx_buffer(),
>> which should be true if a slot is full or a frag will overflow the current slot and
>> doesn't require splitting.
>> > The code in net_rx_action() makes the assumption that each frag will
>> require as many slots as its size requires, i.e. it assumes no packing of
>> multiple frags into a single slot, so it should be a worst case.
>> > Did I miss something in that logic?
>> 
>> Yes.
>> In "xenvif_gop_skb()" this loop:
>> 
>>         for (i = 0; i < nr_frags; i++) {
>>                 xenvif_gop_frag_copy(vif, skb, npo,
>>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
>>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
>>                                      skb_shinfo(skb)->frags[i].page_offset,
>>                                      &head);
>>         }
>> 
>> Is capable of using up (at least) 1 slot more than is anticipated for in
>> "net_rx_action()"  by this code:
>> 
>>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
>>                         unsigned int size;
>>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>>                 }
>> 
>> And this happens when it calls "get_next_rx_buffer()" from
>> "xenvif_gop_frag_copy()" where it's breaking down the frag.
>> 

> The function that determines whether to consume another slot is start_new_rx_buffer() and for each frag I don't see why this would return true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
> It may be called more times than that since the code in xenvif_gop_frag_copy() must also allow for the offset of the frag but should not return true in all cases.
> So, I still cannot see why a frag would ever consume more than DIV_ROUND_UP(size, PAGE_SIZE) slots.

Well here a case were a frag is broken down in 2 pieces:

[ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo->meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096 bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring->req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
[ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring->req_event:2105868 skb->len:66 skb->data_len:0
[ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring->req_event:2105868, reserved_slots_left:0
[ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868 reserved_slots_left:-1
[ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring->req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
[ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096 bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring->req_event:2105868 gso_gaps:0 estimated_slots_needed:4 reserved_slots_left:-1
[ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo->meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1 reserved_slots_left:-1
[ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:3
[ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco->meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2 reserved_slots_left:-1

- When processing an SKB we end up in "xenvif_gop_frag_copy" while prod == cons ... but we still have bytes and size left ..
- start_new_rx_buffer() has returned true ..
- so we end up in get_next_rx_buffer
- this does a RING_GET_REQUEST and ups cons ..
- and we end up with a bad grant reference.

Sometimes we are saved by the bell .. since additional slots have become free (you see cons become > prod in "get_next_rx_buffer" but shortly after that prod is increased ..
just in time to not cause a overrun).

If you need additional / other info, please cook up a debug patch with what you need.

--
Sander






>   Paul

>> Ultimately this results in bad grant reference warnings (and packets marked
>> as "errors" in the interface statistics).
>> 
>> In my case it always seems to be a skb with 1 frag which is broken down in 5
>> or 6 pieces ..
>> 
>> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring with
>> 1 slot, but i'm not sure if that's not coincedence
>> since in the code there seem to be no explicit limitation on how often this
>> code path is taken. So perhaps it's implicitly limited
>> since packets and frags can't be arbitrarily large in comparison with the
>> page_size but that's not something i'm capable of figuring out :-)
>> 
>> 
>> 
>> >   Paul
>> 
>> >> Problem is that a worst case calculation would probably be reverting to
>> the
>> >> old calculation,
>> >> and the problems this patch was trying to solve would reappear, but
>> >> introducing new regressions
>> >> isn't very useful either. And since it seems such a tricky and fragile thing to
>> >> determine, it would
>> >> probably be best to be split into a distinct function with a comment to
>> explain
>> >> the rationale used.
>> >>
>> >> Since this doesn't seem to progress very fast .. CC'ed some more folks ..
>> you
>> >> never know ..
>> >>
>> >> --
>> >> Sander
>> >>
>> >>
>> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
>> >>
>> >>
>> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
>> >>
>> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
>> >> >> [...]
>> >> >>> > Yes there is only one frag .. but it seems to be much larger than
>> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
>> smaller
>> >> bits .. hence the calculation in xenvif_rx_action determining the slots
>> needed
>> >> by doing:
>> >> >>>
>> >> >>> >                 for (i = 0; i < nr_frags; i++) {
>> >> >>> >                         unsigned int size;
>> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
>> PAGE_SIZE);
>> >> >>> >                 }
>> >> >>>
>> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
>> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
>> >> seems involved in that ..
>> >> >>>
>> >> >>> Hmm looked again .. and it seems this is it .. when your frags are large
>> >> enough you have the chance of running into this.
>> >> >>>
>> >>
>> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see
>> any
>> >> >> problem with that implementation?
>> >> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
>> >> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
>> >> > don't count in this possibility. So it's not the guarding of
>> >> "start_new_rx_buffer" that is at fault. It's the ones early in
>> >> "xenvif_rx_action".
>> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
>> >> calculated value that should be a "slim fit".
>> >>
>> >> > The problem is in determining upfront in "xenvif_rx_action" when and
>> how
>> >> often the "get_next_rx_buffer" path will be taken.
>> >> > Unless there are other non direct restrictions (from a size point of view)
>> it
>> >> can be called multiple times per frag per skb.
>> >>
>> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
>> empirical
>> >> test doesn't seem to be the worst case either (i think):
>> >> >>>
>> >> >>> - In my case the packets that hit this only have 1 frag, but i could have
>> >> had more frags.
>> >> >>>   I also think you can't rule out the possibility of doing the
>> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
>> >> >>>   so in the worst (and perhaps even from a single frag since it's looped
>> >> over a split of it in what seems PAGE_SIZE pieces.)
>> >> >>>   - So an exact calculation of how much slots we are going to need for
>> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
>> >> unfeasible.
>> >> >>>   - A worst case gamble seems impossible either .. if you take multiple
>> >> frags * multiple times the "get_next_rx_buffer" ... you would probably be
>> >> back at just
>> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
>> >> >>>
>> >> >>> - Other thing would be checking for the available slots before doing
>> the
>> >> "get_next_rx_buffer" .. how ever .. we don't account for how many slots
>> we
>> >> still need to
>> >> >>>   just process the remaining frags.
>> >> >>>
>> >>
>> >> >> We've done a worst case estimation for whole SKB (linear area + all
>> >> >> frags) already, at the very first beginning. That's what
>> >> >> max_slots_needed is for.
>> >>
>> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
>> >> seems the "get_next_rx_buffer" is necessary ..  when i make
>> >> start_new_rx_buffer always return false
>> >> >>>   i hit the bug below :S
>> >> >>>
>> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
>> necessary
>> >> ?
>> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does
>> that
>> >> outweigh the additional cost of accounting
>> >> >>>      of needed_slots for the frags that have yet to be processed ?
>> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
>> >> returning to using MAX_SKB_FRAGS ?
>> >> >>>
>> >>
>> >> >> I think you need to answer several questions first:
>> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
>> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
>> multiple
>> >> times when processing the SKB
>> >> >         you have the chance of overrunning (or be saved because prod gets
>> >> upped before you overrun).
>> >>
>> >> >> 2. is the test of ring slot availability acurate?
>> >> >         Seems to be.
>> >>
>> >> >> 3. is the number of ring slots consumed larger than
>> max_slots_needed? (I
>> >> >>    guess the answer is yes)
>> >> >         Yes that was the whole point.
>> >>
>> >> >> 4. which step in the break-down procedure causes backend to overrun
>> >> the
>> >> >>    ring?
>> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which can
>> be
>> >> called
>> >> >         multiple times per frag (unless some other none obvious size
>> >> restriction limits this
>> >> >         to one time per frag or one time per SKB).
>> >> >         In my errorneous case it is called one time, but it would be nice if
>> there
>> >> would be some theoretical proof
>> >> >         instead of just some emperical test.
>> >>
>> >>
>> >> >> It doesn't matter how many frags your SKB has and how big a frag is. If
>> >> >> every step is correct then you're fine. The code you're looking at
>> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
>> >> >> carefully examined.
>> >>
>> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
>> >> (and that also what i'm seeing because it doesn't happen
>> >> > continously.
>> >>
>> >> > Question is shouldn't this max_needed_slots calc be reverted to what it
>> >> was before 3.14 and take the time in 3.15 to figure out a
>> >> > the theoretical max (if it can be calculated upfront) .. or another way to
>> >> guarantee the code is able to process the whole SKB  ?
>> >>
>> >> >> Wei.
>> >>
>> >>
>> >>
>> >>
>> 
>> 

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 4:50:30 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 15:23
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 11:11
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >> Paul,
>> >>
>> >> You have been awfully silent for this whole thread while this is a
>> regression
>> >> caused by a patch of you
>> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier
>> in
>> >> this thread).
>> >>
>> 
>> > Sorry, I've been distracted...
>> 
>> >> The commit messages states:
>> >>     "net_rx_action() is the place where we could do with an accurate
>> >> predicition but,
>> >>     since that has proven tricky to calculate, a cheap worse-case (but not
>> too
>> >> bad)
>> >>     estimate is all we really need since the only thing we *must* prevent is
>> >> xenvif_gop_skb()
>> >>     consuming more slots than are available."
>> >>
>> >> Your "worst-case" calculation stated in the commit message is clearly not
>> the
>> >> worst case,
>> >> since it doesn't take calls to "get_next_rx_buffer" into account.
>> >>
>> 
>> > It should be taking into account the behaviour of start_new_rx_buffer(),
>> which should be true if a slot is full or a frag will overflow the current slot and
>> doesn't require splitting.
>> > The code in net_rx_action() makes the assumption that each frag will
>> require as many slots as its size requires, i.e. it assumes no packing of
>> multiple frags into a single slot, so it should be a worst case.
>> > Did I miss something in that logic?
>> 
>> Yes.
>> In "xenvif_gop_skb()" this loop:
>> 
>>         for (i = 0; i < nr_frags; i++) {
>>                 xenvif_gop_frag_copy(vif, skb, npo,
>>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
>>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
>>                                      skb_shinfo(skb)->frags[i].page_offset,
>>                                      &head);
>>         }
>> 
>> Is capable of using up (at least) 1 slot more than is anticipated for in
>> "net_rx_action()"  by this code:
>> 
>>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
>>                         unsigned int size;
>>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>>                 }
>> 
>> And this happens when it calls "get_next_rx_buffer()" from
>> "xenvif_gop_frag_copy()" where it's breaking down the frag.
>> 

> The function that determines whether to consume another slot is start_new_rx_buffer() and for each frag I don't see why this would return true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
> It may be called more times than that since the code in xenvif_gop_frag_copy() must also allow for the offset of the frag but should not return true in all cases.
> So, I still cannot see why a frag would ever consume more than DIV_ROUND_UP(size, PAGE_SIZE) slots.

Well here a case were a frag is broken down in 2 pieces:

[ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo->meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096 bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring->req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
[ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !  min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring->req_event:2105868 skb->len:66 skb->data_len:0
[ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo->meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring->req_event:2105868, reserved_slots_left:0
[ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo->meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868 reserved_slots_left:-1
[ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo->meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096 bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring->req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
[ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096 bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring->req_event:2105868 gso_gaps:0 estimated_slots_needed:4 reserved_slots_left:-1
[ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo->meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1 reserved_slots_left:-1
[ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:3
[ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco->meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2 reserved_slots_left:-1

- When processing an SKB we end up in "xenvif_gop_frag_copy" while prod == cons ... but we still have bytes and size left ..
- start_new_rx_buffer() has returned true ..
- so we end up in get_next_rx_buffer
- this does a RING_GET_REQUEST and ups cons ..
- and we end up with a bad grant reference.

Sometimes we are saved by the bell .. since additional slots have become free (you see cons become > prod in "get_next_rx_buffer" but shortly after that prod is increased ..
just in time to not cause a overrun).

If you need additional / other info, please cook up a debug patch with what you need.

--
Sander






>   Paul

>> Ultimately this results in bad grant reference warnings (and packets marked
>> as "errors" in the interface statistics).
>> 
>> In my case it always seems to be a skb with 1 frag which is broken down in 5
>> or 6 pieces ..
>> 
>> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring with
>> 1 slot, but i'm not sure if that's not coincedence
>> since in the code there seem to be no explicit limitation on how often this
>> code path is taken. So perhaps it's implicitly limited
>> since packets and frags can't be arbitrarily large in comparison with the
>> page_size but that's not something i'm capable of figuring out :-)
>> 
>> 
>> 
>> >   Paul
>> 
>> >> Problem is that a worst case calculation would probably be reverting to
>> the
>> >> old calculation,
>> >> and the problems this patch was trying to solve would reappear, but
>> >> introducing new regressions
>> >> isn't very useful either. And since it seems such a tricky and fragile thing to
>> >> determine, it would
>> >> probably be best to be split into a distinct function with a comment to
>> explain
>> >> the rationale used.
>> >>
>> >> Since this doesn't seem to progress very fast .. CC'ed some more folks ..
>> you
>> >> never know ..
>> >>
>> >> --
>> >> Sander
>> >>
>> >>
>> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
>> >>
>> >>
>> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
>> >>
>> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
>> >> >> [...]
>> >> >>> > Yes there is only one frag .. but it seems to be much larger than
>> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
>> smaller
>> >> bits .. hence the calculation in xenvif_rx_action determining the slots
>> needed
>> >> by doing:
>> >> >>>
>> >> >>> >                 for (i = 0; i < nr_frags; i++) {
>> >> >>> >                         unsigned int size;
>> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
>> PAGE_SIZE);
>> >> >>> >                 }
>> >> >>>
>> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
>> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
>> >> seems involved in that ..
>> >> >>>
>> >> >>> Hmm looked again .. and it seems this is it .. when your frags are large
>> >> enough you have the chance of running into this.
>> >> >>>
>> >>
>> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see
>> any
>> >> >> problem with that implementation?
>> >> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
>> >> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
>> >> > don't count in this possibility. So it's not the guarding of
>> >> "start_new_rx_buffer" that is at fault. It's the ones early in
>> >> "xenvif_rx_action".
>> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
>> >> calculated value that should be a "slim fit".
>> >>
>> >> > The problem is in determining upfront in "xenvif_rx_action" when and
>> how
>> >> often the "get_next_rx_buffer" path will be taken.
>> >> > Unless there are other non direct restrictions (from a size point of view)
>> it
>> >> can be called multiple times per frag per skb.
>> >>
>> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
>> empirical
>> >> test doesn't seem to be the worst case either (i think):
>> >> >>>
>> >> >>> - In my case the packets that hit this only have 1 frag, but i could have
>> >> had more frags.
>> >> >>>   I also think you can't rule out the possibility of doing the
>> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
>> >> >>>   so in the worst (and perhaps even from a single frag since it's looped
>> >> over a split of it in what seems PAGE_SIZE pieces.)
>> >> >>>   - So an exact calculation of how much slots we are going to need for
>> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
>> >> unfeasible.
>> >> >>>   - A worst case gamble seems impossible either .. if you take multiple
>> >> frags * multiple times the "get_next_rx_buffer" ... you would probably be
>> >> back at just
>> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
>> >> >>>
>> >> >>> - Other thing would be checking for the available slots before doing
>> the
>> >> "get_next_rx_buffer" .. how ever .. we don't account for how many slots
>> we
>> >> still need to
>> >> >>>   just process the remaining frags.
>> >> >>>
>> >>
>> >> >> We've done a worst case estimation for whole SKB (linear area + all
>> >> >> frags) already, at the very first beginning. That's what
>> >> >> max_slots_needed is for.
>> >>
>> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
>> >> seems the "get_next_rx_buffer" is necessary ..  when i make
>> >> start_new_rx_buffer always return false
>> >> >>>   i hit the bug below :S
>> >> >>>
>> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
>> necessary
>> >> ?
>> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does
>> that
>> >> outweigh the additional cost of accounting
>> >> >>>      of needed_slots for the frags that have yet to be processed ?
>> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
>> >> returning to using MAX_SKB_FRAGS ?
>> >> >>>
>> >>
>> >> >> I think you need to answer several questions first:
>> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
>> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
>> multiple
>> >> times when processing the SKB
>> >> >         you have the chance of overrunning (or be saved because prod gets
>> >> upped before you overrun).
>> >>
>> >> >> 2. is the test of ring slot availability acurate?
>> >> >         Seems to be.
>> >>
>> >> >> 3. is the number of ring slots consumed larger than
>> max_slots_needed? (I
>> >> >>    guess the answer is yes)
>> >> >         Yes that was the whole point.
>> >>
>> >> >> 4. which step in the break-down procedure causes backend to overrun
>> >> the
>> >> >>    ring?
>> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which can
>> be
>> >> called
>> >> >         multiple times per frag (unless some other none obvious size
>> >> restriction limits this
>> >> >         to one time per frag or one time per SKB).
>> >> >         In my errorneous case it is called one time, but it would be nice if
>> there
>> >> would be some theoretical proof
>> >> >         instead of just some emperical test.
>> >>
>> >>
>> >> >> It doesn't matter how many frags your SKB has and how big a frag is. If
>> >> >> every step is correct then you're fine. The code you're looking at
>> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
>> >> >> carefully examined.
>> >>
>> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
>> >> (and that also what i'm seeing because it doesn't happen
>> >> > continously.
>> >>
>> >> > Question is shouldn't this max_needed_slots calc be reverted to what it
>> >> was before 3.14 and take the time in 3.15 to figure out a
>> >> > the theoretical max (if it can be calculated upfront) .. or another way to
>> >> guarantee the code is able to process the whole SKB  ?
>> >>
>> >> >> Wei.
>> >>
>> >>
>> >>
>> >>
>> 
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 16:07
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 15:23
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >>
> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
> >>
> >> >> -----Original Message-----
> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> Sent: 26 March 2014 11:11
> >> >> To: Paul Durrant
> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> >> kernel; netdev@vger.kernel.org
> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> >> troubles "bisected"
> >> >>
> >> >> Paul,
> >> >>
> >> >> You have been awfully silent for this whole thread while this is a
> >> regression
> >> >> caused by a patch of you
> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
> earlier
> >> in
> >> >> this thread).
> >> >>
> >>
> >> > Sorry, I've been distracted...
> >>
> >> >> The commit messages states:
> >> >>     "net_rx_action() is the place where we could do with an accurate
> >> >> predicition but,
> >> >>     since that has proven tricky to calculate, a cheap worse-case (but not
> >> too
> >> >> bad)
> >> >>     estimate is all we really need since the only thing we *must* prevent
> is
> >> >> xenvif_gop_skb()
> >> >>     consuming more slots than are available."
> >> >>
> >> >> Your "worst-case" calculation stated in the commit message is clearly
> not
> >> the
> >> >> worst case,
> >> >> since it doesn't take calls to "get_next_rx_buffer" into account.
> >> >>
> >>
> >> > It should be taking into account the behaviour of
> start_new_rx_buffer(),
> >> which should be true if a slot is full or a frag will overflow the current slot
> and
> >> doesn't require splitting.
> >> > The code in net_rx_action() makes the assumption that each frag will
> >> require as many slots as its size requires, i.e. it assumes no packing of
> >> multiple frags into a single slot, so it should be a worst case.
> >> > Did I miss something in that logic?
> >>
> >> Yes.
> >> In "xenvif_gop_skb()" this loop:
> >>
> >>         for (i = 0; i < nr_frags; i++) {
> >>                 xenvif_gop_frag_copy(vif, skb, npo,
> >>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
> >>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
> >>                                      skb_shinfo(skb)->frags[i].page_offset,
> >>                                      &head);
> >>         }
> >>
> >> Is capable of using up (at least) 1 slot more than is anticipated for in
> >> "net_rx_action()"  by this code:
> >>
> >>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> >>                         unsigned int size;
> >>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >>                 }
> >>
> >> And this happens when it calls "get_next_rx_buffer()" from
> >> "xenvif_gop_frag_copy()" where it's breaking down the frag.
> >>
> 
> > The function that determines whether to consume another slot is
> start_new_rx_buffer() and for each frag I don't see why this would return
> true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
> > It may be called more times than that since the code in
> xenvif_gop_frag_copy() must also allow for the offset of the frag but should
> not return true in all cases.
> > So, I still cannot see why a frag would ever consume more than
> DIV_ROUND_UP(size, PAGE_SIZE) slots.
> 
> Well here a case were a frag is broken down in 2 pieces:
> 
> [ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo-
> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
> npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096
> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
> [ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !
> min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif-
> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring-
> >req_event:2105868 skb->len:66 skb->data_len:0
> [ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo-
> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
> vif->rx.sring->req_event:2105868, reserved_slots_left:0
> [ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo-
> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868
> reserved_slots_left:-1
> [ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo-
> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096
> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo-
> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
> reserved_slots_left:-1
> [ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo-
> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1
> reserved_slots_left:-1
> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> used_fragloop:3
> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> reserved_slots_left:-1
> 
> - When processing an SKB we end up in "xenvif_gop_frag_copy" while prod
> == cons ... but we still have bytes and size left ..
> - start_new_rx_buffer() has returned true ..
> - so we end up in get_next_rx_buffer
> - this does a RING_GET_REQUEST and ups cons ..
> - and we end up with a bad grant reference.
> 
> Sometimes we are saved by the bell .. since additional slots have become
> free (you see cons become > prod in "get_next_rx_buffer" but shortly after
> that prod is increased ..
> just in time to not cause a overrun).
> 

Ah, but hang on... There's a BUG_ON meta_slots_used > max_slots_needed, so if we are overflowing the worst-case calculation then why is that BUG_ON not firing?

  Paul

> If you need additional / other info, please cook up a debug patch with what
> you need.
> 
> --
> Sander
> 
> 
> 
> 
> 
> 
> >   Paul
> 
> >> Ultimately this results in bad grant reference warnings (and packets
> marked
> >> as "errors" in the interface statistics).
> >>
> >> In my case it always seems to be a skb with 1 frag which is broken down in
> 5
> >> or 6 pieces ..
> >>
> >> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring
> with
> >> 1 slot, but i'm not sure if that's not coincedence
> >> since in the code there seem to be no explicit limitation on how often this
> >> code path is taken. So perhaps it's implicitly limited
> >> since packets and frags can't be arbitrarily large in comparison with the
> >> page_size but that's not something i'm capable of figuring out :-)
> >>
> >>
> >>
> >> >   Paul
> >>
> >> >> Problem is that a worst case calculation would probably be reverting to
> >> the
> >> >> old calculation,
> >> >> and the problems this patch was trying to solve would reappear, but
> >> >> introducing new regressions
> >> >> isn't very useful either. And since it seems such a tricky and fragile
> thing to
> >> >> determine, it would
> >> >> probably be best to be split into a distinct function with a comment to
> >> explain
> >> >> the rationale used.
> >> >>
> >> >> Since this doesn't seem to progress very fast .. CC'ed some more folks
> ..
> >> you
> >> >> never know ..
> >> >>
> >> >> --
> >> >> Sander
> >> >>
> >> >>
> >> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> >> >>
> >> >>
> >> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> >> >>
> >> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom
> wrote:
> >> >> >> [...]
> >> >> >>> > Yes there is only one frag .. but it seems to be much larger than
> >> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
> >> smaller
> >> >> bits .. hence the calculation in xenvif_rx_action determining the slots
> >> needed
> >> >> by doing:
> >> >> >>>
> >> >> >>> >                 for (i = 0; i < nr_frags; i++) {
> >> >> >>> >                         unsigned int size;
> >> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
> >> PAGE_SIZE);
> >> >> >>> >                 }
> >> >> >>>
> >> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing
> one
> >> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
> >> >> seems involved in that ..
> >> >> >>>
> >> >> >>> Hmm looked again .. and it seems this is it .. when your frags are
> large
> >> >> enough you have the chance of running into this.
> >> >> >>>
> >> >>
> >> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you
> see
> >> any
> >> >> >> problem with that implementation?
> >> >> > In general no, but "get_next_rx_buffer" up's cons .. and the
> calculations
> >> >> done in "xenvif_rx_action" for max_slots_needed to prevent the
> overrun
> >> >> > don't count in this possibility. So it's not the guarding of
> >> >> "start_new_rx_buffer" that is at fault. It's the ones early in
> >> >> "xenvif_rx_action".
> >> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS
> to a
> >> >> calculated value that should be a "slim fit".
> >> >>
> >> >> > The problem is in determining upfront in "xenvif_rx_action" when
> and
> >> how
> >> >> often the "get_next_rx_buffer" path will be taken.
> >> >> > Unless there are other non direct restrictions (from a size point of
> view)
> >> it
> >> >> can be called multiple times per frag per skb.
> >> >>
> >> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
> >> empirical
> >> >> test doesn't seem to be the worst case either (i think):
> >> >> >>>
> >> >> >>> - In my case the packets that hit this only have 1 frag, but i could
> have
> >> >> had more frags.
> >> >> >>>   I also think you can't rule out the possibility of doing the
> >> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
> >> >> >>>   so in the worst (and perhaps even from a single frag since it's
> looped
> >> >> over a split of it in what seems PAGE_SIZE pieces.)
> >> >> >>>   - So an exact calculation of how much slots we are going to need
> for
> >> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
> >> >> unfeasible.
> >> >> >>>   - A worst case gamble seems impossible either .. if you take
> multiple
> >> >> frags * multiple times the "get_next_rx_buffer" ... you would probably
> be
> >> >> back at just
> >> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >> >> >>>
> >> >> >>> - Other thing would be checking for the available slots before
> doing
> >> the
> >> >> "get_next_rx_buffer" .. how ever .. we don't account for how many
> slots
> >> we
> >> >> still need to
> >> >> >>>   just process the remaining frags.
> >> >> >>>
> >> >>
> >> >> >> We've done a worst case estimation for whole SKB (linear area + all
> >> >> >> frags) already, at the very first beginning. That's what
> >> >> >> max_slots_needed is for.
> >> >>
> >> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but
> it
> >> >> seems the "get_next_rx_buffer" is necessary ..  when i make
> >> >> start_new_rx_buffer always return false
> >> >> >>>   i hit the bug below :S
> >> >> >>>
> >> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
> >> necessary
> >> >> ?
> >> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and
> does
> >> that
> >> >> outweigh the additional cost of accounting
> >> >> >>>      of needed_slots for the frags that have yet to be processed ?
> >> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
> >> >> returning to using MAX_SKB_FRAGS ?
> >> >> >>>
> >> >>
> >> >> >> I think you need to answer several questions first:
> >> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
> >> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
> >> multiple
> >> >> times when processing the SKB
> >> >> >         you have the chance of overrunning (or be saved because prod
> gets
> >> >> upped before you overrun).
> >> >>
> >> >> >> 2. is the test of ring slot availability acurate?
> >> >> >         Seems to be.
> >> >>
> >> >> >> 3. is the number of ring slots consumed larger than
> >> max_slots_needed? (I
> >> >> >>    guess the answer is yes)
> >> >> >         Yes that was the whole point.
> >> >>
> >> >> >> 4. which step in the break-down procedure causes backend to
> overrun
> >> >> the
> >> >> >>    ring?
> >> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which
> can
> >> be
> >> >> called
> >> >> >         multiple times per frag (unless some other none obvious size
> >> >> restriction limits this
> >> >> >         to one time per frag or one time per SKB).
> >> >> >         In my errorneous case it is called one time, but it would be nice if
> >> there
> >> >> would be some theoretical proof
> >> >> >         instead of just some emperical test.
> >> >>
> >> >>
> >> >> >> It doesn't matter how many frags your SKB has and how big a frag
> is. If
> >> >> >> every step is correct then you're fine. The code you're looking at
> >> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to
> be
> >> >> >> carefully examined.
> >> >>
> >> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases
> ..
> >> >> (and that also what i'm seeing because it doesn't happen
> >> >> > continously.
> >> >>
> >> >> > Question is shouldn't this max_needed_slots calc be reverted to
> what it
> >> >> was before 3.14 and take the time in 3.15 to figure out a
> >> >> > the theoretical max (if it can be calculated upfront) .. or another way
> to
> >> >> guarantee the code is able to process the whole SKB  ?
> >> >>
> >> >> >> Wei.
> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
> 
> 

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 16:07
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 15:23
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >>
> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
> >>
> >> >> -----Original Message-----
> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> Sent: 26 March 2014 11:11
> >> >> To: Paul Durrant
> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> >> kernel; netdev@vger.kernel.org
> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> >> troubles "bisected"
> >> >>
> >> >> Paul,
> >> >>
> >> >> You have been awfully silent for this whole thread while this is a
> >> regression
> >> >> caused by a patch of you
> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
> earlier
> >> in
> >> >> this thread).
> >> >>
> >>
> >> > Sorry, I've been distracted...
> >>
> >> >> The commit messages states:
> >> >>     "net_rx_action() is the place where we could do with an accurate
> >> >> predicition but,
> >> >>     since that has proven tricky to calculate, a cheap worse-case (but not
> >> too
> >> >> bad)
> >> >>     estimate is all we really need since the only thing we *must* prevent
> is
> >> >> xenvif_gop_skb()
> >> >>     consuming more slots than are available."
> >> >>
> >> >> Your "worst-case" calculation stated in the commit message is clearly
> not
> >> the
> >> >> worst case,
> >> >> since it doesn't take calls to "get_next_rx_buffer" into account.
> >> >>
> >>
> >> > It should be taking into account the behaviour of
> start_new_rx_buffer(),
> >> which should be true if a slot is full or a frag will overflow the current slot
> and
> >> doesn't require splitting.
> >> > The code in net_rx_action() makes the assumption that each frag will
> >> require as many slots as its size requires, i.e. it assumes no packing of
> >> multiple frags into a single slot, so it should be a worst case.
> >> > Did I miss something in that logic?
> >>
> >> Yes.
> >> In "xenvif_gop_skb()" this loop:
> >>
> >>         for (i = 0; i < nr_frags; i++) {
> >>                 xenvif_gop_frag_copy(vif, skb, npo,
> >>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
> >>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
> >>                                      skb_shinfo(skb)->frags[i].page_offset,
> >>                                      &head);
> >>         }
> >>
> >> Is capable of using up (at least) 1 slot more than is anticipated for in
> >> "net_rx_action()"  by this code:
> >>
> >>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> >>                         unsigned int size;
> >>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >>                 }
> >>
> >> And this happens when it calls "get_next_rx_buffer()" from
> >> "xenvif_gop_frag_copy()" where it's breaking down the frag.
> >>
> 
> > The function that determines whether to consume another slot is
> start_new_rx_buffer() and for each frag I don't see why this would return
> true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
> > It may be called more times than that since the code in
> xenvif_gop_frag_copy() must also allow for the offset of the frag but should
> not return true in all cases.
> > So, I still cannot see why a frag would ever consume more than
> DIV_ROUND_UP(size, PAGE_SIZE) slots.
> 
> Well here a case were a frag is broken down in 2 pieces:
> 
> [ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo-
> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
> npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096
> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
> [ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !
> min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif-
> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring-
> >req_event:2105868 skb->len:66 skb->data_len:0
> [ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo-
> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
> vif->rx.sring->req_event:2105868, reserved_slots_left:0
> [ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo-
> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868
> reserved_slots_left:-1
> [ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo-
> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096
> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo-
> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
> reserved_slots_left:-1
> [ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo-
> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1
> reserved_slots_left:-1
> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> used_fragloop:3
> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> reserved_slots_left:-1
> 
> - When processing an SKB we end up in "xenvif_gop_frag_copy" while prod
> == cons ... but we still have bytes and size left ..
> - start_new_rx_buffer() has returned true ..
> - so we end up in get_next_rx_buffer
> - this does a RING_GET_REQUEST and ups cons ..
> - and we end up with a bad grant reference.
> 
> Sometimes we are saved by the bell .. since additional slots have become
> free (you see cons become > prod in "get_next_rx_buffer" but shortly after
> that prod is increased ..
> just in time to not cause a overrun).
> 

Ah, but hang on... There's a BUG_ON meta_slots_used > max_slots_needed, so if we are overflowing the worst-case calculation then why is that BUG_ON not firing?

  Paul

> If you need additional / other info, please cook up a debug patch with what
> you need.
> 
> --
> Sander
> 
> 
> 
> 
> 
> 
> >   Paul
> 
> >> Ultimately this results in bad grant reference warnings (and packets
> marked
> >> as "errors" in the interface statistics).
> >>
> >> In my case it always seems to be a skb with 1 frag which is broken down in
> 5
> >> or 6 pieces ..
> >>
> >> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring
> with
> >> 1 slot, but i'm not sure if that's not coincedence
> >> since in the code there seem to be no explicit limitation on how often this
> >> code path is taken. So perhaps it's implicitly limited
> >> since packets and frags can't be arbitrarily large in comparison with the
> >> page_size but that's not something i'm capable of figuring out :-)
> >>
> >>
> >>
> >> >   Paul
> >>
> >> >> Problem is that a worst case calculation would probably be reverting to
> >> the
> >> >> old calculation,
> >> >> and the problems this patch was trying to solve would reappear, but
> >> >> introducing new regressions
> >> >> isn't very useful either. And since it seems such a tricky and fragile
> thing to
> >> >> determine, it would
> >> >> probably be best to be split into a distinct function with a comment to
> >> explain
> >> >> the rationale used.
> >> >>
> >> >> Since this doesn't seem to progress very fast .. CC'ed some more folks
> ..
> >> you
> >> >> never know ..
> >> >>
> >> >> --
> >> >> Sander
> >> >>
> >> >>
> >> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> >> >>
> >> >>
> >> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> >> >>
> >> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom
> wrote:
> >> >> >> [...]
> >> >> >>> > Yes there is only one frag .. but it seems to be much larger than
> >> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
> >> smaller
> >> >> bits .. hence the calculation in xenvif_rx_action determining the slots
> >> needed
> >> >> by doing:
> >> >> >>>
> >> >> >>> >                 for (i = 0; i < nr_frags; i++) {
> >> >> >>> >                         unsigned int size;
> >> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
> >> PAGE_SIZE);
> >> >> >>> >                 }
> >> >> >>>
> >> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing
> one
> >> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
> >> >> seems involved in that ..
> >> >> >>>
> >> >> >>> Hmm looked again .. and it seems this is it .. when your frags are
> large
> >> >> enough you have the chance of running into this.
> >> >> >>>
> >> >>
> >> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you
> see
> >> any
> >> >> >> problem with that implementation?
> >> >> > In general no, but "get_next_rx_buffer" up's cons .. and the
> calculations
> >> >> done in "xenvif_rx_action" for max_slots_needed to prevent the
> overrun
> >> >> > don't count in this possibility. So it's not the guarding of
> >> >> "start_new_rx_buffer" that is at fault. It's the ones early in
> >> >> "xenvif_rx_action".
> >> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS
> to a
> >> >> calculated value that should be a "slim fit".
> >> >>
> >> >> > The problem is in determining upfront in "xenvif_rx_action" when
> and
> >> how
> >> >> often the "get_next_rx_buffer" path will be taken.
> >> >> > Unless there are other non direct restrictions (from a size point of
> view)
> >> it
> >> >> can be called multiple times per frag per skb.
> >> >>
> >> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
> >> empirical
> >> >> test doesn't seem to be the worst case either (i think):
> >> >> >>>
> >> >> >>> - In my case the packets that hit this only have 1 frag, but i could
> have
> >> >> had more frags.
> >> >> >>>   I also think you can't rule out the possibility of doing the
> >> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
> >> >> >>>   so in the worst (and perhaps even from a single frag since it's
> looped
> >> >> over a split of it in what seems PAGE_SIZE pieces.)
> >> >> >>>   - So an exact calculation of how much slots we are going to need
> for
> >> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
> >> >> unfeasible.
> >> >> >>>   - A worst case gamble seems impossible either .. if you take
> multiple
> >> >> frags * multiple times the "get_next_rx_buffer" ... you would probably
> be
> >> >> back at just
> >> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >> >> >>>
> >> >> >>> - Other thing would be checking for the available slots before
> doing
> >> the
> >> >> "get_next_rx_buffer" .. how ever .. we don't account for how many
> slots
> >> we
> >> >> still need to
> >> >> >>>   just process the remaining frags.
> >> >> >>>
> >> >>
> >> >> >> We've done a worst case estimation for whole SKB (linear area + all
> >> >> >> frags) already, at the very first beginning. That's what
> >> >> >> max_slots_needed is for.
> >> >>
> >> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but
> it
> >> >> seems the "get_next_rx_buffer" is necessary ..  when i make
> >> >> start_new_rx_buffer always return false
> >> >> >>>   i hit the bug below :S
> >> >> >>>
> >> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
> >> necessary
> >> >> ?
> >> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and
> does
> >> that
> >> >> outweigh the additional cost of accounting
> >> >> >>>      of needed_slots for the frags that have yet to be processed ?
> >> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
> >> >> returning to using MAX_SKB_FRAGS ?
> >> >> >>>
> >> >>
> >> >> >> I think you need to answer several questions first:
> >> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
> >> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
> >> multiple
> >> >> times when processing the SKB
> >> >> >         you have the chance of overrunning (or be saved because prod
> gets
> >> >> upped before you overrun).
> >> >>
> >> >> >> 2. is the test of ring slot availability acurate?
> >> >> >         Seems to be.
> >> >>
> >> >> >> 3. is the number of ring slots consumed larger than
> >> max_slots_needed? (I
> >> >> >>    guess the answer is yes)
> >> >> >         Yes that was the whole point.
> >> >>
> >> >> >> 4. which step in the break-down procedure causes backend to
> overrun
> >> >> the
> >> >> >>    ring?
> >> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which
> can
> >> be
> >> >> called
> >> >> >         multiple times per frag (unless some other none obvious size
> >> >> restriction limits this
> >> >> >         to one time per frag or one time per SKB).
> >> >> >         In my errorneous case it is called one time, but it would be nice if
> >> there
> >> >> would be some theoretical proof
> >> >> >         instead of just some emperical test.
> >> >>
> >> >>
> >> >> >> It doesn't matter how many frags your SKB has and how big a frag
> is. If
> >> >> >> every step is correct then you're fine. The code you're looking at
> >> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to
> be
> >> >> >> carefully examined.
> >> >>
> >> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases
> ..
> >> >> (and that also what i'm seeing because it doesn't happen
> >> >> > continously.
> >> >>
> >> >> > Question is shouldn't this max_needed_slots calc be reverted to
> what it
> >> >> was before 3.14 and take the time in 3.15 to figure out a
> >> >> > the theoretical max (if it can be calculated upfront) .. or another way
> to
> >> >> guarantee the code is able to process the whole SKB  ?
> >> >>
> >> >> >> Wei.
> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 5:25:21 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 16:07
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 15:23
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >>
>> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> Sent: 26 March 2014 11:11
>> >> >> To: Paul Durrant
>> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> >> kernel; netdev@vger.kernel.org
>> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> >> troubles "bisected"
>> >> >>
>> >> >> Paul,
>> >> >>
>> >> >> You have been awfully silent for this whole thread while this is a
>> >> regression
>> >> >> caused by a patch of you
>> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
>> earlier
>> >> in
>> >> >> this thread).
>> >> >>
>> >>
>> >> > Sorry, I've been distracted...
>> >>
>> >> >> The commit messages states:
>> >> >>     "net_rx_action() is the place where we could do with an accurate
>> >> >> predicition but,
>> >> >>     since that has proven tricky to calculate, a cheap worse-case (but not
>> >> too
>> >> >> bad)
>> >> >>     estimate is all we really need since the only thing we *must* prevent
>> is
>> >> >> xenvif_gop_skb()
>> >> >>     consuming more slots than are available."
>> >> >>
>> >> >> Your "worst-case" calculation stated in the commit message is clearly
>> not
>> >> the
>> >> >> worst case,
>> >> >> since it doesn't take calls to "get_next_rx_buffer" into account.
>> >> >>
>> >>
>> >> > It should be taking into account the behaviour of
>> start_new_rx_buffer(),
>> >> which should be true if a slot is full or a frag will overflow the current slot
>> and
>> >> doesn't require splitting.
>> >> > The code in net_rx_action() makes the assumption that each frag will
>> >> require as many slots as its size requires, i.e. it assumes no packing of
>> >> multiple frags into a single slot, so it should be a worst case.
>> >> > Did I miss something in that logic?
>> >>
>> >> Yes.
>> >> In "xenvif_gop_skb()" this loop:
>> >>
>> >>         for (i = 0; i < nr_frags; i++) {
>> >>                 xenvif_gop_frag_copy(vif, skb, npo,
>> >>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
>> >>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
>> >>                                      skb_shinfo(skb)->frags[i].page_offset,
>> >>                                      &head);
>> >>         }
>> >>
>> >> Is capable of using up (at least) 1 slot more than is anticipated for in
>> >> "net_rx_action()"  by this code:
>> >>
>> >>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
>> >>                         unsigned int size;
>> >>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>> >>                 }
>> >>
>> >> And this happens when it calls "get_next_rx_buffer()" from
>> >> "xenvif_gop_frag_copy()" where it's breaking down the frag.
>> >>
>> 
>> > The function that determines whether to consume another slot is
>> start_new_rx_buffer() and for each frag I don't see why this would return
>> true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
>> > It may be called more times than that since the code in
>> xenvif_gop_frag_copy() must also allow for the offset of the frag but should
>> not return true in all cases.
>> > So, I still cannot see why a frag would ever consume more than
>> DIV_ROUND_UP(size, PAGE_SIZE) slots.
>> 
>> Well here a case were a frag is broken down in 2 pieces:
>> 
>> [ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo-
>> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
>> npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096
>> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
>> >req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
>> [ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !
>> min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif-
>> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring-
>> >req_event:2105868 skb->len:66 skb->data_len:0
>> [ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo-
>> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
>> vif->rx.sring->req_event:2105868, reserved_slots_left:0
>> [ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo-
>> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
>> req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868
>> reserved_slots_left:-1
>> [ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo-
>> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
>> npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096
>> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
>> >req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
>> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo-
>> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
>> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
>> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
>> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
>> reserved_slots_left:-1
>> [ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo-
>> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1
>> reserved_slots_left:-1
>> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
>> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
>> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
>> used_fragloop:3
>> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
>> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
>> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
>> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
>> reserved_slots_left:-1
>> 
>> - When processing an SKB we end up in "xenvif_gop_frag_copy" while prod
>> == cons ... but we still have bytes and size left ..
>> - start_new_rx_buffer() has returned true ..
>> - so we end up in get_next_rx_buffer
>> - this does a RING_GET_REQUEST and ups cons ..
>> - and we end up with a bad grant reference.
>> 
>> Sometimes we are saved by the bell .. since additional slots have become
>> free (you see cons become > prod in "get_next_rx_buffer" but shortly after
>> that prod is increased ..
>> just in time to not cause a overrun).
>> 

> Ah, but hang on... There's a BUG_ON meta_slots_used > max_slots_needed, so if we are overflowing the worst-case calculation then why is that BUG_ON not firing?

You mean:
                sco = (struct skb_cb_overlay *)skb->cb;
                sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
                BUG_ON(sco->meta_slots_used > max_slots_needed);

in "get_next_rx_buffer" ?

I don't know .. at least now it doesn't crash dom0 and therefore not my complete machine and since tcp is recovering from a failed packet  :-)

But probably because "npo->copy_prod++" seems to be used for the frags .. and it isn't added to  npo->meta_prod ?

--
Sander

>   Paul

>> If you need additional / other info, please cook up a debug patch with what
>> you need.
>> 
>> --
>> Sander
>> 
>> 
>> 
>> 
>> 
>> 
>> >   Paul
>> 
>> >> Ultimately this results in bad grant reference warnings (and packets
>> marked
>> >> as "errors" in the interface statistics).
>> >>
>> >> In my case it always seems to be a skb with 1 frag which is broken down in
>> 5
>> >> or 6 pieces ..
>> >>
>> >> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring
>> with
>> >> 1 slot, but i'm not sure if that's not coincedence
>> >> since in the code there seem to be no explicit limitation on how often this
>> >> code path is taken. So perhaps it's implicitly limited
>> >> since packets and frags can't be arbitrarily large in comparison with the
>> >> page_size but that's not something i'm capable of figuring out :-)
>> >>
>> >>
>> >>
>> >> >   Paul
>> >>
>> >> >> Problem is that a worst case calculation would probably be reverting to
>> >> the
>> >> >> old calculation,
>> >> >> and the problems this patch was trying to solve would reappear, but
>> >> >> introducing new regressions
>> >> >> isn't very useful either. And since it seems such a tricky and fragile
>> thing to
>> >> >> determine, it would
>> >> >> probably be best to be split into a distinct function with a comment to
>> >> explain
>> >> >> the rationale used.
>> >> >>
>> >> >> Since this doesn't seem to progress very fast .. CC'ed some more folks
>> ..
>> >> you
>> >> >> never know ..
>> >> >>
>> >> >> --
>> >> >> Sander
>> >> >>
>> >> >>
>> >> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
>> >> >>
>> >> >>
>> >> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
>> >> >>
>> >> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom
>> wrote:
>> >> >> >> [...]
>> >> >> >>> > Yes there is only one frag .. but it seems to be much larger than
>> >> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
>> >> smaller
>> >> >> bits .. hence the calculation in xenvif_rx_action determining the slots
>> >> needed
>> >> >> by doing:
>> >> >> >>>
>> >> >> >>> >                 for (i = 0; i < nr_frags; i++) {
>> >> >> >>> >                         unsigned int size;
>> >> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
>> >> PAGE_SIZE);
>> >> >> >>> >                 }
>> >> >> >>>
>> >> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing
>> one
>> >> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
>> >> >> seems involved in that ..
>> >> >> >>>
>> >> >> >>> Hmm looked again .. and it seems this is it .. when your frags are
>> large
>> >> >> enough you have the chance of running into this.
>> >> >> >>>
>> >> >>
>> >> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you
>> see
>> >> any
>> >> >> >> problem with that implementation?
>> >> >> > In general no, but "get_next_rx_buffer" up's cons .. and the
>> calculations
>> >> >> done in "xenvif_rx_action" for max_slots_needed to prevent the
>> overrun
>> >> >> > don't count in this possibility. So it's not the guarding of
>> >> >> "start_new_rx_buffer" that is at fault. It's the ones early in
>> >> >> "xenvif_rx_action".
>> >> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS
>> to a
>> >> >> calculated value that should be a "slim fit".
>> >> >>
>> >> >> > The problem is in determining upfront in "xenvif_rx_action" when
>> and
>> >> how
>> >> >> often the "get_next_rx_buffer" path will be taken.
>> >> >> > Unless there are other non direct restrictions (from a size point of
>> view)
>> >> it
>> >> >> can be called multiple times per frag per skb.
>> >> >>
>> >> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
>> >> empirical
>> >> >> test doesn't seem to be the worst case either (i think):
>> >> >> >>>
>> >> >> >>> - In my case the packets that hit this only have 1 frag, but i could
>> have
>> >> >> had more frags.
>> >> >> >>>   I also think you can't rule out the possibility of doing the
>> >> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
>> >> >> >>>   so in the worst (and perhaps even from a single frag since it's
>> looped
>> >> >> over a split of it in what seems PAGE_SIZE pieces.)
>> >> >> >>>   - So an exact calculation of how much slots we are going to need
>> for
>> >> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
>> >> >> unfeasible.
>> >> >> >>>   - A worst case gamble seems impossible either .. if you take
>> multiple
>> >> >> frags * multiple times the "get_next_rx_buffer" ... you would probably
>> be
>> >> >> back at just
>> >> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
>> >> >> >>>
>> >> >> >>> - Other thing would be checking for the available slots before
>> doing
>> >> the
>> >> >> "get_next_rx_buffer" .. how ever .. we don't account for how many
>> slots
>> >> we
>> >> >> still need to
>> >> >> >>>   just process the remaining frags.
>> >> >> >>>
>> >> >>
>> >> >> >> We've done a worst case estimation for whole SKB (linear area + all
>> >> >> >> frags) already, at the very first beginning. That's what
>> >> >> >> max_slots_needed is for.
>> >> >>
>> >> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but
>> it
>> >> >> seems the "get_next_rx_buffer" is necessary ..  when i make
>> >> >> start_new_rx_buffer always return false
>> >> >> >>>   i hit the bug below :S
>> >> >> >>>
>> >> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
>> >> necessary
>> >> >> ?
>> >> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and
>> does
>> >> that
>> >> >> outweigh the additional cost of accounting
>> >> >> >>>      of needed_slots for the frags that have yet to be processed ?
>> >> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
>> >> >> returning to using MAX_SKB_FRAGS ?
>> >> >> >>>
>> >> >>
>> >> >> >> I think you need to answer several questions first:
>> >> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
>> >> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
>> >> multiple
>> >> >> times when processing the SKB
>> >> >> >         you have the chance of overrunning (or be saved because prod
>> gets
>> >> >> upped before you overrun).
>> >> >>
>> >> >> >> 2. is the test of ring slot availability acurate?
>> >> >> >         Seems to be.
>> >> >>
>> >> >> >> 3. is the number of ring slots consumed larger than
>> >> max_slots_needed? (I
>> >> >> >>    guess the answer is yes)
>> >> >> >         Yes that was the whole point.
>> >> >>
>> >> >> >> 4. which step in the break-down procedure causes backend to
>> overrun
>> >> >> the
>> >> >> >>    ring?
>> >> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which
>> can
>> >> be
>> >> >> called
>> >> >> >         multiple times per frag (unless some other none obvious size
>> >> >> restriction limits this
>> >> >> >         to one time per frag or one time per SKB).
>> >> >> >         In my errorneous case it is called one time, but it would be nice if
>> >> there
>> >> >> would be some theoretical proof
>> >> >> >         instead of just some emperical test.
>> >> >>
>> >> >>
>> >> >> >> It doesn't matter how many frags your SKB has and how big a frag
>> is. If
>> >> >> >> every step is correct then you're fine. The code you're looking at
>> >> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to
>> be
>> >> >> >> carefully examined.
>> >> >>
>> >> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases
>> ..
>> >> >> (and that also what i'm seeing because it doesn't happen
>> >> >> > continously.
>> >> >>
>> >> >> > Question is shouldn't this max_needed_slots calc be reverted to
>> what it
>> >> >> was before 3.14 and take the time in 3.15 to figure out a
>> >> >> > the theoretical max (if it can be calculated upfront) .. or another way
>> to
>> >> >> guarantee the code is able to process the whole SKB  ?
>> >> >>
>> >> >> >> Wei.
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> 
>> 

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 5:25:21 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 16:07
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 15:23
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >>
>> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> Sent: 26 March 2014 11:11
>> >> >> To: Paul Durrant
>> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> >> kernel; netdev@vger.kernel.org
>> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> >> troubles "bisected"
>> >> >>
>> >> >> Paul,
>> >> >>
>> >> >> You have been awfully silent for this whole thread while this is a
>> >> regression
>> >> >> caused by a patch of you
>> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
>> earlier
>> >> in
>> >> >> this thread).
>> >> >>
>> >>
>> >> > Sorry, I've been distracted...
>> >>
>> >> >> The commit messages states:
>> >> >>     "net_rx_action() is the place where we could do with an accurate
>> >> >> predicition but,
>> >> >>     since that has proven tricky to calculate, a cheap worse-case (but not
>> >> too
>> >> >> bad)
>> >> >>     estimate is all we really need since the only thing we *must* prevent
>> is
>> >> >> xenvif_gop_skb()
>> >> >>     consuming more slots than are available."
>> >> >>
>> >> >> Your "worst-case" calculation stated in the commit message is clearly
>> not
>> >> the
>> >> >> worst case,
>> >> >> since it doesn't take calls to "get_next_rx_buffer" into account.
>> >> >>
>> >>
>> >> > It should be taking into account the behaviour of
>> start_new_rx_buffer(),
>> >> which should be true if a slot is full or a frag will overflow the current slot
>> and
>> >> doesn't require splitting.
>> >> > The code in net_rx_action() makes the assumption that each frag will
>> >> require as many slots as its size requires, i.e. it assumes no packing of
>> >> multiple frags into a single slot, so it should be a worst case.
>> >> > Did I miss something in that logic?
>> >>
>> >> Yes.
>> >> In "xenvif_gop_skb()" this loop:
>> >>
>> >>         for (i = 0; i < nr_frags; i++) {
>> >>                 xenvif_gop_frag_copy(vif, skb, npo,
>> >>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
>> >>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
>> >>                                      skb_shinfo(skb)->frags[i].page_offset,
>> >>                                      &head);
>> >>         }
>> >>
>> >> Is capable of using up (at least) 1 slot more than is anticipated for in
>> >> "net_rx_action()"  by this code:
>> >>
>> >>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
>> >>                         unsigned int size;
>> >>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>> >>                 }
>> >>
>> >> And this happens when it calls "get_next_rx_buffer()" from
>> >> "xenvif_gop_frag_copy()" where it's breaking down the frag.
>> >>
>> 
>> > The function that determines whether to consume another slot is
>> start_new_rx_buffer() and for each frag I don't see why this would return
>> true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
>> > It may be called more times than that since the code in
>> xenvif_gop_frag_copy() must also allow for the offset of the frag but should
>> not return true in all cases.
>> > So, I still cannot see why a frag would ever consume more than
>> DIV_ROUND_UP(size, PAGE_SIZE) slots.
>> 
>> Well here a case were a frag is broken down in 2 pieces:
>> 
>> [ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo-
>> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
>> npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096
>> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
>> >req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
>> [ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !
>> min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif-
>> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring-
>> >req_event:2105868 skb->len:66 skb->data_len:0
>> [ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo-
>> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105867
>> vif->rx.sring->req_event:2105868, reserved_slots_left:0
>> [ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo-
>> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
>> req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868
>> reserved_slots_left:-1
>> [ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo-
>> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
>> npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096
>> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
>> >req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
>> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo-
>> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
>> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
>> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
>> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
>> reserved_slots_left:-1
>> [ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo-
>> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1
>> reserved_slots_left:-1
>> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
>> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
>> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
>> used_fragloop:3
>> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
>> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
>> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
>> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
>> reserved_slots_left:-1
>> 
>> - When processing an SKB we end up in "xenvif_gop_frag_copy" while prod
>> == cons ... but we still have bytes and size left ..
>> - start_new_rx_buffer() has returned true ..
>> - so we end up in get_next_rx_buffer
>> - this does a RING_GET_REQUEST and ups cons ..
>> - and we end up with a bad grant reference.
>> 
>> Sometimes we are saved by the bell .. since additional slots have become
>> free (you see cons become > prod in "get_next_rx_buffer" but shortly after
>> that prod is increased ..
>> just in time to not cause a overrun).
>> 

> Ah, but hang on... There's a BUG_ON meta_slots_used > max_slots_needed, so if we are overflowing the worst-case calculation then why is that BUG_ON not firing?

You mean:
                sco = (struct skb_cb_overlay *)skb->cb;
                sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
                BUG_ON(sco->meta_slots_used > max_slots_needed);

in "get_next_rx_buffer" ?

I don't know .. at least now it doesn't crash dom0 and therefore not my complete machine and since tcp is recovering from a failed packet  :-)

But probably because "npo->copy_prod++" seems to be used for the frags .. and it isn't added to  npo->meta_prod ?

--
Sander

>   Paul

>> If you need additional / other info, please cook up a debug patch with what
>> you need.
>> 
>> --
>> Sander
>> 
>> 
>> 
>> 
>> 
>> 
>> >   Paul
>> 
>> >> Ultimately this results in bad grant reference warnings (and packets
>> marked
>> >> as "errors" in the interface statistics).
>> >>
>> >> In my case it always seems to be a skb with 1 frag which is broken down in
>> 5
>> >> or 6 pieces ..
>> >>
>> >> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring
>> with
>> >> 1 slot, but i'm not sure if that's not coincedence
>> >> since in the code there seem to be no explicit limitation on how often this
>> >> code path is taken. So perhaps it's implicitly limited
>> >> since packets and frags can't be arbitrarily large in comparison with the
>> >> page_size but that's not something i'm capable of figuring out :-)
>> >>
>> >>
>> >>
>> >> >   Paul
>> >>
>> >> >> Problem is that a worst case calculation would probably be reverting to
>> >> the
>> >> >> old calculation,
>> >> >> and the problems this patch was trying to solve would reappear, but
>> >> >> introducing new regressions
>> >> >> isn't very useful either. And since it seems such a tricky and fragile
>> thing to
>> >> >> determine, it would
>> >> >> probably be best to be split into a distinct function with a comment to
>> >> explain
>> >> >> the rationale used.
>> >> >>
>> >> >> Since this doesn't seem to progress very fast .. CC'ed some more folks
>> ..
>> >> you
>> >> >> never know ..
>> >> >>
>> >> >> --
>> >> >> Sander
>> >> >>
>> >> >>
>> >> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
>> >> >>
>> >> >>
>> >> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
>> >> >>
>> >> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom
>> wrote:
>> >> >> >> [...]
>> >> >> >>> > Yes there is only one frag .. but it seems to be much larger than
>> >> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
>> >> smaller
>> >> >> bits .. hence the calculation in xenvif_rx_action determining the slots
>> >> needed
>> >> >> by doing:
>> >> >> >>>
>> >> >> >>> >                 for (i = 0; i < nr_frags; i++) {
>> >> >> >>> >                         unsigned int size;
>> >> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>> >> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
>> >> PAGE_SIZE);
>> >> >> >>> >                 }
>> >> >> >>>
>> >> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing
>> one
>> >> >> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
>> >> >> seems involved in that ..
>> >> >> >>>
>> >> >> >>> Hmm looked again .. and it seems this is it .. when your frags are
>> large
>> >> >> enough you have the chance of running into this.
>> >> >> >>>
>> >> >>
>> >> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you
>> see
>> >> any
>> >> >> >> problem with that implementation?
>> >> >> > In general no, but "get_next_rx_buffer" up's cons .. and the
>> calculations
>> >> >> done in "xenvif_rx_action" for max_slots_needed to prevent the
>> overrun
>> >> >> > don't count in this possibility. So it's not the guarding of
>> >> >> "start_new_rx_buffer" that is at fault. It's the ones early in
>> >> >> "xenvif_rx_action".
>> >> >> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS
>> to a
>> >> >> calculated value that should be a "slim fit".
>> >> >>
>> >> >> > The problem is in determining upfront in "xenvif_rx_action" when
>> and
>> >> how
>> >> >> often the "get_next_rx_buffer" path will be taken.
>> >> >> > Unless there are other non direct restrictions (from a size point of
>> view)
>> >> it
>> >> >> can be called multiple times per frag per skb.
>> >> >>
>> >> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
>> >> empirical
>> >> >> test doesn't seem to be the worst case either (i think):
>> >> >> >>>
>> >> >> >>> - In my case the packets that hit this only have 1 frag, but i could
>> have
>> >> >> had more frags.
>> >> >> >>>   I also think you can't rule out the possibility of doing the
>> >> >> "get_next_rx_buffer" for multiple subsequent frags from one packet,
>> >> >> >>>   so in the worst (and perhaps even from a single frag since it's
>> looped
>> >> >> over a split of it in what seems PAGE_SIZE pieces.)
>> >> >> >>>   - So an exact calculation of how much slots we are going to need
>> for
>> >> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
>> >> >> unfeasible.
>> >> >> >>>   - A worst case gamble seems impossible either .. if you take
>> multiple
>> >> >> frags * multiple times the "get_next_rx_buffer" ... you would probably
>> be
>> >> >> back at just
>> >> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
>> >> >> >>>
>> >> >> >>> - Other thing would be checking for the available slots before
>> doing
>> >> the
>> >> >> "get_next_rx_buffer" .. how ever .. we don't account for how many
>> slots
>> >> we
>> >> >> still need to
>> >> >> >>>   just process the remaining frags.
>> >> >> >>>
>> >> >>
>> >> >> >> We've done a worst case estimation for whole SKB (linear area + all
>> >> >> >> frags) already, at the very first beginning. That's what
>> >> >> >> max_slots_needed is for.
>> >> >>
>> >> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but
>> it
>> >> >> seems the "get_next_rx_buffer" is necessary ..  when i make
>> >> >> start_new_rx_buffer always return false
>> >> >> >>>   i hit the bug below :S
>> >> >> >>>
>> >> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
>> >> necessary
>> >> >> ?
>> >> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and
>> does
>> >> that
>> >> >> outweigh the additional cost of accounting
>> >> >> >>>      of needed_slots for the frags that have yet to be processed ?
>> >> >> >>>    - if yes, erhmmm what would be the best acceptable solution ..
>> >> >> returning to using MAX_SKB_FRAGS ?
>> >> >> >>>
>> >> >>
>> >> >> >> I think you need to answer several questions first:
>> >> >> >> 1. is max_slots_needed actually large enough to cover whole SKB?
>> >> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
>> >> multiple
>> >> >> times when processing the SKB
>> >> >> >         you have the chance of overrunning (or be saved because prod
>> gets
>> >> >> upped before you overrun).
>> >> >>
>> >> >> >> 2. is the test of ring slot availability acurate?
>> >> >> >         Seems to be.
>> >> >>
>> >> >> >> 3. is the number of ring slots consumed larger than
>> >> max_slots_needed? (I
>> >> >> >>    guess the answer is yes)
>> >> >> >         Yes that was the whole point.
>> >> >>
>> >> >> >> 4. which step in the break-down procedure causes backend to
>> overrun
>> >> >> the
>> >> >> >>    ring?
>> >> >> >         The "get_next_rx_buffer" call .. that is not accounted for (which
>> can
>> >> be
>> >> >> called
>> >> >> >         multiple times per frag (unless some other none obvious size
>> >> >> restriction limits this
>> >> >> >         to one time per frag or one time per SKB).
>> >> >> >         In my errorneous case it is called one time, but it would be nice if
>> >> there
>> >> >> would be some theoretical proof
>> >> >> >         instead of just some emperical test.
>> >> >>
>> >> >>
>> >> >> >> It doesn't matter how many frags your SKB has and how big a frag
>> is. If
>> >> >> >> every step is correct then you're fine. The code you're looking at
>> >> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to
>> be
>> >> >> >> carefully examined.
>> >> >>
>> >> >> > Well it seems it only calls "get_next_rx_buffer" in some special cases
>> ..
>> >> >> (and that also what i'm seeing because it doesn't happen
>> >> >> > continously.
>> >> >>
>> >> >> > Question is shouldn't this max_needed_slots calc be reverted to
>> what it
>> >> >> was before 3.14 and take the time in 3.15 to figure out a
>> >> >> > the theoretical max (if it can be calculated upfront) .. or another way
>> to
>> >> >> guarantee the code is able to process the whole SKB  ?
>> >> >>
>> >> >> >> Wei.
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> 
>> 

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 16:54
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 5:25:21 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 16:07
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >>
> >> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
> >>
> >> >> -----Original Message-----
> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> Sent: 26 March 2014 15:23
> >> >> To: Paul Durrant
> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> >> kernel; netdev@vger.kernel.org
> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> >> troubles "bisected"
> >> >>
> >> >>
> >> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
> >> >>
> >> >> >> -----Original Message-----
> >> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> >> Sent: 26 March 2014 11:11
> >> >> >> To: Paul Durrant
> >> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian
> Campbell;
> >> >> linux-
> >> >> >> kernel; netdev@vger.kernel.org
> >> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13
> Network
> >> >> >> troubles "bisected"
> >> >> >>
> >> >> >> Paul,
> >> >> >>
> >> >> >> You have been awfully silent for this whole thread while this is a
> >> >> regression
> >> >> >> caused by a patch of you
> >> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
> >> earlier
> >> >> in
> >> >> >> this thread).
> >> >> >>
> >> >>
> >> >> > Sorry, I've been distracted...
> >> >>
> >> >> >> The commit messages states:
> >> >> >>     "net_rx_action() is the place where we could do with an accurate
> >> >> >> predicition but,
> >> >> >>     since that has proven tricky to calculate, a cheap worse-case (but
> not
> >> >> too
> >> >> >> bad)
> >> >> >>     estimate is all we really need since the only thing we *must*
> prevent
> >> is
> >> >> >> xenvif_gop_skb()
> >> >> >>     consuming more slots than are available."
> >> >> >>
> >> >> >> Your "worst-case" calculation stated in the commit message is
> clearly
> >> not
> >> >> the
> >> >> >> worst case,
> >> >> >> since it doesn't take calls to "get_next_rx_buffer" into account.
> >> >> >>
> >> >>
> >> >> > It should be taking into account the behaviour of
> >> start_new_rx_buffer(),
> >> >> which should be true if a slot is full or a frag will overflow the current
> slot
> >> and
> >> >> doesn't require splitting.
> >> >> > The code in net_rx_action() makes the assumption that each frag will
> >> >> require as many slots as its size requires, i.e. it assumes no packing of
> >> >> multiple frags into a single slot, so it should be a worst case.
> >> >> > Did I miss something in that logic?
> >> >>
> >> >> Yes.
> >> >> In "xenvif_gop_skb()" this loop:
> >> >>
> >> >>         for (i = 0; i < nr_frags; i++) {
> >> >>                 xenvif_gop_frag_copy(vif, skb, npo,
> >> >>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
> >> >>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
> >> >>                                      skb_shinfo(skb)->frags[i].page_offset,
> >> >>                                      &head);
> >> >>         }
> >> >>
> >> >> Is capable of using up (at least) 1 slot more than is anticipated for in
> >> >> "net_rx_action()"  by this code:
> >> >>
> >> >>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> >> >>                         unsigned int size;
> >> >>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >> >>                 }
> >> >>
> >> >> And this happens when it calls "get_next_rx_buffer()" from
> >> >> "xenvif_gop_frag_copy()" where it's breaking down the frag.
> >> >>
> >>
> >> > The function that determines whether to consume another slot is
> >> start_new_rx_buffer() and for each frag I don't see why this would return
> >> true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
> >> > It may be called more times than that since the code in
> >> xenvif_gop_frag_copy() must also allow for the offset of the frag but
> should
> >> not return true in all cases.
> >> > So, I still cannot see why a frag would ever consume more than
> >> DIV_ROUND_UP(size, PAGE_SIZE) slots.
> >>
> >> Well here a case were a frag is broken down in 2 pieces:
> >>
> >> [ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo-
> >> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105867
> >> npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096
> >> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >> >req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
> >> [ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !
> >> min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif-
> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring-
> >> >req_event:2105868 skb->len:66 skb->data_len:0
> >> [ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo-
> >> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105867
> >> vif->rx.sring->req_event:2105868, reserved_slots_left:0
> >> [ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo-
> >> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868
> >> req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868
> >> reserved_slots_left:-1
> >> [ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo-
> >> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868
> >> npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096
> >> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >> >req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
> >> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end
> npo-
> >> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868
> >> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
> >> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
> >> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
> >> reserved_slots_left:-1
> >> [ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo-
> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> >> req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1
> >> reserved_slots_left:-1
> >> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> >> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> >> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> >> used_fragloop:3
> >> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> >> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> >> reserved_slots_left:-1
> >>
> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> prod
> >> == cons ... but we still have bytes and size left ..
> >> - start_new_rx_buffer() has returned true ..
> >> - so we end up in get_next_rx_buffer
> >> - this does a RING_GET_REQUEST and ups cons ..
> >> - and we end up with a bad grant reference.
> >>
> >> Sometimes we are saved by the bell .. since additional slots have become
> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> after
> >> that prod is increased ..
> >> just in time to not cause a overrun).
> >>
> 
> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> max_slots_needed, so if we are overflowing the worst-case calculation then
> why is that BUG_ON not firing?
> 
> You mean:
>                 sco = (struct skb_cb_overlay *)skb->cb;
>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> 
> in "get_next_rx_buffer" ?
> 

That code excerpt is from net_rx_action(),isn't it?

> I don't know .. at least now it doesn't crash dom0 and therefore not my
> complete machine and since tcp is recovering from a failed packet  :-)
> 

Well, if the code calculating max_slots_needed were underestimating then the BUG_ON() should fire. If it is not firing in your case then this suggests your problem lies elsewhere, or that meta_slots_used is not equal to the number of ring slots consumed.

> But probably because "npo->copy_prod++" seems to be used for the frags ..
> and it isn't added to  npo->meta_prod ?
> 

meta_slots_used is calculated as the value of meta_prod at return (from xenvif_gop_skb()) minus the value on entry , and if you look back up the code then you can see that meta_prod is incremented every time RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without evaluating RING_GET_REQUEST() and I think that's exactly what's happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is simply incremented in the case of a GSO. So the BUG_ON() is indeed off by one.

  Paul

> --
> Sander
> 
> >   Paul
> 
> >> If you need additional / other info, please cook up a debug patch with
> what
> >> you need.
> >>
> >> --
> >> Sander
> >>
> >>
> >>
> >>
> >>
> >>
> >> >   Paul
> >>
> >> >> Ultimately this results in bad grant reference warnings (and packets
> >> marked
> >> >> as "errors" in the interface statistics).
> >> >>
> >> >> In my case it always seems to be a skb with 1 frag which is broken
> down in
> >> 5
> >> >> or 6 pieces ..
> >> >>
> >> >> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring
> >> with
> >> >> 1 slot, but i'm not sure if that's not coincedence
> >> >> since in the code there seem to be no explicit limitation on how often
> this
> >> >> code path is taken. So perhaps it's implicitly limited
> >> >> since packets and frags can't be arbitrarily large in comparison with the
> >> >> page_size but that's not something i'm capable of figuring out :-)
> >> >>
> >> >>
> >> >>
> >> >> >   Paul
> >> >>
> >> >> >> Problem is that a worst case calculation would probably be reverting
> to
> >> >> the
> >> >> >> old calculation,
> >> >> >> and the problems this patch was trying to solve would reappear,
> but
> >> >> >> introducing new regressions
> >> >> >> isn't very useful either. And since it seems such a tricky and fragile
> >> thing to
> >> >> >> determine, it would
> >> >> >> probably be best to be split into a distinct function with a comment
> to
> >> >> explain
> >> >> >> the rationale used.
> >> >> >>
> >> >> >> Since this doesn't seem to progress very fast .. CC'ed some more
> folks
> >> ..
> >> >> you
> >> >> >> never know ..
> >> >> >>
> >> >> >> --
> >> >> >> Sander
> >> >> >>
> >> >> >>
> >> >> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> >> >> >>
> >> >> >>
> >> >> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> >> >> >>
> >> >> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom
> >> wrote:
> >> >> >> >> [...]
> >> >> >> >>> > Yes there is only one frag .. but it seems to be much larger
> than
> >> >> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
> >> >> smaller
> >> >> >> bits .. hence the calculation in xenvif_rx_action determining the
> slots
> >> >> needed
> >> >> >> by doing:
> >> >> >> >>>
> >> >> >> >>> >                 for (i = 0; i < nr_frags; i++) {
> >> >> >> >>> >                         unsigned int size;
> >> >> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
> >> >> PAGE_SIZE);
> >> >> >> >>> >                 }
> >> >> >> >>>
> >> >> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing
> >> one
> >> >> >> more slot (from the emperical test) .. and calling
> "get_next_rx_buffer"
> >> >> >> seems involved in that ..
> >> >> >> >>>
> >> >> >> >>> Hmm looked again .. and it seems this is it .. when your frags
> are
> >> large
> >> >> >> enough you have the chance of running into this.
> >> >> >> >>>
> >> >> >>
> >> >> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you
> >> see
> >> >> any
> >> >> >> >> problem with that implementation?
> >> >> >> > In general no, but "get_next_rx_buffer" up's cons .. and the
> >> calculations
> >> >> >> done in "xenvif_rx_action" for max_slots_needed to prevent the
> >> overrun
> >> >> >> > don't count in this possibility. So it's not the guarding of
> >> >> >> "start_new_rx_buffer" that is at fault. It's the ones early in
> >> >> >> "xenvif_rx_action".
> >> >> >> > The ones that were changed by Paul's patch from
> MAX_SKB_FRAGS
> >> to a
> >> >> >> calculated value that should be a "slim fit".
> >> >> >>
> >> >> >> > The problem is in determining upfront in "xenvif_rx_action" when
> >> and
> >> >> how
> >> >> >> often the "get_next_rx_buffer" path will be taken.
> >> >> >> > Unless there are other non direct restrictions (from a size point of
> >> view)
> >> >> it
> >> >> >> can be called multiple times per frag per skb.
> >> >> >>
> >> >> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
> >> >> empirical
> >> >> >> test doesn't seem to be the worst case either (i think):
> >> >> >> >>>
> >> >> >> >>> - In my case the packets that hit this only have 1 frag, but i
> could
> >> have
> >> >> >> had more frags.
> >> >> >> >>>   I also think you can't rule out the possibility of doing the
> >> >> >> "get_next_rx_buffer" for multiple subsequent frags from one
> packet,
> >> >> >> >>>   so in the worst (and perhaps even from a single frag since it's
> >> looped
> >> >> >> over a split of it in what seems PAGE_SIZE pieces.)
> >> >> >> >>>   - So an exact calculation of how much slots we are going to
> need
> >> for
> >> >> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action"
> seems
> >> >> >> unfeasible.
> >> >> >> >>>   - A worst case gamble seems impossible either .. if you take
> >> multiple
> >> >> >> frags * multiple times the "get_next_rx_buffer" ... you would
> probably
> >> be
> >> >> >> back at just
> >> >> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >> >> >> >>>
> >> >> >> >>> - Other thing would be checking for the available slots before
> >> doing
> >> >> the
> >> >> >> "get_next_rx_buffer" .. how ever .. we don't account for how
> many
> >> slots
> >> >> we
> >> >> >> still need to
> >> >> >> >>>   just process the remaining frags.
> >> >> >> >>>
> >> >> >>
> >> >> >> >> We've done a worst case estimation for whole SKB (linear area +
> all
> >> >> >> >> frags) already, at the very first beginning. That's what
> >> >> >> >> max_slots_needed is for.
> >> >> >>
> >> >> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it
> but
> >> it
> >> >> >> seems the "get_next_rx_buffer" is necessary ..  when i make
> >> >> >> start_new_rx_buffer always return false
> >> >> >> >>>   i hit the bug below :S
> >> >> >> >>>
> >> >> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
> >> >> necessary
> >> >> >> ?
> >> >> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and
> >> does
> >> >> that
> >> >> >> outweigh the additional cost of accounting
> >> >> >> >>>      of needed_slots for the frags that have yet to be processed
> ?
> >> >> >> >>>    - if yes, erhmmm what would be the best acceptable solution
> ..
> >> >> >> returning to using MAX_SKB_FRAGS ?
> >> >> >> >>>
> >> >> >>
> >> >> >> >> I think you need to answer several questions first:
> >> >> >> >> 1. is max_slots_needed actually large enough to cover whole
> SKB?
> >> >> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
> >> >> multiple
> >> >> >> times when processing the SKB
> >> >> >> >         you have the chance of overrunning (or be saved because
> prod
> >> gets
> >> >> >> upped before you overrun).
> >> >> >>
> >> >> >> >> 2. is the test of ring slot availability acurate?
> >> >> >> >         Seems to be.
> >> >> >>
> >> >> >> >> 3. is the number of ring slots consumed larger than
> >> >> max_slots_needed? (I
> >> >> >> >>    guess the answer is yes)
> >> >> >> >         Yes that was the whole point.
> >> >> >>
> >> >> >> >> 4. which step in the break-down procedure causes backend to
> >> overrun
> >> >> >> the
> >> >> >> >>    ring?
> >> >> >> >         The "get_next_rx_buffer" call .. that is not accounted for
> (which
> >> can
> >> >> be
> >> >> >> called
> >> >> >> >         multiple times per frag (unless some other none obvious size
> >> >> >> restriction limits this
> >> >> >> >         to one time per frag or one time per SKB).
> >> >> >> >         In my errorneous case it is called one time, but it would be
> nice if
> >> >> there
> >> >> >> would be some theoretical proof
> >> >> >> >         instead of just some emperical test.
> >> >> >>
> >> >> >>
> >> >> >> >> It doesn't matter how many frags your SKB has and how big a
> frag
> >> is. If
> >> >> >> >> every step is correct then you're fine. The code you're looking at
> >> >> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs
> to
> >> be
> >> >> >> >> carefully examined.
> >> >> >>
> >> >> >> > Well it seems it only calls "get_next_rx_buffer" in some special
> cases
> >> ..
> >> >> >> (and that also what i'm seeing because it doesn't happen
> >> >> >> > continously.
> >> >> >>
> >> >> >> > Question is shouldn't this max_needed_slots calc be reverted to
> >> what it
> >> >> >> was before 3.14 and take the time in 3.15 to figure out a
> >> >> >> > the theoretical max (if it can be calculated upfront) .. or another
> way
> >> to
> >> >> >> guarantee the code is able to process the whole SKB  ?
> >> >> >>
> >> >> >> >> Wei.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
> 
> 

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Hi Paul,

Seems your last mail arrived in pretty bad shape (truncated) in my mailbox ..

--
Sander

Wednesday, March 26, 2014, 6:16:49 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 16:54
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 5:25:21 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 16:07
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >>
>> >> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> Sent: 26 March 2014 15:23
>> >> >> To: Paul Durrant
>> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> >> kernel; netdev@vger.kernel.org
>> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> >> troubles "bisected"
>> >> >>
>> >> >>
>> >> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
>> >> >>
>> >> >> >> -----Original Message-----
>> >> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> >> Sent: 26 March 2014 11:11
>> >> >> >> To: Paul Durrant
>> >> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian
>> Campbell;
>> >> >> linux-
>> >> >> >> kernel; netdev@vger.kernel.org
>> >> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13
>> Network
>> >> >> >> troubles "bisected"
>> >> >> >>
>> >> >> >> Paul,
>> >> >> >>
>> >> >> >> You have been awfully silent for this whole thread while this is a
>> >> >> regression
>> >> >> >> caused by a patch of you
>> >> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
>> >> earlier
>> >> >> in
>> >> >> >> this thread).
>> >> >> >>
>> >> >>
>> >> >> > Sorry, I've been distracted...
>> >> >>
>> >> >> >> The commit messages states:
>> >> >> >>     "net_rx_action() is the place where we could do with an accurate
>> >> >> >> predicition but,
>> >> >> >>     since that has proven tricky to calculate, a cheap worse-case (but
>> not
>> >> >> too
>> >> >> >> bad)
>> >> >> >>     estimate is all we really need since the only thing we *must*
>> prevent
>> >> is
>> >> >> >> xenvif_gop_skb()
>> >> >> >>     consuming more slots than are available."
>> >> >> >>
>> >> >> >> Your "worst-case" calculation stated in the commit message is
>> clearly
>> >> not
>> >> >> the
>> >> >> >> worst case,
>> >> >> >> since it doesn't take calls to "get_next_rx_buffer" into account

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

Re-send shortened version...

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 16:54
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
[snip]
> >>
> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> prod
> >> == cons ... but we still have bytes and size left ..
> >> - start_new_rx_buffer() has returned true ..
> >> - so we end up in get_next_rx_buffer
> >> - this does a RING_GET_REQUEST and ups cons ..
> >> - and we end up with a bad grant reference.
> >>
> >> Sometimes we are saved by the bell .. since additional slots have become
> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> after
> >> that prod is increased ..
> >> just in time to not cause a overrun).
> >>
> 
> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> max_slots_needed, so if we are overflowing the worst-case calculation then
> why is that BUG_ON not firing?
> 
> You mean:
>                 sco = (struct skb_cb_overlay *)skb->cb;
>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> 
> in "get_next_rx_buffer" ?
> 

That code excerpt is from net_rx_action(),isn't it?

> I don't know .. at least now it doesn't crash dom0 and therefore not my
> complete machine and since tcp is recovering from a failed packet  :-)
> 

Well, if the code calculating max_slots_needed were underestimating then the BUG_ON() should fire. If it is not firing in your case then this suggests your problem lies elsewhere, or that meta_slots_used is not equal to the number of ring slots consumed.

> But probably because "npo->copy_prod++" seems to be used for the frags ..
> and it isn't added to  npo->meta_prod ?
> 

meta_slots_used is calculated as the value of meta_prod at return (from xenvif_gop_skb()) minus the value on entry , and if you look back up the code then you can see that meta_prod is incremented every time RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without evaluating RING_GET_REQUEST() and I think that's exactly what's happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is simply incremented in the case of a GSO. So the BUG_ON() is indeed off by one.

  Paul

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

Re-send shortened version...

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 16:54
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
[snip]
> >>
> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> prod
> >> == cons ... but we still have bytes and size left ..
> >> - start_new_rx_buffer() has returned true ..
> >> - so we end up in get_next_rx_buffer
> >> - this does a RING_GET_REQUEST and ups cons ..
> >> - and we end up with a bad grant reference.
> >>
> >> Sometimes we are saved by the bell .. since additional slots have become
> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> after
> >> that prod is increased ..
> >> just in time to not cause a overrun).
> >>
> 
> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> max_slots_needed, so if we are overflowing the worst-case calculation then
> why is that BUG_ON not firing?
> 
> You mean:
>                 sco = (struct skb_cb_overlay *)skb->cb;
>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> 
> in "get_next_rx_buffer" ?
> 

That code excerpt is from net_rx_action(),isn't it?

> I don't know .. at least now it doesn't crash dom0 and therefore not my
> complete machine and since tcp is recovering from a failed packet  :-)
> 

Well, if the code calculating max_slots_needed were underestimating then the BUG_ON() should fire. If it is not firing in your case then this suggests your problem lies elsewhere, or that meta_slots_used is not equal to the number of ring slots consumed.

> But probably because "npo->copy_prod++" seems to be used for the frags ..
> and it isn't added to  npo->meta_prod ?
> 

meta_slots_used is calculated as the value of meta_prod at return (from xenvif_gop_skb()) minus the value on entry , and if you look back up the code then you can see that meta_prod is incremented every time RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without evaluating RING_GET_REQUEST() and I think that's exactly what's happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is simply incremented in the case of a GSO. So the BUG_ON() is indeed off by one.

  Paul

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 6:46:06 PM, you wrote:

> Re-send shortened version...

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 16:54
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
> [snip]
>> >>
>> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
>> prod
>> >> == cons ... but we still have bytes and size left ..
>> >> - start_new_rx_buffer() has returned true ..
>> >> - so we end up in get_next_rx_buffer
>> >> - this does a RING_GET_REQUEST and ups cons ..
>> >> - and we end up with a bad grant reference.
>> >>
>> >> Sometimes we are saved by the bell .. since additional slots have become
>> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
>> after
>> >> that prod is increased ..
>> >> just in time to not cause a overrun).
>> >>
>> 
>> > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> max_slots_needed, so if we are overflowing the worst-case calculation then
>> why is that BUG_ON not firing?
>> 
>> You mean:
>>                 sco = (struct skb_cb_overlay *)skb->cb;
>>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> 
>> in "get_next_rx_buffer" ?
>> 

> That code excerpt is from net_rx_action(),isn't it?

Yes

>> I don't know .. at least now it doesn't crash dom0 and therefore not my
>> complete machine and since tcp is recovering from a failed packet  :-)
>> 

> Well, if the code calculating max_slots_needed were underestimating then the BUG_ON() should fire. If it is not firing in your case then this suggests your problem lies elsewhere, or that meta_slots_used is not equal to the number of ring slots consumed.

It's seem to be the last ..

[ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:3
[ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco->meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2 reserved_slots_left:-1

net_rx_action() calculated we would need 4 slots .. and sco->meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON ..

The 4 slots we calculated are:
  1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE)
  2 slots for the single frag in this SKB from: DIV_ROUND_UP(size, PAGE_SIZE)
  1 slot since GSO

In the debug code i annotated all cons++, and the code uses 1 slot to process the data from the SKB as expected but uses 3 slots in the frag chopping loop.
And when it reaches the state  were cons > prod it is always in "get_next_rx_buffer".

>> But probably because "npo->copy_prod++" seems to be used for the frags ..
>> and it isn't added to  npo->meta_prod ?
>> 

> meta_slots_used is calculated as the value of meta_prod at return (from xenvif_gop_skb()) minus the value on entry ,
> and if you look back up the code then you can see that meta_prod is incremented every time RING_GET_REQUEST() is evaluated.
> So, we must be consuming a slot without evaluating RING_GET_REQUEST() and I think that's exactly what's happening...
> Right at the bottom of xenvif_gop_frag_copy() req_cons is simply incremented in the case of a GSO. So the BUG_ON() is indeed off by one.

That is probably only done on first iteration / frag ?
Because i don't see my warn there trigger .. but it could be that's because at that moment we still have cons <= prod.


>   Paul

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 18:08
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
> 
> > Re-send shortened version...
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 16:54
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> > [snip]
> >> >>
> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> >> prod
> >> >> == cons ... but we still have bytes and size left ..
> >> >> - start_new_rx_buffer() has returned true ..
> >> >> - so we end up in get_next_rx_buffer
> >> >> - this does a RING_GET_REQUEST and ups cons ..
> >> >> - and we end up with a bad grant reference.
> >> >>
> >> >> Sometimes we are saved by the bell .. since additional slots have
> become
> >> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> >> after
> >> >> that prod is increased ..
> >> >> just in time to not cause a overrun).
> >> >>
> >>
> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> >> max_slots_needed, so if we are overflowing the worst-case calculation
> then
> >> why is that BUG_ON not firing?
> >>
> >> You mean:
> >>                 sco = (struct skb_cb_overlay *)skb->cb;
> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >>
> >> in "get_next_rx_buffer" ?
> >>
> 
> > That code excerpt is from net_rx_action(),isn't it?
> 
> Yes
> 
> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
> >> complete machine and since tcp is recovering from a failed packet  :-)
> >>
> 
> > Well, if the code calculating max_slots_needed were underestimating then
> the BUG_ON() should fire. If it is not firing in your case then this suggests
> your problem lies elsewhere, or that meta_slots_used is not equal to the
> number of ring slots consumed.
> 
> It's seem to be the last ..
> 
> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> used_fragloop:3
> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> reserved_slots_left:-1
> 
> net_rx_action() calculated we would need 4 slots .. and sco-
> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON ..
> 
> The 4 slots we calculated are:
>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
> skb_headlen(skb), PAGE_SIZE)
>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size, PAGE_SIZE)
>   1 slot since GSO
> 
> In the debug code i annotated all cons++, and the code uses 1 slot to process
> the data from the SKB as expected but uses 3 slots in the frag chopping loop.
> And when it reaches the state  were cons > prod it is always in
> "get_next_rx_buffer".
> 
> >> But probably because "npo->copy_prod++" seems to be used for the
> frags ..
> >> and it isn't added to  npo->meta_prod ?
> >>
> 
> > meta_slots_used is calculated as the value of meta_prod at return (from
> xenvif_gop_skb()) minus the value on entry ,
> > and if you look back up the code then you can see that meta_prod is
> incremented every time RING_GET_REQUEST() is evaluated.
> > So, we must be consuming a slot without evaluating RING_GET_REQUEST()
> and I think that's exactly what's happening...
> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
> incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
> 
> That is probably only done on first iteration / frag ?

Yes, the extra slot is accounted for right after the head frag is processed.

  Paul

> Because i don't see my warn there trigger .. but it could be that's because at
> that moment we still have cons <= prod.
> 
> 
> >   Paul
> 
> 

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 18:08
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
> 
> > Re-send shortened version...
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 16:54
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> > [snip]
> >> >>
> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> >> prod
> >> >> == cons ... but we still have bytes and size left ..
> >> >> - start_new_rx_buffer() has returned true ..
> >> >> - so we end up in get_next_rx_buffer
> >> >> - this does a RING_GET_REQUEST and ups cons ..
> >> >> - and we end up with a bad grant reference.
> >> >>
> >> >> Sometimes we are saved by the bell .. since additional slots have
> become
> >> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> >> after
> >> >> that prod is increased ..
> >> >> just in time to not cause a overrun).
> >> >>
> >>
> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> >> max_slots_needed, so if we are overflowing the worst-case calculation
> then
> >> why is that BUG_ON not firing?
> >>
> >> You mean:
> >>                 sco = (struct skb_cb_overlay *)skb->cb;
> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >>
> >> in "get_next_rx_buffer" ?
> >>
> 
> > That code excerpt is from net_rx_action(),isn't it?
> 
> Yes
> 
> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
> >> complete machine and since tcp is recovering from a failed packet  :-)
> >>
> 
> > Well, if the code calculating max_slots_needed were underestimating then
> the BUG_ON() should fire. If it is not firing in your case then this suggests
> your problem lies elsewhere, or that meta_slots_used is not equal to the
> number of ring slots consumed.
> 
> It's seem to be the last ..
> 
> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> used_fragloop:3
> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> reserved_slots_left:-1
> 
> net_rx_action() calculated we would need 4 slots .. and sco-
> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON ..
> 
> The 4 slots we calculated are:
>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
> skb_headlen(skb), PAGE_SIZE)
>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size, PAGE_SIZE)
>   1 slot since GSO
> 
> In the debug code i annotated all cons++, and the code uses 1 slot to process
> the data from the SKB as expected but uses 3 slots in the frag chopping loop.
> And when it reaches the state  were cons > prod it is always in
> "get_next_rx_buffer".
> 
> >> But probably because "npo->copy_prod++" seems to be used for the
> frags ..
> >> and it isn't added to  npo->meta_prod ?
> >>
> 
> > meta_slots_used is calculated as the value of meta_prod at return (from
> xenvif_gop_skb()) minus the value on entry ,
> > and if you look back up the code then you can see that meta_prod is
> incremented every time RING_GET_REQUEST() is evaluated.
> > So, we must be consuming a slot without evaluating RING_GET_REQUEST()
> and I think that's exactly what's happening...
> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
> incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
> 
> That is probably only done on first iteration / frag ?

Yes, the extra slot is accounted for right after the head frag is processed.

  Paul

> Because i don't see my warn there trigger .. but it could be that's because at
> that moment we still have cons <= prod.
> 
> 
> >   Paul
> 
> 

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Paul Durrant
> Sent: 26 March 2014 18:16
> To: Sander Eikelenboom
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> > -----Original Message-----
> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > Sent: 26 March 2014 18:08
> > To: Paul Durrant
> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> > kernel; netdev@vger.kernel.org
> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> > troubles "bisected"
> >
> >
> > Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
> >
> > > Re-send shortened version...
> >
> > >> -----Original Message-----
> > >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > >> Sent: 26 March 2014 16:54
> > >> To: Paul Durrant
> > >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> > linux-
> > >> kernel; netdev@vger.kernel.org
> > >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> > >> troubles "bisected"
> > >>
> > > [snip]
> > >> >>
> > >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
> while
> > >> prod
> > >> >> == cons ... but we still have bytes and size left ..
> > >> >> - start_new_rx_buffer() has returned true ..
> > >> >> - so we end up in get_next_rx_buffer
> > >> >> - this does a RING_GET_REQUEST and ups cons ..
> > >> >> - and we end up with a bad grant reference.
> > >> >>
> > >> >> Sometimes we are saved by the bell .. since additional slots have
> > become
> > >> >> free (you see cons become > prod in "get_next_rx_buffer" but
> shortly
> > >> after
> > >> >> that prod is increased ..
> > >> >> just in time to not cause a overrun).
> > >> >>
> > >>
> > >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> > >> max_slots_needed, so if we are overflowing the worst-case calculation
> > then
> > >> why is that BUG_ON not firing?
> > >>
> > >> You mean:
> > >>                 sco = (struct skb_cb_overlay *)skb->cb;
> > >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> > >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> > >>
> > >> in "get_next_rx_buffer" ?
> > >>
> >
> > > That code excerpt is from net_rx_action(),isn't it?
> >
> > Yes
> >
> > >> I don't know .. at least now it doesn't crash dom0 and therefore not my
> > >> complete machine and since tcp is recovering from a failed packet  :-)
> > >>
> >
> > > Well, if the code calculating max_slots_needed were underestimating
> then
> > the BUG_ON() should fire. If it is not firing in your case then this suggests
> > your problem lies elsewhere, or that meta_slots_used is not equal to the
> > number of ring slots consumed.
> >
> > It's seem to be the last ..
> >
> > [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> > >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> > >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> > req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> > reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> > used_fragloop:3
> > [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> > >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> > >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> > max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> > reserved_slots_left:-1
> >
> > net_rx_action() calculated we would need 4 slots .. and sco-
> > >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON
> ..
> >
> > The 4 slots we calculated are:
> >   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
> > skb_headlen(skb), PAGE_SIZE)
> >   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size,
> PAGE_SIZE)
> >   1 slot since GSO
> >
> > In the debug code i annotated all cons++, and the code uses 1 slot to
> process
> > the data from the SKB as expected but uses 3 slots in the frag chopping
> loop.

So, we must have done something like fill an existing slot, fill the next, but then had some left over requiring a third. Could you try the following additional patch?

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback
index 58e5eb4..dfd8cea 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -499,7 +499,7 @@ static void xenvif_rx_action(struct xenvif *vif)
                for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
                        unsigned int size;
                        size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
-                       max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
+                       max_slots_needed += (size / PAGE_SIZE) + 2;
                }
                if (skb_is_gso(skb) &&
                   (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||

  Paul

> > And when it reaches the state  were cons > prod it is always in
> > "get_next_rx_buffer".
> >
> > >> But probably because "npo->copy_prod++" seems to be used for the
> > frags ..
> > >> and it isn't added to  npo->meta_prod ?
> > >>
> >
> > > meta_slots_used is calculated as the value of meta_prod at return (from
> > xenvif_gop_skb()) minus the value on entry ,
> > > and if you look back up the code then you can see that meta_prod is
> > incremented every time RING_GET_REQUEST() is evaluated.
> > > So, we must be consuming a slot without evaluating
> RING_GET_REQUEST()
> > and I think that's exactly what's happening...
> > > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
> > incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
> >
> > That is probably only done on first iteration / frag ?
> 
> Yes, the extra slot is accounted for right after the head frag is processed.
> 
>   Paul
> 
> > Because i don't see my warn there trigger .. but it could be that's because
> at
> > that moment we still have cons <= prod.
> >
> >
> > >   Paul
> >
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Paul Durrant
> Sent: 26 March 2014 18:16
> To: Sander Eikelenboom
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> > -----Original Message-----
> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > Sent: 26 March 2014 18:08
> > To: Paul Durrant
> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> > kernel; netdev@vger.kernel.org
> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> > troubles "bisected"
> >
> >
> > Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
> >
> > > Re-send shortened version...
> >
> > >> -----Original Message-----
> > >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > >> Sent: 26 March 2014 16:54
> > >> To: Paul Durrant
> > >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> > linux-
> > >> kernel; netdev@vger.kernel.org
> > >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> > >> troubles "bisected"
> > >>
> > > [snip]
> > >> >>
> > >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
> while
> > >> prod
> > >> >> == cons ... but we still have bytes and size left ..
> > >> >> - start_new_rx_buffer() has returned true ..
> > >> >> - so we end up in get_next_rx_buffer
> > >> >> - this does a RING_GET_REQUEST and ups cons ..
> > >> >> - and we end up with a bad grant reference.
> > >> >>
> > >> >> Sometimes we are saved by the bell .. since additional slots have
> > become
> > >> >> free (you see cons become > prod in "get_next_rx_buffer" but
> shortly
> > >> after
> > >> >> that prod is increased ..
> > >> >> just in time to not cause a overrun).
> > >> >>
> > >>
> > >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> > >> max_slots_needed, so if we are overflowing the worst-case calculation
> > then
> > >> why is that BUG_ON not firing?
> > >>
> > >> You mean:
> > >>                 sco = (struct skb_cb_overlay *)skb->cb;
> > >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> > >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> > >>
> > >> in "get_next_rx_buffer" ?
> > >>
> >
> > > That code excerpt is from net_rx_action(),isn't it?
> >
> > Yes
> >
> > >> I don't know .. at least now it doesn't crash dom0 and therefore not my
> > >> complete machine and since tcp is recovering from a failed packet  :-)
> > >>
> >
> > > Well, if the code calculating max_slots_needed were underestimating
> then
> > the BUG_ON() should fire. If it is not firing in your case then this suggests
> > your problem lies elsewhere, or that meta_slots_used is not equal to the
> > number of ring slots consumed.
> >
> > It's seem to be the last ..
> >
> > [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> > >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> > >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> > req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> > reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> > used_fragloop:3
> > [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> > >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> > >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> > max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> > reserved_slots_left:-1
> >
> > net_rx_action() calculated we would need 4 slots .. and sco-
> > >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON
> ..
> >
> > The 4 slots we calculated are:
> >   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
> > skb_headlen(skb), PAGE_SIZE)
> >   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size,
> PAGE_SIZE)
> >   1 slot since GSO
> >
> > In the debug code i annotated all cons++, and the code uses 1 slot to
> process
> > the data from the SKB as expected but uses 3 slots in the frag chopping
> loop.

So, we must have done something like fill an existing slot, fill the next, but then had some left over requiring a third. Could you try the following additional patch?

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback
index 58e5eb4..dfd8cea 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -499,7 +499,7 @@ static void xenvif_rx_action(struct xenvif *vif)
                for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
                        unsigned int size;
                        size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
-                       max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
+                       max_slots_needed += (size / PAGE_SIZE) + 2;
                }
                if (skb_is_gso(skb) &&
                   (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||

  Paul

> > And when it reaches the state  were cons > prod it is always in
> > "get_next_rx_buffer".
> >
> > >> But probably because "npo->copy_prod++" seems to be used for the
> > frags ..
> > >> and it isn't added to  npo->meta_prod ?
> > >>
> >
> > > meta_slots_used is calculated as the value of meta_prod at return (from
> > xenvif_gop_skb()) minus the value on entry ,
> > > and if you look back up the code then you can see that meta_prod is
> > incremented every time RING_GET_REQUEST() is evaluated.
> > > So, we must be consuming a slot without evaluating
> RING_GET_REQUEST()
> > and I think that's exactly what's happening...
> > > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
> > incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
> >
> > That is probably only done on first iteration / frag ?
> 
> Yes, the extra slot is accounted for right after the head frag is processed.
> 
>   Paul
> 
> > Because i don't see my warn there trigger .. but it could be that's because
> at
> > that moment we still have cons <= prod.
> >
> >
> > >   Paul
> >
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 7:15:30 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 18:08
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
>> 
>> > Re-send shortened version...
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 16:54
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> > [snip]
>> >> >>
>> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
>> >> prod
>> >> >> == cons ... but we still have bytes and size left ..
>> >> >> - start_new_rx_buffer() has returned true ..
>> >> >> - so we end up in get_next_rx_buffer
>> >> >> - this does a RING_GET_REQUEST and ups cons ..
>> >> >> - and we end up with a bad grant reference.
>> >> >>
>> >> >> Sometimes we are saved by the bell .. since additional slots have
>> become
>> >> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
>> >> after
>> >> >> that prod is increased ..
>> >> >> just in time to not cause a overrun).
>> >> >>
>> >>
>> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> >> max_slots_needed, so if we are overflowing the worst-case calculation
>> then
>> >> why is that BUG_ON not firing?
>> >>
>> >> You mean:
>> >>                 sco = (struct skb_cb_overlay *)skb->cb;
>> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >>
>> >> in "get_next_rx_buffer" ?
>> >>
>> 
>> > That code excerpt is from net_rx_action(),isn't it?
>> 
>> Yes
>> 
>> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
>> >> complete machine and since tcp is recovering from a failed packet  :-)
>> >>
>> 
>> > Well, if the code calculating max_slots_needed were underestimating then
>> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> number of ring slots consumed.
>> 
>> It's seem to be the last ..
>> 
>> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
>> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
>> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
>> used_fragloop:3
>> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
>> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
>> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
>> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
>> reserved_slots_left:-1
>> 
>> net_rx_action() calculated we would need 4 slots .. and sco-
>> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON ..
>> 
>> The 4 slots we calculated are:
>>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
>> skb_headlen(skb), PAGE_SIZE)
>>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size, PAGE_SIZE)
>>   1 slot since GSO
>> 
>> In the debug code i annotated all cons++, and the code uses 1 slot to process
>> the data from the SKB as expected but uses 3 slots in the frag chopping loop.
>> And when it reaches the state  were cons > prod it is always in
>> "get_next_rx_buffer".
>> 
>> >> But probably because "npo->copy_prod++" seems to be used for the
>> frags ..
>> >> and it isn't added to  npo->meta_prod ?
>> >>
>> 
>> > meta_slots_used is calculated as the value of meta_prod at return (from
>> xenvif_gop_skb()) minus the value on entry ,
>> > and if you look back up the code then you can see that meta_prod is
>> incremented every time RING_GET_REQUEST() is evaluated.
>> > So, we must be consuming a slot without evaluating RING_GET_REQUEST()
>> and I think that's exactly what's happening...
>> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
>> incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
>> 
>> That is probably only done on first iteration / frag ?

> Yes, the extra slot is accounted for right after the head frag is processed

Ok so we are talking about:

if (*head && ((1 << gso_type) & vif->gso_mask)){
                        vif->rx.req_cons++;

Well it had some debug code in place to detect if that path is taken as well:

[ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096 bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring->req_event:2105868 gso_gaps:0 estimated_slots_needed:4 reserved_slots_left:-1

Well "gso_gaps:0" indicates that in this case that path in "xenvif_gop_frag_copy()" has not been taken in any iteration of that frag.

However i=3 .. so we have done 3 iterations of the loop while we expected to do only 2 ...

So that would mean that somehow the code in "xenvif_gop_frag_copy()" needs more slots
to brake this frag down than the loop with DIV_ROUND_UP(size, PAGE_SIZE) in net_rx_action() has accounted for.

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 20:18
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 7:15:30 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 18:08
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >>
> >> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
> >>
> >> > Re-send shortened version...
> >>
> >> >> -----Original Message-----
> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> Sent: 26 March 2014 16:54
> >> >> To: Paul Durrant
> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> >> kernel; netdev@vger.kernel.org
> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> >> troubles "bisected"
> >> >>
> >> > [snip]
> >> >> >>
> >> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
> while
> >> >> prod
> >> >> >> == cons ... but we still have bytes and size left ..
> >> >> >> - start_new_rx_buffer() has returned true ..
> >> >> >> - so we end up in get_next_rx_buffer
> >> >> >> - this does a RING_GET_REQUEST and ups cons ..
> >> >> >> - and we end up with a bad grant reference.
> >> >> >>
> >> >> >> Sometimes we are saved by the bell .. since additional slots have
> >> become
> >> >> >> free (you see cons become > prod in "get_next_rx_buffer" but
> shortly
> >> >> after
> >> >> >> that prod is increased ..
> >> >> >> just in time to not cause a overrun).
> >> >> >>
> >> >>
> >> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> >> >> max_slots_needed, so if we are overflowing the worst-case calculation
> >> then
> >> >> why is that BUG_ON not firing?
> >> >>
> >> >> You mean:
> >> >>                 sco = (struct skb_cb_overlay *)skb->cb;
> >> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >> >>
> >> >> in "get_next_rx_buffer" ?
> >> >>
> >>
> >> > That code excerpt is from net_rx_action(),isn't it?
> >>
> >> Yes
> >>
> >> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
> >> >> complete machine and since tcp is recovering from a failed packet  :-)
> >> >>
> >>
> >> > Well, if the code calculating max_slots_needed were underestimating
> then
> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
> >> number of ring slots consumed.
> >>
> >> It's seem to be the last ..
> >>
> >> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> >> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> >> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> >> used_fragloop:3
> >> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> >> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> >> reserved_slots_left:-1
> >>
> >> net_rx_action() calculated we would need 4 slots .. and sco-
> >> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON
> ..
> >>
> >> The 4 slots we calculated are:
> >>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
> >> skb_headlen(skb), PAGE_SIZE)
> >>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size,
> PAGE_SIZE)
> >>   1 slot since GSO
> >>
> >> In the debug code i annotated all cons++, and the code uses 1 slot to
> process
> >> the data from the SKB as expected but uses 3 slots in the frag chopping
> loop.
> >> And when it reaches the state  were cons > prod it is always in
> >> "get_next_rx_buffer".
> >>
> >> >> But probably because "npo->copy_prod++" seems to be used for the
> >> frags ..
> >> >> and it isn't added to  npo->meta_prod ?
> >> >>
> >>
> >> > meta_slots_used is calculated as the value of meta_prod at return
> (from
> >> xenvif_gop_skb()) minus the value on entry ,
> >> > and if you look back up the code then you can see that meta_prod is
> >> incremented every time RING_GET_REQUEST() is evaluated.
> >> > So, we must be consuming a slot without evaluating
> RING_GET_REQUEST()
> >> and I think that's exactly what's happening...
> >> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
> >> incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
> >>
> >> That is probably only done on first iteration / frag ?
> 
> > Yes, the extra slot is accounted for right after the head frag is processed
> 
> Ok so we are talking about:
> 
> if (*head && ((1 << gso_type) & vif->gso_mask)){
>                         vif->rx.req_cons++;
> 
> Well it had some debug code in place to detect if that path is taken as well:
> 
> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo-
> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
> reserved_slots_left:-1
> 
> Well "gso_gaps:0" indicates that in this case that path in
> "xenvif_gop_frag_copy()" has not been taken in any iteration of that frag.
> 
> However i=3 .. so we have done 3 iterations of the loop while we expected
> to do only 2 ...
> 
> So that would mean that somehow the code in "xenvif_gop_frag_copy()"
> needs more slots
> to brake this frag down than the loop with DIV_ROUND_UP(size, PAGE_SIZE)
> in net_rx_action() has accounted for.
> 

Yes. The code is not assuming worst-case page-spanning and it looks like it needs to.

I also notice a bogus clause in this if statement in start_new_rx_buffer():

        if ((offset + size > MAX_BUFFER_OFFSET) &&
            (size <= MAX_BUFFER_OFFSET) && offset && !head)
                return true;

MAX_BUFFER_OFFSET is defined to be PAGE_SIZE and xenvif_gop_frag_copy() never passes a value of size > PAGE_SIZE, so that 2nd clause is completely pointless.
I'll come up with some patches shortly.

  Paul

> 
> 
> 
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Hrmm i don't know if it's your mailer or my mailer .. but i seem to get a lot of your mails truncated somehow :S
though the xen-devel list archive seem to have them in complete form .. so it's probably my mailer tripping over something

> I'll come up with some patches shortly.

OK will test them ASAP.


Thursday, March 27, 2014, 10:54:09 AM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 20:18
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 7:15:30 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 18:08
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >>
>> >> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
>> >>
>> >> > Re-send shortened version...
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> Sent: 26 March 2014 16:54
>> >> >> To: Paul Durrant
>> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> >> kernel; netdev@vger.kernel.org
>> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> >> troubles "bisected"
>> >> >>
>> >> > [snip]
>> >> >> >>
>> >> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
>> while
>> >> >> prod
>> >> >> >> == cons ... but we still have bytes and size left ..
>> >> >> >> - start_new_rx_buffer() has returned true ..
>> >> >> >> - so we end up in get_next_rx_buffer
>> >> >> >> - this does a RING_GET_REQUEST and ups cons ..
>> >> >> >> - and we end up with a bad grant reference.
>> >> >> >>
>> >> >> >> Sometimes we are saved by the bell .. since additional slots have
>> >> become
>> >> >> >> free (you see cons become > prod in "get_next_rx_buffer" but
>> shortly
>> >> >> after
>> >> >> >> that prod is increased ..
>> >> >> >> just in time to not cause a overrun).
>> >> >> >>
>> >> >>
>> >> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> >> >> max_slots_needed, so if we are overflowing the worst-case calculation
>> >> then
>> >> >> why is that BUG_ON not firing?
>> >> >>
>> >> >> You mean:
>> >> >>                 sco = (struct skb_cb_overlay *)skb->cb;
>> >> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >> >>
>> >> >> in "get_next_rx_buffer" ?
>> >> >>
>> >>
>> >> > That code excerpt is from net_rx_action(),isn't it?
>> >>
>> >> Yes
>> >>
>> >> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
>> >> >> complete machine and since tcp is recovering from a failed packet  :-)
>> >> >>
>> >>
>> >> > Well, if the code calculating max_slots_needed were underestimating
>> then
>> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> >> number of ring slots consumed.
>> >>
>> >> It's seem to be the last ..
>> >>
>> >> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
>> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> >> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
>> >> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
>> >> used_fragloop:3
>> >> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
>> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
>> >> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
>> >> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
>> >> reserved_slots_left:-1
>> >>
>> >> net_rx_action() calculated we would need 4 slots .. and sco-
>> >> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON
>> ..
>> >>
>> >> The 4 slots we calculated are:
>> >>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
>> >> skb_headlen(skb), PAGE_SIZE)
>> >>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size,
>> PAGE_SIZE)
>> >>   1 slot since GSO
>> >>
>> >> In the debug code i annotated all cons++, and the code uses 1 slot to
>> process
>> >> the data from the SKB as expected but uses 3 slots in the frag chopping
>> loop.
>> >> And when it reaches the state  were cons > prod it is always in
>> >> "get_next_rx_buffer".
>> >>
>> >> >> But probably because "npo->copy_prod++" seems to be used for the
>> >> frags ..
>> >> >> and it isn't added to  npo->meta_prod ?
>> >> >>
>> >>
>> >> > meta_slots_used is calculated as the value of meta_prod at return
>> (from
>> >> xenvif_gop_skb()) minus the value on entry ,
>> >> > and if you look back up the code then you can see that meta_prod is
>> >> incremented every time RING_GET_REQUEST() is evaluated.
>> >> > So, we must be consuming a slot without evaluating
>> RING_GET_REQUEST()
>> >> and I think that's exactly what's happening...
>> >> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
>> >> incremented in the case of a GSO. So the BUG_ON() is indeed off by one

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Hrmm i don't know if it's your mailer or my mailer .. but i seem to get a lot of your mails truncated somehow :S
though the xen-devel list archive seem to have them in complete form .. so it's probably my mailer tripping over something

> I'll come up with some patches shortly.

OK will test them ASAP.


Thursday, March 27, 2014, 10:54:09 AM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 20:18
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 7:15:30 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 18:08
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >>
>> >> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
>> >>
>> >> > Re-send shortened version...
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> Sent: 26 March 2014 16:54
>> >> >> To: Paul Durrant
>> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> >> kernel; netdev@vger.kernel.org
>> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> >> troubles "bisected"
>> >> >>
>> >> > [snip]
>> >> >> >>
>> >> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
>> while
>> >> >> prod
>> >> >> >> == cons ... but we still have bytes and size left ..
>> >> >> >> - start_new_rx_buffer() has returned true ..
>> >> >> >> - so we end up in get_next_rx_buffer
>> >> >> >> - this does a RING_GET_REQUEST and ups cons ..
>> >> >> >> - and we end up with a bad grant reference.
>> >> >> >>
>> >> >> >> Sometimes we are saved by the bell .. since additional slots have
>> >> become
>> >> >> >> free (you see cons become > prod in "get_next_rx_buffer" but
>> shortly
>> >> >> after
>> >> >> >> that prod is increased ..
>> >> >> >> just in time to not cause a overrun).
>> >> >> >>
>> >> >>
>> >> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> >> >> max_slots_needed, so if we are overflowing the worst-case calculation
>> >> then
>> >> >> why is that BUG_ON not firing?
>> >> >>
>> >> >> You mean:
>> >> >>                 sco = (struct skb_cb_overlay *)skb->cb;
>> >> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >> >>
>> >> >> in "get_next_rx_buffer" ?
>> >> >>
>> >>
>> >> > That code excerpt is from net_rx_action(),isn't it?
>> >>
>> >> Yes
>> >>
>> >> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
>> >> >> complete machine and since tcp is recovering from a failed packet  :-)
>> >> >>
>> >>
>> >> > Well, if the code calculating max_slots_needed were underestimating
>> then
>> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> >> number of ring slots consumed.
>> >>
>> >> It's seem to be the last ..
>> >>
>> >> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
>> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> >> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
>> >> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
>> >> used_fragloop:3
>> >> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
>> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
>> >> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
>> >> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
>> >> reserved_slots_left:-1
>> >>
>> >> net_rx_action() calculated we would need 4 slots .. and sco-
>> >> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON
>> ..
>> >>
>> >> The 4 slots we calculated are:
>> >>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
>> >> skb_headlen(skb), PAGE_SIZE)
>> >>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size,
>> PAGE_SIZE)
>> >>   1 slot since GSO
>> >>
>> >> In the debug code i annotated all cons++, and the code uses 1 slot to
>> process
>> >> the data from the SKB as expected but uses 3 slots in the frag chopping
>> loop.
>> >> And when it reaches the state  were cons > prod it is always in
>> >> "get_next_rx_buffer".
>> >>
>> >> >> But probably because "npo->copy_prod++" seems to be used for the
>> >> frags ..
>> >> >> and it isn't added to  npo->meta_prod ?
>> >> >>
>> >>
>> >> > meta_slots_used is calculated as the value of meta_prod at return
>> (from
>> >> xenvif_gop_skb()) minus the value on entry ,
>> >> > and if you look back up the code then you can see that meta_prod is
>> >> incremented every time RING_GET_REQUEST() is evaluated.
>> >> > So, we must be consuming a slot without evaluating
>> RING_GET_REQUEST()
>> >> and I think that's exactly what's happening...
>> >> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
>> >> incremented in the case of a GSO. So the BUG_ON() is indeed off by one

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 20:18
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 7:15:30 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 18:08
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >>
> >> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
> >>
> >> > Re-send shortened version...
> >>
> >> >> -----Original Message-----
> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> Sent: 26 March 2014 16:54
> >> >> To: Paul Durrant
> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> >> kernel; netdev@vger.kernel.org
> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> >> troubles "bisected"
> >> >>
> >> > [snip]
> >> >> >>
> >> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
> while
> >> >> prod
> >> >> >> == cons ... but we still have bytes and size left ..
> >> >> >> - start_new_rx_buffer() has returned true ..
> >> >> >> - so we end up in get_next_rx_buffer
> >> >> >> - this does a RING_GET_REQUEST and ups cons ..
> >> >> >> - and we end up with a bad grant reference.
> >> >> >>
> >> >> >> Sometimes we are saved by the bell .. since additional slots have
> >> become
> >> >> >> free (you see cons become > prod in "get_next_rx_buffer" but
> shortly
> >> >> after
> >> >> >> that prod is increased ..
> >> >> >> just in time to not cause a overrun).
> >> >> >>
> >> >>
> >> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> >> >> max_slots_needed, so if we are overflowing the worst-case calculation
> >> then
> >> >> why is that BUG_ON not firing?
> >> >>
> >> >> You mean:
> >> >>                 sco = (struct skb_cb_overlay *)skb->cb;
> >> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >> >>
> >> >> in "get_next_rx_buffer" ?
> >> >>
> >>
> >> > That code excerpt is from net_rx_action(),isn't it?
> >>
> >> Yes
> >>
> >> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
> >> >> complete machine and since tcp is recovering from a failed packet  :-)
> >> >>
> >>
> >> > Well, if the code calculating max_slots_needed were underestimating
> then
> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
> >> number of ring slots consumed.
> >>
> >> It's seem to be the last ..
> >>
> >> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> >> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> >> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> >> used_fragloop:3
> >> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> >> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> >> reserved_slots_left:-1
> >>
> >> net_rx_action() calculated we would need 4 slots .. and sco-
> >> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON
> ..
> >>
> >> The 4 slots we calculated are:
> >>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
> >> skb_headlen(skb), PAGE_SIZE)
> >>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size,
> PAGE_SIZE)
> >>   1 slot since GSO
> >>
> >> In the debug code i annotated all cons++, and the code uses 1 slot to
> process
> >> the data from the SKB as expected but uses 3 slots in the frag chopping
> loop.
> >> And when it reaches the state  were cons > prod it is always in
> >> "get_next_rx_buffer".
> >>
> >> >> But probably because "npo->copy_prod++" seems to be used for the
> >> frags ..
> >> >> and it isn't added to  npo->meta_prod ?
> >> >>
> >>
> >> > meta_slots_used is calculated as the value of meta_prod at return
> (from
> >> xenvif_gop_skb()) minus the value on entry ,
> >> > and if you look back up the code then you can see that meta_prod is
> >> incremented every time RING_GET_REQUEST() is evaluated.
> >> > So, we must be consuming a slot without evaluating
> RING_GET_REQUEST()
> >> and I think that's exactly what's happening...
> >> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
> >> incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
> >>
> >> That is probably only done on first iteration / frag ?
> 
> > Yes, the extra slot is accounted for right after the head frag is processed
> 
> Ok so we are talking about:
> 
> if (*head && ((1 << gso_type) & vif->gso_mask)){
>                         vif->rx.req_cons++;
> 
> Well it had some debug code in place to detect if that path is taken as well:
> 
> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo-
> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868
> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
> reserved_slots_left:-1
> 
> Well "gso_gaps:0" indicates that in this case that path in
> "xenvif_gop_frag_copy()" has not been taken in any iteration of that frag.
> 
> However i=3 .. so we have done 3 iterations of the loop while we expected
> to do only 2 ...
> 
> So that would mean that somehow the code in "xenvif_gop_frag_copy()"
> needs more slots
> to brake this frag down than the loop with DIV_ROUND_UP(size, PAGE_SIZE)
> in net_rx_action() has accounted for.
> 

Yes. The code is not assuming worst-case page-spanning and it looks like it needs to.

I also notice a bogus clause in this if statement in start_new_rx_buffer():

        if ((offset + size > MAX_BUFFER_OFFSET) &&
            (size <= MAX_BUFFER_OFFSET) && offset && !head)
                return true;

MAX_BUFFER_OFFSET is defined to be PAGE_SIZE and xenvif_gop_frag_copy() never passes a value of size > PAGE_SIZE, so that 2nd clause is completely pointless.
I'll come up with some patches shortly.

  Paul

> 
> 
> 
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 7:15:30 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 18:08
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 6:46:06 PM, you wrote:
>> 
>> > Re-send shortened version...
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 16:54
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> > [snip]
>> >> >>
>> >> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
>> >> prod
>> >> >> == cons ... but we still have bytes and size left ..
>> >> >> - start_new_rx_buffer() has returned true ..
>> >> >> - so we end up in get_next_rx_buffer
>> >> >> - this does a RING_GET_REQUEST and ups cons ..
>> >> >> - and we end up with a bad grant reference.
>> >> >>
>> >> >> Sometimes we are saved by the bell .. since additional slots have
>> become
>> >> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
>> >> after
>> >> >> that prod is increased ..
>> >> >> just in time to not cause a overrun).
>> >> >>
>> >>
>> >> > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> >> max_slots_needed, so if we are overflowing the worst-case calculation
>> then
>> >> why is that BUG_ON not firing?
>> >>
>> >> You mean:
>> >>                 sco = (struct skb_cb_overlay *)skb->cb;
>> >>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >>
>> >> in "get_next_rx_buffer" ?
>> >>
>> 
>> > That code excerpt is from net_rx_action(),isn't it?
>> 
>> Yes
>> 
>> >> I don't know .. at least now it doesn't crash dom0 and therefore not my
>> >> complete machine and since tcp is recovering from a failed packet  :-)
>> >>
>> 
>> > Well, if the code calculating max_slots_needed were underestimating then
>> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> number of ring slots consumed.
>> 
>> It's seem to be the last ..
>> 
>> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
>> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
>> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
>> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
>> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
>> used_fragloop:3
>> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
>> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
>> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
>> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
>> reserved_slots_left:-1
>> 
>> net_rx_action() calculated we would need 4 slots .. and sco-
>> >meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON ..
>> 
>> The 4 slots we calculated are:
>>   1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) +
>> skb_headlen(skb), PAGE_SIZE)
>>   2 slots for the single frag in this SKB from: DIV_ROUND_UP(size, PAGE_SIZE)
>>   1 slot since GSO
>> 
>> In the debug code i annotated all cons++, and the code uses 1 slot to process
>> the data from the SKB as expected but uses 3 slots in the frag chopping loop.
>> And when it reaches the state  were cons > prod it is always in
>> "get_next_rx_buffer".
>> 
>> >> But probably because "npo->copy_prod++" seems to be used for the
>> frags ..
>> >> and it isn't added to  npo->meta_prod ?
>> >>
>> 
>> > meta_slots_used is calculated as the value of meta_prod at return (from
>> xenvif_gop_skb()) minus the value on entry ,
>> > and if you look back up the code then you can see that meta_prod is
>> incremented every time RING_GET_REQUEST() is evaluated.
>> > So, we must be consuming a slot without evaluating RING_GET_REQUEST()
>> and I think that's exactly what's happening...
>> > Right at the bottom of xenvif_gop_frag_copy() req_cons is simply
>> incremented in the case of a GSO. So the BUG_ON() is indeed off by one.
>> 
>> That is probably only done on first iteration / frag ?

> Yes, the extra slot is accounted for right after the head frag is processed

Ok so we are talking about:

if (*head && ((1 << gso_type) & vif->gso_mask)){
                        vif->rx.req_cons++;

Well it had some debug code in place to detect if that path is taken as well:

[ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end npo->meta_prod:40 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096 bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring->req_event:2105868 gso_gaps:0 estimated_slots_needed:4 reserved_slots_left:-1

Well "gso_gaps:0" indicates that in this case that path in "xenvif_gop_frag_copy()" has not been taken in any iteration of that frag.

However i=3 .. so we have done 3 iterations of the loop while we expected to do only 2 ...

So that would mean that somehow the code in "xenvif_gop_frag_copy()" needs more slots
to brake this frag down than the loop with DIV_ROUND_UP(size, PAGE_SIZE) in net_rx_action() has accounted for.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 6:46:06 PM, you wrote:

> Re-send shortened version...

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 16:54
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
> [snip]
>> >>
>> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
>> prod
>> >> == cons ... but we still have bytes and size left ..
>> >> - start_new_rx_buffer() has returned true ..
>> >> - so we end up in get_next_rx_buffer
>> >> - this does a RING_GET_REQUEST and ups cons ..
>> >> - and we end up with a bad grant reference.
>> >>
>> >> Sometimes we are saved by the bell .. since additional slots have become
>> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
>> after
>> >> that prod is increased ..
>> >> just in time to not cause a overrun).
>> >>
>> 
>> > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> max_slots_needed, so if we are overflowing the worst-case calculation then
>> why is that BUG_ON not firing?
>> 
>> You mean:
>>                 sco = (struct skb_cb_overlay *)skb->cb;
>>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> 
>> in "get_next_rx_buffer" ?
>> 

> That code excerpt is from net_rx_action(),isn't it?

Yes

>> I don't know .. at least now it doesn't crash dom0 and therefore not my
>> complete machine and since tcp is recovering from a failed packet  :-)
>> 

> Well, if the code calculating max_slots_needed were underestimating then the BUG_ON() should fire. If it is not firing in your case then this suggests your problem lies elsewhere, or that meta_slots_used is not equal to the number of ring slots consumed.

It's seem to be the last ..

[ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo->meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1 req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1 reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 .. used_fragloop:3
[ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif->rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco->meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1 max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2 reserved_slots_left:-1

net_rx_action() calculated we would need 4 slots .. and sco->meta_slots_used == 4 when we return so it doesn't trigger you BUG_ON ..

The 4 slots we calculated are:
  1 slot for the data part: DIV_ROUND_UP(offset_in_page(skb->data) + skb_headlen(skb), PAGE_SIZE)
  2 slots for the single frag in this SKB from: DIV_ROUND_UP(size, PAGE_SIZE)
  1 slot since GSO

In the debug code i annotated all cons++, and the code uses 1 slot to process the data from the SKB as expected but uses 3 slots in the frag chopping loop.
And when it reaches the state  were cons > prod it is always in "get_next_rx_buffer".

>> But probably because "npo->copy_prod++" seems to be used for the frags ..
>> and it isn't added to  npo->meta_prod ?
>> 

> meta_slots_used is calculated as the value of meta_prod at return (from xenvif_gop_skb()) minus the value on entry ,
> and if you look back up the code then you can see that meta_prod is incremented every time RING_GET_REQUEST() is evaluated.
> So, we must be consuming a slot without evaluating RING_GET_REQUEST() and I think that's exactly what's happening...
> Right at the bottom of xenvif_gop_frag_copy() req_cons is simply incremented in the case of a GSO. So the BUG_ON() is indeed off by one.

That is probably only done on first iteration / frag ?
Because i don't see my warn there trigger .. but it could be that's because at that moment we still have cons <= prod.


>   Paul

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Paul Durrant
> Sent: 26 March 2014 17:47
> To: 'Sander Eikelenboom'
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> Re-send shortened version...
> 
> > -----Original Message-----
> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > Sent: 26 March 2014 16:54
> > To: Paul Durrant
> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> > kernel; netdev@vger.kernel.org
> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> > troubles "bisected"
> >
> [snip]
> > >>
> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> > prod
> > >> == cons ... but we still have bytes and size left ..
> > >> - start_new_rx_buffer() has returned true ..
> > >> - so we end up in get_next_rx_buffer
> > >> - this does a RING_GET_REQUEST and ups cons ..
> > >> - and we end up with a bad grant reference.
> > >>
> > >> Sometimes we are saved by the bell .. since additional slots have
> become
> > >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> > after
> > >> that prod is increased ..
> > >> just in time to not cause a overrun).
> > >>
> >
> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
> > max_slots_needed, so if we are overflowing the worst-case calculation
> then
> > why is that BUG_ON not firing?
> >
> > You mean:
> >                 sco = (struct skb_cb_overlay *)skb->cb;
> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >
> > in "get_next_rx_buffer" ?
> >
> 
> That code excerpt is from net_rx_action(),isn't it?
> 
> > I don't know .. at least now it doesn't crash dom0 and therefore not my
> > complete machine and since tcp is recovering from a failed packet  :-)
> >
> 
> Well, if the code calculating max_slots_needed were underestimating then
> the BUG_ON() should fire. If it is not firing in your case then this suggests
> your problem lies elsewhere, or that meta_slots_used is not equal to the
> number of ring slots consumed.
> 
> > But probably because "npo->copy_prod++" seems to be used for the frags
> ..
> > and it isn't added to  npo->meta_prod ?
> >
> 
> meta_slots_used is calculated as the value of meta_prod at return (from
> xenvif_gop_skb()) minus the value on entry , and if you look back up the
> code then you can see that meta_prod is incremented every time
> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without
> evaluating RING_GET_REQUEST() and I think that's exactly what's
> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
> simply incremented in the case of a GSO. So the BUG_ON() is indeed off by
> one.
> 

Can you re-test with the following patch applied?

  Paul

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback
index 438d0c0..4f24220 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)

        while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
                RING_IDX max_slots_needed;
+               RING_IDX old_req_cons;
+               RING_IDX ring_slots_used;
                int i;

                /* We need a cheap worse case estimate for the number of
@@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
                        vif->rx_last_skb_slots = 0;

                sco = (struct skb_cb_overlay *)skb->cb;
+
+               old_req_cons = vif->rx.req_cons;
                sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
-               BUG_ON(sco->meta_slots_used > max_slots_needed);
+               ring_slots_used = vif->rx.req_cons - old_req_cons;
+
+               BUG_ON(ring_slots_used > max_slots_needed);

                __skb_queue_tail(&rxq, skb);
        }

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 6:48:15 PM, you wrote:

>> -----Original Message-----
>> From: Paul Durrant
>> Sent: 26 March 2014 17:47
>> To: 'Sander Eikelenboom'
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> Re-send shortened version...
>> 
>> > -----Original Message-----
>> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> > Sent: 26 March 2014 16:54
>> > To: Paul Durrant
>> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> > kernel; netdev@vger.kernel.org
>> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> > troubles "bisected"
>> >
>> [snip]
>> > >>
>> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
>> > prod
>> > >> == cons ... but we still have bytes and size left ..
>> > >> - start_new_rx_buffer() has returned true ..
>> > >> - so we end up in get_next_rx_buffer
>> > >> - this does a RING_GET_REQUEST and ups cons ..
>> > >> - and we end up with a bad grant reference.
>> > >>
>> > >> Sometimes we are saved by the bell .. since additional slots have
>> become
>> > >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
>> > after
>> > >> that prod is increased ..
>> > >> just in time to not cause a overrun).
>> > >>
>> >
>> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> > max_slots_needed, so if we are overflowing the worst-case calculation
>> then
>> > why is that BUG_ON not firing?
>> >
>> > You mean:
>> >                 sco = (struct skb_cb_overlay *)skb->cb;
>> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >
>> > in "get_next_rx_buffer" ?
>> >
>> 
>> That code excerpt is from net_rx_action(),isn't it?
>> 
>> > I don't know .. at least now it doesn't crash dom0 and therefore not my
>> > complete machine and since tcp is recovering from a failed packet  :-)
>> >
>> 
>> Well, if the code calculating max_slots_needed were underestimating then
>> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> number of ring slots consumed.
>> 
>> > But probably because "npo->copy_prod++" seems to be used for the frags
>> ..
>> > and it isn't added to  npo->meta_prod ?
>> >
>> 
>> meta_slots_used is calculated as the value of meta_prod at return (from
>> xenvif_gop_skb()) minus the value on entry , and if you look back up the
>> code then you can see that meta_prod is incremented every time
>> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without
>> evaluating RING_GET_REQUEST() and I think that's exactly what's
>> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
>> simply incremented in the case of a GSO. So the BUG_ON() is indeed off by
>> one.
>> 

> Can you re-test with the following patch applied?

>   Paul

> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback
> index 438d0c0..4f24220 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)

>         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
>                 RING_IDX max_slots_needed;
> +               RING_IDX old_req_cons;
> +               RING_IDX ring_slots_used;
>                 int i;

>                 /* We need a cheap worse case estimate for the number of
> @@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
>                         vif->rx_last_skb_slots = 0;

>                 sco = (struct skb_cb_overlay *)skb->cb;
> +
> +               old_req_cons = vif->rx.req_cons;
>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> -               BUG_ON(sco->meta_slots_used > max_slots_needed);
> +               ring_slots_used = vif->rx.req_cons - old_req_cons;
> +
> +               BUG_ON(ring_slots_used > max_slots_needed);

>                 __skb_queue_tail(&rxq, skb);
>         }

That blew pretty fast .. on that BUG_ON

[  290.218182] ------------[ cut here ]------------
[  290.225425] kernel BUG at drivers/net/xen-netback/netback.c:664!
[  290.232717] invalid opcode: 0000 [#1] SMP
[  290.239875] Modules linked in:
[  290.246923] CPU: 0 PID: 10447 Comm: vif7.0 Not tainted 3.13.6-20140326-nbdebug35+ #1
[  290.254040] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS V1.8B1 09/13/2010
[  290.261313] task: ffff880055d16480 ti: ffff88004cb7e000 task.ti: ffff88004cb7e000
[  290.268713] RIP: e030:[<ffffffff81780430>]  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
[  290.276193] RSP: e02b:ffff88004cb7fc28  EFLAGS: 00010202
[  290.283555] RAX: 0000000000000006 RBX: ffff88004c630000 RCX: 3fffffffffffffff
[  290.290908] RDX: 00000000ffffffff RSI: ffff88004c630940 RDI: 0000000000048e7b
[  290.298325] RBP: ffff88004cb7fde8 R08: 0000000000007bc9 R09: 0000000000000005
[  290.305809] R10: ffff88004cb7fd28 R11: ffffc90012690600 R12: 0000000000000004
[  290.313217] R13: ffff8800536a84e0 R14: 0000000000000001 R15: ffff88004c637618
[  290.320521] FS:  00007f1d3030c700(0000) GS:ffff88005f600000(0000) knlGS:0000000000000000
[  290.327839] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  290.335216] CR2: ffffffffff600400 CR3: 0000000058537000 CR4: 0000000000000660
[  290.342732] Stack:
[  290.350129]  ffff88004cb7fd2c ffff880000000005 ffff88004cb7fd28 ffffffff810f7fc8
[  290.357652]  ffff880055d16b50 ffffffff00000407 ffff880000000000 ffffffff00000000
[  290.365048]  ffff880055d16b50 ffff880000000001 ffff880000000001 ffffffff00000000
[  290.372461] Call Trace:
[  290.379806]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
[  290.387211]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
[  290.394552]  [<ffffffff81781400>] xenvif_kthread+0x40/0x190
[  290.401808]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
[  290.408993]  [<ffffffff817813c0>] ? xenvif_stop_queue+0x60/0x60
[  290.416238]  [<ffffffff810d4f24>] kthread+0xe4/0x100
[  290.423428]  [<ffffffff81b4cf30>] ? _raw_spin_unlock_irq+0x30/0x50
[  290.430615]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
[  290.437793]  [<ffffffff81b4e13c>] ret_from_fork+0x7c/0xb0
[  290.444945]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
[  290.452091] Code: fd ff ff 48 8b b5 f0 fe ff ff 48 c7 c2 10 98 ce 81 31 c0 48 8b be c8 7c 00 00 48 c7 c6 f0 f1 fd 81 e8 35 be 24 00 e9 ba f8 ff ff <0f> 0b 0f 0b 41 bf 01 00 00 00 e9 55 f6 ff ff 0f 0b 66 66 66 66
[  290.467121] RIP  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
[  290.474436]  RSP <ffff88004cb7fc28>
[  290.482400] ---[ end trace 2fcf9e9ae26950b3 ]---

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 26, 2014, 6:48:15 PM, you wrote:

>> -----Original Message-----
>> From: Paul Durrant
>> Sent: 26 March 2014 17:47
>> To: 'Sander Eikelenboom'
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> Re-send shortened version...
>> 
>> > -----Original Message-----
>> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> > Sent: 26 March 2014 16:54
>> > To: Paul Durrant
>> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> > kernel; netdev@vger.kernel.org
>> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> > troubles "bisected"
>> >
>> [snip]
>> > >>
>> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
>> > prod
>> > >> == cons ... but we still have bytes and size left ..
>> > >> - start_new_rx_buffer() has returned true ..
>> > >> - so we end up in get_next_rx_buffer
>> > >> - this does a RING_GET_REQUEST and ups cons ..
>> > >> - and we end up with a bad grant reference.
>> > >>
>> > >> Sometimes we are saved by the bell .. since additional slots have
>> become
>> > >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
>> > after
>> > >> that prod is increased ..
>> > >> just in time to not cause a overrun).
>> > >>
>> >
>> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> > max_slots_needed, so if we are overflowing the worst-case calculation
>> then
>> > why is that BUG_ON not firing?
>> >
>> > You mean:
>> >                 sco = (struct skb_cb_overlay *)skb->cb;
>> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >
>> > in "get_next_rx_buffer" ?
>> >
>> 
>> That code excerpt is from net_rx_action(),isn't it?
>> 
>> > I don't know .. at least now it doesn't crash dom0 and therefore not my
>> > complete machine and since tcp is recovering from a failed packet  :-)
>> >
>> 
>> Well, if the code calculating max_slots_needed were underestimating then
>> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> number of ring slots consumed.
>> 
>> > But probably because "npo->copy_prod++" seems to be used for the frags
>> ..
>> > and it isn't added to  npo->meta_prod ?
>> >
>> 
>> meta_slots_used is calculated as the value of meta_prod at return (from
>> xenvif_gop_skb()) minus the value on entry , and if you look back up the
>> code then you can see that meta_prod is incremented every time
>> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without
>> evaluating RING_GET_REQUEST() and I think that's exactly what's
>> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
>> simply incremented in the case of a GSO. So the BUG_ON() is indeed off by
>> one.
>> 

> Can you re-test with the following patch applied?

>   Paul

> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback
> index 438d0c0..4f24220 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)

>         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
>                 RING_IDX max_slots_needed;
> +               RING_IDX old_req_cons;
> +               RING_IDX ring_slots_used;
>                 int i;

>                 /* We need a cheap worse case estimate for the number of
> @@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
>                         vif->rx_last_skb_slots = 0;

>                 sco = (struct skb_cb_overlay *)skb->cb;
> +
> +               old_req_cons = vif->rx.req_cons;
>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> -               BUG_ON(sco->meta_slots_used > max_slots_needed);
> +               ring_slots_used = vif->rx.req_cons - old_req_cons;
> +
> +               BUG_ON(ring_slots_used > max_slots_needed);

>                 __skb_queue_tail(&rxq, skb);
>         }

That blew pretty fast .. on that BUG_ON

[  290.218182] ------------[ cut here ]------------
[  290.225425] kernel BUG at drivers/net/xen-netback/netback.c:664!
[  290.232717] invalid opcode: 0000 [#1] SMP
[  290.239875] Modules linked in:
[  290.246923] CPU: 0 PID: 10447 Comm: vif7.0 Not tainted 3.13.6-20140326-nbdebug35+ #1
[  290.254040] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS V1.8B1 09/13/2010
[  290.261313] task: ffff880055d16480 ti: ffff88004cb7e000 task.ti: ffff88004cb7e000
[  290.268713] RIP: e030:[<ffffffff81780430>]  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
[  290.276193] RSP: e02b:ffff88004cb7fc28  EFLAGS: 00010202
[  290.283555] RAX: 0000000000000006 RBX: ffff88004c630000 RCX: 3fffffffffffffff
[  290.290908] RDX: 00000000ffffffff RSI: ffff88004c630940 RDI: 0000000000048e7b
[  290.298325] RBP: ffff88004cb7fde8 R08: 0000000000007bc9 R09: 0000000000000005
[  290.305809] R10: ffff88004cb7fd28 R11: ffffc90012690600 R12: 0000000000000004
[  290.313217] R13: ffff8800536a84e0 R14: 0000000000000001 R15: ffff88004c637618
[  290.320521] FS:  00007f1d3030c700(0000) GS:ffff88005f600000(0000) knlGS:0000000000000000
[  290.327839] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  290.335216] CR2: ffffffffff600400 CR3: 0000000058537000 CR4: 0000000000000660
[  290.342732] Stack:
[  290.350129]  ffff88004cb7fd2c ffff880000000005 ffff88004cb7fd28 ffffffff810f7fc8
[  290.357652]  ffff880055d16b50 ffffffff00000407 ffff880000000000 ffffffff00000000
[  290.365048]  ffff880055d16b50 ffff880000000001 ffff880000000001 ffffffff00000000
[  290.372461] Call Trace:
[  290.379806]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
[  290.387211]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
[  290.394552]  [<ffffffff81781400>] xenvif_kthread+0x40/0x190
[  290.401808]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
[  290.408993]  [<ffffffff817813c0>] ? xenvif_stop_queue+0x60/0x60
[  290.416238]  [<ffffffff810d4f24>] kthread+0xe4/0x100
[  290.423428]  [<ffffffff81b4cf30>] ? _raw_spin_unlock_irq+0x30/0x50
[  290.430615]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
[  290.437793]  [<ffffffff81b4e13c>] ret_from_fork+0x7c/0xb0
[  290.444945]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
[  290.452091] Code: fd ff ff 48 8b b5 f0 fe ff ff 48 c7 c2 10 98 ce 81 31 c0 48 8b be c8 7c 00 00 48 c7 c6 f0 f1 fd 81 e8 35 be 24 00 e9 ba f8 ff ff <0f> 0b 0f 0b 41 bf 01 00 00 00 e9 55 f6 ff ff 0f 0b 66 66 66 66
[  290.467121] RIP  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
[  290.474436]  RSP <ffff88004cb7fc28>
[  290.482400] ---[ end trace 2fcf9e9ae26950b3 ]---

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 19:57
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 6:48:15 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Paul Durrant
> >> Sent: 26 March 2014 17:47
> >> To: 'Sander Eikelenboom'
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >> Re-send shortened version...
> >>
> >> > -----Original Message-----
> >> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> > Sent: 26 March 2014 16:54
> >> > To: Paul Durrant
> >> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> > kernel; netdev@vger.kernel.org
> >> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> > troubles "bisected"
> >> >
> >> [snip]
> >> > >>
> >> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
> while
> >> > prod
> >> > >> == cons ... but we still have bytes and size left ..
> >> > >> - start_new_rx_buffer() has returned true ..
> >> > >> - so we end up in get_next_rx_buffer
> >> > >> - this does a RING_GET_REQUEST and ups cons ..
> >> > >> - and we end up with a bad grant reference.
> >> > >>
> >> > >> Sometimes we are saved by the bell .. since additional slots have
> >> become
> >> > >> free (you see cons become > prod in "get_next_rx_buffer" but
> shortly
> >> > after
> >> > >> that prod is increased ..
> >> > >> just in time to not cause a overrun).
> >> > >>
> >> >
> >> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
> >> > max_slots_needed, so if we are overflowing the worst-case calculation
> >> then
> >> > why is that BUG_ON not firing?
> >> >
> >> > You mean:
> >> >                 sco = (struct skb_cb_overlay *)skb->cb;
> >> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >> >
> >> > in "get_next_rx_buffer" ?
> >> >
> >>
> >> That code excerpt is from net_rx_action(),isn't it?
> >>
> >> > I don't know .. at least now it doesn't crash dom0 and therefore not my
> >> > complete machine and since tcp is recovering from a failed packet  :-)
> >> >
> >>
> >> Well, if the code calculating max_slots_needed were underestimating
> then
> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
> >> number of ring slots consumed.
> >>
> >> > But probably because "npo->copy_prod++" seems to be used for the
> frags
> >> ..
> >> > and it isn't added to  npo->meta_prod ?
> >> >
> >>
> >> meta_slots_used is calculated as the value of meta_prod at return (from
> >> xenvif_gop_skb()) minus the value on entry , and if you look back up the
> >> code then you can see that meta_prod is incremented every time
> >> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot
> without
> >> evaluating RING_GET_REQUEST() and I think that's exactly what's
> >> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
> >> simply incremented in the case of a GSO. So the BUG_ON() is indeed off
> by
> >> one.
> >>
> 
> > Can you re-test with the following patch applied?
> 
> >   Paul
> 
> > diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-
> netback/netback
> > index 438d0c0..4f24220 100644
> > --- a/drivers/net/xen-netback/netback.c
> > +++ b/drivers/net/xen-netback/netback.c
> > @@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)
> 
> >         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
> >                 RING_IDX max_slots_needed;
> > +               RING_IDX old_req_cons;
> > +               RING_IDX ring_slots_used;
> >                 int i;
> 
> >                 /* We need a cheap worse case estimate for the number of
> > @@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
> >                         vif->rx_last_skb_slots = 0;
> 
> >                 sco = (struct skb_cb_overlay *)skb->cb;
> > +
> > +               old_req_cons = vif->rx.req_cons;
> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> > -               BUG_ON(sco->meta_slots_used > max_slots_needed);
> > +               ring_slots_used = vif->rx.req_cons - old_req_cons;
> > +
> > +               BUG_ON(ring_slots_used > max_slots_needed);
> 
> >                 __skb_queue_tail(&rxq, skb);
> >         }
> 
> That blew pretty fast .. on that BUG_ON
> 

Good. That's what should have happened :-)

  Paul

> [  290.218182] ------------[ cut here ]------------
> [  290.225425] kernel BUG at drivers/net/xen-netback/netback.c:664!
> [  290.232717] invalid opcode: 0000 [#1] SMP
> [  290.239875] Modules linked in:
> [  290.246923] CPU: 0 PID: 10447 Comm: vif7.0 Not tainted 3.13.6-20140326-
> nbdebug35+ #1
> [  290.254040] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS
> V1.8B1 09/13/2010
> [  290.261313] task: ffff880055d16480 ti: ffff88004cb7e000 task.ti:
> ffff88004cb7e000
> [  290.268713] RIP: e030:[<ffffffff81780430>]  [<ffffffff81780430>]
> xenvif_rx_action+0x1650/0x1670
> [  290.276193] RSP: e02b:ffff88004cb7fc28  EFLAGS: 00010202
> [  290.283555] RAX: 0000000000000006 RBX: ffff88004c630000 RCX:
> 3fffffffffffffff
> [  290.290908] RDX: 00000000ffffffff RSI: ffff88004c630940 RDI:
> 0000000000048e7b
> [  290.298325] RBP: ffff88004cb7fde8 R08: 0000000000007bc9 R09:
> 0000000000000005
> [  290.305809] R10: ffff88004cb7fd28 R11: ffffc90012690600 R12:
> 0000000000000004
> [  290.313217] R13: ffff8800536a84e0 R14: 0000000000000001 R15:
> ffff88004c637618
> [  290.320521] FS:  00007f1d3030c700(0000) GS:ffff88005f600000(0000)
> knlGS:0000000000000000
> [  290.327839] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  290.335216] CR2: ffffffffff600400 CR3: 0000000058537000 CR4:
> 0000000000000660
> [  290.342732] Stack:
> [  290.350129]  ffff88004cb7fd2c ffff880000000005 ffff88004cb7fd28
> ffffffff810f7fc8
> [  290.357652]  ffff880055d16b50 ffffffff00000407 ffff880000000000
> ffffffff00000000
> [  290.365048]  ffff880055d16b50 ffff880000000001 ffff880000000001
> ffffffff00000000
> [  290.372461] Call Trace:
> [  290.379806]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
> [  290.387211]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
> [  290.394552]  [<ffffffff81781400>] xenvif_kthread+0x40/0x190
> [  290.401808]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
> [  290.408993]  [<ffffffff817813c0>] ? xenvif_stop_queue+0x60/0x60
> [  290.416238]  [<ffffffff810d4f24>] kthread+0xe4/0x100
> [  290.423428]  [<ffffffff81b4cf30>] ? _raw_spin_unlock_irq+0x30/0x50
> [  290.430615]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
> [  290.437793]  [<ffffffff81b4e13c>] ret_from_fork+0x7c/0xb0
> [  290.444945]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
> [  290.452091] Code: fd ff ff 48 8b b5 f0 fe ff ff 48 c7 c2 10 98 ce 81 31 c0 48 8b
> be c8 7c 00 00 48 c7 c6 f0 f1 fd 81 e8 35 be 24 00 e9 ba f8 ff ff <0f> 0b 0f 0b 41
> bf 01 00 00 00 e9 55 f6 ff ff 0f 0b 66 66 66 66
> [  290.467121] RIP  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
> [  290.474436]  RSP <ffff88004cb7fc28>
> [  290.482400] ---[ end trace 2fcf9e9ae26950b3 ]---

RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 19:57
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 6:48:15 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Paul Durrant
> >> Sent: 26 March 2014 17:47
> >> To: 'Sander Eikelenboom'
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >> Re-send shortened version...
> >>
> >> > -----Original Message-----
> >> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> > Sent: 26 March 2014 16:54
> >> > To: Paul Durrant
> >> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> > kernel; netdev@vger.kernel.org
> >> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> > troubles "bisected"
> >> >
> >> [snip]
> >> > >>
> >> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
> while
> >> > prod
> >> > >> == cons ... but we still have bytes and size left ..
> >> > >> - start_new_rx_buffer() has returned true ..
> >> > >> - so we end up in get_next_rx_buffer
> >> > >> - this does a RING_GET_REQUEST and ups cons ..
> >> > >> - and we end up with a bad grant reference.
> >> > >>
> >> > >> Sometimes we are saved by the bell .. since additional slots have
> >> become
> >> > >> free (you see cons become > prod in "get_next_rx_buffer" but
> shortly
> >> > after
> >> > >> that prod is increased ..
> >> > >> just in time to not cause a overrun).
> >> > >>
> >> >
> >> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
> >> > max_slots_needed, so if we are overflowing the worst-case calculation
> >> then
> >> > why is that BUG_ON not firing?
> >> >
> >> > You mean:
> >> >                 sco = (struct skb_cb_overlay *)skb->cb;
> >> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >> >
> >> > in "get_next_rx_buffer" ?
> >> >
> >>
> >> That code excerpt is from net_rx_action(),isn't it?
> >>
> >> > I don't know .. at least now it doesn't crash dom0 and therefore not my
> >> > complete machine and since tcp is recovering from a failed packet  :-)
> >> >
> >>
> >> Well, if the code calculating max_slots_needed were underestimating
> then
> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
> >> number of ring slots consumed.
> >>
> >> > But probably because "npo->copy_prod++" seems to be used for the
> frags
> >> ..
> >> > and it isn't added to  npo->meta_prod ?
> >> >
> >>
> >> meta_slots_used is calculated as the value of meta_prod at return (from
> >> xenvif_gop_skb()) minus the value on entry , and if you look back up the
> >> code then you can see that meta_prod is incremented every time
> >> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot
> without
> >> evaluating RING_GET_REQUEST() and I think that's exactly what's
> >> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
> >> simply incremented in the case of a GSO. So the BUG_ON() is indeed off
> by
> >> one.
> >>
> 
> > Can you re-test with the following patch applied?
> 
> >   Paul
> 
> > diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-
> netback/netback
> > index 438d0c0..4f24220 100644
> > --- a/drivers/net/xen-netback/netback.c
> > +++ b/drivers/net/xen-netback/netback.c
> > @@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)
> 
> >         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
> >                 RING_IDX max_slots_needed;
> > +               RING_IDX old_req_cons;
> > +               RING_IDX ring_slots_used;
> >                 int i;
> 
> >                 /* We need a cheap worse case estimate for the number of
> > @@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
> >                         vif->rx_last_skb_slots = 0;
> 
> >                 sco = (struct skb_cb_overlay *)skb->cb;
> > +
> > +               old_req_cons = vif->rx.req_cons;
> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> > -               BUG_ON(sco->meta_slots_used > max_slots_needed);
> > +               ring_slots_used = vif->rx.req_cons - old_req_cons;
> > +
> > +               BUG_ON(ring_slots_used > max_slots_needed);
> 
> >                 __skb_queue_tail(&rxq, skb);
> >         }
> 
> That blew pretty fast .. on that BUG_ON
> 

Good. That's what should have happened :-)

  Paul

> [  290.218182] ------------[ cut here ]------------
> [  290.225425] kernel BUG at drivers/net/xen-netback/netback.c:664!
> [  290.232717] invalid opcode: 0000 [#1] SMP
> [  290.239875] Modules linked in:
> [  290.246923] CPU: 0 PID: 10447 Comm: vif7.0 Not tainted 3.13.6-20140326-
> nbdebug35+ #1
> [  290.254040] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS
> V1.8B1 09/13/2010
> [  290.261313] task: ffff880055d16480 ti: ffff88004cb7e000 task.ti:
> ffff88004cb7e000
> [  290.268713] RIP: e030:[<ffffffff81780430>]  [<ffffffff81780430>]
> xenvif_rx_action+0x1650/0x1670
> [  290.276193] RSP: e02b:ffff88004cb7fc28  EFLAGS: 00010202
> [  290.283555] RAX: 0000000000000006 RBX: ffff88004c630000 RCX:
> 3fffffffffffffff
> [  290.290908] RDX: 00000000ffffffff RSI: ffff88004c630940 RDI:
> 0000000000048e7b
> [  290.298325] RBP: ffff88004cb7fde8 R08: 0000000000007bc9 R09:
> 0000000000000005
> [  290.305809] R10: ffff88004cb7fd28 R11: ffffc90012690600 R12:
> 0000000000000004
> [  290.313217] R13: ffff8800536a84e0 R14: 0000000000000001 R15:
> ffff88004c637618
> [  290.320521] FS:  00007f1d3030c700(0000) GS:ffff88005f600000(0000)
> knlGS:0000000000000000
> [  290.327839] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  290.335216] CR2: ffffffffff600400 CR3: 0000000058537000 CR4:
> 0000000000000660
> [  290.342732] Stack:
> [  290.350129]  ffff88004cb7fd2c ffff880000000005 ffff88004cb7fd28
> ffffffff810f7fc8
> [  290.357652]  ffff880055d16b50 ffffffff00000407 ffff880000000000
> ffffffff00000000
> [  290.365048]  ffff880055d16b50 ffff880000000001 ffff880000000001
> ffffffff00000000
> [  290.372461] Call Trace:
> [  290.379806]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
> [  290.387211]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
> [  290.394552]  [<ffffffff81781400>] xenvif_kthread+0x40/0x190
> [  290.401808]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
> [  290.408993]  [<ffffffff817813c0>] ? xenvif_stop_queue+0x60/0x60
> [  290.416238]  [<ffffffff810d4f24>] kthread+0xe4/0x100
> [  290.423428]  [<ffffffff81b4cf30>] ? _raw_spin_unlock_irq+0x30/0x50
> [  290.430615]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
> [  290.437793]  [<ffffffff81b4e13c>] ret_from_fork+0x7c/0xb0
> [  290.444945]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
> [  290.452091] Code: fd ff ff 48 8b b5 f0 fe ff ff 48 c7 c2 10 98 ce 81 31 c0 48 8b
> be c8 7c 00 00 48 c7 c6 f0 f1 fd 81 e8 35 be 24 00 e9 ba f8 ff ff <0f> 0b 0f 0b 41
> bf 01 00 00 00 e9 55 f6 ff ff 0f 0b 66 66 66 66
> [  290.467121] RIP  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
> [  290.474436]  RSP <ffff88004cb7fc28>
> [  290.482400] ---[ end trace 2fcf9e9ae26950b3 ]---

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Thursday, March 27, 2014, 10:47:02 AM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 19:57
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 6:48:15 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Paul Durrant
>> >> Sent: 26 March 2014 17:47
>> >> To: 'Sander Eikelenboom'
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >> Re-send shortened version...
>> >>
>> >> > -----Original Message-----
>> >> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> > Sent: 26 March 2014 16:54
>> >> > To: Paul Durrant
>> >> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> > kernel; netdev@vger.kernel.org
>> >> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> > troubles "bisected"
>> >> >
>> >> [snip]
>> >> > >>
>> >> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
>> while
>> >> > prod
>> >> > >> == cons ... but we still have bytes and size left ..
>> >> > >> - start_new_rx_buffer() has returned true ..
>> >> > >> - so we end up in get_next_rx_buffer
>> >> > >> - this does a RING_GET_REQUEST and ups cons ..
>> >> > >> - and we end up with a bad grant reference.
>> >> > >>
>> >> > >> Sometimes we are saved by the bell .. since additional slots have
>> >> become
>> >> > >> free (you see cons become > prod in "get_next_rx_buffer" but
>> shortly
>> >> > after
>> >> > >> that prod is increased ..
>> >> > >> just in time to not cause a overrun).
>> >> > >>
>> >> >
>> >> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> >> > max_slots_needed, so if we are overflowing the worst-case calculation
>> >> then
>> >> > why is that BUG_ON not firing?
>> >> >
>> >> > You mean:
>> >> >                 sco = (struct skb_cb_overlay *)skb->cb;
>> >> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >> >
>> >> > in "get_next_rx_buffer" ?
>> >> >
>> >>
>> >> That code excerpt is from net_rx_action(),isn't it?
>> >>
>> >> > I don't know .. at least now it doesn't crash dom0 and therefore not my
>> >> > complete machine and since tcp is recovering from a failed packet  :-)
>> >> >
>> >>
>> >> Well, if the code calculating max_slots_needed were underestimating
>> then
>> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> >> number of ring slots consumed.
>> >>
>> >> > But probably because "npo->copy_prod++" seems to be used for the
>> frags
>> >> ..
>> >> > and it isn't added to  npo->meta_prod ?
>> >> >
>> >>
>> >> meta_slots_used is calculated as the value of meta_prod at return (from
>> >> xenvif_gop_skb()) minus the value on entry , and if you look back up the
>> >> code then you can see that meta_prod is incremented every time
>> >> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot
>> without
>> >> evaluating RING_GET_REQUEST() and I think that's exactly what's
>> >> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
>> >> simply incremented in the case of a GSO. So the BUG_ON() is indeed off
>> by
>> >> one.
>> >>
>> 
>> > Can you re-test with the following patch applied?
>> 
>> >   Paul
>> 
>> > diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-
>> netback/netback
>> > index 438d0c0..4f24220 100644
>> > --- a/drivers/net/xen-netback/netback.c
>> > +++ b/drivers/net/xen-netback/netback.c
>> > @@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)
>> 
>> >         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
>> >                 RING_IDX max_slots_needed;
>> > +               RING_IDX old_req_cons;
>> > +               RING_IDX ring_slots_used;
>> >                 int i;
>> 
>> >                 /* We need a cheap worse case estimate for the number of
>> > @@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
>> >                         vif->rx_last_skb_slots = 0;
>> 
>> >                 sco = (struct skb_cb_overlay *)skb->cb;
>> > +
>> > +               old_req_cons = vif->rx.req_cons;
>> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> > -               BUG_ON(sco->meta_slots_used > max_slots_needed);
>> > +               ring_slots_used = vif->rx.req_cons - old_req_cons;
>> > +
>> > +               BUG_ON(ring_slots_used > max_slots_needed);
>> 
>> >                 __skb_queue_tail(&rxq, skb);
>> >         }
>> 
>> That blew pretty fast .. on that BUG_ON
>> 

> Good. That's what should have happened :-)

Yes .. and No ..
We shouldn't be there in the first place :-)

Since now every miscalculation in the needed slots leads to a nice remote DOS attack ..
(since we now crash the vif kthread)
it would be nice to have a worst case slot calculation .. with some theoretical guarantees

--
Sander

>   Paul

>> [  290.218182] ------------[ cut here ]------------
>> [  290.225425] kernel BUG at drivers/net/xen-netback/netback.c:664!
>> [  290.232717] invalid opcode: 0000 [#1] SMP
>> [  290.239875] Modules linked in:
>> [  290.246923] CPU: 0 PID: 10447 Comm: vif7.0 Not tainted 3.13.6-20140326-
>> nbdebug35+ #1
>> [  290.254040] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS
>> V1.8B1 09/13/2010
>> [  290.261313] task: ffff880055d16480 ti: ffff88004cb7e000 task.ti:
>> ffff88004cb7e000
>> [  290.268713] RIP: e030:[<ffffffff81780430>]  [<ffffffff81780430>]
>> xenvif_rx_action+0x1650/0x1670
>> [  290.276193] RSP: e02b:ffff88004cb7fc28  EFLAGS: 00010202
>> [  290.283555] RAX: 0000000000000006 RBX: ffff88004c630000 RCX:
>> 3fffffffffffffff
>> [  290.290908] RDX: 00000000ffffffff RSI: ffff88004c630940 RDI:
>> 0000000000048e7b
>> [  290.298325] RBP: ffff88004cb7fde8 R08: 0000000000007bc9 R09:
>> 0000000000000005
>> [  290.305809] R10: ffff88004cb7fd28 R11: ffffc90012690600 R12:
>> 0000000000000004
>> [  290.313217] R13: ffff8800536a84e0 R14: 0000000000000001 R15:
>> ffff88004c637618
>> [  290.320521] FS:  00007f1d3030c700(0000) GS:ffff88005f600000(0000)
>> knlGS:0000000000000000
>> [  290.327839] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [  290.335216] CR2: ffffffffff600400 CR3: 0000000058537000 CR4:
>> 0000000000000660
>> [  290.342732] Stack:
>> [  290.350129]  ffff88004cb7fd2c ffff880000000005 ffff88004cb7fd28
>> ffffffff810f7fc8
>> [  290.357652]  ffff880055d16b50 ffffffff00000407 ffff880000000000
>> ffffffff00000000
>> [  290.365048]  ffff880055d16b50 ffff880000000001 ffff880000000001
>> ffffffff00000000
>> [  290.372461] Call Trace:
>> [  290.379806]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
>> [  290.387211]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
>> [  290.394552]  [<ffffffff81781400>] xenvif_kthread+0x40/0x190
>> [  290.401808]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
>> [  290.408993]  [<ffffffff817813c0>] ? xenvif_stop_queue+0x60/0x60
>> [  290.416238]  [<ffffffff810d4f24>] kthread+0xe4/0x100
>> [  290.423428]  [<ffffffff81b4cf30>] ? _raw_spin_unlock_irq+0x30/0x50
>> [  290.430615]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
>> [  290.437793]  [<ffffffff81b4e13c>] ret_from_fork+0x7c/0xb0
>> [  290.444945]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
>> [  290.452091] Code: fd ff ff 48 8b b5 f0 fe ff ff 48 c7 c2 10 98 ce 81 31 c0 48 8b
>> be c8 7c 00 00 48 c7 c6 f0 f1 fd 81 e8 35 be 24 00 e9 ba f8 ff ff <0f> 0b 0f 0b 41
>> bf 01 00 00 00 e9 55 f6 ff ff 0f 0b 66 66 66 66
>> [  290.467121] RIP  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
>> [  290.474436]  RSP <ffff88004cb7fc28>
>> [  290.482400] ---[ end trace 2fcf9e9ae26950b3 ]---

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Thursday, March 27, 2014, 10:47:02 AM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 19:57
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 6:48:15 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Paul Durrant
>> >> Sent: 26 March 2014 17:47
>> >> To: 'Sander Eikelenboom'
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >> Re-send shortened version...
>> >>
>> >> > -----Original Message-----
>> >> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> > Sent: 26 March 2014 16:54
>> >> > To: Paul Durrant
>> >> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> > kernel; netdev@vger.kernel.org
>> >> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> > troubles "bisected"
>> >> >
>> >> [snip]
>> >> > >>
>> >> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy"
>> while
>> >> > prod
>> >> > >> == cons ... but we still have bytes and size left ..
>> >> > >> - start_new_rx_buffer() has returned true ..
>> >> > >> - so we end up in get_next_rx_buffer
>> >> > >> - this does a RING_GET_REQUEST and ups cons ..
>> >> > >> - and we end up with a bad grant reference.
>> >> > >>
>> >> > >> Sometimes we are saved by the bell .. since additional slots have
>> >> become
>> >> > >> free (you see cons become > prod in "get_next_rx_buffer" but
>> shortly
>> >> > after
>> >> > >> that prod is increased ..
>> >> > >> just in time to not cause a overrun).
>> >> > >>
>> >> >
>> >> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
>> >> > max_slots_needed, so if we are overflowing the worst-case calculation
>> >> then
>> >> > why is that BUG_ON not firing?
>> >> >
>> >> > You mean:
>> >> >                 sco = (struct skb_cb_overlay *)skb->cb;
>> >> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> >> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
>> >> >
>> >> > in "get_next_rx_buffer" ?
>> >> >
>> >>
>> >> That code excerpt is from net_rx_action(),isn't it?
>> >>
>> >> > I don't know .. at least now it doesn't crash dom0 and therefore not my
>> >> > complete machine and since tcp is recovering from a failed packet  :-)
>> >> >
>> >>
>> >> Well, if the code calculating max_slots_needed were underestimating
>> then
>> >> the BUG_ON() should fire. If it is not firing in your case then this suggests
>> >> your problem lies elsewhere, or that meta_slots_used is not equal to the
>> >> number of ring slots consumed.
>> >>
>> >> > But probably because "npo->copy_prod++" seems to be used for the
>> frags
>> >> ..
>> >> > and it isn't added to  npo->meta_prod ?
>> >> >
>> >>
>> >> meta_slots_used is calculated as the value of meta_prod at return (from
>> >> xenvif_gop_skb()) minus the value on entry , and if you look back up the
>> >> code then you can see that meta_prod is incremented every time
>> >> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot
>> without
>> >> evaluating RING_GET_REQUEST() and I think that's exactly what's
>> >> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
>> >> simply incremented in the case of a GSO. So the BUG_ON() is indeed off
>> by
>> >> one.
>> >>
>> 
>> > Can you re-test with the following patch applied?
>> 
>> >   Paul
>> 
>> > diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-
>> netback/netback
>> > index 438d0c0..4f24220 100644
>> > --- a/drivers/net/xen-netback/netback.c
>> > +++ b/drivers/net/xen-netback/netback.c
>> > @@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)
>> 
>> >         while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
>> >                 RING_IDX max_slots_needed;
>> > +               RING_IDX old_req_cons;
>> > +               RING_IDX ring_slots_used;
>> >                 int i;
>> 
>> >                 /* We need a cheap worse case estimate for the number of
>> > @@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
>> >                         vif->rx_last_skb_slots = 0;
>> 
>> >                 sco = (struct skb_cb_overlay *)skb->cb;
>> > +
>> > +               old_req_cons = vif->rx.req_cons;
>> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>> > -               BUG_ON(sco->meta_slots_used > max_slots_needed);
>> > +               ring_slots_used = vif->rx.req_cons - old_req_cons;
>> > +
>> > +               BUG_ON(ring_slots_used > max_slots_needed);
>> 
>> >                 __skb_queue_tail(&rxq, skb);
>> >         }
>> 
>> That blew pretty fast .. on that BUG_ON
>> 

> Good. That's what should have happened :-)

Yes .. and No ..
We shouldn't be there in the first place :-)

Since now every miscalculation in the needed slots leads to a nice remote DOS attack ..
(since we now crash the vif kthread)
it would be nice to have a worst case slot calculation .. with some theoretical guarantees

--
Sander

>   Paul

>> [  290.218182] ------------[ cut here ]------------
>> [  290.225425] kernel BUG at drivers/net/xen-netback/netback.c:664!
>> [  290.232717] invalid opcode: 0000 [#1] SMP
>> [  290.239875] Modules linked in:
>> [  290.246923] CPU: 0 PID: 10447 Comm: vif7.0 Not tainted 3.13.6-20140326-
>> nbdebug35+ #1
>> [  290.254040] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS
>> V1.8B1 09/13/2010
>> [  290.261313] task: ffff880055d16480 ti: ffff88004cb7e000 task.ti:
>> ffff88004cb7e000
>> [  290.268713] RIP: e030:[<ffffffff81780430>]  [<ffffffff81780430>]
>> xenvif_rx_action+0x1650/0x1670
>> [  290.276193] RSP: e02b:ffff88004cb7fc28  EFLAGS: 00010202
>> [  290.283555] RAX: 0000000000000006 RBX: ffff88004c630000 RCX:
>> 3fffffffffffffff
>> [  290.290908] RDX: 00000000ffffffff RSI: ffff88004c630940 RDI:
>> 0000000000048e7b
>> [  290.298325] RBP: ffff88004cb7fde8 R08: 0000000000007bc9 R09:
>> 0000000000000005
>> [  290.305809] R10: ffff88004cb7fd28 R11: ffffc90012690600 R12:
>> 0000000000000004
>> [  290.313217] R13: ffff8800536a84e0 R14: 0000000000000001 R15:
>> ffff88004c637618
>> [  290.320521] FS:  00007f1d3030c700(0000) GS:ffff88005f600000(0000)
>> knlGS:0000000000000000
>> [  290.327839] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [  290.335216] CR2: ffffffffff600400 CR3: 0000000058537000 CR4:
>> 0000000000000660
>> [  290.342732] Stack:
>> [  290.350129]  ffff88004cb7fd2c ffff880000000005 ffff88004cb7fd28
>> ffffffff810f7fc8
>> [  290.357652]  ffff880055d16b50 ffffffff00000407 ffff880000000000
>> ffffffff00000000
>> [  290.365048]  ffff880055d16b50 ffff880000000001 ffff880000000001
>> ffffffff00000000
>> [  290.372461] Call Trace:
>> [  290.379806]  [<ffffffff810f7fc8>] ? __lock_acquire+0x418/0x2220
>> [  290.387211]  [<ffffffff810df5f6>] ? finish_task_switch+0x46/0xf0
>> [  290.394552]  [<ffffffff81781400>] xenvif_kthread+0x40/0x190
>> [  290.401808]  [<ffffffff810f05e0>] ? __init_waitqueue_head+0x60/0x60
>> [  290.408993]  [<ffffffff817813c0>] ? xenvif_stop_queue+0x60/0x60
>> [  290.416238]  [<ffffffff810d4f24>] kthread+0xe4/0x100
>> [  290.423428]  [<ffffffff81b4cf30>] ? _raw_spin_unlock_irq+0x30/0x50
>> [  290.430615]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
>> [  290.437793]  [<ffffffff81b4e13c>] ret_from_fork+0x7c/0xb0
>> [  290.444945]  [<ffffffff810d4e40>] ? __init_kthread_worker+0x70/0x70
>> [  290.452091] Code: fd ff ff 48 8b b5 f0 fe ff ff 48 c7 c2 10 98 ce 81 31 c0 48 8b
>> be c8 7c 00 00 48 c7 c6 f0 f1 fd 81 e8 35 be 24 00 e9 ba f8 ff ff <0f> 0b 0f 0b 41
>> bf 01 00 00 00 e9 55 f6 ff ff 0f 0b 66 66 66 66
>> [  290.467121] RIP  [<ffffffff81780430>] xenvif_rx_action+0x1650/0x1670
>> [  290.474436]  RSP <ffff88004cb7fc28>
>> [  290.482400] ---[ end trace 2fcf9e9ae26950b3 ]---

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Paul Durrant
> Sent: 26 March 2014 17:47
> To: 'Sander Eikelenboom'
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: RE: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> Re-send shortened version...
> 
> > -----Original Message-----
> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > Sent: 26 March 2014 16:54
> > To: Paul Durrant
> > Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> > kernel; netdev@vger.kernel.org
> > Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> > troubles "bisected"
> >
> [snip]
> > >>
> > >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> > prod
> > >> == cons ... but we still have bytes and size left ..
> > >> - start_new_rx_buffer() has returned true ..
> > >> - so we end up in get_next_rx_buffer
> > >> - this does a RING_GET_REQUEST and ups cons ..
> > >> - and we end up with a bad grant reference.
> > >>
> > >> Sometimes we are saved by the bell .. since additional slots have
> become
> > >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> > after
> > >> that prod is increased ..
> > >> just in time to not cause a overrun).
> > >>
> >
> > > Ah, but hang on... There's a BUG_ON meta_slots_used >
> > max_slots_needed, so if we are overflowing the worst-case calculation
> then
> > why is that BUG_ON not firing?
> >
> > You mean:
> >                 sco = (struct skb_cb_overlay *)skb->cb;
> >                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
> >                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> >
> > in "get_next_rx_buffer" ?
> >
> 
> That code excerpt is from net_rx_action(),isn't it?
> 
> > I don't know .. at least now it doesn't crash dom0 and therefore not my
> > complete machine and since tcp is recovering from a failed packet  :-)
> >
> 
> Well, if the code calculating max_slots_needed were underestimating then
> the BUG_ON() should fire. If it is not firing in your case then this suggests
> your problem lies elsewhere, or that meta_slots_used is not equal to the
> number of ring slots consumed.
> 
> > But probably because "npo->copy_prod++" seems to be used for the frags
> ..
> > and it isn't added to  npo->meta_prod ?
> >
> 
> meta_slots_used is calculated as the value of meta_prod at return (from
> xenvif_gop_skb()) minus the value on entry , and if you look back up the
> code then you can see that meta_prod is incremented every time
> RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without
> evaluating RING_GET_REQUEST() and I think that's exactly what's
> happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is
> simply incremented in the case of a GSO. So the BUG_ON() is indeed off by
> one.
> 

Can you re-test with the following patch applied?

  Paul

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback
index 438d0c0..4f24220 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -482,6 +482,8 @@ static void xenvif_rx_action(struct xenvif *vif)

        while ((skb = skb_dequeue(&vif->rx_queue)) != NULL) {
                RING_IDX max_slots_needed;
+               RING_IDX old_req_cons;
+               RING_IDX ring_slots_used;
                int i;

                /* We need a cheap worse case estimate for the number of
@@ -511,8 +513,12 @@ static void xenvif_rx_action(struct xenvif *vif)
                        vif->rx_last_skb_slots = 0;

                sco = (struct skb_cb_overlay *)skb->cb;
+
+               old_req_cons = vif->rx.req_cons;
                sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
-               BUG_ON(sco->meta_slots_used > max_slots_needed);
+               ring_slots_used = vif->rx.req_cons - old_req_cons;
+
+               BUG_ON(ring_slots_used > max_slots_needed);

                __skb_queue_tail(&rxq, skb);
        }

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Hi Paul,

Seems your last mail arrived in pretty bad shape (truncated) in my mailbox ..

--
Sander

Wednesday, March 26, 2014, 6:16:49 PM, you wrote:

>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Sent: 26 March 2014 16:54
>> To: Paul Durrant
>> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
>> kernel; netdev@vger.kernel.org
>> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> troubles "bisected"
>> 
>> 
>> Wednesday, March 26, 2014, 5:25:21 PM, you wrote:
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: 26 March 2014 16:07
>> >> To: Paul Durrant
>> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> linux-
>> >> kernel; netdev@vger.kernel.org
>> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> troubles "bisected"
>> >>
>> >>
>> >> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> Sent: 26 March 2014 15:23
>> >> >> To: Paul Durrant
>> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
>> >> linux-
>> >> >> kernel; netdev@vger.kernel.org
>> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
>> >> >> troubles "bisected"
>> >> >>
>> >> >>
>> >> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
>> >> >>
>> >> >> >> -----Original Message-----
>> >> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> >> Sent: 26 March 2014 11:11
>> >> >> >> To: Paul Durrant
>> >> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian
>> Campbell;
>> >> >> linux-
>> >> >> >> kernel; netdev@vger.kernel.org
>> >> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13
>> Network
>> >> >> >> troubles "bisected"
>> >> >> >>
>> >> >> >> Paul,
>> >> >> >>
>> >> >> >> You have been awfully silent for this whole thread while this is a
>> >> >> regression
>> >> >> >> caused by a patch of you
>> >> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
>> >> earlier
>> >> >> in
>> >> >> >> this thread).
>> >> >> >>
>> >> >>
>> >> >> > Sorry, I've been distracted...
>> >> >>
>> >> >> >> The commit messages states:
>> >> >> >>     "net_rx_action() is the place where we could do with an accurate
>> >> >> >> predicition but,
>> >> >> >>     since that has proven tricky to calculate, a cheap worse-case (but
>> not
>> >> >> too
>> >> >> >> bad)
>> >> >> >>     estimate is all we really need since the only thing we *must*
>> prevent
>> >> is
>> >> >> >> xenvif_gop_skb()
>> >> >> >>     consuming more slots than are available."
>> >> >> >>
>> >> >> >> Your "worst-case" calculation stated in the commit message is
>> clearly
>> >> not
>> >> >> the
>> >> >> >> worst case,
>> >> >> >> since it doesn't take calls to "get_next_rx_buffer" into account

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 16:54
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Wednesday, March 26, 2014, 5:25:21 PM, you wrote:
> 
> >> -----Original Message-----
> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> Sent: 26 March 2014 16:07
> >> To: Paul Durrant
> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> linux-
> >> kernel; netdev@vger.kernel.org
> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> troubles "bisected"
> >>
> >>
> >> Wednesday, March 26, 2014, 4:50:30 PM, you wrote:
> >>
> >> >> -----Original Message-----
> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> Sent: 26 March 2014 15:23
> >> >> To: Paul Durrant
> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell;
> >> linux-
> >> >> kernel; netdev@vger.kernel.org
> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> >> >> troubles "bisected"
> >> >>
> >> >>
> >> >> Wednesday, March 26, 2014, 3:44:42 PM, you wrote:
> >> >>
> >> >> >> -----Original Message-----
> >> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> >> >> >> Sent: 26 March 2014 11:11
> >> >> >> To: Paul Durrant
> >> >> >> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian
> Campbell;
> >> >> linux-
> >> >> >> kernel; netdev@vger.kernel.org
> >> >> >> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13
> Network
> >> >> >> troubles "bisected"
> >> >> >>
> >> >> >> Paul,
> >> >> >>
> >> >> >> You have been awfully silent for this whole thread while this is a
> >> >> regression
> >> >> >> caused by a patch of you
> >> >> >> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much
> >> earlier
> >> >> in
> >> >> >> this thread).
> >> >> >>
> >> >>
> >> >> > Sorry, I've been distracted...
> >> >>
> >> >> >> The commit messages states:
> >> >> >>     "net_rx_action() is the place where we could do with an accurate
> >> >> >> predicition but,
> >> >> >>     since that has proven tricky to calculate, a cheap worse-case (but
> not
> >> >> too
> >> >> >> bad)
> >> >> >>     estimate is all we really need since the only thing we *must*
> prevent
> >> is
> >> >> >> xenvif_gop_skb()
> >> >> >>     consuming more slots than are available."
> >> >> >>
> >> >> >> Your "worst-case" calculation stated in the commit message is
> clearly
> >> not
> >> >> the
> >> >> >> worst case,
> >> >> >> since it doesn't take calls to "get_next_rx_buffer" into account.
> >> >> >>
> >> >>
> >> >> > It should be taking into account the behaviour of
> >> start_new_rx_buffer(),
> >> >> which should be true if a slot is full or a frag will overflow the current
> slot
> >> and
> >> >> doesn't require splitting.
> >> >> > The code in net_rx_action() makes the assumption that each frag will
> >> >> require as many slots as its size requires, i.e. it assumes no packing of
> >> >> multiple frags into a single slot, so it should be a worst case.
> >> >> > Did I miss something in that logic?
> >> >>
> >> >> Yes.
> >> >> In "xenvif_gop_skb()" this loop:
> >> >>
> >> >>         for (i = 0; i < nr_frags; i++) {
> >> >>                 xenvif_gop_frag_copy(vif, skb, npo,
> >> >>                                      skb_frag_page(&skb_shinfo(skb)->frags[i]),
> >> >>                                      skb_frag_size(&skb_shinfo(skb)->frags[i]),
> >> >>                                      skb_shinfo(skb)->frags[i].page_offset,
> >> >>                                      &head);
> >> >>         }
> >> >>
> >> >> Is capable of using up (at least) 1 slot more than is anticipated for in
> >> >> "net_rx_action()"  by this code:
> >> >>
> >> >>                 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> >> >>                         unsigned int size;
> >> >>                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >>                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >> >>                 }
> >> >>
> >> >> And this happens when it calls "get_next_rx_buffer()" from
> >> >> "xenvif_gop_frag_copy()" where it's breaking down the frag.
> >> >>
> >>
> >> > The function that determines whether to consume another slot is
> >> start_new_rx_buffer() and for each frag I don't see why this would return
> >> true more than DIV_ROUND_UP(size, PAGE_SIZE) times.
> >> > It may be called more times than that since the code in
> >> xenvif_gop_frag_copy() must also allow for the offset of the frag but
> should
> >> not return true in all cases.
> >> > So, I still cannot see why a frag would ever consume more than
> >> DIV_ROUND_UP(size, PAGE_SIZE) slots.
> >>
> >> Well here a case were a frag is broken down in 2 pieces:
> >>
> >> [ 1156.870372] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 1  npo-
> >> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105867
> >> npo->copy_gref:760  npo->copy_off:4096  MAX_BUFFER_OFFSET:4096
> >> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >> >req_event:2104275 estimated_slots_needed:4 reserved_slots_left:0
> >> [ 1156.871971] vif vif-7-0 vif7.0: ?!? xenvif_start_xmit stopping queue !
> >> min_slots_needed:1 min_slots_needed_2:0 min_slots_needed_3:0 vif-
> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105867 vif->rx.sring-
> >> >req_event:2105868 skb->len:66 skb->data_len:0
> >> [ 1156.964316] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer before req npo-
> >> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105867
> >> vif->rx.sring->req_event:2105868, reserved_slots_left:0
> >> [ 1157.001635] vif vif-7-0 vif7.0: ?!? get_next_rx_buffer after req npo-
> >> >meta_prod:39 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868
> >> req->gref:4325379 req->id:11 vif->rx.sring->req_event:2105868
> >> reserved_slots_left:-1
> >> [ 1157.039095] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here 2  npo-
> >> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868
> >> npo->copy_gref:4325379  npo->copy_off:0  MAX_BUFFER_OFFSET:4096
> >> bytes:560 size:560  offset:0 head:1273462060 i:2 vif->rx.sring-
> >> >req_event:2105868 estimated_slots_needed:4 reserved_slots_left:-1
> >> [ 1157.095216] vif vif-7-0 vif7.0: ?!? xenvif_gop_frag_copy Me here end
> npo-
> >> >meta_prod:40 vif->rx.sring->req_prod:2105867 vif-
> >rx.req_cons:2105868
> >> npo->copy_gref:4325379 npo->copy_off:560  MAX_BUFFER_OFFSET:4096
> >> bytes:560 size:0  offset:560 head:1273462060 i:3 vif->rx.sring-
> >> >req_event:2105868 gso_gaps:0 estimated_slots_needed:4
> >> reserved_slots_left:-1
> >> [ 1157.151338] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 4 after npo-
> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> >> req->gref:657 req->id:7 estimated_slots_needed:4 i(frag):0 j(data):1
> >> reserved_slots_left:-1
> >> [ 1157.188908] vif vif-7-0 vif7.0: ?!? xenvif_gop_skb Me here 5 npo-
> >> >meta_prod:40 old_meta_prod:36 vif->rx.sring->req_prod:2105867 vif-
> >> >rx.req_cons:2105868 meta->gso_type:1 meta->gso_size:1448 nr_frags:1
> >> req->gref:657 req->id:7 estimated_slots_needed:4 j(data):1
> >> reserved_slots_left:-1    used in funcstart: 0 + 1 .. used_dataloop:1 ..
> >> used_fragloop:3
> >> [ 1157.244975] vif vif-7-0 vif7.0: ?!? xenvif_rx_action me here 2 ..  vif-
> >> >rx.sring->req_prod:2105867 vif->rx.req_cons:2105868 sco-
> >> >meta_slots_used:4 max_upped_gso:1 skb_is_gso(skb):1
> >> max_slots_needed:4 j:6 is_gso:1 nr_frags:1 firstpart:1 secondpart:2
> >> reserved_slots_left:-1
> >>
> >> - When processing an SKB we end up in "xenvif_gop_frag_copy" while
> prod
> >> == cons ... but we still have bytes and size left ..
> >> - start_new_rx_buffer() has returned true ..
> >> - so we end up in get_next_rx_buffer
> >> - this does a RING_GET_REQUEST and ups cons ..
> >> - and we end up with a bad grant reference.
> >>
> >> Sometimes we are saved by the bell .. since additional slots have become
> >> free (you see cons become > prod in "get_next_rx_buffer" but shortly
> after
> >> that prod is increased ..
> >> just in time to not cause a overrun).
> >>
> 
> > Ah, but hang on... There's a BUG_ON meta_slots_used >
> max_slots_needed, so if we are overflowing the worst-case calculation then
> why is that BUG_ON not firing?
> 
> You mean:
>                 sco = (struct skb_cb_overlay *)skb->cb;
>                 sco->meta_slots_used = xenvif_gop_skb(skb, &npo);
>                 BUG_ON(sco->meta_slots_used > max_slots_needed);
> 
> in "get_next_rx_buffer" ?
> 

That code excerpt is from net_rx_action(),isn't it?

> I don't know .. at least now it doesn't crash dom0 and therefore not my
> complete machine and since tcp is recovering from a failed packet  :-)
> 

Well, if the code calculating max_slots_needed were underestimating then the BUG_ON() should fire. If it is not firing in your case then this suggests your problem lies elsewhere, or that meta_slots_used is not equal to the number of ring slots consumed.

> But probably because "npo->copy_prod++" seems to be used for the frags ..
> and it isn't added to  npo->meta_prod ?
> 

meta_slots_used is calculated as the value of meta_prod at return (from xenvif_gop_skb()) minus the value on entry , and if you look back up the code then you can see that meta_prod is incremented every time RING_GET_REQUEST() is evaluated. So, we must be consuming a slot without evaluating RING_GET_REQUEST() and I think that's exactly what's happening... Right at the bottom of xenvif_gop_frag_copy() req_cons is simply incremented in the case of a GSO. So the BUG_ON() is indeed off by one.

  Paul

> --
> Sander
> 
> >   Paul
> 
> >> If you need additional / other info, please cook up a debug patch with
> what
> >> you need.
> >>
> >> --
> >> Sander
> >>
> >>
> >>
> >>
> >>
> >>
> >> >   Paul
> >>
> >> >> Ultimately this results in bad grant reference warnings (and packets
> >> marked
> >> >> as "errors" in the interface statistics).
> >> >>
> >> >> In my case it always seems to be a skb with 1 frag which is broken
> down in
> >> 5
> >> >> or 6 pieces ..
> >> >>
> >> >> So "get_next_rx_buffer()" is called once .. and i'm overrunning the ring
> >> with
> >> >> 1 slot, but i'm not sure if that's not coincedence
> >> >> since in the code there seem to be no explicit limitation on how often
> this
> >> >> code path is taken. So perhaps it's implicitly limited
> >> >> since packets and frags can't be arbitrarily large in comparison with the
> >> >> page_size but that's not something i'm capable of figuring out :-)
> >> >>
> >> >>
> >> >>
> >> >> >   Paul
> >> >>
> >> >> >> Problem is that a worst case calculation would probably be reverting
> to
> >> >> the
> >> >> >> old calculation,
> >> >> >> and the problems this patch was trying to solve would reappear,
> but
> >> >> >> introducing new regressions
> >> >> >> isn't very useful either. And since it seems such a tricky and fragile
> >> thing to
> >> >> >> determine, it would
> >> >> >> probably be best to be split into a distinct function with a comment
> to
> >> >> explain
> >> >> >> the rationale used.
> >> >> >>
> >> >> >> Since this doesn't seem to progress very fast .. CC'ed some more
> folks
> >> ..
> >> >> you
> >> >> >> never know ..
> >> >> >>
> >> >> >> --
> >> >> >> Sander
> >> >> >>
> >> >> >>
> >> >> >> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> >> >> >>
> >> >> >>
> >> >> >> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> >> >> >>
> >> >> >> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom
> >> wrote:
> >> >> >> >> [...]
> >> >> >> >>> > Yes there is only one frag .. but it seems to be much larger
> than
> >> >> >> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into
> >> >> smaller
> >> >> >> bits .. hence the calculation in xenvif_rx_action determining the
> slots
> >> >> needed
> >> >> >> by doing:
> >> >> >> >>>
> >> >> >> >>> >                 for (i = 0; i < nr_frags; i++) {
> >> >> >> >>> >                         unsigned int size;
> >> >> >> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >> >> >>> >                         max_slots_needed += DIV_ROUND_UP(size,
> >> >> PAGE_SIZE);
> >> >> >> >>> >                 }
> >> >> >> >>>
> >> >> >> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing
> >> one
> >> >> >> more slot (from the emperical test) .. and calling
> "get_next_rx_buffer"
> >> >> >> seems involved in that ..
> >> >> >> >>>
> >> >> >> >>> Hmm looked again .. and it seems this is it .. when your frags
> are
> >> large
> >> >> >> enough you have the chance of running into this.
> >> >> >> >>>
> >> >> >>
> >> >> >> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you
> >> see
> >> >> any
> >> >> >> >> problem with that implementation?
> >> >> >> > In general no, but "get_next_rx_buffer" up's cons .. and the
> >> calculations
> >> >> >> done in "xenvif_rx_action" for max_slots_needed to prevent the
> >> overrun
> >> >> >> > don't count in this possibility. So it's not the guarding of
> >> >> >> "start_new_rx_buffer" that is at fault. It's the ones early in
> >> >> >> "xenvif_rx_action".
> >> >> >> > The ones that were changed by Paul's patch from
> MAX_SKB_FRAGS
> >> to a
> >> >> >> calculated value that should be a "slim fit".
> >> >> >>
> >> >> >> > The problem is in determining upfront in "xenvif_rx_action" when
> >> and
> >> >> how
> >> >> >> often the "get_next_rx_buffer" path will be taken.
> >> >> >> > Unless there are other non direct restrictions (from a size point of
> >> view)
> >> >> it
> >> >> >> can be called multiple times per frag per skb.
> >> >> >>
> >> >> >> >>> Problem is .. i don't see an easy fix, the "one more slot" of the
> >> >> empirical
> >> >> >> test doesn't seem to be the worst case either (i think):
> >> >> >> >>>
> >> >> >> >>> - In my case the packets that hit this only have 1 frag, but i
> could
> >> have
> >> >> >> had more frags.
> >> >> >> >>>   I also think you can't rule out the possibility of doing the
> >> >> >> "get_next_rx_buffer" for multiple subsequent frags from one
> packet,
> >> >> >> >>>   so in the worst (and perhaps even from a single frag since it's
> >> looped
> >> >> >> over a split of it in what seems PAGE_SIZE pieces.)
> >> >> >> >>>   - So an exact calculation of how much slots we are going to
> need
> >> for
> >> >> >> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action"
> seems
> >> >> >> unfeasible.
> >> >> >> >>>   - A worst case gamble seems impossible either .. if you take
> >> multiple
> >> >> >> frags * multiple times the "get_next_rx_buffer" ... you would
> probably
> >> be
> >> >> >> back at just
> >> >> >> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >> >> >> >>>
> >> >> >> >>> - Other thing would be checking for the available slots before
> >> doing
> >> >> the
> >> >> >> "get_next_rx_buffer" .. how ever .. we don't account for how
> many
> >> slots
> >> >> we
> >> >> >> still need to
> >> >> >> >>>   just process the remaining frags.
> >> >> >> >>>
> >> >> >>
> >> >> >> >> We've done a worst case estimation for whole SKB (linear area +
> all
> >> >> >> >> frags) already, at the very first beginning. That's what
> >> >> >> >> max_slots_needed is for.
> >> >> >>
> >> >> >> >>> - Just remove the whole "get_next_rx_buffer".. just tested it
> but
> >> it
> >> >> >> seems the "get_next_rx_buffer" is necessary ..  when i make
> >> >> >> start_new_rx_buffer always return false
> >> >> >> >>>   i hit the bug below :S
> >> >> >> >>>
> >> >> >> >>> So the questions are ... is the "get_next_rx_buffer" part really
> >> >> necessary
> >> >> >> ?
> >> >> >> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and
> >> does
> >> >> that
> >> >> >> outweigh the additional cost of accounting
> >> >> >> >>>      of needed_slots for the frags that have yet to be processed
> ?
> >> >> >> >>>    - if yes, erhmmm what would be the best acceptable solution
> ..
> >> >> >> returning to using MAX_SKB_FRAGS ?
> >> >> >> >>>
> >> >> >>
> >> >> >> >> I think you need to answer several questions first:
> >> >> >> >> 1. is max_slots_needed actually large enough to cover whole
> SKB?
> >> >> >> >         No it's not if, you end up calling "get_next_rx_buffer" one or
> >> >> multiple
> >> >> >> times when processing the SKB
> >> >> >> >         you have the chance of overrunning (or be saved because
> prod
> >> gets
> >> >> >> upped before you overrun).
> >> >> >>
> >> >> >> >> 2. is the test of ring slot availability acurate?
> >> >> >> >         Seems to be.
> >> >> >>
> >> >> >> >> 3. is the number of ring slots consumed larger than
> >> >> max_slots_needed? (I
> >> >> >> >>    guess the answer is yes)
> >> >> >> >         Yes that was the whole point.
> >> >> >>
> >> >> >> >> 4. which step in the break-down procedure causes backend to
> >> overrun
> >> >> >> the
> >> >> >> >>    ring?
> >> >> >> >         The "get_next_rx_buffer" call .. that is not accounted for
> (which
> >> can
> >> >> be
> >> >> >> called
> >> >> >> >         multiple times per frag (unless some other none obvious size
> >> >> >> restriction limits this
> >> >> >> >         to one time per frag or one time per SKB).
> >> >> >> >         In my errorneous case it is called one time, but it would be
> nice if
> >> >> there
> >> >> >> would be some theoretical proof
> >> >> >> >         instead of just some emperical test.
> >> >> >>
> >> >> >>
> >> >> >> >> It doesn't matter how many frags your SKB has and how big a
> frag
> >> is. If
> >> >> >> >> every step is correct then you're fine. The code you're looking at
> >> >> >> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs
> to
> >> be
> >> >> >> >> carefully examined.
> >> >> >>
> >> >> >> > Well it seems it only calls "get_next_rx_buffer" in some special
> cases
> >> ..
> >> >> >> (and that also what i'm seeing because it doesn't happen
> >> >> >> > continously.
> >> >> >>
> >> >> >> > Question is shouldn't this max_needed_slots calc be reverted to
> >> what it
> >> >> >> was before 3.14 and take the time in 3.15 to figure out a
> >> >> >> > the theoretical max (if it can be calculated upfront) .. or another
> way
> >> to
> >> >> >> guarantee the code is able to process the whole SKB  ?
> >> >> >>
> >> >> >> >> Wei.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 26 March 2014 11:11
> To: Paul Durrant
> Cc: Wei Liu; annie li; Zoltan Kiss; xen-devel@lists.xen.org; Ian Campbell; linux-
> kernel; netdev@vger.kernel.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> Paul,
> 
> You have been awfully silent for this whole thread while this is a regression
> caused by a patch of you
> (ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier in
> this thread).
> 

Sorry, I've been distracted...

> The commit messages states:
>     "net_rx_action() is the place where we could do with an accurate
> predicition but,
>     since that has proven tricky to calculate, a cheap worse-case (but not too
> bad)
>     estimate is all we really need since the only thing we *must* prevent is
> xenvif_gop_skb()
>     consuming more slots than are available."
> 
> Your "worst-case" calculation stated in the commit message is clearly not the
> worst case,
> since it doesn't take calls to "get_next_rx_buffer" into account.
> 

It should be taking into account the behaviour of start_new_rx_buffer(), which should be true if a slot is full or a frag will overflow the current slot and doesn't require splitting. The code in net_rx_action() makes the assumption that each frag will require as many slots as its size requires, i.e. it assumes no packing of multiple frags into a single slot, so it should be a worst case. Did I miss something in that logic?

  Paul

> Problem is that a worst case calculation would probably be reverting to the
> old calculation,
> and the problems this patch was trying to solve would reappear, but
> introducing new regressions
> isn't very useful either. And since it seems such a tricky and fragile thing to
> determine, it would
> probably be best to be split into a distinct function with a comment to explain
> the rationale used.
> 
> Since this doesn't seem to progress very fast .. CC'ed some more folks .. you
> never know ..
> 
> --
> Sander
> 
> 
> Tuesday, March 25, 2014, 4:29:42 PM, you wrote:
> 
> 
> > Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> 
> >> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
> >> [...]
> >>> > Yes there is only one frag .. but it seems to be much larger than
> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller
> bits .. hence the calculation in xenvif_rx_action determining the slots needed
> by doing:
> >>>
> >>> >                 for (i = 0; i < nr_frags; i++) {
> >>> >                         unsigned int size;
> >>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >>> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >>> >                 }
> >>>
> >>> > But the code in xenvif_gop_frag_copy .. seems to be needing one
> more slot (from the emperical test) .. and calling "get_next_rx_buffer"
> seems involved in that ..
> >>>
> >>> Hmm looked again .. and it seems this is it .. when your frags are large
> enough you have the chance of running into this.
> >>>
> 
> >> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
> >> problem with that implementation?
> > In general no, but "get_next_rx_buffer" up's cons .. and the calculations
> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
> > don't count in this possibility. So it's not the guarding of
> "start_new_rx_buffer" that is at fault. It's the ones early in
> "xenvif_rx_action".
> > The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
> calculated value that should be a "slim fit".
> 
> > The problem is in determining upfront in "xenvif_rx_action" when and how
> often the "get_next_rx_buffer" path will be taken.
> > Unless there are other non direct restrictions (from a size point of view) it
> can be called multiple times per frag per skb.
> 
> >>> Problem is .. i don't see an easy fix, the "one more slot" of the empirical
> test doesn't seem to be the worst case either (i think):
> >>>
> >>> - In my case the packets that hit this only have 1 frag, but i could have
> had more frags.
> >>>   I also think you can't rule out the possibility of doing the
> "get_next_rx_buffer" for multiple subsequent frags from one packet,
> >>>   so in the worst (and perhaps even from a single frag since it's looped
> over a split of it in what seems PAGE_SIZE pieces.)
> >>>   - So an exact calculation of how much slots we are going to need for
> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
> unfeasible.
> >>>   - A worst case gamble seems impossible either .. if you take multiple
> frags * multiple times the "get_next_rx_buffer" ... you would probably be
> back at just
> >>>     setting the needed_slots to MAX_SKB_FRAGS.
> >>>
> >>> - Other thing would be checking for the available slots before doing the
> "get_next_rx_buffer" .. how ever .. we don't account for how many slots we
> still need to
> >>>   just process the remaining frags.
> >>>
> 
> >> We've done a worst case estimation for whole SKB (linear area + all
> >> frags) already, at the very first beginning. That's what
> >> max_slots_needed is for.
> 
> >>> - Just remove the whole "get_next_rx_buffer".. just tested it but it
> seems the "get_next_rx_buffer" is necessary ..  when i make
> start_new_rx_buffer always return false
> >>>   i hit the bug below :S
> >>>
> >>> So the questions are ... is the "get_next_rx_buffer" part really necessary
> ?
> >>>    - if not, what is the benefit of the "get_next_rx_buffer" and does that
> outweigh the additional cost of accounting
> >>>      of needed_slots for the frags that have yet to be processed ?
> >>>    - if yes, erhmmm what would be the best acceptable solution ..
> returning to using MAX_SKB_FRAGS ?
> >>>
> 
> >> I think you need to answer several questions first:
> >> 1. is max_slots_needed actually large enough to cover whole SKB?
> >         No it's not if, you end up calling "get_next_rx_buffer" one or multiple
> times when processing the SKB
> >         you have the chance of overrunning (or be saved because prod gets
> upped before you overrun).
> 
> >> 2. is the test of ring slot availability acurate?
> >         Seems to be.
> 
> >> 3. is the number of ring slots consumed larger than max_slots_needed? (I
> >>    guess the answer is yes)
> >         Yes that was the whole point.
> 
> >> 4. which step in the break-down procedure causes backend to overrun
> the
> >>    ring?
> >         The "get_next_rx_buffer" call .. that is not accounted for (which can be
> called
> >         multiple times per frag (unless some other none obvious size
> restriction limits this
> >         to one time per frag or one time per SKB).
> >         In my errorneous case it is called one time, but it would be nice if there
> would be some theoretical proof
> >         instead of just some emperical test.
> 
> 
> >> It doesn't matter how many frags your SKB has and how big a frag is. If
> >> every step is correct then you're fine. The code you're looking at
> >> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
> >> carefully examined.
> 
> > Well it seems it only calls "get_next_rx_buffer" in some special cases ..
> (and that also what i'm seeing because it doesn't happen
> > continously.
> 
> > Question is shouldn't this max_needed_slots calc be reverted to what it
> was before 3.14 and take the time in 3.15 to figure out a
> > the theoretical max (if it can be calculated upfront) .. or another way to
> guarantee the code is able to process the whole SKB  ?
> 
> >> Wei.
> 
> 
> 
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Paul,

You have been awfully silent for this whole thread while this is a regression caused by a patch of you
(ca2f09f2b2c6c25047cfc545d057c4edfcfe561c as clearly stated much earlier in this thread).

The commit messages states:
    "net_rx_action() is the place where we could do with an accurate predicition but,
    since that has proven tricky to calculate, a cheap worse-case (but not too bad)
    estimate is all we really need since the only thing we *must* prevent is xenvif_gop_skb()
    consuming more slots than are available."

Your "worst-case" calculation stated in the commit message is clearly not the worst case,
since it doesn't take calls to "get_next_rx_buffer" into account.

Problem is that a worst case calculation would probably be reverting to the old calculation,
and the problems this patch was trying to solve would reappear, but introducing new regressions
isn't very useful either. And since it seems such a tricky and fragile thing to determine, it would
probably be best to be split into a distinct function with a comment to explain the rationale used.

Since this doesn't seem to progress very fast .. CC'ed some more folks .. you never know ..

--
Sander


Tuesday, March 25, 2014, 4:29:42 PM, you wrote:


> Tuesday, March 25, 2014, 4:15:39 PM, you wrote:

>> On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
>> [...]
>>> > Yes there is only one frag .. but it seems to be much larger than PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller bits .. hence the calculation in xenvif_rx_action determining the slots needed by doing:
>>> 
>>> >                 for (i = 0; i < nr_frags; i++) {
>>> >                         unsigned int size;
>>> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
>>> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
>>> >                 }
>>> 
>>> > But the code in xenvif_gop_frag_copy .. seems to be needing one more slot (from the emperical test) .. and calling "get_next_rx_buffer" seems involved in that ..
>>> 
>>> Hmm looked again .. and it seems this is it .. when your frags are large enough you have the chance of running into this.
>>> 

>> get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
>> problem with that implementation?
> In general no, but "get_next_rx_buffer" up's cons .. and the calculations done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
> don't count in this possibility. So it's not the guarding of "start_new_rx_buffer" that is at fault. It's the ones early in "xenvif_rx_action".
> The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a calculated value that should be a "slim fit".

> The problem is in determining upfront in "xenvif_rx_action" when and how often the "get_next_rx_buffer" path will be taken.
> Unless there are other non direct restrictions (from a size point of view) it can be called multiple times per frag per skb.

>>> Problem is .. i don't see an easy fix, the "one more slot" of the empirical test doesn't seem to be the worst case either (i think):
>>> 
>>> - In my case the packets that hit this only have 1 frag, but i could have had more frags.
>>>   I also think you can't rule out the possibility of doing the "get_next_rx_buffer" for multiple subsequent frags from one packet,
>>>   so in the worst (and perhaps even from a single frag since it's looped over a split of it in what seems PAGE_SIZE pieces.)
>>>   - So an exact calculation of how much slots we are going to need for hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems unfeasible.
>>>   - A worst case gamble seems impossible either .. if you take multiple frags * multiple times the "get_next_rx_buffer" ... you would probably be back at just
>>>     setting the needed_slots to MAX_SKB_FRAGS.
>>> 
>>> - Other thing would be checking for the available slots before doing the  "get_next_rx_buffer" .. how ever .. we don't account for how many slots we still need to
>>>   just process the remaining frags.
>>> 

>> We've done a worst case estimation for whole SKB (linear area + all
>> frags) already, at the very first beginning. That's what
>> max_slots_needed is for.

>>> - Just remove the whole "get_next_rx_buffer".. just tested it but it seems the "get_next_rx_buffer" is necessary ..  when i make start_new_rx_buffer always return false
>>>   i hit the bug below :S
>>> 
>>> So the questions are ... is the "get_next_rx_buffer" part really necessary ?
>>>    - if not, what is the benefit of the "get_next_rx_buffer" and does that outweigh the additional cost of accounting
>>>      of needed_slots for the frags that have yet to be processed ?
>>>    - if yes, erhmmm what would be the best acceptable solution .. returning to using MAX_SKB_FRAGS ?
>>> 

>> I think you need to answer several questions first:
>> 1. is max_slots_needed actually large enough to cover whole SKB?
>         No it's not if, you end up calling "get_next_rx_buffer" one or multiple times when processing the SKB
>         you have the chance of overrunning (or be saved because prod gets upped before you overrun).

>> 2. is the test of ring slot availability acurate?
>         Seems to be.

>> 3. is the number of ring slots consumed larger than max_slots_needed? (I
>>    guess the answer is yes)
>         Yes that was the whole point.

>> 4. which step in the break-down procedure causes backend to overrun the
>>    ring?
>         The "get_next_rx_buffer" call .. that is not accounted for (which can be called
>         multiple times per frag (unless some other none obvious size restriction limits this
>         to one time per frag or one time per SKB).
>         In my errorneous case it is called one time, but it would be nice if there would be some theoretical proof
>         instead of just some emperical test.


>> It doesn't matter how many frags your SKB has and how big a frag is. If
>> every step is correct then you're fine. The code you're looking at
>> (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
>> carefully examined.

> Well it seems it only calls "get_next_rx_buffer" in some special cases .. (and that also what i'm seeing because it doesn't happen
> continously.

> Question is shouldn't this max_needed_slots calc be reverted to what it was before 3.14 and take the time in 3.15 to figure out a
> the theoretical max (if it can be calculated upfront) .. or another way to guarantee the code is able to process the whole SKB  ?

>> Wei.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Paul Durrant]

> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Sent: 25 March 2014 15:30
> To: Wei Liu
> Cc: annie li; Paul Durrant; Zoltan Kiss; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network
> troubles "bisected"
> 
> 
> Tuesday, March 25, 2014, 4:15:39 PM, you wrote:
> 
> > On Sat, Mar 22, 2014 at 07:28:34PM +0100, Sander Eikelenboom wrote:
> > [...]
> >> > Yes there is only one frag .. but it seems to be much larger than
> PAGE_SIZE .. and xenvif_gop_frag_copy brakes that frag down into smaller
> bits .. hence the calculation in xenvif_rx_action determining the slots needed
> by doing:
> >>
> >> >                 for (i = 0; i < nr_frags; i++) {
> >> >                         unsigned int size;
> >> >                         size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> >> >                         max_slots_needed += DIV_ROUND_UP(size, PAGE_SIZE);
> >> >                 }
> >>
> >> > But the code in xenvif_gop_frag_copy .. seems to be needing one more
> slot (from the emperical test) .. and calling "get_next_rx_buffer" seems
> involved in that ..
> >>
> >> Hmm looked again .. and it seems this is it .. when your frags are large
> enough you have the chance of running into this.
> >>
> 
> > get_next_rx_buffer is guarded by start_new_rx_buffer. Do you see any
> > problem with that implementation?
> In general no, but "get_next_rx_buffer" up's cons .. and the calculations
> done in "xenvif_rx_action" for max_slots_needed to prevent the overrun
> don't count in this possibility. So it's not the guarding of
> "start_new_rx_buffer" that is at fault. It's the ones early in
> "xenvif_rx_action".
> The ones that were changed by Paul's patch from MAX_SKB_FRAGS to a
> calculated value that should be a "slim fit".
> 
> The problem is in determining upfront in "xenvif_rx_action" when and how
> often the "get_next_rx_buffer" path will be taken.
> Unless there are other non direct restrictions (from a size point of view) it
> can be called multiple times per frag per skb.
> 
> >> Problem is .. i don't see an easy fix, the "one more slot" of the empirical
> test doesn't seem to be the worst case either (i think):
> >>
> >> - In my case the packets that hit this only have 1 frag, but i could have had
> more frags.
> >>   I also think you can't rule out the possibility of doing the
> "get_next_rx_buffer" for multiple subsequent frags from one packet,
> >>   so in the worst (and perhaps even from a single frag since it's looped
> over a split of it in what seems PAGE_SIZE pieces.)
> >>   - So an exact calculation of how much slots we are going to need for
> hitting this "get_next_rx_buffer"  upfront in "xenvif_rx_action" seems
> unfeasible.
> >>   - A worst case gamble seems impossible either .. if you take multiple
> frags * multiple times the "get_next_rx_buffer" ... you would probably be
> back at just
> >>     setting the needed_slots to MAX_SKB_FRAGS.
> >>
> >> - Other thing would be checking for the available slots before doing the
> "get_next_rx_buffer" .. how ever .. we don't account for how many slots we
> still need to
> >>   just process the remaining frags.
> >>
> 
> > We've done a worst case estimation for whole SKB (linear area + all
> > frags) already, at the very first beginning. That's what
> > max_slots_needed is for.
> 
> >> - Just remove the whole "get_next_rx_buffer".. just tested it but it
> seems the "get_next_rx_buffer" is necessary ..  when i make
> start_new_rx_buffer always return false
> >>   i hit the bug below :S
> >>
> >> So the questions are ... is the "get_next_rx_buffer" part really necessary
> ?
> >>    - if not, what is the benefit of the "get_next_rx_buffer" and does that
> outweigh the additional cost of accounting
> >>      of needed_slots for the frags that have yet to be processed ?
> >>    - if yes, erhmmm what would be the best acceptable solution ..
> returning to using MAX_SKB_FRAGS ?
> >>
> 
> > I think you need to answer several questions first:
> > 1. is max_slots_needed actually large enough to cover whole SKB?
>         No it's not if, you end up calling "get_next_rx_buffer" one or multiple
> times when processing the SKB
>         you have the chance of overrunning (or be saved because prod gets
> upped before you overrun).
> 
> > 2. is the test of ring slot availability acurate?
>         Seems to be.
> 
> > 3. is the number of ring slots consumed larger than max_slots_needed? (I
> >    guess the answer is yes)
>         Yes that was the whole point.
> 
> > 4. which step in the break-down procedure causes backend to overrun the
> >    ring?
>         The "get_next_rx_buffer" call .. that is not accounted for (which can be
> called
>         multiple times per frag (unless some other none obvious size restriction
> limits this
>         to one time per frag or one time per SKB).
>         In my errorneous case it is called one time, but it would be nice if there
> would be some theoretical proof
>         instead of just some emperical test.
> 
> 
> > It doesn't matter how many frags your SKB has and how big a frag is. If
> > every step is correct then you're fine. The code you're looking at
> > (start_new_rx_buffer / get_next_rx_buffer and friend) needs to be
> > carefully examined.
> 
> Well it seems it only calls "get_next_rx_buffer" in some special cases .. (and
> that also what i'm seeing because it doesn't happen
> continously.
> 
> Question is shouldn't this max_needed_slots calc be reverted to what it was
> before 3.14 and take the time in 3.15 to figure out a
> the theoretical max (if it can be calculated upfront) .. or another way to
> guarantee the code is able to process the whole SKB  ?
> 

Reverting that patch and falling back to the old slot counting code will simply awaken the bugs that that patch fixed. There is no reason why a worst case skb fragmentation (i.e. assuming no packing of multiple frags into a slot) cannot be calculated up-front.

  Paul

> > Wei.
> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles "bisected"

[Sander Eikelenboom]

Wednesday, March 12, 2014, 3:59:15 PM, you wrote:

> On Wed, Mar 12, 2014 at 03:49:46PM +0100, Sander Eikelenboom wrote:
>> 
>> Wednesday, March 12, 2014, 3:48:26 PM, you wrote:
>> 
>> > On Wed, Mar 12, 2014 at 03:23:36PM +0100, Sander Eikelenboom wrote:
>> > [...]
>> >> > From the look of the code, 0 looks correct to me. Netback won't complain
>> >> > about "bad gref" because it has no idea what a "good gref" looks like.
>> >> > Only Xen has the knowledge whether a gref is legit. All netback sees is
>> >> > the hypercall fails and it tries to push corresponding response to
>> >> > netfront. But you can probably safely assume a gref larger than several
>> >> > thousands be bad.
>> >> 
>> >> OK it's indeed netback setting the status to -1:
>> >> 
>> >> [  976.431585] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> [ 1030.855057] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:2 flags:7 size:2974
>> >> [ 1030.861759] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:1 flags:0 size:1460
>> >> [ 1030.868499] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> [ 1030.875278] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:2974
>> >> [ 1068.199650] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:8 flags:7 size:1634
>> >> [ 1068.206479] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:0 nr_meta_slots:7 flags:4 size:2848
>> >> [ 1068.213158] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:1 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.219701] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:2 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.226194] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:3 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.232728] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:4 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.239163] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:5 nr_meta_slots:7 flags:4 size:4096
>> >> [ 1068.245397] vif vif-7-0 vif7.0: ?!? xenvif_add_frag_responses status err? status:-1 i:6 nr_meta_slots:7 flags:0 size:1168
>> >> [ 1068.251457] vif vif-7-0 vif7.0: ?!? xenvif_rx_action status err? status:-1 meta_slots_used:1 flags:3 size:958
>> >> 
>> >> Now to find out why ..
>> >> 
>> 
>> > -1 is the error code documented in netif.h, XEN_NETIF_RSP_ERROR.
>> 
>> grant copy fails ->> netback sets response status to -1.
>> 
>> > Assuming the traffic volumn is quite high both way (TX and RX), probably
>> > worth checking if RX ring is overflowed. That is, if consumer pointer(s)
>> > advance before producer pointer(s).
>> 
>> In what function ?
>> 

> That should be in RX path. Look for "RING_GET_REQUESTS(&vif->rx". That's
> the macro to get a slot in RX ring; second argument is consumer index.
> You can see if it advances before producer index.

Ah  RING_GET_REQUEST it is ... compiling again ..
> Wei.

>> > Wei.
>> 

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Roger Pau Monné]

On 27/02/14 15:18, Wei Liu wrote:
> On Wed, Feb 26, 2014 at 04:11:23PM +0100, Sander Eikelenboom wrote:
>>
>> Wednesday, February 26, 2014, 10:14:42 AM, you wrote:
>>
>>
>>> Friday, February 21, 2014, 7:32:08 AM, you wrote:
>>
>>
>>>> On 2014/2/20 19:18, Sander Eikelenboom wrote:
>>>>> Thursday, February 20, 2014, 10:49:58 AM, you wrote:
>>>>>
>>>>>
>>>>>> On 2014/2/19 5:25, Sander Eikelenboom wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I'm currently having some network troubles with Xen and recent linux kernels.
>>>>>>>
>>>>>>> - When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
>>>>>>>     I get what seems to be described in this thread: http://www.spinics.net/lists/netdev/msg242953.html
>>>>>>>
>>>>>>>     In the guest:
>>>>>>>     [57539.859584] net eth0: rx->offset: 0, size: 4294967295
>>>>>>>     [57539.859599] net eth0: rx->offset: 0, size: 4294967295
>>>>>>>     [57539.859605] net eth0: rx->offset: 0, size: 4294967295
>>>>>>>     [57539.859610] net eth0: Need more slots
>>>>>>>     [58157.675939] net eth0: Need more slots
>>>>>>>     [58725.344712] net eth0: Need more slots
>>>>>>>     [61815.849180] net eth0: rx->offset: 0, size: 4294967295
>>>>>>>     [61815.849205] net eth0: rx->offset: 0, size: 4294967295
>>>>>>>     [61815.849216] net eth0: rx->offset: 0, size: 4294967295
>>>>>>>     [61815.849225] net eth0: Need more slots
>>>>>> This issue is familiar... and I thought it get fixed.
>>>>>>   From original analysis for similar issue I hit before, the root cause
>>>>>> is netback still creates response when the ring is full. I remember
>>>>>> larger MTU can trigger this issue before, what is the MTU size?
>>>>> In dom0 both for the physical nics and the guest vif's MTU=1500
>>>>> In domU the eth0 also has MTU=1500.
>>>>>
>>>>> So it's not jumbo frames .. just everywhere the same plain defaults ..
>>>>>
>>>>> With the patch from Wei that solves the other issue, i'm still seeing the Need more slots issue on 3.14-rc3+wei's patch now.
>>>>> I have extended the "need more slots warn" to also print the cons, slots, max,  rx->offset, size, hope that gives some more insight.
>>>>> But it indeed is the VM were i had similar issues before, the primary thing this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes of data.
>>>>>
>>>>> This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference " as seen below, don't know if it's a cause or a effect though.
>>
>>>> The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
>>>> Probably the response overlaps the request and grantcopy return error 
>>>> when using wrong grant reference, Netback returns resp->status with 
>>>> ||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend.
>>>> Would it be possible to print log in xenvif_rx_action of netback to see 
>>>> whether something wrong with max slots and used slots?
>>
>>>> Thanks
>>>> Annie
>>
>>> Looking more closely it are perhaps 2 different issues ... the bad grant references do not happen
>>> at the same time as the netfront messages in the guest.
>>
>>> I added some debugpatches to the kernel netback, netfront and xen granttable code (see below)
>>> One of the things was to simplify the code for the debug key to print the granttables, the present code
>>> takes too long to execute and brings down the box due to stalls and NMI's. So it now only prints
>>> the nr of entries per domain.
>>
>>
>>> Issue 1: grant_table.c:1858:d0 Bad grant reference
>>
>>> After running the box for just one night (with 15 VM's) i get these mentions of "Bad grant reference".
>>> The maptrack also seems to increase quite fast and the number of entries seem to have gone up quite fast as well.
>>
>>> Most domains have just one disk(blkfront/blkback) and one nic, a few have a second disk.
>>> The blk drivers use persistent grants so i would assume it would reuse those and not increase it (by much).
>>
> 
> As far as I can tell netfront has a pool of grant references and it
> will BUG_ON() if there's no grefs in the pool when you request one.
> Since your DomU didn't crash so I suspect the book-keeping is still
> intact.
> 
>>> Domain 1 seems to have increased it's nr_grant_entries from 2048 to 3072 somewhere this night.
>>> Domain 7 is the domain that happens to give the netfront messages.
>>
>>> I also don't get why it is reporting the "Bad grant reference" for domain 0, which seems to have 0 active entries ..
>>> Also is this amount of grant entries "normal" ? or could it be a leak somewhere ?
>>
> 
> I suppose Dom0 expanding its maptrack is normal. I see as well when I
> increase the number of domains. But if it keeps increasing while the
> number of DomUs stay the same then it is not normal.

blkfront/blkback will allocate persistent grants on demand, so it's not
strange to see the number of grants increasing while the domain is
running (although it should reach a stable state at some point).

Roger.

Re: Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles

[Wei Liu]

On Thu, Feb 27, 2014 at 04:36:52PM +0100, Roger Pau Monné wrote:
[...]
> >>
> > 
> > I suppose Dom0 expanding its maptrack is normal. I see as well when I
> > increase the number of domains. But if it keeps increasing while the
> > number of DomUs stay the same then it is not normal.
> 
> blkfront/blkback will allocate persistent grants on demand, so it's not
> strange to see the number of grants increasing while the domain is
> running (although it should reach a stable state at some point).
> 

Yes, that's exactly what I meant. :-)

Wei.

> Roger.