compound skb frag pages appearing in start_xmit

compound skb frag pages appearing in start_xmit

[Ian Campbell]

Hi Eric,

Sander has discovered an issue where xen-netback is given a compound
page as one of the skb frag pages to transmit. Currently netback can
only handle PAGE_SIZE'd frags and bugs out.

I suspect this is something to do with 69b08f62e174 "net: use bigger
pages in __netdev_alloc_frag", although perhaps not because it looks
like only tg3 uses it and Sander has an r8169. Also tg3 seems to only
call netdev_alloc_frag for sizes < PAGE_SIZE. I'm probably missing
something.

Are all net drivers expected to be able to handle compound pages in the
frags? Obviously it is to their benefit to do so, so it is something
I'll want to look into for netback.

I expect the main factor here is bridging/forwarding, since the
receiving NIC and its driver appear to support compound pages but the
outgoing NIC (netback in this case) does not.

I guess my question is should I be rushing to fix netback ASAP or should
I rather be looking for a bug somewhere which caused a frag of this type
to get as far as netback's start_xmit in the first place?

Or am I just barking up the wrong tree to start with?

Thanks,
Ian.

Re: compound skb frag pages appearing in start_xmit

[Eric Dumazet]

On Tue, 2012-10-09 at 14:47 +0100, Ian Campbell wrote:
> Hi Eric,
> 

Hi Ian

> Sander has discovered an issue where xen-netback is given a compound
> page as one of the skb frag pages to transmit. Currently netback can
> only handle PAGE_SIZE'd frags and bugs out.
> 
> I suspect this is something to do with 69b08f62e174 "net: use bigger
> pages in __netdev_alloc_frag", although perhaps not because it looks
> like only tg3 uses it and Sander has an r8169. Also tg3 seems to only
> call netdev_alloc_frag for sizes < PAGE_SIZE. I'm probably missing
> something.


Its not the commit you want ;)

> 
> Are all net drivers expected to be able to handle compound pages in the
> frags? Obviously it is to their benefit to do so, so it is something
> I'll want to look into for netback.
> 

Not sure why a net driver would care of COMPOUND page at all ?

a Fragment has a struct page *, and a size.

a page can be order-0, order-1, order-2, order-3, ...

> I expect the main factor here is bridging/forwarding, since the
> receiving NIC and its driver appear to support compound pages but the
> outgoing NIC (netback in this case) does not.
> 
> I guess my question is should I be rushing to fix netback ASAP or should
> I rather be looking for a bug somewhere which caused a frag of this type
> to get as far as netback's start_xmit in the first place?
> 
> Or am I just barking up the wrong tree to start with?



The problem comes because of 

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=5640f7685831e088fe6c2e1f863a6805962f8e81

And yes, we must find a way to cope with this problem in your driver,
because you can also benefit from increase of performance once fixed ;)

And yes I can certainly help, as I am the author of this patch ;)

Thanks

Re: compound skb frag pages appearing in start_xmit

[Eric Dumazet]

On Tue, 2012-10-09 at 15:54 +0200, Eric Dumazet wrote:
> On Tue, 2012-10-09 at 14:47 +0100, Ian Campbell wrote:
> > Hi Eric,
> > 
> 
> Hi Ian
> 
> > Sander has discovered an issue where xen-netback is given a compound
> > page as one of the skb frag pages to transmit. Currently netback can
> > only handle PAGE_SIZE'd frags and bugs out.
> > 
> > I suspect this is something to do with 69b08f62e174 "net: use bigger
> > pages in __netdev_alloc_frag", although perhaps not because it looks
> > like only tg3 uses it and Sander has an r8169. Also tg3 seems to only
> > call netdev_alloc_frag for sizes < PAGE_SIZE. I'm probably missing
> > something.
> 
> 
> Its not the commit you want ;)

Hmm, I take it back. It also can give you the same problem :

We use this allocator for rx path of drivers : 

 __netdev_alloc_skb() 

So its now absolutely possible that one skb->head is backed by a order-3
page.

Is the problem coming from xen_netbk_count_skb_slots() ?

Give me more information if you want me to help.

Re: compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Tue, 2012-10-09 at 14:54 +0100, Eric Dumazet wrote:
> On Tue, 2012-10-09 at 14:47 +0100, Ian Campbell wrote:
> > Hi Eric,
> > 
> 
> Hi Ian
> 
> > Sander has discovered an issue where xen-netback is given a compound
> > page as one of the skb frag pages to transmit. Currently netback can
> > only handle PAGE_SIZE'd frags and bugs out.
> > 
> > I suspect this is something to do with 69b08f62e174 "net: use bigger
> > pages in __netdev_alloc_frag", although perhaps not because it looks
> > like only tg3 uses it and Sander has an r8169. Also tg3 seems to only
> > call netdev_alloc_frag for sizes < PAGE_SIZE. I'm probably missing
> > something.
> 
> 
> Its not the commit you want ;)
> 
> > 
> > Are all net drivers expected to be able to handle compound pages in the
> > frags? Obviously it is to their benefit to do so, so it is something
> > I'll want to look into for netback.
> > 
> 
> Not sure why a net driver would care of COMPOUND page at all ?
> 
> a Fragment has a struct page *, and a size.
> 
> a page can be order-0, order-1, order-2, order-3, ...

I keep falling into this trap that a struct page * can be order > 0.

The Xen PV interfaces deal in order-0 pages only. Also things which are
contiguous in physical space may not be contiguous in DMA space (which
we call "machine memory" in Xen terminology).

The first is probably a specific quirk of Xen, but I thought there were
other architectures where physical and DMA space we not necessarily
contiguous and which would therefore need special handling (I guess
those platforms all have IOMMUs)

> > I expect the main factor here is bridging/forwarding, since the
> > receiving NIC and its driver appear to support compound pages but the
> > outgoing NIC (netback in this case) does not.
> > 
> > I guess my question is should I be rushing to fix netback ASAP or should
> > I rather be looking for a bug somewhere which caused a frag of this type
> > to get as far as netback's start_xmit in the first place?
> > 
> > Or am I just barking up the wrong tree to start with?
> 
> 
> 
> The problem comes because of 
> 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=5640f7685831e088fe6c2e1f863a6805962f8e81
> 
> And yes, we must find a way to cope with this problem in your driver,
> because you can also benefit from increase of performance once fixed ;)
> 
> And yes I can certainly help, as I am the author of this patch ;)

I think I can mostly deal with this in the same way netback deals with
large skb heads i.e. by busting the multipage page into individual 4096
page chunks.

Does the higher order pages effectively reduce the number of frags which
are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
have 64K worth of frag data.

If we switch to order-3 pages everywhere then can the skb contain 512K
of data, or does the effective maximum number of frags in an skb reduce
to 2?

If it's the latter then I think fixing netback is simple, if it's the
former then I might need to think a bit harder.

Ian.

Re: compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Tue, 2012-10-09 at 15:01 +0100, Eric Dumazet wrote:
> On Tue, 2012-10-09 at 15:54 +0200, Eric Dumazet wrote:
> > On Tue, 2012-10-09 at 14:47 +0100, Ian Campbell wrote:
> > > Hi Eric,
> > > 
> > 
> > Hi Ian
> > 
> > > Sander has discovered an issue where xen-netback is given a compound
> > > page as one of the skb frag pages to transmit. Currently netback can
> > > only handle PAGE_SIZE'd frags and bugs out.
> > > 
> > > I suspect this is something to do with 69b08f62e174 "net: use bigger
> > > pages in __netdev_alloc_frag", although perhaps not because it looks
> > > like only tg3 uses it and Sander has an r8169. Also tg3 seems to only
> > > call netdev_alloc_frag for sizes < PAGE_SIZE. I'm probably missing
> > > something.
> > 
> > 
> > Its not the commit you want ;)
> 
> Hmm, I take it back. It also can give you the same problem :
> 
> We use this allocator for rx path of drivers : 
> 
>  __netdev_alloc_skb() 
> 
> So its now absolutely possible that one skb->head is backed by a order-3
> page.
> 
> Is the problem coming from xen_netbk_count_skb_slots() ?
> 
> Give me more information if you want me to help.

The interesting code is in netbk_gop_skb(), specifically the two calls
to netbk_gop_frag_copy.

netbk_gop_frag_copy can only copy order-0 pages to the peer since they
go over a shared ring transport which can only deal in order-0 pages.

For the SKB head there is a loop which handles order>0 heads, I suspect
we just need something similar for the frag case.

Although see my question in the other response about the maximum number
of frags we can have when order is > 0 since if using larger pages
causes us to end up with a much larger number of order-0 pages once
we've broken them up then we have a problem and I need to put my
thinking cap on a bit (perhaps substantially) tighter.

Konrad, it looks like netfront has a similar issue in
xennet_make_frags() since it doesn't shatter large order mappings
either.

Ian.

Re: compound skb frag pages appearing in start_xmit

[Eric Dumazet]

On Tue, 2012-10-09 at 15:17 +0100, Ian Campbell wrote:

> Does the higher order pages effectively reduce the number of frags which
> are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
> have 64K worth of frag data.
> 
> If we switch to order-3 pages everywhere then can the skb contain 512K
> of data, or does the effective maximum number of frags in an skb reduce
> to 2?

effective number of frags reduce to 2 or 3

(We still limit GSO packets to ~63536 bytes)

Re: compound skb frag pages appearing in start_xmit

[Eric Dumazet]

On Tue, 2012-10-09 at 15:23 +0100, Ian Campbell wrote:
> On Tue, 2012-10-09 at 15:01 +0100, Eric Dumazet wrote:
> > On Tue, 2012-10-09 at 15:54 +0200, Eric Dumazet wrote:
> > > On Tue, 2012-10-09 at 14:47 +0100, Ian Campbell wrote:
> > > > Hi Eric,
> > > > 
> > > 
> > > Hi Ian
> > > 
> > > > Sander has discovered an issue where xen-netback is given a compound
> > > > page as one of the skb frag pages to transmit. Currently netback can
> > > > only handle PAGE_SIZE'd frags and bugs out.
> > > > 
> > > > I suspect this is something to do with 69b08f62e174 "net: use bigger
> > > > pages in __netdev_alloc_frag", although perhaps not because it looks
> > > > like only tg3 uses it and Sander has an r8169. Also tg3 seems to only
> > > > call netdev_alloc_frag for sizes < PAGE_SIZE. I'm probably missing
> > > > something.
> > > 
> > > 
> > > Its not the commit you want ;)
> > 
> > Hmm, I take it back. It also can give you the same problem :
> > 
> > We use this allocator for rx path of drivers : 
> > 
> >  __netdev_alloc_skb() 
> > 
> > So its now absolutely possible that one skb->head is backed by a order-3
> > page.
> > 
> > Is the problem coming from xen_netbk_count_skb_slots() ?
> > 
> > Give me more information if you want me to help.
> 
> The interesting code is in netbk_gop_skb(), specifically the two calls
> to netbk_gop_frag_copy.
> 
> netbk_gop_frag_copy can only copy order-0 pages to the peer since they
> go over a shared ring transport which can only deal in order-0 pages.
> 
> For the SKB head there is a loop which handles order>0 heads, I suspect
> we just need something similar for the frag case.
> 
> Although see my question in the other response about the maximum number
> of frags we can have when order is > 0 since if using larger pages
> causes us to end up with a much larger number of order-0 pages once
> we've broken them up then we have a problem and I need to put my
> thinking cap on a bit (perhaps substantially) tighter.
> 
> Konrad, it looks like netfront has a similar issue in
> xennet_make_frags() since it doesn't shatter large order mappings
> either.

Hmm...

In theory, if a skb has 16+1 frags backed by compound pages, you could
need ~48 order-0 frags.

(4098 bytes could need 1-4096-1 (3 frags))

In practice, it should be around ~17 order-0 frags as before.

Re: compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Tue, 2012-10-09 at 15:27 +0100, Eric Dumazet wrote:
> On Tue, 2012-10-09 at 15:17 +0100, Ian Campbell wrote:
> 
> > Does the higher order pages effectively reduce the number of frags which
> > are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
> > have 64K worth of frag data.
> > 
> > If we switch to order-3 pages everywhere then can the skb contain 512K
> > of data, or does the effective maximum number of frags in an skb reduce
> > to 2?
> 
> effective number of frags reduce to 2 or 3
> 
> (We still limit GSO packets to ~63536 bytes)

Great! Then I think the fix is more/less trivial...

As an aside, when the skb head is < 4096 bytes is that necessarily a
compound page or might it just be a large kmalloc area?

Only really relevant since it impacts the possibility for code sharing
between the head and the frags sending.

Ian

Re: compound skb frag pages appearing in start_xmit

[Eric Dumazet]

On Tue, 2012-10-09 at 15:40 +0100, Ian Campbell wrote:
> On Tue, 2012-10-09 at 15:27 +0100, Eric Dumazet wrote:
> > On Tue, 2012-10-09 at 15:17 +0100, Ian Campbell wrote:
> > 
> > > Does the higher order pages effectively reduce the number of frags which
> > > are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
> > > have 64K worth of frag data.
> > > 
> > > If we switch to order-3 pages everywhere then can the skb contain 512K
> > > of data, or does the effective maximum number of frags in an skb reduce
> > > to 2?
> > 
> > effective number of frags reduce to 2 or 3
> > 
> > (We still limit GSO packets to ~63536 bytes)
> 
> Great! Then I think the fix is more/less trivial...
> 
> As an aside, when the skb head is < 4096 bytes is that necessarily a
> compound page or might it just be a large kmalloc area?
> 

skb->head can be either allocated by kmalloc() (standard alloc_skb()) or
a page frag (if allocated in rx path)

Not sure its related to headlen/size...

Re: compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Tue, 2012-10-09 at 15:33 +0100, Eric Dumazet wrote:
> On Tue, 2012-10-09 at 15:23 +0100, Ian Campbell wrote:
> > On Tue, 2012-10-09 at 15:01 +0100, Eric Dumazet wrote:
> > > On Tue, 2012-10-09 at 15:54 +0200, Eric Dumazet wrote:
> > > > On Tue, 2012-10-09 at 14:47 +0100, Ian Campbell wrote:
> > > > > Hi Eric,
> > > > > 
> > > > 
> > > > Hi Ian
> > > > 
> > > > > Sander has discovered an issue where xen-netback is given a compound
> > > > > page as one of the skb frag pages to transmit. Currently netback can
> > > > > only handle PAGE_SIZE'd frags and bugs out.
> > > > > 
> > > > > I suspect this is something to do with 69b08f62e174 "net: use bigger
> > > > > pages in __netdev_alloc_frag", although perhaps not because it looks
> > > > > like only tg3 uses it and Sander has an r8169. Also tg3 seems to only
> > > > > call netdev_alloc_frag for sizes < PAGE_SIZE. I'm probably missing
> > > > > something.
> > > > 
> > > > 
> > > > Its not the commit you want ;)
> > > 
> > > Hmm, I take it back. It also can give you the same problem :
> > > 
> > > We use this allocator for rx path of drivers : 
> > > 
> > >  __netdev_alloc_skb() 
> > > 
> > > So its now absolutely possible that one skb->head is backed by a order-3
> > > page.
> > > 
> > > Is the problem coming from xen_netbk_count_skb_slots() ?
> > > 
> > > Give me more information if you want me to help.
> > 
> > The interesting code is in netbk_gop_skb(), specifically the two calls
> > to netbk_gop_frag_copy.
> > 
> > netbk_gop_frag_copy can only copy order-0 pages to the peer since they
> > go over a shared ring transport which can only deal in order-0 pages.
> > 
> > For the SKB head there is a loop which handles order>0 heads, I suspect
> > we just need something similar for the frag case.
> > 
> > Although see my question in the other response about the maximum number
> > of frags we can have when order is > 0 since if using larger pages
> > causes us to end up with a much larger number of order-0 pages once
> > we've broken them up then we have a problem and I need to put my
> > thinking cap on a bit (perhaps substantially) tighter.
> > 
> > Konrad, it looks like netfront has a similar issue in
> > xennet_make_frags() since it doesn't shatter large order mappings
> > either.
> 
> Hmm...
> 
> In theory, if a skb has 16+1 frags backed by compound pages, you could
> need ~48 order-0 frags.
> 
> (4098 bytes could need 1-4096-1 (3 frags))
> 
> In practice, it should be around ~17 order-0 frags as before.

Right, thanks. I think I can cope with that without needing to change
the PV protocol in any way.

Ian.

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Tue, 2012-10-09 at 15:40 +0100, Ian Campbell wrote:
> On Tue, 2012-10-09 at 15:27 +0100, Eric Dumazet wrote:
> > On Tue, 2012-10-09 at 15:17 +0100, Ian Campbell wrote:
> > 
> > > Does the higher order pages effectively reduce the number of frags which
> > > are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
> > > have 64K worth of frag data.
> > > 
> > > If we switch to order-3 pages everywhere then can the skb contain 512K
> > > of data, or does the effective maximum number of frags in an skb reduce
> > > to 2?
> > 
> > effective number of frags reduce to 2 or 3
> > 
> > (We still limit GSO packets to ~63536 bytes)
> 
> Great! Then I think the fix is more/less trivial...

The following seems to work for me.

I haven't tackled netfront yet.

8<--------------------------------------------------------------

>From 551e42e3dd203f2eb97cb082985013bb33b8f020 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Tue, 9 Oct 2012 15:51:20 +0100
Subject: [PATCH] xen: netback: handle compound page fragments on transmit.

An SKB paged fragment can consist of a compound page with order > 0.
However the netchannel protocol deals only in PAGE_SIZE frames.

Handle this in netbk_gop_frag_copy and xen_netbk_count_skb_slots by
iterating over the frames which make up the page.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
---
 drivers/net/xen-netback/netback.c |   40 ++++++++++++++++++++++++++++++++----
 1 files changed, 35 insertions(+), 5 deletions(-)

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 4ebfcf3..d747e30 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -335,21 +335,35 @@ unsigned int xen_netbk_count_skb_slots(struct xenvif *vif, struct sk_buff *skb)
 
 	for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
 		unsigned long size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
+		unsigned long offset = skb_shinfo(skb)->frags[i].page_offset;
 		unsigned long bytes;
+
+		offset &= ~PAGE_MASK;
+
 		while (size > 0) {
+			BUG_ON(offset >= PAGE_SIZE);
 			BUG_ON(copy_off > MAX_BUFFER_OFFSET);
 
-			if (start_new_rx_buffer(copy_off, size, 0)) {
+			bytes = PAGE_SIZE - offset;
+
+			if (bytes > size)
+				bytes = size;
+
+			if (start_new_rx_buffer(copy_off, bytes, 0)) {
 				count++;
 				copy_off = 0;
 			}
 
-			bytes = size;
 			if (copy_off + bytes > MAX_BUFFER_OFFSET)
 				bytes = MAX_BUFFER_OFFSET - copy_off;
 
 			copy_off += bytes;
+
+			offset += bytes;
 			size -= bytes;
+
+			if (offset == PAGE_SIZE)
+				offset = 0;
 		}
 	}
 	return count;
@@ -403,14 +417,24 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
 	unsigned long bytes;
 
 	/* Data must not cross a page boundary. */
-	BUG_ON(size + offset > PAGE_SIZE);
+	BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
 
 	meta = npo->meta + npo->meta_prod - 1;
 
+	/* Skip unused frames from start of page */
+	page += offset >> PAGE_SHIFT;
+	offset &= ~PAGE_MASK;
+
 	while (size > 0) {
+		BUG_ON(offset >= PAGE_SIZE);
 		BUG_ON(npo->copy_off > MAX_BUFFER_OFFSET);
 
-		if (start_new_rx_buffer(npo->copy_off, size, *head)) {
+		bytes = PAGE_SIZE - offset;
+
+		if (bytes > size)
+			bytes = size;
+
+		if (start_new_rx_buffer(npo->copy_off, bytes, *head)) {
 			/*
 			 * Netfront requires there to be some data in the head
 			 * buffer.
@@ -420,7 +444,6 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
 			meta = get_next_rx_buffer(vif, npo);
 		}
 
-		bytes = size;
 		if (npo->copy_off + bytes > MAX_BUFFER_OFFSET)
 			bytes = MAX_BUFFER_OFFSET - npo->copy_off;
 
@@ -453,6 +476,13 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
 		offset += bytes;
 		size -= bytes;
 
+		/* Next frame */
+		if (offset == PAGE_SIZE) {
+			BUG_ON(!PageCompound(page));
+			page++;
+			offset = 0;
+		}
+
 		/* Leave a gap for the GSO descriptor. */
 		if (*head && skb_shinfo(skb)->gso_size && !vif->gso_prefix)
 			vif->rx.req_cons++;
-- 
1.7.2.5

Re: compound skb frag pages appearing in start_xmit

[Sander Eikelenboom]

Wednesday, October 10, 2012, 12:13:04 PM, you wrote:

> On Tue, 2012-10-09 at 15:40 +0100, Ian Campbell wrote:
>> On Tue, 2012-10-09 at 15:27 +0100, Eric Dumazet wrote:
>> > On Tue, 2012-10-09 at 15:17 +0100, Ian Campbell wrote:
>> > 
>> > > Does the higher order pages effectively reduce the number of frags which
>> > > are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
>> > > have 64K worth of frag data.
>> > > 
>> > > If we switch to order-3 pages everywhere then can the skb contain 512K
>> > > of data, or does the effective maximum number of frags in an skb reduce
>> > > to 2?
>> > 
>> > effective number of frags reduce to 2 or 3
>> > 
>> > (We still limit GSO packets to ~63536 bytes)
>> 
>> Great! Then I think the fix is more/less trivial...

> The following seems to work for me.

But it doesn't seem to work for me ... dmesg attached.

I don't know if the "mcelog:4359 map pfn expected mapping type write-back for [mem 0x0009f000-0x000a0fff], got uncached-minus"
is related, is shows up right after the nics get initialized ?

netback still fails with:

[  191.777994] ------------[ cut here ]------------
[  191.784245] kernel BUG at drivers/net/xen-netback/netback.c:481!
[  191.790423] invalid opcode: 0000 [#1] PREEMPT SMP 
[  191.796462] Modules linked in:
[  191.802315] CPU 1 
[  191.802367] Pid: 1177, comm: netback/1 Tainted: G        W    3.6.0pre-rc1-20121010 #1 MSI MS-7640/890FXA-GD70 (MS-7640)  
[  191.814043] RIP: e030:[<ffffffff8146de61>]  [<ffffffff8146de61>] netbk_gop_frag_copy+0x3f1/0x400
[  191.820171] RSP: e02b:ffff880037c6bb98  EFLAGS: 00010246
[  191.826271] RAX: 0000000000000244 RBX: ffffc90010827f98 RCX: ffff880031ed9880
[  191.832450] RDX: 00000000000000a8 RSI: ffff880037c6bd24 RDI: ffffea0000b03f80
[  191.838581] RBP: ffff880037c6bc28 R08: ffff8800319f8100 R09: 0000000000001000
[  191.844739] R10: 0000000000000000 R11: 0000000000000132 R12: 00000000000000a8
[  191.850785] R13: ffff880037c6bcd8 R14: 0000000000001000 R15: ffffc9001082cf70
[  191.856741] FS:  00007f9f3c944700(0000) GS:ffff88003f840000(0000) knlGS:0000000000000000
[  191.862841] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  191.868901] CR2: 0000000001337ca0 CR3: 0000000032cec000 CR4: 0000000000000660
[  191.875053] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  191.881175] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  191.887247] Process netback/1 (pid: 1177, threadinfo ffff880037c6a000, task ffff880039984140)
[  191.893325] Stack:
[  191.899328]  ffff880037c6bd24 00000000000000a8 ffff8800319f8100 ffff880031ed9880
[  191.905534]  ffffc90000000000 0000000000001000 0000000000000000 0000000000000000
[  191.911742]  ffff880000000000 ffffffff817459f3 ffffc90010823420 ffffea0000b03f80
[  191.917898] Call Trace:
[  191.923939]  [<ffffffff817459f3>] ? _raw_spin_unlock_irqrestore+0x53/0xa0
[  191.930141]  [<ffffffff8146e1cb>] xen_netbk_rx_action+0x30b/0x830
[  191.936543]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  191.942942]  [<ffffffff8146f6da>] xen_netbk_kthread+0xba/0xa90
[  191.949147]  [<ffffffff81095b06>] ? try_to_wake_up+0x1b6/0x310
[  191.955250]  [<ffffffff81086b40>] ? wake_up_bit+0x40/0x40
[  191.961421]  [<ffffffff8146f620>] ? xen_netbk_tx_build_gops+0xa70/0xa70
[  191.967660]  [<ffffffff810864d6>] kthread+0xd6/0xe0
[  191.973834]  [<ffffffff81086400>] ? __init_kthread_worker+0x70/0x70
[  191.979953]  [<ffffffff8174677c>] ret_from_fork+0x7c/0x90
[  191.986107]  [<ffffffff81086400>] ? __init_kthread_worker+0x70/0x70
[  191.992174] Code: b8 b3 00 00 48 8d 8c f1 60 01 00 00 48 3b 14 01 0f 85 72 fc ff ff e9 7a fc ff ff 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe <0f> 0b eb fe 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 
[  192.005230] RIP  [<ffffffff8146de61>] netbk_gop_frag_copy+0x3f1/0x400
[  192.011786]  RSP <ffff880037c6bb98>
[  192.018402] ---[ end trace c51ab5e2c2c918fc ]---


--

Sander

> I haven't tackled netfront yet.

> 8<--------------------------------------------------------------

> From 551e42e3dd203f2eb97cb082985013bb33b8f020 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Tue, 9 Oct 2012 15:51:20 +0100
> Subject: [PATCH] xen: netback: handle compound page fragments on transmit.

> An SKB paged fragment can consist of a compound page with order > 0.
> However the netchannel protocol deals only in PAGE_SIZE frames.

> Handle this in netbk_gop_frag_copy and xen_netbk_count_skb_slots by
> iterating over the frames which make up the page.

> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: Eric Dumazet <eric.dumazet@gmail.com>
> Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>
> Cc: Sander Eikelenboom <linux@eikelenboom.it>
> ---
>  drivers/net/xen-netback/netback.c |   40 ++++++++++++++++++++++++++++++++----
>  1 files changed, 35 insertions(+), 5 deletions(-)

> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> index 4ebfcf3..d747e30 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -335,21 +335,35 @@ unsigned int xen_netbk_count_skb_slots(struct xenvif *vif, struct sk_buff *skb)
>  
>         for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
>                 unsigned long size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
> +               unsigned long offset = skb_shinfo(skb)->frags[i].page_offset;
>                 unsigned long bytes;
> +
> +               offset &= ~PAGE_MASK;
> +
>                 while (size > 0) {
> +                       BUG_ON(offset >= PAGE_SIZE);
>                         BUG_ON(copy_off > MAX_BUFFER_OFFSET);
>  
> -                       if (start_new_rx_buffer(copy_off, size, 0)) {
> +                       bytes = PAGE_SIZE - offset;
> +
> +                       if (bytes > size)
> +                               bytes = size;
> +
> +                       if (start_new_rx_buffer(copy_off, bytes, 0)) {
>                                 count++;
>                                 copy_off = 0;
>                         }
>  
> -                       bytes = size;
>                         if (copy_off + bytes > MAX_BUFFER_OFFSET)
>                                 bytes = MAX_BUFFER_OFFSET - copy_off;
>  
>                         copy_off += bytes;
> +
> +                       offset += bytes;
>                         size -= bytes;
> +
> +                       if (offset == PAGE_SIZE)
> +                               offset = 0;
>                 }
>         }
>         return count;
> @@ -403,14 +417,24 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
>         unsigned long bytes;
>  
>         /* Data must not cross a page boundary. */
> -       BUG_ON(size + offset > PAGE_SIZE);
> +       BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
>  
>         meta = npo->meta + npo->meta_prod - 1;
>  
> +       /* Skip unused frames from start of page */
> +       page += offset >> PAGE_SHIFT;
> +       offset &= ~PAGE_MASK;
> +
>         while (size > 0) {
> +               BUG_ON(offset >= PAGE_SIZE);
>                 BUG_ON(npo->copy_off > MAX_BUFFER_OFFSET);
>  
> -               if (start_new_rx_buffer(npo->copy_off, size, *head)) {
> +               bytes = PAGE_SIZE - offset;
> +
> +               if (bytes > size)
> +                       bytes = size;
> +
> +               if (start_new_rx_buffer(npo->copy_off, bytes, *head)) {
>                         /*
>                          * Netfront requires there to be some data in the head
>                          * buffer.
> @@ -420,7 +444,6 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
>                         meta = get_next_rx_buffer(vif, npo);
>                 }
>  
> -               bytes = size;
>                 if (npo->copy_off + bytes > MAX_BUFFER_OFFSET)
>                         bytes = MAX_BUFFER_OFFSET - npo->copy_off;
>  
> @@ -453,6 +476,13 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
>                 offset += bytes;
>                 size -= bytes;
>  
> +               /* Next frame */
> +               if (offset == PAGE_SIZE) {
> +                       BUG_ON(!PageCompound(page));
> +                       page++;
> +                       offset = 0;
> +               }
> +
>                 /* Leave a gap for the GSO descriptor. */
>                 if (*head && skb_shinfo(skb)->gso_size && !vif->gso_prefix)
>                         vif->rx.req_cons++;

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Wed, 2012-10-10 at 13:24 +0100, Sander Eikelenboom wrote:
> Wednesday, October 10, 2012, 12:13:04 PM, you wrote:
> 
> > On Tue, 2012-10-09 at 15:40 +0100, Ian Campbell wrote:
> >> On Tue, 2012-10-09 at 15:27 +0100, Eric Dumazet wrote:
> >> > On Tue, 2012-10-09 at 15:17 +0100, Ian Campbell wrote:
> >> > 
> >> > > Does the higher order pages effectively reduce the number of frags which
> >> > > are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
> >> > > have 64K worth of frag data.
> >> > > 
> >> > > If we switch to order-3 pages everywhere then can the skb contain 512K
> >> > > of data, or does the effective maximum number of frags in an skb reduce
> >> > > to 2?
> >> > 
> >> > effective number of frags reduce to 2 or 3
> >> > 
> >> > (We still limit GSO packets to ~63536 bytes)
> >> 
> >> Great! Then I think the fix is more/less trivial...
> 
> > The following seems to work for me.
> 
> But it doesn't seem to work for me ... dmesg attached.

> [  191.777994] ------------[ cut here ]------------
> [  191.784245] kernel BUG at drivers/net/xen-netback/netback.c:481!

Looks like that BUG_ON is a little aggressive. It'll trigger if the data
happens to end on a frame boundary. Hopefully this will fix it for you:

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index d747e30..f2d6b78 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -477,7 +477,7 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
 		size -= bytes;
 
 		/* Next frame */
-		if (offset == PAGE_SIZE) {
+		if (offset == PAGE_SIZE && size) {
 			BUG_ON(!PageCompound(page));
 			page++;
 			offset = 0;

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Wed, 2012-10-10 at 11:13 +0100, Ian Campbell wrote:
> I haven't tackled netfront yet. 

I seem to be totally unable to reproduce the equivalent issue on the
netfront xmit side, even though it seems like the loop in
xennet_make_frags ought to be obviously susceptible to it.

Konrad, Sander, are either of you able to repro, e.g. with:

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index b06ef81..8a3f770 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -462,6 +462,8 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
 		ref = gnttab_claim_grant_reference(&np->gref_tx_head);
 		BUG_ON((signed short)ref < 0);
 
+		BUG_ON(PageCompound(skb_frag_page(frag)));
+
 		mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
 		gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
 						mfn, GNTMAP_readonly);

My repro for netback was just to netcat a wodge of data from dom0->domU
but going the other way doesn't seem to trigger.

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Sander Eikelenboom]

Wednesday, October 10, 2012, 2:29:09 PM, you wrote:

> On Wed, 2012-10-10 at 13:24 +0100, Sander Eikelenboom wrote:
>> Wednesday, October 10, 2012, 12:13:04 PM, you wrote:
>> 
>> > On Tue, 2012-10-09 at 15:40 +0100, Ian Campbell wrote:
>> >> On Tue, 2012-10-09 at 15:27 +0100, Eric Dumazet wrote:
>> >> > On Tue, 2012-10-09 at 15:17 +0100, Ian Campbell wrote:
>> >> > 
>> >> > > Does the higher order pages effectively reduce the number of frags which
>> >> > > are in use? e.g if MAX_SKB_FRAGS is 16, then for order-0 pages you could
>> >> > > have 64K worth of frag data.
>> >> > > 
>> >> > > If we switch to order-3 pages everywhere then can the skb contain 512K
>> >> > > of data, or does the effective maximum number of frags in an skb reduce
>> >> > > to 2?
>> >> > 
>> >> > effective number of frags reduce to 2 or 3
>> >> > 
>> >> > (We still limit GSO packets to ~63536 bytes)
>> >> 
>> >> Great! Then I think the fix is more/less trivial...
>> 
>> > The following seems to work for me.
>> 
>> But it doesn't seem to work for me ... dmesg attached.

>> [  191.777994] ------------[ cut here ]------------
>> [  191.784245] kernel BUG at drivers/net/xen-netback/netback.c:481!

> Looks like that BUG_ON is a little aggressive. It'll trigger if the data
> happens to end on a frame boundary. Hopefully this will fix it for you:

Yes it does !
Thanks .. will recompile and test the netfront case as well

--
Sander

> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> index d747e30..f2d6b78 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -477,7 +477,7 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
>                 size -= bytes;
>  
>                 /* Next frame */
> -               if (offset == PAGE_SIZE) {
> +               if (offset == PAGE_SIZE && size) {
>                         BUG_ON(!PageCompound(page));
>                         page++;
>                         offset = 0;

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Sander Eikelenboom]

Wednesday, October 10, 2012, 3:09:58 PM, you wrote:

> On Wed, 2012-10-10 at 11:13 +0100, Ian Campbell wrote:
>> I haven't tackled netfront yet. 

> I seem to be totally unable to reproduce the equivalent issue on the
> netfront xmit side, even though it seems like the loop in
> xennet_make_frags ought to be obviously susceptible to it.

> Konrad, Sander, are either of you able to repro, e.g. with:


Hmrrrmm i don't see any traces, only strange behaviour ..

- i can connect to guests by ssh, but it's sluggish, and sometimes stops working
- The guest seem to keep trying to connect to netback:

[  658.276719] xen_bridge: port 2(vif40.0) entered forwarding state
[  658.282258] xen_bridge: port 2(vif40.0) entered forwarding state
[  663.945964] xen_bridge: port 7(vif39.0) entered forwarding state
[  669.674277] xen_bridge: port 2(vif40.0) entered disabled state
[  669.680290] device vif40.0 left promiscuous mode
[  669.685464] xen_bridge: port 2(vif40.0) entered disabled state
[  672.857222] device vif41.0 entered promiscuous mode
[  673.166254] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
[  673.176368] xen_bridge: port 2(vif41.0) entered forwarding state
[  673.182042] xen_bridge: port 2(vif41.0) entered forwarding state
[  674.439725] xen_bridge: port 7(vif39.0) entered disabled state
[  674.445708] device vif39.0 left promiscuous mode
[  674.450955] xen_bridge: port 7(vif39.0) entered disabled state
[  677.726040] device vif42.0 entered promiscuous mode
[  678.053381] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
[  678.062804] xen_bridge: port 7(vif42.0) entered forwarding state
[  678.068433] xen_bridge: port 7(vif42.0) entered forwarding state
[  688.224736] xen_bridge: port 2(vif41.0) entered forwarding state
[  693.080557] xen_bridge: port 7(vif42.0) entered forwarding state
[  700.786276] xen_bridge: port 7(vif42.0) entered disabled state
[  700.792484] device vif42.0 left promiscuous mode
[  700.802409] xen_bridge: port 7(vif42.0) entered disabled state
[  704.133606] device vif43.0 entered promiscuous mode
[  704.460160] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
[  704.469800] xen_bridge: port 7(vif43.0) entered forwarding state
[  704.475303] xen_bridge: port 7(vif43.0) entered forwarding state
[  719.493788] xen_bridge: port 7(vif43.0) entered forwarding state
[  726.302456] xen_bridge: port 7(vif43.0) entered disabled state
[  726.308898] device vif43.0 left promiscuous mode
[  726.314029] xen_bridge: port 7(vif43.0) entered disabled state

All the guests are already up, but this keeps on going and going and going ....



> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index b06ef81..8a3f770 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -462,6 +462,8 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
>                 ref = gnttab_claim_grant_reference(&np->gref_tx_head);
>                 BUG_ON((signed short)ref < 0);
>  
> +               BUG_ON(PageCompound(skb_frag_page(frag)));
> +
>                 mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
>                 gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
>                                                 mfn, GNTMAP_readonly);

> My repro for netback was just to netcat a wodge of data from dom0->domU
> but going the other way doesn't seem to trigger.

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Wed, 2012-10-10 at 15:49 +0100, Sander Eikelenboom wrote:
> Wednesday, October 10, 2012, 3:09:58 PM, you wrote:
> 
> > On Wed, 2012-10-10 at 11:13 +0100, Ian Campbell wrote:
> >> I haven't tackled netfront yet. 
> 
> > I seem to be totally unable to reproduce the equivalent issue on the
> > netfront xmit side, even though it seems like the loop in
> > xennet_make_frags ought to be obviously susceptible to it.
> 
> > Konrad, Sander, are either of you able to repro, e.g. with:
> 
> 
> Hmrrrmm i don't see any traces, only strange behaviour ..
> 
> - i can connect to guests by ssh, but it's sluggish, and sometimes stops working

I saw something like this (ssh sluggish) even with dom0 itself. I'm
trying to see if I can characterise it enough to reliably bisect it.

I already switched out xen-unstable for 4.2-testing but that didn't make
any difference.

> - The guest seem to keep trying to connect to netback:
> 
> [  658.276719] xen_bridge: port 2(vif40.0) entered forwarding state
> [  658.282258] xen_bridge: port 2(vif40.0) entered forwarding state
> [  663.945964] xen_bridge: port 7(vif39.0) entered forwarding state
> [  669.674277] xen_bridge: port 2(vif40.0) entered disabled state
> [  669.680290] device vif40.0 left promiscuous mode
> [  669.685464] xen_bridge: port 2(vif40.0) entered disabled state
> [  672.857222] device vif41.0 entered promiscuous mode
> [  673.166254] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> [  673.176368] xen_bridge: port 2(vif41.0) entered forwarding state
> [  673.182042] xen_bridge: port 2(vif41.0) entered forwarding state
> [  674.439725] xen_bridge: port 7(vif39.0) entered disabled state
> [  674.445708] device vif39.0 left promiscuous mode
> [  674.450955] xen_bridge: port 7(vif39.0) entered disabled state
> [  677.726040] device vif42.0 entered promiscuous mode
> [  678.053381] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> [  678.062804] xen_bridge: port 7(vif42.0) entered forwarding state
> [  678.068433] xen_bridge: port 7(vif42.0) entered forwarding state
> [  688.224736] xen_bridge: port 2(vif41.0) entered forwarding state
> [  693.080557] xen_bridge: port 7(vif42.0) entered forwarding state
> [  700.786276] xen_bridge: port 7(vif42.0) entered disabled state
> [  700.792484] device vif42.0 left promiscuous mode
> [  700.802409] xen_bridge: port 7(vif42.0) entered disabled state
> [  704.133606] device vif43.0 entered promiscuous mode
> [  704.460160] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> [  704.469800] xen_bridge: port 7(vif43.0) entered forwarding state
> [  704.475303] xen_bridge: port 7(vif43.0) entered forwarding state
> [  719.493788] xen_bridge: port 7(vif43.0) entered forwarding state
> [  726.302456] xen_bridge: port 7(vif43.0) entered disabled state
> [  726.308898] device vif43.0 left promiscuous mode
> [  726.314029] xen_bridge: port 7(vif43.0) entered disabled state
> 
> All the guests are already up, but this keeps on going and going and going ....

The domain number seems to be climbing, are you sure something isn't
(crashing and) restarting?

> > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> > index b06ef81..8a3f770 100644
> > --- a/drivers/net/xen-netfront.c
> > +++ b/drivers/net/xen-netfront.c
> > @@ -462,6 +462,8 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
> >                 ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> >                 BUG_ON((signed short)ref < 0);
> >  
> > +               BUG_ON(PageCompound(skb_frag_page(frag)));
> > +
> >                 mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
> >                 gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> >                                                 mfn, GNTMAP_readonly);
> 
> > My repro for netback was just to netcat a wodge of data from dom0->domU
> > but going the other way doesn't seem to trigger.
> 
> 
> 

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Sander Eikelenboom]

[-- Attachment #1: Type: text/plain, Size: 6593 bytes --]


Thursday, October 11, 2012, 10:02:26 AM, you wrote:

> On Wed, 2012-10-10 at 15:49 +0100, Sander Eikelenboom wrote:
>> Wednesday, October 10, 2012, 3:09:58 PM, you wrote:
>> 
>> > On Wed, 2012-10-10 at 11:13 +0100, Ian Campbell wrote:
>> >> I haven't tackled netfront yet. 
>> 
>> > I seem to be totally unable to reproduce the equivalent issue on the
>> > netfront xmit side, even though it seems like the loop in
>> > xennet_make_frags ought to be obviously susceptible to it.
>> 
>> > Konrad, Sander, are either of you able to repro, e.g. with:
>> 
>> 
>> Hmrrrmm i don't see any traces, only strange behaviour ..
>> 
>> - i can connect to guests by ssh, but it's sluggish, and sometimes stops working

> I saw something like this (ssh sluggish) even with dom0 itself. I'm
> trying to see if I can characterise it enough to reliably bisect it.

> I already switched out xen-unstable for 4.2-testing but that didn't make
> any difference.



>> - The guest seem to keep trying to connect to netback:
>> 
>> [  658.276719] xen_bridge: port 2(vif40.0) entered forwarding state
>> [  658.282258] xen_bridge: port 2(vif40.0) entered forwarding state
>> [  663.945964] xen_bridge: port 7(vif39.0) entered forwarding state
>> [  669.674277] xen_bridge: port 2(vif40.0) entered disabled state
>> [  669.680290] device vif40.0 left promiscuous mode
>> [  669.685464] xen_bridge: port 2(vif40.0) entered disabled state
>> [  672.857222] device vif41.0 entered promiscuous mode
>> [  673.166254] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
>> [  673.176368] xen_bridge: port 2(vif41.0) entered forwarding state
>> [  673.182042] xen_bridge: port 2(vif41.0) entered forwarding state
>> [  674.439725] xen_bridge: port 7(vif39.0) entered disabled state
>> [  674.445708] device vif39.0 left promiscuous mode
>> [  674.450955] xen_bridge: port 7(vif39.0) entered disabled state
>> [  677.726040] device vif42.0 entered promiscuous mode
>> [  678.053381] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
>> [  678.062804] xen_bridge: port 7(vif42.0) entered forwarding state
>> [  678.068433] xen_bridge: port 7(vif42.0) entered forwarding state
>> [  688.224736] xen_bridge: port 2(vif41.0) entered forwarding state
>> [  693.080557] xen_bridge: port 7(vif42.0) entered forwarding state
>> [  700.786276] xen_bridge: port 7(vif42.0) entered disabled state
>> [  700.792484] device vif42.0 left promiscuous mode
>> [  700.802409] xen_bridge: port 7(vif42.0) entered disabled state
>> [  704.133606] device vif43.0 entered promiscuous mode
>> [  704.460160] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
>> [  704.469800] xen_bridge: port 7(vif43.0) entered forwarding state
>> [  704.475303] xen_bridge: port 7(vif43.0) entered forwarding state
>> [  719.493788] xen_bridge: port 7(vif43.0) entered forwarding state
>> [  726.302456] xen_bridge: port 7(vif43.0) entered disabled state
>> [  726.308898] device vif43.0 left promiscuous mode
>> [  726.314029] xen_bridge: port 7(vif43.0) entered disabled state
>> 
>> All the guests are already up, but this keeps on going and going and going ....

> The domain number seems to be climbing, are you sure something isn't
> (crashing and) restarting?

Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
And i seem to hit it, but only in one of the guests at the moment and it triggers quite irregularly.

[   34.298549] ------------[ cut here ]------------
[   34.298567] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.298574] Modules linked in:
[   34.298597] Pid: 1580, comm: sshd Not tainted 3.6.0pre-rc1-20121011 #1
[   34.298603] Call Trace:
[   34.298611]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.298617]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.298623]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.298631]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.298637]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.298643]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.298649]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.298656]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[   34.298663]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.298668]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.298674]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.298680]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.298687]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.298692]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.298699]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.298705]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.298711]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.298717]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[   34.298723]  [<ffffffff816d225b>] tcp_push_one+0x2b/0x40
[   34.298728]  [<ffffffff816c2dec>] tcp_sendmsg+0x8dc/0xe20
[   34.298735]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[   34.298740]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[   34.298746]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.298753]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[   34.298762]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[   34.298768]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.298774]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[   34.298779]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[   34.298786]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   34.298792]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[   34.298797] ---[ end trace 2e28eec93b7a8b74 ]---


Complete dmesg from guest attached.



>> > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
>> > index b06ef81..8a3f770 100644
>> > --- a/drivers/net/xen-netfront.c
>> > +++ b/drivers/net/xen-netfront.c
>> > @@ -462,6 +462,8 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
>> >                 ref = gnttab_claim_grant_reference(&np->gref_tx_head);
>> >                 BUG_ON((signed short)ref < 0);
>> >  
>> > +               BUG_ON(PageCompound(skb_frag_page(frag)));
>> > +
>> >                 mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
>> >                 gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
>> >                                                 mfn, GNTMAP_readonly);
>> 
>> > My repro for netback was just to netcat a wodge of data from dom0->domU
>> > but going the other way doesn't seem to trigger.
>> 
>> 
>> 



[-- Attachment #2: dmesg-netfront.txt --]
[-- Type: text/plain, Size: 177798 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.6.0pre-rc1-20121011 (root@serveerstertje) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP PREEMPT Thu Oct 11 11:04:38 CEST 2012
[    0.000000] Command line: root=/dev/xvda1 ro 
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000001fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
[    0.000000] initial memory mapped: [mem 0x00000000-0x037c9fff]
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x1fffffff]
[    0.000000]  [mem 0x00000000-0x1fffffff] page 4k
[    0.000000] kernel direct mapping tables up to 0x1fffffff @ [mem 0x029c4000-0x02ac5fff]
[    0.000000] xen: setting RW the range 2aa6000 - 2ac6000
[    0.000000] RAMDISK: [mem 0x02ac6000-0x037c9fff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1fffffff]
[    0.000000]   NODE_DATA [mem 0x1fff5000-0x1fffffff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00010000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00010000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x1fffffff]
[    0.000000] On node 0 totalpages: 130960
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 6 pages reserved
[    0.000000]   DMA zone: 3914 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1984 pages used for memmap
[    0.000000]   DMA32 zone: 124992 pages, LIFO batch:31
[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] e820: [mem 0x20000000-0xffffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.3-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88001fc00000 s81088 r8192 d21312 u2097152
[    0.000000] pcpu-alloc: s81088 r8192 d21312 u2097152 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 128906
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/xvda1 ro 
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] __ex_table already sorted, skipping sort
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 476724k/524288k available (7472k kernel code, 448k absent, 47116k reserved, 5626k data, 696k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	Additional per-CPU info printed with stalls.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
[    0.000000] NR_IRQS:4352 nr_irqs:256 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.000000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.000000] ... MAX_LOCK_DEPTH:          48
[    0.000000] ... MAX_LOCKDEP_KEYS:        8191
[    0.000000] ... CLASSHASH_SIZE:          4096
[    0.000000] ... MAX_LOCKDEP_ENTRIES:     16384
[    0.000000] ... MAX_LOCKDEP_CHAINS:      32768
[    0.000000] ... CHAINHASH_SIZE:          16384
[    0.000000]  memory used by lock dependency info: 5855 kB
[    0.000000]  per task-struct memory footprint: 1920 bytes
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 3200.144 MHz processor
[    0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 6402.96 BogoMIPS (lpj=10667146)
[    0.003333] pid_max: default: 32768 minimum: 301
[    0.003333] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.003333] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.003333] Mount-cache hash table entries: 256
[    0.003333] Initializing cgroup subsys cpuacct
[    0.003333] Initializing cgroup subsys freezer
[    0.003333] Initializing cgroup subsys blkio
[    0.003333] tseg: 0000000000
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8
[    0.003333] Last level dTLB entries: 4KB 512, 2MB 128, 4MB 64
[    0.003333] tlb_flushall_shift: 4
[    0.025363] Freeing SMP alternatives: 24k freed
[    0.026174] Performance Events: 
[    0.026187] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.026193] no hardware sampling interrupt available.
[    0.026215] Broken PMU hardware detected, using software events only.
[    0.026220] Failed to access perfctr msr (MSR c0010004 is 2185c52b638f)
[    0.043549] Brought up 1 CPUs
[    0.043634] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.044079] Grant tables using version 2 layout.
[    0.044103] Grant table initialized
[    0.044225] NET: Registered protocol family 16
[    0.045499] PCI: setting up Xen PCI frontend stub
[    0.045499] PCI: pci_cache_line_size set to 64 bytes
[    0.063094] bio: create slab <bio-0> at 0
[    0.063323] ACPI: Interpreter disabled.
[    0.063517] xen/balloon: Initialising balloon driver.
[    0.063594] xen-balloon: Initialising balloon driver.
[    0.063594] vgaarb: loaded
[    0.063795] SCSI subsystem initialized
[    0.063931] libata version 3.00 loaded.
[    0.064210] usbcore: registered new interface driver usbfs
[    0.064270] usbcore: registered new interface driver hub
[    0.064346] usbcore: registered new device driver usb
[    0.064482] Linux video capture interface: v2.00
[    0.064809] Advanced Linux Sound Architecture Driver Initialized.
[    0.064815] PCI: System does not support PCI
[    0.064819] PCI: System does not support PCI
[    0.067137] Switching to clocksource xen
[    0.067388] pnp: PnP ACPI: disabled
[    0.071247] NET: Registered protocol family 2
[    0.071479] TCP established hash table entries: 16384 (order: 6, 262144 bytes)
[    0.071680] TCP bind hash table entries: 16384 (order: 8, 1048576 bytes)
[    0.072505] TCP: Hash tables configured (established 16384 bind 16384)
[    0.072557] TCP: reno registered
[    0.072574] UDP hash table entries: 256 (order: 3, 40960 bytes)
[    0.072612] UDP-Lite hash table entries: 256 (order: 3, 40960 bytes)
[    0.072755] NET: Registered protocol family 1
[    0.072767] PCI: CLS 0 bytes, default 64
[    0.072912] Trying to unpack rootfs image as initramfs...
[    0.094234] Freeing initrd memory: 13328k freed
[    0.101194] DMA-API: preallocated 32768 debug entries
[    0.101206] DMA-API: debugging enabled by kernel config
[    0.101420] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.102549] sha1_ssse3: Neither AVX nor SSSE3 is available/usable.
[    0.102915] audit: initializing netlink socket (disabled)
[    0.102948] type=2000 audit(1349948050.825:1): initialized
[    0.103470] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.108912] VFS: Disk quotas dquot_6.5.2
[    0.109021] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.110540] NTFS driver 2.1.30 [Flags: R/W].
[    0.110758] fuse init (API version 7.20)
[    0.110960] msgmni has been set to 957
[    0.111673] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.111695] io scheduler noop registered
[    0.111699] io scheduler deadline registered
[    0.111880] io scheduler cfq registered (default)
[    0.112908] crc32: CRC_LE_BITS = 64, CRC_BE BITS = 64
[    0.112912] crc32: self tests passed, processed 225944 bytes in 112580 nsec
[    0.113021] crc32c: CRC_LE_BITS = 64
[    0.117226] crc32c: self tests passed, processed 225944 bytes in 50333 nsec
[    0.117576] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.117797] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.118013] usbcore: registered new interface driver udlfb
[    0.118467] Event-channel device installed.
[    0.119213] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.120055] Linux agpgart interface v0.103
[    0.120374] Hangcheck: starting hangcheck timer 0.9.1 (tick is 180 seconds, margin is 60 seconds).
[    0.120383] Hangcheck: Using getrawmonotonic().
[    0.120443] [drm] Initialized drm 1.1.0 20060810
[    0.123268] brd: module loaded
[    0.124859] loop: module loaded
[    0.127778] tun: Universal TUN/TAP device driver, 1.6
[    0.127792] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    0.127873] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    0.127880] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    0.128004] Initialising Xen virtual ethernet driver.
[    0.133058] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.133071] ehci_hcd: block sizes: qh 104 qtd 96 itd 192 sitd 96
[    0.133130] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.133136] ohci_hcd: block sizes: ed 80 td 96
[    0.133188] uhci_hcd: USB Universal Host Controller Interface driver
[    0.133301] usbcore: registered new interface driver usblp
[    0.133306] Initializing USB Mass Storage driver...
[    0.133359] usbcore: registered new interface driver usb-storage
[    0.133364] USB Mass Storage support registered.
[    0.133497] usbcore: registered new interface driver usbserial
[    0.133568] usbcore: registered new interface driver cp210x
[    0.133701] usbserial: USB Serial support registered for cp210x
[    0.133758] usbcore: registered new interface driver cypress_m8
[    0.133815] usbserial: USB Serial support registered for DeLorme Earthmate USB
[    0.133872] usbserial: USB Serial support registered for HID->COM RS232 Adapter
[    0.133931] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter
[    0.133989] usbcore: registered new interface driver mos7720
[    0.134045] usbserial: USB Serial support registered for Moschip 2 port adapter
[    0.134102] usbcore: registered new interface driver mos7840
[    0.134158] usbserial: USB Serial support registered for Moschip 7840/7820 USB Serial Driver
[    0.134360] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    0.135190] i8042: No controller found
[    0.135284] mousedev: PS/2 mouse device common for all mice
[    0.137520] blkfront: xvda: flush diskcache: enabled
[    0.196054] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.196438] ida_remove called for id=0 which is not allocated.
[    0.196448] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.197009] lirc_dev: IR Remote Control driver registered, major 250 
[    0.197027] IR NEC protocol handler initialized
[    0.197031] IR RC5(x) protocol handler initialized
[    0.197035] IR RC6 protocol handler initialized
[    0.197038] IR JVC protocol handler initialized
[    0.197041] IR Sony protocol handler initialized
[    0.197045] IR RC5 (streamzap) protocol handler initialized
[    0.197049] IR SANYO protocol handler initialized
[    0.197052] IR MCE Keyboard/mouse protocol handler initialized
[    0.197057] IR LIRC bridge handler initialized
[    0.197268] sp5100_tco: SP5100 TCO WatchDog Timer Driver v0.01
[    0.197382] xen_wdt: Xen WatchDog Timer Driver v0.01
[    0.197578] xen_wdt: initialized (timeout=60s, nowayout=0)
[    0.197820] device-mapper: ioctl: 4.23.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com
[    0.199123] usbcore: registered new interface driver usbhid
[    0.199129] usbhid: USB HID core driver
[    0.202313] usbcore: registered new interface driver snd-usb-audio
[    0.202373] usbcore: registered new interface driver snd-ua101
[    0.202434] usbcore: registered new interface driver snd-usb-usx2y
[    0.202488] usbcore: registered new interface driver snd-usb-caiaq
[    0.202543] usbcore: registered new interface driver snd-usb-6fire
[    0.202575] Netfilter messages via NETLINK v0.30.
[    0.202583] nfnl_acct: registering with nfnetlink.
[    0.202626] nf_conntrack version 0.5.0 (3828 buckets, 15312 max)
[    0.202950] ctnetlink v0.93: registering with nfnetlink.
[    0.203250] xt_time: kernel timezone is -0000
[    0.316848] ip_set: protocol 6
[    0.316875] IPVS: Registered protocols ()
[    0.316996] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[    0.317068] IPVS: Creating netns size=1896 id=0
[    0.317100] IPVS: ipvs loaded.
[    0.317213] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.317279] TCP: cubic registered
[    0.317285] NET: Registered protocol family 17
[    0.317340] Bridge firewalling registered
[    0.317359] Ebtables v2.0 registered
[    0.317617] registered taskstats version 1
[    0.782134]  xvda: xvda1 xvda2
[    0.817140] console [netcon0] enabled
[    0.817166] netconsole: network logging started
[    0.817368] ALSA device list:
[    0.817385]   No soundcards found.
[    0.820425] Freeing unused kernel memory: 696k freed
[    0.820887] Write protecting the kernel read-only data: 12288k
[    0.837830] Freeing unused kernel memory: 708k freed
[    0.839937] Freeing unused kernel memory: 1076k freed
[    3.871581] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
[   13.486115] Adding 769020k swap on /dev/xvda2.  Priority:-1 extents:1 across:769020k SS
[   13.529134] EXT4-fs (xvda1): re-mounted. Opts: (null)
[   14.449206] EXT4-fs (xvda1): re-mounted. Opts: errors=remount-ro
[   22.967397] tty_init_dev: 2 callbacks suppressed
[   22.967692] ttyS0: LSR safety check engaged!
[   22.970334] ttyS0: LSR safety check engaged!
[   22.972956] ttyS1: LSR safety check engaged!
[   22.974811] ttyS1: LSR safety check engaged!
[   34.298549] ------------[ cut here ]------------
[   34.298567] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.298574] Modules linked in:
[   34.298597] Pid: 1580, comm: sshd Not tainted 3.6.0pre-rc1-20121011 #1
[   34.298603] Call Trace:
[   34.298611]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.298617]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.298623]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.298631]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.298637]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.298643]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.298649]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.298656]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[   34.298663]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.298668]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.298674]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.298680]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.298687]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.298692]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.298699]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.298705]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.298711]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.298717]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[   34.298723]  [<ffffffff816d225b>] tcp_push_one+0x2b/0x40
[   34.298728]  [<ffffffff816c2dec>] tcp_sendmsg+0x8dc/0xe20
[   34.298735]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[   34.298740]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[   34.298746]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.298753]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[   34.298762]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[   34.298768]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.298774]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[   34.298779]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[   34.298786]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   34.298792]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[   34.298797] ---[ end trace 2e28eec93b7a8b74 ]---
[   34.298825] ------------[ cut here ]------------
[   34.298832] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.298838] Modules linked in:
[   34.298843] Pid: 1580, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   34.298848] Call Trace:
[   34.298852]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.298858]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.298864]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.298870]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.298876]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.298881]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.298887]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.298893]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[   34.298899]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.298904]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.298910]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.298916]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.298921]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.298927]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.298932]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.298938]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.298943]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.298949]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[   34.298955]  [<ffffffff816d229d>] __tcp_push_pending_frames+0x2d/0x90
[   34.298961]  [<ffffffff816c2693>] tcp_sendmsg+0x183/0xe20
[   34.298966]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[   34.298972]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[   34.298977]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.298983]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[   34.298988]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[   34.298994]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.299000]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[   34.299005]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[   34.299011]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   34.299017]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[   34.299022] ---[ end trace 2e28eec93b7a8b75 ]---
[   34.299036] ------------[ cut here ]------------
[   34.299042] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.299047] Modules linked in:
[   34.299052] Pid: 1580, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   34.299057] Call Trace:
[   34.299061]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.299071]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.299077]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.299084]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.299089]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.299095]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.299101]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.299106]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[   34.299112]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.299118]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.299124]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.299129]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.299135]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.299140]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.299146]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.299152]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.299157]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.299163]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[   34.299169]  [<ffffffff816d229d>] __tcp_push_pending_frames+0x2d/0x90
[   34.299174]  [<ffffffff816c2693>] tcp_sendmsg+0x183/0xe20
[   34.299180]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[   34.299185]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[   34.299191]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.299196]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[   34.299202]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[   34.299208]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.299213]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[   34.299219]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[   34.299224]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   34.299239]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[   34.299243] ---[ end trace 2e28eec93b7a8b76 ]---
[   34.794592] ------------[ cut here ]------------
[   34.794615] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.794623] Modules linked in:
[   34.794632] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   34.794640] Call Trace:
[   34.794644]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.794660]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.794667]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.794677]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.794685]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.794693]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.794701]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.794710]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   34.794719]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.794730]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.794742]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.794750]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.794757]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.794765]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.794773]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.794781]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.794788]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.794796]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   34.794804]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[   34.794812]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[   34.794819]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[   34.794827]  [<ffffffff816cd858>] tcp_rcv_established+0x318/0x640
[   34.794838]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.794845]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[   34.794854]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[   34.794862]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.794869]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[   34.794876]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.794884]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.794892]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[   34.794900]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.794908]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[   34.794915]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[   34.794922]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[   34.794930]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[   34.794937]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[   34.794945]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.794953]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[   34.794961]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[   34.794968]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[   34.794976]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[   34.794990]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[   34.794998]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[   34.795006]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   34.795014]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   34.795021]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   34.795028]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   34.795037]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   34.795044]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   34.795050]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   34.795063]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   34.795072]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   34.795080]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   34.795087]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   34.795094]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   34.795101]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   34.795111]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   34.795119]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   34.795126]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   34.795135]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   34.795142] ---[ end trace 2e28eec93b7a8b77 ]---
[   34.795170] ------------[ cut here ]------------
[   34.795178] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.795186] Modules linked in:
[   34.795192] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   34.795199] Call Trace:
[   34.795203]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.795215]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.795223]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.795231]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.795239]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.795247]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.795254]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.795262]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   34.795270]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.795278]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.795285]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.795293]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.795300]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.795307]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.795315]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.795322]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.795330]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.795337]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   34.795345]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[   34.795354]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[   34.795362]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[   34.795369]  [<ffffffff816cd858>] tcp_rcv_established+0x318/0x640
[   34.795377]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.795384]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[   34.795392]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[   34.795399]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.795406]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[   34.795413]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.795420]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.795429]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[   34.795440]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.795452]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[   34.795459]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[   34.795466]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[   34.795473]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[   34.795482]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[   34.795490]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.795497]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[   34.795505]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[   34.795513]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[   34.795520]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[   34.795529]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[   34.795536]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[   34.795544]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   34.795551]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   34.795558]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   34.795565]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   34.795572]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   34.795580]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   34.795586]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   34.795598]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   34.795606]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   34.795613]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   34.795620]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   34.795627]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   34.795633]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   34.795641]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   34.795649]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   34.795656]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   34.795665]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   34.795672] ---[ end trace 2e28eec93b7a8b78 ]---
[   34.797450] ------------[ cut here ]------------
[   34.797464] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.797474] Modules linked in:
[   34.797482] Pid: 1532, comm: klogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   34.797489] Call Trace:
[   34.797493]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.797506]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.797513]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.797522]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.797530]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.797537]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.797545]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.797554]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   34.797562]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.797570]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.797578]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.797587]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.797594]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.797601]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.797609]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.797617]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.797624]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.797632]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   34.797640]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[   34.797648]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[   34.797656]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[   34.797663]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[   34.797671]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[   34.797678]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.797686]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[   34.797693]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[   34.797701]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.797708]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[   34.797715]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.797723]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.797731]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[   34.797743]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.797751]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[   34.797759]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[   34.797766]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[   34.797773]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[   34.797780]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[   34.797788]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.797796]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[   34.797804]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[   34.797811]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[   34.797819]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[   34.797827]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[   34.797835]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[   34.797842]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   34.797849]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   34.797856]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   34.797863]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   34.797871]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   34.797879]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   34.797885]  <EOI> 
[   34.797889] ---[ end trace 2e28eec93b7a8b79 ]---
[   34.797917] ------------[ cut here ]------------
[   34.797925] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.797932] Modules linked in:
[   34.797939] Pid: 1532, comm: klogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   34.797947] Call Trace:
[   34.797950]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.797962]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.797970]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.797978]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.797986]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.797994]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.798001]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.798012]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   34.798019]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.798027]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.798035]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.798043]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.798050]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.798057]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.798065]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.798072]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.798080]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.798087]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   34.798095]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[   34.798103]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[   34.798111]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[   34.798119]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[   34.798126]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[   34.798134]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.798141]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[   34.798149]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[   34.798156]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.798164]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[   34.798171]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.798178]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.798186]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[   34.798194]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.798202]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[   34.798210]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[   34.798220]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[   34.798227]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[   34.798234]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[   34.798242]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.798249]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[   34.798257]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[   34.798265]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[   34.798276]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[   34.798285]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[   34.798292]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[   34.798300]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   34.798307]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   34.798314]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   34.798321]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   34.798328]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   34.798335]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   34.798342]  <EOI> 
[   34.798345] ---[ end trace 2e28eec93b7a8b7a ]---
[   34.798368] ------------[ cut here ]------------
[   34.798375] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   34.798383] Modules linked in:
[   34.798388] Pid: 1532, comm: klogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   34.798396] Call Trace:
[   34.798400]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.798411]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.798418]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.798426]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.798434]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.798442]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.798449]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.798457]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   34.798465]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.798472]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.798480]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.798487]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.798494]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.798502]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.798510]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.798518]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.798526]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.798534]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   34.798541]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[   34.798550]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[   34.798558]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[   34.798565]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[   34.798573]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[   34.798580]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.798588]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[   34.798595]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[   34.798603]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[   34.798610]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[   34.798617]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.798624]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.798633]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[   34.798640]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[   34.798648]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[   34.798656]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[   34.798663]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[   34.798670]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[   34.798678]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[   34.798685]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.798693]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[   34.798701]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[   34.798708]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[   34.798716]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[   34.798725]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[   34.798732]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[   34.798739]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   34.798747]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   34.798754]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   34.798761]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   34.798768]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   34.798776]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   34.798782]  <EOI> 
[   34.798786] ---[ end trace 2e28eec93b7a8b7b ]---
[   35.363444] ------------[ cut here ]------------
[   35.363461] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   35.363468] Modules linked in:
[   35.363474] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   35.363480] Call Trace:
[   35.363483]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   35.363495]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   35.363500]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   35.363509]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   35.363515]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   35.363522]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   35.363528]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   35.363535]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   35.363542]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   35.363548]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   35.363555]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   35.363560]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   35.363566]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   35.363571]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   35.363578]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   35.363585]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   35.363591]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   35.363597]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   35.363603]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   35.363609]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[   35.363615]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[   35.363620]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[   35.363626]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[   35.363632]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[   35.363637]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   35.363643]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   35.363649]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[   35.363656]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   35.363662]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   35.363668]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   35.363673]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   35.363680]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   35.363686]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   35.363691]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   35.363701]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   35.363707]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   35.363713]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   35.363719]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   35.363724]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   35.363729]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   35.363737]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   35.363743]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   35.363749]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   35.363759]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   35.363764] ---[ end trace 2e28eec93b7a8b7c ]---
[   36.500221] ------------[ cut here ]------------
[   36.500267] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   36.500293] Modules linked in:
[   36.500315] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   36.500339] Call Trace:
[   36.500351]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   36.500396]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   36.500420]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   36.500447]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   36.500471]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   36.500494]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   36.500516]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   36.500541]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   36.500570]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   36.500594]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   36.500619]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   36.500641]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   36.500662]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   36.500684]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   36.500709]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   36.500732]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   36.500755]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   36.500779]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   36.500803]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   36.500827]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[   36.500850]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[   36.500874]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[   36.500897]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[   36.500917]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[   36.500938]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   36.500962]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   36.500985]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[   36.501011]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   36.501034]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   36.501055]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   36.501077]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   36.501101]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   36.501124]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   36.501143]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   36.501181]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   36.501205]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   36.501228]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   36.501249]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   36.501270]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   36.501290]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   36.501318]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   36.501341]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   36.501364]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   36.501388]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   36.501409] ---[ end trace 2e28eec93b7a8b7d ]---
[   38.773510] ------------[ cut here ]------------
[   38.773528] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   38.773535] Modules linked in:
[   38.773542] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   38.773548] Call Trace:
[   38.773552]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   38.773564]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   38.773570]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   38.773582]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   38.773589]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   38.773594]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   38.773600]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   38.773607]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   38.773613]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   38.773619]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   38.773625]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   38.773631]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   38.773636]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   38.773643]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   38.773649]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   38.773656]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   38.773662]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   38.773668]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   38.773674]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   38.773680]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[   38.773685]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[   38.773691]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[   38.773697]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[   38.773702]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[   38.773707]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   38.773713]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   38.773719]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[   38.773726]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   38.773732]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   38.773738]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   38.773743]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   38.773751]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   38.773756]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   38.773761]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   38.773771]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   38.773778]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   38.773784]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   38.773789]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   38.773794]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   38.773800]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   38.773807]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   38.773813]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   38.773819]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   38.773825]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   38.773831] ---[ end trace 2e28eec93b7a8b7e ]---
[   43.320207] ------------[ cut here ]------------
[   43.320235] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   43.320244] Modules linked in:
[   43.320252] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   43.320261] Call Trace:
[   43.320265]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   43.320280]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   43.320288]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   43.320298]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   43.320307]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   43.320315]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   43.320323]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   43.320337]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   43.320346]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   43.320354]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   43.320362]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   43.320370]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   43.320377]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   43.320385]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   43.320393]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   43.320402]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   43.320409]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   43.320417]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   43.320425]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   43.320433]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[   43.320441]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[   43.320449]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[   43.320456]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[   43.320463]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[   43.320470]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   43.320479]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   43.320487]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[   43.320496]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   43.320503]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   43.320511]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   43.320518]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   43.320528]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   43.320535]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   43.320542]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   43.320555]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   43.320564]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   43.320571]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   43.320578]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   43.320586]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   43.320592]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   43.320602]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   43.320610]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   43.320617]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   43.320626]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   43.320633] ---[ end trace 2e28eec93b7a8b7f ]---
[   52.400126] ------------[ cut here ]------------
[   52.400146] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   52.400152] Modules linked in:
[   52.400158] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   52.400164] Call Trace:
[   52.400167]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   52.400178]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   52.400183]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   52.400191]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   52.400197]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   52.400202]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   52.400208]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   52.400214]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[   52.400221]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   52.400226]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   52.400232]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   52.400237]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   52.400242]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   52.400247]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   52.400254]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   52.400261]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   52.400266]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   52.400272]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   52.400278]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   52.400283]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[   52.400288]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[   52.400293]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[   52.400299]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[   52.400303]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[   52.400308]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   52.400313]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   52.400319]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[   52.400325]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   52.400330]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   52.400336]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   52.400341]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   52.400347]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   52.400352]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   52.400356]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   52.400366]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   52.400371]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   52.400377]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   52.400382]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   52.400387]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   52.400391]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   52.400398]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   52.400403]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   52.400408]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   52.400414]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   52.400420] ---[ end trace 2e28eec93b7a8b80 ]---
[   70.560242] ------------[ cut here ]------------
[   70.560332] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[   70.560360] Modules linked in:
[   70.560381] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[   70.560408] Call Trace:
[   70.560421]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   70.560462]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   70.560485]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   70.560512]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   70.560536]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   70.560559]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   70.560581]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   70.560605]  [<ffffffff81629a77>] neigh_resolve_output+0x127/0x250
[   70.560630]  [<ffffffff816b96ad>] ip_finish_output+0x39d/0x530
[   70.560653]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   70.560679]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   70.560701]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   70.560722]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   70.560744]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   70.560768]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   70.560791]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   70.560812]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   70.560835]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[   70.560859]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   70.560882]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[   70.560905]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[   70.560928]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[   70.560951]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[   70.560972]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[   70.560993]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   70.561017]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[   70.561040]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[   70.561064]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[   70.561087]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[   70.561108]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[   70.561130]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[   70.561153]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[   70.561176]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[   70.561195]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   70.561232]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[   70.561256]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[   70.561278]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[   70.561299]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[   70.561319]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[   70.561339]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[   70.561390]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[   70.561414]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[   70.561436]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[   70.561462]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[   70.561483] ---[ end trace 2e28eec93b7a8b81 ]---
[  106.933420] ------------[ cut here ]------------
[  106.933445] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  106.933452] Modules linked in:
[  106.933458] Pid: 1954, comm: gzip Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  106.933463] Call Trace:
[  106.933466]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  106.933476]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  106.933482]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  106.933490]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  106.933496]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  106.933501]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  106.933506]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  106.933512]  [<ffffffff81629a77>] neigh_resolve_output+0x127/0x250
[  106.933519]  [<ffffffff816b96ad>] ip_finish_output+0x39d/0x530
[  106.933525]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  106.933530]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  106.933535]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  106.933540]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  106.933545]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  106.933551]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  106.933557]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  106.933562]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  106.933567]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  106.933573]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  106.933578]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  106.933584]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  106.933590]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  106.933595]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  106.933600]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  106.933604]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  106.933610]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  106.933615]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  106.933621]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  106.933627]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  106.933632]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  106.933637]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  106.933643]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  106.933648]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  106.933653]  <EOI> 
[  106.933655] ---[ end trace 2e28eec93b7a8b82 ]---
[  179.680315] ------------[ cut here ]------------
[  179.680344] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  179.680357] Modules linked in:
[  179.680369] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  179.680381] Call Trace:
[  179.680387]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  179.680409]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  179.680420]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  179.680437]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  179.680449]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  179.680461]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  179.680472]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  179.680485]  [<ffffffff81629a77>] neigh_resolve_output+0x127/0x250
[  179.680498]  [<ffffffff816b96ad>] ip_finish_output+0x39d/0x530
[  179.680511]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  179.680523]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  179.680533]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  179.680544]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  179.680555]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  179.680567]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  179.680579]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  179.680591]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  179.680603]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  179.680615]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  179.680627]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  179.680639]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  179.680651]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  179.680662]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  179.680673]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  179.680683]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  179.680696]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  179.680707]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  179.680720]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  179.680732]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  179.680743]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  179.680754]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  179.680766]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  179.680777]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  179.680786]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  179.680806]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  179.680818]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  179.680830]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  179.680841]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  179.680851]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  179.680861]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  179.680875]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  179.680886]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  179.680898]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  179.680911]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  179.680922] ---[ end trace 2e28eec93b7a8b83 ]---
[  199.302239] tty_init_dev: 19 callbacks suppressed
[  300.148171] ------------[ cut here ]------------
[  300.148226] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  300.148255] Modules linked in:
[  300.148278] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  300.148304] Call Trace:
[  300.148318]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  300.148362]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  300.148386]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  300.148415]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  300.148441]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  300.148464]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  300.148488]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  300.148514]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  300.148539]  [<ffffffff81629a77>] neigh_resolve_output+0x127/0x250
[  300.148565]  [<ffffffff816ad1be>] ? ipv4_neigh_lookup+0x3e/0x150
[  300.148588]  [<ffffffff8162ad37>] neigh_update+0x297/0x5e0
[  300.148610]  [<ffffffff8162adc2>] ? neigh_update+0x322/0x5e0
[  300.148633]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  300.148661]  [<ffffffff8106e756>] ? local_bh_enable+0xc6/0x160
[  300.148702]  [<ffffffff816e2f55>] arp_process+0x2e5/0x700
[  300.148725]  [<ffffffff810ad345>] ? debug_check_no_locks_freed+0x115/0x1d0
[  300.148751]  [<ffffffff816e3445>] arp_rcv+0xd5/0x140
[  300.148773]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  300.148796]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  300.148820]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  300.148845]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  300.148870]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  300.148896]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  300.148920]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  300.148947]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  300.148971]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  300.148995]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  300.149019]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  300.149042]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  300.149064]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  300.149089]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  300.149113]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  300.149133]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  300.149172]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  300.149198]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  300.149222]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  300.149245]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  300.149267]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  300.149288]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  300.149316]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  300.149340]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  300.149364]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  300.149391]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  300.149413] ---[ end trace 2e28eec93b7a8b84 ]---
[  312.591132] ------------[ cut here ]------------
[  312.591164] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  312.591171] Modules linked in:
[  312.591177] Pid: 2159, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  312.591183] Call Trace:
[  312.591191]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  312.591213]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  312.591219]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  312.591226]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  312.591233]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  312.591239]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  312.591244]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  312.591251]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[  312.591259]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  312.591264]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  312.591270]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  312.591276]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  312.591281]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  312.591286]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  312.591293]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  312.591299]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  312.591306]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  312.591312]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[  312.591317]  [<ffffffff816d225b>] tcp_push_one+0x2b/0x40
[  312.591324]  [<ffffffff816c2dec>] tcp_sendmsg+0x8dc/0xe20
[  312.591331]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[  312.591336]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[  312.591342]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  312.591349]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[  312.591356]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[  312.591365]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  312.591371]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[  312.591376]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[  312.591383]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[  312.591391]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[  312.591396] ---[ end trace 2e28eec93b7a8b85 ]---
[  312.591429] ------------[ cut here ]------------
[  312.591435] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  312.591441] Modules linked in:
[  312.591445] Pid: 2159, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  312.591451] Call Trace:
[  312.591457]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  312.591463]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  312.591468]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  312.591475]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  312.591481]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  312.591487]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  312.591492]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  312.591498]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[  312.591504]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  312.591510]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  312.591516]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  312.591521]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  312.591526]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  312.591532]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  312.591538]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  312.591544]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  312.591549]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  312.591555]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[  312.591560]  [<ffffffff816d229d>] __tcp_push_pending_frames+0x2d/0x90
[  312.591566]  [<ffffffff816c2693>] tcp_sendmsg+0x183/0xe20
[  312.591572]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[  312.591578]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[  312.591583]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  312.591588]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[  312.591594]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[  312.591600]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  312.591606]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[  312.591611]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[  312.591617]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[  312.591623]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[  312.591627] ---[ end trace 2e28eec93b7a8b86 ]---
[  312.591645] ------------[ cut here ]------------
[  312.591651] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  312.591657] Modules linked in:
[  312.591662] Pid: 2159, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  312.591667] Call Trace:
[  312.591671]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  312.591677]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  312.591682]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  312.591688]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  312.591694]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  312.591700]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  312.591705]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  312.591711]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[  312.591717]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  312.591726]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  312.591732]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  312.591737]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  312.591743]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  312.591748]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  312.591754]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  312.591759]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  312.591765]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  312.591770]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[  312.591776]  [<ffffffff816d229d>] __tcp_push_pending_frames+0x2d/0x90
[  312.591782]  [<ffffffff816c2693>] tcp_sendmsg+0x183/0xe20
[  312.591788]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[  312.591793]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[  312.591799]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  312.591804]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[  312.591810]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[  312.591816]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  312.591822]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[  312.591827]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[  312.591833]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[  312.591839]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[  312.591843] ---[ end trace 2e28eec93b7a8b87 ]---
[  313.048431] ------------[ cut here ]------------
[  313.048487] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  313.048514] Modules linked in:
[  313.048536] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  313.048560] Call Trace:
[  313.048573]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  313.048614]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  313.048638]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  313.048665]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  313.048689]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  313.048711]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  313.048734]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  313.048758]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  313.048783]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  313.048806]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  313.048829]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  313.048850]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  313.048872]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  313.048896]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  313.048920]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  313.048943]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  313.048965]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  313.048988]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  313.049011]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  313.049034]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  313.049057]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  313.049079]  [<ffffffff816cd858>] tcp_rcv_established+0x318/0x640
[  313.049102]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.049123]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  313.049147]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  313.049169]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.049191]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  313.049212]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  313.049233]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.049257]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  313.049280]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.049304]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  313.049326]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  313.049347]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  313.049368]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  313.049393]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  313.049416]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  313.049440]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  313.049463]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  313.049486]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  313.049508]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  313.049534]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  313.049557]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  313.049579]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  313.049602]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  313.049623]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  313.049645]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  313.049668]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  313.049690]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  313.049709]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  313.049745]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  313.049769]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  313.049791]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  313.049812]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  313.049832]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  313.049852]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  313.049878]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  313.049901]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  313.049928]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  313.049952]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  313.049974] ---[ end trace 2e28eec93b7a8b88 ]---
[  313.050054] ------------[ cut here ]------------
[  313.050077] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  313.050102] Modules linked in:
[  313.050121] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  313.050143] Call Trace:
[  313.050153]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  313.050187]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  313.050210]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  313.050234]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  313.050257]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  313.050280]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  313.050302]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  313.050325]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  313.050348]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  313.050373]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  313.050397]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  313.050418]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  313.050439]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  313.050461]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  313.050484]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  313.050506]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  313.050527]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  313.050550]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  313.050574]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  313.050598]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  313.050621]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  313.050643]  [<ffffffff816cd858>] tcp_rcv_established+0x318/0x640
[  313.050666]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.050688]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  313.050710]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  313.050733]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.050755]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  313.050776]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  313.050797]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.050821]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  313.050844]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.050869]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  313.050891]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  313.050913]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  313.050934]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  313.050956]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  313.050979]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  313.051001]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  313.051024]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  313.051047]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  313.051070]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  313.051095]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  313.051117]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  313.051139]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  313.051161]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  313.051181]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  313.051202]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  313.051223]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  313.051246]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  313.051265]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  313.051301]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  313.051325]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  313.051347]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  313.051369]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  313.051388]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  313.051408]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  313.051432]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  313.051454]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  313.051476]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  313.051503]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  313.051523] ---[ end trace 2e28eec93b7a8b89 ]---
[  313.053645] ------------[ cut here ]------------
[  313.053684] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  313.053707] Modules linked in:
[  313.053729] Pid: 1523, comm: syslogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  313.053751] Call Trace:
[  313.053762]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  313.053798]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  313.053820]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  313.053845]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  313.053869]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  313.053891]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  313.053913]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  313.053936]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  313.053960]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  313.053985]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  313.054010]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  313.054031]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  313.054053]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  313.054074]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  313.054097]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  313.054120]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  313.054141]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  313.054164]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  313.054186]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[  313.054211]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  313.054234]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  313.054257]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  313.054280]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[  313.054303]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.054325]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  313.054347]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  313.054370]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.054391]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  313.054412]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  313.054433]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.054458]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  313.054481]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.054505]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  313.054527]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  313.054548]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  313.054569]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  313.054591]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  313.054613]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  313.054636]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  313.054659]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  313.054682]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  313.054705]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  313.054730]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  313.054752]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  313.054774]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  313.054795]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  313.054816]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  313.054836]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  313.054858]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  313.054881]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  313.054899]  <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  313.054937]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  313.054961]  [<ffffffff8100864d>] ? xen_force_evtchn_callback+0xd/0x10
[  313.054985]  [<ffffffff81008ff2>] ? check_events+0x12/0x20
[  313.055007]  [<ffffffff81008f99>] ? xen_irq_enable_direct_reloc+0x4/0x4
[  313.055031]  [<ffffffff81749146>] ? ia32_cstar_target+0x16/0x8c
[  313.055049] ---[ end trace 2e28eec93b7a8b8a ]---
[  313.055115] ------------[ cut here ]------------
[  313.055137] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  313.055160] Modules linked in:
[  313.055179] Pid: 1523, comm: syslogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  313.055204] Call Trace:
[  313.055215]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  313.055248]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  313.055271]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  313.055296]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  313.055319]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  313.055342]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  313.055364]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  313.055387]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  313.055411]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  313.055434]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  313.055458]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  313.055479]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  313.055501]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  313.055524]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  313.055547]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  313.055569]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  313.055591]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  313.055615]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  313.055637]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[  313.055662]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  313.055686]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  313.055709]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  313.055732]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[  313.055760]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.055782]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  313.055804]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  313.055828]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.055849]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  313.055870]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  313.055892]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.055917]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  313.055940]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.055965]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  313.055987]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  313.056009]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  313.056030]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  313.056052]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  313.056075]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  313.056098]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  313.056121]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  313.056145]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  313.056168]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  313.056193]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  313.056216]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  313.056238]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  313.056260]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  313.056281]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  313.056302]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  313.056324]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  313.056347]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  313.056366]  <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  313.056402]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  313.056427]  [<ffffffff8100864d>] ? xen_force_evtchn_callback+0xd/0x10
[  313.056450]  [<ffffffff81008ff2>] ? check_events+0x12/0x20
[  313.056473]  [<ffffffff81008f99>] ? xen_irq_enable_direct_reloc+0x4/0x4
[  313.056496]  [<ffffffff81749146>] ? ia32_cstar_target+0x16/0x8c
[  313.056515] ---[ end trace 2e28eec93b7a8b8b ]---
[  313.056578] ------------[ cut here ]------------
[  313.056601] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  313.056624] Modules linked in:
[  313.056641] Pid: 1523, comm: syslogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  313.056664] Call Trace:
[  313.056675]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  313.056708]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  313.056708]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  313.056708]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  313.056708]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  313.056708]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  313.056708]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  313.056708]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  313.056708]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  313.056708]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  313.056708]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  313.056708]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  313.056708]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  313.056708]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  313.056708]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  313.056708]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  313.056708]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  313.056708]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  313.056708]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[  313.056708]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  313.056708]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  313.056708]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  313.056708]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[  313.056708]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.056708]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  313.056708]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  313.056708]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  313.056708]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  313.056708]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  313.056708]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.056708]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  313.056708]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  313.056708]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  313.056708]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  313.056708]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  313.056708]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  313.056708]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  313.056708]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  313.056708]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  313.056708]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  313.056708]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  313.056708]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  313.056708]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  313.056708]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  313.056708]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  313.056708]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  313.056708]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  313.056708]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  313.056708]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  313.056708]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  313.056708]  <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  313.056708]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  313.056708]  [<ffffffff8100864d>] ? xen_force_evtchn_callback+0xd/0x10
[  313.056708]  [<ffffffff81008ff2>] ? check_events+0x12/0x20
[  313.056708]  [<ffffffff81008f99>] ? xen_irq_enable_direct_reloc+0x4/0x4
[  313.056708]  [<ffffffff81749146>] ? ia32_cstar_target+0x16/0x8c
[  313.056708] ---[ end trace 2e28eec93b7a8b8c ]---
[  313.540200] ------------[ cut here ]------------
[  313.540218] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  313.540240] Modules linked in:
[  313.540246] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  313.540252] Call Trace:
[  313.540256]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  313.540269]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  313.540274]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  313.540283]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  313.540290]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  313.540295]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  313.540301]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  313.540308]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  313.540316]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  313.540321]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  313.540327]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  313.540333]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  313.540338]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  313.540344]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  313.540350]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  313.540357]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  313.540363]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  313.540369]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  313.540375]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  313.540381]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  313.540386]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  313.540392]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  313.540398]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  313.540403]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  313.540408]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  313.540414]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  313.540468]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  313.540475]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  313.540483]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  313.540490]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  313.540495]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  313.540502]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  313.540508]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  313.540513]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  313.540523]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  313.540529]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  313.540535]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  313.540541]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  313.540546]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  313.540551]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  313.540560]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  313.540566]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  313.540571]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  313.540590]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  313.540595] ---[ end trace 2e28eec93b7a8b8d ]---
[  314.516817] ------------[ cut here ]------------
[  314.516845] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  314.516858] Modules linked in:
[  314.516870] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  314.516883] Call Trace:
[  314.516890]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  314.516911]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  314.516923]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  314.516937]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  314.516949]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  314.516960]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  314.516972]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  314.516984]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  314.516999]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  314.517011]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  314.517023]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  314.517033]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  314.517044]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  314.517055]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  314.517067]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  314.517079]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  314.517090]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  314.517102]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  314.517114]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  314.517126]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  314.517138]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  314.517150]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  314.517161]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  314.517171]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  314.517181]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  314.517194]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  314.517205]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  314.517218]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  314.517230]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  314.517241]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  314.517252]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  314.517265]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  314.517277]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  314.517286]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  314.517306]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  314.517318]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  314.517329]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  314.517340]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  314.517351]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  314.517361]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  314.517375]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  314.517386]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  314.517397]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  314.517410]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  314.517421] ---[ end trace 2e28eec93b7a8b8e ]---
[  316.466838] ------------[ cut here ]------------
[  316.466867] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  316.466881] Modules linked in:
[  316.466894] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  316.466906] Call Trace:
[  316.466912]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  316.466934]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  316.466946]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  316.466960]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  316.466973]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  316.466984]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  316.466999]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  316.467012]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  316.467025]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  316.467037]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  316.467049]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  316.467061]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  316.467071]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  316.467083]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  316.467096]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  316.467108]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  316.467119]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  316.467131]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  316.467143]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  316.467155]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  316.467166]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  316.467178]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  316.467190]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  316.467200]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  316.467211]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  316.467223]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  316.467235]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  316.467248]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  316.467260]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  316.467270]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  316.467281]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  316.467293]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  316.467304]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  316.467314]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  316.467334]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  316.467347]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  316.467359]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  316.467369]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  316.467380]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  316.467390]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  316.467403]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  316.467415]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  316.467426]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  316.467439]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  316.467449] ---[ end trace 2e28eec93b7a8b8f ]---
[  320.373445] ------------[ cut here ]------------
[  320.373465] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  320.373472] Modules linked in:
[  320.373478] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  320.373489] Call Trace:
[  320.373492]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  320.373505]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  320.373511]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  320.373519]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  320.373526]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  320.373532]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  320.373538]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  320.373544]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  320.373552]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  320.373557]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  320.373564]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  320.373570]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  320.373575]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  320.373581]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  320.373587]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  320.373594]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  320.373600]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  320.373606]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  320.373613]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  320.373619]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  320.373625]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  320.373631]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  320.373640]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  320.373647]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  320.373653]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  320.373659]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  320.373665]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  320.373672]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  320.373679]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  320.373685]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  320.373690]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  320.373697]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  320.373703]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  320.373708]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  320.373718]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  320.373727]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  320.373733]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  320.373739]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  320.373744]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  320.373749]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  320.373758]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  320.373764]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  320.373769]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  320.373776]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  320.373782] ---[ end trace 2e28eec93b7a8b90 ]---
[  328.186900] ------------[ cut here ]------------
[  328.186948] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  328.186974] Modules linked in:
[  328.186996] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  328.187020] Call Trace:
[  328.187033]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  328.187074]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  328.187097]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  328.187124]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  328.187148]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  328.187171]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  328.187193]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  328.187218]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  328.187242]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  328.187268]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  328.187291]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  328.187316]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  328.187342]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  328.187364]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  328.187389]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  328.187412]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  328.187434]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  328.187457]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  328.187481]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  328.187505]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  328.187528]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  328.187551]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  328.187574]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  328.187594]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  328.187615]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  328.187640]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  328.187663]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  328.187688]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  328.187711]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  328.187732]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  328.187753]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  328.187778]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  328.187801]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  328.187820]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  328.187859]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  328.187883]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  328.187905]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  328.187927]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  328.187947]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  328.187967]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  328.187994]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  328.188017]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  328.188040]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  328.188065]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  328.188087] ---[ end trace 2e28eec93b7a8b91 ]---
[  343.786883] ------------[ cut here ]------------
[  343.786934] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  343.786960] Modules linked in:
[  343.786982] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  343.787006] Call Trace:
[  343.787019]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  343.787060]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  343.787083]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  343.787110]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  343.787135]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  343.787157]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  343.787180]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  343.787203]  [<ffffffff81629a77>] neigh_resolve_output+0x127/0x250
[  343.787229]  [<ffffffff816b96ad>] ip_finish_output+0x39d/0x530
[  343.787252]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  343.787277]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  343.787298]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  343.787320]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  343.787342]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  343.787367]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  343.787390]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  343.787412]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  343.787435]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  343.787459]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  343.787482]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  343.787506]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  343.787529]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  343.787550]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  343.787571]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  343.787591]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  343.787615]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  343.787638]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  343.787663]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  343.787686]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  343.787707]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  343.787729]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  343.787752]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  343.787774]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  343.787793]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  343.787830]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  343.787855]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  343.787877]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  343.787898]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  343.787918]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  343.787938]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  343.787964]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  343.787987]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  343.788011]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  343.788035]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  343.788056] ---[ end trace 2e28eec93b7a8b92 ]---
[  369.138108] ------------[ cut here ]------------
[  369.138148] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.138159] Modules linked in:
[  369.138165] Pid: 2169, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.138172] Call Trace:
[  369.138179]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.138186]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.138191]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.138199]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.138205]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.138211]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.138217]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.138223]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[  369.138231]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.138236]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.138242]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.138247]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.138253]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.138258]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.138265]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.138271]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.138277]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.138283]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[  369.138289]  [<ffffffff816d225b>] tcp_push_one+0x2b/0x40
[  369.138295]  [<ffffffff816c2dec>] tcp_sendmsg+0x8dc/0xe20
[  369.138302]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[  369.138307]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[  369.138312]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.138319]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[  369.138326]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[  369.138331]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.138337]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[  369.138376]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[  369.138384]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[  369.138391]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[  369.138396] ---[ end trace 2e28eec93b7a8b93 ]---
[  369.138422] ------------[ cut here ]------------
[  369.138428] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.138435] Modules linked in:
[  369.138439] Pid: 2169, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.138445] Call Trace:
[  369.138450]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.138455]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.138461]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.138468]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.138474]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.138479]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.138485]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.138491]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[  369.138497]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.138503]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.138509]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.138514]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.138520]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.138525]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.138531]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.138537]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.138542]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.138548]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[  369.138554]  [<ffffffff816d229d>] __tcp_push_pending_frames+0x2d/0x90
[  369.138560]  [<ffffffff816c2693>] tcp_sendmsg+0x183/0xe20
[  369.138566]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[  369.138571]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[  369.138577]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.138583]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[  369.138589]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[  369.138594]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.138600]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[  369.138605]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[  369.138611]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[  369.138617]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[  369.138621] ---[ end trace 2e28eec93b7a8b94 ]---
[  369.138648] ------------[ cut here ]------------
[  369.138653] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.138659] Modules linked in:
[  369.138664] Pid: 2169, comm: sshd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.138670] Call Trace:
[  369.138674]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.138680]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.138686]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.138692]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.138698]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.138704]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.138709]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.138715]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[  369.138721]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.138727]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.138733]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.138739]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.138744]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.138750]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.138756]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.138762]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.138767]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.138773]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[  369.138779]  [<ffffffff816d229d>] __tcp_push_pending_frames+0x2d/0x90
[  369.138785]  [<ffffffff816c2693>] tcp_sendmsg+0x183/0xe20
[  369.138791]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[  369.138796]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[  369.138802]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.138808]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[  369.138814]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[  369.138819]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.138825]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[  369.138830]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[  369.138836]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[  369.138842]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[  369.138847] ---[ end trace 2e28eec93b7a8b95 ]---
[  369.571785] ------------[ cut here ]------------
[  369.571811] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.571818] Modules linked in:
[  369.571825] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.571832] Call Trace:
[  369.571835]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.571848]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.571854]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.571863]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.571871]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.571877]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.571883]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.571890]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  369.571898]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.571903]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.571909]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.571915]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.571922]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.571927]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.571934]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.571941]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.571947]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.571953]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  369.571959]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  369.571965]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  369.571971]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  369.571977]  [<ffffffff816cd858>] tcp_rcv_established+0x318/0x640
[  369.571983]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.571989]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  369.571996]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  369.572002]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.572007]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  369.572013]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.572019]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.572026]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  369.572031]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.572037]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  369.572043]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  369.572048]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  369.572054]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  369.572060]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  369.572066]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.572073]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  369.572080]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  369.572086]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  369.572092]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  369.572098]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  369.572105]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  369.572110]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  369.572116]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  369.572124]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  369.572129]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  369.572137]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  369.572143]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  369.572147]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  369.572158]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  369.572165]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  369.572171]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  369.572176]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  369.572186]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  369.572192]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  369.572200]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  369.572206]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  369.572212]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  369.572218]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  369.572224] ---[ end trace 2e28eec93b7a8b96 ]---
[  369.572251] ------------[ cut here ]------------
[  369.572257] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.572264] Modules linked in:
[  369.572268] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.572274] Call Trace:
[  369.572277]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.572286]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.572291]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.572298]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.572304]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.572309]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.572315]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.572321]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  369.572327]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.572333]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.572339]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.572344]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.572350]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.572355]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.572361]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.572370]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.572376]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.572382]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  369.572388]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  369.572394]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  369.572400]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  369.572406]  [<ffffffff816cd858>] tcp_rcv_established+0x318/0x640
[  369.572411]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.572417]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  369.572423]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  369.572429]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.572434]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  369.572439]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.572445]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.572451]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  369.572457]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.572463]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  369.572469]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  369.572474]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  369.572480]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  369.572486]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  369.572492]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.572497]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  369.572503]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  369.572509]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  369.572515]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  369.572521]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  369.572527]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  369.572533]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  369.572538]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  369.572543]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  369.572549]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  369.572555]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  369.572560]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  369.572566]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  369.572575]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  369.572581]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  369.572586]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  369.572592]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  369.572597]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  369.572602]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  369.572608]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  369.572614]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  369.572619]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  369.572626]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  369.572631] ---[ end trace 2e28eec93b7a8b97 ]---
[  369.574579] ------------[ cut here ]------------
[  369.574599] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.574605] Modules linked in:
[  369.574611] Pid: 1523, comm: syslogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.574617] Call Trace:
[  369.574620]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.574629]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.574635]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.574642]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.574648]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.574653]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.574659]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.574665]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  369.574671]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.574677]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.574683]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.574688]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.574694]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.574699]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.574706]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.574712]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.574718]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.574728]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  369.574734]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[  369.574740]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  369.574746]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  369.574752]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  369.574758]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[  369.574763]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.574769]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  369.574775]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  369.574781]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.574786]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  369.574792]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.574797]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.574804]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  369.574810]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.574816]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  369.574821]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  369.574827]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  369.574832]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  369.574838]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  369.574843]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.574850]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  369.574856]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  369.574862]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  369.574867]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  369.574874]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  369.574879]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  369.574885]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  369.574891]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  369.574896]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  369.574901]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  369.574907]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  369.574913]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  369.574918]  <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  369.574928]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  369.574934]  [<ffffffff8100864d>] ? xen_force_evtchn_callback+0xd/0x10
[  369.574940]  [<ffffffff81008ff2>] ? check_events+0x12/0x20
[  369.574946]  [<ffffffff81008fdf>] ? xen_restore_fl_direct_reloc+0x4/0x4
[  369.574952]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.574958]  [<ffffffff810f2a40>] ? find_get_pages+0x190/0x190
[  369.574964]  [<ffffffff810f2a82>] ? find_get_page+0x42/0x100
[  369.574969]  [<ffffffff810f2a40>] ? find_get_pages+0x190/0x190
[  369.574975]  [<ffffffff810f2dc1>] ? find_lock_page+0x21/0x80
[  369.574980]  [<ffffffff810f331a>] ? grab_cache_page_write_begin+0x6a/0xe0
[  369.574988]  [<ffffffff811d6df7>] ? ext4_da_write_begin+0x97/0x1a0
[  369.574993]  [<ffffffff810b054b>] ? __lock_acquire+0x46b/0xdd0
[  369.574999]  [<ffffffff8161034c>] ? skb_free_head+0x4c/0x60
[  369.575005]  [<ffffffff810f1ebb>] ? generic_file_buffered_write+0x10b/0x2b0
[  369.575012]  [<ffffffff8106d742>] ? current_fs_time+0x22/0x30
[  369.575017]  [<ffffffff810f409b>] ? __generic_file_aio_write+0x1bb/0x3c0
[  369.575023]  [<ffffffff810f431c>] ? generic_file_aio_write+0x7c/0xf0
[  369.575029]  [<ffffffff811d04c4>] ? ext4_file_write+0x64/0x4c0
[  369.575036]  [<ffffffff8108eb1d>] ? lg_local_unlock+0x3d/0x70
[  369.575042]  [<ffffffff811d0460>] ? ext4_unwritten_wait+0xc0/0xc0
[  369.575049]  [<ffffffff811434cb>] ? do_sync_readv_writev+0x9b/0xe0
[  369.575056]  [<ffffffff8118d07c>] ? compat_do_readv_writev+0x1cc/0x210
[  369.575062]  [<ffffffff81749219>] ? sysretl_from_sys_call+0x2e/0x38
[  369.575068]  [<ffffffff8118d0f7>] ? compat_writev+0x37/0x70
[  369.575074]  [<ffffffff8118d2b4>] ? compat_sys_writev+0x54/0x90
[  369.575079]  [<ffffffff817491cc>] ? cstar_dispatch+0x7/0x26
[  369.575084] ---[ end trace 2e28eec93b7a8b98 ]---
[  369.575105] ------------[ cut here ]------------
[  369.575111] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.575117] Modules linked in:
[  369.575122] Pid: 1523, comm: syslogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.575128] Call Trace:
[  369.575130]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.575139]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.575145]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.575151]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.575157]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.575166]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.575172]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.575178]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  369.575184]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.575190]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.575196]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.575202]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.575207]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.575213]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.575219]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.575225]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.575230]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.575236]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  369.575242]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[  369.575248]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  369.575255]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  369.575260]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  369.575266]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[  369.575272]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.575278]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  369.575284]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  369.575289]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.575295]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  369.575300]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.575309]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.575316]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  369.575322]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.575328]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  369.575334]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  369.575339]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  369.575344]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  369.575350]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  369.575356]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.575362]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  369.575369]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  369.575375]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  369.575384]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  369.575391]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  369.575397]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  369.575402]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  369.575408]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  369.575413]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  369.575419]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  369.575428]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  369.575434]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  369.575439]  <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  369.575448]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  369.575454]  [<ffffffff8100864d>] ? xen_force_evtchn_callback+0xd/0x10
[  369.575460]  [<ffffffff81008ff2>] ? check_events+0x12/0x20
[  369.575466]  [<ffffffff81008fdf>] ? xen_restore_fl_direct_reloc+0x4/0x4
[  369.575472]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.575478]  [<ffffffff810f2a40>] ? find_get_pages+0x190/0x190
[  369.575485]  [<ffffffff810f2a82>] ? find_get_page+0x42/0x100
[  369.575491]  [<ffffffff810f2a40>] ? find_get_pages+0x190/0x190
[  369.575497]  [<ffffffff810f2dc1>] ? find_lock_page+0x21/0x80
[  369.575502]  [<ffffffff810f331a>] ? grab_cache_page_write_begin+0x6a/0xe0
[  369.575508]  [<ffffffff811d6df7>] ? ext4_da_write_begin+0x97/0x1a0
[  369.575514]  [<ffffffff810b054b>] ? __lock_acquire+0x46b/0xdd0
[  369.575520]  [<ffffffff8161034c>] ? skb_free_head+0x4c/0x60
[  369.575526]  [<ffffffff810f1ebb>] ? generic_file_buffered_write+0x10b/0x2b0
[  369.575532]  [<ffffffff8106d742>] ? current_fs_time+0x22/0x30
[  369.575538]  [<ffffffff810f409b>] ? __generic_file_aio_write+0x1bb/0x3c0
[  369.575544]  [<ffffffff810f431c>] ? generic_file_aio_write+0x7c/0xf0
[  369.575550]  [<ffffffff811d04c4>] ? ext4_file_write+0x64/0x4c0
[  369.575556]  [<ffffffff8108eb1d>] ? lg_local_unlock+0x3d/0x70
[  369.575562]  [<ffffffff811d0460>] ? ext4_unwritten_wait+0xc0/0xc0
[  369.575567]  [<ffffffff811434cb>] ? do_sync_readv_writev+0x9b/0xe0
[  369.575574]  [<ffffffff8118d07c>] ? compat_do_readv_writev+0x1cc/0x210
[  369.575582]  [<ffffffff81749219>] ? sysretl_from_sys_call+0x2e/0x38
[  369.575588]  [<ffffffff8118d0f7>] ? compat_writev+0x37/0x70
[  369.575594]  [<ffffffff8118d2b4>] ? compat_sys_writev+0x54/0x90
[  369.575600]  [<ffffffff817491cc>] ? cstar_dispatch+0x7/0x26
[  369.575604] ---[ end trace 2e28eec93b7a8b99 ]---
[  369.575626] ------------[ cut here ]------------
[  369.575632] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  369.575638] Modules linked in:
[  369.575642] Pid: 1523, comm: syslogd Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  369.575648] Call Trace:
[  369.575651]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  369.575660]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  369.575665]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  369.575672]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  369.575678]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  369.575684]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  369.575690]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  369.575696]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  369.575702]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  369.575707]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  369.575714]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  369.575719]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  369.575725]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  369.575730]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  369.575736]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  369.575742]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  369.575748]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  369.575754]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  369.575759]  [<ffffffff81746cb5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[  369.575766]  [<ffffffff816d167e>] tcp_xmit_retransmit_queue+0x19e/0x300
[  369.575772]  [<ffffffff816c99ff>] tcp_fastretrans_alert+0x94f/0xcb0
[  369.575778]  [<ffffffff816ca70c>] tcp_ack+0x9ac/0x1150
[  369.575783]  [<ffffffff816cd8ce>] tcp_rcv_established+0x38e/0x640
[  369.575789]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.575795]  [<ffffffff816d54d5>] tcp_v4_do_rcv+0x135/0x480
[  369.575803]  [<ffffffff817461d2>] ? _raw_spin_lock_nested+0x42/0x50
[  369.575809]  [<ffffffff816d5eef>] ? tcp_v4_rcv+0x6cf/0xb10
[  369.575815]  [<ffffffff816d617d>] tcp_v4_rcv+0x95d/0xb10
[  369.575823]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.575828]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.575835]  [<ffffffff816b2a6a>] ip_local_deliver_finish+0x11a/0x230
[  369.575841]  [<ffffffff816b2995>] ? ip_local_deliver_finish+0x45/0x230
[  369.575847]  [<ffffffff816b2bb8>] ip_local_deliver+0x38/0x80
[  369.575852]  [<ffffffff816b217a>] ip_rcv_finish+0x15a/0x630
[  369.575858]  [<ffffffff816b2868>] ip_rcv+0x218/0x300
[  369.575863]  [<ffffffff8161ac8d>] __netif_receive_skb+0x65d/0x8d0
[  369.575869]  [<ffffffff8161a775>] ? __netif_receive_skb+0x145/0x8d0
[  369.575875]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[  369.575881]  [<ffffffff810f9973>] ? free_hot_cold_page+0x1b3/0x1e0
[  369.575887]  [<ffffffff8161d1f8>] netif_receive_skb+0x28/0xf0
[  369.575893]  [<ffffffff81612d03>] ? __pskb_pull_tail+0x253/0x340
[  369.576726]  [<ffffffff8146e4c5>] xennet_poll+0xad5/0xe10
[  369.576726]  [<ffffffff8161dfa6>] net_rx_action+0x136/0x260
[  369.576726]  [<ffffffff8106e381>] ? __do_softirq+0x71/0x1a0
[  369.576726]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  369.576726]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  369.576726]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  369.576726]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  369.576726]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  369.576726]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  369.576726]  <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  369.576726]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[  369.576726]  [<ffffffff8100864d>] ? xen_force_evtchn_callback+0xd/0x10
[  369.576726]  [<ffffffff81008ff2>] ? check_events+0x12/0x20
[  369.576726]  [<ffffffff81008fdf>] ? xen_restore_fl_direct_reloc+0x4/0x4
[  369.576726]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[  369.576726]  [<ffffffff810f2a40>] ? find_get_pages+0x190/0x190
[  369.576726]  [<ffffffff810f2a82>] ? find_get_page+0x42/0x100
[  369.576726]  [<ffffffff810f2a40>] ? find_get_pages+0x190/0x190
[  369.576726]  [<ffffffff810f2dc1>] ? find_lock_page+0x21/0x80
[  369.576726]  [<ffffffff810f331a>] ? grab_cache_page_write_begin+0x6a/0xe0
[  369.576726]  [<ffffffff811d6df7>] ? ext4_da_write_begin+0x97/0x1a0
[  369.576726]  [<ffffffff810b054b>] ? __lock_acquire+0x46b/0xdd0
[  369.576726]  [<ffffffff8161034c>] ? skb_free_head+0x4c/0x60
[  369.576726]  [<ffffffff810f1ebb>] ? generic_file_buffered_write+0x10b/0x2b0
[  369.576726]  [<ffffffff8106d742>] ? current_fs_time+0x22/0x30
[  369.576726]  [<ffffffff810f409b>] ? __generic_file_aio_write+0x1bb/0x3c0
[  369.576726]  [<ffffffff810f431c>] ? generic_file_aio_write+0x7c/0xf0
[  369.576726]  [<ffffffff811d04c4>] ? ext4_file_write+0x64/0x4c0
[  369.576726]  [<ffffffff8108eb1d>] ? lg_local_unlock+0x3d/0x70
[  369.576726]  [<ffffffff811d0460>] ? ext4_unwritten_wait+0xc0/0xc0
[  369.576726]  [<ffffffff811434cb>] ? do_sync_readv_writev+0x9b/0xe0
[  369.576726]  [<ffffffff8118d07c>] ? compat_do_readv_writev+0x1cc/0x210
[  369.576726]  [<ffffffff81749219>] ? sysretl_from_sys_call+0x2e/0x38
[  369.576726]  [<ffffffff8118d0f7>] ? compat_writev+0x37/0x70
[  369.576726]  [<ffffffff8118d2b4>] ? compat_sys_writev+0x54/0x90
[  369.576726]  [<ffffffff817491cc>] ? cstar_dispatch+0x7/0x26
[  369.576726] ---[ end trace 2e28eec93b7a8b9a ]---
[  370.046880] ------------[ cut here ]------------
[  370.046904] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  370.046910] Modules linked in:
[  370.046917] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  370.046923] Call Trace:
[  370.046927]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  370.046939]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  370.046945]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  370.046954]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  370.046961]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  370.046966]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  370.046972]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  370.046979]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  370.046986]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  370.046992]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  370.046998]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  370.047003]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  370.047009]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  370.047014]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  370.047020]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  370.047027]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  370.047033]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  370.047039]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  370.047045]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  370.047051]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  370.047057]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  370.047063]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  370.047068]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  370.047073]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  370.047079]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  370.047085]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  370.047234]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  370.047241]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  370.047248]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  370.047254]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  370.047259]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  370.047266]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  370.047272]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  370.047277]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  370.047287]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  370.047293]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  370.047300]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  370.047305]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  370.047311]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  370.047316]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  370.047325]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  370.047331]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  370.047337]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  370.047343]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  370.047349] ---[ end trace 2e28eec93b7a8b9b ]---
[  370.996789] ------------[ cut here ]------------
[  370.996807] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  370.996814] Modules linked in:
[  370.996820] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  370.996826] Call Trace:
[  370.996829]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  370.996840]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  370.996845]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  370.996852]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  370.996858]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  370.996863]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  370.996868]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  370.996875]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  370.996881]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  370.996886]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  370.996892]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  370.996896]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  370.996901]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  370.996906]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  370.996912]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  370.996926]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  370.996941]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  370.996947]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  370.996953]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  370.996959]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  370.996965]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  370.996971]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  370.996977]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  370.996983]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  370.996988]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  370.996998]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  370.997004]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  370.997011]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  370.997017]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  370.997023]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  370.997029]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  370.997036]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  370.997042]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  370.997046]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  370.997057]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  370.997064]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  370.997070]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  370.997075]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  370.997081]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  370.997093]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  370.997102]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  370.997108]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  370.997114]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  370.997120]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  370.997126] ---[ end trace 2e28eec93b7a8b9c ]---
[  372.893570] ------------[ cut here ]------------
[  372.893594] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  372.893601] Modules linked in:
[  372.893607] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  372.893613] Call Trace:
[  372.893617]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  372.893630]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  372.893636]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  372.893644]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  372.893651]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  372.893657]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  372.893663]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  372.893669]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  372.893676]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  372.893682]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  372.893688]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  372.893694]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  372.893699]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  372.893705]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  372.893711]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  372.893718]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  372.893724]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  372.893730]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  372.893737]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  372.893743]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  372.893748]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  372.893754]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  372.893760]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  372.893765]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  372.893771]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  372.893777]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  372.893939]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  372.893946]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  372.893952]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  372.893958]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  372.893963]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  372.893970]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  372.893976]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  372.893980]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  372.893990]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  372.893997]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  372.894003]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  372.894008]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  372.894014]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  372.894019]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  372.894027]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  372.894036]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  372.894042]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  372.894048]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  372.894054] ---[ end trace 2e28eec93b7a8b9d ]---
[  374.986770] ------------[ cut here ]------------
[  374.986795] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  374.986802] Modules linked in:
[  374.986810] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  374.986816] Call Trace:
[  374.986819]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  374.986833]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  374.986838]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  374.986847]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  374.986854]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  374.986859]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  374.986864]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  374.986871]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  374.986879]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  374.986884]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  374.986890]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  374.986894]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  374.986899]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  374.986904]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  374.986911]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  374.986918]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  374.986926]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  374.986931]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  374.986937]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  374.986942]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  374.986947]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  374.986953]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  374.986958]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  374.986963]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  374.986967]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  374.986973]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  374.986978]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  374.986985]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  374.986991]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  374.986997]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  374.987002]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  374.987009]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  374.987014]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  374.987018]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  374.987028]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  374.987039]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  374.987059]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  374.987064]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  374.987070]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  374.987075]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  374.987084]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  374.987091]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  374.987097]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  374.987104]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  374.987110] ---[ end trace 2e28eec93b7a8b9e ]---
[  376.693529] ------------[ cut here ]------------
[  376.693582] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  376.693607] Modules linked in:
[  376.693630] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  376.693654] Call Trace:
[  376.693666]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  376.693708]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  376.693731]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  376.693758]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  376.693783]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  376.693806]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  376.693828]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  376.693852]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  376.693877]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  376.693902]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  376.693926]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  376.693948]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  376.693970]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  376.693991]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  376.694017]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  376.694040]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  376.694062]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  376.694085]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  376.694110]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  376.694133]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  376.694156]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  376.694179]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  376.694201]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  376.694222]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  376.694242]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  376.694267]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  376.694290]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  376.694315]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  376.694337]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  376.694358]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  376.694380]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  376.694404]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  376.694427]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  376.694446]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  376.694484]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  376.694509]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  376.694531]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  376.694552]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  376.694573]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  376.694594]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  376.694621]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  376.694644]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  376.694666]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  376.694691]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  376.694712] ---[ end trace 2e28eec93b7a8b9f ]---
[  384.293592] ------------[ cut here ]------------
[  384.293622] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  384.293629] Modules linked in:
[  384.293637] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  384.293643] Call Trace:
[  384.293647]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  384.293662]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  384.293667]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  384.293676]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  384.293683]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  384.293689]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  384.293696]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  384.293704]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  384.293711]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  384.293717]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  384.293724]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  384.293729]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  384.293735]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  384.293741]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  384.293747]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  384.293754]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  384.293760]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  384.293767]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  384.293775]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  384.293781]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  384.293787]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  384.293793]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  384.293800]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  384.293805]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  384.293810]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  384.293824]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  384.293830]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  384.293838]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  384.293845]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  384.293851]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  384.293857]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  384.293864]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  384.293870]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  384.293875]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  384.293886]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  384.293893]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  384.293900]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  384.293905]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  384.293911]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  384.293916]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  384.293925]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  384.293932]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  384.293938]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  384.293945]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  384.293950] ---[ end trace 2e28eec93b7a8ba0 ]---
[  399.466773] ------------[ cut here ]------------
[  399.466791] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  399.466798] Modules linked in:
[  399.466804] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  399.466809] Call Trace:
[  399.466813]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  399.466824]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  399.466830]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  399.466837]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  399.466843]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  399.466848]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  399.466853]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  399.466859]  [<ffffffff81629a77>] neigh_resolve_output+0x127/0x250
[  399.466865]  [<ffffffff816b96ad>] ip_finish_output+0x39d/0x530
[  399.466871]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  399.466876]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  399.466881]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  399.466886]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  399.466891]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  399.466896]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  399.466903]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  399.466909]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  399.466914]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  399.466919]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  399.466925]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  399.466930]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  399.466935]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  399.466940]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  399.466945]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  399.466949]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  399.466955]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  399.466960]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  399.466966]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  399.466972]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  399.466977]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  399.466982]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  399.466989]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  399.466994]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  399.466998]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  399.467008]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  399.467013]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  399.467019]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  399.467024]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  399.467028]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  399.467033]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  399.467040]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  399.467045]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  399.467050]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  399.467055]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  399.467060] ---[ end trace 2e28eec93b7a8ba1 ]---
[  429.813440] ------------[ cut here ]------------
[  429.813477] WARNING: at drivers/net/xen-netfront.c:465 xennet_start_xmit+0x7fe/0x860()
[  429.813484] Modules linked in:
[  429.813491] Pid: 0, comm: swapper/0 Tainted: G        W    3.6.0pre-rc1-20121011 #1
[  429.813497] Call Trace:
[  429.813501]  <IRQ>  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[  429.813513]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[  429.813519]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[  429.813527]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[  429.813534]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[  429.813539]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[  429.813545]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[  429.813552]  [<ffffffff810b1417>] ? lock_release+0x117/0x250
[  429.813559]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[  429.813565]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[  429.813571]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[  429.813576]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[  429.813582]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[  429.813588]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[  429.813594]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[  429.813600]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[  429.813606]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[  429.813614]  [<ffffffff816d1106>] tcp_retransmit_skb+0x1c6/0x5a0
[  429.813623]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  429.813631]  [<ffffffff816d2f38>] tcp_retransmit_timer+0x358/0x630
[  429.813637]  [<ffffffff816d334d>] tcp_write_timer_handler+0x13d/0x1a0
[  429.813643]  [<ffffffff816d3428>] tcp_write_timer+0x78/0x80
[  429.813649]  [<ffffffff81073f7c>] call_timer_fn+0x7c/0x100
[  429.813654]  [<ffffffff81073f00>] ? cascade+0xa0/0xa0
[  429.813659]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  429.813665]  [<ffffffff816d33b0>] ? tcp_write_timer_handler+0x1a0/0x1a0
[  429.813728]  [<ffffffff81074217>] run_timer_softirq+0x217/0x250
[  429.813735]  [<ffffffff8106e3d9>] __do_softirq+0xc9/0x1a0
[  429.813742]  [<ffffffff81748d3c>] call_softirq+0x1c/0x30
[  429.813749]  [<ffffffff8100edb5>] do_softirq+0x85/0xf0
[  429.813754]  [<ffffffff8106e24e>] irq_exit+0x9e/0xd0
[  429.813762]  [<ffffffff81339f5f>] xen_evtchn_do_upcall+0x2f/0x40
[  429.813767]  [<ffffffff81748d9e>] xen_do_hypervisor_callback+0x1e/0x30
[  429.813772]  <EOI>  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  429.813783]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[  429.813789]  [<ffffffff81008690>] ? xen_safe_halt+0x10/0x20
[  429.813795]  [<ffffffff810160d0>] ? default_idle+0x40/0x90
[  429.813801]  [<ffffffff81016486>] ? cpu_idle+0x96/0xf0
[  429.813807]  [<ffffffff81722d3c>] ? rest_init+0xbc/0xd0
[  429.813813]  [<ffffffff81722c80>] ? csum_partial_copy_generic+0x170/0x170
[  429.813821]  [<ffffffff81ce0df2>] ? start_kernel+0x390/0x39d
[  429.813827]  [<ffffffff81ce0882>] ? kernel_init+0x20b/0x20b
[  429.813833]  [<ffffffff81ce0356>] ? x86_64_start_reservations+0x131/0x136
[  429.813839]  [<ffffffff81ce3d60>] ? xen_start_kernel+0x70d/0x70f
[  429.813845] ---[ end trace 2e28eec93b7a8ba2 ]---

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Eric Dumazet]

On Thu, 2012-10-11 at 12:00 +0200, Sander Eikelenboom wrote:

> Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
> And i seem to hit it, but only in one of the guests at the moment and it triggers quite irregularly.

xennet_make_frags() is able to split the skb->head in multiple page-size
chunks.

It should do the same for fragments

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Thu, 2012-10-11 at 11:05 +0100, Eric Dumazet wrote:
> On Thu, 2012-10-11 at 12:00 +0200, Sander Eikelenboom wrote:
> 
> > Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
> > And i seem to hit it, but only in one of the guests at the moment and it triggers quite irregularly.
> 
> xennet_make_frags() is able to split the skb->head in multiple page-size
> chunks.
> 
> It should do the same for fragments

Right, I just want to be reproduce the issue so I can know I've fixed it
properly ;-)

Ian.

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Sander Eikelenboom]

Thursday, October 11, 2012, 12:14:54 PM, you wrote:

> On Thu, 2012-10-11 at 11:05 +0100, Eric Dumazet wrote:
>> On Thu, 2012-10-11 at 12:00 +0200, Sander Eikelenboom wrote:
>> 
>> > Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
>> > And i seem to hit it, but only in one of the guests at the moment and it triggers quite irregularly.
>> 
>> xennet_make_frags() is able to split the skb->head in multiple page-size
>> chunks.
>> 
>> It should do the same for fragments

> Right, I just want to be reproduce the issue so I can know I've fixed it
> properly ;-)

Trying to scp/sftp files from a guest seems to trigger it for me ..

> Ian.

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[ANNIE LI]

On 2012-10-11 18:14, Ian Campbell wrote:
> On Thu, 2012-10-11 at 11:05 +0100, Eric Dumazet wrote:
>> On Thu, 2012-10-11 at 12:00 +0200, Sander Eikelenboom wrote:
>>
>>> Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
>>> And i seem to hit it, but only in one of the guests at the moment and it triggers quite irregularly.
>> xennet_make_frags() is able to split the skb->head in multiple page-size
>> chunks.
>>
>> It should do the same for fragments
> Right, I just want to be reproduce the issue so I can know I've fixed it
> properly ;-)
Hi Ian,

I can reproduce this BUG_ON when running netperf/netserver test between 
two domus running on the same dom0. The domu and dom0 all use v3.7-rc1.

When I tried to rebase my persistent grant netfront/netback patch on 
latest kernel, netperf/netserver test never succeeded. I did some test 
to find out that v3.6-rc7 works fine, but v3.7-rc1, v3.7-rc2 and 
v3.7-rc4 does not succeed in netperf/netserver test. So I keep my 
persistent grant patch only based on v3.4-rc3 now.

Konrad thought about commit 6a8ed462f16b8455eec5ae00eb6014159a6721f0 in 
v3.7-rc1, and suggested me to test your debug patch in netfront. This 
BUG_ON happens soon after running the netperf/netserver test case.

Thanks
Annie
>
> Ian.
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: compound skb frag pages appearing in start_xmit

[Sander Eikelenboom]

Thursday, November 15, 2012, 3:31:42 AM, you wrote:

> On 2012-10-11 18:14, Ian Campbell wrote:
>> On Thu, 2012-10-11 at 11:05 +0100, Eric Dumazet wrote:
>>> On Thu, 2012-10-11 at 12:00 +0200, Sander Eikelenboom wrote:
>>>
>>>> Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
>>>> And i seem to hit it, but only in one of the guests at the moment and it triggers quite irregularly.
>>> xennet_make_frags() is able to split the skb->head in multiple page-size
>>> chunks.
>>>
>>> It should do the same for fragments
>> Right, I just want to be reproduce the issue so I can know I've fixed it
>> properly ;-)
> Hi Ian,

> I can reproduce this BUG_ON when running netperf/netserver test between 
> two domus running on the same dom0. The domu and dom0 all use v3.7-rc1.

> When I tried to rebase my persistent grant netfront/netback patch on 
> latest kernel, netperf/netserver test never succeeded. I did some test 
> to find out that v3.6-rc7 works fine, but v3.7-rc1, v3.7-rc2 and 
> v3.7-rc4 does not succeed in netperf/netserver test. So I keep my 
> persistent grant patch only based on v3.4-rc3 now.

> Konrad thought about commit 6a8ed462f16b8455eec5ae00eb6014159a6721f0 in 
> v3.7-rc1, and suggested me to test your debug patch in netfront. This 
> BUG_ON happens soon after running the netperf/netserver test case.

> Thanks
> Annie

Is there any progression with this bug (rc6 is out the door, so the release of 3.7-final seems to be eminent and this bug completely cripples any networking with guests) ?

--
Sander

>>
>> Ian.
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Stefan Bader]

[-- Attachment #1: Type: text/plain, Size: 2248 bytes --]

On 19.11.2012 16:43, Sander Eikelenboom wrote:
> 
> Thursday, November 15, 2012, 3:31:42 AM, you wrote:
> 
>> On 2012-10-11 18:14, Ian Campbell wrote:
>>> On Thu, 2012-10-11 at 11:05 +0100, Eric Dumazet wrote:
>>>> On Thu, 2012-10-11 at 12:00 +0200, Sander Eikelenboom wrote:
>>>>
>>>>> Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
>>>>> And i seem to hit it, but only in one of the guests at the moment and it triggers quite irregularly.
>>>> xennet_make_frags() is able to split the skb->head in multiple page-size
>>>> chunks.
>>>>
>>>> It should do the same for fragments
>>> Right, I just want to be reproduce the issue so I can know I've fixed it
>>> properly ;-)
>> Hi Ian,
> 
>> I can reproduce this BUG_ON when running netperf/netserver test between 
>> two domus running on the same dom0. The domu and dom0 all use v3.7-rc1.
> 
>> When I tried to rebase my persistent grant netfront/netback patch on 
>> latest kernel, netperf/netserver test never succeeded. I did some test 
>> to find out that v3.6-rc7 works fine, but v3.7-rc1, v3.7-rc2 and 
>> v3.7-rc4 does not succeed in netperf/netserver test. So I keep my 
>> persistent grant patch only based on v3.4-rc3 now.
> 
>> Konrad thought about commit 6a8ed462f16b8455eec5ae00eb6014159a6721f0 in 
>> v3.7-rc1, and suggested me to test your debug patch in netfront. This 
>> BUG_ON happens soon after running the netperf/netserver test case.
> 
>> Thanks
>> Annie
> 
> Is there any progression with this bug (rc6 is out the door, so the release of 3.7-final seems to be eminent and this bug completely cripples any networking with guests) ?
> 

+1 on that. I was testing yesterday with a PVM domU running 3.7-rc5 on Xen 4.2
(but also reported from EC2 running Xen 3.4.3) c with one VCPU. I actually can
trigger it by just ssh'ing into the domU (from another machine) and then run
"find /". Output starts to stutter and then stops completely. When this happens
a new connection still can be made and as long as only shorter output is
generated the ssh connection is ok. From a dump taken it looks like user-space
is waiting in some select call (without any warnon I rather won't see the tx path).

-Stefan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 897 bytes --]

Re: compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Tue, 2012-11-20 at 08:30 +0000, Stefan Bader wrote:
> >> When I tried to rebase my persistent grant netfront/netback patch on 
> >> latest kernel, netperf/netserver test never succeeded. I did some test 
> >> to find out that v3.6-rc7 works fine, but v3.7-rc1, v3.7-rc2 and 
> >> v3.7-rc4 does not succeed in netperf/netserver test. So I keep my 
> >> persistent grant patch only based on v3.4-rc3 now.
> > 
> >> Konrad thought about commit 6a8ed462f16b8455eec5ae00eb6014159a6721f0 in 
> >> v3.7-rc1, and suggested me to test your debug patch in netfront. This 
> >> BUG_ON happens soon after running the netperf/netserver test case.
> > 
> >> Thanks
> >> Annie
> > 
> > Is there any progression with this bug (rc6 is out the door, so the
> release of 3.7-final seems to be eminent and this bug completely
> cripples any networking with guests) ?
> > 
> 
> +1 on that. I was testing yesterday with a PVM domU running 3.7-rc5 on Xen 4.2
> (but also reported from EC2 running Xen 3.4.3) c with one VCPU. I actually can
> trigger it by just ssh'ing into the domU (from another machine) and then run
> "find /". Output starts to stutter and then stops completely. When this happens
> a new connection still can be made and as long as only shorter output is
> generated the ssh connection is ok. From a dump taken it looks like user-space
> is waiting in some select call (without any warnon I rather won't see the tx path).

Annie, are you still looking into this or shall I?

Ian.

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Ian Campbell]

On Tue, 2012-11-20 at 09:21 +0000, Ian Campbell wrote:
> On Tue, 2012-11-20 at 08:30 +0000, Stefan Bader wrote:
> > >> When I tried to rebase my persistent grant netfront/netback patch on 
> > >> latest kernel, netperf/netserver test never succeeded. I did some test 
> > >> to find out that v3.6-rc7 works fine, but v3.7-rc1, v3.7-rc2 and 
> > >> v3.7-rc4 does not succeed in netperf/netserver test. So I keep my 
> > >> persistent grant patch only based on v3.4-rc3 now.
> > > 
> > >> Konrad thought about commit 6a8ed462f16b8455eec5ae00eb6014159a6721f0 in 
> > >> v3.7-rc1, and suggested me to test your debug patch in netfront. This 
> > >> BUG_ON happens soon after running the netperf/netserver test case.
> > > 
> > >> Thanks
> > >> Annie
> > > 
> > > Is there any progression with this bug (rc6 is out the door, so the
> > release of 3.7-final seems to be eminent and this bug completely
> > cripples any networking with guests) ?
> > > 
> > 
> > +1 on that. I was testing yesterday with a PVM domU running 3.7-rc5 on Xen 4.2
> > (but also reported from EC2 running Xen 3.4.3) c with one VCPU. I actually can
> > trigger it by just ssh'ing into the domU (from another machine) and then run
> > "find /". Output starts to stutter and then stops completely. When this happens
> > a new connection still can be made and as long as only shorter output is
> > generated the ssh connection is ok. From a dump taken it looks like user-space
> > is waiting in some select call (without any warnon I rather won't see the tx path).
> 
> Annie, are you still looking into this or shall I?

I'll assume that silence == No. Will post a patch shortly.

Ian.

[PATCH] xen/netfront: handle compound page fragments on transmit

[Ian Campbell]

An SKB paged fragment can consist of a compound page with order > 0.
However the netchannel protocol deals only in PAGE_SIZE frames.

Handle this in xennet_make_frags by iterating over the frames which
make up the page.

This is the netfront equivalent to 6a8ed462f16b for netback.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: netdev@vger.kernel.org
Cc: xen-devel@lists.xen.org
Cc: Eric Dumazet <edumazet@google.com>
Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>
Cc: ANNIE LI <annie.li@oracle.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
Cc: Stefan Bader <stefan.bader@canonical.com>
---
 drivers/net/xen-netfront.c |   58 +++++++++++++++++++++++++++++++++----------
 1 files changed, 44 insertions(+), 14 deletions(-)

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index caa0110..a12b99a 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -452,24 +452,54 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
 	/* Grant backend access to each skb fragment page. */
 	for (i = 0; i < frags; i++) {
 		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
+		struct page *page = skb_frag_page(frag);
+		unsigned long size = skb_frag_size(frag);
+		unsigned long offset = frag->page_offset;
 
-		tx->flags |= XEN_NETTXF_more_data;
+		/* Data must not cross a page boundary. */
+		BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
 
-		id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
-		np->tx_skbs[id].skb = skb_get(skb);
-		tx = RING_GET_REQUEST(&np->tx, prod++);
-		tx->id = id;
-		ref = gnttab_claim_grant_reference(&np->gref_tx_head);
-		BUG_ON((signed short)ref < 0);
+		/* Skip unused frames from start of page */
+		page += offset >> PAGE_SHIFT;
+		offset &= ~PAGE_MASK;
 
-		mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
-		gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
-						mfn, GNTMAP_readonly);
+		while (size > 0) {
+			unsigned long bytes;
 
-		tx->gref = np->grant_tx_ref[id] = ref;
-		tx->offset = frag->page_offset;
-		tx->size = skb_frag_size(frag);
-		tx->flags = 0;
+			BUG_ON(offset >= PAGE_SIZE);
+
+			bytes = PAGE_SIZE - offset;
+			if (bytes > size)
+				bytes = size;
+
+			tx->flags |= XEN_NETTXF_more_data;
+
+			id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
+			np->tx_skbs[id].skb = skb_get(skb);
+			tx = RING_GET_REQUEST(&np->tx, prod++);
+			tx->id = id;
+			ref = gnttab_claim_grant_reference(&np->gref_tx_head);
+			BUG_ON((signed short)ref < 0);
+
+			mfn = pfn_to_mfn(page_to_pfn(page));
+			gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
+							mfn, GNTMAP_readonly);
+
+			tx->gref = np->grant_tx_ref[id] = ref;
+			tx->offset = offset;
+			tx->size = bytes;
+			tx->flags = 0;
+
+			offset += bytes;
+			size -= bytes;
+
+			/* Next frame */
+			if (offset == PAGE_SIZE && size) {
+				BUG_ON(!PageCompound(page));
+				page++;
+				offset = 0;
+			}
+		}
 	}
 
 	np->tx.req_prod_pvt = prod;
-- 
1.7.2.5

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Jan Beulich]

>>> On 20.11.12 at 12:40, Ian Campbell <ian.campbell@citrix.com> wrote:
> An SKB paged fragment can consist of a compound page with order > 0.
> However the netchannel protocol deals only in PAGE_SIZE frames.
> 
> Handle this in xennet_make_frags by iterating over the frames which
> make up the page.
> 
> This is the netfront equivalent to 6a8ed462f16b for netback.

Wouldn't you need to be at least a little more conservative here
with respect to resource use: I realize that get_id_from_freelist()
return values were never checked, and failure of
gnttab_claim_grant_reference() was always dealt with via
BUG_ON(), but considering that netfront_tx_slot_available()
doesn't account for compound page fragments, I think this (lack
of) error handling needs improvement in the course of the
change here (regardless of - I think - someone having said that
usually the sum of all pages referenced from an skb's fragments
would not exceed MAX_SKB_FRAGS - "usually" just isn't enough
imo).

Jan

> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: netdev@vger.kernel.org 
> Cc: xen-devel@lists.xen.org 
> Cc: Eric Dumazet <edumazet@google.com>
> Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>
> Cc: ANNIE LI <annie.li@oracle.com>
> Cc: Sander Eikelenboom <linux@eikelenboom.it>
> Cc: Stefan Bader <stefan.bader@canonical.com>
> ---
>  drivers/net/xen-netfront.c |   58 +++++++++++++++++++++++++++++++++----------
>  1 files changed, 44 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index caa0110..a12b99a 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -452,24 +452,54 @@ static void xennet_make_frags(struct sk_buff *skb, 
> struct net_device *dev,
>  	/* Grant backend access to each skb fragment page. */
>  	for (i = 0; i < frags; i++) {
>  		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
> +		struct page *page = skb_frag_page(frag);
> +		unsigned long size = skb_frag_size(frag);
> +		unsigned long offset = frag->page_offset;
>  
> -		tx->flags |= XEN_NETTXF_more_data;
> +		/* Data must not cross a page boundary. */
> +		BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
>  
> -		id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> -		np->tx_skbs[id].skb = skb_get(skb);
> -		tx = RING_GET_REQUEST(&np->tx, prod++);
> -		tx->id = id;
> -		ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> -		BUG_ON((signed short)ref < 0);
> +		/* Skip unused frames from start of page */
> +		page += offset >> PAGE_SHIFT;
> +		offset &= ~PAGE_MASK;
>  
> -		mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
> -		gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> -						mfn, GNTMAP_readonly);
> +		while (size > 0) {
> +			unsigned long bytes;
>  
> -		tx->gref = np->grant_tx_ref[id] = ref;
> -		tx->offset = frag->page_offset;
> -		tx->size = skb_frag_size(frag);
> -		tx->flags = 0;
> +			BUG_ON(offset >= PAGE_SIZE);
> +
> +			bytes = PAGE_SIZE - offset;
> +			if (bytes > size)
> +				bytes = size;
> +
> +			tx->flags |= XEN_NETTXF_more_data;
> +
> +			id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> +			np->tx_skbs[id].skb = skb_get(skb);
> +			tx = RING_GET_REQUEST(&np->tx, prod++);
> +			tx->id = id;
> +			ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> +			BUG_ON((signed short)ref < 0);
> +
> +			mfn = pfn_to_mfn(page_to_pfn(page));
> +			gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> +							mfn, GNTMAP_readonly);
> +
> +			tx->gref = np->grant_tx_ref[id] = ref;
> +			tx->offset = offset;
> +			tx->size = bytes;
> +			tx->flags = 0;
> +
> +			offset += bytes;
> +			size -= bytes;
> +
> +			/* Next frame */
> +			if (offset == PAGE_SIZE && size) {
> +				BUG_ON(!PageCompound(page));
> +				page++;
> +				offset = 0;
> +			}
> +		}
>  	}
>  
>  	np->tx.req_prod_pvt = prod;
> -- 
> 1.7.2.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 

Re: [PATCH] xen/netfront: handle compound page fragments on transmit

[Stefan Bader]

[-- Attachment #1: Type: text/plain, Size: 3619 bytes --]

Aside from Jans comments about error handling, I tried below patch and it seems
to solve the problem with transfers out of the domU for me (though only shallow
testing done, otoh 5 times is more than getting stuck the first time).

-Stefan

On 20.11.2012 12:40, Ian Campbell wrote:
> An SKB paged fragment can consist of a compound page with order > 0.
> However the netchannel protocol deals only in PAGE_SIZE frames.
> 
> Handle this in xennet_make_frags by iterating over the frames which
> make up the page.
> 
> This is the netfront equivalent to 6a8ed462f16b for netback.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: netdev@vger.kernel.org
> Cc: xen-devel@lists.xen.org
> Cc: Eric Dumazet <edumazet@google.com>
> Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>
> Cc: ANNIE LI <annie.li@oracle.com>
> Cc: Sander Eikelenboom <linux@eikelenboom.it>
> Cc: Stefan Bader <stefan.bader@canonical.com>
Tested-by: Stefan Bader <stefan.bader@canonical.com>
> ---
>  drivers/net/xen-netfront.c |   58 +++++++++++++++++++++++++++++++++----------
>  1 files changed, 44 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index caa0110..a12b99a 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -452,24 +452,54 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
>  	/* Grant backend access to each skb fragment page. */
>  	for (i = 0; i < frags; i++) {
>  		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
> +		struct page *page = skb_frag_page(frag);
> +		unsigned long size = skb_frag_size(frag);
> +		unsigned long offset = frag->page_offset;
>  
> -		tx->flags |= XEN_NETTXF_more_data;
> +		/* Data must not cross a page boundary. */
> +		BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
>  
> -		id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> -		np->tx_skbs[id].skb = skb_get(skb);
> -		tx = RING_GET_REQUEST(&np->tx, prod++);
> -		tx->id = id;
> -		ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> -		BUG_ON((signed short)ref < 0);
> +		/* Skip unused frames from start of page */
> +		page += offset >> PAGE_SHIFT;
> +		offset &= ~PAGE_MASK;
>  
> -		mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
> -		gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> -						mfn, GNTMAP_readonly);
> +		while (size > 0) {
> +			unsigned long bytes;
>  
> -		tx->gref = np->grant_tx_ref[id] = ref;
> -		tx->offset = frag->page_offset;
> -		tx->size = skb_frag_size(frag);
> -		tx->flags = 0;
> +			BUG_ON(offset >= PAGE_SIZE);
> +
> +			bytes = PAGE_SIZE - offset;
> +			if (bytes > size)
> +				bytes = size;
> +
> +			tx->flags |= XEN_NETTXF_more_data;
> +
> +			id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> +			np->tx_skbs[id].skb = skb_get(skb);
> +			tx = RING_GET_REQUEST(&np->tx, prod++);
> +			tx->id = id;
> +			ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> +			BUG_ON((signed short)ref < 0);
> +
> +			mfn = pfn_to_mfn(page_to_pfn(page));
> +			gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> +							mfn, GNTMAP_readonly);
> +
> +			tx->gref = np->grant_tx_ref[id] = ref;
> +			tx->offset = offset;
> +			tx->size = bytes;
> +			tx->flags = 0;
> +
> +			offset += bytes;
> +			size -= bytes;
> +
> +			/* Next frame */
> +			if (offset == PAGE_SIZE && size) {
> +				BUG_ON(!PageCompound(page));
> +				page++;
> +				offset = 0;
> +			}
> +		}
>  	}
>  
>  	np->tx.req_prod_pvt = prod;
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 897 bytes --]

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Ian Campbell]

On Tue, 2012-11-20 at 12:28 +0000, Jan Beulich wrote:
> >>> On 20.11.12 at 12:40, Ian Campbell <ian.campbell@citrix.com> wrote:
> > An SKB paged fragment can consist of a compound page with order > 0.
> > However the netchannel protocol deals only in PAGE_SIZE frames.
> > 
> > Handle this in xennet_make_frags by iterating over the frames which
> > make up the page.
> > 
> > This is the netfront equivalent to 6a8ed462f16b for netback.
> 
> Wouldn't you need to be at least a little more conservative here
> with respect to resource use: I realize that get_id_from_freelist()
> return values were never checked, and failure of
> gnttab_claim_grant_reference() was always dealt with via
> BUG_ON(), but considering that netfront_tx_slot_available()
> doesn't account for compound page fragments, I think this (lack
> of) error handling needs improvement in the course of the
> change here (regardless of - I think - someone having said that
> usually the sum of all pages referenced from an skb's fragments
> would not exceed MAX_SKB_FRAGS - "usually" just isn't enough
> imo).

I think it is more than "usually", it is derived from the number of
pages needed to contain 64K of data which is the maximum size of the
data associated with an skb (AIUI).

Unwinding from failure in xennet_make_frags looks pretty tricky, but how
about this incremental patch:

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index a12b99a..06d0a84 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -505,6 +505,46 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
 	np->tx.req_prod_pvt = prod;
 }
 
+/*
+ * Count how many ring slots are required to send the frags of this
+ * skb. Each frag might be a compound page.
+ */
+static int xennet_count_skb_frag_pages(struct sk_buff *skb)
+{
+	int i, frags = skb_shinfo(skb)->nr_frags;
+	int pages = 0;
+
+	for (i = 0; i < frags; i++) {
+		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
+		unsigned long size = skb_frag_size(frag);
+		unsigned long offset = frag->page_offset;
+
+		/* Skip unused frames from start of page */
+		offset &= ~PAGE_MASK;
+
+		while (size > 0) {
+			unsigned long bytes;
+
+			BUG_ON(offset >= PAGE_SIZE);
+
+			bytes = PAGE_SIZE - offset;
+			if (bytes > size)
+				bytes = size;
+
+			offset += bytes;
+			size -= bytes;
+
+			/* Next frame */
+			if (offset == PAGE_SIZE && size) {
+				pages++;
+				offset = 0;
+			}
+		}
+	}
+
+	return pages;
+}
+
 static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
 {
 	unsigned short id;
@@ -517,12 +557,13 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	grant_ref_t ref;
 	unsigned long mfn;
 	int notify;
-	int frags = skb_shinfo(skb)->nr_frags;
+	int frags;
 	unsigned int offset = offset_in_page(data);
 	unsigned int len = skb_headlen(skb);
 	unsigned long flags;
 
-	frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
+	frags = xennet_count_skb_frag_pages(skb) +
+		DIV_ROUND_UP(offset + len, PAGE_SIZE);
 	if (unlikely(frags > MAX_SKB_FRAGS + 1)) {
 		printk(KERN_ALERT "xennet: skb rides the rocket: %d frags\n",
 		       frags);

Re: [PATCH] xen/netfront: handle compound page fragments on transmit

[Sander Eikelenboom]

Tuesday, November 20, 2012, 2:30:54 PM, you wrote:

> Aside from Jans comments about error handling, I tried below patch and it seems
> to solve the problem with transfers out of the domU for me (though only shallow
> testing done, otoh 5 times is more than getting stuck the first time).

> -Stefan

I'm running with this patch now, it seems to fix the problems for me as well.

--
Sander

> On 20.11.2012 12:40, Ian Campbell wrote:
>> An SKB paged fragment can consist of a compound page with order > 0.
>> However the netchannel protocol deals only in PAGE_SIZE frames.
>> 
>> Handle this in xennet_make_frags by iterating over the frames which
>> make up the page.
>> 
>> This is the netfront equivalent to 6a8ed462f16b for netback.
>> 
>> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
>> Cc: netdev@vger.kernel.org
>> Cc: xen-devel@lists.xen.org
>> Cc: Eric Dumazet <edumazet@google.com>
>> Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>
>> Cc: ANNIE LI <annie.li@oracle.com>
>> Cc: Sander Eikelenboom <linux@eikelenboom.it>
>> Cc: Stefan Bader <stefan.bader@canonical.com>
> Tested-by: Stefan Bader <stefan.bader@canonical.com>
>> ---
>>  drivers/net/xen-netfront.c |   58 +++++++++++++++++++++++++++++++++----------
>>  1 files changed, 44 insertions(+), 14 deletions(-)
>> 
>> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
>> index caa0110..a12b99a 100644
>> --- a/drivers/net/xen-netfront.c
>> +++ b/drivers/net/xen-netfront.c
>> @@ -452,24 +452,54 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
>>       /* Grant backend access to each skb fragment page. */
>>       for (i = 0; i < frags; i++) {
>>               skb_frag_t *frag = skb_shinfo(skb)->frags + i;
>> +             struct page *page = skb_frag_page(frag);
>> +             unsigned long size = skb_frag_size(frag);
>> +             unsigned long offset = frag->page_offset;
>>  
>> -             tx->flags |= XEN_NETTXF_more_data;
>> +             /* Data must not cross a page boundary. */
>> +             BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
>>  
>> -             id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
>> -             np->tx_skbs[id].skb = skb_get(skb);
>> -             tx = RING_GET_REQUEST(&np->tx, prod++);
>> -             tx->id = id;
>> -             ref = gnttab_claim_grant_reference(&np->gref_tx_head);
>> -             BUG_ON((signed short)ref < 0);
>> +             /* Skip unused frames from start of page */
>> +             page += offset >> PAGE_SHIFT;
>> +             offset &= ~PAGE_MASK;
>>  
>> -             mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
>> -             gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
>> -                                             mfn, GNTMAP_readonly);
>> +             while (size > 0) {
>> +                     unsigned long bytes;
>>  
>> -             tx->gref = np->grant_tx_ref[id] = ref;
>> -             tx->offset = frag->page_offset;
>> -             tx->size = skb_frag_size(frag);
>> -             tx->flags = 0;
>> +                     BUG_ON(offset >= PAGE_SIZE);
>> +
>> +                     bytes = PAGE_SIZE - offset;
>> +                     if (bytes > size)
>> +                             bytes = size;
>> +
>> +                     tx->flags |= XEN_NETTXF_more_data;
>> +
>> +                     id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
>> +                     np->tx_skbs[id].skb = skb_get(skb);
>> +                     tx = RING_GET_REQUEST(&np->tx, prod++);
>> +                     tx->id = id;
>> +                     ref = gnttab_claim_grant_reference(&np->gref_tx_head);
>> +                     BUG_ON((signed short)ref < 0);
>> +
>> +                     mfn = pfn_to_mfn(page_to_pfn(page));
>> +                     gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
>> +                                                     mfn, GNTMAP_readonly);
>> +
>> +                     tx->gref = np->grant_tx_ref[id] = ref;
>> +                     tx->offset = offset;
>> +                     tx->size = bytes;
>> +                     tx->flags = 0;
>> +
>> +                     offset += bytes;
>> +                     size -= bytes;
>> +
>> +                     /* Next frame */
>> +                     if (offset == PAGE_SIZE && size) {
>> +                             BUG_ON(!PageCompound(page));
>> +                             page++;
>> +                             offset = 0;
>> +                     }
>> +             }
>>       }
>>  
>>       np->tx.req_prod_pvt = prod;
>> 

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Jan Beulich]

>>> On 20.11.12 at 14:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2012-11-20 at 12:28 +0000, Jan Beulich wrote:
>> >>> On 20.11.12 at 12:40, Ian Campbell <ian.campbell@citrix.com> wrote:
>> > An SKB paged fragment can consist of a compound page with order > 0.
>> > However the netchannel protocol deals only in PAGE_SIZE frames.
>> > 
>> > Handle this in xennet_make_frags by iterating over the frames which
>> > make up the page.
>> > 
>> > This is the netfront equivalent to 6a8ed462f16b for netback.
>> 
>> Wouldn't you need to be at least a little more conservative here
>> with respect to resource use: I realize that get_id_from_freelist()
>> return values were never checked, and failure of
>> gnttab_claim_grant_reference() was always dealt with via
>> BUG_ON(), but considering that netfront_tx_slot_available()
>> doesn't account for compound page fragments, I think this (lack
>> of) error handling needs improvement in the course of the
>> change here (regardless of - I think - someone having said that
>> usually the sum of all pages referenced from an skb's fragments
>> would not exceed MAX_SKB_FRAGS - "usually" just isn't enough
>> imo).
> 
> I think it is more than "usually", it is derived from the number of
> pages needed to contain 64K of data which is the maximum size of the
> data associated with an skb (AIUI).
> 
> Unwinding from failure in xennet_make_frags looks pretty tricky,

Yes, I agree.

> but how about this incremental patch:

Looks good, but can probably be simplified quite a bit:

> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -505,6 +505,46 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
>  	np->tx.req_prod_pvt = prod;
>  }
>  
> +/*
> + * Count how many ring slots are required to send the frags of this
> + * skb. Each frag might be a compound page.
> + */
> +static int xennet_count_skb_frag_pages(struct sk_buff *skb)
> +{
> +	int i, frags = skb_shinfo(skb)->nr_frags;
> +	int pages = 0;
> +
> +	for (i = 0; i < frags; i++) {
> +		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
> +		unsigned long size = skb_frag_size(frag);
> +		unsigned long offset = frag->page_offset;
> +
> +		/* Skip unused frames from start of page */
> +		offset &= ~PAGE_MASK;
> +
> +		while (size > 0) {
> +			unsigned long bytes;
> +
> +			BUG_ON(offset >= PAGE_SIZE);
> +
> +			bytes = PAGE_SIZE - offset;
> +			if (bytes > size)
> +				bytes = size;
> +
> +			offset += bytes;
> +			size -= bytes;
> +
> +			/* Next frame */
> +			if (offset == PAGE_SIZE && size) {
> +				pages++;
> +				offset = 0;
> +			}
> +		}

Isn't the whole loop equivalent to 

		pages = PFN_UP(offset + size);

(at least as long as size is not zero)?

Plus I think the increment of pages would need to be pulled out
of the if() body.

> +	}
> +
> +	return pages;
> +}
> +
>  static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
>  {
>  	unsigned short id;
> @@ -517,12 +557,13 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
>  	grant_ref_t ref;
>  	unsigned long mfn;
>  	int notify;
> -	int frags = skb_shinfo(skb)->nr_frags;
> +	int frags;
>  	unsigned int offset = offset_in_page(data);
>  	unsigned int len = skb_headlen(skb);
>  	unsigned long flags;
>  
> -	frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
> +	frags = xennet_count_skb_frag_pages(skb) +
> +		DIV_ROUND_UP(offset + len, PAGE_SIZE);
>  	if (unlikely(frags > MAX_SKB_FRAGS + 1)) {

This condition would now need adjustment, though (because
"frags" is no longer what its name says).

Jan

>  		printk(KERN_ALERT "xennet: skb rides the rocket: %d frags\n",
>  		       frags);

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Ian Campbell]

On Tue, 2012-11-20 at 13:51 +0000, Jan Beulich wrote:
> >>> On 20.11.12 at 14:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Tue, 2012-11-20 at 12:28 +0000, Jan Beulich wrote:
> >> >>> On 20.11.12 at 12:40, Ian Campbell <ian.campbell@citrix.com> wrote:
> >> > An SKB paged fragment can consist of a compound page with order > 0.
> >> > However the netchannel protocol deals only in PAGE_SIZE frames.
> >> > 
> >> > Handle this in xennet_make_frags by iterating over the frames which
> >> > make up the page.
> >> > 
> >> > This is the netfront equivalent to 6a8ed462f16b for netback.
> >> 
> >> Wouldn't you need to be at least a little more conservative here
> >> with respect to resource use: I realize that get_id_from_freelist()
> >> return values were never checked, and failure of
> >> gnttab_claim_grant_reference() was always dealt with via
> >> BUG_ON(), but considering that netfront_tx_slot_available()
> >> doesn't account for compound page fragments, I think this (lack
> >> of) error handling needs improvement in the course of the
> >> change here (regardless of - I think - someone having said that
> >> usually the sum of all pages referenced from an skb's fragments
> >> would not exceed MAX_SKB_FRAGS - "usually" just isn't enough
> >> imo).
> > 
> > I think it is more than "usually", it is derived from the number of
> > pages needed to contain 64K of data which is the maximum size of the
> > data associated with an skb (AIUI).
> > 
> > Unwinding from failure in xennet_make_frags looks pretty tricky,
> 
> Yes, I agree.
> 
> > but how about this incremental patch:
> 
> Looks good, but can probably be simplified quite a bit:
> 
> > --- a/drivers/net/xen-netfront.c
> > +++ b/drivers/net/xen-netfront.c
> > @@ -505,6 +505,46 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
> >  	np->tx.req_prod_pvt = prod;
> >  }
> >  
> > +/*
> > + * Count how many ring slots are required to send the frags of this
> > + * skb. Each frag might be a compound page.
> > + */
> > +static int xennet_count_skb_frag_pages(struct sk_buff *skb)
> > +{
> > +	int i, frags = skb_shinfo(skb)->nr_frags;
> > +	int pages = 0;
> > +
> > +	for (i = 0; i < frags; i++) {
> > +		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
> > +		unsigned long size = skb_frag_size(frag);
> > +		unsigned long offset = frag->page_offset;
> > +
> > +		/* Skip unused frames from start of page */
> > +		offset &= ~PAGE_MASK;
> > +
> > +		while (size > 0) {
> > +			unsigned long bytes;
> > +
> > +			BUG_ON(offset >= PAGE_SIZE);
> > +
> > +			bytes = PAGE_SIZE - offset;
> > +			if (bytes > size)
> > +				bytes = size;
> > +
> > +			offset += bytes;
> > +			size -= bytes;
> > +
> > +			/* Next frame */
> > +			if (offset == PAGE_SIZE && size) {
> > +				pages++;
> > +				offset = 0;
> > +			}
> > +		}
> 
> Isn't the whole loop equivalent to 
> 
> 		pages = PFN_UP(offset + size);
> 
> (at least as long as size is not zero)?

Er, yes. Wood for the trees etc...

I think using PFN_UP overcounts a bit since the data needed start in the
first frame of a compound frame, but if you keep the 
        /* Skip unused frames from start of page */
        offset &= ~PAGE_MASK;
        
I think that does the right thing

> > @@ -517,12 +557,13 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
> >  	grant_ref_t ref;
> >  	unsigned long mfn;
> >  	int notify;
> > -	int frags = skb_shinfo(skb)->nr_frags;
> > +	int frags;
> >  	unsigned int offset = offset_in_page(data);
> >  	unsigned int len = skb_headlen(skb);
> >  	unsigned long flags;
> >  
> > -	frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
> > +	frags = xennet_count_skb_frag_pages(skb) +
> > +		DIV_ROUND_UP(offset + len, PAGE_SIZE);
> >  	if (unlikely(frags > MAX_SKB_FRAGS + 1)) {
> 
> This condition would now need adjustment, though (because
> "frags" is no longer what its name says).

I think it already wasn't what the name says, since it included the skb
head too. Perhaps "slots" would be a better name?

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index a12b99a..b744875 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -505,6 +505,29 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
 	np->tx.req_prod_pvt = prod;
 }
 
+/*
+ * Count how many ring slots are required to send the frags of this
+ * skb. Each frag might be a compound page.
+ */
+static int xennet_count_skb_frag_slots(struct sk_buff *skb)
+{
+	int i, frags = skb_shinfo(skb)->nr_frags;
+	int pages = 0;
+
+	for (i = 0; i < frags; i++) {
+		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
+		unsigned long size = skb_frag_size(frag);
+		unsigned long offset = frag->page_offset;
+
+		/* Skip unused frames from start of page */
+		offset &= ~PAGE_MASK;
+
+		pages += PFN_UP(offset + size);
+	}
+
+	return pages;
+}
+
 static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
 {
 	unsigned short id;
@@ -517,15 +540,16 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	grant_ref_t ref;
 	unsigned long mfn;
 	int notify;
-	int frags = skb_shinfo(skb)->nr_frags;
+	int slots;
 	unsigned int offset = offset_in_page(data);
 	unsigned int len = skb_headlen(skb);
 	unsigned long flags;
 
-	frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
-	if (unlikely(frags > MAX_SKB_FRAGS + 1)) {
-		printk(KERN_ALERT "xennet: skb rides the rocket: %d frags\n",
-		       frags);
+	slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
+		xennet_count_skb_frag_slots(skb);
+	if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
+		printk(KERN_ALERT "xennet: skb rides the rocket: %d slots\n",
+		       slots);
 		dump_stack();
 		goto drop;
 	}
@@ -533,7 +557,7 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	spin_lock_irqsave(&np->tx_lock, flags);
 
 	if (unlikely(!netif_carrier_ok(dev) ||
-		     (frags > 1 && !xennet_can_sg(dev)) ||
+		     (slots > 1 && !xennet_can_sg(dev)) ||
 		     netif_needs_gso(skb, netif_skb_features(skb)))) {
 		spin_unlock_irqrestore(&np->tx_lock, flags);
 		goto drop;

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Jan Beulich]

>>> On 20.11.12 at 15:14, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2012-11-20 at 13:51 +0000, Jan Beulich wrote:
>> >>> On 20.11.12 at 14:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Tue, 2012-11-20 at 12:28 +0000, Jan Beulich wrote:
>> >> >>> On 20.11.12 at 12:40, Ian Campbell <ian.campbell@citrix.com> wrote:
>> >> > An SKB paged fragment can consist of a compound page with order > 0.
>> >> > However the netchannel protocol deals only in PAGE_SIZE frames.
>> >> > 
>> >> > Handle this in xennet_make_frags by iterating over the frames which
>> >> > make up the page.
>> >> > 
>> >> > This is the netfront equivalent to 6a8ed462f16b for netback.
>> >> 
>> >> Wouldn't you need to be at least a little more conservative here
>> >> with respect to resource use: I realize that get_id_from_freelist()
>> >> return values were never checked, and failure of
>> >> gnttab_claim_grant_reference() was always dealt with via
>> >> BUG_ON(), but considering that netfront_tx_slot_available()
>> >> doesn't account for compound page fragments, I think this (lack
>> >> of) error handling needs improvement in the course of the
>> >> change here (regardless of - I think - someone having said that
>> >> usually the sum of all pages referenced from an skb's fragments
>> >> would not exceed MAX_SKB_FRAGS - "usually" just isn't enough
>> >> imo).
>> > 
>> > I think it is more than "usually", it is derived from the number of
>> > pages needed to contain 64K of data which is the maximum size of the
>> > data associated with an skb (AIUI).
>> > 
>> > Unwinding from failure in xennet_make_frags looks pretty tricky,
>> 
>> Yes, I agree.
>> 
>> > but how about this incremental patch:
>> 
>> Looks good, but can probably be simplified quite a bit:
>> 
>> > --- a/drivers/net/xen-netfront.c
>> > +++ b/drivers/net/xen-netfront.c
>> > @@ -505,6 +505,46 @@ static void xennet_make_frags(struct sk_buff *skb, 
> struct net_device *dev,
>> >  	np->tx.req_prod_pvt = prod;
>> >  }
>> >  
>> > +/*
>> > + * Count how many ring slots are required to send the frags of this
>> > + * skb. Each frag might be a compound page.
>> > + */
>> > +static int xennet_count_skb_frag_pages(struct sk_buff *skb)
>> > +{
>> > +	int i, frags = skb_shinfo(skb)->nr_frags;
>> > +	int pages = 0;
>> > +
>> > +	for (i = 0; i < frags; i++) {
>> > +		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
>> > +		unsigned long size = skb_frag_size(frag);
>> > +		unsigned long offset = frag->page_offset;
>> > +
>> > +		/* Skip unused frames from start of page */
>> > +		offset &= ~PAGE_MASK;
>> > +
>> > +		while (size > 0) {
>> > +			unsigned long bytes;
>> > +
>> > +			BUG_ON(offset >= PAGE_SIZE);
>> > +
>> > +			bytes = PAGE_SIZE - offset;
>> > +			if (bytes > size)
>> > +				bytes = size;
>> > +
>> > +			offset += bytes;
>> > +			size -= bytes;
>> > +
>> > +			/* Next frame */
>> > +			if (offset == PAGE_SIZE && size) {
>> > +				pages++;
>> > +				offset = 0;
>> > +			}
>> > +		}
>> 
>> Isn't the whole loop equivalent to 
>> 
>> 		pages = PFN_UP(offset + size);
>> 
>> (at least as long as size is not zero)?
> 
> Er, yes. Wood for the trees etc...
> 
> I think using PFN_UP overcounts a bit since the data needed start in the
> first frame of a compound frame, but if you keep the 
>         /* Skip unused frames from start of page */
>         offset &= ~PAGE_MASK;
>         
> I think that does the right thing

Right, that's what I said (I only wanted the loop to be replaced, not
what was prior to it).

> @@ -517,15 +540,16 @@ static int xennet_start_xmit(struct sk_buff *skb, 
> struct net_device *dev)
>  	grant_ref_t ref;
>  	unsigned long mfn;
>  	int notify;
> -	int frags = skb_shinfo(skb)->nr_frags;
> +	int slots;
>  	unsigned int offset = offset_in_page(data);
>  	unsigned int len = skb_headlen(skb);
>  	unsigned long flags;
>  
> -	frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
> -	if (unlikely(frags > MAX_SKB_FRAGS + 1)) {
> -		printk(KERN_ALERT "xennet: skb rides the rocket: %d frags\n",
> -		       frags);
> +	slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
> +		xennet_count_skb_frag_slots(skb);
> +	if (unlikely(slots > MAX_SKB_FRAGS + 1)) {

But still - isn't this wrong now (i.e. can't it now validly exceed the
boundary checked for)?

Jan

> +		printk(KERN_ALERT "xennet: skb rides the rocket: %d slots\n",
> +		       slots);
>  		dump_stack();
>  		goto drop;
>  	}

Re: [PATCH] xen/netfront: handle compound page fragments on transmit

[Eric Dumazet]

On Tue, 2012-11-20 at 11:40 +0000, Ian Campbell wrote:
> An SKB paged fragment can consist of a compound page with order > 0.
> However the netchannel protocol deals only in PAGE_SIZE frames.
> 
> Handle this in xennet_make_frags by iterating over the frames which
> make up the page.
> 
> This is the netfront equivalent to 6a8ed462f16b for netback.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: netdev@vger.kernel.org
> Cc: xen-devel@lists.xen.org
> Cc: Eric Dumazet <edumazet@google.com>
> Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>
> Cc: ANNIE LI <annie.li@oracle.com>
> Cc: Sander Eikelenboom <linux@eikelenboom.it>
> Cc: Stefan Bader <stefan.bader@canonical.com>
> ---
>  drivers/net/xen-netfront.c |   58 +++++++++++++++++++++++++++++++++----------
>  1 files changed, 44 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index caa0110..a12b99a 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -452,24 +452,54 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
>  	/* Grant backend access to each skb fragment page. */
>  	for (i = 0; i < frags; i++) {
>  		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
> +		struct page *page = skb_frag_page(frag);
> +		unsigned long size = skb_frag_size(frag);
> +		unsigned long offset = frag->page_offset;
>  
> -		tx->flags |= XEN_NETTXF_more_data;
> +		/* Data must not cross a page boundary. */
> +		BUG_ON(size + offset > PAGE_SIZE<<compound_order(page));
>  
> -		id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> -		np->tx_skbs[id].skb = skb_get(skb);
> -		tx = RING_GET_REQUEST(&np->tx, prod++);
> -		tx->id = id;
> -		ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> -		BUG_ON((signed short)ref < 0);
> +		/* Skip unused frames from start of page */

'frame' in the comment means an order-0 page ?

> +		page += offset >> PAGE_SHIFT;
> +		offset &= ~PAGE_MASK;
>  
> -		mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
> -		gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> -						mfn, GNTMAP_readonly);
> +		while (size > 0) {
> +			unsigned long bytes;
>  
> -		tx->gref = np->grant_tx_ref[id] = ref;
> -		tx->offset = frag->page_offset;
> -		tx->size = skb_frag_size(frag);
> -		tx->flags = 0;
> +			BUG_ON(offset >= PAGE_SIZE);
> +
> +			bytes = PAGE_SIZE - offset;
> +			if (bytes > size)
> +				bytes = size;
> +
> +			tx->flags |= XEN_NETTXF_more_data;
> +
> +			id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> +			np->tx_skbs[id].skb = skb_get(skb);

BTW this skb_get() means extra atomic operations for every 4096 bytes
unit, and an extra atomic op (and test for final 0) at TX completion.
This could be avoided, by setting np->tx_skbs[id].skb = skb only for the
very last unit.

> +			tx = RING_GET_REQUEST(&np->tx, prod++);
> +			tx->id = id;
> +			ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> +			BUG_ON((signed short)ref < 0);
> +
> +			mfn = pfn_to_mfn(page_to_pfn(page));
> +			gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> +							mfn, GNTMAP_readonly);
> +
> +			tx->gref = np->grant_tx_ref[id] = ref;
> +			tx->offset = offset;
> +			tx->size = bytes;
> +			tx->flags = 0;
> +
> +			offset += bytes;
> +			size -= bytes;
> +
> +			/* Next frame */
> +			if (offset == PAGE_SIZE && size) {
> +				BUG_ON(!PageCompound(page));
> +				page++;
> +				offset = 0;
> +			}
> +		}
>  	}
>  
>  	np->tx.req_prod_pvt = prod;

Acked-by: Eric Dumazet <edumazet@google.com>

Thanks !

Re: [PATCH] xen/netfront: handle compound page fragments on transmit

[Ian Campbell]

On Tue, 2012-11-20 at 14:45 +0000, Eric Dumazet wrote:
> > +		/* Skip unused frames from start of page */
> 
> 'frame' in the comment means an order-0 page ?

Yes. Confusing in the context of a network driver I know! I couldn't
think of a better term.

> > +			id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> > +			np->tx_skbs[id].skb = skb_get(skb);
> 
> BTW this skb_get() means extra atomic operations for every 4096 bytes
> unit, and an extra atomic op (and test for final 0) at TX completion.
> This could be avoided, by setting np->tx_skbs[id].skb = skb only for the
> very last unit.

Thanks. Might be tricky because guests can ack the individual requests
in any order but it's something worth having a look at.

> >  	np->tx.req_prod_pvt = prod;
> 
> Acked-by: Eric Dumazet <edumazet@google.com>
> 
> Thanks !

Thanks for the review.

Ian.

> 
> 

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Ian Campbell]

On Tue, 2012-11-20 at 14:32 +0000, Jan Beulich wrote:
> > @@ -517,15 +540,16 @@ static int xennet_start_xmit(struct sk_buff *skb, 
> > struct net_device *dev)
> >  	grant_ref_t ref;
> >  	unsigned long mfn;
> >  	int notify;
> > -	int frags = skb_shinfo(skb)->nr_frags;
> > +	int slots;
> >  	unsigned int offset = offset_in_page(data);
> >  	unsigned int len = skb_headlen(skb);
> >  	unsigned long flags;
> >  
> > -	frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
> > -	if (unlikely(frags > MAX_SKB_FRAGS + 1)) {
> > -		printk(KERN_ALERT "xennet: skb rides the rocket: %d frags\n",
> > -		       frags);
> > +	slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
> > +		xennet_count_skb_frag_slots(skb);
> > +	if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
> 
> But still - isn't this wrong now (i.e. can't it now validly exceed the
> boundary checked for)?

In practice no because of the property that the number of pages backing
the frags is <= MAX_SKB_FRAGS even if you are using compound pages as
the frags.

Ian.

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Eric Dumazet]

On Tue, 2012-11-20 at 15:06 +0000, Ian Campbell wrote:

> In practice no because of the property that the number of pages backing
> the frags is <= MAX_SKB_FRAGS even if you are using compound pages as
> the frags.

Yes, but you can make this test trigger with some hacks from userland
(since the frag allocator is per task instead of per socket), so you
should remove the dump_stack() ?

Best way would be to count exact number of slots.

This could be something like 48 slots for a single skb

(if each frag is 4098 (1+4096+1)bytes, only the last one is around 4000
bytes)

MAX_SKB_FRAGS is really number of frags, while your driver needs a count
of 'order-0' 'frames'

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Jan Beulich]

>>> On 20.11.12 at 16:06, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2012-11-20 at 14:32 +0000, Jan Beulich wrote:
>> > @@ -517,15 +540,16 @@ static int xennet_start_xmit(struct sk_buff *skb, 
>> > struct net_device *dev)
>> >  	grant_ref_t ref;
>> >  	unsigned long mfn;
>> >  	int notify;
>> > -	int frags = skb_shinfo(skb)->nr_frags;
>> > +	int slots;
>> >  	unsigned int offset = offset_in_page(data);
>> >  	unsigned int len = skb_headlen(skb);
>> >  	unsigned long flags;
>> >  
>> > -	frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
>> > -	if (unlikely(frags > MAX_SKB_FRAGS + 1)) {
>> > -		printk(KERN_ALERT "xennet: skb rides the rocket: %d frags\n",
>> > -		       frags);
>> > +	slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
>> > +		xennet_count_skb_frag_slots(skb);
>> > +	if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
>> 
>> But still - isn't this wrong now (i.e. can't it now validly exceed the
>> boundary checked for)?
> 
> In practice no because of the property that the number of pages backing
> the frags is <= MAX_SKB_FRAGS even if you are using compound pages as
> the frags.

So are you saying that there is something in the system
preventing up to MAX_SKB_FRAGS * SKB_FRAG_PAGE_ORDER
(or NETDEV_FRAG_PAGE_MAX_ORDER) skb-s to be created? I
didn't find any. I do notice that __netdev_alloc_frag() currently
never gets called with a size larger than PAGE_SIZE, but
considering that the function just recently got made capable of
that, I'm sure respective users will show up rather sooner than
later.

Jan

Re: [Xen-devel] [PATCH] xen/netfront: handle compound page fragments on transmit

[Ian Campbell]

On Tue, 2012-11-20 at 15:28 +0000, Eric Dumazet wrote:
> On Tue, 2012-11-20 at 15:06 +0000, Ian Campbell wrote:
> 
> > In practice no because of the property that the number of pages backing
> > the frags is <= MAX_SKB_FRAGS even if you are using compound pages as
> > the frags.
> 
> Yes, but you can make this test trigger with some hacks from userland
> (since the frag allocator is per task instead of per socket), so you
> should remove the dump_stack() ?
> 
> Best way would be to count exact number of slots.
> 
> This could be something like 48 slots for a single skb
> 
> (if each frag is 4098 (1+4096+1)bytes, only the last one is around 4000
> bytes)
> 
> MAX_SKB_FRAGS is really number of frags, while your driver needs a count
> of 'order-0' 'frames'

The use of MAX_SKB_FRAGS is a bit misleading here, it's really the max
number of slots which the other end will be willing to receive as a
single frame (in the Ethernet sense), as defined by the PV protocol. It
happens to be the same as MAX_SKB_FRAGS (or it is at least
MAX_SKB_FRAGS, I'm not too sure).

I'll nuke the dump_stack() though -- it's not clear what sort of useful
context it would contain anyway.

Ian.

Re: [PATCH] xen/netfront: handle compound page fragments on transmit

[Eric Dumazet]

[-- Attachment #1.1: Type: text/plain, Size: 1985 bytes --]

At least TCP skbs are limited to 65536 bytes in tcp_sendmsg()
(around 45 1460-bytes MSS segments)

Thats probably because IPv4  stack only copes with this limit.

This probably could be relaxed for TCP friends, but this kind of skbs
should not hit a driver.




On Tue, Nov 20, 2012 at 7:44 AM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 20.11.12 at 16:06, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Tue, 2012-11-20 at 14:32 +0000, Jan Beulich wrote:
> >> > @@ -517,15 +540,16 @@ static int xennet_start_xmit(struct sk_buff
> *skb,
> >> > struct net_device *dev)
> >> >    grant_ref_t ref;
> >> >    unsigned long mfn;
> >> >    int notify;
> >> > -  int frags = skb_shinfo(skb)->nr_frags;
> >> > +  int slots;
> >> >    unsigned int offset = offset_in_page(data);
> >> >    unsigned int len = skb_headlen(skb);
> >> >    unsigned long flags;
> >> >
> >> > -  frags += DIV_ROUND_UP(offset + len, PAGE_SIZE);
> >> > -  if (unlikely(frags > MAX_SKB_FRAGS + 1)) {
> >> > -          printk(KERN_ALERT "xennet: skb rides the rocket: %d
> frags\n",
> >> > -                 frags);
> >> > +  slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
> >> > +          xennet_count_skb_frag_slots(skb);
> >> > +  if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
> >>
> >> But still - isn't this wrong now (i.e. can't it now validly exceed the
> >> boundary checked for)?
> >
> > In practice no because of the property that the number of pages backing
> > the frags is <= MAX_SKB_FRAGS even if you are using compound pages as
> > the frags.
>
> So are you saying that there is something in the system
> preventing up to MAX_SKB_FRAGS * SKB_FRAG_PAGE_ORDER
> (or NETDEV_FRAG_PAGE_MAX_ORDER) skb-s to be created? I
> didn't find any. I do notice that __netdev_alloc_frag() currently
> never gets called with a size larger than PAGE_SIZE, but
> considering that the function just recently got made capable of
> that, I'm sure respective users will show up rather sooner than
> later.
>
> Jan
>
>

[-- Attachment #1.2: Type: text/html, Size: 2945 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[ANNIE LI]

On 2012-11-20 19:36, Ian Campbell wrote:
> On Tue, 2012-11-20 at 09:21 +0000, Ian Campbell wrote:
>> On Tue, 2012-11-20 at 08:30 +0000, Stefan Bader wrote:
>>>>> When I tried to rebase my persistent grant netfront/netback patch on
>>>>> latest kernel, netperf/netserver test never succeeded. I did some test
>>>>> to find out that v3.6-rc7 works fine, but v3.7-rc1, v3.7-rc2 and
>>>>> v3.7-rc4 does not succeed in netperf/netserver test. So I keep my
>>>>> persistent grant patch only based on v3.4-rc3 now.
>>>>> Konrad thought about commit 6a8ed462f16b8455eec5ae00eb6014159a6721f0 in
>>>>> v3.7-rc1, and suggested me to test your debug patch in netfront. This
>>>>> BUG_ON happens soon after running the netperf/netserver test case.
>>>>> Thanks
>>>>> Annie
>>>> Is there any progression with this bug (rc6 is out the door, so the
>>> release of 3.7-final seems to be eminent and this bug completely
>>> cripples any networking with guests) ?
>>> +1 on that. I was testing yesterday with a PVM domU running 3.7-rc5 on Xen 4.2
>>> (but also reported from EC2 running Xen 3.4.3) c with one VCPU. I actually can
>>> trigger it by just ssh'ing into the domU (from another machine) and then run
>>> "find /". Output starts to stutter and then stops completely. When this happens
>>> a new connection still can be made and as long as only shorter output is
>>> generated the ssh connection is ok. From a dump taken it looks like user-space
>>> is waiting in some select call (without any warnon I rather won't see the tx path).
>> Annie, are you still looking into this or shall I?
> I'll assume that silence == No. Will post a patch shortly.
Sorry for the delay response, I did create a patch, but did not post it 
out in time.

Thanks
Annie
> Ian.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH] xen/netfront: handle compound page fragments on transmit

[ANNIE LI]

On 2012-11-20 19:40, Ian Campbell wrote:
> An SKB paged fragment can consist of a compound page with order>  0.
> However the netchannel protocol deals only in PAGE_SIZE frames.
>
> Handle this in xennet_make_frags by iterating over the frames which
> make up the page.
>
> This is the netfront equivalent to 6a8ed462f16b for netback.
>
> Signed-off-by: Ian Campbell<ian.campbell@citrix.com>
> Cc: netdev@vger.kernel.org
> Cc: xen-devel@lists.xen.org
> Cc: Eric Dumazet<edumazet@google.com>
> Cc: Konrad Rzeszutek Wilk<konrad@kernel.org>
> Cc: ANNIE LI<annie.li@oracle.com>
> Cc: Sander Eikelenboom<linux@eikelenboom.it>
> Cc: Stefan Bader<stefan.bader@canonical.com>
> ---
>   drivers/net/xen-netfront.c |   58 +++++++++++++++++++++++++++++++++----------
>   1 files changed, 44 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index caa0110..a12b99a 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -452,24 +452,54 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
>   	/* Grant backend access to each skb fragment page. */
>   	for (i = 0; i<  frags; i++) {
>   		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
> +		struct page *page = skb_frag_page(frag);
> +		unsigned long size = skb_frag_size(frag);
> +		unsigned long offset = frag->page_offset;

There are following definitions at the beginning of xennet_make_frags,

         unsigned int offset = offset_in_page(data);
         unsigned int len = skb_headlen(skb);

Is it better to reuse those definitions, and not define new size and 
offset again in this for loop? And unsigned int is enough here, right?

>
> -		tx->flags |= XEN_NETTXF_more_data;
> +		/* Data must not cross a page boundary. */
> +		BUG_ON(size + offset>  PAGE_SIZE<<compound_order(page));
>
> -		id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
> -		np->tx_skbs[id].skb = skb_get(skb);
> -		tx = RING_GET_REQUEST(&np->tx, prod++);
> -		tx->id = id;
> -		ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> -		BUG_ON((signed short)ref<  0);
> +		/* Skip unused frames from start of page */
> +		page += offset>>  PAGE_SHIFT;
> +		offset&= ~PAGE_MASK;
>
> -		mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
> -		gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> -						mfn, GNTMAP_readonly);
> +		while (size>  0) {
> +			unsigned long bytes;
>
> -		tx->gref = np->grant_tx_ref[id] = ref;
> -		tx->offset = frag->page_offset;
> -		tx->size = skb_frag_size(frag);
> -		tx->flags = 0;
> +			BUG_ON(offset>= PAGE_SIZE);
> +
> +			bytes = PAGE_SIZE - offset;
> +			if (bytes>  size)
> +				bytes = size;
> +
> +			tx->flags |= XEN_NETTXF_more_data;
> +
> +			id = get_id_from_freelist(&np->tx_skb_freelist, np->tx_skbs);
Over 80 characters?
> +			np->tx_skbs[id].skb = skb_get(skb);
> +			tx = RING_GET_REQUEST(&np->tx, prod++);
> +			tx->id = id;
> +			ref = gnttab_claim_grant_reference(&np->gref_tx_head);
> +			BUG_ON((signed short)ref<  0);
> +
> +			mfn = pfn_to_mfn(page_to_pfn(page));
> +			gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
> +							mfn, GNTMAP_readonly);
Over 80 characters?

Thanks
Annie
> +
> +			tx->gref = np->grant_tx_ref[id] = ref;
> +			tx->offset = offset;
> +			tx->size = bytes;
> +			tx->flags = 0;
> +
> +			offset += bytes;
> +			size -= bytes;
> +
> +			/* Next frame */
> +			if (offset == PAGE_SIZE&&  size) {
> +				BUG_ON(!PageCompound(page));
> +				page++;
> +				offset = 0;
> +			}
> +		}
>   	}
>
>   	np->tx.req_prod_pvt = prod;

Re: [PATCH] xen/netfront: handle compound page fragments on transmit

[Ian Campbell]

On Wed, 2012-11-21 at 02:52 +0000, ANNIE LI wrote:
> 
> On 2012-11-20 19:40, Ian Campbell wrote:
> > An SKB paged fragment can consist of a compound page with order>  0.
> > However the netchannel protocol deals only in PAGE_SIZE frames.
> >
> > Handle this in xennet_make_frags by iterating over the frames which
> > make up the page.
> >
> > This is the netfront equivalent to 6a8ed462f16b for netback.
> >
> > Signed-off-by: Ian Campbell<ian.campbell@citrix.com>
> > Cc: netdev@vger.kernel.org
> > Cc: xen-devel@lists.xen.org
> > Cc: Eric Dumazet<edumazet@google.com>
> > Cc: Konrad Rzeszutek Wilk<konrad@kernel.org>
> > Cc: ANNIE LI<annie.li@oracle.com>
> > Cc: Sander Eikelenboom<linux@eikelenboom.it>
> > Cc: Stefan Bader<stefan.bader@canonical.com>
> > ---
> >   drivers/net/xen-netfront.c |   58 +++++++++++++++++++++++++++++++++----------
> >   1 files changed, 44 insertions(+), 14 deletions(-)
> >
> > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> > index caa0110..a12b99a 100644
> > --- a/drivers/net/xen-netfront.c
> > +++ b/drivers/net/xen-netfront.c
> > @@ -452,24 +452,54 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
> >   	/* Grant backend access to each skb fragment page. */
> >   	for (i = 0; i<  frags; i++) {
> >   		skb_frag_t *frag = skb_shinfo(skb)->frags + i;
> > +		struct page *page = skb_frag_page(frag);
> > +		unsigned long size = skb_frag_size(frag);
> > +		unsigned long offset = frag->page_offset;
> 
> There are following definitions at the beginning of xennet_make_frags,
> 
>          unsigned int offset = offset_in_page(data);
>          unsigned int len = skb_headlen(skb);

So they are, well spotted.

> Is it better to reuse those definitions, and not define new size and 
> offset again in this for loop? And unsigned int is enough here, right?

Yes to both.
> [...]
> Over 80 characters?
[...]
> Over 80 characters?

Both fixed, thanks for your review.

Ian.