All of lore.kernel.org
 help / color / mirror / Atom feed
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Brad Campbell <brad@fnarfbargle.com>
Cc: Patrick McHardy <kaber@trash.net>,
	Bart De Schuymer <bdschuym@pandora.be>,
	kvm@vger.kernel.org, linux-mm@kvack.org,
	linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
	netfilter-devel@vger.kernel.org
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39
Date: Wed, 08 Jun 2011 05:59:01 +0200	[thread overview]
Message-ID: <1307505541.3102.12.camel@edumazet-laptop> (raw)
In-Reply-To: <4DEEBFC2.4060102@fnarfbargle.com>

Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;



WARNING: multiple messages have this Message-ID (diff)
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Brad Campbell <brad@fnarfbargle.com>
Cc: Patrick McHardy <kaber@trash.net>,
	Bart De Schuymer <bdschuym@pandora.be>,
	 kvm@vger.kernel.org, linux-mm@kvack.org,
	linux-kernel@vger.kernel.org,  netdev@vger.kernel.org,
	netfilter-devel@vger.kernel.org
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39
Date: Wed, 08 Jun 2011 05:59:01 +0200	[thread overview]
Message-ID: <1307505541.3102.12.camel@edumazet-laptop> (raw)
In-Reply-To: <4DEEBFC2.4060102@fnarfbargle.com>

Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

WARNING: multiple messages have this Message-ID (diff)
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Brad Campbell <brad@fnarfbargle.com>
Cc: Patrick McHardy <kaber@trash.net>,
	Bart De Schuymer <bdschuym@pandora.be>,
	kvm@vger.kernel.org, linux-mm@kvack.org,
	linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
	netfilter-devel@vger.kernel.org
Subject: Re: KVM induced panic on 2.6.38[2367] & 2.6.39
Date: Wed, 08 Jun 2011 05:59:01 +0200	[thread overview]
Message-ID: <1307505541.3102.12.camel@edumazet-laptop> (raw)
In-Reply-To: <4DEEBFC2.4060102@fnarfbargle.com>

Le mercredi 08 juin 2011 A  08:18 +0800, Brad Campbell a A(C)crit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 A  17:35 +0200, Patrick McHardy a A(C)crit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

  reply	other threads:[~2011-06-08  3:59 UTC|newest]

Thread overview: 111+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-05-31  1:24 KVM induced panic on 2.6.38[2367] & 2.6.39 Brad Campbell
2011-05-31  5:47 ` Borislav Petkov
2011-05-31  5:47   ` Borislav Petkov
2011-05-31  9:26   ` Brad Campbell
2011-05-31  9:26     ` Brad Campbell
2011-05-31 10:38     ` Borislav Petkov
2011-05-31 10:38       ` Borislav Petkov
2011-05-31 14:24       ` Brad Campbell
2011-05-31 14:24         ` Brad Campbell
2011-05-31 14:24         ` Brad Campbell
2011-05-31 22:31         ` Hugh Dickins
2011-05-31 22:31           ` Hugh Dickins
2011-06-01  0:18           ` Brad Campbell
2011-06-01  0:18             ` Brad Campbell
2011-06-01  0:37           ` Brad Campbell
2011-06-01  0:37             ` Brad Campbell
2011-06-01  1:15             ` Andrea Arcangeli
2011-06-01  1:15               ` Andrea Arcangeli
2011-06-01  2:03               ` Brad Campbell
2011-06-01  2:03                 ` Brad Campbell
2011-06-01  4:52               ` Hugh Dickins
2011-06-01  4:52                 ` Hugh Dickins
2011-06-01  6:31                 ` Brad Campbell
2011-06-01  6:31                   ` Brad Campbell
2011-06-01  6:56                   ` Avi Kivity
2011-06-01  6:56                     ` Avi Kivity
2011-06-01  9:29                     ` Brad Campbell
2011-06-01  9:29                       ` Brad Campbell
2011-06-01  9:29                       ` Brad Campbell
2011-06-01  9:40                       ` Avi Kivity
2011-06-01  9:40                         ` Avi Kivity
2011-06-01  9:41                         ` Avi Kivity
2011-06-01  9:41                           ` Avi Kivity
2011-06-01 10:53                           ` Brad Campbell
2011-06-01 10:53                             ` Brad Campbell
2011-06-01 11:09                             ` Avi Kivity
2011-06-01 11:09                               ` Avi Kivity
2011-06-01 11:18                             ` CaT
2011-06-01 11:18                               ` CaT
2011-06-01 11:52                               ` Brad Campbell
2011-06-01 11:52                                 ` Brad Campbell
2011-06-01 23:03                                 ` CaT
2011-06-01 23:03                                   ` CaT
2011-06-03 13:38                                   ` Brad Campbell
2011-06-03 13:38                                     ` Brad Campbell
2011-06-03 15:50                                     ` Bernhard Held
2011-06-03 15:50                                       ` Bernhard Held
2011-06-03 15:50                                       ` Bernhard Held
2011-06-03 16:07                                       ` Brad Campbell
2011-06-03 16:07                                         ` Brad Campbell
2011-06-06 20:10                                         ` Bart De Schuymer
2011-06-06 20:10                                           ` Bart De Schuymer
2011-06-06 20:23                                           ` Eric Dumazet
2011-06-06 20:23                                             ` Eric Dumazet
2011-06-06 20:23                                             ` Eric Dumazet
2011-06-07  3:33                                           ` Brad Campbell
2011-06-07  3:33                                             ` Brad Campbell
2011-06-07 13:30                                             ` Patrick McHardy
2011-06-07 13:30                                               ` Patrick McHardy
2011-06-07 14:40                                               ` Brad Campbell
2011-06-07 14:40                                                 ` Brad Campbell
2011-06-07 15:35                                                 ` Patrick McHardy
2011-06-07 15:35                                                   ` Patrick McHardy
2011-06-07 18:31                                                   ` Eric Dumazet
2011-06-07 18:31                                                     ` Eric Dumazet
2011-06-07 18:31                                                     ` Eric Dumazet
2011-06-07 22:57                                                     ` Patrick McHardy
2011-06-07 22:57                                                       ` Patrick McHardy
2011-06-07 22:57                                                       ` Patrick McHardy
2011-06-08  0:18                                                       ` Brad Campbell
2011-06-08  0:18                                                         ` Brad Campbell
2011-06-08  0:18                                                         ` Brad Campbell
2011-06-08  3:59                                                         ` Eric Dumazet [this message]
2011-06-08  3:59                                                           ` Eric Dumazet
2011-06-08  3:59                                                           ` Eric Dumazet
2011-06-08 17:02                                                           ` Brad Campbell
2011-06-08 17:02                                                             ` Brad Campbell
2011-06-08 17:02                                                             ` Brad Campbell
2011-06-08 21:22                                                             ` Eric Dumazet
2011-06-08 21:22                                                               ` Eric Dumazet
2011-06-08 21:22                                                               ` Eric Dumazet
2011-06-10  2:52                                                             ` Simon Horman
2011-06-10  2:52                                                               ` Simon Horman
2011-06-10 12:37                                                               ` Mark Lord
2011-06-10 12:37                                                                 ` Mark Lord
2011-06-10 16:43                                                                 ` Henrique de Moraes Holschuh
2011-06-10 16:43                                                                   ` Henrique de Moraes Holschuh
2011-06-12 15:38                                                               ` Avi Kivity
2011-06-12 15:38                                                                 ` Avi Kivity
2011-06-07 23:43                                                   ` Brad Campbell
2011-06-07 23:43                                                     ` Brad Campbell
2011-06-07 18:04                                                 ` Bart De Schuymer
2011-06-07 18:04                                                   ` Bart De Schuymer
2011-06-08  0:15                                                   ` Brad Campbell
2011-06-08  0:15                                                     ` Brad Campbell
2011-06-05  8:14                                     ` Avi Kivity
2011-06-05  8:14                                       ` Avi Kivity
2011-06-05 13:45                                       ` Brad Campbell
2011-06-05 13:45                                         ` Brad Campbell
2011-06-05 13:58                                         ` Avi Kivity
2011-06-05 13:58                                           ` Avi Kivity
2011-06-06 20:22                                         ` Eric Dumazet
2011-06-06 20:22                                           ` Eric Dumazet
2011-06-06 20:22                                           ` Eric Dumazet
2011-06-07 13:27                                           ` Brad Campbell
2011-06-07 13:37                                             ` Eric Dumazet
2011-06-07 15:15                                               ` Brad Campbell
2011-08-20 13:16                                               ` Brad Campbell
2011-08-22  6:36                                                 ` Avi Kivity
2011-08-22  6:45                                                   ` Eric Dumazet
2011-08-22 11:45                                                     ` Brad Campbell

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1307505541.3102.12.camel@edumazet-laptop \
    --to=eric.dumazet@gmail.com \
    --cc=bdschuym@pandora.be \
    --cc=brad@fnarfbargle.com \
    --cc=kaber@trash.net \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=netdev@vger.kernel.org \
    --cc=netfilter-devel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.