VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Rick Jones]

Hi -

I've been messing about with VxLAN with a pair of two socket systems 
with ixgbe-driven NICs (HP 560FLRs for the curious) and have noticed 
that with a kernel from a davem net-next tree I refreshed a couple days 
ago I can create the vlxan link, and ifconfig it, but once I start 
trying to push packets through it the system appears to hang and I have 
to reset it.

I went back to checkout v3.19 and tried that kernel and all was 
happiness and joy.  So, I started down the bisect path, but after the 
first iteration I ran into:

   INSTALL debian/headertmp/usr/include/asm/ (65 files)
parsechangelog/debian: warning:     debian/changelog(l2): found eof 
where expected first heading
parsechangelog/debian: error: fatal error occurred while parsing 
debian/changelog
dpkg-gencontrol: error: changelog parser 
/usr/lib/dpkg/parsechangelog/debian gave error exit status 255
scripts/package/Makefile:90: recipe for target 'deb-pkg' failed
make[1]: *** [deb-pkg] Error 255
Makefile:1238: recipe for target 'deb-pkg' failed
make: *** [deb-pkg] Error 2

Which I'm afraid goes beyond my knowledge of how to deal.  Does any of 
that - either the VxLAN or the build problem - sound familiar?

thanks,

rick jones

Re: VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Rick Jones]

On 04/03/2015 05:11 PM, Rick Jones wrote:

>    INSTALL debian/headertmp/usr/include/asm/ (65 files)
> parsechangelog/debian: warning:     debian/changelog(l2): found eof
> where expected first heading
> parsechangelog/debian: error: fatal error occurred while parsing
> debian/changelog
> dpkg-gencontrol: error: changelog parser
> /usr/lib/dpkg/parsechangelog/debian gave error exit status 255
> scripts/package/Makefile:90: recipe for target 'deb-pkg' failed
> make[1]: *** [deb-pkg] Error 255
> Makefile:1238: recipe for target 'deb-pkg' failed
> make: *** [deb-pkg] Error 2
>
> Which I'm afraid goes beyond my knowledge of how to deal.  Does any of
> that - either the VxLAN or the build problem - sound familiar?

Seems the build failure relates to the particulars of the lsb-release 
package on the system on which I was building.

rick

Re: VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Marcelo Ricardo Leitner]

Hi,

On Fri, Apr 03, 2015 at 05:11:12PM -0700, Rick Jones wrote:
> Hi -
> 
> I've been messing about with VxLAN with a pair of two socket systems with
> ixgbe-driven NICs (HP 560FLRs for the curious) and have noticed that with a
> kernel from a davem net-next tree I refreshed a couple days ago I can create
> the vlxan link, and ifconfig it, but once I start trying to push packets
> through it the system appears to hang and I have to reset it.
> 
> I went back to checkout v3.19 and tried that kernel and all was happiness
> and joy.  So, I started down the bisect path, but after the first iteration
> I ran into:

That means you could send/receive some packets on it before the hang, right?

  Marcelo

Re: VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Rick Jones]

On 04/06/2015 12:07 PM, Marcelo Ricardo Leitner wrote:
> Hi,
>
> On Fri, Apr 03, 2015 at 05:11:12PM -0700, Rick Jones wrote:
>> Hi -
>>
>> I've been messing about with VxLAN with a pair of two socket systems with
>> ixgbe-driven NICs (HP 560FLRs for the curious) and have noticed that with a
>> kernel from a davem net-next tree I refreshed a couple days ago I can create
>> the vlxan link, and ifconfig it, but once I start trying to push packets
>> through it the system appears to hang and I have to reset it.
>>
>> I went back to checkout v3.19 and tried that kernel and all was happiness
>> and joy.  So, I started down the bisect path, but after the first iteration
>> I ran into:
>
> That means you could send/receive some packets on it before the hang, right?

On v3.19 I could send and receive lots of packets without a hang.  I 
could even run netperf through the vxlan tunnel :)

Now that I've addressed my build issue I'm in the midst of a bisect, but 
if someone has faster git-fu I'll not be disappointed if they are able 
to bisect before me.

rick

Re: VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Rick Jones]

On 04/06/2015 01:00 PM, Rick Jones wrote:
> On 04/06/2015 12:07 PM, Marcelo Ricardo Leitner wrote:
>> That means you could send/receive some packets on it before the hang,
>> right?
>
> On v3.19 I could send and receive lots of packets without a hang.  I
> could even run netperf through the vxlan tunnel :)
>
> Now that I've addressed my build issue I'm in the midst of a bisect, but
> if someone has faster git-fu I'll not be disappointed if they are able
> to bisect before me.

The git bisect has implicated:

b6a7719aedd7e5c0f2df7641aa47386111682df4 is the first bad commit
commit b6a7719aedd7e5c0f2df7641aa47386111682df4
Author: Hannes Frederic Sowa <hannes@stressinduktion.org>
Date:   Wed Mar 25 17:07:44 2015 +0100

     ipv4: hash net ptr into fragmentation bucket selection

     As namespaces are sometimes used with overlapping ip address ranges,
     we should also use the namespace as input to the hash to select the ip
     fragmentation counter bucket.

     Cc: Eric Dumazet <edumazet@google.com>
     Cc: Flavio Leitner <fbl@redhat.com>
     Signed-off-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
     Signed-off-by: David S. Miller <davem@davemloft.net>

:040000 040000 fd2a733ad8775b6d5f6096c15187a25515f93812 
7122024878daef58138f67bf47287c3bb8f23caa M	drivers
:040000 040000 89268ed81d87ec7d4f21e45bc22f1f4103fbe829 
759611eb84770f27d8f06f63eaba5e228344cb18 M	include
:040000 040000 7be4960e2db1a2975ee25eddf78f126fc9468662 
e0e86454b5681f6cd9d738f9bc05e7bb828910b4 M	net

I cloned a "fresh" davem net-next tree (before today's accepts and 
message to Linus) which made kernels tagged as 4.0.0-rc6+ and that still 
hung.  When I reverted the commit implicated above, the resulting kernel 
didn't hang when I tried to ping through a vxlan interface.

rick

Re: VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Hannes Frederic Sowa]

On Tue, Apr 7, 2015, at 00:51, Rick Jones wrote:
> On 04/06/2015 01:00 PM, Rick Jones wrote:
> > On 04/06/2015 12:07 PM, Marcelo Ricardo Leitner wrote:
> >> That means you could send/receive some packets on it before the hang,
> >> right?
> >
> > On v3.19 I could send and receive lots of packets without a hang.  I
> > could even run netperf through the vxlan tunnel :)
> >
> > Now that I've addressed my build issue I'm in the midst of a bisect, but
> > if someone has faster git-fu I'll not be disappointed if they are able
> > to bisect before me.
> 
> The git bisect has implicated:
> 
> b6a7719aedd7e5c0f2df7641aa47386111682df4 is the first bad commit
> commit b6a7719aedd7e5c0f2df7641aa47386111682df4
> Author: Hannes Frederic Sowa <hannes@stressinduktion.org>
> Date:   Wed Mar 25 17:07:44 2015 +0100

Thanks for the report, I try to reproduce this right now!

Bye,
Hannes

Re: VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Hannes Frederic Sowa]

On Tue, Apr 7, 2015, at 12:14, Hannes Frederic Sowa wrote:
> On Tue, Apr 7, 2015, at 00:51, Rick Jones wrote:
> > On 04/06/2015 01:00 PM, Rick Jones wrote:
> > > On 04/06/2015 12:07 PM, Marcelo Ricardo Leitner wrote:
> > >> That means you could send/receive some packets on it before the hang,
> > >> right?
> > >
> > > On v3.19 I could send and receive lots of packets without a hang.  I
> > > could even run netperf through the vxlan tunnel :)
> > >
> > > Now that I've addressed my build issue I'm in the midst of a bisect, but
> > > if someone has faster git-fu I'll not be disappointed if they are able
> > > to bisect before me.
> > 
> > The git bisect has implicated:
> > 
> > b6a7719aedd7e5c0f2df7641aa47386111682df4 is the first bad commit
> > commit b6a7719aedd7e5c0f2df7641aa47386111682df4
> > Author: Hannes Frederic Sowa <hannes@stressinduktion.org>
> > Date:   Wed Mar 25 17:07:44 2015 +0100
> 
> Thanks for the report, I try to reproduce this right now!

Rick, can you send me more details on your setup? I am not able to
reproduce it here, I am able to send a lot of fragments via vxlan
interfaces here.

Thanks,
Hannes

Re: VxLAN system wedge in 4.0.0-rc5+ from davem net-next?

[Rick Jones]

On 04/07/2015 05:28 AM, Hannes Frederic Sowa wrote:
> On Tue, Apr 7, 2015, at 12:14, Hannes Frederic Sowa wrote:
>> On Tue, Apr 7, 2015, at 00:51, Rick Jones wrote:
>>> The git bisect has implicated:
>>>
>>> b6a7719aedd7e5c0f2df7641aa47386111682df4 is the first bad commit
>>> commit b6a7719aedd7e5c0f2df7641aa47386111682df4
>>> Author: Hannes Frederic Sowa <hannes@stressinduktion.org>
>>> Date:   Wed Mar 25 17:07:44 2015 +0100
>>
>> Thanks for the report, I try to reproduce this right now!
>
> Rick, can you send me more details on your setup? I am not able to
> reproduce it here, I am able to send a lot of fragments via vxlan
> interfaces here.

Pair of HP Sl230s Gen8 systems with HP 560FLR 10GbE NICs (Intel
82599ES) connected to an HP 5900 series switch running Comware 7.11.035, 
Release 2208P01.  On one side I have a 3.14.mumble kernel (which was 
also the source of the config file used to build the upstream kernels). 
  On the other side I have the test kernels from davem's net-next tree. 
  I create the vxlan0 interface using:

  ip li add vxlan0 type vxlan id 42 group 239.1.1.1 dev eth2 dstport 4789

Which is taken almost verbatim from Documentation/networking/vxlan.txt
and then I assign an IP address to the interface with ifconfig.  Then I 
ping the other system (which I have setup vxlan similarly) and before I 
even see ping report one reply, the system ceases to respond via the 
network or console.

I suspect my present mail client would munge it as inline so I'll send 
you a copy of the config file as a direct email with an attachment.

rick