net packet storms with multiple NICs

net packet storms with multiple NICs

[Michael Tokarev]

Hello.

I vaguely remember something like this has been reported and/or
discussed already, but I can't find anything related. I'm also
not sure if it's kvm-specific or exists in qemu too.

I want some clarification wrt vlan= parameter in -net definition.

What started this all is a problem report I had with kvm package.
It turns out that the OP had 2 network adaptors defined for one
of his guests, and right when his guest started getting an IP
address over DHCP for one of them, the network saw huge packet
storm consisting of DHCP and ARP packets.

I immediately reproduced the problem locally.  It turns out that
kvm reflects packets coming from one guest NIC on another guest
NIC, and since both are connected to the same bridge we're getting
endless packet storm.  To a level when kvm process becomes 100%
busy and does not respond to anything but `kill -9'.

The problem is easily solvable by specifying explicit different
vlan indexes for different pairs of host/guest -net components.

I've two questions:

o what's the intended usage of all-vlan-equal case, when kvm (or qemu)
   reflects packets from one interface to another?  It's what bridge
   in linux is for, I think.

o why different -net guest -net host pairs are not getting different
   vlan= indexes by default, to stop the above-mentioned packet
   storms right away?  I think it's a wise default to assign different
   pairs to different vlans, by counting -net host and -net guest
   sequences.

Thanks!

/mjt

[Qemu-devel] net packet storms with multiple NICs

[Michael Tokarev]

Hello.

I vaguely remember something like this has been reported and/or
discussed already, but I can't find anything related. I'm also
not sure if it's kvm-specific or exists in qemu too.

I want some clarification wrt vlan= parameter in -net definition.

What started this all is a problem report I had with kvm package.
It turns out that the OP had 2 network adaptors defined for one
of his guests, and right when his guest started getting an IP
address over DHCP for one of them, the network saw huge packet
storm consisting of DHCP and ARP packets.

I immediately reproduced the problem locally.  It turns out that
kvm reflects packets coming from one guest NIC on another guest
NIC, and since both are connected to the same bridge we're getting
endless packet storm.  To a level when kvm process becomes 100%
busy and does not respond to anything but `kill -9'.

The problem is easily solvable by specifying explicit different
vlan indexes for different pairs of host/guest -net components.

I've two questions:

o what's the intended usage of all-vlan-equal case, when kvm (or qemu)
   reflects packets from one interface to another?  It's what bridge
   in linux is for, I think.

o why different -net guest -net host pairs are not getting different
   vlan= indexes by default, to stop the above-mentioned packet
   storms right away?  I think it's a wise default to assign different
   pairs to different vlans, by counting -net host and -net guest
   sequences.

Thanks!

/mjt

Re: net packet storms with multiple NICs

[Andreas Plesner Jacobsen]

On Fri, Oct 23, 2009 at 08:25:39PM +0400, Michael Tokarev wrote:
>
> o why different -net guest -net host pairs are not getting different
>   vlan= indexes by default, to stop the above-mentioned packet
>   storms right away?  I think it's a wise default to assign different
>   pairs to different vlans, by counting -net host and -net guest
>   sequences.

I think the major issue is that the definition of a "pair" is when the
vlan-options match. There's no requirement to define the two after each
other.

-- 
Andreas

[Qemu-devel] Re: net packet storms with multiple NICs

[Andreas Plesner Jacobsen]

On Fri, Oct 23, 2009 at 08:25:39PM +0400, Michael Tokarev wrote:
>
> o why different -net guest -net host pairs are not getting different
>   vlan= indexes by default, to stop the above-mentioned packet
>   storms right away?  I think it's a wise default to assign different
>   pairs to different vlans, by counting -net host and -net guest
>   sequences.

I think the major issue is that the definition of a "pair" is when the
vlan-options match. There's no requirement to define the two after each
other.

-- 
Andreas

Re: [Qemu-devel] net packet storms with multiple NICs

[Mark McLoughlin]

On Fri, 2009-10-23 at 20:25 +0400, Michael Tokarev wrote:
> I've two questions:
> 
> o what's the intended usage of all-vlan-equal case, when kvm (or qemu)
>    reflects packets from one interface to another?  It's what bridge
>    in linux is for, I think.

I don't think it's necessarily an intended use-case for the vlan feature

> o why different -net guest -net host pairs are not getting different
>    vlan= indexes by default, to stop the above-mentioned packet
>    storms right away?  I think it's a wise default to assign different
>    pairs to different vlans, by counting -net host and -net guest
>    sequences.

With 0.12, we're going to be de-emphasising the vlan feature and instead
have NICs directly connected to host backends. The vlan feature will be
just another host backend, but optional

You'll start guests with e.g.:

  -netdev tap,id=tap.0 -device virtio-net-pci,netdev=tap.0

Cheers,
Mark.

Re: net packet storms with multiple NICs

[Michael Tokarev]

Andreas Plesner Jacobsen wrote:
> On Fri, Oct 23, 2009 at 08:25:39PM +0400, Michael Tokarev wrote:
>> o why different -net guest -net host pairs are not getting different
>>   vlan= indexes by default, to stop the above-mentioned packet
>>   storms right away?  I think it's a wise default to assign different
>>   pairs to different vlans, by counting -net host and -net guest
>>   sequences.
> 
> I think the major issue is that the definition of a "pair" is when the
> vlan-options match. There's no requirement to define the two after each
> other.

I mean something like this (rough approximation:

  int host_vlan_no = 0, guest_vlan_no = 0;

  while(parse_option()) {
     if (option == host-side-net) {
       if (!explicit_vlan) vlan = host_vlan_no;
       ++host_vlan_no;
     }
     if (option == guest-side-net) {
       if (!explicit_vlan) vlan = guest_vlan_no;
       ++guest_vlan_no;
     }
  }

this way, consecutive host-side -net will receive
consecutive vlan=NN, and consecutive guest-side -net
also consecutive vlan=NN numbers.

But having in mind Marc's reply I don't think it's
necessary anymore.

Thanks!

/mjt

[Qemu-devel] Re: net packet storms with multiple NICs

[Michael Tokarev]

Andreas Plesner Jacobsen wrote:
> On Fri, Oct 23, 2009 at 08:25:39PM +0400, Michael Tokarev wrote:
>> o why different -net guest -net host pairs are not getting different
>>   vlan= indexes by default, to stop the above-mentioned packet
>>   storms right away?  I think it's a wise default to assign different
>>   pairs to different vlans, by counting -net host and -net guest
>>   sequences.
> 
> I think the major issue is that the definition of a "pair" is when the
> vlan-options match. There's no requirement to define the two after each
> other.

I mean something like this (rough approximation:

  int host_vlan_no = 0, guest_vlan_no = 0;

  while(parse_option()) {
     if (option == host-side-net) {
       if (!explicit_vlan) vlan = host_vlan_no;
       ++host_vlan_no;
     }
     if (option == guest-side-net) {
       if (!explicit_vlan) vlan = guest_vlan_no;
       ++guest_vlan_no;
     }
  }

this way, consecutive host-side -net will receive
consecutive vlan=NN, and consecutive guest-side -net
also consecutive vlan=NN numbers.

But having in mind Marc's reply I don't think it's
necessary anymore.

Thanks!

/mjt

Re: [Qemu-devel] net packet storms with multiple NICs

[Avi Kivity]

On 10/23/2009 06:43 PM, Mark McLoughlin wrote:
> On Fri, 2009-10-23 at 20:25 +0400, Michael Tokarev wrote:
>    
>> I've two questions:
>>
>> o what's the intended usage of all-vlan-equal case, when kvm (or qemu)
>>     reflects packets from one interface to another?  It's what bridge
>>     in linux is for, I think.
>>      
> I don't think it's necessarily an intended use-case for the vlan feature
>    

Well, it is.  vlan=x really means "the ethernet segment named x".  If 
you connect all your guest nics to one vlan, you are connecting them all 
to one ethernet segment, so any packet transmitted on one will be 
reflected on others.

Whether this is a useful feature is another matter, but the code is 
functioning as expected.

-- 
error compiling committee.c: too many arguments to function

RE: [Qemu-devel] net packet storms with multiple NICs

[Krumme, Chris]

> -----Original Message-----
> From: 
> qemu-devel-bounces+chris.krumme=windriver.com@nongnu.org 
> [mailto:qemu-devel-bounces+chris.krumme=windriver.com@nongnu.o
rg] On Behalf Of Avi Kivity
> Sent: Sunday, October 25, 2009 9:23 AM
> To: Mark McLoughlin
> Cc: Michael Tokarev; qemu-devel@nongnu.org; KVM list
> Subject: Re: [Qemu-devel] net packet storms with multiple NICs
> 
> On 10/23/2009 06:43 PM, Mark McLoughlin wrote:
> > On Fri, 2009-10-23 at 20:25 +0400, Michael Tokarev wrote:
> >    
> >> I've two questions:
> >>
> >> o what's the intended usage of all-vlan-equal case, when 
> kvm (or qemu)
> >>     reflects packets from one interface to another?  It's 
> what bridge
> >>     in linux is for, I think.
> >>      
> > I don't think it's necessarily an intended use-case for the 
> vlan feature
> >    
> 
> Well, it is.  vlan=x really means "the ethernet segment named x".  If 
> you connect all your guest nics to one vlan, you are 
> connecting them all 
> to one ethernet segment, so any packet transmitted on one will be 
> reflected on others.
> 
> Whether this is a useful feature is another matter, but the code is 
> functioning as expected.

Hello,

We had one environment where the NIC understood by u-boot and the NIC
understood by the kernel where different.  We just attached both to the
same VLAN.  During u-boot one was used for downloading the kernel, then
once the kernel booted the other was used.  Not ideal, and maybe not
important enough to keep the "feature" around, but it does get used now
and again.

Thanks

Chris 

> 
> -- 
> error compiling committee.c: too many arguments to function
> 
> 
> 
> 

RE: [Qemu-devel] net packet storms with multiple NICs

[Krumme, Chris]

> -----Original Message-----
> From: 
> qemu-devel-bounces+chris.krumme=windriver.com@nongnu.org 
> [mailto:qemu-devel-bounces+chris.krumme=windriver.com@nongnu.o
rg] On Behalf Of Avi Kivity
> Sent: Sunday, October 25, 2009 9:23 AM
> To: Mark McLoughlin
> Cc: Michael Tokarev; qemu-devel@nongnu.org; KVM list
> Subject: Re: [Qemu-devel] net packet storms with multiple NICs
> 
> On 10/23/2009 06:43 PM, Mark McLoughlin wrote:
> > On Fri, 2009-10-23 at 20:25 +0400, Michael Tokarev wrote:
> >    
> >> I've two questions:
> >>
> >> o what's the intended usage of all-vlan-equal case, when 
> kvm (or qemu)
> >>     reflects packets from one interface to another?  It's 
> what bridge
> >>     in linux is for, I think.
> >>      
> > I don't think it's necessarily an intended use-case for the 
> vlan feature
> >    
> 
> Well, it is.  vlan=x really means "the ethernet segment named x".  If 
> you connect all your guest nics to one vlan, you are 
> connecting them all 
> to one ethernet segment, so any packet transmitted on one will be 
> reflected on others.
> 
> Whether this is a useful feature is another matter, but the code is 
> functioning as expected.

Hello,

We had one environment where the NIC understood by u-boot and the NIC
understood by the kernel where different.  We just attached both to the
same VLAN.  During u-boot one was used for downloading the kernel, then
once the kernel booted the other was used.  Not ideal, and maybe not
important enough to keep the "feature" around, but it does get used now
and again.

Thanks

Chris 

> 
> -- 
> error compiling committee.c: too many arguments to function
> 
> 
> 
> 

Re: [Qemu-devel] net packet storms with multiple NICs

[Avi Kivity]

On 10/26/2009 03:40 PM, Krumme, Chris wrote:
>
>> Well, it is.  vlan=x really means "the ethernet segment named x".  If
>> you connect all your guest nics to one vlan, you are
>> connecting them all
>> to one ethernet segment, so any packet transmitted on one will be
>> reflected on others.
>>
>> Whether this is a useful feature is another matter, but the code is
>> functioning as expected.
>>      
> Hello,
>
> We had one environment where the NIC understood by u-boot and the NIC
> understood by the kernel where different.  We just attached both to the
> same VLAN.  During u-boot one was used for downloading the kernel, then
> once the kernel booted the other was used.  Not ideal, and maybe not
> important enough to keep the "feature" around, but it does get used now
> and again.
>    

You could get the same behaviour by using two different vlans connected 
to the same bridge.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] net packet storms with multiple NICs

[Paul Brook]

> I immediately reproduced the problem locally.  It turns out that
> kvm reflects packets coming from one guest NIC on another guest
> NIC, and since both are connected to the same bridge we're getting
> endless packet storm.  To a level when kvm process becomes 100%
> busy and does not respond to anything but `kill -9'.

You created a network loop. It is working exactly as expected.
Create the same topology with a physical network hub and a pair of NICs and 
you'll get the same end result.

Paul

Re: [Qemu-devel] net packet storms with multiple NICs

[Anthony Liguori]

Mark McLoughlin wrote:
> On Fri, 2009-10-23 at 20:25 +0400, Michael Tokarev wrote:
>   
>> I've two questions:
>>
>> o what's the intended usage of all-vlan-equal case, when kvm (or qemu)
>>    reflects packets from one interface to another?  It's what bridge
>>    in linux is for, I think.
>>     
>
> I don't think it's necessarily an intended use-case for the vlan feature
>
>   
>> o why different -net guest -net host pairs are not getting different
>>    vlan= indexes by default, to stop the above-mentioned packet
>>    storms right away?  I think it's a wise default to assign different
>>    pairs to different vlans, by counting -net host and -net guest
>>    sequences.
>>     
>
> With 0.12, we're going to be de-emphasising the vlan feature and instead
> have NICs directly connected to host backends. The vlan feature will be
> just another host backend, but optional
>
> You'll start guests with e.g.:
>
>   -netdev tap,id=tap.0 -device virtio-net-pci,netdev=tap.0
>   

Which is not necessarily more friendly to a user, but it lays the 
foundation to be able to do a config file which in turns could be easily 
generated from a nic GUI/command line tool.

Regards,

Anthony Liguori

Re: [Qemu-devel] net packet storms with multiple NICs

[Michael Tokarev]

Paul Brook wrote:
>> I immediately reproduced the problem locally.  It turns out that
>> kvm reflects packets coming from one guest NIC on another guest
>> NIC, and since both are connected to the same bridge we're getting
>> endless packet storm.  To a level when kvm process becomes 100%
>> busy and does not respond to anything but `kill -9'.
> 
> You created a network loop. It is working exactly as expected.
> Create the same topology with a physical network hub and a pair of NICs and 
> you'll get the same end result.

Paul, I know what the result is.  But mind you, I talked about something
different, namely: why, using "obvious" way, you get a packet storm/loop
instead of a working network.  And if that loop is useful in the first
place.

I've got answers to both already :)

Thanks

/mjt