[Qemu-devel] Multicast network breaks under high load

[Qemu-devel] Multicast network breaks under high load

[Brian Rak]

I have two Ubuntu 14.04 KVM instances (kernel version 
3.16.0-29-generic).  (Running under qemu 2.2.0)  They're connected using 
qemu's 'multicast' network type, and basic connectivity works fine.  I 
can ping between the guests, they can transfer data, etc.

I'll refer to them this way:

guest1 - 10.99.0.10
guest2 - 10.99.0.11

The problem is when I send a large amount of traffic from guest2 to 
guest1, guest2's network stack just stops transmitting packets. This 
issue persists until I reboot guest2 (rebooting from the OS itself is 
enough, I do not need to terminate and relaunch the qemu process).

Running tcpdump from within guest2, it sees incoming ARP packets:

16:09:29.197379 ARP, Request who-has 10.99.0.11 tell 10.99.0.10, length 28
16:09:30.196671 ARP, Request who-has 10.99.0.11 tell 10.99.0.10, length 28

but, it doesn't respond to them (even though the network config is fine):

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP group default qlen 1000
     inet 10.99.0.11/8 brd 10.255.255.255 scope global eth1
        valid_lft forever preferred_lft forever

Once this breaks, if I try to ping guest1 from guest2 I see no bytes 
transmitted (via ifconfig & tcpdump).  If I let ping run for a minute or 
two, I start seeing:

ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available

Things that I've tried, that haven't prevented the issue:

* Kernel upgrade of guest2 to 3.17.1-031701-generic
* Disabling GRO on guest2

Once this breaks, none of the following will fix it:
* ifconfig eth1 down; ifconfig eth1 up

It looks to me like the virtio driver is just giving up on transmitting 
packets for some reason.  I'm not really sure how I would begin 
troubleshooting this.  Any suggestions?

For reference, guest2's network definition looks like this:

-netdev socket,mcast=230.0.0.0:6000,id=hostnet1