[Qemu-devel] Questions about networking

[Qemu-devel] Questions about networking

[Peter Niessen]

Dear List,

I'm trying to set up a testbed for batch systems using qemu-kvm. So far,
I've created two machines, a master ("torque") and an execution host
("mom") for use with torque. I'm using the following command lines to
start up the virtual machines:

qemu-kvm -smp 2 -m 768 -hda ./torque.qcow2 -net
nic,vlan=1,macaddr=52:54:00:12:34:56 -net
nic,vlan=2,macaddr=52:54:00:12:34:57 -net user,vlan=2 -net
socket,vlan=1,listen=localhost:1234 -redir tcp:26022::22 -nographic
-daemonize

qemu-kvm -smp 2 -m 768 -hda ./mom.qcow2 -net
nic,vlan=1,macaddr=52:54:00:12:34:58 -net
socket,vlan=1,connect=localhost:1234 -nographic -daemonize

which I took from http://www.h7.dion.ne.jp/~qemu-win/HowToNetwork-en.html.

Everything works fine, I can see the internet from "mom" via "torque"
and NFS mount the users home directory from "torque" on "mom" and
resolve users via NIS.

Here's the ifconfig of the nodes:

torque:~ # ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56
          inet addr:192.168.42.250  Bcast:192.168.42.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:707 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1873 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:44388 (43.3 Kb)  TX bytes:2539091 (2.4 Mb)
          Interrupt:11 Base address:0x2000

eth1      Link encap:Ethernet  HWaddr 52:54:00:12:34:57
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3457/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:69 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7837 (7.6 Kb)  TX bytes:13548 (13.2 Kb)
          Interrupt:10 Base address:0xc000

And "mom":

mom:~ # ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:58
          inet addr:192.168.42.1  Bcast:192.168.42.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3458/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1888 errors:0 dropped:0 overruns:0 frame:0
          TX packets:752 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2514373 (2.3 Mb)  TX bytes:60325 (58.9 Kb)
          Interrupt:11 Base address:0x2000


The ping times between the servers are the following:

torque:~ # ping mom
PING mom.qemu (192.168.42.1) 56(84) bytes of data.
64 bytes from mom.qemu (192.168.42.1): icmp_seq=1 ttl=64 time=39.6 ms
64 bytes from mom.qemu (192.168.42.1): icmp_seq=2 ttl=64 time=39.4 ms
64 bytes from mom.qemu (192.168.42.1): icmp_seq=3 ttl=64 time=39.7 ms
64 bytes from mom.qemu (192.168.42.1): icmp_seq=4 ttl=64 time=39.8 ms
64 bytes from mom.qemu (192.168.42.1): icmp_seq=5 ttl=64 time=39.8 ms
64 bytes from mom.qemu (192.168.42.1): icmp_seq=6 ttl=64 time=39.8 ms
64 bytes from mom.qemu (192.168.42.1): icmp_seq=7 ttl=64 time=39.8 ms

Do these times make sense?

However, batch operations are not working properly. Jobs start fine and
produce the right output, but when it comes to tidying up, the "mom"
machine can't contact the "torque":

Aug  3 10:10:26 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:27 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:28 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:29 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:29 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:29 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:30 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:31 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:32 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:33 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused
Aug  3 10:10:34 mom pbs_mom: LOG_ERROR::Operation now in progress (115)
in scan_for_exiting, cannot connect to port 1023 in client_to_svr -
connection refused


At this time, tcpdump on the "torque" machine says:


10:10:17.072582 IP mom.qemu.1023 > torque.qemu.pbs: Flags [S], seq
25915729, win 5840, options [mss 1460,sackOK,TS val 719328 ecr
0,nop,wscale 6], length 0
10:10:17.072647 IP torque.qemu.pbs > mom.qemu.1023: Flags [S.], seq
18959859, ack 25915730, win 5792, options [mss 1460,sackOK,TS val 756722
ecr 719328,nop,wscale 6], length 0
10:10:17.152568 IP mom.qemu.1023 > torque.qemu.pbs: Flags [R], seq
25915730, win 0, length 0
10:10:18.084234 IP mom.qemu.1023 > torque.qemu.pbs: Flags [S], seq
41724490, win 5840, options [mss 1460,sackOK,TS val 720340 ecr
0,nop,wscale 6], length 0
10:10:18.084297 IP torque.qemu.pbs > mom.qemu.1023: Flags [S.], seq
34766899, ack 41724491, win 5792, options [mss 1460,sackOK,TS val 757734
ecr 720340,nop,wscale 6], length 0
10:10:18.163568 IP mom.qemu.1023 > torque.qemu.pbs: Flags [R], seq
41724491, win 0, length 0
10:10:19.095909 IP mom.qemu.1023 > torque.qemu.pbs: Flags [S], seq
57533379, win 5840, options [mss 1460,sackOK,TS val 721352 ecr
0,nop,wscale 6], length 0
10:10:19.095947 IP torque.qemu.pbs > mom.qemu.1023: Flags [S.], seq
50574033, ack 57533380, win 5792, options [mss 1460,sackOK,TS val 758745
ecr 721352,nop,wscale 6], length 0
10:10:19.175628 IP mom.qemu.1023 > torque.qemu.pbs: Flags [R], seq
57533380, win 0, length 0

netstat says:

torque:~ # netstat | grep 1023
tcp        0      0 torque.qemu:1023        mom.qemu:pbs_mom
TIME_WAIT
tcp        0      0 torque.qemu:1023        mom.qemu:pbs_mom
TIME_WAIT

Might the performance of my internal network conection (192.168.42.0/24)
not be sufficient?

Thanks for your help,

Cheers, Peter.

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Re: [Qemu-devel] Questions about networking

[Mulyadi Santosa]

Hi Peter...

On Tue, Aug 3, 2010 at 15:13, Peter Niessen <p.niessen@fz-juelich.de> wrote:
> Dear List,
>
> I'm trying to set up a testbed for batch systems using qemu-kvm. So far,
> I've created two machines, a master ("torque") and an execution host
> ("mom") for use with torque. I'm using the following command lines to
> start up the virtual machines:
>
> qemu-kvm -smp 2 -m 768 -hda ./torque.qcow2 -net
> nic,vlan=1,macaddr=52:54:00:12:34:56 -net
> nic,vlan=2,macaddr=52:54:00:12:34:57 -net user,vlan=2 -net
> socket,vlan=1,listen=localhost:1234 -redir tcp:26022::22 -nographic
> -daemonize
>
> qemu-kvm -smp 2 -m 768 -hda ./mom.qcow2 -net
> nic,vlan=1,macaddr=52:54:00:12:34:58 -net
> socket,vlan=1,connect=localhost:1234 -nographic -daemonize
>
> which I took from http://www.h7.dion.ne.jp/~qemu-win/HowToNetwork-en.html.

Sorry, I took out most of your messages.


IIRC, I once met the same situation too. And it gets worse when using
non TCP transmission (e.g UDP, multicast etc).

So, uhm, perhaps better to use tun/tap and hook them together, maybe
by simply put them all in same network id but separated from host
network device...




-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

RE: [Qemu-devel] Questions about networking

[Niessen, Peter]

Hello Mulyadi,

thanks, I'll give it a spin. I'm off for the next few days, so I won't have results immediately.

Cheers, have a good weekend,

Peter.
________________________________________
From: Mulyadi Santosa [mulyadi.santosa@gmail.com]
Sent: 07 August 2010 07:37
To: Niessen, Peter
Cc: qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] Questions about networking

Hi Peter...

On Tue, Aug 3, 2010 at 15:13, Peter Niessen <p.niessen@fz-juelich.de> wrote:
> Dear List,
>
> I'm trying to set up a testbed for batch systems using qemu-kvm. So far,
> I've created two machines, a master ("torque") and an execution host
> ("mom") for use with torque. I'm using the following command lines to
> start up the virtual machines:
>
> qemu-kvm -smp 2 -m 768 -hda ./torque.qcow2 -net
> nic,vlan=1,macaddr=52:54:00:12:34:56 -net
> nic,vlan=2,macaddr=52:54:00:12:34:57 -net user,vlan=2 -net
> socket,vlan=1,listen=localhost:1234 -redir tcp:26022::22 -nographic
> -daemonize
>
> qemu-kvm -smp 2 -m 768 -hda ./mom.qcow2 -net
> nic,vlan=1,macaddr=52:54:00:12:34:58 -net
> socket,vlan=1,connect=localhost:1234 -nographic -daemonize
>
> which I took from http://www.h7.dion.ne.jp/~qemu-win/HowToNetwork-en.html.

Sorry, I took out most of your messages.


IIRC, I once met the same situation too. And it gets worse when using
non TCP transmission (e.g UDP, multicast etc).

So, uhm, perhaps better to use tun/tap and hook them together, maybe
by simply put them all in same network id but separated from host
network device...




--
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Re: [Qemu-devel] Questions about networking

[Peter Niessen]

Hello again,

using the hints on

http://blog.cynapses.org/2007/07/12/qemu-kvm-internal-network-setup/

(on openSuSE 11.3) I could get my ping times down to 0.8 ms between the
virtual machines.

Now my torque batch setup works without problems.

Here's my startup script, I include a second interface towards the outside:

qemu-kvm -smp 2 -m 768 -hda ./torque.qcow2 -net
nic,macaddr=52:54:00:12:34:56 -net tap,ifname=kvmnet0,script=no
 -net nic,macaddr=52:54:00:12:34:57 -net user -redir tcp:26022::22

and

qemu-kvm -smp 2 -m 768 -hda ./mom.qcow2 -net
nic,macaddr=52:54:00:12:34:58 -net tap,ifname=kvmnet1,script=no

So it seems you can mix user mode and tapping, although I see a lot of
martian source messages. I would re-configure 52:54:00:12:34:57 to use
another tap/bridge combination.


So, problem solved.

Thanks,

Cheers, Peter.

On 08/07/10 12:42, Niessen, Peter wrote:
> Hello Mulyadi,
>
> thanks, I'll give it a spin. I'm off for the next few days, so I won't have results immediately.
>
> Cheers, have a good weekend,
>
> Peter.
> ________________________________________
> From: Mulyadi Santosa [mulyadi.santosa@gmail.com]
> Sent: 07 August 2010 07:37
> To: Niessen, Peter
> Cc: qemu-devel@nongnu.org
> Subject: Re: [Qemu-devel] Questions about networking
>
> Hi Peter...
>
> On Tue, Aug 3, 2010 at 15:13, Peter Niessen <p.niessen@fz-juelich.de> wrote:
>> Dear List,
>>
>> I'm trying to set up a testbed for batch systems using qemu-kvm. So far,
>> I've created two machines, a master ("torque") and an execution host
>> ("mom") for use with torque. I'm using the following command lines to
>> start up the virtual machines:
>>
>> qemu-kvm -smp 2 -m 768 -hda ./torque.qcow2 -net
>> nic,vlan=1,macaddr=52:54:00:12:34:56 -net
>> nic,vlan=2,macaddr=52:54:00:12:34:57 -net user,vlan=2 -net
>> socket,vlan=1,listen=localhost:1234 -redir tcp:26022::22 -nographic
>> -daemonize
>>
>> qemu-kvm -smp 2 -m 768 -hda ./mom.qcow2 -net
>> nic,vlan=1,macaddr=52:54:00:12:34:58 -net
>> socket,vlan=1,connect=localhost:1234 -nographic -daemonize
>>
>> which I took from http://www.h7.dion.ne.jp/~qemu-win/HowToNetwork-en.html.
>
> Sorry, I took out most of your messages.
>
>
> IIRC, I once met the same situation too. And it gets worse when using
> non TCP transmission (e.g UDP, multicast etc).
>
> So, uhm, perhaps better to use tun/tap and hook them together, maybe
> by simply put them all in same network id but separated from host
> network device...
>
>
>
>
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Re: [Qemu-devel] Questions about networking

[Mulyadi Santosa]

On Thu, Aug 12, 2010 at 16:46, Peter Niessen <p.niessen@fz-juelich.de> wrote:
> Hello again,
>
> using the hints on
>
> http://blog.cynapses.org/2007/07/12/qemu-kvm-internal-network-setup/
>
> (on openSuSE 11.3) I could get my ping times down to 0.8 ms between the
> virtual machines.

Thanks for sharing. I personally greatly appreciate that, since I can
learn new ideas too :)

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com