HVM guest sees two NICs even though just one vif configured

HVM guest sees two NICs even though just one vif configured

[Fischer, Anna]

I am running SLES10 with Xen 3.2 and I have configured an HVM guest. 

I configured the guest with a single virtual NIC:

vif = [ 'bridge=eth0, mac=00:16:3e:00:31:13' ]

I have not configured type=ioemu or type=netfront here, as you can see.

Now what happens is that on Dom0 Xen creates a tap device connecting to QEMU emulating a RTL 8139 NIC, and additionally a vifX device for the traditional PV approach. Both devices are connected to the bridge eth0.

Now within my HVM guest I can see the RTL 8139 under lspci. However, I also have the xen-vnif driver installed inside the guest OS as it provides much better performance. On boot xen-vnif successfully connects the netfront to the netback driver and basically I have a proper PV connection that my guest can now use. It shows up as say eth0. At this point the guest will use eth0 and the PV drivers to do networking.

But still the guests sees as second NIC, the RTL 8139, and if I load the 8139too driver inside the guest, then the guest can additionally activate that second NIC, say as eth1. By default it is configured with the same MAC address as eth0 but I can easily change that within the guest OS. From that point on, my guest has two working NICs, with different MACs and so different IPs. 

For me this does not sound right. If I configure my guest with only a single NIC in the configuration file, then there should not be any way for the guest to set up and use two NICs at the same time. 

Is this a known issue? How is this supposed to be handled?

Thanks,
Anna


My Xen version:

xm info
release                : 2.6.16.60-0.21-xen
version                : #1 SMP Tue May 6 12:41:02 UTC 2008
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2700
hw_caps                : 178bf3ff:efc3fbff:00000000:00000110:00802001:00000000:000037ff
total_memory           : 65533
free_memory            : 4141
max_free_memory        : 54085
max_para_memory        : 54081
max_hvm_memory         : 53946
node_to_cpu            : node0:0-7
xen_major              : 3
xen_minor              : 2
xen_extra              : .0_16718_14-0.4
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 16718
cc_compiler            : gcc version 4.1.2 20070115 (SUSE Linux)
cc_compile_by          : abuild
cc_compile_domain      : suse.de
cc_compile_date        : Tue May  6 15:20:41 UTC 2008
xend_config_format     : 4

Re: HVM guest sees two NICs even though just one vif configured

[Keir Fraser]

On 06/08/2009 10:32, "Fischer, Anna" <anna.fischer@hp.com> wrote:

> For me this does not sound right. If I configure my guest with only a single
> NIC in the configuration file, then there should not be any way for the guest
> to set up and use two NICs at the same time.
> 
> Is this a known issue? How is this supposed to be handled?

I think it's deliberate, to allow install of an OS with emulated driver,
followed by switchover to PV drivers, wth no need for config changes.

 -- Keir

RE: HVM guest sees two NICs even though just one vif configured

[Fischer, Anna]

> Subject: Re: [Xen-devel] HVM guest sees two NICs even though just one
> vif configured
> 
> On 06/08/2009 10:32, "Fischer, Anna" <anna.fischer@hp.com> wrote:
> 
> > For me this does not sound right. If I configure my guest with only a
> single
> > NIC in the configuration file, then there should not be any way for
> the guest
> > to set up and use two NICs at the same time.
> >
> > Is this a known issue? How is this supposed to be handled?
> 
> I think it's deliberate, to allow install of an OS with emulated
> driver,
> followed by switchover to PV drivers, wth no need for config changes.

I assumed that was the reason behind this. However, a switchover would need to guarantee that a guest cannot use two NICs at the same time, doesn't it? Because otherwise the guest can send out of two separate devices at the same time and it could cause unexpected behaviour unless it is explicitly prevented at the backends in Dom0, e.g. through filtering or anti-spoofing I guess.

Anna

RE: HVM guest sees two NICs even though just one vif configured

[Stefano Stabellini]

On Thu, 6 Aug 2009, Fischer, Anna wrote:
> > Subject: Re: [Xen-devel] HVM guest sees two NICs even though just one
> > vif configured
> > 
> > On 06/08/2009 10:32, "Fischer, Anna" <anna.fischer@hp.com> wrote:
> > 
> > > For me this does not sound right. If I configure my guest with only a
> > single
> > > NIC in the configuration file, then there should not be any way for
> > the guest
> > > to set up and use two NICs at the same time.
> > >
> > > Is this a known issue? How is this supposed to be handled?
> > 
> > I think it's deliberate, to allow install of an OS with emulated
> > driver,
> > followed by switchover to PV drivers, wth no need for config changes.
> 
> I assumed that was the reason behind this. However, a switchover would need to guarantee that a guest cannot use two NICs at the same time, doesn't it? Because otherwise the guest can send out of two separate devices at the same time and it could cause unexpected behaviour unless it is explicitly prevented at the backends in Dom0, e.g. through filtering or anti-spoofing I guess.
> 

I more recent qemu allows the pv drivers in the guest to disconnect the
emulated devices to avoid these problems.

RE: HVM guest sees two NICs even though just one vif configured

[Fischer, Anna]

> Subject: RE: [Xen-devel] HVM guest sees two NICs even though just one
> vif configured
> 
> On Thu, 6 Aug 2009, Fischer, Anna wrote:
> > > Subject: Re: [Xen-devel] HVM guest sees two NICs even though just
> one
> > > vif configured
> > >
> > > On 06/08/2009 10:32, "Fischer, Anna" <anna.fischer@hp.com> wrote:
> > >
> > > > For me this does not sound right. If I configure my guest with
> only a
> > > single
> > > > NIC in the configuration file, then there should not be any way
> for
> > > the guest
> > > > to set up and use two NICs at the same time.
> > > >
> > > > Is this a known issue? How is this supposed to be handled?
> > >
> > > I think it's deliberate, to allow install of an OS with emulated
> > > driver,
> > > followed by switchover to PV drivers, wth no need for config
> changes.
> >
> > I assumed that was the reason behind this. However, a switchover
> would need to guarantee that a guest cannot use two NICs at the same
> time, doesn't it? Because otherwise the guest can send out of two
> separate devices at the same time and it could cause unexpected
> behaviour unless it is explicitly prevented at the backends in Dom0,
> e.g. through filtering or anti-spoofing I guess.
> >
> 
> I more recent qemu allows the pv drivers in the guest to disconnect the
> emulated devices to avoid these problems.

I am not too confident with having to rely on the guest to make such a disconnect happen. Is there no way to do this in Xen/Dom0?

RE: HVM guest sees two NICs even though just one vif configured

[Stefano Stabellini]

On Thu, 6 Aug 2009, Fischer, Anna wrote:
> > Subject: RE: [Xen-devel] HVM guest sees two NICs even though just one
> > vif configured
> > 
> > On Thu, 6 Aug 2009, Fischer, Anna wrote:
> > > > Subject: Re: [Xen-devel] HVM guest sees two NICs even though just
> > one
> > > > vif configured
> > > >
> > > > On 06/08/2009 10:32, "Fischer, Anna" <anna.fischer@hp.com> wrote:
> > > >
> > > > > For me this does not sound right. If I configure my guest with
> > only a
> > > > single
> > > > > NIC in the configuration file, then there should not be any way
> > for
> > > > the guest
> > > > > to set up and use two NICs at the same time.
> > > > >
> > > > > Is this a known issue? How is this supposed to be handled?
> > > >
> > > > I think it's deliberate, to allow install of an OS with emulated
> > > > driver,
> > > > followed by switchover to PV drivers, wth no need for config
> > changes.
> > >
> > > I assumed that was the reason behind this. However, a switchover
> > would need to guarantee that a guest cannot use two NICs at the same
> > time, doesn't it? Because otherwise the guest can send out of two
> > separate devices at the same time and it could cause unexpected
> > behaviour unless it is explicitly prevented at the backends in Dom0,
> > e.g. through filtering or anti-spoofing I guess.
> > >
> > 
> > I more recent qemu allows the pv drivers in the guest to disconnect the
> > emulated devices to avoid these problems.
> 
> I am not too confident with having to rely on the guest to make such a disconnect happen. Is there no way to do this in Xen/Dom0?
>

The idea is that the pv drivers should load only after having
disabled the emulated devices.
After all if the pv drivers are badly written you can end up with a
corrupted disk anyway.

Re: HVM guest sees two NICs even though just one vif configured

[Keir Fraser]

On 06/08/2009 12:12, "Fischer, Anna" <anna.fischer@hp.com> wrote:

>> I more recent qemu allows the pv drivers in the guest to disconnect the
>> emulated devices to avoid these problems.
> 
> I am not too confident with having to rely on the guest to make such a
> disconnect happen. Is there no way to do this in Xen/Dom0?

The disconnect would be triggered by the PV drivers you need in order to be
able to use the PV devices. You generally want to disconnect the emulated
devcies only when the PV drivers are ready, so it's the sensibel place to
trigger disconnect.

 -- Keir

RE: HVM guest sees two NICs even though just one vif configured

[Fischer, Anna]

> Subject: RE: [Xen-devel] HVM guest sees two NICs even though just one
> vif configured
> 
> On Thu, 6 Aug 2009, Fischer, Anna wrote:
> > > Subject: RE: [Xen-devel] HVM guest sees two NICs even though just
> one
> > > vif configured
> > >
> > > On Thu, 6 Aug 2009, Fischer, Anna wrote:
> > > > > Subject: Re: [Xen-devel] HVM guest sees two NICs even though
> just
> > > one
> > > > > vif configured
> > > > >
> > > > > On 06/08/2009 10:32, "Fischer, Anna" <anna.fischer@hp.com>
> wrote:
> > > > >
> > > > > > For me this does not sound right. If I configure my guest
> with
> > > only a
> > > > > single
> > > > > > NIC in the configuration file, then there should not be any
> way
> > > for
> > > > > the guest
> > > > > > to set up and use two NICs at the same time.
> > > > > >
> > > > > > Is this a known issue? How is this supposed to be handled?
> > > > >
> > > > > I think it's deliberate, to allow install of an OS with
> emulated
> > > > > driver,
> > > > > followed by switchover to PV drivers, wth no need for config
> > > changes.
> > > >
> > > > I assumed that was the reason behind this. However, a switchover
> > > would need to guarantee that a guest cannot use two NICs at the
> same
> > > time, doesn't it? Because otherwise the guest can send out of two
> > > separate devices at the same time and it could cause unexpected
> > > behaviour unless it is explicitly prevented at the backends in
> Dom0,
> > > e.g. through filtering or anti-spoofing I guess.
> > > >
> > >
> > > I more recent qemu allows the pv drivers in the guest to disconnect
> the
> > > emulated devices to avoid these problems.
> >
> > I am not too confident with having to rely on the guest to make such
> a disconnect happen. Is there no way to do this in Xen/Dom0?
> >
> 
> The idea is that the pv drivers should load only after having
> disabled the emulated devices.
> After all if the pv drivers are badly written you can end up with a
> corrupted disk anyway.

A corrupted PV NIC cannot that easily send packet through. Having the guest transmitting through two interfaces while only one is "officially" configured on the system is a different scenario though, I would say. I am not saying it is not an (easily) solvable issue. But it does require some mechanisms to be set up in Dom0 explicitly. The running instantiation of the guest (having two NICs) simply differs from what is specified on the system (my config file says the guest has one NIC). That inconsistency might cause problems.

RE: HVM guest sees two NICs even though just one vif configured

[Fischer, Anna]

> Subject: Re: [Xen-devel] HVM guest sees two NICs even though just one
> vif configured
> 
> On 06/08/2009 12:12, "Fischer, Anna" <anna.fischer@hp.com> wrote:
> 
> >> I more recent qemu allows the pv drivers in the guest to disconnect
> the
> >> emulated devices to avoid these problems.
> >
> > I am not too confident with having to rely on the guest to make such
> a
> > disconnect happen. Is there no way to do this in Xen/Dom0?
> 
> The disconnect would be triggered by the PV drivers you need in order
> to be
> able to use the PV devices. You generally want to disconnect the
> emulated
> devcies only when the PV drivers are ready, so it's the sensibel place
> to
> trigger disconnect.

Yes, I understand that. But at least on the backend side, shouldn't there only be a single device associated with the guest if I configured a single NIC for my guest? Because otherwise, how do I know which one is active if the guest can switch back and forth between them? I don't think that having the two devices in Dom0 is an ideal solution for other management components on the system.

Re: HVM guest sees two NICs even though just one vif configured

[Ky Srinivasan]

>>> On 8/6/2009 at  5:32 AM, in message
<0199E0D51A61344794750DC57738F58E6D6A501125@GVW1118EXC.americas.hpqcorp.net>,
"Fischer, Anna" <anna.fischer@hp.com> wrote: 
> I am running SLES10 with Xen 3.2 and I have configured an HVM guest. 
What version of sles10 are you installing? In sles10 sp3, if you install with our tools you automatically get PV drivers (both NIC and disk) enabled. Our tools will generate the appropriate config file (with type=netfront) and appropriate kernel boot option to disable probing of the hard disk. Prior to sles10 SP3, one had to manually enable PV drivers after installing using the emulated devices.

Regards,

K. Y
> 
> I configured the guest with a single virtual NIC:
> 
> vif = [ 'bridge=eth0, mac=00:16:3e:00:31:13' ]
> 
> I have not configured type=ioemu or type=netfront here, as you can see.
> 
> Now what happens is that on Dom0 Xen creates a tap device connecting to QEMU 
> emulating a RTL 8139 NIC, and additionally a vifX device for the traditional 
> PV approach. Both devices are connected to the bridge eth0.
> 
> Now within my HVM guest I can see the RTL 8139 under lspci. However, I also 
> have the xen-vnif driver installed inside the guest OS as it provides much 
> better performance. On boot xen-vnif successfully connects the netfront to the 
> netback driver and basically I have a proper PV connection that my guest can 
> now use. It shows up as say eth0. At this point the guest will use eth0 and 
> the PV drivers to do networking.
> 
> But still the guests sees as second NIC, the RTL 8139, and if I load the 
> 8139too driver inside the guest, then the guest can additionally activate 
> that second NIC, say as eth1. By default it is configured with the same MAC 
> address as eth0 but I can easily change that within the guest OS. From that 
> point on, my guest has two working NICs, with different MACs and so different 
> IPs. 
> 
> For me this does not sound right. If I configure my guest with only a single 
> NIC in the configuration file, then there should not be any way for the guest 
> to set up and use two NICs at the same time. 
> 
> Is this a known issue? How is this supposed to be handled?
> 
> Thanks,
> Anna
> 
> 
> My Xen version:
> 
> xm info
> release                : 2.6.16.60-0.21-xen
> version                : #1 SMP Tue May 6 12:41:02 UTC 2008
> machine                : x86_64
> nr_cpus                : 8
> nr_nodes               : 1
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 2700
> hw_caps                : 
> 178bf3ff:efc3fbff:00000000:00000110:00802001:00000000:000037ff
> total_memory           : 65533
> free_memory            : 4141
> max_free_memory        : 54085
> max_para_memory        : 54081
> max_hvm_memory         : 53946
> node_to_cpu            : node0:0-7
> xen_major              : 3
> xen_minor              : 2
> xen_extra              : .0_16718_14-0.4
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
> hvm-3.0-x86_32p hvm-3.0-x86_64 
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : 16718
> cc_compiler            : gcc version 4.1.2 20070115 (SUSE Linux)
> cc_compile_by          : abuild
> cc_compile_domain      : suse.de
> cc_compile_date        : Tue May  6 15:20:41 UTC 2008
> xend_config_format     : 4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: HVM guest sees two NICs even though just one vif configured

[Fischer, Anna]

> Subject: Re: [Xen-devel] HVM guest sees two NICs even though just one
> vif configured
> 
> 
> 
> >>> On 8/6/2009 at  5:32 AM, in message
> <0199E0D51A61344794750DC57738F58E6D6A501125@GVW1118EXC.americas.hpqcorp
> .net>,
> "Fischer, Anna" <anna.fischer@hp.com> wrote:
> > I am running SLES10 with Xen 3.2 and I have configured an HVM guest.
> What version of sles10 are you installing? In sles10 sp3, if you
> install with our tools you automatically get PV drivers (both NIC and
> disk) enabled. Our tools will generate the appropriate config file
> (with type=netfront) and appropriate kernel boot option to disable
> probing of the hard disk. Prior to sles10 SP3, one had to manually
> enable PV drivers after installing using the emulated devices.

But how can the tools in Dom0 know whether your guest runs PV drivers or not? This assumes that your guest runs SLES10 as well and has your PV driver packages installed. Only then you can set type=netfront. Otherwise, if you set type=netfront, and your guest does not have PV drivers installed, then your guest will not have any networking device that works...

You are assuming that you know what the guest has installed, only then you can set the type=[netfront|ioemu] correctly. In my case though, I don't know what the guest runs, so I would need it to be detected automatically. It works to some extend right now, and it can switch between emulated NIC and PV NIC automatically, if the guest does it properly. However, if the guest does not do it right (on purpose or by mistake), then I suddenly have a guest with two active NICs attached.

Anna

RE: HVM guest sees two NICs even though just one vifconfigured

[Ky Srinivasan]

>>> On 8/6/2009 at 11:27 AM, in message
<0199E0D51A61344794750DC57738F58E6D6A5013C9@GVW1118EXC.americas.hpqcorp.net>,
"Fischer, Anna" <anna.fischer@hp.com> wrote: 
>>  Subject: Re: [Xen-devel] HVM guest sees two NICs even though just one
>> vif configured
>> 
>> 
>> 
>> >>> On 8/6/2009 at  5:32 AM, in message
>> <0199E0D51A61344794750DC57738F58E6D6A501125@GVW1118EXC.americas.hpqcorp
>> .net>,
>> "Fischer, Anna" <anna.fischer@hp.com> wrote:
>> > I am running SLES10 with Xen 3.2 and I have configured an HVM guest.
>> What version of sles10 are you installing? In sles10 sp3, if you
>> install with our tools you automatically get PV drivers (both NIC and
>> disk) enabled. Our tools will generate the appropriate config file
>> (with type=netfront) and appropriate kernel boot option to disable
>> probing of the hard disk. Prior to sles10 SP3, one had to manually
>> enable PV drivers after installing using the emulated devices.
> 
> But how can the tools in Dom0 know whether your guest runs PV drivers or 
> not? This assumes that your guest runs SLES10 as well and has your PV driver 
> packages installed. Only then you can set type=netfront. Otherwise, if you 
> set type=netfront, and your guest does not have PV drivers installed, then 
> your guest will not have any networking device that works...

When you install SLES11 or sles10 sp3 as a hvm guest, we automatically install the guest with the PV drivers enabled in the guest since PV drivers give you the best user experience for HVM guests.
> 
> You are assuming that you know what the guest has installed, only then you 
> can set the type=[netfront|ioemu] correctly.
When you install sles11 or sles10 sp3 as hvm guest on our platform (sles dom0), we will by default enable PV drivers.

Regards,

K. Y

> In my case though, I don't know 
> what the guest runs, so I would need it to be detected automatically. It 
> works to some extend right now, and it can switch between emulated NIC and PV 
> NIC automatically, if the guest does it properly. However, if the guest does 
> not do it right (on purpose or by mistake), then I suddenly have a guest with 
> two active NICs attached.
> 
> Anna
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: HVM guest sees two NICs even though just one vif configured

[Ian Jackson]

Fischer, Anna writes ("RE: [Xen-devel] HVM guest sees two NICs even though just one vif configured"):
> I am not too confident with having to rely on the guest to make such
> a disconnect happen. Is there no way to do this in Xen/Dom0?

>From a security point of view, you should tread both virtual
interfaces the same way - as controlled by the (untrusted) guest.  If
you do that it won't matter which interface the guest uses or even if
it is confused and uses both.

Fischer, Anna writes ("RE: [Xen-devel] HVM guest sees two NICs even though just one vif configured"):
> Yes, I understand that. But at least on the backend side, shouldn't
> there only be a single device associated with the guest if I
> configured a single NIC for my guest?

No, because that's not the way it's implemented.  Because of the
possibility of both HVM emulation, and PV drivers, it is necessary to
do some multiplexing.  Rather than write our own custom multiplexer
(which would, ultimately, be a kind of braindead ethernet switch) we
chose to use the dom0's kernel which already has all the necessary
functionality.  I think this is a sensible design decision.

>  Because otherwise, how do I know which one is active if the guest
> can switch back and forth between them?

You need to treat both as potentially active.

> I don't think that having the two devices in Dom0 is
> an ideal solution for other management components on the system.

>From the point of view of management tools, this situation is little
different to the guest having two network interfaces for redundancy,
with automatic failover.

Ian.