xen 2.0, networking, bridging, and nfsroot

xen 2.0, networking, bridging, and nfsroot

[Adam Heath]

Our current setup(based on 1.0 and 1.2) uses completely diskless machines.
They pxeboot, which then runs grub, which then loads xen.  dom0 and domN all
do nfsroot, and everything is happy.

So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
However, the bridge has a hole, where the network does not exist, while it
copies the addresses from eth0 to br0, and changes all the routes.

In nfsroot mode, this fails, as suddenly the network is inaccessible, so
brctl(and friends) can no longer be found.

It would be more safe, to not use a bridge to configure access to the other
domains; this would leave networking running at all times on dom0. Otherwise,
if the bridge setup code breaks, you would end up with dom0 not accessible
from the net, even in normal disk-bound situations.






-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Brian Wolfe]

Interesting. If you are correct, then this will also break my machines.
8-(  Are you sayign that the old method is no longer there? that only
the domain that grabs the net card will have direct access, the rest
must go through the bridge? Not good.

BTW, I've redone the debian/rules file to build any number of cpu-arch
for 2.4 and 2.6 as well as priv and unpriv modes. I'll toss you the
patch once I have it at least building the .deb packages. Probably
tonight or tomorrow morning. I've also changed up a bunch of
dependancies and added xen-dev as well as the individual XenLinux image
headers, source, and standard debian kernel image/modules/etc files.

As for others pulling my debs (if anyone does), I'll be placing the new
2.0-pre debs at the standard terabox.com/debian location in a couple
days for playing with.

It's getting kinda large during compile to create all of the .deb
packages. :) Adam, also need to chat with you on some solutions to the
modules in debian linux domains....


On Tue, 2004-08-10 at 17:29, Adam Heath wrote:
> Our current setup(based on 1.0 and 1.2) uses completely diskless machines.
> They pxeboot, which then runs grub, which then loads xen.  dom0 and domN all
> do nfsroot, and everything is happy.
> 
> So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
> However, the bridge has a hole, where the network does not exist, while it
> copies the addresses from eth0 to br0, and changes all the routes.
> 
> In nfsroot mode, this fails, as suddenly the network is inaccessible, so
> brctl(and friends) can no longer be found.
> 
> It would be more safe, to not use a bridge to configure access to the other
> domains; this would leave networking running at all times on dom0. Otherwise,
> if the bridge setup code breaks, you would end up with dom0 not accessible
> from the net, even in normal disk-bound situations.
> 
> 
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Adam Heath]

On Tue, 10 Aug 2004, Brian Wolfe wrote:

> Interesting. If you are correct, then this will also break my machines.
> 8-(  Are you sayign that the old method is no longer there? that only
> the domain that grabs the net card will have direct access, the rest
> must go through the bridge? Not good.

The ones going thru the bridge can still do dhcp, if that's what you're
asking.

The problem is that the domain with real access to the nic looses access to
the nic momentarily while the bridge is being configured.  This breaks nfsroot
setups.



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Ian Pratt]

> So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
> However, the bridge has a hole, where the network does not exist, while it
> copies the addresses from eth0 to br0, and changes all the routes.
> 
> In nfsroot mode, this fails, as suddenly the network is inaccessible, so
> brctl(and friends) can no longer be found.

It's an unfortunate mis-feature of the Linux bridge code that
when adding an interface to the bridge it doesn't inherit the IP
addresses associated with the interface: As I recall, one of
either tx or rx breaks, but the other direction is OK.

I presume 2.6 exhibits the same behaviour? 

We've tried to work around this in the /etc/xen/network script,
but it's certainly a problem for nfsroot dom0 systems. 

One option is to come up with a patch to the linux bridge code to
'fix' the current arguably broken behaviour. It would be
interesting to take this up with the bridge code maintainer. Any
volumteers?

The other alternative is to route rather than bridge VIF's onto
the real network. We've supplied example scripts for bridging,
but it would be good to include example scripts for a routed
setup too. 

Just edit the network-script and vif-script parameters in
/etc/xen/xend-config.sxp to point at a pair of new scripts.

I've had routed setups working just fine. The only slight
annoyance is that I had to configure a dummy IP address for the
backend (vifX.Y) interfaces to point routes through. I was hoping
to set them up as explicit point-to-point links and avoid this,
but ifconfig wouldn't let me. Perhaps there's some device flag
that our backend driver should be setting to allow this? If so, a
patch for this would be great.

Thanks,
Ian



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Sachin Goyal]

I have never tried pxeboot or diskless linux, but I guess you can include
bridge tools (e.g brctl) in initrd image and configure the network card initially
itself in bridge mode before nfs mounting the real root.

sachin


On Tue, 10 Aug 2004, Adam Heath wrote:

> Our current setup(based on 1.0 and 1.2) uses completely diskless machines.
> They pxeboot, which then runs grub, which then loads xen.  dom0 and domN all
> do nfsroot, and everything is happy.
>
> So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
> However, the bridge has a hole, where the network does not exist, while it
> copies the addresses from eth0 to br0, and changes all the routes.
>
> In nfsroot mode, this fails, as suddenly the network is inaccessible, so
> brctl(and friends) can no longer be found.
>
> It would be more safe, to not use a bridge to configure access to the other
> domains; this would leave networking running at all times on dom0. Otherwise,
> if the bridge setup code breaks, you would end up with dom0 not accessible
> from the net, even in normal disk-bound situations.
>
>
>
>
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Brian Wolfe]

Hmm, that is a good idea. I have been pursuing the idea of a stripped
down linux OS just for running as the Domain0. I'll be taking the base
from the Debian install CD and adding the necessary libraries and
whatnot that Domain0 requires to perform it's duties as the privileged
domain. I have a partially working stripped down copy of the standard
Debian unstable base install on a test server. Still needs work to
reduce it further so that it will fit into a small initrd.

 The end goal is to have Domain0 not need ANY network or physical disk
access. The config files for xen domains would be pulled either from a
http based config server, or loaded from a temporarily mounted source at
boot time, or pre-exist int he initrd image.

Don't expect an image for several weeks. 8-P I've got my hands full
already getting the latest Xen packaged for Debian unstable and iSCSI
packaged since I could not find any packages pre-made.

The good news is that I have a working debian/rules file for compiling
all of the targets and the start of .config files for 2.6.7 priv and
unpriv. I'll probably toss up some very broken .deb files Wednesday
night (CST). I'm certain that Adam will find PLENTY of packaging flaws
for me to fix up. :) I'va mangled his build scripts quite well by now.
*grin*

On Tue, 2004-08-10 at 22:05, Sachin Goyal wrote:
> I have never tried pxeboot or diskless linux, but I guess you can include
> bridge tools (e.g brctl) in initrd image and configure the network card initially
> itself in bridge mode before nfs mounting the real root.
> 
> sachin
> 
> 
> On Tue, 10 Aug 2004, Adam Heath wrote:
> 
> > Our current setup(based on 1.0 and 1.2) uses completely diskless machines.
> > They pxeboot, which then runs grub, which then loads xen.  dom0 and domN all
> > do nfsroot, and everything is happy.
> >
> > So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
> > However, the bridge has a hole, where the network does not exist, while it
> > copies the addresses from eth0 to br0, and changes all the routes.
> >
> > In nfsroot mode, this fails, as suddenly the network is inaccessible, so
> > brctl(and friends) can no longer be found.
> >
> > It would be more safe, to not use a bridge to configure access to the other
> > domains; this would leave networking running at all times on dom0. Otherwise,
> > if the bridge setup code breaks, you would end up with dom0 not accessible
> > from the net, even in normal disk-bound situations.
> >
> >
> >
> >
> >
> >
> > -------------------------------------------------------
> > SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> > 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> > Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> > http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Wm]

On Tue, Aug 10, 2004 at 05:29:30PM -0500, Adam Heath wrote:
> Our current setup(based on 1.0 and 1.2) uses completely diskless machines.
> They pxeboot, which then runs grub, which then loads xen.  dom0 and domN all
> do nfsroot, and everything is happy.
> 
> So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
> However, the bridge has a hole, where the network does not exist, while it
> copies the addresses from eth0 to br0, and changes all the routes.

Can you use initramfs to configure the bridge/networking and mount the rootfs?

My work arround was to add and configure the bridged interface at boot
time, also i found the script for copying over the IP addresses
was never reliable especailly when multiple addresses had been added
to the old interface, and it made no attempt to deal with v6 addresses
and routing completely breaks.

> In nfsroot mode, this fails, as suddenly the network is inaccessible, so
> brctl(and friends) can no longer be found.

Can you run brctl before you mount the rootfs via nfs?


-- 
Bill Boughton


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

RE: xen 2.0, networking, bridging, and nfsroot

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 3442 bytes --]

Isn't the misfeature you describe a function of the brctl utilities? Couldn't they be modified to add an option to 'brctl addif', eg 'inherit' so to make br0 inherit eth0's address you might use this syntax:

brctl addif br0 eth0 inherit

brctl could then internally do it this way:
1. create bridge br0
2. give br0 eth0's ip address
3. up br0
4. add eth0 to br0
5. remove address from eth0

but maybe the problem is that 3+4+5 need to be done in an atomic operation and there's no way to do that from userspace.

another solution which would maybe solve the nfsroot problem is to have the kernel configure a bridge at boot time via kernel parameters, eg bridge=br0,eth0 and then do everything on br0 from the start rather than starting with eth0 and moving to br0 at some later stage.

failing all of that, initrd might solve all the problems.

I think it would be incorrect to just always automatically add the ip address of an interface being added though.

James


From: Ian Pratt
Sent: Wed 11/08/2004 10:43 AM
To: brianw@terrabox.com; Adam Heath; xen-devel@lists.sourceforge.net
Cc: Ian.Pratt@cl.cam.ac.uk
Subject: Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot


> So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
> However, the bridge has a hole, where the network does not exist, while it
> copies the addresses from eth0 to br0, and changes all the routes.
> 
> In nfsroot mode, this fails, as suddenly the network is inaccessible, so
> brctl(and friends) can no longer be found.

It's an unfortunate mis-feature of the Linux bridge code that
when adding an interface to the bridge it doesn't inherit the IP
addresses associated with the interface: As I recall, one of
either tx or rx breaks, but the other direction is OK.

I presume 2.6 exhibits the same behaviour? 

We've tried to work around this in the /etc/xen/network script,
but it's certainly a problem for nfsroot dom0 systems. 

One option is to come up with a patch to the linux bridge code to
'fix' the current arguably broken behaviour. It would be
interesting to take this up with the bridge code maintainer. Any
volumteers?

The other alternative is to route rather than bridge VIF's onto
the real network. We've supplied example scripts for bridging,
but it would be good to include example scripts for a routed
setup too. 

Just edit the network-script and vif-script parameters in
/etc/xen/xend-config.sxp to point at a pair of new scripts.

I've had routed setups working just fine. The only slight
annoyance is that I had to configure a dummy IP address for the
backend (vifX.Y) interfaces to point routes through. I was hoping
to set them up as explicit point-to-point links and avoid this,
but ifconfig wouldn't let me. Perhaps there's some device flag
that our backend driver should be setting to allow this? If so, a
patch for this would be great.

Thanks,
Ian



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

[-- Attachment #2: Type: text/html, Size: 4775 bytes --]

Re: xen 2.0, networking, bridging, and nfsroot

[Ian Pratt]

> Isn't the misfeature you describe a function of the brctl utilities? Couldn't they be modified to add an option to 'brctl addif', eg 'inherit' so to make br0 inherit eth0's address you might use this syntax:

The current situation as implemented by the bridge code is
inconsistent (at least in my view).

If I create a bridge and add eth0 to it, I can still send packets
via eth0 with the original IP address -- I just can't receive
them until I move the IP address to the bridge.

> brctl could then internally do it this way:
> 1. create bridge br0
> 2. give br0 eth0's ip address
> 3. up br0
> 4. add eth0 to br0
> 5. remove address from eth0
> 
> but maybe the problem is that 3+4+5 need to be done in an atomic operation and there's no way to do that from userspace.

I'm not sure how atomic it would need to be to keep nfsroot happy
-- certainly all the pages of the brctl binary and libraries had
better already be paged in.

Also, you need to worry about transferring all the routes that
were pointing through eth0...
 
> another solution which would maybe solve the nfsroot problem is to have the kernel configure a bridge at boot time via kernel parameters, eg bridge=br0,eth0 and then do everything on br0 from the start rather than starting with eth0 and moving to br0 at some later stage.

That could possibly work, though I believe such boot time options
are becoming deprecated in favour of initrd's in 2.6.

> I think it would be incorrect to just always automatically add the ip address of an interface being added though.

You haven't totally convinced me ;-)  The fact that you can still
TX makes me think it might just be a bug.

Has anyone got the time/inclination to take this up with
<bridge@osdl.org> or search the archives ?
http://lists.osdl.org/pipermail/bridge/


Ian


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Derek Glidden]

On Aug 10, 2004, at 6:29 PM, Adam Heath wrote:

> Our current setup(based on 1.0 and 1.2) uses completely diskless 
> machines.
> They pxeboot, which then runs grub, which then loads xen.  dom0 and 
> domN all
> do nfsroot, and everything is happy.
>
> So, along comes 2.0.  It now uses a normal bridge to connect dom0 and 
> domN.
> However, the bridge has a hole, where the network does not exist, 
> while it
> copies the addresses from eth0 to br0, and changes all the routes.
>
> In nfsroot mode, this fails, as suddenly the network is inaccessible, 
> so
> brctl(and friends) can no longer be found.
>
> It would be more safe, to not use a bridge to configure access to the 
> other
> domains; this would leave networking running at all times on dom0. 
> Otherwise,
> if the bridge setup code breaks, you would end up with dom0 not 
> accessible
> from the net, even in normal disk-bound situations.

The bridge setup is just a function of the network script that xend 
calls when you start it.  You can easily change the network script to 
do "something else".  There's nothing inherent in the xen kernel or 
xend (AFAIK; I haven't had a lot of time to play with it for a couple 
of weeks) that requires you use bridging to do your network stuff, or 
that you do it specifically the way it comes out of the box.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
"I think that's what they mean by   |
"nickels a day can feed a child."   |       http://www.eff.org/
I thought, "How can food be so      | http://www.anti-dmca.org/
cheap over there?"  It's not, they  |--------------------------
just eat the nickels." -- Peter Nguyen


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
"I think that's what they mean by   |
"nickels a day can feed a child."   |       http://www.eff.org/
I thought, "How can food be so      | http://www.anti-dmca.org/
cheap over there?"  It's not, they  |--------------------------
just eat the nickels." -- Peter Nguyen



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: xen 2.0, networking, bridging, and nfsroot

[Ian Pratt]

> On Tue, Aug 10, 2004 at 05:29:30PM -0500, Adam Heath wrote:
> > Our current setup(based on 1.0 and 1.2) uses completely diskless machines.
> > They pxeboot, which then runs grub, which then loads xen.  dom0 and domN all
> > do nfsroot, and everything is happy.
> > 
> > So, along comes 2.0.  It now uses a normal bridge to connect dom0 and domN.
> > However, the bridge has a hole, where the network does not exist, while it
> > copies the addresses from eth0 to br0, and changes all the routes.
> 
> Can you use initramfs to configure the bridge/networking and mount the rootfs?

This should work fine.  I believe using an initramfs is now the
preferred way to setup things like NFS root these days, though
using the in-kernel functions is certainly more convenient.
 
> My work arround was to add and configure the bridged interface at boot
> time, also i found the script for copying over the IP addresses
> was never reliable especailly when multiple addresses had been added
> to the old interface, and it made no attempt to deal with v6 addresses
> and routing completely breaks.

The script manages to copy multiple IPv4 addresses for me -- I'd
be grateful for a an example of it failing.

Fixing it for IPv6 shouldn't be hard, particularly if you could
post the output of 'ip addr show ethX' and 'ip route list'. 

Ian


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285