Re: UML is now self-hosting!

Re: UML is now self-hosting!

[Pavel Machek]

Hi!

> Yup.  And that sort of thing doesn't come close to reaching the potential
> of a virtual OS.
> 
> A couple of the intermediate-term things I'm most interested in are
> 	embedding UML in things like Apache to provide a standard internal
> 	development and execution environment

This is little perverted, right? What is it good for?

> 	spreading a SMP UML instance across multiple hosts

Now this looks very interesting to me.
								Pavel

-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

UML is now self-hosting!

[Jeff Dike]

UML is now able to run nested inside itself.  This works as of UML 2.4.18-21,
which isn't released yet, but will be soon.  See the log below for the gory 
details, and also see http://user-mode-linux.sf.net/nesting.html for how to
do it yourself.

This is a sign of UML maturity rather than any new magic functionality having
been added.  UML is a demanding process, so even though it uses only the Linux
system call interface, it takes some maturity for a Linux kernel to host it.

The missing pieces were a couple of signal delivery bugs that were still 
lurking.  Once these were fixed, UML booted right up.

				Jeff

Here's the log of the first nested boot.

Notes:
	'usermode:~#' is the UML shell prompt. 
	/dev/ubd/1 has been attached to a tomsrtbt image on the host
	~/linux/2.4/nest/linux is the specially build nested UML. 

Also note the back-to-back UML shutdowns at the very end. 

usermode:~# scp jdike@192.168.0.254:linux/2.4/nest/linux .
jdike@192.168.0.254's password: 
scp: warning: Executing scp1 compatibility.
linux                100% |*****************************|  8227 KB    00:00 ETA
usermode:~# ls -l linux
-rwxr--r--    1 root     root      8424800 May  3 21:09 linux
usermode:~# ./linux ubd0=/dev/ubd/1
tracing thread pid = 146
Linux version 2.4.18-21um (jdike@uml.karaya.com) (gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-81)) #1 Wed May 1 21:07:32 EDT 2002
On node 0 totalpages: 8192
zone(0): 0 pages.
zone(1): 8192 pages.
zone(2): 0 pages.
Kernel command line: ubd0=/dev/ubd/1 root=/dev/ubd0
Calibrating delay loop... 707.26 BogoMIPS
Memory: 32244k available
Dentry-cache hash table entries: 4096 (order: 3, 32768 bytes)
Inode-cache hash table entries: 2048 (order: 2, 16384 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer-cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 8192 (order: 3, 32768 bytes)
Checking for host processor cmov support...Reading /proc/cpuinfo failed, errno = 2
Yes
Checking for host processor xmm support...Reading /proc/cpuinfo failed, errno = 2
No
Checking that ptrace can change system call numbers...OK
Checking that host ptys support output SIGIO...No, enabling workaround
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
VFS: Diskquotas version dquot_6.4.0 initialized
devfs: v1.10 (20020120) Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x1
JFFS version 1.0, (C) 1999, 2000  Axis Communications AB
JFFS2 version 2.1. (C) 2001 Red Hat, Inc., designed by Axis Communications AB.
pty: 256 Unix98 ptys configured
block: 64 slots per queue, batch=16
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256).
loop: loaded (max 8 devices)
PPP generic driver version 2.4.1
Universal TUN/TAP device driver 1.4 (C)1999-2001 Maxim Krasnyansky
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 2048 bind 2048)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
Initializing stdio console driver
Initializing software serial port version 1
mconsole (version 1) initialized on /root/.uml/yfWc2E/mconsole
Partition check:
 ubda: unknown partition table
UML Audio Relay: May  1 2002 21:17:10
VFS: Mounted root (ext2 filesystem) readonly.
Mounted devfs on /dev
INIT: version 2.84 booting
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
INIT: Entering runlevel: 5


Welcome to the uml version of Tom's root/boot.

This is a customized version, so the notice below is displayed
at Tom's request.

*******************************************************************************
* If you base something on it, use any of the scripts, distribute binaries or *
* libraries from it, or distribute customized versions of it: You must credit *
* tomsrtbt and include a pointer to http://www.toms.net/rb/ and tom@toms.net, *
* and include this notice verbatim. Copyright Tom Oehser 1999. This notice in *
* no way supercedes or nullifies any other protections on the component parts *
* such as the BSD and GPL copyrights which apply to practically everything!!! *
* Within these strictures you may redistribute, incorporate, copy, modify, or *
* do anything else to it or with it that you like. Tomsrtbt has no warranties *
* not even implied fitness or usefulness.  If it breaks you keep both pieces. *
*******************************************************************************





        What you have is...

3c589_cs advansys agetty aha152x aha152x_cs aha1542 aic7xxx ash awk badblocks
bdflush buildit.s busLogic busybox bzip2 cardbus cardmgr cat ce ce.help chain.b
chattr chgrp chmod chown chroot clear clone.s cmp common config cp cpio cut date
dd ddate debugfs df dhcpcd dirname dmesg ds du dumpe2fs e2fsck eata echo echo.c
elvis emacs ex extend false fdflush fdformat fdisk fdomain filesize find
findsuper fmt fsck.ext2 fsck.msdos fstab grep group gzip halt head hexdump
hexedit host.conf hostname hosts i82365 ifconfig ifport ile image init init.old
inittab insmod install.s ioctl.save issue kill killall5 ksyms ld ld-linux length
less libc libcom_err libe2p libext2fs libss libtermcap libuuid lilo lilo.conf ln
loadkeys login losetup ls lsattr man mawk md5sum memtest miterm mkdir mkdosfs
mke2fs mkfifo mkfs.minix mklost+found mknod mkswap mnsed more more.help mount mt
mtab mv nc ncr53c8xx network networks nmclan_cs ntfs passwd pax pcmcia
pcmcia_core pcnet_cs ping plip ppa printf profile protocols ps pwd qlogic_cs
qlogicfas rc.0 rc.6 rc.M rc.S rc.custom rc.custom.gz rc.custom~ reboot rescuept
reset resolv.conf rm rmdir rmmod route rsh rshd script scsi scsi_info seagate
sed serial serial_cs services setserial settings.s sh shared shutdown slattach
sleep snarf sort split stty swapoff swapon sync tail tar tcic tee telnet telnetd
termcap test tomcr.txt tomshexd tomsrtbt.FAQ touch true tune2fs umount undeb
undeb-- unpack.s unrpm-- update utmp vi vi.help view wc wtmplock

        ...Login as root.

 ttys/0 tomsrtbt login: root
Password: ile rev.2.01
Today is Pungenday, the 50th day of Discord in the YOLD 3168
Celebrate Discoflux
# ps uax
1       S       (init) init 
2       S       (keventd) 
3       S       (ksoftirqd_CPU0) 
4       S       (kswapd) 
5       S       (bdflush) 
6       S       (kupdated) 
7       S       (mtdblockd) 
55      S       (ile) ile /bin/sh -c . /etc/profile -si 
59      S       (sh) /bin/sh -c . /etc/profile -si 
61      S       (ps) /bin/sh /usr/bin/ps uax 
62      R       (ps) /bin/sh /usr/bin/ps uax 
63      R       (sed) sed -e s/\ / /g 
# halt
INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal

halt
System halted.
nbd: module cleaned up.

usermode:~# halt

Broadcast message from root (vc/0) Fri May  3 21:24:09 2002...

The system is going down for system halt NOW !!
INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
Stopping internet superserver: inetd.
Stopping OpenBSD Secure Shell server: sshd.
Saving the System Clock time to the Hardware Clock...
hwclock: Can't open /dev/tty1, errno=2: No such file or directory.
hwclock is unable to get I/O port access:  the iopl(3) call failed.
Hardware Clock updated to Fri May  3 21:25:17 CEST 2002.
Stopping portmap daemon: portmap.
Stopping NFS kernel daemon: mountd nfsd.
Unexporting directories for NFS kernel daemon...done.
Stopping NFS common utilities: statd.
Stopping system log daemon: klogd syslogd.
Sending all processes the TERM signal... done.
Sending all processes the KILL signal... done.
Saving random seed... done.
Unmounting remote filesystems... done.
Deconfiguring network interfaces: done.
Deactivating swap... done.
Unmounting local filesystems... done.
* route del -host 192.168.0.253 dev tap0
* bash -c echo 0 > /proc/sys/net/ipv4/conf/tap0/proxy_arp
* arp -i eth0 -d 192.168.0.253 pub
Power down.
nbd: module cleaned up.

um 1012: 

Re: UML is now self-hosting!

[Guest section DW]

Congratulations!

[Reminds me of the good old times 30 years ago -
had a tower of three virtual machines on top of a
real PDP 8/I. Now that you can run UML under UML,
can you run UML under UML under UML?]

Re: UML is now self-hosting!

[Jeff Dike]

Some moron who can't keep track of his own patches said:
>  This works as of UML 2.4.18-21, which isn't released yet, but will be
> soon.

Correction - 2.4.18-22, which has just been released.

				Jeff

Re: UML is now self-hosting!

[Gerrit Huizenga]

In message <20020503215102.GA24653@win.tue.nl>, > : Guest section DW writes:
> Congratulations!
> 
> [Reminds me of the good old times 30 years ago -
> had a tower of three virtual machines on top of a
> real PDP 8/I. Now that you can run UML under UML,
> can you run UML under UML under UML?]

Fun stuff!  With PTX we were doing something very similar near
the end of our days with PTX:

PTX could run Linux Binaries
PTX could run a System 390 emulator (Flex/ES ?)
PTX could *almost* run VMWare (might be able to run Win4Lin or Boochs...)
PTX could sever as a Citrix (Windows NT) server

Picture Windows running in VMWare, talking to an OS/390 emulator
on the same hardware.  You might have been able to run Linux on
390, as well as VM/SP or whatever...

Add on all the other Linux emulators and you had quite a few applications
you could run on a single platform, all able to talk to each other.  ;-)

Customers wanted to run legacy OS/390 apps that they had lost the
binaries for, with a fast, modern database (Oracle or DB2) running
at native speed, with either Linux or Windows applications.  Add
UML and you can do development and client support like System 390
can do with Linux and you have an interesting (if a bit perverted ;-)
world.

Sick and twisted...

gerrit

Re: UML is now self-hosting!

[Jeff Dike]

dwguest@win.tue.nl said:
> Congratulations!

Thanks! 

> Now that you can run UML
> under UML, can you run UML under UML under UML?] 

Heh, I haven't tried it.  Feel free, it should work now.

gh@us.ibm.com said:
> Customers wanted to run legacy OS/390 apps that they had lost the
> binaries for, with a fast, modern database (Oracle or DB2) running at
> native speed, with either Linux or Windows applications.  Add UML and
> you can do development and client support like System 390 can do with
> Linux and you have an interesting (if a bit perverted ;-) world.

Yup.  And that sort of thing doesn't come close to reaching the potential
of a virtual OS.

A couple of the intermediate-term things I'm most interested in are
	embedding UML in things like Apache to provide a standard internal
	development and execution environment

	spreading a SMP UML instance across multiple hosts

				Jeff

Re: UML is now self-hosting!

[Christoph Hellwig]

On Fri, May 03, 2002 at 03:28:03PM -0700, Gerrit Huizenga wrote:
> PTX could *almost* run VMWare (might be able to run Win4Lin or Boochs...)

Umm, you have ported the VMWare and Win4Lin kernel modules?
For Win4Lin I could almost image it as it is ported UnixWare code..

Re: UML is now self-hosting!

[Gerrit Huizenga]

In message <20020504072849.B2295@infradead.org>, > : Christoph Hellwig writes:
> On Fri, May 03, 2002 at 03:28:03PM -0700, Gerrit Huizenga wrote:
> > PTX could *almost* run VMWare (might be able to run Win4Lin or Boochs...)
> 
> Umm, you have ported the VMWare and Win4Lin kernel modules?
> For Win4Lin I could almost image it as it is ported UnixWare code..

Nope - that's where the part of the *almost* comes from.  There were
a few other things we didn't get to and the only time I remember someone
trying it was when we still didn't have modify_ldt() ported, so that
also broke.  VMWare wasn't actually a goal - on the same hardware
we could statically partition nodes and some nodes could natively
run NT while some ran PTX.  Now those same nodes can also run Linux
as well.  And we quit putting major development effort into PTX about
two years ago, so we'll probably never find out just how close we
were on VMWare.

gerrit

Re: UML is now self-hosting!

[Mike Fedyk]

On Fri, May 03, 2002 at 08:32:27PM -0500, Jeff Dike wrote:
> A couple of the intermediate-term things I'm most interested in are
> 	embedding UML in things like Apache to provide a standard internal
> 	development and execution environment

How would that benefit Apache, and why does is have any use for an imbedded
kernel?

> 
> 	spreading a SMP UML instance across multiple hosts

How would this be better than MOSIX, or other clustering solutions?

Any URLs you may have on this would be quite helpful.

Mike

Re: UML is now self-hosting!

[Vikram]

>
> How would this be better than MOSIX, or other clustering solutions?
>
> Any URLs you may have on this would be quite helpful.

uh-huh, you miss the pt maybe? uml offers a great testing, debugging,
developing platform. the whole idea is to replace the real thing (say like
kernel devel) with UML and your qn is more like why cant we use the real
thing itself....:)

			Vikram

Re: UML is now self-hosting!

[Mike Fedyk]

On Sun, May 05, 2002 at 01:29:43AM -0700, Vikram wrote:
> 
> >
> > How would this be better than MOSIX, or other clustering solutions?
> >
> > Any URLs you may have on this would be quite helpful.
> 
> uh-huh, you miss the pt maybe? uml offers a great testing, debugging,
> developing platform. the whole idea is to replace the real thing (say like
> kernel devel) with UML and your qn is more like why cant we use the real
> thing itself....:)

If you want to test clustering (or "UML SMP over several seperate hosts"
-JDike) with UML, why not just create a UML kernel with the clustering
support (ie, MOSIX) in that UML kernel?  

Really, I'm just asking what the benifit is to use UML for clustering as
oposed to MOSIX.  I can think of one, testing NUMA without special
hardware...

Re: UML is now self-hosting!

[Jeff Dike]

mfedyk@matchmail.com said:
> How would that benefit Apache, and why does is have any use for an
> imbedded kernel?

See http://user-mode-linux.sourceforge.net/slides/wvu2002/wvu2002.htm, which 
are the slides from a talk I gave at WVU and MorLUG in which I explained this
more clearly than I had managed before.  The section that's relevant here
starts at http://user-mode-linux.sourceforge.net/slides/wvu2002/img9.htm

> How would this be better than MOSIX, or other clustering solutions? 

MOSIX (or Compaq's SSI) would certainly be a way of doing it.  It happens
that there's a particularly simple way of doing it with UML.  You'd partition
UML's 'physical' memory between the hosts, and use the fact that those pages
are really virtual to fault them between hosts as needed.  This would perform
particularly badly, but its simplicity appeals to me.

				Jeff

Re: UML is now self-hosting!

[Larry McVoy]

On Sun, May 05, 2002 at 07:25:00AM -0500, Jeff Dike wrote:
> mfedyk@matchmail.com said:
> > How would this be better than MOSIX, or other clustering solutions? 
> 
> MOSIX (or Compaq's SSI) would certainly be a way of doing it.  It happens
> that there's a particularly simple way of doing it with UML.  You'd partition
> UML's 'physical' memory between the hosts, and use the fact that those pages
> are really virtual to fault them between hosts as needed.  This would perform
> particularly badly, but its simplicity appeals to me.

See http://www.bitmover.com/cc-pitch/ for some more on this idea.  I think 
the UML approach would be very cool.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: [uml-devel] Re: UML is now self-hosting!

[Jeff Dike]

lm@bitmover.com said:
> See http://www.bitmover.com/cc-pitch/ for some more on this idea.  I
> think  the UML approach would be very cool. 

Actually, what I outlined is different from your CC-smp.  A lot simpler
and a lot less likely to be practical :-)

http://www.bitmover.com/cc-pitch/ also deserves to be looked at along with
MOSIX and Compaq's SSI project for UML (and physical) clustering.

				Jeff

Re: [uml-devel] Re: UML is now self-hosting!

[Lars Marowsky-Bree]

On 2002-05-05T07:25:00,
   Jeff Dike <jdike@karaya.com> said:

> MOSIX (or Compaq's SSI) would certainly be a way of doing it.  It happens
> that there's a particularly simple way of doing it with UML.  You'd partition
> UML's 'physical' memory between the hosts, and use the fact that those pages
> are really virtual to fault them between hosts as needed.  This would perform
> particularly badly, but its simplicity appeals to me.

An interesting and simple approach indeed; but spreading an instance across
multiple nodes is nowhere as simple as it seems; where do you keep OS data, IO
access, scheduling decisions, inter-node communication in the first place, how
to deal with node failure etc...

However, I believe it could potentially be implemented cleaner than currently
with the Compaq SSI stuff, because the encapsulation is better etc; but I have
been known to be wrong ;-)

It would certainly be very interesting. If you _really_ want to open this can
of worms, you should consider joining linux-cluster mailing list for this, or
the Open Clustering Framework list (because you are going to stumble into the
madness which is "interoperability and lack of standards" here).


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
Immortality is an adequate definition of high availability for me.
	--- Gregory F. Pfister

Re: UML is now self-hosting!

[Jeff Dike]

pavel@suse.cz said:
> > 	embedding UML in things like Apache to provide a standard internal
> > 	development and execution environment
> This is little perverted, right?

Maybe :-)

> What is it good for? 

Secure mod_perl - i.e. mod_perl in an Apache that's shared with other web sites

mod_perl in any language supported by Linux

Interactive debugging of your perl on live requests inside a live Apache

See http://user-mode-linux.sourceforge.net/slides/wvu2002/wvu2002.htm

The section that's relevant here starts at 
	http://user-mode-linux.sourceforge.net/slides/wvu2002/img9.htm

There are also some wackier possibilities which I'm not sure are terribly
useful or practical, but would still be interesting to look at, such as:

Treating HTTP requests as processes, so
	# ps uax
	...
	apache    1120  0.0  0.6  4388 1536 ?        S    13:46   0:00 GET / HTTP/1.0
	...
	# kill -9 1120
or
	# nice -20 1120

Treating HTTP requests as packets and using Netfilter to manipulate them.

				Jeff

Re: [uml-devel] Re: UML is now self-hosting!

[Jeff Dike]

lmb@suse.de said:
> but spreading an instance across multiple nodes is nowhere as simple
> as it seems; 

It is if you want to be sufficiently stupid about it :-)

> where do you keep OS data, 

It gets faulted from host to host as needed.

> IO access, 

This scheme (and any clustering scheme, I think) would need back channels
for one node to access the devices of another

> scheduling decisions, 

This machine thinks it's a normal SMP box, so scheduling happens as normal

> inter-node communication in the first place, how to deal
> with node failure etc...

Maybe I'm not familiar enough with the clustering world, but I was under the
impression that with a normal SSI cluster, the nodes are like CPUs in an
SMP box - if one fails, the whole thing dies.  In other words, that SSI
clustering and HA clustering are pretty disjoint.

				Jeff

Re: [uml-devel] Re: UML is now self-hosting!

[Rob Landley]

On Tuesday 07 May 2002 12:26 pm, Lars Marowsky-Bree wrote:

> > This machine thinks it's a normal SMP box, so scheduling happens as
> > normal
>
> Ugh ugh ugh. Too many page faults; you need a scheduler capable of keeping
> node affinity.

O(1)?

Thought it did.  Might have to teach the load balancer to be a bit more 
clever...

Rob

Re: [uml-devel] Re: UML is now self-hosting!

[Lars Marowsky-Bree]

On 2002-05-06T15:55:52,
   Jeff Dike <jdike@karaya.com> said:

> > but spreading an instance across multiple nodes is nowhere as simple
> > as it seems; 
> It is if you want to be sufficiently stupid about it :-)

Ugh.

> > where do you keep OS data, 
> It gets faulted from host to host as needed.

Ugh Ugh Ugh. You need coherency algorithms for these.

> > IO access, 
> This scheme (and any clustering scheme, I think) would need back channels
> for one node to access the devices of another

Right. You need communication services and coherency algorithms for these ;-)

> > scheduling decisions, 
> This machine thinks it's a normal SMP box, so scheduling happens as normal

Ugh ugh ugh. Too many page faults; you need a scheduler capable of keeping
node affinity.

> > inter-node communication in the first place, how to deal
> > with node failure etc...
> Maybe I'm not familiar enough with the clustering world, but I was under the
> impression that with a normal SSI cluster, the nodes are like CPUs in an
> SMP box - if one fails, the whole thing dies.  In other words, that SSI
> clustering and HA clustering are pretty disjoint.

No. In the optimal case, nothing dies. ;-) In the less than optimal case, only
the processes affected directly by the failure die (the processes which had
dirty pages there etc).

In the really useless case, the entire cluster goes down like on a SMP box or
an unpartitioned CC-NUMA.

"In search of clusters" is definetely highly recommended reading. It is very
entertaining, too.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
Immortality is an adequate definition of high availability for me.
	--- Gregory F. Pfister

Re: [uml-devel] Re: UML is now self-hosting!

[Jeff Dike]

lmb@suse.de said:
> Ugh. 

> Ugh Ugh Ugh.

> Ugh ugh ugh. Too many page faults;

You will note that the only favorable adjective I've used in describing this
scheme is 'easy' :-)

The too many page faults is covered by my 'performance will suck'...

				Jeff