inserting a Forth-like language into the Linux kernel

inserting a Forth-like language into the Linux kernel

[Rick Hohensee]

( H3sm is my 3-stack Forth variant. )

I have a Linux 2.4.0-test10 kernel building and booting nicely with H3sm
partially installed as an in-kernel thread ala kswapd, the virtual memory
swapper. This means H3sm runs in cooperative multitasking with the Linux
schuduler, i.e. it's something of a co-kernel with Linux. H3sm looks like
this from the Linux side in the unix /proc pseudo-filesystem of system
information...

Name:   kspamd
State:  Z (zombie)
Pid:    4
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 0
Groups:
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000000000000
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 00000000ffffffff
CapEff: 00000000fffffeff

H3sm is of course the 4th process ;o)  (kspamd) is the Linux-side wrapper
for H3sm....

:;ps
PID     TTY     STAT    RSS     COMMAND
1       0       S       89      (init)
2       0       S       0       (kswapd)
3       0       S       0       (kreclaimd)
4       0       Z       0       (kspamd)
5       0       S       0       (kflushd)
6       0       S       0       (kupdate)
12      1026    S       157     (bash)
19      1027    S       143     (bash)
20      1028    S       141     (bash)
21      1029    S       137     (bash)
22      1030    S       142     (bash)
23      1031    S       137     (bash)
24      1032    S       137     (bash)
25      1033    S       137     (bash)
26      1034    S       137     (bash)
27      1035    S       137     (bash)
28      1036    S       137     (bash)
44      0       S       83      (mouse)
67      0       S       101     (syslogd)
80      1029    S       140     (bash)
81      1029    S       339     (browse)
83      1088    S       148     (pppd)
85      1028    S       84      (irc)
88      1028    S       311     (ircii-EPIC4pre2)
89      1028    S       63      (in.identd)
91      1027    S       1287    (slrn)
92      1027    S       108     (edit)
94      1030    S       345     (browse)
95      1031    R       2       (ps)

H3sm doesn't yet survive the boot process, and gets killed and becomes a a
unix "zombie process" (the Z's in the listings). I just spent 3 days
running around in circles at the lowest level to no avail, other than I
think I may have arrived at a realization that I badly mis-concieved
H3sm's behavior requirements trying to do "file" reads as part of the
kernel. H3sm opens the file namespace representation of the first virtual
console just fine, namely /dev/tty1. It can beep the console before going
zombie, which is a "file" write of one character, ASCII 7. (I've left
/dev/tty1 unallocated in the usual init process on the Linux side, so that
when H3sm does work it won't be competing with a getty or shell for user
input.)

For reading user input, the regular userland H3sm was doing a blocking
read, i.e. it would call the form of read syscall that pauses the process
until the "file" replies somehow. This is asking a bit much of kernel
code, to say the least. The userland H3sm lets the console driver provide
it with line-wise communications from the user. I just hope it's possible
to do a character-wise version from the kernel, or some other work-around.

If it is possible to read a virtual terminal from the kernel the
possibilities are outrageous. The more general cases of file IO and
terminal IO are proven possible. This approach could be applied to various
Forths, for those who like thier stacks in matched pairs.

H3sm is built into the kernel by sneaking in behind the real in-kernel
daemons in mm/vmscan.c. This avoids a lot of #include complexity.  A
wrapper mimicing the outermost code of kpiod or kreclaimd is what the
kspamd is on the Linux side. Playing around in there you see neat examples
of the cooperative multi-tasking at the core of Linux. You can hog the
CPU, grab it periodically, mark your self runnable and keep CPU use at
100% while still switching to Linux, and so on. You relinquish the CPU
from the H3sm/Forth/whatever side with the Linux "schedule" call. Syscalls
are as per usual, except with the more fundamental coordination issues.
Here's some of H3sm in x86 gas and m4...
......................................................................
top_loop_of_H3sm:
        pushf
        pusha
                call schedule   # exuent to host
        popa
        popf

        call token
                YES(            mozygote)
                        call interpret
mov A to 0  [note to c.l.f: This is my debugging hook. Do a Linux "oops"
                here.]
ELL(                                    top_loop_of_H3sm)
        #zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
mozygote:
                call beep
#               call Source
 #                      call zero
  #              call store
   #             call pdrop
#                call cr
                call refill
                call drop
ELL(                    top_loop_of_H3sm)
initialdp:

......................................................................

When the H3sm zombies the rest of Linux boots without it.

Here's some of the mm/vmscan.c in question....
...................................................................
int kspamd(void *unused)
{
        struct task_struct *tsk = current;
        pg_data_t *pgdat;

        tsk->session = 1;
        tsk->pgrp = 1;
        strcpy(tsk->comm, "kspamd");
/*      sigfillset(&tsk->blocked);
        current->flags |= PF_MEMALLOC;
*/
        mainH3sm();
}


static int __init kswapd_init(void)
{
        printk("Starting kswapd v1.8\n");
        swap_setup();
        kernel_thread(kswapd, NULL, CLONE_FS | CLONE_FILES |
                        CLONE_SIGNAL);
        kernel_thread(kreclaimd, NULL, CLONE_FS | CLONE_FILES |
                        CLONE_SIGNAL);
        kernel_thread(kspamd, NULL, CLONE_FS | CLONE_SIGNAL );
        printk("Starting kspamd v0.00.00.00.01\n");
        return 0;
}
........................................................................

That code is left over from some tests without H3sm that accepted the CPU
on every scheduling opportunity, resulting in the CPU use pegged at 1.0 or
more, but otherwise normal operation of Linux.

An ed script and some hand renaming suffixes all the labels in H3sm with
"H3sm" to eliminate name conflicts with Linux labels.

This is not how kdb works. That is an in-kernel debugger invoked on
breakpoint traps and so on. I don't know about kgdb, which is a wrapper
for the userland GNU symbolic debugger. gdb requires compiling the kernel
with debugging information inserted everywhere.

Rick Hohensee
www.clienux.com

Re: inserting a Forth-like language into the Linux kernel

[Alexander Viro]

On Tue, 1 May 2001, Rick Hohensee wrote:
 
> (kspamd) is the Linux-side wrapper for H3sm....

C|N>K

OK, you owe me a new keyboard. And thanks for new .sig.

BTW, Rick, out of curiosity - how many Greencard Lawyers does it take
to upgrade the thing to full-blown H1 S&M?

Re: inserting a Forth-like language into the Linux kernel

[Rick Hohensee]

> 

I thought my mail client was doing "reply to all recipients". If it _was_
then this is redundant and I apologize.

> 
> 
> On Tue, 1 May 2001, Rick Hohensee wrote:
>  
> > (kspamd) is the Linux-side wrapper for H3sm....
> 
> C|N>K
> 
> OK, you owe me a new keyboard. And thanks for new .sig.

Oh, uh, sorry about that, Chief.

> BTW, Rick, out of curiosity - how many Greencard Lawyers does it take
> to upgrade the thing to full-blown H1 S&M?
> 

I dono. How many you got?

Rick Hohensee
www.clienux.com

Re: inserting a Forth-like language into the Linux kernel

[Rick Hohensee]

read() of a tty from the kernel does behave as per the tty settings.
I now have a kernel thread with stdin, out and err, FD's 0, 1 and 2,
open and reading lines of typed input. The "interpreter" adds 0x30 
(ASCII 0) to the return value from read and prints it, which means 
it occurs at the beginning of the next line. Here's what's sitting 
on my tty1 screen right now...
.................................................................
1
1
1
1aaoeutasoeu aoeu oaeusntaoesutoaeut oeuoaeuo uoaeuoaeu
heeee
6e
2
1
1
1
1eeeeee
7ee
3
1onetu euaoeuaoeuoa eu aoeu ao eu ao eu aoeu  aoeu ao eu aoe u aoe u
te e e e e
;e e e e
9e e e e
9e e e
7ee
3
...................................................................

This should simplify hanging an interpreter off this simple little top 
loop. That is, userland H3sm was using cooked mode, so this is nice. 
This is the top loop code, which is x86 assembly and H3sm 
subroutines...

.................................................................
abort_H3sm:
        call buffer

top_loop_of_H3sm:
                                # stick a sleep in here.
                                # Do not melt the CPU, do not slow down
                                #       the test cycle.
        call timespec
        call pdup
        call pplusc
        call pplusc
        call nanosleep
        call twopdrop
        call drop

        HANDOFF

        call zero               # FD
        call read
        call literal
        .byte 4
        .int  0x30
        call plus
        call emit
ELL(                                                    top_loop_of_H3sm)
...................................................................

emit is similar to Forth EMIT. It is a one-byte write().
buffer and timespec put addresses on
the H3sm pointer stack. nanosleep and read are H3sm stack-passing
wrappers for the syscalls. HANDOFF is a macro for a stack-push-wrapped
schedule(). It's a macro because I suspect I'm going to have to sprinkle
it judiciously around H3sm. The nanosleep keeps my load average normal.

Rick Hohensee
www.clienux.com

Re: inserting a Forth-like language into the Linux kernel

[Jesse Pollard]

On Sat, 05 May 2001, Rick Hohensee wrote:
>kspamd/H3sm is now making continuous writes to tty1 from an 
>in-kernel thread. It was locking on a write to /dev/console by
>init, so I made /dev/console a plain file. This is after 
>hollowing out sys_syslog to be a null routine, and various 
>other minor destruction.
>
>I am now typing at you on tty4 or so while the kernel itself 
>sends an endless stream of d's to tty1. It will scroll-lock 
>and un-scroll-lock, which is how I can tell it's not just a 
>static screen of d's.
>
>I don't know about H1 S&M, but the ability to open a tty
>normally directly into kernelspace may prove popular, particularly 
>with a Forth on that tty in that kernelspace. Persons with actual 
>kernel clue may want to look at allowing /dev/console users and 
>an in-kernel tty user to play nice. For my purposes I'll do without 
>a real /dev/console and syslogging for now. 
>
>Now I get to find out how many worlds of trouble I didn't foresee
>in _reading_ a tty from the kernel :o)
>
>If someone knows of another example of interpreter-like behavior 
>directly in a unix in-kernel thread I'd like to know about it.  

Only in reference to allowing for virus infection of the kernel.

It isn't a good idea.

-- 
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: jesse@cats-chateau.net

Any opinions expressed are solely my own.

Re: inserting a Forth-like language into the Linux kernel

[Rick Hohensee]

> 
> 
> 
> > > > If someone knows of another example of interpreter-like behavior 
> > > > directly in a unix in-kernel thread I'd like to know about it.  
> > > 
> > > kdb
> > > 
> > 
> > That runs in trap handlers doesn't it? I don't think it's a 
> > kernel daemon.
> 
> and there's the hangman-in-kernel patch...
> interpreter or daemon or app-in-system-space ???

Thanks, I'll look for that.

Rick Hohensee

> 
> ~Randy
> 

RE: inserting a Forth-like language into the Linux kernel

[Dunlap, Randy]

> > > If someone knows of another example of interpreter-like behavior 
> > > directly in a unix in-kernel thread I'd like to know about it.  
> > 
> > kdb
> > 
> 
> That runs in trap handlers doesn't it? I don't think it's a 
> kernel daemon.

and there's the hangman-in-kernel patch...
interpreter or daemon or app-in-system-space ???

~Randy

Re: inserting a Forth-like language into the Linux kernel

[Rick Hohensee]

> 
> > I don't know about H1 S&M, but the ability to open a tty
> > normally directly into kernelspace may prove popular, particularly 
> > with a Forth on that tty in that kernelspace. Persons with actual 
> 
> With anything other than Forth, LISP, and COBOL... yes.

Nice little sensibility scale for kspamd-like things,

	Forth 1.0     Lisp  0.5     COBOL   0.0

> 
> > If someone knows of another example of interpreter-like behavior 
> > directly in a unix in-kernel thread I'd like to know about it.  
> 
> kdb
> 

That runs in trap handlers doesn't it? I don't think it's a kernel daemon.


Rick Hohensee
www.clienux.com

Re: inserting a Forth-like language into the Linux kernel

[Rick Hohensee]

kspamd/H3sm is now making continuous writes to tty1 from an 
in-kernel thread. It was locking on a write to /dev/console by
init, so I made /dev/console a plain file. This is after 
hollowing out sys_syslog to be a null routine, and various 
other minor destruction.

I am now typing at you on tty4 or so while the kernel itself 
sends an endless stream of d's to tty1. It will scroll-lock 
and un-scroll-lock, which is how I can tell it's not just a 
static screen of d's.

I don't know about H1 S&M, but the ability to open a tty
normally directly into kernelspace may prove popular, particularly 
with a Forth on that tty in that kernelspace. Persons with actual 
kernel clue may want to look at allowing /dev/console users and 
an in-kernel tty user to play nice. For my purposes I'll do without 
a real /dev/console and syslogging for now. 

Now I get to find out how many worlds of trouble I didn't foresee
in _reading_ a tty from the kernel :o)

If someone knows of another example of interpreter-like behavior 
directly in a unix in-kernel thread I'd like to know about it.  

Rick Hohensee
www.clienux.com
The userland H3sm is in 
ftp://ftp.gwdg.de/pub/linux/include/clienux/interim

Re: inserting a Forth-like language into the Linux kernel

[Rick Hohensee]

In 2.4.0-test10, in my kspamd kernel-thread that's like a hollowed-out
kswapd, I have x86 asm code like this...

[enter from in-kernel init]

[open 3 FDs on tty1]

LOOP:
	pushf
	pusha
	call schedule
	popa
	popf

	[ here is some code to use the write syscall to send 
		one byte to FD 1. A d, for example. ]

jmp LOOP


The d's get written until a syslog happens. If I do do_syslog(6,6,6); in
the C kspamd wrapper code it's about like so...



<gobs of d's>ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
dddddddddddddddINIT: entering runlevel 8

	[more normal logging stuff, ext2 warnings and so on]



If I don't do the do_syslog() the same thing happens somewhat earlier,
i.e. on a from-the-kernel-itself syslog.

Boot is normal. I can't shift&pg_up vt1, which normally I can, but I can
switch vt's, email you, etc.  /proc/4/status for kspamd shows

Name:   kspamd
State:  R (running)
Pid:    4
PPid:   1

PIDs 1, 2 and 3 don't have any open FD's. My simplistic wrapper pegs CPU
use at 1.0. 


Why don't my d's continue?

Might this be easier to do in a 2.0 kernel?


Rick Hohensee
www.clienux.com