Re: PROPOSAL: dot-proc interface [was: /proc stuff]

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, Daniel Phillips wrote:

> Doing 'top -d .1' eats 18% of a 1GHz cpu, which is abominable.  A kernel
> profile courtesy of sgi's kernprof shows that scanning pages does not move
> the needle, whereas sprintf does.  Notice that the biggest chunk of time

Huh?  Scanning pages is statm_pgd_range().  I'd say that it takes
seriously more than vsnprintf() - look at your own results.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

On November 5, 2001 12:42 am, Alexander Viro wrote:
> On Sun, 4 Nov 2001, Daniel Phillips wrote:
> 
> > Doing 'top -d .1' eats 18% of a 1GHz cpu, which is abominable.  A kernel
> > profile courtesy of sgi's kernprof shows that scanning pages does not move
> > the needle, whereas sprintf does.  Notice that the biggest chunk of time
> 
> Huh?  Scanning pages is statm_pgd_range().  I'd say that it takes
> seriously more than vsnprintf() - look at your own results.

Yes, true, 2.6 seconds for the statm_pgd_range vs 1.2 for sprintf.  Still, 
sprintf is definitely burning cycles, pretty much the whole 1.2 seconds would 
be recovered with a binary interface.

Now look at the total time we spend in the kernel: 10.4 seconds, 4 times the 
page scanning overhead.  This is really wasteful.

For top does it really matter?  (yes, think slow computer)  What happens when 
proc stabilizes and applications start relying on it heavily as a kernel 
interface?  If we're still turning in this kind of stunningly poor 
performance, it won't be nice.

It's not that it doesn't work, it's just that it isn't the best.

--
Daniel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Craig Thrall]

> Problem:  Could it be made simpler to parse from scripting languages,
> without making it less elegant to parse in plain C ?

Yes.  At one point, somebody suggested XML.  Now, as much as I hate the fact
that people somehow equate high-tech with tags, I think whomever originally
suggested it might be on to something.  :)

Fact is, just about EVERY language out there has some sort of utility to
parse XML.  There's expat for C, Perl and Python have libs, etc.  We could
even write a proc DTD that could specify the valid data types.

There are two problems:

1. Performance - it's slower to go through a library that outputs XML than
do a printf("%d", pid) or the like.

2. Space - based on a little experience using XML as a transport, the space
used by the tags adds up.

3. Work - writing a good package to do this, and rewriting bits of the
kernel to use it.  I'll volunteer my time.

Just a thought,

Craig

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 04:06:25PM -0700, Craig Thrall wrote:
> > Problem:  Could it be made simpler to parse from scripting languages,
> > without making it less elegant to parse in plain C ?
> 
> Yes.  At one point, somebody suggested XML.  Now, as much as I hate the fact
> that people somehow equate high-tech with tags, I think whomever originally
> suggested it might be on to something.  :)
> 
> Fact is, just about EVERY language out there has some sort of utility to
> parse XML.  There's expat for C, Perl and Python have libs, etc.  We could
> even write a proc DTD that could specify the valid data types.

I would say that it's "less elegant" to have to depend on yet another (big, 
complex, still evolving) library just to read out system metrics.

> 
> There are two problems:
> 
> 1. Performance - it's slower to go through a library that outputs XML than
> do a printf("%d", pid) or the like.

Indeed.

> 
> 2. Space - based on a little experience using XML as a transport, the space
> used by the tags adds up.

Yep.

> 
> 3. Work - writing a good package to do this, and rewriting bits of the
> kernel to use it.  I'll volunteer my time.

4. Stability - A good XML parsing library cannot be "simple" or "small". At
least not when written in C   ;)

5. Lack of benefits - we already have structure because of the filesystem in
which the information would live. The actual "tags" could be so incredibly
simple that using XML would just be shooting birds with tactical nukes. E.g.
lots of fun, but a little expensive and not really necessary.

But maybe I'm just a pessimist and should stop bitching and start coding  ;)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

On November 4, 2001 08:46 pm, Linus Torvalds wrote:
> On Sun, 4 Nov 2001, Daniel Phillips wrote:
> > >
> > > The computer can parse anything.
> >
> > OK, then lets keep the 'current' variable in ASCII.
> 
> Yeah, the old "argument by absurdity".
> 
> Did you ever take logics class? It isn't a valid argument at all.
> 
> My argument is: humans want the data they want in a readable format. What
> the _hell_ does that have to do with the "current" variable?




> > > Handling spaces and newlines is easy enough - see the patches from Al
> > > Viro, for example.
> >
> > Why are we doing this parsing in the kernel when it can be done in user
> > space?
> 
> We're not parsing anything.
> 
> We're marshalling the data into a format that is independent of whatever
> internal representation the kernel happens to have for it that particular
> day.
> 
> A representation that is valid across architectures, and a representation
> that is unambiguous. A representation that various scripts can trivially
> use, and a representation that is not bound by fixed-sized fields or other
> idiocy.
> 
> In short, text strings.
> 
> They have advantages even for a computer. Fixed-size binary interfaces are
> BAD for information interchange. They are bad as a word document file
> format, they are bad for email, and they are bad for /proc. Get it?
> 
> Would you prefer doc-files to be standard text, marshalled into some
> logical form? Or do you prefer binary blobs of data that is limited by the
> binary format?
> 
> 		Linus
> 
> 

[PATCH] 2.5 PROPOSAL: Replacement for current /proc of shit.

[Rusty Russell]

[Sorry: been wanting to use that phrase here for the longest time]

	This proof-of-concept implementation is pretty poor but the
principle (and interface) is simple:

	int my_foo;

	static int __init initfn(void)
	{
		return proc("net", "foo", my_foo, int, 0644);
	}

	static void __exit exitfn(void)
	{
		unproc("net", "foo");
	}

No kernel-formatted tables: use a directory.  (eg. kernel symbols
become a directory of symbol names, each containing the symbol value).

For cases when you don't want to take the overhead of creating a new
proc entry (eg. tcp socket creation), you can create directories on
demand when a user reads them using:

	proc_dir("net", "subdir", dirfunc, NULL);
	unproc_dir("net", "subdir");

Note that with kbuild 2.5, you can do something like:

	proc(KBUILD_OBJECT, "foo", my_foo, int, 0644);

And with my previous parameter patch:
	PARAM(foo, int, 0444);

declares a boot time parameter "KBUILD_OBJECT.foo=", or a module
parameter "foo=", and places it as readable in
/proc/KBUILD_OBJECT/foo.

I believe that rewriting /proc (and /proc/sys should simply die) is a
better solution than extending the interface, or avoiding it
altogether by using a new filesystem.

Of course, I don't care if it's *NOT* under /proc...
Rusty.
--
Premature optmztion is rt of all evl. --DK

diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.4.13-uml/include/linux/simpleproc.h working-2.4.13-uml-proc/include/linux/simpleproc.h
--- linux-2.4.13-uml/include/linux/simpleproc.h	Thu Jan  1 10:00:00 1970
+++ working-2.4.13-uml-proc/include/linux/simpleproc.h	Thu Nov  1 20:17:42 2001
@@ -0,0 +1,206 @@
+/* Dynamic proc filesystem that doesn't suck.  (C) 2001 Rusty Russell. */
+#ifndef _LINUX_SIMPLE_PROC_H
+#define _LINUX_SIMPLE_PROC_H
+#include <linux/config.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/stat.h>
+
+/* Commit the contents of this (NUL-terminated) buffer if possible.
+   -errno indicates error. */
+typedef int (proc_commitfn_t)(const char *dirname,
+			      const char *filename,
+			      const char *buffer,
+			      unsigned int size,
+			      void *arg);
+/* Fetch the contents into buffer: return size used (or needed), or
+   -errno. */
+typedef int (proc_fetchfn_t)(const char *dirname,
+			     const char *filename,
+			     char *buffer,
+			     unsigned int size,
+			     void *arg);
+
+/* If we're a dynamic directory, this routine gets dir contents:
+   returns size used (or needed), or -errno. */
+struct proc_dircontents;
+typedef int (proc_dirfn_t)(const char *dirname,
+			   const char *filename,
+			   struct proc_dircontents *buffer,
+			   unsigned int maxlen,
+			   void *arg);
+
+/* Register a proc entry of the given type. */
+#define proc(dir, fname, var, type, perms)				 \
+	__proc(dir, fname, S_IFREG|(perms),				 \
+	       __new_proc(&var,						 \
+			  ((perms)&S_IRUGO) ? proc_fetch_##type : NULL,	 \
+			  ((perms)&S_IWUGO) ? proc_commit_##type : NULL, \
+			  NULL))
+
+/* Register a proc entry protected by a spinlock. */
+#define proc_spinlock(dir, fname, var, type, lock, p)			   \
+	__proc(dir, fname, S_IFREG|(p),					   \
+	       __new_proc_lock(&var, lock,				   \
+			       ((p)&S_IRUGO) ? proc_fetch_##type : NULL,   \
+			       ((p)&S_IWUGO) ? proc_commit_##type : NULL))
+
+/* Register a proc entry protected by a semaphore. */
+#define proc_sem(dir, fname, var, type, sem, p)				  \
+	__proc(dir, fname, S_IFREG|(p),					  \
+	       __new_proc_sem(&var, sem,				  \
+			      ((p)&S_IRUGO) ? proc_fetch_##type : NULL,	  \
+			      ((p)&S_IWUGO) ? proc_commit_##type : NULL))
+
+/* These exist, believe me */
+struct semaphore;
+struct proc_data;
+
+#ifdef CONFIG_SIMPLE_PROC_FS
+/* Low level functions */
+int __proc(const char *dirname, const char *fname, int mode,
+	   struct proc_data *pdata);
+struct proc_data *__new_proc(void *arg, proc_fetchfn_t *, proc_commitfn_t *,
+			     proc_dirfn_t *);
+struct proc_data *__new_proc_lock(void *arg, spinlock_t *lock,
+				  proc_fetchfn_t *, proc_commitfn_t *);
+struct proc_data *__new_proc_sem(void *arg, struct semaphore *sem,
+				 proc_fetchfn_t *, proc_commitfn_t *);
+
+/* Register a whole dynamic directory */
+static inline int proc_dir(const char *dir, const char *dirname,
+			   proc_dirfn_t *dirfunc, void *arg)
+{
+	return __proc(dir, dirname, S_IFDIR|0555, 
+		      __new_proc(arg, NULL, NULL, dirfunc));
+}
+
+/* Release a dynamic proc directory */
+void unproc_dir(const char *dir, const char *fname);
+
+/* Release a proc entry */
+void unproc(const char *dir, const char *fname);
+
+#else
+static inline int proc_dir(const char *dir, const char *dirname,
+			   proc_dirfn_t *dirfunc, void *arg)
+{
+	return 0;
+}
+
+static inline void unproc(const char *dir, const char *fname)
+{
+}
+
+static inline void unproc_dir(const char *dir, const char *fname)
+{
+}
+
+static inline int __proc(const char *dirname, const char *fname, int mode,
+			 struct proc_data *pdata)
+{
+	return 0;
+}
+
+struct proc_data *__new_proc(void *arg,
+			     proc_fetchfn_t *fetch,
+			     proc_commitfn_t *commit,
+			     proc_dirfn_t *dir)
+{
+	return (struct proc_data *)-1;
+}
+struct proc_data *__new_proc_lock(void *arg, spinlock_t *lock,
+				  proc_fetchfn_t *fetch,
+				  proc_commitfn_t *commit)
+{
+	return (struct proc_data *)-1;
+}
+struct proc_data *__new_proc_sem(void *arg, struct semaphore *sem,
+				 proc_fetchfn_t *fetch,
+				 proc_commitfn_t *commit)
+{
+	return (struct proc_data *)-1;
+}
+#endif /*CONFIG_PROC_FS*/
+
+/* Helper parsing routines.  You can write your own, too. */
+proc_fetchfn_t proc_fetch_short;
+proc_fetchfn_t proc_fetch_ushort;
+proc_fetchfn_t proc_fetch_int;
+proc_fetchfn_t proc_fetch_uint;
+proc_fetchfn_t proc_fetch_long;
+proc_fetchfn_t proc_fetch_ulong;
+proc_fetchfn_t proc_fetch_bool;
+
+proc_commitfn_t proc_commit_short;
+proc_commitfn_t proc_commit_ushort;
+proc_commitfn_t proc_commit_int;
+proc_commitfn_t proc_commit_uint;
+proc_commitfn_t proc_commit_long;
+proc_commitfn_t proc_commit_ulong;
+proc_commitfn_t proc_commit_bool;
+
+/* Filled in by dir functions */
+struct proc_dircontents
+{
+	/* Mode of file.  0 terminates list. */
+	int mode;
+
+	/* Fetch, commit and dir functions for entry. */
+	proc_fetchfn_t *fetch;
+	proc_commitfn_t *commit;
+	proc_dirfn_t *dir;
+
+	/* Arg */
+	void *arg;
+
+	/* Name is nul-terminated, and padded to alignof this struct */
+	char name[0];
+};
+
+/* Helper to add another dircontents to the list, return updated "used" */
+static inline unsigned int proc_add_dircontents(struct proc_dircontents *pd,
+						unsigned int used,
+						unsigned int maxlen,
+						int mode,
+						proc_fetchfn_t *fetch,
+						proc_commitfn_t *commit,
+						proc_dirfn_t *dir,
+						void *arg,
+						const char *name)
+{
+	unsigned int thislen;
+
+	thislen = sizeof(*pd) + strlen(name) + 1;
+	thislen = (thislen + __alignof__(*pd) - 1) & ~(__alignof__(*pd) - 1);
+	if (used + thislen <= maxlen) {
+		pd = (void *)pd + used;
+		pd->mode = mode;
+		pd->fetch = fetch;
+		pd->commit = commit;
+		pd->dir = dir;
+		pd->arg = arg;
+		strcpy(pd->name, name);
+	}
+	return used + thislen;
+}
+
+static inline unsigned int proc_end_dircontents(struct proc_dircontents *pd,
+						unsigned int used,
+						unsigned int maxlen)
+{
+	return proc_add_dircontents(pd, used, maxlen, 0,
+				    NULL, NULL, NULL, NULL, "");
+}
+
+/* Internal use */
+struct proc_data
+{
+	/* User-defined argument for routines */
+	void *arg;
+
+	proc_dirfn_t *dir;
+	proc_commitfn_t *commit;
+	proc_fetchfn_t *fetch;
+};
+#endif /* _LINUX_SIMPLE_PROC_H */
diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.4.13-uml/fs/Config.in working-2.4.13-uml-proc/fs/Config.in
--- linux-2.4.13-uml/fs/Config.in	Thu Oct 25 11:29:49 2001
+++ working-2.4.13-uml-proc/fs/Config.in	Tue Oct 30 12:47:11 2001
@@ -50,7 +50,10 @@
 
 tristate 'OS/2 HPFS file system support' CONFIG_HPFS_FS
 
-bool '/proc file system support' CONFIG_PROC_FS
+bool 'Simple /proc file system support (EXPERIMENTAL)' CONFIG_SIMPLE_PROC_FS
+if [ "$CONFIG_SIMPLE_PROC_FS" != y ]; then
+   bool '/proc file system support' CONFIG_PROC_FS
+fi
 
 dep_bool '/dev file system support (EXPERIMENTAL)' CONFIG_DEVFS_FS $CONFIG_EXPERIMENTAL
 dep_bool '  Automatically mount at boot' CONFIG_DEVFS_MOUNT $CONFIG_DEVFS_FS
diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.4.13-uml/fs/Makefile working-2.4.13-uml-proc/fs/Makefile
--- linux-2.4.13-uml/fs/Makefile	Thu Oct 25 11:29:49 2001
+++ working-2.4.13-uml-proc/fs/Makefile	Tue Oct 30 12:47:11 2001
@@ -23,6 +23,7 @@
 endif
 
 subdir-$(CONFIG_PROC_FS)	+= proc
+subdir-$(CONFIG_SIMPLE_PROC_FS)	+= simpleproc
 subdir-y			+= partitions
 
 # Do not add any filesystems before this line
diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.4.13-uml/fs/proc/simple_proc.c working-2.4.13-uml-proc/fs/proc/simple_proc.c
--- linux-2.4.13-uml/fs/proc/simple_proc.c	Thu Jan  1 10:00:00 1970
+++ working-2.4.13-uml-proc/fs/proc/simple_proc.c	Tue Oct 30 12:47:11 2001
@@ -0,0 +1,517 @@
+/* Those of you who read this, give quiet thanks that you did not
+   suffer the endless frustration of dealing with the old /proc
+   interface.
+
+   Copyright (C) 2001 Rusty Russell.
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
+#include <linux/proc.h>
+#include <linux/proc_fs.h>
+#include <asm/uaccess.h>
+
+/* Simplistic approach: semaphore protects all proc accesses. */
+static DECLARE_MUTEX(simple_proc_sem);
+
+/* FIXME: Use reference counts and "dead" marker to return -ENOENT if
+   unregistered while open -RR */
+struct proc_data
+{
+	void *arg;
+	int (*get)(void *, char *, int);
+	int (*set)(void *, const char *);
+
+	/* FIXME: Belongs in struct file --RR */
+	int readlen, maxreadlen, writelen, maxwritelen;
+	char *readdata, *writedata;
+};
+
+static int fill_buffer(char **buffer,
+			int *maxlen,
+			struct proc_data *pdata)
+{
+	int len;
+
+	for (;;) {
+		len = pdata->get(pdata->arg, *buffer, *maxlen);
+		/* Need more room? */
+		if (len > *maxlen) {
+			/* We need some restriction here, to avoid
+			   DoS.  fs/proc/generic.c wants this, but we
+			   should make one or two pages eventually. */
+			if (len > PAGE_SIZE - 1024)
+				BUG();
+			kfree(*buffer);
+			*buffer = kmalloc(len, GFP_KERNEL);
+			if (!*buffer) return -ENOMEM;
+			*maxlen = len;
+		} else
+			return len;
+	}
+}
+
+/* FIXME: Get the struct file, and we can use ->private_data to store
+   this per file descriptor, rather than per file --RR */
+static int simple_read(char *page, char **start, off_t off, int count, 
+		       int *eof, void *data)
+{
+	struct proc_data *pdata = data;
+	int ret;
+
+	/* Start of read?  Get fresh buffer */
+	if (off == 0) {
+		int readlen, maxreadlen;
+		char *buffer;
+
+		maxreadlen = pdata->maxreadlen;
+		buffer = kmalloc(maxreadlen, GFP_KERNEL);
+		if (!buffer) {
+			*eof = 1;
+			return -ENOMEM;
+		}
+		readlen = fill_buffer(&buffer, &maxreadlen, pdata);
+		if (readlen < 0) {
+			*eof = 1;
+			kfree(buffer);
+			return readlen;
+		}
+
+		/* Substitute buffer */
+		if (down_interruptible(&simple_proc_sem) != 0) {
+			*eof = 1;
+			kfree(buffer);
+			return -EINTR;
+		}
+		kfree(pdata->readdata);
+		pdata->maxreadlen = maxreadlen;
+		pdata->readlen = readlen;
+		pdata->readdata = buffer;
+		up(&simple_proc_sem);
+	}
+
+	/* Serve from buffer */
+	if (down_interruptible(&simple_proc_sem) != 0) {
+		*eof = 1;
+		return -EINTR;
+	}
+
+	if (off <= pdata->readlen) {
+		ret = pdata->readlen - off;
+		memcpy(page + off, pdata->readdata + off, ret);
+	} else {
+		*eof = 1;
+		ret = 0;
+	}
+	up(&simple_proc_sem);
+
+	return ret;
+}
+
+/* FIXME: Don't share the write buffer: use file->private_data */
+static int simple_write(struct file *file,
+			const char *userbuffer,
+			unsigned long count, 
+			void *data)
+{
+	struct proc_data *pdata = data;
+
+	/* FIXME: commit the write(s) on close or seek.  We don't have
+	   that control under the current proc system, so simply
+	   terminate on \n. --RR */
+	if (file->f_pos + count > pdata->maxwritelen) {
+		char *newbuffer;
+		int newmax = file->f_pos + count;
+
+		/* As in read, we need some limit, and this is from
+                   fs/proc/generic.c */
+		if (newmax > PAGE_SIZE - 1024)
+			return -ENOSPC;
+
+		newbuffer = kmalloc(newmax, GFP_KERNEL);
+		if (!newbuffer)
+			return -ENOMEM;
+
+		/* Substitute buffer */
+		if (down_interruptible(&simple_proc_sem) != 0) {
+			kfree(newbuffer);
+			return -EINTR;
+		}
+		memcpy(newbuffer, pdata->writedata, pdata->writelen);
+		kfree(pdata->writedata);
+		pdata->maxwritelen = newmax;
+		pdata->writedata = newbuffer;
+		up(&simple_proc_sem);
+	}
+
+	/* Copy into buffer */
+	if (down_interruptible(&simple_proc_sem) != 0)
+		return -EINTR;
+
+	if (copy_from_user(pdata->writedata+file->f_pos, userbuffer, count)
+	    != 0) {
+		up(&simple_proc_sem);
+		return -EFAULT;
+	}
+
+	file->f_pos += count;
+
+	/* If there is now a '\n' at the end of the buffer, commit */
+	if (file->f_pos > 0 && pdata->writedata[file->f_pos-1] == '\n') {
+		int set;
+		pdata->writedata[file->f_pos-1] = '\0';
+		set = pdata->set(pdata->arg, pdata->writedata);
+
+		if (set < 0) {
+			up(&simple_proc_sem);
+			return set;
+		}
+	}
+	up(&simple_proc_sem);
+	return count;
+}
+
+/* This implementation serves only as a demonstration. --RR */
+static int do_register(const char *dir,
+		       const char *fname,
+		       int perms,
+		       struct proc_data *pdata)
+{
+	struct proc_dir_entry *entry;
+	char fullpath[strlen(dir) + 1 + strlen(fname) + 1];
+
+	sprintf(fullpath, "%s/%s", dir, fname);
+	entry = create_proc_entry(fullpath, perms, NULL);
+	if (!entry) return -EINVAL; /* -ERANDOM */
+
+	/* Populate data */
+	entry->data = pdata;
+
+	/* Set up read and write callbacks */
+	if (pdata->set) entry->read_proc = &simple_read;
+	if (pdata->get) entry->write_proc = &simple_write;
+	return 0;
+}
+
+int __register_proc(const char *dir,
+		    const char *fname,
+		    void *arg,
+		    unsigned int perms,
+		    int (*get)(void *arg, char *, int),
+		    int (*set)(void *arg, const char *))
+{
+	struct proc_data *pdata;
+	int ret;
+
+	pdata = kmalloc(sizeof(*pdata), GFP_KERNEL);
+	if (!pdata)
+		return -ENOMEM;
+	
+	pdata->arg = arg;
+	pdata->get = get;
+	pdata->set = set;
+	pdata->writelen = pdata->readlen = 0;
+	pdata->readdata = pdata->writedata = NULL;
+
+	ret = do_register(dir, fname, perms, pdata);
+	if (ret < 0)
+		kfree(pdata);
+	return ret;
+}
+
+/* Wrapper for user's real proc functions */
+struct pdata_wrapper
+{
+	struct proc_data pdata;
+	int (*get)(void *, char *, int);
+	int (*set)(void *, const char *);
+	void *lock;
+	void *userarg;
+};
+
+static struct pdata_wrapper *
+new_pdata_wrapper(void *arg,
+		  int (*userget)(void *, char *, int),
+		  int (*userset)(void *, const char *),
+		  int (*wrapperget)(void *, char *, int),
+		  int (*wrapperset)(void *, const char *),
+		  void *lock)
+{
+	struct pdata_wrapper *pwrap;
+
+	pwrap = kmalloc(sizeof(*pwrap), GFP_KERNEL);
+	if (pwrap) {
+		pwrap->pdata.arg = pwrap;
+		pwrap->pdata.writelen = pwrap->pdata.readlen = 0;
+		pwrap->pdata.readdata = pwrap->pdata.writedata = NULL;
+		pwrap->pdata.get = wrapperget;
+		pwrap->pdata.set = wrapperset;
+		pwrap->lock = lock;
+		pwrap->userarg = arg;
+		pwrap->get = userget;
+		pwrap->set = userset;
+	}
+	return pwrap;
+}
+
+static int do_register_wrap(const char *dir,
+			    const char *fname,
+			    int perms,
+			    void *arg,
+			    int (*userget)(void *, char *, int),
+			    int (*userset)(void *, const char *),
+			    int (*wrapperget)(void *, char *, int),
+			    int (*wrapperset)(void *, const char *),
+			    void *lock)
+{
+	struct pdata_wrapper *pwrap;
+	int ret;
+
+	pwrap = new_pdata_wrapper(arg, userget, userset, wrapperget,
+				  wrapperset, lock);
+	if (!pwrap)
+		return -ENOMEM;
+	ret = do_register(dir, fname, perms, &pwrap->pdata);
+	if (ret < 0)
+		kfree(pwrap);
+	return ret;
+}
+
+static int spinlock_get(void *arg, char *buffer, int size)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	spin_lock_irq(pwrap->lock);
+	ret = pwrap->get(pwrap->userarg, buffer, size);
+	spin_unlock_irq(pwrap->lock);
+
+	return ret;
+}
+
+static int spinlock_set(void *arg, const char *buffer)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	spin_lock_irq(pwrap->lock);
+	ret = pwrap->set(pwrap->userarg, buffer);
+	spin_unlock_irq(pwrap->lock);
+
+	return ret;
+}
+
+int __register_proc_spinlock(const char *dir,
+			     const char *fname,
+			     void *arg,
+			     unsigned int perms,
+			     spinlock_t *lock,
+			     int (*get)(void *arg, char *, int),
+			     int (*set)(void *arg, const char *))
+{
+	return do_register_wrap(dir, fname, perms, arg, get, set,
+				spinlock_get, spinlock_set, lock);
+}
+
+static int rwlock_get(void *arg, char *buffer, int size)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	read_lock_irq(pwrap->lock);
+	ret = pwrap->get(pwrap->userarg, buffer, size);
+	read_unlock_irq(pwrap->lock);
+
+	return ret;
+}
+
+static int rwlock_set(void *arg, const char *buffer)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	write_lock_irq(pwrap->lock);
+	ret = pwrap->set(pwrap->userarg, buffer);
+	write_unlock_irq(pwrap->lock);
+
+	return ret;
+}
+
+int __register_proc_rwlock(const char *dir,
+			   const char *fname,
+			   void *arg,
+			   unsigned int perms,
+			   rwlock_t *lock,
+			   int (*get)(void *arg, char *, int),
+			   int (*set)(void *arg, const char *))
+{
+	return do_register_wrap(dir, fname, perms, arg, get, set,
+				rwlock_get, rwlock_set, lock);
+}
+
+static int semaphore_get(void *arg, char *buffer, int size)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	if (down_interruptible(pwrap->lock) != 0)
+		return -EINTR;
+	ret = pwrap->get(pwrap->userarg, buffer, size);
+	up(pwrap->lock);
+
+	return ret;
+}
+
+static int semaphore_set(void *arg, const char *buffer)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	if (down_interruptible(pwrap->lock) != 0)
+		return -EINTR;
+	ret = pwrap->set(pwrap->userarg, buffer);
+	up(pwrap->lock);
+
+	return ret;
+}
+
+int __register_proc_semaphore(const char *dir,
+			      const char *fname,
+			      void *arg,
+			      unsigned int perms,
+			      struct semaphore *lock,
+			      int (*get)(void *arg, char *, int),
+			      int (*set)(void *arg, const char *))
+{
+	return do_register_wrap(dir, fname, perms, arg, get, set,
+				semaphore_get, semaphore_set, lock);
+}
+	
+static int rwsemaphore_get(void *arg, char *buffer, int size)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	down_read(pwrap->lock);
+	ret = pwrap->get(pwrap->userarg, buffer, size);
+	up_read(pwrap->lock);
+
+	return ret;
+}
+
+static int rwsemaphore_set(void *arg, const char *buffer)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	down_write(pwrap->lock);
+	ret = pwrap->set(pwrap->userarg, buffer);
+	up_write(pwrap->lock);
+
+	return ret;
+}
+
+int __register_proc_rwsemaphore(const char *dir,
+				const char *fname,
+				void *arg,
+				unsigned int perms,
+				struct rw_semaphore *lock,
+				int (*get)(void *arg, char *, int),
+				int (*set)(void *arg, const char *))
+{
+	return do_register_wrap(dir, fname, perms, arg, get, set,
+				rwsemaphore_get, rwsemaphore_set, lock);
+}
+
+int __proc_read_short(void *shortp, char *outbuf, int len)
+{
+	return snprintf(outbuf, len, "%hi", *(short *)shortp);
+}
+
+int __proc_write_short(void *shortp, const char *inbuf)
+{
+	if (sscanf(inbuf, "%hi", (short *)shortp) != 1) return -EINVAL;
+	return 0;
+}
+
+int __proc_read_ushort(void *ushortp, char *outbuf, int len)
+{
+	return snprintf(outbuf, len, "%hu", *(unsigned short *)ushortp);
+}
+
+int __proc_write_ushort(void *ushortp, const char *inbuf)
+{
+	if (sscanf(inbuf, "%hu", (unsigned short *)ushortp) != 1)
+		return -EINVAL;
+	return 0;
+}
+
+int __proc_read_int(void *intp, char *outbuf, int len)
+{
+	return snprintf(outbuf, len, "%i", *(int *)intp);
+}
+
+int __proc_write_int(void *intp, const char *inbuf)
+{
+	if (sscanf(inbuf, "%i", (int *)intp) != 1) return -EINVAL;
+	return 0;
+}
+
+int __proc_read_uint(void *uintp, char *outbuf, int len)
+{
+	return snprintf(outbuf, len, "%u", *(unsigned int *)uintp);
+}
+
+int __proc_write_uint(void *uintp, const char *inbuf)
+{
+	if (sscanf(inbuf, "%u", (unsigned int *)uintp) != 1) return -EINVAL;
+	return 0;
+}
+
+int __proc_read_long(void *longp, char *outbuf, int len)
+{
+	return snprintf(outbuf, len, "%li", *(long *)longp);
+}
+
+int __proc_write_long(void *longp, const char *inbuf)
+{
+	if (sscanf(inbuf, "%li", (long *)longp) != 1) return -EINVAL;
+	return 0;
+}
+
+int __proc_read_ulong(void *ulongp, char *outbuf, int len)
+{
+	return snprintf(outbuf, len, "%lu", *(long *)ulongp);
+}
+
+int __proc_write_ulong(void *ulongp, const char *inbuf)
+{
+	if (sscanf(inbuf, "%lu", (unsigned long *)ulongp) != 1) return -EINVAL;
+	return 0;
+}
+
+int __proc_read_bool(void *boolp, char *outbuf, int len)
+{
+	if (*(int *)boolp) return snprintf(outbuf, len, "y");
+	else return snprintf(outbuf, len, "n");
+}
+
+int __proc_write_bool(void *boolp, const char *inbuf)
+{
+	if (inbuf[0] == 'y' || inbuf[0] == 'Y')
+		*(int *)boolp = 1;
+	else if (inbuf[0] == 'n' || inbuf[0] == 'N')
+		*(int *)boolp = 0;
+	else return __proc_write_int(boolp, inbuf);
+	return 0;
+}
diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.4.13-uml/fs/simpleproc/Makefile working-2.4.13-uml-proc/fs/simpleproc/Makefile
--- linux-2.4.13-uml/fs/simpleproc/Makefile	Thu Jan  1 10:00:00 1970
+++ working-2.4.13-uml-proc/fs/simpleproc/Makefile	Tue Oct 30 12:47:11 2001
@@ -0,0 +1,14 @@
+#
+# Makefile for the Linux proc filesystem routines.
+#
+# Note! Dependencies are done automagically by 'make dep', which also
+# removes any old dependencies. DON'T put your own dependencies here
+# unless it's something special (not a .c file).
+#
+# Note 2! The CFLAGS definitions are now in the main makefile.
+
+O_TARGET := simpleproc.o
+
+obj-y    := inode.o helper.o
+
+include $(TOPDIR)/Rules.make
diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.4.13-uml/fs/simpleproc/helper.c working-2.4.13-uml-proc/fs/simpleproc/helper.c
--- linux-2.4.13-uml/fs/simpleproc/helper.c	Thu Jan  1 10:00:00 1970
+++ working-2.4.13-uml-proc/fs/simpleproc/helper.c	Thu Nov  1 20:58:13 2001
@@ -0,0 +1,277 @@
+/* Those of you who read this, give quiet thanks that you did not
+   suffer the endless frustration of dealing with the old /proc
+   interface.
+
+   Copyright (C) 2001 Rusty Russell.
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; if not, write to the Free Software
+    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
+#include <linux/simpleproc.h>
+#include <linux/slab.h>
+#include <linux/errno.h>
+#include <linux/dcache.h>
+#include <linux/init.h>
+
+/* Wrapper for user's real proc functions */
+struct pdata_wrapper
+{
+	struct proc_data pdata;
+	proc_fetchfn_t *fetch;
+	proc_commitfn_t *commit;
+	void *lock;
+	void *userarg;
+};
+
+static struct proc_data *new_wrapper(proc_fetchfn_t *userfetch,
+				     proc_commitfn_t *usercommit,
+				     void *userarg,
+				     proc_fetchfn_t *wrapfetch,
+				     proc_commitfn_t *wrapcommit,
+				     void *lock)
+{
+	struct pdata_wrapper *pwrap;
+
+	pwrap = kmalloc(sizeof(*pwrap), GFP_KERNEL);
+	if (pwrap) {
+		pwrap->pdata.arg = pwrap;
+		pwrap->pdata.fetch = wrapfetch;
+		pwrap->pdata.commit = wrapcommit;
+		pwrap->pdata.dir = NULL;
+		pwrap->fetch = userfetch;
+		pwrap->commit = usercommit;
+		pwrap->lock = lock;
+		pwrap->userarg = userarg;
+	}
+	return &pwrap->pdata;
+}
+
+static int lock_fetch(const char *dirname, const char *fname,
+		      char *buffer, unsigned int size, void *arg)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	spin_lock_irq(pwrap->lock);
+	ret = pwrap->fetch(dirname, fname, pwrap->userarg, size, buffer);
+	spin_unlock_irq(pwrap->lock);
+
+	return ret;
+}
+
+static int lock_commit(const char *dirname, const char *fname,
+		       const char *buffer, unsigned int size, void *arg)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	spin_lock_irq(pwrap->lock);
+	ret = pwrap->commit(dirname, fname, buffer, size, pwrap->userarg);
+	spin_unlock_irq(pwrap->lock);
+
+	return ret;
+}
+
+struct proc_data *__new_proc_lock(void *arg, spinlock_t *lock,
+				  proc_fetchfn_t *fetch,
+				  proc_commitfn_t *commit)
+{
+	return new_wrapper(fetch, commit, arg, lock_fetch, lock_commit, lock);
+}
+
+static int sem_fetch(const char *dirname, const char *fname,
+		     char *buffer, unsigned int size, void *arg)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	if (down_interruptible(pwrap->lock) != 0)
+		return -EINTR;
+	ret = pwrap->fetch(dirname, fname, pwrap->userarg, size, buffer);
+	up(pwrap->lock);
+
+	return ret;
+}
+
+static int sem_commit(const char *dirname, const char *fname,
+		      const char *buffer, unsigned int size, void *arg)
+{
+	struct pdata_wrapper *pwrap = arg;
+	int ret;
+
+	if (down_interruptible(pwrap->lock) != 0)
+		return -EINTR;
+	ret = pwrap->commit(dirname, fname, buffer, size, pwrap->userarg);
+	up(pwrap->lock);
+
+	return ret;
+}
+
+struct proc_data *__new_proc_sem(void *arg, struct semaphore *sem,
+				 proc_fetchfn_t *fetch,
+				 proc_commitfn_t *commit)
+{
+	return new_wrapper(fetch, commit, arg, sem_fetch, sem_commit, sem);
+}
+
+int proc_fetch_short(const char *dir, const char *fname,
+		     char *outbuf, unsigned int size, void *shortp)
+{
+	return snprintf(outbuf, size, "%hi\n", *(short *)shortp);
+}
+
+int proc_commit_short(const char *dir, const char *fname,
+		      const char *inbuf, unsigned int size, void *shortp)
+{
+	if (sscanf(inbuf, "%hi", (short *)shortp) != 1) return -EINVAL;
+	return 0;
+}
+
+int proc_fetch_ushort(const char *dir, const char *fname,
+		      char *outbuf, unsigned int size, void *ushortp)
+{
+	return snprintf(outbuf, size, "%hu\n", *(unsigned short *)ushortp);
+}
+
+int proc_commit_ushort(const char *dir, const char *fname,
+		       const char *inbuf, unsigned int size, void *ushortp)
+{
+	if (sscanf(inbuf, "%hu", (unsigned short *)ushortp) != 1)
+		return -EINVAL;
+	return 0;
+}
+
+int proc_fetch_int(const char *dir, const char *fname,
+		    char *outbuf, unsigned int size, void *intp)
+{
+	return snprintf(outbuf, size, "%i\n", *(int *)intp);
+}
+
+int proc_commit_int(const char *dir, const char *fname,
+		    const char *inbuf, unsigned int size, void *intp)
+{
+	if (sscanf(inbuf, "%i", (int *)intp) != 1) return -EINVAL;
+	return 0;
+}
+
+int proc_fetch_uint(const char *dir, const char *fname,
+		    char *outbuf, unsigned int size, void *uintp)
+{
+	return snprintf(outbuf, size, "%u\n", *(unsigned int *)uintp);
+}
+
+int proc_commit_uint(const char *dir, const char *fname,
+		     const char *inbuf, unsigned int size, void *uintp)
+{
+	if (sscanf(inbuf, "%u", (unsigned int *)uintp) != 1) return -EINVAL;
+	return 0;
+}
+
+int proc_fetch_long(const char *dir, const char *fname,
+		    char *outbuf, unsigned int size, void *longp)
+{
+	return snprintf(outbuf, size, "%li\n", *(long *)longp);
+}
+
+int proc_commit_long(const char *dir, const char *fname,
+		     const char *inbuf, unsigned int size, void *longp)
+{
+	if (sscanf(inbuf, "%li", (long *)longp) != 1) return -EINVAL;
+	return 0;
+}
+
+int proc_fetch_ulong(const char *dir, const char *fname,
+		     char *outbuf, unsigned int size, void *ulongp)
+{
+	return snprintf(outbuf, size, "%lu\n", *(long *)ulongp);
+}
+
+int proc_commit_ulong(const char *dir, const char *fname,
+		      const char *inbuf, unsigned int size, void *ulongp)
+{
+	if (sscanf(inbuf, "%lu", (unsigned long *)ulongp) != 1) return -EINVAL;
+	return 0;
+}
+
+int proc_fetch_bool(const char *dir, const char *fname,
+		    char *outbuf, unsigned int size, void *boolp)
+{
+	if (*(int *)boolp) return snprintf(outbuf, size, "y\n");
+	else return snprintf(outbuf, size, "n\n");
+}
+
+int proc_commit_bool(const char *dir, const char *fname,
+		     const char *inbuf, unsigned int size, void *boolp)
+{
+	if (inbuf[0] == 'y' || inbuf[0] == 'Y')
+		*(int *)boolp = 1;
+	else if (inbuf[0] == 'n' || inbuf[0] == 'N')
+		*(int *)boolp = 0;
+	else return proc_commit_int(dir, fname, inbuf, size, boolp);
+	return 0;
+}
+
+/* Test code: delete me */
+static int number = 7;
+
+static int testfetch(const char *dirname,
+		     const char *filename,
+		     char *buffer,
+		     unsigned int size,
+		     void *arg)
+{
+	/* As an example, each one holds its own name */
+	return snprintf(buffer, size, "%s/%s\n", dirname, filename);
+}
+
+static int dirfunc(const char *dirname,
+		   const char *filename,
+		   struct proc_dircontents *buffer,
+		   unsigned int maxlen,
+		   void *arg)
+{
+	unsigned int used = 0;
+	char name[100];
+	unsigned int i;
+
+	for (i = 0; i < 10; i++) {
+		sprintf(name, "file-%u", i);
+		used = proc_add_dircontents(buffer, used, maxlen,
+					    S_IFREG|0400, testfetch,
+					    NULL, NULL, NULL, name);
+	}
+	/* And one infinite subdirectory example */
+	used = proc_add_dircontents(buffer, used, maxlen,
+				    S_IFDIR|0555, NULL, NULL, dirfunc, NULL,
+				    "subdir");
+	return proc_end_dircontents(buffer, used, maxlen);
+}
+
+static int __init init_test(void)
+{
+	int ret;
+	ret = proc("testdir", "number", number, int, 0644);
+	if (ret)
+		printk("Proc registration failed: %i\n", ret);
+	proc_dir("testdir", "subdir", dirfunc, NULL);
+	return 0;
+}
+
+static void __exit exit_test(void)
+{
+	unproc("testdir", "number");
+	unproc_dir("testdir", "subdir");
+}
+
+module_init(init_test);
+module_exit(exit_test);
diff -urN -I \$.*\$ --exclude TAGS -X /home/rusty/current-dontdiff --minimal linux-2.4.13-uml/fs/simpleproc/inode.c working-2.4.13-uml-proc/fs/simpleproc/inode.c
--- linux-2.4.13-uml/fs/simpleproc/inode.c	Thu Jan  1 10:00:00 1970
+++ working-2.4.13-uml-proc/fs/simpleproc/inode.c	Thu Nov  1 21:29:14 2001
@@ -0,0 +1,790 @@
+/*
+ * Simple /proc filesystem for Linux.
+ *
+ * Conceptually, there are two types of directories here: static
+ * (entries are created and deleted using
+ * register_proc/unregister_proc), and dynamic (contents are created
+ * on demand using a callback).
+ *
+ *   Copyright (C) 2001 Rusty Russell.
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 
+ */
+#include <linux/module.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/string.h>
+#include <linux/stat.h>
+#include <linux/slab.h>
+#include <linux/pagemap.h>
+#include <linux/simpleproc.h>
+#include <asm/semaphore.h>
+
+#include <asm/uaccess.h>
+
+/* Proc mount point */
+struct vfsmount *proc_mnt;
+
+/* Serialize insert/delete and mounts */
+static DECLARE_MUTEX(proc_semaphore);
+
+#define SIMPLE_PROCFS_MAGIC	0x62121174
+
+/* Start with this many bytes allocated for file read */
+#define PROCFS_START_FILE	64
+/* Start with this many bytes allocated for directory read */
+#define PROCFS_START_DIR	PAGE_SIZE
+/* Approximate upper ceiling for memory usage per fs */
+#define PROCFS_MAX_SIZE		PAGE_SIZE
+
+/* Pre-decls for assigning */
+static struct inode_operations proc_punt_inodeops;
+static struct file_operations proc_helper_fileops;
+static struct file_operations proc_helper_dirops;
+static struct file_operations proc_punt_dirops;
+static struct inode_operations proc_helper_inodeops;
+static struct super_operations proc_ops;
+static struct dentry_operations proc_dentry_ops;
+
+struct proc_buffer
+{
+	unsigned int maxlen;
+	unsigned int len;
+	/* One is for the nul terminator */
+	char buffer[1];
+};
+
+struct proc_data *__new_proc(void *arg,
+			     proc_fetchfn_t *fetch,
+			     proc_commitfn_t *commit,
+			     proc_dirfn_t *dir)
+{
+	struct proc_data *pdata;
+
+	pdata = kmalloc(sizeof(*pdata), GFP_KERNEL);
+	if (pdata) {
+		pdata->arg = arg;
+		pdata->fetch = fetch;
+		pdata->commit = commit;
+		pdata->dir = dir;
+	}
+	return pdata;
+}
+
+/* FIXME: I have no idea what all this does: stolen from old /proc --RR */
+static int proc_statfs(struct super_block *sb, struct statfs *buf)
+{
+	buf->f_type = SIMPLE_PROCFS_MAGIC;
+	buf->f_bsize = PAGE_SIZE/sizeof(long);
+	buf->f_bfree = 0;
+	buf->f_bavail = 0;
+	buf->f_ffree = 0;
+	buf->f_namelen = NAME_MAX;
+	return 0;
+}
+
+/* Convenience routine to make an inode */
+static struct inode *
+new_proc_inode(struct super_block *sb, int mode, int is_dynamic)
+{
+	struct inode * inode = new_inode(sb);
+
+	if (!inode)
+		return NULL;
+
+	inode->i_mode = mode;
+	inode->i_uid = 0;
+	inode->i_gid = 0;
+	inode->i_blksize = PAGE_CACHE_SIZE;
+	inode->i_blocks = 0;
+	inode->i_rdev = NODEV;
+	inode->i_mapping->a_ops = NULL;
+	inode->i_atime = inode->i_mtime = inode->i_ctime = CURRENT_TIME;
+	switch (mode & S_IFMT) {
+	case S_IFREG:
+		inode->i_fop = &proc_helper_fileops;
+		break;
+	case S_IFDIR:
+		if (is_dynamic) {
+			inode->i_fop = &proc_punt_dirops;
+			inode->i_op = &proc_punt_inodeops;
+		} else {
+			inode->i_fop = &proc_helper_dirops;
+			inode->i_op = &proc_helper_inodeops;
+		}
+		break;
+	default:
+		BUG();
+		break;
+	}
+
+	return inode;
+}
+
+/* Make a new proc entry in this directory: must be holding proc_semapore */
+static int make_proc_entry(struct dentry *dir,
+			   const char *fname,
+			   int mode,
+			   struct proc_data *pdata,
+			   int is_dynamic)
+{
+	struct inode *inode;
+	struct dentry *dentry;
+	struct qstr qstr;
+
+	/* Create qstr for this entry */
+	qstr.name = fname;
+	qstr.len = strlen(fname);
+	qstr.hash = full_name_hash(qstr.name, qstr.len);
+
+	/* You can't put a static proc entry in a dynamic dir */
+	if (dir->d_inode->i_op == &proc_punt_inodeops)
+		BUG();
+
+	/* Does it already exist? */
+	dentry = d_lookup(dir, &qstr);
+	if (dentry) {
+		dput(dentry);
+		return -EEXIST;
+	}
+
+	/* Doesn't exist: create inode */
+	inode = new_proc_inode(dir->d_sb, mode, is_dynamic);
+	if (!inode)
+		return -ENOMEM;
+
+	/* Create dentry */
+	dentry = d_alloc(dir, &qstr);
+	if (!dentry) {
+		iput(inode);
+		return -ENOMEM;
+	}
+	dentry->d_op = &proc_dentry_ops;
+	dentry->d_fsdata = pdata;
+	d_add(dentry, inode);
+
+	/* Pin the dentry here, so it doesn't get pruned */
+	dget(dentry);
+	return 0;
+}
+
+/* Create (static) proc directory if neccessary. */
+/* FIXME: Keep refcnt, so we can delete when no more users */
+static struct dentry *get_proc_dir(const char *dirname)
+{
+	struct dentry *dentry;
+	struct qstr qstr;
+	const char *delim;
+
+	/* FIXME: Definitely need a better way --RR */
+	dentry = dget(proc_mnt->mnt_sb->s_root);
+	delim = dirname;
+
+	for (;;) {
+		struct dentry *newdentry;
+
+		/* Ignore multiple slashes */ 
+		while (*delim == '/') delim++;
+		qstr.name = delim;
+		delim = strchr(qstr.name, '/');
+		if (!delim) delim = qstr.name + strlen(qstr.name);
+		qstr.len = delim-(char *)qstr.name;
+		qstr.hash = full_name_hash(qstr.name, qstr.len);
+
+		if (qstr.len == 0)
+			break;
+
+		/* If entry doesn't exist, create it */
+		while (!(newdentry = d_lookup(dentry, &qstr))) {
+			char fname[qstr.len+1];
+			int ret;
+
+			strncpy(fname, qstr.name, qstr.len);
+			fname[qstr.len] = '\0';
+			down(&proc_semaphore);
+			ret = make_proc_entry(dentry, fname, S_IFDIR|0555,
+					       NULL, 0);
+			up(&proc_semaphore);
+
+			if (ret < 0) {
+				dput(dentry);
+				return ERR_PTR(ret);
+			}
+		}
+		dput(dentry);
+		dentry = newdentry;
+	}
+	return dentry;
+}
+
+/* Actually add a proc file or dynamic directory */
+int __proc(const char *dirname, const char *fname, int mode,
+	   struct proc_data *pdata)
+{
+	struct dentry *dir;
+	int ret;
+
+	if (!pdata)
+		return -ENOMEM;
+
+	dir = get_proc_dir(dirname);
+	if (IS_ERR(dir))
+		return PTR_ERR(dir);
+
+	ret = make_proc_entry(dir, fname, mode, pdata, S_ISDIR(mode) ? 1 : 0);
+	dput(dir);
+	return ret;
+}
+
+static int proc_nofetch(const char *dirname, const char *fname,
+			char *outbuf, unsigned int len, void *arg)
+{
+	return -ENOENT;
+}
+
+static int proc_nocommit(const char *dirname, const char *fname,
+			 const char *inbuf, unsigned int len, void *arg)
+{
+	return -ENOENT;
+}
+
+static int proc_nodir(const char *dirname,
+		      const char *filename,
+		      struct proc_dircontents *buffer,
+		      unsigned int size,
+		      void *arg)
+{
+	return -ENOENT;
+}
+
+/* Release a proc entry */
+void unproc(const char *dir, const char *fname)
+{
+	struct dentry *dentry;
+	const char *delim;
+	struct qstr qstr;
+	struct proc_data *pdata;
+
+	/* FIXME: There's a better way, right? --RR */
+	dentry = dget(proc_mnt->mnt_sb->s_root);
+
+	delim = dir;
+	for (;;) {
+		/* Ignore multiple slashes */ 
+		while (*delim == '/') delim++;
+		qstr.name = delim;
+		delim = strchr(qstr.name, '/');
+		if (!delim) delim = qstr.name + strlen(qstr.name);
+		qstr.len = delim-(char *)qstr.name;
+		qstr.hash = full_name_hash(qstr.name, qstr.len);
+
+		if (qstr.len == 0)
+			break;
+
+		dentry = d_lookup(dentry, &qstr);
+		if (!dentry)
+			BUG();
+		dput(dentry->d_parent);
+	}
+
+	qstr.name = fname;
+	qstr.len = strlen(fname);
+	qstr.hash = full_name_hash(qstr.name, qstr.len);
+	dentry = d_lookup(dentry, &qstr);
+	if (!dentry)
+		BUG();
+	dput(dentry->d_parent);
+
+	/* We have the dentry: change the private area so it doesn't
+           enter the caller any more. */
+	pdata = dentry->d_fsdata;
+	pdata->commit = proc_nocommit;
+	pdata->fetch = proc_nofetch;
+	pdata->dir = proc_nodir;
+
+	/* This will probably free the dentry immediately, but if not,
+           too bad. */
+	dput(dentry);
+	dput(dentry);
+}
+
+void unproc_dir(const char *dir, const char *fname)
+{
+	unproc(dir, fname);
+}
+
+/* See if /proc entry exists (entries registered in directory). */
+static struct dentry *proc_lookup(struct inode *dir,
+					 struct dentry *dentry)
+{
+	/* Since we place new staticn entries in the dcache, if we get
+	   here, we know the entry does not exist.  Create a negative
+	   dentry, and return NULL */
+	d_add(dentry, NULL);
+	return NULL;
+}
+
+/* Call callback to get directory contents */
+static struct proc_dircontents *get_dir_contents(const char *dirname,
+						 const char *filename,
+						 struct proc_data *pdata)
+{
+	struct proc_dircontents *ret;
+	unsigned int size = PROCFS_START_DIR;
+
+	ret = kmalloc(size, GFP_KERNEL);
+	while (ret) {
+		int used;
+		used = pdata->dir(dirname, filename, ret, size, pdata->arg);
+		if (used < 0) {
+			kfree(ret);
+			return ERR_PTR(used);
+		}
+		if (used <= size)
+			return ret;
+
+		/* Realloc larger and loop */
+		kfree(ret);
+		size = used;
+		ret = kmalloc(size, GFP_KERNEL);
+	}
+	return ERR_PTR(-ENOMEM);
+}
+
+/* Incrementing is a little tricky: round up to alignment */
+static struct proc_dircontents *next_dcont(struct proc_dircontents *dcontents)
+{
+	unsigned int len;
+
+	len = ((sizeof(*dcontents) + strlen(dcontents->name) + 1
+		+ __alignof__(*dcontents) - 1)
+	       & ~(__alignof__(*dcontents) - 1));
+	return (void *)dcontents + len;
+}
+
+/* Search results from callback for this name, and if found create inode */
+static struct proc_dircontents *
+find_dcontents(struct proc_dircontents *dir_contents,
+	       struct dentry *dentry)
+{
+	while (dir_contents->mode) {
+		if (strcmp(dentry->d_name.name, dir_contents->name) == 0)
+			return dir_contents;
+		dir_contents = next_dcont(dir_contents);
+	}
+	/* Not found... */
+	return NULL;
+}
+
+/* Since there are no hard links in this filesystem, we can simply map
+   inodes to dentries.  This is not possibly in general! */
+static struct dentry *inode_to_dentry(struct inode *inode)
+{
+	if (inode->i_dentry.next->next != &inode->i_dentry)
+		BUG();
+	return list_entry(inode->i_dentry.next, struct dentry, d_alias);
+}
+
+/* See if /proc entry exists (user controls contents of directory). */
+static struct dentry *proc_punt_lookup(struct inode *dir,
+				       struct dentry *dentry)
+{
+	/* We do the whole callback on every lookup. */
+	struct proc_data *pdata;
+	struct proc_dircontents *dir_contents, *dc;
+	struct inode *inode;
+	struct dentry *parent;
+
+	/* Since we know the inode is a directory, there is only one
+           inode in the dentry alias list, so mapping inode -> dentry
+           is easy */
+	parent = inode_to_dentry(dir);
+	dir_contents = get_dir_contents(parent->d_name.name,
+					dentry->d_name.name,
+					parent->d_fsdata);
+	if (!dir_contents || IS_ERR(dir_contents))
+		return (struct dentry *)dir_contents;
+
+	/* Looks through callback-supplied list for this dentry */ 
+	dc = find_dcontents(dir_contents, dentry);
+	if (!dc) {
+		kfree(dir_contents);
+		return NULL;
+	}
+	inode = new_proc_inode(dentry->d_sb, dc->mode, 1);
+	if (!inode) {
+		kfree(dir_contents);
+		return ERR_PTR(-ENOMEM);
+	}
+	pdata = __new_proc(dc->arg, dc->fetch, dc->commit, dc->dir);
+	if (!pdata) {
+		iput(inode);
+		kfree(dir_contents);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	dentry->d_op = &proc_dentry_ops;
+	dentry->d_fsdata = pdata;
+	d_add(dentry, inode);
+	kfree(dir_contents);
+	return NULL;
+}
+
+/* On open, we grab contents if we're readable... */
+static int proc_file_snapshot(struct inode *inode, struct file *filp)
+{
+	unsigned int size;
+	struct proc_buffer *buf;
+	struct proc_data *pdata;
+	char *dirname;
+	unsigned long page;
+
+	pdata = filp->f_dentry->d_fsdata;
+	/* Start at this, and work up */
+	size = PROCFS_START_FILE;
+
+	if (!(filp->f_mode & FMODE_READ)) {
+		/* Allocate write buffer: */
+		buf = kmalloc(sizeof(*buf) + size, GFP_KERNEL);
+		if (!buf)
+			return -ENOMEM;
+		buf->maxlen = size;
+		buf->len = 0;
+		filp->private_data = buf;
+		return 0;
+	}
+
+	/* For the moment, you can't open for read & write.  Later,
+           when seek resets snapshot/commit, we should allow this. */
+	if (filp->f_mode & FMODE_WRITE)
+		return -EINVAL;
+
+	page = __get_free_page(GFP_USER);
+	if (!page)
+		return -ENOMEM;
+
+	/* FIXME: This is not right: the callbacks don't care what the
+	   process's idea of root is, it only wants path after proc/. */
+	dirname = d_path(filp->f_dentry, filp->f_vfsmnt,
+			 (char *)page, PAGE_SIZE);
+	if (dirname < (char *)page)
+		BUG();
+
+	/* buf[0] holds the size */
+	buf = kmalloc(sizeof(*buf)+size, GFP_KERNEL);
+	while (buf) {
+		int used;
+		used = pdata->fetch(dirname, filp->f_dentry->d_name.name,
+				    buf->buffer, size, pdata->arg);
+		if (used < 0) {
+			kfree(buf);
+			free_page(page);
+			/* FIXME: if used == -ENOENT, destroy dcache entry */
+			return used;
+		}
+		if (used <= size) {
+			free_page(page);
+			filp->private_data = buf;
+			/* Nul terminate and save size */
+			buf->maxlen = size;
+			buf->len = used;
+			buf->buffer[used] = '\0';
+			return 0;
+		}
+
+		/* Realloc larger and loop */
+		kfree(buf);
+		size = used;
+		if (size > PROCFS_MAX_SIZE)
+			break;
+		buf = kmalloc(sizeof(*buf)+size, GFP_KERNEL);
+	}
+	free_page(page);
+	return -ENOMEM;
+}
+
+/* On close, we commit contents if we've been written to... */
+static int proc_file_commit(struct inode *inode, struct file *filp)
+{
+	int ret;
+	struct proc_data *pdata;
+	struct proc_buffer *buf;
+	char *dirname;
+	unsigned long page;
+
+	pdata = filp->f_dentry->d_fsdata;
+	if (!(filp->f_mode & FMODE_WRITE)) {
+		kfree(filp->private_data);
+		return 0;
+	}
+
+	page = __get_free_page(GFP_USER);
+	if (!page) {
+		kfree(filp->private_data);
+		return -ENOMEM;
+	}
+
+	/* FIXME: This is not right: the callbacks don't care what the
+	   process's idea of root is, it only wants path after proc/. */
+	dirname = d_path(filp->f_dentry, filp->f_vfsmnt,
+			 (char *)page, PAGE_SIZE);
+	if (dirname < (char *)page)
+		BUG();
+
+	/* nul-terminate buffer */
+	buf = filp->private_data;
+	buf->buffer[buf->len] = '\0';
+	ret = pdata->commit(dirname, filp->f_dentry->d_name.name,
+			    buf->buffer, buf->len, pdata->arg);
+	
+	kfree(filp->private_data);
+	free_page(page);
+	return ret;
+}
+
+/* Copy from buffer */
+static ssize_t proc_file_read(struct file *filp, char *ubuf, size_t size,
+			      loff_t *off)
+{
+	struct proc_buffer *buf;
+	struct inode *inode;
+
+	/* Use inode semaphore to serialize against writes. */
+	inode = filp->f_dentry->d_inode;
+	if (down_interruptible(&inode->i_sem) != 0)
+		return -EINTR;
+
+	buf = filp->private_data;
+	if (size + *off > buf->len)
+		size = buf->len - *off;
+
+	/* Copy from static buffer */
+	if (copy_to_user(ubuf, buf->buffer, size) != 0) {
+		up(&inode->i_sem);
+		return -EFAULT;
+	}
+	up(&inode->i_sem);
+
+	*off += size;
+	return (ssize_t)size;
+}
+
+/* Copy to buffer */
+static ssize_t proc_file_write(struct file *filp,
+			       const char *ubuf,
+			       size_t size,
+			       loff_t *off)
+{
+	struct inode *inode;
+	struct proc_buffer *buf;
+	struct proc_data *pdata;
+
+	pdata = filp->f_dentry->d_fsdata;
+
+	/* Use inode semaphore to serialize writes & reads. */
+	inode = filp->f_dentry->d_inode;
+	if (down_interruptible(&inode->i_sem) != 0)
+		return -EINTR;
+
+	buf = filp->private_data;
+	if (*off + size > buf->maxlen) {
+		struct proc_buffer *newbuffer;
+		/* Prevent them using too much memory */
+		if (*off + size > PROCFS_MAX_SIZE) {
+			up(&inode->i_sem);
+			return -ENOSPC;
+		}
+		/* Room for count at head */
+		newbuffer = kmalloc(sizeof(*newbuffer) + *off + size,
+				    GFP_USER);
+		if (!newbuffer) {
+			up(&inode->i_sem);
+			return -ENOMEM;
+		}
+		memcpy(newbuffer, buf, sizeof(*buf) + buf->len);
+		kfree(filp->private_data);
+		filp->private_data = buf = newbuffer;
+	}
+
+	/* Do actual copy */
+	if (copy_from_user(buf->buffer + *off, ubuf, size) != 0) {
+		up(&inode->i_sem);
+		return -EFAULT;
+	}
+	up(&inode->i_sem);
+	buf->len += size;
+	*off += size;
+
+	return size;
+}
+
+/* Call the user's callback to get contents of this directory.
+   Generate . and .. automagically. */
+static int proc_dynamic_readdir(struct file *filp,
+				void *dirent,
+				filldir_t filldir)
+{
+	int i;
+	struct proc_dircontents *dcontents, *dp;
+	char *dirname;
+	unsigned long page;
+	struct proc_data *pdata;
+	struct dentry *dentry = filp->f_dentry;
+
+	pdata = filp->f_dentry->d_fsdata;
+
+	i = filp->f_pos;
+	switch (i) {
+	case 0:
+		if (filldir(dirent, ".", 1, 0, dentry->d_inode->i_ino, DT_DIR)
+		    < 0)
+			break;
+		i++;
+		filp->f_pos++;
+		/* fallthrough */
+	case 1:
+		if (filldir(dirent, "..", 2, 0,
+			    dentry->d_parent->d_inode->i_ino, DT_DIR) < 0)
+			break;
+		i++;
+		filp->f_pos++;
+	}
+
+	page = __get_free_page(GFP_USER);
+	if (!page) return -ENOMEM;
+
+	/* FIXME: This is not right: the callbacks don't care what the
+	   process's idea of root is, it only wants path after proc/. */
+	dirname = d_path(filp->f_dentry, filp->f_vfsmnt,
+			 (char *)page, PAGE_SIZE);
+	if (dirname < (char *)page)
+		BUG();
+
+	/* Call user callback to get directory */
+	dcontents = get_dir_contents(dirname,
+				     dentry->d_name.name,
+				     pdata);
+	if (IS_ERR(dcontents)) {
+		free_page(page);
+		return PTR_ERR(dcontents);
+	}
+
+	/* Skip any already-read entries... */
+	for (dp = dcontents, i -= 2; dp->mode && i; dp = next_dcont(dp), i++);
+
+	for (; dp->mode; dp = next_dcont(dp)) {
+		/* FIXME: Use non-zero inode numbers */
+		if (filldir(dirent, dp->name, strlen(dp->name),
+			    filp->f_pos,
+			    filp->f_pos,
+			    S_ISDIR(dp->mode) ? DT_DIR : DT_REG) < 0)
+			break;
+		filp->f_pos++;
+	}
+	free_page(page);
+	kfree(dcontents);
+	return 0;
+}
+
+/* Free the private area when dentry is freed. */
+static void proc_release(struct dentry *dentry)
+{
+	kfree(dentry->d_fsdata);
+}
+
+static struct super_block *proc_read_super(struct super_block *s,
+					   void *data, 
+					   int silent)
+{
+	struct inode * root_inode;
+
+	s->s_blocksize = 1024;
+	s->s_blocksize_bits = 10;
+	s->s_magic = SIMPLE_PROCFS_MAGIC;
+	s->s_op = &proc_ops;
+
+	root_inode = new_proc_inode(s, S_IFDIR|0555, 0);
+	if (!root_inode) return NULL;
+
+	/* Block concurrent mounts */
+	down(&proc_semaphore);
+
+	s->s_root = d_alloc_root(root_inode);
+	if (!s->s_root) {
+		iput(root_inode);
+		up(&proc_semaphore);
+		return NULL;
+	}
+	up(&proc_semaphore);
+	return s;
+}
+
+/* Proc files use these wrappers */
+static struct file_operations proc_helper_fileops = {
+	open:		proc_file_snapshot,
+	release:	proc_file_commit,
+	read:		proc_file_read,
+	write:		proc_file_write,
+};
+
+/* Directories which use normal registration mechanism, which sit in
+   the dcache */
+static struct file_operations proc_helper_dirops = {
+	read:		generic_read_dir,
+	readdir:	dcache_readdir,
+};
+
+/* Directories which have their own dynamic content */
+static struct file_operations proc_punt_dirops = {
+	read:		generic_read_dir,
+	readdir:	proc_dynamic_readdir,
+};
+
+/* You can only do lookups through these dirs: dynamic ones do callbacks... */
+static struct inode_operations proc_punt_inodeops = {
+	lookup:		proc_punt_lookup,
+};
+
+/* ... static ones look up registrations */
+static struct inode_operations proc_helper_inodeops = {
+	lookup:		proc_lookup,
+};
+
+static struct super_operations proc_ops = {
+	statfs:		proc_statfs,
+	put_inode:	force_delete,
+};
+
+static struct dentry_operations proc_dentry_ops = {
+	d_release:	proc_release,
+};
+
+static DECLARE_FSTYPE(proc_fs_type, "proc", proc_read_super, FS_SINGLE);
+
+static int __init init_proc_fs(void)
+{
+	register_filesystem(&proc_fs_type);
+	proc_mnt = kern_mount(&proc_fs_type);
+	return 0;
+}
+
+static void __exit exit_proc_fs(void)
+{
+	unregister_filesystem(&proc_fs_type);
+}
+
+module_init(init_proc_fs);
+module_exit(exit_proc_fs);
+

Re: [PATCH] 2.5 PROPOSAL: Replacement for current /proc of shit.

[Tim Jansen]

On Sunday 04 November 2001 03:08, Jakob Østergaard wrote:
> Now, if established files in proc could just be stable, so that they would
> not change unless non-backwards-compatible information absolutely must be
> presented, that would be a major step in the right direction.  Further, if
> we could find some acceptable compromise between human- and machine-
> readability, as has happened in the past...

The problem is that it is almost impossible to offer human-readable 
interfaces that will be backward-compatible. As soon as you have a 
well-formatted output, like /proc/partitions, you can not add a new field 
without breaking user-space applications. 
What you could do is to establish rules for files like /proc/partitions ("if 
there are more than 4 space-separated alphanumeric strings per line in 
/proc/partitions then ignore the additional fields"), but you won't find such 
a rule that is useful for every file and still offers a nice human-readable 
format. And it will be quite hard to be sure that everybody really sticks to 
these rules. Alternatively you could use a semi-human-readable format like 
XML, which several people have proposed, but it seemed like almost nobody 
liked it.

IMHO there shouldn't be any 'presentation logic' in the kernel. If you need 
the things in a human-friendly format, write a 3 line shell script:

for I in `ls -d /proc/partitions/*` ; do
	echo `cat $I/major` `cat $I/minor` `cat $I/blocks` `cat $I/name`
done

bye...

Re: [PATCH] 2.5 PROPOSAL: Replacement for current /proc of shit.

[Daniel Kobras]

On Sun, Nov 04, 2001 at 01:30:06PM +0100, Tim Jansen wrote:
> The problem is that it is almost impossible to offer human-readable 
> interfaces that will be backward-compatible. As soon as you have a 
> well-formatted output, like /proc/partitions, you can not add a new field 
> without breaking user-space applications. 
> What you could do is to establish rules for files like /proc/partitions ("if 
> there are more than 4 space-separated alphanumeric strings per line in 
> /proc/partitions then ignore the additional fields"), but you won't find such 
> a rule that is useful for every file and still offers a nice human-readable 
> format.

Certainly you can further fields without breaking (well-written) apps. That's
what the first line in /proc/partitions is for. When adding a new column,
you also give it a new tag in the header. Ask RedHat how many apps broke
when they started patching sard into their kernels.

Adding new fields is even easier with /proc/stat-style key:value pairs. Both
styles are human- as well as machine readable. Problems only arise when
someone changes the semantics of a certain field without changing the tag.
But luckily these kinds of changes never happen in a stable kernel series...

Regards,

Daniel.

Re: [PATCH] 2.5 PROPOSAL: Replacement for current /proc of shit.

[Tim Jansen]

On Sunday 04 November 2001 14:36, Daniel Kobras wrote:
> Certainly you can further fields without breaking (well-written) apps.
> That's what the first line in /proc/partitions is for. When adding a new
> column, you also give it a new tag in the header. Ask RedHat how many apps
> broke when they started patching sard into their kernels.

The format won't help you when you have strings with whitespace or if you 
want to export a list for each partition. 

> Adding new fields is even easier with /proc/stat-style key:value pairs.
> Both styles are human- as well as machine readable. Problems only arise
> when someone changes the semantics of a certain field without changing the
> tag. But luckily these kinds of changes never happen in a stable kernel
> series...

I don't think that this format is very user friendly, and it has the same 
limitations as /proc/partitions. 

The problem is not that it is impossible to invent a new format for every 
file. The problem is that you need a different format for each file.

bye...

PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

Here's my stab at the problems - please comment,

We want to avoid these problems:
 1)  It is hard to parse (some) /proc files from userspace
 2)  As /proc files change, parsers must be changed in userspace

Still, we want to keep on offering
 3)  Human readable /proc files with some amount of pretty-printing
 4)  A /proc fs that can be changed as the kernel needs those changes


Taking care of (3) and (4):

Maintaining the current /proc files is very simple, and it offers the system
administrator a lot of functionality that isn't reasonable to take away now. 

       * They should stay in a form close to the current one *


Taking care of (1) and (2):

For each file "f" in /proc, there will be a ".f" file which is a
machine-readable version of "f", with the difference that it may contain extra
information that one may not want to present to the user in "f".

The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it is a
list of elements, wherein an element can itself be a list (or a character string,
or a host-native numeric type.  Thus, (key,value) pairs and lists thereof are
possible, as well as tree structures etc.

All data types are stored in the architecture-native format, and a simple
library should be sufficient to parse any dot-proc file.


So, we need a small change in procfs that does not in any way break
compatibility - and we need a few lines of C under LGPL to interface with it.

Tell me what you think - It is possible that I could do this (or something
close) in the near future, unless someone shows me the problem with the
approach.

Thank you,

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Gábor Lénárt]

On Sun, Nov 04, 2001 at 04:33:54PM +0100, Jakob ?stergaard wrote:
> For each file "f" in /proc, there will be a ".f" file which is a
> machine-readable version of "f", with the difference that it may contain extra
> information that one may not want to present to the user in "f".
> 
> The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it is a
> list of elements, wherein an element can itself be a list (or a character string,
> or a host-native numeric type.  Thus, (key,value) pairs and lists thereof are
> possible, as well as tree structures etc.
> 
> All data types are stored in the architecture-native format, and a simple
> library should be sufficient to parse any dot-proc file.

Hmmmm. If someone would be able to implement new architecture which can
provide 1:1 sysctl/procfs support, there would be need for user space
programs parse proc filesystem. Then, /proc would be only good to administrators
to echo to/cat entries. So compatibility with old design can remain, and
new programs would be able to use the much more versatile sysctl support.
OK, it's a hard guess only. ;-)


- Gabor

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

On November 4, 2001 04:33 pm, Jakob Østergaard wrote:
> Here's my stab at the problems - please comment,
> 
> We want to avoid these problems:
>  1)  It is hard to parse (some) /proc files from userspace
>  2)  As /proc files change, parsers must be changed in userspace
> 
> Still, we want to keep on offering
>  3)  Human readable /proc files with some amount of pretty-printing
>  4)  A /proc fs that can be changed as the kernel needs those changes
> 
> 
> Taking care of (3) and (4):
> 
> Maintaining the current /proc files is very simple, and it offers the system
> administrator a lot of functionality that isn't reasonable to take away 
now. 
> 
>        * They should stay in a form close to the current one *
> 
> 
> Taking care of (1) and (2):
> 
> For each file "f" in /proc, there will be a ".f" file which is a
> machine-readable version of "f", with the difference that it may contain 
extra
> information that one may not want to present to the user in "f".
> 
> The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it 
is a
> list of elements, wherein an element can itself be a list (or a character 
string,
> or a host-native numeric type.  Thus, (key,value) pairs and lists thereof 
are
> possible, as well as tree structures etc.
> 
> All data types are stored in the architecture-native format, and a simple
> library should be sufficient to parse any dot-proc file.
> 
> 
> So, we need a small change in procfs that does not in any way break
> compatibility - and we need a few lines of C under LGPL to interface with 
it.
> 
> Tell me what you think - It is possible that I could do this (or something
> close) in the near future, unless someone shows me the problem with the
> approach.
> 
> Thank you,

While the basic idea is attractive for a number of reasons, there are more 
than a few questions to answer.  Take a look at a typical proc function, 
meminfo_read_proc for example.  Its active ingredient is basically a sprintf 
function:

        len += sprintf(page+len,
                "MemTotal:     %8lu kB\n"
                "MemFree:      %8lu kB\n"
                "MemShared:    %8lu kB\n"
		...,
                K(i.totalram),
                K(i.freeram),
                K(i.sharedram),
		...);

What does the equivalent look like under your scheme?   Does it remain 
localized in one proc routine, or does it get spread out over a few 
locations, possibibly with a part of the specification outside the 
kernel?  Do the titles end up in your dotfile?  How do you specify whatever 
formatting is necessary to transform a dotfile into normal /proc output?  Is 
this transformation handled in user space or the kernel?  How much library 
support is needed?

--
Daniel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 05:31:50PM +0100, Daniel Phillips wrote:
...
> While the basic idea is attractive for a number of reasons, there are more 
> than a few questions to answer.  Take a look at a typical proc function, 
> meminfo_read_proc for example.  Its active ingredient is basically a sprintf 
> function:
> 
>         len += sprintf(page+len,
>                 "MemTotal:     %8lu kB\n"
>                 "MemFree:      %8lu kB\n"
>                 "MemShared:    %8lu kB\n"
> 		...,
>                 K(i.totalram),
>                 K(i.freeram),
>                 K(i.sharedram),
> 		...);
> 
> What does the equivalent look like under your scheme?

Well, in my previous mail I gave a vague description of the "media" but
the actual semantics of the informaiton.

I *suppose* we would represent the above like:
( ("version", 1),
  ("totalram", 1123412),
  ("freeram",  1243),
  ("sharedram", 234) )

This would be encoded in some simple binary form. Let's not get hung up in
those details for now.

Your example was very simple, and my solution is simple too - a list of
key/value pairs.  But my scheme will allow for tree structures, longer
lists, etc. etc.  too.

The API would probably be something like (I have 45 seconds of thought behind
this API so don't consider this a final draft)

dp_list * itall = new_dp_list();
add_dp_key(itall, "version", new_dp_int(1));
add_dp_key(itall, "totalram", new_dp_int(i.totalram));
add_dp_key(itall, "freeram", new_dp_int(i.freeram));
add_dp_key(itall, "sharedram", new_dp_int(i.sharedram));
len += dp_commit(page+len, itall);


dp_list * new_dp_list(void) 
 Creates a new empty list

void add_dp_key(dp_list*, const char*, dp_element*)
 Creates a two-element list, filling in the two elements
 with a string and a value element of arbitrary type.
 This list is appended as the new last element in the
 list given as the first argument.

int dp_commit(char*, dp_list*)
 Will encode the list argument into the buffer given,
 and free the entire list.

---

Now I agree that this API hurts the eyes already now - it should
not require a myriad of small structures with pointers to everywhere
to be allocated and de-allocated.   I can do better than this,
trust me,  but I can't do that until tomorrow  ;)

>   Does it remain 
> localized in one proc routine, or does it get spread out over a few 
> locations, possibibly with a part of the specification outside the 
> kernel?  Do the titles end up in your dotfile?  How do you specify whatever 
> formatting is necessary to transform a dotfile into normal /proc output?  Is 
> this transformation handled in user space or the kernel?  How much library 
> support is needed?

Wherever you have a routine for filling out a proc file, you'd add a
routine for filling out the dot-proc file.

Simple.  Keeping the routines together makes it more likely that the
maintainers and occational hackers will keep the two fairly well in sync.

(remember - this scheme does not *require* anything to be in sync though)


The dot-proc file does not generate the normal proc output.  Let the pretty
printing happen as it does today - changing a semicolon anywhere near an
existing proc routine is going to break half a gazillion programs in userland
anyway.

The *only* library support you need is
1)  Parsing of the dot-proc files into structures convenient for the
    language at hand (C first).
2)  Handling of the resulting structures, eg. allocation and de-allocation,
    if this is at all necessary - it may not be...


I'm not pushing style-sheets into the kernel  :)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 16:33, you wrote:
> Maintaining the current /proc files is very simple, and it offers the
> system administrator a lot of functionality that isn't reasonable to take
> away now.
>        * They should stay in a form close to the current one *

I doubt that it is worthwhile to keep them in the current form for any other 
reason than compatibility (with existing software and people's habits). 
It doesn't make sense to describe things in 200 different formats, you won't 
help anybody with that. It also violates the good old principle of keeping 
policy out of the kernel. And, for me, layout is clearly policy.

The reason for proc's popularity is clearly that you can use any tool, from 
cat over more/less to the text editor of choice, and read the files. There 
should be ways to achieve this without putting things into the kernel.  Is 
there is a way to implement a filesystem in user-space? What you could do is 
to export the raw data using single-value-files, XML or whatever and then 
provide an emulation of the old /proc files and possibly new ones in user 
space. This could be as simple as writing a shell-script for each emulated 
file.


> The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it
> is a list of elements, wherein an element can itself be a list (or a

Why would anybody want a binary encoding? 
It needs special parsers and will be almost impossible to access from shell 
scripts. 

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

On November 4, 2001 05:45 pm, Tim Jansen wrote:
> > The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it
> > is a list of elements, wherein an element can itself be a list (or a
> 
> Why would anybody want a binary encoding? 

Because they have a computer?

> It needs special parsers and will be almost impossible to access from shell 
> scripts. 

No, look, he's proposing to put the binary encoding in hidden .files.  The 
good old /proc files will continue to appear and operate as they do now.

--
Daniel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 06:28:47PM +0100, Daniel Phillips wrote:
> On November 4, 2001 05:45 pm, Tim Jansen wrote:
> > > The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it
> > > is a list of elements, wherein an element can itself be a list (or a
> > 
> > Why would anybody want a binary encoding? 
> 
> Because they have a computer?

Yes - good reason  :)

The "fuzzy parsing" userland has to do today to get useful information
out of many proc files today is not nice at all.  It eats CPU, it's 
error-prone, and all in all it's just "wrong".

However - having a human-readable /proc that you can use directly with
cat, echo,  your scripts,  simple programs using read(), etc.   is absolutely
a *very* cool feature that I don't want to let go.  It is just too damn
practical.

But building a piece of software that needs to reliably read out status
information from a system providing something more and more resembling a GUI in
text-files is becoming unnecessarily time-consuming and error-prone.

> 
> > It needs special parsers and will be almost impossible to access from shell 
> > scripts. 
> 
> No, look, he's proposing to put the binary encoding in hidden .files.  The 
> good old /proc files will continue to appear and operate as they do now.
> 

Exactly.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[SpaceWalker]

Jakob Østergaard wrote:
> 
> On Sun, Nov 04, 2001 at 06:28:47PM +0100, Daniel Phillips wrote:
> > On November 4, 2001 05:45 pm, Tim Jansen wrote:
> > > > The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it
> > > > is a list of elements, wherein an element can itself be a list (or a
> > >
> > > Why would anybody want a binary encoding?
> >
> > Because they have a computer?
> 
> Yes - good reason  :)
> 
> The "fuzzy parsing" userland has to do today to get useful information
> out of many proc files today is not nice at all.  It eats CPU, it's
> error-prone, and all in all it's just "wrong".
> 
> However - having a human-readable /proc that you can use directly with
> cat, echo,  your scripts,  simple programs using read(), etc.   is absolutely
> a *very* cool feature that I don't want to let go.  It is just too damn
> practical.
> 
> But building a piece of software that needs to reliably read out status
> information from a system providing something more and more resembling a GUI in
> text-files is becoming unnecessarily time-consuming and error-prone.
> 
> >
> > > It needs special parsers and will be almost impossible to access from shell
> > > scripts.
> >
> > No, look, he's proposing to put the binary encoding in hidden .files.  The
> > good old /proc files will continue to appear and operate as they do now.
> >
> 
> Exactly.
> 
> --
> ................................................................
> :   jakob@unthought.net   : And I see the elder races,         :
> :.........................: putrid forms of man                :
> :   Jakob Østergaard      : See him rise and claim the earth,  :
> :        OZ9ABN           : his downfall is at hand.           :
> :.........................:............{Konkhra}...............:
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

A good reason could be that a simple ps -aux uses hundreds of system
calls to get the list of all the processes ...
-- 
SpaceWalker

spacewalker@altern.org
ICQ 36157579

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Albert D. Cahalan]

SpaceWalker writes:

> A good reason could be that a simple ps -aux uses hundreds of system
> calls to get the list of all the processes ...

First of all, "ps -aux" isn't correct usage. It is accepted only
as long as you don't have a username "x". Try "ps aux" instead.
(good versions of ps will print a warning -- use Debian)

Second of all, if you want a really fast ps, look here:
http://lwn.net/2000/0420/a/atomicps.html

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[John Levon]

On Sun, Nov 04, 2001 at 06:41:59PM +0100, Jakob Østergaard wrote:

> The "fuzzy parsing" userland has to do today to get useful information
> out of many proc files today is not nice at all.  It eats CPU, it's 
> error-prone, and all in all it's just "wrong".

This is because the files are human-readable, nothing to do with binary vs. plain
text. proc should be made (entirely ?) of value-per-file trees, and a back-compat
compatprocfs union mounted for the files people and programs are expecting.

> However - having a human-readable /proc that you can use directly with
> cat, echo,  your scripts,  simple programs using read(), etc.   is absolutely
> a *very* cool feature that I don't want to let go.  It is just too damn
> practical.

I don't see that it's at all useful: it just makes life harder. You yourself
state above that read(2) parsing of human readable files is "not nice at all",
and now you're saying it is "just too damn practical".

Just drop the human-readable stuff from the new /proc, please.

In what way is parsing /proc/meminfo in a script more practical than 
cat /proc/meminfo/total ?

> > No, look, he's proposing to put the binary encoding in hidden .files.  The 
> > good old /proc files will continue to appear and operate as they do now.
> > 
>
> Exactly.

This just seems needless duplication, and fragile. Representing things as directory
hierarchies and single-value files in text seems to me to be much nicer, just as
convenient, and much nicer for fs/proc/ source...

IMHO
john

-- 
"All this just amounts to more grist for the mill of the ill."
	- Elizabeth Wurtzel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 05:59:45PM +0000, John Levon wrote:
> On Sun, Nov 04, 2001 at 06:41:59PM +0100, Jakob Østergaard wrote:
> 
> > The "fuzzy parsing" userland has to do today to get useful information
> > out of many proc files today is not nice at all.  It eats CPU, it's 
> > error-prone, and all in all it's just "wrong".
> 
> This is because the files are human-readable, nothing to do with binary vs. plain
> text. proc should be made (entirely ?) of value-per-file trees, and a back-compat
> compatprocfs union mounted for the files people and programs are expecting.

So you want generaiton and parsing of text strings whenever we pass an int from
the kernel ?

> 
> > However - having a human-readable /proc that you can use directly with
> > cat, echo,  your scripts,  simple programs using read(), etc.   is absolutely
> > a *very* cool feature that I don't want to let go.  It is just too damn
> > practical.
> 
> I don't see that it's at all useful: it just makes life harder. You yourself
> state above that read(2) parsing of human readable files is "not nice at all",
> and now you're saying it is "just too damn practical".

cat /proc/mdstat    - that's practical !
cat /proc/cpuinfo   - equally so

Anyway - I won't involve myself in the argument whether we should keep
the old /proc or not - I wanted to present my idea how we could overcome
some fundamental problems in the existing framework,  non-intrusively.

> 
> Just drop the human-readable stuff from the new /proc, please.

I don't care enough about it to discuss it now, but I'm sure others do  ;)

> 
> In what way is parsing /proc/meminfo in a script more practical than 
> cat /proc/meminfo/total ?

I see your point.

There's some system overhead when converting text/integer values, but
if you're polling so often I guess you have other problems anyway...

...
> 
> This just seems needless duplication, and fragile. Representing things as directory
> hierarchies and single-value files in text seems to me to be much nicer, just as
> convenient, and much nicer for fs/proc/ source...

I like the idea of single-value files.

But then how do we get the nice summary information we have today ?

Hmm...   How about:

  /proc/meminfo    - as it was
  /proc/.meminfo/  - as you suggested

That way we keep /proc looking like it was, while offering the very nice
single-value file interface to apps that needs it.

I could even live with text encoding of the values - I just hate not being able
to tell if it's supposed to be i32/u32/i64/u64/float/double/...  from looking
at the variable.   Type-less interfaces with implicitly typed values are
*evil*.

I'd love to have type information passed along with the value.   Of course
you could add a "f"_t file for each "f", and handle eventual discrepancies
at run-time in your application.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, [iso-8859-1] Jakob ьstergaard wrote:
> So you want generaiton and parsing of text strings whenever we pass an int from
> the kernel ?

"scanf is tough" --- programmer Barbie...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 01:40:22PM -0500, Alexander Viro wrote:
> 
> 
> On Sun, 4 Nov 2001, [iso-8859-1] Jakob %stergaard wrote:
> > So you want generaiton and parsing of text strings whenever we pass an int from
> > the kernel ?
> 
> "scanf is tough" --- programmer Barbie...

I'm a little scared when our VFS guy claims he never heard of excellent
programmers using scanf in a way that led to parse errors.

/me hopes VFS doesn't use scanf...

Come on Al, if you have real arguments let hear them, if you want to insult
people you gotta do better than that above.   :)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alex Bligh - linux-kernel]

--On Sunday, 04 November, 2001 8:04 PM +0100 Jakob Østergaard 
<jakob@unthought.net> wrote:

> I'm a little scared when our VFS guy claims he never heard of excellent
> programmers using scanf in a way that led to parse errors.

I'd be far more scared if Al claimed he'd never heard of excellent
programmers reading binary formats, compatible between multiple
code revisions both forward and backwards, endian-ness etc., which
had never lead to parse errors of the binary structure.

If you feel it's too hard to write use scanf(), use sh, awk, perl
etc. which all have their own implementations that appear to have
served UNIX quite well for a long while.

Constructive suggestions:

1. use a textual format, make minimal
   changes from current (duplicate new stuff where necessary),
   but ensure each /proc interface has something which spits
   out a format line (header line or whatever, perhaps an
   interface version number). This at least
   means that userspace tools can check this against known
   previous formats, and don't have to be clairvoyant to
   tell what future kernels have the same /proc interfaces.

2. Flag those entries which are sysctl mirrors as such
   (perhaps in each /proc directory /proc/foo/bar/, a
   /proc/foo/bar/ctl with them all in). Duplicate for the
   time being rather than move. Make reading them (at
   least those in the ctl directory) have a comment line
   starting with a '#' at the top describing the format
   (integer, boolean, string, whatever), what it does.
   Ignore comment lines on write.

3. Try and rearrange all the /proc entries this way, which
   means sysctl can be implemented by a straight ASCII
   write - nice and easy to parse files. Accept that some
   /proc reads (especially) are going to be hard.

--
Alex Bligh

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 07:24:36PM -0000, Alex Bligh - linux-kernel wrote:
> 
> 
> --On Sunday, 04 November, 2001 8:04 PM +0100 Jakob Østergaard 
> <jakob@unthought.net> wrote:
> 
> > I'm a little scared when our VFS guy claims he never heard of excellent
> > programmers using scanf in a way that led to parse errors.
> 
> I'd be far more scared if Al claimed he'd never heard of excellent
> programmers reading binary formats, compatible between multiple
> code revisions both forward and backwards, endian-ness etc., which
> had never lead to parse errors of the binary structure.

Sure there is potential for error anywhere.  And maybe your compiler's
type-check is broken too.  But that's not an argument for not trying
to improve on things.

Please tell me,  is "1610612736" a 32-bit integer, a 64-bit integer, is
it signed or unsigned   ?

I could even live with parsing ASCII, as long as there'd just be type
information to go with the values.  But I see no point in using ASCII
for something intended purely for machine-to-machine communication.

/proc text "GUI" files will stay, don't worry  :)

> If you feel it's too hard to write use scanf(), use sh, awk, perl
> etc. which all have their own implementations that appear to have
> served UNIX quite well for a long while.

Witness ten lines of vmstat output taking 300+ millions of clock cycles.

> Constructive suggestions:
> 
> 1. use a textual format, make minimal
>    changes from current (duplicate new stuff where necessary),
>    but ensure each /proc interface has something which spits
>    out a format line (header line or whatever, perhaps an
>    interface version number). This at least
>    means that userspace tools can check this against known
>    previous formats, and don't have to be clairvoyant to
>    tell what future kernels have the same /proc interfaces.

Then we have text strings as values - some with spaces, some with quotes in
them.   Then we escape our way out of that (which isn't done today by the way),
and then we start implementing a parser for that in every /proc using
application out there.

These interfaces need to be "correct", not "mostly correct".

Example:   I make a symlink from "cat" to "c)(t" (sick example, but that doesn't
change my point), and do a "./c)(t /proc/self/stat":

[albatros:joe] $ ./c\)\(a /proc/self/stat
22482 (c)(a) R 22444 22482 22444 34816 22482 0 20 0 126 0 0 0 0 0 14 0 0 0 24933425 1654784 129 4294967295 134512640 134525684 3221223504 3221223112 1074798884 0 0 0 0 0 0 0 17 0

Go parse that one !  What's the name of my applications ? 

It's good enough for human readers - we have the ability to reason and
make qualified quesses.   Now go implement that in every single piece of
/proc reading software out there   :)

If you want ASCII, we should at least have some approved parsing library
to parse this into native-machine binary structures that can be used 
safely in applications.  I see little point in ASCII then, but maybe it's
just me.

> 
> 2. Flag those entries which are sysctl mirrors as such
>    (perhaps in each /proc directory /proc/foo/bar/, a
>    /proc/foo/bar/ctl with them all in). Duplicate for the
>    time being rather than move. Make reading them (at
>    least those in the ctl directory) have a comment line
>    starting with a '#' at the top describing the format
>    (integer, boolean, string, whatever), what it does.
>    Ignore comment lines on write.
> 
> 3. Try and rearrange all the /proc entries this way, which
>    means sysctl can be implemented by a straight ASCII
>    write - nice and easy to parse files. Accept that some
>    /proc reads (especially) are going to be hard.

I just hate to implement a fuzzy parser with an A.I. that makes HAL look like
kid's toys, every d*mn time I need to get information from the system.

I'm not a big fan of huge re-arrangements. I do like the idea of providing
a machine-readable version of /proc.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, [iso-8859-1] Jakob ьstergaard wrote:

> > If you feel it's too hard to write use scanf(), use sh, awk, perl
> > etc. which all have their own implementations that appear to have
> > served UNIX quite well for a long while.
> 
> Witness ten lines of vmstat output taking 300+ millions of clock cycles.

Would the esteemed sir care to check where these cycles are spent?
How about "traversing page tables of every damn process out there"?
Doesn't sound like a string operation to me...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 02:52:56PM -0500, Alexander Viro wrote:
> 
> 
> On Sun, 4 Nov 2001, [iso-8859-1] Jakob %stergaard wrote:
> 
> > > If you feel it's too hard to write use scanf(), use sh, awk, perl
> > > etc. which all have their own implementations that appear to have
> > > served UNIX quite well for a long while.
> > 
> > Witness ten lines of vmstat output taking 300+ millions of clock cycles.
> 
> Would the esteemed sir care to check where these cycles are spent?
> How about "traversing page tables of every damn process out there"?
> Doesn't sound like a string operation to me...
> 

I'm sure your're right.   It's probably not just string operations. And maybe
then don't even dominate.

And I'm sure that vmstat doesn't use sh, awk, and perl either.

Anyway, the efficiency issues was mainly me getting side-tracked from the main
issue as I see it.

The point I wanted to make was, that we need an interface thats possible to
parse "correctly", not "mostly correctly", and we need to be able to parse it
in a way so that we do not have to rely on a myriad of small tools (that change
over time too).

You need something that's simple and correct.  If it's ASCII, well let it be
ASCII. But /proc as it is today is not possible to parse reliably.  See my "cat
vs. c)(a" example.   You can parse it "mostly reliable", but that's just not
good enough.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

(Whoops, sorry about the 200K spam the first time, hope that was bounced
from the list, here's the abridged version.)

On November 4, 2001 08:52 pm, Alexander Viro wrote:
> On Sun, 4 Nov 2001, [iso-8859-1] Jakob ьstergaard wrote:
> 
> > > If you feel it's too hard to write use scanf(), use sh, awk, perl
> > > etc. which all have their own implementations that appear to have
> > > served UNIX quite well for a long while.
> > 
> > Witness ten lines of vmstat output taking 300+ millions of clock cycles.
> 
> Would the esteemed sir care to check where these cycles are spent?
> How about "traversing page tables of every damn process out there"?
> Doesn't sound like a string operation to me...

Doing 'top -d .1' eats 18% of a 1GHz cpu, which is abominable.  A kernel
profile courtesy of sgi's kernprof shows that scanning pages does not move
the needle, whereas sprintf does.  Notice that the biggest chunk of time
is spent in user space, possibly decoding proc values.  I didn't profile
user space, and I should but I'm not set up to do that just now.  Another
main cause of this embarrassing waste of cpu cycles is the bazillions of
file operations.  Enjoy:

		     Call graph (explanation follows)

granularity: each sample hit covers 4 byte(s) for 0.00% of 38.05 seconds

index % time    self  children    called     name
                                                 <spontaneous>
[1]     78.0    0.00   29.67                 cpu_idle [1]
               29.65    0.00   54034/54034       default_idle [2]
                0.01    0.00    1805/3791        schedule [81]
                0.00    0.00    1804/1806        check_pgt_cache [225]
-----------------------------------------------
               29.65    0.00   54034/54034       cpu_idle [1]
[2]     77.9   29.65    0.00   54034         default_idle [2]
-----------------------------------------------
                                                 <spontaneous>
[3]     11.2    4.26    0.00                 USER [3]
-----------------------------------------------
                                                 <spontaneous>
[4]     10.4    0.04    3.94                 system_call [4]
                0.01    2.85   19776/19776       sys_read [5]
                0.01    0.48   17031/17031       sys_open [14]
                0.00    0.15    5742/5742        sys_stat64 [24]
                0.00    0.12    4554/4554        sys_write [28]
                0.00    0.09     642/642         sys_getdents64 [30]
                0.01    0.08    1544/1544        sys_select [32]
                0.00    0.03     767/767         sys_poll [69]
                0.01    0.02   16901/16903       sys_close [75]
                0.00    0.02    3969/3969        sys_fcntl64 [82]
                0.00    0.01    1046/1046        sys_ioctl [137]
                0.00    0.01     134/134         old_mmap [142]
                0.00    0.01    5376/5376        sys_alarm [145]
                0.00    0.00    3644/3644        sys_gettimeofday [154]
                0.00    0.00       1/1           sys_fork [165]
                0.00    0.00     128/128         sys_access [176]
                0.00    0.00     130/130         sys_munmap [188]
                0.00    0.00       1/1           sys_execve [201]
                0.00    0.00     385/385         sys_lseek [214]
                0.00    0.00     263/263         sys_fstat64 [219]
                0.00    0.00    3615/3615        sys_rt_sigaction [224]
                0.00    0.00     128/128         sys_llseek [232]
                0.00    0.00      17/17          sys_wait4 [237]
                0.00    0.00       1/1           sys_exit [239]
                0.00    0.00       4/4           sys_brk [297]
                0.00    0.00       6/6           sys_writev [304]
                0.00    0.00       1/1           sys_mprotect [350]
                0.00    0.00     148/148         sys_time [425]
                0.00    0.00      34/34          sys_rt_sigprocmask [433]
                0.00    0.00       2/2           sys_setpgid [504]
                0.00    0.00       1/1           sys_newuname [550]
                0.00    0.00       1/1           sys_getpid [549]
                0.00    0.00       1/1           sys_sigreturn [551]
-----------------------------------------------
                0.01    2.85   19776/19776       system_call [4]
[5]      7.5    0.01    2.85   19776         sys_read [5]
                0.01    1.50   16512/16512       proc_info_read [6]
                0.00    1.27     524/524         proc_file_read [7]
                0.00    0.02     781/781         tty_read [92]
                0.00    0.02    1793/1798        generic_file_read [98]
                0.01    0.01   19776/80631       fput [45]
                0.00    0.01     166/166         sock_read [121]
                0.01    0.00   19776/52388       fget [90]
-----------------------------------------------
                0.01    1.50   16512/16512       sys_read [5]
[6]      4.0    0.01    1.50   16512         proc_info_read [6]
                0.03    1.04    5504/5504        proc_pid_statm [8]
                0.12    0.12    5504/5504        proc_pid_stat [23]
                0.09    0.00   15360/27661       _generic_copy_to_user [25]
                0.01    0.04    5504/5504        proc_pid_cmdline [48]
                0.00    0.04   16512/20003       _get_free_pages [50]
                0.02    0.00   16512/20160       _free_pages_ok [77]
                0.00    0.00   16512/87748       _free_pages [107]
                0.00    0.00   16512/80548       free_pages [167]
-----------------------------------------------
                0.00    1.27     524/524         sys_read [5]
[7]      3.3    0.00    1.27     524         proc_file_read [7]
                0.00    0.92     128/128         meminfo_read_proc [10]
                0.01    0.29     128/128         kstat_read_proc [21]
                0.00    0.03      12/12          ksyms_read_proc [68]
                0.00    0.00     524/27661       _generic_copy_to_user [25]
                0.00    0.00     128/128         loadavg_read_proc [230]
                0.00    0.00     128/128         uptime_read_proc [231]
                0.00    0.00     524/20003       _get_free_pages [50]
                0.00    0.00     524/20160       _free_pages_ok [77]
                0.00    0.00     524/87748       _free_pages [107]
                0.00    0.00     524/80548       free_pages [167]
                0.00    0.00     384/512         proc_calc_metrics [417]
-----------------------------------------------
                0.03    1.04    5504/5504        proc_info_read [6]
[8]      2.8    0.03    1.04    5504         proc_pid_statm [8]
                0.98    0.00  177792/177792      statm_pgd_range [9]
                0.00    0.05    5504/44307       sprintf [16]
                0.00    0.00    4352/17928       mmput [152]
-----------------------------------------------
                0.98    0.00  177792/177792      proc_pid_statm [8]
[9]      2.6    0.98    0.00  177792         statm_pgd_range [9]
-----------------------------------------------
                0.00    0.92     128/128         proc_file_read [7]
[10]     2.4    0.00    0.92     128         meminfo_read_proc [10]
                0.92    0.00     128/128         si_swapinfo [11]
                0.00    0.00     256/44307       sprintf [16]
                0.00    0.00     128/128         si_meminfo [271]
                0.00    0.00     128/128         nr_inactive_clean_pages [429]
                0.00    0.00     128/512         proc_calc_metrics [417]
-----------------------------------------------
                0.92    0.00     128/128         meminfo_read_proc [10]
[11]     2.4    0.92    0.00     128         si_swapinfo [11]
-----------------------------------------------
[12]     1.3    0.07    0.44   22903+4       <cycle 1 as a whole> [12]
                0.07    0.44   22905             path_walk <cycle 1> [13]
-----------------------------------------------
                                   2             vfs_follow_link <cycle 1> [506]
                0.00    0.00       2/22903       open_exec [326]
                0.02    0.11    5870/22903       _user_walk [26]
                0.05    0.33   17031/22903       open_namei [19]
[13]     1.3    0.07    0.44   22905         path_walk <cycle 1> [13]
                0.03    0.24   38529/38529       real_lookup [22]
                0.02    0.08   85429/108334      dput [27]
                0.01    0.02   63040/63041       cached_lookup [74]
                0.01    0.01   63042/80076       vfs_permission [71]
                0.01    0.00   63042/80076       permission [120]
                0.00    0.00   22389/22389       lookup_mnt [200]
                0.00    0.00     244/5999        path_release [136]
                0.00    0.00       2/1809        update_atime [402]
                0.00    0.00       2/2           ext2_follow_link [492]
                                   2             vfs_follow_link <cycle 1> [506]
-----------------------------------------------
                0.01    0.48   17031/17031       system_call [4]
[14]     1.3    0.01    0.48   17031         sys_open [14]
                0.00    0.44   17031/17031       filp_open [15]
                0.01    0.01   17031/22902       getname [86]
                0.01    0.00   17031/17032       get_unused_fd [110]
                0.00    0.00   17031/111161      kmem_cache_free [70]
-----------------------------------------------
                0.00    0.44   17031/17031       sys_open [14]
[15]     1.2    0.00    0.44   17031         filp_open [15]
                0.01    0.40   17031/17031       open_namei [19]
                0.01    0.01   16902/16904       dentry_open [84]
-----------------------------------------------
                0.00    0.00     128/44307       loadavg_read_proc [230]
                0.00    0.00     128/44307       uptime_read_proc [231]
                0.00    0.00     256/44307       meminfo_read_proc [10]
                0.00    0.03    3347/44307       get_ksyms_list [67]
                0.00    0.05    5504/44307       proc_pid_stat [23]
                0.00    0.05    5504/44307       proc_pid_statm [8]
                0.00    0.29   29440/44307       kstat_read_proc [21]
[16]     1.2    0.00    0.44   44307         sprintf [16]
                0.00    0.43   44307/44310       vsprintf [17]
-----------------------------------------------
                0.00    0.00       3/44310       printk [327]
                0.00    0.43   44307/44310       sprintf [16]
[17]     1.1    0.00    0.43   44310         vsprintf [17]
                0.09    0.34   44310/44310       vsnprintf [18]
-----------------------------------------------
                0.09    0.34   44310/44310       vsprintf [17]
[18]     1.1    0.09    0.34   44310         vsnprintf [18]
                0.34    0.00  281878/281878      number [20]
                0.00    0.00    3843/3843        skip_atoi [280]
-----------------------------------------------
                0.01    0.40   17031/17031       filp_open [15]
[19]     1.1    0.01    0.40   17031         open_namei [19]
                0.05    0.33   17031/22903       path_walk <cycle 1> [13]
                0.01    0.00   17031/22903       path_init [109]
                0.00    0.00   17030/80076       vfs_permission [71]
                0.00    0.00   17030/80076       permission [120]
                0.00    0.00     129/5999        path_release [136]
                0.00    0.00       1/1           do_truncate [356]
                0.00    0.00       1/108334      dput [27]
                0.00    0.00       1/1           lookup_hash [380]
                0.00    0.00       1/2           get_write_access [494]
-----------------------------------------------
                0.34    0.00  281878/281878      vsnprintf [18]
[20]     0.9    0.34    0.00  281878         number [20]
-----------------------------------------------
                0.01    0.29     128/128         proc_file_read [7]
[21]     0.8    0.01    0.29     128         kstat_read_proc [21]
                0.00    0.29   29440/44307       sprintf [16]
-----------------------------------------------
                0.03    0.24   38529/38529       path_walk <cycle 1> [13]
[22]     0.7    0.03    0.24   38529         real_lookup [22]
                0.03    0.05   22016/22016       proc_pid_lookup [31]
                0.00    0.05   22017/22017       proc_root_lookup [41]
                0.01    0.04   16512/16512       proc_base_lookup [47]
                0.03    0.01   38529/38529       d_alloc [65]
                0.01    0.00   38529/101570      d_lookup [57]
-----------------------------------------------
                0.12    0.12    5504/5504        proc_info_read [6]
[23]     0.6    0.12    0.12    5504         proc_pid_stat [23]
                0.00    0.05    5504/44307       sprintf [16]
                0.05    0.00    5504/5504        collect_sigign_sigcatch [44]
                0.01    0.00    5504/5504        get_wchan [130]
                0.00    0.00    4352/17928       mmput [152]
-----------------------------------------------
                0.00    0.15    5742/5742        system_call [4]
[24]     0.4    0.00    0.15    5742         sys_stat64 [24]
                0.00    0.14    5742/5870        _user_walk [26]
                0.00    0.01    5626/5999        path_release [136]
                0.01    0.00    5626/5889        cp_new_stat64 [157]
-----------------------------------------------
                0.00    0.00     166/27661       memcpy_toiovec [263]
                0.00    0.00     524/27661       proc_file_read [7]
                0.00    0.00     857/27661       read_chan [99]
                0.06    0.00   10754/27661       filldir64 [39]
                0.09    0.00   15360/27661       proc_info_read [6]
[25]     0.4    0.16    0.00   27661         _generic_copy_to_user [25]
-----------------------------------------------
                0.00    0.00     128/5870        sys_access [176]
                0.00    0.14    5742/5870        sys_stat64 [24]
[26]     0.4    0.00    0.14    5870         _user_walk [26]
                0.02    0.11    5870/22903       path_walk <cycle 1> [13]
                0.00    0.00    5870/22902       getname [86]
                0.00    0.00    5870/22903       path_init [109]
                0.00    0.00    5870/111161      kmem_cache_free [70]
-----------------------------------------------
                0.00    0.00       1/108334      open_namei [19]
                0.00    0.00       2/108334      do_exit [238]
                0.00    0.01    5999/108334      path_release [136]
                0.00    0.02   16903/108334      fput [45]
                0.02    0.08   85429/108334      path_walk <cycle 1> [13]
[27]     0.3    0.02    0.10  108334         dput [27]
                0.02    0.04   38529/38529       iput [42]
                0.04    0.00  146863/203321      atomic_dec_and_lock [49]
                0.01    0.00   38529/111161      kmem_cache_free [70]
                0.00    0.00   38528/38528       pid_delete_dentry [212]
                0.00    0.00       1/1           proc_delete_dentry [532]
-----------------------------------------------
                0.00    0.12    4554/4554        system_call [4]
[28]     0.3    0.00    0.12    4554         sys_write [28]
                0.01    0.08    4237/4237        tty_write [29]
                0.00    0.02     317/317         sock_write [87]
                0.00    0.00    4554/80631       fput [45]
                0.00    0.00    4554/52388       fget [90]
-----------------------------------------------
                0.01    0.08    4237/4237        sys_write [28]
[29]     0.2    0.01    0.08    4237         tty_write [29]
                0.00    0.08    4237/4237        write_chan [34]
-----------------------------------------------
                0.00    0.09     642/642         system_call [4]
[30]     0.2    0.00    0.09     642         sys_getdents64 [30]
                0.00    0.08     642/642         vfs_readdir [33]
                0.00    0.00     642/80631       fput [45]
                0.00    0.00     642/52388       fget [90]
-----------------------------------------------
                0.03    0.05   22016/22016       real_lookup [22]
[31]     0.2    0.03    0.05   22016         proc_pid_lookup [31]
                0.01    0.03   22016/38528       proc_pid_make_inode [40]
                0.01    0.00   22016/38529       d_rehash [131]
                0.00    0.00   22016/38529       d_instantiate [140]
                0.00    0.00   22016/87748       _free_pages [107]
                0.00    0.00   22016/80548       free_pages [167]
-----------------------------------------------
                0.01    0.08    1544/1544        system_call [4]
[32]     0.2    0.01    0.08    1544         sys_select [32]
                0.01    0.07    1544/1544        do_select [36]
                0.00    0.00    1544/1544        select_bits_alloc [205]
                0.00    0.00    1545/2963        kfree [192]
                0.00    0.00    1545/1545        select_bits_free [405]
-----------------------------------------------
                0.00    0.08     642/642         sys_getdents64 [30]
[33]     0.2    0.00    0.08     642         vfs_readdir [33]
                0.00    0.05     512/512         proc_pid_readdir [51]
                0.00    0.04     640/640         proc_root_readdir [61]
                0.00    0.00       2/2           ext2_readdir [338]
-----------------------------------------------
                0.00    0.08    4237/4237        tty_write [29]
[34]     0.2    0.00    0.08    4237         write_chan [34]
                0.02    0.02    4361/4361        opost_block [56]
                0.00    0.03    4225/4225        opost [72]
                0.00    0.00    4237/16341       add_wait_queue [129]
                0.00    0.00    4237/16343       remove_wait_queue [156]
                0.00    0.00    4237/8714        tty_hung_up_p [223]
                0.00    0.00       4/12687       pty_write [59]
-----------------------------------------------
                0.00    0.00       1/2050        smp_apic_timer_interrupt [329]
                0.01    0.01     313/2050        stext_lock [95]
                0.04    0.03    1736/2050        do_IRQ [37]
[35]     0.2    0.05    0.03    2050         do_softirq [35]
                0.00    0.02     488/488         net_rx_action [83]
                0.00    0.01    1428/1428        tasklet_hi_action [132]
                0.00    0.00     132/132         net_tx_action [308]
                0.00    0.00       2/2           tasklet_action [346]
-----------------------------------------------
                0.01    0.07    1544/1544        sys_select [32]
[36]     0.2    0.01    0.07    1544         do_select [36]
                0.00    0.01    4473/4473        tty_poll [108]
                0.00    0.01    1545/2312        poll_freewait [106]
                0.00    0.01    1228/1958        schedule_timeout [104]
                0.00    0.01    5626/9901        sock_poll [101]
                0.01    0.00   13757/80631       fput [45]
                0.01    0.00   13757/52388       fget [90]
                0.00    0.00    3658/3658        pipe_poll [164]
                0.00    0.00    1544/1544        max_select_fd [193]
-----------------------------------------------
                0.00    0.07    1760/1760        ret_from_intr [38]
[37]     0.2    0.00    0.07    1760         do_IRQ [37]
                0.04    0.03    1736/2050        do_softirq [35]
                0.00    0.00    1760/1760        handle_IRQ_event [216]
                0.00    0.00    1428/1428        ack_edge_ioapic_irq [406]
                0.00    0.00    1428/1428        end_edge_ioapic_irq [408]
                0.00    0.00     332/332         mask_and_ack_level_ioapic_irq [420]
                0.00    0.00     332/332         end_level_ioapic_irq [419]
-----------------------------------------------
                                                 <spontaneous>
[38]     0.2    0.00    0.07                 ret_from_intr [38]
                0.00    0.07    1760/1760        do_IRQ [37]
-----------------------------------------------
                0.00    0.00       2/10882       ext2_readdir [338]
                0.00    0.03    5248/10882       proc_readdir [62]
                0.00    0.03    5632/10882       proc_pid_readdir [51]
[39]     0.2    0.01    0.06   10882         filldir64 [39]
                0.06    0.00   10754/27661       _generic_copy_to_user [25]
-----------------------------------------------
                0.01    0.02   16512/38528       proc_base_lookup [47]
                0.01    0.03   22016/38528       proc_pid_lookup [31]
[40]     0.2    0.01    0.06   38528         proc_pid_make_inode [40]
                0.01    0.04   38528/38528       get_empty_inode [46]
                0.01    0.00   16512/16512       task_dumpable [158]
-----------------------------------------------
                0.00    0.05   22017/22017       real_lookup [22]
[41]     0.2    0.00    0.05   22017         proc_root_lookup [41]
                0.05    0.00   22017/22017       proc_lookup [43]
-----------------------------------------------
                0.02    0.04   38529/38529       dput [27]
[42]     0.1    0.02    0.04   38529         iput [42]
                0.00    0.01   38529/38529       destroy_inode [117]
                0.01    0.00   38529/203321      atomic_dec_and_lock [49]
                0.01    0.00   38528/87748       _free_pages [107]
                0.01    0.00   38529/38529       force_delete [148]
                0.00    0.00   38528/38528       proc_pid_delete_inode [199]
                0.00    0.00   38528/80548       free_pages [167]
                0.00    0.00   38529/38529       proc_delete_inode [260]
-----------------------------------------------
                0.05    0.00   22017/22017       proc_root_lookup [41]
[43]     0.1    0.05    0.00   22017         proc_lookup [43]
                0.00    0.00       1/1           proc_get_inode [349]
                0.00    0.00       1/38529       d_rehash [131]
                0.00    0.00       1/38529       d_instantiate [140]
-----------------------------------------------
                0.05    0.00    5504/5504        proc_pid_stat [23]
[44]     0.1    0.05    0.00    5504         collect_sigign_sigcatch [44]
-----------------------------------------------
                0.00    0.00       1/80631       search_binary_handler [217]
                0.00    0.00       1/80631       load_elf_binary [218]
                0.00    0.00       2/80631       unmap_fixup [272]
                0.00    0.00       3/80631       old_mmap [142]
                0.00    0.00       6/80631       sys_writev [304]
                0.00    0.00      31/80631       exit_mmap [241]
                0.00    0.00     128/80631       sys_llseek [232]
                0.00    0.00     263/80631       sys_fstat64 [219]
                0.00    0.00     384/80631       sys_lseek [214]
                0.00    0.00     642/80631       sys_getdents64 [30]
                0.00    0.00    1046/80631       sys_ioctl [137]
                0.00    0.00    3584/80631       fcntl_setlk [96]
                0.00    0.00    3969/80631       sys_fcntl64 [82]
                0.00    0.00    4275/80631       do_pollfd [113]
                0.00    0.00    4554/80631       sys_write [28]
                0.00    0.00   11303/80631       poll_freewait [106]
                0.01    0.00   13757/80631       do_select [36]
                0.01    0.00   16906/80631       filp_close [100]
                0.01    0.01   19776/80631       sys_read [5]
[45]     0.1    0.03    0.02   80631         fput [45]
                0.00    0.02   16903/108334      dput [27]
                0.00    0.00   16903/16903       locks_remove_flock [221]
                0.00    0.00     132/132         ext2_release_file [278]
-----------------------------------------------
                0.01    0.04   38528/38528       proc_pid_make_inode [40]
[46]     0.1    0.01    0.04   38528         get_empty_inode [46]
                0.03    0.00   38528/38529       clean_inode [79]
                0.01    0.00   38528/111164      kmem_cache_alloc [78]
-----------------------------------------------
                0.01    0.04   16512/16512       real_lookup [22]
[47]     0.1    0.01    0.04   16512         proc_base_lookup [47]
                0.01    0.02   16512/38528       proc_pid_make_inode [40]
                0.00    0.00   16512/38529       d_rehash [131]
                0.00    0.00   16512/38529       d_instantiate [140]
-----------------------------------------------
                0.01    0.04    5504/5504        proc_info_read [6]
[48]     0.1    0.01    0.04    5504         proc_pid_cmdline [48]
                0.00    0.04    4736/4736        access_process_vm [55]
                0.00    0.00    4352/17928       mmput [152]
-----------------------------------------------
                0.00    0.00       1/203321      free_uid [393]
                0.00    0.00   17928/203321      mmput [152]
                0.01    0.00   38529/203321      iput [42]
                0.04    0.00  146863/203321      dput [27]
[49]     0.1    0.05    0.00  203321         atomic_dec_and_lock [49]
-----------------------------------------------
                0.00    0.00       1/20003       mm_init [275]
                0.00    0.00       1/20003       do_fork [166]
                0.00    0.00       6/20003       pte_alloc [240]
                0.00    0.00     524/20003       proc_file_read [7]
                0.00    0.00     726/20003       sys_poll [69]
                0.00    0.00    2233/20003       _pollwait [139]
                0.00    0.04   16512/20003       proc_info_read [6]
[50]     0.1    0.00    0.04   20003         _get_free_pages [50]
                0.01    0.04   20003/20162       _alloc_pages [54]
                0.00    0.00   20003/20162       alloc_pages [302]
-----------------------------------------------
                0.00    0.05     512/512         vfs_readdir [33]
[51]     0.1    0.00    0.05     512         proc_pid_readdir [51]
                0.00    0.03    5632/10882       filldir64 [39]
                0.01    0.00     512/512         get_pid_list [118]
-----------------------------------------------
                0.01    0.04     293/293         error_code [53]
[52]     0.1    0.01    0.04     293         do_page_fault [52]
                0.00    0.04     293/293         handle_mm_fault [58]
                0.00    0.00     293/5429        find_vma [243]
-----------------------------------------------
                                                 <spontaneous>
[53]     0.1    0.00    0.05                 error_code [53]
                0.01    0.04     293/293         do_page_fault [52]
-----------------------------------------------
                0.00    0.00       1/20162       copy_strings [265]
                0.00    0.00       2/20162       filemap_nopage [342]
                0.00    0.00       2/20162       grow_buffers [363]
                0.00    0.00      20/20162       do_wp_page [151]
                0.00    0.00     134/20162       do_anonymous_page [76]
                0.01    0.04   20003/20162       _get_free_pages [50]
[54]     0.1    0.01    0.04   20162         _alloc_pages [54]
                0.04    0.00   20162/20162       rmqueue [63]
-----------------------------------------------
                0.00    0.04    4736/4736        proc_pid_cmdline [48]
[55]     0.1    0.00    0.04    4736         access_process_vm [55]
                0.00    0.04    4736/4736        access_mm [60]
                0.00    0.00    4736/17928       mmput [152]
                0.00    0.00    4736/4736        find_extend_vma [264]
-----------------------------------------------
                0.02    0.02    4361/4361        write_chan [34]
[56]     0.1    0.02    0.02    4361         opost_block [56]
                0.00    0.01    4361/12687       pty_write [59]
                0.00    0.00    4361/5138        _generic_copy_from_user [173]
                0.00    0.00    4361/8586        pty_write_room [203]
-----------------------------------------------
                0.01    0.00   38529/101570      real_lookup [22]
                0.02    0.00   63041/101570      cached_lookup [74]
[57]     0.1    0.04    0.00  101570         d_lookup [57]
-----------------------------------------------
                0.00    0.04     293/293         do_page_fault [52]
[58]     0.1    0.00    0.04     293         handle_mm_fault [58]
                0.00    0.03     242/242         do_no_page [73]
                0.01    0.00      51/51          do_wp_page [151]
                0.00    0.00     293/325         pte_alloc [240]
-----------------------------------------------
                0.00    0.00       4/12687       write_chan [34]
                0.00    0.01    4361/12687       opost_block [56]
                0.00    0.02    8322/12687       tty_default_put_char [80]
[59]     0.1    0.00    0.04   12687         pty_write [59]
                0.02    0.02   12687/12687       n_tty_receive_buf [66]
                0.00    0.00   12691/33964       n_tty_receive_room [155]
                0.00    0.00       4/5138        _generic_copy_from_user [173]
-----------------------------------------------
                0.00    0.04    4736/4736        access_process_vm [55]
[60]     0.1    0.00    0.04    4736         access_mm [60]
                0.03    0.01    4736/4736        access_one_page [64]
-----------------------------------------------
                0.00    0.04     640/640         vfs_readdir [33]
[61]     0.1    0.00    0.04     640         proc_root_readdir [61]
                0.00    0.03     256/256         proc_readdir [62]
-----------------------------------------------
                0.00    0.03     256/256         proc_root_readdir [61]
[62]     0.1    0.00    0.03     256         proc_readdir [62]
                0.00    0.03    5248/10882       filldir64 [39]
-----------------------------------------------
                0.04    0.00   20162/20162       _alloc_pages [54]
[63]     0.1    0.04    0.00   20162         rmqueue [63]
-----------------------------------------------
                0.03    0.01    4736/4736        access_mm [60]
[64]     0.1    0.03    0.01    4736         access_one_page [64]
                0.01    0.00    4736/6752        kunmap_high [135]
                0.00    0.00    4736/6752        kmap_high [181]
                0.00    0.00    4736/87748       _free_pages [107]
-----------------------------------------------
                0.03    0.01   38529/38529       real_lookup [22]
[65]     0.1    0.03    0.01   38529         d_alloc [65]
                0.01    0.00   38529/111164      kmem_cache_alloc [78]
-----------------------------------------------
                0.02    0.02   12687/12687       pty_write [59]
[66]     0.1    0.02    0.02   12687         n_tty_receive_buf [66]
                0.01    0.00    1548/2506        _wake_up [112]
                0.01    0.00   12686/12686       kill_fasync [143]
                0.00    0.00   12687/33964       n_tty_receive_room [155]
                0.00    0.00       6/6           n_tty_receive_char [466]
-----------------------------------------------
                0.00    0.03      12/12          ksyms_read_proc [68]
[67]     0.1    0.00    0.03      12         get_ksyms_list [67]
                0.00    0.03    3347/44307       sprintf [16]
-----------------------------------------------
                0.00    0.03      12/12          proc_file_read [7]
[68]     0.1    0.00    0.03      12         ksyms_read_proc [68]
                0.00    0.03      12/12          get_ksyms_list [67]
-----------------------------------------------
                0.00    0.03     767/767         system_call [4]
[69]     0.1    0.00    0.03     767         sys_poll [69]
                0.00    0.02     767/767         do_poll [93]
                0.00    0.00     767/2312        poll_freewait [106]
                0.00    0.00     726/20003       _get_free_pages [50]
                0.00    0.00     726/2963        kmalloc [161]
                0.00    0.00     726/20160       _free_pages_ok [77]
                0.00    0.00     726/2963        kfree [192]
                0.00    0.00     726/5138        _generic_copy_from_user [173]
                0.00    0.00     726/87748       _free_pages [107]
                0.00    0.00     726/80548       free_pages [167]
-----------------------------------------------
                0.00    0.00       1/111161      sys_execve [201]
                0.00    0.00       1/111161      _mmdrop [377]
                0.00    0.00       1/111161      put_files_struct [365]
                0.00    0.00       1/111161      do_exit [238]
                0.00    0.00       1/111161      exit_sighand [277]
                0.00    0.00       1/111161      collect_signal [386]
                0.00    0.00      40/111161      exit_mmap [241]
                0.00    0.00     130/111161      unmap_fixup [272]
                0.00    0.00     132/111161      do_munmap [159]
                0.00    0.00     142/111161      kfree_skbmem [234]
                0.00    0.00    1792/111161      locks_delete_lock [261]
                0.00    0.00    3584/111161      fcntl_setlk [96]
                0.00    0.00    5376/111161      posix_lock_file [122]
                0.00    0.00    5870/111161      _user_walk [26]
                0.00    0.00   17031/111161      sys_open [14]
                0.01    0.00   38529/111161      dput [27]
                0.01    0.00   38529/111161      destroy_inode [117]
[70]     0.1    0.03    0.00  111161         kmem_cache_free [70]
                0.00    0.00       2/12          free_block [439]
-----------------------------------------------
                0.00    0.00       1/80076       flush_old_exec [242]
                0.00    0.00       1/80076       lookup_hash [380]
                0.00    0.00       2/80076       open_exec [326]
                0.00    0.00   17030/80076       open_namei [19]
                0.01    0.01   63042/80076       path_walk <cycle 1> [13]
[71]     0.1    0.02    0.01   80076         vfs_permission [71]
                0.00    0.01   72794/72794       in_group_p [114]
-----------------------------------------------
                0.00    0.03    4225/4225        write_chan [34]
[72]     0.1    0.00    0.03    4225         opost [72]
                0.00    0.03    8322/8322        tty_default_put_char [80]
                0.00    0.00    4225/8586        pty_write_room [203]
-----------------------------------------------
                0.00    0.03     242/242         handle_mm_fault [58]
[73]     0.1    0.00    0.03     242         do_no_page [73]
                0.03    0.00     136/136         do_anonymous_page [76]
                0.00    0.00     106/106         filemap_nopage [342]
-----------------------------------------------
                0.00    0.00       1/63041       lookup_hash [380]
                0.01    0.02   63040/63041       path_walk <cycle 1> [13]
[74]     0.1    0.01    0.02   63041         cached_lookup [74]
                0.02    0.00   63041/101570      d_lookup [57]
-----------------------------------------------
                0.00    0.00       1/16903       flush_old_exec [242]
                0.00    0.00       1/16903       load_elf_binary [218]
                0.01    0.02   16901/16903       system_call [4]
[75]     0.1    0.01    0.02   16903         sys_close [75]
                0.00    0.02   16903/16906       filp_close [100]
-----------------------------------------------
                0.03    0.00     136/136         do_no_page [73]
[76]     0.1    0.03    0.00     136         do_anonymous_page [76]
                0.00    0.00     134/20162       _alloc_pages [54]
                0.00    0.00     134/6752        kunmap_high [135]
                0.00    0.00     134/6752        kmap_high [181]
                0.00    0.00     134/20162       alloc_pages [302]
-----------------------------------------------
                0.00    0.00       1/20160       _mmdrop [377]
                0.00    0.00       1/20160       sys_wait4 [237]
                0.00    0.00       1/20160       do_wp_page [151]
                0.00    0.00       6/20160       clear_page_tables [341]
                0.00    0.00     155/20160       free_page_and_swap_cache [266]
                0.00    0.00     524/20160       proc_file_read [7]
                0.00    0.00     726/20160       sys_poll [69]
                0.00    0.00    2234/20160       poll_freewait [106]
                0.02    0.00   16512/20160       proc_info_read [6]
[77]     0.1    0.03    0.00   20160         _free_pages_ok [77]
-----------------------------------------------
                0.00    0.00       1/111164      get_new_inode [347]
                0.00    0.00       1/111164      copy_files [276]
                0.00    0.00       1/111164      send_signal [392]
                0.00    0.00       1/111164      mprotect_fixup [352]
                0.00    0.00       1/111164      setup_arg_pages [361]
                0.00    0.00       2/111164      do_fork [166]
                0.00    0.00       2/111164      get_unused_buffer_head [384]
                0.00    0.00      21/111164      skb_clone [190]
                0.00    0.00      32/111164      copy_mm [209]
                0.00    0.00     122/111164      alloc_skb [187]
                0.00    0.00     132/111164      do_munmap [159]
                0.00    0.00     137/111164      do_mmap_pgoff [150]
                0.00    0.00   10752/111164      locks_alloc_lock [171]
                0.01    0.00   22902/111164      getname [86]
                0.01    0.00   38528/111164      get_empty_inode [46]
                0.01    0.00   38529/111164      d_alloc [65]
[78]     0.1    0.03    0.00  111164         kmem_cache_alloc [78]
                0.00    0.00       1/11          kmem_cache_alloc_batch [440]
-----------------------------------------------
                0.00    0.00       1/38529       get_new_inode [347]
                0.03    0.00   38528/38529       get_empty_inode [46]
[79]     0.1    0.03    0.00   38529         clean_inode [79]
-----------------------------------------------
                0.00    0.03    8322/8322        opost [72]
[80]     0.1    0.00    0.03    8322         tty_default_put_char [80]
                0.00    0.02    8322/12687       pty_write [59]
-----------------------------------------------
                0.00    0.00       1/3791        do_exit [238]
                0.00    0.00       1/3791        sys_wait4 [237]
                0.00    0.00       3/3791        _wait_on_buffer [331]
                0.00    0.00      23/3791        reschedule [315]
                0.01    0.00    1805/3791        cpu_idle [1]
                0.01    0.00    1958/3791        schedule_timeout [104]
[81]     0.1    0.02    0.00    3791         schedule [81]
                0.00    0.00    3764/3765        _switch_to [184]
                0.00    0.00       1/1           _mmdrop [377]
-----------------------------------------------
                0.00    0.02    3969/3969        system_call [4]
[82]     0.1    0.00    0.02    3969         sys_fcntl64 [82]
                0.00    0.02    3969/3969        do_fcntl [94]
                0.00    0.00    3969/80631       fput [45]
                0.00    0.00    3969/52388       fget [90]
-----------------------------------------------
                0.00    0.02     488/488         do_softirq [35]
[83]     0.1    0.00    0.02     488         net_rx_action [83]
                0.00    0.02     488/488         ip_rcv [91]
-----------------------------------------------
                0.00    0.00       2/16904       open_exec [326]
                0.01    0.01   16902/16904       filp_open [15]
[84]     0.1    0.01    0.01   16904         dentry_open [84]
                0.01    0.00   16904/16904       get_empty_filp [127]
                0.00    0.00   16904/16904       file_move [172]
                0.00    0.00     133/133         ext2_open_file [427]
                0.00    0.00       1/2           get_write_access [494]
                0.00    0.00       1/1           chrdev_open [511]
-----------------------------------------------
                0.00    0.00       6/323         sock_readv_writev [306]
                0.00    0.02     317/323         sock_write [87]
[85]     0.1    0.00    0.02     323         sock_sendmsg [85]
                0.00    0.02     323/323         inet_sendmsg [88]
-----------------------------------------------
                0.00    0.00       1/22902       sys_execve [201]
                0.00    0.00    5870/22902       _user_walk [26]
                0.01    0.01   17031/22902       sys_open [14]
[86]     0.1    0.01    0.02   22902         getname [86]
                0.01    0.00   22902/22902       strncpy_from_user [119]
                0.01    0.00   22902/111164      kmem_cache_alloc [78]
-----------------------------------------------
                0.00    0.02     317/317         sys_write [28]
[87]     0.1    0.00    0.02     317         sock_write [87]
                0.00    0.02     317/323         sock_sendmsg [85]
-----------------------------------------------
                0.00    0.02     323/323         sock_sendmsg [85]
[88]     0.1    0.00    0.02     323         inet_sendmsg [88]
                0.01    0.01     323/323         tcp_sendmsg [89]
-----------------------------------------------
                0.01    0.01     323/323         inet_sendmsg [88]
[89]     0.1    0.01    0.01     323         tcp_sendmsg [89]
                0.00    0.01     318/322         tcp_write_xmit [123]
                0.00    0.00     349/689         alloc_skb [187]
                0.00    0.00      27/27          tcp_push_one [269]
                0.00    0.00      15/144         tcp_mem_schedule [292]
                0.00    0.00       1/1           _release_sock [333]
-----------------------------------------------
                0.00    0.00       3/52388       old_mmap [142]
                0.00    0.00       6/52388       sys_writev [304]
                0.00    0.00     128/52388       sys_llseek [232]
                0.00    0.00     263/52388       sys_fstat64 [219]
                0.00    0.00     385/52388       sys_lseek [214]
                0.00    0.00     642/52388       sys_getdents64 [30]
                0.00    0.00    1046/52388       sys_ioctl [137]
                0.00    0.00    3584/52388       fcntl_setlk [96]
                0.00    0.00    3969/52388       sys_fcntl64 [82]
                0.00    0.00    4275/52388       do_pollfd [113]
                0.00    0.00    4554/52388       sys_write [28]
                0.01    0.00   13757/52388       do_select [36]
                0.01    0.00   19776/52388       sys_read [5]
[90]     0.1    0.02    0.00   52388         fget [90]
-----------------------------------------------
                0.00    0.02     488/488         net_rx_action [83]
[91]     0.1    0.00    0.02     488         ip_rcv [91]
                0.00    0.02     488/488         ip_local_deliver [102]
                0.00    0.00     176/176         ip_route_input [198]
-----------------------------------------------
                0.00    0.02     781/781         sys_read [5]
[92]     0.1    0.00    0.02     781         tty_read [92]
                0.01    0.01     781/781         read_chan [99]
-----------------------------------------------
                0.00    0.02     767/767         sys_poll [69]
[93]     0.1    0.00    0.02     767         do_poll [93]
                0.00    0.01    1435/1435        do_pollfd [113]
                0.00    0.01     709/1958        schedule_timeout [104]
-----------------------------------------------
                0.00    0.02    3969/3969        sys_fcntl64 [82]
[94]     0.1    0.00    0.02    3969         do_fcntl [94]
                0.00    0.02    3584/3584        fcntl_setlk [96]
-----------------------------------------------
                                                 <spontaneous>
[95]     0.1    0.01    0.01                 stext_lock [95]
                0.01    0.01     313/2050        do_softirq [35]
-----------------------------------------------
                0.00    0.02    3584/3584        do_fcntl [94]
[96]     0.1    0.00    0.02    3584         fcntl_setlk [96]
                0.00    0.01    3584/3584        posix_lock_file [122]
                0.00    0.00    3584/80631       fput [45]
                0.00    0.00    3584/52388       fget [90]
                0.00    0.00    3584/10752       locks_alloc_lock [171]
                0.00    0.00    3584/3584        flock_to_posix_lock [245]
                0.00    0.00    3584/111161      kmem_cache_free [70]
-----------------------------------------------
                0.00    0.02    1798/1798        generic_file_read [98]
[97]     0.1    0.00    0.02    1798         do_generic_file_read [97]
                0.01    0.00    1798/1798        file_read_actor [103]
                0.00    0.00    1798/1798        generic_file_readahead [247]
                0.00    0.00    1798/87748       _free_pages [107]
                0.00    0.00    1798/1809        update_atime [402]
-----------------------------------------------
                0.00    0.00       5/1798        kernel_read [325]
                0.00    0.02    1793/1798        sys_read [5]
[98]     0.1    0.00    0.02    1798         generic_file_read [98]
                0.00    0.02    1798/1798        do_generic_file_read [97]
-----------------------------------------------
                0.01    0.01     781/781         tty_read [92]
[99]     0.1    0.01    0.01     781         read_chan [99]
                0.00    0.00     857/27661       _generic_copy_to_user [25]
                0.00    0.00     781/781         check_unthrottle [162]
                0.00    0.00     781/16341       add_wait_queue [129]
                0.00    0.00     781/5256        n_tty_chars_in_buffer [210]
                0.00    0.00     781/16343       remove_wait_queue [156]
                0.00    0.00       4/1958        schedule_timeout [104]
                0.00    0.00       4/8714        tty_hung_up_p [223]
-----------------------------------------------

--
Daniel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Albert D. Cahalan]

=?iso-8859-1?Q?Jak writes:

> Please tell me,  is "1610612736" a 32-bit integer, a 64-bit integer, is
> it signed or unsigned   ?
> 
> I could even live with parsing ASCII, as long as there'd just be type
> information to go with the values.

You are looking for something called the registry. It's something
that was introduced with Windows 95. It's basically a filesystem
with typed files: char, int, string, string array, etc.

> These interfaces need to be "correct", not "mostly correct".
>
> Example:   I make a symlink from "cat" to "c)(t" (sick example,
> but that doesn't change my point), and do a "./c)(t /proc/self/stat":
>
> [albatros:joe] $ ./c\)\(a /proc/self/stat
> 22482 (c)(a) R 22444 22482 22444 34816 22482 0 20 0 126 0 0 0 0 0 14 0 0 0 24933425 1654784 129 4294967295 134512640 134525684 3221223504 3221223112 1074798884 0 0 0 0 0 0 0 17 0
>
> Go parse that one !  What's the name of my applications ? 

Funny you should mention that one. I wrote the code used by procps
to read this file. I love that file! The parentheses issue is just
a beauty wart. People rarely feel the urge to screw with raw numbers.
In all the other files, idiots like to: add headers, change the
spelling of field names, change the order, add spaces and random
punctuation, etc. Nothing is as stable and easy to use as the
/proc/self/stat file.

> If you want ASCII, we should at least have some approved parsing
> library to parse this into native-machine binary structures

No.

>> 2. Flag those entries which are sysctl mirrors as such
>>    (perhaps in each /proc directory /proc/foo/bar/, a
>>    /proc/foo/bar/ctl with them all in). Duplicate for the
>>    time being rather than move. Make reading them (at
>>    least those in the ctl directory) have a comment line
>>    starting with a '#' at the top describing the format
>>    (integer, boolean, string, whatever), what it does.
>>    Ignore comment lines on write.

Now you are proposing to dink with the format. See above comments.

>> 3. Try and rearrange all the /proc entries this way, which
>>    means sysctl can be implemented by a straight ASCII
>>    write - nice and easy to parse files.

This is exactly what the sysctl command does.

> I'm not a big fan of huge re-arrangements. I do like the idea of providing
> a machine-readable version of /proc.

Linus clearly doesn't give a fuck about /proc performance.
That's his right, and you are welcome to patch your kernel to
have something better: http://lwn.net/2000/0420/a/atomicps.html

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 04:12:23PM -0500, Albert D. Cahalan wrote:
> =?iso-8859-1?Q?Jak writes:
> 
> > Please tell me,  is "1610612736" a 32-bit integer, a 64-bit integer, is
> > it signed or unsigned   ?
> > 
> > I could even live with parsing ASCII, as long as there'd just be type
> > information to go with the values.
> 
> You are looking for something called the registry. It's something
> that was introduced with Windows 95. It's basically a filesystem
> with typed files: char, int, string, string array, etc.

Nope   :)

It does not have "char, int, string, string array, etc." it has "String, binary
and DWORD".

Having read out 64 bit values, floating point data etc. from the registry, I'm
old enough to know that it is *NOT* what I'm looking for   :)

...
> Funny you should mention that one. I wrote the code used by procps
> to read this file. I love that file! The parentheses issue is just
> a beauty wart. People rarely feel the urge to screw with raw numbers.
> In all the other files, idiots like to: add headers, change the
> spelling of field names, change the order, add spaces and random
> punctuation, etc. Nothing is as stable and easy to use as the
> /proc/self/stat file.

Imagine every field in a file by itself, with well-defined type
information and unit informaiton.

...
> Linus clearly doesn't give a fuck about /proc performance.
> That's his right, and you are welcome to patch your kernel to
> have something better: http://lwn.net/2000/0420/a/atomicps.html

Performance is one thing.  Not being able to know whether numbers are i32, u32,
u64, or measured in Kilobytes or carrots is another ting.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 22:20, Jakob Østergaard wrote:
> > > I could even live with parsing ASCII, as long as there'd just be type
> > > information to go with the values.
> > You are looking for something called the registry. It's something
> > that was introduced with Windows 95. It's basically a filesystem
> > with typed files: char, int, string, string array, etc.
> Having read out 64 bit values, floating point data etc. from the registry,
> I'm old enough to know that it is *NOT* what I'm looking for   :)

Why? It's not bad only because it is from MS. IMHO the disadvantages of the 
registry are:
- you need special software/syscalls to access it
- because of that making backups etc is hard
- the organization of the data is horrible

Assuming you could mount it as a regular filesystem and use it for kernel 
configuration, what else are its disadvantages?

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Albert D. Cahalan]

=?iso-8859-1?Q?Jak writes:
> On Sun, Nov 04, 2001 at 04:12:23PM -0500, Albert D. Cahalan wrote:

>> You are looking for something called the registry. It's something
>> that was introduced with Windows 95. It's basically a filesystem
>> with typed files: char, int, string, string array, etc.
>
> Nope   :)
>
> It does not have "char, int, string, string array, etc." it
> has "String, binary and DWORD".

I'm pretty sure that newer implementations have additional types.
BTW, we could call the persistent part of our registry "reiserfs4".

> Imagine every field in a file by itself, with well-defined type
> information and unit informaiton.

I suppose I could print a warning if the type or unit info
isn't what was expected. That's insignificantly useful.

Individual files are nice, until you realize: open, read, close

> Performance is one thing.  Not being able to know whether
> numbers are i32, u32, u64, or measured in Kilobytes or
> carrots is another ting.

I don't see what the code is supposed to do if it was expecting
kilobytes and you serve it carrots. Certainly nothing useful can
be done when this happens.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Martin Dalecki]

"Albert D. Cahalan" wrote:

Every BASTARD out there telling the world, that parsing ASCII formatted
files
is easy should be punished to providing a BNF definition of it's syntax.
Otherwise I won't trust him. Having a struct {} with a version field,
indicating
possible semantical changes wil always be easier faster more immune
to errors to use in user level programs.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Mon, 5 Nov 2001, Martin Dalecki wrote:

> "Albert D. Cahalan" wrote:
> 
> Every BASTARD out there telling the world, that parsing ASCII formatted
> files

What was your username, again?

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Martin Dalecki]

Alexander Viro wrote:
> 
> On Mon, 5 Nov 2001, Martin Dalecki wrote:
> 
> > "Albert D. Cahalan" wrote:
> >
> > Every BASTARD out there telling the world, that parsing ASCII formatted
> > files
> 
> What was your username, again?

root, with uid != 0 and on a masquaraded host, who cares?

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Albert D. Cahalan]

You wrote:
> Alexander Viro wrote:
>> On Mon, 5 Nov 2001, Martin Dalecki wrote:
>>> "Albert D. Cahalan" wrote:
>>>
>>> Every BASTARD out there telling the world, that parsing ASCII formatted
>>> files
>>
>> What was your username, again?
>
> root, with uid != 0 and on a masquaraded host, who cares?

I think the point is that it looks like to attributed your own
words to me. Your post didn't quote anything from me, but it
started off as follows:

--------------------------------------
"Albert D. Cahalan" wrote:

Every BASTARD out there telling the world, that parsing ASCII formatted
--------------------------------------

Well, I didn't write that or anything else in your post.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Martin Dalecki]

"Albert D. Cahalan" wrote:
> 
> You wrote:
> > Alexander Viro wrote:
> >> On Mon, 5 Nov 2001, Martin Dalecki wrote:
> >>> "Albert D. Cahalan" wrote:
> >>>
> >>> Every BASTARD out there telling the world, that parsing ASCII formatted
> >>> files
> >>
> >> What was your username, again?
> >
> > root, with uid != 0 and on a masquaraded host, who cares?
> 
> I think the point is that it looks like to attributed your own
> words to me. Your post didn't quote anything from me, but it
> started off as follows:

Oh excuse me I didn't intend to put the pun on you personally 
or disguise myself as somebody else. In fact I was kind of
 supporting your point of view.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Gerhard Mack]

On Mon, 5 Nov 2001, Albert D. Cahalan wrote:

> You wrote:
> > Alexander Viro wrote:
> >> On Mon, 5 Nov 2001, Martin Dalecki wrote:
> >>> "Albert D. Cahalan" wrote:
> >>>
> >>> Every BASTARD out there telling the world, that parsing ASCII formatted
> >>> files
> >>
> >> What was your username, again?
> >
> > root, with uid != 0 and on a masquaraded host, who cares?
> 
> I think the point is that it looks like to attributed your own
> words to me. Your post didn't quote anything from me, but it
> started off as follows:
> 
> --------------------------------------
> "Albert D. Cahalan" wrote:
> 
> Every BASTARD out there telling the world, that parsing ASCII formatted
> --------------------------------------
> 
> Well, I didn't write that or anything else in your post.
> 

Actually Al Viro's reply makes a lot more sense if your familiar with:
http://bofh.ntk.net/bastard.html

	Gerhard


--
Gerhard Mack

gmack@innerfire.net

<>< As a computer I find your faith in technology amusing.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Stephen Satchell]

At 12:23 PM 11/5/01 +0100, Martin Dalecki wrote:
>Every BASTARD out there telling the world, that parsing ASCII formatted
>files
>is easy should be punished to providing a BNF definition of it's syntax.
>Otherwise I won't trust him. Having a struct {} with a version field,
>indicating
>possible semantical changes wil always be easier faster more immune
>to errors to use in user level programs.

I would love for the people who write the code that generates the /proc 
info to be required to document the layout of the information.  The best 
place for that documentation is the source, and in English or other 
accepted human language, in a comment block.  Not in "header lines" or 
other such nonsense.  I don't need no stinkin' BNF, just a reasonable 
description of what each field is would suffice.  I would go so far as to 
say there needs to be a standard established in how /proc data is formatted 
so that we can create templates for the standard tools.

(I have to ask, have you ever used flex?  I used to hand-code scanners, but 
I find that flex is so much easier and generates smaller faster code than I 
can do by hand.  Changes are easy, too)

As for version fields:  I HATE THEM.  So much of my older code has bloat 
because of "version fields" that require that I have multiple blocks of 
code for the same damn thing.  POSIX code that has to determine which 
version of POSIX is implemented, and tailor the code at run-time to the 
whims of the OS gods.  BLOAT BLOAT BLOAT.  Besides, you already have a 
"version field", or is the release level of Linux too coarse for you?

As for easier:  EASIER FOR WHOM?  The sysadmin who is trying to figure out 
why his system is behaving in a strange manner?  You expect sysadmins to 
grow C compilers and header files in order to read /proc?  You can bet that 
my next point will require sysadmins to look at the hidden proc files 
sooner or later.  To wit:

The absolute worst part of this proposal is that it provides yet another 
for separate mechanism to do the same thing, and there is no clean way to 
use the ASCII /proc mechanism to generate the binary.  That inflates the 
opportunities for error by an unmanageable amount -- you will end up 
breaking BOTH methods of extracting information.  Which is more 
important:  getting the right information at the cost of a flex/bison 
scanner and some CPU time, or getting the WRONG information in the blink of 
the CPU's eye?  What happens when the ASCII version is broken and the 
binary version is right?  Who is going to take up the task of verifying 
that the ASCII and the binary match?

That version field thing:  you have to be willing to guarantee complete 
backward compatibility of your structures, so that you can only extend the 
structures, not manipulate already-defined fields.  In addition, you would 
need to define, for every single field, a value that indicates that no 
value is present.  This means that fields that are deprecated will still 
have a value, but the value that would be returned would be "no 
value"...and the applications that use your structures would have to know 
and understand and test for this not-a-value value and react appropriately.

One think I like about SNMP is that I can parse a MIB and probe for the 
information I need without worrying about versions.  It's there, I know its 
type, and I know what to expect in the way of values.  I'm also told when 
there is no value to report, either because the OS chooses not to return 
one, or because the state of the system says that returning a value is 
meaningless.  There is already a BNF definition of a MIB, too, which 
satisfies your other requirement.  New version?  New 
MIB.  Cross-checking?  Yes, I can be sure that the version of the MIB I'm 
using matches the version the system is using to generate the data.

Oh, the bloat thing:  why do you want to bloat the kernel even more than it 
is?  /proc is not cheap, and there have been times when I have been tempted 
to generate kernels without it.  Doubling up on the /proc filesystem may 
drive me to do it yet, and explore the wonders of sysctl.

I applaud the proponents of the idea for identifying a problem and 
proposing an interesting fix.  It's the wrong fix, but interesting anyway.

Stephen Satchell

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Martin Dalecki]

Stephen Satchell wrote:
> 
> At 12:23 PM 11/5/01 +0100, Martin Dalecki wrote:
> >Every BASTARD out there telling the world, that parsing ASCII formatted
> >files
> >is easy should be punished to providing a BNF definition of it's syntax.
> >Otherwise I won't trust him. Having a struct {} with a version field,
> >indicating
> >possible semantical changes wil always be easier faster more immune
> >to errors to use in user level programs.
> 
> I would love for the people who write the code that generates the /proc
> info to be required to document the layout of the information.  The best
> place for that documentation is the source, and in English or other
> accepted human language, in a comment block.  Not in "header lines" or
> other such nonsense.  I don't need no stinkin' BNF, just a reasonable

I don't agree. BNF is basically the only proper and efficient way for a 
nice formal descrition of a LR parsable language. No accident most
programming
languages out there are defined in some sort of BNF.

> description of what each field is would suffice.  I would go so far as to
> say there needs to be a standard established in how /proc data is formatted
> so that we can create templates for the standard tools.
> 
> (I have to ask, have you ever used flex?  I used to hand-code scanners, but
> I find that flex is so much easier and generates smaller faster code than I
> can do by hand.  Changes are easy, too)

Short answer: yes I know them, yacc bison pure flex and lex whatever,
and 
I used to use them for job projects not just toys. Trust me they are
the only proper practical way to define the syntax of something parsable
and beeing complete about it. Unless you wan't to reach the stability of
the usual perl-web hackkery.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Ben Greear]

Martin Dalecki wrote:

> Stephen Satchell wrote:
> 
>>At 12:23 PM 11/5/01 +0100, Martin Dalecki wrote:
>>
>>>Every BASTARD out there telling the world, that parsing ASCII formatted
>>>files
>>>is easy should be punished to providing a BNF definition of it's syntax.
>>>Otherwise I won't trust him. Having a struct {} with a version field,
>>>indicating
>>>possible semantical changes wil always be easier faster more immune
>>>to errors to use in user level programs.
>>>
>>I would love for the people who write the code that generates the /proc
>>info to be required to document the layout of the information.  The best
>>place for that documentation is the source, and in English or other
>>accepted human language, in a comment block.  Not in "header lines" or
>>other such nonsense.  I don't need no stinkin' BNF, just a reasonable


I would rather have a header block, as well as docs in the source.
If the header cannot easily explain it, then the header can have a URL
or other link to the full explanation.  I don't expect to be able to parse
every /proc interface with a single tool, but I would like to be able to
easily parse individual ones with perl, sscanf, etc...

Ben



-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Rik van Riel]

On Mon, 5 Nov 2001, Ben Greear wrote:

> I would rather have a header block, as well as docs in the
> source. If the header cannot easily explain it, then the header
> can have a URL or other link to the full explanation.

I think you've hit the core of the problem. There is no magical
bullet which will stop badly written userland programs from
breaking, but the kernel developers should have the courtesy
of providing documentation for the /proc files so the writers
of userland programs can have an idea what to expect.

The inline docbook stuff in the kernel should make it easy for
kernel developers to keep code and documentation in sync, while
also making it easy to generate documentation in a format which
is nice to read ;)

regards,

Rik
-- 
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/  (volunteers needed)

http://www.surriel.com/		http://distro.conectiva.com/

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Monday 05 November 2001 19:40, Rik van Riel wrote:
> I think you've hit the core of the problem. There is no magical
> bullet which will stop badly written userland programs from
> breaking, but the kernel developers should have the courtesy
> of providing documentation for the /proc files so the writers
> of userland programs can have an idea what to expect.

I think the core insight is that if the kernel continues to have dozens of 
"human-readable" file formats in /proc, each should to be documented using a 
BNF description that can guarantee that the format is still valid in the 
future, even if there is the need to add additional fields. 
The result of this is, of course, that it may be very hard to write 
shell scripts that won't break sooner or later and that accessing the data in 
C is much more work than a simple scanf. 

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Ben Greear]

Tim Jansen wrote:

> On Monday 05 November 2001 19:40, Rik van Riel wrote:
> 
>>I think you've hit the core of the problem. There is no magical
>>bullet which will stop badly written userland programs from
>>breaking, but the kernel developers should have the courtesy
>>of providing documentation for the /proc files so the writers
>>of userland programs can have an idea what to expect.
>>
> 
> I think the core insight is that if the kernel continues to have dozens of 
> "human-readable" file formats in /proc, each should to be documented using a 
> BNF description that can guarantee that the format is still valid in the 
> future, even if there is the need to add additional fields. 
> The result of this is, of course, that it may be very hard to write 
> shell scripts that won't break sooner or later and that accessing the data in 
> C is much more work than a simple scanf.


So if BNF makes it harder for shell scripts and sscanf, and harder for
the kernel developers...what good does it do???  I definately don't advocate
anything more than some simple documentation about whatever format the proc
module writer uses.  All of these interfaces (proc, ioctl, ...) end up being
hacks at some point, but a _documented_ hack can be called a feature :)


> 
> bye...
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Monday 05 November 2001 22:58, Ben Greear wrote:
> So if BNF makes it harder for shell scripts and sscanf, and harder for
> the kernel developers...what good does it do???  

You know how to parse the file. 
Take a look at /proc/partitions. Is its exact syntax obvious without 
examining the source in the kernel? Can it happen that there is a space or 
another unusual character in the device path and what happens then? Could it 
be that someone decides that an additional column is neccessary and how can 
my parser stay compatible then? Are there any suprises or special conditions 
that I don't know about? Maybe one of the fields is hexadecimal but I think 
it is decimal, I can't see it from looking at the file's content.

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Erik Andersen]

On Mon Nov 05, 2001 at 11:51:52PM +0100, Tim Jansen wrote:
> On Monday 05 November 2001 22:58, Ben Greear wrote:
> > So if BNF makes it harder for shell scripts and sscanf, and harder for
> > the kernel developers...what good does it do???  
> 
> You know how to parse the file. 
> Take a look at /proc/partitions. Is its exact syntax obvious without 
> examining the source in the kernel? Can it happen that there is a space or 
> another unusual character in the device path and what happens then? Could it 

Come now, it really isn't that difficult: 

    char name[80];
    unsigned long long size;
    unsigned int major, minor;
    
    if (sscanf(line, "%4u %4u %llu %s", &major, &minor, &size, name) == 4)
    {       
	add_partition(name, size, major, minor);
    }

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Monday 05 November 2001 23:59, Erik Andersen wrote:
> Come now, it really isn't that difficult:
>     if (sscanf(line, "%4u %4u %llu %s", &major, &minor, &size, name) == 4)
>     {
> 	add_partition(name, size, major, minor);
>     }

But how can the user know this without looking into the kernel? Compare it to 
/proc/mounts. Proc mounts escapes spaces and other special characters in 
strings with an octal encoding (so spaces are replaced by '\040'). 

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Tue, 6 Nov 2001, Tim Jansen wrote:

> But how can the user know this without looking into the kernel? Compare it to 
> /proc/mounts. Proc mounts escapes spaces and other special characters in 
> strings with an octal encoding (so spaces are replaced by '\040'). 

Ah, yes - the horrible /proc/mounts.  Check that code in 2.4.13-ac8, will
you?

Yes, current procfs sucks.  We got a decent infrastructure that allows
to write that code easily.  Again, see -ac8 and watch fs/namespace.c
code dealing with /proc/mounts.

No need to play silly buggers with "one value per file" (and invite the
Elder Ones with applications trying to use getdents()).  Sigh...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Martin Dalecki]

Alexander Viro wrote:
> 
> On Tue, 6 Nov 2001, Tim Jansen wrote:
> 
> > But how can the user know this without looking into the kernel? Compare it to
> > /proc/mounts. Proc mounts escapes spaces and other special characters in
> > strings with an octal encoding (so spaces are replaced by '\040').
> 
> Ah, yes - the horrible /proc/mounts.  Check that code in 2.4.13-ac8, will
> you?
> 
> Yes, current procfs sucks.  We got a decent infrastructure that allows
> to write that code easily.  Again, see -ac8 and watch fs/namespace.c
> code dealing with /proc/mounts.
> 
> No need to play silly buggers with "one value per file" (and invite the
> Elder Ones with applications trying to use getdents()).  Sigh...

Getdents() can be removed since 2.0 times. I never noticed *any*
application
actually using it.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[dank]

In article <20011105155955.A16505@codepoet.org> Erik Anderson wrote:
> Come now, it really isn't that difficult: 

>    char name[80];
>    if (sscanf(line, "%4u %4u %llu %s", &major, &minor, &size, name) == 4)

if it's so easy to do, why do you have a great big buffer overflow here?

-- 
nicholas black (dank@trellisinc.com)    developer, trellis network security

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Erik Andersen]

On Tue Nov 06, 2001 at 02:49:23PM -0500, dank@trellisinc.com wrote:
> In article <20011105155955.A16505@codepoet.org> Erik Anderson wrote:
> > Come now, it really isn't that difficult: 
> 
> >    char name[80];
> >    if (sscanf(line, "%4u %4u %llu %s", &major, &minor, &size, name) == 4)
> 
> if it's so easy to do, why do you have a great big buffer overflow here?


Sorry, no doughnut for you.  drivers/block/genhd.c:

    #ifdef CONFIG_PROC_FS
    int get_partition_list(char *page, char **start, off_t offset, int count)
    {
	...
	char buf[64];

	...

	len += snprintf(page + len, 63, "%4d  %4d %10d %s\n", gp->major, n, 
		gp->sizes[n], disk_name(gp, n, buf));

so each /proc/partitions line maxes out at 63 bytes.  So not only
is there no overflow, I am providing 16 extra bytes of padding.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[dank]

In article <20011106152215.A31923@codepoet.org> you wrote:
> Sorry, no doughnut for you.  drivers/block/genhd.c:

>    #ifdef CONFIG_PROC_FS
>    int get_partition_list(char *page, char **start, off_t offset, int count)
>        char buf[64];
> so each /proc/partitions line maxes out at 63 bytes.  So not only
> is there no overflow, I am providing 16 extra bytes of padding.

"code poet?"  you've plucked an 80 from the air.  regardless of what the
kernel prints now and how it's limited (deep within drivers/block/genhd.c),
there is no reference to this silent 63 via either explicit comment or
pure code.  your code remains happily ignorant of any modification to this
postcondition, and when that changes (as it surely will), you lose.  it's
uninspired coding like the above that keeps the buffer overflow
technique alive.

now, i imagine you're more skilled than this, and would have invested
the time to do it properly the first time around (certainly *my*
managers wouldn't accept "buried within the backend is a hardcoded
constant...", but i work in network security).  others, however, may
not be so skilled as you, and what of when they're writing your server?

c string processing is all of doable, mature, and meticulous.  "done
properly by beginners" is not how i would describe it.

-- 
nicholas black (dank@trellisinc.com)                      http://trellisinc.com

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Erik Andersen]

On Tue Nov 06, 2001 at 05:47:53PM -0500, dank@trellisinc.com wrote:
> In article <20011106152215.A31923@codepoet.org> you wrote:
> > Sorry, no doughnut for you.  drivers/block/genhd.c:
> 
> >    #ifdef CONFIG_PROC_FS
> >    int get_partition_list(char *page, char **start, off_t offset, int count)
> >        char buf[64];
> > so each /proc/partitions line maxes out at 63 bytes.  So not only
> > is there no overflow, I am providing 16 extra bytes of padding.
> 
> "code poet?"  you've plucked an 80 from the air.  regardless of what the

Yup, you are right.  You found me out.  I'm a complete impostor
and I know nothing about programming because I spent the exactly
4 seconds to write a simple example without first researching the
underlying interface.  Know why?  Because it was an _example_,
not a dissertation on string processing.  If I was actually going
to write that code, I would have spent the extra two minute it
would have taken to read the kernel source first.  Yes, fixed
buffers suck.  But that is the current interface, so get over it
and get over the pointless ad hominem attacks. 

> constant...", but i work in network security).  others, however,
> may not be so skilled as you, and what of when they're writing
> your server?

Then I should be spending more time interviewing so I don't hire
dolts, and I would spend more time auditing their code and
teaching them how to program. 

I actually avoid using /proc as much as possible in all my code
(I even wrote a patch to replace /proc with a char device about a
year ago, rejected of course for the same reasons Linus expressed
on this thread).  There are many valid reasons why /proc sucks
(especially for embedded systems).  But I don't consider the
ASCII-is-hard-to-parse argument valid.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Ricky Beam]

On Tue, 6 Nov 2001 dank@trellisinc.com wrote:
>"code poet?"  you've plucked an 80 from the air.  regardless of what the
>kernel prints now and how it's limited (deep within drivers/block/genhd.c),
>there is no reference to this silent 63 via either explicit comment or
>pure code.  your code remains happily ignorant of any modification to this
>postcondition, and when that changes (as it surely will), you lose.  it's
>uninspired coding like the above that keeps the buffer overflow
>technique alive.

Exactly.  Just because the code _currently_ won't generate more than 63
chars doesn't mean it always will.  And who says the application will see
the true, kernel generated "/proc/partitions"? <raises eyebrow>

>c string processing is all of doable, mature, and meticulous.  "done
>properly by beginners" is not how i would describe it.

Experience shows beginners rarely get thing right the first time out. (Or
the second or third time if they are like some of my previous students.)

--Ricky

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Remco Post]

Hi,

still,

char name[80]
if (sscanf(line, "%4u %4u %llu %80s", &major, &minor, &size, name) == 4)

would safeguard you agains whatever you might read from your line, this is 
better protection agains buffer overflows than looking up the code that you 
think you'll be reading the output from. This code is NOT under your control, 
usually...

> On Tue Nov 06, 2001 at 02:49:23PM -0500, dank@trellisinc.com wrote:
> > In article <20011105155955.A16505@codepoet.org> Erik Anderson wrote:
> > > Come now, it really isn't that difficult: 
> > 
> > >    char name[80];
> > >    if (sscanf(line, "%4u %4u %llu %s", &major, &minor, &size, name) == 4)
> > 
> > if it's so easy to do, why do you have a great big buffer overflow here?
> 
> 
> Sorry, no doughnut for you.  drivers/block/genhd.c:
> 
>     #ifdef CONFIG_PROC_FS
>     int get_partition_list(char *page, char **start, off_t offset, int count)
>     {
> 	...
> 	char buf[64];
> 
> 	...
> 
> 	len += snprintf(page + len, 63, "%4d  %4d %10d %s\n", gp->major, n, 
> 		gp->sizes[n], disk_name(gp, n, buf));
> 
> so each /proc/partitions line maxes out at 63 bytes.  So not only
> is there no overflow, I am providing 16 extra bytes of padding.
> 
>  -Erik
> 
> --
> Erik B. Andersen             http://codepoet-consulting.com/
> --This message was written using 73% post-consumer electrons--
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer industry
didn't even foresee that the century was going to end." -- Douglas Adams

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[George Greer]

On Tue, 6 Nov 2001 dank@trellisinc.com wrote:

>In article <20011105155955.A16505@codepoet.org> Erik Anderson wrote:
>> Come now, it really isn't that difficult: 
>
>>    char name[80];
>>    if (sscanf(line, "%4u %4u %llu %s", &major, &minor, &size, name) == 4)
>
>if it's so easy to do, why do you have a great big buffer overflow here?

Because he forgot about "%80s"?  But if he forgot that he may accidently
use strcpy, strcat, and gets, so...

Or maybe it was just an exercise for the reader?

-- 
George Greer, greerga@m-l.org
http://www.m-l.org/~greerga/

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jonathan Lundell]

At 8:38 AM -0800 11/5/01, Stephen Satchell wrote:
>As for version fields:  I HATE THEM.  So much of my older code has 
>bloat because of "version fields" that require that I have multiple 
>blocks of code for the same damn thing.  POSIX code that has to 
>determine which version of POSIX is implemented, and tailor the code 
>at run-time to the whims of the OS gods.  BLOAT BLOAT BLOAT. 
>Besides, you already have a "version field", or is the release level 
>of Linux too coarse for you?

Either too coarse or too fine, often enough, when we're talking about 
a semi-independent module. Consider, though, a more legitimate 
non-bloating use of a version field. Rather than try to support all 
versions, use it to determine whether the two ends of the 
communication channel are compatible, and fail gracefully because of 
the incompatible version. Tell the user to update the app, or 
whatever.
-- 
/Jonathan Lundell.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Stephen Satchell]

At 11:58 AM 11/5/01 -0800, Jonathan Lundell wrote:
>Either too coarse or too fine, often enough, when we're talking about a 
>semi-independent module. Consider, though, a more legitimate non-bloating 
>use of a version field. Rather than try to support all versions, use it to 
>determine whether the two ends of the communication channel are 
>compatible, and fail gracefully because of the incompatible version. Tell 
>the user to update the app, or whatever.

I have software out in the field that has been around for more than ten 
years.  Some of it has been maintenance-free (other than the 
every-other-fortnight bug report that requires a fix) because the 
underlying operating system didn't change.  Some of it has been a 
nightmare, requiring changes for each OS release and in some cases with 
each sub-release in order to keep the feature bloat from knocking out the 
functionality of the program.

Unlike many of you, my client base doesn't upgrade on a whim.  They stick 
with what works.  That means all my software has to be able to run up and 
down the version tree, and I have a real problem maintaining parallel 
versions of code.  In Linux, I have people on 2.0.34 still.  I have people 
running some of my software on old versions of Ultrix on hardware that 
hasn't seen sales for over a decade.  I just found out that software I 
wrote 20 years ago is STILL in use, and customers were inquiring if I was 
available to make changes!

And then there is the problem of who pays for my time to make the app 
update.  I don't charge people for updates as a rule -- that rule may have 
to change for my Linux apps if this ill-thought-out idea goes into the 
kernel.  I expend enough effort trying to keep up with the crap coming out 
of Redmond and Cupertino.

Apologies for the vent, but I just swatted another bug caused by an 
undocumented change in Windows 2000 that nailed one of my apps but good.  I 
shudder to think what XP is going to look like when my clients start 
thinking of "upgrading" their hardware and have XP foisted on them...

Satch

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Ragnar Hojland Espinosa]

On Mon, Nov 05, 2001 at 01:43:11PM -0800, Stephen Satchell wrote:
> At 11:58 AM 11/5/01 -0800, Jonathan Lundell wrote:
> >use of a version field. Rather than try to support all versions, use it to 
> >determine whether the two ends of the communication channel are 
> >compatible, and fail gracefully because of the incompatible version. Tell 
> >the user to update the app, or whatever.

[snip]

> And then there is the problem of who pays for my time to make the app 
> update.  I don't charge people for updates as a rule -- that rule may have 
> to change for my Linux apps if this ill-thought-out idea goes into the 
> kernel.  I expend enough effort trying to keep up with the crap coming out 

I hope you just don't mean the version number idea.  Because I don't see
reason for not, instead of adding a version number to every /proc file and
breaknig everything, adding all them to a /proc/proc-version file which 
would still let clients make some sanity checks.

-- 
____/|  Ragnar Højland      Freedom - Linux - OpenGL |    Brainbench MVP
\ o.O|  PGP94C4B2F0D27DE025BE2302C104B78C56 B72F0822 | for Unix Programming
 =(_)=  "Thou shalt not follow the NULL pointer for  | (www.brainbench.com)
   U     chaos and madness await thee at its end."

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alex Bligh - linux-kernel]

--On Sunday, 04 November, 2001 4:12 PM -0500 "Albert D. Cahalan" 
<acahalan@cs.uml.edu> wrote:

>> Now you are proposing to dink with the format. See above comments.

Attribution error: that was me, disagreeing with Jakob - the point was
if you want to dink with the format to achieve the objectives
he seemed to be after (which I thought were to do at least
in part with consistency etc.), it is theoretically possible
to do such dinking with minimal change & certainly retain
text format (and note I said retain original /proc files too). Whether
it's worth it as a practical exercize, with all the inherent
disruption it would no doubt cause, and questionable net benefit
is a completely different question. I was just saying that
binary format wasn't necessary to achieve what I think
Jakob wanted to achieve. The full thought
experiment was in a later email. I suspect you don't disagree
given your previous post.

>>> 3. Try and rearrange all the /proc entries this way, which
>>>    means sysctl can be implemented by a straight ASCII
>>>    write - nice and easy to parse files.
>
> This is exactly what the sysctl command does.

Sorry, I meant 'this way a consistent interface cf
sysctl could be used for more of what's currently
done through /proc'. Last time I looked there was
stuff you could read/write to through /proc which
couldn't be done through sysctl.

--
Alex Bligh

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Stuart Young]

At 02:52 PM 4/11/01 -0500, Alexander Viro wrote:
>Would the esteemed sir care to check where these cycles are spent?
>How about "traversing page tables of every damn process out there"?
>Doesn't sound like a string operation to me...

Just a quickie....

Any reason we can't move all the process info into something like 
/proc/pid/* instead of in the root /proc tree?

Should be pretty easy to do, could still have the pid's in the root /proc 
tree, and if they get read, do what /proc/pci does, and log a warning about 
"xxx is using old /proc interfaces". Makes it just that little bit easier 
to parse processes without fiddling around if you know all the dir's are 
always processes. It's also a bit of a visual cleanup when you have lots of 
processes and do a 'ls /proc'.

There is probably a few other things in /proc/* that could be moved out and 
put in more sensible places (eg: interrupts, irq, devices, mtrr, slabinfo, 
mounts, modules, stat, etc), that really define what they belong to (a 
/proc/kernel/* mebbe). Having /proc basically full of directories would 
clean things up a bit. Some things don't need to change though (eg: uptime, 
version).


AMC Enterprises P/L    - Stuart Young
First Floor            - Network and Systems Admin
3 Chesterville Rd      - sgy@amc.com.au
Cheltenham Vic 3192    - Ph:  (03) 9584-2700
http://www.amc.com.au/ - Fax: (03) 9584-2755

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Mon, 5 Nov 2001, Stuart Young wrote:

> Any reason we can't move all the process info into something like 
> /proc/pid/* instead of in the root /proc tree?

Thanks, but no thanks.  If we are starting to move stuff around, we
would be much better off leaving in /proc only what it was supposed
to contain - per-process information.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Stuart Young]

At 11:05 PM 4/11/01 -0500, Alexander Viro wrote:


>On Mon, 5 Nov 2001, Stuart Young wrote:
>
> > Any reason we can't move all the process info into something like
> > /proc/pid/* instead of in the root /proc tree?
>
>Thanks, but no thanks.  If we are starting to move stuff around, we
>would be much better off leaving in /proc only what it was supposed
>to contain - per-process information.

That's fair.. so (this is all speculation of course) move everything else 
but process info out of there? I could handle that, makes sense, long as we 
had some backward "transitional" interface, that warned about using old 
interfaces. Only question is, where would we put this information in the 
file system tree?


AMC Enterprises P/L    - Stuart Young
First Floor            - Network and Systems Admin
3 Chesterville Rd      - sgy@amc.com.au
Cheltenham Vic 3192    - Ph:  (03) 9584-2700
http://www.amc.com.au/ - Fax: (03) 9584-2755

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[SpaceWalker]

Stuart Young wrote:
> 
> At 11:05 PM 4/11/01 -0500, Alexander Viro wrote:
> 
> >On Mon, 5 Nov 2001, Stuart Young wrote:
> >
> > > Any reason we can't move all the process info into something like
> > > /proc/pid/* instead of in the root /proc tree?
> >
> >Thanks, but no thanks.  If we are starting to move stuff around, we
> >would be much better off leaving in /proc only what it was supposed
> >to contain - per-process information.
> 

We could add a file into /proc like /proc/processes that contains once
all process informations that some programs like top or ps can read only
Once.
It could save a lot of time in kernel mode scanning the process list for
each process.
later, a new version of ps or top could simply stat /proc/processes and
if it exists uses it to give informations to the user.
What do you think of this idea ?


SpaceWalker

spacewalker@altern.org
ICQ 36157579

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Mon, Nov 05, 2001 at 05:32:34PM +0100, SpaceWalker wrote:
> Stuart Young wrote:
> > 
> > At 11:05 PM 4/11/01 -0500, Alexander Viro wrote:
> > 
> > >On Mon, 5 Nov 2001, Stuart Young wrote:
> > >
> > > > Any reason we can't move all the process info into something like
> > > > /proc/pid/* instead of in the root /proc tree?
> > >
> > >Thanks, but no thanks.  If we are starting to move stuff around, we
> > >would be much better off leaving in /proc only what it was supposed
> > >to contain - per-process information.
> > 
> 
> We could add a file into /proc like /proc/processes that contains once
> all process informations that some programs like top or ps can read only
> Once.
> It could save a lot of time in kernel mode scanning the process list for
> each process.
> later, a new version of ps or top could simply stat /proc/processes and
> if it exists uses it to give informations to the user.
> What do you think of this idea ?

We would have the same "changing format of /proc/processes" parsing
problems as we have now with the rest of /proc.

Why not implement all of top in the kernel, so that you could do a
 cat /dev/top   and have the usual top output nicely shown ?   ;)

(yes, the last one was a joke!)

Your suggestion may improve the performance of one or two userland
applications, but it does not attack the real problem: that /proc is not
machine readable.   

We would be maintaining yet another /proc file, but we'd still have the
problems we have now.   Implementing an A.I. in every CPU meter applet out
there, while still having to accept that the A.I. gives up on us every now and
then (when someone decides to add an ASCII art visualization of the utilization
of the various ALUs in /proc/cpuinfo for example - the worst part being that
this example is probably not even far fetched!)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, [iso-8859-1] Jakob ьstergaard wrote:

> I'm a little scared when our VFS guy claims he never heard of excellent
> programmers using scanf in a way that led to parse errors.

I've seen excellent programmers fscking up use of && and ||.

I've also seen quite a few guys coming (from their experience) to
the conclusions that look an awful lot similar to mine.  Like, say it,
dmr and ken.  Or Linus.  Or Rob Pike.  Or Brian Kernighan.

And frankly, when I hear about "typed" interfaces, two things come to
mind - "typed files" and CORBA.  Both - architectural failures.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 02:29:06PM -0500, Alexander Viro wrote:
> 
> 
> On Sun, 4 Nov 2001, [iso-8859-1] Jakob %stergaard wrote:
> 
> > I'm a little scared when our VFS guy claims he never heard of excellent
> > programmers using scanf in a way that led to parse errors.
> 
> I've seen excellent programmers fscking up use of && and ||.
> 
> I've also seen quite a few guys coming (from their experience) to
> the conclusions that look an awful lot similar to mine.  Like, say it,
> dmr and ken.  Or Linus.  Or Rob Pike.  Or Brian Kernighan.
> 
> And frankly, when I hear about "typed" interfaces, two things come to
> mind - "typed files" and CORBA.  Both - architectural failures.

Architectural failures like "C" or "C++" for example.

Strong type information (in one form or the other) is absolutely fundamental
for achieving correctness in this kind of software.

You can't just have a "data thingy" and hope your "operator thingys" do
what you suppose they do, and that your "storage thingy" has enough room
for what you shovel into it.

C has types for a reason.  C++ improved the type system for a reason.  Perl
and PHP programs have run-time failures for a reason.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, [iso-8859-1] Jakob ьstergaard wrote:

> Strong type information (in one form or the other) is absolutely fundamental
> for achieving correctness in this kind of software.

Like, say it, all shell programming?  Or the whole idea of "file as stream of
characters"?  Or pipes, for that matter...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 03:01:12PM -0500, Alexander Viro wrote:
> 
> 
> On Sun, 4 Nov 2001, [iso-8859-1] Jakob %stergaard wrote:
> 
> > Strong type information (in one form or the other) is absolutely fundamental
> > for achieving correctness in this kind of software.
> 
> Like, say it, all shell programming?  Or the whole idea of "file as stream of
> characters"?  Or pipes, for that matter...
> 

Shell programming is great for small programs. You don't need type information
in the language when you can fit it all in your head.

Now, go write 100K lines of shell, something that does something that is not
just shoveling lines from one app into a grep and into another app.  Let's say,
a database.  Go implement the next Oracle replacement in bash, and tell me you
don't care about types in your language.

Why do we have "file formats", well that is because files are just streams
of characters, and we need more structure than just that.

This is exactly what I'm proposing - having "just" files in proc is fine,
but not knowing the type of the information they present is catastrophic.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Kai Henningsen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 1108 bytes --]

jakob@unthought.net (Jakob ¥stergaard)  wrote on 04.11.01 in <20011104210936.T14001@unthought.net>:

> On Sun, Nov 04, 2001 at 03:01:12PM -0500, Alexander Viro wrote:
> >
> >
> > On Sun, 4 Nov 2001, [iso-8859-1] Jakob %stergaard wrote:
> >
> > > Strong type information (in one form or the other) is absolutely
> > > fundamental for achieving correctness in this kind of software.
> >
> > Like, say it, all shell programming?  Or the whole idea of "file as stream
> > of characters"?  Or pipes, for that matter...
> >
>
> Shell programming is great for small programs. You don't need type
> information in the language when you can fit it all in your head.
>
> Now, go write 100K lines of shell, something that does something that is not
> just shoveling lines from one app into a grep and into another app.  Let's
> say, a database.  Go implement the next Oracle replacement in bash, and tell
> me you don't care about types in your language.

And now look at how large typical /proc-using code parts are. Do they  
match better with your first or your second paragraph?

The first?

I thought so.

MfG Kai

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Tue, Nov 06, 2001 at 09:23:00AM +0200, Kai Henningsen wrote:
> jakob@unthought.net (Jakob ¥stergaard)  wrote on 04.11.01 in <20011104210936.T14001@unthought.net>:
> 
...
> >
> > Shell programming is great for small programs. You don't need type
> > information in the language when you can fit it all in your head.
> >
> > Now, go write 100K lines of shell, something that does something that is not
> > just shoveling lines from one app into a grep and into another app.  Let's
> > say, a database.  Go implement the next Oracle replacement in bash, and tell
> > me you don't care about types in your language.
> 
> And now look at how large typical /proc-using code parts are. Do they  
> match better with your first or your second paragraph?

If you write in C, you need type information.  No matter if it's 5 lines or 50K.

How many of your shell languages use arbitrary precision arithmetic *always* ?
If they only do "sometimes" (for some operations) you'll be up shit creek without
a paddle once some value you thought was 32 bits turns out to be 64, and your
scripts, lacking type informaiton, handle this error "gracefully" (accounting
scripts for example where you don't check the output every day, but discover at
the end of the quarter that you're fucked because you only have the lower 32
bits of the user's network usage).

My argument with the 100K of shell was more to emphasize that type information
is necessary in complex systems.

Even if you just have 5 lines of Perl, you have a kernel too - it is a complex
system already.

> 
> The first?
> 
> I thought so.

Well, working for a company that makes a living of reading in /proc (and being
fairly good at it), it would be more like the second   ;)

But I have also coded for HP-UX, Solaris, NT and others.   I have seen how
others attack the problems of getting information out of systems, and I can see
that /proc as it is today is *not* a good answer to that problem.

There are worse systems out there than Linux, but there are better ones as
well.   I see no reason why Linux shouldn't excel in this area too.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 18:41, you wrote:
> The "fuzzy parsing" userland has to do today to get useful information
> out of many proc files today is not nice at all. 

I agree, but you dont need a binary format to achieve this. A WELL-DEFINED 
format is sufficient. XML is one of them, one-value-files another one. The 
"fuzzy parsing" only happens because the files try to be friendly for human 
readers.


> It eats CPU, it's error-prone, and all in all it's just "wrong".

How much of your CPU time is spent parsing /proc files?


> However - having a human-readable /proc that you can use directly with
> cat, echo,  your scripts,  simple programs using read(), etc.   is
> absolutely a *very* cool feature that I don't want to let go.  It is just
> too damn practical.

You shouldn't use them in scripts because they are likely to break. That's 
the whole point. At least not when you want to distribute the scripts to 
others. And BTW the one-value-files are much easier to parse for scripts than 
any other solution that I have seen so far, including the current /proc 
interface.

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, Tim Jansen wrote:

> On Sunday 04 November 2001 18:41, you wrote:
> > The "fuzzy parsing" userland has to do today to get useful information
> > out of many proc files today is not nice at all. 
> 
> I agree, but you dont need a binary format to achieve this. A WELL-DEFINED 
> format is sufficient. XML is one of them, one-value-files another one. The 

And thinking before defining an API is the third.  Guess why XML is so
popular alternative...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 07:27:16PM +0100, Tim Jansen wrote:
> On Sunday 04 November 2001 18:41, you wrote:
> > The "fuzzy parsing" userland has to do today to get useful information
> > out of many proc files today is not nice at all. 
> 
> I agree, but you dont need a binary format to achieve this. A WELL-DEFINED 
> format is sufficient. XML is one of them, one-value-files another one. The 
> "fuzzy parsing" only happens because the files try to be friendly for human 
> readers.

You need syntax or "transport", and then you need semantics.  My approach
is identical to XML except it doesn't give you kilobytes of human-unreadable
text.

You could use text, with binary you save the extra conversions along with
errors from parsers or bad use of sscanf()/sprintf()/...   K.I.S.S.   :)

I see a good point in using one-value-files though, except I think there's
some type information missing.

> 
> 
> > It eats CPU, it's error-prone, and all in all it's just "wrong".
> 
> How much of your CPU time is spent parsing /proc files?

[albatros:joe] $ time vmstat 1
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0 113908   3184   1892 130584   1   1     3     3   61    43   9   5  86
 1  0  0 113908   3064   1896 130700   0   0     8     0 2301  1148   8   2  90
 0  0  0 113908   3064   1896 130700   0   0     0     0 2026   893   7   2  91
 0  0  0 113908   3064   1896 130700   0   0     0     0 1877   829   3   4  93
 0  0  0 113908   3068   1896 130696   0   0     0     0 1946   942   5   3  92
 0  0  0 113908   3072   1896 130696   0   0     0     0 2009  1034   7   5  88
 0  0  0 113908   3064   1896 130704   0   0     0     0 3706  2336   4   5  90
 0  0  0 113908   3064   1900 130688   0   0     0     0 2341  1671  10   3  87
 0  0  0 113908   3064   1900 130736   0   0     0     0 2431  1869  15   5  79
 2  0  0 113908   3064   1900 130764   0   0     0    88 2346  1440  12   3  85
^C

real	0m9.486s
user	0m0.070s
sys	0m0.120s

[albatros:joe] $ 


A *very* simple program (top is probably a lot worse!) uses 1% on my Dual 1.4
GHz Athlon.

Those TEN lines of status output cost me 336 MILLION clock cycles.

> 
> 
> > However - having a human-readable /proc that you can use directly with
> > cat, echo,  your scripts,  simple programs using read(), etc.   is
> > absolutely a *very* cool feature that I don't want to let go.  It is just
> > too damn practical.
> 
> You shouldn't use them in scripts because they are likely to break. That's 
> the whole point. At least not when you want to distribute the scripts to 
> others. And BTW the one-value-files are much easier to parse for scripts than 
> any other solution that I have seen so far, including the current /proc 
> interface.

I agree that there's some really good points in using single-value files.


-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Pavel Machek]

Hi!

> > It eats CPU, it's error-prone, and all in all it's just "wrong".
> 
> How much of your CPU time is spent parsing /proc files?

30% of 486 if you run top... That's way too much and top is unusable on slower
machines. 
"Not fast enough for showing processes" sounds wery wrong.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Rik van Riel]

On Wed, 7 Nov 2001, Pavel Machek wrote:

> > > It eats CPU, it's error-prone, and all in all it's just "wrong".
> >
> > How much of your CPU time is spent parsing /proc files?
>
> 30% of 486 if you run top... That's way too much and top is unusable
> on slower machines.
> "Not fast enough for showing processes" sounds wery wrong.

Is this time actually spent parsing ascii, or is it procfs
walking all the page tables of all processes ? ;)

Rik
-- 
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/

http://www.surriel.com/		http://distro.conectiva.com/

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Pavel Machek]

Hi!

> > > > It eats CPU, it's error-prone, and all in all it's just "wrong".
> > >
> > > How much of your CPU time is spent parsing /proc files?
> >
> > 30% of 486 if you run top... That's way too much and top is unusable
> > on slower machines.
> > "Not fast enough for showing processes" sounds wery wrong.
> 
> Is this time actually spent parsing ascii, or is it procfs
> walking all the page tables of all processes ? ;)

About 1:1, probably. Readdir of /proc and open/read/parse/close is 
pretty expensive.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 18:28, Daniel Phillips wrote:
> > It needs special parsers and will be almost impossible to access from
> > shell scripts.
> No, look, he's proposing to put the binary encoding in hidden .files.  The
> good old /proc files will continue to appear and operate as they do now.

But as he already said:
2)  As /proc files change, parsers must be changed in userspace

So if only some programs use the 'dot-files' and the other still use the 
crappy text interface we still have the old problem for scripts, only with a 
much larger effort.

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, Tim Jansen wrote:

> So if only some programs use the 'dot-files' and the other still use the 
> crappy text interface we still have the old problem for scripts, only with a 
> much larger effort.

Folks, could we please deep-six the "ASCII is tough" mentality?  Idea of
native-endian data is so broken that it's not even funny.  Exercise:
try to export such thing over the network.  Another one: try to use
that in a shell script.  One more: try to do it portably in Perl script.

It had been tried.  Many times.  It had backfired 100 times out 100.
We have the same idiocy to thank for fun trying to move a disk with UFS
volume from Solaris sparc to Solaris x86.  We have the same idiocy to
thank for a lot of ugliness in X.

At the very least, use canonical bytesex and field sizes.  Anything less
is just begging for trouble.  And in case of procfs or its equivalents,
_use_ the_ _damn_ _ASCII_ _representations_.  scanf(3) is there for
purpose.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 01:30:38PM -0500, Alexander Viro wrote:
> 
> 
> On Sun, 4 Nov 2001, Tim Jansen wrote:
> 
> > So if only some programs use the 'dot-files' and the other still use the 
> > crappy text interface we still have the old problem for scripts, only with a 
> > much larger effort.
> 
> Folks, could we please deep-six the "ASCII is tough" mentality?  Idea of
> native-endian data is so broken that it's not even funny.  Exercise:
> try to export such thing over the network.  Another one: try to use
> that in a shell script.  One more: try to do it portably in Perl script.

So make it network byte order.

How many bugs have you heard of with bad use of sscanf() ?

The counters *are* host specific. Available memory is 32 bits somewhere, 64
other places.  That's the world we live in and hiding the difficulties in ASCII
that *can* be parsed so that it only breaks "sometimes" doesn't help the
application developers.

Better to face the facts, and get over it.

> It had been tried.  Many times.  It had backfired 100 times out 100.
> We have the same idiocy to thank for fun trying to move a disk with UFS
> volume from Solaris sparc to Solaris x86.  We have the same idiocy to
> thank for a lot of ugliness in X.
> 
> At the very least, use canonical bytesex and field sizes.  Anything less
> is just begging for trouble.  And in case of procfs or its equivalents,
> _use_ the_ _damn_ _ASCII_ _representations_.  scanf(3) is there for
> purpose.
> 

scanf can be used wrongly in more ways than the two of us can imagine
together, even if we try.

I disagree with harmonizing field sizes - that doesn't make sense. What's
64 bits today is 128 tomorrow (IPv6 related things, crypto, ...), what
used to fit in 32 is in 64, some places.

Having a library that gives you either compile-time errors if you use it
wrong, or barfs loudly at run-time is one hell of a lot better than having
silent mis-parsing of ASCII values.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

On November 4, 2001 07:52 pm, Jakob Østergaard wrote:
> On Sun, Nov 04, 2001 at 01:30:38PM -0500, Alexander Viro wrote:
> > Folks, could we please deep-six the "ASCII is tough" mentality?  Idea of
> > native-endian data is so broken that it's not even funny.  Exercise:
> > try to export such thing over the network.  Another one: try to use
> > that in a shell script.  One more: try to do it portably in Perl script.
> 
> So make it network byte order.
> 
> How many bugs have you heard of with bad use of sscanf() ?

Yes, and it's easy for those to be buffer overflow bugs.  The extra security 
risk is even more of a reason to avoid ASCII strings in internal interfaces 
than the gross overhead.  Do the ASCII conversions in user space, please.

No, ASCII isn't tough, it just sucks as an internal transport.

--
Daniel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Albert D. Cahalan]

Alexander Viro writes:

> Folks, could we please deep-six the "ASCII is tough" mentality?

Sure. How about:
It's a PITA to break out the dragon book, bloat sucks,
and I'd just rather not have to write the code.

I also like:
Trusting every little status app to not have exploitable
buffer overruns is worrisome.

The more serious problem:
ASCII formats change in unpredictable ways. (see below)

>  Idea of
> native-endian data is so broken that it's not even funny.  Exercise:
> try to export such thing over the network.

Ooh! You're making /proc NFS exportable?

>  Another one: try to use
> that in a shell script.  One more: try to do it portably in Perl script.

FOO=`ps -o foo= -p $$`    # Get our FOO value out of binary /proc

> It had been tried.  Many times.  It had backfired 100 times out 100.
> We have the same idiocy to thank for fun trying to move a disk with UFS
> volume from Solaris sparc to Solaris x86.

Disks are slow, so native endian UFS was indeed a poor choice.

>  We have the same idiocy to
> thank for a lot of ugliness in X.

This was a necessary performance hack. Hopefully nobody wrote
an X server or client that would do conditional byte swaps
all over the code.

> At the very least, use canonical bytesex and field sizes.  Anything less
> is just begging for trouble.  And in case of procfs or its equivalents,
> _use_ the_ _damn_ _ASCII_ _representations_.  scanf(3) is there for
> purpose.

SigCgt in /proc/self/status wasn't always spelled that way.
It wasn't always in the same location either. ASCII bites
because people can't resist screwing with it.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Martin Dalecki]

Alexander Viro wrote:
> 
> On Sun, 4 Nov 2001, Tim Jansen wrote:
> 
> > So if only some programs use the 'dot-files' and the other still use the
> > crappy text interface we still have the old problem for scripts, only with a
> > much larger effort.
> 
> Folks, could we please deep-six the "ASCII is tough" mentality?  Idea of
> native-endian data is so broken that it's not even funny.  Exercise:
> try to export such thing over the network.  Another one: try to use
> that in a shell script.  One more: try to do it portably in Perl script.
> 
> It had been tried.  Many times.  It had backfired 100 times out 100.
> We have the same idiocy to thank for fun trying to move a disk with UFS
> volume from Solaris sparc to Solaris x86.  We have the same idiocy to
> thank for a lot of ugliness in X.
> 
> At the very least, use canonical bytesex and field sizes.  Anything less
> is just begging for trouble.  And in case of procfs or its equivalents,
> _use_ the_ _damn_ _ASCII_ _representations_.  scanf(3) is there for
> purpose.

And the purpose of scanf in system level applications is to introduce
nice
opportunities for buffer overruns and string formatting bugs.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

On November 5, 2001 12:06 pm, Martin Dalecki wrote:
> Alexander Viro wrote:
> > At the very least, use canonical bytesex and field sizes.  Anything less
> > is just begging for trouble.  And in case of procfs or its equivalents,
> > _use_ the_ _damn_ _ASCII_ _representations_.  scanf(3) is there for
> > purpose.
> 
> And the purpose of scanf in system level applications is to introduce
> nice opportunities for buffer overruns and string formatting bugs.

I've done quite a bit more kernel profiling and I've found that overhead for 
converting numbers to ascii for transport to proc is significant, and there 
are other overheads as well, such as the sprintf and proc file open.  These 
must be matched by corresponding overhead on the user space side, which I 
have not profiled.  I'll take some time and present these numbers properly at 
some point.

Not that I think we are going to change this way of doing things any time
soon - Linus has spoken - but at least we should know what the overheads are.
Programmers should not labor under the misaprehension that this is an 
efficient interface.

--
Daniel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Albert D. Cahalan]

Daniel Phillips writes:

> I've done quite a bit more kernel profiling and I've found that
> overhead for converting numbers to ascii for transport to proc is
> significant, and there are other overheads as well, such as the
> sprintf and proc file open.  These must be matched by corresponding
> overhead on the user space side, which I have not profiled.  I'll
> take some time and present these numbers properly at some point.

You said "top -d .1" was 18%, with 11% user, and konsole at 9%.
So that gives:

9% konsole
7% kernel
2% top
0% X server ????

If konsole is well-written, that 9% should drop greatly as konsole
falls behind on a busy system. For example, when scrolling rapidly
it might skip whole screenfuls of data. Hopefully those characters
are rendered in a reasonably efficient way.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Daniel Phillips]

On November 5, 2001 11:46 pm, Albert D. Cahalan wrote:
> Daniel Phillips writes:
> 
> > I've done quite a bit more kernel profiling and I've found that
> > overhead for converting numbers to ascii for transport to proc is
> > significant, and there are other overheads as well, such as the
> > sprintf and proc file open.  These must be matched by corresponding
> > overhead on the user space side, which I have not profiled.  I'll
> > take some time and present these numbers properly at some point.
> 
> You said "top -d .1" was 18%, with 11% user, and konsole at 9%.
> So that gives:
> 
> 9% konsole
> 7% kernel
> 2% top
> 0% X server ????

No, the konsole 9% is outside of top's 18%.

> If konsole is well-written, that 9% should drop greatly as konsole
> falls behind on a busy system. For example, when scrolling rapidly
> it might skip whole screenfuls of data. Hopefully those characters
> are rendered in a reasonably efficient way.

I don't think I'll try to optimize konsole/QT/X today, thanks ;-)

Lets just not lose sight of the overhead connected with ASCII proc IO, it's a 
lot more than some seem to think.

--
Daniel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Stephen Satchell]

At 01:54 AM 11/6/01 +0100, Daniel Phillips wrote:
>Lets just not lose sight of the overhead connected with ASCII proc IO, it's a
>lot more than some seem to think.

Any idea what the overhead connected with a binary proc IO would be?  From 
looking at some of the code, it would appear that you have a lot of 
overhead no matter what you do.

Satch

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 07:20:39PM +0100, Tim Jansen wrote:
> On Sunday 04 November 2001 18:28, Daniel Phillips wrote:
> > > It needs special parsers and will be almost impossible to access from
> > > shell scripts.
> > No, look, he's proposing to put the binary encoding in hidden .files.  The
> > good old /proc files will continue to appear and operate as they do now.
> 
> But as he already said:
> 2)  As /proc files change, parsers must be changed in userspace
> 
> So if only some programs use the 'dot-files' and the other still use the 
> crappy text interface we still have the old problem for scripts, only with a 
> much larger effort.

So we have a gradual transition - nothing breaks more than it does already,
and applications can migrate to the stable API over time.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Linus Torvalds]

In article <20011104172742Z16629-26013+37@humbolt.nl.linux.org>,
Daniel Phillips  <phillips@bonn-fries.net> wrote:
>On November 4, 2001 05:45 pm, Tim Jansen wrote:
>> > The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it
>> > is a list of elements, wherein an element can itself be a list (or a
>> 
>> Why would anybody want a binary encoding? 
>
>Because they have a computer?

That's a stupid argument.

The computer can parse anything. 

It's us _humans_ that are limited at parsing. We like text interfaces,
because that's how we are brought up. We aren't good at binary, and
we're not good at non-linear, "structured" interfaces.

In contrast, a program can be taught to parse the ascii files quite
well, and does not have the inherent limitations we humans have. Sure,
it has _other_ limitations, but /proc being ASCII is sure as hell not
one of them.

In short: /proc is ASCII, and will so remain while I maintain a kernel.
Anything else is stupid.

Handling spaces and newlines is easy enough - see the patches from Al
Viro, for example.

		Linus

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 07:07:53PM +0000, Linus Torvalds wrote:
> In article <20011104172742Z16629-26013+37@humbolt.nl.linux.org>,
> Daniel Phillips  <phillips@bonn-fries.net> wrote:
> >On November 4, 2001 05:45 pm, Tim Jansen wrote:
> >> > The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it
> >> > is a list of elements, wherein an element can itself be a list (or a
> >> 
> >> Why would anybody want a binary encoding? 
> >
> >Because they have a computer?
> 
> That's a stupid argument.
> 
> The computer can parse anything. 
> 
> It's us _humans_ that are limited at parsing. We like text interfaces,
> because that's how we are brought up. We aren't good at binary, and
> we're not good at non-linear, "structured" interfaces.
> 
> In contrast, a program can be taught to parse the ascii files quite
> well, and does not have the inherent limitations we humans have. Sure,
> it has _other_ limitations, but /proc being ASCII is sure as hell not
> one of them.
> 
> In short: /proc is ASCII, and will so remain while I maintain a kernel.
> Anything else is stupid.

I agree that it would be stupid *not* to have an ASCII proc.

But why not make a machine-readable /proc as well ?

> 
> Handling spaces and newlines is easy enough - see the patches from Al
> Viro, for example.

Obviously none of you have parsed something like:

[albatros:joe] $ cat /proc/mdstat 
Personalities : [raid0] [raid1] 
read_ahead 1024 sectors
md0 : active raid1 hdc1[1] hda1[0]
      51264 blocks [2/2] [UU]
      
md1 : active raid1 hdc5[1] hda5[0]
      10240128 blocks [2/2] [UU]
      
md2 : active raid0 hdc7[1] hda6[0]
      6661184 blocks 64k chunks
      
unused devices: <none>
[albatros:joe] $ 

Now this isn't even bad - the fun begins when a resync is running, when
mdstat contains *progress meters* like  "[====>      ] 42%".  While being
nicely readable for a human, this is a parsing nightmare.  Especially
because stuff like this changes over time.

The worst thing is, that you'll often see that your parser isn't strict
enough, and therefore won't fail loudly, but rather "mis-parse" the "GUI"
that somehow got put into /proc.

I think it's great to put these things in /proc, but not having a machine
readable form too is stupid, especially because this could be done with
*no* harm to the existing interface, and with very little code.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Dave Jones]

On Sun, 4 Nov 2001, Jakob Østergaard wrote:

> Now this isn't even bad - the fun begins when a resync is running, when
> mdstat contains *progress meters* like  "[====>      ] 42%".  While being
> nicely readable for a human, this is a parsing nightmare.  Especially
> because stuff like this changes over time.

Any program needing to parse this would just ignore the bits between [],
and convert the percentage to an int. Hardly a 'nightmare'.

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 08:32:17PM +0100, Dave Jones wrote:
> On Sun, 4 Nov 2001, Jakob Østergaard wrote:
> 
> > Now this isn't even bad - the fun begins when a resync is running, when
> > mdstat contains *progress meters* like  "[====>      ] 42%".  While being
> > nicely readable for a human, this is a parsing nightmare.  Especially
> > because stuff like this changes over time.
> 
> Any program needing to parse this would just ignore the bits between [],
> and convert the percentage to an int. Hardly a 'nightmare'.

You didn't read the output then.  The information about which disks are up
and which are failed, is put between square brackets too.  You don't want
to ignore that.

So just ignore square brackets that have "=" " " and ">" between them ?

What happens when someone decides  "[---->   ]" looks cooler ?

Or just parse the brackets with "U" "F" and "_" between them ?  What then
when someone decides that disks being resynced are marked with "R"  ?

Fuzzy matching.

Nightmare.


-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Sun, 4 Nov 2001, [iso-8859-1] Jakob ьstergaard wrote:

> So just ignore square brackets that have "=" " " and ">" between them ?
> 
> What happens when someone decides  "[---->   ]" looks cooler ?

First of all, whoever had chosen that output did a fairly idiotic thing.
But as for your question - you _do_ know what regular expressions are,
don't you?  And you do know how to do this particular regex without
any use of library functions, right?

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 03:06:27PM -0500, Alexander Viro wrote:
> 
> 
> On Sun, 4 Nov 2001, [iso-8859-1] Jakob %stergaard wrote:
> 
> > So just ignore square brackets that have "=" " " and ">" between them ?
> > 
> > What happens when someone decides  "[---->   ]" looks cooler ?
> 
> First of all, whoever had chosen that output did a fairly idiotic thing.
> But as for your question - you _do_ know what regular expressions are,
> don't you?  And you do know how to do this particular regex without
> any use of library functions, right?

A regex won't tell me if  345987 is a signed or unsigned 32-bit or 64-bit
integer,  or if it's a double.

Sure, implement arbitrary precision arithmetic in every single app out there
using /proc....

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Kai Henningsen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 1318 bytes --]

jakob@unthought.net (Jakob ¥stergaard)  wrote on 04.11.01 in <20011104211118.U14001@unthought.net>:
[quoteto.xps]
> On Sun, Nov 04, 2001 at 03:06:27PM -0500, Alexander Viro wrote:
> >
> >
> > On Sun, 4 Nov 2001, [iso-8859-1] Jakob %stergaard wrote:
> >
> > > So just ignore square brackets that have "=" " " and ">" between them ?
> > >
> > > What happens when someone decides  "[---->   ]" looks cooler ?
> >
> > First of all, whoever had chosen that output did a fairly idiotic thing.
> > But as for your question - you _do_ know what regular expressions are,
> > don't you?  And you do know how to do this particular regex without
> > any use of library functions, right?
>
> A regex won't tell me if  345987 is a signed or unsigned 32-bit or 64-bit
> integer,  or if it's a double.

You do not *need* that information at runtime. If you think you do, you're  
doing something badly wrong.

I cannot even imagine what program would want that information.

> Sure, implement arbitrary precision arithmetic in every single app out there
> using /proc....

Bullshit. Implement whatever arithmetic is right *for your problem*. And  
notice when the value you get doesn't fit so you can tell the user he  
needs a newer version. That's all.

There's no reason whatsoever to care what data type the kernel used.

MfG Kai

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 11, 2001 at 12:06:00PM +0200, Kai Henningsen wrote:
> jakob@unthought.net (Jakob ¥stergaard)  wrote on 04.11.01 in <20011104211118.U14001@unthought.net>:
...
> >
> > A regex won't tell me if  345987 is a signed or unsigned 32-bit or 64-bit
> > integer,  or if it's a double.
> 
> You do not *need* that information at runtime. If you think you do, you're  
> doing something badly wrong.

I would prefer to have the information at compile-time, so that I would get
a compiler error if I did something wrong.

But that's unrealistic - some counter could change it's type from kernel
release to kernel release.

Now, my program needs to deal with the data, perform operations on it,
so naturally I need to know what kind of data I'm dealing with.  Most likely,
my software will *expect* some certain type, but if I have no way of verifying
that my assumption is correct, I will lose sooner or later...

> 
> I cannot even imagine what program would want that information.

Uh. Any program using /proc data ?

> 
> > Sure, implement arbitrary precision arithmetic in every single app out there
> > using /proc....
> 
> Bullshit. Implement whatever arithmetic is right *for your problem*. And  
> notice when the value you get doesn't fit so you can tell the user he  
> needs a newer version. That's all.
> 
> There's no reason whatsoever to care what data type the kernel used.

So my program runs for two months and then aborts with an error because
some counter just happened to no longer fit into whatever type I assumed
it was ?

Come on - you just can't code like that...

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Pascal Schmidt]

On Sun, 11 Nov 2001, Jakob Østergaard wrote:

> Now, my program needs to deal with the data, perform operations on it,
> so naturally I need to know what kind of data I'm dealing with.  Most likely,
> my software will *expect* some certain type, but if I have no way of verifying
> that my assumption is correct, I will lose sooner or later...

Why not read everything into a 1024-bit signed variable? Will work for 
every numeric value in /proc. It's a bit of a hassle to code, but it is 
possible. You only need to know the type if you want to write a numerical 
value to a file in /proc, and even then the driver behind that /proc entry 
should do sanity checks.

-- 
Ciao, Pascal

-<[ pharao90@tzi.de, netmail 2:241/215.72, home http://cobol.cjb.net/) ]>-

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Mon, Nov 12, 2001 at 02:43:41PM +0100, Pascal Schmidt wrote:
> On Sun, 11 Nov 2001, Jakob Østergaard wrote:
> 
> > Now, my program needs to deal with the data, perform operations on it,
> > so naturally I need to know what kind of data I'm dealing with.  Most likely,
> > my software will *expect* some certain type, but if I have no way of verifying
> > that my assumption is correct, I will lose sooner or later...
> 
> Why not read everything into a 1024-bit signed variable? Will work for 
> every numeric value in /proc. It's a bit of a hassle to code, but it is 
> possible. You only need to know the type if you want to write a numerical 
> value to a file in /proc, and even then the driver behind that /proc entry 
> should do sanity checks.

So for 99.9% of all cases my program will do much much more work than is
actually needed.

I may still save the data in a database, or go over the network with it,
so I should implement 1024 bit signed integers in all of that code too ?

And what happens when we do crypto and 1024 bits is not enough ?

I think the "use rediculously large datatypes" solution is a poor one,
as it can never cover all cases in the future, and it will impose a large
overhead on existing and new applications.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Riley Williams]

Hi Jakob.

>>> Sure, implement arbitrary precision arithmetic in every single app
>>> out there using /proc....

>> Bullshit. Implement whatever arithmetic is right *for your problem*.
>> And notice when the value you get doesn't fit so you can tell the
>> user he needs a newer version. That's all.
>> 
>> There's no reason whatsoever to care what data type the kernel used.

> So my program runs for two months and then aborts with an error
> because some counter just happened to no longer fit into whatever
> type I assumed it was ?
>
> Come on - you just can't code like that...

There are certain assumptions you can make about any given variable
without even seeing a specific value for it. For example:

 1. Does it make sense for the value to be negative? If not, use an
    unsigned variable.

    As an example, no systems can validly have a negative uptime, as
    that implies that it hasn't yet started running. It is for this
    very reason that a supposedly Roman coin inscribed with the date
    "37 BC" was known to be counterfeit - who measures time from an
    event that hasn't yet happenned?

 2. Does it make sense for the variable to report fractional values?
    If not, use integral variables.

    As an example, it makes no sense to have a fractional number of
    CPU's in a particular system - or, for that matter, for a given
    family to have the fabled 2.4 children !!!

 3. If fractional values do make sense, what accuracy is needed, and
    would it make sense to use scaled integers rather than reals?

    As an example, fractional values make sense for the current time
    but the need for accuracy indicates that scaled integers rather
    than reals are to be preferred, with the integers recording time
    in units of whatever fraction of a second is deemed sufficiently
    accurate for the intended purpose whilst still giving a practical
    range that can be stored.

    To take this one step further, and push the next version of the
    Y2K problem as far into the future as possible whilst providing
    a sufficient accuracy for most tasks nowadays, one could use a
    64-bit unsigned variable for the current time, but, rather than
    storing the number of seconds since epoch in it, store the
    number of xths of a second instead.

    As an example of this, a 64-bit unsigned value that measures the
    number of 40 ns intervals from Jan 1 00:00:00 UTC 1970 onwards
    will roll over at Jan 29 15:31:14 UTC 13661. This is a over 45%
    further in the future than the Y10K rollover seen elsewhere...

		( 13661 - 2000 )
		---------------- * 100 % = 145.762 %
		( 10000 - 2000 )

    With an interval of 40 ns one can accurately convert to seconds
    for backwards compatibility by simply dividing by 25,000,000.

 4. Is there any inherent limit on the range it can take? If not, use
    the largest available variables of the relevant type.

I've been doing this for 25 years now, and I've never regretted it.

Best wishes from Riley.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Luigi Genoni]

On Sun, 4 Nov 2001, Linus Torvalds wrote:

> In article <20011104172742Z16629-26013+37@humbolt.nl.linux.org>,
> Daniel Phillips  <phillips@bonn-fries.net> wrote:
> >On November 4, 2001 05:45 pm, Tim Jansen wrote:
> >> > The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it
> >> > is a list of elements, wherein an element can itself be a list (or a
> >>
> >> Why would anybody want a binary encoding?
> >
> >Because they have a computer?
>
> That's a stupid argument.
>
> The computer can parse anything.
>
> It's us _humans_ that are limited at parsing. We like text interfaces,
> because that's how we are brought up. We aren't good at binary, and
> we're not good at non-linear, "structured" interfaces.
>
> In contrast, a program can be taught to parse the ascii files quite
> well, and does not have the inherent limitations we humans have. Sure,
> it has _other_ limitations, but /proc being ASCII is sure as hell not
> one of them.
>
> In short: /proc is ASCII, and will so remain while I maintain a kernel.
> Anything else is stupid.
>
OHHH,
good sense at last!!
I was starting to worry

> Handling spaces and newlines is easy enough - see the patches from Al
> Viro, for example.
>
> 		Linus
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 05:45:45PM +0100, Tim Jansen wrote:
> On Sunday 04 November 2001 16:33, you wrote:
> > Maintaining the current /proc files is very simple, and it offers the
> > system administrator a lot of functionality that isn't reasonable to take
> > away now.
> >        * They should stay in a form close to the current one *
> 
> I doubt that it is worthwhile to keep them in the current form for any other 
> reason than compatibility (with existing software and people's habits). 

It's an essential feature for *many* sysadmins.  It's just so *easy* to hack
up a script to act on the information in some file - or to take a look with
"cat" to see how you RAID resync is coming along.

> It doesn't make sense to describe things in 200 different formats, you won't 
> help anybody with that. It also violates the good old principle of keeping 
> policy out of the kernel. And, for me, layout is clearly policy.

User-readable,  and machine-readable.   I think that covers everything. And
that's two formats.

Where's the policy ?   The only policy I see is the text-mode GUI in the
existing proc interface - and that is one place where I actually *like* the
policy as a user (sysadmin), but hate it as an application programmer.

> 
> The reason for proc's popularity is clearly that you can use any tool, from 
> cat over more/less to the text editor of choice, and read the files.

That's the reason why I want to keep the old proc files.

> There 
> should be ways to achieve this without putting things into the kernel.  Is 
> there is a way to implement a filesystem in user-space? What you could do is 
> to export the raw data using single-value-files, XML or whatever and then 
> provide an emulation of the old /proc files and possibly new ones in user 
> space. This could be as simple as writing a shell-script for each emulated 
> file.

You're proposing a replacement of /proc ?

My proposal was at least intended to be very simple and non-intrusive.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[John Levon]

On Sun, Nov 04, 2001 at 06:48:39PM +0100, Jakob Østergaard wrote:

> Where's the policy ?   The only policy I see is the text-mode GUI in the
> existing proc interface

well exactly

> - and that is one place where I actually *like* the
> policy as a user (sysadmin)

"meminfo" versus "cat /proc/meminfo" is not too great a leap in 2.7, when
the back-compat stuff gets dropped.

regards
john

-- 
"All this just amounts to more grist for the mill of the ill."
	- Elizabeth Wurtzel

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 18:48, you wrote:
> > Is there is a way to implement a filesystem in user-space? What you could
> You're proposing a replacement of /proc ?

I was asking whether there is a way to do compatibility stuff and human 
readable interfaces in user space. 

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 07:34:00PM +0100, Tim Jansen wrote:
> On Sunday 04 November 2001 18:48, you wrote:
> > > Is there is a way to implement a filesystem in user-space? What you could
> > You're proposing a replacement of /proc ?
> 
> I was asking whether there is a way to do compatibility stuff and human 
> readable interfaces in user space. 

Probably.

I'm just trying to:
1)  Supplement an unstable "API sort-of-thing" with something
    that's stable and can last.
2)  Come up with a realistic idea that can be implemented, tested,
    and deemed "obviously correct" and "good" in finite time
3)  Not break stuff more than it already is, and allow for a gradual
    transition to something that won't break mysteriously every ten
    kernel releases.

The idea is that if the userland application does it's parsing wrong, it should
either not compile at all, or abort loudly at run-time, instead of getting bad
values "sometimes".

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 19:59, you wrote:
> The idea is that if the userland application does it's parsing wrong, it
> should either not compile at all, or abort loudly at run-time, instead of
> getting bad values "sometimes".

All the XML parser interfaces that I have seen so far allow you to do things 
that will cause the code to fail when you do stupid things or are not 
prepared that there may appear unknown elements. Or you use a DTD, and then 
your code is guaranteed to fail after a change, which may be even worse.

One-value-files are a noticable exception, you must be VERY stupid if your 
code breaks because of an additional file.

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 08:19:39PM +0100, Tim Jansen wrote:
> On Sunday 04 November 2001 19:59, you wrote:
> > The idea is that if the userland application does it's parsing wrong, it
> > should either not compile at all, or abort loudly at run-time, instead of
> > getting bad values "sometimes".
> 
> All the XML parser interfaces that I have seen so far allow you to do things 
> that will cause the code to fail when you do stupid things or are not 
> prepared that there may appear unknown elements. Or you use a DTD, and then 
> your code is guaranteed to fail after a change, which may be even worse.

XML is pretty far from light-weight.  And it's only human readable in theory.

I like the *idea*, but XML is the wrong implementation of that idea.  Other than
that I think we could agree   ;)

> 
> One-value-files are a noticable exception, you must be VERY stupid if your 
> code breaks because of an additional file.

hehe, agreed.   The problem then is type information.

Consider:
-------------
int mf = open("/proc/meminfo/totalmem",O_RDONLY);
int32 mem;
read(mf, &mem, sizeof(mem));
-------------

Does this work ?   Yes of course.  But what if I ported my program to
a 64 bit arch...  The program still compiles.  It also runs.  But the
values are no longer correct.   Now *that* is hell.

Same story with ASCII.

I want type information.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 20:24, Jakob Østergaard wrote:
> Does this work ?   Yes of course.  But what if I ported my program to
> a 64 bit arch...  The program still compiles.  It also runs.  But the
> values are no longer correct.   Now *that* is hell.

Actually I worry more about those programs that are already compiled and will 
break when the kernel changes. But even if you recompile the code, how can 
you be sure that the programmer uses longs instead of ints for those 64 bit 
types? The C compiler allows the implicit conversion without warning. If you 
change the type the program has to be changed, no matter what you do.

> I want type information.

BTW nobody says to one-value-files can not have types (see my earlier posts 
in this thread).

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 08:41:34PM +0100, Tim Jansen wrote:
> On Sunday 04 November 2001 20:24, Jakob Østergaard wrote:
> > Does this work ?   Yes of course.  But what if I ported my program to
> > a 64 bit arch...  The program still compiles.  It also runs.  But the
> > values are no longer correct.   Now *that* is hell.
> 
> Actually I worry more about those programs that are already compiled and will 
> break when the kernel changes. But even if you recompile the code, how can 
> you be sure that the programmer uses longs instead of ints for those 64 bit 
> types? The C compiler allows the implicit conversion without warning. If you 
> change the type the program has to be changed, no matter what you do.

int get(result_t * result);

u32 a;
get(&a);

This will fail at compile time if result_t is 64 bits.

In C++ you could even do overloading where conversion is possible and
still have compile time errors when it's not possible.

> 
> > I want type information.
> 
> BTW nobody says to one-value-files can not have types (see my earlier posts 
> in this thread).

I don't dislike one-value-files - please tell me how you get type information

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Sunday 04 November 2001 20:55, Jakob Østergaard wrote:
> > BTW nobody says to one-value-files can not have types (see my earlier
> > posts in this thread).
> I don't dislike one-value-files - please tell me how you get type
> information

Using a ioctl that returns the type. 

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 09:13:35PM +0100, Tim Jansen wrote:
> On Sunday 04 November 2001 20:55, Jakob Østergaard wrote:
> > > BTW nobody says to one-value-files can not have types (see my earlier
> > > posts in this thread).
> > I don't dislike one-value-files - please tell me how you get type
> > information
> 
> Using a ioctl that returns the type. 

But that's not pretty   :)

Can't we think of something else ?

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alex Bligh - linux-kernel]

>> Using a ioctl that returns the type.
>
> But that's not pretty   :)
>
> Can't we think of something else ?

Well this sure isn't perfect, but to
illustrate it can be done with a text
interface (and the only restriction
is strings can't contain \n):

cat /proc/widget
# Format: '%l'
# Params: Number_of_Widgets
37

echo '38' > /proc/widget

cat /proc/widget
# Format: '%l'
# Params: Number_of_Widgets
38

cat /proc/widget | egrep -v '^#'
38

cat /proc/sprocket
# Format: '%l' '%s'
# Params: Number_of_Sprockets Master_Sprocket_Name
21
Foo Bar Baz

echo '22' > /proc/sprocket
# writes first value if no \n character written before
# close - all writes done simultaneously on close

cat /proc/sprocket | egrep -v '^#'
22
Foo Bar Baz

echo 'Master_Sprocket_Name\nBaz Foo Bar' > /proc/sprocket

cat /proc/sprocket | egrep -v '^#'
22
Baz Foo Bar

echo 'Master_Sprocket_Name\nFoo Foo Foo\nNumber_of_Sprockets\n111' > 
/proc/sprocket
# Simultaneous commit if /proc driver needs it
# i.e. it has get_lock() and release_lock()
# entries
cat /proc/sprocket | egrep -v '^#'
111
Foo Foo Foo

& nice user tools look at the '# Params:' line to find
what number param they want to read / alter.

--
Alex Bligh

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Sun, Nov 04, 2001 at 08:47:53PM -0000, Alex Bligh - linux-kernel wrote:
> >> Using a ioctl that returns the type.
> >
> > But that's not pretty   :)
> >
> > Can't we think of something else ?
> 
> Well this sure isn't perfect, but to
> illustrate it can be done with a text
> interface (and the only restriction
> is strings can't contain \n):

Such limitations are not acceptable.

> 
> cat /proc/widget
> # Format: '%l'
> # Params: Number_of_Widgets
> 37
> 
> echo '38' > /proc/widget
> 
> cat /proc/widget
> # Format: '%l'
> # Params: Number_of_Widgets
> 38

Good point with the parsing  :)

> 
> cat /proc/widget | egrep -v '^#'
> 38
> 
> cat /proc/sprocket
> # Format: '%l' '%s'
> # Params: Number_of_Sprockets Master_Sprocket_Name
> 21
> Foo Bar Baz

Not one value per file ?

> 
> echo '22' > /proc/sprocket
> # writes first value if no \n character written before
> # close - all writes done simultaneously on close
> 
> cat /proc/sprocket | egrep -v '^#'
> 22
> Foo Bar Baz
> 
> echo 'Master_Sprocket_Name\nBaz Foo Bar' > /proc/sprocket
> 
> cat /proc/sprocket | egrep -v '^#'
> 22
> Baz Foo Bar
> 
> echo 'Master_Sprocket_Name\nFoo Foo Foo\nNumber_of_Sprockets\n111' > 
> /proc/sprocket
> # Simultaneous commit if /proc driver needs it
> # i.e. it has get_lock() and release_lock()
> # entries
> cat /proc/sprocket | egrep -v '^#'
> 111
> Foo Foo Foo
> 
> & nice user tools look at the '# Params:' line to find
> what number param they want to read / alter.

How about:

We keep old proc files.

For each file, we make a .directory.

For example - for /proc/meminfo, we make a /proc/.meminfo/ directory
that contains the files
 MemTotal
 MemFree
 MemShared
 etc.

cat /proc/.meminfo/MemTotal gives you
"u32:KB:513276"

The kernel code for printing this is something like
 sprintf(..., "%s:%s:%u", DPI_T_U32, DPI_U_KB, i.memtotal);

The types and the units are necessary. But furthermore we do not
want various developers to be using different ways of writing the
types and units (KB vs. kB, vs. KiB).  Defines will ensure that
(if they are used - but they lend themselves to being used), and
once a new define is introduced it is fairly easy to document and
export to userland.

Not only does this format tell us exactly what's in the file (and
therefore how we should parse it), it also defines what we can write
to it (assuming we write the same types as we read - but that's a
reasonable assumption I suppose).

Problem:  Could it be made simpler to parse from scripting languages,
without making it less elegant to parse in plain C ?

If the values is a string, the string will begin after the second
semicolon (safe, since no type or unit can contain a colon and won't
have to, ever), and ends at the end of the file.  Voila, any character can be
in the string value.

And Al gets his #%^# text files   ;)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Stephen Satchell]

At 09:11 PM 11/4/01 +0100, Jakob Østergaard wrote:
>On Sun, Nov 04, 2001 at 09:13:35PM +0100, Tim Jansen wrote:
> > On Sunday 04 November 2001 20:55, Jakob Østergaard wrote:
> > > > BTW nobody says to one-value-files can not have types (see my earlier
> > > > posts in this thread).
> > > I don't dislike one-value-files - please tell me how you get type
> > > information
> >
> > Using a ioctl that returns the type.
>
>But that's not pretty   :)
>
>Can't we think of something else ?

I absolutely love how people want to re-invent the wheel.  If you want 
typed access (both read AND write) in a version-independent manner, then 
you really need to take a look at Simple Network Management Protocol, or 
SNMP.  It has everything you want:  named access, types, binary data or 
ASCII data or whatever data, and the ability for vendor, distribution, and 
version differences to be caught quickly and easily.  As new stuff is added 
or changed, all you need is a replacement MIB to be able to use the stuff.

Furthermore, SNMP is script friendly in that access to the data can be 
automated, with all conversions being done in userspace.

Finally, SNMP works over networks.

There are many, many security issues surrounding SNMP, but at least it 
exists, is well-understood, is already implemented in multiple systems, and 
it WORKS.

Why invent yet another replacement for sysctl?

My pair-o-pennies(tm) to this discussion...

Satch

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[zmwillow]

Jakob ?tergaard wrote:

>Here's my stab at the problems - please comment,
>
>We want to avoid these problems:
> 1)  It is hard to parse (some) /proc files from userspace
> 2)  As /proc files change, parsers must be changed in userspace
>
>Still, we want to keep on offering
> 3)  Human readable /proc files with some amount of pretty-printing
> 4)  A /proc fs that can be changed as the kernel needs those changes
>
>
>Taking care of (3) and (4):
>
>Maintaining the current /proc files is very simple, and it offers the system
>administrator a lot of functionality that isn't reasonable to take away now. 
>
>       * They should stay in a form close to the current one *
>
>
>Taking care of (1) and (2):
>
>For each file "f" in /proc, there will be a ".f" file which is a
>machine-readable version of "f", with the difference that it may contain extra
>information that one may not want to present to the user in "f".
>
>The dot-proc file is basically a binary encoding of Lisp (or XML), e.g. it is a
>list of elements, wherein an element can itself be a list (or a character string,
>or a host-native numeric type.  Thus, (key,value) pairs and lists thereof are
>possible, as well as tree structures etc.
>
>All data types are stored in the architecture-native format, and a simple
>library should be sufficient to parse any dot-proc file.
>
>
>So, we need a small change in procfs that does not in any way break
>compatibility - and we need a few lines of C under LGPL to interface with it.
>
>Tell me what you think - It is possible that I could do this (or something
>close) in the near future, unless someone shows me the problem with the
>approach.
>
>Thank you,
>

see http://sourceforge.net/projects/xmlprocfs/
i think this is a good idea that make the kernel output xml format 
informations.


best regards.
zmwillow

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Petr Baudis]

Hi,

> We want to avoid these problems:
>  1)  It is hard to parse (some) /proc files from userspace
>  2)  As /proc files change, parsers must be changed in userspace
> 
> Still, we want to keep on offering
>  3)  Human readable /proc files with some amount of pretty-printing
>  4)  A /proc fs that can be changed as the kernel needs those changes

  I've read the whole thread, but i still don't get it. Your solution doesn't
improve (1) for parsers in scripting languages, where it is frequently far
easier to parse ASCII stuff than messing with binary things, when not almost
impossible. So we don't make any progress here.  And for languages like C,
where this will have most use, there actually is solution and it is working.
So, please, can you enlighten me, what's so wrong on sysctl? It actually
provides exactly what do you want, and you even don't need to bother yourself
with open() etc ;). So it would be maybe better improving sysctl interface,
especially mirroring of all /proc stuff there, instead of arguing about scanf()
:-).

  So can you please explain me merits of your approach against sysctl?

-- 

				Petr "Pasky" Baudis

UN*X programmer, UN*X administrator, hobbies = IPv6, IRC
Real Users hate Real Programmers.
Public PGP key, geekcode and stuff: http://pasky.ji.cz/~pasky/

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Tim Jansen]

On Monday 05 November 2001 14:41, Petr Baudis wrote:
> So, please, can you enlighten me, what's so wrong on sysctl?

It doesn't work for complex data, especially lists. How do you want to 
configure devices, for example? 

bye...

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Ben Greear]

Tim Jansen wrote:

> On Monday 05 November 2001 14:41, Petr Baudis wrote:
> 
>>So, please, can you enlighten me, what's so wrong on sysctl?
>>
> 
> It doesn't work for complex data, especially lists. How do you want to 
> configure devices, for example? 


How about this:

struct ioctl_payload {
    int how_many;
    int* numbers;
};


User space sets things up normally, with a valid pointer in 'numbers'.
Kernel space copy_from_user the structure, then copy_from_user the number's
memory...

Can that work?  If so, numbers could of course be any data structure we want...


> 
> bye...
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Petr Baudis]

> So far sysctl is only used to configure kernel parameters, so there exists 
> one parameter in the system (per kernel). 
Not true.. see net.ipv* stuff - for each device, same parameters are to be set.
I see no problem in this.

> An example for devices would be mass storage devices. You may want to switch 
> DMA on and off per device. Using one-value-files you would have directories 
> called /somepath/0/dma, /somepath/1/dma and so on, and could turn on DMA on 
> device 1 by executing "echo 1 > /somepath/1/dma".
Set 1 for dev.ide.host0.bus0.target0.lun0.dma (we should stay consistent at
least with devfs, or we can give it up completely ;)

> Beside that there is the good old problem "who manages the sysctl namespace" 
> problem that is even more important if you want to use sysctl for device 
> drivers that may not even be in the kernel.
Well, why not maintainers of appropriate kernel sections, or even special
maintainer, like the one for device numbers. For each section of sysctl
namespace, subset of required ctls should be defined, obviously not restricting
optional bloat ;).

-- 

				Petr "Pasky" Baudis

UN*X programmer, UN*X administrator, hobbies = IPv6, IRC
Real Users hate Real Programmers.
Public PGP key, geekcode and stuff: http://pasky.ji.cz/~pasky/

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Mon, Nov 05, 2001 at 02:41:12PM +0100, Petr Baudis wrote:
> Hi,
> 
> > We want to avoid these problems:
> >  1)  It is hard to parse (some) /proc files from userspace
> >  2)  As /proc files change, parsers must be changed in userspace
> > 
> > Still, we want to keep on offering
> >  3)  Human readable /proc files with some amount of pretty-printing
> >  4)  A /proc fs that can be changed as the kernel needs those changes
> 
>   I've read the whole thread, but i still don't get it. Your solution doesn't
> improve (1) for parsers in scripting languages, where it is frequently far
> easier to parse ASCII stuff than messing with binary things, when not almost
> impossible. So we don't make any progress here.  And for languages like C,
> where this will have most use, there actually is solution and it is working.
> So, please, can you enlighten me, what's so wrong on sysctl? It actually
> provides exactly what do you want, and you even don't need to bother yourself
> with open() etc ;). So it would be maybe better improving sysctl interface,
> especially mirroring of all /proc stuff there, instead of arguing about scanf()
> :-).
> 
>   So can you please explain me merits of your approach against sysctl?

As far as I can see, I cannot read /proc/[pid]/* info using sysctl.

Then I need the other /proc/* files as well, not just /proc/sys/*

It seems to me that the sysctl interface does not have any type checking,
so if for example I want to read the jiffies counter and supply a 32-bit
field, sysctl will happily give me the first/last 32 bits of a field that
could as well be 64 bits (bit widths do sometimes change, even on architectures
that do not change).   How am I to know ?

If you look in kernel/sysctl.c, you'll see code like 

if (oldval && oldlenp) {
        get_user(len, oldlenp);
        if (len) {
                if (len > table->maxlen)
                        len = table->maxlen;
                if(copy_to_user(oldval, table->data, len))
                        return -EFAULT;
                if(put_user(len, oldlenp))
                        return -EFAULT;
        }
}
if (newval && newlen) {
        len = newlen;
        if (len > table->maxlen)
                len = table->maxlen;
        if(copy_from_user(table->data, newval, len))
                return -EFAULT;
}

Now is that pretty or what - Imagine someone trying to configure a 64-bit
kernel parameter with a 32 bit value   ;)

This could be tightened up of course - but the problem of knowing which
unit some number may be represented in is still there.  For example,
assuming I could read out partition sizes, well are they in blocks, bytes,
kilobytes, or what ?    Oh, I'm supposed to *know*, and *assume* such things
never change ?

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Petr Baudis]

> As far as I can see, I cannot read /proc/[pid]/* info using sysctl.
That can be added. We just have existing interface, and I don't propose to
stick on its actual state as it isn't convenient, but to extend it to cope our
needs.

> Then I need the other /proc/* files as well, not just /proc/sys/*
IMHO whole /proc should be mirrored by sysctl. Then, we can in 2.7 slowly move
those only to /proc/sys. Ideally, only /proc/[pid]/ and /proc/sys/ should be
present in /proc/.

> It seems to me that the sysctl interface does not have any type checking,
> so if for example I want to read the jiffies counter and supply a 32-bit
> field, sysctl will happily give me the first/last 32 bits of a field that
> could as well be 64 bits (bit widths do sometimes change, even on architectures
> that do not change).   How am I to know ?
 From the sysctl() manpage:

 size_t *oldlenp;  /* available room for old value,
                      overwritten by actual size of old value */

So, it will happily give you first/last 32 bits (at least something and your
application won't crash UNhappily ;), however it will write to location pointed
by oldlenp that jiffies are 64 bits.

> This could be tightened up of course - but the problem of knowing which
> unit some number may be represented in is still there.  For example,
> assuming I could read out partition sizes, well are they in blocks, bytes,
> kilobytes, or what ?    Oh, I'm supposed to *know*, and *assume* such things
> never change ?
Obviously they shouldn't. Any reason to change them? If you decide it would
be nice to give you also size in something different from blocks, you can just
introduce new ctl with suffix e.g. kb or so.

-- 

				Petr "Pasky" Baudis

UN*X programmer, UN*X administrator, hobbies = IPv6, IRC
Real Users hate Real Programmers.
Public PGP key, geekcode and stuff: http://pasky.ji.cz/~pasky/

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Alexander Viro]

On Tue, 6 Nov 2001, Petr Baudis wrote:

> > As far as I can see, I cannot read /proc/[pid]/* info using sysctl.
> That can be added. We just have existing interface, and I don't propose to
> stick on its actual state as it isn't convenient, but to extend it to cope our
> needs.

No, that cannot.  Guys, you've been told: it won't happen.  I think that
was loud and clear enough.

Can it.  Get a dictionary and look up the meaning of "veto".

Oh, and as for "let's extend existing interfaces just because we had flunked
'strings in C'" - if you need Hurd, you know where to find it.

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Jakob Østergaard]

On Tue, Nov 06, 2001 at 03:34:40AM -0500, Alexander Viro wrote:
> 
> 
> On Tue, 6 Nov 2001, Petr Baudis wrote:
> 
> > > As far as I can see, I cannot read /proc/[pid]/* info using sysctl.
> > That can be added. We just have existing interface, and I don't propose to
> > stick on its actual state as it isn't convenient, but to extend it to cope our
> > needs.
> 
> No, that cannot.  Guys, you've been told: it won't happen.  I think that
> was loud and clear enough.

Al, sure no half-assed ad-hoc /proc substitute should go in, but there *are*
*real* problems, and just because you don't see them in your daily life doesn't
mean they don't exist.

These real problems could use a real solution.   And *some* of us are at least
going to *discuss* what such a solution could be.

If, or when, we arrive at something where at least some of us agree, then we
will see if it will be your decision to include it at all.  At this stage in
the discussion the final (draft) solution may not have anything to do with
filessytems at all.  We don't know - or at least I don't know.

> 
> Can it.  Get a dictionary and look up the meaning of "veto".

Just because data is in a filesystem doesn't mean it doesn't need structure
*in* the data too.

Get over it Al.

> 
> Oh, and as for "let's extend existing interfaces just because we had flunked
> 'strings in C'" - if you need Hurd, you know where to find it.

My approach would be more like making another interface that could eventually
gradually obsolete an older and inadequate one.   I see nothing in /proc that's
worth extending on, as it stands today.

Clearly you have no comprehension of the problems that people are working on
solving with the new proc changes (or, rather, ideas for changes).

That's too bad.  It would have been great to have constructive critisism from
someone with your experience.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: PROPOSAL: dot-proc interface [was: /proc stuff]

[Petr Baudis]

> > > As far as I can see, I cannot read /proc/[pid]/* info using sysctl.
> > That can be added. We just have existing interface, and I don't propose to
> > stick on its actual state as it isn't convenient, but to extend it to cope
> > our needs.
> No, that cannot.  Guys, you've been told: it won't happen.  I think that was
> loud and clear enough.
So, if we want to be clear, we should freeze sysctl interface and focus to
/proc/? And sysctl is expected to disappear from the kernel by the time? If
not, I admit that I wasn't very much sure if exactly [pid] should go there.  If
answer is not, fine, as specially /proc/[pid]/ should be parsed with no
problems with scanf() (expect "(procname)" in /proc/[pid]/stat ;), as a
difference to some nightmares in device specific proc files etc. _Those_ are
which I propose to mirror in sysctl tree. You still can put nice progress bars
here to help humans (which is great), and you won't make programmers run around
crying something about linux [developers] stupidity. And I don't see any
disadvantage in this - /proc/ should remain supported forever and nothing stops
you using it, and you won't fill it with .bloat files.. (and that actually was
what Linus told he won't accept, iirc)

> Can it.  Get a dictionary and look up the meaning of "veto".
Well, 'veto' was for binary **** in /proc/. This is something completely
different. And actually done ;).

-- 

				Petr "Pasky" Baudis

UN*X programmer, UN*X administrator, hobbies = IPv6, IRC
Real Users hate Real Programmers.
Public PGP key, geekcode and stuff: http://pasky.ji.cz/~pasky/