archive mirror
 help / color / mirror / Atom feed
From: Linus Torvalds <>
To: Rusty Russell <>
Cc: Zach Brown <>,, Ingo Molnar <>,
	Ulrich Drepper <>,
	Arjan van de Ven <>,
	Andrew Morton <>,
	Alan Cox <>,
	Evgeniy Polyakov <>,
	"David S. Miller" <>,
	Suparna Bhattacharya <>,
	Davide Libenzi <>,
	Jens Axboe <>,
	Thomas Gleixner <>,
	Dan Williams <>,
	Jeff Moyer <>,
	Simon Holm Thogersen <>,
Subject: Re: [PATCH 5/6] syslets: add generic syslets infrastructure
Date: Wed, 9 Jan 2008 14:58:10 -0800 (PST)	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On Thu, 10 Jan 2008, Rusty Russell wrote:
> I'd have to read his original statement, but eventfd doesn't build up state, 
> so I think it qualifies.

How about you guys battle it out by giving an example program usign the 

Here's a favourite really simple load of mine:

 - do the equivalent of "ls -lR" or "find /usr" as quickly as possible, 
   without playing "sort by the inode numbers" games (that don't work in 
   general, but are great for some filesystems)

   Do this on a directory that isn't newly created, but has had files 
   added and removed over time (so that the return order of "readdir()" 
   isn't dense and sorted in the inode tables already). The classic 
   example is "ls -l /usr/bin" or similar.

This is actually a fairly hard load, because there's two easily tested 
cases that are both equally important: hot-cache (no IO at all, just CPU), 
and cold-cache (all about trying to get concurrent IO on inode lookups 
going, to get the disk elevator working in the *absense* of any sorting).

And almost all of the operations are operations that right now have no 
asynchronous model except for full threads (ie neither filename nor inode 
lookup have any aio_xyz() things).

How can we do something like *that*? It's about as simple an IO test 
program you can imagine, while I'd argue that it is still a reasonably 
"realistic" thing to do, and interesting for asynchronous operations.

How would do you something like this, striving to allow overlap of IO, and 
getting (hopefully) the block layer to create bigger request sizes?

To make it simple, let's make the *only* operation we care about being 
asynchronous be that "lstat()". And instead of printing out each file, 
just add up the sizes or something (ie make this approximate "du -sh"). 

But the important thing is that if things are cached, it should be the 
same speed as this trivial program, ie the async interfaces shouldn't slow 
things down. But if things require IO, hopefully we can get at least some 
reasonable number of concurrent IO's going, and making the program not 
seek back and forth quite so much).

Try this with and without a "echo 3 > /proc/sys/vm/drop_caches" in 
between. Both are real and relevant usage cases, I think.

And *simplicity* of the end result really does matter. If it's not simple 
to use, people won't use it.


#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <dirent.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>

static void find(const char *base, int baselen);

static void handle(const char *name, int namelen)
	struct stat st;

	if (lstat(name, &st) < 0)
	printf("%8lu %s\n", st.st_size, name);
	if (S_ISDIR(st.st_mode))
		find(name, namelen);

static void find(const char *base, int baselen)
	DIR *dir;
	char *name = malloc(baselen + 255 + 2);

	if (!name)
	memcpy(name, base, baselen);
	name[baselen] = '/';
	dir = opendir(base);
	if (dir) {
		struct dirent *de;
		while ((de = readdir(dir)) != NULL) {
			const char *p = de->d_name;
			int len = strlen(p);
			if (len > 255)
			if (p[0] == '.') {
				if (len == 1)
				if (len == 2 && p[1] == '.')
			memcpy(name + baselen + 1, de->d_name, len+1);
			handle(name, baselen + 1 + len);

int main(int argc, char **argv)
	return 0;

  reply	other threads:[~2008-01-09 23:00 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-12-06 23:20 syslets v7: back to basics Zach Brown
2007-12-06 23:20 ` [PATCH 1/6] indirect: use asmlinkage in i386 syscall table prototype Zach Brown
2007-12-06 23:20   ` [PATCH 2/6] syslet: asm-generic support to disable syslets Zach Brown
2007-12-06 23:20     ` [PATCH 3/6] syslet: introduce abi structs Zach Brown
2007-12-06 23:20       ` [PATCH 4/6] syslets: add indirect args Zach Brown
2007-12-06 23:20         ` [PATCH 5/6] syslets: add generic syslets infrastructure Zach Brown
2007-12-06 23:20           ` [PATCH 6/6] syslets: add both 32bit and 64bit x86 syslet support Zach Brown
2007-12-07 11:55           ` [PATCH 5/6] syslets: add generic syslets infrastructure Evgeniy Polyakov
2007-12-07 18:24             ` Zach Brown
2008-01-09  2:03           ` Rusty Russell
2008-01-09  3:00             ` Zach Brown
2008-01-09  3:48               ` Rusty Russell
2008-01-09 18:16                 ` Zach Brown
2008-01-09 22:04                   ` Rusty Russell
2008-01-09 22:58                     ` Linus Torvalds [this message]
2008-01-09 23:05                       ` Linus Torvalds
2008-01-09 23:47                       ` Zach Brown
2008-01-10  1:18                       ` Rusty Russell
2008-01-09 23:15                     ` Davide Libenzi
2008-01-10  5:41                   ` Jeff Garzik
2007-12-08 12:40   ` [PATCH 1/6] indirect: use asmlinkage in i386 syscall table prototype Simon Holm Thøgersen
2007-12-08 21:22     ` Zach Brown
2007-12-08 12:52 ` [PATCH] Fix casting on architectures with 32-bit pointers/longs Simon Holm Thøgersen
2007-12-10 19:46 ` syslets v7: back to basics Jens Axboe
2007-12-10 21:30 ` Phillip Susi
2007-12-10 22:15   ` Zach Brown

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).