Re: Linux 2.6.29

[Linus Torvalds <torvalds@linux-foundation.org>]

On Wed, 25 Mar 2009, Kyle Moffett wrote:
> 
> Well, I think the goal is not to *replace* the POSIX API or even
> provide "transactional" guarantees.  The performance penalty for
> atomic transactions is pretty high, and most programs (like GIT) don't
> really give a damn, as they provide that on a higher level.

Speaking with my 'git' hat on, I can tell that

 - git was designed to have almost minimal requirements from the 
   filesystem, and to not do anything even half-way clever.

 - despite that, we've hit an absolute metric sh*tload of filesystem bugs 
   and misfeatures. Some very much in Linux. And some I bet git was the 
   first to ever notice, exactly because git tries to be really anal, in 
   ways that I can pretty much guarantee no normal program _ever_ is.

For example, the latest one came from git actually checking the error code 
from 'close()'. Tell me the last time you saw anybody do that in a real 
program. Hint: it's just not done. EVER. Git does it (and even then, git 
does it only for the core git object files that we care about so much), 
and we found a real data-loss CIFS bug thanks to that. Afaik, the bug has 
been there for a year and half. Don't tell me nobody uses cifs.

Before that, we had cross-directory rename bugs. Or the inexplicable 
"pread() doesn't work correctly on HP-UX". Or the "readdir() returns the 
same entry multiple times" bug. And all of this without ever doing 
anything even _remotely_ odd. No file locking, no rewriting of old files, 
no lseek()ing in directories, no nothing.

Anybody who wants more complex and subtle filesystem interfaces is just 
crazy. Not only will they never get used, they'll definitely not be 
stable.

> To be honest I think we could provide much better data consistency
> guarantees and remove a lot of fsync() calls with just a basic
> per-filesystem barrier() call.

The problem is not that we have a lot of fsync() calls. Quite the reverse. 
fsync() is really really rare. So is being careful in general. The number 
of applications that do even the _minimal_ safety-net of "create new file, 
rename it atomically over an old one" is basically zero. Almost everybody 
ends up rewriting files with something like

	open(name, O_CREAT | O_TRUNC, 0666)
	write();
	close();

where there isn't an fsync in sight, nor any "create temp file", nor 
likely even any real error checking on the write(), much less the 
close().

And if we have a Linux-specific magic system call or sync action, it's 
going to be even more rarely used than fsync(). Do you think anybody 
really uses the OS X FSYNC_FULL ioctl? Nope. Outside of a few databases, 
it is almost certainly not going to be used, and fsync() will not be 
reliable in general.

So rather than come up with new barriers that nobody will use, filesystem 
people should aim to make "badly written" code "just work" unless people 
are really really unlucky. Because like it or not, that's what 99% of all 
code is.

The undeniable FACT that people don't tend to check errors from close() 
should, for example, mean that delayed allocation must still track disk 
full conditions, for example. If your filesystem returns ENOSPC at close() 
rather than at write(), you just lost error coverage for disk full cases 
from 90% of all apps. It's that simple.

Crying that it's an application bug is like crying over the speed of 
light: you should deal with *reality*, not what you wish reality was. Same 
goes for any complaints that "people should write a temp-file, fsync it, 
and rename it over the original". You may wish that was what they did, but 
reality is that "open(filename, O_TRUNC | O_CREAT, 0666)" thing.

Harsh, I know. And in the end, even the _good_ applications will decide 
that it's not worth the performance penalty of doing an fsync(). In git, 
for example, where we generally try to be very very very careful, 
'fsync()' on the object files is turned off by default.

Why? Because turning it on results in unacceptable behavior on ext3. Now, 
admittedly, the git design means that a lost new DB file isn't deadly, 
just potentially very very annoying and confusing - you may have to roll 
back and re-do your operation by hand, and you have to know enough to be 
able to do it in the first place.

The point here? Sometimes those filesystem people who say "you must use 
fsync() to get well-defined semantics" are the same people who SCREWED IT 
UP SO DAMN BADLY THAT FSYNC ISN'T ACTUALLY REALISTICALLY USEABLE!

Theory and practice sometimes clash. And when that happens, theory loses. 
Every single time. 

		Linus