Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Bodo Eggert]

Pavel Machek <pavel@ucw.cz> wrote:

> My proposal is
> 
> rename() stays.
> 
> replace(src, bar) is rename that ensures that bar will contain valid
> data after powerfail.

This can be done using implicit logic:

->E.g. on close(), mark inodes without being sync()ed as poisoned.
(I can think of more sophisticated logic, but ...)
->On completing the inode with the delayed allocations, unpoison it.
->Don't commit rename()s if the corresponding inode is poisoned.

Et Voila, everything replace is supposed to guarantee is guaranteed.

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Theodore Tso]

On Tue, Mar 31, 2009 at 11:27:33PM +0200, Bodo Eggert wrote:
> This can be done using implicit logic:
> 
> ->E.g. on close(), mark inodes without being sync()ed as poisoned.
> (I can think of more sophisticated logic, but ...)
> ->On completing the inode with the delayed allocations, unpoison it.
> ->Don't commit rename()s if the corresponding inode is poisoned.

Send us patches if you think it's that easy to do what you are
proposing.   I assure it's not easy.

				- Ted

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Pavel Machek]

On Tue 2009-03-31 20:06:57, Theodore Tso wrote:
> On Tue, Mar 31, 2009 at 11:27:33PM +0200, Bodo Eggert wrote:
> > This can be done using implicit logic:
> > 
> > ->E.g. on close(), mark inodes without being sync()ed as poisoned.
> > (I can think of more sophisticated logic, but ...)
> > ->On completing the inode with the delayed allocations, unpoison it.
> > ->Don't commit rename()s if the corresponding inode is poisoned.
> 
> Send us patches if you think it's that easy to do what you are
> proposing.   I assure it's not easy.

Well, implementing replace() syscall would be quite easy. Would you
want such a patch?
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Bodo Eggert]

On Wed, 1 Apr 2009, Pavel Machek wrote:
> On Tue 2009-03-31 20:06:57, Theodore Tso wrote:
> > On Tue, Mar 31, 2009 at 11:27:33PM +0200, Bodo Eggert wrote:

> > > This can be done using implicit logic:
> > > 
> > > ->E.g. on close(), mark inodes without being sync()ed as poisoned.
> > > (I can think of more sophisticated logic, but ...)
> > > ->On completing the inode with the delayed allocations, unpoison it.
> > > ->Don't commit rename()s if the corresponding inode is poisoned.
> > 
> > Send us patches if you think it's that easy to do what you are
> > proposing.   I assure it's not easy.
> 
> Well, implementing replace() syscall would be quite easy. Would you
> want such a patch?

You'd need - minus setting the flag on close - about the same logic:
- detecting if the inode are dirty
- detecting when the inode gets into a clean state
- delaying the commit of the rename() until then
- not replaying parts of the journal

As I understand, the replace() should not behave different from rename(),
except for making a guaranty. Since this guaranty is desired anyway, I 
think making it automatically would be a good thing. Especially until
all applications are rewritten ...

Off cause if you call replace, you know it's going to be OK, but you could 
also use pathconf($configdir, _PC_SYNC_BEFORE_RENAME)
(== 0: Sync or barrier implicit, (data = ordered)
    1: small chance of data loss due to not syncing, (vfat?)
    2: guarantee of data loss without sync (ext4))

// userspace.c
static inline int sync_before_rename(char * where, int is_important_data)
{
	#ifndef _PC_SYNC_BEFORE_RENAME
		// be optimistic for unimportant data
		return is_important_data;
	#else
		int ret = pathconf(where, _PC_SYNC_BEFORE_RENAME);
		if (is_important_data)
			return ret;
		return (ret > 1);
	#endif
}
-- 
How do I set my laser printer on stun?

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Pavel Machek]

Hi!

>> My proposal is 
>>
>> rename() stays.
>>
>> replace(src, bar) is rename that ensures that bar will contain valid
>> data after powerfail.
>>   
>
> Surely the only way to "insure" this is to spin up the drive, write the  
> meta-data and data back and make sure that it is not held in volatile  
> write cache?

Well, no. "will contain valid data" but may contain _old_ valid data.

So the way to do that would be "wait until you have to spin disk up
anyway or until timeout, then write data first, then do rename".

AFAICT that's semantics gnome (etc) wants.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Ric Wheeler]

Pavel Machek wrote:
>>>> We have a problem that user-space people do not want to
>>>> use 'fsync()', even when they are pointed to their code
>>>> which is doing create/write/rename/close without fsync().
>>>>         
>>> Well... they really don't want to spin the disk up for the
>>> fsync(). I'm not sure if fsync() is really sensible operation to use
>>> there.
>>>       
>> I'm personally concerned about hand-held, and in case of UBIFS
>> fsync is not too expensive - we work on flash and on fsync() we
>> write back only the stuff belonging to inode in question, and
>> nothing else.
>>     
>
> Well, I'm more concerned about spinning disks, having one even in my
> zaurus. And I do believe that fsync() will write more data than
> neccessary even in flash case.
>
>   
>>>> 1. truncate/write/close leads to empty files
>>>>         
>>> this is buggy.
>>>       
>> In FS, or in application?
>>     
>
> Application is buggy; no way kernel can help there.
>
>   
>>>> 2. create/write/rename leads to empty files
>>>>         
>>> ..but this should not be. If we want to make that explicit, we should
>>> provide "replace()" operation; where replace is rename that makes sure
>>> that source file is completely on media before commiting the rename.
>>>       
>> Well, OK, we can fsync() before rename, we just need clean rules
>> for this, so that all Linux FSes would follow them. Would be nice
>> to have final agreement on all this stuff.
>>     
>
> My proposal is 
>
> rename() stays.
>
> replace(src, bar) is rename that ensures that bar will contain valid
> data after powerfail.
>   

Surely the only way to "insure" this is to spin up the drive, write the 
meta-data and data back and make sure that it is not held in volatile 
write cache?

Why would calling this replace be better or more power efficient than 
what you need to do today?

ric

>   
>>> It is somehow similar to fsync()/rename(), but does not force disk
>>> spin up immediately -- it only inserts "barrier" between data blocks
>>> and rename. (And yes, it should be implemented as fsync()+rename() for
>>> filesystems like xfs. It can be implemented as plain rename for ext3
>>> and ext4 after the fixes...)
>>>       
>> Right. But I guess only few file-systems would really implement
>> this, because this is complex.
>>     
>
> Complex yes, but at least ext3+ext4+btrfs should, and they really have
> 90% of "market share" :-). ext3 and ext4 implementations are already
> done :-).
> 								Pavel
>   

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Diego Calleja]

On Domingo 29 Marzo 2009 14:26:00 Pavel Machek escribió:

> ...but this should not be. If we want to make that explicit, we should
> provide "replace()" operation; where replace is rename that makes sure
> that source file is completely on media before commiting the rename.

An "ad Linus-em" counterexample:

"And if we have a Linux-specific magic system call or sync action, it's 
going to be even more rarely used than fsync(). Do you think anybody 
really uses the OS X FSYNC_FULL ioctl? Nope. Outside of a few databases, 
it is almost certainly not going to be used, and fsync() will not be 
reliable in general.

So rather than come up with new barriers that nobody will use, filesystem 
people should aim to make "badly written" code "just work" unless people 
are really really unlucky. Because like it or not, that's what 99% of all 
code is."

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Pavel Machek]

On Sun 2009-03-29 16:57:06, Artem Bityutskiy wrote:
> ext Pavel Machek wrote:
>> On Sun 2009-03-29 16:07:35, Artem Bityutskiy wrote:
>>> Pavel Machek wrote:
>>>> On Sun 2009-03-29 16:00:45, Artem Bityutskiy wrote:
>>>>> Pavel Machek wrote:
>>>>>>>>> 2. create/write/rename leads to empty files
>>>>>>>> ..but this should not be. If we want to make that explicit, we should
>>>>>>>> provide "replace()" operation; where replace is rename that makes sure
>>>>>>>> that source file is completely on media before commiting the rename.
>>>>>>> Well, OK, we can fsync() before rename, we just need clean rules
>>>>>>> for this, so that all Linux FSes would follow them. Would be nice
>>>>>>> to have final agreement on all this stuff.
>>>>>> My proposal is 
>>>>>>
>>>>>> rename() stays.
>>>>> It stays and:
>>>>>
>>>>> 1. does _not_ fsync
>>>> Does not fsync. If someone wants to make sure one of the files is on
>>>> the disk, he should use replace(). [On non-linux systems, replace()
>>>> should be implemented as fsync/rename in libc or something.]
>>> I would be happy with these rules. But the fact is, application
>>> people just refuse to add fsync before rename. They say that the
>>> FS has to do this. And they say that even Linus supports them,
>>
>> That's good. fsync before rename would be ugly regression (on ext3 at
>> least). We should get them to use replace() syscall, not get them to
>> add fsyncs. [Of course, that means we need replace syscall first. :-)]
>
> I'd say it is better to fix ext3 then.

? I don't get this.

ext3's rename() is already equivalent to proposed replace(). The
problem is that btrfs's and ubifs's renames are not.

So doing extra fsync() on ext3 is actually an performance regression
-> we do not want applications to randomly add open-coded fsyncs().

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Artem Bityutskiy]

ext Pavel Machek wrote:
> On Sun 2009-03-29 16:07:35, Artem Bityutskiy wrote:
>> Pavel Machek wrote:
>>> On Sun 2009-03-29 16:00:45, Artem Bityutskiy wrote:
>>>> Pavel Machek wrote:
>>>>>>>> 2. create/write/rename leads to empty files
>>>>>>> ..but this should not be. If we want to make that explicit, we should
>>>>>>> provide "replace()" operation; where replace is rename that makes sure
>>>>>>> that source file is completely on media before commiting the rename.
>>>>>> Well, OK, we can fsync() before rename, we just need clean rules
>>>>>> for this, so that all Linux FSes would follow them. Would be nice
>>>>>> to have final agreement on all this stuff.
>>>>> My proposal is 
>>>>>
>>>>> rename() stays.
>>>> It stays and:
>>>>
>>>> 1. does _not_ fsync
>>> Does not fsync. If someone wants to make sure one of the files is on
>>> the disk, he should use replace(). [On non-linux systems, replace()
>>> should be implemented as fsync/rename in libc or something.]
>> I would be happy with these rules. But the fact is, application
>> people just refuse to add fsync before rename. They say that the
>> FS has to do this. And they say that even Linus supports them,
> 
> That's good. fsync before rename would be ugly regression (on ext3 at
> least). We should get them to use replace() syscall, not get them to
> add fsyncs. [Of course, that means we need replace syscall first. :-)]

I'd say it is better to fix ext3 then.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Artem Bityutskiy]

Andreas T.Auer wrote:
> On 29.03.2009 15:07 Artem Bityutskiy wrote:
>> Pavel Machek wrote:
>>> Does not fsync. If someone wants to make sure one of the files is on
>>> the disk, he should use replace(). [On non-linux systems, replace()
>>> should be implemented as fsync/rename in libc or something.]
> As a user I will avoid using any fs, which requires the tons of
> applications to be changed for a reasonable amount of data safety.
>> I would be happy with these rules. But the fact is, application
>> people just refuse to add fsync before rename.
> Because it slows down the performance.
>> They say that the
>> FS has to do this. 
> They say that FS should not write metadata for non-existing data and
> even overwrite "clean" metadata with "dirty" metadata. It is up to the
> fs to decide, whether fsync is needed to achieve this.
 
Well, this makes sense, but the fact is that FS developers did
not keep this in mind. And when we have been developing UBIFS,
we also naively assumed that user-space would just call fsync
if needed. And it was easier to implement stuff this way. And
it looked like POSIX and other Linux FSes assumed that.

But well, we can change UBIFS behavior, but it would be nice
to have some agreement on all this.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Pavel Machek]

On Sun 2009-03-29 16:07:35, Artem Bityutskiy wrote:
> Pavel Machek wrote:
>> On Sun 2009-03-29 16:00:45, Artem Bityutskiy wrote:
>>> Pavel Machek wrote:
>>>>>>> 2. create/write/rename leads to empty files
>>>>>> ..but this should not be. If we want to make that explicit, we should
>>>>>> provide "replace()" operation; where replace is rename that makes sure
>>>>>> that source file is completely on media before commiting the rename.
>>>>> Well, OK, we can fsync() before rename, we just need clean rules
>>>>> for this, so that all Linux FSes would follow them. Would be nice
>>>>> to have final agreement on all this stuff.
>>>> My proposal is 
>>>>
>>>> rename() stays.
>>> It stays and:
>>>
>>> 1. does _not_ fsync
>>
>> Does not fsync. If someone wants to make sure one of the files is on
>> the disk, he should use replace(). [On non-linux systems, replace()
>> should be implemented as fsync/rename in libc or something.]
>
> I would be happy with these rules. But the fact is, application
> people just refuse to add fsync before rename. They say that the
> FS has to do this. And they say that even Linus supports them,

That's good. fsync before rename would be ugly regression (on ext3 at
least). We should get them to use replace() syscall, not get them to
add fsyncs. [Of course, that means we need replace syscall first. :-)]
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Andreas T.Auer]

On 29.03.2009 15:07 Artem Bityutskiy wrote:
> Pavel Machek wrote:
>>
>> Does not fsync. If someone wants to make sure one of the files is on
>> the disk, he should use replace(). [On non-linux systems, replace()
>> should be implemented as fsync/rename in libc or something.]
>
As a user I will avoid using any fs, which requires the tons of
applications to be changed for a reasonable amount of data safety.
> I would be happy with these rules. But the fact is, application
> people just refuse to add fsync before rename.
Because it slows down the performance.
> They say that the
> FS has to do this. 
They say that FS should not write metadata for non-existing data and
even overwrite "clean" metadata with "dirty" metadata. It is up to the
fs to decide, whether fsync is needed to achieve this.

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Artem Bityutskiy]

Pavel Machek wrote:
> On Sun 2009-03-29 16:00:45, Artem Bityutskiy wrote:
>> Pavel Machek wrote:
>>>>>> 2. create/write/rename leads to empty files
>>>>> ..but this should not be. If we want to make that explicit, we should
>>>>> provide "replace()" operation; where replace is rename that makes sure
>>>>> that source file is completely on media before commiting the rename.
>>>> Well, OK, we can fsync() before rename, we just need clean rules
>>>> for this, so that all Linux FSes would follow them. Would be nice
>>>> to have final agreement on all this stuff.
>>> My proposal is 
>>>
>>> rename() stays.
>> It stays and:
>>
>> 1. does _not_ fsync
> 
> Does not fsync. If someone wants to make sure one of the files is on
> the disk, he should use replace(). [On non-linux systems, replace()
> should be implemented as fsync/rename in libc or something.]

I would be happy with these rules. But the fact is, application
people just refuse to add fsync before rename. They say that the
FS has to do this. And they say that even Linus supports them,
which is an argument I find difficult to fight against. This is
why I want clean rules.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Artem Bityutskiy]

ext Andreas T.Auer wrote:
> 
> On 29.03.2009 14:42 Artem Bityutskiy wrote:
>> Pavel Machek wrote:
>>
>>>> 1. truncate/write/close leads to empty files
>>> this is buggy.
>> In FS, or in application?
> In application of course. If you rewrite a huge file that way, you have
> a long-time risk of loosing data in a crash, even with sychronous writes.

You know, after reading all these blogs and discussions,
I will not be surprised if someone says this is an FS bug.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Pavel Machek]

On Sun 2009-03-29 16:00:45, Artem Bityutskiy wrote:
> Pavel Machek wrote:
>>>>> 2. create/write/rename leads to empty files
>>>> ..but this should not be. If we want to make that explicit, we should
>>>> provide "replace()" operation; where replace is rename that makes sure
>>>> that source file is completely on media before commiting the rename.
>>> Well, OK, we can fsync() before rename, we just need clean rules
>>> for this, so that all Linux FSes would follow them. Would be nice
>>> to have final agreement on all this stuff.
>>
>> My proposal is 
>>
>> rename() stays.
>
> It stays and:
>
> 1. does _not_ fsync

Does not fsync. If someone wants to make sure one of the files is on
the disk, he should use replace(). [On non-linux systems, replace()
should be implemented as fsync/rename in libc or something.]
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Andreas T.Auer]

On 29.03.2009 14:42 Artem Bityutskiy wrote:
> Pavel Machek wrote:
>
>>> 1. truncate/write/close leads to empty files
>>
>> this is buggy.
>
> In FS, or in application?
In application of course. If you rewrite a huge file that way, you have
a long-time risk of loosing data in a crash, even with sychronous writes.
>
>>> 2. create/write/rename leads to empty files
In the that case the time for the risk is reduced to the rename from the
viewpoint of the application developers, which don't know modern
re-ordering filesystems.

>> ..but this should not be. If we want to make that explicit, we should
>> provide "replace()" operation; where replace is rename that makes sure
>> that source file is completely on media before commiting the rename.
It is a hard task to change all the applications, there a lot of
orphaned projects, which are still used.
> Well, OK, we can fsync() before rename, we just need clean rules
> for this, so that all Linux FSes would follow them. Would be nice
> to have final agreement on all this stuff.
>
This slows down things, but you could also delay the writing of the
metadata pointing to non-existing data. Or is there any use for it after
the crash?

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Artem Bityutskiy]

Pavel Machek wrote:
>>>> 2. create/write/rename leads to empty files
>>> ..but this should not be. If we want to make that explicit, we should
>>> provide "replace()" operation; where replace is rename that makes sure
>>> that source file is completely on media before commiting the rename.
>> Well, OK, we can fsync() before rename, we just need clean rules
>> for this, so that all Linux FSes would follow them. Would be nice
>> to have final agreement on all this stuff.
> 
> My proposal is 
> 
> rename() stays.

It stays and:

1. does _not_ fsync
2. has synchronous fsync added
3. stays and have asynchronous fsync added?

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Pavel Machek]

>>> We have a problem that user-space people do not want to
>>> use 'fsync()', even when they are pointed to their code
>>> which is doing create/write/rename/close without fsync().
>>
>> Well... they really don't want to spin the disk up for the
>> fsync(). I'm not sure if fsync() is really sensible operation to use
>> there.
>
> I'm personally concerned about hand-held, and in case of UBIFS
> fsync is not too expensive - we work on flash and on fsync() we
> write back only the stuff belonging to inode in question, and
> nothing else.

Well, I'm more concerned about spinning disks, having one even in my
zaurus. And I do believe that fsync() will write more data than
neccessary even in flash case.

>>> 1. truncate/write/close leads to empty files
>>
>> this is buggy.
>
> In FS, or in application?

Application is buggy; no way kernel can help there.

>>> 2. create/write/rename leads to empty files
>>
>> ..but this should not be. If we want to make that explicit, we should
>> provide "replace()" operation; where replace is rename that makes sure
>> that source file is completely on media before commiting the rename.
>
> Well, OK, we can fsync() before rename, we just need clean rules
> for this, so that all Linux FSes would follow them. Would be nice
> to have final agreement on all this stuff.

My proposal is 

rename() stays.

replace(src, bar) is rename that ensures that bar will contain valid
data after powerfail.

>> It is somehow similar to fsync()/rename(), but does not force disk
>> spin up immediately -- it only inserts "barrier" between data blocks
>> and rename. (And yes, it should be implemented as fsync()+rename() for
>> filesystems like xfs. It can be implemented as plain rename for ext3
>> and ext4 after the fixes...)
>
> Right. But I guess only few file-systems would really implement
> this, because this is complex.

Complex yes, but at least ext3+ext4+btrfs should, and they really have
90% of "market share" :-). ext3 and ext4 implementations are already
done :-).
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Artem Bityutskiy]

Pavel Machek wrote:
> On Fri 2009-03-27 14:48:10, Artem Bityutskiy wrote:
>> UBIFS has exactly the same properties like ext4 - in case
>> of power cuts:
>>
>> 1. truncate/write/close leads to empty files
>> 2. create/write/rename leads to empty files
>>
>> UBIFS is used in hand-held and and power-cuts are very
>> often there, because users just remove battery often.
>>
>> I realize the "reality is different" argument, and already
>> concluded that we need a similar changes as Theo has done
>> for ext4:
>> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=bf1b69c0db7f9b9d8f02e94d40b19fca8336b991
>> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=f32b730a69bd56c5c9d704d8b75f03e90e290971
>> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=8411e347c3306ed36b8ca88611bf5fbf4d27d705
>>
>> We have a problem that user-space people do not want to
>> use 'fsync()', even when they are pointed to their code
>> which is doing create/write/rename/close without fsync().
> 
> Well... they really don't want to spin the disk up for the
> fsync(). I'm not sure if fsync() is really sensible operation to use
> there.

I'm personally concerned about hand-held, and in case of UBIFS
fsync is not too expensive - we work on flash and on fsync() we
write back only the stuff belonging to inode in question, and
nothing else.

>> 1. truncate/write/close leads to empty files
> 
> this is buggy.

In FS, or in application?

>> 2. create/write/rename leads to empty files
> 
> ..but this should not be. If we want to make that explicit, we should
> provide "replace()" operation; where replace is rename that makes sure
> that source file is completely on media before commiting the rename.

Well, OK, we can fsync() before rename, we just need clean rules
for this, so that all Linux FSes would follow them. Would be nice
to have final agreement on all this stuff.

> It is somehow similar to fsync()/rename(), but does not force disk
> spin up immediately -- it only inserts "barrier" between data blocks
> and rename. (And yes, it should be implemented as fsync()+rename() for
> filesystems like xfs. It can be implemented as plain rename for ext3
> and ext4 after the fixes...)

Right. But I guess only few file-systems would really implement
this, because this is complex.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

replace() system call needed (was Re: EXT4-ish "fixes" in UBIFS)

[Pavel Machek]

On Fri 2009-03-27 14:48:10, Artem Bityutskiy wrote:
> UBIFS has exactly the same properties like ext4 - in case
> of power cuts:
>
> 1. truncate/write/close leads to empty files
> 2. create/write/rename leads to empty files
>
> UBIFS is used in hand-held and and power-cuts are very
> often there, because users just remove battery often.
>
> I realize the "reality is different" argument, and already
> concluded that we need a similar changes as Theo has done
> for ext4:
> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=bf1b69c0db7f9b9d8f02e94d40b19fca8336b991
> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=f32b730a69bd56c5c9d704d8b75f03e90e290971
> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=8411e347c3306ed36b8ca88611bf5fbf4d27d705
>
> We have a problem that user-space people do not want to
> use 'fsync()', even when they are pointed to their code
> which is doing create/write/rename/close without fsync().

Well... they really don't want to spin the disk up for the
fsync(). I'm not sure if fsync() is really sensible operation to use
there.

> 1. truncate/write/close leads to empty files

this is buggy.

> 2. create/write/rename leads to empty files

...but this should not be. If we want to make that explicit, we should
provide "replace()" operation; where replace is rename that makes sure
that source file is completely on media before commiting the rename.

It is somehow similar to fsync()/rename(), but does not force disk
spin up immediately -- it only inserts "barrier" between data blocks
and rename. (And yes, it should be implemented as fsync()+rename() for
filesystems like xfs. It can be implemented as plain rename for ext3
and ext4 after the fixes...)

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html