fsync() does not flush all the file

fsync() does not flush all the file

[Berthier, Emmanuel]

Hi,

I work on Android platform and get a strange issue: sometimes the Sqlite journal is corrupted.
I've reproduced the issue by a simple C test program which mimics Sqlite behavior:
	Create a file
	Write around 40KB of data on it
	Flush it using fsync() or fdatasync()
	Close it
Then I power-cut the system within a couple of seconds: result: the end of the file is missing on disk.

I've reproduced the issue on different handsets (x86|arm) and different FS (ext3|ext4/barrier=1/data=write_back|ordered|journal) with Android 2.3.2 and kernel 2.6.35. These handset use eMMC memory.
I've checked with traces that the issue is upper eMMC block driver.

Some infos:
- the end of the file is flushed on eMMC only after around 5s.
- I can force the complete flush using the shell "sync" command.
- Laptop_mode is not enabled.
- dirty_write_back_centisecs has an impact on the flushing delay.
- ext3/commit option has not impact


What's wrong with fsync()?

Thanks,

Emmanuel BERTHIER.
Intel
---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: fsync() does not flush all the file

[Christoph Hellwig]

On Thu, Sep 01, 2011 at 06:12:51PM +0000, Berthier, Emmanuel wrote:
> I've checked with traces that the issue is upper eMMC block driver.

What do you mean with this sentence?

Does your problem reproduce without eMMC in the game?  Does writing
directly to the block device and then fsync the data get it to disk?

RE: fsync() does not flush all the file

[Berthier, Emmanuel]

From: Christoph Hellwig [mailto:hch@infradead.org] :
> On Thu, Sep 01, 2011 at 06:12:51PM +0000, Berthier, Emmanuel wrote:
> > I've checked with traces that the issue is upper eMMC block driver.
>
> What do you mean with this sentence?
>
> Does your problem reproduce without eMMC in the game?  Does writing
> directly to the block device and then fsync the data get it to disk?

Thanks for the answer.

I've patches the mmcd block driver in order to catch the start and the end of the written file, so I can detect when the flush is completed.

I've reproduced the issue on a PC with Ubuntu 10.04, but as the SSD drive contains cache, it's clear that this configuration can't guaranty the immediate flushing of the file on the flash memory.

A new info of today: the issue disappears if I call fflush() before fsync(): all the file is well flushed at a time.
The problem is that this is not the behavior of Sqlite3.6.2 (I've trace sys calls with strace, no fflush() on journal) (same for last 3.7.7.1).

Who's wrong? FS or Sqlite?
---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: fsync() does not flush all the file

[Christoph Hellwig]

On Fri, Sep 02, 2011 at 07:59:56AM +0000, Berthier, Emmanuel wrote:
> 
> I've reproduced the issue on a PC with Ubuntu 10.04, but as the SSD drive contains cache, it's clear that this configuration can't guaranty the immediate flushing of the file on the flash memory.
> 
> A new info of today: the issue disappears if I call fflush() before fsync(): all the file is well flushed at a time.
> The problem is that this is not the behavior of Sqlite3.6.2 (I've trace sys calls with strace, no fflush() on journal) (same for last 3.7.7.1).
> 
> Who's wrong? FS or Sqlite?

Oh. that actually sounds very much like a bug in sqlite.

If you use the fread/fwrite family of userspace-buffered stream I/O you
do need the fflush call to actually make the written data visible to the
kernel, fsync has no chance to pick it up before that.

It really surprise me sqlite uses fwrite for its intent log, as loggin
generall is absolutely latency an integrity critical, and the streaming
I/O APIs just make that more complicated.

RE: fsync() does not flush all the file

[Berthier, Emmanuel]

> From: linux-fsdevel-owner@vger.kernel.org [mailto:linux-fsdevel-owner@vger.kernel.org] On Behalf Of Christoph Hellwig
> On Fri, Sep 02, 2011 at 07:59:56AM +0000, Berthier, Emmanuel wrote:
> > 
> > I've reproduced the issue on a PC with Ubuntu 10.04, but as the SSD drive contains cache, it's clear that this configuration can't guaranty the > immediate flushing of the file on the flash memory.
> > 
> > A new info of today: the issue disappears if I call fflush() before fsync(): all the file is well flushed at a time.
> > The problem is that this is not the behavior of Sqlite3.6.2 (I've trace sys calls with strace, no fflush() on journal) (same for last 3.7.7.1).
> > 
> > Who's wrong? FS or Sqlite?
>
> Oh. that actually sounds very much like a bug in sqlite.
>
> If you use the fread/fwrite family of userspace-buffered stream I/O you
> do need the fflush call to actually make the written data visible to the
> kernel, fsync has no chance to pick it up before that.
>
> It really surprise me sqlite uses fwrite for its intent log, as loggin
> generall is absolutely latency an integrity critical, and the streaming
> I/O APIs just make that more complicated.

You're right, I used fprintf(). After replacing these calls by write() ones, no more issue!
And yes, Sqlite uses write() calls also.

Thanks a lot!

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.