msync() more expensive than fsync()?

msync() more expensive than fsync()?

[Chris Friesen]

I have some code that runs on a ramdisk-based filesystem.  Through a special 
device it mmaps a section of memory that is persistant over reboots, to be used 
for logging.  In order to guarantee that the logs were flushed to the memory 
area, I  assume that I need to use some kind of sync operation.  This is where 
things get a bit odd.

I did some testing with relatively small messages, 50 bytes or so, with results 
as follows:


Without any explicit flushing it takes 8 usec to log a message.

If I msync() only the pages that were touched in writing (usually 3 pages) it 
takes 39 usecs to log a message.

If I fsync() the entire file (200KB) it takes 12 usec to log a message.

Why the additional cost for msync()?  I would have thought it would be faster 
since it is explicitely for mmapped memory areas.  As a side note, the 
difference is even more extreme if a file is used on a disk-backed filesystem.

The kernel was 2.4.18.

Chris


-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: msync() more expensive than fsync()?

[Rik van Riel]

On Tue, 8 Apr 2003, Chris Friesen wrote:

> If I msync() only the pages that were touched in writing (usually 3
> pages) it takes 39 usecs to log a message.
> 
> If I fsync() the entire file (200KB) it takes 12 usec to log a message.
> 
> Why the additional cost for msync()?

If you only write into the file through an mmap()d area, then
I guess that fsync() is a NOP since it doesn't check the page
tables for dirty bits, while msync() does check the page tables.

This also means that fsync() won't see any dirty bits as long
as they're still only in the page tables, and consequently won't
write anything to disk, while msync() will write.

Note that I don't remember the details 100%, but IIRC it was
something like this.

Re: msync() more expensive than fsync()?

[H. Peter Anvin]

Followup to:  <3E92FAE6.8000300@nortelnetworks.com>
By author:    Chris Friesen <cfriesen@nortelnetworks.com>
In newsgroup: linux.dev.kernel
> 
> Without any explicit flushing it takes 8 usec to log a message.
> 
> If I msync() only the pages that were touched in writing (usually 3 pages) it 
> takes 39 usecs to log a message.
> 
> If I fsync() the entire file (200KB) it takes 12 usec to log a message.
> 
> Why the additional cost for msync()?  I would have thought it would be faster 
> since it is explicitely for mmapped memory areas.  As a side note, the 
> difference is even more extreme if a file is used on a disk-backed filesystem.
> 

Because fsync() does less work.  In particular, fsync() doesn't do the
work you want: unlike msync() it doesn't guarantee that memory maps
are consistent with the filesystem.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64