RE: Memory mapped files question

RE: Memory mapped files question

[Perez-Gonzalez, Inaky]

> From: Bryan Shumsky [mailto:bzs@via.com]

> Hi, everyone.  I'm running into a problem that I hope someone else has
seen,
> and maybe can help solve.  We're using the mmap system function for memory
> mapped files, but our updates never get flushed until we munmap or msysnc.

I thought that was the way it was supposed to work.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)

Re: Memory mapped files question

[Bryan Shumsky]

Hi, everyone.  Thanks for all your responses.  Our confusion is that in Unix
environments, when we modify memory in memory-mapped files the underlying
system flusher manages to flush the files for us before the files are
munmap'ed or msysnc'ed.

Rewriting all of our code to manually handle the flushing is a MAJOR
undertaking, so I was hoping there might be some sneaky solution you could
come up with.  Any ideas?

Thanks again,

-- Bryan Shumsky
Director of Engineering
Via Systems, Inc.

----- Original Message -----
From: "Perez-Gonzalez, Inaky" <inaky.perez-gonzalez@intel.com>
To: "'Bryan Shumsky'" <bzs@via.com>; <linux-kernel@vger.kernel.org>
Sent: Monday, April 14, 2003 12:31 PM
Subject: RE: Memory mapped files question


>
> > From: Bryan Shumsky [mailto:bzs@via.com]
>
> > Hi, everyone.  I'm running into a problem that I hope someone else has
> seen,
> > and maybe can help solve.  We're using the mmap system function for
memory
> > mapped files, but our updates never get flushed until we munmap or
msysnc.
>
> I thought that was the way it was supposed to work.
>
> Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
> (and my fault)
>
>

Re: Memory mapped files question

[Alan Cox]

On Llu, 2003-04-14 at 20:42, Bryan Shumsky wrote:
> Rewriting all of our code to manually handle the flushing is a MAJOR
> undertaking, so I was hoping there might be some sneaky solution you could
> come up with.  Any ideas?

Create a thread that does msync's every so often. Its that simple

Re: Memory mapped files question

[Richard B. Johnson]

On Mon, 14 Apr 2003, Bryan Shumsky wrote:

> Hi, everyone.  Thanks for all your responses.  Our confusion is that in Unix
> environments, when we modify memory in memory-mapped files the underlying
> system flusher manages to flush the files for us before the files are
> munmap'ed or msysnc'ed.
>

So? Why should you care?

> Rewriting all of our code to manually handle the flushing is a MAJOR
> undertaking, so I was hoping there might be some sneaky solution you could
> come up with.  Any ideas?
>
> Thanks again,
>

Memory mapped files are supposed to be accessed through memory!
Any program that needs to know what's on the physical disk is
broken. If you need to write to files and know when they are
written to the physical media, you use a journaled file-system.

Anytime you write to a memory mapped file, anybody else who
also mapped that file will see the changes as long as the file
was mapped MAP_SHARED. If your program mapped it MAP_PRIVATE, it's
probably broken.

There is nothing to work-around. All working databases work
this way.

Memory mapped files are used to speed access to large files. Only
the portion of the file actually being accessed needs to be resident
in memory. Multiple readers and writers can access these memory-
mapped portions without enduring the time for long seeks to various
pieces of the file. The snippets of the file get written to the
media using a LRU (least recently used) mechanism so that stuff that
isn't being actively updated is the stuff that gets written to the
device (actually replaced with something that needs to be accessed).
Therefore, there is absolutely no possible way for this interchange
to be somehow handled by the user. It doesn't happen in any memory-
mapped file --ever.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.

Re: Memory mapped files question

[Frank van Maarseveen]

On Mon, Apr 14, 2003 at 04:13:52PM -0400, Richard B. Johnson wrote:
> 
> Memory mapped files are supposed to be accessed through memory!
> Any program that needs to know what's on the physical disk is
> broken. If you need to write to files and know when they are
> written to the physical media, you use a journaled file-system.

It is not that simple.
Shared mmaped files are _never_ flushed, at least in 2.4.x. So,
without an explicit msync() a process (innd comes to mind) may loose
years of updates upon a system crash or power outage.

I have learned to live with it but I still find this a bit awkward.

-- 
Frank

Re: Memory mapped files question

[Richard B. Johnson]

On Mon, 14 Apr 2003, Frank van Maarseveen wrote:

> On Mon, Apr 14, 2003 at 04:13:52PM -0400, Richard B. Johnson wrote:
> >
> > Memory mapped files are supposed to be accessed through memory!
> > Any program that needs to know what's on the physical disk is
> > broken. If you need to write to files and know when they are
> > written to the physical media, you use a journaled file-system.
>
> It is not that simple.
> Shared mmaped files are _never_ flushed, at least in 2.4.x. So,
> without an explicit msync() a process (innd comes to mind) may loose
> years of updates upon a system crash or power outage.
>
> I have learned to live with it but I still find this a bit awkward.
>
> --
> Frank
>

But it is that simple. If you need to update the file with the
memory copy an explicit msync() must be used. Also note that
msync() takes some parameters that may be different in different
processes that access that mmapped file. Therefore, even msync()
goesn't guarantee that everything gets updated for the next
power outage. So, if you absolutely-posix-a-tively need to get
that data onto a disk, they you need to use some kind of signaling
mechanism (to pause all writers), explicitly write everything from
the lowest to the highest, update your check-point file and close
it, then restart all writers. Upon restart, you can unwind to
the check-point file and restart.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.

Re: Memory mapped files question

[Chris Friesen]

Alan Cox wrote:
> On Llu, 2003-04-14 at 20:42, Bryan Shumsky wrote:
> 
>>Rewriting all of our code to manually handle the flushing is a MAJOR
>>undertaking, so I was hoping there might be some sneaky solution you could
>>come up with.  Any ideas?
>>
> 
> Create a thread that does msync's every so often. Its that simple

How do you deal with ensuring (or even trying to ensure) that the stuff *on 
disk* is sane?

If I understand correctly, msync() doesn't guarantee order of writes, so 
randomly firing off msync() calls doesn't help.

If I want to update an entry and then set a flag saying that the entry is 
correct, I need to have two msyncs, one for the entry data, and one for the 
flag.  I had hoped that I could avoid this by opening the file with O_SYNC, but 
hpa just disabused me of that notion...

Are the mmap semantics different for devices?

Chris



-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: Memory mapped files question

[H. Peter Anvin]

Followup to:  <004301c302bd$ed548680$fe64a8c0@webserver>
By author:    "Bryan Shumsky" <bzs@via.com>
In newsgroup: linux.dev.kernel
>
> Hi, everyone.  Thanks for all your responses.  Our confusion is that in Unix
> environments, when we modify memory in memory-mapped files the underlying
> system flusher manages to flush the files for us before the files are
> munmap'ed or msysnc'ed.
> 

Bullshit.  It might work on one particular Unix implementation, but
the definition of Unix, the Single Unix Standard, does explicitly
*not* require this behavior.

> Rewriting all of our code to manually handle the flushing is a MAJOR
> undertaking, so I was hoping there might be some sneaky solution you could
> come up with.  Any ideas?

Your code is fundamentally broken.  You need to fix it.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: Memory mapped files question

[Jamie Lokier]

H. Peter Anvin wrote:
> > Hi, everyone.  Thanks for all your responses.  Our confusion is
> > that in Unix environments, when we modify memory in memory-mapped
> > files the underlying system flusher manages to flush the files for
> > us before the files are munmap'ed or msysnc'ed.
> 
> Bullshit.  It might work on one particular Unix implementation, but
> the definition of Unix, the Single Unix Standard, does explicitly
> *not* require this behavior.

I presume that if you do write(), the Single Unix Standard allows the
data to remain dirty in RAM for an arbitrary duration too.

If I write() a file I expect it to be automatically written to disk
within a few minutes at most, where that is plausible.

Frank van Maarseveen wrote:
> Shared mmaped files are _never_ flushed, at least in 2.4.x. So,
> without an explicit msync() a process (innd comes to mind) may loose
> years of updates upon a system crash or power outage.

It's a quality of implementation issue if data can remain dirty in RAM
forever without ever being flushed.

Can this really happen with normal open/mmap/munmap/close usage, or
does it only occur with long-lived processes like innd which mmap a
file, dirty the pages but never munmap them?

If the former case does happen, I'd say we're failing on quality of
implementation.  If it's only the latter case, though, fair enough: the
application writer will have to use msync().

-- Jamie

Re: Memory mapped files question

[H. Peter Anvin]

Jamie Lokier wrote:
> 
> It's a quality of implementation issue if data can remain dirty in RAM
> forever without ever being flushed.
> 
> Can this really happen with normal open/mmap/munmap/close usage, or
> does it only occur with long-lived processes like innd which mmap a
> file, dirty the pages but never munmap them?
> 
> If the former case does happen, I'd say we're failing on quality of
> implementation.  If it's only the latter case, though, fair enough: the
> application writer will have to use msync().
> 

The latter, I'm pretty sure.  After all, that's what pgflush/bdflush is 
all about.

Re: Memory mapped files question

[H. Peter Anvin]

Chris Friesen wrote:
> 
>> munmap() and fsync() or msync() will flush it to disk; there is no reason
>> munmap() should unless perhaps the file was opened O_SYNC.
> 
> Wait a minute.  Shouldn't a file opened O_SYNC flush the writes as they
> happen,
> removing the requirement for any explicit syncing?  If it doesn't there
> are some very broken apps around.
> 

Not for mmap().

	-hpa

Re: Memory mapped files question

[Chris Friesen]

H. Peter Anvin wrote:
 > Followup to:  <11640000.1050332688@[10.10.2.4]> By author:    "Martin J.
 > Bligh" <mbligh@aracnet.com> In newsgroup: linux.dev.kernel
 >
 >>> Martin, something which was not mentioned last week (I've just checked).
 >>>
 >>> It's OK if we never write to disk unless explicitely told, but will we
 >>> writeback when we munmap?

 > munmap() and fsync() or msync() will flush it to disk; there is no reason
 > munmap() should unless perhaps the file was opened O_SYNC.

Wait a minute.  Shouldn't a file opened O_SYNC flush the writes as they happen,
removing the requirement for any explicit syncing?  If it doesn't there are some 
very broken apps around.

Chris


-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

RE: Memory mapped files question

[Perez-Gonzalez, Inaky]

> From: Bryan Shumsky [mailto:bzs@via.com]
> 
> Hi, everyone.  Thanks for all your responses.  Our confusion is that in
Unix
> environments, when we modify memory in memory-mapped files the underlying
> system flusher manages to flush the files for us before the files are
> munmap'ed or msysnc'ed.
> 
> Rewriting all of our code to manually handle the flushing is a MAJOR
> undertaking, so I was hoping there might be some sneaky solution you could
> come up with.  Any ideas?

Have a high prio thread msync()ing every now and then? OOps forget it,
I never said that, it is really a lame solution :) you can run in all
kinds of trouble.

Or maybe it is not that lame if you know *exactly* when do you need
the flush (for example, some other program is going to access the data);
this way you can signal the process who did the modification for it
to issue an msync(). However, this is going to be some delicate
synchronization
between the two apps.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)

Re: Memory mapped files question

[H. Peter Anvin]

Followup to:  <11640000.1050332688@[10.10.2.4]>
By author:    "Martin J. Bligh" <mbligh@aracnet.com>
In newsgroup: linux.dev.kernel
>
> > Martin, something which was not mentioned last week (I've just checked).
> > 
> > It's OK if we never write to disk unless explicitely told, but will we writeback
> > when we munmap?
> 
> Don't know for sure - you'd have to read the code (do_munmap) ... I couldn't
> see anything there at a quick glance. However, I'd guess we don't write it, 
> as multiple people could have the file mapped, or we could remap it
> again from somewhere. Presumably the standard LRU will just flush it out.
> 

munmap() and fsync() or msync() will flush it to disk; there is no
reason munmap() should unless perhaps the file was opened O_SYNC.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: Memory mapped files question

[Antonio Vargas]

On Mon, Apr 14, 2003 at 07:36:39AM -0700, Martin J. Bligh wrote:
> > Hi, everyone.  I'm running into a problem that I hope someone else has seen,
> > and maybe can help solve.  We're using the mmap system function for memory
> > mapped files, but our updates never get flushed until we munmap or msysnc.
> > Are we missing something?  Is there a tunable parameter in the kernel or
> > filing system that can be set to flag these updates as 'write required'?
> 
> This was discussed about a week ago on either linux-kernel or linux-mm.
> The short answer is "yes, that's deliberate", but an archive search would 
> probably be fruitful.
> 
> M.

Martin, something which was not mentioned last week (I've just checked).

It's OK if we never write to disk unless explicitely told, but will we writeback
when we munmap?

Greets, Antonio.

Re: Memory mapped files question

[Martin J. Bligh]

> Martin, something which was not mentioned last week (I've just checked).
> 
> It's OK if we never write to disk unless explicitely told, but will we writeback
> when we munmap?

Don't know for sure - you'd have to read the code (do_munmap) ... I couldn't
see anything there at a quick glance. However, I'd guess we don't write it, 
as multiple people could have the file mapped, or we could remap it
again from somewhere. Presumably the standard LRU will just flush it out.

M.

Re: Memory mapped files question

[Martin J. Bligh]

> Hi, everyone.  I'm running into a problem that I hope someone else has seen,
> and maybe can help solve.  We're using the mmap system function for memory
> mapped files, but our updates never get flushed until we munmap or msysnc.
> Are we missing something?  Is there a tunable parameter in the kernel or
> filing system that can be set to flag these updates as 'write required'?

This was discussed about a week ago on either linux-kernel or linux-mm.
The short answer is "yes, that's deliberate", but an archive search would 
probably be fruitful.

M.

Memory mapped files question

[Bryan Shumsky]

Hi, everyone.  I'm running into a problem that I hope someone else has seen,
and maybe can help solve.  We're using the mmap system function for memory
mapped files, but our updates never get flushed until we munmap or msysnc.
Are we missing something?  Is there a tunable parameter in the kernel or
filing system that can be set to flag these updates as 'write required'?

Any advice or suggestions would be appreciated.

Regards,

-- Bryan Shumsky
Director of Engineering
Via Systems, Inc.