[Xenomai-help] Xenomai 2.5.3 process doubles its memory footprint when spawning a subprocess?

[Xenomai-help] Xenomai 2.5.3 process doubles its memory footprint when spawning a subprocess?

[Jeremy Friesner]

Hi all, 

I've got a program that runs under Xenomai 2.5.3 .... its process contains a Xenomai real-time thread, a few dozen non-real-time threads (used for disk I/O), and the main() thread which uses Linux's SCHED_RR at a high priority but is not, of course, a Xenomai real-time thread.

In general this all works quite well... there is one problem that I am seeing, though.  Occasionally the main() thread needs to spawn a sub-process to handle certain non-real-time chores (for example, it might spawn an rsync process to back up some files to a remote machine).  Depending on the feature being exercised, does this via the standard fork()/exec() calling sequence, or via forkpty(), or in some instances it just calls system().

The problem is that when the program has a lot of RAM allocated (e.g. a gigabyte or so) and/or a lot of I/O threads present, the spawning of the sub-process becomes very resource-intensive.  For example, in my testing I can do a system("/bin/sleep 1") and see the computer's RAM usage briefly shoot up from ~55% to ~95%, while the CPU usage spikes to 100% for a second or two, before the CPU usage and RAM usage both come back down again.  If there is a bit more RAM allocated by my program, this can even run the computer completely out of memory, causing my program and/or various other processes to report errors, crash, or exit.

I suspect that is has something to do with mlockall() and/or some other aspect of Xenomai partially or fully defeating the fork() call's Copy-On-Write feature, meaning that when the sub-process is spawned, most or all of the RAM pages allocated by the parent process are being duplicated, even though they are all going to be thrown away again immediately because I'm going to call exec() right afterwards.

My question is, is there some recommended way to launch a sub-process from a large Xenomai process that will avoid massive amounts of overhead?  I suppose I could launch a separate, persistent, non-Xenomai process that would act as a "non-Xenomai process server", and connect to it via TCP and ask it to spawn processes on my behalf.... but that seems like an awful lot of extra work.  Is there an easier way around this problem?

Thanks,
Jeremy

Re: [Xenomai-help] Xenomai 2.5.3 process doubles its memory footprint when spawning a subprocess?

[Gilles Chanteperdrix]

On 08/26/2011 10:54 PM, Jeremy Friesner wrote:
> I suspect that is has something to do with mlockall() and/or some
> other aspect of Xenomai partially or fully defeating the fork()
> call's Copy-On-Write feature, meaning that when the sub-process is
> spawned, most or all of the RAM pages allocated by the parent process
> are being duplicated, even though they are all going to be thrown
> away again immediately because I'm going to call exec() right
> afterwards.

The "copy-on-write" feature causes faults both in the parent and child
process, which causes threads to exit primary mode. So, in order to
avoid it, we duplicate the memory space when forking.

> 
> My question is, is there some recommended way to launch a sub-process
> from a large Xenomai process that will avoid massive amounts of
> overhead?  I suppose I could launch a separate, persistent,
> non-Xenomai process that would act as a "non-Xenomai process server",
> and connect to it via TCP and ask it to spawn processes on my
> behalf.... but that seems like an awful lot of extra work.  Is there
> an easier way around this problem?

If what you intend is to execute another thread, you do not need to use
fork(), you can use vfork() which will avoid the issue you are
mentioning. Normally, if you are using a recent libc, system() already
uses vfork().

It is also worth mentioning that the reason why your process consumes so
much memory in the first place may be because of the space allocated for
the threads stacks (2MB by default), because of mlockall. So, if you are
using the posix API, think about setting the stacks sizes to a more
reasonable size with pthread_attr_setstacksize.

-- 
                                                                Gilles.

Re: [Xenomai-help] Xenomai 2.5.3 process doubles its memory footprint when spawning a subprocess?

[Jeremy Friesner]

Hi Gilles,

Thanks, your suggestions were very helpful.  I reduced my threads' stack sizes as you suggested, and the RAM usage of my test dropped from 55% to 17%!  :^)  This made the overhead of the fork() call correspondingly smaller, so perhaps I can ignore that overhead for now.  

Jeremy

On Aug 26, 2011, at 2:04 PM, Gilles Chanteperdrix wrote:

> On 08/26/2011 10:54 PM, Jeremy Friesner wrote:
>> I suspect that is has something to do with mlockall() and/or some
>> other aspect of Xenomai partially or fully defeating the fork()
>> call's Copy-On-Write feature, meaning that when the sub-process is
>> spawned, most or all of the RAM pages allocated by the parent process
>> are being duplicated, even though they are all going to be thrown
>> away again immediately because I'm going to call exec() right
>> afterwards.
> 
> The "copy-on-write" feature causes faults both in the parent and child
> process, which causes threads to exit primary mode. So, in order to
> avoid it, we duplicate the memory space when forking.
> 
>> 
>> My question is, is there some recommended way to launch a sub-process
>> from a large Xenomai process that will avoid massive amounts of
>> overhead?  I suppose I could launch a separate, persistent,
>> non-Xenomai process that would act as a "non-Xenomai process server",
>> and connect to it via TCP and ask it to spawn processes on my
>> behalf.... but that seems like an awful lot of extra work.  Is there
>> an easier way around this problem?
> 
> If what you intend is to execute another thread, you do not need to use
> fork(), you can use vfork() which will avoid the issue you are
> mentioning. Normally, if you are using a recent libc, system() already
> uses vfork().
> 
> It is also worth mentioning that the reason why your process consumes so
> much memory in the first place may be because of the space allocated for
> the threads stacks (2MB by default), because of mlockall. So, if you are
> using the posix API, think about setting the stacks sizes to a more
> reasonable size with pthread_attr_setstacksize.
> 
> -- 
>                                                                Gilles.