RE: [External] - Re: Conflicting EVL Processing Loops

[Russell Johnson <russell.johnson@kratosdefense.com>]

[-- Attachment #1: Type: text/plain, Size: 2876 bytes --]

Philippe,

An update and question for you.

First, as you know, we found that the built-in heap management in Xenomai
4/EVL was causing us substantial performance problems, due to the need to
perform locking on all memory allocations. The nolock API is not an option
for us since we perform memory operations in most of our threads. We also
tried the TLSF manager, but saw similar performance issues.

The good news is that we've adapted the mimalloc memory management library,
which I believe implements something like the fast bins you mention in your
earlier email. The performance of mimalloc looks to be very good, and we are
able to get our dual processing loop system running within our real-time
constraints. The current implementation is still a bit "hackish", and we're
continuing to test it and clean it up. I am hoping to get you the specifics
about what we had to do to implement it, since I think it could be a good
option for other X4/EVL users. At a high level, it is essentially a
"go-between" with the Xenomai heap management at the bottom layer, replacing
the low-level sbrk() used to get memory in a Linux run-time environment.

One nagging problem is that we're still plagued by occasional page faults.
We have tried to prefault everything we can think of, but we're obviously
missing something. I go through each accessible section in the
/proc/self/maps file, prefaulting each one (this includes all code and data
segments). I have also added a hack to the kernel so that when the "switched
inband (fault)" occurs, the faulting address is displayed in dmesg. So far,
all of the runtime page faults we see are in the heap section, which I have
attempted to prefault completely, even doing it multiple times during
startup, since the heap section seems to be growing as we start up our real
time threads. 

So, I'm looking at one of 2 possibilities:
1. My prefaulting code, which touches one memory location in each page, is
not actually doing what it is supposed to. I've declared variables in the
prefaulting function to be "volatile" so that they don't get optimized out.
But I don't know any way to really verify that the pages are being mapped in
and locked.
2. A kernel bug, where the pages are not, in fact, being locked into memory.
We're calling "mlockall(MCL_CURRENT | MCL_FUTURE)", so, even if the heap is
growing, I don't understand why any future pages are not being populated and
locked into memory at the very beginning. And, the kernel should not be
unmapping any of our pages, but perhaps it is?

I know this isn't likely a problem with the EVL code, but we're just about
out of ideas for how to find and kill this problem. I'm not much of a kernel
expert. If you have any ideas for how to isolate this problem, especially if
there's a way to verify whether our process pages are really locked or not,
they would be greatly appreciated.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]