Re: [Xenomai] [RFC] Xenomai3 tuneables replaced by environment variables

[Philippe Gerum <rpm@xenomai.org>]

On 08/25/2017 07:16 PM, Henning Schild wrote:
> Am Wed, 16 Aug 2017 20:13:34 +0200
> schrieb Philippe Gerum <rpm@xenomai.org>:
> 
>> On 08/08/2017 09:27 PM, Henning Schild wrote:
>>>   
>>>> Could you sketch the requirements of the app with respect to
>>>> option parsing (namespace left aside)? Is there some main
>>>> non-Xenomai executable dlopening a Xenomai-based solib defining a
>>>> set of rt-specific options?  
>>>
>>> Yes the application runs its own main() and later dlopen() s
>>> multiple libs, some of which may be linked to xenomai. The order of
>>> the dlopens seems hard to control because of static c++
>>> constructors or something alike. But i suggest we discuss the
>>> internals of that particular app after we have finished the
>>> discussion. 
>>
>> No need for this; I was not asking for details about the internals,
>> only for explicit requirements about receiving parameters / options
>> among the software components present in your app. Anyway, I think I
>> figured it out from what you've just said.
>>
>>>> It seems that you are implicitly referring to ancillaries settings
>>>> such as --silent, --verbose and --quiet as being problematic. Is it
>>>> correct or are you referring to other parameters?  
>>>
>>> If your main comes before xenomai_init() it either has to be able to
>>> ignore xenomai args or it will need to change argv/argc before
>>> dlopening.
>>>
>>> In the first case it has to know all valid args  
>>
>> ... the application accepts. Do some (non-Xenomai) dlopened libs
>> define their own option namespace the main component does not know
>> about? If not, this is only a matter of excluding/ignoring the
>> options which the application does not accept when parsing the
>> argument vector.
>>
>> xenomai_init() will do the same afterwards, ignoring the non-Xenomai
>> options transparently. The point being: who should call xenomai_init()
>> in the plugin.so file, and when.
>>
>>  and the second one
>>> does not sound very nice. And for the second one i am not sure
>>> there is always a time window in the face of static ctors launching
>>> threads and dlopening.  
>>
>> That is not an issue if Xenomai setup descriptors are used properly.
>> The "tuning" phase which may override parameters always precedes the
>> "init" phase which considers them when allocating and bootstrapping
>> the resources.
>>
>> Practically: say you have plugin.so, explicitly set dependent on
>> libcobalt.so at link time, you will nevertheless have all ->tune()
>> handlers run first, before the first ->init() handler is invoked. And
>> this will happen only when xenomai_init() is called; all despite the
>> fact that solibs are initialized in reverse dependency order (i.e.
>> from the least dependent lib to the most dependent one).
>>
>> So, the way that should work with your app would be to load a proxy
>> solib which was made dependent on libcobalt.so at build time. That
>> proxy shared lib would define a setup descriptor, whose ->tune()
>> handler could pull values from the environment for setting whatever
>> runtime/config Xenomai tunable that one sees fit. That process is
>> guaranteed to happen _before_ the Xenomai core is initialized. The
>> proxy solib would include a Xenomai bootstrap module, calling
>> xenomai_init().
>>
>> The entire process would then look like:
>>
>> - main() entered
>> -- application parses argv[], no Xenomai arg in the way
>> ...
>> -- dlopen(plugin.so)
>> --- libcobalt.so dependency is loaded, all setup descriptors
>> registered --- libplugin.so is loaded, setup descriptor registered
>> --- libplugin's bootstrap module calls xenomai_init_dso()
>> ---- plugin->tune() is called, pulls args from the environment
>> ---- Xenomai ->init() handlers run (using arg settings)
>> -- back to main()
>>
>> With all Xenomai base parameters exclusively pulled from the
>> environment, you could omit any Xenomai-specific option switch from
>> the command line when spawning the app, reserving the entire option
>> namespace to the application.
>>
>> I have demoed this in the attached code (the proxy lib is implemented
>> by plugin.c). The demo code only retrieves the verbosity_level
>> parameter from the environment, which is normally affected by
>> --silent, --quiet and --verbose command line options, but you get the
>> idea.
> 
> I only just really looked at the example, after i asked my customer to
> look at your response. And they kept telling me that it will not work.

This sounds like nobody tried it actually.

> 
> Now i gave them a similar example a few months ago. Put the tuning in
> the lib init and you are done ...
> 
> As far as i understand the situation:
> We are talking about more than one "plugin" that pulls in libcobalt.
> And the dlopens are not in main but in static ctors before main, so our
> plugin load order is "unknown/random". So the tuning would have to go
> into all plugins or we will need to find a way to have a defined "first
> plugin".
> 

This explanation is very confusing. Unlike dlmopen(), dlopen()ing
multiple times the same shared object (and likewise its dependencies)
only creates a single mapping at the initial opening, and only runs the
library constructors once at that moment. This would include the call to
xenomai_init() if the bootstrap module is present in the DSO.

You could dlopen() libplugin.so directly or indirectly via the
dependency list a hundred times, you would only get a single bootstrap
if the bootstrap module is glued to the DSO. If not, then calling
xenomai_init() multiple times from your app would simply lead to nop
after the first call (with Xenomai 3.0.5).

> I guess we can invent libtune.so and LD_PRELOAD it. Or put that as
> proxy into all plugins.  ... and all that vs "export XENO_foo"
> 

I guess that your customer may want to clearly explain what they want to
achieve by sketching their init process and the software components in
the picture, because at this point, I could not figure out what their
problem is. I'm not keen on continuing this discussion only based on
guesses.

On a general note, I believe that this discussion took the wrong track
by mentioning orthogonal and often unrelated issues such as constructor
priority, dlopen ordering, namespace conflicts. I do believe that these
are not the issues at stake here. The real issue is about understanding
how dynamic inits do work with Xenomai, how setup descriptors can help
introducing initialization order when there was none, and progress from
there.

Whether adding support for getting parameters from envvars should be
implemented is a separate issue. Doing so would not solve every problem
regarding dynamic loading and static C++ constructors, and people
dealing with that should not put away their tin foil hat, but this can
be done elegantly, and as I said earlier, I'm not opposed to merge such
feature.

>> I understand you would like a permanent way of doing this without
>> having to provide any code.
> 
> No, but i anticipated a discussion and decided to write code when the
> outcome of the discussion is clear.
> 
>> Although I don't buy a second the argument about envvars being "more
>> standard" than command switches,
> 
> Command switches always become a problem when two or more components
> have to share the namespace. Removing your traces and not bothering

And envvars do introduce a painful issue too: submitting parameters is
not explicit anymore, and one may end up running with broken settings
from a dusted set of conflicting login / setup scripts, inheriting the
mess all way through the process hierarchy.

I do acknowledge that proper use do exist too, but let's not sell
envvars the as perfect, simple and foolproof option: with complex
systems developed and configured by many people over a long period of
time, they are not. With simple applications with only a few
user-settable parameters, they are just inconvenient compared to plain
command switches.

Besides, a real-time Xenomai application acting as a server forking
other applications is not quite the common case. the app could also
achieve that export by running setenv() on any set of parameters
received on the command line it would want to pass to the forkees. Your
point is that you would like the Xenomai libs to consider envvars
directly, which is fair enough from your standpoint, but this is
definitely not most people's requirement and as such this will never
become a default.

> the other/imposing on the other only works if the order is known. In
> Xenomai the order depends on how people link

This affirmation seems wrong when considering the way setup descriptors
are dealt with internally, actually they have been designed precisely
not to depend on the link order, and also to allow unrelated components
to initialize in proper sequence.

So I can only assume that your remark on the link order is about static
dependency vs dlopen. In that case, I still fail to see what Xenomai
libs impose which any other random library does not, when it comes to
using dlopen().

Again, could you be specific and illustrate your concern with a simple
code example that would not work with the current Xenomai implementation?

> so we can not know it for
> the general case.
> > I would not want to be the poor guy writing main() for an app where
> multiple libs think that argv is for them to use, especially with
> tricks to access it before main(). Just the fact that xenomai is
> stealing this out of /proc/self/cmdline should suggest how non-standard
> that is.
> 

I can see the rant above, but I don't get the reasoning.

First, your original mail mentioned name clashes as a reason to drop
command line options or to prefix them with --xeno, followed by a
suggestion to drop setup descriptors in favor of a generalized use of
environment variables.

If your customer's application wants to exclusively use environment
variables for its own parameters, then there is no way such variables
could conflict with any existing Xenomai command line switch.
Conversely, there is nothing in Xenomai that will prevent any piece of
application code to read the environment.

So you must be referring to the general case instead, where "some"
application - i.e. not your customer's - with requirements to use
command line switches, would clash with existing ones defined by the
Xenomai libraries.

To answer this point, I will repeat that only a handful of Xenomai
command switches have generic names (e.g. --verbose, --quiet, --silent,
--help). The rest is either prefixed, or specific enough to prevent any
confusion regarding their meaning, leaving a lot of space for variants.

So maybe the time has come to be more specific about the namespace clash
issue too: which existing Xenomai option are your referring to, which
would create a conflict with your application namespace _today_?

> Env variables are just there to look at, if you are not interested
> nobody will tell you to look at argc more of them. They survive a
> fork() if your main script needs to launch multiple xenomai apps with
> the same tuning.
> 

Reading /proc/self/cmdline is exclusively part of the auto-bootstrap
feature of a Xenomai application, and the latter is NOT required. You
can disable it at will at build time by not including the bootstrap
module in the DSO or main executable, and receive the unfiltered set of
command line switches into main() for the latter.

The auto-bootstrap is there because most applications are getting
benefit from not having to care for calling xenomai_init() explicitly,
and are just fine with receiving only the command line switches they
want to process in their main() routine, expunged from all Xenomai
switches.

The only requirement in this case would be that at some point, the
application calls xenomai_init() explicitly, before calling into Xenomai
services. If your application knows when it should dlopen() libcobalt,
it therefore knows when to call xenomai_init() too.

You could argue that enabling auto-bootstrap should not be a
default setting of the build switches returned by xeno-config,
preferring manual init of the real-time services by default, but that
is pretty much all of the extent of the issue.

>> I'm not opposed to get this kind of feature in, but only if:
>>
>> - the implementation does not reinvent the wheel of parameter parsing,
>> this would be fairly silly.
>> - the natural way of setting Xenomai parameters remains based on
>> command line switches. Most people do NOT care a dime about deferred
>> binding to libcobalt, so there is no reason to make this use case the
>> preferred one.
> 
> If we can not agree to drop at least the tunables, introducing another
> way would be insane.
>

We can't agree because this is a misconception. Tunables are entirely
orthogonal to command line switches and environment variable.
Functionally speaking, you can't replace tunables with envvars.

Each setting of a Xenomai application - whether it is defined by the
real-time libs or the user code makes no difference - can get its init
value as follows, by increasing level of precedence:

level 1- core default, this is a sane value set by the base
implementation. Usually only appropriate for simple, small apps. For
instance, the default memory pool size is 1Mb, not much for an hungry
app, but enough for most simple apps.

level 2- tunable, this is a _factory setting_ aimed at the application
developer. Quite often, only the application developer really knows how
to determine the proper value of such setting because it depends on
intimate knowledge of the innards, like the appropriate amount of memory
a given user-defined configuration would require. Such a value should be
determined by the tuning handler of a setup descriptor. Providing a
factory setting via a tunable allows you to deliver a binary executable
that will always allocate a sane amount of resources, without requiring
the end user to figure out. Unlike the core default, this one is
dynamic, it can adapt to runtime configuration bits. Whether those bits
are obtained from command line switches, envvars, config file, pigeon
post or telepathy is not relevant. At all.

level 3- command line switches, which are aimed at the end user, for
overriding any previous setting.

My proposal to automatically turn each command line defined by setup
descriptors into an implicit envvar would insert a 4th level, between
the tunable and the command line switch.

For instance, we could ask xenomai_init() to assume that since
"--mem-pool-size" was defined (in libcopperplate's setup descriptor),
there might an envvar called "XENO_MEM_POOL_SIZE" with the exact same
semantics, pulling it if present.

>> The sanest way to provide such feature would be to implicitly turn
>> every option label defined by setup descriptors into a potential
>> envvar, e.g. --cpu-affinity would be unambiguously matched by a
>> variable named "cpu_affinity". The core would look for such a
>> variable in the environment if the command line switch is not found,
>> hence giving precedence to the latter implicitly.
> 
> Yes the command line switch should still win, if they really need to
> stay. And a prefix like XENO_ should be added.
> 
>> At least, this would limit the risk of having totally unrelated ways
>> of setting core parameters, with less braindamage side-effects on
>> people with unsuitable daily amount of caffeine for the job.
>>
>> There is a way to do this properly, but this cannot be high on my todo
>> list ATM, given the ongoing work I have on my plate. Anyway, issue
>> logged.
> 
> Would you be willing to review/consider a patch that replaces the
> tuneables with env vars?
> 
No, for the reason mentioned above. However I would consider a patch
that implicitly turns options obtained from registered setup descriptors
into envvars, with lower precedence than the former, so that command
switches can always override tunables and envvars.

This said, I doubt this would fix the issue with your customer's
application: as you described it, that one is not about command line vs
envvar, this is about dealing in an orderly manner with the uncertain
combination of lazy code binding and C++ runtime mechanisms.

-- 
Philippe.