[Xenomai] RTDM syscalls & switching

[Xenomai] RTDM syscalls & switching

[Jan Kiszka]

Gilles,

regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
current) - I remember a discussion on that topic, but I do not find its
traces any more. Do you have a pointer

In any case, I'm confronted with a use case for the old (Xenomai 2),
lazy switching behaviour: lightweight, performance sensitive IOCTL
services that can (and should) be called without any switching from both
domains.

What were the arguments in favour of migrating threads to real-time first?

I currently see the real need only for IOCTLs, but the question is then
if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
avoid unwanted migration costs (which are significantly higher than
syscall restarts).

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> Gilles,
> 
> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> current) - I remember a discussion on that topic, but I do not find its
> traces any more. Do you have a pointer
> 
> In any case, I'm confronted with a use case for the old (Xenomai 2),
> lazy switching behaviour: lightweight, performance sensitive IOCTL
> services that can (and should) be called without any switching from both
> domains.

Why not using a plain linux driver? ioctl_nrt callbacks are
redundant with plain linux drivers.

> 
> What were the arguments in favour of migrating threads to real-time first?
> 
> I currently see the real need only for IOCTLs, but the question is then
> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> avoid unwanted migration costs (which are significantly higher than
> syscall restarts).

I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
not remember merging this. However I find commit
13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
explain the reason pretty clear.

At the time of the discussion we had concluded that it was the way
to go. With __xn_exec_current you may enter the ioctl_rt callback
from secondary domain, which is counter-intuitive, error-prone, and
forces you to cripple driver code for checks for the current domain.
It optimizes the secondary mode case at the expense of the primary
mode case which is the inverse of what should be done. With
__xn_exec_conforming, you have the guarantee to enter ioctl_rt over
primary domain and ioctl_nrt over secondary domain.

See Philippe prose here also:
https://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Adaptive_syscalls

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>> Gilles,
>>
>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>> current) - I remember a discussion on that topic, but I do not find its
>> traces any more. Do you have a pointer
>>
>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>> services that can (and should) be called without any switching from both
>> domains.
> 
> Why not using a plain linux driver? ioctl_nrt callbacks are
> redundant with plain linux drivers.

Because that enforces the calling layer to either call the same service
via a plain Linux device if the calling thread is currently relaxed or
go for the RT device if the caller is in primary. Doable, but I would
really like to avoid this pain for the users.

> 
>>
>> What were the arguments in favour of migrating threads to real-time first?
>>
>> I currently see the real need only for IOCTLs, but the question is then
>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>> avoid unwanted migration costs (which are significantly higher than
>> syscall restarts).
> 
> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do

Xenomai 2 is still following the lazy scheme - we reverted that commit
later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.

> not remember merging this. However I find commit
> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> explain the reason pretty clear.
> 
> At the time of the discussion we had concluded that it was the way
> to go. With __xn_exec_current you may enter the ioctl_rt callback
> from secondary domain, which is counter-intuitive, error-prone, and
> forces you to cripple driver code for checks for the current domain.

Nope, normal drivers are not affected as they just implement those
services in the respective mode they want to support there and have a
simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
leaving out the implementation of the counterpart handler).

> It optimizes the secondary mode case at the expense of the primary
> mode case which is the inverse of what should be done. With
> __xn_exec_conforming, you have the guarantee to enter ioctl_rt over
> primary domain and ioctl_nrt over secondary domain.
> 
> See Philippe prose here also:
> https://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Adaptive_syscalls
> 

Yes, unfortunately I didn't have the current use case at hand by then to
argue against this change:

We do not always need real-time context in a driver, specifically to
perform certain IOCTLs. We do typically need a caller path that does not
migrate RT threads to Linux without a reason, though. As migrating is
far from free, migration ping-pong (we first migrate a conforming caller
and then find out that there is no RT version for an IOCTL service) or
needless switching for those services that can handle both domains has
to be avoided.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> > On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >> Gilles,
> >>
> >> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >> current) - I remember a discussion on that topic, but I do not find its
> >> traces any more. Do you have a pointer
> >>
> >> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >> services that can (and should) be called without any switching from both
> >> domains.
> > 
> > Why not using a plain linux driver? ioctl_nrt callbacks are
> > redundant with plain linux drivers.
> 
> Because that enforces the calling layer to either call the same service
> via a plain Linux device if the calling thread is currently relaxed or
> go for the RT device if the caller is in primary. Doable, but I would
> really like to avoid this pain for the users.
> 
> > 
> >>
> >> What were the arguments in favour of migrating threads to real-time first?
> >>
> >> I currently see the real need only for IOCTLs, but the question is then
> >> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >> avoid unwanted migration costs (which are significantly higher than
> >> syscall restarts).
> > 
> > I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> 
> Xenomai 2 is still following the lazy scheme - we reverted that commit
> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> 
> > not remember merging this. However I find commit
> > 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> > explain the reason pretty clear.
> > 
> > At the time of the discussion we had concluded that it was the way
> > to go. With __xn_exec_current you may enter the ioctl_rt callback
> > from secondary domain, which is counter-intuitive, error-prone, and
> > forces you to cripple driver code for checks for the current domain.
> 
> Nope, normal drivers are not affected as they just implement those
> services in the respective mode they want to support there and have a
> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> leaving out the implementation of the counterpart handler).

Yes, I got mixed up trying to remember. I think the crux of the
problem is that if a thread running in primary mode gets
(temporarily) switched to secondary mode by gdb, the ioctl_nrt
handler gets invoked, which is almost certainly the wrong thing to
do. You want the thread to migrate to primary mode to execute
ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
application in gdb causes the application to behave differently.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> > On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> > > On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> > >> Gilles,
> > >>
> > >> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> > >> current) - I remember a discussion on that topic, but I do not find its
> > >> traces any more. Do you have a pointer
> > >>
> > >> In any case, I'm confronted with a use case for the old (Xenomai 2),
> > >> lazy switching behaviour: lightweight, performance sensitive IOCTL
> > >> services that can (and should) be called without any switching from both
> > >> domains.
> > > 
> > > Why not using a plain linux driver? ioctl_nrt callbacks are
> > > redundant with plain linux drivers.
> > 
> > Because that enforces the calling layer to either call the same service
> > via a plain Linux device if the calling thread is currently relaxed or
> > go for the RT device if the caller is in primary. Doable, but I would
> > really like to avoid this pain for the users.
> > 
> > > 
> > >>
> > >> What were the arguments in favour of migrating threads to real-time first?
> > >>
> > >> I currently see the real need only for IOCTLs, but the question is then
> > >> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> > >> avoid unwanted migration costs (which are significantly higher than
> > >> syscall restarts).
> > > 
> > > I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> > 
> > Xenomai 2 is still following the lazy scheme - we reverted that commit
> > later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> > 
> > > not remember merging this. However I find commit
> > > 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> > > explain the reason pretty clear.
> > > 
> > > At the time of the discussion we had concluded that it was the way
> > > to go. With __xn_exec_current you may enter the ioctl_rt callback
> > > from secondary domain, which is counter-intuitive, error-prone, and
> > > forces you to cripple driver code for checks for the current domain.
> > 
> > Nope, normal drivers are not affected as they just implement those
> > services in the respective mode they want to support there and have a
> > simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> > leaving out the implementation of the counterpart handler).
> 
> Yes, I got mixed up trying to remember. I think the crux of the
> problem is that if a thread running in primary mode gets
> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> handler gets invoked, which is almost certainly the wrong thing to
> do. You want the thread to migrate to primary mode to execute
> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> application in gdb causes the application to behave differently.

And trying and avoiding this issue indeed cripple codes with checks
for rtdm_in_rt_context:
https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 18:59, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>> Gilles,
>>>>
>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>> current) - I remember a discussion on that topic, but I do not find its
>>>> traces any more. Do you have a pointer
>>>>
>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>> services that can (and should) be called without any switching from both
>>>> domains.
>>>
>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>> redundant with plain linux drivers.
>>
>> Because that enforces the calling layer to either call the same service
>> via a plain Linux device if the calling thread is currently relaxed or
>> go for the RT device if the caller is in primary. Doable, but I would
>> really like to avoid this pain for the users.
>>
>>>
>>>>
>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>
>>>> I currently see the real need only for IOCTLs, but the question is then
>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>> avoid unwanted migration costs (which are significantly higher than
>>>> syscall restarts).
>>>
>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>
>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>
>>> not remember merging this. However I find commit
>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>> explain the reason pretty clear.
>>>
>>> At the time of the discussion we had concluded that it was the way
>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>> from secondary domain, which is counter-intuitive, error-prone, and
>>> forces you to cripple driver code for checks for the current domain.
>>
>> Nope, normal drivers are not affected as they just implement those
>> services in the respective mode they want to support there and have a
>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>> leaving out the implementation of the counterpart handler).
> 
> Yes, I got mixed up trying to remember. I think the crux of the
> problem is that if a thread running in primary mode gets
> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> handler gets invoked, which is almost certainly the wrong thing to
> do. You want the thread to migrate to primary mode to execute
> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> application in gdb causes the application to behave differently.

Well, if that service is not handling nrt, the only difference under gdb
will be the extra switch. But gdb does these kind of things to
applications all time.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>> Gilles,
>>>>>
>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>> traces any more. Do you have a pointer
>>>>>
>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>> services that can (and should) be called without any switching from both
>>>>> domains.
>>>>
>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>> redundant with plain linux drivers.
>>>
>>> Because that enforces the calling layer to either call the same service
>>> via a plain Linux device if the calling thread is currently relaxed or
>>> go for the RT device if the caller is in primary. Doable, but I would
>>> really like to avoid this pain for the users.
>>>
>>>>
>>>>>
>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>
>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>> syscall restarts).
>>>>
>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>
>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>
>>>> not remember merging this. However I find commit
>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>> explain the reason pretty clear.
>>>>
>>>> At the time of the discussion we had concluded that it was the way
>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>> forces you to cripple driver code for checks for the current domain.
>>>
>>> Nope, normal drivers are not affected as they just implement those
>>> services in the respective mode they want to support there and have a
>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>> leaving out the implementation of the counterpart handler).
>>
>> Yes, I got mixed up trying to remember. I think the crux of the
>> problem is that if a thread running in primary mode gets
>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>> handler gets invoked, which is almost certainly the wrong thing to
>> do. You want the thread to migrate to primary mode to execute
>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>> application in gdb causes the application to behave differently.
> 
> And trying and avoiding this issue indeed cripple codes with checks
> for rtdm_in_rt_context:
> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
> 

I don't remember details here, but this is a special case: The driver
provides also read_nrt - is that really useful for Analogy?

In most cases, you are fine with not providing the nrt (or rt) handler,
or with a simple

default:
	return -ENOSYS;

in your ioctl dispatcher.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 05/12/2016 06:50 PM, Jan Kiszka wrote:
> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>> Gilles,
>>>
>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>> current) - I remember a discussion on that topic, but I do not find its
>>> traces any more. Do you have a pointer
>>>
>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>> services that can (and should) be called without any switching from both
>>> domains.
>>
>> Why not using a plain linux driver? ioctl_nrt callbacks are
>> redundant with plain linux drivers.
> 
> Because that enforces the calling layer to either call the same service
> via a plain Linux device if the calling thread is currently relaxed or
> go for the RT device if the caller is in primary. Doable, but I would
> really like to avoid this pain for the users.
> 
>>
>>>
>>> What were the arguments in favour of migrating threads to real-time first?
>>>
>>> I currently see the real need only for IOCTLs, but the question is then
>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>> avoid unwanted migration costs (which are significantly higher than
>>> syscall restarts).
>>
>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> 
> Xenomai 2 is still following the lazy scheme - we reverted that commit
> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> 
>> not remember merging this. However I find commit
>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>> explain the reason pretty clear.
>>
>> At the time of the discussion we had concluded that it was the way
>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>> from secondary domain, which is counter-intuitive, error-prone, and
>> forces you to cripple driver code for checks for the current domain.
> 
> Nope, normal drivers are not affected as they just implement those
> services in the respective mode they want to support there and have a
> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> leaving out the implementation of the counterpart handler).
> 
>> It optimizes the secondary mode case at the expense of the primary
>> mode case which is the inverse of what should be done. With
>> __xn_exec_conforming, you have the guarantee to enter ioctl_rt over
>> primary domain and ioctl_nrt over secondary domain.
>>
>> See Philippe prose here also:
>> https://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Adaptive_syscalls
>>
> 
> Yes, unfortunately I didn't have the current use case at hand by then to
> argue against this change:
> 
> We do not always need real-time context in a driver, specifically to
> perform certain IOCTLs. We do typically need a caller path that does not
> migrate RT threads to Linux without a reason, though. As migrating is
> far from free, migration ping-pong (we first migrate a conforming caller
> and then find out that there is no RT version for an IOCTL service) or
> needless switching for those services that can handle both domains has
> to be avoided.
> 

The logic of your driver is broken: it wants to optimize on the number
of migration a thread does. That is wrong:

- either a thread has real-time requirements, in which case all services
it calls during normal operations are best served from primary mode by
definition, which the current RTDM logic honors.

- or, a thread does not have such requirements (e.g. SCHED_WEAK), in
which case the preferred mode of operation is secondary mode, which the
current RTDM logic honors too.

If an application switches mode frequently enough to be adversely
affected by the migration cost, then it is certainly mixing real-time
and regular Linux code within a thread's inner loop, which is not a RTDM
or Cobalt issue.

Cobalt is not migrating the threads without a reason, it is making sure
that your application knows a priori which context a call will be
handled from, without asking you to track the current mode, crippling
the driver code with in_nrt() or forced mode switches from the
application, like Gilles mentioned.

>From that point of view, __xn_exec_current has always been a bad idea,
and has no more in-tree users nowadays for that reason.

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 19:38, Philippe Gerum wrote:
> On 05/12/2016 06:50 PM, Jan Kiszka wrote:
>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>> Gilles,
>>>>
>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>> current) - I remember a discussion on that topic, but I do not find its
>>>> traces any more. Do you have a pointer
>>>>
>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>> services that can (and should) be called without any switching from both
>>>> domains.
>>>
>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>> redundant with plain linux drivers.
>>
>> Because that enforces the calling layer to either call the same service
>> via a plain Linux device if the calling thread is currently relaxed or
>> go for the RT device if the caller is in primary. Doable, but I would
>> really like to avoid this pain for the users.
>>
>>>
>>>>
>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>
>>>> I currently see the real need only for IOCTLs, but the question is then
>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>> avoid unwanted migration costs (which are significantly higher than
>>>> syscall restarts).
>>>
>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>
>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>
>>> not remember merging this. However I find commit
>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>> explain the reason pretty clear.
>>>
>>> At the time of the discussion we had concluded that it was the way
>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>> from secondary domain, which is counter-intuitive, error-prone, and
>>> forces you to cripple driver code for checks for the current domain.
>>
>> Nope, normal drivers are not affected as they just implement those
>> services in the respective mode they want to support there and have a
>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>> leaving out the implementation of the counterpart handler).
>>
>>> It optimizes the secondary mode case at the expense of the primary
>>> mode case which is the inverse of what should be done. With
>>> __xn_exec_conforming, you have the guarantee to enter ioctl_rt over
>>> primary domain and ioctl_nrt over secondary domain.
>>>
>>> See Philippe prose here also:
>>> https://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Adaptive_syscalls
>>>
>>
>> Yes, unfortunately I didn't have the current use case at hand by then to
>> argue against this change:
>>
>> We do not always need real-time context in a driver, specifically to
>> perform certain IOCTLs. We do typically need a caller path that does not
>> migrate RT threads to Linux without a reason, though. As migrating is
>> far from free, migration ping-pong (we first migrate a conforming caller
>> and then find out that there is no RT version for an IOCTL service) or
>> needless switching for those services that can handle both domains has
>> to be avoided.
>>
> 
> The logic of your driver is broken: it wants to optimize on the number
> of migration a thread does. That is wrong:
> 
> - either a thread has real-time requirements, in which case all services
> it calls during normal operations are best served from primary mode by
> definition, which the current RTDM logic honors.
> 
> - or, a thread does not have such requirements (e.g. SCHED_WEAK), in
> which case the preferred mode of operation is secondary mode, which the
> current RTDM logic honors too.

Where is SCHED_WEAK honoured in the syscall handling? To my
understanding, the only thing that matters for __xn_exec_conforming is
if the caller has a shadow or not.

> 
> If an application switches mode frequently enough to be adversely
> affected by the migration cost, then it is certainly mixing real-time
> and regular Linux code within a thread's inner loop, which is not a RTDM
> or Cobalt issue.
> 
> Cobalt is not migrating the threads without a reason, it is making sure
> that your application knows a priori which context a call will be
> handled from, without asking you to track the current mode, crippling
> the driver code with in_nrt() or forced mode switches from the
> application, like Gilles mentioned.

As pointed out, normal drivers (the majority) were and will never be
affected by the laziness or eagerness of the syscall entry path - they
need to handle both cases independently of the caller's preference any
way. The cited Analogy case is an exception.

Really, the current switching model has too many disadvantages. Just
think what happens to a shadowed thread, currently in Linux mode to do
some non-RT stuff, now issuing a read() or ioctl() on a normal Linux
device: it will first be migrated to RT, because RTDM may handle the
file descriptor, but it doesn't, and so it will return to Linux when
calling the regular syscall afterwards. Such inefficiencies didn't exist
with the original switching scheme.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
> > On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
> >> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> >>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> >>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >>>>> Gilles,
> >>>>>
> >>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >>>>> current) - I remember a discussion on that topic, but I do not find its
> >>>>> traces any more. Do you have a pointer
> >>>>>
> >>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >>>>> services that can (and should) be called without any switching from both
> >>>>> domains.
> >>>>
> >>>> Why not using a plain linux driver? ioctl_nrt callbacks are
> >>>> redundant with plain linux drivers.
> >>>
> >>> Because that enforces the calling layer to either call the same service
> >>> via a plain Linux device if the calling thread is currently relaxed or
> >>> go for the RT device if the caller is in primary. Doable, but I would
> >>> really like to avoid this pain for the users.
> >>>
> >>>>
> >>>>>
> >>>>> What were the arguments in favour of migrating threads to real-time first?
> >>>>>
> >>>>> I currently see the real need only for IOCTLs, but the question is then
> >>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >>>>> avoid unwanted migration costs (which are significantly higher than
> >>>>> syscall restarts).
> >>>>
> >>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> >>>
> >>> Xenomai 2 is still following the lazy scheme - we reverted that commit
> >>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> >>>
> >>>> not remember merging this. However I find commit
> >>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> >>>> explain the reason pretty clear.
> >>>>
> >>>> At the time of the discussion we had concluded that it was the way
> >>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
> >>>> from secondary domain, which is counter-intuitive, error-prone, and
> >>>> forces you to cripple driver code for checks for the current domain.
> >>>
> >>> Nope, normal drivers are not affected as they just implement those
> >>> services in the respective mode they want to support there and have a
> >>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> >>> leaving out the implementation of the counterpart handler).
> >>
> >> Yes, I got mixed up trying to remember. I think the crux of the
> >> problem is that if a thread running in primary mode gets
> >> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> >> handler gets invoked, which is almost certainly the wrong thing to
> >> do. You want the thread to migrate to primary mode to execute
> >> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> >> application in gdb causes the application to behave differently.
> > 
> > And trying and avoiding this issue indeed cripple codes with checks
> > for rtdm_in_rt_context:
> > https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
> > 
> 
> I don't remember details here, but this is a special case: The driver
> provides also read_nrt - is that really useful for Analogy?
> 
> In most cases, you are fine with not providing the nrt (or rt) handler,
> or with a simple
> 
> default:
> 	return -ENOSYS;
> 
> in your ioctl dispatcher.

You are missing the point: if you enter read_nrt, there are two
cases:
- either the thread is real-time capable and has been relaxed by gdb
and you want to switch to read_rt for the reasons I already
explained, in that case, you must return -ENOSYS;
- or the thread is not real-time capable and the nrt handler
applies.

So, you need at least

read_nrt()
{
	if (rt_capable)
	     return -ENOSYS;

	/* Do the normal case here */
}

Having that code in the driver is plain wrong: this logic should be
handled by the core, not by every driver. The case you describe is a
corner case, this case is the normal case, so it makes sense that
the normal case does not require such cruft, and that the corner
case suffers from imperfections.

In the case of analogy it so happens that the normal case can be
handled by the same code as the rt case. So you need

read()
{
	if (!in_rt_context() /* test whether entering over nrt */
	       && rt_capable()) /* yes, over nrt, have we been
	       	  			       relaxed by gdb ? */
	       return -ENOSYS;

	/* Do the normal case here */
}


-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 07:51:20PM +0200, Jan Kiszka wrote:
> On 2016-05-12 19:38, Philippe Gerum wrote:
> > On 05/12/2016 06:50 PM, Jan Kiszka wrote:
> >> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> >>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >>>> Gilles,
> >>>>
> >>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >>>> current) - I remember a discussion on that topic, but I do not find its
> >>>> traces any more. Do you have a pointer
> >>>>
> >>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >>>> services that can (and should) be called without any switching from both
> >>>> domains.
> >>>
> >>> Why not using a plain linux driver? ioctl_nrt callbacks are
> >>> redundant with plain linux drivers.
> >>
> >> Because that enforces the calling layer to either call the same service
> >> via a plain Linux device if the calling thread is currently relaxed or
> >> go for the RT device if the caller is in primary. Doable, but I would
> >> really like to avoid this pain for the users.
> >>
> >>>
> >>>>
> >>>> What were the arguments in favour of migrating threads to real-time first?
> >>>>
> >>>> I currently see the real need only for IOCTLs, but the question is then
> >>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >>>> avoid unwanted migration costs (which are significantly higher than
> >>>> syscall restarts).
> >>>
> >>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> >>
> >> Xenomai 2 is still following the lazy scheme - we reverted that commit
> >> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> >>
> >>> not remember merging this. However I find commit
> >>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> >>> explain the reason pretty clear.
> >>>
> >>> At the time of the discussion we had concluded that it was the way
> >>> to go. With __xn_exec_current you may enter the ioctl_rt callback
> >>> from secondary domain, which is counter-intuitive, error-prone, and
> >>> forces you to cripple driver code for checks for the current domain.
> >>
> >> Nope, normal drivers are not affected as they just implement those
> >> services in the respective mode they want to support there and have a
> >> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> >> leaving out the implementation of the counterpart handler).
> >>
> >>> It optimizes the secondary mode case at the expense of the primary
> >>> mode case which is the inverse of what should be done. With
> >>> __xn_exec_conforming, you have the guarantee to enter ioctl_rt over
> >>> primary domain and ioctl_nrt over secondary domain.
> >>>
> >>> See Philippe prose here also:
> >>> https://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Adaptive_syscalls
> >>>
> >>
> >> Yes, unfortunately I didn't have the current use case at hand by then to
> >> argue against this change:
> >>
> >> We do not always need real-time context in a driver, specifically to
> >> perform certain IOCTLs. We do typically need a caller path that does not
> >> migrate RT threads to Linux without a reason, though. As migrating is
> >> far from free, migration ping-pong (we first migrate a conforming caller
> >> and then find out that there is no RT version for an IOCTL service) or
> >> needless switching for those services that can handle both domains has
> >> to be avoided.
> >>
> > 
> > The logic of your driver is broken: it wants to optimize on the number
> > of migration a thread does. That is wrong:
> > 
> > - either a thread has real-time requirements, in which case all services
> > it calls during normal operations are best served from primary mode by
> > definition, which the current RTDM logic honors.
> > 
> > - or, a thread does not have such requirements (e.g. SCHED_WEAK), in
> > which case the preferred mode of operation is secondary mode, which the
> > current RTDM logic honors too.
> 
> Where is SCHED_WEAK honoured in the syscall handling? To my
> understanding, the only thing that matters for __xn_exec_conforming is
> if the caller has a shadow or not.
> 
> > 
> > If an application switches mode frequently enough to be adversely
> > affected by the migration cost, then it is certainly mixing real-time
> > and regular Linux code within a thread's inner loop, which is not a RTDM
> > or Cobalt issue.
> > 
> > Cobalt is not migrating the threads without a reason, it is making sure
> > that your application knows a priori which context a call will be
> > handled from, without asking you to track the current mode, crippling
> > the driver code with in_nrt() or forced mode switches from the
> > application, like Gilles mentioned.
> 
> As pointed out, normal drivers (the majority) were and will never be
> affected by the laziness or eagerness of the syscall entry path - they
> need to handle both cases independently of the caller's preference any
> way. The cited Analogy case is an exception.

They are. See previous post.

I wonder however, since RTDM drivers are now also Linux driver,
could not the Linux driver branch to the _nrt handlers ?

This way you could use __real_ioctl to indicate that you know that
the thread is running in secondary mode and that you do not want it
to switch to primary mode even though it has a shadow.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>> Gilles,
>>>>>>>
>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>> traces any more. Do you have a pointer
>>>>>>>
>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>> services that can (and should) be called without any switching from both
>>>>>>> domains.
>>>>>>
>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>> redundant with plain linux drivers.
>>>>>
>>>>> Because that enforces the calling layer to either call the same service
>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>> really like to avoid this pain for the users.
>>>>>
>>>>>>
>>>>>>>
>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>
>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>> syscall restarts).
>>>>>>
>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>
>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>
>>>>>> not remember merging this. However I find commit
>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>> explain the reason pretty clear.
>>>>>>
>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>
>>>>> Nope, normal drivers are not affected as they just implement those
>>>>> services in the respective mode they want to support there and have a
>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>> leaving out the implementation of the counterpart handler).
>>>>
>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>> problem is that if a thread running in primary mode gets
>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>> do. You want the thread to migrate to primary mode to execute
>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>> application in gdb causes the application to behave differently.
>>>
>>> And trying and avoiding this issue indeed cripple codes with checks
>>> for rtdm_in_rt_context:
>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>
>>
>> I don't remember details here, but this is a special case: The driver
>> provides also read_nrt - is that really useful for Analogy?
>>
>> In most cases, you are fine with not providing the nrt (or rt) handler,
>> or with a simple
>>
>> default:
>> 	return -ENOSYS;
>>
>> in your ioctl dispatcher.
> 
> You are missing the point: if you enter read_nrt, there are two
> cases:
> - either the thread is real-time capable and has been relaxed by gdb
> and you want to switch to read_rt for the reasons I already
> explained, in that case, you must return -ENOSYS;
> - or the thread is not real-time capable and the nrt handler
> applies.
> 
> So, you need at least
> 
> read_nrt()
> {
> 	if (rt_capable)
> 	     return -ENOSYS;
> 
> 	/* Do the normal case here */
> }

Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
understand why, though). And having some special code in the exceptional
case is probably better then the side effects we get from eagerly
switching now.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 08:24:54PM +0200, Jan Kiszka wrote:
> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
> > On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
> >> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
> >>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
> >>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> >>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> >>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >>>>>>> Gilles,
> >>>>>>>
> >>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >>>>>>> current) - I remember a discussion on that topic, but I do not find its
> >>>>>>> traces any more. Do you have a pointer
> >>>>>>>
> >>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >>>>>>> services that can (and should) be called without any switching from both
> >>>>>>> domains.
> >>>>>>
> >>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
> >>>>>> redundant with plain linux drivers.
> >>>>>
> >>>>> Because that enforces the calling layer to either call the same service
> >>>>> via a plain Linux device if the calling thread is currently relaxed or
> >>>>> go for the RT device if the caller is in primary. Doable, but I would
> >>>>> really like to avoid this pain for the users.
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> What were the arguments in favour of migrating threads to real-time first?
> >>>>>>>
> >>>>>>> I currently see the real need only for IOCTLs, but the question is then
> >>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >>>>>>> avoid unwanted migration costs (which are significantly higher than
> >>>>>>> syscall restarts).
> >>>>>>
> >>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> >>>>>
> >>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
> >>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> >>>>>
> >>>>>> not remember merging this. However I find commit
> >>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> >>>>>> explain the reason pretty clear.
> >>>>>>
> >>>>>> At the time of the discussion we had concluded that it was the way
> >>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
> >>>>>> from secondary domain, which is counter-intuitive, error-prone, and
> >>>>>> forces you to cripple driver code for checks for the current domain.
> >>>>>
> >>>>> Nope, normal drivers are not affected as they just implement those
> >>>>> services in the respective mode they want to support there and have a
> >>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> >>>>> leaving out the implementation of the counterpart handler).
> >>>>
> >>>> Yes, I got mixed up trying to remember. I think the crux of the
> >>>> problem is that if a thread running in primary mode gets
> >>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> >>>> handler gets invoked, which is almost certainly the wrong thing to
> >>>> do. You want the thread to migrate to primary mode to execute
> >>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> >>>> application in gdb causes the application to behave differently.
> >>>
> >>> And trying and avoiding this issue indeed cripple codes with checks
> >>> for rtdm_in_rt_context:
> >>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
> >>>
> >>
> >> I don't remember details here, but this is a special case: The driver
> >> provides also read_nrt - is that really useful for Analogy?
> >>
> >> In most cases, you are fine with not providing the nrt (or rt) handler,
> >> or with a simple
> >>
> >> default:
> >> 	return -ENOSYS;
> >>
> >> in your ioctl dispatcher.
> > 
> > You are missing the point: if you enter read_nrt, there are two
> > cases:
> > - either the thread is real-time capable and has been relaxed by gdb
> > and you want to switch to read_rt for the reasons I already
> > explained, in that case, you must return -ENOSYS;
> > - or the thread is not real-time capable and the nrt handler
> > applies.
> > 
> > So, you need at least
> > 
> > read_nrt()
> > {
> > 	if (rt_capable)
> > 	     return -ENOSYS;
> > 
> > 	/* Do the normal case here */
> > }
> 
> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
> understand why, though). And having some special code in the exceptional
> case is probably better then the side effects we get from eagerly
> switching now.

The normal case is for a driver to have _rt handlers and for the
core code to switch as fast as possible to primary mode to call
them. With your awkward solution, that normal case gets slowed down
in order to be able to handle a corner case.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 20:22, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 07:51:20PM +0200, Jan Kiszka wrote:
>> On 2016-05-12 19:38, Philippe Gerum wrote:
>>> On 05/12/2016 06:50 PM, Jan Kiszka wrote:
>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>> Gilles,
>>>>>>
>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>> traces any more. Do you have a pointer
>>>>>>
>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>> services that can (and should) be called without any switching from both
>>>>>> domains.
>>>>>
>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>> redundant with plain linux drivers.
>>>>
>>>> Because that enforces the calling layer to either call the same service
>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>> really like to avoid this pain for the users.
>>>>
>>>>>
>>>>>>
>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>
>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>> syscall restarts).
>>>>>
>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>
>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>
>>>>> not remember merging this. However I find commit
>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>> explain the reason pretty clear.
>>>>>
>>>>> At the time of the discussion we had concluded that it was the way
>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>> forces you to cripple driver code for checks for the current domain.
>>>>
>>>> Nope, normal drivers are not affected as they just implement those
>>>> services in the respective mode they want to support there and have a
>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>> leaving out the implementation of the counterpart handler).
>>>>
>>>>> It optimizes the secondary mode case at the expense of the primary
>>>>> mode case which is the inverse of what should be done. With
>>>>> __xn_exec_conforming, you have the guarantee to enter ioctl_rt over
>>>>> primary domain and ioctl_nrt over secondary domain.
>>>>>
>>>>> See Philippe prose here also:
>>>>> https://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Adaptive_syscalls
>>>>>
>>>>
>>>> Yes, unfortunately I didn't have the current use case at hand by then to
>>>> argue against this change:
>>>>
>>>> We do not always need real-time context in a driver, specifically to
>>>> perform certain IOCTLs. We do typically need a caller path that does not
>>>> migrate RT threads to Linux without a reason, though. As migrating is
>>>> far from free, migration ping-pong (we first migrate a conforming caller
>>>> and then find out that there is no RT version for an IOCTL service) or
>>>> needless switching for those services that can handle both domains has
>>>> to be avoided.
>>>>
>>>
>>> The logic of your driver is broken: it wants to optimize on the number
>>> of migration a thread does. That is wrong:
>>>
>>> - either a thread has real-time requirements, in which case all services
>>> it calls during normal operations are best served from primary mode by
>>> definition, which the current RTDM logic honors.
>>>
>>> - or, a thread does not have such requirements (e.g. SCHED_WEAK), in
>>> which case the preferred mode of operation is secondary mode, which the
>>> current RTDM logic honors too.
>>
>> Where is SCHED_WEAK honoured in the syscall handling? To my
>> understanding, the only thing that matters for __xn_exec_conforming is
>> if the caller has a shadow or not.
>>
>>>
>>> If an application switches mode frequently enough to be adversely
>>> affected by the migration cost, then it is certainly mixing real-time
>>> and regular Linux code within a thread's inner loop, which is not a RTDM
>>> or Cobalt issue.
>>>
>>> Cobalt is not migrating the threads without a reason, it is making sure
>>> that your application knows a priori which context a call will be
>>> handled from, without asking you to track the current mode, crippling
>>> the driver code with in_nrt() or forced mode switches from the
>>> application, like Gilles mentioned.
>>
>> As pointed out, normal drivers (the majority) were and will never be
>> affected by the laziness or eagerness of the syscall entry path - they
>> need to handle both cases independently of the caller's preference any
>> way. The cited Analogy case is an exception.
> 
> They are. See previous post.
> 
> I wonder however, since RTDM drivers are now also Linux driver,
> could not the Linux driver branch to the _nrt handlers ?

We could if there is a value.

> 
> This way you could use __real_ioctl to indicate that you know that
> the thread is running in secondary mode and that you do not want it
> to switch to primary mode even though it has a shadow.

If you have a layer (device abstraction library etc.) in your
application that does not know if a caller is an RT thread or rather
runs in Linux and should go for the __real version, this won't work. You
can write detection code, yes, but I really don't see why userspace
should do this as long as we can solve the problem much simpler via the
switching strategy.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 20:30, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 08:24:54PM +0200, Jan Kiszka wrote:
>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>> Gilles,
>>>>>>>>>
>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>
>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>> domains.
>>>>>>>>
>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>> redundant with plain linux drivers.
>>>>>>>
>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>> really like to avoid this pain for the users.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>
>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>> syscall restarts).
>>>>>>>>
>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>
>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>
>>>>>>>> not remember merging this. However I find commit
>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>> explain the reason pretty clear.
>>>>>>>>
>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>
>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>> services in the respective mode they want to support there and have a
>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>
>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>> problem is that if a thread running in primary mode gets
>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>> application in gdb causes the application to behave differently.
>>>>>
>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>> for rtdm_in_rt_context:
>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>
>>>>
>>>> I don't remember details here, but this is a special case: The driver
>>>> provides also read_nrt - is that really useful for Analogy?
>>>>
>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>> or with a simple
>>>>
>>>> default:
>>>> 	return -ENOSYS;
>>>>
>>>> in your ioctl dispatcher.
>>>
>>> You are missing the point: if you enter read_nrt, there are two
>>> cases:
>>> - either the thread is real-time capable and has been relaxed by gdb
>>> and you want to switch to read_rt for the reasons I already
>>> explained, in that case, you must return -ENOSYS;
>>> - or the thread is not real-time capable and the nrt handler
>>> applies.
>>>
>>> So, you need at least
>>>
>>> read_nrt()
>>> {
>>> 	if (rt_capable)
>>> 	     return -ENOSYS;
>>>
>>> 	/* Do the normal case here */
>>> }
>>
>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>> understand why, though). And having some special code in the exceptional
>> case is probably better then the side effects we get from eagerly
>> switching now.
> 
> The normal case is for a driver to have _rt handlers and for the
> core code to switch as fast as possible to primary mode to call
> them. With your awkward solution, that normal case gets slowed down
> in order to be able to handle a corner case.

The normal case is surely not calling from non-RT into an RT service
under timing constraints. And even then, the costs for the additional
dispatching attempt to nrt is negligibly compared to the
migration-related context switches.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 05/12/2016 08:24 PM, Jan Kiszka wrote:
> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>> Gilles,
>>>>>>>>
>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>
>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>> domains.
>>>>>>>
>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>> redundant with plain linux drivers.
>>>>>>
>>>>>> Because that enforces the calling layer to either call the same service
>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>> really like to avoid this pain for the users.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>
>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>> syscall restarts).
>>>>>>>
>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>
>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>
>>>>>>> not remember merging this. However I find commit
>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>> explain the reason pretty clear.
>>>>>>>
>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>
>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>> services in the respective mode they want to support there and have a
>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>> leaving out the implementation of the counterpart handler).
>>>>>
>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>> problem is that if a thread running in primary mode gets
>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>> do. You want the thread to migrate to primary mode to execute
>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>> application in gdb causes the application to behave differently.
>>>>
>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>> for rtdm_in_rt_context:
>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>
>>>
>>> I don't remember details here, but this is a special case: The driver
>>> provides also read_nrt - is that really useful for Analogy?
>>>
>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>> or with a simple
>>>
>>> default:
>>> 	return -ENOSYS;
>>>
>>> in your ioctl dispatcher.
>>
>> You are missing the point: if you enter read_nrt, there are two
>> cases:
>> - either the thread is real-time capable and has been relaxed by gdb
>> and you want to switch to read_rt for the reasons I already
>> explained, in that case, you must return -ENOSYS;
>> - or the thread is not real-time capable and the nrt handler
>> applies.
>>
>> So, you need at least
>>
>> read_nrt()
>> {
>> 	if (rt_capable)
>> 	     return -ENOSYS;
>>
>> 	/* Do the normal case here */
>> }
> 
> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
> understand why, though). And having some special code in the exceptional
> case is probably better then the side effects we get from eagerly
> switching now.
> 

Sorry, that is exactly the opposite: your use case is exceptional and I
believe is wrong. The normal use case is the one that does not ask the
user to track the current mode for knowing what any random driver would
eventually do depending on the calling context.

I'm not opposed to finding a way to cover your use case too if possible,
but certainly not at the expense of reintroducing a broken and
error-prone logic in the core for everyone.

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 20:35, Philippe Gerum wrote:
> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>> Gilles,
>>>>>>>>>
>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>
>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>> domains.
>>>>>>>>
>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>> redundant with plain linux drivers.
>>>>>>>
>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>> really like to avoid this pain for the users.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>
>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>> syscall restarts).
>>>>>>>>
>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>
>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>
>>>>>>>> not remember merging this. However I find commit
>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>> explain the reason pretty clear.
>>>>>>>>
>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>
>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>> services in the respective mode they want to support there and have a
>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>
>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>> problem is that if a thread running in primary mode gets
>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>> application in gdb causes the application to behave differently.
>>>>>
>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>> for rtdm_in_rt_context:
>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>
>>>>
>>>> I don't remember details here, but this is a special case: The driver
>>>> provides also read_nrt - is that really useful for Analogy?
>>>>
>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>> or with a simple
>>>>
>>>> default:
>>>> 	return -ENOSYS;
>>>>
>>>> in your ioctl dispatcher.
>>>
>>> You are missing the point: if you enter read_nrt, there are two
>>> cases:
>>> - either the thread is real-time capable and has been relaxed by gdb
>>> and you want to switch to read_rt for the reasons I already
>>> explained, in that case, you must return -ENOSYS;
>>> - or the thread is not real-time capable and the nrt handler
>>> applies.
>>>
>>> So, you need at least
>>>
>>> read_nrt()
>>> {
>>> 	if (rt_capable)
>>> 	     return -ENOSYS;
>>>
>>> 	/* Do the normal case here */
>>> }
>>
>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>> understand why, though). And having some special code in the exceptional
>> case is probably better then the side effects we get from eagerly
>> switching now.
>>
> 
> Sorry, that is exactly the opposite: your use case is exceptional and I
> believe is wrong. The normal use case is the one that does not ask the
> user to track the current mode for knowing what any random driver would
> eventually do depending on the calling context.

You still miss the point that this is not required in 99% of the cases.
There is no such problem. There only Analogy.

Every driver must ensure that a service is only exposed to users in the
right mode. That is a functional requirement, and drivers that fail to
do so only work by chance (thus with the restricted workload they are
tested against). If that is fulfilled, it doesn't matter to the driver
when the switch happens. It's pure optimization.

> 
> I'm not opposed to finding a way to cover your use case too if possible,
> but certainly not at the expense of reintroducing a broken and
> error-prone logic in the core for everyone.

Then explain to me how to fix the needless migration overhead when
calling regular Linux drivers - There are multiple issues and
limitations with conforming.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 05/12/2016 08:42 PM, Jan Kiszka wrote:
> On 2016-05-12 20:35, Philippe Gerum wrote:
>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>> Gilles,
>>>>>>>>>>
>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>
>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>> domains.
>>>>>>>>>
>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>
>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>
>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>> syscall restarts).
>>>>>>>>>
>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>
>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>
>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>
>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>
>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>
>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>> application in gdb causes the application to behave differently.
>>>>>>
>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>> for rtdm_in_rt_context:
>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>
>>>>>
>>>>> I don't remember details here, but this is a special case: The driver
>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>
>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>> or with a simple
>>>>>
>>>>> default:
>>>>> 	return -ENOSYS;
>>>>>
>>>>> in your ioctl dispatcher.
>>>>
>>>> You are missing the point: if you enter read_nrt, there are two
>>>> cases:
>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>> and you want to switch to read_rt for the reasons I already
>>>> explained, in that case, you must return -ENOSYS;
>>>> - or the thread is not real-time capable and the nrt handler
>>>> applies.
>>>>
>>>> So, you need at least
>>>>
>>>> read_nrt()
>>>> {
>>>> 	if (rt_capable)
>>>> 	     return -ENOSYS;
>>>>
>>>> 	/* Do the normal case here */
>>>> }
>>>
>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>> understand why, though). And having some special code in the exceptional
>>> case is probably better then the side effects we get from eagerly
>>> switching now.
>>>
>>
>> Sorry, that is exactly the opposite: your use case is exceptional and I
>> believe is wrong. The normal use case is the one that does not ask the
>> user to track the current mode for knowing what any random driver would
>> eventually do depending on the calling context.
> 
> You still miss the point that this is not required in 99% of the cases.
> There is no such problem. There only Analogy.
>

I'm not discussing Analogy at all, those drivers are still biased by the
legacy 2.x logic for dealing with modes and need fixing. I have never
been convinced by the reasoning behind rtdm_in_rt_context(), which
perfectly illustrates why messing with the call mode is not the
application's business.

> Every driver must ensure that a service is only exposed to users in the
> right mode. That is a functional requirement, and drivers that fail to
> do so only work by chance (thus with the restricted workload they are
> tested against). If that is fulfilled, it doesn't matter to the driver
> when the switch happens. It's pure optimization.
> 

You don't seem to get my point either. Let's proceed differently, please
sketch the application code that would require __xn_exec_current for
RTDM calls.

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 08:24:54PM +0200, Jan Kiszka wrote:
> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
> > On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
> >> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
> >>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
> >>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> >>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> >>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >>>>>>> Gilles,
> >>>>>>>
> >>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >>>>>>> current) - I remember a discussion on that topic, but I do not find its
> >>>>>>> traces any more. Do you have a pointer
> >>>>>>>
> >>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >>>>>>> services that can (and should) be called without any switching from both
> >>>>>>> domains.
> >>>>>>
> >>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
> >>>>>> redundant with plain linux drivers.
> >>>>>
> >>>>> Because that enforces the calling layer to either call the same service
> >>>>> via a plain Linux device if the calling thread is currently relaxed or
> >>>>> go for the RT device if the caller is in primary. Doable, but I would
> >>>>> really like to avoid this pain for the users.
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> What were the arguments in favour of migrating threads to real-time first?
> >>>>>>>
> >>>>>>> I currently see the real need only for IOCTLs, but the question is then
> >>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >>>>>>> avoid unwanted migration costs (which are significantly higher than
> >>>>>>> syscall restarts).
> >>>>>>
> >>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> >>>>>
> >>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
> >>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> >>>>>
> >>>>>> not remember merging this. However I find commit
> >>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> >>>>>> explain the reason pretty clear.
> >>>>>>
> >>>>>> At the time of the discussion we had concluded that it was the way
> >>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
> >>>>>> from secondary domain, which is counter-intuitive, error-prone, and
> >>>>>> forces you to cripple driver code for checks for the current domain.
> >>>>>
> >>>>> Nope, normal drivers are not affected as they just implement those
> >>>>> services in the respective mode they want to support there and have a
> >>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> >>>>> leaving out the implementation of the counterpart handler).
> >>>>
> >>>> Yes, I got mixed up trying to remember. I think the crux of the
> >>>> problem is that if a thread running in primary mode gets
> >>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> >>>> handler gets invoked, which is almost certainly the wrong thing to
> >>>> do. You want the thread to migrate to primary mode to execute
> >>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> >>>> application in gdb causes the application to behave differently.
> >>>
> >>> And trying and avoiding this issue indeed cripple codes with checks
> >>> for rtdm_in_rt_context:
> >>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
> >>>
> >>
> >> I don't remember details here, but this is a special case: The driver
> >> provides also read_nrt - is that really useful for Analogy?
> >>
> >> In most cases, you are fine with not providing the nrt (or rt) handler,
> >> or with a simple
> >>
> >> default:
> >> 	return -ENOSYS;
> >>
> >> in your ioctl dispatcher.
> > 
> > You are missing the point: if you enter read_nrt, there are two
> > cases:
> > - either the thread is real-time capable and has been relaxed by gdb
> > and you want to switch to read_rt for the reasons I already
> > explained, in that case, you must return -ENOSYS;
> > - or the thread is not real-time capable and the nrt handler
> > applies.
> > 
> > So, you need at least
> > 
> > read_nrt()
> > {
> > 	if (rt_capable)
> > 	     return -ENOSYS;
> > 
> > 	/* Do the normal case here */
> > }
> 
> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
> understand why, though). And having some special code in the exceptional
> case is probably better then the side effects we get from eagerly
> switching now.


Analogy is special, but is in-tree. How many in-tree drivers
require an nrt handler that does not switch back to primary mode?

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 21:08, Philippe Gerum wrote:
> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>> Gilles,
>>>>>>>>>>>
>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>
>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>> domains.
>>>>>>>>>>
>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>
>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>
>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>> syscall restarts).
>>>>>>>>>>
>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>
>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>
>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>
>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>
>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>
>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>
>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>> for rtdm_in_rt_context:
>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>
>>>>>>
>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>
>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>> or with a simple
>>>>>>
>>>>>> default:
>>>>>> 	return -ENOSYS;
>>>>>>
>>>>>> in your ioctl dispatcher.
>>>>>
>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>> cases:
>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>> and you want to switch to read_rt for the reasons I already
>>>>> explained, in that case, you must return -ENOSYS;
>>>>> - or the thread is not real-time capable and the nrt handler
>>>>> applies.
>>>>>
>>>>> So, you need at least
>>>>>
>>>>> read_nrt()
>>>>> {
>>>>> 	if (rt_capable)
>>>>> 	     return -ENOSYS;
>>>>>
>>>>> 	/* Do the normal case here */
>>>>> }
>>>>
>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>> understand why, though). And having some special code in the exceptional
>>>> case is probably better then the side effects we get from eagerly
>>>> switching now.
>>>>
>>>
>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>> believe is wrong. The normal use case is the one that does not ask the
>>> user to track the current mode for knowing what any random driver would
>>> eventually do depending on the calling context.
>>
>> You still miss the point that this is not required in 99% of the cases.
>> There is no such problem. There only Analogy.
>>
> 
> I'm not discussing Analogy at all, those drivers are still biased by the
> legacy 2.x logic for dealing with modes and need fixing. I have never
> been convinced by the reasoning behind rtdm_in_rt_context(), which
> perfectly illustrates why messing with the call mode is not the
> application's business.

You still need rtdm_in_rt_context() for the (rare) case of having the
same handler for both service_rt and service_nrt. That didn't change
with any switching strategy adjustment. It can't as long as there are
services behind a syscall that may handle any mode, thus that syscall is
unable to filter for the service in the background. We really need to
differentiate here.

> 
>> Every driver must ensure that a service is only exposed to users in the
>> right mode. That is a functional requirement, and drivers that fail to
>> do so only work by chance (thus with the restricted workload they are
>> tested against). If that is fulfilled, it doesn't matter to the driver
>> when the switch happens. It's pure optimization.
>>
> 
> You don't seem to get my point either. Let's proceed differently, please
> sketch the application code that would require __xn_exec_current for
> RTDM calls.

You cut the more interesting case (migration ping-pong when calling
non-RT drivers from relaxed threads), and I hope you will not forget to
answer this.

But let's go to our case:

We have a non-blocking service in the driver, the classic case of
accessing a privileged resource that userspace can't or shouldn't touch
directly. Think of some kind of register access that requires low-level
synchronization with other threads and interrupt handlers. That service
is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
(some thousand times per second). The RT threads are obviously on the
time critical path, must not migrate, and that can be achieved perfectly
already by providing that service under ioctl_rt. The non-RT threads
could be migrated to RT, but then they would pay an unneeded price,
contributing to a higher system load, in the worst case overload.
Therefore, the very same service shall be provided under ioctl_nrt as
well. Makes sense?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-12 21:11, Gilles Chanteperdrix wrote:
> On Thu, May 12, 2016 at 08:24:54PM +0200, Jan Kiszka wrote:
>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>> Gilles,
>>>>>>>>>
>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>
>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>> domains.
>>>>>>>>
>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>> redundant with plain linux drivers.
>>>>>>>
>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>> really like to avoid this pain for the users.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>
>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>> syscall restarts).
>>>>>>>>
>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>
>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>
>>>>>>>> not remember merging this. However I find commit
>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>> explain the reason pretty clear.
>>>>>>>>
>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>
>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>> services in the respective mode they want to support there and have a
>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>
>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>> problem is that if a thread running in primary mode gets
>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>> application in gdb causes the application to behave differently.
>>>>>
>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>> for rtdm_in_rt_context:
>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>
>>>>
>>>> I don't remember details here, but this is a special case: The driver
>>>> provides also read_nrt - is that really useful for Analogy?
>>>>
>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>> or with a simple
>>>>
>>>> default:
>>>> 	return -ENOSYS;
>>>>
>>>> in your ioctl dispatcher.
>>>
>>> You are missing the point: if you enter read_nrt, there are two
>>> cases:
>>> - either the thread is real-time capable and has been relaxed by gdb
>>> and you want to switch to read_rt for the reasons I already
>>> explained, in that case, you must return -ENOSYS;
>>> - or the thread is not real-time capable and the nrt handler
>>> applies.
>>>
>>> So, you need at least
>>>
>>> read_nrt()
>>> {
>>> 	if (rt_capable)
>>> 	     return -ENOSYS;
>>>
>>> 	/* Do the normal case here */
>>> }
>>
>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>> understand why, though). And having some special code in the exceptional
>> case is probably better then the side effects we get from eagerly
>> switching now.
> 
> 
> Analogy is special, but is in-tree. How many in-tree drivers
> require an nrt handler that does not switch back to primary mode?

That is not an argument because Xenomai never designed its APIs only for
in-tree usage.

And even if it were, I pointed out problems that are unrelated to RTDM
drivers, rather to wrapping all POSIX I/O functions and probing for the
target driver and domain.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 09:31:02PM +0200, Jan Kiszka wrote:
> >> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
> >> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
> >> understand why, though). And having some special code in the exceptional
> >> case is probably better then the side effects we get from eagerly
> >> switching now.
> > 
> > 
> > Analogy is special, but is in-tree. How many in-tree drivers
> > require an nrt handler that does not switch back to primary mode?
> 
> That is not an argument because Xenomai never designed its APIs only for
> in-tree usage.

You were the one who restricted the discussion to in-tree drivers.
See the quoted part above. I will stop answering this thread now,
and let you discuss the issue with Philippe.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Thu, May 12, 2016 at 09:27:43PM +0200, Jan Kiszka wrote:
> On 2016-05-12 21:08, Philippe Gerum wrote:
> > On 05/12/2016 08:42 PM, Jan Kiszka wrote:
> >> On 2016-05-12 20:35, Philippe Gerum wrote:
> >>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
> >>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
> >>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
> >>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
> >>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
> >>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> >>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> >>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >>>>>>>>>>> Gilles,
> >>>>>>>>>>>
> >>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
> >>>>>>>>>>> traces any more. Do you have a pointer
> >>>>>>>>>>>
> >>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >>>>>>>>>>> services that can (and should) be called without any switching from both
> >>>>>>>>>>> domains.
> >>>>>>>>>>
> >>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
> >>>>>>>>>> redundant with plain linux drivers.
> >>>>>>>>>
> >>>>>>>>> Because that enforces the calling layer to either call the same service
> >>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
> >>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
> >>>>>>>>> really like to avoid this pain for the users.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
> >>>>>>>>>>>
> >>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
> >>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
> >>>>>>>>>>> syscall restarts).
> >>>>>>>>>>
> >>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> >>>>>>>>>
> >>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
> >>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> >>>>>>>>>
> >>>>>>>>>> not remember merging this. However I find commit
> >>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> >>>>>>>>>> explain the reason pretty clear.
> >>>>>>>>>>
> >>>>>>>>>> At the time of the discussion we had concluded that it was the way
> >>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
> >>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
> >>>>>>>>>> forces you to cripple driver code for checks for the current domain.
> >>>>>>>>>
> >>>>>>>>> Nope, normal drivers are not affected as they just implement those
> >>>>>>>>> services in the respective mode they want to support there and have a
> >>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> >>>>>>>>> leaving out the implementation of the counterpart handler).
> >>>>>>>>
> >>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
> >>>>>>>> problem is that if a thread running in primary mode gets
> >>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> >>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
> >>>>>>>> do. You want the thread to migrate to primary mode to execute
> >>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> >>>>>>>> application in gdb causes the application to behave differently.
> >>>>>>>
> >>>>>>> And trying and avoiding this issue indeed cripple codes with checks
> >>>>>>> for rtdm_in_rt_context:
> >>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
> >>>>>>>
> >>>>>>
> >>>>>> I don't remember details here, but this is a special case: The driver
> >>>>>> provides also read_nrt - is that really useful for Analogy?
> >>>>>>
> >>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
> >>>>>> or with a simple
> >>>>>>
> >>>>>> default:
> >>>>>> 	return -ENOSYS;
> >>>>>>
> >>>>>> in your ioctl dispatcher.
> >>>>>
> >>>>> You are missing the point: if you enter read_nrt, there are two
> >>>>> cases:
> >>>>> - either the thread is real-time capable and has been relaxed by gdb
> >>>>> and you want to switch to read_rt for the reasons I already
> >>>>> explained, in that case, you must return -ENOSYS;
> >>>>> - or the thread is not real-time capable and the nrt handler
> >>>>> applies.
> >>>>>
> >>>>> So, you need at least
> >>>>>
> >>>>> read_nrt()
> >>>>> {
> >>>>> 	if (rt_capable)
> >>>>> 	     return -ENOSYS;
> >>>>>
> >>>>> 	/* Do the normal case here */
> >>>>> }
> >>>>
> >>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
> >>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
> >>>> understand why, though). And having some special code in the exceptional
> >>>> case is probably better then the side effects we get from eagerly
> >>>> switching now.
> >>>>
> >>>
> >>> Sorry, that is exactly the opposite: your use case is exceptional and I
> >>> believe is wrong. The normal use case is the one that does not ask the
> >>> user to track the current mode for knowing what any random driver would
> >>> eventually do depending on the calling context.
> >>
> >> You still miss the point that this is not required in 99% of the cases.
> >> There is no such problem. There only Analogy.
> >>
> > 
> > I'm not discussing Analogy at all, those drivers are still biased by the
> > legacy 2.x logic for dealing with modes and need fixing. I have never
> > been convinced by the reasoning behind rtdm_in_rt_context(), which
> > perfectly illustrates why messing with the call mode is not the
> > application's business.
> 
> You still need rtdm_in_rt_context() for the (rare) case of having the
> same handler for both service_rt and service_nrt. That didn't change
> with any switching strategy adjustment. It can't as long as there are
> services behind a syscall that may handle any mode, thus that syscall is
> unable to filter for the service in the background. We really need to
> differentiate here.

You do not need rtdm_in_rt_context. By writing a separate _rt and
_nrt driver you achieve exactly the same goal. If the the two
handlers share some code, put it in a common function. It has
always been a choice to use rtdm_in_rt_context. And not a very good
one in fact: you get potentially two instruction cache misses
instead of just one.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 05/12/2016 09:27 PM, Jan Kiszka wrote:
> On 2016-05-12 21:08, Philippe Gerum wrote:
>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>
>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>
>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>> domains.
>>>>>>>>>>>
>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>
>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>
>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>
>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>
>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>
>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>
>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>
>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>
>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>
>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>> for rtdm_in_rt_context:
>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>
>>>>>>>
>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>
>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>> or with a simple
>>>>>>>
>>>>>>> default:
>>>>>>> 	return -ENOSYS;
>>>>>>>
>>>>>>> in your ioctl dispatcher.
>>>>>>
>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>> cases:
>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>> applies.
>>>>>>
>>>>>> So, you need at least
>>>>>>
>>>>>> read_nrt()
>>>>>> {
>>>>>> 	if (rt_capable)
>>>>>> 	     return -ENOSYS;
>>>>>>
>>>>>> 	/* Do the normal case here */
>>>>>> }
>>>>>
>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>> understand why, though). And having some special code in the exceptional
>>>>> case is probably better then the side effects we get from eagerly
>>>>> switching now.
>>>>>
>>>>
>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>> believe is wrong. The normal use case is the one that does not ask the
>>>> user to track the current mode for knowing what any random driver would
>>>> eventually do depending on the calling context.
>>>
>>> You still miss the point that this is not required in 99% of the cases.
>>> There is no such problem. There only Analogy.
>>>
>>
>> I'm not discussing Analogy at all, those drivers are still biased by the
>> legacy 2.x logic for dealing with modes and need fixing. I have never
>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>> perfectly illustrates why messing with the call mode is not the
>> application's business.
> 
> You still need rtdm_in_rt_context() for the (rare) case of having the
> same handler for both service_rt and service_nrt. That didn't change
> with any switching strategy adjustment. It can't as long as there are
> services behind a syscall that may handle any mode, thus that syscall is
> unable to filter for the service in the background. We really need to
> differentiate here.
> 
>>
>>> Every driver must ensure that a service is only exposed to users in the
>>> right mode. That is a functional requirement, and drivers that fail to
>>> do so only work by chance (thus with the restricted workload they are
>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>> when the switch happens. It's pure optimization.
>>>
>>
>> You don't seem to get my point either. Let's proceed differently, please
>> sketch the application code that would require __xn_exec_current for
>> RTDM calls.
> 
> You cut the more interesting case (migration ping-pong when calling
> non-RT drivers from relaxed threads), and I hope you will not forget to
> answer this.
> 

I'm not ignoring the question, I have been postponing the answer until I
understand why the application could be put in a situation making this
migration a problem, and whether another approach would exist for
solving that problem within the current scheme.

> But let's go to our case:
> 
> We have a non-blocking service in the driver, the classic case of
> accessing a privileged resource that userspace can't or shouldn't touch
> directly. Think of some kind of register access that requires low-level
> synchronization with other threads and interrupt handlers. That service
> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
> (some thousand times per second). The RT threads are obviously on the
> time critical path, must not migrate, and that can be achieved perfectly
> already by providing that service under ioctl_rt. The non-RT threads
> could be migrated to RT, but then they would pay an unneeded price,
> contributing to a higher system load, in the worst case overload.
> Therefore, the very same service shall be provided under ioctl_nrt as
> well. Makes sense?
> 

I understand the conflict with the "rt-always-has-precedence" rule
implemented by the conforming state, then I have another question:

assuming the nrt thread undergoes the SCHED_WEAK policy because it is
mainly operating from the Linux space but still needs to synchronize
with the rt side at some point, which kind of high frequency interaction
with the rt side is this?

Sharing some resource requiring mutual exclusion via a Cobalt synchro,
waiting for rt events, something else?

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-13 00:26, Philippe Gerum wrote:
> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>
>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>
>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>> domains.
>>>>>>>>>>>>
>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>
>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>
>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>
>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>
>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>
>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>
>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>
>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>
>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>
>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>> or with a simple
>>>>>>>>
>>>>>>>> default:
>>>>>>>> 	return -ENOSYS;
>>>>>>>>
>>>>>>>> in your ioctl dispatcher.
>>>>>>>
>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>> cases:
>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>> applies.
>>>>>>>
>>>>>>> So, you need at least
>>>>>>>
>>>>>>> read_nrt()
>>>>>>> {
>>>>>>> 	if (rt_capable)
>>>>>>> 	     return -ENOSYS;
>>>>>>>
>>>>>>> 	/* Do the normal case here */
>>>>>>> }
>>>>>>
>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>> understand why, though). And having some special code in the exceptional
>>>>>> case is probably better then the side effects we get from eagerly
>>>>>> switching now.
>>>>>>
>>>>>
>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>> user to track the current mode for knowing what any random driver would
>>>>> eventually do depending on the calling context.
>>>>
>>>> You still miss the point that this is not required in 99% of the cases.
>>>> There is no such problem. There only Analogy.
>>>>
>>>
>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>> perfectly illustrates why messing with the call mode is not the
>>> application's business.
>>
>> You still need rtdm_in_rt_context() for the (rare) case of having the
>> same handler for both service_rt and service_nrt. That didn't change
>> with any switching strategy adjustment. It can't as long as there are
>> services behind a syscall that may handle any mode, thus that syscall is
>> unable to filter for the service in the background. We really need to
>> differentiate here.
>>
>>>
>>>> Every driver must ensure that a service is only exposed to users in the
>>>> right mode. That is a functional requirement, and drivers that fail to
>>>> do so only work by chance (thus with the restricted workload they are
>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>> when the switch happens. It's pure optimization.
>>>>
>>>
>>> You don't seem to get my point either. Let's proceed differently, please
>>> sketch the application code that would require __xn_exec_current for
>>> RTDM calls.
>>
>> You cut the more interesting case (migration ping-pong when calling
>> non-RT drivers from relaxed threads), and I hope you will not forget to
>> answer this.
>>
> 
> I'm not ignoring the question, I have been postponing the answer until I
> understand why the application could be put in a situation making this
> migration a problem, and whether another approach would exist for
> solving that problem within the current scheme.

These two scenarios are unrelated: this migration issue would still be
there even if we solved the one below via a different application/driver
design.

> 
>> But let's go to our case:
>>
>> We have a non-blocking service in the driver, the classic case of
>> accessing a privileged resource that userspace can't or shouldn't touch
>> directly. Think of some kind of register access that requires low-level
>> synchronization with other threads and interrupt handlers. That service
>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>> (some thousand times per second). The RT threads are obviously on the
>> time critical path, must not migrate, and that can be achieved perfectly
>> already by providing that service under ioctl_rt. The non-RT threads
>> could be migrated to RT, but then they would pay an unneeded price,
>> contributing to a higher system load, in the worst case overload.
>> Therefore, the very same service shall be provided under ioctl_nrt as
>> well. Makes sense?
>>
> 
> I understand the conflict with the "rt-always-has-precedence" rule
> implemented by the conforming state, then I have another question:
> 
> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
> mainly operating from the Linux space but still needs to synchronize
> with the rt side at some point, which kind of high frequency interaction
> with the rt side is this?
> 
> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
> waiting for rt events, something else?
> 

There synchronization need is first of all only on the hardware access
(thus inside the driver), not necessarily at application level. In fact,
there are even scenarios where you only want to exploit the driver as
permission checker on privileged resource accesses (userspace shall only
access certain MMIO registers in a page, thus the driver acts as
gatekeeper). Then there could be no synchronization at all but still the
need to provide migration-free accesses.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 05/13/2016 07:54 AM, Jan Kiszka wrote:
> On 2016-05-13 00:26, Philippe Gerum wrote:
>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>
>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>
>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>
>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>
>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>
>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>
>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>
>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>
>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>> or with a simple
>>>>>>>>>
>>>>>>>>> default:
>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>
>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>
>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>> cases:
>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>> applies.
>>>>>>>>
>>>>>>>> So, you need at least
>>>>>>>>
>>>>>>>> read_nrt()
>>>>>>>> {
>>>>>>>> 	if (rt_capable)
>>>>>>>> 	     return -ENOSYS;
>>>>>>>>
>>>>>>>> 	/* Do the normal case here */
>>>>>>>> }
>>>>>>>
>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>> switching now.
>>>>>>>
>>>>>>
>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>> user to track the current mode for knowing what any random driver would
>>>>>> eventually do depending on the calling context.
>>>>>
>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>> There is no such problem. There only Analogy.
>>>>>
>>>>
>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>> perfectly illustrates why messing with the call mode is not the
>>>> application's business.
>>>
>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>> same handler for both service_rt and service_nrt. That didn't change
>>> with any switching strategy adjustment. It can't as long as there are
>>> services behind a syscall that may handle any mode, thus that syscall is
>>> unable to filter for the service in the background. We really need to
>>> differentiate here.
>>>
>>>>
>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>> do so only work by chance (thus with the restricted workload they are
>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>> when the switch happens. It's pure optimization.
>>>>>
>>>>
>>>> You don't seem to get my point either. Let's proceed differently, please
>>>> sketch the application code that would require __xn_exec_current for
>>>> RTDM calls.
>>>
>>> You cut the more interesting case (migration ping-pong when calling
>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>> answer this.
>>>
>>
>> I'm not ignoring the question, I have been postponing the answer until I
>> understand why the application could be put in a situation making this
>> migration a problem, and whether another approach would exist for
>> solving that problem within the current scheme.
> 
> These two scenarios are unrelated: this migration issue would still be
> there even if we solved the one below via a different application/driver
> design.
> 

Which starts to be an issue only because the caller is a Cobalt shadow
undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
operation very frequently. For this reason, those two scenarii are very
much related.

>>
>>> But let's go to our case:
>>>
>>> We have a non-blocking service in the driver, the classic case of
>>> accessing a privileged resource that userspace can't or shouldn't touch
>>> directly. Think of some kind of register access that requires low-level
>>> synchronization with other threads and interrupt handlers. That service
>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>> (some thousand times per second). The RT threads are obviously on the
>>> time critical path, must not migrate, and that can be achieved perfectly
>>> already by providing that service under ioctl_rt. The non-RT threads
>>> could be migrated to RT, but then they would pay an unneeded price,
>>> contributing to a higher system load, in the worst case overload.
>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>> well. Makes sense?
>>>
>>
>> I understand the conflict with the "rt-always-has-precedence" rule
>> implemented by the conforming state, then I have another question:
>>
>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>> mainly operating from the Linux space but still needs to synchronize
>> with the rt side at some point, which kind of high frequency interaction
>> with the rt side is this?
>>
>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>> waiting for rt events, something else?
>>
> 
> There synchronization need is first of all only on the hardware access
> (thus inside the driver), not necessarily at application level. In fact,
> there are even scenarios where you only want to exploit the driver as
> permission checker on privileged resource accesses (userspace shall only
> access certain MMIO registers in a page, thus the driver acts as
> gatekeeper). Then there could be no synchronization at all but still the
> need to provide migration-free accesses.
> 

I get the idea of the resource gatekeeper, which does make a lot of sense.

However I still don't get which benefit your caller has in undergoing
the SCHED_WEAK policy - which implies that it has to share
synchronization points with Cobalt - compared to running as a regular
(glibc) thread, under whichever policy that could fit?

Leaving the non-RT ioctl call aside, which are those Cobalt calls the
SCHED_WEAK thread needs to invoke for synchronizing with rt threads?

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-13 15:38, Philippe Gerum wrote:
> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>
>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>
>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>
>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>> or with a simple
>>>>>>>>>>
>>>>>>>>>> default:
>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>
>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>
>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>> cases:
>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>> applies.
>>>>>>>>>
>>>>>>>>> So, you need at least
>>>>>>>>>
>>>>>>>>> read_nrt()
>>>>>>>>> {
>>>>>>>>> 	if (rt_capable)
>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>
>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>> }
>>>>>>>>
>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>> switching now.
>>>>>>>>
>>>>>>>
>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>> eventually do depending on the calling context.
>>>>>>
>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>> There is no such problem. There only Analogy.
>>>>>>
>>>>>
>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>> perfectly illustrates why messing with the call mode is not the
>>>>> application's business.
>>>>
>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>> same handler for both service_rt and service_nrt. That didn't change
>>>> with any switching strategy adjustment. It can't as long as there are
>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>> unable to filter for the service in the background. We really need to
>>>> differentiate here.
>>>>
>>>>>
>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>> when the switch happens. It's pure optimization.
>>>>>>
>>>>>
>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>> sketch the application code that would require __xn_exec_current for
>>>>> RTDM calls.
>>>>
>>>> You cut the more interesting case (migration ping-pong when calling
>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>> answer this.
>>>>
>>>
>>> I'm not ignoring the question, I have been postponing the answer until I
>>> understand why the application could be put in a situation making this
>>> migration a problem, and whether another approach would exist for
>>> solving that problem within the current scheme.
>>
>> These two scenarios are unrelated: this migration issue would still be
>> there even if we solved the one below via a different application/driver
>> design.
>>
> 
> Which starts to be an issue only because the caller is a Cobalt shadow
> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
> operation very frequently. For this reason, those two scenarii are very
> much related.

Not SCHED_WEAK, but being a shadow in the first place. Unless you
enforce non-shadow thread creation, all are shadowed in a Xenomai
application, thus are affected. However, asking our users to user
__real_pthread_create extensively may not lead to the desired portable
designs.

> 
>>>
>>>> But let's go to our case:
>>>>
>>>> We have a non-blocking service in the driver, the classic case of
>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>> directly. Think of some kind of register access that requires low-level
>>>> synchronization with other threads and interrupt handlers. That service
>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>> (some thousand times per second). The RT threads are obviously on the
>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>> contributing to a higher system load, in the worst case overload.
>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>> well. Makes sense?
>>>>
>>>
>>> I understand the conflict with the "rt-always-has-precedence" rule
>>> implemented by the conforming state, then I have another question:
>>>
>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>> mainly operating from the Linux space but still needs to synchronize
>>> with the rt side at some point, which kind of high frequency interaction
>>> with the rt side is this?
>>>
>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>> waiting for rt events, something else?
>>>
>>
>> There synchronization need is first of all only on the hardware access
>> (thus inside the driver), not necessarily at application level. In fact,
>> there are even scenarios where you only want to exploit the driver as
>> permission checker on privileged resource accesses (userspace shall only
>> access certain MMIO registers in a page, thus the driver acts as
>> gatekeeper). Then there could be no synchronization at all but still the
>> need to provide migration-free accesses.
>>
> 
> I get the idea of the resource gatekeeper, which does make a lot of sense.
> 
> However I still don't get which benefit your caller has in undergoing
> the SCHED_WEAK policy - which implies that it has to share
> synchronization points with Cobalt - compared to running as a regular
> (glibc) thread, under whichever policy that could fit?

See above: it's additional, non-portable instrumentation of your code to
tag non-shadowed threads. And then you may easily run into troubles in
larger, layered application designs that a non-shadowed thread will
still need a blocking Xenomai service, e.g. via some hidden dependency.

> 
> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?

I don't have these details at hand, but let's consider a large layered
application that also does significant work against Linux APIs during
runtime. You can't always enforce the complete separation. Because if
you can, you could also move the non-RT part into a separate process
that has nothing to do with Xenomai.

We promote the transparency of the Xenomai POSIX interface, and that
should not make the usage of non-Xenomai services needlessly expensive
or require extensive non-portable tagging via __real_ prefixes.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-05-13 17:32, Jan Kiszka wrote:
> On 2016-05-13 15:38, Philippe Gerum wrote:
>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>>
>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>>
>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>>> or with a simple
>>>>>>>>>>>
>>>>>>>>>>> default:
>>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>>
>>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>>
>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>>> cases:
>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>>> applies.
>>>>>>>>>>
>>>>>>>>>> So, you need at least
>>>>>>>>>>
>>>>>>>>>> read_nrt()
>>>>>>>>>> {
>>>>>>>>>> 	if (rt_capable)
>>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>>
>>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>>> switching now.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>>> eventually do depending on the calling context.
>>>>>>>
>>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>>> There is no such problem. There only Analogy.
>>>>>>>
>>>>>>
>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>>> perfectly illustrates why messing with the call mode is not the
>>>>>> application's business.
>>>>>
>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>>> same handler for both service_rt and service_nrt. That didn't change
>>>>> with any switching strategy adjustment. It can't as long as there are
>>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>>> unable to filter for the service in the background. We really need to
>>>>> differentiate here.
>>>>>
>>>>>>
>>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>>> when the switch happens. It's pure optimization.
>>>>>>>
>>>>>>
>>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>>> sketch the application code that would require __xn_exec_current for
>>>>>> RTDM calls.
>>>>>
>>>>> You cut the more interesting case (migration ping-pong when calling
>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>>> answer this.
>>>>>
>>>>
>>>> I'm not ignoring the question, I have been postponing the answer until I
>>>> understand why the application could be put in a situation making this
>>>> migration a problem, and whether another approach would exist for
>>>> solving that problem within the current scheme.
>>>
>>> These two scenarios are unrelated: this migration issue would still be
>>> there even if we solved the one below via a different application/driver
>>> design.
>>>
>>
>> Which starts to be an issue only because the caller is a Cobalt shadow
>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
>> operation very frequently. For this reason, those two scenarii are very
>> much related.
> 
> Not SCHED_WEAK, but being a shadow in the first place. Unless you
> enforce non-shadow thread creation, all are shadowed in a Xenomai
> application, thus are affected. However, asking our users to user
> __real_pthread_create extensively may not lead to the desired portable
> designs.
> 
>>
>>>>
>>>>> But let's go to our case:
>>>>>
>>>>> We have a non-blocking service in the driver, the classic case of
>>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>>> directly. Think of some kind of register access that requires low-level
>>>>> synchronization with other threads and interrupt handlers. That service
>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>>> (some thousand times per second). The RT threads are obviously on the
>>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>>> contributing to a higher system load, in the worst case overload.
>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>>> well. Makes sense?
>>>>>
>>>>
>>>> I understand the conflict with the "rt-always-has-precedence" rule
>>>> implemented by the conforming state, then I have another question:
>>>>
>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>>> mainly operating from the Linux space but still needs to synchronize
>>>> with the rt side at some point, which kind of high frequency interaction
>>>> with the rt side is this?
>>>>
>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>>> waiting for rt events, something else?
>>>>
>>>
>>> There synchronization need is first of all only on the hardware access
>>> (thus inside the driver), not necessarily at application level. In fact,
>>> there are even scenarios where you only want to exploit the driver as
>>> permission checker on privileged resource accesses (userspace shall only
>>> access certain MMIO registers in a page, thus the driver acts as
>>> gatekeeper). Then there could be no synchronization at all but still the
>>> need to provide migration-free accesses.
>>>
>>
>> I get the idea of the resource gatekeeper, which does make a lot of sense.
>>
>> However I still don't get which benefit your caller has in undergoing
>> the SCHED_WEAK policy - which implies that it has to share
>> synchronization points with Cobalt - compared to running as a regular
>> (glibc) thread, under whichever policy that could fit?
> 
> See above: it's additional, non-portable instrumentation of your code to
> tag non-shadowed threads. And then you may easily run into troubles in
> larger, layered application designs that a non-shadowed thread will
> still need a blocking Xenomai service, e.g. via some hidden dependency.
> 
>>
>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
> 
> I don't have these details at hand, but let's consider a large layered
> application that also does significant work against Linux APIs during
> runtime. You can't always enforce the complete separation. Because if
> you can, you could also move the non-RT part into a separate process
> that has nothing to do with Xenomai.
> 
> We promote the transparency of the Xenomai POSIX interface, and that
> should not make the usage of non-Xenomai services needlessly expensive
> or require extensive non-portable tagging via __real_ prefixes.
> 

Ping on this still open topic (will now have to introduce a local patch
that restores the original behaviour). Can we resolve the issue upstream
as well?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 06/14/2016 05:09 PM, Jan Kiszka wrote:
> On 2016-05-13 17:32, Jan Kiszka wrote:
>> On 2016-05-13 15:38, Philippe Gerum wrote:
>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>>>
>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>>>> or with a simple
>>>>>>>>>>>>
>>>>>>>>>>>> default:
>>>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>>>
>>>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>>>
>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>>>> cases:
>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>>>> applies.
>>>>>>>>>>>
>>>>>>>>>>> So, you need at least
>>>>>>>>>>>
>>>>>>>>>>> read_nrt()
>>>>>>>>>>> {
>>>>>>>>>>> 	if (rt_capable)
>>>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>>>
>>>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>>>> switching now.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>>>> eventually do depending on the calling context.
>>>>>>>>
>>>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>>>> There is no such problem. There only Analogy.
>>>>>>>>
>>>>>>>
>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>>>> perfectly illustrates why messing with the call mode is not the
>>>>>>> application's business.
>>>>>>
>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>>>> same handler for both service_rt and service_nrt. That didn't change
>>>>>> with any switching strategy adjustment. It can't as long as there are
>>>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>>>> unable to filter for the service in the background. We really need to
>>>>>> differentiate here.
>>>>>>
>>>>>>>
>>>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>>>> when the switch happens. It's pure optimization.
>>>>>>>>
>>>>>>>
>>>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>>>> sketch the application code that would require __xn_exec_current for
>>>>>>> RTDM calls.
>>>>>>
>>>>>> You cut the more interesting case (migration ping-pong when calling
>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>>>> answer this.
>>>>>>
>>>>>
>>>>> I'm not ignoring the question, I have been postponing the answer until I
>>>>> understand why the application could be put in a situation making this
>>>>> migration a problem, and whether another approach would exist for
>>>>> solving that problem within the current scheme.
>>>>
>>>> These two scenarios are unrelated: this migration issue would still be
>>>> there even if we solved the one below via a different application/driver
>>>> design.
>>>>
>>>
>>> Which starts to be an issue only because the caller is a Cobalt shadow
>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
>>> operation very frequently. For this reason, those two scenarii are very
>>> much related.
>>
>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
>> enforce non-shadow thread creation, all are shadowed in a Xenomai
>> application, thus are affected. However, asking our users to user
>> __real_pthread_create extensively may not lead to the desired portable
>> designs.
>>
>>>
>>>>>
>>>>>> But let's go to our case:
>>>>>>
>>>>>> We have a non-blocking service in the driver, the classic case of
>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>>>> directly. Think of some kind of register access that requires low-level
>>>>>> synchronization with other threads and interrupt handlers. That service
>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>>>> (some thousand times per second). The RT threads are obviously on the
>>>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>>>> contributing to a higher system load, in the worst case overload.
>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>>>> well. Makes sense?
>>>>>>
>>>>>
>>>>> I understand the conflict with the "rt-always-has-precedence" rule
>>>>> implemented by the conforming state, then I have another question:
>>>>>
>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>>>> mainly operating from the Linux space but still needs to synchronize
>>>>> with the rt side at some point, which kind of high frequency interaction
>>>>> with the rt side is this?
>>>>>
>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>>>> waiting for rt events, something else?
>>>>>
>>>>
>>>> There synchronization need is first of all only on the hardware access
>>>> (thus inside the driver), not necessarily at application level. In fact,
>>>> there are even scenarios where you only want to exploit the driver as
>>>> permission checker on privileged resource accesses (userspace shall only
>>>> access certain MMIO registers in a page, thus the driver acts as
>>>> gatekeeper). Then there could be no synchronization at all but still the
>>>> need to provide migration-free accesses.
>>>>
>>>
>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
>>>
>>> However I still don't get which benefit your caller has in undergoing
>>> the SCHED_WEAK policy - which implies that it has to share
>>> synchronization points with Cobalt - compared to running as a regular
>>> (glibc) thread, under whichever policy that could fit?
>>
>> See above: it's additional, non-portable instrumentation of your code to
>> tag non-shadowed threads. And then you may easily run into troubles in
>> larger, layered application designs that a non-shadowed thread will
>> still need a blocking Xenomai service, e.g. via some hidden dependency.
>>
>>>
>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
>>
>> I don't have these details at hand, but let's consider a large layered
>> application that also does significant work against Linux APIs during
>> runtime. You can't always enforce the complete separation. Because if
>> you can, you could also move the non-RT part into a separate process
>> that has nothing to do with Xenomai.
>>
>> We promote the transparency of the Xenomai POSIX interface, and that
>> should not make the usage of non-Xenomai services needlessly expensive
>> or require extensive non-portable tagging via __real_ prefixes.
>>
> 
> Ping on this still open topic (will now have to introduce a local patch
> that restores the original behaviour). Can we resolve the issue upstream
> as well?
> 

Restoring the original behavior unconditionally would not be a fix but
only a work-around for your own issue. Finding a better way acceptable
to all parties is on my todo list for the upcoming 3.0.3.

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 17:23, Philippe Gerum wrote:
> On 06/14/2016 05:09 PM, Jan Kiszka wrote:
>> On 2016-05-13 17:32, Jan Kiszka wrote:
>>> On 2016-05-13 15:38, Philippe Gerum wrote:
>>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>>>>
>>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>>>>> or with a simple
>>>>>>>>>>>>>
>>>>>>>>>>>>> default:
>>>>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>>>>
>>>>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>>>>
>>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>>>>> cases:
>>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>>>>> applies.
>>>>>>>>>>>>
>>>>>>>>>>>> So, you need at least
>>>>>>>>>>>>
>>>>>>>>>>>> read_nrt()
>>>>>>>>>>>> {
>>>>>>>>>>>> 	if (rt_capable)
>>>>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>>>>
>>>>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>>>>> switching now.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>>>>> eventually do depending on the calling context.
>>>>>>>>>
>>>>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>>>>> There is no such problem. There only Analogy.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>>>>> perfectly illustrates why messing with the call mode is not the
>>>>>>>> application's business.
>>>>>>>
>>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>>>>> same handler for both service_rt and service_nrt. That didn't change
>>>>>>> with any switching strategy adjustment. It can't as long as there are
>>>>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>>>>> unable to filter for the service in the background. We really need to
>>>>>>> differentiate here.
>>>>>>>
>>>>>>>>
>>>>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>>>>> when the switch happens. It's pure optimization.
>>>>>>>>>
>>>>>>>>
>>>>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>>>>> sketch the application code that would require __xn_exec_current for
>>>>>>>> RTDM calls.
>>>>>>>
>>>>>>> You cut the more interesting case (migration ping-pong when calling
>>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>>>>> answer this.
>>>>>>>
>>>>>>
>>>>>> I'm not ignoring the question, I have been postponing the answer until I
>>>>>> understand why the application could be put in a situation making this
>>>>>> migration a problem, and whether another approach would exist for
>>>>>> solving that problem within the current scheme.
>>>>>
>>>>> These two scenarios are unrelated: this migration issue would still be
>>>>> there even if we solved the one below via a different application/driver
>>>>> design.
>>>>>
>>>>
>>>> Which starts to be an issue only because the caller is a Cobalt shadow
>>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
>>>> operation very frequently. For this reason, those two scenarii are very
>>>> much related.
>>>
>>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
>>> enforce non-shadow thread creation, all are shadowed in a Xenomai
>>> application, thus are affected. However, asking our users to user
>>> __real_pthread_create extensively may not lead to the desired portable
>>> designs.
>>>
>>>>
>>>>>>
>>>>>>> But let's go to our case:
>>>>>>>
>>>>>>> We have a non-blocking service in the driver, the classic case of
>>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>>>>> directly. Think of some kind of register access that requires low-level
>>>>>>> synchronization with other threads and interrupt handlers. That service
>>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>>>>> (some thousand times per second). The RT threads are obviously on the
>>>>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>>>>> contributing to a higher system load, in the worst case overload.
>>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>>>>> well. Makes sense?
>>>>>>>
>>>>>>
>>>>>> I understand the conflict with the "rt-always-has-precedence" rule
>>>>>> implemented by the conforming state, then I have another question:
>>>>>>
>>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>>>>> mainly operating from the Linux space but still needs to synchronize
>>>>>> with the rt side at some point, which kind of high frequency interaction
>>>>>> with the rt side is this?
>>>>>>
>>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>>>>> waiting for rt events, something else?
>>>>>>
>>>>>
>>>>> There synchronization need is first of all only on the hardware access
>>>>> (thus inside the driver), not necessarily at application level. In fact,
>>>>> there are even scenarios where you only want to exploit the driver as
>>>>> permission checker on privileged resource accesses (userspace shall only
>>>>> access certain MMIO registers in a page, thus the driver acts as
>>>>> gatekeeper). Then there could be no synchronization at all but still the
>>>>> need to provide migration-free accesses.
>>>>>
>>>>
>>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
>>>>
>>>> However I still don't get which benefit your caller has in undergoing
>>>> the SCHED_WEAK policy - which implies that it has to share
>>>> synchronization points with Cobalt - compared to running as a regular
>>>> (glibc) thread, under whichever policy that could fit?
>>>
>>> See above: it's additional, non-portable instrumentation of your code to
>>> tag non-shadowed threads. And then you may easily run into troubles in
>>> larger, layered application designs that a non-shadowed thread will
>>> still need a blocking Xenomai service, e.g. via some hidden dependency.
>>>
>>>>
>>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
>>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
>>>
>>> I don't have these details at hand, but let's consider a large layered
>>> application that also does significant work against Linux APIs during
>>> runtime. You can't always enforce the complete separation. Because if
>>> you can, you could also move the non-RT part into a separate process
>>> that has nothing to do with Xenomai.
>>>
>>> We promote the transparency of the Xenomai POSIX interface, and that
>>> should not make the usage of non-Xenomai services needlessly expensive
>>> or require extensive non-portable tagging via __real_ prefixes.
>>>
>>
>> Ping on this still open topic (will now have to introduce a local patch
>> that restores the original behaviour). Can we resolve the issue upstream
>> as well?
>>
> 
> Restoring the original behavior unconditionally would not be a fix but
> only a work-around for your own issue. Finding a better way acceptable
> to all parties is on my todo list for the upcoming 3.0.3.

We don't have all the issues, as I pointed out. It is a significant
deficit of current Xenomai that you now have to create non-Xenomai
threads explicitly (__real_pthread_create) in order to use Linux I/O
syscalls efficiently (because of the otherwise enforces migration
ping-pong). We didn't have that problem with the original design.

BTW, we is corectl a probing syscall? It doesn't make use of this
feature at all.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Tue, Jun 14, 2016 at 05:27:54PM +0200, Jan Kiszka wrote:
> On 2016-06-14 17:23, Philippe Gerum wrote:
> > On 06/14/2016 05:09 PM, Jan Kiszka wrote:
> >> On 2016-05-13 17:32, Jan Kiszka wrote:
> >>> On 2016-05-13 15:38, Philippe Gerum wrote:
> >>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
> >>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
> >>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
> >>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
> >>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
> >>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
> >>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
> >>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
> >>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> >>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >>>>>>>>>>>>>>>>>> Gilles,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
> >>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
> >>>>>>>>>>>>>>>>>> domains.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
> >>>>>>>>>>>>>>>>> redundant with plain linux drivers.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
> >>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
> >>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
> >>>>>>>>>>>>>>>> really like to avoid this pain for the users.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
> >>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
> >>>>>>>>>>>>>>>>>> syscall restarts).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
> >>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> not remember merging this. However I find commit
> >>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> >>>>>>>>>>>>>>>>> explain the reason pretty clear.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
> >>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
> >>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
> >>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
> >>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
> >>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> >>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
> >>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
> >>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> >>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
> >>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
> >>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> >>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
> >>>>>>>>>>>>>> for rtdm_in_rt_context:
> >>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
> >>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
> >>>>>>>>>>>>> or with a simple
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> default:
> >>>>>>>>>>>>> 	return -ENOSYS;
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> in your ioctl dispatcher.
> >>>>>>>>>>>>
> >>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
> >>>>>>>>>>>> cases:
> >>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
> >>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
> >>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
> >>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
> >>>>>>>>>>>> applies.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So, you need at least
> >>>>>>>>>>>>
> >>>>>>>>>>>> read_nrt()
> >>>>>>>>>>>> {
> >>>>>>>>>>>> 	if (rt_capable)
> >>>>>>>>>>>> 	     return -ENOSYS;
> >>>>>>>>>>>>
> >>>>>>>>>>>> 	/* Do the normal case here */
> >>>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
> >>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
> >>>>>>>>>>> understand why, though). And having some special code in the exceptional
> >>>>>>>>>>> case is probably better then the side effects we get from eagerly
> >>>>>>>>>>> switching now.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
> >>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
> >>>>>>>>>> user to track the current mode for knowing what any random driver would
> >>>>>>>>>> eventually do depending on the calling context.
> >>>>>>>>>
> >>>>>>>>> You still miss the point that this is not required in 99% of the cases.
> >>>>>>>>> There is no such problem. There only Analogy.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
> >>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
> >>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
> >>>>>>>> perfectly illustrates why messing with the call mode is not the
> >>>>>>>> application's business.
> >>>>>>>
> >>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
> >>>>>>> same handler for both service_rt and service_nrt. That didn't change
> >>>>>>> with any switching strategy adjustment. It can't as long as there are
> >>>>>>> services behind a syscall that may handle any mode, thus that syscall is
> >>>>>>> unable to filter for the service in the background. We really need to
> >>>>>>> differentiate here.
> >>>>>>>
> >>>>>>>>
> >>>>>>>>> Every driver must ensure that a service is only exposed to users in the
> >>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
> >>>>>>>>> do so only work by chance (thus with the restricted workload they are
> >>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
> >>>>>>>>> when the switch happens. It's pure optimization.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> You don't seem to get my point either. Let's proceed differently, please
> >>>>>>>> sketch the application code that would require __xn_exec_current for
> >>>>>>>> RTDM calls.
> >>>>>>>
> >>>>>>> You cut the more interesting case (migration ping-pong when calling
> >>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
> >>>>>>> answer this.
> >>>>>>>
> >>>>>>
> >>>>>> I'm not ignoring the question, I have been postponing the answer until I
> >>>>>> understand why the application could be put in a situation making this
> >>>>>> migration a problem, and whether another approach would exist for
> >>>>>> solving that problem within the current scheme.
> >>>>>
> >>>>> These two scenarios are unrelated: this migration issue would still be
> >>>>> there even if we solved the one below via a different application/driver
> >>>>> design.
> >>>>>
> >>>>
> >>>> Which starts to be an issue only because the caller is a Cobalt shadow
> >>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
> >>>> operation very frequently. For this reason, those two scenarii are very
> >>>> much related.
> >>>
> >>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
> >>> enforce non-shadow thread creation, all are shadowed in a Xenomai
> >>> application, thus are affected. However, asking our users to user
> >>> __real_pthread_create extensively may not lead to the desired portable
> >>> designs.
> >>>
> >>>>
> >>>>>>
> >>>>>>> But let's go to our case:
> >>>>>>>
> >>>>>>> We have a non-blocking service in the driver, the classic case of
> >>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
> >>>>>>> directly. Think of some kind of register access that requires low-level
> >>>>>>> synchronization with other threads and interrupt handlers. That service
> >>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
> >>>>>>> (some thousand times per second). The RT threads are obviously on the
> >>>>>>> time critical path, must not migrate, and that can be achieved perfectly
> >>>>>>> already by providing that service under ioctl_rt. The non-RT threads
> >>>>>>> could be migrated to RT, but then they would pay an unneeded price,
> >>>>>>> contributing to a higher system load, in the worst case overload.
> >>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
> >>>>>>> well. Makes sense?
> >>>>>>>
> >>>>>>
> >>>>>> I understand the conflict with the "rt-always-has-precedence" rule
> >>>>>> implemented by the conforming state, then I have another question:
> >>>>>>
> >>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
> >>>>>> mainly operating from the Linux space but still needs to synchronize
> >>>>>> with the rt side at some point, which kind of high frequency interaction
> >>>>>> with the rt side is this?
> >>>>>>
> >>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
> >>>>>> waiting for rt events, something else?
> >>>>>>
> >>>>>
> >>>>> There synchronization need is first of all only on the hardware access
> >>>>> (thus inside the driver), not necessarily at application level. In fact,
> >>>>> there are even scenarios where you only want to exploit the driver as
> >>>>> permission checker on privileged resource accesses (userspace shall only
> >>>>> access certain MMIO registers in a page, thus the driver acts as
> >>>>> gatekeeper). Then there could be no synchronization at all but still the
> >>>>> need to provide migration-free accesses.
> >>>>>
> >>>>
> >>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
> >>>>
> >>>> However I still don't get which benefit your caller has in undergoing
> >>>> the SCHED_WEAK policy - which implies that it has to share
> >>>> synchronization points with Cobalt - compared to running as a regular
> >>>> (glibc) thread, under whichever policy that could fit?
> >>>
> >>> See above: it's additional, non-portable instrumentation of your code to
> >>> tag non-shadowed threads. And then you may easily run into troubles in
> >>> larger, layered application designs that a non-shadowed thread will
> >>> still need a blocking Xenomai service, e.g. via some hidden dependency.
> >>>
> >>>>
> >>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
> >>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
> >>>
> >>> I don't have these details at hand, but let's consider a large layered
> >>> application that also does significant work against Linux APIs during
> >>> runtime. You can't always enforce the complete separation. Because if
> >>> you can, you could also move the non-RT part into a separate process
> >>> that has nothing to do with Xenomai.
> >>>
> >>> We promote the transparency of the Xenomai POSIX interface, and that
> >>> should not make the usage of non-Xenomai services needlessly expensive
> >>> or require extensive non-portable tagging via __real_ prefixes.
> >>>
> >>
> >> Ping on this still open topic (will now have to introduce a local patch
> >> that restores the original behaviour). Can we resolve the issue upstream
> >> as well?
> >>
> > 
> > Restoring the original behavior unconditionally would not be a fix but
> > only a work-around for your own issue. Finding a better way acceptable
> > to all parties is on my todo list for the upcoming 3.0.3.
> 
> It is a significant deficit of current Xenomai that you now have
> to create non-Xenomai threads explicitly (__real_pthread_create)
> in order to use Linux I/O syscalls efficiently (because of the
> otherwise enforces migration ping-pong).

Please don't spread misconceptions, there is no ping-pong when using
Linux I/O syscalls. There is ping-pong when using RTDM I/O with _nrt
handlers.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 17:38, Gilles Chanteperdrix wrote:
> On Tue, Jun 14, 2016 at 05:27:54PM +0200, Jan Kiszka wrote:
>> On 2016-06-14 17:23, Philippe Gerum wrote:
>>> On 06/14/2016 05:09 PM, Jan Kiszka wrote:
>>>> On 2016-05-13 17:32, Jan Kiszka wrote:
>>>>> On 2016-05-13 15:38, Philippe Gerum wrote:
>>>>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>>>>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>>>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>>>>>>> or with a simple
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> default:
>>>>>>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>>>>>>> cases:
>>>>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>>>>>>> applies.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, you need at least
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> read_nrt()
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> 	if (rt_capable)
>>>>>>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>>>>>>> switching now.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>>>>>>> eventually do depending on the calling context.
>>>>>>>>>>>
>>>>>>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>>>>>>> There is no such problem. There only Analogy.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>>>>>>> perfectly illustrates why messing with the call mode is not the
>>>>>>>>>> application's business.
>>>>>>>>>
>>>>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>>>>>>> same handler for both service_rt and service_nrt. That didn't change
>>>>>>>>> with any switching strategy adjustment. It can't as long as there are
>>>>>>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>>>>>>> unable to filter for the service in the background. We really need to
>>>>>>>>> differentiate here.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>>>>>>> when the switch happens. It's pure optimization.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>>>>>>> sketch the application code that would require __xn_exec_current for
>>>>>>>>>> RTDM calls.
>>>>>>>>>
>>>>>>>>> You cut the more interesting case (migration ping-pong when calling
>>>>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>>>>>>> answer this.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not ignoring the question, I have been postponing the answer until I
>>>>>>>> understand why the application could be put in a situation making this
>>>>>>>> migration a problem, and whether another approach would exist for
>>>>>>>> solving that problem within the current scheme.
>>>>>>>
>>>>>>> These two scenarios are unrelated: this migration issue would still be
>>>>>>> there even if we solved the one below via a different application/driver
>>>>>>> design.
>>>>>>>
>>>>>>
>>>>>> Which starts to be an issue only because the caller is a Cobalt shadow
>>>>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
>>>>>> operation very frequently. For this reason, those two scenarii are very
>>>>>> much related.
>>>>>
>>>>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
>>>>> enforce non-shadow thread creation, all are shadowed in a Xenomai
>>>>> application, thus are affected. However, asking our users to user
>>>>> __real_pthread_create extensively may not lead to the desired portable
>>>>> designs.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>>> But let's go to our case:
>>>>>>>>>
>>>>>>>>> We have a non-blocking service in the driver, the classic case of
>>>>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>>>>>>> directly. Think of some kind of register access that requires low-level
>>>>>>>>> synchronization with other threads and interrupt handlers. That service
>>>>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>>>>>>> (some thousand times per second). The RT threads are obviously on the
>>>>>>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>>>>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>>>>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>>>>>>> contributing to a higher system load, in the worst case overload.
>>>>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>>>>>>> well. Makes sense?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I understand the conflict with the "rt-always-has-precedence" rule
>>>>>>>> implemented by the conforming state, then I have another question:
>>>>>>>>
>>>>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>>>>>>> mainly operating from the Linux space but still needs to synchronize
>>>>>>>> with the rt side at some point, which kind of high frequency interaction
>>>>>>>> with the rt side is this?
>>>>>>>>
>>>>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>>>>>>> waiting for rt events, something else?
>>>>>>>>
>>>>>>>
>>>>>>> There synchronization need is first of all only on the hardware access
>>>>>>> (thus inside the driver), not necessarily at application level. In fact,
>>>>>>> there are even scenarios where you only want to exploit the driver as
>>>>>>> permission checker on privileged resource accesses (userspace shall only
>>>>>>> access certain MMIO registers in a page, thus the driver acts as
>>>>>>> gatekeeper). Then there could be no synchronization at all but still the
>>>>>>> need to provide migration-free accesses.
>>>>>>>
>>>>>>
>>>>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
>>>>>>
>>>>>> However I still don't get which benefit your caller has in undergoing
>>>>>> the SCHED_WEAK policy - which implies that it has to share
>>>>>> synchronization points with Cobalt - compared to running as a regular
>>>>>> (glibc) thread, under whichever policy that could fit?
>>>>>
>>>>> See above: it's additional, non-portable instrumentation of your code to
>>>>> tag non-shadowed threads. And then you may easily run into troubles in
>>>>> larger, layered application designs that a non-shadowed thread will
>>>>> still need a blocking Xenomai service, e.g. via some hidden dependency.
>>>>>
>>>>>>
>>>>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
>>>>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
>>>>>
>>>>> I don't have these details at hand, but let's consider a large layered
>>>>> application that also does significant work against Linux APIs during
>>>>> runtime. You can't always enforce the complete separation. Because if
>>>>> you can, you could also move the non-RT part into a separate process
>>>>> that has nothing to do with Xenomai.
>>>>>
>>>>> We promote the transparency of the Xenomai POSIX interface, and that
>>>>> should not make the usage of non-Xenomai services needlessly expensive
>>>>> or require extensive non-portable tagging via __real_ prefixes.
>>>>>
>>>>
>>>> Ping on this still open topic (will now have to introduce a local patch
>>>> that restores the original behaviour). Can we resolve the issue upstream
>>>> as well?
>>>>
>>>
>>> Restoring the original behavior unconditionally would not be a fix but
>>> only a work-around for your own issue. Finding a better way acceptable
>>> to all parties is on my todo list for the upcoming 3.0.3.
>>
>> It is a significant deficit of current Xenomai that you now have
>> to create non-Xenomai threads explicitly (__real_pthread_create)
>> in order to use Linux I/O syscalls efficiently (because of the
>> otherwise enforces migration ping-pong).
> 
> Please don't spread misconceptions, there is no ping-pong when using
> Linux I/O syscalls. There is ping-pong when using RTDM I/O with _nrt
> handlers.

Sorry, there *is*: Shadowed thread (anything created by wrapped
pthread_create) calls, say, read() on some Linux file descriptor, read()
is wrapped, first probes the call on RTDM, which means migration to RT
(for currently relaxed threads, like SCHED_WEAK), no RTDM match in the
kernel, and finally the migration to NRT in order to do the Linux read()
syscall. That didn't happen with the original design.

Jan

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 17:27, Jan Kiszka wrote:
> BTW, we is corectl a probing syscall? It doesn't make use of this
> feature at all.

Oh, there is - oversaw one code path in do_conf_option.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Tue, Jun 14, 2016 at 05:43:45PM +0200, Jan Kiszka wrote:
> On 2016-06-14 17:38, Gilles Chanteperdrix wrote:
> > On Tue, Jun 14, 2016 at 05:27:54PM +0200, Jan Kiszka wrote:
> >> On 2016-06-14 17:23, Philippe Gerum wrote:
> >>> On 06/14/2016 05:09 PM, Jan Kiszka wrote:
> >>>> On 2016-05-13 17:32, Jan Kiszka wrote:
> >>>>> On 2016-05-13 15:38, Philippe Gerum wrote:
> >>>>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
> >>>>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
> >>>>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
> >>>>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
> >>>>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
> >>>>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
> >>>>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
> >>>>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
> >>>>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
> >>>>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
> >>>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
> >>>>>>>>>>>>>>>>>>>> Gilles,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
> >>>>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
> >>>>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
> >>>>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
> >>>>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
> >>>>>>>>>>>>>>>>>>>> domains.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
> >>>>>>>>>>>>>>>>>>> redundant with plain linux drivers.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
> >>>>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
> >>>>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
> >>>>>>>>>>>>>>>>>> really like to avoid this pain for the users.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
> >>>>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
> >>>>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
> >>>>>>>>>>>>>>>>>>>> syscall restarts).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
> >>>>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> not remember merging this. However I find commit
> >>>>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
> >>>>>>>>>>>>>>>>>>> explain the reason pretty clear.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
> >>>>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
> >>>>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
> >>>>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
> >>>>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
> >>>>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
> >>>>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
> >>>>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
> >>>>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
> >>>>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
> >>>>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
> >>>>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
> >>>>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
> >>>>>>>>>>>>>>>> for rtdm_in_rt_context:
> >>>>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
> >>>>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
> >>>>>>>>>>>>>>> or with a simple
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> default:
> >>>>>>>>>>>>>>> 	return -ENOSYS;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> in your ioctl dispatcher.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
> >>>>>>>>>>>>>> cases:
> >>>>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
> >>>>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
> >>>>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
> >>>>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
> >>>>>>>>>>>>>> applies.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So, you need at least
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> read_nrt()
> >>>>>>>>>>>>>> {
> >>>>>>>>>>>>>> 	if (rt_capable)
> >>>>>>>>>>>>>> 	     return -ENOSYS;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 	/* Do the normal case here */
> >>>>>>>>>>>>>> }
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
> >>>>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
> >>>>>>>>>>>>> understand why, though). And having some special code in the exceptional
> >>>>>>>>>>>>> case is probably better then the side effects we get from eagerly
> >>>>>>>>>>>>> switching now.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
> >>>>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
> >>>>>>>>>>>> user to track the current mode for knowing what any random driver would
> >>>>>>>>>>>> eventually do depending on the calling context.
> >>>>>>>>>>>
> >>>>>>>>>>> You still miss the point that this is not required in 99% of the cases.
> >>>>>>>>>>> There is no such problem. There only Analogy.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
> >>>>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
> >>>>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
> >>>>>>>>>> perfectly illustrates why messing with the call mode is not the
> >>>>>>>>>> application's business.
> >>>>>>>>>
> >>>>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
> >>>>>>>>> same handler for both service_rt and service_nrt. That didn't change
> >>>>>>>>> with any switching strategy adjustment. It can't as long as there are
> >>>>>>>>> services behind a syscall that may handle any mode, thus that syscall is
> >>>>>>>>> unable to filter for the service in the background. We really need to
> >>>>>>>>> differentiate here.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Every driver must ensure that a service is only exposed to users in the
> >>>>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
> >>>>>>>>>>> do so only work by chance (thus with the restricted workload they are
> >>>>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
> >>>>>>>>>>> when the switch happens. It's pure optimization.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> You don't seem to get my point either. Let's proceed differently, please
> >>>>>>>>>> sketch the application code that would require __xn_exec_current for
> >>>>>>>>>> RTDM calls.
> >>>>>>>>>
> >>>>>>>>> You cut the more interesting case (migration ping-pong when calling
> >>>>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
> >>>>>>>>> answer this.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> I'm not ignoring the question, I have been postponing the answer until I
> >>>>>>>> understand why the application could be put in a situation making this
> >>>>>>>> migration a problem, and whether another approach would exist for
> >>>>>>>> solving that problem within the current scheme.
> >>>>>>>
> >>>>>>> These two scenarios are unrelated: this migration issue would still be
> >>>>>>> there even if we solved the one below via a different application/driver
> >>>>>>> design.
> >>>>>>>
> >>>>>>
> >>>>>> Which starts to be an issue only because the caller is a Cobalt shadow
> >>>>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
> >>>>>> operation very frequently. For this reason, those two scenarii are very
> >>>>>> much related.
> >>>>>
> >>>>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
> >>>>> enforce non-shadow thread creation, all are shadowed in a Xenomai
> >>>>> application, thus are affected. However, asking our users to user
> >>>>> __real_pthread_create extensively may not lead to the desired portable
> >>>>> designs.
> >>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>>> But let's go to our case:
> >>>>>>>>>
> >>>>>>>>> We have a non-blocking service in the driver, the classic case of
> >>>>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
> >>>>>>>>> directly. Think of some kind of register access that requires low-level
> >>>>>>>>> synchronization with other threads and interrupt handlers. That service
> >>>>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
> >>>>>>>>> (some thousand times per second). The RT threads are obviously on the
> >>>>>>>>> time critical path, must not migrate, and that can be achieved perfectly
> >>>>>>>>> already by providing that service under ioctl_rt. The non-RT threads
> >>>>>>>>> could be migrated to RT, but then they would pay an unneeded price,
> >>>>>>>>> contributing to a higher system load, in the worst case overload.
> >>>>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
> >>>>>>>>> well. Makes sense?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> I understand the conflict with the "rt-always-has-precedence" rule
> >>>>>>>> implemented by the conforming state, then I have another question:
> >>>>>>>>
> >>>>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
> >>>>>>>> mainly operating from the Linux space but still needs to synchronize
> >>>>>>>> with the rt side at some point, which kind of high frequency interaction
> >>>>>>>> with the rt side is this?
> >>>>>>>>
> >>>>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
> >>>>>>>> waiting for rt events, something else?
> >>>>>>>>
> >>>>>>>
> >>>>>>> There synchronization need is first of all only on the hardware access
> >>>>>>> (thus inside the driver), not necessarily at application level. In fact,
> >>>>>>> there are even scenarios where you only want to exploit the driver as
> >>>>>>> permission checker on privileged resource accesses (userspace shall only
> >>>>>>> access certain MMIO registers in a page, thus the driver acts as
> >>>>>>> gatekeeper). Then there could be no synchronization at all but still the
> >>>>>>> need to provide migration-free accesses.
> >>>>>>>
> >>>>>>
> >>>>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
> >>>>>>
> >>>>>> However I still don't get which benefit your caller has in undergoing
> >>>>>> the SCHED_WEAK policy - which implies that it has to share
> >>>>>> synchronization points with Cobalt - compared to running as a regular
> >>>>>> (glibc) thread, under whichever policy that could fit?
> >>>>>
> >>>>> See above: it's additional, non-portable instrumentation of your code to
> >>>>> tag non-shadowed threads. And then you may easily run into troubles in
> >>>>> larger, layered application designs that a non-shadowed thread will
> >>>>> still need a blocking Xenomai service, e.g. via some hidden dependency.
> >>>>>
> >>>>>>
> >>>>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
> >>>>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
> >>>>>
> >>>>> I don't have these details at hand, but let's consider a large layered
> >>>>> application that also does significant work against Linux APIs during
> >>>>> runtime. You can't always enforce the complete separation. Because if
> >>>>> you can, you could also move the non-RT part into a separate process
> >>>>> that has nothing to do with Xenomai.
> >>>>>
> >>>>> We promote the transparency of the Xenomai POSIX interface, and that
> >>>>> should not make the usage of non-Xenomai services needlessly expensive
> >>>>> or require extensive non-portable tagging via __real_ prefixes.
> >>>>>
> >>>>
> >>>> Ping on this still open topic (will now have to introduce a local patch
> >>>> that restores the original behaviour). Can we resolve the issue upstream
> >>>> as well?
> >>>>
> >>>
> >>> Restoring the original behavior unconditionally would not be a fix but
> >>> only a work-around for your own issue. Finding a better way acceptable
> >>> to all parties is on my todo list for the upcoming 3.0.3.
> >>
> >> It is a significant deficit of current Xenomai that you now have
> >> to create non-Xenomai threads explicitly (__real_pthread_create)
> >> in order to use Linux I/O syscalls efficiently (because of the
> >> otherwise enforces migration ping-pong).
> > 
> > Please don't spread misconceptions, there is no ping-pong when using
> > Linux I/O syscalls. There is ping-pong when using RTDM I/O with _nrt
> > handlers.
> 
> Sorry, there *is*: Shadowed thread (anything created by wrapped
> pthread_create) calls, say, read() on some Linux file descriptor, read()
> is wrapped, first probes the call on RTDM, which means migration to RT
> (for currently relaxed threads, like SCHED_WEAK), no RTDM match in the
> kernel, and finally the migration to NRT in order to do the Linux read()
> syscall. That didn't happen with the original design.

The wrapped read does not get ping-pong when calling the Linux I/O
syscalls. The term syscall means something very precise, and
__wrap_read is not a syscall. It gets ping-pong because it calls
RTDM I/O. But if you call directly Linux I/O syscall, with say
__real_read, you do not get ping-pong. Linux I/O syscall work as
they have always have: they require xenomai threads to run in
secondary mode and will cause them to switch to secondary mode to
handle the syscall.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 17:51, Gilles Chanteperdrix wrote:
>> Sorry, there *is*: Shadowed thread (anything created by wrapped
>> pthread_create) calls, say, read() on some Linux file descriptor, read()
>> is wrapped, first probes the call on RTDM, which means migration to RT
>> (for currently relaxed threads, like SCHED_WEAK), no RTDM match in the
>> kernel, and finally the migration to NRT in order to do the Linux read()
>> syscall. That didn't happen with the original design.
> 
> The wrapped read does not get ping-pong when calling the Linux I/O
> syscalls. The term syscall means something very precise, and
> __wrap_read is not a syscall. It gets ping-pong because it calls
> RTDM I/O. But if you call directly Linux I/O syscall, with say
> __real_read, you do not get ping-pong. Linux I/O syscall work as
> they have always have: they require xenomai threads to run in
> secondary mode and will cause them to switch to secondary mode to
> handle the syscall.

__real_* are a non-portable workarounds for special cases. It's not what
Xenomai developers are supposed to use in their applications.

If it were, we could also give up on wrapping and have rt_task_create
etc. again, ie. separate APIs for RT and non-RT.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Tue, Jun 14, 2016 at 06:03:20PM +0200, Jan Kiszka wrote:
> On 2016-06-14 17:51, Gilles Chanteperdrix wrote:
> >> Sorry, there *is*: Shadowed thread (anything created by wrapped
> >> pthread_create) calls, say, read() on some Linux file descriptor, read()
> >> is wrapped, first probes the call on RTDM, which means migration to RT
> >> (for currently relaxed threads, like SCHED_WEAK), no RTDM match in the
> >> kernel, and finally the migration to NRT in order to do the Linux read()
> >> syscall. That didn't happen with the original design.
> > 
> > The wrapped read does not get ping-pong when calling the Linux I/O
> > syscalls. The term syscall means something very precise, and
> > __wrap_read is not a syscall. It gets ping-pong because it calls
> > RTDM I/O. But if you call directly Linux I/O syscall, with say
> > __real_read, you do not get ping-pong. Linux I/O syscall work as
> > they have always have: they require xenomai threads to run in
> > secondary mode and will cause them to switch to secondary mode to
> > handle the syscall.
> 
> __real_* are a non-portable workarounds for special cases. It's not what
> Xenomai developers are supposed to use in their applications.

My argument is that you do not get ping-pong with Linux I/O
syscalls. And whether __real_* is portable or not does not change
that fact.

> 
> If it were, we could also give up on wrapping and have rt_task_create
> etc. again, ie. separate APIs for RT and non-RT.

Well, actually, I have been thinking about doing exactly that when
not wrapping: when the application has to call __RT(read) to call
Xenomai read, there is no reason for this call to fall back to Linux
read call. The idea came to late, but who knows, I may use it for a
later version.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 18:12, Gilles Chanteperdrix wrote:
> On Tue, Jun 14, 2016 at 06:03:20PM +0200, Jan Kiszka wrote:
>> On 2016-06-14 17:51, Gilles Chanteperdrix wrote:
>>>> Sorry, there *is*: Shadowed thread (anything created by wrapped
>>>> pthread_create) calls, say, read() on some Linux file descriptor, read()
>>>> is wrapped, first probes the call on RTDM, which means migration to RT
>>>> (for currently relaxed threads, like SCHED_WEAK), no RTDM match in the
>>>> kernel, and finally the migration to NRT in order to do the Linux read()
>>>> syscall. That didn't happen with the original design.
>>>
>>> The wrapped read does not get ping-pong when calling the Linux I/O
>>> syscalls. The term syscall means something very precise, and
>>> __wrap_read is not a syscall. It gets ping-pong because it calls
>>> RTDM I/O. But if you call directly Linux I/O syscall, with say
>>> __real_read, you do not get ping-pong. Linux I/O syscall work as
>>> they have always have: they require xenomai threads to run in
>>> secondary mode and will cause them to switch to secondary mode to
>>> handle the syscall.
>>
>> __real_* are a non-portable workarounds for special cases. It's not what
>> Xenomai developers are supposed to use in their applications.
> 
> My argument is that you do not get ping-pong with Linux I/O
> syscalls. And whether __real_* is portable or not does not change
> that fact.

...or by not doing eager migration. The failing RTDM syscall is not for
free either, but it's a different order of magnitude (if not two)
compared to the current execution flow.

> 
>>
>> If it were, we could also give up on wrapping and have rt_task_create
>> etc. again, ie. separate APIs for RT and non-RT.
> 
> Well, actually, I have been thinking about doing exactly that when
> not wrapping: when the application has to call __RT(read) to call
> Xenomai read, there is no reason for this call to fall back to Linux
> read call. The idea came to late, but who knows, I may use it for a
> later version.

Sure, such ideas existed a long time ago already, but then we rather
moved towards seamless integration with Linux and easy porting to/from
Xenomai. If basic things like I/O already requires special care, then we
can indeed go back directly to requiring explicit call tagging. Not
portable either, though a tag like __RT() can be more easily defined
away on native real-time.

That said, such a requirement will not make developers of existing
large, complex, layered applications happy. Like in our case.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Tue, Jun 14, 2016 at 06:25:43PM +0200, Jan Kiszka wrote:
> On 2016-06-14 18:12, Gilles Chanteperdrix wrote:
> > On Tue, Jun 14, 2016 at 06:03:20PM +0200, Jan Kiszka wrote:
> >> On 2016-06-14 17:51, Gilles Chanteperdrix wrote:
> >>>> Sorry, there *is*: Shadowed thread (anything created by wrapped
> >>>> pthread_create) calls, say, read() on some Linux file descriptor, read()
> >>>> is wrapped, first probes the call on RTDM, which means migration to RT
> >>>> (for currently relaxed threads, like SCHED_WEAK), no RTDM match in the
> >>>> kernel, and finally the migration to NRT in order to do the Linux read()
> >>>> syscall. That didn't happen with the original design.
> >>>
> >>> The wrapped read does not get ping-pong when calling the Linux I/O
> >>> syscalls. The term syscall means something very precise, and
> >>> __wrap_read is not a syscall. It gets ping-pong because it calls
> >>> RTDM I/O. But if you call directly Linux I/O syscall, with say
> >>> __real_read, you do not get ping-pong. Linux I/O syscall work as
> >>> they have always have: they require xenomai threads to run in
> >>> secondary mode and will cause them to switch to secondary mode to
> >>> handle the syscall.
> >>
> >> __real_* are a non-portable workarounds for special cases. It's not what
> >> Xenomai developers are supposed to use in their applications.
> > 
> > My argument is that you do not get ping-pong with Linux I/O
> > syscalls. And whether __real_* is portable or not does not change
> > that fact.
> 
> ...or by not doing eager migration.

Once again: there is no eager migration to primary mode for Linux
I/O syscalls, this would not make any sense. Only eager migration to
secondary mode, and this has always been like that, nothing changed
with 3.x.

> > Well, actually, I have been thinking about doing exactly that when
> > not wrapping: when the application has to call __RT(read) to call
> > Xenomai read, there is no reason for this call to fall back to Linux
> > read call. The idea came to late, but who knows, I may use it for a
> > later version.
> 
> Sure, such ideas existed a long time ago already, but then we rather
> moved towards seamless integration with Linux and easy porting to/from
> Xenomai. If basic things like I/O already requires special care, then we
> can indeed go back directly to requiring explicit call tagging. Not
> portable either, though a tag like __RT() can be more easily defined
> away on native real-time.
> 
> That said, such a requirement will not make developers of existing
> large, complex, layered applications happy. Like in our case.

I have some experience porting POSIX applications to Xenomai, and
when I did it I knew precisely at each point in the code whether the
underlying thread was a Xenomai thread, or a Linux thread. It looks
to me like a pre-requisite for programming applications based on
dual-kernels. So, when you say "large, complex, layered
application", I just hear "badly designed". The fallback in the
"wrapped read" is to allow plain Linux threads to be able to
continue to use read, not for Xenomai threads running in secondary
mode to have a high-performance Linux read, this use of read is a
corner case, and getting that corner case to work like it worked in
2.x had really bad side effects. But this is not the point of my
e-mail at all. I will let Philippe work with you on this issue.

The original point of my mail was that your assertion that "Linux
I/O syscalls cause ping pong" is false. How Linux syscalls work, I/O
or otherwise, with Xenomai, has always been the same.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 18:42, Gilles Chanteperdrix wrote:
> The original point of my mail was that your assertion that "Linux
> I/O syscalls cause ping pong" is false. How Linux syscalls work, I/O
> or otherwise, with Xenomai, has always been the same.

Yes, s/syscall/library call/ if you want to be that precise. No one
calls syscalls directly from the application, but the standard POSIX
function we are talking about are not doing much more than that. That's
what matters.

Jan

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 17:23, Philippe Gerum wrote:
> Restoring the original behavior unconditionally would not be a fix but
> only a work-around for your own issue. Finding a better way acceptable
> to all parties is on my todo list for the upcoming 3.0.3.

An alternative design to a plain revert of the current->conforming
switch could be to enhance conforming to take the scheduling class into
account: SCHED_WEAK and SCHED_OTHER should have a NRT as conforming
domain while real-time scheduling classes obviously target RT. But I
didn't check yet what side effects that may have, nor if there could be
relevant impact on syscall performance (unlikely, though).

Jan

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 06/14/2016 05:27 PM, Jan Kiszka wrote:
> On 2016-06-14 17:23, Philippe Gerum wrote:
>> On 06/14/2016 05:09 PM, Jan Kiszka wrote:
>>> On 2016-05-13 17:32, Jan Kiszka wrote:
>>>> On 2016-05-13 15:38, Philippe Gerum wrote:
>>>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>>>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>>>>>> or with a simple
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> default:
>>>>>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>>>>>> cases:
>>>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>>>>>> applies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, you need at least
>>>>>>>>>>>>>
>>>>>>>>>>>>> read_nrt()
>>>>>>>>>>>>> {
>>>>>>>>>>>>> 	if (rt_capable)
>>>>>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>>>>>
>>>>>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>>>>>> switching now.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>>>>>> eventually do depending on the calling context.
>>>>>>>>>>
>>>>>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>>>>>> There is no such problem. There only Analogy.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>>>>>> perfectly illustrates why messing with the call mode is not the
>>>>>>>>> application's business.
>>>>>>>>
>>>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>>>>>> same handler for both service_rt and service_nrt. That didn't change
>>>>>>>> with any switching strategy adjustment. It can't as long as there are
>>>>>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>>>>>> unable to filter for the service in the background. We really need to
>>>>>>>> differentiate here.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>>>>>> when the switch happens. It's pure optimization.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>>>>>> sketch the application code that would require __xn_exec_current for
>>>>>>>>> RTDM calls.
>>>>>>>>
>>>>>>>> You cut the more interesting case (migration ping-pong when calling
>>>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>>>>>> answer this.
>>>>>>>>
>>>>>>>
>>>>>>> I'm not ignoring the question, I have been postponing the answer until I
>>>>>>> understand why the application could be put in a situation making this
>>>>>>> migration a problem, and whether another approach would exist for
>>>>>>> solving that problem within the current scheme.
>>>>>>
>>>>>> These two scenarios are unrelated: this migration issue would still be
>>>>>> there even if we solved the one below via a different application/driver
>>>>>> design.
>>>>>>
>>>>>
>>>>> Which starts to be an issue only because the caller is a Cobalt shadow
>>>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
>>>>> operation very frequently. For this reason, those two scenarii are very
>>>>> much related.
>>>>
>>>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
>>>> enforce non-shadow thread creation, all are shadowed in a Xenomai
>>>> application, thus are affected. However, asking our users to user
>>>> __real_pthread_create extensively may not lead to the desired portable
>>>> designs.
>>>>
>>>>>
>>>>>>>
>>>>>>>> But let's go to our case:
>>>>>>>>
>>>>>>>> We have a non-blocking service in the driver, the classic case of
>>>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>>>>>> directly. Think of some kind of register access that requires low-level
>>>>>>>> synchronization with other threads and interrupt handlers. That service
>>>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>>>>>> (some thousand times per second). The RT threads are obviously on the
>>>>>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>>>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>>>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>>>>>> contributing to a higher system load, in the worst case overload.
>>>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>>>>>> well. Makes sense?
>>>>>>>>
>>>>>>>
>>>>>>> I understand the conflict with the "rt-always-has-precedence" rule
>>>>>>> implemented by the conforming state, then I have another question:
>>>>>>>
>>>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>>>>>> mainly operating from the Linux space but still needs to synchronize
>>>>>>> with the rt side at some point, which kind of high frequency interaction
>>>>>>> with the rt side is this?
>>>>>>>
>>>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>>>>>> waiting for rt events, something else?
>>>>>>>
>>>>>>
>>>>>> There synchronization need is first of all only on the hardware access
>>>>>> (thus inside the driver), not necessarily at application level. In fact,
>>>>>> there are even scenarios where you only want to exploit the driver as
>>>>>> permission checker on privileged resource accesses (userspace shall only
>>>>>> access certain MMIO registers in a page, thus the driver acts as
>>>>>> gatekeeper). Then there could be no synchronization at all but still the
>>>>>> need to provide migration-free accesses.
>>>>>>
>>>>>
>>>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
>>>>>
>>>>> However I still don't get which benefit your caller has in undergoing
>>>>> the SCHED_WEAK policy - which implies that it has to share
>>>>> synchronization points with Cobalt - compared to running as a regular
>>>>> (glibc) thread, under whichever policy that could fit?
>>>>
>>>> See above: it's additional, non-portable instrumentation of your code to
>>>> tag non-shadowed threads. And then you may easily run into troubles in
>>>> larger, layered application designs that a non-shadowed thread will
>>>> still need a blocking Xenomai service, e.g. via some hidden dependency.
>>>>
>>>>>
>>>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
>>>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
>>>>
>>>> I don't have these details at hand, but let's consider a large layered
>>>> application that also does significant work against Linux APIs during
>>>> runtime. You can't always enforce the complete separation. Because if
>>>> you can, you could also move the non-RT part into a separate process
>>>> that has nothing to do with Xenomai.
>>>>
>>>> We promote the transparency of the Xenomai POSIX interface, and that
>>>> should not make the usage of non-Xenomai services needlessly expensive
>>>> or require extensive non-portable tagging via __real_ prefixes.
>>>>
>>>
>>> Ping on this still open topic (will now have to introduce a local patch
>>> that restores the original behaviour). Can we resolve the issue upstream
>>> as well?
>>>
>>
>> Restoring the original behavior unconditionally would not be a fix but
>> only a work-around for your own issue. Finding a better way acceptable
>> to all parties is on my todo list for the upcoming 3.0.3.
> 
> We don't have all the issues, as I pointed out. It is a significant
> deficit of current Xenomai that you now have to create non-Xenomai
> threads explicitly (__real_pthread_create) in order to use Linux I/O
> syscalls efficiently (because of the otherwise enforces migration
> ping-pong). We didn't have that problem with the original design.
> 

Again, your application has a problem: it expects a real-time API to
operate as a non real-time API. This is wrong, really. The __real prefix
is a selector to pick the non-conforming API when symbol wrapping is in
effect, nothing more.

You are clearly misinterpreting what Xenomai is about in that case, it
is about providing a subset of the real-time POSIX API which can be
seamlessly used conforming to the standard, for real-time purposes.

It is definitely not about providing a way to run a non real-time thread
by calling the real-time API. This is why your application sees mode
switches, otherwise it would not. Please let's move on away from this
flawed argument, it does not help thinking constructively.

I'm ok to help finding a solution, but please don't try to sell me such
broken idea as a de facto Xenomai standard. It has never been so.

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Jan Kiszka]

On 2016-06-14 21:48, Philippe Gerum wrote:
> On 06/14/2016 05:27 PM, Jan Kiszka wrote:
>> On 2016-06-14 17:23, Philippe Gerum wrote:
>>> On 06/14/2016 05:09 PM, Jan Kiszka wrote:
>>>> On 2016-05-13 17:32, Jan Kiszka wrote:
>>>>> On 2016-05-13 15:38, Philippe Gerum wrote:
>>>>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>>>>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>>>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>>>>>>> or with a simple
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> default:
>>>>>>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>>>>>>> cases:
>>>>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>>>>>>> applies.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, you need at least
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> read_nrt()
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> 	if (rt_capable)
>>>>>>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>>>>>>> switching now.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>>>>>>> eventually do depending on the calling context.
>>>>>>>>>>>
>>>>>>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>>>>>>> There is no such problem. There only Analogy.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>>>>>>> perfectly illustrates why messing with the call mode is not the
>>>>>>>>>> application's business.
>>>>>>>>>
>>>>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>>>>>>> same handler for both service_rt and service_nrt. That didn't change
>>>>>>>>> with any switching strategy adjustment. It can't as long as there are
>>>>>>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>>>>>>> unable to filter for the service in the background. We really need to
>>>>>>>>> differentiate here.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>>>>>>> when the switch happens. It's pure optimization.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>>>>>>> sketch the application code that would require __xn_exec_current for
>>>>>>>>>> RTDM calls.
>>>>>>>>>
>>>>>>>>> You cut the more interesting case (migration ping-pong when calling
>>>>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>>>>>>> answer this.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not ignoring the question, I have been postponing the answer until I
>>>>>>>> understand why the application could be put in a situation making this
>>>>>>>> migration a problem, and whether another approach would exist for
>>>>>>>> solving that problem within the current scheme.
>>>>>>>
>>>>>>> These two scenarios are unrelated: this migration issue would still be
>>>>>>> there even if we solved the one below via a different application/driver
>>>>>>> design.
>>>>>>>
>>>>>>
>>>>>> Which starts to be an issue only because the caller is a Cobalt shadow
>>>>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
>>>>>> operation very frequently. For this reason, those two scenarii are very
>>>>>> much related.
>>>>>
>>>>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
>>>>> enforce non-shadow thread creation, all are shadowed in a Xenomai
>>>>> application, thus are affected. However, asking our users to user
>>>>> __real_pthread_create extensively may not lead to the desired portable
>>>>> designs.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>>> But let's go to our case:
>>>>>>>>>
>>>>>>>>> We have a non-blocking service in the driver, the classic case of
>>>>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>>>>>>> directly. Think of some kind of register access that requires low-level
>>>>>>>>> synchronization with other threads and interrupt handlers. That service
>>>>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>>>>>>> (some thousand times per second). The RT threads are obviously on the
>>>>>>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>>>>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>>>>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>>>>>>> contributing to a higher system load, in the worst case overload.
>>>>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>>>>>>> well. Makes sense?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I understand the conflict with the "rt-always-has-precedence" rule
>>>>>>>> implemented by the conforming state, then I have another question:
>>>>>>>>
>>>>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>>>>>>> mainly operating from the Linux space but still needs to synchronize
>>>>>>>> with the rt side at some point, which kind of high frequency interaction
>>>>>>>> with the rt side is this?
>>>>>>>>
>>>>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>>>>>>> waiting for rt events, something else?
>>>>>>>>
>>>>>>>
>>>>>>> There synchronization need is first of all only on the hardware access
>>>>>>> (thus inside the driver), not necessarily at application level. In fact,
>>>>>>> there are even scenarios where you only want to exploit the driver as
>>>>>>> permission checker on privileged resource accesses (userspace shall only
>>>>>>> access certain MMIO registers in a page, thus the driver acts as
>>>>>>> gatekeeper). Then there could be no synchronization at all but still the
>>>>>>> need to provide migration-free accesses.
>>>>>>>
>>>>>>
>>>>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
>>>>>>
>>>>>> However I still don't get which benefit your caller has in undergoing
>>>>>> the SCHED_WEAK policy - which implies that it has to share
>>>>>> synchronization points with Cobalt - compared to running as a regular
>>>>>> (glibc) thread, under whichever policy that could fit?
>>>>>
>>>>> See above: it's additional, non-portable instrumentation of your code to
>>>>> tag non-shadowed threads. And then you may easily run into troubles in
>>>>> larger, layered application designs that a non-shadowed thread will
>>>>> still need a blocking Xenomai service, e.g. via some hidden dependency.
>>>>>
>>>>>>
>>>>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
>>>>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
>>>>>
>>>>> I don't have these details at hand, but let's consider a large layered
>>>>> application that also does significant work against Linux APIs during
>>>>> runtime. You can't always enforce the complete separation. Because if
>>>>> you can, you could also move the non-RT part into a separate process
>>>>> that has nothing to do with Xenomai.
>>>>>
>>>>> We promote the transparency of the Xenomai POSIX interface, and that
>>>>> should not make the usage of non-Xenomai services needlessly expensive
>>>>> or require extensive non-portable tagging via __real_ prefixes.
>>>>>
>>>>
>>>> Ping on this still open topic (will now have to introduce a local patch
>>>> that restores the original behaviour). Can we resolve the issue upstream
>>>> as well?
>>>>
>>>
>>> Restoring the original behavior unconditionally would not be a fix but
>>> only a work-around for your own issue. Finding a better way acceptable
>>> to all parties is on my todo list for the upcoming 3.0.3.
>>
>> We don't have all the issues, as I pointed out. It is a significant
>> deficit of current Xenomai that you now have to create non-Xenomai
>> threads explicitly (__real_pthread_create) in order to use Linux I/O
>> syscalls efficiently (because of the otherwise enforces migration
>> ping-pong). We didn't have that problem with the original design.
>>
> 
> Again, your application has a problem: it expects a real-time API to
> operate as a non real-time API. This is wrong, really. The __real prefix
> is a selector to pick the non-conforming API when symbol wrapping is in
> effect, nothing more.
> 
> You are clearly misinterpreting what Xenomai is about in that case, it
> is about providing a subset of the real-time POSIX API which can be
> seamlessly used conforming to the standard, for real-time purposes.
> 
> It is definitely not about providing a way to run a non real-time thread
> by calling the real-time API. This is why your application sees mode
> switches, otherwise it would not. Please let's move on away from this
> flawed argument, it does not help thinking constructively.
> 
> I'm ok to help finding a solution, but please don't try to sell me such
> broken idea as a de facto Xenomai standard. It has never been so.
> 

I don't need to sell this "idea" to you. It is not my idea. It is a
typical valid use case of Xenomai - colocated RT and non-RT workload in
the same process (that is a value of Xenomai!) - in a typical valid
scenario - porting existing large code bases over Xenomai and keeping
them portable to other POSIX environment. Calling this "broken" is not
appropriate.

But let's focus on the solution space. I expanded it already.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 06/14/2016 07:13 PM, Jan Kiszka wrote:
> On 2016-06-14 17:23, Philippe Gerum wrote:
>> Restoring the original behavior unconditionally would not be a fix but
>> only a work-around for your own issue. Finding a better way acceptable
>> to all parties is on my todo list for the upcoming 3.0.3.
> 
> An alternative design to a plain revert of the current->conforming
> switch could be to enhance conforming to take the scheduling class into
> account: SCHED_WEAK and SCHED_OTHER should have a NRT as conforming
> domain while real-time scheduling classes obviously target RT. But I
> didn't check yet what side effects that may have, nor if there could be
> relevant impact on syscall performance (unlikely, though).
> 

That would not make more sense: SCHED_WEAK/OTHER is about having
_Xenomai_ threads interfacing with the _Xenomai_ system, without
competing with real-time threads priority-wise. But this is still about
running Xenomai services, for synchronizing on real-time events or
receiving messages from the real-time side. Basically, this requires to
run from primary mode.

We need a scheme that:

- does not allow user-space to select the RTDM handler being called only
by manipulating its runtime mode, because this would certainly lead to a
massive mess for the application.

- does not ask the driver to deal with mode detection on each and every
(ioctl) request it implements. With the conforming mode applicable to
all RTDM I/O calls, only the _rt side should return ENOSYS to hand over
some requests to the nrt side. When a request cannot be processed from
nrt, then we know the request is wrong/invalid, returning ENOSYS makes
no sense.

- possibly restrict the former "current" behavior to some ioctl() calls,
as specified by the driver, not decided by the application.

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Philippe Gerum]

On 06/14/2016 10:03 PM, Jan Kiszka wrote:
> On 2016-06-14 21:48, Philippe Gerum wrote:
>> On 06/14/2016 05:27 PM, Jan Kiszka wrote:
>>> On 2016-06-14 17:23, Philippe Gerum wrote:
>>>> On 06/14/2016 05:09 PM, Jan Kiszka wrote:
>>>>> On 2016-05-13 17:32, Jan Kiszka wrote:
>>>>>> On 2016-05-13 15:38, Philippe Gerum wrote:
>>>>>>> On 05/13/2016 07:54 AM, Jan Kiszka wrote:
>>>>>>>> On 2016-05-13 00:26, Philippe Gerum wrote:
>>>>>>>>> On 05/12/2016 09:27 PM, Jan Kiszka wrote:
>>>>>>>>>> On 2016-05-12 21:08, Philippe Gerum wrote:
>>>>>>>>>>> On 05/12/2016 08:42 PM, Jan Kiszka wrote:
>>>>>>>>>>>> On 2016-05-12 20:35, Philippe Gerum wrote:
>>>>>>>>>>>>> On 05/12/2016 08:24 PM, Jan Kiszka wrote:
>>>>>>>>>>>>>> On 2016-05-12 20:20, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 07:17:15PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>> On 2016-05-12 19:12, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:59:04PM +0200, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:50:03PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>>> On 2016-05-12 18:31, Gilles Chanteperdrix wrote:
>>>>>>>>>>>>>>>>>>>> On Thu, May 12, 2016 at 06:06:16PM +0200, Jan Kiszka wrote:
>>>>>>>>>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> regarding commit bec5d0dd42 (rtdm: make syscalls conforming rather than
>>>>>>>>>>>>>>>>>>>>> current) - I remember a discussion on that topic, but I do not find its
>>>>>>>>>>>>>>>>>>>>> traces any more. Do you have a pointer
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In any case, I'm confronted with a use case for the old (Xenomai 2),
>>>>>>>>>>>>>>>>>>>>> lazy switching behaviour: lightweight, performance sensitive IOCTL
>>>>>>>>>>>>>>>>>>>>> services that can (and should) be called without any switching from both
>>>>>>>>>>>>>>>>>>>>> domains.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Why not using a plain linux driver? ioctl_nrt callbacks are
>>>>>>>>>>>>>>>>>>>> redundant with plain linux drivers.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Because that enforces the calling layer to either call the same service
>>>>>>>>>>>>>>>>>>> via a plain Linux device if the calling thread is currently relaxed or
>>>>>>>>>>>>>>>>>>> go for the RT device if the caller is in primary. Doable, but I would
>>>>>>>>>>>>>>>>>>> really like to avoid this pain for the users.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What were the arguments in favour of migrating threads to real-time first?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I currently see the real need only for IOCTLs, but the question is then
>>>>>>>>>>>>>>>>>>>>> if we shouldn't go back to "__xn_exec_current" in all RTDM cases to
>>>>>>>>>>>>>>>>>>>>> avoid unwanted migration costs (which are significantly higher than
>>>>>>>>>>>>>>>>>>>>> syscall restarts).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I do not find commit bec5d0dd42 in xenomai-2.6 git tree, and I do
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Xenomai 2 is still following the lazy scheme - we reverted that commit
>>>>>>>>>>>>>>>>>>> later on in 7df0c1d96b. Xenomai 3 changed it again with the commit above.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> not remember merging this. However I find commit
>>>>>>>>>>>>>>>>>>>> 13bfdd477ab880499d2e8f3b82c49ef4d2cccff0 from 2010 which seems to
>>>>>>>>>>>>>>>>>>>> explain the reason pretty clear.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> At the time of the discussion we had concluded that it was the way
>>>>>>>>>>>>>>>>>>>> to go. With __xn_exec_current you may enter the ioctl_rt callback
>>>>>>>>>>>>>>>>>>>> from secondary domain, which is counter-intuitive, error-prone, and
>>>>>>>>>>>>>>>>>>>> forces you to cripple driver code for checks for the current domain.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Nope, normal drivers are not affected as they just implement those
>>>>>>>>>>>>>>>>>>> services in the respective mode they want to support there and have a
>>>>>>>>>>>>>>>>>>> simple -ENOSYS for the rest (explicitly in IOCTLs or implicitly by
>>>>>>>>>>>>>>>>>>> leaving out the implementation of the counterpart handler).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes, I got mixed up trying to remember. I think the crux of the
>>>>>>>>>>>>>>>>>> problem is that if a thread running in primary mode gets
>>>>>>>>>>>>>>>>>> (temporarily) switched to secondary mode by gdb, the ioctl_nrt
>>>>>>>>>>>>>>>>>> handler gets invoked, which is almost certainly the wrong thing to
>>>>>>>>>>>>>>>>>> do. You want the thread to migrate to primary mode to execute
>>>>>>>>>>>>>>>>>> ioctl_rt, which __xn_exec_conforming achieves. Otherwise running an
>>>>>>>>>>>>>>>>>> application in gdb causes the application to behave differently.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And trying and avoiding this issue indeed cripple codes with checks
>>>>>>>>>>>>>>>>> for rtdm_in_rt_context:
>>>>>>>>>>>>>>>>> https://git.xenomai.org/xenomai-2.6.git/tree/ksrc/drivers/analogy/rtdm_interface.c#n194
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I don't remember details here, but this is a special case: The driver
>>>>>>>>>>>>>>>> provides also read_nrt - is that really useful for Analogy?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In most cases, you are fine with not providing the nrt (or rt) handler,
>>>>>>>>>>>>>>>> or with a simple
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> default:
>>>>>>>>>>>>>>>> 	return -ENOSYS;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> in your ioctl dispatcher.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You are missing the point: if you enter read_nrt, there are two
>>>>>>>>>>>>>>> cases:
>>>>>>>>>>>>>>> - either the thread is real-time capable and has been relaxed by gdb
>>>>>>>>>>>>>>> and you want to switch to read_rt for the reasons I already
>>>>>>>>>>>>>>> explained, in that case, you must return -ENOSYS;
>>>>>>>>>>>>>>> - or the thread is not real-time capable and the nrt handler
>>>>>>>>>>>>>>> applies.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, you need at least
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> read_nrt()
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> 	if (rt_capable)
>>>>>>>>>>>>>>> 	     return -ENOSYS;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 	/* Do the normal case here */
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now tell me how many drivers have read_nrt, write_nrt? 1 in-tree.
>>>>>>>>>>>>>> recvmsg_nrt, sendmsg_nrt? 0 in-tree. Analogy is special (still like to
>>>>>>>>>>>>>> understand why, though). And having some special code in the exceptional
>>>>>>>>>>>>>> case is probably better then the side effects we get from eagerly
>>>>>>>>>>>>>> switching now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry, that is exactly the opposite: your use case is exceptional and I
>>>>>>>>>>>>> believe is wrong. The normal use case is the one that does not ask the
>>>>>>>>>>>>> user to track the current mode for knowing what any random driver would
>>>>>>>>>>>>> eventually do depending on the calling context.
>>>>>>>>>>>>
>>>>>>>>>>>> You still miss the point that this is not required in 99% of the cases.
>>>>>>>>>>>> There is no such problem. There only Analogy.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm not discussing Analogy at all, those drivers are still biased by the
>>>>>>>>>>> legacy 2.x logic for dealing with modes and need fixing. I have never
>>>>>>>>>>> been convinced by the reasoning behind rtdm_in_rt_context(), which
>>>>>>>>>>> perfectly illustrates why messing with the call mode is not the
>>>>>>>>>>> application's business.
>>>>>>>>>>
>>>>>>>>>> You still need rtdm_in_rt_context() for the (rare) case of having the
>>>>>>>>>> same handler for both service_rt and service_nrt. That didn't change
>>>>>>>>>> with any switching strategy adjustment. It can't as long as there are
>>>>>>>>>> services behind a syscall that may handle any mode, thus that syscall is
>>>>>>>>>> unable to filter for the service in the background. We really need to
>>>>>>>>>> differentiate here.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Every driver must ensure that a service is only exposed to users in the
>>>>>>>>>>>> right mode. That is a functional requirement, and drivers that fail to
>>>>>>>>>>>> do so only work by chance (thus with the restricted workload they are
>>>>>>>>>>>> tested against). If that is fulfilled, it doesn't matter to the driver
>>>>>>>>>>>> when the switch happens. It's pure optimization.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> You don't seem to get my point either. Let's proceed differently, please
>>>>>>>>>>> sketch the application code that would require __xn_exec_current for
>>>>>>>>>>> RTDM calls.
>>>>>>>>>>
>>>>>>>>>> You cut the more interesting case (migration ping-pong when calling
>>>>>>>>>> non-RT drivers from relaxed threads), and I hope you will not forget to
>>>>>>>>>> answer this.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm not ignoring the question, I have been postponing the answer until I
>>>>>>>>> understand why the application could be put in a situation making this
>>>>>>>>> migration a problem, and whether another approach would exist for
>>>>>>>>> solving that problem within the current scheme.
>>>>>>>>
>>>>>>>> These two scenarios are unrelated: this migration issue would still be
>>>>>>>> there even if we solved the one below via a different application/driver
>>>>>>>> design.
>>>>>>>>
>>>>>>>
>>>>>>> Which starts to be an issue only because the caller is a Cobalt shadow
>>>>>>> undergoing the SCHED_WEAK policy, calling a RTDM driver for a non-rt
>>>>>>> operation very frequently. For this reason, those two scenarii are very
>>>>>>> much related.
>>>>>>
>>>>>> Not SCHED_WEAK, but being a shadow in the first place. Unless you
>>>>>> enforce non-shadow thread creation, all are shadowed in a Xenomai
>>>>>> application, thus are affected. However, asking our users to user
>>>>>> __real_pthread_create extensively may not lead to the desired portable
>>>>>> designs.
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>>> But let's go to our case:
>>>>>>>>>>
>>>>>>>>>> We have a non-blocking service in the driver, the classic case of
>>>>>>>>>> accessing a privileged resource that userspace can't or shouldn't touch
>>>>>>>>>> directly. Think of some kind of register access that requires low-level
>>>>>>>>>> synchronization with other threads and interrupt handlers. That service
>>>>>>>>>> is called by both RT and non-RT threads (SCHED_WEAK) at higher frequency
>>>>>>>>>> (some thousand times per second). The RT threads are obviously on the
>>>>>>>>>> time critical path, must not migrate, and that can be achieved perfectly
>>>>>>>>>> already by providing that service under ioctl_rt. The non-RT threads
>>>>>>>>>> could be migrated to RT, but then they would pay an unneeded price,
>>>>>>>>>> contributing to a higher system load, in the worst case overload.
>>>>>>>>>> Therefore, the very same service shall be provided under ioctl_nrt as
>>>>>>>>>> well. Makes sense?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I understand the conflict with the "rt-always-has-precedence" rule
>>>>>>>>> implemented by the conforming state, then I have another question:
>>>>>>>>>
>>>>>>>>> assuming the nrt thread undergoes the SCHED_WEAK policy because it is
>>>>>>>>> mainly operating from the Linux space but still needs to synchronize
>>>>>>>>> with the rt side at some point, which kind of high frequency interaction
>>>>>>>>> with the rt side is this?
>>>>>>>>>
>>>>>>>>> Sharing some resource requiring mutual exclusion via a Cobalt synchro,
>>>>>>>>> waiting for rt events, something else?
>>>>>>>>>
>>>>>>>>
>>>>>>>> There synchronization need is first of all only on the hardware access
>>>>>>>> (thus inside the driver), not necessarily at application level. In fact,
>>>>>>>> there are even scenarios where you only want to exploit the driver as
>>>>>>>> permission checker on privileged resource accesses (userspace shall only
>>>>>>>> access certain MMIO registers in a page, thus the driver acts as
>>>>>>>> gatekeeper). Then there could be no synchronization at all but still the
>>>>>>>> need to provide migration-free accesses.
>>>>>>>>
>>>>>>>
>>>>>>> I get the idea of the resource gatekeeper, which does make a lot of sense.
>>>>>>>
>>>>>>> However I still don't get which benefit your caller has in undergoing
>>>>>>> the SCHED_WEAK policy - which implies that it has to share
>>>>>>> synchronization points with Cobalt - compared to running as a regular
>>>>>>> (glibc) thread, under whichever policy that could fit?
>>>>>>
>>>>>> See above: it's additional, non-portable instrumentation of your code to
>>>>>> tag non-shadowed threads. And then you may easily run into troubles in
>>>>>> larger, layered application designs that a non-shadowed thread will
>>>>>> still need a blocking Xenomai service, e.g. via some hidden dependency.
>>>>>>
>>>>>>>
>>>>>>> Leaving the non-RT ioctl call aside, which are those Cobalt calls the
>>>>>>> SCHED_WEAK thread needs to invoke for synchronizing with rt threads?
>>>>>>
>>>>>> I don't have these details at hand, but let's consider a large layered
>>>>>> application that also does significant work against Linux APIs during
>>>>>> runtime. You can't always enforce the complete separation. Because if
>>>>>> you can, you could also move the non-RT part into a separate process
>>>>>> that has nothing to do with Xenomai.
>>>>>>
>>>>>> We promote the transparency of the Xenomai POSIX interface, and that
>>>>>> should not make the usage of non-Xenomai services needlessly expensive
>>>>>> or require extensive non-portable tagging via __real_ prefixes.
>>>>>>
>>>>>
>>>>> Ping on this still open topic (will now have to introduce a local patch
>>>>> that restores the original behaviour). Can we resolve the issue upstream
>>>>> as well?
>>>>>
>>>>
>>>> Restoring the original behavior unconditionally would not be a fix but
>>>> only a work-around for your own issue. Finding a better way acceptable
>>>> to all parties is on my todo list for the upcoming 3.0.3.
>>>
>>> We don't have all the issues, as I pointed out. It is a significant
>>> deficit of current Xenomai that you now have to create non-Xenomai
>>> threads explicitly (__real_pthread_create) in order to use Linux I/O
>>> syscalls efficiently (because of the otherwise enforces migration
>>> ping-pong). We didn't have that problem with the original design.
>>>
>>
>> Again, your application has a problem: it expects a real-time API to
>> operate as a non real-time API. This is wrong, really. The __real prefix
>> is a selector to pick the non-conforming API when symbol wrapping is in
>> effect, nothing more.
>>
>> You are clearly misinterpreting what Xenomai is about in that case, it
>> is about providing a subset of the real-time POSIX API which can be
>> seamlessly used conforming to the standard, for real-time purposes.
>>
>> It is definitely not about providing a way to run a non real-time thread
>> by calling the real-time API. This is why your application sees mode
>> switches, otherwise it would not. Please let's move on away from this
>> flawed argument, it does not help thinking constructively.
>>
>> I'm ok to help finding a solution, but please don't try to sell me such
>> broken idea as a de facto Xenomai standard. It has never been so.
>>
> 
> I don't need to sell this "idea" to you. It is not my idea. It is a
> typical valid use case of Xenomai - colocated RT and non-RT workload in
> the same process (that is a value of Xenomai!

Correct. This means using the proper API for that. This is basically
what the dual kernel approach is all about: the real-time purpose
implies to use a separate API.

-- 
Philippe.

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Tue, Jun 14, 2016 at 10:11:30PM +0200, Philippe Gerum wrote:
> On 06/14/2016 07:13 PM, Jan Kiszka wrote:
> > On 2016-06-14 17:23, Philippe Gerum wrote:
> >> Restoring the original behavior unconditionally would not be a fix but
> >> only a work-around for your own issue. Finding a better way acceptable
> >> to all parties is on my todo list for the upcoming 3.0.3.
> > 
> > An alternative design to a plain revert of the current->conforming
> > switch could be to enhance conforming to take the scheduling class into
> > account: SCHED_WEAK and SCHED_OTHER should have a NRT as conforming
> > domain while real-time scheduling classes obviously target RT. But I
> > didn't check yet what side effects that may have, nor if there could be
> > relevant impact on syscall performance (unlikely, though).
> > 
> 
> That would not make more sense: SCHED_WEAK/OTHER is about having
> _Xenomai_ threads interfacing with the _Xenomai_ system, without
> competing with real-time threads priority-wise. But this is still about
> running Xenomai services, for synchronizing on real-time events or
> receiving messages from the real-time side. Basically, this requires to
> run from primary mode.
> 
> We need a scheme that:
> 
> - does not allow user-space to select the RTDM handler being called only
> by manipulating its runtime mode, because this would certainly lead to a
> massive mess for the application.
> 
> - does not ask the driver to deal with mode detection on each and every
> (ioctl) request it implements. With the conforming mode applicable to
> all RTDM I/O calls, only the _rt side should return ENOSYS to hand over
> some requests to the nrt side. When a request cannot be processed from
> nrt, then we know the request is wrong/invalid, returning ENOSYS makes
> no sense.
> 
> - possibly restrict the former "current" behavior to some ioctl() calls,
> as specified by the driver, not decided by the application.

I am not sure I have already proposed that, but I am going to try,
sorry if this is the second time. Since RTDM file descriptors are
plain Linux file descriptors, could not the ioctl callback of these
plain linux file descriptors give access to the ioctl_nrt RTDM
callback? This way, in the case which is a problem for Jan, he could
call __real_ioctl and be done with it.

The fact that __real is not portable is not really an issue, you can
wrap that in a macro that adds __real or not like the __RT macro
does.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] RTDM syscalls & switching

[Gilles Chanteperdrix]

On Tue, Jun 14, 2016 at 06:59:48PM +0200, Jan Kiszka wrote:
> On 2016-06-14 18:42, Gilles Chanteperdrix wrote:
> > The original point of my mail was that your assertion that "Linux
> > I/O syscalls cause ping pong" is false. How Linux syscalls work, I/O
> > or otherwise, with Xenomai, has always been the same.
> 
> Yes, s/syscall/library call/ if you want to be that precise. No one
> calls syscalls directly from the application, but the standard POSIX
> function we are talking about are not doing much more than that. That's
> what matters.

Yes and /Linux/Xenomai/ because these library calls are implemented
in Xenomai libraries, not in glibc. Whereas glibc library calls
branch almost unambiguously on Linux syscalls, Xenomai library calls
can either branch to Xenomai syscall or to Linux syscall. So,
calling Xenomai library calls "Linux I/O syscalls" is very
misleading.

-- 
					    Gilles.
https://click-hack.org