Towards Xenomai on Dovetail

Towards Xenomai on Dovetail

[Jan Kiszka]

Hi Philippe,

could you give a rough overview what areas would be just mapping API 
calls, what would be more different and what is simply missing in 
Dovetail in order to place Xenomai on top of it?

BTW, something like a folded Dovetail patch queue does not exist yet, 
does it? Would it make sense to create something like that, or is it 
still too early?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Towards Xenomai on Dovetail

[Philippe Gerum]

On 2/10/20 7:57 PM, Jan Kiszka wrote:
> Hi Philippe,
> 
> could you give a rough overview what areas would be just mapping API
> calls, what would be more different and what is simply missing in
> Dovetail in order to place Xenomai on top of it?

By order of trickiness:

- Dovetail assumes the companion core shares the same timekeeping bases
and unit with the common kernel, providing some help in doing so. The
core should switch to nanosecs, ktime_t internally. Typically, MMIO
clock sources which are not architected can be exported directly by the
vDSO (e.g. ARM with its truckload of different clock sources). IOW, no
more tsc-based timekeeping, CLOCK_HOST_REALTIME becomes useless as
clock_gettime() is usable from any context. This deeply affects the
Xenomai timer framework, and the way libcobalt reads time (no need for
tsc->ns conversion, the vDSO delivers nanosecs via clock_gettime()). In
the wake of such change, there would be no reason to keep the arith
support in Xenomai.

- kernel/cobalt/irq.c and related can go. Dovetail integrates the
so-called "out-of-band" IRQ management directly into the mainline genirq
logic. This means Xenomai can switch to the request/free_irq() interface
or related.

- kernel/cobalt/arch/*/ and related can go, almost entirely. There is
direct support for context switching including fpu management for
threads controlled by the companion (e.g. real-time) core. This means
that Xenomai's scheduler core should be adapted to use the available
dovetail interface for this. The workings are explained in the
description of the so-called "alternate scheduling" Dovetail implements [3].

- support for synchronous breakpoints would need to be reworked a bit,
around the ptrace()-related events Dovetail triggers (TASK_PTSTOP,
PTSTEP, PTCONT). As a consequence, the most ugly KEVT_SCHEDULE event is
gone.

- although this would not be a requirement for a port, you could reuse
the native VFS support for out-of-band I/O handling in character drivers
Dovetail brings, in order to drop the RTDM-specific file descriptor
management [2], simplifying things even further.

- the I-pipe tracer has gone in dovetail. ftrace's regular "function
tracer" is safe in all contexts, directly adds information related to
the IRQ pipelining logic. This means the trace* API in Xenomai would
have to be enabled in a different way.

- the APC interface can go entirely, along with the
ipipe_post_root_work() call, all this can be obtained using the regular
irq_work() API now.

- straightforward renames such as ipipe_spin_lock to hard_spin_lock etc.

Although there may be never enough of it, there is still a fair amount
of documentation already available for all of this at [1]. I'm gathering
the bits for writing a document which could be titled "How to write your
own Linux co-kernel on top of Dovetail". This needs time, I'm a slow writer.

The detailed note I sent you in July '18 about how Dovetail was
diverging from the I-pipe is still relevant for the most part. The
update logic for an irqchip to support interrupt pipelining has been
simplified compared to the description in this note though (no more
pipeline-specific handlers basically).

[1] https://evlproject.org/dovetail/
[2] https://evlproject.org/core/kernel-api/
[3] https://evlproject.org/dovetail/altsched/

> 
> BTW, something like a folded Dovetail patch queue does not exist yet,
> does it? Would it make sense to create something like that, or is it
> still too early?
> 

I could add this. Please note that Dovetail API stability across kernel
releases is not guaranteed, although I tend to minimize such changes
when this does not impede improving the code.

-- 
Philippe.

Re: Towards Xenomai on Dovetail

[Jan Kiszka]

On 18.02.20 18:36, Philippe Gerum wrote:
> On 2/10/20 7:57 PM, Jan Kiszka wrote:
>> Hi Philippe,
>>
>> could you give a rough overview what areas would be just mapping API
>> calls, what would be more different and what is simply missing in
>> Dovetail in order to place Xenomai on top of it?
> 
> By order of trickiness:
> 
> - Dovetail assumes the companion core shares the same timekeeping bases
> and unit with the common kernel, providing some help in doing so. The
> core should switch to nanosecs, ktime_t internally. Typically, MMIO
> clock sources which are not architected can be exported directly by the
> vDSO (e.g. ARM with its truckload of different clock sources). IOW, no
> more tsc-based timekeeping, CLOCK_HOST_REALTIME becomes useless as
> clock_gettime() is usable from any context. This deeply affects the
> Xenomai timer framework, and the way libcobalt reads time (no need for
> tsc->ns conversion, the vDSO delivers nanosecs via clock_gettime()). In
> the wake of such change, there would be no reason to keep the arith
> support in Xenomai.
> 
> - kernel/cobalt/irq.c and related can go. Dovetail integrates the
> so-called "out-of-band" IRQ management directly into the mainline genirq
> logic. This means Xenomai can switch to the request/free_irq() interface
> or related.
> 
> - kernel/cobalt/arch/*/ and related can go, almost entirely. There is
> direct support for context switching including fpu management for
> threads controlled by the companion (e.g. real-time) core. This means
> that Xenomai's scheduler core should be adapted to use the available
> dovetail interface for this. The workings are explained in the
> description of the so-called "alternate scheduling" Dovetail implements [3].
> 
> - support for synchronous breakpoints would need to be reworked a bit,
> around the ptrace()-related events Dovetail triggers (TASK_PTSTOP,
> PTSTEP, PTCONT). As a consequence, the most ugly KEVT_SCHEDULE event is
> gone.
> 
> - although this would not be a requirement for a port, you could reuse
> the native VFS support for out-of-band I/O handling in character drivers
> Dovetail brings, in order to drop the RTDM-specific file descriptor
> management [2], simplifying things even further.

One after the other, indeed. For RTDM, I will have to look into more 
details to asses the user-visible impact. Currently, your approach is 
not yet up-to-speed with RTDM, specifically due to a missing socket 
interface (RTnet, RT-CAN, ...).

> 
> - the I-pipe tracer has gone in dovetail. ftrace's regular "function
> tracer" is safe in all contexts, directly adds information related to
> the IRQ pipelining logic. This means the trace* API in Xenomai would
> have to be enabled in a different way.

I suppose you already enabled visualization of the oob-specific states 
(virtual interrupt flag, in-band / out-of-band context) in ftrace, 
right? The other reason I didn't suggest to drop the I-pipe function 
tracer in favor of the (already working) ftrace one is the "stop on 
highest latency" mode that you can model with it. But the former aspect 
was actually way more critical, to debug subtle ipipe state corruptions.

> 
> - the APC interface can go entirely, along with the
> ipipe_post_root_work() call, all this can be obtained using the regular
> irq_work() API now.
> 
> - straightforward renames such as ipipe_spin_lock to hard_spin_lock etc.
> 
> Although there may be never enough of it, there is still a fair amount
> of documentation already available for all of this at [1]. I'm gathering
> the bits for writing a document which could be titled "How to write your
> own Linux co-kernel on top of Dovetail". This needs time, I'm a slow writer.
> 
> The detailed note I sent you in July '18 about how Dovetail was
> diverging from the I-pipe is still relevant for the most part. The
> update logic for an irqchip to support interrupt pipelining has been
> simplified compared to the description in this note though (no more
> pipeline-specific handlers basically).
> 
> [1] https://evlproject.org/dovetail/
> [2] https://evlproject.org/core/kernel-api/
> [3] https://evlproject.org/dovetail/altsched/
> 

Thanks for the (updated) summery, will surely be helpful!

>>
>> BTW, something like a folded Dovetail patch queue does not exist yet,
>> does it? Would it make sense to create something like that, or is it
>> still too early?
>>
> 
> I could add this. Please note that Dovetail API stability across kernel
> releases is not guaranteed, although I tend to minimize such changes
> when this does not impede improving the code.
> 

I think it will help in explaining as well as porting the series. It's 
not urgent ATM, though - I plan to start over your latest revision.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Towards Xenomai on Dovetail

[Philippe Gerum]

On 2/19/20 1:48 PM, Jan Kiszka via Xenomai wrote:
> On 18.02.20 18:36, Philippe Gerum wrote:
>> On 2/10/20 7:57 PM, Jan Kiszka wrote:
>>> Hi Philippe,
>>>
>>> could you give a rough overview what areas would be just mapping API
>>> calls, what would be more different and what is simply missing in
>>> Dovetail in order to place Xenomai on top of it?
>>
>> By order of trickiness:
>>
>> - Dovetail assumes the companion core shares the same timekeeping bases
>> and unit with the common kernel, providing some help in doing so. The
>> core should switch to nanosecs, ktime_t internally. Typically, MMIO
>> clock sources which are not architected can be exported directly by the
>> vDSO (e.g. ARM with its truckload of different clock sources). IOW, no
>> more tsc-based timekeeping, CLOCK_HOST_REALTIME becomes useless as
>> clock_gettime() is usable from any context. This deeply affects the
>> Xenomai timer framework, and the way libcobalt reads time (no need for
>> tsc->ns conversion, the vDSO delivers nanosecs via clock_gettime()). In
>> the wake of such change, there would be no reason to keep the arith
>> support in Xenomai.
>>
>> - kernel/cobalt/irq.c and related can go. Dovetail integrates the
>> so-called "out-of-band" IRQ management directly into the mainline genirq
>> logic. This means Xenomai can switch to the request/free_irq() interface
>> or related.
>>
>> - kernel/cobalt/arch/*/ and related can go, almost entirely. There is
>> direct support for context switching including fpu management for
>> threads controlled by the companion (e.g. real-time) core. This means
>> that Xenomai's scheduler core should be adapted to use the available
>> dovetail interface for this. The workings are explained in the
>> description of the so-called "alternate scheduling" Dovetail
>> implements [3].
>>
>> - support for synchronous breakpoints would need to be reworked a bit,
>> around the ptrace()-related events Dovetail triggers (TASK_PTSTOP,
>> PTSTEP, PTCONT). As a consequence, the most ugly KEVT_SCHEDULE event is
>> gone.
>>
>> - although this would not be a requirement for a port, you could reuse
>> the native VFS support for out-of-band I/O handling in character drivers
>> Dovetail brings, in order to drop the RTDM-specific file descriptor
>> management [2], simplifying things even further.
> 
> One after the other, indeed. For RTDM, I will have to look into more
> details to asses the user-visible impact. Currently, your approach is
> not yet up-to-speed with RTDM, specifically due to a missing socket
> interface (RTnet, RT-CAN, ...).
>

RTDM sockets are fundamentally a user-space layer aimed at presenting a
POSIX-like API to application; the way this layer talks to Cobalt could
be implemented fully using a chardev interface internally. Actually,
more than 2/3 of the services exposed by a RTDM protocol driver are
currently attached to the Cobalt core via a chardev interface
(connect(), bind(), set/getsockopt(), etc).

The fact that EVL does not aim at POSIX compliance explains why there is
no pressure in having this. Dovetail may extend the net_device
abstraction with support for out-of-band I/O, whether this should go up
to the socket layer as a service access point is undecided. I just don't
know yet. Still musing with the idea. Bottom line for me is keep it simple.

-- 
Philippe.

Re: Towards Xenomai on Dovetail

[Jan Kiszka]

On 19.02.20 15:58, Philippe Gerum wrote:
> On 2/19/20 1:48 PM, Jan Kiszka via Xenomai wrote:
>> On 18.02.20 18:36, Philippe Gerum wrote:
>>> On 2/10/20 7:57 PM, Jan Kiszka wrote:
>>>> Hi Philippe,
>>>>
>>>> could you give a rough overview what areas would be just mapping API
>>>> calls, what would be more different and what is simply missing in
>>>> Dovetail in order to place Xenomai on top of it?
>>>
>>> By order of trickiness:
>>>
>>> - Dovetail assumes the companion core shares the same timekeeping bases
>>> and unit with the common kernel, providing some help in doing so. The
>>> core should switch to nanosecs, ktime_t internally. Typically, MMIO
>>> clock sources which are not architected can be exported directly by the
>>> vDSO (e.g. ARM with its truckload of different clock sources). IOW, no
>>> more tsc-based timekeeping, CLOCK_HOST_REALTIME becomes useless as
>>> clock_gettime() is usable from any context. This deeply affects the
>>> Xenomai timer framework, and the way libcobalt reads time (no need for
>>> tsc->ns conversion, the vDSO delivers nanosecs via clock_gettime()). In
>>> the wake of such change, there would be no reason to keep the arith
>>> support in Xenomai.
>>>
>>> - kernel/cobalt/irq.c and related can go. Dovetail integrates the
>>> so-called "out-of-band" IRQ management directly into the mainline genirq
>>> logic. This means Xenomai can switch to the request/free_irq() interface
>>> or related.
>>>
>>> - kernel/cobalt/arch/*/ and related can go, almost entirely. There is
>>> direct support for context switching including fpu management for
>>> threads controlled by the companion (e.g. real-time) core. This means
>>> that Xenomai's scheduler core should be adapted to use the available
>>> dovetail interface for this. The workings are explained in the
>>> description of the so-called "alternate scheduling" Dovetail
>>> implements [3].
>>>
>>> - support for synchronous breakpoints would need to be reworked a bit,
>>> around the ptrace()-related events Dovetail triggers (TASK_PTSTOP,
>>> PTSTEP, PTCONT). As a consequence, the most ugly KEVT_SCHEDULE event is
>>> gone.
>>>
>>> - although this would not be a requirement for a port, you could reuse
>>> the native VFS support for out-of-band I/O handling in character drivers
>>> Dovetail brings, in order to drop the RTDM-specific file descriptor
>>> management [2], simplifying things even further.
>>
>> One after the other, indeed. For RTDM, I will have to look into more
>> details to asses the user-visible impact. Currently, your approach is
>> not yet up-to-speed with RTDM, specifically due to a missing socket
>> interface (RTnet, RT-CAN, ...).
>>
> 
> RTDM sockets are fundamentally a user-space layer aimed at presenting a
> POSIX-like API to application; the way this layer talks to Cobalt could
> be implemented fully using a chardev interface internally. Actually,
> more than 2/3 of the services exposed by a RTDM protocol driver are
> currently attached to the Cobalt core via a chardev interface
> (connect(), bind(), set/getsockopt(), etc).
> 
> The fact that EVL does not aim at POSIX compliance explains why there is
> no pressure in having this. Dovetail may extend the net_device
> abstraction with support for out-of-band I/O, whether this should go up
> to the socket layer as a service access point is undecided. I just don't
> know yet. Still musing with the idea. Bottom line for me is keep it simple.
> 

Having been there already with RTDM, hmm, 15 years ago, I will predict 
that this approach won't be the final one. That said, it may indeed be a 
good idea to stay at that simple level while trying to sell things upstream.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Towards Xenomai on Dovetail

[steven.seeger flightsystems.net]

On Tuesday, February 18, 2020 12:36:45 PM EST Philippe Gerum wrote:
> - kernel/cobalt/arch/*/ and related can go, almost entirely. There is
> direct support for context switching including fpu management for
> threads controlled by the companion (e.g. real-time) core. This means
> that Xenomai's scheduler core should be adapted to use the available
> dovetail interface for this. The workings are explained in the
> description of the so-called "alternate scheduling" Dovetail implements [3].

Sorry for joining this discussion late.

Is there still a syscall interface (and fast ipipe syscall path) for dovetail? 
Or is this done away with?

Steven

Re: Towards Xenomai on Dovetail

[Philippe Gerum]

On 2/19/20 4:33 PM, steven.seeger flightsystems.net via Xenomai wrote:
> On Tuesday, February 18, 2020 12:36:45 PM EST Philippe Gerum wrote:
>> - kernel/cobalt/arch/*/ and related can go, almost entirely. There is
>> direct support for context switching including fpu management for
>> threads controlled by the companion (e.g. real-time) core. This means
>> that Xenomai's scheduler core should be adapted to use the available
>> dovetail interface for this. The workings are explained in the
>> description of the so-called "alternate scheduling" Dovetail implements [3].
> 
> Sorry for joining this discussion late.
> 
> Is there still a syscall interface (and fast ipipe syscall path) for dovetail? 
> Or is this done away with?
> 

Yes, there is, no difference compared to the I-pipe in this respect.

-- 
Philippe.