[lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[MONTET Julien via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 2178 bytes --]

Hi the developers !

CONTEXT
I am currently working on a Raspberry pi 3B with Xenomai and LTTng tools.
Raspbian 10.9 Buster - kernel 4.19.85
uname -a : Linux raspberrypi 4.19.85-v7+ #5 SMP PREEMPT Wed May 12 10:13:37
Both tools are working, but I wonder about the accuracy of LTTng libraries.


METHOD
The code used is quite simple, it is written with the alchemy skin.
A rt_task_spawn calls a function that has rt_task_set_periodic(NULL, TM_NOW, period) and rt_task_wait_period(NULL).
->The rt_task_set_periodic is based on 1ms.
->The  rt_task_wait_period(NULL) is of course inside a while loop (see below at the very end).

My goal is to get accurate traces from Xenomai.
I took two methods to do so :
-> lttng
-> basic calculation based on  rt_timer_read()

What a surprise when I found both method have two different results.
-> LTTng shows me traces [0.870;1.13] ms (or even less precise)
-> rt_time_read shows me traces [0.980;1.020] ms

Thing to note :
-> The use of LTTng has no influence on rt_time_read(), you can use both methods at the same time.

Then, I saved the output of rt_time_read inside a tracepoint.
It appeared the LTTng is always called at the right time because the value got by rt_time_read () is really good.


QUESTIONS
These are now my questions :
- What is the method I should trust ?
- I have searched on the forum and I found LTTng uses a MONOTONIC clock for the timestamp. Can/Should I modify it ?


CODE
-----------------------------------------------------------------------
A small part of my function called by rt_task_spawn :
[...]
    RTIME period = 1000*1000; // in ns
    RTIME now;
    RTIME previous = 0;
    RTIME duration;
[...]
 while(1)
    {
        overruns = 0;
        err = rt_task_wait_period(&overruns);
        now = rt_timer_read();
        tracepoint(tp_provider, tracepoint_tick_ms, now, "tick");

        if (previous != 0)
        {
            duration=now-previous;
            rt_printf("%llu\n \n", duration/1000);
        }
       previous=now;
   [...]
}
------------------------------------------------------------------------

Regards,

Julien

[-- Attachment #1.2: Type: text/html, Size: 8380 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Norbert Lange via lttng-dev]

Am Do., 20. Mai 2021 um 09:58 Uhr schrieb MONTET Julien via lttng-dev
<lttng-dev@lists.lttng.org>:
>
> Hi the developers !
>
> CONTEXT
> I am currently working on a Raspberry pi 3B with Xenomai and LTTng tools.
> Raspbian 10.9 Buster - kernel 4.19.85
> uname -a : Linux raspberrypi 4.19.85-v7+ #5 SMP PREEMPT Wed May 12 10:13:37
> Both tools are working, but I wonder about the accuracy of LTTng libraries.
>
>
> METHOD
> The code used is quite simple, it is written with the alchemy skin.
> A rt_task_spawn calls a function that has rt_task_set_periodic(NULL, TM_NOW, period) and rt_task_wait_period(NULL).
> ->The rt_task_set_periodic is based on 1ms.
> ->The  rt_task_wait_period(NULL) is of course inside a while loop (see below at the very end).
>
> My goal is to get accurate traces from Xenomai.
> I took two methods to do so :
> -> lttng
> -> basic calculation based on  rt_timer_read()
>
> What a surprise when I found both method have two different results.
> -> LTTng shows me traces [0.870;1.13] ms (or even less precise)
> -> rt_time_read shows me traces [0.980;1.020] ms
>
> Thing to note :
> -> The use of LTTng has no influence on rt_time_read(), you can use both methods at the same time.
>
> Then, I saved the output of rt_time_read inside a tracepoint.
> It appeared the LTTng is always called at the right time because the value got by rt_time_read () is really good.
>
>
> QUESTIONS
> These are now my questions :
> - What is the method I should trust ?
> - I have searched on the forum and I found LTTng uses a MONOTONIC clock for the timestamp. Can/Should I modify it ?
>
>
> CODE
> -----------------------------------------------------------------------
> A small part of my function called by rt_task_spawn :
> [...]
>     RTIME period = 1000*1000; // in ns
>     RTIME now;
>     RTIME previous = 0;
>     RTIME duration;
> [...]
>  while(1)
>     {
>         overruns = 0;
>         err = rt_task_wait_period(&overruns);
>         now = rt_timer_read();
>         tracepoint(tp_provider, tracepoint_tick_ms, now, "tick");
>
>         if (previous != 0)
>         {
>             duration=now-previous;
>             rt_printf("%llu\n \n", duration/1000);
>         }
>        previous=now;
>    [...]
> }

Are you using the Xenomai kernel ("Cobalt"), or just skins via
copperplate ("Mercury")?
You have some file /proc/xenomai/version?

The Xenomai kernel has his own clock, which in general is not
correlated to the linux monotonic clock.
(Under some circumstances it might be identical).

My plan is to use a clock plugin for Lttng, particularly because if
lttng uses the linux monotonic clock from a realtime thread
it might deadlock really good ;)

Norbert
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[MONTET Julien via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 3382 bytes --]

Hi Norbert,

Thank you for your answer !

Yes, I am using a Xenomai cobalt - xenomai is 3.1
cat /proc/xenomai/version => 3.1

After the installation, I tested "test tools" in /proc/xenomai/ and it worked nice.

What do you mean by "it might deadlock really good" ?

Cheers,


________________________________
De : Norbert Lange <nolange79@gmail.com>
Envoyé : jeudi 20 mai 2021 10:20
À : MONTET Julien <julien.montet@reseau.eseo.fr>
Cc : lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>
Objet : Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

Am Do., 20. Mai 2021 um 09:58 Uhr schrieb MONTET Julien via lttng-dev
<lttng-dev@lists.lttng.org>:
>
> Hi the developers !
>
> CONTEXT
> I am currently working on a Raspberry pi 3B with Xenomai and LTTng tools.
> Raspbian 10.9 Buster - kernel 4.19.85
> uname -a : Linux raspberrypi 4.19.85-v7+ #5 SMP PREEMPT Wed May 12 10:13:37
> Both tools are working, but I wonder about the accuracy of LTTng libraries.
>
>
> METHOD
> The code used is quite simple, it is written with the alchemy skin.
> A rt_task_spawn calls a function that has rt_task_set_periodic(NULL, TM_NOW, period) and rt_task_wait_period(NULL).
> ->The rt_task_set_periodic is based on 1ms.
> ->The  rt_task_wait_period(NULL) is of course inside a while loop (see below at the very end).
>
> My goal is to get accurate traces from Xenomai.
> I took two methods to do so :
> -> lttng
> -> basic calculation based on  rt_timer_read()
>
> What a surprise when I found both method have two different results.
> -> LTTng shows me traces [0.870;1.13] ms (or even less precise)
> -> rt_time_read shows me traces [0.980;1.020] ms
>
> Thing to note :
> -> The use of LTTng has no influence on rt_time_read(), you can use both methods at the same time.
>
> Then, I saved the output of rt_time_read inside a tracepoint.
> It appeared the LTTng is always called at the right time because the value got by rt_time_read () is really good.
>
>
> QUESTIONS
> These are now my questions :
> - What is the method I should trust ?
> - I have searched on the forum and I found LTTng uses a MONOTONIC clock for the timestamp. Can/Should I modify it ?
>
>
> CODE
> -----------------------------------------------------------------------
> A small part of my function called by rt_task_spawn :
> [...]
>     RTIME period = 1000*1000; // in ns
>     RTIME now;
>     RTIME previous = 0;
>     RTIME duration;
> [...]
>  while(1)
>     {
>         overruns = 0;
>         err = rt_task_wait_period(&overruns);
>         now = rt_timer_read();
>         tracepoint(tp_provider, tracepoint_tick_ms, now, "tick");
>
>         if (previous != 0)
>         {
>             duration=now-previous;
>             rt_printf("%llu\n \n", duration/1000);
>         }
>        previous=now;
>    [...]
> }

Are you using the Xenomai kernel ("Cobalt"), or just skins via
copperplate ("Mercury")?
You have some file /proc/xenomai/version?

The Xenomai kernel has his own clock, which in general is not
correlated to the linux monotonic clock.
(Under some circumstances it might be identical).

My plan is to use a clock plugin for Lttng, particularly because if
lttng uses the linux monotonic clock from a realtime thread
it might deadlock really good ;)

Norbert

[-- Attachment #1.2: Type: text/html, Size: 7508 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Norbert Lange via lttng-dev]

Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
<julien.montet@reseau.eseo.fr>:
>
> Hi Norbert,
>
> Thank you for your answer !
>
> Yes, I am using a Xenomai cobalt - xenomai is 3.1
> cat /proc/xenomai/version => 3.1
>
> After the installation, I tested "test tools" in /proc/xenomai/ and it worked nice.

Just asked to make sure, thought the scripts usual add some -xeno tag
to the kernel version.

> What do you mean by "it might deadlock really good" ?

clock_gettime will either use a syscall (kills realtime always) or is
optimized via VDSO (which very likely is your case).

What happens is that the kernel will take a spinlock, then write new
values, then releases the spinlock.
your program will aswell spin (but just to see if the spinlock is
free), read the values and interpolates them.

But if your program interrupts the kernel while the kernel holds the
lock (all on the same cpu core), then it will spin forever and the
kernel will never execute.

The ugly truth is, that any library you use could call those
functions, you need to very closely look at your realtime path.
Some checkers can help: https://github.com/nolange/preload_checkers



Norbert.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Mathieu Desnoyers via lttng-dev]

----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:

> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> <julien.montet@reseau.eseo.fr>:
>>
>> Hi Norbert,
>>
>> Thank you for your answer !
>>
>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
>> cat /proc/xenomai/version => 3.1
>>
>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
>> nice.
> 
> Just asked to make sure, thought the scripts usual add some -xeno tag
> to the kernel version.
> 
>> What do you mean by "it might deadlock really good" ?
> 
> clock_gettime will either use a syscall (kills realtime always) or is
> optimized via VDSO (which very likely is your case).
> 
> What happens is that the kernel will take a spinlock, then write new
> values, then releases the spinlock.
> your program will aswell spin (but just to see if the spinlock is
> free), read the values and interpolates them.
> 
> But if your program interrupts the kernel while the kernel holds the
> lock (all on the same cpu core), then it will spin forever and the
> kernel will never execute.

Just one clarification: the specific locking strategy used by the
Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
sets a bit which keeps concurrent readers looping until they observe
a consistent value. With Xenomai it indeed appears to be prone to
deadlock if a high priority Xenomai thread interrupts the kernel
while the write seqlock is held, and then proceeds to loop forever
on the read-side of the seqlock.

Note that for the in-kernel tracer clock read use-case, which
needs to be able to happen from NMI context, I've contributed a
modified version of the seqlock to the Linux kernel:

https://lwn.net/Articles/831540/ The seqcount latch lock type

It basically keeps two copies of the clock data structures, so the
read-side never has to loop waiting for the updater: it simply gets
redirected to the "stable" copy of the data.

The trade-off here is that with the latch lock used for clocks, a
reader may observe time going slightly backwards between two clock
reads when reading while specific clock rate adjustments are made
by an updater. The clock user needs to be aware of this.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Mathieu Desnoyers via lttng-dev]

----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:

> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> 
>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
>> <julien.montet@reseau.eseo.fr>:
>>>
>>> Hi Norbert,
>>>
>>> Thank you for your answer !
>>>
>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
>>> cat /proc/xenomai/version => 3.1
>>>
>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
>>> nice.
>> 
>> Just asked to make sure, thought the scripts usual add some -xeno tag
>> to the kernel version.
>> 
>>> What do you mean by "it might deadlock really good" ?
>> 
>> clock_gettime will either use a syscall (kills realtime always) or is
>> optimized via VDSO (which very likely is your case).
>> 
>> What happens is that the kernel will take a spinlock, then write new
>> values, then releases the spinlock.
>> your program will aswell spin (but just to see if the spinlock is
>> free), read the values and interpolates them.
>> 
>> But if your program interrupts the kernel while the kernel holds the
>> lock (all on the same cpu core), then it will spin forever and the
>> kernel will never execute.
> 
> Just one clarification: the specific locking strategy used by the
> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> sets a bit which keeps concurrent readers looping until they observe

When I say "sets a bit", I actually mean "increment a sequence counter",
and readers observe either odd or even state, thus knowing whether
they need to retry, and whether the value read before/after reading
the data structure changed.

Thanks,

Mathieu

> a consistent value. With Xenomai it indeed appears to be prone to
> deadlock if a high priority Xenomai thread interrupts the kernel
> while the write seqlock is held, and then proceeds to loop forever
> on the read-side of the seqlock.
> 
> Note that for the in-kernel tracer clock read use-case, which
> needs to be able to happen from NMI context, I've contributed a
> modified version of the seqlock to the Linux kernel:
> 
> https://lwn.net/Articles/831540/ The seqcount latch lock type
> 
> It basically keeps two copies of the clock data structures, so the
> read-side never has to loop waiting for the updater: it simply gets
> redirected to the "stable" copy of the data.
> 
> The trade-off here is that with the latch lock used for clocks, a
> reader may observe time going slightly backwards between two clock
> reads when reading while specific clock rate adjustments are made
> by an updater. The clock user needs to be aware of this.
> 
> Thanks,
> 
> Mathieu
> 
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Mathieu Desnoyers via lttng-dev]

----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:

> ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> 
>> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>> 
>>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
>>> <julien.montet@reseau.eseo.fr>:
>>>>
>>>> Hi Norbert,
>>>>
>>>> Thank you for your answer !
>>>>
>>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
>>>> cat /proc/xenomai/version => 3.1
>>>>
>>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
>>>> nice.
>>> 
>>> Just asked to make sure, thought the scripts usual add some -xeno tag
>>> to the kernel version.
>>> 
>>>> What do you mean by "it might deadlock really good" ?
>>> 
>>> clock_gettime will either use a syscall (kills realtime always) or is
>>> optimized via VDSO (which very likely is your case).
>>> 
>>> What happens is that the kernel will take a spinlock, then write new
>>> values, then releases the spinlock.
>>> your program will aswell spin (but just to see if the spinlock is
>>> free), read the values and interpolates them.
>>> 
>>> But if your program interrupts the kernel while the kernel holds the
>>> lock (all on the same cpu core), then it will spin forever and the
>>> kernel will never execute.
>> 
>> Just one clarification: the specific locking strategy used by the
>> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
>> sets a bit which keeps concurrent readers looping until they observe
> 
> When I say "sets a bit", I actually mean "increment a sequence counter",
> and readers observe either odd or even state, thus knowing whether
> they need to retry, and whether the value read before/after reading
> the data structure changed.

Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
of vdso_update_{begin,end}, I notice that interrupts are disabled across
the entire update. So I understand that the Interrupt pipeline (I-pipe)
interrupt gets delivered even when the kernel disables interrupts. Did
you consider modifying the I-pipe kernel patch to change the vdso update so
it updates the vdso from within an I-pipe virq handler ?

AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
clock sources.

Thanks,

Mathieu

> 
> Thanks,
> 
> Mathieu
> 
>> a consistent value. With Xenomai it indeed appears to be prone to
>> deadlock if a high priority Xenomai thread interrupts the kernel
>> while the write seqlock is held, and then proceeds to loop forever
>> on the read-side of the seqlock.
>> 
>> Note that for the in-kernel tracer clock read use-case, which
>> needs to be able to happen from NMI context, I've contributed a
>> modified version of the seqlock to the Linux kernel:
>> 
>> https://lwn.net/Articles/831540/ The seqcount latch lock type
>> 
>> It basically keeps two copies of the clock data structures, so the
>> read-side never has to loop waiting for the updater: it simply gets
>> redirected to the "stable" copy of the data.
>> 
>> The trade-off here is that with the latch lock used for clocks, a
>> reader may observe time going slightly backwards between two clock
>> reads when reading while specific clock rate adjustments are made
>> by an updater. The clock user needs to be aware of this.
>> 
>> Thanks,
>> 
>> Mathieu
>> 
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Jan Kiszka via lttng-dev]

On 20.05.21 17:09, Mathieu Desnoyers wrote:
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
> 
>> ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>
>>> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>>
>>>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
>>>> <julien.montet@reseau.eseo.fr>:
>>>>>
>>>>> Hi Norbert,
>>>>>
>>>>> Thank you for your answer !
>>>>>
>>>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
>>>>> cat /proc/xenomai/version => 3.1
>>>>>
>>>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
>>>>> nice.
>>>>
>>>> Just asked to make sure, thought the scripts usual add some -xeno tag
>>>> to the kernel version.
>>>>
>>>>> What do you mean by "it might deadlock really good" ?
>>>>
>>>> clock_gettime will either use a syscall (kills realtime always) or is
>>>> optimized via VDSO (which very likely is your case).
>>>>
>>>> What happens is that the kernel will take a spinlock, then write new
>>>> values, then releases the spinlock.
>>>> your program will aswell spin (but just to see if the spinlock is
>>>> free), read the values and interpolates them.
>>>>
>>>> But if your program interrupts the kernel while the kernel holds the
>>>> lock (all on the same cpu core), then it will spin forever and the
>>>> kernel will never execute.
>>>
>>> Just one clarification: the specific locking strategy used by the
>>> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
>>> sets a bit which keeps concurrent readers looping until they observe
>>
>> When I say "sets a bit", I actually mean "increment a sequence counter",
>> and readers observe either odd or even state, thus knowing whether
>> they need to retry, and whether the value read before/after reading
>> the data structure changed.
> 
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?
> 
> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.

In fact, this is what happens with upcoming Xenomai 3.2, based on the
Dovetail kernel patch (replacement of I-pipe). Implies kernel 5.10.

For I-pipe, we have the CLOCK_HOST_REALTIME infrastructure to obtain the
kernel's view on CLOCK_REALTIME from within an Xenomai task. That is
available up to kernel 5.4.

HTH,
Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Norbert Lange via lttng-dev]

Am Do., 20. Mai 2021 um 17:09 Uhr schrieb Mathieu Desnoyers
<mathieu.desnoyers@efficios.com>:
>
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
> > ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >
> >> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >>
> >>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> >>> <julien.montet@reseau.eseo.fr>:
> >>>>
> >>>> Hi Norbert,
> >>>>
> >>>> Thank you for your answer !
> >>>>
> >>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
> >>>> cat /proc/xenomai/version => 3.1
> >>>>
> >>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
> >>>> nice.
> >>>
> >>> Just asked to make sure, thought the scripts usual add some -xeno tag
> >>> to the kernel version.
> >>>
> >>>> What do you mean by "it might deadlock really good" ?
> >>>
> >>> clock_gettime will either use a syscall (kills realtime always) or is
> >>> optimized via VDSO (which very likely is your case).
> >>>
> >>> What happens is that the kernel will take a spinlock, then write new
> >>> values, then releases the spinlock.
> >>> your program will aswell spin (but just to see if the spinlock is
> >>> free), read the values and interpolates them.
> >>>
> >>> But if your program interrupts the kernel while the kernel holds the
> >>> lock (all on the same cpu core), then it will spin forever and the
> >>> kernel will never execute.
> >>
> >> Just one clarification: the specific locking strategy used by the
> >> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> >> sets a bit which keeps concurrent readers looping until they observe
> >
> > When I say "sets a bit", I actually mean "increment a sequence counter",
> > and readers observe either odd or even state, thus knowing whether
> > they need to retry, and whether the value read before/after reading
> > the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?

Yes, I did use an non-upstreamed patch for a while to get things in order:
https://www.xenomai.org/pipermail/xenomai/2018-December/040134.html

I would prefer just a NMI safe source that might jump back a bit, no matter how.

> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.

The Xenomai folks are trying to get their next-gen abstraction "dovetail" closer
coupled to the kernel, AFAIR their will be VDSO support and
unification of the clock sources.

Still need to get stuff running today =)

Norbert
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[MONTET Julien via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 4539 bytes --]

Hello Mathieu, Norbert and Jan,

Thank you for all of your explainations and the overview of the system.
No I didn't change the ipipe patch for the vDSO, I may try this.
If I have correctly understood, this patch prevents Cobalt from entering in a deadlock when the kernel is using the vDSO and the program interrupts the kernel at the same time. On which kernel does it word (aroubd 4.19) ?
I currently try to avoid kernel 5.4 because I remember I faced some boot issues (but it is on another topic).

Here the issues i faced (drawn on TraceCompass). Are these the deadlocks we are talking about ?
https://postimg.cc/BP4G3bF0 (on 11:02:56:380)
https://postimg.cc/q6wHvrcC

Regards,


________________________________
De : Norbert Lange <nolange79@gmail.com>
Envoyé : jeudi 20 mai 2021 17:39
À : Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc : MONTET Julien <julien.montet@reseau.eseo.fr>; lttng-dev <lttng-dev@lists.lttng.org>; Jan Kiszka <jan.kiszka@siemens.com>; Xenomai <xenomai@xenomai.org>
Objet : Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

Am Do., 20. Mai 2021 um 17:09 Uhr schrieb Mathieu Desnoyers
<mathieu.desnoyers@efficios.com>:
>
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
> > ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >
> >> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >>
> >>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> >>> <julien.montet@reseau.eseo.fr>:
> >>>>
> >>>> Hi Norbert,
> >>>>
> >>>> Thank you for your answer !
> >>>>
> >>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
> >>>> cat /proc/xenomai/version => 3.1
> >>>>
> >>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
> >>>> nice.
> >>>
> >>> Just asked to make sure, thought the scripts usual add some -xeno tag
> >>> to the kernel version.
> >>>
> >>>> What do you mean by "it might deadlock really good" ?
> >>>
> >>> clock_gettime will either use a syscall (kills realtime always) or is
> >>> optimized via VDSO (which very likely is your case).
> >>>
> >>> What happens is that the kernel will take a spinlock, then write new
> >>> values, then releases the spinlock.
> >>> your program will aswell spin (but just to see if the spinlock is
> >>> free), read the values and interpolates them.
> >>>
> >>> But if your program interrupts the kernel while the kernel holds the
> >>> lock (all on the same cpu core), then it will spin forever and the
> >>> kernel will never execute.
> >>
> >> Just one clarification: the specific locking strategy used by the
> >> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> >> sets a bit which keeps concurrent readers looping until they observe
> >
> > When I say "sets a bit", I actually mean "increment a sequence counter",
> > and readers observe either odd or even state, thus knowing whether
> > they need to retry, and whether the value read before/after reading
> > the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?

Yes, I did use an non-upstreamed patch for a while to get things in order:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.xenomai.org%2Fpipermail%2Fxenomai%2F2018-December%2F040134.html&amp;data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7Cef0b71ac314f4ab2321f08d91ba57c9d%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637571219835495365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=dOiRFzeKFQA%2B25R6aqrjL2ZJMkV5c782DSBGiHHoYZc%3D&amp;reserved=0

I would prefer just a NMI safe source that might jump back a bit, no matter how.

> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.

The Xenomai folks are trying to get their next-gen abstraction "dovetail" closer
coupled to the kernel, AFAIR their will be VDSO support and
unification of the clock sources.

Still need to get stuff running today =)

Norbert

[-- Attachment #1.2: Type: text/html, Size: 8477 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()

[Norbert Lange via lttng-dev]

Am Fr., 21. Mai 2021 um 12:13 Uhr schrieb MONTET Julien
<julien.montet@reseau.eseo.fr>:
>
> Hello Mathieu, Norbert and Jan,
>
> Thank you for all of your explainations and the overview of the system.
> No I didn't change the ipipe patch for the vDSO, I may try this.
> If I have correctly understood, this patch prevents Cobalt from entering in a deadlock when the kernel is using the vDSO and the program interrupts the kernel at the same time. On which kernel does it word (aroubd 4.19) ?
> I currently try to avoid kernel 5.4 because I remember I faced some boot issues (but it is on another topic).

That patch was for 4.19 AFAIR, but its specific to x86. The Linux
kernel cleaned up the vdso handling to have a common base with some
5.x version,
but back then its been separate for each arch

>
> Here the issues i faced (drawn on TraceCompass). Are these the deadlocks we are talking about ?
> https://postimg.cc/BP4G3bF0 (on 11:02:56:380)
> https://postimg.cc/q6wHvrcC

Nope, if you get such a deadlock, then your only hope is Xenomai's
Watchdog killing the process.
It should happen very rarely (but thats not an argument if your
software should run for years).

Norbert
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev