* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
@ 2021-05-20 15:09 ` Mathieu Desnoyers
0 siblings, 0 replies; 16+ messages in thread
From: Mathieu Desnoyers @ 2021-05-20 15:09 UTC (permalink / raw)
To: Norbert Lange
Cc: MONTET Julien, lttng-dev, Mathieu Desnoyers, Jan Kiszka, Xenomai
----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
> ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>
>> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>
>>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
>>> <julien.montet@reseau.eseo.fr>:
>>>>
>>>> Hi Norbert,
>>>>
>>>> Thank you for your answer !
>>>>
>>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
>>>> cat /proc/xenomai/version => 3.1
>>>>
>>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
>>>> nice.
>>>
>>> Just asked to make sure, thought the scripts usual add some -xeno tag
>>> to the kernel version.
>>>
>>>> What do you mean by "it might deadlock really good" ?
>>>
>>> clock_gettime will either use a syscall (kills realtime always) or is
>>> optimized via VDSO (which very likely is your case).
>>>
>>> What happens is that the kernel will take a spinlock, then write new
>>> values, then releases the spinlock.
>>> your program will aswell spin (but just to see if the spinlock is
>>> free), read the values and interpolates them.
>>>
>>> But if your program interrupts the kernel while the kernel holds the
>>> lock (all on the same cpu core), then it will spin forever and the
>>> kernel will never execute.
>>
>> Just one clarification: the specific locking strategy used by the
>> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
>> sets a bit which keeps concurrent readers looping until they observe
>
> When I say "sets a bit", I actually mean "increment a sequence counter",
> and readers observe either odd or even state, thus knowing whether
> they need to retry, and whether the value read before/after reading
> the data structure changed.
Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
of vdso_update_{begin,end}, I notice that interrupts are disabled across
the entire update. So I understand that the Interrupt pipeline (I-pipe)
interrupt gets delivered even when the kernel disables interrupts. Did
you consider modifying the I-pipe kernel patch to change the vdso update so
it updates the vdso from within an I-pipe virq handler ?
AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
clock sources.
Thanks,
Mathieu
>
> Thanks,
>
> Mathieu
>
>> a consistent value. With Xenomai it indeed appears to be prone to
>> deadlock if a high priority Xenomai thread interrupts the kernel
>> while the write seqlock is held, and then proceeds to loop forever
>> on the read-side of the seqlock.
>>
>> Note that for the in-kernel tracer clock read use-case, which
>> needs to be able to happen from NMI context, I've contributed a
>> modified version of the seqlock to the Linux kernel:
>>
>> https://lwn.net/Articles/831540/ The seqcount latch lock type
>>
>> It basically keeps two copies of the clock data structures, so the
>> read-side never has to loop waiting for the updater: it simply gets
>> redirected to the "stable" copy of the data.
>>
>> The trade-off here is that with the latch lock used for clocks, a
>> reader may observe time going slightly backwards between two clock
>> reads when reading while specific clock rate adjustments are made
>> by an updater. The clock user needs to be aware of this.
>>
>> Thanks,
>>
>> Mathieu
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
2021-05-20 15:09 ` Mathieu Desnoyers
@ 2021-05-20 15:34 ` Jan Kiszka
-1 siblings, 0 replies; 16+ messages in thread
From: Jan Kiszka via lttng-dev @ 2021-05-20 15:34 UTC (permalink / raw)
To: Mathieu Desnoyers, Norbert Lange; +Cc: MONTET Julien, lttng-dev, Xenomai
On 20.05.21 17:09, Mathieu Desnoyers wrote:
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
>> ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>
>>> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>>
>>>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
>>>> <julien.montet@reseau.eseo.fr>:
>>>>>
>>>>> Hi Norbert,
>>>>>
>>>>> Thank you for your answer !
>>>>>
>>>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
>>>>> cat /proc/xenomai/version => 3.1
>>>>>
>>>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
>>>>> nice.
>>>>
>>>> Just asked to make sure, thought the scripts usual add some -xeno tag
>>>> to the kernel version.
>>>>
>>>>> What do you mean by "it might deadlock really good" ?
>>>>
>>>> clock_gettime will either use a syscall (kills realtime always) or is
>>>> optimized via VDSO (which very likely is your case).
>>>>
>>>> What happens is that the kernel will take a spinlock, then write new
>>>> values, then releases the spinlock.
>>>> your program will aswell spin (but just to see if the spinlock is
>>>> free), read the values and interpolates them.
>>>>
>>>> But if your program interrupts the kernel while the kernel holds the
>>>> lock (all on the same cpu core), then it will spin forever and the
>>>> kernel will never execute.
>>>
>>> Just one clarification: the specific locking strategy used by the
>>> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
>>> sets a bit which keeps concurrent readers looping until they observe
>>
>> When I say "sets a bit", I actually mean "increment a sequence counter",
>> and readers observe either odd or even state, thus knowing whether
>> they need to retry, and whether the value read before/after reading
>> the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?
>
> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.
In fact, this is what happens with upcoming Xenomai 3.2, based on the
Dovetail kernel patch (replacement of I-pipe). Implies kernel 5.10.
For I-pipe, we have the CLOCK_HOST_REALTIME infrastructure to obtain the
kernel's view on CLOCK_REALTIME from within an Xenomai task. That is
available up to kernel 5.4.
HTH,
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
@ 2021-05-20 15:34 ` Jan Kiszka
0 siblings, 0 replies; 16+ messages in thread
From: Jan Kiszka @ 2021-05-20 15:34 UTC (permalink / raw)
To: Mathieu Desnoyers, Norbert Lange; +Cc: MONTET Julien, lttng-dev, Xenomai
On 20.05.21 17:09, Mathieu Desnoyers wrote:
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
>> ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>
>>> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
>>>
>>>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
>>>> <julien.montet@reseau.eseo.fr>:
>>>>>
>>>>> Hi Norbert,
>>>>>
>>>>> Thank you for your answer !
>>>>>
>>>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
>>>>> cat /proc/xenomai/version => 3.1
>>>>>
>>>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
>>>>> nice.
>>>>
>>>> Just asked to make sure, thought the scripts usual add some -xeno tag
>>>> to the kernel version.
>>>>
>>>>> What do you mean by "it might deadlock really good" ?
>>>>
>>>> clock_gettime will either use a syscall (kills realtime always) or is
>>>> optimized via VDSO (which very likely is your case).
>>>>
>>>> What happens is that the kernel will take a spinlock, then write new
>>>> values, then releases the spinlock.
>>>> your program will aswell spin (but just to see if the spinlock is
>>>> free), read the values and interpolates them.
>>>>
>>>> But if your program interrupts the kernel while the kernel holds the
>>>> lock (all on the same cpu core), then it will spin forever and the
>>>> kernel will never execute.
>>>
>>> Just one clarification: the specific locking strategy used by the
>>> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
>>> sets a bit which keeps concurrent readers looping until they observe
>>
>> When I say "sets a bit", I actually mean "increment a sequence counter",
>> and readers observe either odd or even state, thus knowing whether
>> they need to retry, and whether the value read before/after reading
>> the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?
>
> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.
In fact, this is what happens with upcoming Xenomai 3.2, based on the
Dovetail kernel patch (replacement of I-pipe). Implies kernel 5.10.
For I-pipe, we have the CLOCK_HOST_REALTIME infrastructure to obtain the
kernel's view on CLOCK_REALTIME from within an Xenomai task. That is
available up to kernel 5.4.
HTH,
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
2021-05-20 15:09 ` Mathieu Desnoyers
@ 2021-05-20 15:39 ` Norbert Lange
-1 siblings, 0 replies; 16+ messages in thread
From: Norbert Lange via lttng-dev @ 2021-05-20 15:39 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: MONTET Julien, lttng-dev, Jan Kiszka, Xenomai
Am Do., 20. Mai 2021 um 17:09 Uhr schrieb Mathieu Desnoyers
<mathieu.desnoyers@efficios.com>:
>
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
> > ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >
> >> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >>
> >>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> >>> <julien.montet@reseau.eseo.fr>:
> >>>>
> >>>> Hi Norbert,
> >>>>
> >>>> Thank you for your answer !
> >>>>
> >>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
> >>>> cat /proc/xenomai/version => 3.1
> >>>>
> >>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
> >>>> nice.
> >>>
> >>> Just asked to make sure, thought the scripts usual add some -xeno tag
> >>> to the kernel version.
> >>>
> >>>> What do you mean by "it might deadlock really good" ?
> >>>
> >>> clock_gettime will either use a syscall (kills realtime always) or is
> >>> optimized via VDSO (which very likely is your case).
> >>>
> >>> What happens is that the kernel will take a spinlock, then write new
> >>> values, then releases the spinlock.
> >>> your program will aswell spin (but just to see if the spinlock is
> >>> free), read the values and interpolates them.
> >>>
> >>> But if your program interrupts the kernel while the kernel holds the
> >>> lock (all on the same cpu core), then it will spin forever and the
> >>> kernel will never execute.
> >>
> >> Just one clarification: the specific locking strategy used by the
> >> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> >> sets a bit which keeps concurrent readers looping until they observe
> >
> > When I say "sets a bit", I actually mean "increment a sequence counter",
> > and readers observe either odd or even state, thus knowing whether
> > they need to retry, and whether the value read before/after reading
> > the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?
Yes, I did use an non-upstreamed patch for a while to get things in order:
https://www.xenomai.org/pipermail/xenomai/2018-December/040134.html
I would prefer just a NMI safe source that might jump back a bit, no matter how.
> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.
The Xenomai folks are trying to get their next-gen abstraction "dovetail" closer
coupled to the kernel, AFAIR their will be VDSO support and
unification of the clock sources.
Still need to get stuff running today =)
Norbert
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
@ 2021-05-20 15:39 ` Norbert Lange
0 siblings, 0 replies; 16+ messages in thread
From: Norbert Lange @ 2021-05-20 15:39 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: MONTET Julien, lttng-dev, Jan Kiszka, Xenomai
Am Do., 20. Mai 2021 um 17:09 Uhr schrieb Mathieu Desnoyers
<mathieu.desnoyers@efficios.com>:
>
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
> > ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >
> >> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >>
> >>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> >>> <julien.montet@reseau.eseo.fr>:
> >>>>
> >>>> Hi Norbert,
> >>>>
> >>>> Thank you for your answer !
> >>>>
> >>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
> >>>> cat /proc/xenomai/version => 3.1
> >>>>
> >>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
> >>>> nice.
> >>>
> >>> Just asked to make sure, thought the scripts usual add some -xeno tag
> >>> to the kernel version.
> >>>
> >>>> What do you mean by "it might deadlock really good" ?
> >>>
> >>> clock_gettime will either use a syscall (kills realtime always) or is
> >>> optimized via VDSO (which very likely is your case).
> >>>
> >>> What happens is that the kernel will take a spinlock, then write new
> >>> values, then releases the spinlock.
> >>> your program will aswell spin (but just to see if the spinlock is
> >>> free), read the values and interpolates them.
> >>>
> >>> But if your program interrupts the kernel while the kernel holds the
> >>> lock (all on the same cpu core), then it will spin forever and the
> >>> kernel will never execute.
> >>
> >> Just one clarification: the specific locking strategy used by the
> >> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> >> sets a bit which keeps concurrent readers looping until they observe
> >
> > When I say "sets a bit", I actually mean "increment a sequence counter",
> > and readers observe either odd or even state, thus knowing whether
> > they need to retry, and whether the value read before/after reading
> > the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?
Yes, I did use an non-upstreamed patch for a while to get things in order:
https://www.xenomai.org/pipermail/xenomai/2018-December/040134.html
I would prefer just a NMI safe source that might jump back a bit, no matter how.
> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.
The Xenomai folks are trying to get their next-gen abstraction "dovetail" closer
coupled to the kernel, AFAIR their will be VDSO support and
unification of the clock sources.
Still need to get stuff running today =)
Norbert
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
2021-05-20 15:39 ` Norbert Lange
@ 2021-05-21 10:13 ` MONTET Julien
-1 siblings, 0 replies; 16+ messages in thread
From: MONTET Julien via lttng-dev @ 2021-05-21 10:13 UTC (permalink / raw)
To: Norbert Lange, Mathieu Desnoyers; +Cc: lttng-dev, Jan Kiszka, Xenomai
[-- Attachment #1.1: Type: text/plain, Size: 4539 bytes --]
Hello Mathieu, Norbert and Jan,
Thank you for all of your explainations and the overview of the system.
No I didn't change the ipipe patch for the vDSO, I may try this.
If I have correctly understood, this patch prevents Cobalt from entering in a deadlock when the kernel is using the vDSO and the program interrupts the kernel at the same time. On which kernel does it word (aroubd 4.19) ?
I currently try to avoid kernel 5.4 because I remember I faced some boot issues (but it is on another topic).
Here the issues i faced (drawn on TraceCompass). Are these the deadlocks we are talking about ?
https://postimg.cc/BP4G3bF0 (on 11:02:56:380)
https://postimg.cc/q6wHvrcC
Regards,
________________________________
De : Norbert Lange <nolange79@gmail.com>
Envoyé : jeudi 20 mai 2021 17:39
À : Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc : MONTET Julien <julien.montet@reseau.eseo.fr>; lttng-dev <lttng-dev@lists.lttng.org>; Jan Kiszka <jan.kiszka@siemens.com>; Xenomai <xenomai@xenomai.org>
Objet : Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
Am Do., 20. Mai 2021 um 17:09 Uhr schrieb Mathieu Desnoyers
<mathieu.desnoyers@efficios.com>:
>
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
> > ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >
> >> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >>
> >>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> >>> <julien.montet@reseau.eseo.fr>:
> >>>>
> >>>> Hi Norbert,
> >>>>
> >>>> Thank you for your answer !
> >>>>
> >>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
> >>>> cat /proc/xenomai/version => 3.1
> >>>>
> >>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
> >>>> nice.
> >>>
> >>> Just asked to make sure, thought the scripts usual add some -xeno tag
> >>> to the kernel version.
> >>>
> >>>> What do you mean by "it might deadlock really good" ?
> >>>
> >>> clock_gettime will either use a syscall (kills realtime always) or is
> >>> optimized via VDSO (which very likely is your case).
> >>>
> >>> What happens is that the kernel will take a spinlock, then write new
> >>> values, then releases the spinlock.
> >>> your program will aswell spin (but just to see if the spinlock is
> >>> free), read the values and interpolates them.
> >>>
> >>> But if your program interrupts the kernel while the kernel holds the
> >>> lock (all on the same cpu core), then it will spin forever and the
> >>> kernel will never execute.
> >>
> >> Just one clarification: the specific locking strategy used by the
> >> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> >> sets a bit which keeps concurrent readers looping until they observe
> >
> > When I say "sets a bit", I actually mean "increment a sequence counter",
> > and readers observe either odd or even state, thus knowing whether
> > they need to retry, and whether the value read before/after reading
> > the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?
Yes, I did use an non-upstreamed patch for a while to get things in order:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.xenomai.org%2Fpipermail%2Fxenomai%2F2018-December%2F040134.html&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7Cef0b71ac314f4ab2321f08d91ba57c9d%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637571219835495365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dOiRFzeKFQA%2B25R6aqrjL2ZJMkV5c782DSBGiHHoYZc%3D&reserved=0
I would prefer just a NMI safe source that might jump back a bit, no matter how.
> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.
The Xenomai folks are trying to get their next-gen abstraction "dovetail" closer
coupled to the kernel, AFAIR their will be VDSO support and
unification of the clock sources.
Still need to get stuff running today =)
Norbert
[-- Attachment #1.2: Type: text/html, Size: 8477 bytes --]
[-- Attachment #2: Type: text/plain, Size: 156 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
@ 2021-05-21 10:13 ` MONTET Julien
0 siblings, 0 replies; 16+ messages in thread
From: MONTET Julien @ 2021-05-21 10:13 UTC (permalink / raw)
To: Norbert Lange, Mathieu Desnoyers; +Cc: lttng-dev, Jan Kiszka, Xenomai
Hello Mathieu, Norbert and Jan,
Thank you for all of your explainations and the overview of the system.
No I didn't change the ipipe patch for the vDSO, I may try this.
If I have correctly understood, this patch prevents Cobalt from entering in a deadlock when the kernel is using the vDSO and the program interrupts the kernel at the same time. On which kernel does it word (aroubd 4.19) ?
I currently try to avoid kernel 5.4 because I remember I faced some boot issues (but it is on another topic).
Here the issues i faced (drawn on TraceCompass). Are these the deadlocks we are talking about ?
https://postimg.cc/BP4G3bF0 (on 11:02:56:380)
https://postimg.cc/q6wHvrcC
Regards,
________________________________
De : Norbert Lange <nolange79@gmail.com>
Envoyé : jeudi 20 mai 2021 17:39
À : Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc : MONTET Julien <julien.montet@reseau.eseo.fr>; lttng-dev <lttng-dev@lists.lttng.org>; Jan Kiszka <jan.kiszka@siemens.com>; Xenomai <xenomai@xenomai.org>
Objet : Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
Am Do., 20. Mai 2021 um 17:09 Uhr schrieb Mathieu Desnoyers
<mathieu.desnoyers@efficios.com>:
>
> ----- On May 20, 2021, at 9:56 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:
>
> > ----- On May 20, 2021, at 9:54 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >
> >> ----- On May 20, 2021, at 5:11 AM, lttng-dev lttng-dev@lists.lttng.org wrote:
> >>
> >>> Am Do., 20. Mai 2021 um 10:28 Uhr schrieb MONTET Julien
> >>> <julien.montet@reseau.eseo.fr>:
> >>>>
> >>>> Hi Norbert,
> >>>>
> >>>> Thank you for your answer !
> >>>>
> >>>> Yes, I am using a Xenomai cobalt - xenomai is 3.1
> >>>> cat /proc/xenomai/version => 3.1
> >>>>
> >>>> After the installation, I tested "test tools" in /proc/xenomai/ and it worked
> >>>> nice.
> >>>
> >>> Just asked to make sure, thought the scripts usual add some -xeno tag
> >>> to the kernel version.
> >>>
> >>>> What do you mean by "it might deadlock really good" ?
> >>>
> >>> clock_gettime will either use a syscall (kills realtime always) or is
> >>> optimized via VDSO (which very likely is your case).
> >>>
> >>> What happens is that the kernel will take a spinlock, then write new
> >>> values, then releases the spinlock.
> >>> your program will aswell spin (but just to see if the spinlock is
> >>> free), read the values and interpolates them.
> >>>
> >>> But if your program interrupts the kernel while the kernel holds the
> >>> lock (all on the same cpu core), then it will spin forever and the
> >>> kernel will never execute.
> >>
> >> Just one clarification: the specific locking strategy used by the
> >> Linux kernel monotonic clock vDSO is a "seqlock", where the kernel
> >> sets a bit which keeps concurrent readers looping until they observe
> >
> > When I say "sets a bit", I actually mean "increment a sequence counter",
> > and readers observe either odd or even state, thus knowing whether
> > they need to retry, and whether the value read before/after reading
> > the data structure changed.
>
> Looking again at the Linux kernel's kernel/time/vsyscall.c implementation
> of vdso_update_{begin,end}, I notice that interrupts are disabled across
> the entire update. So I understand that the Interrupt pipeline (I-pipe)
> interrupt gets delivered even when the kernel disables interrupts. Did
> you consider modifying the I-pipe kernel patch to change the vdso update so
> it updates the vdso from within an I-pipe virq handler ?
Yes, I did use an non-upstreamed patch for a while to get things in order:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.xenomai.org%2Fpipermail%2Fxenomai%2F2018-December%2F040134.html&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7Cef0b71ac314f4ab2321f08d91ba57c9d%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637571219835495365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dOiRFzeKFQA%2B25R6aqrjL2ZJMkV5c782DSBGiHHoYZc%3D&reserved=0
I would prefer just a NMI safe source that might jump back a bit, no matter how.
> AFAIU this would allow Xenomai userspace to use the Linux kernel vDSO
> clock sources.
The Xenomai folks are trying to get their next-gen abstraction "dovetail" closer
coupled to the kernel, AFAIR their will be VDSO support and
unification of the clock sources.
Still need to get stuff running today =)
Norbert
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
2021-05-21 10:13 ` MONTET Julien
@ 2021-05-25 8:46 ` Norbert Lange
-1 siblings, 0 replies; 16+ messages in thread
From: Norbert Lange via lttng-dev @ 2021-05-25 8:46 UTC (permalink / raw)
To: MONTET Julien; +Cc: Jan Kiszka, lttng-dev, Xenomai
Am Fr., 21. Mai 2021 um 12:13 Uhr schrieb MONTET Julien
<julien.montet@reseau.eseo.fr>:
>
> Hello Mathieu, Norbert and Jan,
>
> Thank you for all of your explainations and the overview of the system.
> No I didn't change the ipipe patch for the vDSO, I may try this.
> If I have correctly understood, this patch prevents Cobalt from entering in a deadlock when the kernel is using the vDSO and the program interrupts the kernel at the same time. On which kernel does it word (aroubd 4.19) ?
> I currently try to avoid kernel 5.4 because I remember I faced some boot issues (but it is on another topic).
That patch was for 4.19 AFAIR, but its specific to x86. The Linux
kernel cleaned up the vdso handling to have a common base with some
5.x version,
but back then its been separate for each arch
>
> Here the issues i faced (drawn on TraceCompass). Are these the deadlocks we are talking about ?
> https://postimg.cc/BP4G3bF0 (on 11:02:56:380)
> https://postimg.cc/q6wHvrcC
Nope, if you get such a deadlock, then your only hope is Xenomai's
Watchdog killing the process.
It should happen very rarely (but thats not an argument if your
software should run for years).
Norbert
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [lttng-dev] LTTng - Xenomai : different results between timestamp-lttng and rt_time_read()
@ 2021-05-25 8:46 ` Norbert Lange
0 siblings, 0 replies; 16+ messages in thread
From: Norbert Lange @ 2021-05-25 8:46 UTC (permalink / raw)
To: MONTET Julien; +Cc: Mathieu Desnoyers, lttng-dev, Jan Kiszka, Xenomai
Am Fr., 21. Mai 2021 um 12:13 Uhr schrieb MONTET Julien
<julien.montet@reseau.eseo.fr>:
>
> Hello Mathieu, Norbert and Jan,
>
> Thank you for all of your explainations and the overview of the system.
> No I didn't change the ipipe patch for the vDSO, I may try this.
> If I have correctly understood, this patch prevents Cobalt from entering in a deadlock when the kernel is using the vDSO and the program interrupts the kernel at the same time. On which kernel does it word (aroubd 4.19) ?
> I currently try to avoid kernel 5.4 because I remember I faced some boot issues (but it is on another topic).
That patch was for 4.19 AFAIR, but its specific to x86. The Linux
kernel cleaned up the vdso handling to have a common base with some
5.x version,
but back then its been separate for each arch
>
> Here the issues i faced (drawn on TraceCompass). Are these the deadlocks we are talking about ?
> https://postimg.cc/BP4G3bF0 (on 11:02:56:380)
> https://postimg.cc/q6wHvrcC
Nope, if you get such a deadlock, then your only hope is Xenomai's
Watchdog killing the process.
It should happen very rarely (but thats not an argument if your
software should run for years).
Norbert
^ permalink raw reply [flat|nested] 16+ messages in thread