* EVL Thread Priorities
@ 2022-11-17 15:57 Russell Johnson
2022-11-17 16:20 ` Philippe Gerum
0 siblings, 1 reply; 3+ messages in thread
From: Russell Johnson @ 2022-11-17 15:57 UTC (permalink / raw)
To: xenomai, Philippe Gerum; +Cc: Bryan Butler, Shawn McManus
[-- Attachment #1.1: Type: text/plain, Size: 4980 bytes --]
Hello,
We are seeing an issue with setting thread priorities. I'm not sure if this
is a real problem that is contributing to our issues, or if it's just
something weird happening.
Here is a list of the real time threads and the priority and CPU affinity
settings we're asking for.
Thread Priority CPU
___________________________
Main 50 0
TxStartup 52 0
RxStartup 51 0
DummyTxConfig 89 2
DummyTxService 88 3
TxMain 87 4
TxWorker(n) 86 8-11
DummyTxSample 85 5
DummyRxConfig 79 2
DummyRxService 78 3
RxMain 77 4
RxWorker(n) 76 8-11
If we run 'top' we can see the priority settings:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32130 root 20 0 3731268 2.5g 110748 S 0.0 16.2 0:05.54
realtime_sfhm
32140 root 20 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 sigthread
32144 root -51 0 3731268 2.5g 110748 S 0.0 16.2 0:00.09 Main
32150 root -53 0 3731268 2.5g 110748 S 0.0 16.2 0:00.21 TxStartup
32151 root -86 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00
DummyTxSample
32152 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker0
32153 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker1
32154 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker2
32155 root -52 0 3731268 2.5g 110748 S 0.0 16.2 0:00.13 RxStartup
32156 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker3
32157 root -88 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxMain
32158 root -89 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00
DummyTxService
32159 root -80 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00
DummyRxConfig
32160 root -78 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 RxMain
32161 root -77 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 RxWorker0
32162 root -79 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00
DummyRxService
Top apparently has this weirdness where it displays real-time thread
priorities as -(P+1), which I guess is how it's maintained in the kernel.
This seems to be a known issue, but it appears that, with this caveat, that
the thread priorities are being set properly within the pthreads
implementation. The two non-realtime threads are part of our startup
process. They never execute within the EVL context.
However, if we look at the same threads using 'evl ps', we get this result:
CPU PID SCHED PRIO ISW CTXSW SYS RWA STAT
TIMEOUT %CPU CPUTIME NAME
0 32144 fifo 50 34 1316 2910 0 Dto
011.587 0.0 00:027.437 Main
0 32150 fifo 52 4 1286 1974 0 Dto
075.003 0.0 00:023.977 TxStartup
5 32151 fifo 87 0 51201 587379 49931 wbto
907.752 1.0 01:343.085 DummyTxSample
8 32152 fifo 87 0 19580 207489 19653 wbto
998.796 1.0 01:240.410 TxWorker0
9 32153 fifo 87 0 19000 206602 19071 wbto
905.257 1.0 01:242.255 TxWorker1
10 32154 fifo 87 0 16976 197880 17049 wbto
901.719 1.0 01:314.756 TxWorker2
0 32155 fifo 87 4 141 646 8 Dbto
966.055 0.0 00:007.204 RxStartup
11 32156 fifo 87 0 16372 180026 16439 wbto
907.961 1.0 01:273.781 TxWorker3
4 32157 fifo 87 0 13443 851659 13390 wto
902.939 1.5 01:929.911 TxMain
3 32158 fifo 88 0 1281 3845 0 Dto
004.172 0.0 00:039.957 DummyTxService
2 32159 fifo 87 0 6515 260458 5314 wbto
073.624 0.7 00:858.282 DummyRxConfig
4 32160 fifo 87 0 49059 626499 49185 wbto
972.854 2.4 03:082.805 RxMain
8 32161 fifo 87 0 33436 386429 32535 wbto
065.114 1.8 02:317.974 RxWorker0
3 32162 fifo 87 0 37243 378069 36132 Dbto
071.350 1.9 02:374.748 DummyRxService
CPU affinity settings look to be correct, but the priorities shown by 'evl
ps' are not consistent with those from 'top'. Furthermore, there seems to be
some difference between runs. Sometimes the priorities will be closer to
those we asked for. At other times it looks more like this, where they all
appear to be set to 87.
I don't know if this is just a mistake by the 'evl ps' command, or if the
EVL scheduler is actually using the wrong priorities for scheduling.
Relative to our other issue that you're looking at, this one is definitely
of lower importance, unless it is related in some way to that other problem.
Thanks,
Russell
[-- Attachment #1.2: Type: text/html, Size: 13805 bytes --]
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: EVL Thread Priorities
2022-11-17 15:57 EVL Thread Priorities Russell Johnson
@ 2022-11-17 16:20 ` Philippe Gerum
2022-11-17 16:43 ` Philippe Gerum
0 siblings, 1 reply; 3+ messages in thread
From: Philippe Gerum @ 2022-11-17 16:20 UTC (permalink / raw)
To: Russell Johnson; +Cc: xenomai, Bryan Butler, Shawn McManus
Russell Johnson <russell.johnson@kratosdefense.com> writes:
> [[S/MIME Signed Part:Undecided]]
> Hello,
>
>
>
> We are seeing an issue with setting thread priorities. I’m not sure if this is a real problem that is contributing to our issues, or if it’s just
> something weird happening.
>
>
>
> Here is a list of the real time threads and the priority and CPU affinity settings we’re asking for.
>
>
>
> Thread Priority CPU
>
> ___________________________
>
> Main 50 0
>
> TxStartup 52 0
>
> RxStartup 51 0
>
> DummyTxConfig 89 2
>
> DummyTxService 88 3
>
> TxMain 87 4
>
> TxWorker(n) 86 8-11
>
> DummyTxSample 85 5
>
> DummyRxConfig 79 2
>
> DummyRxService 78 3
>
> RxMain 77 4
>
> RxWorker(n) 76 8-11
>
>
>
> If we run ‘top’ we can see the priority settings:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 32130 root 20 0 3731268 2.5g 110748 S 0.0 16.2 0:05.54 realtime_sfhm
>
> 32140 root 20 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 sigthread
>
> 32144 root -51 0 3731268 2.5g 110748 S 0.0 16.2 0:00.09 Main
>
> 32150 root -53 0 3731268 2.5g 110748 S 0.0 16.2 0:00.21 TxStartup
>
> 32151 root -86 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyTxSample
>
> 32152 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker0
>
> 32153 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker1
>
> 32154 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker2
>
> 32155 root -52 0 3731268 2.5g 110748 S 0.0 16.2 0:00.13 RxStartup
>
> 32156 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker3
>
> 32157 root -88 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxMain
>
> 32158 root -89 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyTxService
>
> 32159 root -80 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyRxConfig
>
> 32160 root -78 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 RxMain
>
> 32161 root -77 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 RxWorker0
>
> 32162 root -79 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyRxService
>
>
>
> Top apparently has this weirdness where it displays real-time thread priorities as –(P+1), which I guess is how it’s maintained in the kernel.
> This seems to be a known issue, but it appears that, with this caveat, that the thread priorities are being set properly within the pthreads
> implementation. The two non-realtime threads are part of our startup process. They never execute within the EVL context.
>
>
>
> However, if we look at the same threads using ‘evl ps’, we get this result:
>
> CPU PID SCHED PRIO ISW CTXSW SYS RWA STAT TIMEOUT %CPU CPUTIME NAME
>
> 0 32144 fifo 50 34 1316 2910 0 Dto 011.587 0.0 00:027.437 Main
>
> 0 32150 fifo 52 4 1286 1974 0 Dto 075.003 0.0 00:023.977 TxStartup
>
> 5 32151 fifo 87 0 51201 587379 49931 wbto 907.752 1.0 01:343.085 DummyTxSample
>
> 8 32152 fifo 87 0 19580 207489 19653 wbto 998.796 1.0 01:240.410 TxWorker0
>
> 9 32153 fifo 87 0 19000 206602 19071 wbto 905.257 1.0 01:242.255 TxWorker1
>
> 10 32154 fifo 87 0 16976 197880 17049 wbto 901.719 1.0 01:314.756 TxWorker2
>
> 0 32155 fifo 87 4 141 646 8 Dbto 966.055 0.0 00:007.204 RxStartup
>
> 11 32156 fifo 87 0 16372 180026 16439 wbto 907.961 1.0 01:273.781 TxWorker3
>
> 4 32157 fifo 87 0 13443 851659 13390 wto 902.939 1.5 01:929.911 TxMain
>
> 3 32158 fifo 88 0 1281 3845 0 Dto 004.172 0.0 00:039.957 DummyTxService
>
> 2 32159 fifo 87 0 6515 260458 5314 wbto 073.624 0.7 00:858.282 DummyRxConfig
>
> 4 32160 fifo 87 0 49059 626499 49185 wbto 972.854 2.4 03:082.805 RxMain
>
> 8 32161 fifo 87 0 33436 386429 32535 wbto 065.114 1.8 02:317.974 RxWorker0
>
> 3 32162 fifo 87 0 37243 378069 36132 Dbto 071.350 1.9 02:374.748 DummyRxService
>
>
>
> CPU affinity settings look to be correct, but the priorities shown by ‘evl ps’ are not consistent with those from ‘top’. Furthermore, there seems
> to be some difference between runs. Sometimes the priorities will be closer to those we asked for. At other times it looks more like this, where
> they all appear to be set to 87.
>
>
>
> I don’t know if this is just a mistake by the ‘evl ps’ command, or if the EVL scheduler is actually using the wrong priorities for scheduling.
> Relative to our other issue that you’re looking at, this one is definitely of lower importance, unless it is related in some way to that other
> problem.
The 'b' flag appearing in the STAT field for a thread means that it is
undergoing a priority boost due to mutex priority
inheritance/protection, PRIO displays the effective priority which may
be boosted - not the base priority.
By the way, if evl ps is able to catch this output frequently, this
means that your app triggers *a lot* of priority inheritance, maybe a
bit too much/frequent sharing of resources between threads which belong
to distinct priority groups?
--
Philippe.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: EVL Thread Priorities
2022-11-17 16:20 ` Philippe Gerum
@ 2022-11-17 16:43 ` Philippe Gerum
0 siblings, 0 replies; 3+ messages in thread
From: Philippe Gerum @ 2022-11-17 16:43 UTC (permalink / raw)
To: Russell Johnson; +Cc: xenomai, Bryan Butler, Shawn McManus
Philippe Gerum <rpm@xenomai.org> writes:
> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>
>> [[S/MIME Signed Part:Undecided]]
>> Hello,
>>
>>
>>
>> We are seeing an issue with setting thread priorities. I’m not sure if this is a real problem that is contributing to our issues, or if it’s just
>> something weird happening.
>>
>>
>>
>> Here is a list of the real time threads and the priority and CPU affinity settings we’re asking for.
>>
>>
>>
>> Thread Priority CPU
>>
>> ___________________________
>>
>> Main 50 0
>>
>> TxStartup 52 0
>>
>> RxStartup 51 0
>>
>> DummyTxConfig 89 2
>>
>> DummyTxService 88 3
>>
>> TxMain 87 4
>>
>> TxWorker(n) 86 8-11
>>
>> DummyTxSample 85 5
>>
>> DummyRxConfig 79 2
>>
>> DummyRxService 78 3
>>
>> RxMain 77 4
>>
>> RxWorker(n) 76 8-11
>>
>>
>>
>> If we run ‘top’ we can see the priority settings:
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>> 32130 root 20 0 3731268 2.5g 110748 S 0.0 16.2 0:05.54 realtime_sfhm
>>
>> 32140 root 20 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 sigthread
>>
>> 32144 root -51 0 3731268 2.5g 110748 S 0.0 16.2 0:00.09 Main
>>
>> 32150 root -53 0 3731268 2.5g 110748 S 0.0 16.2 0:00.21 TxStartup
>>
>> 32151 root -86 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyTxSample
>>
>> 32152 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker0
>>
>> 32153 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker1
>>
>> 32154 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker2
>>
>> 32155 root -52 0 3731268 2.5g 110748 S 0.0 16.2 0:00.13 RxStartup
>>
>> 32156 root -87 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxWorker3
>>
>> 32157 root -88 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 TxMain
>>
>> 32158 root -89 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyTxService
>>
>> 32159 root -80 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyRxConfig
>>
>> 32160 root -78 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 RxMain
>>
>> 32161 root -77 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 RxWorker0
>>
>> 32162 root -79 0 3731268 2.5g 110748 S 0.0 16.2 0:00.00 DummyRxService
>>
>>
>>
>> Top apparently has this weirdness where it displays real-time thread priorities as –(P+1), which I guess is how it’s maintained in the kernel.
>> This seems to be a known issue, but it appears that, with this caveat, that the thread priorities are being set properly within the pthreads
>> implementation. The two non-realtime threads are part of our startup process. They never execute within the EVL context.
>>
>>
>>
>> However, if we look at the same threads using ‘evl ps’, we get this result:
>>
>> CPU PID SCHED PRIO ISW CTXSW SYS RWA STAT TIMEOUT %CPU CPUTIME NAME
>>
>> 0 32144 fifo 50 34 1316 2910 0 Dto 011.587 0.0 00:027.437 Main
>>
>> 0 32150 fifo 52 4 1286 1974 0 Dto 075.003 0.0 00:023.977 TxStartup
>>
>> 5 32151 fifo 87 0 51201 587379 49931 wbto 907.752 1.0 01:343.085 DummyTxSample
>>
>> 8 32152 fifo 87 0 19580 207489 19653 wbto 998.796 1.0 01:240.410 TxWorker0
>>
>> 9 32153 fifo 87 0 19000 206602 19071 wbto 905.257 1.0 01:242.255 TxWorker1
>>
>> 10 32154 fifo 87 0 16976 197880 17049 wbto 901.719 1.0 01:314.756 TxWorker2
>>
>> 0 32155 fifo 87 4 141 646 8 Dbto 966.055 0.0 00:007.204 RxStartup
>>
>> 11 32156 fifo 87 0 16372 180026 16439 wbto 907.961 1.0 01:273.781 TxWorker3
>>
>> 4 32157 fifo 87 0 13443 851659 13390 wto 902.939 1.5 01:929.911 TxMain
>>
>> 3 32158 fifo 88 0 1281 3845 0 Dto 004.172 0.0 00:039.957 DummyTxService
>>
>> 2 32159 fifo 87 0 6515 260458 5314 wbto 073.624 0.7 00:858.282 DummyRxConfig
>>
>> 4 32160 fifo 87 0 49059 626499 49185 wbto 972.854 2.4 03:082.805 RxMain
>>
>> 8 32161 fifo 87 0 33436 386429 32535 wbto 065.114 1.8 02:317.974 RxWorker0
>>
>> 3 32162 fifo 87 0 37243 378069 36132 Dbto 071.350 1.9 02:374.748 DummyRxService
>>
>>
>>
>> CPU affinity settings look to be correct, but the priorities shown by ‘evl ps’ are not consistent with those from ‘top’. Furthermore, there seems
>> to be some difference between runs. Sometimes the priorities will be closer to those we asked for. At other times it looks more like this, where
>> they all appear to be set to 87.
>>
>>
>>
>> I don’t know if this is just a mistake by the ‘evl ps’ command, or if the EVL scheduler is actually using the wrong priorities for scheduling.
>> Relative to our other issue that you’re looking at, this one is definitely of lower importance, unless it is related in some way to that other
>> problem.
>
> The 'b' flag appearing in the STAT field for a thread means that it is
> undergoing a priority boost due to mutex priority
> inheritance/protection, PRIO displays the effective priority which may
> be boosted - not the base priority.
>
> By the way, if evl ps is able to catch this output frequently, this
> means that your app triggers *a lot* of priority inheritance, maybe a
> bit too much/frequent sharing of resources between threads which belong
> to distinct priority groups?
CAUTION: if these priorities remain unchanged over time, then there
could be an issue in the core, i.e. some threads not being deboosted
properly. You may want to check this by running a display loop, e.g.
# watch -n1 evl ps -l
--
Philippe.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-11-17 16:45 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-17 15:57 EVL Thread Priorities Russell Johnson
2022-11-17 16:20 ` Philippe Gerum
2022-11-17 16:43 ` Philippe Gerum
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.