Re: EVL Thread Priorities

[Philippe Gerum <rpm@xenomai.org>]

Philippe Gerum <rpm@xenomai.org> writes:

> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>
>> [[S/MIME Signed Part:Undecided]]
>> Hello,
>>
>>  
>>
>> We are seeing an issue with setting thread priorities. I’m not sure if this is a real problem that is contributing to our issues, or if it’s just
>> something weird happening. 
>>
>>  
>>
>> Here is a list of the real time threads and the priority and CPU affinity settings we’re asking for.
>>
>>  
>>
>> Thread               Priority  CPU
>>
>> ___________________________
>>
>> Main                         50     0
>>
>> TxStartup                 52     0
>>
>> RxStartup                 51     0
>>
>> DummyTxConfig     89     2
>>
>> DummyTxService    88     3
>>
>> TxMain                      87     4
>>
>> TxWorker(n)             86    8-11
>>
>> DummyTxSample    85     5
>>
>> DummyRxConfig     79     2
>>
>> DummyRxService    78     3
>>
>> RxMain                      77     4
>>
>> RxWorker(n)             76    8-11
>>
>>  
>>
>> If we run ‘top’ we can see the priority settings:
>>
>>   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
>>
>> 32130 root      20   0 3731268   2.5g 110748 S  0.0 16.2   0:05.54 realtime_sfhm
>>
>> 32140 root      20   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 sigthread
>>
>> 32144 root     -51   0 3731268   2.5g 110748 S  0.0 16.2   0:00.09 Main
>>
>> 32150 root     -53   0 3731268   2.5g 110748 S  0.0 16.2   0:00.21 TxStartup
>>
>> 32151 root     -86   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 DummyTxSample
>>
>> 32152 root     -87   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 TxWorker0
>>
>> 32153 root     -87   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 TxWorker1
>>
>> 32154 root     -87   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 TxWorker2
>>
>> 32155 root     -52   0 3731268   2.5g 110748 S  0.0 16.2   0:00.13 RxStartup
>>
>> 32156 root     -87   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 TxWorker3
>>
>> 32157 root     -88   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 TxMain
>>
>> 32158 root     -89   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 DummyTxService
>>
>> 32159 root     -80   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 DummyRxConfig
>>
>> 32160 root     -78   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 RxMain
>>
>> 32161 root     -77   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 RxWorker0
>>
>> 32162 root     -79   0 3731268   2.5g 110748 S  0.0 16.2   0:00.00 DummyRxService
>>
>>  
>>
>> Top apparently has this weirdness where it displays real-time thread priorities as –(P+1), which I guess is how it’s maintained in the kernel.
>> This seems to be a known issue, but it appears that, with this caveat, that the thread priorities are being set properly within the pthreads
>> implementation. The two non-realtime threads are part of our startup process. They never execute within the EVL context. 
>>
>>  
>>
>> However, if we look at the same threads using ‘evl ps’, we get this result:
>>
>> CPU   PID   SCHED   PRIO  ISW     CTXSW     SYS       RWA       STAT     TIMEOUT      %CPU   CPUTIME     NAME
>>
>>   0   32144  fifo    50   34      1316      2910      0          Dto     011.587        0.0  00:027.437  Main
>>
>>   0   32150  fifo    52   4       1286      1974      0          Dto     075.003        0.0  00:023.977  TxStartup
>>
>>   5   32151  fifo    87   0       51201     587379    49931      wbto    907.752        1.0  01:343.085  DummyTxSample
>>
>>   8   32152  fifo    87   0       19580     207489    19653      wbto    998.796        1.0  01:240.410  TxWorker0
>>
>>   9   32153  fifo    87   0       19000     206602    19071      wbto    905.257        1.0  01:242.255  TxWorker1
>>
>> 10   32154  fifo    87   0       16976     197880    17049      wbto    901.719        1.0  01:314.756  TxWorker2
>>
>>   0   32155  fifo    87   4       141       646       8          Dbto    966.055        0.0  00:007.204  RxStartup
>>
>> 11   32156  fifo    87   0       16372     180026    16439      wbto    907.961        1.0  01:273.781  TxWorker3
>>
>>   4   32157  fifo    87   0       13443     851659    13390      wto     902.939        1.5  01:929.911  TxMain
>>
>>   3   32158  fifo    88   0       1281      3845      0          Dto     004.172        0.0  00:039.957  DummyTxService
>>
>>   2   32159  fifo    87   0       6515      260458    5314       wbto    073.624        0.7  00:858.282  DummyRxConfig
>>
>>   4   32160  fifo    87   0       49059     626499    49185      wbto    972.854        2.4  03:082.805  RxMain
>>
>>   8   32161  fifo    87   0       33436     386429    32535      wbto    065.114        1.8  02:317.974  RxWorker0
>>
>>   3   32162  fifo    87   0       37243     378069    36132      Dbto    071.350        1.9  02:374.748  DummyRxService
>>
>>  
>>
>> CPU affinity settings look to be correct, but the priorities shown by ‘evl ps’ are not consistent with those from ‘top’. Furthermore, there seems
>> to be some difference between runs. Sometimes the priorities will be closer to those we asked for. At other times it looks more like this, where
>> they all appear to be set to 87.
>>
>>  
>>
>> I don’t know if this is just a mistake by the ‘evl ps’ command, or if the EVL scheduler is actually using the wrong priorities for scheduling.
>> Relative to our other issue that you’re looking at, this one is definitely of lower importance, unless it is related in some way to that other
>> problem.
>
> The 'b' flag appearing in the STAT field for a thread means that it is
> undergoing a priority boost due to mutex priority
> inheritance/protection, PRIO displays the effective priority which may
> be boosted - not the base priority.
>
> By the way, if evl ps is able to catch this output frequently, this
> means that your app triggers *a lot* of priority inheritance, maybe a
> bit too much/frequent sharing of resources between threads which belong
> to distinct priority groups?

CAUTION: if these priorities remain unchanged over time, then there
could be an issue in the core, i.e. some threads not being deboosted
properly. You may want to check this by running a display loop, e.g.

# watch -n1 evl ps -l

-- 
Philippe.