high-priority RT CPU-hog applications and kernel threads

[Chris Friesen <chris.friesen@windriver.com>]

Hi,

I'm on the CentOS 7 RT kernel but I'm hoping the general principles 
still apply.

I've got a system with a number of CPUs set aside for housekeeping and 
others dedicated to low-latency applications.  We have "irqaffinity", 
"rcu_nocbs", "nohz_full", and "isolcpus" set up to try to isolate the 
application CPUs as much as possible, with RCU threads affined to the 
housekeeping CPUs where possible.  (rcuc/N is still on the application 
CPUs.)  We have CONFIG_RCU_BOOST=y and CONFIG_RCU_KTHREAD_PRIO=2.

One packet-processing application wants to run in a busy-loop pulling 
packets off queues and shoving them onto other queues.  No actual system 
calls that context-switch into the kernel, only calls to get a timestamp 
which are handled by the vDSO.  Testing has shown better jitter if we 
run this higher priority than all the kernel threads, but I wanted to 
make sure this was kosher since the various tuning guides seem to 
recommend against running high-RT-priority CPU hogs.

Is it okay to run the application higher-priority than all the RCU 
threads since the bulk of the work is offloaded to the housekeeping 
CPUs?  Would we need "rcu_nocb_poll" to run the application 
higher-priority than rcuc/N?

Is it okay to run the application higher-priority than ktimersoftd/N 
given that the application doesn't use timers and nothing else should be 
running on that CPU?

Is it okay run the application higher-priority than the irq threads or 
is that a moot point as long as we affine all the irq threads to the 
housekeeping CPUs?

Is there anything else I'm missing?  Basically as much as possible I 
just want this CPU to be entirely dedicated to the application, with 
nothing else running on it.  Most of the tuning guides want the kernel 
threads higher-priority than the application though, and I'd like to 
avoid that if possible.

Thanks,
Chris