Thread goes into deep-deep-state and not woken-up

Thread goes into deep-deep-state and not woken-up

[Ananda Sathyanarayana]

Hi All,

We have a customer critical issue, where one of the application thread  is not getting woken up by the OS after calling usleep. This is causing the appliance and in turn the services to go  down. When the customer connects a console cable, the application thread is woken up and the services are restored.

Appliance : 8 core Atom Rangeley based Intel(R) Atom(TM) CPU  C2758  @ 2.40GHz
Linux OS :  Linux versa-flexvnf 4.4.0-142-generic, x86_64 x86_64 x86_64 GNU/Linux
Isolcpu is enabled. Core 1-7 are isolated. 
Application : User space application

Cores 1 to 7 poll for x (in micro sec) time and if no work sleep for y amount of time (in micro sec). What we are observing is 1 out 7 cores goes into this "non-responsive" state. This issue is not seen not very often (it is very hard to recreate). Once we connect the console and press few key strokes, the CPU in non-responsive state become active.

Even though c-state is disabled in the BIOS, we do see c-state transitions under ./sys/bus/cpu/drivers/processor/cpux/cpuidle directory

We also observed the local timer interrupts were stopped for that core when the issue happened.

Could you please help us understand 
-	Possible reasons for the kernel not to schedule one of the threads at all (not waking from the deep sleep state)
-	If there is a theory, how to recreate this issue on a consistent basis

Your help is very much appreciated.

Thanks
Ananda