sysemu SMP scheduling

sysemu SMP scheduling

[Brian Cain]

For some hexagon use cases, we would prefer to have finer grained scheduling among multiple guest cores/threads.  We haven't been able to determine exactly what kind of scheduling algorithm is operating in the baseline case.  If the current hw thread is ready-to-run and is spinning over a tight loop that never hits any exceptions, would it ever yield to another thread after so-many iterations/TBs executed?  Or perhaps since we're executing translated blocks there's just no yield opportunity available?

We came up with a design for this finer-grained scheduling feature, but are re-examining whether or not it should be necessary and if it is necessary, whether it should have been designed like so.  We haven't seen a similar example in other targets, so we'd love to get feedback on the approach.

In the TranslatorOps .tb_stop() we generate code like so:

	if the current count of ready-to-run threads >= 2:
		tb_count++
		if tb_count > THRESHOLD:
			gen_exception(EXCP_YIELD);
			tb_count = 0
			gen exit_tb

- "current count of ready-to-run threads" is based on the values in the CPU state.  When entering a wait/halt mode, we set the appropriate state and call cpu_stop_current().
- Is EXCP_YIELD an appropriate mechanism for this feature?  It seems like maybe it has no special handling, but any exception can trigger a yield?

-Brian

RE: sysemu SMP scheduling

[Brian Cain]

> -----Original Message-----
> From: Brian Cain
> Sent: Wednesday, April 28, 2021 10:06 PM
> To: qemu-devel@nongnu.org
> Cc: Richard Henderson <richard.henderson@linaro.org>; Taylor Simpson
> <tsimpson@quicinc.com>; Michael Lambert <mlambert@quicinc.com>;
> Manning, Sid <sidneym@quicinc.com>
> Subject: sysemu SMP scheduling
> 
> For some hexagon use cases, we would prefer to have finer grained scheduling
> among multiple guest cores/threads.  We haven't been able to determine
> exactly what kind of scheduling algorithm is operating in the baseline case.  If
> the current hw thread is ready-to-run and is spinning over a tight loop that
> never hits any exceptions, would it ever yield to another thread after so-many
> iterations/TBs executed?  Or perhaps since we're executing translated blocks
> there's just no yield opportunity available?
> 
> We came up with a design for this finer-grained scheduling feature, but are re-
> examining whether or not it should be necessary and if it is necessary, whether
> it should have been designed like so.  We haven't seen a similar example in
> other targets, so we'd love to get feedback on the approach.
> 
> In the TranslatorOps .tb_stop() we generate code like so:
> 
> 	if the current count of ready-to-run threads >= 2:
> 		tb_count++
> 		if tb_count > THRESHOLD:
> 			gen_exception(EXCP_YIELD);
> 			tb_count = 0
> 			gen exit_tb

Hmm, I omitted some critical details of this algorithm.

tb_stop, corrected:

 	if the current count of ready-to-run threads >= 2:
 		consecutive_tb_count++
 		if consecutive_tb_count > THRESHOLD:
 			gen_exception(EXCP_YIELD);
 			consecutive_tb_count = 0
 			gen exit_tb
	last_cpu_id = this_cpu_id

tb_start generates code like this:

	if this_cpu_id != last_cpu_id:
		consecutive_tb_count = 0

> - "current count of ready-to-run threads" is based on the values in the CPU
> state.  When entering a wait/halt mode, we set the appropriate state and call
> cpu_stop_current().
> - Is EXCP_YIELD an appropriate mechanism for this feature?  It seems like
> maybe it has no special handling, but any exception can trigger a yield?
> 
> -Brian