From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:58894)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <alex.bennee@linaro.org>) id 1bDCAD-0003h9-S7
	for qemu-devel@nongnu.org; Wed, 15 Jun 2016 10:51:34 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <alex.bennee@linaro.org>) id 1bDCA9-0003hh-Iu
	for qemu-devel@nongnu.org; Wed, 15 Jun 2016 10:51:29 -0400
Received: from mail-lf0-x22a.google.com ([2a00:1450:4010:c07::22a]:35398)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <alex.bennee@linaro.org>) id 1bDCA8-0003h6-OQ
	for qemu-devel@nongnu.org; Wed, 15 Jun 2016 10:51:25 -0400
Received: by mail-lf0-x22a.google.com with SMTP id l188so13289318lfe.2
	for <qemu-devel@nongnu.org>; Wed, 15 Jun 2016 07:51:24 -0700 (PDT)
From: Alex =?utf-8?Q?Benn=C3=A9e?= <alex.bennee@linaro.org>
In-reply-to: <CAH47eN0WNnUY-bjunu9KULfdo3czJhjgDmEswS4ThjaEN6vDuw@mail.gmail.com>
Date: Wed, 15 Jun 2016 15:51:26 +0100
Message-ID: <8760tah5e9.fsf@linaro.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Subject: Re: [Qemu-devel] exec: Safe work in quiescent state
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel/>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: alvise rigo <a.rigo@virtualopensystems.com>
Cc: Sergey Fedorov <serge.fdrv@gmail.com>, QEMU Developers <qemu-devel@nongnu.org>, MTTCG Devel <mttcg@listserver.greensocs.com>, KONRAD =?utf-8?B?RnLDqWTDqXJpYw==?= <fred.konrad@greensocs.com>, "Emilio G.
	Cota" <cota@braap.org>, Paolo Bonzini <pbonzini@redhat.com>, Richard Henderson <rth@twiddle.net>, Peter Maydell <peter.maydell@linaro.org>


alvise rigo <a.rigo@virtualopensystems.com> writes:

> Hi Sergey,
>
> Nice review of the implementations we have so far.
> Just few comments below.
>
> On Wed, Jun 15, 2016 at 2:59 PM, Sergey Fedorov <serge.fdrv@gmail.com> wrote:
>> On 10/06/16 00:51, Sergey Fedorov wrote:
>>> For certain kinds of tasks we might need a quiescent state to perform an
>>> operation safely. Quiescent state means no CPU thread executing, and
>>> probably BQL held as well. The tasks could include:
<snip>
>>
>> Alvise's async_wait_run_on_cpu() [3]:
>> - uses the same queue as async_run_on_cpu();
>> - the CPU that requested the job is recorded in qemu_work_item;
>> - each CPU has a counter of such jobs it has requested;
>> - the counter is decremented upon job completion;
>> - only the target CPU is forced to exit the execution loop, i.e. the job
>> is not run in quiescent state;
>
> async_wait_run_on_cpu() kicks the target VCPU before calling
> cpu_exit() on the current VCPU, so all the VCPUs are forced to exit.
> Moreover, the current VCPU waits for all the tasks to be completed.

The effect of qemu_cpu_kick() for TCG is effectively just doing a
cpu_exit() anyway. Once done any TCG code will exit on it's next
intra-block transition.

>
<snip>
>> Distilling the requirements, safe work mechanism should:
>> - support both system and user-mode emulation;
>> - allow to schedule an asynchronous operation to be performed out of CPU
>> execution loop;
>> - guarantee that all CPUs are out of execution loop before the operation
>> can begin;
>
> This requirement is probably not necessary if we need to query TLB
> flushes to other VCPUs, since every VCPU will flush its own TLB.
> For this reason we probably need to mechanisms:
> - The first allows a VCPU to query a job to all the others and wait
> for all of them to be done (like for global TLB flush)

Do we need to wait?

> - The second allows a VCPU to perform a task in quiescent state i.e.
> the task starts and finishes when all VCPUs are out of the execution
> loop (translation buffer flush)

If you really want to ensure everything is done then you can exit the
block early. To get the sort of dsb() flush semantics mentioned you
simply:

  - queue your async safe work
  - exit block on dsb()

  This ensures by the time the TCG thread restarts for the next
  instruction all pending work has been flushed.

> Does this make sense?

I think we want one way of doing things for anything that is Cross CPU
and requires a degree of synchronisation. If it ends up being too
expensive then we can look at more efficient special casing solutions.

>
>> - guarantee that no CPU enters execution loop before all the scheduled
>> operations are complete.
>
> This is probably too much in some cases for the reasons of before.
>
> Best regards,
> alvise
>
>>
>> If that sounds like a sane approach, I'll come up with a more specific
>> solution to discuss. The solution could be merged into v2.7 along with
>> safe translation buffer flush in user-mode as an actual use case. Safe
>> cross-CPU TLB flush would become a part of MTTCG work. Comments,
>> suggestions, arguments etc. are welcome!
>>
>> [1] http://thread.gmane.org/gmane.comp.emulators.qemu/355323/focus=355632
>> [2] http://thread.gmane.org/gmane.comp.emulators.qemu/407030/focus=407039
>> [3] http://thread.gmane.org/gmane.comp.emulators.qemu/413978/focus=413982
>> [4] http://thread.gmane.org/gmane.comp.emulators.qemu/356765/focus=356789
>> [5] http://thread.gmane.org/gmane.comp.emulators.qemu/397295/focus=397301
>> [6] http://thread.gmane.org/gmane.comp.emulators.qemu/413978/focus=417231
>>
>> Kind regards,
>> Sergey


--
Alex Bennée