[Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[解壁伟]

Hi there:

I've read this an article discribe the thread model in Qemu.
http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
Is there really one thread per vcpu in qemu?

When I run the qemu using "-smp 16", I can see only two threads using
"ps -xH".

So I want to know that if qemu excuting the vcpu in one thread per vcpu way?

or, qemu simply excuting all the vcpus in one thread?

How can I trace the generation of the vcpu thread?

I'll be appreciate for your help~

puck
2013-3-12

Re: [Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[Laszlo Ersek]

On 03/12/13 10:34, 解壁伟 wrote:
> Hi there:
> 
> I've read this an article discribe the thread model in Qemu.
> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
> Is there really one thread per vcpu in qemu?
> 
> When I run the qemu using "-smp 16", I can see only two threads using
> "ps -xH".

What ps is that? I'd try ps -efL.

> 
> So I want to know that if qemu excuting the vcpu in one thread per vcpu way?
> 
> or, qemu simply excuting all the vcpus in one thread?
> 
> How can I trace the generation of the vcpu thread?

qemu_init_vcpu() --> qemu_kvm_start_vcpu() [cpus.c]

Re: [Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[Stefan Hajnoczi]

On Tue, Mar 12, 2013 at 05:34:24PM +0800, 解壁伟 wrote:
> Hi there:
> 
> I've read this an article discribe the thread model in Qemu.
> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
> Is there really one thread per vcpu in qemu?
> 
> When I run the qemu using "-smp 16", I can see only two threads using
> "ps -xH".

"Note that TCG is not thread-safe so even under the iothread model it
multiplexes vcpus across a single QEMU thread. Only KVM can take
advantage of per-vcpu threads."

If you do not have -enable-kvm then you're probably using TCG.

Stefan

Re: [Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[解壁伟]

Thank you very much.

I've find the generation of thread in qemu, following your advice.

qemu_init_vcpu() --> qemu_kvm_start_vcpu() [cpus.c]

puck

于 2013/3/13 6:26, Laszlo Ersek 写道:
> On 03/12/13 10:34, 解壁伟 wrote:
>> Hi there:
>>
>> I've read this an article discribe the thread model in Qemu.
>> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
>> Is there really one thread per vcpu in qemu?
>>
>> When I run the qemu using "-smp 16", I can see only two threads using
>> "ps -xH".
> What ps is that? I'd try ps -efL.
>
>> So I want to know that if qemu excuting the vcpu in one thread per vcpu way?
>>
>> or, qemu simply excuting all the vcpus in one thread?
>>
>> How can I trace the generation of the vcpu thread?
> qemu_init_vcpu() --> qemu_kvm_start_vcpu() [cpus.c]
>
>

Re: [Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[解壁伟]

I read the article again, and I get the answer to my question in that 
paragraph.
Thank you very much.
As you've said, TCG's not thread-safe.
Is it only the shared data structure(e.g TB) that prevent TCG to be 
thread-safe? I'm seeking a method to parallize TCG in order to parallize 
Qemu.

And,how do the multi-thread in kvm mode Qemu synchronize to each other? 
For some threads maybe excute faster or slower than others, how do they 
synchronize the clock with each other? Using a huge lock or a global clock?

I'll be appreciate if you can give me some hints for further exploration.

For now, I need a emulator to emulate more cores efficiently.I want to 
analyze whether Qemu can be faster when it emulate more cores, while 
parallelism maybe a good choice.


puck




于 2013/3/13 16:45, Stefan Hajnoczi 写道:
> On Tue, Mar 12, 2013 at 05:34:24PM +0800, 解壁伟 wrote:
>> Hi there:
>>
>> I've read this an article discribe the thread model in Qemu.
>> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
>> Is there really one thread per vcpu in qemu?
>>
>> When I run the qemu using "-smp 16", I can see only two threads using
>> "ps -xH".
> "Note that TCG is not thread-safe so even under the iothread model it
> multiplexes vcpus across a single QEMU thread. Only KVM can take
> advantage of per-vcpu threads."
>
> If you do not have -enable-kvm then you're probably using TCG.
>
> Stefan
>

Re: [Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[Peter Maydell]

On 13 March 2013 09:32, 解壁伟 <xiebiwei@ict.ac.cn> wrote:
> I read the article again, and I get the answer to my question in that
> paragraph.
> Thank you very much.
> As you've said, TCG's not thread-safe.
> Is it only the shared data structure(e.g TB) that prevent TCG to be
> thread-safe? I'm seeking a method to parallize TCG in order to parallize
> Qemu.

Yes, the major problem is the non-thread-safe code generation
data structures. (I fixed one set of non-threadsafe code recently
but there are still a lot to do. Needs thought about the right
design principles.)

> For now, I need a emulator to emulate more cores efficiently.I want to
> analyze whether Qemu can be faster when it emulate more cores, while
> parallelism maybe a good choice.

There has been some other research work with QEMU in this
area, you should probably look it up. However it has all been
"we fixed enough of the big problems to be able to run some
benchmarks and write our research paper", so none of this is
actually in the QEMU sources. (The short answer I think is
"probably but it will depend a lot on workload, and there's
a lot of software engineering required to do it reliably".)

-- PMM

Re: [Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[Laurent Desnogues]

On Wed, Mar 13, 2013 at 10:51 AM, Peter Maydell
<peter.maydell@linaro.org> wrote:
[...]
>> For now, I need a emulator to emulate more cores efficiently.I want to
>> analyze whether Qemu can be faster when it emulate more cores, while
>> parallelism maybe a good choice.
>
> There has been some other research work with QEMU in this
> area, you should probably look it up. However it has all been
> "we fixed enough of the big problems to be able to run some
> benchmarks and write our research paper", so none of this is
> actually in the QEMU sources. (The short answer I think is
> "probably but it will depend a lot on workload, and there's
> a lot of software engineering required to do it reliably".)

I know of two such variants: COREMU and HQEMU, but as far
as I know the latter isn't freely available.


Laurent

Re: [Qemu-devel] About the Thread model in Qemu【 Is it one thread per vcpu?】

[Paolo Bonzini]

Il 13/03/2013 10:56, Laurent Desnogues ha scritto:
>> > There has been some other research work with QEMU in this
>> > area, you should probably look it up. However it has all been
>> > "we fixed enough of the big problems to be able to run some
>> > benchmarks and write our research paper", so none of this is
>> > actually in the QEMU sources. (The short answer I think is
>> > "probably but it will depend a lot on workload, and there's
>> > a lot of software engineering required to do it reliably".)
> I know of two such variants: COREMU and HQEMU, but as far
> as I know the latter isn't freely available.

COREMU had one code generator per VCPU, if I remember correctly.

Jan Kiszka also posted an RFC patch that removed the "lockstep" behavior
of TCG, where the VCPU will always block the iothread even when it is
not executing MMIO.  Getting this upstream would be an even bigger step
than making the code generator thread-safe.

Paolo

[Qemu-devel] Round robin of multi-vcpu in qemu

[解壁伟]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

Hi there:

I'm studying the execution details of multi-vcpu in TCG mode.

The vcpus in TCG mode are executed one by one in a sequencial way.

I read the function [tcg_exec_all] in Qemu 1.3.0 as bellow, but the
implementation is not as I expected.

[tcg_exec_all] will finally call [cpu_exec] to excute a loop to excute
the TBs in code cache

So,how does these functions control the running time of each VCPU?

That is,when will the execution of one VCPU return, in order to execute
the next_cpu in the loop of [tcg_exec_all].

I've examined the alarm timmer, this timer is used to interrupt the
execution of VCPUs to handle interruptions.

Does Qemu use a similar way (timer) to control the round robin of these
VCPUs?

Or, does Qemu use the iCount to prevent a VCPU from running too long?

        staticvoidtcg_exec_all(void)
        {
        intr;

        /* Account partial waits to the vm_clock. */
        qemu_clock_warp(vm_clock);

        if(next_cpu==NULL){
        next_cpu=first_cpu;
        }
        for(;next_cpu!=NULL&&!exit_request;next_cpu=next_cpu->next_cpu){
        CPUArchState*env=next_cpu;
        CPUState*cpu=ENV_GET_CPU(env);

        qemu_clock_enable(vm_clock,
        (env->singlestep_enabled&SSTEP_NOTIMER)==0);

        if(cpu_can_run(cpu)){
        r=tcg_cpu_exec(env);
        if(r==EXCP_DEBUG){
        cpu_handle_guest_debug(env);
        break;
        }
        }elseif(cpu->stop||cpu->stopped){
        break;
        }
        }
        exit_request=0;
        }

Yours
Biwei

[-- Attachment #2: Type: text/html, Size: 11091 bytes --]

Re: [Qemu-devel] Round robin of multi-vcpu in qemu

[陳韋任 (Wei-Ren Chen)]

> I've examined the alarm timmer, this timer is used to interrupt the
> execution of VCPUs to handle interruptions.

  IIRC, alarm timer will interrupt VCPU to execute the next one.
You can check where QEMU register the alarm timer, and see how it
works.

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj