[Qemu-devel] QEMU and Real Time OS

[Qemu-devel] QEMU and Real Time OS

[Marc Marí]

Hi

I tried to run a Linux Kernel with Preempt RT patch on a QEMU emulated
machine using TCG (ARM guest, x86_64 guest).

I expected the guest to have high latencies, but more or less constant.
But I found, using cyclictest, a lot of randomness and a lot of
difference between the average and the maximum latency.

I thought this may be due to I/O overhead. The current system has a root
filesystem through NFS, and I'm launching the tests through a serial
terminal and saving the results in files. But the high priority thread
saves the result in memory, it is the low priority thread that saves it
in files.

Is this an expected behaviour? I can't see why.

I'd like to know if there is a certain reason why it doesn't work. Or
if it should work and the problem is too much I/O overhead. Or any
other hint to understand it.

Thanks
Marc

Re: [Qemu-devel] QEMU and Real Time OS

[Paolo Bonzini]

On 29/01/2015 20:37, Marc Marí wrote:
> Is this an expected behaviour? I can't see why.
> 
> I'd like to know if there is a certain reason why it doesn't work. Or
> if it should work and the problem is too much I/O overhead. Or any
> other hint to understand it.

It is due to latencies in the host.  You need at least to use preempt-rt
kernels in the host as well.

Paolo

Re: [Qemu-devel] QEMU and Real Time OS

[Jan Kiszka]

On 2015-01-30 00:06, Paolo Bonzini wrote:
> 
> 
> On 29/01/2015 20:37, Marc Marí wrote:
>> Is this an expected behaviour? I can't see why.
>>
>> I'd like to know if there is a certain reason why it doesn't work. Or
>> if it should work and the problem is too much I/O overhead. Or any
>> other hint to understand it.
> 
> It is due to latencies in the host.  You need at least to use preempt-rt
> kernels in the host as well.

That alone won't help much. You also need to fine-tune the guest to
avoid running into QEMU locks that continuously synchronizes the guest
on things like VGA or disk I/O emulation.

When using KVM, thus being able to run VCPUs widely independent of each
other and the device models, you need to push cyclictest on an isolated
second virtual CPU of the guest. Luiz and Marcelo can probably confirm
this based on their ongoing experiments.

With TCG, we would first of all have to make it true SMP and independent
of the I/O device lock. That's what Frederic is working on [1].

Jan

[1] http://permalink.gmane.org/gmane.comp.emulators.qemu/314406

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] QEMU and Real Time OS

[Marc Marí]

El Fri, 30 Jan 2015 08:37:47 +0100
Jan Kiszka <jan.kiszka@siemens.com> escribió:
> On 2015-01-30 00:06, Paolo Bonzini wrote:
> > 
> > 
> > On 29/01/2015 20:37, Marc Marí wrote:
> >> Is this an expected behaviour? I can't see why.
> >>
> >> I'd like to know if there is a certain reason why it doesn't work.
> >> Or if it should work and the problem is too much I/O overhead. Or
> >> any other hint to understand it.
> > 
> > It is due to latencies in the host.  You need at least to use
> > preempt-rt kernels in the host as well.
> 
> That alone won't help much. You also need to fine-tune the guest to
> avoid running into QEMU locks that continuously synchronizes the guest
> on things like VGA or disk I/O emulation.
> 
> When using KVM, thus being able to run VCPUs widely independent of
> each other and the device models, you need to push cyclictest on an
> isolated second virtual CPU of the guest. Luiz and Marcelo can
> probably confirm this based on their ongoing experiments.
> 
> With TCG, we would first of all have to make it true SMP and
> independent of the I/O device lock. That's what Frederic is working
> on [1].
> 
> Jan
> 
> [1] http://permalink.gmane.org/gmane.comp.emulators.qemu/314406
> 

Thanks for the answers. I think I'm stuck with ARM926, which I think is
not prepared for SMP. I'll have to look if I can use Cortex for my
experiments.

I'll continue interested with the improvements for RT on TCG, but for
the moment I'll go to work on real harware, even though is easier to
run and debug on an emulator.

Thanks
Marc

Re: [Qemu-devel] QEMU and Real Time OS

[Frederic Konrad]

On 30/01/2015 11:26, Marc Marí wrote:
> El Fri, 30 Jan 2015 08:37:47 +0100
> Jan Kiszka <jan.kiszka@siemens.com> escribió:
>> On 2015-01-30 00:06, Paolo Bonzini wrote:
>>>
>>> On 29/01/2015 20:37, Marc Marí wrote:
>>>> Is this an expected behaviour? I can't see why.
>>>>
>>>> I'd like to know if there is a certain reason why it doesn't work.
>>>> Or if it should work and the problem is too much I/O overhead. Or
>>>> any other hint to understand it.
>>> It is due to latencies in the host.  You need at least to use
>>> preempt-rt kernels in the host as well.
>> That alone won't help much. You also need to fine-tune the guest to
>> avoid running into QEMU locks that continuously synchronizes the guest
>> on things like VGA or disk I/O emulation.
>>
>> When using KVM, thus being able to run VCPUs widely independent of
>> each other and the device models, you need to push cyclictest on an
>> isolated second virtual CPU of the guest. Luiz and Marcelo can
>> probably confirm this based on their ongoing experiments.
>>
>> With TCG, we would first of all have to make it true SMP and
>> independent of the I/O device lock. That's what Frederic is working
>> on [1].
>>
>> Jan
>>
>> [1] http://permalink.gmane.org/gmane.comp.emulators.qemu/314406
>>
> Thanks for the answers. I think I'm stuck with ARM926, which I think is
> not prepared for SMP. I'll have to look if I can use Cortex for my
> experiments.
>
> I'll continue interested with the improvements for RT on TCG, but for
> the moment I'll go to work on real harware, even though is easier to
> run and debug on an emulator.
>
> Thanks
> Marc
Hi Marc,

I think the important point here is "TCG thread independent of the I/O 
device lock".
I need it for multithread TCG but that doesn't mean you need an SMP guest
platform for that.

Fred

Re: [Qemu-devel] QEMU and Real Time OS

[Marc Marí]

El Fri, 30 Jan 2015 11:36:37 +0100
Frederic Konrad <fred.konrad@greensocs.com> escribió:
> On 30/01/2015 11:26, Marc Marí wrote:
> > El Fri, 30 Jan 2015 08:37:47 +0100
> > Jan Kiszka <jan.kiszka@siemens.com> escribió:
> >> On 2015-01-30 00:06, Paolo Bonzini wrote:
> >>>
> >>> On 29/01/2015 20:37, Marc Marí wrote:
> >>>> Is this an expected behaviour? I can't see why.
> >>>>
> >>>> I'd like to know if there is a certain reason why it doesn't
> >>>> work. Or if it should work and the problem is too much I/O
> >>>> overhead. Or any other hint to understand it.
> >>> It is due to latencies in the host.  You need at least to use
> >>> preempt-rt kernels in the host as well.
> >> That alone won't help much. You also need to fine-tune the guest to
> >> avoid running into QEMU locks that continuously synchronizes the
> >> guest on things like VGA or disk I/O emulation.
> >>
> >> When using KVM, thus being able to run VCPUs widely independent of
> >> each other and the device models, you need to push cyclictest on an
> >> isolated second virtual CPU of the guest. Luiz and Marcelo can
> >> probably confirm this based on their ongoing experiments.
> >>
> >> With TCG, we would first of all have to make it true SMP and
> >> independent of the I/O device lock. That's what Frederic is working
> >> on [1].
> >>
> >> Jan
> >>
> >> [1] http://permalink.gmane.org/gmane.comp.emulators.qemu/314406
> >>
> > Thanks for the answers. I think I'm stuck with ARM926, which I
> > think is not prepared for SMP. I'll have to look if I can use
> > Cortex for my experiments.
> >
> > I'll continue interested with the improvements for RT on TCG, but
> > for the moment I'll go to work on real harware, even though is
> > easier to run and debug on an emulator.
> >
> > Thanks
> > Marc
> Hi Marc,
> 
> I think the important point here is "TCG thread independent of the
> I/O device lock".
> I need it for multithread TCG but that doesn't mean you need an SMP
> guest platform for that.
> 
> Fred
> 

I read too fast, sorry. What has to be multicore is the host, so the
I/O can run independently of the TCG. But the guest can be multi- or
uni-core.

Marc

Re: [Qemu-devel] QEMU and Real Time OS

[Jan Kiszka]

On 2015-01-30 11:36, Frederic Konrad wrote:
> On 30/01/2015 11:26, Marc Marí wrote:
>> El Fri, 30 Jan 2015 08:37:47 +0100
>> Jan Kiszka <jan.kiszka@siemens.com> escribió:
>>> On 2015-01-30 00:06, Paolo Bonzini wrote:
>>>>
>>>> On 29/01/2015 20:37, Marc Marí wrote:
>>>>> Is this an expected behaviour? I can't see why.
>>>>>
>>>>> I'd like to know if there is a certain reason why it doesn't work.
>>>>> Or if it should work and the problem is too much I/O overhead. Or
>>>>> any other hint to understand it.
>>>> It is due to latencies in the host.  You need at least to use
>>>> preempt-rt kernels in the host as well.
>>> That alone won't help much. You also need to fine-tune the guest to
>>> avoid running into QEMU locks that continuously synchronizes the guest
>>> on things like VGA or disk I/O emulation.
>>>
>>> When using KVM, thus being able to run VCPUs widely independent of
>>> each other and the device models, you need to push cyclictest on an
>>> isolated second virtual CPU of the guest. Luiz and Marcelo can
>>> probably confirm this based on their ongoing experiments.
>>>
>>> With TCG, we would first of all have to make it true SMP and
>>> independent of the I/O device lock. That's what Frederic is working
>>> on [1].
>>>
>>> Jan
>>>
>>> [1] http://permalink.gmane.org/gmane.comp.emulators.qemu/314406
>>>
>> Thanks for the answers. I think I'm stuck with ARM926, which I think is
>> not prepared for SMP. I'll have to look if I can use Cortex for my
>> experiments.
>>
>> I'll continue interested with the improvements for RT on TCG, but for
>> the moment I'll go to work on real harware, even though is easier to
>> run and debug on an emulator.
>>
>> Thanks
>> Marc
> Hi Marc,
> 
> I think the important point here is "TCG thread independent of the I/O
> device lock".
> I need it for multithread TCG but that doesn't mean you need an SMP guest
> platform for that.

Provided the guest is not stressing any costly device emulation. That
can cause priority inversions when an urgent event should rather be
delivered in the meantime. That can easily happen when running real-time
Linux as guest, therefore the SMP proposal.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] QEMU and Real Time OS

[Marcelo Tosatti]

On Fri, Jan 30, 2015 at 08:37:47AM +0100, Jan Kiszka wrote:
> On 2015-01-30 00:06, Paolo Bonzini wrote:
> > 
> > 
> > On 29/01/2015 20:37, Marc Marí wrote:
> >> Is this an expected behaviour? I can't see why.
> >>
> >> I'd like to know if there is a certain reason why it doesn't work. Or
> >> if it should work and the problem is too much I/O overhead. Or any
> >> other hint to understand it.
> > 
> > It is due to latencies in the host.  You need at least to use preempt-rt
> > kernels in the host as well.
> 
> That alone won't help much. You also need to fine-tune the guest to
> avoid running into QEMU locks that continuously synchronizes the guest
> on things like VGA or disk I/O emulation.
> 
> When using KVM, thus being able to run VCPUs widely independent of each
> other and the device models, you need to push cyclictest on an isolated
> second virtual CPU of the guest. Luiz and Marcelo can probably confirm
> this based on their ongoing experiments.

Yes, we have achieved low latencies by using a dedicated pCPU for a
guest vCPU. 

This also avoids iothread <-> guest vCPU -RT priority issues.

> With TCG, we would first of all have to make it true SMP and independent
> of the I/O device lock. That's what Frederic is working on [1].
> 
> Jan
> 
> [1] http://permalink.gmane.org/gmane.comp.emulators.qemu/314406
> 
> -- 
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux