[Qemu-devel] Questions about the flow of interrupt simulation

[Qemu-devel] Questions about the flow of interrupt simulation

[Eva Chen]

Hi,

I have added a new virtual device under qemu/hw, and a device driver in the
source code of guest OS (I use arm linux).
Since I want to add an interrupt to the virtual device, I am tracing the
flow of interrupt simulation of QEMU.
However, I have some questions about the flow of interrupt, and wondering
if I could ask here for some help.

I use $qemu-system-aarch64 -machine virt.
Take the flow of interrupt that generated from the UART pl011 device as
example.
First an interrupt, CPU_INTERRUPT_HARD, will be sent from device to GIC for
telling CPU that this device finished some job, then this interrupt wiil be
transformed into EXCP_IRQ in aarch64_cpu_do_interrupt() and interrupt
handled by CPU.

- Details of function call:
1. Device to GIC : pl011_write() => pl011_update() => gic_set_irq()
2. CPU handling : cpu_exec() => arm_cpu_exec_interrupt() =>
aarch64_cpu_do_interrupt()
Guest CPU's pc (env->pc) will be changed to the address of interrupt
service routine in aarch64_cpu_do_interrupt().

The question is:
Will CPU execute the ISR that env->pc assigned?
If yes, is the ISR defined in the guest OS (I use arm-linux kernel image)?
How can I see the code that handling the EXCP_IRQ? Is there any method I
can set some breakpoint in gdb or set some log flags to see the handler?
If not, then is that all guest ISR will not be called by using QEMU?

The reason that I have this question is that I look into the cpu_exec(),
find that tb_find() will be executed after handling an interrupt. At this
time, the ISR address in env->pc should be replaced by the TB that is going
to be executed. However, I don't know how to check my thought since I can't
debug of execution in the TB so I ask for some help here.

I will be grateful for any advice, thanks.

Sincerely,
Eva

Re: [Qemu-devel] Questions about the flow of interrupt simulation

[Peter Maydell]

On 23 May 2018 at 11:45, Eva Chen <debby83729@gmail.com> wrote:
> Hi,
>
> I have added a new virtual device under qemu/hw, and a device driver in the
> source code of guest OS (I use arm linux).
> Since I want to add an interrupt to the virtual device, I am tracing the
> flow of interrupt simulation of QEMU.
> However, I have some questions about the flow of interrupt, and wondering
> if I could ask here for some help.
>
> I use $qemu-system-aarch64 -machine virt.
> Take the flow of interrupt that generated from the UART pl011 device as
> example.
> First an interrupt, CPU_INTERRUPT_HARD, will be sent from device to GIC for
> telling CPU that this device finished some job, then this interrupt wiil be
> transformed into EXCP_IRQ in aarch64_cpu_do_interrupt() and interrupt
> handled by CPU.

This isn't quite the way it works.

What happens is more like:
 * pl011 device calls qemu_set_irq() on its outbound IRQ line
 * the board model has connected that line up to one of the GIC's
   input lines
 * the GIC function on the other end of the qemu_irq is called
   and does the necessary processing of the input line state change.
   (For the gicv2 this is the gic_set_irq() function.)
 * if the GIC state is such that it should now signal an IRQ to
   the CPU it will do that. (This is not necessarily going to happen:
   for instance the interrupt might be disabled in the GIC, or
   another higher priority interrupt might already be being signaled.)
   The GIC does this by calling qemu_set_irq() on its outbound
   IRQ or FIQ line.
 * the CPU object on the other end of that IRQ or FIQ line will have
   its arm_cpu_set_irq() function called. All this function does is
   make a note so that we stop executing code and start doing
   something else; we do that by calling cpu_interrupt(). We use
   CPU_INTERRUPT_HARD/FIQ/VIRQ/VFIQ here to distinguish which of our
   4 interrupt lines this is; nothing outside the CPU cares about
   those names.
 * cpu_interrupt() itself doesn't do anything, it just sets a flag
   which the thread executing guest cpu code will check.
 * in the guest-cpu-execution code path you have traced the relevant
   flow here:
   > 2. CPU handling : cpu_exec() => arm_cpu_exec_interrupt() =>
   > aarch64_cpu_do_interrupt()
   > Guest CPU's pc (env->pc) will be changed to the address of interrupt
   > service routine in aarch64_cpu_do_interrupt().

Some notes:
 (1) I've assumed GICv2 in the above; the GICv3 is a bit more complex
but the basic idea is similar
 (2) The GIC is a device model that is not "special" just because
it happens to be an interrupt controller. It's just a model of a
bit of hardware that has some input lines and some output lines,
and sets the output lines as needed depending on what the input lines do.
It happens that for an interrupt controller the input lines are
interrupt lines from devices and the outputs are IRQ and FIQ to the CPU.
 (3) qemu_irq is a generic facility that roughly models a signal or
'wire' between two QEMU devices. On the sending side, you call
qemu_set_irq() to set the level. On the receiving side, a function
which you registered is called whenever the level changes. In the
board model code, you have to connect the two ends together.

> The question is:
> Will CPU execute the ISR that env->pc assigned?

Yes. The emulated CPU here behaves exactly like hardware. When
an interrupt is taken, the PC is set to the exception vector for
it. (The exact address depends on whether the exception is a
synchronous exception, SError, IRQ or FIQ; which exception level
we are taking the exception to; and the value in the VBAR register.)

It is up to the guest to have put some useful code at that address.
If it has not, then we will start executing whatever garbage was
at that address. (if that garbage is all-zeroes, then we'll take
an immediate exception because that's not a valid A64 instruction,
and likely go into an infinite loop of taking exceptions. This is
the same as what the hardware does in that situation.)

If your guest OS is Linux then it will have installed handlers
for these exceptions.

> The reason that I have this question is that I look into the cpu_exec(),
> find that tb_find() will be executed after handling an interrupt. At this
> time, the ISR address in env->pc should be replaced by the TB that is going
> to be executed. However, I don't know how to check my thought since I can't
> debug of execution in the TB so I ask for some help here.

This is correct, but you shouldn't usually need to look into QEMU's
internals at this low a level. You can debug either using QEMU's
gdb debugstub, or by using the -d options to enable tracing of
events like execution, interrupts and so on.

thanks
-- PMM

Re: [Qemu-devel] Questions about the flow of interrupt simulation

[Eva Chen]

​​
Thanks deeply for your clear explanation!
Let me try to conclude what I learned from your description.

                    qemu_set_irq()                       qemu_set_irq()

[ device ]   ----------------->  [ GIC ]  ---------------------> [ CPU ]


Detailed:
[device]
=>qemu_set_irq()
[GIC]
=>gic_set_irq()
=>gic_update()
=>qemu_set_irq()
[CPU]
=> arm_cpu_set_irq()

GIC decides to send an irq to CPU or not depends on the level that device
sends or others reason like interrupt mask.
GIC turns the interrupt into an interrupt type:  ARM_CPU_HARD/FIQ/VIRQ/VFIQ.
In arm_cpu_set_irq(), it turns   ARM_CPU_HARD/FIQ/VIRQ/VFIQ into
CPU_INTERRUPT_HARD/FIQ/VIRQ/VFIQ.
cpu_interrupt() sets cpu->interrupt_request.

Then I have some following questions:
1. There are two kinds of interrupt:  edge triggered and level triggered.
I have seen two code segment related to the level: gic_set_irq() and
arm_cpu_set_irq().
In gic_set_irq(), if level == GIC_TEST_LEVEL(irq, cm), which means the
level is not changed, will return.
In arm_cpu_set_irq() said that if level ==1, call cpu_interrupt(). if
level==0, call cpu_reset_interrupt(), which will clean up that irq bits..
Does that mean all interrupt in arm are level triggered(high level)?
How to know the triggered type of interrupt?

2. interrupt signal will be passed through GIC from device to CPU. There
are four types of interrupt in CPU: CPU_INTERRUPT_HARD/FIQ/VIRQ/VFIQ.
Where exactly define the CPU_INTERRUPT_{type} that device's interrupt
corresponded?

3. I have seen others device's code under qemu/hw directory. Almost all
device will call qemu_set_irq() at the end of device's read/write. Is that
for the purpose of a device to tell CPU that it has done some works?
but the second parameter of qemu_set_irq(), level, will be set to a
different value(not always 1 or 0), which sometimes will cause the
interrupt return at gic_set_irq() instead of passing to CPU.
What does the interrupt at the end of device_read/write(device_update())
mean?

I think I might ask some basic questions of interrupt. If there is any
documents or tutorial about interrupt, please share with me,  I am willing
to study it.

Sincerely,
Eva


2018-05-24 21:40 GMT+08:00 Peter Maydell <peter.maydell@linaro.org>:

> On 23 May 2018 at 11:45, Eva Chen <debby83729@gmail.com> wrote:
> > Hi,
> >
> > I have added a new virtual device under qemu/hw, and a device driver in
> the
> > source code of guest OS (I use arm linux).
> > Since I want to add an interrupt to the virtual device, I am tracing the
> > flow of interrupt simulation of QEMU.
> > However, I have some questions about the flow of interrupt, and wondering
> > if I could ask here for some help.
> >
> > I use $qemu-system-aarch64 -machine virt.
> > Take the flow of interrupt that generated from the UART pl011 device as
> > example.
> > First an interrupt, CPU_INTERRUPT_HARD, will be sent from device to GIC
> for
> > telling CPU that this device finished some job, then this interrupt wiil
> be
> > transformed into EXCP_IRQ in aarch64_cpu_do_interrupt() and interrupt
> > handled by CPU.
>
> This isn't quite the way it works.
>
> What happens is more like:
>  * pl011 device calls qemu_set_irq() on its outbound IRQ line
>  * the board model has connected that line up to one of the GIC's
>    input lines
>  * the GIC function on the other end of the qemu_irq is called
>    and does the necessary processing of the input line state change.
>    (For the gicv2 this is the gic_set_irq() function.)
>  * if the GIC state is such that it should now signal an IRQ to
>    the CPU it will do that. (This is not necessarily going to happen:
>    for instance the interrupt might be disabled in the GIC, or
>    another higher priority interrupt might already be being signaled.)
>    The GIC does this by calling qemu_set_irq() on its outbound
>    IRQ or FIQ line.
>  * the CPU object on the other end of that IRQ or FIQ line will have
>    its arm_cpu_set_irq() function called. All this function does is
>    make a note so that we stop executing code and start doing
>    something else; we do that by calling cpu_interrupt(). We use
>    CPU_INTERRUPT_HARD/FIQ/VIRQ/VFIQ here to distinguish which of our
>    4 interrupt lines this is; nothing outside the CPU cares about
>    those names.
>  * cpu_interrupt() itself doesn't do anything, it just sets a flag
>    which the thread executing guest cpu code will check.
>  * in the guest-cpu-execution code path you have traced the relevant
>    flow here:
>    > 2. CPU handling : cpu_exec() => arm_cpu_exec_interrupt() =>
>    > aarch64_cpu_do_interrupt()
>    > Guest CPU's pc (env->pc) will be changed to the address of interrupt
>    > service routine in aarch64_cpu_do_interrupt().
>
> Some notes:
>  (1) I've assumed GICv2 in the above; the GICv3 is a bit more complex
> but the basic idea is similar
>  (2) The GIC is a device model that is not "special" just because
> it happens to be an interrupt controller. It's just a model of a
> bit of hardware that has some input lines and some output lines,
> and sets the output lines as needed depending on what the input lines do.
> It happens that for an interrupt controller the input lines are
> interrupt lines from devices and the outputs are IRQ and FIQ to the CPU.
>  (3) qemu_irq is a generic facility that roughly models a signal or
> 'wire' between two QEMU devices. On the sending side, you call
> qemu_set_irq() to set the level. On the receiving side, a function
> which you registered is called whenever the level changes. In the
> board model code, you have to connect the two ends together.
>
> > The question is:
> > Will CPU execute the ISR that env->pc assigned?
>
> Yes. The emulated CPU here behaves exactly like hardware. When
> an interrupt is taken, the PC is set to the exception vector for
> it. (The exact address depends on whether the exception is a
> synchronous exception, SError, IRQ or FIQ; which exception level
> we are taking the exception to; and the value in the VBAR register.)
>
> It is up to the guest to have put some useful code at that address.
> If it has not, then we will start executing whatever garbage was
> at that address. (if that garbage is all-zeroes, then we'll take
> an immediate exception because that's not a valid A64 instruction,
> and likely go into an infinite loop of taking exceptions. This is
> the same as what the hardware does in that situation.)
>
> If your guest OS is Linux then it will have installed handlers
> for these exceptions.
>
> > The reason that I have this question is that I look into the cpu_exec(),
> > find that tb_find() will be executed after handling an interrupt. At this
> > time, the ISR address in env->pc should be replaced by the TB that is
> going
> > to be executed. However, I don't know how to check my thought since I
> can't
> > debug of execution in the TB so I ask for some help here.
>
> This is correct, but you shouldn't usually need to look into QEMU's
> internals at this low a level. You can debug either using QEMU's
> gdb debugstub, or by using the -d options to enable tracing of
> events like execution, interrupts and so on.
>
> thanks
> -- PMM
>

Re: [Qemu-devel] Questions about the flow of interrupt simulation

[Peter Maydell]

On 1 June 2018 at 07:17, Eva Chen <debby83729@gmail.com> wrote:
> 1. There are two kinds of interrupt:  edge triggered and level triggered.
> I have seen two code segment related to the level: gic_set_irq() and
> arm_cpu_set_irq().
> In gic_set_irq(), if level == GIC_TEST_LEVEL(irq, cm), which means the
> level is not changed, will return.
> In arm_cpu_set_irq() said that if level ==1, call cpu_interrupt(). if
> level==0, call cpu_reset_interrupt(), which will clean up that irq bits..
> Does that mean all interrupt in arm are level triggered(high level)?
> How to know the triggered type of interrupt?

This is mixing up interrupts in two different places. For the Arm
architecture, IRQ and FIQ are always level-sensitive: the thing
which sets them (the GIC, typically) has to set them and keep them
set until the CPU acknowledges them.

For the GIC, its input interrupts may be either level sensitive or
edge sensitive. This is configurable for each interrupt on GICv2 by
writing to the GICD_ICFGRn registers. The gic_set_irq() code implements
the behaviour that the GIC specification requires, depending on whether
the ICFGRn register says that interrupt should be edge or level
triggered.

Other interrupt controllers that QEMU models may behave differently.
(For instance the ARMv7M NVIC is different again.)

> 2. interrupt signal will be passed through GIC from device to CPU. There
> are four types of interrupt in CPU: CPU_INTERRUPT_HARD/FIQ/VIRQ/VFIQ.
> Where exactly define the CPU_INTERRUPT_{type} that device's interrupt
> corresponded?

Again, this is configurable by the guest by writing to GIC registers.
In the GICv2, the GICD_IGROUPRn registers set the whether the interrupt
should be in "group 0" or "group 1". Group 1 interrupts always
cause an IRQ; group 0 interrupts cause either IRQ or FIQ depending
on the setting of the GICC_CTLR FIQEn bit. (The expected use is that
interrupts configured for use with the TrustZone Secure World will
use FIQ and those configured for use with the NonSecure World will
use IRQ.)

VIRQ and VFIQ are for when the GIC and CPU support the Virtualization
Extension.

The behaviour of all of this is defined by the GIC specification;
QEMU just has to implement what the hardware does.

> 3. I have seen others device's code under qemu/hw directory. Almost all
> device will call qemu_set_irq() at the end of device's read/write. Is that
> for the purpose of a device to tell CPU that it has done some works?
> but the second parameter of qemu_set_irq(), level, will be set to a
> different value(not always 1 or 0), which sometimes will cause the
> interrupt return at gic_set_irq() instead of passing to CPU.
> What does the interrupt at the end of device_read/write(device_update())
> mean?

The best way to think of this is not to try to think about whether
the interrupt line is connected to the CPU or anything else. Just
think about a device model as being an emulation of a particular
bit of hardware. For instance, take the pl011 UART. The specification
for that UART says that when certain conditions inside the device are
true, the UART will assert its outgoing interrupt line. So our model
also must check those conditions and call qemu_set_irq() to raise
and lower the interrupt at the right time. The common way to code
this is to have a function which is called whenever any of the
relevant state has changed, which rechecks the conditions and calls
qemu_set_irq(). The "purpose" of this code is just to behave the
way the hardware behaves.

Commonly, the output IRQ line from a device is connected to an
interrupt controller and thus to a CPU, but it doesn't have to be.
On some boards, the IRQ line might not be connected to anything.
Or it might be connected up to some other device which provides its
value to the guest via a status register. Or perhaps it's ORed together
with lines from other devices and the output of the OR gate goes
to the interrupt controller. If you're designing a board in real
hardware, you are taking various components (UART, interrupt
controller, etc) and connecting them up to implement a useful design.
In QEMU, we also take various components and connect them up, to
produce the same design the hardware has.

qemu_set_irq() is usually just modelling what in the hardware is a
simple wire. The source end calls this function to say "the wire
is at logical 0/1", and on the destination end a function is called
to handle that. It is also possible to pass in a value other than
0 or 1. This happens in two cases:
 (1) a bug, where the source end really ought to be using 0 or 1;
often this doesn't have any visible bad effects because the
destination end is testing 0 vs not-0, rather than 0 vs 1
 (2) we really do want to transfer an integer rather than just
a 0-vs-1 level. This is less common and only happens when both
ends of the "wire" know that that's the convention they want to use.

I think the overall theme of my reply is that fundamentally
QEMU is modelling hardware. If you want to understand why
parts of QEMU are connected the way they are, or why they
behave in a particular way, then looking at what the
specification of the hardware is and how the hardware is
connected, will often help a lot.

thanks
-- PMM