* Errors on MMIO read access on VM suspend / resume operations
@ 2011-01-11 16:19 Stefan Berger
2011-01-13 10:22 ` Avi Kivity
0 siblings, 1 reply; 22+ messages in thread
From: Stefan Berger @ 2011-01-11 16:19 UTC (permalink / raw)
To: kvm
Hi!
I am currently doing some long-term testing of a device model using
memory mapped IO (TPM TIS) and am seeing some strange errors when the
suspend occurs in the middle of a read operation in the Linux TPM TIS
device driver where the driver reads the result packet from the mmio
location.
Short background: The TPM response packet is read in 2 chunks. First
the first 10 bytes are read containing the response's header.
Subsequently the rest of the packet is read using knowledge of the total
size of the response packet from the header (bytes 2-5 in big endian
format). The corresponding code reading the data from the hardware
interface is here:
http://lxr.linux.no/#linux+v2.6.37/drivers/char/tpm/tpm_tis.c#L228
The test I am running is setup as follows:
- inside the VM keys are permanently generated by sending commands to
the TPM; packets read from the interface are dumped to the screen
- on the host a script suspends the VM every 6 seconds and resumes it
immediately afterwards (using libvirt)
As it happens, sometimes the VM is suspended in the middle of a read
operation on the TPM TIS interface -- see above code reference. I see
that because I do dump the state of the TPM TIS when suspending and see
that the read offset is pointing to a location somewhere in the middle
of the packet - so the TPM TIS Linux driver is in the above loop
currently reading the data. I am observing two types of results if this
happens:
- either the result read by the Linux TPM TIS driver is ok, so no
problem here
- or the problematic case where the TPM TIS driver reads a packet with a
byte missing and then at the end gets a zero byte from the TPM TIS
interface indicating that it read beyond the available data. If the
suspend happened while reading the first chunk of data (header), the TPM
TIS driver will also complain that the available data for the 2nd chunk
(burst size) is less than what's expected -- it's an off-by-one error
So, I then modified the TPM TIS device model to decrement the read
offset pointer by '1' in case it was detected that the suspend happened
in the middle of the read operation -- in Qemu I do this in the
post-load 'method'. This then leads to the following types of results:
- the problematic(!) case where the read packet was ok
- the expected case where the TPM TIS driver reads the packet and ends
up having two same bytes in the result in consecutive array locations;
besides that the TPM TIS driver will in this case complain that it has
left-over data
So my conclusion from the above tests are:
- for some reason the memory read to the MMIO location happens as the
last instruction executed on suspend and again as the very first on
executed on resume. This explains to me that the TPM TIS model internal
pointer into the packet was advanced by '1' (the packet is read by
subsequently reading from the same memory location) and the above
problematic cases make sense
- the other instruction in the Linux TPM TIS drivers that for example
advance the buffer location do not execute twice, i.e., size++ in the
buf[size++] = ... in the Linux driver.
What puzzles me is that the read operation may be run twice but others
don't.
If you have insights as why the above may be occurring, please let me
know. A simple solution to work around this may be to introduce a
register holding the index into the result packet where to read the next
byte from (rather than advancing an internal pointer to the next byte),
though this would deviate the driver from the standard interface the
model currently implements.
Regards,
Stefan
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-11 16:19 Errors on MMIO read access on VM suspend / resume operations Stefan Berger
@ 2011-01-13 10:22 ` Avi Kivity
2011-01-14 19:27 ` Stefan Berger
0 siblings, 1 reply; 22+ messages in thread
From: Avi Kivity @ 2011-01-13 10:22 UTC (permalink / raw)
To: Stefan Berger; +Cc: kvm
On 01/11/2011 06:19 PM, Stefan Berger wrote:
> Hi!
>
> I am currently doing some long-term testing of a device model using
> memory mapped IO (TPM TIS) and am seeing some strange errors when the
> suspend occurs in the middle of a read operation in the Linux TPM TIS
> device driver where the driver reads the result packet from the mmio
> location.
>
> Short background: The TPM response packet is read in 2 chunks. First
> the first 10 bytes are read containing the response's header.
> Subsequently the rest of the packet is read using knowledge of the
> total size of the response packet from the header (bytes 2-5 in big
> endian format). The corresponding code reading the data from the
> hardware interface is here:
>
>
> http://lxr.linux.no/#linux+v2.6.37/drivers/char/tpm/tpm_tis.c#L228
>
>
> The test I am running is setup as follows:
>
> - inside the VM keys are permanently generated by sending commands to
> the TPM; packets read from the interface are dumped to the screen
>
> - on the host a script suspends the VM every 6 seconds and resumes it
> immediately afterwards (using libvirt)
>
>
> As it happens, sometimes the VM is suspended in the middle of a read
> operation on the TPM TIS interface -- see above code reference. I see
> that because I do dump the state of the TPM TIS when suspending and
> see that the read offset is pointing to a location somewhere in the
> middle of the packet - so the TPM TIS Linux driver is in the above
> loop currently reading the data. I am observing two types of results
> if this happens:
>
>
> - either the result read by the Linux TPM TIS driver is ok, so no
> problem here
>
> - or the problematic case where the TPM TIS driver reads a packet with
> a byte missing and then at the end gets a zero byte from the TPM TIS
> interface indicating that it read beyond the available data. If the
> suspend happened while reading the first chunk of data (header), the
> TPM TIS driver will also complain that the available data for the 2nd
> chunk (burst size) is less than what's expected -- it's an off-by-one
> error
>
>
> So, I then modified the TPM TIS device model to decrement the read
> offset pointer by '1' in case it was detected that the suspend
> happened in the middle of the read operation -- in Qemu I do this in
> the post-load 'method'. This then leads to the following types of
> results:
>
>
> - the problematic(!) case where the read packet was ok
>
> - the expected case where the TPM TIS driver reads the packet and ends
> up having two same bytes in the result in consecutive array locations;
> besides that the TPM TIS driver will in this case complain that it has
> left-over data
>
>
> So my conclusion from the above tests are:
>
> - for some reason the memory read to the MMIO location happens as the
> last instruction executed on suspend and again as the very first on
> executed on resume. This explains to me that the TPM TIS model
> internal pointer into the packet was advanced by '1' (the packet is
> read by subsequently reading from the same memory location) and the
> above problematic cases make sense
Most likely this is qemu-kvm failing to obey this snippet from
Documentation/kvm/api.txt:
> NOTE: For KVM_EXIT_IO, KVM_EXIT_MMIO and KVM_EXIT_OSI, the corresponding
> operations are complete (and guest state is consistent) only after
> userspace
> has re-entered the kernel with KVM_RUN. The kernel side will first finish
> incomplete operations and then check for pending signals. Userspace
> can re-enter the guest with an unmasked signal pending to complete
> pending operations.
However, the code appears to be correct. kvm_run() calls handle_mmio(),
which returns 0. The following bit
if (!r) {
goto again;
}
at the end of kvm_run() makes it enter the kernel again (delivering a
signal to itself in case we want to stop).
>
> - the other instruction in the Linux TPM TIS drivers that for example
> advance the buffer location do not execute twice, i.e., size++ in the
> buf[size++] = ... in the Linux driver.
>
>
> What puzzles me is that the read operation may be run twice but others
> don't.
>
Reads have split execution: kvm emulates the mmio instruction, notices
that it cannot satisfy the read request, exits to qemu, then restarts
the instruction. If the last step is omitted due to savevm, then kvm
will exit back to qemu again and your device will see the read duplicated.
> If you have insights as why the above may be occurring, please let me
> know. A simple solution to work around this may be to introduce a
> register holding the index into the result packet where to read the
> next byte from (rather than advancing an internal pointer to the next
> byte), though this would deviate the driver from the standard
> interface the model currently implements.
Most undesirable, I'd like to fix the bug.
Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to verify
this?
Good pattern:
ioctl(KVM_RUN)
-> KVM_EXIT_MMIO
ioctl(KVM_RUN)
-> ENTR
no further KVM_RUNs
or
ioctl(KVM_RUN)
-> something other than KVM_EXIT_MMIO
no further KVM_RUNs
Bad pattern:
ioctl(KVM_RUN)
-> KVM_EXIT_MMIO
no further KVM_RUNs
(in the save portion of your test)
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-13 10:22 ` Avi Kivity
@ 2011-01-14 19:27 ` Stefan Berger
2011-01-16 14:43 ` Avi Kivity
0 siblings, 1 reply; 22+ messages in thread
From: Stefan Berger @ 2011-01-14 19:27 UTC (permalink / raw)
To: Avi Kivity; +Cc: kvm
On 01/13/2011 05:22 AM, Avi Kivity wrote:
> On 01/11/2011 06:19 PM, Stefan Berger wrote:
>>
>> What puzzles me is that the read operation may be run twice but
>> others don't.
>>
>
> Reads have split execution: kvm emulates the mmio instruction, notices
> that it cannot satisfy the read request, exits to qemu, then restarts
> the instruction. If the last step is omitted due to savevm, then kvm
> will exit back to qemu again and your device will see the read
> duplicated.
That would then explain it on the read-side. MMIO writes seem to be ok.
>
>> If you have insights as why the above may be occurring, please let me
>> know. A simple solution to work around this may be to introduce a
>> register holding the index into the result packet where to read the
>> next byte from (rather than advancing an internal pointer to the next
>> byte), though this would deviate the driver from the standard
>> interface the model currently implements.
>
> Most undesirable, I'd like to fix the bug.
In the meantime I modified the tpm_tis driver making use of the fact
that the packet can be re-read. I reread the packet when I recognize
that the TIS is running out of bytes early (due to double-read). That
makes it work. Debugging output of the driver shows the retrying,
implying that the double-reads do occur. This work-around is just
'masking' the problem.
>
> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
> verify this?
>
Here's what I did:
diff --git a/kvm-all.c b/kvm-all.c
index cae24bb..7ba222a 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -31,7 +31,7 @@
/* KVM uses PAGE_SIZE in it's definition of COALESCED_MMIO_MAX */
#define PAGE_SIZE TARGET_PAGE_SIZE
-//#define DEBUG_KVM
+#define DEBUG_KVM
#ifdef DEBUG_KVM
#define DPRINTF(fmt, ...) \
@@ -874,6 +874,9 @@ int kvm_cpu_exec(CPUState *env)
kvm_arch_pre_run(env, run);
cpu_single_env = NULL;
qemu_mutex_unlock_iothread();
+#ifdef DEBUG_KVM
+ fprintf(stderr,"ioctl(KVM_RUN)\n");
+#endif
ret = kvm_vcpu_ioctl(env, KVM_RUN, 0);
qemu_mutex_lock_iothread();
cpu_single_env = env;
> Good pattern:
>
> ioctl(KVM_RUN)
> -> KVM_EXIT_MMIO
> ioctl(KVM_RUN)
> -> ENTR
> no further KVM_RUNs
>
> or
>
> ioctl(KVM_RUN)
> -> something other than KVM_EXIT_MMIO
> no further KVM_RUNs
>
> Bad pattern:
>
> ioctl(KVM_RUN)
> -> KVM_EXIT_MMIO
> no further KVM_RUNs
>
Here's what I see in the failure case:
ioctl(KVM_RUN)
handle_mmio
ioctl(KVM_RUN)
handle_mmio
ioctl(KVM_RUN)
handle_mmio
ioctl(KVM_RUN)
handle_mmio
interrupt exit requested
0+0 records in
0+0 records out
0 bytes (0 B) copied, 9.18e-06 s, 0.0 kB/s
tpm_tis: suspend: locty 0 : r_offset = 61, w_offset = 0
tpm_tis: active locality : 0
tpm_tis: state of locality 0 : 0x2
tpm_tis: register dump:
tpm_tis: 0x0000 : 000000a0
tpm_tis: 0x0008 : 0x8000008d
tpm_tis: 0x000c : 0x00000003
tpm_tis: 0x0010 : 0x00000000
tpm_tis: 0x0014 : 0x00000095
tpm_tis: 0x0018 : 0x00011290
tpm_tis: 0x0f00 : 0x00010001
tpm_tis: 0x0f04 : 0x00000001
tpm_tis: read offset : 61
tpm_tis: result buffer : 00 c5 00 00 01 4f 00 00 00 00 00 00 00 01 00 03
tpm_tis: 00 01 00 00 00 0c 00 00 08 00 00 00 00 02 00 00
tpm_tis: 00 00 00 00 01 00 ca 47 12 16 3b e9 14 25 f7 f4
tpm_tis: b1 5e 69 c7 da 2d 5e f8 a3 82 04 bc e6>17 9e c5
tpm_tis: fd 2a f0 c0 01 13 c9 f2 c7 fb 5d e6 63 a7 db 48
tpm_tis: f2 85 bb 93 f1 b3 40 00 5b 27 70 5d 8c 28 79 89
tpm_tis: 36 49 7d 2f 56 b8 28 d5 61 74 4d ff 1c cf 4b a9
tpm_tis: a1 a8 b1 9e 6f 89 02 a3 21 02 86 8b ab 73 14 9b
tpm_tis: 79 6e 86 27 da a8 2d 2c 0f 9c 28 ae 8d b7 b8 66
tpm_tis: 6a 20 92 3d 48 d0 c9 42 84 f6 2a ad e2 bd d3 11
tpm_tis: 2b e9 90 8e 1f 64 e4 de 9d 12 75 4a db 3f de d7
tpm_tis: 70 06 b0 95 47 56 9f a7 32 b6 20 1a 12 ea 8a ed
tpm_tis: 24 b1 17 6b ad 4a 64 4b 64 b7 2c 1f 44 02 fe 2c
tpm_tis: d8 72 1b bb 9c 42 8c 64 46 f4 29 ad 9c ff ea 37
tpm_tis: 48 77 26 4c 75 53 52 5a cb 6a 58 d5 d6 81 17 17
tpm_tis: 27 97 44 bf d7 0d 46 3c 9f b3 70 3f 5c 5c b1 ba
tpm_tis: 86 5e 9c 78 bc 4c 40 41 90 ed 79 b2 06 64 73 6e
tpm_tis: 64 18 e7 d0 9f 4d 7d d6 bb 2d 39 18 e1 41 3f 34
tpm_tis: 2b 9e a4 8e 70 a1 fa 35 1d f3 cf be 5a 73 4d dd
tpm_tis: 49 dd 57 79 5b be 48 c6 44 3c 00 01 68 07 b3 2d
tpm_tis: 42 08 2e 51 93 1a cf db 34 de 10 1d 94 fe 9a
tpm_tis: write offset : 0
tpm_tis: request buffer:>00 c2 00 00 00 3b 00 00 00 81 40 00 00 06 78 e8
tpm_tis: a5 47 c3 e9 67 27 ac 27 5a a7 9c 53 19 f9 f0 cd
tpm_tis: 8b 93 4a 82 85 af 00 e9 b8 c6 80 92 42 fb c6 55
tpm_tis: 06 00 f2 62 52 7b 8b cd 64 65 95
48+11535 records in
48+11535 records out
500881189 bytes (501 MB) copied, 14.282 s, 35.1 MB/s
2011-01-14 13:55:57.871: shutting down
Immediately following is the restore part:
2011-01-14 13:55:59.575: starting up
LC_ALL=C PATH=/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none [...]
kvm_init_vcpu
tpm_tis: resume : locty 0 : r_offset = 61, w_offset = 0
kvm_cpu_exec()
interrupt exit requested
kvm_cpu_exec()
ioctl(KVM_RUN)
irq_window_open
kvm_cpu_exec()
ioctl(KVM_RUN)
handle_mmio
ioctl(KVM_RUN)
handle_mmio
Cheers!
Stefan
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-14 19:27 ` Stefan Berger
@ 2011-01-16 14:43 ` Avi Kivity
2011-01-18 3:03 ` [Qemu-devel] " Stefan Berger
0 siblings, 1 reply; 22+ messages in thread
From: Avi Kivity @ 2011-01-16 14:43 UTC (permalink / raw)
To: Stefan Berger; +Cc: kvm
On 01/14/2011 09:27 PM, Stefan Berger wrote:
>
>>
>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>> verify this?
>>
> Here's what I did:
>
>
> interrupt exit requested
It appears from this you're using qemu.git. Please try qemu-kvm.git,
where the code appears to be correct.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-16 14:43 ` Avi Kivity
@ 2011-01-18 3:03 ` Stefan Berger
0 siblings, 0 replies; 22+ messages in thread
From: Stefan Berger @ 2011-01-18 3:03 UTC (permalink / raw)
To: Avi Kivity; +Cc: kvm, qemu-devel
On 01/16/2011 09:43 AM, Avi Kivity wrote:
> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>
>>>
>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>> verify this?
>>>
>> Here's what I did:
>>
>>
>> interrupt exit requested
>
> It appears from this you're using qemu.git. Please try qemu-kvm.git,
> where the code appears to be correct.
>
Cc'ing qemu-devel now. For reference, here the initial problem description:
http://www.spinics.net/lists/kvm/msg48274.html
I didn't know there was another tree...
I have seen now a couple of suspends-while-reading with patches applied
to the qemu-kvm.git tree and indeed, when run with the same host kernel
and VM I do not see the debugging dumps due to double-reads that I would
have anticipated seeing by now. Now what? Can this be easily fixed in
the other Qemu tree as well?
One thing I'd like to mention is that I have seen what I think are
interrupt stalls when running my tests inside the qemu-kvm.git tree
version and not suspending at all. A some point the interrupt counter in
the guest kernel does not increase anymore even though I see the device
model raising the IRQ and lowering it. The same tests run literally
forever in the qemu.git tree version of Qemu.
Regards,
Stefan
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
@ 2011-01-18 3:03 ` Stefan Berger
0 siblings, 0 replies; 22+ messages in thread
From: Stefan Berger @ 2011-01-18 3:03 UTC (permalink / raw)
To: Avi Kivity; +Cc: qemu-devel, kvm
On 01/16/2011 09:43 AM, Avi Kivity wrote:
> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>
>>>
>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>> verify this?
>>>
>> Here's what I did:
>>
>>
>> interrupt exit requested
>
> It appears from this you're using qemu.git. Please try qemu-kvm.git,
> where the code appears to be correct.
>
Cc'ing qemu-devel now. For reference, here the initial problem description:
http://www.spinics.net/lists/kvm/msg48274.html
I didn't know there was another tree...
I have seen now a couple of suspends-while-reading with patches applied
to the qemu-kvm.git tree and indeed, when run with the same host kernel
and VM I do not see the debugging dumps due to double-reads that I would
have anticipated seeing by now. Now what? Can this be easily fixed in
the other Qemu tree as well?
One thing I'd like to mention is that I have seen what I think are
interrupt stalls when running my tests inside the qemu-kvm.git tree
version and not suspending at all. A some point the interrupt counter in
the guest kernel does not increase anymore even though I see the device
model raising the IRQ and lowering it. The same tests run literally
forever in the qemu.git tree version of Qemu.
Regards,
Stefan
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-18 3:03 ` [Qemu-devel] " Stefan Berger
@ 2011-01-18 8:53 ` Jan Kiszka
-1 siblings, 0 replies; 22+ messages in thread
From: Jan Kiszka @ 2011-01-18 8:53 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
On 2011-01-18 04:03, Stefan Berger wrote:
> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>
>>>>
>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>> verify this?
>>>>
>>> Here's what I did:
>>>
>>>
>>> interrupt exit requested
>>
>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>> where the code appears to be correct.
>>
> Cc'ing qemu-devel now. For reference, here the initial problem description:
>
> http://www.spinics.net/lists/kvm/msg48274.html
>
> I didn't know there was another tree...
>
> I have seen now a couple of suspends-while-reading with patches applied
> to the qemu-kvm.git tree and indeed, when run with the same host kernel
> and VM I do not see the debugging dumps due to double-reads that I would
> have anticipated seeing by now. Now what? Can this be easily fixed in
> the other Qemu tree as well?
Please give this a try:
git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
fixes the issue for you. If other problems pop up with that tree, also
try resetting to that particular commit.
I'm currently trying to shake all those hidden or forgotten bug fixes
out of qemu-kvm and port them upstream. Most of those subtle differences
should hopefully soon be history.
>
> One thing I'd like to mention is that I have seen what I think are
> interrupt stalls when running my tests inside the qemu-kvm.git tree
> version and not suspending at all. A some point the interrupt counter in
> the guest kernel does not increase anymore even though I see the device
> model raising the IRQ and lowering it. The same tests run literally
> forever in the qemu.git tree version of Qemu.
What about qemu-kmv and -no-kvm-irqchip?
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
@ 2011-01-18 8:53 ` Jan Kiszka
0 siblings, 0 replies; 22+ messages in thread
From: Jan Kiszka @ 2011-01-18 8:53 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
On 2011-01-18 04:03, Stefan Berger wrote:
> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>
>>>>
>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>> verify this?
>>>>
>>> Here's what I did:
>>>
>>>
>>> interrupt exit requested
>>
>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>> where the code appears to be correct.
>>
> Cc'ing qemu-devel now. For reference, here the initial problem description:
>
> http://www.spinics.net/lists/kvm/msg48274.html
>
> I didn't know there was another tree...
>
> I have seen now a couple of suspends-while-reading with patches applied
> to the qemu-kvm.git tree and indeed, when run with the same host kernel
> and VM I do not see the debugging dumps due to double-reads that I would
> have anticipated seeing by now. Now what? Can this be easily fixed in
> the other Qemu tree as well?
Please give this a try:
git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
fixes the issue for you. If other problems pop up with that tree, also
try resetting to that particular commit.
I'm currently trying to shake all those hidden or forgotten bug fixes
out of qemu-kvm and port them upstream. Most of those subtle differences
should hopefully soon be history.
>
> One thing I'd like to mention is that I have seen what I think are
> interrupt stalls when running my tests inside the qemu-kvm.git tree
> version and not suspending at all. A some point the interrupt counter in
> the guest kernel does not increase anymore even though I see the device
> model raising the IRQ and lowering it. The same tests run literally
> forever in the qemu.git tree version of Qemu.
What about qemu-kmv and -no-kvm-irqchip?
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-18 8:53 ` [Qemu-devel] " Jan Kiszka
@ 2011-01-24 18:27 ` Stefan Berger
-1 siblings, 0 replies; 22+ messages in thread
From: Stefan Berger @ 2011-01-24 18:27 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Avi Kivity, kvm, qemu-devel
On 01/18/2011 03:53 AM, Jan Kiszka wrote:
> On 2011-01-18 04:03, Stefan Berger wrote:
>> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>>> verify this?
>>>>>
>>>> Here's what I did:
>>>>
>>>>
>>>> interrupt exit requested
>>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>>> where the code appears to be correct.
>>>
>> Cc'ing qemu-devel now. For reference, here the initial problem description:
>>
>> http://www.spinics.net/lists/kvm/msg48274.html
>>
>> I didn't know there was another tree...
>>
>> I have seen now a couple of suspends-while-reading with patches applied
>> to the qemu-kvm.git tree and indeed, when run with the same host kernel
>> and VM I do not see the debugging dumps due to double-reads that I would
>> have anticipated seeing by now. Now what? Can this be easily fixed in
>> the other Qemu tree as well?
> Please give this a try:
>
> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
>
> I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
> fixes the issue for you. If other problems pop up with that tree, also
> try resetting to that particular commit.
>
> I'm currently trying to shake all those hidden or forgotten bug fixes
> out of qemu-kvm and port them upstream. Most of those subtle differences
> should hopefully soon be history.
>
I did the same test as I did with Avi's tree and haven't seen the
consequences of possible double-reads. So, I would say that you should
upstream those patches...
I searched for the text you mention above using 'gitk' but couldn't find
a patch with that headline in your tree. There were others that seem to
be related:
Gleb Natapov: "do not enter vcpu again if it was stopped during IO"
>> One thing I'd like to mention is that I have seen what I think are
>> interrupt stalls when running my tests inside the qemu-kvm.git tree
>> version and not suspending at all. A some point the interrupt counter in
>> the guest kernel does not increase anymore even though I see the device
>> model raising the IRQ and lowering it. The same tests run literally
>> forever in the qemu.git tree version of Qemu.
> What about qemu-kmv and -no-kvm-irqchip?
That seems to be necessary for both trees, yours and the one Avi pointed
me to. If applied, then I did not see the interrupt problem.
Stefan
> Jan
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
@ 2011-01-24 18:27 ` Stefan Berger
0 siblings, 0 replies; 22+ messages in thread
From: Stefan Berger @ 2011-01-24 18:27 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Avi Kivity, kvm, qemu-devel
On 01/18/2011 03:53 AM, Jan Kiszka wrote:
> On 2011-01-18 04:03, Stefan Berger wrote:
>> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>>> verify this?
>>>>>
>>>> Here's what I did:
>>>>
>>>>
>>>> interrupt exit requested
>>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>>> where the code appears to be correct.
>>>
>> Cc'ing qemu-devel now. For reference, here the initial problem description:
>>
>> http://www.spinics.net/lists/kvm/msg48274.html
>>
>> I didn't know there was another tree...
>>
>> I have seen now a couple of suspends-while-reading with patches applied
>> to the qemu-kvm.git tree and indeed, when run with the same host kernel
>> and VM I do not see the debugging dumps due to double-reads that I would
>> have anticipated seeing by now. Now what? Can this be easily fixed in
>> the other Qemu tree as well?
> Please give this a try:
>
> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
>
> I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
> fixes the issue for you. If other problems pop up with that tree, also
> try resetting to that particular commit.
>
> I'm currently trying to shake all those hidden or forgotten bug fixes
> out of qemu-kvm and port them upstream. Most of those subtle differences
> should hopefully soon be history.
>
I did the same test as I did with Avi's tree and haven't seen the
consequences of possible double-reads. So, I would say that you should
upstream those patches...
I searched for the text you mention above using 'gitk' but couldn't find
a patch with that headline in your tree. There were others that seem to
be related:
Gleb Natapov: "do not enter vcpu again if it was stopped during IO"
>> One thing I'd like to mention is that I have seen what I think are
>> interrupt stalls when running my tests inside the qemu-kvm.git tree
>> version and not suspending at all. A some point the interrupt counter in
>> the guest kernel does not increase anymore even though I see the device
>> model raising the IRQ and lowering it. The same tests run literally
>> forever in the qemu.git tree version of Qemu.
> What about qemu-kmv and -no-kvm-irqchip?
That seems to be necessary for both trees, yours and the one Avi pointed
me to. If applied, then I did not see the interrupt problem.
Stefan
> Jan
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-24 18:27 ` [Qemu-devel] " Stefan Berger
@ 2011-01-24 22:34 ` Jan Kiszka
-1 siblings, 0 replies; 22+ messages in thread
From: Jan Kiszka @ 2011-01-24 22:34 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 3043 bytes --]
On 2011-01-24 19:27, Stefan Berger wrote:
> On 01/18/2011 03:53 AM, Jan Kiszka wrote:
>> On 2011-01-18 04:03, Stefan Berger wrote:
>>> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>>>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>>>> verify this?
>>>>>>
>>>>> Here's what I did:
>>>>>
>>>>>
>>>>> interrupt exit requested
>>>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>>>> where the code appears to be correct.
>>>>
>>> Cc'ing qemu-devel now. For reference, here the initial problem
>>> description:
>>>
>>> http://www.spinics.net/lists/kvm/msg48274.html
>>>
>>> I didn't know there was another tree...
>>>
>>> I have seen now a couple of suspends-while-reading with patches applied
>>> to the qemu-kvm.git tree and indeed, when run with the same host kernel
>>> and VM I do not see the debugging dumps due to double-reads that I would
>>> have anticipated seeing by now. Now what? Can this be easily fixed in
>>> the other Qemu tree as well?
>> Please give this a try:
>>
>> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
>>
>> I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
>> fixes the issue for you. If other problems pop up with that tree, also
>> try resetting to that particular commit.
>>
>> I'm currently trying to shake all those hidden or forgotten bug fixes
>> out of qemu-kvm and port them upstream. Most of those subtle differences
>> should hopefully soon be history.
>>
> I did the same test as I did with Avi's tree and haven't seen the
> consequences of possible double-reads. So, I would say that you should
> upstream those patches...
>
> I searched for the text you mention above using 'gitk' but couldn't find
> a patch with that headline in your tree. There were others that seem to
> be related:
>
> Gleb Natapov: "do not enter vcpu again if it was stopped during IO"
Err, I don't think you checked out queues/kvm-upstream. I bet you just
ran my master branch which is a version of qemu-kvm's master. Am I right? :)
>>> One thing I'd like to mention is that I have seen what I think are
>>> interrupt stalls when running my tests inside the qemu-kvm.git tree
>>> version and not suspending at all. A some point the interrupt counter in
>>> the guest kernel does not increase anymore even though I see the device
>>> model raising the IRQ and lowering it. The same tests run literally
>>> forever in the qemu.git tree version of Qemu.
>> What about qemu-kmv and -no-kvm-irqchip?
> That seems to be necessary for both trees, yours and the one Avi pointed
> me to. If applied, then I did not see the interrupt problem.
And the fact that you were able to call qemu from my tree with
-no-kvm-irqchip just underlines my assumption: that switch is refused by
upstream. Please retry with the latest kvm-upstream queue.
Besides that, this other bug you may see in the in-kernel IRQ path - how
can we reproduce it?
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
@ 2011-01-24 22:34 ` Jan Kiszka
0 siblings, 0 replies; 22+ messages in thread
From: Jan Kiszka @ 2011-01-24 22:34 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 3043 bytes --]
On 2011-01-24 19:27, Stefan Berger wrote:
> On 01/18/2011 03:53 AM, Jan Kiszka wrote:
>> On 2011-01-18 04:03, Stefan Berger wrote:
>>> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>>>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>>>> verify this?
>>>>>>
>>>>> Here's what I did:
>>>>>
>>>>>
>>>>> interrupt exit requested
>>>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>>>> where the code appears to be correct.
>>>>
>>> Cc'ing qemu-devel now. For reference, here the initial problem
>>> description:
>>>
>>> http://www.spinics.net/lists/kvm/msg48274.html
>>>
>>> I didn't know there was another tree...
>>>
>>> I have seen now a couple of suspends-while-reading with patches applied
>>> to the qemu-kvm.git tree and indeed, when run with the same host kernel
>>> and VM I do not see the debugging dumps due to double-reads that I would
>>> have anticipated seeing by now. Now what? Can this be easily fixed in
>>> the other Qemu tree as well?
>> Please give this a try:
>>
>> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
>>
>> I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
>> fixes the issue for you. If other problems pop up with that tree, also
>> try resetting to that particular commit.
>>
>> I'm currently trying to shake all those hidden or forgotten bug fixes
>> out of qemu-kvm and port them upstream. Most of those subtle differences
>> should hopefully soon be history.
>>
> I did the same test as I did with Avi's tree and haven't seen the
> consequences of possible double-reads. So, I would say that you should
> upstream those patches...
>
> I searched for the text you mention above using 'gitk' but couldn't find
> a patch with that headline in your tree. There were others that seem to
> be related:
>
> Gleb Natapov: "do not enter vcpu again if it was stopped during IO"
Err, I don't think you checked out queues/kvm-upstream. I bet you just
ran my master branch which is a version of qemu-kvm's master. Am I right? :)
>>> One thing I'd like to mention is that I have seen what I think are
>>> interrupt stalls when running my tests inside the qemu-kvm.git tree
>>> version and not suspending at all. A some point the interrupt counter in
>>> the guest kernel does not increase anymore even though I see the device
>>> model raising the IRQ and lowering it. The same tests run literally
>>> forever in the qemu.git tree version of Qemu.
>> What about qemu-kmv and -no-kvm-irqchip?
> That seems to be necessary for both trees, yours and the one Avi pointed
> me to. If applied, then I did not see the interrupt problem.
And the fact that you were able to call qemu from my tree with
-no-kvm-irqchip just underlines my assumption: that switch is refused by
upstream. Please retry with the latest kvm-upstream queue.
Besides that, this other bug you may see in the in-kernel IRQ path - how
can we reproduce it?
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-24 22:34 ` [Qemu-devel] " Jan Kiszka
(?)
@ 2011-01-25 3:13 ` Stefan Berger
2011-01-25 7:26 ` Jan Kiszka
-1 siblings, 1 reply; 22+ messages in thread
From: Stefan Berger @ 2011-01-25 3:13 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Avi Kivity, kvm, qemu-devel
On 01/24/2011 05:34 PM, Jan Kiszka wrote:
> On 2011-01-24 19:27, Stefan Berger wrote:
>> On 01/18/2011 03:53 AM, Jan Kiszka wrote:
>>> On 2011-01-18 04:03, Stefan Berger wrote:
>>>> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>>>>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>>>>> verify this?
>>>>>>>
>>>>>> Here's what I did:
>>>>>>
>>>>>>
>>>>>> interrupt exit requested
>>>>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>>>>> where the code appears to be correct.
>>>>>
>>>> Cc'ing qemu-devel now. For reference, here the initial problem
>>>> description:
>>>>
>>>> http://www.spinics.net/lists/kvm/msg48274.html
>>>>
>>>> I didn't know there was another tree...
>>>>
>>>> I have seen now a couple of suspends-while-reading with patches applied
>>>> to the qemu-kvm.git tree and indeed, when run with the same host kernel
>>>> and VM I do not see the debugging dumps due to double-reads that I would
>>>> have anticipated seeing by now. Now what? Can this be easily fixed in
>>>> the other Qemu tree as well?
>>> Please give this a try:
>>>
>>> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
>>>
>>> I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
>>> fixes the issue for you. If other problems pop up with that tree, also
>>> try resetting to that particular commit.
>>>
>>> I'm currently trying to shake all those hidden or forgotten bug fixes
>>> out of qemu-kvm and port them upstream. Most of those subtle differences
>>> should hopefully soon be history.
>>>
>> I did the same test as I did with Avi's tree and haven't seen the
>> consequences of possible double-reads. So, I would say that you should
>> upstream those patches...
>>
>> I searched for the text you mention above using 'gitk' but couldn't find
>> a patch with that headline in your tree. There were others that seem to
>> be related:
>>
>> Gleb Natapov: "do not enter vcpu again if it was stopped during IO"
> Err, I don't think you checked out queues/kvm-upstream. I bet you just
> ran my master branch which is a version of qemu-kvm's master. Am I right? :)
>
You're right. :-) my lack of git knowledge - checked out the branch now.
I redid the testing and it passed. No double-reads and lost bytes from
what I could see.
>>>> One thing I'd like to mention is that I have seen what I think are
>>>> interrupt stalls when running my tests inside the qemu-kvm.git tree
>>>> version and not suspending at all. A some point the interrupt counter in
>>>> the guest kernel does not increase anymore even though I see the device
>>>> model raising the IRQ and lowering it. The same tests run literally
>>>> forever in the qemu.git tree version of Qemu.
>>> What about qemu-kmv and -no-kvm-irqchip?
>> That seems to be necessary for both trees, yours and the one Avi pointed
>> me to. If applied, then I did not see the interrupt problem.
> And the fact that you were able to call qemu from my tree with
> -no-kvm-irqchip just underlines my assumption: that switch is refused by
> upstream. Please retry with the latest kvm-upstream queue.
>
> Besides that, this other bug you may see in the in-kernel IRQ path - how
> can we reproduce it?
Unfortunately I don't know. Some things have to come together for the
code I am working on to become available and useful for everyone. It's
going to be a while.
Thanks!
Stefan
> Jan
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-25 3:13 ` Stefan Berger
@ 2011-01-25 7:26 ` Jan Kiszka
2011-01-25 16:49 ` Stefan Berger
0 siblings, 1 reply; 22+ messages in thread
From: Jan Kiszka @ 2011-01-25 7:26 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 3841 bytes --]
On 2011-01-25 04:13, Stefan Berger wrote:
> On 01/24/2011 05:34 PM, Jan Kiszka wrote:
>> On 2011-01-24 19:27, Stefan Berger wrote:
>>> On 01/18/2011 03:53 AM, Jan Kiszka wrote:
>>>> On 2011-01-18 04:03, Stefan Berger wrote:
>>>>> On 01/16/2011 09:43 AM, Avi Kivity wrote:
>>>>>> On 01/14/2011 09:27 PM, Stefan Berger wrote:
>>>>>>>> Can you sprinkle some printfs() arount kvm_run (in qemu-kvm.c) to
>>>>>>>> verify this?
>>>>>>>>
>>>>>>> Here's what I did:
>>>>>>>
>>>>>>>
>>>>>>> interrupt exit requested
>>>>>> It appears from this you're using qemu.git. Please try qemu-kvm.git,
>>>>>> where the code appears to be correct.
>>>>>>
>>>>> Cc'ing qemu-devel now. For reference, here the initial problem
>>>>> description:
>>>>>
>>>>> http://www.spinics.net/lists/kvm/msg48274.html
>>>>>
>>>>> I didn't know there was another tree...
>>>>>
>>>>> I have seen now a couple of suspends-while-reading with patches
>>>>> applied
>>>>> to the qemu-kvm.git tree and indeed, when run with the same host
>>>>> kernel
>>>>> and VM I do not see the debugging dumps due to double-reads that I
>>>>> would
>>>>> have anticipated seeing by now. Now what? Can this be easily fixed in
>>>>> the other Qemu tree as well?
>>>> Please give this a try:
>>>>
>>>> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream
>>>>
>>>> I bet (& hope) "kvm: Unconditionally reenter kernel after IO exits"
>>>> fixes the issue for you. If other problems pop up with that tree, also
>>>> try resetting to that particular commit.
>>>>
>>>> I'm currently trying to shake all those hidden or forgotten bug fixes
>>>> out of qemu-kvm and port them upstream. Most of those subtle
>>>> differences
>>>> should hopefully soon be history.
>>>>
>>> I did the same test as I did with Avi's tree and haven't seen the
>>> consequences of possible double-reads. So, I would say that you should
>>> upstream those patches...
>>>
>>> I searched for the text you mention above using 'gitk' but couldn't find
>>> a patch with that headline in your tree. There were others that seem to
>>> be related:
>>>
>>> Gleb Natapov: "do not enter vcpu again if it was stopped during IO"
>> Err, I don't think you checked out queues/kvm-upstream. I bet you just
>> ran my master branch which is a version of qemu-kvm's master. Am I
>> right? :)
>>
>
> You're right. :-) my lack of git knowledge - checked out the branch now.
>
> I redid the testing and it passed. No double-reads and lost bytes from
> what I could see.
Great, thanks.
>
>>>>> One thing I'd like to mention is that I have seen what I think are
>>>>> interrupt stalls when running my tests inside the qemu-kvm.git tree
>>>>> version and not suspending at all. A some point the interrupt
>>>>> counter in
>>>>> the guest kernel does not increase anymore even though I see the
>>>>> device
>>>>> model raising the IRQ and lowering it. The same tests run literally
>>>>> forever in the qemu.git tree version of Qemu.
>>>> What about qemu-kmv and -no-kvm-irqchip?
>>> That seems to be necessary for both trees, yours and the one Avi pointed
>>> me to. If applied, then I did not see the interrupt problem.
>> And the fact that you were able to call qemu from my tree with
>> -no-kvm-irqchip just underlines my assumption: that switch is refused by
>> upstream. Please retry with the latest kvm-upstream queue.
>>
>> Besides that, this other bug you may see in the in-kernel IRQ path - how
>> can we reproduce it?
> Unfortunately I don't know. Some things have to come together for the
> code I am working on to become available and useful for everyone. It's
> going to be a while.
Do you see a chance to look closer at the issue yourself? E.g.
instrument the kernel's irqchip models and dump their states once your
guest is stuck?
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-25 7:26 ` Jan Kiszka
@ 2011-01-25 16:49 ` Stefan Berger
2011-01-26 8:14 ` Jan Kiszka
0 siblings, 1 reply; 22+ messages in thread
From: Stefan Berger @ 2011-01-25 16:49 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Avi Kivity, kvm, qemu-devel
On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>
> Do you see a chance to look closer at the issue yourself? E.g.
> instrument the kernel's irqchip models and dump their states once your
> guest is stuck?
The device runs on iRQ 3. So I applied this patch here.
diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
index 3cece05..8f4f94c 100644
--- a/arch/x86/kvm/i8259.c
+++ b/arch/x86/kvm/i8259.c
@@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct kvm_kpic_state *s, int irq, int level)
{
int mask, ret = 1;
mask = 1<< irq;
- if (s->elcr& mask) /* level triggered */
+ if (s->elcr& mask) /* level triggered */ {
if (level) {
ret = !(s->irr& mask);
s->irr |= mask;
@@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct kvm_kpic_state *s, int irq, int level)
s->irr&= ~mask;
s->last_irr&= ~mask;
}
- else /* edge triggered */
+if (irq == 3)
+ printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level, s->irr);
+ }
+ else /* edge triggered */ {
if (level) {
if ((s->last_irr& mask) == 0) {
ret = !(s->irr& mask);
@@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct kvm_kpic_state *s, int irq, int level)
s->last_irr |= mask;
} else
s->last_irr&= ~mask;
-
+if (irq == 3)
+ printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level, s->irr);
+ }
return (s->imr& mask) ? -1 : ret;
}
@@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int level)
pic_lock(s);
if (irq>= 0&& irq< PIC_NUM_PINS) {
+if (irq == 3)
+printk("%s\n", __FUNCTION__);
ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
pic_update_irq(s);
trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>> 3].elcr,
While it's still working I see this here with the levels changing 0-1-0.
Though then it stops and levels are only at '1'.
[ 1773.833824] kvm_pic_set_irq
[ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
[ 1773.834161] kvm_pic_set_irq
[ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
[ 1773.834193] kvm_pic_set_irq
[ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
[ 1773.835028] kvm_pic_set_irq
[ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
[ 1773.835542] kvm_pic_set_irq
[ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
[ 1773.889892] kvm_pic_set_irq
[ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
[ 1791.258793] pic_set_irq1 119: level=1, irr = d9
[ 1791.258824] pic_set_irq1 119: level=0, irr = d1
[ 1791.402476] pic_set_irq1 119: level=1, irr = d9
[ 1791.402534] pic_set_irq1 119: level=0, irr = d1
[ 1791.402538] pic_set_irq1 119: level=1, irr = d9
[...]
I believe the last 5 shown calls can be ignored. After that the
interrupts don't go through anymore.
In the device model I see interrupts being raised and cleared. After the
last one was cleared in 'my' device model, only interrupts are raised.
This looks like as if the interrupt handler in the guest Linux was never
run, thus the IRQ is never cleared and we're stuck.
Regards,
Stefan
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-25 16:49 ` Stefan Berger
@ 2011-01-26 8:14 ` Jan Kiszka
2011-01-26 12:05 ` Stefan Berger
0 siblings, 1 reply; 22+ messages in thread
From: Jan Kiszka @ 2011-01-26 8:14 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 3746 bytes --]
On 2011-01-25 17:49, Stefan Berger wrote:
> On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>>
>> Do you see a chance to look closer at the issue yourself? E.g.
>> instrument the kernel's irqchip models and dump their states once your
>> guest is stuck?
> The device runs on iRQ 3. So I applied this patch here.
>
> diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
> index 3cece05..8f4f94c 100644
> --- a/arch/x86/kvm/i8259.c
> +++ b/arch/x86/kvm/i8259.c
> @@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct kvm_kpic_state
> *s, int irq, int level)
> {
> int mask, ret = 1;
> mask = 1<< irq;
> - if (s->elcr& mask) /* level triggered */
> + if (s->elcr& mask) /* level triggered */ {
> if (level) {
> ret = !(s->irr& mask);
> s->irr |= mask;
> @@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct
> kvm_kpic_state *s, int irq, int level)
> s->irr&= ~mask;
> s->last_irr&= ~mask;
> }
> - else /* edge triggered */
> +if (irq == 3)
> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
> s->irr);
> + }
> + else /* edge triggered */ {
> if (level) {
> if ((s->last_irr& mask) == 0) {
> ret = !(s->irr& mask);
> @@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct kvm_kpic_state
> *s, int irq, int level)
> s->last_irr |= mask;
> } else
> s->last_irr&= ~mask;
> -
> +if (irq == 3)
> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
> s->irr);
> + }
> return (s->imr& mask) ? -1 : ret;
> }
>
> @@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int level)
>
> pic_lock(s);
> if (irq>= 0&& irq< PIC_NUM_PINS) {
> +if (irq == 3)
> +printk("%s\n", __FUNCTION__);
> ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
> pic_update_irq(s);
> trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>> 3].elcr,
>
>
>
> While it's still working I see this here with the levels changing 0-1-0.
> Though then it stops and levels are only at '1'.
>
> [ 1773.833824] kvm_pic_set_irq
> [ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
> [ 1773.834161] kvm_pic_set_irq
> [ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
> [ 1773.834193] kvm_pic_set_irq
> [ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
> [ 1773.835028] kvm_pic_set_irq
> [ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
> [ 1773.835542] kvm_pic_set_irq
> [ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
> [ 1773.889892] kvm_pic_set_irq
> [ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
> [ 1791.258793] pic_set_irq1 119: level=1, irr = d9
> [ 1791.258824] pic_set_irq1 119: level=0, irr = d1
> [ 1791.402476] pic_set_irq1 119: level=1, irr = d9
> [ 1791.402534] pic_set_irq1 119: level=0, irr = d1
> [ 1791.402538] pic_set_irq1 119: level=1, irr = d9
> [...]
>
>
> I believe the last 5 shown calls can be ignored. After that the
> interrupts don't go through anymore.
>
> In the device model I see interrupts being raised and cleared. After the
> last one was cleared in 'my' device model, only interrupts are raised.
> This looks like as if the interrupt handler in the guest Linux was never
> run, thus the IRQ is never cleared and we're stuck.
>
User space is responsible for both setting and clearing that line. IRQ3
means you are using some serial device model? Then you should check what
its state is.
Moreover, a complete picture of the kernel/user space interaction should
be obtainable by using fstrace for capturing kvm events.
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-26 8:14 ` Jan Kiszka
@ 2011-01-26 12:05 ` Stefan Berger
2011-01-26 12:09 ` Jan Kiszka
0 siblings, 1 reply; 22+ messages in thread
From: Stefan Berger @ 2011-01-26 12:05 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Avi Kivity, kvm, qemu-devel
On 01/26/2011 03:14 AM, Jan Kiszka wrote:
> On 2011-01-25 17:49, Stefan Berger wrote:
>> On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>>> Do you see a chance to look closer at the issue yourself? E.g.
>>> instrument the kernel's irqchip models and dump their states once your
>>> guest is stuck?
>> The device runs on iRQ 3. So I applied this patch here.
>>
>> diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
>> index 3cece05..8f4f94c 100644
>> --- a/arch/x86/kvm/i8259.c
>> +++ b/arch/x86/kvm/i8259.c
>> @@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct kvm_kpic_state
>> *s, int irq, int level)
>> {
>> int mask, ret = 1;
>> mask = 1<< irq;
>> - if (s->elcr& mask) /* level triggered */
>> + if (s->elcr& mask) /* level triggered */ {
>> if (level) {
>> ret = !(s->irr& mask);
>> s->irr |= mask;
>> @@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct
>> kvm_kpic_state *s, int irq, int level)
>> s->irr&= ~mask;
>> s->last_irr&= ~mask;
>> }
>> - else /* edge triggered */
>> +if (irq == 3)
>> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
>> s->irr);
>> + }
>> + else /* edge triggered */ {
>> if (level) {
>> if ((s->last_irr& mask) == 0) {
>> ret = !(s->irr& mask);
>> @@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct kvm_kpic_state
>> *s, int irq, int level)
>> s->last_irr |= mask;
>> } else
>> s->last_irr&= ~mask;
>> -
>> +if (irq == 3)
>> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
>> s->irr);
>> + }
>> return (s->imr& mask) ? -1 : ret;
>> }
>>
>> @@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int level)
>>
>> pic_lock(s);
>> if (irq>= 0&& irq< PIC_NUM_PINS) {
>> +if (irq == 3)
>> +printk("%s\n", __FUNCTION__);
>> ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
>> pic_update_irq(s);
>> trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>> 3].elcr,
>>
>>
>>
>> While it's still working I see this here with the levels changing 0-1-0.
>> Though then it stops and levels are only at '1'.
>>
>> [ 1773.833824] kvm_pic_set_irq
>> [ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
>> [ 1773.834161] kvm_pic_set_irq
>> [ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
>> [ 1773.834193] kvm_pic_set_irq
>> [ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
>> [ 1773.835028] kvm_pic_set_irq
>> [ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
>> [ 1773.835542] kvm_pic_set_irq
>> [ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
>> [ 1773.889892] kvm_pic_set_irq
>> [ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
>> [ 1791.258793] pic_set_irq1 119: level=1, irr = d9
>> [ 1791.258824] pic_set_irq1 119: level=0, irr = d1
>> [ 1791.402476] pic_set_irq1 119: level=1, irr = d9
>> [ 1791.402534] pic_set_irq1 119: level=0, irr = d1
>> [ 1791.402538] pic_set_irq1 119: level=1, irr = d9
>> [...]
>>
>>
>> I believe the last 5 shown calls can be ignored. After that the
>> interrupts don't go through anymore.
>>
>> In the device model I see interrupts being raised and cleared. After the
>> last one was cleared in 'my' device model, only interrupts are raised.
>> This looks like as if the interrupt handler in the guest Linux was never
>> run, thus the IRQ is never cleared and we're stuck.
>>
> User space is responsible for both setting and clearing that line. IRQ3
> means you are using some serial device model? Then you should check what
> its state is.
Good hint. I moved it now to IRQ11 and it works fine now (with kvm-git)
from what I can see. There was no UART on IRQ3 before, though, but
certainly it was the wrong IRQ for it.
> Moreover, a complete picture of the kernel/user space interaction should
> be obtainable by using fstrace for capturing kvm events.
>
Should it be working on IRQ3? If so, I'd look into it when I get a chance...
Stefan
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-26 12:05 ` Stefan Berger
@ 2011-01-26 12:09 ` Jan Kiszka
2011-01-26 13:08 ` Stefan Berger
0 siblings, 1 reply; 22+ messages in thread
From: Jan Kiszka @ 2011-01-26 12:09 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 4538 bytes --]
On 2011-01-26 13:05, Stefan Berger wrote:
> On 01/26/2011 03:14 AM, Jan Kiszka wrote:
>> On 2011-01-25 17:49, Stefan Berger wrote:
>>> On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>>>> Do you see a chance to look closer at the issue yourself? E.g.
>>>> instrument the kernel's irqchip models and dump their states once your
>>>> guest is stuck?
>>> The device runs on iRQ 3. So I applied this patch here.
>>>
>>> diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
>>> index 3cece05..8f4f94c 100644
>>> --- a/arch/x86/kvm/i8259.c
>>> +++ b/arch/x86/kvm/i8259.c
>>> @@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct kvm_kpic_state
>>> *s, int irq, int level)
>>> {
>>> int mask, ret = 1;
>>> mask = 1<< irq;
>>> - if (s->elcr& mask) /* level triggered */
>>> + if (s->elcr& mask) /* level triggered */ {
>>> if (level) {
>>> ret = !(s->irr& mask);
>>> s->irr |= mask;
>>> @@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct
>>> kvm_kpic_state *s, int irq, int level)
>>> s->irr&= ~mask;
>>> s->last_irr&= ~mask;
>>> }
>>> - else /* edge triggered */
>>> +if (irq == 3)
>>> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
>>> s->irr);
>>> + }
>>> + else /* edge triggered */ {
>>> if (level) {
>>> if ((s->last_irr& mask) == 0) {
>>> ret = !(s->irr& mask);
>>> @@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct kvm_kpic_state
>>> *s, int irq, int level)
>>> s->last_irr |= mask;
>>> } else
>>> s->last_irr&= ~mask;
>>> -
>>> +if (irq == 3)
>>> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
>>> s->irr);
>>> + }
>>> return (s->imr& mask) ? -1 : ret;
>>> }
>>>
>>> @@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int
>>> level)
>>>
>>> pic_lock(s);
>>> if (irq>= 0&& irq< PIC_NUM_PINS) {
>>> +if (irq == 3)
>>> +printk("%s\n", __FUNCTION__);
>>> ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
>>> pic_update_irq(s);
>>> trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>>
>>> 3].elcr,
>>>
>>>
>>>
>>> While it's still working I see this here with the levels changing 0-1-0.
>>> Though then it stops and levels are only at '1'.
>>>
>>> [ 1773.833824] kvm_pic_set_irq
>>> [ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
>>> [ 1773.834161] kvm_pic_set_irq
>>> [ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
>>> [ 1773.834193] kvm_pic_set_irq
>>> [ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
>>> [ 1773.835028] kvm_pic_set_irq
>>> [ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
>>> [ 1773.835542] kvm_pic_set_irq
>>> [ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
>>> [ 1773.889892] kvm_pic_set_irq
>>> [ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
>>> [ 1791.258793] pic_set_irq1 119: level=1, irr = d9
>>> [ 1791.258824] pic_set_irq1 119: level=0, irr = d1
>>> [ 1791.402476] pic_set_irq1 119: level=1, irr = d9
>>> [ 1791.402534] pic_set_irq1 119: level=0, irr = d1
>>> [ 1791.402538] pic_set_irq1 119: level=1, irr = d9
>>> [...]
>>>
>>>
>>> I believe the last 5 shown calls can be ignored. After that the
>>> interrupts don't go through anymore.
>>>
>>> In the device model I see interrupts being raised and cleared. After the
>>> last one was cleared in 'my' device model, only interrupts are raised.
>>> This looks like as if the interrupt handler in the guest Linux was never
>>> run, thus the IRQ is never cleared and we're stuck.
>>>
>> User space is responsible for both setting and clearing that line. IRQ3
>> means you are using some serial device model? Then you should check what
>> its state is.
> Good hint. I moved it now to IRQ11 and it works fine now (with kvm-git)
> from what I can see. There was no UART on IRQ3 before, though, but
> certainly it was the wrong IRQ for it.
>> Moreover, a complete picture of the kernel/user space interaction should
>> be obtainable by using fstrace for capturing kvm events.
>>
> Should it be working on IRQ3? If so, I'd look into it when I get a
> chance...
I don't know your customizations, so it's hard to tell if that should
work or not. IRQ3 is intended to be used by ISA devices on the PC
machine. Are you adding an ISA model, or what is your use case?
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-26 12:09 ` Jan Kiszka
@ 2011-01-26 13:08 ` Stefan Berger
2011-01-26 13:15 ` Jan Kiszka
0 siblings, 1 reply; 22+ messages in thread
From: Stefan Berger @ 2011-01-26 13:08 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Avi Kivity, kvm, qemu-devel
On 01/26/2011 07:09 AM, Jan Kiszka wrote:
> On 2011-01-26 13:05, Stefan Berger wrote:
>> On 01/26/2011 03:14 AM, Jan Kiszka wrote:
>>> On 2011-01-25 17:49, Stefan Berger wrote:
>>>> On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>>>>> Do you see a chance to look closer at the issue yourself? E.g.
>>>>> instrument the kernel's irqchip models and dump their states once your
>>>>> guest is stuck?
>>>> The device runs on iRQ 3. So I applied this patch here.
>>>>
>>>> diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
>>>> index 3cece05..8f4f94c 100644
>>>> --- a/arch/x86/kvm/i8259.c
>>>> +++ b/arch/x86/kvm/i8259.c
>>>> @@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct kvm_kpic_state
>>>> *s, int irq, int level)
>>>> {
>>>> int mask, ret = 1;
>>>> mask = 1<< irq;
>>>> - if (s->elcr& mask) /* level triggered */
>>>> + if (s->elcr& mask) /* level triggered */ {
>>>> if (level) {
>>>> ret = !(s->irr& mask);
>>>> s->irr |= mask;
>>>> @@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct
>>>> kvm_kpic_state *s, int irq, int level)
>>>> s->irr&= ~mask;
>>>> s->last_irr&= ~mask;
>>>> }
>>>> - else /* edge triggered */
>>>> +if (irq == 3)
>>>> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
>>>> s->irr);
>>>> + }
>>>> + else /* edge triggered */ {
>>>> if (level) {
>>>> if ((s->last_irr& mask) == 0) {
>>>> ret = !(s->irr& mask);
>>>> @@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct kvm_kpic_state
>>>> *s, int irq, int level)
>>>> s->last_irr |= mask;
>>>> } else
>>>> s->last_irr&= ~mask;
>>>> -
>>>> +if (irq == 3)
>>>> + printk("%s %d: level=%d, irr = %x\n", __FUNCTION__,__LINE__,level,
>>>> s->irr);
>>>> + }
>>>> return (s->imr& mask) ? -1 : ret;
>>>> }
>>>>
>>>> @@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int
>>>> level)
>>>>
>>>> pic_lock(s);
>>>> if (irq>= 0&& irq< PIC_NUM_PINS) {
>>>> +if (irq == 3)
>>>> +printk("%s\n", __FUNCTION__);
>>>> ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
>>>> pic_update_irq(s);
>>>> trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>>
>>>> 3].elcr,
>>>>
>>>>
>>>>
>>>> While it's still working I see this here with the levels changing 0-1-0.
>>>> Though then it stops and levels are only at '1'.
>>>>
>>>> [ 1773.833824] kvm_pic_set_irq
>>>> [ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
>>>> [ 1773.834161] kvm_pic_set_irq
>>>> [ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
>>>> [ 1773.834193] kvm_pic_set_irq
>>>> [ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
>>>> [ 1773.835028] kvm_pic_set_irq
>>>> [ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
>>>> [ 1773.835542] kvm_pic_set_irq
>>>> [ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
>>>> [ 1773.889892] kvm_pic_set_irq
>>>> [ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
>>>> [ 1791.258793] pic_set_irq1 119: level=1, irr = d9
>>>> [ 1791.258824] pic_set_irq1 119: level=0, irr = d1
>>>> [ 1791.402476] pic_set_irq1 119: level=1, irr = d9
>>>> [ 1791.402534] pic_set_irq1 119: level=0, irr = d1
>>>> [ 1791.402538] pic_set_irq1 119: level=1, irr = d9
>>>> [...]
>>>>
>>>>
>>>> I believe the last 5 shown calls can be ignored. After that the
>>>> interrupts don't go through anymore.
>>>>
>>>> In the device model I see interrupts being raised and cleared. After the
>>>> last one was cleared in 'my' device model, only interrupts are raised.
>>>> This looks like as if the interrupt handler in the guest Linux was never
>>>> run, thus the IRQ is never cleared and we're stuck.
>>>>
>>> User space is responsible for both setting and clearing that line. IRQ3
>>> means you are using some serial device model? Then you should check what
>>> its state is.
>> Good hint. I moved it now to IRQ11 and it works fine now (with kvm-git)
>> from what I can see. There was no UART on IRQ3 before, though, but
>> certainly it was the wrong IRQ for it.
>>> Moreover, a complete picture of the kernel/user space interaction should
>>> be obtainable by using fstrace for capturing kvm events.
>>>
>> Should it be working on IRQ3? If so, I'd look into it when I get a
>> chance...
> I don't know your customizations, so it's hard to tell if that should
> work or not. IRQ3 is intended to be used by ISA devices on the PC
> machine. Are you adding an ISA model, or what is your use case?
>
The use case is to add a TPM device interface.
http://xenbits.xensource.com/xen-unstable.hg?file/1e56ac73b9b9/tools/ioemu/hw/tpm_tis.c
This one typically is connected to the LPC bus.
Stefan
> Jan
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-26 13:08 ` Stefan Berger
@ 2011-01-26 13:15 ` Jan Kiszka
2011-01-26 13:31 ` Jan Kiszka
0 siblings, 1 reply; 22+ messages in thread
From: Jan Kiszka @ 2011-01-26 13:15 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 5275 bytes --]
On 2011-01-26 14:08, Stefan Berger wrote:
> On 01/26/2011 07:09 AM, Jan Kiszka wrote:
>> On 2011-01-26 13:05, Stefan Berger wrote:
>>> On 01/26/2011 03:14 AM, Jan Kiszka wrote:
>>>> On 2011-01-25 17:49, Stefan Berger wrote:
>>>>> On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>>>>>> Do you see a chance to look closer at the issue yourself? E.g.
>>>>>> instrument the kernel's irqchip models and dump their states once
>>>>>> your
>>>>>> guest is stuck?
>>>>> The device runs on iRQ 3. So I applied this patch here.
>>>>>
>>>>> diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
>>>>> index 3cece05..8f4f94c 100644
>>>>> --- a/arch/x86/kvm/i8259.c
>>>>> +++ b/arch/x86/kvm/i8259.c
>>>>> @@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct
>>>>> kvm_kpic_state
>>>>> *s, int irq, int level)
>>>>> {
>>>>> int mask, ret = 1;
>>>>> mask = 1<< irq;
>>>>> - if (s->elcr& mask) /* level triggered */
>>>>> + if (s->elcr& mask) /* level triggered */ {
>>>>> if (level) {
>>>>> ret = !(s->irr& mask);
>>>>> s->irr |= mask;
>>>>> @@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct
>>>>> kvm_kpic_state *s, int irq, int level)
>>>>> s->irr&= ~mask;
>>>>> s->last_irr&= ~mask;
>>>>> }
>>>>> - else /* edge triggered */
>>>>> +if (irq == 3)
>>>>> + printk("%s %d: level=%d, irr = %x\n",
>>>>> __FUNCTION__,__LINE__,level,
>>>>> s->irr);
>>>>> + }
>>>>> + else /* edge triggered */ {
>>>>> if (level) {
>>>>> if ((s->last_irr& mask) == 0) {
>>>>> ret = !(s->irr& mask);
>>>>> @@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct
>>>>> kvm_kpic_state
>>>>> *s, int irq, int level)
>>>>> s->last_irr |= mask;
>>>>> } else
>>>>> s->last_irr&= ~mask;
>>>>> -
>>>>> +if (irq == 3)
>>>>> + printk("%s %d: level=%d, irr = %x\n",
>>>>> __FUNCTION__,__LINE__,level,
>>>>> s->irr);
>>>>> + }
>>>>> return (s->imr& mask) ? -1 : ret;
>>>>> }
>>>>>
>>>>> @@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int
>>>>> level)
>>>>>
>>>>> pic_lock(s);
>>>>> if (irq>= 0&& irq< PIC_NUM_PINS) {
>>>>> +if (irq == 3)
>>>>> +printk("%s\n", __FUNCTION__);
>>>>> ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
>>>>> pic_update_irq(s);
>>>>> trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>>
>>>>> 3].elcr,
>>>>>
>>>>>
>>>>>
>>>>> While it's still working I see this here with the levels changing
>>>>> 0-1-0.
>>>>> Though then it stops and levels are only at '1'.
>>>>>
>>>>> [ 1773.833824] kvm_pic_set_irq
>>>>> [ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
>>>>> [ 1773.834161] kvm_pic_set_irq
>>>>> [ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
>>>>> [ 1773.834193] kvm_pic_set_irq
>>>>> [ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
>>>>> [ 1773.835028] kvm_pic_set_irq
>>>>> [ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
>>>>> [ 1773.835542] kvm_pic_set_irq
>>>>> [ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
>>>>> [ 1773.889892] kvm_pic_set_irq
>>>>> [ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
>>>>> [ 1791.258793] pic_set_irq1 119: level=1, irr = d9
>>>>> [ 1791.258824] pic_set_irq1 119: level=0, irr = d1
>>>>> [ 1791.402476] pic_set_irq1 119: level=1, irr = d9
>>>>> [ 1791.402534] pic_set_irq1 119: level=0, irr = d1
>>>>> [ 1791.402538] pic_set_irq1 119: level=1, irr = d9
>>>>> [...]
>>>>>
>>>>>
>>>>> I believe the last 5 shown calls can be ignored. After that the
>>>>> interrupts don't go through anymore.
>>>>>
>>>>> In the device model I see interrupts being raised and cleared.
>>>>> After the
>>>>> last one was cleared in 'my' device model, only interrupts are raised.
>>>>> This looks like as if the interrupt handler in the guest Linux was
>>>>> never
>>>>> run, thus the IRQ is never cleared and we're stuck.
>>>>>
>>>> User space is responsible for both setting and clearing that line. IRQ3
>>>> means you are using some serial device model? Then you should check
>>>> what
>>>> its state is.
>>> Good hint. I moved it now to IRQ11 and it works fine now (with kvm-git)
>>> from what I can see. There was no UART on IRQ3 before, though, but
>>> certainly it was the wrong IRQ for it.
>>>> Moreover, a complete picture of the kernel/user space interaction
>>>> should
>>>> be obtainable by using fstrace for capturing kvm events.
>>>>
>>> Should it be working on IRQ3? If so, I'd look into it when I get a
>>> chance...
>> I don't know your customizations, so it's hard to tell if that should
>> work or not. IRQ3 is intended to be used by ISA devices on the PC
>> machine. Are you adding an ISA model, or what is your use case?
>>
> The use case is to add a TPM device interface.
>
> http://xenbits.xensource.com/xen-unstable.hg?file/1e56ac73b9b9/tools/ioemu/hw/tpm_tis.c
>
>
> This one typically is connected to the LPC bus.
I see. Do you also have the xen-free version of it? Maybe there are
still issues with proper qdev integration etc.
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-26 13:15 ` Jan Kiszka
@ 2011-01-26 13:31 ` Jan Kiszka
2011-01-26 13:52 ` Stefan Berger
0 siblings, 1 reply; 22+ messages in thread
From: Jan Kiszka @ 2011-01-26 13:31 UTC (permalink / raw)
To: Stefan Berger; +Cc: Avi Kivity, kvm, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 6159 bytes --]
On 2011-01-26 14:15, Jan Kiszka wrote:
> On 2011-01-26 14:08, Stefan Berger wrote:
>> On 01/26/2011 07:09 AM, Jan Kiszka wrote:
>>> On 2011-01-26 13:05, Stefan Berger wrote:
>>>> On 01/26/2011 03:14 AM, Jan Kiszka wrote:
>>>>> On 2011-01-25 17:49, Stefan Berger wrote:
>>>>>> On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>>>>>>> Do you see a chance to look closer at the issue yourself? E.g.
>>>>>>> instrument the kernel's irqchip models and dump their states once
>>>>>>> your
>>>>>>> guest is stuck?
>>>>>> The device runs on iRQ 3. So I applied this patch here.
>>>>>>
>>>>>> diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
>>>>>> index 3cece05..8f4f94c 100644
>>>>>> --- a/arch/x86/kvm/i8259.c
>>>>>> +++ b/arch/x86/kvm/i8259.c
>>>>>> @@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct
>>>>>> kvm_kpic_state
>>>>>> *s, int irq, int level)
>>>>>> {
>>>>>> int mask, ret = 1;
>>>>>> mask = 1<< irq;
>>>>>> - if (s->elcr& mask) /* level triggered */
>>>>>> + if (s->elcr& mask) /* level triggered */ {
>>>>>> if (level) {
>>>>>> ret = !(s->irr& mask);
>>>>>> s->irr |= mask;
>>>>>> @@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct
>>>>>> kvm_kpic_state *s, int irq, int level)
>>>>>> s->irr&= ~mask;
>>>>>> s->last_irr&= ~mask;
>>>>>> }
>>>>>> - else /* edge triggered */
>>>>>> +if (irq == 3)
>>>>>> + printk("%s %d: level=%d, irr = %x\n",
>>>>>> __FUNCTION__,__LINE__,level,
>>>>>> s->irr);
>>>>>> + }
>>>>>> + else /* edge triggered */ {
>>>>>> if (level) {
>>>>>> if ((s->last_irr& mask) == 0) {
>>>>>> ret = !(s->irr& mask);
>>>>>> @@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct
>>>>>> kvm_kpic_state
>>>>>> *s, int irq, int level)
>>>>>> s->last_irr |= mask;
>>>>>> } else
>>>>>> s->last_irr&= ~mask;
>>>>>> -
>>>>>> +if (irq == 3)
>>>>>> + printk("%s %d: level=%d, irr = %x\n",
>>>>>> __FUNCTION__,__LINE__,level,
>>>>>> s->irr);
>>>>>> + }
>>>>>> return (s->imr& mask) ? -1 : ret;
>>>>>> }
>>>>>>
>>>>>> @@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int
>>>>>> level)
>>>>>>
>>>>>> pic_lock(s);
>>>>>> if (irq>= 0&& irq< PIC_NUM_PINS) {
>>>>>> +if (irq == 3)
>>>>>> +printk("%s\n", __FUNCTION__);
>>>>>> ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
>>>>>> pic_update_irq(s);
>>>>>> trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>>
>>>>>> 3].elcr,
>>>>>>
>>>>>>
>>>>>>
>>>>>> While it's still working I see this here with the levels changing
>>>>>> 0-1-0.
>>>>>> Though then it stops and levels are only at '1'.
>>>>>>
>>>>>> [ 1773.833824] kvm_pic_set_irq
>>>>>> [ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
>>>>>> [ 1773.834161] kvm_pic_set_irq
>>>>>> [ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
>>>>>> [ 1773.834193] kvm_pic_set_irq
>>>>>> [ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
>>>>>> [ 1773.835028] kvm_pic_set_irq
>>>>>> [ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
>>>>>> [ 1773.835542] kvm_pic_set_irq
>>>>>> [ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
>>>>>> [ 1773.889892] kvm_pic_set_irq
>>>>>> [ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
>>>>>> [ 1791.258793] pic_set_irq1 119: level=1, irr = d9
>>>>>> [ 1791.258824] pic_set_irq1 119: level=0, irr = d1
>>>>>> [ 1791.402476] pic_set_irq1 119: level=1, irr = d9
>>>>>> [ 1791.402534] pic_set_irq1 119: level=0, irr = d1
>>>>>> [ 1791.402538] pic_set_irq1 119: level=1, irr = d9
>>>>>> [...]
>>>>>>
>>>>>>
>>>>>> I believe the last 5 shown calls can be ignored. After that the
>>>>>> interrupts don't go through anymore.
>>>>>>
>>>>>> In the device model I see interrupts being raised and cleared.
>>>>>> After the
>>>>>> last one was cleared in 'my' device model, only interrupts are raised.
>>>>>> This looks like as if the interrupt handler in the guest Linux was
>>>>>> never
>>>>>> run, thus the IRQ is never cleared and we're stuck.
>>>>>>
>>>>> User space is responsible for both setting and clearing that line. IRQ3
>>>>> means you are using some serial device model? Then you should check
>>>>> what
>>>>> its state is.
>>>> Good hint. I moved it now to IRQ11 and it works fine now (with kvm-git)
>>>> from what I can see. There was no UART on IRQ3 before, though, but
>>>> certainly it was the wrong IRQ for it.
>>>>> Moreover, a complete picture of the kernel/user space interaction
>>>>> should
>>>>> be obtainable by using fstrace for capturing kvm events.
>>>>>
>>>> Should it be working on IRQ3? If so, I'd look into it when I get a
>>>> chance...
>>> I don't know your customizations, so it's hard to tell if that should
>>> work or not. IRQ3 is intended to be used by ISA devices on the PC
>>> machine. Are you adding an ISA model, or what is your use case?
>>>
>> The use case is to add a TPM device interface.
>>
>> http://xenbits.xensource.com/xen-unstable.hg?file/1e56ac73b9b9/tools/ioemu/hw/tpm_tis.c
>>
>>
>> This one typically is connected to the LPC bus.
>
> I see. Do you also have the xen-free version of it? Maybe there are
> still issues with proper qdev integration etc.
>
Without knowing the hardware spec or what is actually behind set_irq,
this looks at least suspicious:
[...]
if (off == TPM_REG_INT_STATUS) {
/* clearing of interrupt flags */
if ((val & INTERRUPTS_SUPPORTED) &&
(s->loc[locty].ints & INTERRUPTS_SUPPORTED)) {
s->set_irq(s->irq_opaque, s->irq, 0);
s->irq_pending = 0;
}
s->loc[locty].ints &= ~(val & INTERRUPTS_SUPPORTED);
} else
[...]
The code does no
t check if there are ints left after masking out those provided in val.
Does that device already de-asserts the line if you only clear a single
interrupt reason?
BTW, irq_pending looks redundant, at least when using the qemu irq
subsystem.
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] Re: Errors on MMIO read access on VM suspend / resume operations
2011-01-26 13:31 ` Jan Kiszka
@ 2011-01-26 13:52 ` Stefan Berger
0 siblings, 0 replies; 22+ messages in thread
From: Stefan Berger @ 2011-01-26 13:52 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Avi Kivity, kvm, qemu-devel
On 01/26/2011 08:31 AM, Jan Kiszka wrote:
> On 2011-01-26 14:15, Jan Kiszka wrote:
>> On 2011-01-26 14:08, Stefan Berger wrote:
>>> On 01/26/2011 07:09 AM, Jan Kiszka wrote:
>>>> On 2011-01-26 13:05, Stefan Berger wrote:
>>>>> On 01/26/2011 03:14 AM, Jan Kiszka wrote:
>>>>>> On 2011-01-25 17:49, Stefan Berger wrote:
>>>>>>> On 01/25/2011 02:26 AM, Jan Kiszka wrote:
>>>>>>>> Do you see a chance to look closer at the issue yourself? E.g.
>>>>>>>> instrument the kernel's irqchip models and dump their states once
>>>>>>>> your
>>>>>>>> guest is stuck?
>>>>>>> The device runs on iRQ 3. So I applied this patch here.
>>>>>>>
>>>>>>> diff --git a/arch/x86/kvm/i8259.c b/arch/x86/kvm/i8259.c
>>>>>>> index 3cece05..8f4f94c 100644
>>>>>>> --- a/arch/x86/kvm/i8259.c
>>>>>>> +++ b/arch/x86/kvm/i8259.c
>>>>>>> @@ -106,7 +106,7 @@ static inline int pic_set_irq1(struct
>>>>>>> kvm_kpic_state
>>>>>>> *s, int irq, int level)
>>>>>>> {
>>>>>>> int mask, ret = 1;
>>>>>>> mask = 1<< irq;
>>>>>>> - if (s->elcr& mask) /* level triggered */
>>>>>>> + if (s->elcr& mask) /* level triggered */ {
>>>>>>> if (level) {
>>>>>>> ret = !(s->irr& mask);
>>>>>>> s->irr |= mask;
>>>>>>> @@ -115,7 +115,10 @@ static inline int pic_set_irq1(struct
>>>>>>> kvm_kpic_state *s, int irq, int level)
>>>>>>> s->irr&= ~mask;
>>>>>>> s->last_irr&= ~mask;
>>>>>>> }
>>>>>>> - else /* edge triggered */
>>>>>>> +if (irq == 3)
>>>>>>> + printk("%s %d: level=%d, irr = %x\n",
>>>>>>> __FUNCTION__,__LINE__,level,
>>>>>>> s->irr);
>>>>>>> + }
>>>>>>> + else /* edge triggered */ {
>>>>>>> if (level) {
>>>>>>> if ((s->last_irr& mask) == 0) {
>>>>>>> ret = !(s->irr& mask);
>>>>>>> @@ -124,7 +127,9 @@ static inline int pic_set_irq1(struct
>>>>>>> kvm_kpic_state
>>>>>>> *s, int irq, int level)
>>>>>>> s->last_irr |= mask;
>>>>>>> } else
>>>>>>> s->last_irr&= ~mask;
>>>>>>> -
>>>>>>> +if (irq == 3)
>>>>>>> + printk("%s %d: level=%d, irr = %x\n",
>>>>>>> __FUNCTION__,__LINE__,level,
>>>>>>> s->irr);
>>>>>>> + }
>>>>>>> return (s->imr& mask) ? -1 : ret;
>>>>>>> }
>>>>>>>
>>>>>>> @@ -206,6 +211,8 @@ int kvm_pic_set_irq(void *opaque, int irq, int
>>>>>>> level)
>>>>>>>
>>>>>>> pic_lock(s);
>>>>>>> if (irq>= 0&& irq< PIC_NUM_PINS) {
>>>>>>> +if (irq == 3)
>>>>>>> +printk("%s\n", __FUNCTION__);
>>>>>>> ret = pic_set_irq1(&s->pics[irq>> 3], irq& 7, level);
>>>>>>> pic_update_irq(s);
>>>>>>> trace_kvm_pic_set_irq(irq>> 3, irq& 7, s->pics[irq>>
>>>>>>> 3].elcr,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> While it's still working I see this here with the levels changing
>>>>>>> 0-1-0.
>>>>>>> Though then it stops and levels are only at '1'.
>>>>>>>
>>>>>>> [ 1773.833824] kvm_pic_set_irq
>>>>>>> [ 1773.833827] pic_set_irq1 131: level=0, irr = 5b
>>>>>>> [ 1773.834161] kvm_pic_set_irq
>>>>>>> [ 1773.834163] pic_set_irq1 131: level=1, irr = 5b
>>>>>>> [ 1773.834193] kvm_pic_set_irq
>>>>>>> [ 1773.834195] pic_set_irq1 131: level=0, irr = 5b
>>>>>>> [ 1773.835028] kvm_pic_set_irq
>>>>>>> [ 1773.835031] pic_set_irq1 131: level=1, irr = 5b
>>>>>>> [ 1773.835542] kvm_pic_set_irq
>>>>>>> [ 1773.835545] pic_set_irq1 131: level=1, irr = 5b
>>>>>>> [ 1773.889892] kvm_pic_set_irq
>>>>>>> [ 1773.889894] pic_set_irq1 131: level=1, irr = 5b
>>>>>>> [ 1791.258793] pic_set_irq1 119: level=1, irr = d9
>>>>>>> [ 1791.258824] pic_set_irq1 119: level=0, irr = d1
>>>>>>> [ 1791.402476] pic_set_irq1 119: level=1, irr = d9
>>>>>>> [ 1791.402534] pic_set_irq1 119: level=0, irr = d1
>>>>>>> [ 1791.402538] pic_set_irq1 119: level=1, irr = d9
>>>>>>> [...]
>>>>>>>
>>>>>>>
>>>>>>> I believe the last 5 shown calls can be ignored. After that the
>>>>>>> interrupts don't go through anymore.
>>>>>>>
>>>>>>> In the device model I see interrupts being raised and cleared.
>>>>>>> After the
>>>>>>> last one was cleared in 'my' device model, only interrupts are raised.
>>>>>>> This looks like as if the interrupt handler in the guest Linux was
>>>>>>> never
>>>>>>> run, thus the IRQ is never cleared and we're stuck.
>>>>>>>
>>>>>> User space is responsible for both setting and clearing that line. IRQ3
>>>>>> means you are using some serial device model? Then you should check
>>>>>> what
>>>>>> its state is.
>>>>> Good hint. I moved it now to IRQ11 and it works fine now (with kvm-git)
>>>>> from what I can see. There was no UART on IRQ3 before, though, but
>>>>> certainly it was the wrong IRQ for it.
>>>>>> Moreover, a complete picture of the kernel/user space interaction
>>>>>> should
>>>>>> be obtainable by using fstrace for capturing kvm events.
>>>>>>
>>>>> Should it be working on IRQ3? If so, I'd look into it when I get a
>>>>> chance...
>>>> I don't know your customizations, so it's hard to tell if that should
>>>> work or not. IRQ3 is intended to be used by ISA devices on the PC
>>>> machine. Are you adding an ISA model, or what is your use case?
>>>>
>>> The use case is to add a TPM device interface.
>>>
>>> http://xenbits.xensource.com/xen-unstable.hg?file/1e56ac73b9b9/tools/ioemu/hw/tpm_tis.c
>>>
>>>
>>> This one typically is connected to the LPC bus.
>> I see. Do you also have the xen-free version of it? Maybe there are
>> still issues with proper qdev integration etc.
>>
> Without knowing the hardware spec or what is actually behind set_irq,
> this looks at least suspicious:
>
> [...]
> if (off == TPM_REG_INT_STATUS) {
> /* clearing of interrupt flags */
> if ((val& INTERRUPTS_SUPPORTED)&&
> (s->loc[locty].ints& INTERRUPTS_SUPPORTED)) {
> s->set_irq(s->irq_opaque, s->irq, 0);
> s->irq_pending = 0;
> }
> s->loc[locty].ints&= ~(val& INTERRUPTS_SUPPORTED);
> } else
> [...]
>
> The code does no
> t check if there are ints left after masking out those provided in val.
> Does that device already de-asserts the line if you only clear a single
> interrupt reason?
>
> BTW, irq_pending looks redundant, at least when using the qemu irq
> subsystem.
The code has substantially changed in the meantime -- the Xen repository
code is from > 3 years ago - I had to go backwards in the xen unstable
repository to find it. The link was merely meant to show what type of
device is being added. As said, some other things need to come together
first before this will become available.
Stefan
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2011-01-26 13:52 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-01-11 16:19 Errors on MMIO read access on VM suspend / resume operations Stefan Berger
2011-01-13 10:22 ` Avi Kivity
2011-01-14 19:27 ` Stefan Berger
2011-01-16 14:43 ` Avi Kivity
2011-01-18 3:03 ` Stefan Berger
2011-01-18 3:03 ` [Qemu-devel] " Stefan Berger
2011-01-18 8:53 ` Jan Kiszka
2011-01-18 8:53 ` [Qemu-devel] " Jan Kiszka
2011-01-24 18:27 ` Stefan Berger
2011-01-24 18:27 ` [Qemu-devel] " Stefan Berger
2011-01-24 22:34 ` Jan Kiszka
2011-01-24 22:34 ` [Qemu-devel] " Jan Kiszka
2011-01-25 3:13 ` Stefan Berger
2011-01-25 7:26 ` Jan Kiszka
2011-01-25 16:49 ` Stefan Berger
2011-01-26 8:14 ` Jan Kiszka
2011-01-26 12:05 ` Stefan Berger
2011-01-26 12:09 ` Jan Kiszka
2011-01-26 13:08 ` Stefan Berger
2011-01-26 13:15 ` Jan Kiszka
2011-01-26 13:31 ` Jan Kiszka
2011-01-26 13:52 ` Stefan Berger
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.