Frame buffer corruptions with KVM >= 2.6.36

Frame buffer corruptions with KVM >= 2.6.36

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

Hi,

I'm seeing quite frequent corruptions of the VESA frame buffer with
Linux guests (vga=0x317) that are starting with KVM kernel modules of
upcoming 2.6.36 (I'm currently running -rc7). Effects disappears when
downgrading to kvm-kmod-2.6.35.6. Will see if I can bisect later, but
maybe someone already has an idea or wants to reproduce (just run
something like "find /" on one text console and witch to another one -
text fragments will remain on the screen on every few switches).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Frame buffer corruptions with KVM >= 2.6.36

[Avi Kivity]

  On 10/14/2010 09:27 AM, Jan Kiszka wrote:
> Hi,
>
> I'm seeing quite frequent corruptions of the VESA frame buffer with
> Linux guests (vga=0x317) that are starting with KVM kernel modules of
> upcoming 2.6.36 (I'm currently running -rc7). Effects disappears when
> downgrading to kvm-kmod-2.6.35.6. Will see if I can bisect later, but
> maybe someone already has an idea or wants to reproduce (just run
> something like "find /" on one text console and witch to another one -
> text fragments will remain on the screen on every few switches).
>

Reproduces on kvm.git.  I wonder what's going on.

Looks like vesafb uses the bios to switch the display start, so I expect 
a problem in qemu reacting to this.

-- 
error compiling committee.c: too many arguments to function

Re: Frame buffer corruptions with KVM >= 2.6.36

[Jan Kiszka]

Am 14.10.2010 09:27, Jan Kiszka wrote:
> Hi,
> 
> I'm seeing quite frequent corruptions of the VESA frame buffer with
> Linux guests (vga=0x317) that are starting with KVM kernel modules of
> upcoming 2.6.36 (I'm currently running -rc7). Effects disappears when
> downgrading to kvm-kmod-2.6.35.6. Will see if I can bisect later, but
> maybe someone already has an idea or wants to reproduce (just run
> something like "find /" on one text console and witch to another one -
> text fragments will remain on the screen on every few switches).

Commit d25f31f488e5f7597c17a3ac7d82074de8138e3b in kvm.git ("KVM: x86:
avoid unnecessary bitmap allocation when memslot is clean") is at least
magnifying the issue. With this patch applied, I can easily trigger
display corruptions when switching between VGA consoles while one of
them is undergoing heavy updates.

However, I once saw a much smaller inconsistency during my tests even
with a previous revision. Maybe there is a fundamental issue in when and
how the coalesced backlog is replayed, and this commit just makes the
corruptions more likely. This may even be a QEMU issue in the cirrus/vga
model (both qemu-kvm and upstream show the effect).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Frame buffer corruptions with KVM >= 2.6.36

[Avi Kivity]

  On 10/14/2010 02:04 PM, Avi Kivity wrote:
>  On 10/14/2010 09:27 AM, Jan Kiszka wrote:
>> Hi,
>>
>> I'm seeing quite frequent corruptions of the VESA frame buffer with
>> Linux guests (vga=0x317) that are starting with KVM kernel modules of
>> upcoming 2.6.36 (I'm currently running -rc7). Effects disappears when
>> downgrading to kvm-kmod-2.6.35.6. Will see if I can bisect later, but
>> maybe someone already has an idea or wants to reproduce (just run
>> something like "find /" on one text console and witch to another one -
>> text fragments will remain on the screen on every few switches).
>>
>
> Reproduces on kvm.git.  I wonder what's going on.
>
> Looks like vesafb uses the bios to switch the display start, so I 
> expect a problem in qemu reacting to this.
>

Hm, you said it is a kernel regression.  Maybe it's an issue with dirty 
bit tracking.

-- 
error compiling committee.c: too many arguments to function

Re: Frame buffer corruptions with KVM >= 2.6.36

[Jan Kiszka]

Am 14.10.2010 14:11, Avi Kivity wrote:
>  On 10/14/2010 02:04 PM, Avi Kivity wrote:
>>  On 10/14/2010 09:27 AM, Jan Kiszka wrote:
>>> Hi,
>>>
>>> I'm seeing quite frequent corruptions of the VESA frame buffer with
>>> Linux guests (vga=0x317) that are starting with KVM kernel modules of
>>> upcoming 2.6.36 (I'm currently running -rc7). Effects disappears when
>>> downgrading to kvm-kmod-2.6.35.6. Will see if I can bisect later, but
>>> maybe someone already has an idea or wants to reproduce (just run
>>> something like "find /" on one text console and witch to another one -
>>> text fragments will remain on the screen on every few switches).
>>>
>>
>> Reproduces on kvm.git.  I wonder what's going on.
>>
>> Looks like vesafb uses the bios to switch the display start, so I
>> expect a problem in qemu reacting to this.
>>
> 
> Hm, you said it is a kernel regression.  Maybe it's an issue with dirty
> bit tracking.
> 

Ah, cross-posting...

It need not be a kernel thing, see my other mail.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Frame buffer corruptions with KVM >= 2.6.36

[Avi Kivity]

  On 10/14/2010 02:10 PM, Jan Kiszka wrote:
> Am 14.10.2010 09:27, Jan Kiszka wrote:
> >  Hi,
> >
> >  I'm seeing quite frequent corruptions of the VESA frame buffer with
> >  Linux guests (vga=0x317) that are starting with KVM kernel modules of
> >  upcoming 2.6.36 (I'm currently running -rc7). Effects disappears when
> >  downgrading to kvm-kmod-2.6.35.6. Will see if I can bisect later, but
> >  maybe someone already has an idea or wants to reproduce (just run
> >  something like "find /" on one text console and witch to another one -
> >  text fragments will remain on the screen on every few switches).
>
> Commit d25f31f488e5f7597c17a3ac7d82074de8138e3b in kvm.git ("KVM: x86:
> avoid unnecessary bitmap allocation when memslot is clean") is at least
> magnifying the issue. With this patch applied, I can easily trigger
> display corruptions when switching between VGA consoles while one of
> them is undergoing heavy updates.
>
> However, I once saw a much smaller inconsistency during my tests even
> with a previous revision. Maybe there is a fundamental issue in when and
> how the coalesced backlog is replayed,

I didn't see any mmio writes to the framebuffer, so I don't think 
coalescing plays a part here.

> and this commit just makes the
> corruptions more likely. This may even be a QEMU issue in the cirrus/vga
> model (both qemu-kvm and upstream show the effect).
>

What about -no-kvm?

-- 
error compiling committee.c: too many arguments to function

Re: Frame buffer corruptions with KVM >= 2.6.36

[Avi Kivity]

  On 10/14/2010 02:13 PM, Avi Kivity wrote:
>
>> and this commit just makes the
>> corruptions more likely. This may even be a QEMU issue in the cirrus/vga
>> model (both qemu-kvm and upstream show the effect).
>>
>
> What about -no-kvm?
>

Doesn't happen there.

My guess is the initial dirty log after the switch does not contain all 
1s so memory isn't updated.  Probably need to force a full redraw in 
that case.

-- 
error compiling committee.c: too many arguments to function

Re: Frame buffer corruptions with KVM >= 2.6.36

[Jan Kiszka]

Am 14.10.2010 14:13, Avi Kivity wrote:
>   On 10/14/2010 02:10 PM, Jan Kiszka wrote:
>> Am 14.10.2010 09:27, Jan Kiszka wrote:
>>>  Hi,
>>>
>>>  I'm seeing quite frequent corruptions of the VESA frame buffer with
>>>  Linux guests (vga=0x317) that are starting with KVM kernel modules of
>>>  upcoming 2.6.36 (I'm currently running -rc7). Effects disappears when
>>>  downgrading to kvm-kmod-2.6.35.6. Will see if I can bisect later, but
>>>  maybe someone already has an idea or wants to reproduce (just run
>>>  something like "find /" on one text console and witch to another one -
>>>  text fragments will remain on the screen on every few switches).
>>
>> Commit d25f31f488e5f7597c17a3ac7d82074de8138e3b in kvm.git ("KVM: x86:
>> avoid unnecessary bitmap allocation when memslot is clean") is at least
>> magnifying the issue. With this patch applied, I can easily trigger
>> display corruptions when switching between VGA consoles while one of
>> them is undergoing heavy updates.
>>
>> However, I once saw a much smaller inconsistency during my tests even
>> with a previous revision. Maybe there is a fundamental issue in when and
>> how the coalesced backlog is replayed,
> 
> I didn't see any mmio writes to the framebuffer, so I don't think 
> coalescing plays a part here.
> 
>> and this commit just makes the
>> corruptions more likely. This may even be a QEMU issue in the cirrus/vga
>> model (both qemu-kvm and upstream show the effect).
>>
> 
> What about -no-kvm?

Just booted it (took ages), and the result was actually a completely
black screen. Kind of persistent corruption. This really looks like a
qemu issue now, maybe even a regression as I don't remember running into
such effects a while back.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Frame buffer corruptions with KVM >= 2.6.36

[Avi Kivity]

  On 10/14/2010 02:36 PM, Jan Kiszka wrote:
> >
> >>  and this commit just makes the
> >>  corruptions more likely. This may even be a QEMU issue in the cirrus/vga
> >>  model (both qemu-kvm and upstream show the effect).
> >>
> >
> >  What about -no-kvm?
>
> Just booted it (took ages), and the result was actually a completely
> black screen. Kind of persistent corruption. This really looks like a
> qemu issue now, maybe even a regression as I don't remember running into
> such effects a while back.

Worked fine for me (though yes it was slow - did tcg regress?).

-- 
error compiling committee.c: too many arguments to function

Re: Frame buffer corruptions with KVM >= 2.6.36

[Takuya Yoshikawa]

(2010/10/14 21:38), Avi Kivity wrote:
> On 10/14/2010 02:36 PM, Jan Kiszka wrote:
>> >
>> >> and this commit just makes the
>> >> corruptions more likely. This may even be a QEMU issue in the cirrus/vga
>> >> model (both qemu-kvm and upstream show the effect).
>> >>
>> >
>> > What about -no-kvm?
>>
>> Just booted it (took ages), and the result was actually a completely
>> black screen. Kind of persistent corruption. This really looks like a
>> qemu issue now, maybe even a regression as I don't remember running into
>> such effects a while back.
>
> Worked fine for me (though yes it was slow - did tcg regress?).
>

I reread my commit but could not find any reason to make this corruption
in kernel side.

But at least, I want to make it clear whether my commit was just a magnifier
of this problem, I know that magnifier itself is bad, or not.

Though I'm now trying some debugging but have not got a way to show
kvm_vm_ioctl_get_dirty_log() is 100% correct yet.


If you have any idea please tell me!

   Takuya

Re: Frame buffer corruptions with KVM >= 2.6.36

[Takuya Yoshikawa]

(2010/10/15 15:41), Takuya Yoshikawa wrote:
> (2010/10/14 21:38), Avi Kivity wrote:
>> On 10/14/2010 02:36 PM, Jan Kiszka wrote:
>>> >
>>> >> and this commit just makes the
>>> >> corruptions more likely. This may even be a QEMU issue in the cirrus/vga
>>> >> model (both qemu-kvm and upstream show the effect).
>>> >>
>>> >
>>> > What about -no-kvm?
>>>
>>> Just booted it (took ages), and the result was actually a completely
>>> black screen. Kind of persistent corruption. This really looks like a
>>> qemu issue now, maybe even a regression as I don't remember running into
>>> such effects a while back.
>>
>> Worked fine for me (though yes it was slow - did tcg regress?).
>>
>
> I reread my commit but could not find any reason to make this corruption
> in kernel side.
>
> But at least, I want to make it clear whether my commit was just a magnifier
> of this problem, I know that magnifier itself is bad, or not.
>
> Though I'm now trying some debugging but have not got a way to show
> kvm_vm_ioctl_get_dirty_log() is 100% correct yet.
>

I tried some tests and found another issue.

   I opened a terminal in one workspace and did "find /" on it. Then I switched
   to another workspace. When I switched back to the first one, the display of
   "find /" result was frozen.

I also replaced kvm_vm_ioctl_get_dirty_log() with the version before my patch was
applied and got the same result.

I don't know this is related to your report, but something seems wrong.

Thanks,
   Takuya

Re: Frame buffer corruptions with KVM >= 2.6.36

[Jan Kiszka]

Am 18.10.2010 14:14, Takuya Yoshikawa wrote:
> (2010/10/15 15:41), Takuya Yoshikawa wrote:
>> (2010/10/14 21:38), Avi Kivity wrote:
>>> On 10/14/2010 02:36 PM, Jan Kiszka wrote:
>>>>>
>>>>>> and this commit just makes the
>>>>>> corruptions more likely. This may even be a QEMU issue in the cirrus/vga
>>>>>> model (both qemu-kvm and upstream show the effect).
>>>>>>
>>>>>
>>>>> What about -no-kvm?
>>>>
>>>> Just booted it (took ages), and the result was actually a completely
>>>> black screen. Kind of persistent corruption. This really looks like a
>>>> qemu issue now, maybe even a regression as I don't remember running into
>>>> such effects a while back.
>>>
>>> Worked fine for me (though yes it was slow - did tcg regress?).
>>>
>>
>> I reread my commit but could not find any reason to make this corruption
>> in kernel side.
>>
>> But at least, I want to make it clear whether my commit was just a magnifier
>> of this problem, I know that magnifier itself is bad, or not.
>>
>> Though I'm now trying some debugging but have not got a way to show
>> kvm_vm_ioctl_get_dirty_log() is 100% correct yet.
>>
> 
> I tried some tests and found another issue.
> 
>    I opened a terminal in one workspace and did "find /" on it. Then I switched
>    to another workspace. When I switched back to the first one, the display of
>    "find /" result was frozen.
> 
> I also replaced kvm_vm_ioctl_get_dirty_log() with the version before my patch was
> applied and got the same result.
> 
> I don't know this is related to your report, but something seems wrong.

Besides corruptions, I'm getting those freezes from time to time as well
(fairly annoying now). Once that happened, (SDL) display updates can
only be achieved by pressing CTRL-ATL-1 (ie. switching qemu consoles).

Thanks for looking into this. I also think your patch is not guilty. But
we still need to understand the real issue.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux