kvm-autotest: weird memory error during stepmaker test

kvm-autotest: weird memory error during stepmaker test

[Ryan Harper]

Wondering if anyone else using kvm-autotest stepmaker has ever seen this
error:

Traceback (most recent call last):
  File "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepmaker.py", line 146, in update
    self.set_image_from_file(self.screendump_filename)
  File "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor.py", line 499, in set_image_from_file
    self.set_image(w, h, data)
  File "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor.py", line 485, in set_image
    w, h, w*3))
MemoryError

The guest is still running, but stepmaker isn't recording any more so it's
boned at that point.  And of course, it's near the end of a guest install so
one has lost a decent amount of time...


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: kvm-autotest: weird memory error during stepmaker test

[Eduardo Habkost]

Excerpts from Ryan Harper's message of Qua Abr 01 12:55:58 -0300 2009:
> Wondering if anyone else using kvm-autotest stepmaker has ever seen this
> error:
> 
> Traceback (most recent call last):
>   File
> "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepmaker.
> py", line 146, in update
>     self.set_image_from_file(self.screendump_filename)
>   File
> "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor
> .py", line 499, in set_image_from_file
>     self.set_image(w, h, data)
>   File
> "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor
> .py", line 485, in set_image
>     w, h, w*3))
> MemoryError

I've seen this error twice today, while trying to create a step file to
install a Windows 2008 R2 64-bit guest (the Win2008-64 step file
available on the git repository doesn't work for me). This happened when
the guest was being rebooted by the windows installer. The contents of
the screen dump file are this:

$ cat /home/ehabkost/autotest/kvm-autotest/client/results/default/kvm_runtest_2.Win2008.64.install/debug/scrdump.ppm
P6
0 0
255
$ 

And the 0x0 pixmap really makes gdk panic:

>>> (w, h, data) =  ppm_utils.image_read_from_ppm_file('/home/ehabkost/autotest/kvm-autotest/client/results/default/kvm_runtest_2.Win2008.64.install/debug/scrdump.ppm')
>>> w,h,data
(0, 0, '')
>>> gtk.gdk.pixbuf_new_from_data(data, gtk.gdk.COLORSPACE_RGB, False, 8,  w, h, w*3)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
MemoryError
>>> 



> 
> The guest is still running, but stepmaker isn't recording any more so it's
> boned at that point.  And of course, it's near the end of a guest install so
> one has lost a decent amount of time...
> 
-- 
Eduardo

[PATCH] kvm-autotest: stepeditor: clear image if width, height, or data are invalid

[Eduardo Habkost]

This patch fixes the following issue:

Excerpts from Eduardo Habkost's message of Fri Apr 03 17:37:56 -0300 2009:
> Excerpts from Ryan Harper's message of Wed Apr 01 12:55:58 -0300 2009:
> > Wondering if anyone else using kvm-autotest stepmaker has ever seen this
> > error:
> > 
> > Traceback (most recent call last):
> >   File
> > "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepmaker.
> > py", line 146, in update
> >     self.set_image_from_file(self.screendump_filename)
> >   File
> > "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor
> > .py", line 499, in set_image_from_file
> >     self.set_image(w, h, data)
> >   File
> > "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor
> > .py", line 485, in set_image
> >     w, h, w*3))
> > MemoryError
> 
> I've seen this error twice today, while trying to create a step file to
> install a Windows 2008 R2 64-bit guest (the Win2008-64 step file
> available on the git repository doesn't work for me). This happened when
> the guest was being rebooted by the windows installer. The contents of
> the screen dump file are this:
> 
> $ cat
> /home/ehabkost/autotest/kvm-autotest/client/results/default/kvm_runtest_2.Win200
> 8.64.install/debug/scrdump.ppm
> P6
> 0 0
> 255
> $ 
> 
> And the 0x0 pixmap really makes gdk panic:
> 
> >>> (w, h, data) =  ppm_utils.image_read_from_ppm_file('/home/ehabkost/autotest/kvm-autotest/client/results/default/kvm_runtest_2.Win2008.64.install/debug/scrdump.ppm')
> >>> w,h,data
> (0, 0, '')
> >>> gtk.gdk.pixbuf_new_from_data(data, gtk.gdk.COLORSPACE_RGB, False, 8,  w, h, w*3)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> MemoryError
> >>> 


Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 client/tests/kvm_runtest_2/stepeditor.py |   14 +++++++++-----
 1 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/client/tests/kvm_runtest_2/stepeditor.py b/client/tests/kvm_runtest_2/stepeditor.py
index caaf47b..383834b 100755
--- a/client/tests/kvm_runtest_2/stepeditor.py
+++ b/client/tests/kvm_runtest_2/stepeditor.py
@@ -488,14 +488,18 @@ Utilities
         vscrollbar = self.scrolledwindow.get_vscrollbar()
         vscrollbar.set_range(0, h)
 
+    def clear_image(self):
+        self.image.clear()
+        self.image_width = 0
+        self.image_height = 0
+        self.image_data = ""
+        
     def set_image_from_file(self, filename):
         if not filename or not os.path.exists(filename):
-            self.image.clear()
-            self.image_width = 0
-            self.image_height = 0
-            self.image_data = ""
-            return
+            return self.clear_image()
         (w, h, data) = ppm_utils.image_read_from_ppm_file(filename)
+        if w <= 0 or h <= 0 or not data:
+            return self.clear_image()
         self.set_image(w, h, data)
 
     def get_step_lines(self, output_dir=None, current_step=None):

-- 
1.5.5.6
-- 
Eduardo

Re: [PATCH] kvm-autotest: stepeditor: clear image if width, height, or data are invalid

[Avi Kivity]

Eduardo Habkost wrote:
>>>>> SPACE_RGB, False, 8,  w, h, w*3)
>>>>>           
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in ?
>> MemoryError
>>     
>
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  client/tests/kvm_runtest_2/stepeditor.py |   14 +++++++++-----
>  1 files changed, 9 insertions(+), 5 deletions(-)
>
> diff --git a/client/tests/kvm_runtest_2/stepeditor.py b/client/tests/kvm_runtest_2/stepeditor.py
> index caaf47b..383834b 100755
> --- a/client/tests/kvm_runtest_2/stepeditor.py
> +++ b/client/tests/kvm_runtest_2/stepeditor.py
> @@ -488,14 +488,18 @@ Utilities
>          vscrollbar = self.scrolledwindow.get_vscrollbar()
>          vscrollbar.set_range(0, h)
>  
> +    def clear_image(self):
> +        self.image.clear()
> +        self.image_width = 0
> +        self.image_height = 0
> +        self.image_data = ""
> +        
>      def set_image_from_file(self, filename):
>          if not filename or not os.path.exists(filename):
> -            self.image.clear()
> -            self.image_width = 0
> -            self.image_height = 0
> -            self.image_data = ""
> -            return
> +            return self.clear_image()
>          (w, h, data) = ppm_utils.image_read_from_ppm_file(filename)
> +        if w <= 0 or h <= 0 or not data:
> +            return self.clear_image()
>          self.set_image(w, h, data)
>   

Instead of returning an empty image here, we should throw an exception.  
Qemu shouldn't write invalid image.

I've just sent a patch to qemu-devel which may help with this.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH] kvm-autotest: stepeditor: clear image if width, height, or data are invalid

[Eduardo Habkost]

On Sun, Apr 05, 2009 at 03:31:03PM +0300, Avi Kivity wrote:
> Eduardo Habkost wrote:
>>>>>> SPACE_RGB, False, 8,  w, h, w*3)
>>>>>>           
>>> Traceback (most recent call last):
>>>   File "<stdin>", line 1, in ?
>>> MemoryError
>>>     
>>
>>
<snip>
>>   
>
> Instead of returning an empty image here, we should throw an exception.   
> Qemu shouldn't write invalid image.

Throwing an exception is exactly what the current code does, and what I
want to prevent. stepeditor is a GUI application, there is no point in
aborting because the image is invalid during a small time window.

But fixing qemu is welcome, of course.

>
> I've just sent a patch to qemu-devel which may help with this.

-- 
Eduardo

Re: [PATCH] kvm-autotest: stepeditor: clear image if width, height, or data are invalid

[Uri Lublin]

Avi Kivity wrote:
> Eduardo Habkost wrote:
>>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>>  client/tests/kvm_runtest_2/stepeditor.py |   14 +++++++++-----
>>  1 files changed, 9 insertions(+), 5 deletions(-)
>>
>> diff --git a/client/tests/kvm_runtest_2/stepeditor.py 
>> b/client/tests/kvm_runtest_2/stepeditor.py
>> index caaf47b..383834b 100755
>> --- a/client/tests/kvm_runtest_2/stepeditor.py
>> +++ b/client/tests/kvm_runtest_2/stepeditor.py
>> @@ -488,14 +488,18 @@ Utilities
>>          vscrollbar = self.scrolledwindow.get_vscrollbar()
>>          vscrollbar.set_range(0, h)
>>  
>> +    def clear_image(self):
>> +        self.image.clear()
>> +        self.image_width = 0
>> +        self.image_height = 0
>> +        self.image_data = ""
>> +             def set_image_from_file(self, filename):
>>          if not filename or not os.path.exists(filename):
>> -            self.image.clear()
>> -            self.image_width = 0
>> -            self.image_height = 0
>> -            self.image_data = ""
>> -            return
>> +            return self.clear_image()
>>          (w, h, data) = ppm_utils.image_read_from_ppm_file(filename)
>> +        if w <= 0 or h <= 0 or not data:
>> +            return self.clear_image()
>>          self.set_image(w, h, data)
>>   
> 
> Instead of returning an empty image here, we should throw an exception.  
> Qemu shouldn't write invalid image.
> 
> I've just sent a patch to qemu-devel which may help with this.
> 

I'm going to accept this patch.

For step-maker and step-editor, I think we should let the user finish 
creating/editing the step-file (see Ryan's comment about wasting time).

For guest-installation test (kvm-guest-wizard), I agree that we should fail the 
test, and not ignore the error.

Thanks,
     Uri.

Re: kvm-autotest: weird memory error during stepmaker test

[Michael Goldish]

----- "Eduardo Habkost" <ehabkost@raisama.net> wrote:

> Excerpts from Ryan Harper's message of Qua Abr 01 12:55:58 -0300
> 2009:
> > Wondering if anyone else using kvm-autotest stepmaker has ever seen
> this
> > error:
> > 
> > Traceback (most recent call last):
> >   File
> >
> "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepmaker.
> > py", line 146, in update
> >     self.set_image_from_file(self.screendump_filename)
> >   File
> >
> "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor
> > .py", line 499, in set_image_from_file
> >     self.set_image(w, h, data)
> >   File
> >
> "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor
> > .py", line 485, in set_image
> >     w, h, w*3))
> > MemoryError
> 
> I've seen this error twice today, while trying to create a step file
> to
> install a Windows 2008 R2 64-bit guest (the Win2008-64 step file
> available on the git repository doesn't work for me). This happened
> when
> the guest was being rebooted by the windows installer. The contents
> of
> the screen dump file are this:
> 
> $ cat
> /home/ehabkost/autotest/kvm-autotest/client/results/default/kvm_runtest_2.Win2008.64.install/debug/scrdump.ppm
> P6
> 0 0
> 255
> $ 
> 
> And the 0x0 pixmap really makes gdk panic:
> 
> >>> (w, h, data) = 
> ppm_utils.image_read_from_ppm_file('/home/ehabkost/autotest/kvm-autotest/client/results/default/kvm_runtest_2.Win2008.64.install/debug/scrdump.ppm')
> >>> w,h,data
> (0, 0, '')
> >>> gtk.gdk.pixbuf_new_from_data(data, gtk.gdk.COLORSPACE_RGB, False,
> 8,  w, h, w*3)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> MemoryError
> >>> 

This is very useful information. I've seen qemu/kvm produce 0x0 screendumps before,
but it's never happened to me while working with stepmaker.

A reasonable solution would be to make sure a screendump is OK before feeding it
to gdk. I'll try to commit this ASAP so it doesn't bother people any more.

Thanks,
Michael

Re: kvm-autotest: weird memory error during stepmaker test

[Michael Goldish]

I've never seen this exception before (and I've used stepmaker quite a bit).
The last line in the traceback is a call to gtk.gdk.pixbuf_new_from_data, so it might be PyGTK's fault, but I can't be sure.

This is what the docs have to say:
exception MemoryError
Raised when an operation runs out of memory but the situation may still be rescued (by deleting some objects). The associated value is a string indicating what kind of (internal) operation ran out of memory. Note that because of the underlying memory management architecture (C’s malloc function), the interpreter may not always be able to completely recover from this situation; it nevertheless raises an exception so that a stack traceback can be printed, in case a run-away program was the cause.

It seems that in this case the exception had no associated value, so it'll be hard to tell what caused it.

In any case, if the guest is still alive and stepmaker dies for some reason, there are 2 things you can do:

1. Re-run stepmaker with a living VM:
Kill autotest (not with ctrl+c), and if the kvm-autotest postprocessor doesn't kill your VM (either because it wasn't told to with kill_vm=yes, or because autotest was killed before the postprocessor could run) -- the VM will stay alive and untouched. Then you can modify the config file so the preprocessor doesn't touch the VM or the image (e.g. force_create_image=no, restart_vm=no, or just make sure they're not set to 'yes' anywhere), change the name of the steps file to be written (stepmaker won't run if the file already exists), and re-run stepmaker. It will run with the same living VM and continue exactly where you left off. Then you just need to concatenate the two stepfiles with stepeditor, and maybe make a few small fixes.

Note:
  - If you press ctrl+c autotest will kill all subprocesses started in the current run, including your VM. You should kill autotest somehow without ctrl+c. "killall autotest" and "killall -9 autotest" worked for me.
  - If you run stepmaker by changing the 'type' of an install test to 'stepmaker', remember that this test inherits the parameters of the 'install' variant defined at the top of the file, which typically include force_create_image=yes, which will overwrite the disk image with a blank one, which is a bad thing if you want to re-run stepmaker with the same VM.

2. Restart the installation from an advanced point:
If the VM is already dead, and you can somehow resume the installation so that you don't have to start from scratch (maybe start from the first boot?), you can do that with stepmaker and then concatenate the two stepfiles and use stepeditor to remove duplicate steps.

Please let me know if you ever encounter that weird memory exception again.

Thanks,
Michael

----- Original Message -----
From: "Ryan Harper" <ryanh@us.ibm.com>
To: kvm@vger.kernel.org
Sent: Wednesday, April 1, 2009 6:55:58 PM (GMT+0200) Auto-Detected
Subject: kvm-autotest: weird memory error during stepmaker test

Wondering if anyone else using kvm-autotest stepmaker has ever seen this
error:

Traceback (most recent call last):
  File "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepmaker.py", line 146, in update
    self.set_image_from_file(self.screendump_filename)
  File "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor.py", line 499, in set_image_from_file
    self.set_image(w, h, data)
  File "/home/rharper/work/git/build/kvm-autotest/client/tests/kvm_runtest_2/stepeditor.py", line 485, in set_image
    w, h, w*3))
MemoryError

The guest is still running, but stepmaker isn't recording any more so it's
boned at that point.  And of course, it's near the end of a guest install so
one has lost a decent amount of time...


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html