Should 'xm save' work, when using PCI pass through?

Should 'xm save' work, when using PCI pass through?

[Tom Rotenberg]

Hi,

I have a domain with PCI pass-through, and i would like to save it,
and restore it later (maybe even after rebooting the dom0). Is it
possible to do this using 'xm save' & 'xm restore', although my domain
has PCI pass through?

Tom

RE: Should 'xm save' work, when using PCI pass through?

[Ian Pratt]

> I have a domain with PCI pass-through, and i would like to save it,
> and restore it later (maybe even after rebooting the dom0). Is it
> possible to do this using 'xm save' & 'xm restore', although my domain
> has PCI pass through?

You'll need to make sure the physical devices are shutdown. The best way of doing this is to hot unplug them. You could also achieve this by putting the VM into S3 sleep before doing the save.  

Ian

Re: Should 'xm save' work, when using PCI pass through?

[Tom Rotenberg]

Well, i just tried sending the domain to S3, and then saving it's
state, but i got an error saying, that 'migration isn't allowed with
pass-through devices'.
How can i workaround this?

On Wed, Sep 23, 2009 at 1:45 PM, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>> I have a domain with PCI pass-through, and i would like to save it,
>> and restore it later (maybe even after rebooting the dom0). Is it
>> possible to do this using 'xm save' & 'xm restore', although my domain
>> has PCI pass through?
>
> You'll need to make sure the physical devices are shutdown. The best way of doing this is to hot unplug them. You could also achieve this by putting the VM into S3 sleep before doing the save.
>
> Ian
>

Re: Should 'xm save' work, when using PCI pass through?

[Keir Fraser]

I'm not sure xend can save/restore passthrough state. In which case your
only option is to hot-unplug the PCI devices from the guest first.

 -- Keir

On 23/09/2009 12:16, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:

> Well, i just tried sending the domain to S3, and then saving it's
> state, but i got an error saying, that 'migration isn't allowed with
> pass-through devices'.
> How can i workaround this?
> 
> On Wed, Sep 23, 2009 at 1:45 PM, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>>> I have a domain with PCI pass-through, and i would like to save it,
>>> and restore it later (maybe even after rebooting the dom0). Is it
>>> possible to do this using 'xm save' & 'xm restore', although my domain
>>> has PCI pass through?
>> 
>> You'll need to make sure the physical devices are shutdown. The best way of
>> doing this is to hot unplug them. You could also achieve this by putting the
>> VM into S3 sleep before doing the save.
>> 
>> Ian
>> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Should 'xm save' work, when using PCI pass through?

[Tom Rotenberg]

Why? if i send the domain to S3, and then save the PCI config space -
isn't it enough?

On Wed, Sep 23, 2009 at 3:49 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> I'm not sure xend can save/restore passthrough state. In which case your
> only option is to hot-unplug the PCI devices from the guest first.
>
>  -- Keir
>
> On 23/09/2009 12:16, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:
>
>> Well, i just tried sending the domain to S3, and then saving it's
>> state, but i got an error saying, that 'migration isn't allowed with
>> pass-through devices'.
>> How can i workaround this?
>>
>> On Wed, Sep 23, 2009 at 1:45 PM, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>>>> I have a domain with PCI pass-through, and i would like to save it,
>>>> and restore it later (maybe even after rebooting the dom0). Is it
>>>> possible to do this using 'xm save' & 'xm restore', although my domain
>>>> has PCI pass through?
>>>
>>> You'll need to make sure the physical devices are shutdown. The best way of
>>> doing this is to hot unplug them. You could also achieve this by putting the
>>> VM into S3 sleep before doing the save.
>>>
>>> Ian
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>

Re: Should 'xm save' work, when using PCI pass through?

[Keir Fraser]

Well, you could try removing the warning check from xend and see what
happens...

 -- Keir

On 23/09/2009 13:51, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:

> Why? if i send the domain to S3, and then save the PCI config space -
> isn't it enough?
> 
> On Wed, Sep 23, 2009 at 3:49 PM, Keir Fraser <keir.fraser@eu.citrix.com>
> wrote:
>> I'm not sure xend can save/restore passthrough state. In which case your
>> only option is to hot-unplug the PCI devices from the guest first.
>> 
>>  -- Keir
>> 
>> On 23/09/2009 12:16, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:
>> 
>>> Well, i just tried sending the domain to S3, and then saving it's
>>> state, but i got an error saying, that 'migration isn't allowed with
>>> pass-through devices'.
>>> How can i workaround this?
>>> 
>>> On Wed, Sep 23, 2009 at 1:45 PM, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>>>>> I have a domain with PCI pass-through, and i would like to save it,
>>>>> and restore it later (maybe even after rebooting the dom0). Is it
>>>>> possible to do this using 'xm save' & 'xm restore', although my domain
>>>>> has PCI pass through?
>>>> 
>>>> You'll need to make sure the physical devices are shutdown. The best way of
>>>> doing this is to hot unplug them. You could also achieve this by putting
>>>> the
>>>> VM into S3 sleep before doing the save.
>>>> 
>>>> Ian
>>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>> 
>> 
>> 

Re: Should 'xm save' work, when using PCI pass through?

[Tom Rotenberg]

I'll think i'll give it a try...

On Wed, Sep 23, 2009 at 4:02 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> Well, you could try removing the warning check from xend and see what
> happens...
>
>  -- Keir
>
> On 23/09/2009 13:51, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:
>
>> Why? if i send the domain to S3, and then save the PCI config space -
>> isn't it enough?
>>
>> On Wed, Sep 23, 2009 at 3:49 PM, Keir Fraser <keir.fraser@eu.citrix.com>
>> wrote:
>>> I'm not sure xend can save/restore passthrough state. In which case your
>>> only option is to hot-unplug the PCI devices from the guest first.
>>>
>>>  -- Keir
>>>
>>> On 23/09/2009 12:16, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:
>>>
>>>> Well, i just tried sending the domain to S3, and then saving it's
>>>> state, but i got an error saying, that 'migration isn't allowed with
>>>> pass-through devices'.
>>>> How can i workaround this?
>>>>
>>>> On Wed, Sep 23, 2009 at 1:45 PM, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>>>>>> I have a domain with PCI pass-through, and i would like to save it,
>>>>>> and restore it later (maybe even after rebooting the dom0). Is it
>>>>>> possible to do this using 'xm save' & 'xm restore', although my domain
>>>>>> has PCI pass through?
>>>>>
>>>>> You'll need to make sure the physical devices are shutdown. The best way of
>>>>> doing this is to hot unplug them. You could also achieve this by putting
>>>>> the
>>>>> VM into S3 sleep before doing the save.
>>>>>
>>>>> Ian
>>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>>
>
>
>

Re: Should 'xm save' work, when using PCI pass through?

[Tom Rotenberg]

Just tried it... it doesn't work. It looks like, because we have
pass-through devices, the restore is trying to send signals to qemu,
while the qemu instance is None...
The errors i see in the xend.log:

"
...
[2009-09-23 11:43:44 4345] DEBUG (XendDomainInfo:1120)
XendDomainInfo.destroyDevice: deviceClass = console, device =
console/0
[2009-09-23 11:43:44 4345] DEBUG (XendDomainInfo:2196) Removing pci/0
[2009-09-23 11:43:44 4345] DEBUG (XendDomainInfo:1120)
XendDomainInfo.destroyDevice: deviceClass = pci, device = pci/0
[2009-09-23 11:43:44 4345] DEBUG (pciif:418) XendDomainInfo.handleAerStateWatch
[2009-09-23 11:43:44 4345] DEBUG (pciif:537) pci: unregister aer watch
[2009-09-23 11:43:44 4345] ERROR (XendDomain:1155) Restore failed
Traceback (most recent call last):
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomain.py", line
1147, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused,
relocating=relocating)
  File "usr/lib/python2.5/site-packages/xen/xend/XendCheckpoint.py",
line 213, in restore
    dominfo = xd.restore_(vmconfig)
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomain.py", line
509, in restore_
    dominfo = XendDomainInfo.restore(config)
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py",
line 234, in restore
    vm.resume()
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py",
line 501, in resume
    self._createDevices()
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py",
line 2170, in _createDevices
    self.pci_device_configure_boot()
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py",
line 594, in pci_device_configure_boot
    self.pci_device_configure(dev_sxp)
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py",
line 830, in pci_device_configure
    vdevfn = self.hvm_pci_device_insert(dev_config)
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py",
line 722, in hvm_pci_device_insert
    return self.hvm_pci_device_insert_dev(new_dev)
  File "usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py",
line 736, in hvm_pci_device_insert_dev
    self.image.signalDeviceModel('pci-ins', 'pci-inserted', bdf_str)
AttributeError: 'NoneType' object has no attribute 'signalDeviceModel'
"

Shouldn't we develop basic save/restore functionality for domains with
pass-through devices (without migration to a different machine)?

Tom

On Wed, Sep 23, 2009 at 4:02 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> Well, you could try removing the warning check from xend and see what
> happens...
>
>  -- Keir
>
> On 23/09/2009 13:51, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:
>
>> Why? if i send the domain to S3, and then save the PCI config space -
>> isn't it enough?
>>
>> On Wed, Sep 23, 2009 at 3:49 PM, Keir Fraser <keir.fraser@eu.citrix.com>
>> wrote:
>>> I'm not sure xend can save/restore passthrough state. In which case your
>>> only option is to hot-unplug the PCI devices from the guest first.
>>>
>>>  -- Keir
>>>
>>> On 23/09/2009 12:16, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:
>>>
>>>> Well, i just tried sending the domain to S3, and then saving it's
>>>> state, but i got an error saying, that 'migration isn't allowed with
>>>> pass-through devices'.
>>>> How can i workaround this?
>>>>
>>>> On Wed, Sep 23, 2009 at 1:45 PM, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
>>>>>> I have a domain with PCI pass-through, and i would like to save it,
>>>>>> and restore it later (maybe even after rebooting the dom0). Is it
>>>>>> possible to do this using 'xm save' & 'xm restore', although my domain
>>>>>> has PCI pass through?
>>>>>
>>>>> You'll need to make sure the physical devices are shutdown. The best way of
>>>>> doing this is to hot unplug them. You could also achieve this by putting
>>>>> the
>>>>> VM into S3 sleep before doing the save.
>>>>>
>>>>> Ian
>>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>>
>
>
>

Re: Should 'xm save' work, when using PCI pass through?

[Keir Fraser]

On 23/09/2009 16:35, "Tom Rotenberg" <tom.rotenberg@gmail.com> wrote:

> Shouldn't we develop basic save/restore functionality for domains with
> pass-through devices (without migration to a different machine)?

Extending xend in this way would be a great project for someone to pick up.
In the meantime you could try the XCI project
http://xenbits.xensource.com/XCI. Its toolstack should support this already.

 -- Keir