[PATCH 0/6] PM extension to domU

[PATCH 0/6] PM extension to domU

[Tian, Kevin]

Hi, all,
	Current domU suspend/resume infrastructure doesn't satisfy 
physical PM activity like Xen S3, as I posted before:
http://lists.xensource.com/archives/html/xen-devel/2007-01/msg00647.html

	Basically this patch set adds two features:
		- Driver domain suspend/resume support
		- Light-weight suspend/resume, with all old domain
context kept

	For the driver domain, new added logic like process freeze is 
skipped for non-driver domain to avoid impact. One important note 
here is, driver domain suspend will only work correctly for the new 
added commands, where old domain context are still kept intact 
after resume. For save/restore and migration, there's no easy way 
to do when domain is destroyed and re-created. The reason is the 
machine address saved within private driver structures. Unless 
domainU itself takes IOMMU as the default DMA interface for all 
drivers, we may need to explicitly prevent user from 
save/restore/migration on driver domain later.

	For the new added commands "sleep/wake", I'm not sure whether 
it's preferred way. Since it's purpose is very simple by sending suspend

notification to domU, I choose to add instead of intervening complex 
logic of save/restore. Anyway, their names do explain different meaning 
indeed.

	Verified on Cset 13803.

Thanks,
Kevin

Re: [PATCH 0/6] PM extension to domU

[Keir Fraser]

On 6/2/07 08:12, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> For the new added commands "sleep/wake", I'm not sure whether
> it's preferred way. Since it's purpose is very simple by sending suspend
> 
> notification to domU, I choose to add instead of intervening complex
> logic of save/restore. Anyway, their names do explain different meaning
> indeed.
> 
> Verified on Cset 13803.

Have you considered destroying and recreating driver domains rather than
sleeping them? They are stateless after all, and should be quick (enough) to
boot.

We want to support driver domain restart anyway for driver-bug isolation.
The question mark there is whether we have the xenbus protocol sorted out
yet to allow frontends to automatically reconnect. You don't even have to
worry about that if you're prepared to do lightweight suspend/resume of all
non-driver domains across S3.

 -- Keir

RE: [PATCH 0/6] PM extension to domU

[Tian, Kevin]

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年2月7日 5:45
>On 6/2/07 08:12, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> For the new added commands "sleep/wake", I'm not sure whether
>> it's preferred way. Since it's purpose is very simple by sending suspend
>>
>> notification to domU, I choose to add instead of intervening complex
>> logic of save/restore. Anyway, their names do explain different
>meaning
>> indeed.
>>
>> Verified on Cset 13803.
>
>Have you considered destroying and recreating driver domains rather
>than
>sleeping them? They are stateless after all, and should be quick (enough)
>to
>boot.

Sorry that I don't quite understand you. For suspend-to-disk, maybe 
we can use that policy to recreate driver domains. But for 
suspend-to-ram, that process is still slow and we just need suspend 
domain itself. Also I'm not sure why it's stateless... How do you 
recover workloads after re-creation?

>
>We want to support driver domain restart anyway for driver-bug isolation.
>The question mark there is whether we have the xenbus protocol sorted
>out
>yet to allow frontends to automatically reconnect. You don't even have to
>worry about that if you're prepared to do lightweight suspend/resume of
>all
>non-driver domains across S3.
>
> -- Keir

Anyway driver domain restart should be another issue. BTW, I can 
destroy and re-create a driver domain manually now. 

So, could you be more specific which part you'd like to change in 
my patch sets? :-)

Thanks.
Kevin

Re: [PATCH 0/6] PM extension to domU

[Keir Fraser]

On 7/2/07 01:25, "Tian, Kevin" <kevin.tian@intel.com> wrote:

>> Have you considered destroying and recreating driver domains rather
>> than
>> sleeping them? They are stateless after all, and should be quick (enough)
>> to
>> boot.
> 
> Sorry that I don't quite understand you. For suspend-to-disk, maybe
> we can use that policy to recreate driver domains. But for
> suspend-to-ram, that process is still slow and we just need suspend
> domain itself. Also I'm not sure why it's stateless... How do you
> recover workloads after re-creation?

A proper driver domain would be little more than a device driver wrapped in
minios-style Xen-interfacing code. They should be quick to start, and
restart would lose no workload. I suppose there are people running device
drivers as part of a full guest however...

Is it necessary to sleep/wake non driver domains? Could we check in S3
support that would work for systems with all drivers in dom0, and then
consider this patchset as an extension?

 -- Keir

RE: [PATCH 0/6] PM extension to domU

[Tian, Kevin]

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007年2月7日 9:57
>> Sorry that I don't quite understand you. For suspend-to-disk, maybe
>> we can use that policy to recreate driver domains. But for
>> suspend-to-ram, that process is still slow and we just need suspend
>> domain itself. Also I'm not sure why it's stateless... How do you
>> recover workloads after re-creation?
>
>A proper driver domain would be little more than a device driver wrapped
>in
>minios-style Xen-interfacing code. They should be quick to start, and
>restart would lose no workload. I suppose there are people running
>device
>drivers as part of a full guest however...

So this is the point about definition about driver domains. If we 
consider driver domain as the part of I/O virtualization service 
provider like dom0, minios-style is perfectly right and quick. 
However we have to consider the cases where user just wants 
device passthrough for performance. Actually most requirements 
on the list before driver domain was back to xen3.0, are all related 
to a full guest which doesn't provide backend service to others.

Then actually we have two scenarios:
	- Minios-driver domains created automatically to balance I/O 
virtualization, as part of increasing overall xen performance
	- Full guest with direct device assignment just for performance, 
which is created by user manually

I think we need cover both cases.

>
>Is it necessary to sleep/wake non driver domains? Could we check in S3
>support that would work for systems with all drivers in dom0, and then
>consider this patchset as an extension?
>
> -- Keir

It's still necessary to sleep/wake non driver domains, or else simple 
pause/unpause out of domain's knowledge is not enough. For example, 
at least time_resume is necessary after a long sleep/wake period. 

If you prefer to check in basic S3 first, OK I can hold it after that. 
Currently I sent out this patch set because it's self-contained. Anyway, 
we're doing final cleanup and rebase for S3, and should be able to 
send out soon.

Thanks,
Kevin

Re: [PATCH 0/6] PM extension to domU

[Keir Fraser]

On 7/2/07 02:21, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> It's still necessary to sleep/wake non driver domains, or else simple
> pause/unpause out of domain's knowledge is not enough. For example,
> at least time_resume is necessary after a long sleep/wake period.
> 
> If you prefer to check in basic S3 first, OK I can hold it after that.
> Currently I sent out this patch set because it's self-contained. Anyway,
> we're doing final cleanup and rebase for S3, and should be able to
> send out soon.

Sounds good. With feature freeze for 3.0.5 now, all these patches are likely
to be held pending for the next few weeks at least anyway.

 -- Keir

RE: [PATCH 0/6] PM extension to domU

[Puthiyaparambil, Aravindh]

> So this is the point about definition about driver domains. If we
> consider driver domain as the part of I/O virtualization service
> provider like dom0, minios-style is perfectly right and quick.
> However we have to consider the cases where user just wants
> device passthrough for performance. Actually most requirements
> on the list before driver domain was back to xen3.0, are all related
> to a full guest which doesn't provide backend service to others.
> 
> Then actually we have two scenarios:
> 	- Minios-driver domains created automatically to balance I/O
> virtualization, as part of increasing overall xen performance
> 	- Full guest with direct device assignment just for performance,
> which is created by user manually

Kevin, Keir,

Are Mini-OS based driver domains possible today? Is there a group
working on this?

Thanks,
Aravindh

Re: [PATCH 0/6] PM extension to domU

[Keir Fraser]

On 7/2/07 05:01, "Puthiyaparambil, Aravindh"
<aravindh.puthiyaparambil@unisys.com> wrote:

>> Then actually we have two scenarios:
>> - Minios-driver domains created automatically to balance I/O
>> virtualization, as part of increasing overall xen performance
>> - Full guest with direct device assignment just for performance,
>> which is created by user manually
> 
> Kevin, Keir,
> 
> Are Mini-OS based driver domains possible today? Is there a group
> working on this?

Not really. The best you can do right now is a stripped-down Linux kernel.
If you don't boot user space (e.g., have a very small init) then that's
actually not bad.

 -- Keir