Software Suspend Functionality in 2.5

Software Suspend Functionality in 2.5

[Nigel Cunningham]

Hi Linus.

Pavel and I have been discussing what functionality should be in
software suspend, and I wanted to ask your opinion since you're the
final decision maker anyway.

Under 2.4, I've added a lot functionality which has recently been merged
by Florent (swsusp beta 18). This functionality brings us as close as I
can get to the point where what you have in memory when you initiate the
suspend is very close to what you have when you finish resuming.
Settings on the proc interface and the amount of swap available do mean 
the degree to which this is true varies - you can set a soft limit on
the image size, for example - but that's the gist of the changes. I
implemented all the changes because as a user, I wanted the system to be
as responsive as possible upon resume - I didn't want a ton of thrashing
and an unresponsive system while processes tried to get back whatever
had been discarded or forced to swap. Performance with the changes does
not seem to be degraded, even though pages are currently write
synchronously (batching of requests is in the pipeline).

Where the rubber hits the road, I guess, is that the process is a bit
more complicated than the one in the kernel at the moment:
- Rather than freeing memory until no more can be freed, memory is freed
until the image fits in available swap, will be able to be saved and
reloaded and meets the user's size limit. (The limit can be disabled by
setting it to zero. If we can't reduce the image size to the requested
amount (eg the user requests a 1MB image), we continue anyway - hence
the description of 'soft limit' above.
- Pages to be saved are put into two categories - those we definitely
won't need during suspend and those that might/will be needed (simple
algorithm). The two sets of pages are saved separately - those not
needed are saved directly to disk, those needed are later copied and
then the copy is saved (as per current algorithm).
- Since we're saving more pages, we need more pages for the pagedir
(index). Unlike now, we probably won't be able to allocate (say) a
hundred contiguous set of pages, so we need to be able to have a
scattered (discontiguous) pagedir.
- We also need to be careful to free buffers & swap cache generated
during the save/resume process, so as to not corrupt the image.

Naturally there are other changes, but perhaps the simplest comparison
is to say that the 2.5.62 suspend.c is 34549 bytes (suspend.o 25132),
whereas beta18 is 77945 bytes (suspend.o 53756 with all the debugging
code turned off).

So can we please get your thoughts? Would you be willing to accept these
additions when they're ready?

Regards,

Nigel

Re: Software Suspend Functionality in 2.5

[Suparna Bhattacharya]

Nigel,

When I noticed some of your discussions on swsusp mailing
list earlier, a question that crossed my mind was whether
you'd thought about the possibility of compression of data 
at the time of copy. 

Would that have been another way to helped achieve the 
objective you have in mind ? Do any issues come to mind ?

Regards
Suparna

On Wed, Feb 26, 2003 at 07:46:47AM +0000, Nigel Cunningham wrote:
> Hi Linus.
> 
> Pavel and I have been discussing what functionality should be in
> software suspend, and I wanted to ask your opinion since you're the
> final decision maker anyway.
> 
> Under 2.4, I've added a lot functionality which has recently been merged
> by Florent (swsusp beta 18). This functionality brings us as close as I
> can get to the point where what you have in memory when you initiate the
> suspend is very close to what you have when you finish resuming.
> Settings on the proc interface and the amount of swap available do mean 
> the degree to which this is true varies - you can set a soft limit on
> the image size, for example - but that's the gist of the changes. I
> implemented all the changes because as a user, I wanted the system to be
> as responsive as possible upon resume - I didn't want a ton of thrashing
> and an unresponsive system while processes tried to get back whatever
> had been discarded or forced to swap. Performance with the changes does
> not seem to be degraded, even though pages are currently write
> synchronously (batching of requests is in the pipeline).
> 
> Where the rubber hits the road, I guess, is that the process is a bit
> more complicated than the one in the kernel at the moment:
> - Rather than freeing memory until no more can be freed, memory is freed
> until the image fits in available swap, will be able to be saved and
> reloaded and meets the user's size limit. (The limit can be disabled by
> setting it to zero. If we can't reduce the image size to the requested
> amount (eg the user requests a 1MB image), we continue anyway - hence
> the description of 'soft limit' above.
> - Pages to be saved are put into two categories - those we definitely
> won't need during suspend and those that might/will be needed (simple
> algorithm). The two sets of pages are saved separately - those not
> needed are saved directly to disk, those needed are later copied and
> then the copy is saved (as per current algorithm).
> - Since we're saving more pages, we need more pages for the pagedir
> (index). Unlike now, we probably won't be able to allocate (say) a
> hundred contiguous set of pages, so we need to be able to have a
> scattered (discontiguous) pagedir.
> - We also need to be careful to free buffers & swap cache generated
> during the save/resume process, so as to not corrupt the image.
> 
> Naturally there are other changes, but perhaps the simplest comparison
> is to say that the 2.5.62 suspend.c is 34549 bytes (suspend.o 25132),
> whereas beta18 is 77945 bytes (suspend.o 53756 with all the debugging
> code turned off).
> 
> So can we please get your thoughts? Would you be willing to accept these
> additions when they're ready?
> 
> Regards,
> 
> Nigel
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India

Re: Software Suspend Functionality in 2.5

[Nigel Cunningham]

Hi.

I've looked at LKCD and seen that they have a provision for compressing
the pages which are written. Ideally, I'd like to see us sharing code
because I reckon there's a fair bit in common between the two projects.
For the moment, though, I haven't seriously considered implementing
compression. I've just been concentrating on getting things as stable as
possible under 2.4 (given that there's no driver model there), and
beginning to seek to port the changes to 2.5. Perhaps compression could
be added later, but I am worrying about the basics (getting the pages
saved in any form!) first.

On Fri, 2003-02-28 at 01:42, Suparna Bhattacharya wrote:
> Nigel,
> 
> When I noticed some of your discussions on swsusp mailing
> list earlier, a question that crossed my mind was whether
> you'd thought about the possibility of compression of data 
> at the time of copy. 
> 
> Would that have been another way to helped achieve the 
> objective you have in mind ? Do any issues come to mind ?
> 
> Regards
> Suparna
> 

Re: Software Suspend Functionality in 2.5

[Suparna Bhattacharya]

On Fri, Feb 28, 2003 at 07:16:31AM +1300, Nigel Cunningham wrote:
> Hi.
> 
> I've looked at LKCD and seen that they have a provision for compressing
> the pages which are written. Ideally, I'd like to see us sharing code
> because I reckon there's a fair bit in common between the two projects.
> For the moment, though, I haven't seriously considered implementing
> compression. I've just been concentrating on getting things as stable as
> possible under 2.4 (given that there's no driver model there), and
> beginning to seek to port the changes to 2.5. Perhaps compression could
> be added later, but I am worrying about the basics (getting the pages
> saved in any form!) first.

The thought was that with compression you can copy more pages 
into the same free memory space; meaning that you need to 
free up far less pages. If you are lucky enough, you might 
get to resume from very close to the state you suspended and 
hence avoid the initial sluggishness on resume.

But yes, you've rightly guessed that my question did stem from 
what we are doing on LKCD. People have asked me if we could 
reuse swsusp logic, and even I'd thought about that way back 
when swsusp started for 2.5. I remember talking to Pavel about 
it last year at OLS - hoping it could save some work for 
us :) 

The problem was that we have to deal with far more stringent 
conditions in a crash dump situation, and the very things that 
seemed like interesting ideas or neat twists in swsusp (like 
the trick of freeing-up enough pages to be able to an atomic 
suspend-to-ram, or maybe even the refridgerator concept), 
just couldn't work or apply in that situation. 

Aside of the fact that we can't swapout at that point of 
time, we do want a near-exact snapshot of memory. And we are 
ambitious enough to want to be able to snapshot full memory 
or as close as possible to that, if so required.

Since we've had to work on a solution that can be used 
for accurate non-disruptive dumps as well as crash dumps
(the latter using kexec), I was wondering whether it 
was worth exploring possibilities of commonality with 
swsusp down the line ... I know its not probably not 
something very immediate, but just an indication on 
whether we should keep applicability for swsusp (probably 
reuse and share ideas/code back and forth between the 
two efforts) in mind as we move forward. Because we 
have to support a more restrictive situation when it 
comes to dumping, it just may be usable by swsusp too 
if we can get it right.

Regards
Suparna

> 
> On Fri, 2003-02-28 at 01:42, Suparna Bhattacharya wrote:
> > Nigel,
> > 
> > When I noticed some of your discussions on swsusp mailing
> > list earlier, a question that crossed my mind was whether
> > you'd thought about the possibility of compression of data 
> > at the time of copy. 
> > 
> > Would that have been another way to helped achieve the 
> > objective you have in mind ? Do any issues come to mind ?
> > 
> > Regards
> > Suparna
> > 
> 
> 

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India

Re: Software Suspend Functionality in 2.5

[Pavel Machek]

Hi!

> Since we've had to work on a solution that can be used 
> for accurate non-disruptive dumps as well as crash dumps
> (the latter using kexec), I was wondering whether it 
> was worth exploring possibilities of commonality with 
> swsusp down the line ... I know its not probably not 
> something very immediate, but just an indication on 
> whether we should keep applicability for swsusp (probably 
> reuse and share ideas/code back and forth between the 
> two efforts) in mind as we move forward. Because we 
> have to support a more restrictive situation when it 
> comes to dumping, it just may be usable by swsusp too 
> if we can get it right.

Well, less code duplication is always welcome. But notice we need
*atomic* snapshots in swsusp, else we might corrupt data.
-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

Re: Software Suspend Functionality in 2.5

[Suparna Bhattacharya]

On Fri, Feb 28, 2003 at 02:05:48PM +0100, Pavel Machek wrote:
> Hi!
> 
> > Since we've had to work on a solution that can be used 
> > for accurate non-disruptive dumps as well as crash dumps
> > (the latter using kexec), I was wondering whether it 
> > was worth exploring possibilities of commonality with 
> > swsusp down the line ... I know its not probably not 
> > something very immediate, but just an indication on 
> > whether we should keep applicability for swsusp (probably 
> > reuse and share ideas/code back and forth between the 
> > two efforts) in mind as we move forward. Because we 
> > have to support a more restrictive situation when it 
> > comes to dumping, it just may be usable by swsusp too 
> > if we can get it right.
> 
> Well, less code duplication is always welcome. But notice we need
> *atomic* snapshots in swsusp, else we might corrupt data.

Atomic snapshots are what we'd like for dump too, since we desire 
accurate dumps (minimum drift), so its not a conflicting requirement. 
The difference is that while you could do i/o (e.g to flush pages 
to free up memory) before initiating an atomic snapshot, we can't.

Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India

Re: Software Suspend Functionality in 2.5

[Pavel Machek]

Hi!

> > > Since we've had to work on a solution that can be used 
> > > for accurate non-disruptive dumps as well as crash dumps
> > > (the latter using kexec), I was wondering whether it 
> > > was worth exploring possibilities of commonality with 
> > > swsusp down the line ... I know its not probably not 
> > > something very immediate, but just an indication on 
> > > whether we should keep applicability for swsusp (probably 
> > > reuse and share ideas/code back and forth between the 
> > > two efforts) in mind as we move forward. Because we 
> > > have to support a more restrictive situation when it 
> > > comes to dumping, it just may be usable by swsusp too 
> > > if we can get it right.
> > 
> > Well, less code duplication is always welcome. But notice we need
> > *atomic* snapshots in swsusp, else we might corrupt data.
> 
> Atomic snapshots are what we'd like for dump too, since we desire 
> accurate dumps (minimum drift), so its not a conflicting requirement. 
> The difference is that while you could do i/o (e.g to flush pages 
> to free up memory) before initiating an atomic snapshot, we can't.

OTOH "best-effort-atomic" is probably okay for you, while it is not
acceptable for swsusp. Hopefully the code is not going to get too
complicated by "must be atomic" and "must work with crashed system"
requirements...

							Pavel
-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

Re: Software Suspend Functionality in 2.5

[Pavel Machek]

Hi!

> > > Atomic snapshots are what we'd like for dump too, since we desire 
> > > accurate dumps (minimum drift), so its not a conflicting requirement. 
> > > The difference is that while you could do i/o (e.g to flush pages 
> > > to free up memory) before initiating an atomic snapshot, we can't.
> > 
> > OTOH "best-effort-atomic" is probably okay for you, while it is not
> > acceptable for swsusp. Hopefully the code is not going to get too
> > complicated by "must be atomic" and "must work with crashed system"
> > requirements...
> > 
> For the kind of atomicity you need there probably are two
> steps:
> 1) Quiesce the system - get to a point of consistency (when you
>    can take a resumable snapshot)
> 2) Perform an atomic copy / snapshot
> 
> Step (1) would be different for swsusp and crash dump (not
> intended to be common ). But for Step (2), do you think
> what you need/do is complicated by crashed system requirements ?

Well, I guess count_and_copy_data_pages() is easy to share, OTOH it is
really small piece of code. Also do you think you can free half of
memory in crashed system? Thats what swsusp currently does...

[I need really little about LKCD... But you are going to need modified
disk drivers etc, right? I'd like to get away without that in swsusp,
at least in 2.6.X.]

								Pavel
-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

Re: Software Suspend Functionality in 2.5

[Suparna Bhattacharya]

On Fri, Feb 28, 2003 at 02:44:06PM +0100, Pavel Machek wrote:
> > Atomic snapshots are what we'd like for dump too, since we desire 
> > accurate dumps (minimum drift), so its not a conflicting requirement. 
> > The difference is that while you could do i/o (e.g to flush pages 
> > to free up memory) before initiating an atomic snapshot, we can't.
> 
> OTOH "best-effort-atomic" is probably okay for you, while it is not
> acceptable for swsusp. Hopefully the code is not going to get too
> complicated by "must be atomic" and "must work with crashed system"
> requirements...
> 
For the kind of atomicity you need there probably are two
steps:
1) Quiesce the system - get to a point of consistency (when you
   can take a resumable snapshot)
2) Perform an atomic copy / snapshot

Step (1) would be different for swsusp and crash dump (not
intended to be common ). But for Step (2), do you think
what you need/do is complicated by crashed system requirements ?

Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India

Re: Software Suspend Functionality in 2.5

[Nigel Cunningham]

On Sat, 2003-03-01 at 04:17, Pavel Machek wrote:
> > For the kind of atomicity you need there probably are two
> > steps:
> > 1) Quiesce the system - get to a point of consistency (when you
> >    can take a resumable snapshot)
> > 2) Perform an atomic copy / snapshot
> > 
> > Step (1) would be different for swsusp and crash dump (not
> > intended to be common ). But for Step (2), do you think
> > what you need/do is complicated by crashed system requirements ?
> 
> Well, I guess count_and_copy_data_pages() is easy to share, OTOH it is
> really small piece of code. Also do you think you can free half of
> memory in crashed system? Thats what swsusp currently does...
> 
> [I need really little about LKCD... But you are going to need modified
> disk drivers etc, right? I'd like to get away without that in swsusp,
> at least in 2.6.X.]
> 

With the changes I've made, which I'm starting to merge with Pavel, I
think the two are a lot closer to each other.

With regard to quiescing the system, we need the same things stopped
that you need. We can of course use drivers_suspend when you can't, but
we could probably also use the SMP code you have.

I've got swsusp so that freeing memory is not necessary - the whole
image can be written to disk. There is still an option for the user to
aim for a smaller image (a soft limit can be set), and if there's not
enough swap available, that will also cause some memory to be freed, but
LKCD would run into that problem writing to swap too.

Regards,

Nigel

Re: Software Suspend Functionality in 2.5

[Suparna Bhattacharya]

On Sat, Mar 01, 2003 at 07:59:36AM +1300, Nigel Cunningham wrote:
> On Sat, 2003-03-01 at 04:17, Pavel Machek wrote:
> > > For the kind of atomicity you need there probably are two
> > > steps:
> > > 1) Quiesce the system - get to a point of consistency (when you
> > >    can take a resumable snapshot)
> > > 2) Perform an atomic copy / snapshot
> > > 
> > > Step (1) would be different for swsusp and crash dump (not
> > > intended to be common ). But for Step (2), do you think
> > > what you need/do is complicated by crashed system requirements ?
> > 
> > Well, I guess count_and_copy_data_pages() is easy to share, OTOH it is
> > really small piece of code. Also do you think you can free half of
> > memory in crashed system? Thats what swsusp currently does...
> > 
> > [I need really little about LKCD... But you are going to need modified
> > disk drivers etc, right? I'd like to get away without that in swsusp,
> > at least in 2.6.X.]
> > 
> 
> With the changes I've made, which I'm starting to merge with Pavel, I
> think the two are a lot closer to each other.

Yes, I've noticed that, this is why it was in the context of 
your changes that I brought up the question.

> 
> With regard to quiescing the system, we need the same things stopped
> that you need. We can of course use drivers_suspend when you can't, but
> we could probably also use the SMP code you have.
> 
> I've got swsusp so that freeing memory is not necessary - the whole
> image can be written to disk. There is still an option for the user to
> aim for a smaller image (a soft limit can be set), and if there's not
> enough swap available, that will also cause some memory to be freed, but

If you add to that the possibility of being able to save more 
in less space if you have compression, would it be useful ?

Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India

Re: Software Suspend Functionality in 2.5

[Nigel Cunningham]

Hi.

On Mon, 2003-03-03 at 17:28, Suparna Bhattacharya wrote:
> If you add to that the possibility of being able to save more 
> in less space if you have compression, would it be useful ?

I'm not sure that it would because we don't know how much compression
we're going to get ahead of time, so we don't know how many extra pages
we can save. The compression/decompression also takes extra time and
puts more drain on a potentially low battery.

I suppose it could be included as an option if it was going to already
be there for LKCD anyway, but I'm not sure it would be helpful for the
usual use of swsusp.

Regards,

Nigel

Re: Software Suspend Functionality in 2.5

[Suparna Bhattacharya]

On Mon, Mar 03, 2003 at 07:36:49PM +1300, Nigel Cunningham wrote:
> Hi.
> 
> On Mon, 2003-03-03 at 17:28, Suparna Bhattacharya wrote:
> > If you add to that the possibility of being able to save more 
> > in less space if you have compression, would it be useful ?
> 
> I'm not sure that it would because we don't know how much compression
> we're going to get ahead of time, so we don't know how many extra pages

The algorithm could be adjusted to deal with that, however ...

> we can save. The compression/decompression also takes extra time and
> puts more drain on a potentially low battery.

.. I didn't think about the battery drain - valid point !

Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India

Re: Software Suspend Functionality in 2.5

[Nigel Cunningham]

On Mon, 2003-03-03 at 19:54, Suparna Bhattacharya wrote:
> On Mon, Mar 03, 2003 at 07:36:49PM +1300, Nigel Cunningham wrote:
> > Hi.
> > 
> > On Mon, 2003-03-03 at 17:28, Suparna Bhattacharya wrote:
> > > If you add to that the possibility of being able to save more 
> > > in less space if you have compression, would it be useful ?
> > 
> > I'm not sure that it would because we don't know how much compression
> > we're going to get ahead of time, so we don't know how many extra pages
> 
> The algorithm could be adjusted to deal with that, however ...

Ah I see. You're thinking of compressing the pages as we eat memory? I
guess I need to look at LKCD more closely. I think it can wait until we
get the basics of the expanded functionality sorted.

> 
> > we can save. The compression/decompression also takes extra time and
> > puts more drain on a potentially low battery.
> 
> .. I didn't think about the battery drain - valid point !

Of course that can be overcome too. The user simply starts swsusp
earlier or turns off compression if necessary.

Thanks for the thought provoking :>

Regards,

Nigel

Re: Software Suspend Functionality in 2.5

[Pavel Machek]

Hi!

> > On Mon, 2003-03-03 at 17:28, Suparna Bhattacharya wrote:
> > > If you add to that the possibility of being able to save more 
> > > in less space if you have compression, would it be useful ?
> > 
> > I'm not sure that it would because we don't know how much compression
> > we're going to get ahead of time, so we don't know how many extra pages
> 
> The algorithm could be adjusted to deal with that, however ...
> 
> > we can save. The compression/decompression also takes extra time and
> > puts more drain on a potentially low battery.
> 
> .. I didn't think about the battery drain - valid point !

Actually I don't quiet think so. gzip compression is pretty cheap, and
if it makes you suspend faster and with less disk writes...

But I think it adds unneccessary complexity.

								Pavel
-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

Re: Software Suspend Functionality in 2.5

[Suparna Bhattacharya]

On Mon, Mar 03, 2003 at 01:25:26PM +0100, Pavel Machek wrote:
> Hi!
> 
> > > On Mon, 2003-03-03 at 17:28, Suparna Bhattacharya wrote:
> > > > If you add to that the possibility of being able to save more 
> > > > in less space if you have compression, would it be useful ?
> > > 
> > > I'm not sure that it would because we don't know how much compression
> > > we're going to get ahead of time, so we don't know how many extra pages
> > 
> > The algorithm could be adjusted to deal with that, however ...
> > 
> > > we can save. The compression/decompression also takes extra time and
> > > puts more drain on a potentially low battery.
> > 
> > .. I didn't think about the battery drain - valid point !
> 
> Actually I don't quiet think so. gzip compression is pretty cheap, and
> if it makes you suspend faster and with less disk writes...
> 
> But I think it adds unneccessary complexity.

If that's the only concern left, I guess its time for us to go 
back and complete what we have (we still have a few issues to
think over and concentrate on fixing), look at Nigel's patches 
more closely, and then come back and discuss the algo sometime 
later .. You could make up your mind about the actual complexity 
then (both for suspend and resume paths). 

>From what I can gather from this whole discussion, it seems 
worth at least a detailed look on our part. 

Thanks for the inputs and insights.

Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India