linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re:  Re:  Swap Compression
@ 2003-04-25 22:32 rmoser
  2003-04-28 21:35 ` Timothy Miller
  0 siblings, 1 reply; 21+ messages in thread
From: rmoser @ 2003-04-25 22:32 UTC (permalink / raw)
  To: linux-kernel

Yeah you did but I'm going into a bit more detail, and with a very tight algorithm.  Heck the algo was originally designed based on another compression algorithm, but for a 6502 packer.  I aimed at speed, simplicity, and minimal RAM usage (hint:  it used 4k for the code AND the compressed data on a 6502, 336 bytes for code, and if I turn it into just a straight packer I can go under 200 bytes on the 6502).

Honestly, I just never looked.  I look in my kernel.  But still, the stuff I defined about swapon options, swap-on-ram, and how the compression works (yes, compressed without headers) is all the detail you need about it to go do it AFAIK.  Preplanning should be done there--done meaning workable, not "the absolute best."

--Bluefox Icy

---- ORIGINAL MESSAGE ----
List:     linux-kernel
Subject:  Re: Swap Compression
From:     John Bradford <john () grabjohn ! com>
Date:     2003-04-25 21:17:11
[Download message RAW]

> Sorry if this is HTML mailed.  I don't know how to control those settings

HTML mail is automatically filtered from LKML.

> COMPRESSED SWAP

We discussed this on the list quite recently, have a look at:

http://marc.theaimsgroup.com/?l=linux-kernel&m=105018674018129&w=2

and:

http://linuxcompressed.sourceforge.net/

John.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re:  Re:  Swap Compression
@ 2003-04-25 22:48 rmoser
  2003-04-26  9:17 ` Jörn Engel
  0 siblings, 1 reply; 21+ messages in thread
From: rmoser @ 2003-04-25 22:48 UTC (permalink / raw)
  To: linux-kernel

Yeah, I had to mail it 3 times.  Lst time I figured it out.

As for the performance hit, the original idea of that very tiny format was
to pack 6502 programs into 4k of code.  The expansion phase is very
tight and very efficient, and on a ... anything... it will provide no problem.
The swap-on-ram as long as it's not like 200 MB uncompressed SOR and
1 MB RAM will I think work great in the decompression phase.

Compression will take a little overhead.  I think if you use a boyer-moore
fast string search algo for binary strings (yes you can do this), you can
quickly compress the data.  It may be like.. just a guess... 10-30 times
more overhead than the decompression phase.  So use it on at least a
10-30 mhz processor.  If I ever write the code, it won't be kernel; just the
compression/decompression program (userspace).  Take the code and
stuff it into the kernel if I do.  I'll at the point of the algo coming in to
existence make another estimate.

The real power in this is Swap on RAM, but setting that as having precidence
over swap on disk (normal swaps) would decrease disk swap usage by
supplying more RAM in RAM.  And of course swapping RAM to RAM is
a lot faster.  I'm looking at this for PDA's but yes I will be running this on
my desktop the day we see it.

Well, I could work on the compression code, mebbe I can put the tarball up
here.  If I do I'd expect someone to add the code to swap to work with it--in
kernel 2.4 at the very least (port it to dev kernel later!).  As a separate module.
We don't want code that could be mean in the real swap driver. :)

--Bluefox Icy

---- ORIGINAL MESSAGE ----

List:     linux-kernel
Subject:  Re: Swap Compression
From:     =?iso-8859-1?Q?J=F6rn?= Engel <joern () wohnheim ! fh-wedel ! de>
Date:     2003-04-25 21:14:05
[Download message RAW]

On Fri, 25 April 2003 16:48:15 -0400, rmoser wrote:
> 
> Sorry if this is HTML mailed.  I don't know how to control those settings

It is not, but if you could limit lines to <80 characters, that would
be nice.

> COMPRESSED SWAP
> 
> This is mainly for things like linux on iPaq (handhelds.org) and
> people who like to play (me :), but how about compressed swap and
> RAM as swap?  To be plausable, we need a very fast compression
> algorithm.  I'd say use the following back pointer algorithm (this
> is headerless and I coded a decompressor in 6502 assembly in about
> 315 bytes) and 100k block sizes (compress streams of data until they
> are 100k in size, then stop.  Include the cap at the end in the
> 100k).
> 
> [...]
> 
> CONCLUSION
> 
> I think compressed swap and swap-on-ram with compression would be a
> great idea, especially for embedded systems.  High-performance
> systems that can handle the compression/decompression without
> blinking would especially benefit, as the swap-on-ram feature would
> give an almost seamless RAM increase.  Low-performance systems would
> take a performance hit, but embedded devices would still benefit
> from the swap-on-ram with compression RAM boost, considering they
> can probably handle the algorithm.  I am looking forward to seeing
> this implimented in 2.4 and 2.5/2.6 if it is adopted.

Nice idea. This might even benefit normal pc style boxes, if the
performance loss through compression is overcompensated by io gains
(less data transferred).

The tiny problem I see is that most people here have tons of whacky
ideas themselves but lack the time to actually implement them. If you
really want to get it done, do it yourself. It doesn't have to be
perfect, if it works in principle and appears to be promising, you
will likely receive enough help to finish it. But you have to get
there first.

At least, that is how it usually works. Feel free to prove me wrong.

Jörn

-- 
When in doubt, use brute force.
-- Ken Thompson
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: Swap Compression
@ 2003-04-29 20:07 Timothy Miller
  2003-04-29 20:40 ` rmoser
  0 siblings, 1 reply; 21+ messages in thread
From: Timothy Miller @ 2003-04-29 20:07 UTC (permalink / raw)
  To: Linux Kernel Mailing List


Con Kolivas wrote:

>On Tue, 29 Apr 2003 07:35, Timothy Miller wrote:
>  
>
>>    
>>
>The work that Rodrigo De Castro did on compressed caching 
>(linuxcompressed.sf.net) included a minilzo algorithm which I used by default 
>in the -ck patch addon as it performed the best for all the reasons you 
>mention. Why not look at that lzo code for adoption.
>
>  
>
Some time before rmoser started HIS discussion on compressed swap, I 
started a discussion on the same topic.  Out of that discussion came 
mention of an existing project which did EXACTLY what I was proposing. 
 That made me happy.  For some reason rmoser wants to start from 
scratch.  He can do that if he wants.  I had only hoped that my earlier 
mention of it would spark interest in taking the existing implementation 
and adding it to the mainline kernel, after some incompatbilities were 
dealt with.  Unfortunately, I don't have the time to engage directly in 
that particular aspect the kernel development.

>  
>


^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: Swap Compression
@ 2003-05-08  3:17 Perez-Gonzalez, Inaky
  2003-05-08  8:07 ` Jörn Engel
  0 siblings, 1 reply; 21+ messages in thread
From: Perez-Gonzalez, Inaky @ 2003-05-08  3:17 UTC (permalink / raw)
  To: 'jw schultz', 'linux-kernel@vger.kernel.org'



> From: jw schultz [mailto:jw@pegasys.ws]
>
> While we're having thoughts, this thread keeps me thinking
> it would make sense to have a block device driver that would
> be assigned unused memory.
> 
> I don't mean memory on video cards etc.  I'm thinking of the
> 10% of RAM unused when 1GB systems are booted with MEM=900M
> because they run faster with HIGHMEM turned off.
> 
> The primary use for this "device" would be high priority swap.
> Even with whatever overhead it takes to access it should be
> orders of magnitude faster than any spinning media.

This reminds me of some howto I saw somewhere of someway to
use the MTD drivers to access the unused video RAM and turn
it into swap (maybe with blkmtd?) ... probably it can be done
with that too.

I'd really love it ... I don't know if I can blame it on highmem
or not, but since I enabled it, my system 'feels' slower.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: Swap Compression
@ 2003-05-09  3:21 Perez-Gonzalez, Inaky
  0 siblings, 0 replies; 21+ messages in thread
From: Perez-Gonzalez, Inaky @ 2003-05-09  3:21 UTC (permalink / raw)
  To: 'Jörn Engel', Perez-Gonzalez, Inaky
  Cc: 'jw schultz', 'linux-kernel@vger.kernel.org'

> From: Jörn Engel [mailto:joern@wohnheim.fh-wedel.de]
>
> On Wed, 7 May 2003 20:17:32 -0700, Perez-Gonzalez, Inaky wrote:
> > This reminds me of some howto I saw somewhere of someway to
> > use the MTD drivers to access the unused video RAM and turn
> > it into swap (maybe with blkmtd?) ... probably it can be done
> > with that too.
> 
> Jupp, if you know the physical address range of the RAM, it's a piece
> of cake. Except that the slram option parsing is not user-friendly,
> with me being an examplary user.

It should be ... I need to find some time to dig that howto and
try to do it ...

> For memory above 4GB, things are harder. Basically you'd have to write
> a new mtd driver that copies some of the highmem code. Maybe a day or
> two plus testing.

Ok, right to the stack of 'would be nice to work on' stuff. 

The "feeling" thing is going to be groovy to test - I guess some parallel
kernel compiles would do - hmmm ... will think about it.

Thanks,

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)

^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2003-05-09  3:08 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <200304251640110420.0069172B@smtp.comcast.net>
2003-04-25 20:48 ` Swap Compression rmoser
2003-04-25 21:14   ` Jörn Engel
2003-04-25 21:17   ` John Bradford
2003-04-25 22:32 rmoser
2003-04-28 21:35 ` Timothy Miller
2003-04-29  0:43   ` Con Kolivas
2003-04-25 22:48 rmoser
2003-04-26  9:17 ` Jörn Engel
     [not found]   ` <200304261148590300.00CE9372@smtp.comcast.net>
     [not found]     ` <20030426160920.GC21015@wohnheim.fh-wedel.de>
2003-04-27  2:24       ` rmoser
2003-04-27  9:04         ` Jörn Engel
2003-04-27 17:24           ` rmoser
2003-04-27 17:51             ` Jörn Engel
2003-04-27 18:31               ` rmoser
2003-04-27 19:04                 ` Jörn Engel
2003-04-28  8:52                   ` Eric W. Biederman
2003-04-28 10:26                     ` Jörn Engel
     [not found]                 ` <3EAE8899.2010208@techsource.com>
     [not found]                   ` <200304291521120230.0462A551@smtp.comcast.net>
2003-04-29 19:21                     ` rmoser
     [not found]         ` <3EADAA5D.1090408@techsource.com>
     [not found]           ` <200304282258310030.00DED562@smtp.comcast.net>
2003-04-29  2:58             ` rmoser
2003-04-29 20:07 Timothy Miller
2003-04-29 20:40 ` rmoser
2003-04-29 21:14   ` John Bradford
2003-04-30  0:59     ` rmoser
2003-04-30  2:48       ` Con Kolivas
2003-04-30 12:59         ` Jörn Engel
2003-04-30 19:18           ` rmoser
2003-05-01 22:07           ` rmoser
2003-05-02  2:46             ` jw schultz
2003-05-08  3:17 Perez-Gonzalez, Inaky
2003-05-08  8:07 ` Jörn Engel
2003-05-09  3:21 Perez-Gonzalez, Inaky

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).