* Swap Compression
[not found] <200304251640110420.0069172B@smtp.comcast.net>
@ 2003-04-25 20:48 ` rmoser
2003-04-25 21:14 ` Jörn Engel
2003-04-25 21:17 ` John Bradford
0 siblings, 2 replies; 21+ messages in thread
From: rmoser @ 2003-04-25 20:48 UTC (permalink / raw)
To: linux-kernel
please CC a copy to me personally; I am not subscribed.
Sorry if this is HTML mailed. I don't know how to control those settings
COMPRESSED SWAP
This is mainly for things like linux on iPaq (handhelds.org) and people who like to play (me :), but how about compressed swap and RAM as swap? To be plausable, we need a very fast compression algorithm. I'd say use the following back pointer algorithm (this is headerless and I coded a decompressor in 6502 assembly in about 315 bytes) and 100k block sizes (compress streams of data until they are 100k in size, then stop. Include the cap at the end in the 100k).
Here is how the compression works (use fixed width font):
;
; Type Size Description
; Byte 1 Distance to the next backpointer. 0 means end-of-stream.
; Stream X Just straight data, no work needbe performed on it.
; Byte 1 Distance (up to 256, it's base 1) back to seek
; Byte 1 Number of bytes (Up to 255) to copy. 0 means don't
; really do anything (for chaining pointers in large
; areas of nonredundant data)
;
; The above repeats until the end of the stream is reached. Teach by example!
;
; 04 00 01 02 03 03 03 04 00 01 02 03 05 04 06 01 03 00 00 00
; ^ & # ^ & # ^ & # ^
; ^ = Distance to next; & = backseek; # = duplicate length
;
; The above stream uncompresses to:
; 00 01 02 03 00 01 02 00 01 02 03 01 02 00 01 01 03
;
; Note how very small datasets do not compress well. Very large would
; begin to catch up. This is only a 17 byte dataset.
Also, use a booyer-moore algorithm to do the redundancy searching (modify the one in the fdber project at fdber.sourceforge.net to work on binary data, for instance) to further speed it up. In the end I'd project about a 2-4k code increase (compiled/assembled) in the kernel size, plus 256 bytes for the compression/decompression buffer, plus the dedicated 100k to load the compressed block into RAM. So about 2-4k increase in the kernel size and 105k increase in RAM usage.
The feature should be optional, and controlled by swapon options. For instance:
swapon -o compress /dev/hda5
The above could turn a swap partition on under compressed swap mode. Of course we only would want to use 100k blocks of compressed data, but some extra code can allow the user to configure it:
swapon -o compress=1M /dev/hda5
1 MB block size for instance if you want. Don't make this too limited; if such a thing were implimented, with such an option, I'd still expect the following to work:
swapon -o compress=1247299 /dev/hda5
to load a swap partition with 1,247,299 byte blocks. Of course there'd be a rediculous cap on the high end (500 MB block size?!!!!!!!!!!!!), and an impossible cap on both ends (block size > available RAM; block size < 5 (a valid stream fits in 4 but it can't store any data or be processed)).
For predicting available swap, just take statistics and do some calculations, figure out what the average ratio is, and give that times real size as the total swap size. If you really really care that much, discard outliers.
SWAP ON RAM
The other feature, for handheld devices, is for all those iPaqs with 32 or 64 MB RAM: swap-on-RAM. This is useless (i.e. swap on a RAMdisk), but with the swap compression it becomes useful.
Swap on RAM allows the swap driver to use RAM directly as swap, i.e. not as a swap file on a ramdisk. For instance:
swapon -o compress -o swaponram=16M
The above would use 16 MB of my RAM as a compressed swap file, negating the overhead of disk access, filesystem control, ramdisk control, and so on. On an iPaq with 32 MB RAM, I could access say 16 M RAM (physical) and 32 MB swap now, giving me 1.5x my real ram (predicted). With 64 MB, it would be 82 MB RAM (1.25x real RAM), or I could just use 32MB (1.5x = 96 MB) or even 48 MB (1.75x = 112 MB). Can't use all of it of course; and the more I use the more it swaps.
When a real swap and a RAM swap exist, a RAM swap should be the first choice for swapping out (highest precidence). This would keep performence up the most under these odd conditions.
CONCLUSION
I think compressed swap and swap-on-ram with compression would be a great idea, especially for embedded systems. High-performance systems that can handle the compression/decompression without blinking would especially benefit, as the swap-on-ram feature would give an almost seamless RAM increase. Low-performance systems would take a performance hit, but embedded devices would still benefit from the swap-on-ram with compression RAM boost, considering they can probably handle the algorithm. I am looking forward to seeing this implimented in 2.4 and 2.5/2.6 if it is adopted.
-- Bluefox Icy
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-25 20:48 ` Swap Compression rmoser
@ 2003-04-25 21:14 ` Jörn Engel
2003-04-25 21:17 ` John Bradford
1 sibling, 0 replies; 21+ messages in thread
From: Jörn Engel @ 2003-04-25 21:14 UTC (permalink / raw)
To: rmoser; +Cc: linux-kernel
On Fri, 25 April 2003 16:48:15 -0400, rmoser wrote:
>
> Sorry if this is HTML mailed. I don't know how to control those settings
It is not, but if you could limit lines to <80 characters, that would
be nice.
> COMPRESSED SWAP
>
> This is mainly for things like linux on iPaq (handhelds.org) and
> people who like to play (me :), but how about compressed swap and
> RAM as swap? To be plausable, we need a very fast compression
> algorithm. I'd say use the following back pointer algorithm (this
> is headerless and I coded a decompressor in 6502 assembly in about
> 315 bytes) and 100k block sizes (compress streams of data until they
> are 100k in size, then stop. Include the cap at the end in the
> 100k).
>
> [...]
>
> CONCLUSION
>
> I think compressed swap and swap-on-ram with compression would be a
> great idea, especially for embedded systems. High-performance
> systems that can handle the compression/decompression without
> blinking would especially benefit, as the swap-on-ram feature would
> give an almost seamless RAM increase. Low-performance systems would
> take a performance hit, but embedded devices would still benefit
> from the swap-on-ram with compression RAM boost, considering they
> can probably handle the algorithm. I am looking forward to seeing
> this implimented in 2.4 and 2.5/2.6 if it is adopted.
Nice idea. This might even benefit normal pc style boxes, if the
performance loss through compression is overcompensated by io gains
(less data transferred).
The tiny problem I see is that most people here have tons of whacky
ideas themselves but lack the time to actually implement them. If you
really want to get it done, do it yourself. It doesn't have to be
perfect, if it works in principle and appears to be promising, you
will likely receive enough help to finish it. But you have to get
there first.
At least, that is how it usually works. Feel free to prove me wrong.
Jörn
--
When in doubt, use brute force.
-- Ken Thompson
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-25 20:48 ` Swap Compression rmoser
2003-04-25 21:14 ` Jörn Engel
@ 2003-04-25 21:17 ` John Bradford
1 sibling, 0 replies; 21+ messages in thread
From: John Bradford @ 2003-04-25 21:17 UTC (permalink / raw)
To: rmoser; +Cc: linux-kernel
> Sorry if this is HTML mailed. I don't know how to control those settings
HTML mail is automatically filtered from LKML.
> COMPRESSED SWAP
We discussed this on the list quite recently, have a look at:
http://marc.theaimsgroup.com/?l=linux-kernel&m=105018674018129&w=2
and:
http://linuxcompressed.sourceforge.net/
John.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Re: Swap Compression
@ 2003-04-25 22:32 rmoser
2003-04-28 21:35 ` Timothy Miller
0 siblings, 1 reply; 21+ messages in thread
From: rmoser @ 2003-04-25 22:32 UTC (permalink / raw)
To: linux-kernel
Yeah you did but I'm going into a bit more detail, and with a very tight algorithm. Heck the algo was originally designed based on another compression algorithm, but for a 6502 packer. I aimed at speed, simplicity, and minimal RAM usage (hint: it used 4k for the code AND the compressed data on a 6502, 336 bytes for code, and if I turn it into just a straight packer I can go under 200 bytes on the 6502).
Honestly, I just never looked. I look in my kernel. But still, the stuff I defined about swapon options, swap-on-ram, and how the compression works (yes, compressed without headers) is all the detail you need about it to go do it AFAIK. Preplanning should be done there--done meaning workable, not "the absolute best."
--Bluefox Icy
---- ORIGINAL MESSAGE ----
List: linux-kernel
Subject: Re: Swap Compression
From: John Bradford <john () grabjohn ! com>
Date: 2003-04-25 21:17:11
[Download message RAW]
> Sorry if this is HTML mailed. I don't know how to control those settings
HTML mail is automatically filtered from LKML.
> COMPRESSED SWAP
We discussed this on the list quite recently, have a look at:
http://marc.theaimsgroup.com/?l=linux-kernel&m=105018674018129&w=2
and:
http://linuxcompressed.sourceforge.net/
John.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-25 22:32 rmoser
@ 2003-04-28 21:35 ` Timothy Miller
2003-04-29 0:43 ` Con Kolivas
0 siblings, 1 reply; 21+ messages in thread
From: Timothy Miller @ 2003-04-28 21:35 UTC (permalink / raw)
To: rmoser; +Cc: linux-kernel
rmoser wrote:
>Yeah you did but I'm going into a bit more detail, and with a very tight algorithm. Heck the algo was originally designed based on another compression algorithm, but for a 6502 packer. I aimed at speed, simplicity, and minimal RAM usage (hint: it used 4k for the code AND the compressed data on a 6502, 336 bytes for code, and if I turn it into just a straight packer I can go under 200 bytes on the 6502).
>
>Honestly, I just never looked. I look in my kernel. But still, the stuff I defined about swapon options, swap-on-ram, and how the compression works (yes, compressed without headers) is all the detail you need about it to go do it AFAIK. Preplanning should be done there--done meaning workable, not "the absolute best."
>
>
I think we might be able to deal with a somewhat more heavy-weight
compression. Considering how much faster the compression is than the
disk access, the better the compression, the better the performance.
Usually, if you have too much swapping, the CPU usage will drop, because
things aren't getting done. That means we have plenty of head room to
spend time compressing rather than waiting. The speed over-all would go
up. Theoretically, we could run into a situation where the compression
time dominates. In that case, it would be beneficial to have a tuning
options which uses a less CPU-intensive compression algorithm.
>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-28 21:35 ` Timothy Miller
@ 2003-04-29 0:43 ` Con Kolivas
0 siblings, 0 replies; 21+ messages in thread
From: Con Kolivas @ 2003-04-29 0:43 UTC (permalink / raw)
To: Timothy Miller, rmoser; +Cc: linux-kernel
On Tue, 29 Apr 2003 07:35, Timothy Miller wrote:
> rmoser wrote:
> >Yeah you did but I'm going into a bit more detail, and with a very tight
> > algorithm. Heck the algo was originally designed based on another
> > compression algorithm, but for a 6502 packer. I aimed at speed,
> > simplicity, and minimal RAM usage (hint: it used 4k for the code AND the
> > compressed data on a 6502, 336 bytes for code, and if I turn it into just
> > a straight packer I can go under 200 bytes on the 6502).
> >
> >Honestly, I just never looked. I look in my kernel. But still, the stuff
> > I defined about swapon options, swap-on-ram, and how the compression
> > works (yes, compressed without headers) is all the detail you need about
> > it to go do it AFAIK. Preplanning should be done there--done meaning
> > workable, not "the absolute best."
>
> I think we might be able to deal with a somewhat more heavy-weight
> compression. Considering how much faster the compression is than the
> disk access, the better the compression, the better the performance.
>
> Usually, if you have too much swapping, the CPU usage will drop, because
> things aren't getting done. That means we have plenty of head room to
> spend time compressing rather than waiting. The speed over-all would go
> up. Theoretically, we could run into a situation where the compression
> time dominates. In that case, it would be beneficial to have a tuning
> options which uses a less CPU-intensive compression algorithm.
The work that Rodrigo De Castro did on compressed caching
(linuxcompressed.sf.net) included a minilzo algorithm which I used by default
in the -ck patch addon as it performed the best for all the reasons you
mention. Why not look at that lzo code for adoption.
Con
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Re: Swap Compression
@ 2003-04-25 22:48 rmoser
2003-04-26 9:17 ` Jörn Engel
0 siblings, 1 reply; 21+ messages in thread
From: rmoser @ 2003-04-25 22:48 UTC (permalink / raw)
To: linux-kernel
Yeah, I had to mail it 3 times. Lst time I figured it out.
As for the performance hit, the original idea of that very tiny format was
to pack 6502 programs into 4k of code. The expansion phase is very
tight and very efficient, and on a ... anything... it will provide no problem.
The swap-on-ram as long as it's not like 200 MB uncompressed SOR and
1 MB RAM will I think work great in the decompression phase.
Compression will take a little overhead. I think if you use a boyer-moore
fast string search algo for binary strings (yes you can do this), you can
quickly compress the data. It may be like.. just a guess... 10-30 times
more overhead than the decompression phase. So use it on at least a
10-30 mhz processor. If I ever write the code, it won't be kernel; just the
compression/decompression program (userspace). Take the code and
stuff it into the kernel if I do. I'll at the point of the algo coming in to
existence make another estimate.
The real power in this is Swap on RAM, but setting that as having precidence
over swap on disk (normal swaps) would decrease disk swap usage by
supplying more RAM in RAM. And of course swapping RAM to RAM is
a lot faster. I'm looking at this for PDA's but yes I will be running this on
my desktop the day we see it.
Well, I could work on the compression code, mebbe I can put the tarball up
here. If I do I'd expect someone to add the code to swap to work with it--in
kernel 2.4 at the very least (port it to dev kernel later!). As a separate module.
We don't want code that could be mean in the real swap driver. :)
--Bluefox Icy
---- ORIGINAL MESSAGE ----
List: linux-kernel
Subject: Re: Swap Compression
From: =?iso-8859-1?Q?J=F6rn?= Engel <joern () wohnheim ! fh-wedel ! de>
Date: 2003-04-25 21:14:05
[Download message RAW]
On Fri, 25 April 2003 16:48:15 -0400, rmoser wrote:
>
> Sorry if this is HTML mailed. I don't know how to control those settings
It is not, but if you could limit lines to <80 characters, that would
be nice.
> COMPRESSED SWAP
>
> This is mainly for things like linux on iPaq (handhelds.org) and
> people who like to play (me :), but how about compressed swap and
> RAM as swap? To be plausable, we need a very fast compression
> algorithm. I'd say use the following back pointer algorithm (this
> is headerless and I coded a decompressor in 6502 assembly in about
> 315 bytes) and 100k block sizes (compress streams of data until they
> are 100k in size, then stop. Include the cap at the end in the
> 100k).
>
> [...]
>
> CONCLUSION
>
> I think compressed swap and swap-on-ram with compression would be a
> great idea, especially for embedded systems. High-performance
> systems that can handle the compression/decompression without
> blinking would especially benefit, as the swap-on-ram feature would
> give an almost seamless RAM increase. Low-performance systems would
> take a performance hit, but embedded devices would still benefit
> from the swap-on-ram with compression RAM boost, considering they
> can probably handle the algorithm. I am looking forward to seeing
> this implimented in 2.4 and 2.5/2.6 if it is adopted.
Nice idea. This might even benefit normal pc style boxes, if the
performance loss through compression is overcompensated by io gains
(less data transferred).
The tiny problem I see is that most people here have tons of whacky
ideas themselves but lack the time to actually implement them. If you
really want to get it done, do it yourself. It doesn't have to be
perfect, if it works in principle and appears to be promising, you
will likely receive enough help to finish it. But you have to get
there first.
At least, that is how it usually works. Feel free to prove me wrong.
Jörn
--
When in doubt, use brute force.
-- Ken Thompson
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Re: Swap Compression
2003-04-25 22:48 rmoser
@ 2003-04-26 9:17 ` Jörn Engel
[not found] ` <200304261148590300.00CE9372@smtp.comcast.net>
0 siblings, 1 reply; 21+ messages in thread
From: Jörn Engel @ 2003-04-26 9:17 UTC (permalink / raw)
To: rmoser; +Cc: linux-kernel
On Fri, 25 April 2003 18:48:41 -0400, rmoser wrote:
>
> Yeah, I had to mail it 3 times. Lst time I figured it out.
<more formal junk>
- Your mailer doesn't generate In-Reply-To: or References: Headers.
This breaks threading.
- It is usually more readable if you reply below the quoted text you
refer to.
- Most people prefer, if you reply to all. Not everyone here is
actually subscribed to the list.
- Feel free to ignore any of this, as long as the mail contains
interesting information. :)
</more formal junk>
> As for the performance hit, the original idea of that very tiny format was
> to pack 6502 programs into 4k of code. The expansion phase is very
> tight and very efficient, and on a ... anything... it will provide no problem.
> The swap-on-ram as long as it's not like 200 MB uncompressed SOR and
> 1 MB RAM will I think work great in the decompression phase.
>
> Compression will take a little overhead. I think if you use a boyer-moore
> fast string search algo for binary strings (yes you can do this), you can
> quickly compress the data. It may be like.. just a guess... 10-30 times
> more overhead than the decompression phase. So use it on at least a
> 10-30 mhz processor. If I ever write the code, it won't be kernel; just the
> compression/decompression program (userspace). Take the code and
> stuff it into the kernel if I do. I'll at the point of the algo coming in to
> existence make another estimate.
A userspace program would be just fine. Send it to me and I'll convert
it to the kernel, putting it somewhere under lib/.
Do you have any problems with a GPL license to your code (necessary
for kernel port)?
> The real power in this is Swap on RAM, but setting that as having precidence
> over swap on disk (normal swaps) would decrease disk swap usage by
> supplying more RAM in RAM. And of course swapping RAM to RAM is
> a lot faster. I'm looking at this for PDA's but yes I will be running this on
> my desktop the day we see it.
Swapping RAM to RAM sounds interesting, but also quite complicated. As
a first step, I would try to compress the swap data before going to
disk, that should be relatively simple to do.
("I would" means, I will if I find the time for it.)
> Well, I could work on the compression code, mebbe I can put the tarball up
> here. If I do I'd expect someone to add the code to swap to work with it--in
> kernel 2.4 at the very least (port it to dev kernel later!). As a separate module.
> We don't want code that could be mean in the real swap driver. :)
Right. But for 2.4, there is no swap driver, that you can simply
enable or disable. I hacked up a patch, but so far, disabling swap
eats ~100k of memory every second, so that clearly needs more work.
Jörn
--
Do not stop an army on its way home.
-- Sun Tzu
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
@ 2003-04-29 20:07 Timothy Miller
2003-04-29 20:40 ` rmoser
0 siblings, 1 reply; 21+ messages in thread
From: Timothy Miller @ 2003-04-29 20:07 UTC (permalink / raw)
To: Linux Kernel Mailing List
Con Kolivas wrote:
>On Tue, 29 Apr 2003 07:35, Timothy Miller wrote:
>
>
>>
>>
>The work that Rodrigo De Castro did on compressed caching
>(linuxcompressed.sf.net) included a minilzo algorithm which I used by default
>in the -ck patch addon as it performed the best for all the reasons you
>mention. Why not look at that lzo code for adoption.
>
>
>
Some time before rmoser started HIS discussion on compressed swap, I
started a discussion on the same topic. Out of that discussion came
mention of an existing project which did EXACTLY what I was proposing.
That made me happy. For some reason rmoser wants to start from
scratch. He can do that if he wants. I had only hoped that my earlier
mention of it would spark interest in taking the existing implementation
and adding it to the mainline kernel, after some incompatbilities were
dealt with. Unfortunately, I don't have the time to engage directly in
that particular aspect the kernel development.
>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-29 20:07 Timothy Miller
@ 2003-04-29 20:40 ` rmoser
2003-04-29 21:14 ` John Bradford
0 siblings, 1 reply; 21+ messages in thread
From: rmoser @ 2003-04-29 20:40 UTC (permalink / raw)
To: linux-kernel
*********** REPLY SEPARATOR ***********
On 4/29/2003 at 3:39 PM Timothy Miller wrote:
>
>
>Con Kolivas wrote:
>
>On Tue, 29 Apr 2003 07:35, Timothy Miller wrote:
>
>The work that Rodrigo De Castro did on compressed caching
>(linuxcompressed.sf.net) included a minilzo algorithm which I used by
>default
>in the -ck patch addon as it performed the best for all the reasons you
>mention. Why not look at that lzo code for adoption.
>Some time before rmoser started HIS discussion on compressed swap, I
>started a discussion on the same topic. Out of that discussion came
>mention of an existing project which did EXACTLY what I was proposing.
>That made me happy. For some reason rmoser wants to start from scratch.
>He can do that if he wants. I had only hoped that my earlier mention of
>it would spark interest in taking the existing implementation and adding
>it to the mainline kernel, after some incompatbilities were dealt with.
>Unfortunately, I don't have the time to engage directly in that particular
>aspect the kernel development
>
I want to start from scratch because I am trying a different angle, and I don't understand
fully current projects on it. I am not trying to discredit anyone. In fact, I would hope that
the existing implimentation (different as it is) would manage to get itself in as well.
Remember, modularity is all about options. Options aren't just "Do you want this or
that?", but also "How do you want this or that done?" Plus, I am not compressing the
page cache, assuming I understand what this means. AFAIK (from others' help), the
page cache is a cached copy of pages that have gone to the swap recently, or been
pulled off recently. I am not touching that.
Other things (not brought up by me) include making a central compression library for the
kernel (modular, i.e. modprobe zlib and such) (Jorn's idea). That is something that I think
would be essential for swap compression; why should one module horde all the compression
algo's?
--Bluefox
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-29 20:40 ` rmoser
@ 2003-04-29 21:14 ` John Bradford
2003-04-30 0:59 ` rmoser
0 siblings, 1 reply; 21+ messages in thread
From: John Bradford @ 2003-04-29 21:14 UTC (permalink / raw)
To: rmoser; +Cc: linux-kernel
> I want to start from scratch because I am trying a different angle,
> and I don't understand fully current projects on it.
There is nothing wrong from having a separate, parallel project
running - it's a good idea, and generally helps to spark interest in
both projects.
John.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-29 21:14 ` John Bradford
@ 2003-04-30 0:59 ` rmoser
2003-04-30 2:48 ` Con Kolivas
0 siblings, 1 reply; 21+ messages in thread
From: rmoser @ 2003-04-30 0:59 UTC (permalink / raw)
To: John Bradford; +Cc: linux-kernel
*********** REPLY SEPARATOR ***********
On 4/29/2003 at 10:14 PM John Bradford wrote:
>> I want to start from scratch because I am trying a different angle,
>> and I don't understand fully current projects on it.
>
>There is nothing wrong from having a separate, parallel project
>running - it's a good idea, and generally helps to spark interest in
>both projects.
>
THANK YOU!
You've saved me from having to defend meself.
--Bluefox Icy
>John.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-30 0:59 ` rmoser
@ 2003-04-30 2:48 ` Con Kolivas
2003-04-30 12:59 ` Jörn Engel
0 siblings, 1 reply; 21+ messages in thread
From: Con Kolivas @ 2003-04-30 2:48 UTC (permalink / raw)
To: rmoser, John Bradford; +Cc: linux-kernel
On Wed, 30 Apr 2003 10:59, rmoser wrote:
> *********** REPLY SEPARATOR ***********
>
> On 4/29/2003 at 10:14 PM John Bradford wrote:
> >> I want to start from scratch because I am trying a different angle,
> >> and I don't understand fully current projects on it.
> >
> >There is nothing wrong from having a separate, parallel project
> >running - it's a good idea, and generally helps to spark interest in
> >both projects.
>
> THANK YOU!
>
> You've saved me from having to defend meself.
I don't think a parallel project is a bad idea either. I was just suggesting
adding the minilzo algorithm from the linuxcompressed project as one of the
compression algorithms available.
Con
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-30 2:48 ` Con Kolivas
@ 2003-04-30 12:59 ` Jörn Engel
2003-04-30 19:18 ` rmoser
2003-05-01 22:07 ` rmoser
0 siblings, 2 replies; 21+ messages in thread
From: Jörn Engel @ 2003-04-30 12:59 UTC (permalink / raw)
To: Con Kolivas; +Cc: rmoser, John Bradford, linux-kernel
On Wed, 30 April 2003 12:48:07 +1000, Con Kolivas wrote:
>
> I don't think a parallel project is a bad idea either. I was just suggesting
> adding the minilzo algorithm from the linuxcompressed project as one of the
> compression algorithms available.
Actually, I'd like a central compression library with a large
assortment of algorithms. That way the really common code is shared
between both (or more) projects is shared.
Also, yet another unused compression algorithm hurts about as bad, as
yet another unused device driver. It just grows the kernel .tar.bz2.
Jörn
--
Time? What's that? Time is only worth what you do with it.
-- Theo de Raadt
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-30 12:59 ` Jörn Engel
@ 2003-04-30 19:18 ` rmoser
2003-05-01 22:07 ` rmoser
1 sibling, 0 replies; 21+ messages in thread
From: rmoser @ 2003-04-30 19:18 UTC (permalink / raw)
To: =?UNKNOWN?Q?J=F6rn?= Engel; +Cc: linux-kernel
*********** REPLY SEPARATOR ***********
On 4/30/2003 at 2:59 PM Jörn Engel wrote:
>On Wed, 30 April 2003 12:48:07 +1000, Con Kolivas wrote:
>>
>> I don't think a parallel project is a bad idea either. I was just
>suggesting
>> adding the minilzo algorithm from the linuxcompressed project as one of
>the
>> compression algorithms available.
>
>Actually, I'd like a central compression library with a large
>assortment of algorithms. That way the really common code is shared
>between both (or more) projects is shared.
>
That's just great, heh. It will in the future cut kernel code size and matinence
work.
>Also, yet another unused compression algorithm hurts about as bad, as
>yet another unused device driver. It just grows the kernel .tar.bz2.
>
Not to get side tracked, but there is an answer for this.....
Well then let's restructure the kernel's build system soon. It's 15 seconds
since I read this, and already I'm working out in my head the high-level
workings of a modular build system (what it would look like).
For an example, future kernel releases could be in directories, which contain
the kernel .tar.bz2, then separate sets of drivers and such. Simply unpacking
these drivers into your kernel source tree could place files that make would
read, i.e. the kernel build system would adapt. For example:
linux/options/foo.lod <-- Linux Option Definition file.
This file would reference other LO's that it requires, and give configuration
options. These options would have references to LO's that they require.
They would also reference which version ranges of themselves they are
compatible with. Thus:
make menuconfig
<M>Foo
----<M>Foo with Bar support
<SELECT> <EXIT> <HELP> <MISSING>
Hit missing on an option, say Foo:
Dependencies for CONFIG_FOO
< >Foo
----< >Bar
----< >Baz [FAILED DEPENDS!]
Hit baz:
Dependencies for CONFIG_FOO_BAZ:
Foo with Baz support needs Baz compatible with Baz v1.32
So you get the Baz option tarball, or the tarball containing it:
linux/options/baz.lod
make menuconfig:
<M>Foo
----<M>Foo with Bar support
----<M>Foo with Baz support
Yes I know this is a hassle, but it's not like Microsoft drivers
where you spend 12 hours all over the net searching for stuff;
the main kernel source distribution tree would contain the
kernel, and the kernel.org mirrors would all have the Linux
Options. There'd be a tarball for the "default must-have drivers"
separate from the kernel core.
It's just a suggestion for the future.
--Bluefox Icy
>Jörn
>
>--
>Time? What's that? Time is only worth what you do with it.
>-- Theo de Raadt
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-04-30 12:59 ` Jörn Engel
2003-04-30 19:18 ` rmoser
@ 2003-05-01 22:07 ` rmoser
2003-05-02 2:46 ` jw schultz
1 sibling, 1 reply; 21+ messages in thread
From: rmoser @ 2003-05-01 22:07 UTC (permalink / raw)
To: =?UNKNOWN?Q?J=F6rn?= Engel; +Cc: linux-kernel
>Actually, I'd like a central compression library with a large
>assortment of algorithms. That way the really common code is shared
>between both (or more) projects is shared.
>
>Also, yet another unused compression algorithm hurts about as bad, as
>yet another unused device driver. It just grows the kernel .tar.bz2.
>
>Jörn
Had a thought. Why wait for a compression algorithm? Jorn, if you are going
to work on putting the code into the kernel and making the stuff to allow the
swap code to use it, why not start coding it before the compression code is
finished? i.e. get the stuff down for the swap filtering (filtering i.e. passing
through a compression or encryption routine) and swap-on-ram stuff, and later
take the compression algo code and place the module interface on it and make
a kernel module.
At this point, I'd say to allow specified order filters, to allow for swap cyphering
and compression. Security, you know; swap files are a security hazard. Just
power-off, boot a root disk, pull up the swap partition, rip out the blocks, and
look for what looks to be the root password.
--Bluefox Icy
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-05-01 22:07 ` rmoser
@ 2003-05-02 2:46 ` jw schultz
0 siblings, 0 replies; 21+ messages in thread
From: jw schultz @ 2003-05-02 2:46 UTC (permalink / raw)
To: linux-kernel
On Thu, May 01, 2003 at 06:07:59PM -0400, rmoser wrote:
> Had a thought. Why wait for a compression algorithm? Jorn, if you are going
> to work on putting the code into the kernel and making the stuff to allow the
> swap code to use it, why not start coding it before the compression code is
> finished? i.e. get the stuff down for the swap filtering (filtering i.e. passing
> through a compression or encryption routine) and swap-on-ram stuff, and later
> take the compression algo code and place the module interface on it and make
> a kernel module.
>
> At this point, I'd say to allow specified order filters, to allow for swap cyphering
> and compression. Security, you know; swap files are a security hazard. Just
> power-off, boot a root disk, pull up the swap partition, rip out the blocks, and
> look for what looks to be the root password.
While we're having thoughts, this thread keeps me thinking
it would make sense to have a block device driver that would
be assigned unused memory.
I don't mean memory on video cards etc. I'm thinking of the
10% of RAM unused when 1GB systems are booted with MEM=900M
because they run faster with HIGHMEM turned off.
The primary use for this "device" would be high priority swap.
Even with whatever overhead it takes to access it should be
orders of magnitude faster than any spinning media.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: jw@pegasys.ws
Remember Cernan and Schmitt
^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: Swap Compression
@ 2003-05-08 3:17 Perez-Gonzalez, Inaky
2003-05-08 8:07 ` Jörn Engel
0 siblings, 1 reply; 21+ messages in thread
From: Perez-Gonzalez, Inaky @ 2003-05-08 3:17 UTC (permalink / raw)
To: 'jw schultz', 'linux-kernel@vger.kernel.org'
> From: jw schultz [mailto:jw@pegasys.ws]
>
> While we're having thoughts, this thread keeps me thinking
> it would make sense to have a block device driver that would
> be assigned unused memory.
>
> I don't mean memory on video cards etc. I'm thinking of the
> 10% of RAM unused when 1GB systems are booted with MEM=900M
> because they run faster with HIGHMEM turned off.
>
> The primary use for this "device" would be high priority swap.
> Even with whatever overhead it takes to access it should be
> orders of magnitude faster than any spinning media.
This reminds me of some howto I saw somewhere of someway to
use the MTD drivers to access the unused video RAM and turn
it into swap (maybe with blkmtd?) ... probably it can be done
with that too.
I'd really love it ... I don't know if I can blame it on highmem
or not, but since I enabled it, my system 'feels' slower.
Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Swap Compression
2003-05-08 3:17 Perez-Gonzalez, Inaky
@ 2003-05-08 8:07 ` Jörn Engel
0 siblings, 0 replies; 21+ messages in thread
From: Jörn Engel @ 2003-05-08 8:07 UTC (permalink / raw)
To: Perez-Gonzalez, Inaky
Cc: 'jw schultz', 'linux-kernel@vger.kernel.org'
On Wed, 7 May 2003 20:17:32 -0700, Perez-Gonzalez, Inaky wrote:
>
> This reminds me of some howto I saw somewhere of someway to
> use the MTD drivers to access the unused video RAM and turn
> it into swap (maybe with blkmtd?) ... probably it can be done
> with that too.
Jupp, if you know the physical address range of the RAM, it's a piece
of cake. Except that the slram option parsing is not user-friendly,
with me being an examplary user.
For memory above 4GB, things are harder. Basically you'd have to write
a new mtd driver that copies some of the highmem code. Maybe a day or
two plus testing.
> I'd really love it ... I don't know if I can blame it on highmem
> or not, but since I enabled it, my system 'feels' slower.
Go ahead and try it. If it 'feels' faster, it should be possible to
benchmark your feeling into some numbers.
Jörn
--
Victory in war is not repetitious.
-- Sun Tzu
^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: Swap Compression
@ 2003-05-09 3:21 Perez-Gonzalez, Inaky
0 siblings, 0 replies; 21+ messages in thread
From: Perez-Gonzalez, Inaky @ 2003-05-09 3:21 UTC (permalink / raw)
To: 'Jörn Engel', Perez-Gonzalez, Inaky
Cc: 'jw schultz', 'linux-kernel@vger.kernel.org'
> From: Jörn Engel [mailto:joern@wohnheim.fh-wedel.de]
>
> On Wed, 7 May 2003 20:17:32 -0700, Perez-Gonzalez, Inaky wrote:
> > This reminds me of some howto I saw somewhere of someway to
> > use the MTD drivers to access the unused video RAM and turn
> > it into swap (maybe with blkmtd?) ... probably it can be done
> > with that too.
>
> Jupp, if you know the physical address range of the RAM, it's a piece
> of cake. Except that the slram option parsing is not user-friendly,
> with me being an examplary user.
It should be ... I need to find some time to dig that howto and
try to do it ...
> For memory above 4GB, things are harder. Basically you'd have to write
> a new mtd driver that copies some of the highmem code. Maybe a day or
> two plus testing.
Ok, right to the stack of 'would be nice to work on' stuff.
The "feeling" thing is going to be groovy to test - I guess some parallel
kernel compiles would do - hmmm ... will think about it.
Thanks,
Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2003-05-09 3:08 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <200304251640110420.0069172B@smtp.comcast.net>
2003-04-25 20:48 ` Swap Compression rmoser
2003-04-25 21:14 ` Jörn Engel
2003-04-25 21:17 ` John Bradford
2003-04-25 22:32 rmoser
2003-04-28 21:35 ` Timothy Miller
2003-04-29 0:43 ` Con Kolivas
2003-04-25 22:48 rmoser
2003-04-26 9:17 ` Jörn Engel
[not found] ` <200304261148590300.00CE9372@smtp.comcast.net>
[not found] ` <20030426160920.GC21015@wohnheim.fh-wedel.de>
2003-04-27 2:24 ` rmoser
2003-04-27 9:04 ` Jörn Engel
2003-04-27 17:24 ` rmoser
2003-04-27 17:51 ` Jörn Engel
2003-04-27 18:31 ` rmoser
2003-04-27 19:04 ` Jörn Engel
2003-04-28 8:52 ` Eric W. Biederman
2003-04-28 10:26 ` Jörn Engel
[not found] ` <3EAE8899.2010208@techsource.com>
[not found] ` <200304291521120230.0462A551@smtp.comcast.net>
2003-04-29 19:21 ` rmoser
[not found] ` <3EADAA5D.1090408@techsource.com>
[not found] ` <200304282258310030.00DED562@smtp.comcast.net>
2003-04-29 2:58 ` rmoser
2003-04-29 20:07 Timothy Miller
2003-04-29 20:40 ` rmoser
2003-04-29 21:14 ` John Bradford
2003-04-30 0:59 ` rmoser
2003-04-30 2:48 ` Con Kolivas
2003-04-30 12:59 ` Jörn Engel
2003-04-30 19:18 ` rmoser
2003-05-01 22:07 ` rmoser
2003-05-02 2:46 ` jw schultz
2003-05-08 3:17 Perez-Gonzalez, Inaky
2003-05-08 8:07 ` Jörn Engel
2003-05-09 3:21 Perez-Gonzalez, Inaky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).