Creating compressed backing_store as swapfile

Creating compressed backing_store as swapfile

[Pintu Agarwal]

Hi,

I have one requirement:
I wanted to have a swapfile (64MB to 256MB) on my system.
But I wanted the data to be compressed and stored on the disk in my swapfile.
[Similar to zram, but compressed data should be moved to disk, instead of RAM].

Note: I wanted to optimize RAM space, so performance is not important
right now for our requirement.

So, what are the options available, to perform this in 4.x kernel version.
My Kernel: 4.9.x
Board: any - (arm64 mostly).

As I know, following are the choices:
1) ZRAM: But it compresses and store data in RAM itself
2) frontswap + zswap : Didn't explore much on this, not sure if this
is helpful for our case.
3) Manually creating swapfile: but how to compress it ?
4) Any other options ?


Thanks,
Pintu

Re: Creating compressed backing_store as swapfile

[Adam Borowski]

On Mon, Nov 05, 2018 at 08:31:46PM +0530, Pintu Agarwal wrote:
> Hi,
> 
> I have one requirement:
> I wanted to have a swapfile (64MB to 256MB) on my system.
> But I wanted the data to be compressed and stored on the disk in my swapfile.
> [Similar to zram, but compressed data should be moved to disk, instead of RAM].
> 
> Note: I wanted to optimize RAM space, so performance is not important
> right now for our requirement.
> 
> So, what are the options available, to perform this in 4.x kernel version.
> My Kernel: 4.9.x
> Board: any - (arm64 mostly).
> 
> As I know, following are the choices:
> 1) ZRAM: But it compresses and store data in RAM itself
> 2) frontswap + zswap : Didn't explore much on this, not sure if this
> is helpful for our case.
> 3) Manually creating swapfile: but how to compress it ?
> 4) Any other options ?

Loop device on any filesystem that can compress (such as btrfs)?  The
performance would suck, though -- besides the indirection of loop, btrfs
compresses in blocks of 128KB while swap wants 4KB writes.  Other similar
option is qemu-nbd -- it can use compressed disk images and expose them to a
(local) nbd client.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ Have you heard of the Amber Road?  For thousands of years, the
⣾⠁⢰⠒⠀⣿⡁ Romans and co valued amber, hauled through the Europe over the
⢿⡄⠘⠷⠚⠋⠀ mountains and along the Vistula, from Gdańsk.  To where it came
⠈⠳⣄⠀⠀⠀⠀ together with silk (judging by today's amber stalls).

Re: Creating compressed backing_store as swapfile

[Austin S. Hemmelgarn]

On 11/5/2018 10:58 AM, Adam Borowski wrote:
> On Mon, Nov 05, 2018 at 08:31:46PM +0530, Pintu Agarwal wrote:
>> Hi,
>>
>> I have one requirement:
>> I wanted to have a swapfile (64MB to 256MB) on my system.
>> But I wanted the data to be compressed and stored on the disk in my swapfile.
>> [Similar to zram, but compressed data should be moved to disk, instead of RAM].
>>
>> Note: I wanted to optimize RAM space, so performance is not important
>> right now for our requirement.
>>
>> So, what are the options available, to perform this in 4.x kernel version.
>> My Kernel: 4.9.x
>> Board: any - (arm64 mostly).
>>
>> As I know, following are the choices:
>> 1) ZRAM: But it compresses and store data in RAM itself
>> 2) frontswap + zswap : Didn't explore much on this, not sure if this
>> is helpful for our case.
>> 3) Manually creating swapfile: but how to compress it ?
>> 4) Any other options ?
> 
> Loop device on any filesystem that can compress (such as btrfs)?  The
> performance would suck, though -- besides the indirection of loop, btrfs
> compresses in blocks of 128KB while swap wants 4KB writes.  Other similar
> option is qemu-nbd -- it can use compressed disk images and expose them to a
> (local) nbd client.

Swap on any type of a networked storage device (NBD, iSCSI, ATAoE, etc) 
served from the local system is _really_ risky.  The moment the local 
server process for the storage device gets forced out to swap, you deadlock.

Performance isn't _too_ bad for the BTRFS case though (I've actually 
tested this before), just make sure you disable direct I/O mode on the 
loop device, otherwise you run the risk of data corruption.

Re: Creating compressed backing_store as swapfile

[valdis.kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2220 bytes --]

On Mon, 05 Nov 2018 20:31:46 +0530, Pintu Agarwal said:
> I wanted to have a swapfile (64MB to 256MB) on my system.
> But I wanted the data to be compressed and stored on the disk in my swapfile.
> [Similar to zram, but compressed data should be moved to disk, instead of RAM].

What platform are you on that you're both storage constrained enough to need
swap, and also so short on disk space that compressing it makes sense?
Understanding the hardware constraints here would help in advising you.

> Note: I wanted to optimize RAM space, so performance is not important
> right now for our requirement.
>
> So, what are the options available, to perform this in 4.x kernel version.
> My Kernel: 4.9.x

Given that this is a greenfield development, why are you picking a kernel
that's 2 years out of date?  You *do* realize that 4.9.135 does *not* contain
all the bugfixes since then, only that relatively small subset that qualify for
'stable' (see Documentation/process/stable-kernel-rules.rst for the gory
details).

One possible total hack would be to simply use a file-based swap area,
but put the file on a filesystem that supports automatic inline compression.

Note that this will probably *totally* suck on performance, because there's
no good way to find where 4K block 11,493 starts inside the compressed
file, so it would have to read/decompress from the file beginning.  Also,
if you write data to a previously unused location (or even a previously used
spot that compressed the 4K page to a different length), you have a bad time
inserting it.  (Note that zram can avoid most of this because it can (a) keep
a table of pointers to where each page starts and (b) it isn't constrained to
writing to 4K blocks on disk, so if the current compression takes a 4K page down
to 1,283 bytes, it doesn't have to care *too* much if it stores that someplace
that crosses a page boundary.

Another thing that you will need to worry about is what happens in low-memory
situations - the time you *most* need to do a swap operation, you may not have
enough memory to do the I/O.  zram basically makes sure it *has* the memory
needed beforehand, and swap directly to pre-allocated disk doesn't need much
additional memory.

[-- Attachment #2: Type: application/pgp-signature, Size: 486 bytes --]

Re: Creating compressed backing_store as swapfile

[valdis.kletnieks]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

On Mon, 05 Nov 2018 11:07:12 -0500, "Austin S. Hemmelgarn" said:

> Performance isn't _too_ bad for the BTRFS case though (I've actually
> tested this before), just make sure you disable direct I/O mode on the
> loop device, otherwise you run the risk of data corruption.

Did you test that for random-access. or just sequential read/write?
(Also, see the note in my other mail regarding doing a random-access
write to the middle of the file...)


[-- Attachment #2: Type: application/pgp-signature, Size: 486 bytes --]

Re: Creating compressed backing_store as swapfile

[Austin S. Hemmelgarn]

On 11/5/2018 11:14 AM, valdis.kletnieks@vt.edu wrote:
> On Mon, 05 Nov 2018 11:07:12 -0500, "Austin S. Hemmelgarn" said:
> 
>> Performance isn't _too_ bad for the BTRFS case though (I've actually
>> tested this before), just make sure you disable direct I/O mode on the
>> loop device, otherwise you run the risk of data corruption.
> 
> Did you test that for random-access. or just sequential read/write?
> (Also, see the note in my other mail regarding doing a random-access
> write to the middle of the file...)
> 
Actual swap usage.  About 16 months ago, I had been running a couple of 
Intel NUC5PPYH boxes (Pentium N3700 CPU's, 4GB of DDR3-1333 RAM) for 
some network prototyping.  On both, I had swap set up to use a file on 
BTRFS via a loop device, and I made a point to test both with LZ4 inline 
compression and without any compression, and saw negligible performance 
differences (less than 1% in most cases).  It was, of course, 
significantly worse than running on ext4, but on a system that's so 
resource constrained that both storage and memory are at a premium to 
this degree, the performance hit is probably going to be worth it.

Also, it's probably worth noting that BTRFS doesn't need to decompress 
the entire file to read or write blocks in the middle, it splits the 
file into 128k blocks and compresses each of those independent of the 
others, so it can just decompress the 128k block that holds the actual 
block that's needed.

Re: Creating compressed backing_store as swapfile

[valdis.kletnieks]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Mon, 05 Nov 2018 11:28:49 -0500, "Austin S. Hemmelgarn" said:

> Also, it's probably worth noting that BTRFS doesn't need to decompress
> the entire file to read or write blocks in the middle, it splits the
> file into 128k blocks and compresses each of those independent of the
> others, so it can just decompress the 128k block that holds the actual
> block that's needed.

Presumably it does something sane with block allocation for the now-compressed
128K that's presumably much smaller.  Also, that limits the damage from writing to
the middle of a compression unit....

That *does* however increase the memory requirement - you can OOM or
deadlock if your read/write from the swap needs an additional 128K for the
compression buffer at an inconvenient time...


[-- Attachment #2: Type: application/pgp-signature, Size: 486 bytes --]

Re: Creating compressed backing_store as swapfile

[Austin S. Hemmelgarn]

On 11/5/2018 11:53 AM, valdis.kletnieks@vt.edu wrote:
> On Mon, 05 Nov 2018 11:28:49 -0500, "Austin S. Hemmelgarn" said:
> 
>> Also, it's probably worth noting that BTRFS doesn't need to decompress
>> the entire file to read or write blocks in the middle, it splits the
>> file into 128k blocks and compresses each of those independent of the
>> others, so it can just decompress the 128k block that holds the actual
>> block that's needed.
> 
> Presumably it does something sane with block allocation for the now-compressed
> 128K that's presumably much smaller.  Also, that limits the damage from writing to
> the middle of a compression unit....
> 
> That *does* however increase the memory requirement - you can OOM or
> deadlock if your read/write from the swap needs an additional 128K for the
> compression buffer at an inconvenient time...
> 
Indeed, and I can't really comment on how it might behave under those 
circumstances (the systems I did the testing on never saw memory 
pressure quite _that_ bad, and I had them set up to swap things out 
pretty early and really aggressively).

Re: Creating compressed backing_store as swapfile

[Pintu Agarwal]

On Mon, Nov 5, 2018 at 9:42 PM <valdis.kletnieks@vt.edu> wrote:
>
> On Mon, 05 Nov 2018 20:31:46 +0530, Pintu Agarwal said:
> > I wanted to have a swapfile (64MB to 256MB) on my system.
> > But I wanted the data to be compressed and stored on the disk in my swapfile.
> > [Similar to zram, but compressed data should be moved to disk, instead of RAM].
>
> What platform are you on that you're both storage constrained enough to need
> swap, and also so short on disk space that compressing it makes sense?
> Understanding the hardware constraints here would help in advising you.
>

Currently, I am using the minimal platform such as busybox for arm
(kind of a ubuntu based debian platform).
Also I am trying to do this on an arm based embedded board with 8 GB
MMC card and 1 GB RAM.
And I am using the ext4 filesystem with Linux Kernel version 4.9.x.
So, with 8 GB SD card I have 2 GB left on the storage space. Out of
which 64MB - 128MB would be used for swapfile.
However, note that this is not the final end product requirement.
I am just trying to demonstrate a prototype and use cases.
Performance requirement is not that strict right now, as I don't know
the end product. However, the system requirement is as minimal as
this.

The main requirement is, creating a RAM snapshot image, then
compressing some of its data and moving to swapfile, so that snapshot
image size can be reduced.
I guess, ZRAM is not useful here, so I thought to explore some other
option such as zswap, etc. ?
BTRFS is not an option, though, as we use ext4 and vfat filesystem (only).

> > Note: I wanted to optimize RAM space, so performance is not important
> > right now for our requirement.
> >
> > So, what are the options available, to perform this in 4.x kernel version.
> > My Kernel: 4.9.x
>
> Given that this is a greenfield development, why are you picking a kernel
> that's 2 years out of date?  You *do* realize that 4.9.135 does *not* contain
> all the bugfixes since then, only that relatively small subset that qualify for
> 'stable' (see Documentation/process/stable-kernel-rules.rst for the gory
> details).
>
Yes, we want to stick to 4.9 right now, as the end product might be
based on this version.
However, if higher kernel version have some fixes or good features, we
can back port it.

> One possible total hack would be to simply use a file-based swap area,
> but put the file on a filesystem that supports automatic inline compression.
>
I know, squashfs is a compressed filesystem, but it is read-only. So
its ruled out.

> Note that this will probably *totally* suck on performance, because there's
> no good way to find where 4K block 11,493 starts inside the compressed
> file, so it would have to read/decompress from the file beginning.  Also,
> if you write data to a previously unused location (or even a previously used
> spot that compressed the 4K page to a different length), you have a bad time
> inserting it.  (Note that zram can avoid most of this because it can (a) keep
> a table of pointers to where each page starts and (b) it isn't constrained to
> writing to 4K blocks on disk, so if the current compression takes a 4K page down
> to 1,283 bytes, it doesn't have to care *too* much if it stores that someplace
> that crosses a page boundary.
>
> Another thing that you will need to worry about is what happens in low-memory
> situations - the time you *most* need to do a swap operation, you may not have
> enough memory to do the I/O.  zram basically makes sure it *has* the memory
> needed beforehand, and swap directly to pre-allocated disk doesn't need much
> additional memory.
Swap storage requirement would be mostly between 64MB to 256MB (pre-configured).
Yes it can be something similar on ZRAM line, may be zram + zswap ?
Not sure if this right combination ?

Re: Creating compressed backing_store as swapfile

[Pintu Agarwal]

On Mon, Nov 5, 2018 at 9:37 PM Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
>
> On 11/5/2018 10:58 AM, Adam Borowski wrote:
> > On Mon, Nov 05, 2018 at 08:31:46PM +0530, Pintu Agarwal wrote:
> >> Hi,
> >>
> >> I have one requirement:
> >> I wanted to have a swapfile (64MB to 256MB) on my system.
> >> But I wanted the data to be compressed and stored on the disk in my swapfile.
> >> [Similar to zram, but compressed data should be moved to disk, instead of RAM].
> >>
> >> Note: I wanted to optimize RAM space, so performance is not important
> >> right now for our requirement.
> >>
> >> So, what are the options available, to perform this in 4.x kernel version.
> >> My Kernel: 4.9.x
> >> Board: any - (arm64 mostly).
> >>
> >> As I know, following are the choices:
> >> 1) ZRAM: But it compresses and store data in RAM itself
> >> 2) frontswap + zswap : Didn't explore much on this, not sure if this
> >> is helpful for our case.
> >> 3) Manually creating swapfile: but how to compress it ?
> >> 4) Any other options ?
> >
> > Loop device on any filesystem that can compress (such as btrfs)?  The
> > performance would suck, though -- besides the indirection of loop, btrfs
> > compresses in blocks of 128KB while swap wants 4KB writes.  Other similar
> > option is qemu-nbd -- it can use compressed disk images and expose them to a
> > (local) nbd client.
>
> Swap on any type of a networked storage device (NBD, iSCSI, ATAoE, etc)
> served from the local system is _really_ risky.  The moment the local
> server process for the storage device gets forced out to swap, you deadlock.
>
> Performance isn't _too_ bad for the BTRFS case though (I've actually
> tested this before), just make sure you disable direct I/O mode on the
> loop device, otherwise you run the risk of data corruption.

Sorry, btrfs is not an option for us. We want something more lighter
weight as our requirement is just < 200 MBs.