is there a COW inside the kernel ?

is there a COW inside the kernel ?

[roland]

hello !

is there an equivalent of something like

cowloop ( http://www.atconsultancy.nl/cowloop/total.html ) or md based cow 
device ( http://www.cl.cam.ac.uk/users/br260/doc/report.pdf ),

i.e. a feature called "Copy On Write Blockdevice" inside the current or the 
near-future mainline kernel (besides UserModeLinux Arch)?

i'm not sure - i think i remember having read that something like this can 
be probably done , but i don`t remember anymore what it was.

i would find this useful for several purpose, but i don`t want to patch my 
system with 3rd party drivers or "non-standard" stuff -  or even recompile 
the kernel.

can someone help out with some information ?

TIA

roland k.
system engineer 

Re: is there a COW inside the kernel ?

[Jan Engelhardt]

> is there an equivalent of something like
> cowloop ( http://www.atconsultancy.nl/cowloop/total.html ) or md based cow
> device ( http://www.cl.cam.ac.uk/users/br260/doc/report.pdf ),
> i.e. a feature called "Copy On Write Blockdevice" inside the current or the
> near-future mainline kernel (besides UserModeLinux Arch)?

Not directly block-device-based cow, but unionfs is at least a 
filesystem-based cow.


Jan Engelhardt
-- 

Re: is there a COW inside the kernel ?

[Alasdair G Kergon]

On Fri, Mar 03, 2006 at 09:29:02AM +0100, roland wrote:
> is there an equivalent of something like
> cowloop ( http://www.atconsultancy.nl/cowloop/total.html ) or md based cow 
> device ( http://www.cl.cam.ac.uk/users/br260/doc/report.pdf ),
> i.e. a feature called "Copy On Write Blockdevice" inside the current or the 
> near-future mainline kernel (besides UserModeLinux Arch)?
 
device-mapper snapshots?

  Documentation/device-mapper/snapshot.txt

Alasdair
-- 
agk@redhat.com

Re: is there a COW inside the kernel ?

[Kevin Corry]

On Fri March 3 2006 2:29 am, roland wrote:
> hello !
>
> is there an equivalent of something like
>
> cowloop ( http://www.atconsultancy.nl/cowloop/total.html ) or md based cow
> device ( http://www.cl.cam.ac.uk/users/br260/doc/report.pdf ),
>
> i.e. a feature called "Copy On Write Blockdevice" inside the current or the
> near-future mainline kernel (besides UserModeLinux Arch)?

Device-Mapper has a snapshot module, which is used by LVM and EVMS. You can 
also use dmsetup if you want lower-level access than provided by the volume 
managers. To do the equivalent of the cowloop driver that you linked to 
above, you could do something like this:

Say you have a read-only block-device (say a cd-rom) at /dev/hdc. And you have 
a small disk partition, /dev/hdb1, that you want to use for your "COW file". 
Run:

cow_size=`blockdev --getsize /dev/hdc`
chunk_size=64   # Size of each copied-on-write chunk, in 512 byte sectors
cow_name="my_cow_dev"
echo "0 $cow_size snapshot /dev/hdc /dev/hdb1 p $chunk_size" | \
   dmsetup create $cow_name

This will give you a device called /dev/mapper/$cow_name. Presuming /dev/hdc 
has a filesystem on it, you can mount /dev/mapper/$cow_name and get a 
read-write version of the filesystem on /dev/hdc, where updates to the 
filesystem will be stored on /dev/hdb1. The size of /dev/hdb1 can be 
significantly smaller than /dev/hdc, depending on the amount of writes you 
expect to happen on /dev/mapper/$cow_name. While this device is active, don't 
try to mount /dev/hdc read-write (assuming that's possible), or it will 
corrupt the view of /dev/mapper/$cow_name. If you need read-write access to 
both devices simultaneously, you'll probably just want to use LVM or EVMS and 
create snapshot volumes, since manually activating that kind of setup with 
dmsetup is incredibly tricky.

Use "dmsetup remove $cow_name" to deactivate the device.

> i would find this useful for several purpose, but i don`t want to patch my
> system with 3rd party drivers or "non-standard" stuff -  or even recompile
> the kernel.

This should work with any recent 2.6 kernel. You'll also need to have the 
device-mapper package installed, which should be available with any recent 
Linux distro.

-- 
Kevin Corry
kevcorry@us.ibm.com
http://www.ibm.com/linux/
http://evms.sourceforge.net/

Re: is there a COW inside the kernel ?

[Jeff Dike]

On Fri, Mar 03, 2006 at 09:29:02AM +0100, roland wrote:
> hello !
> 
> is there an equivalent of something like
> 
> cowloop ( http://www.atconsultancy.nl/cowloop/total.html ) or md based cow 
> device ( http://www.cl.cam.ac.uk/users/br260/doc/report.pdf ),
> 
> i.e. a feature called "Copy On Write Blockdevice" inside the current or the 
> near-future mainline kernel (besides UserModeLinux Arch)?
> can someone help out with some information ?

Miklos Szeredi announced mountlo a few days ago - this uses a UML to
export a filesystem to the host through FUSE.  It's intended to allow
non-privileged loopback mounting of normal file system images, but
presumably will export a COW block device as well.

I'm doing something similar, and using FUSE to export the entire UML
filesystem to the host.

These aren't specifically COW drivers, but they have the same effect
as long as you have a UML with your COW device mounted.

				Jeff

Re: is there a COW inside the kernel ?

[roland]

hello !

thanks to all for providing that information.

i think i will take a closer look on device-mapper, but i'm unsure if it`s 
perfectly suited.

can i only use devices, not files for the cow?
what about merging a cow-dev/file back to the r/o-dev/file ?
cowloop can do this, and because it can use files, i don`t need to provide a 
fixed amount of diskspace for the cow.  you have direct feedback about how 
big you cow grows....
i'm unsure about dm, but i will dig into the details. (wanted to learn more 
about dm anyway)

any chance of cowloop being merged into mainline ?
loop.c is 1343 lines of code, cowloop.c is around 1000 lines more so it`s 
"reasonable" small....and it has a _really_ nice user manual.

regards
roland

ps:
btw - mountlo looks really "freaky" - i like uml very much, but i think i 
need each bit of performance.
will take a look at mountlo just for personal interest, though.




----- Original Message ----- 
From: "Kevin Corry" <kevcorry@us.ibm.com>
To: <linux-kernel@vger.kernel.org>
Cc: "roland" <devzero@web.de>
Sent: Friday, March 03, 2006 3:28 PM
Subject: Re: is there a COW inside the kernel ?


> On Fri March 3 2006 2:29 am, roland wrote:
>> hello !
>>
>> is there an equivalent of something like
>>
>> cowloop ( http://www.atconsultancy.nl/cowloop/total.html ) or md based 
>> cow
>> device ( http://www.cl.cam.ac.uk/users/br260/doc/report.pdf ),
>>
>> i.e. a feature called "Copy On Write Blockdevice" inside the current or 
>> the
>> near-future mainline kernel (besides UserModeLinux Arch)?
>
> Device-Mapper has a snapshot module, which is used by LVM and EVMS. You 
> can
> also use dmsetup if you want lower-level access than provided by the 
> volume
> managers. To do the equivalent of the cowloop driver that you linked to
> above, you could do something like this:
>
> Say you have a read-only block-device (say a cd-rom) at /dev/hdc. And you 
> have
> a small disk partition, /dev/hdb1, that you want to use for your "COW 
> file".
> Run:
>
> cow_size=`blockdev --getsize /dev/hdc`
> chunk_size=64   # Size of each copied-on-write chunk, in 512 byte sectors
> cow_name="my_cow_dev"
> echo "0 $cow_size snapshot /dev/hdc /dev/hdb1 p $chunk_size" | \
>   dmsetup create $cow_name
>
> This will give you a device called /dev/mapper/$cow_name. Presuming 
> /dev/hdc
> has a filesystem on it, you can mount /dev/mapper/$cow_name and get a
> read-write version of the filesystem on /dev/hdc, where updates to the
> filesystem will be stored on /dev/hdb1. The size of /dev/hdb1 can be
> significantly smaller than /dev/hdc, depending on the amount of writes you
> expect to happen on /dev/mapper/$cow_name. While this device is active, 
> don't
> try to mount /dev/hdc read-write (assuming that's possible), or it will
> corrupt the view of /dev/mapper/$cow_name. If you need read-write access 
> to
> both devices simultaneously, you'll probably just want to use LVM or EVMS 
> and
> create snapshot volumes, since manually activating that kind of setup with
> dmsetup is incredibly tricky.
>
> Use "dmsetup remove $cow_name" to deactivate the device.
>
>> i would find this useful for several purpose, but i don`t want to patch 
>> my
>> system with 3rd party drivers or "non-standard" stuff -  or even 
>> recompile
>> the kernel.
>
> This should work with any recent 2.6 kernel. You'll also need to have the
> device-mapper package installed, which should be available with any recent
> Linux distro.
>
> -- 
> Kevin Corry
> kevcorry@us.ibm.com
> http://www.ibm.com/linux/
> http://evms.sourceforge.net/ 

Re: is there a COW inside the kernel ?

[Kevin Corry]

On Fri March 3 2006 3:29 pm, roland wrote:
> i think i will take a closer look on device-mapper, but i'm unsure if it`s
> perfectly suited.
>
> can i only use devices, not files for the cow?

Yes, Device-Mapper can only map to block-devices. If you need to do this with 
files, you could use losetup to create block-device from them. However, the 
COW device still won't "grow" automatically as you described.

> what about merging a cow-dev/file back to the r/o-dev/file ?

Device-Mapper does not directly support this, but EVMS provides a "rollback" 
function for reverting an origin volume back to the contents of its snapshot 
volume.

-- 
Kevin Corry
kevcorry@us.ibm.com
http://www.ibm.com/linux/
http://evms.sourceforge.net/

Re: is there a COW inside the kernel ?

[Jon Masters]

On 3/3/06, roland <devzero@web.de> wrote:

> i think i will take a closer look on device-mapper, but i'm unsure if it`s
> perfectly suited.

It looks to me that you want to use something more like unionfs with
COW/whiteout mode - there's an implementation based on FUSE available
but I've not really looked at it. Obviously you might just want
something in mainline, in which case you're limited to the FUSE idea
or one of the others already suggested.

> what about merging a cow-dev/file back to the r/o-dev/file ?

This is why I suggest you look at the above.

Jon.