Encryption

Encryption

[Pierre Abbat]

How is filesystem encryption going to work? Or has anyone figured that out 
yet?

phma
-- 
.i toljundi do .ibabo mi'afra tu'a do
.ibabo damba do .ibabo do jinga
.icu'u la ma'atman.

Re: Encryption

[Yury Umanets]

Pierre Abbat wrote:

>How is filesystem encryption going to work? Or has anyone figured that out 
>yet?
>
>phma
>  
>
Don't worry, it is going up :)

-- 
Yury Umanets

Snapshots a la NetApp?

[kend]

I'm just wondering if there's any development being done on NetApp-like
snapshots.  (This differs from LVM snapshots in that it's file-by-file,
instead of volume based.)  If not, has anyone given it any consideration? 
It would be a huge win for the RAID-in-a-box folks, and, speaking as a
sysadmin, is something about which I dream frequently.

Just curious...

Ken D'Ambrosio
Sr. SysAdmin,
Xanoptix, Inc.

Re: Snapshots a la NetApp?

[Heinz-Josef Claes]

Am Son, 2003-03-23 um 04.38 schrieb kend@xanoptix.com:
> I'm just wondering if there's any development being done on NetApp-like
> snapshots.  (This differs from LVM snapshots in that it's file-by-file,
> instead of volume based.)  If not, has anyone given it any consideration? 
> It would be a huge win for the RAID-in-a-box folks, and, speaking as a
> sysadmin, is something about which I dream frequently.
> 
> Just curious...
> 
> Ken D'Ambrosio
> Sr. SysAdmin,
> Xanoptix, Inc.

Hi,
you can look at the URL below. It's a kind of snapshot inspired by
NetApp, but has nothing to do with LVM or the filesystem. It also has a
different behaviour.

-- 
Heinz-Josef Claes                     hjclaes@web.de
project: http://sourceforge.net/projects/storebackup
      -> snapshot-like backup to another disk

Re: Snapshots a la NetApp?

[Lars Marowsky-Bree]

On 2003-03-22T22:38:01,
   kend@xanoptix.com said:

> I'm just wondering if there's any development being done on NetApp-like
> snapshots.  (This differs from LVM snapshots in that it's file-by-file,
> instead of volume based.)  If not, has anyone given it any consideration? 
> It would be a huge win for the RAID-in-a-box folks, and, speaking as a
> sysadmin, is something about which I dream frequently.

Well, implementing it at the LVM level isn't entirely bad though. What kind of
advantage would have implementing at the filesystem level have? It would need
to compensate at least the disadvantage of being less general.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur

Re: Snapshots a la NetApp?

[Oleg Drokin]

Hello!

On Sat, Mar 22, 2003 at 10:38:01PM -0500, kend@xanoptix.com wrote:
> I'm just wondering if there's any development being done on NetApp-like
> snapshots.  (This differs from LVM snapshots in that it's file-by-file,
> instead of volume based.)  If not, has anyone given it any consideration? 
> It would be a huge win for the RAID-in-a-box folks, and, speaking as a
> sysadmin, is something about which I dream frequently.

That sounds like file versioning on filesystem, perhaps
filesystem (binary only and stuff) from ClearCase suite (or what was the name)
from Rational have this.
Though I have not tried it myself.
And their http://www.rational.com does not answers. May be related to the fact
that they were bought by IBM recently ;)

Bye,
    Oleg

Re: Snapshots a la NetApp?

[Ragnar Kjørstad]

On Sun, Mar 23, 2003 at 12:18:31PM +0100, Lars Marowsky-Bree wrote:
> > I'm just wondering if there's any development being done on NetApp-like
> > snapshots.  (This differs from LVM snapshots in that it's file-by-file,
> > instead of volume based.)  If not, has anyone given it any consideration? 
> > It would be a huge win for the RAID-in-a-box folks, and, speaking as a
> > sysadmin, is something about which I dream frequently.
> 
> Well, implementing it at the LVM level isn't entirely bad though. What kind of
> advantage would have implementing at the filesystem level have? It would need
> to compensate at least the disadvantage of being less general.

I think the main advantage of having it in the filesystem is that it
makes the snapshots available to the users. This makes it possible for
regular users to undelete / restore files without having to contact the
sysadmin. 

As for linux implementations of NetApp-like snapshots you should check
out snapfs: 

http://www.mountainviewdata.com/us/product/product_snap_1.html
http://www-mddsp.enel.ucalgary.ca/People/adilger/snapfs/
http://sourcefrog.net/projects/snapfs/


-- 
Ragnar Kjørstad
Zet.no

Re: Encryption

[Hans Reiser]

Yury Umanets wrote:

> Pierre Abbat wrote:
>
>> How is filesystem encryption going to work? Or has anyone figured 
>> that out yet?
>>
>> phma
>>  
>>
> Don't worry, it is going up :)
>
Edward, please provide a serious answer including your design documents, 
so that the list can review your design and comment.  Chances are that 
they will make at least one serious improvement to the design..... and 
it might be nice to get that improvement made before you have made the 
time investment required to debug.....

-- 
Hans

Re: Snapshots a la NetApp?

[Hans Reiser]

Heinz-Josef Claes wrote:

>Am Son, 2003-03-23 um 04.38 schrieb kend@xanoptix.com:
>  
>
>>I'm just wondering if there's any development being done on NetApp-like
>>snapshots.  (This differs from LVM snapshots in that it's file-by-file,
>>instead of volume based.)  If not, has anyone given it any consideration? 
>>It would be a huge win for the RAID-in-a-box folks, and, speaking as a
>>sysadmin, is something about which I dream frequently.
>>
>>Just curious...
>>
>>Ken D'Ambrosio
>>Sr. SysAdmin,
>>Xanoptix, Inc.
>>    
>>
>
>Hi,
>you can look at the URL below. It's a kind of snapshot inspired by
>NetApp, but has nothing to do with LVM or the filesystem. It also has a
>different behaviour.
>
>  
>
I didn't find the magic click that gave me the design doc for 
storebackup, could you post it to the list?

-- 
Hans

Re: Snapshots a la NetApp?

[Heinz-Josef Claes]

Am Mon, 2003-03-24 um 21.49 schrieb Hans Reiser:
> Heinz-Josef Claes wrote:
> 
> >Am Son, 2003-03-23 um 04.38 schrieb kend@xanoptix.com:
> >  
> >
> >>I'm just wondering if there's any development being done on NetApp-like
> >>snapshots.  (This differs from LVM snapshots in that it's file-by-file,
> >>instead of volume based.)  If not, has anyone given it any consideration? 
> >>It would be a huge win for the RAID-in-a-box folks, and, speaking as a
> >>sysadmin, is something about which I dream frequently.
> >>
> >>Just curious...
> >>
> >>Ken D'Ambrosio
> >>Sr. SysAdmin,
> >>Xanoptix, Inc.
> >>    
> >>
> >
> >Hi,
> >you can look at the URL below. It's a kind of snapshot inspired by
> >NetApp, but has nothing to do with LVM or the filesystem. It also has a
> >different behaviour.
> >
> >  
> >
> I didn't find the magic click that gave me the design doc for 
> storebackup, could you post it to the list?

There is no design doc, because it's principal design is so simple :-).
Here is an excerpt from the README file include in the tar file at
sourceforge:

GENERAL FUNCTIONALITY
---------------------

storeBackup is a disk-to-disk backup tool for Linux. It should run on
other Unix like machines. You can directly browse through the backuped
files (locally, via NFS, via SAMBA or whatever). This gives the users
the possibility to restore files absolutely easily and fast. He/She
only has to copy (and possibly uncompress) the file. There is also a
tool for easily restoring (sub) trees for the administrator. Every
single backup of a specific time can be deleted without affecting the
other existing backups.


HOW DOES IT WORK?
-----------------

storeBackup makes a backup of a directory to another direct reachable
location. It does not care about where this location is (same disk,
another disc, via NFS over the network). You should use another disk
or better another computer to store the backup. The target directory
must be on a Unix virtual file system which supports hard links. So
backing up to a SAMBA share is not possible. Naturally, you can also
mount the source directory via NFS and backup in a local
filesystem. In this case, it's good to have a fast network.  The
backup(s) can be seen in a directory in the form date_time
(yyyy.mm.dd_hh.mm.ss) which it creates.

Implemented are several optimizations to reduce disk usage:

- The files to backup are compressed (default bzip2) as discrete files
  in the backup. Files with definable suffixes (like .gz, which is part
  of the default value) will not be compressed. It is possible to
  avoid compression in general.

- If a file with the same contents exists in the previous backup, the
  new backup will only be a hard link to the other one. (This
  mechanism depends on the contents, not on a file name or path!) If you
  rename a file or directory or move sub trees around, it will not cost
  additional space in the backup.

- You can also check older backups than the last one for files with
  the same contents. But this is normally not worth the effort. You
  can also check backups of *other* machines (or backup series)
  for files with the same contents, which can be very efficient.

- If a file with the same contents exists in the same tree to back up,
  it will be hard linked to the other one (and naturally to the older
  ones).

As a result, only changes resulting in different files were stored
(compressed) and require disk space. Normally, the required disk space
for one backup a day for 30 days is less then the required space for the
original. But this depends on the number of changes.


There are several optimizations to improve performance. Normally, the
first backup is much slower than the followings, because all the data
has to be compressed and/or copied. storeBackup is able to take
advantage of multiprocessor machines. From the second run, it should not
be a problem to get more than 100 files/sec with a normal machine of
today. This does not depend on the file size.

There is are special files in the root of the created backup called
.md5CheckSums.info and .md5CheckSums or .md5CheckSums.bz2
(default). You must not delete these files. They contain all
information about the original files. You can use this information to
write you own tools beside the existing to restore or to analyze the
backups.

When started, storeBackup will read .md5CheckSums and creates its own
databases (dbm file) in $TMPDIR or --tmpdir (default is /tmp). If you
back up a large number of files (some millions), the required space can
be several dozens of megabytes. If you do not have enough memory to
cache the dbm file, I recommend using a separate hard disk (if
available) for better performance.


LIMITATIONS
-----------

- storeBackup can backup normal files, directories, symbolic links and
  named pipes. Other file types are not supported up to now and will
  generate a warning.

- The permissions in the backup tree(s) are equal to the permissions
  in the original directory. Under special rare conditions it is
  possible, that a user cannot read one ore more of own his/her files
  in the backup. With the restore tool - storeBackupRecover.pl -
  everything is restored with the original permissions.

- storeBackup uses hard links to save disk space. Linux with ext2 file
  system supports up to 32000, reiserfs up to 64535 hard links. If
  storeBackup needs more hard links, it will write a warning and store
  a new (compressed) copy of the file. If you use ext2 for the backup,
  you have to reserve enough (static) inodes!



Hope this helps. As I wrote, it's not depending on the filesystem. It's
depending on files, while NetApp is depending on blocks. Making a
snapshot of a big database file with NetApp is a good idea. Making
backups of big database files with storebackup costs you a lot of space.

On the other side, if you change a normal text file, the block related
algorithm of NetApp will not save much blocks. Compressing the whole
file is much more efficient. And duplicated files, with exist in a great
number in a "normal" file system containing the data of many users (I
also didn't believe this :-)) are saved only once in the backup.

If you want to backup big DB files on linux, there's a better tool for
this use case which uses the algorithm of rsync and stores the original
file and a series of deltas. If I remember well it's called rdiff.
(Sorry, but I never used it.)

-- 
Heinz-Josef Claes                     hjclaes@web.de
project: http://sourceforge.net/projects/storebackup
      -> snapshot-like backup to another disk

Re: Snapshots a la NetApp?

[Tom Vier]

fwiw, there's pdumpfs, which does much the same thing. it's much like
plan9's dump (which writes snapshots to worm media). pdumpfs has worked
great for me. i just use it for my home directory. note that many programs
don't handle hardlinks well. specifically tar, cpio, and rsync (the last two
use TONS of memory even if you have just a few months worth of snapshots).

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0xE6CB97DA

Re: Snapshots a la NetApp?

[Hans Reiser]

Heinz-Josef Claes wrote:

>Am Mon, 2003-03-24 um 21.49 schrieb Hans Reiser:
>  
>
>>Heinz-Josef Claes wrote:
>>
>>    
>>
>>>Am Son, 2003-03-23 um 04.38 schrieb kend@xanoptix.com:
>>> 
>>>
>>>      
>>>
>>>>I'm just wondering if there's any development being done on NetApp-like
>>>>snapshots.  (This differs from LVM snapshots in that it's file-by-file,
>>>>instead of volume based.)  If not, has anyone given it any consideration? 
>>>>It would be a huge win for the RAID-in-a-box folks, and, speaking as a
>>>>sysadmin, is something about which I dream frequently.
>>>>
>>>>Just curious...
>>>>
>>>>Ken D'Ambrosio
>>>>Sr. SysAdmin,
>>>>Xanoptix, Inc.
>>>>   
>>>>
>>>>        
>>>>
>>>Hi,
>>>you can look at the URL below. It's a kind of snapshot inspired by
>>>NetApp, but has nothing to do with LVM or the filesystem. It also has a
>>>different behaviour.
>>>
>>> 
>>>
>>>      
>>>
>>I didn't find the magic click that gave me the design doc for 
>>storebackup, could you post it to the list?
>>    
>>
>
>There is no design doc, because it's principal design is so simple :-).
>Here is an excerpt from the README file include in the tar file at
>sourceforge:
>
>GENERAL FUNCTIONALITY
>---------------------
>
>storeBackup is a disk-to-disk backup tool for Linux. It should run on
>other Unix like machines. You can directly browse through the backuped
>files (locally, via NFS, via SAMBA or whatever). This gives the users
>the possibility to restore files absolutely easily and fast. He/She
>only has to copy (and possibly uncompress) the file. There is also a
>tool for easily restoring (sub) trees for the administrator. Every
>single backup of a specific time can be deleted without affecting the
>other existing backups.
>
>
>HOW DOES IT WORK?
>-----------------
>
>storeBackup makes a backup of a directory to another direct reachable
>location. It does not care about where this location is (same disk,
>another disc, via NFS over the network). You should use another disk
>or better another computer to store the backup. The target directory
>must be on a Unix virtual file system which supports hard links. So
>backing up to a SAMBA share is not possible. Naturally, you can also
>mount the source directory via NFS and backup in a local
>filesystem. In this case, it's good to have a fast network.  The
>backup(s) can be seen in a directory in the form date_time
>(yyyy.mm.dd_hh.mm.ss) which it creates.
>
>Implemented are several optimizations to reduce disk usage:
>
>- The files to backup are compressed (default bzip2) as discrete files
>  in the backup. Files with definable suffixes (like .gz, which is part
>  of the default value) will not be compressed. It is possible to
>  avoid compression in general.
>
>- If a file with the same contents exists in the previous backup, the
>  new backup will only be a hard link to the other one. (This
>  mechanism depends on the contents, not on a file name or path!) If you
>  rename a file or directory or move sub trees around, it will not cost
>  additional space in the backup.
>
>- You can also check older backups than the last one for files with
>  the same contents. But this is normally not worth the effort. You
>  can also check backups of *other* machines (or backup series)
>  for files with the same contents, which can be very efficient.
>
>- If a file with the same contents exists in the same tree to back up,
>  it will be hard linked to the other one (and naturally to the older
>  ones).
>
>As a result, only changes resulting in different files were stored
>(compressed) and require disk space. Normally, the required disk space
>for one backup a day for 30 days is less then the required space for the
>original. But this depends on the number of changes.
>
>
>There are several optimizations to improve performance. Normally, the
>first backup is much slower than the followings, because all the data
>has to be compressed and/or copied. storeBackup is able to take
>advantage of multiprocessor machines. From the second run, it should not
>be a problem to get more than 100 files/sec with a normal machine of
>today. This does not depend on the file size.
>
>There is are special files in the root of the created backup called
>.md5CheckSums.info and .md5CheckSums or .md5CheckSums.bz2
>(default). You must not delete these files. They contain all
>information about the original files. You can use this information to
>write you own tools beside the existing to restore or to analyze the
>backups.
>
>When started, storeBackup will read .md5CheckSums and creates its own
>databases (dbm file) in $TMPDIR or --tmpdir (default is /tmp). If you
>back up a large number of files (some millions), the required space can
>be several dozens of megabytes. If you do not have enough memory to
>cache the dbm file, I recommend using a separate hard disk (if
>available) for better performance.
>
>
>LIMITATIONS
>-----------
>
>- storeBackup can backup normal files, directories, symbolic links and
>  named pipes. Other file types are not supported up to now and will
>  generate a warning.
>
>- The permissions in the backup tree(s) are equal to the permissions
>  in the original directory. Under special rare conditions it is
>  possible, that a user cannot read one ore more of own his/her files
>  in the backup. With the restore tool - storeBackupRecover.pl -
>  everything is restored with the original permissions.
>
>- storeBackup uses hard links to save disk space. Linux with ext2 file
>  system supports up to 32000, reiserfs up to 64535 hard links. If
>  storeBackup needs more hard links, it will write a warning and store
>  a new (compressed) copy of the file. If you use ext2 for the backup,
>  you have to reserve enough (static) inodes!
>
>
>
>Hope this helps. As I wrote, it's not depending on the filesystem. It's
>depending on files, while NetApp is depending on blocks. Making a
>snapshot of a big database file with NetApp is a good idea. Making
>backups of big database files with storebackup costs you a lot of space.
>
>On the other side, if you change a normal text file, the block related
>algorithm of NetApp will not save much blocks. Compressing the whole
>file is much more efficient. And duplicated files, with exist in a great
>number in a "normal" file system containing the data of many users (I
>also didn't believe this :-)) are saved only once in the backup.
>
>If you want to backup big DB files on linux, there's a better tool for
>this use case which uses the algorithm of rsync and stores the original
>file and a series of deltas. If I remember well it's called rdiff.
>(Sorry, but I never used it.)
>
>  
>
Flx, should we try this for our backups?

-- 
Hans

Re: unsubscribe

[Voicu Liviu]

unsubscribe

Re: Encryption

[Edward Shushkin]

Hans Reiser wrote:
> 
> Yury Umanets wrote:
> 
> > Pierre Abbat wrote:
> >
> >> How is filesystem encryption going to work? Or has anyone figured
> >> that out yet?
> >>
> >> phma
> >>
> >>
> > Don't worry, it is going up :)
> >
> Edward, please provide a serious answer including your design documents,
> so that the list can review your design and comment. 
> Chances are that
> they will make at least one serious improvement to the design...

Ok, it will be very nice.
Edward.

.. and
> it might be nice to get that improvement made before you have made the
> time investment required to debug.....
> 
> --
> Hans

Re: Snapshots a la NetApp?

[Heinz-Josef Claes]

Am Die, 2003-03-25 um 03.35 schrieb Hans Reiser:
> Heinz-Josef Claes wrote:
> 
> >Am Mon, 2003-03-24 um 21.49 schrieb Hans Reiser:
> >  
> >
> >>Heinz-Josef Claes wrote:
> >>
> >>    
> >>
> >>>Am Son, 2003-03-23 um 04.38 schrieb kend@xanoptix.com:
> >>> 
> >>>
> >>>      
> >>>
> >>>>I'm just wondering if there's any development being done on NetApp-like
> >>>>snapshots.  (This differs from LVM snapshots in that it's file-by-file,
> >>>>instead of volume based.)  If not, has anyone given it any consideration? 
> >>>>It would be a huge win for the RAID-in-a-box folks, and, speaking as a
> >>>>sysadmin, is something about which I dream frequently.
> >>>>
> >>>>Just curious...
> >>>>
> >>>>Ken D'Ambrosio
> >>>>Sr. SysAdmin,
> >>>>Xanoptix, Inc.
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>Hi,
> >>>you can look at the URL below. It's a kind of snapshot inspired by
> >>>NetApp, but has nothing to do with LVM or the filesystem. It also has a
> >>>different behaviour.
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >>I didn't find the magic click that gave me the design doc for 
> >>storebackup, could you post it to the list?
> >>    
> >>
> >
> >There is no design doc, because it's principal design is so simple :-).
> >Here is an excerpt from the README file include in the tar file at
> >sourceforge:
> >
> >GENERAL FUNCTIONALITY
> >---------------------
> >
> >storeBackup is a disk-to-disk backup tool for Linux. It should run on
> >other Unix like machines. You can directly browse through the backuped
> >files (locally, via NFS, via SAMBA or whatever). This gives the users
> >the possibility to restore files absolutely easily and fast. He/She
> >only has to copy (and possibly uncompress) the file. There is also a
> >tool for easily restoring (sub) trees for the administrator. Every
> >single backup of a specific time can be deleted without affecting the
> >other existing backups.
> >
> >
> >HOW DOES IT WORK?
> >-----------------
> >
> >storeBackup makes a backup of a directory to another direct reachable
> >location. It does not care about where this location is (same disk,
> >another disc, via NFS over the network). You should use another disk
> >or better another computer to store the backup. The target directory
> >must be on a Unix virtual file system which supports hard links. So
> >backing up to a SAMBA share is not possible. Naturally, you can also
> >mount the source directory via NFS and backup in a local
> >filesystem. In this case, it's good to have a fast network.  The
> >backup(s) can be seen in a directory in the form date_time
> >(yyyy.mm.dd_hh.mm.ss) which it creates.
> >
> >Implemented are several optimizations to reduce disk usage:
> >
> >- The files to backup are compressed (default bzip2) as discrete files
> >  in the backup. Files with definable suffixes (like .gz, which is part
> >  of the default value) will not be compressed. It is possible to
> >  avoid compression in general.
> >
> >- If a file with the same contents exists in the previous backup, the
> >  new backup will only be a hard link to the other one. (This
> >  mechanism depends on the contents, not on a file name or path!) If you
> >  rename a file or directory or move sub trees around, it will not cost
> >  additional space in the backup.
> >
> >- You can also check older backups than the last one for files with
> >  the same contents. But this is normally not worth the effort. You
> >  can also check backups of *other* machines (or backup series)
> >  for files with the same contents, which can be very efficient.
> >
> >- If a file with the same contents exists in the same tree to back up,
> >  it will be hard linked to the other one (and naturally to the older
> >  ones).
> >
> >As a result, only changes resulting in different files were stored
> >(compressed) and require disk space. Normally, the required disk space
> >for one backup a day for 30 days is less then the required space for the
> >original. But this depends on the number of changes.
> >
> >
> >There are several optimizations to improve performance. Normally, the
> >first backup is much slower than the followings, because all the data
> >has to be compressed and/or copied. storeBackup is able to take
> >advantage of multiprocessor machines. From the second run, it should not
> >be a problem to get more than 100 files/sec with a normal machine of
> >today. This does not depend on the file size.
> >
> >There is are special files in the root of the created backup called
> >.md5CheckSums.info and .md5CheckSums or .md5CheckSums.bz2
> >(default). You must not delete these files. They contain all
> >information about the original files. You can use this information to
> >write you own tools beside the existing to restore or to analyze the
> >backups.
> >
> >When started, storeBackup will read .md5CheckSums and creates its own
> >databases (dbm file) in $TMPDIR or --tmpdir (default is /tmp). If you
> >back up a large number of files (some millions), the required space can
> >be several dozens of megabytes. If you do not have enough memory to
> >cache the dbm file, I recommend using a separate hard disk (if
> >available) for better performance.
> >
> >
> >LIMITATIONS
> >-----------
> >
> >- storeBackup can backup normal files, directories, symbolic links and
> >  named pipes. Other file types are not supported up to now and will
> >  generate a warning.
> >
> >- The permissions in the backup tree(s) are equal to the permissions
> >  in the original directory. Under special rare conditions it is
> >  possible, that a user cannot read one ore more of own his/her files
> >  in the backup. With the restore tool - storeBackupRecover.pl -
> >  everything is restored with the original permissions.
> >
> >- storeBackup uses hard links to save disk space. Linux with ext2 file
> >  system supports up to 32000, reiserfs up to 64535 hard links. If
> >  storeBackup needs more hard links, it will write a warning and store
> >  a new (compressed) copy of the file. If you use ext2 for the backup,
> >  you have to reserve enough (static) inodes!
> >
> >
> >
> >Hope this helps. As I wrote, it's not depending on the filesystem. It's
> >depending on files, while NetApp is depending on blocks. Making a
> >snapshot of a big database file with NetApp is a good idea. Making
> >backups of big database files with storebackup costs you a lot of space.
> >
> >On the other side, if you change a normal text file, the block related
> >algorithm of NetApp will not save much blocks. Compressing the whole
> >file is much more efficient. And duplicated files, with exist in a great
> >number in a "normal" file system containing the data of many users (I
> >also didn't believe this :-)) are saved only once in the backup.
> >
> >If you want to backup big DB files on linux, there's a better tool for
> >this use case which uses the algorithm of rsync and stores the original
> >file and a series of deltas. If I remember well it's called rdiff.
> >(Sorry, but I never used it.)
> >
> >  
> >
> Flx, should we try this for our backups?

If you test it, I would be glad to hear about your experiences, because
I believe that you must be a stress tester by birth :-)

btw: The only handycap of the used method is that all the links to a
file can have only *one* set of permissions (chmod, chown). Will it be
possible to change this with a loadable module in reiser4?

best regards,
HJC

Re: Snapshots a la NetApp?

[Hans Reiser]

Heinz-Josef Claes wrote:

>
>btw: The only handycap of the used method is that all the links to a
>file can have only *one* set of permissions (chmod, chown). Will it be
>possible to change this with a loadable module in reiser4?
>
>best regards,
>HJC
>
>
>
>  
>
I am not sure if we will support that.  If someone wanted to write the 
code to do it, then I could examine the problem more closely and it 
could be done.....

-- 
Hans

Re: Snapshots a la NetApp?

[Heinz-Josef Claes]

Am Die, 2003-03-25 um 01.03 schrieb Tom Vier:
> fwiw, there's pdumpfs, which does much the same thing. it's much like
> plan9's dump (which writes snapshots to worm media). pdumpfs has worked
> great for me. i just use it for my home directory. note that many programs
> don't handle hardlinks well. specifically tar, cpio, and rsync (the last two
> use TONS of memory even if you have just a few months worth of snapshots).

Thanks for the hint - I've never heard before about pdumpfs. I wrote storeBackup
because I could not find such a tool.

A very basic idea of both progs are the same: to use hard links. But the
implementation differs.

The difference is, that pdumpfs depends on paths and file names.
StoreBackup depends on the *contents* of the files, compresses optional
and can share files via hard links between independent backups of different
trees (computers).

The parameters are:
        storeBackup.pl -f configFile [-g | --print]
or
        storeBackup.pl -s sourceDir -t targetDir [-T tmpdir] [-L lockFile]
                [--exceptDirs dir1,dir2,dir3] [--exceptDirsSep sep]
                [--precommand job] [--followLinks depth]
                [-c compress] [-u uncompress] [-p postfix]
                [--noCompress number] [--queueCompress number]
                [--noCopy number] [--queueCopy number]
                [--withUserGroupStat] [--userGroupStatFile filename]
                [--exceptSuffix suffixes]
                [--addExceptSuffix suffixes] [--contExceptDirsErr]
                [--compressMD5File yes|no] [--chmodMD5File] [-v] [-d level]
                [--keepAll timescale] [--keepWeekday entry]
                [--keepMinNumber number] [--keepOnlyLastOfDay timescale]
                [--nodelete] [--progressReport number] [--printDepth]
                [--ignoreReadError]
                [-l logFile
                 [--plusLogStdout] [--withTime yes|no] [-m maxFilelen]
                 [[-n noOfOldFiles] | [--saveLogs yes|no]
                 [--compressWith compressprog]]
                [--logInBackupDir yes|no [--compressLogInBackupDir yes|no]
                 [--logInBackupDirFileName logFile]]
                [otherBackupDirs ...]

best regards,

-- 
Heinz-Josef Claes                     hjclaes@web.de
project: http://sourceforge.net/projects/storebackup
      -> snapshot-like backup to another disk

Re: Snapshots a la NetApp?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

On Wed, 26 Mar 2003 21:37:10 +0300, Hans Reiser said:
> Heinz-Josef Claes wrote:
> >btw: The only handycap of the used method is that all the links to a
> >file can have only *one* set of permissions (chmod, chown). Will it be
> >possible to change this with a loadable module in reiser4?

> I am not sure if we will support that.  If someone wanted to write the 
> code to do it, then I could examine the problem more closely and it 
> could be done.....

Please don't.

There is just *WAY* too much Unix/Linux semantics that are dependent on the
concept that an inode describes a file, with all it implies - the fact that all
hard links have the same owner/perms (hint - how would you store 2 inodes that
had different owner/perm/mtime/ctime/etc but the same block list, and still
have it fsck correctly?  If you have a linked list of owners/perms, how do you
determine which one to use? etc etc..)  I'm not even going into the aspects
like programs that creat() and then unlink() in order to get a anonymous temp
file that goes away at last close, etc etc....

You want different permissions, use ACLs.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: encryption

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Mon, Feb 16, 2015 at 06:19:04PM +0100, Henry Noack wrote:
>> it is possible to decrypt a kvm volume only by using the command line after
>> starting it?
>
> Encryption can be done at 3 levels:
[...]
> 2. In QEMU with qcow2, although this feature is not widely used and not
>    up to modern disk encryption standards.

Quoting the fine manual:

          The use of encryption in qcow and qcow2 images is considered
          to be flawed by modern cryptography standards, suffering from
          a number of design problems:

             − The AES-CBC cipher is used with predictable
               initialization vectors based on the sector number.  This
               makes it vulnerable to chosen plaintext attacks which can
               reveal the existence of encrypted data.
             − The user passphrase is directly used as the encryption
               key.  A poorly chosen or short passphrase will compromise
               the security of the encryption.
             − In the event of the passphrase being compromised there is
               no way to change the passphrase to protect data in any
               qcow images.  The files must be cloned, using a different
               encryption passphrase in the new file.  The original file
               must then be securely erased using a program like shred,
               though even this is ineffective with many modern storage
               technologies.

          Use of qcow / qcow2 encryption is thus strongly discouraged.
          Users are recommended to use an alternative encryption
          technology such as the Linux dm-crypt / LUKS system.

[...]

Re: encryption

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Mon, Feb 16, 2015 at 06:19:04PM +0100, Henry Noack wrote:
> it is possible to decrypt a kvm volume only by using the command line after
> starting it?

Encryption can be done at 3 levels:

1. Inside the guest.  Just like you do on a physical machine with LUKS
   (dm-crypt), ecryptfs, TrueCrypt, etc.

2. In QEMU with qcow2, although this feature is not widely used and not
   up to modern disk encryption standards.

3. On the host using LUKS (dm-crypt), ecryptfs, TrueCrypt, etc or on the
   storage appliance.

It depends what you are trying to achieve.

Keep in mind that encrypting the disk image does not stop the host from
seeing inside the guest.  The host is always trusted, today's
virtualization technology has this limitation.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

encryption

[Henry Noack]

Hello you guys,


it is possible to decrypt a kvm volume only by using the command line 
after starting it?


Best regards
Henry

Re: Encryption

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

On Thu, Dec 13, 2012 at 09:23:05AM -0800, merc1984@f-m.fm wrote:
> 
> On Thu, Dec 13, 2012, at 1:17, Sander wrote:
> Forbids? That is just plain wrong.
> I have one btrfs filesystem on top of two encrypted devices. Works just
> fine.
> 
> That's dynamite Sander.
> 
> But I am not going to contravene the instructions, then have problems,
> only to come back here and have fingers wagged in my face telling me
> this is all EXPERIMENTAL!

   Well, I'm afraid that applies to the information on the wiki, too
-- that's also experimental, to a degree. The notes on the wiki about
behaviour of encryption layers weren't added by any of the core
developers. Nobody's published concrete tests *either* way yet, and
those comments are one person's opinion, as far as I'm aware (and note
that they don't actually quote sources, results, or even personal
experience).

   YMMV.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
        --- Great oxymorons of the world, no. 2: Common Sense ---        

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Encryption

[merc1984]

On Thu, Dec 13, 2012, at 1:17, Sander wrote:
Forbids? That is just plain wrong.
I have one btrfs filesystem on top of two encrypted devices. Works just
fine.

That's dynamite Sander.

But I am not going to contravene the instructions, then have problems,
only to come back here and have fingers wagged in my face telling me
this is all EXPERIMENTAL!

-- 
http://www.fastmail.fm - Send your email first class

Re: Encryption

[Sander]

merc1984@f-m.fm wrote (ao):
> Oh pardon me, it's BTRFS RAID that's a no-go, which is just as critical
> to me as I have a 4 disk 8TB array.
> The FAQ goeth on to Say:
> -----------------------------------------------------------
> This pretty much forbids you to use btrfs' cool RAID features if you
> need encryption.

Forbids? That is just plain wrong.

I have one btrfs filesystem on top of two encrypted devices. Works just
fine.

	Sander

Re: Encryption

[cwillu]

On Wed, Dec 12, 2012 at 2:06 PM,  <merc1984@f-m.fm> wrote:
> On Wed, Dec 12, 2012, at 10:48, cwillu wrote:
>> Sayeth the FAQ:
>
> Oh pardon me, it's BTRFS RAID that's a no-go, which is just as critical
> to me as I have a 4 disk 8TB array.
> The FAQ goeth on to Say:
> -----------------------------------------------------------
> This pretty much forbids you to use btrfs' cool RAID features if you
> need encryption. Using a RAID implementation on top of several encrypted
> disks is much slower than using encryption on top of a RAID device. So
> the RAID implementation must be on a lower layer than the encryption,
> which is not possible using btrfs' RAID support.
>  -----------------------------------------------------------
>
> You saw that I need RAID above.  Were you just trying to criticize my
> memory of the FAQ cwillu?

It's not asking for trouble, it's just asking for poor performance,
and I suspect even that will depend greatly on the workload.

Snapshots still have nothing to do with it:  you could have btrfs
(with snapshots) on dm-crypt on mdraid.  Btrfs would just lose the
ability to try alternate mirrors and similar; snapshots would still
work just fine.

Re: Encryption

[merc1984]

On Wed, Dec 12, 2012, at 10:48, cwillu wrote:
> Sayeth the FAQ:

Oh pardon me, it's BTRFS RAID that's a no-go, which is just as critical
to me as I have a 4 disk 8TB array.
The FAQ goeth on to Say:
-----------------------------------------------------------
This pretty much forbids you to use btrfs' cool RAID features if you
need encryption. Using a RAID implementation on top of several encrypted
disks is much slower than using encryption on top of a RAID device. So
the RAID implementation must be on a lower layer than the encryption,
which is not possible using btrfs' RAID support.
 -----------------------------------------------------------

You saw that I need RAID above.  Were you just trying to criticize my
memory of the FAQ cwillu?


-- 
http://www.fastmail.fm - Accessible with your email software
                          or over the web

Re: Encryption

[cwillu]

On Wed, Dec 12, 2012 at 12:38 PM,  <merc1984@f-m.fm> wrote:
>
> On Wed, Dec 12, 2012, at 10:31, Mitch Harder wrote:
>> I run btrfs on top of LUKS encryption on my laptop.  You should be able to do the same.
>>
>> You could then run rsync through ssh.  However, rsync will have no knowledge of any blocks shared under subvolume snapshots.
>>
>> Btrfs does not yet have internal encryption.

> The FAQ says specifically to NOT run BTRFS with any kind of volume
> encryption, so you're asking for trouble.

Sayeth the FAQ:

Does Btrfs work on top of dm-crypt?
This is deemed safe since 3.2 kernels. Corruption has been reported
before that, so you want a recent kernel. The reason was improper
passing of device barriers that are a requirement of the filesystem to
guarantee consistency.

> And clearly encryption is not possible if you need snapshots.

Snapshots don't come into this at all:  btrfs doesn't care where the
block devices it's on come from.  Things like dm-crypt show btrfs (or
whatever filesystem you put on it) a decrypted view of the device.

Re: Encryption

[merc1984]

On Wed, Dec 12, 2012, at 10:31, Mitch Harder wrote:
> I run btrfs on top of LUKS encryption on my laptop.  You should be able to do the same.
>
> You could then run rsync through ssh.  However, rsync will have no knowledge of any blocks shared under subvolume snapshots.
>
> Btrfs does not yet have internal encryption.

The FAQ says specifically to NOT run BTRFS with any kind of volume
encryption, so you're asking for trouble.

And clearly encryption is not possible if you need snapshots.

-- 
http://www.fastmail.fm - One of many happy users:
  http://www.fastmail.fm/help/overview_quotes.html

Re: Encryption

[Mitch Harder]

On Wed, Dec 12, 2012 at 11:12 AM,  <merc1984@f-m.fm> wrote:
>
> So there is no way to have filesystem encryption, while keeping
> snapshots?
>
>

I run btrfs on top of LUKS encryption on my laptop.  You should be
able to do the same.

You could then run rsync through ssh.  However, rsync will have no
knowledge of any blocks shared under subvolume snapshots.

Btrfs does not yet have internal encryption.

Re: Encryption

[merc1984]

So there is no way to have filesystem encryption, while keeping
snapshots?


On Fri, Dec 7, 2012, at 8:16, [2]merc1984@f-m.fm wrote:

> We're using a backups server to back up all machines in a LAN.  Four 2TB disks are assembled in a BTRFS RAID array and mounted as /media/backups.  Under this are subvolumes droog, hex, etc, and snapshots droog_snap-{date1}, hex_snap-{date1}, etc.

> Goal is to encrypt backups, but the concern is with snapshots.  Won't piping rsync through encryption with GPG or somesuch, play havoc with BTRFS snapshot accounting?

> Is there any way to encrypt an array so it is inaccesible while umounted?

> I've already asked on the ecryptfs listserv and it resulted in mass confusion.

--


-- 
http://www.fastmail.fm - Or how I learned to stop worrying and
                          love email again

Re: Encryption

[Josef Bacik]

On Thu, Jan 20, 2011 at 07:05:52AM -0800, Carl Cook wrote:
> 
> Does BTRFS have subvolume encryption built in?  If not, why?
> 

No, and because nobody has done it yet.  Thanks,

Josef

Re: Encryption

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

On Thu, Jan 20, 2011 at 07:05:52AM -0800, Carl Cook wrote:
> 
> Does BTRFS have subvolume encryption built in?  If not, why?

   Not at the moment.

   My opinion on why: Getting crypto right is *hard*. There are far
easier features that people are asking for that we can implement
first.

   There may be technical issues that make it hard to implement within
btrfs, although being able to do compression is harder from a FS
structure point of view, so I suspect that the issues are more about
ensuring correctness of the crypto implementation (not just the basic
symmetric algorithm, because we've got those in the kernel, but all
the key management and block chaining and probably a bunch of things I
don't know about because I'm not a cryptographer -- all of which makes
a big difference to the security of the final system).

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- Once is happenstance; twice is coincidence; three times ---     
                            is enemy action.                             

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Encryption

[Carl Cook]

Does BTRFS have subvolume encryption built in?  If not, why?