Re: Storage and snapshots as historical yearly

["Senén Vidal Blanco" <senenvidal@sgisoft.com>]

[-- Attachment #1: Type: text/plain, Size: 5491 bytes --]

Hi Austin,
Thank you very much for your answer.
I comment a little on your suggestions in the context.

El lunes, 11 de septiembre de 2017 14:49:05 (CEST) Austin S. Hemmelgarn 
escribió:
> On 2017-09-11 14:17, Senén Vidal Blanco wrote:
> > I am trying to implement a system that stores the data in a unit (A) with
> > BTRFS format that is untouchable and that future files and folders created
> > or modified are stored in another physical unit (B) with BTRFS format.
> > Each year the new files will be moved to store A and start over.
> > 
> > The idea is that a duplicate of disk A can be made to keep it in a safe
> > place and that the files stored there can not be modified until the
> > mixture of (A) and (B) is made.
> 
> Before I get into anything further, I would like to comment that this is
> a very odd use case.  Yearly granularity doesn't make sense for backups
> unless you generate very little data throughout the year (otherwise
> backups will take forever) and can afford to lose multiple months of
> that data.
> 
> The timescale you're talking about combined with the requirement that
> files not be modifiable on A except during the times when you sync
> changes indicates you will probably be much better served by proper
> off-line archival storage than some online configuration as you appear
> to be trying to create.

Totally agree with you, although it really is not the objective, for that I 
use Bareos as backup system. It is simply because I need to have stored almost 
2 Terabytes and be able to have a copy of the data outside the enclosure and 3 
years of duplicate disks and be able to access almost instantaneously the old 
data.
With a backup system it would take several days to restore the entire system 
or several hours to rebuild the file system and restore the file you need. While 
with an image from last year you would only have to restore a year at best.

> 
> > I have looked for information on this but I do not see it very clear. Both
> > "SEND" and "RECEIVE" do the opposite case to what interests me.
> 
> If you can afford to operate at a shorter timescale (even monthly), then
> BTRFS snapshots plus send and receive probably are one of the best
> options.  Perhaps you could explain your understanding of send and
> receive and why you think it won't work, and I (and possibly other
> people on the list as well) could confirm whether or not you understand
> correctly.

That is another long-term goal that I would like to implement, since I have 
seen that it can be combined with external storage and able to move the data 
through snapshots. It uses little bandwidth and seems to be more immediate 
than the Bareos, although it would leave both systems for security, since I 
speak of very sensitive and important data to lose them.

> 
> > I have also seen that you could try to get a RAID 0 but I am afraid that
> > the data in A will not remain intact if the system performs a "BALANCE"
> > at some point and mixes data between (A) and (B).
> > 
> > It is assumed that both the (A) and (B) data will be displayed in the same
> > structure transparently, for example in "/ home".
> > 
> > -------------------------------
> > 
> > |             HOME            |
> > | 
> > | ------------   ------------ |
> > | 
> > | | DISK (A) |   | DISK (B) | |
> > | | 
> > | |  BTRFS   |   |  BTRFS   | |
> > | 
> > | ------------   ------------ |
> > 
> > -------------------------------
> 
> What you are describing here is called an overlay or union mount.
> Source A would be your lower directory, and B would be your upper
> directory.  Unmodified data is passed through from the lower directory
> until a file is changed.  When changes are made to the overlay mount,
> they are reflected on the upper directory.  Special files called
> 'whiteout' files are used in the upper directory to represent deleted items.
> 
> Unfortunately, I don't know of any overlay mount implementation that
> works correctly and reliably with BTRFS.  I know for a fact that
> OverlayFS (the upstream in-kernel implementation) does not work, and I
> believe that AUFS3 and UnionFS (the third-party options that are used by
> most LiveCD's) don't work either.  UnionFS-FUSE (a userspace
> implementation completely unrelated to UnionFS) might work, but I've
> never tested it and it will likely have performance issues because it's
> implemented in userspace.  As far as I know, whiteout support is the
> primary missing piece here, but I may be mistaken.
> 
> Alternatively, this could be done with a seed device.  The concept is
> pretty similar to an overlay mount, but it operates at a lower level,
> and it's a BTRFS specific feature.  Unfortunately, it's not well
> documented, and I'm not confident that I could explain how to do it
> correctly.

This part does seem to me very interesting, since one of your colleagues has 
commented on the topic of using "Seed" with BTRFS, which I am seeing its 
operation; and I can see how this would be OverlayFS, since I think the other 
options are not very generalized.

Thank you for the explanation and I will keep you informed if I can get 
something clean, because if documenting my experience I can contribute 
something and I will be satisfied.
Greetings.

-- 
	Senén Vidal Blanco - SGISoft S.L.
 
	Tlf.: 986413322 - 660923711
	GPG ID 466431A8AF01F99A
	http://www.sgisoft.com/
--
 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]