[RFC] btrfs auto snapshot

[RFC] btrfs auto snapshot

[Anand Jain]

sorry forgot to follow the protocol, now included RFC in the subject.


Hi,

  Appears that no one is working on the auto-snapshot feature for btrfs,
  so here I am implementing the same.

  Below is a draft on the feature list.  Any comments / questions /
  suggestions are welcome, please do let me know.

  btrfs auto snapshot feature will include:
  Initially:
      - configurable timely snapshots
      - uses services and crontab to schedule
      - Gnome integration
      - snapshot rollback and cleanups
      - snapshot trashing based on available space
      - snapshot destination will be subvol/.btrfs/snapshot@<time> and
        snapshot/.btrfs/snapshot@<time> for subvolume and snapshot
        respectively

  Later:
      -integration with (a ?) backup software
        - snapshot trashing per backup confirmation
        - rollback from the backup server

  Challenges:
    - rollback per file or dir instead of entire snapshot-rollback ?
    - integrating into the btrfs gui ?
    - some FS (samfs) support continues backup
    - do we need a new role for the snapshot management ? zfs does.
    - we don't need a snapshot if the master file-system didn't change
      at all
    - snapshots are writable by default, hope that sudo-er doesn't
      modify the snapshot - do we need to implement a kind of read-only
      snapshot ?

Thanks, -Anand

Re: [RFC] btrfs auto snapshot

[Anand Jain]

And a rough implementation design is here below. (As of now this does
not include the GNOME integration since I have no idea how to do that).

Further, implementation will contain 2 new files
/etc/init.d/btrfs and /<path>/btrfs-auto-snapshot,

any idea where does a file like btrfs-auto-snapshot should be placed
(with its purpose as mentioned below).

-------
1.
File: /etc/init.d/btrfs :
      This will contain the global start and stop knob
      (something like `service btrfs start`)

2.
Crontab: Upon start, crontab will be updated with something like the 
following.
    every 15min `<path>/btrfs-auto-snapshot cleanup>`
    every 15min `/<path>/btrfs-auto-snapshot 15min`
    every hr   `/<path>/btrfs-auto-snapshot hr`
    every day  `/<path>/btrfs-auto-snapshot day`
    every month `/<path>/btrfs-auto-snapshot month`
    every year  `/<path>/btrfs-auto-snapshot year`


3.
File: /<path>/btrfs-auto-snapshot :
  to process the call from cronjob.
    - check the config and check the target fs for the snapshot
    - check the space in the target FS
    - call btrfs cli to take snapshot (provide src and destination)
    - check if snapshot cleanup is required

  to process the calls from '/etc/init.d/btrfs'

  configuration setting.
   - to set which btrfs fs will participate in the auto-snapshot
   - to indicate if auto snapshot has to stop when target FS is 80% full

---------


Thanks, Anand



On 08/17/2011 10:15 AM, Anand Jain wrote:
>
> sorry forgot to follow the protocol, now included RFC in the subject.
>
>
> Hi,
>
> Appears that no one is working on the auto-snapshot feature for btrfs,
> so here I am implementing the same.
>
> Below is a draft on the feature list. Any comments / questions /
> suggestions are welcome, please do let me know.
>
> btrfs auto snapshot feature will include:
> Initially:
> - configurable timely snapshots
> - uses services and crontab to schedule
> - Gnome integration
> - snapshot rollback and cleanups
> - snapshot trashing based on available space
> - snapshot destination will be subvol/.btrfs/snapshot@<time> and
> snapshot/.btrfs/snapshot@<time> for subvolume and snapshot
> respectively
>
> Later:
> -integration with (a ?) backup software
> - snapshot trashing per backup confirmation
> - rollback from the backup server
>
> Challenges:
> - rollback per file or dir instead of entire snapshot-rollback ?
> - integrating into the btrfs gui ?
> - some FS (samfs) support continues backup
> - do we need a new role for the snapshot management ? zfs does.
> - we don't need a snapshot if the master file-system didn't change
> at all
> - snapshots are writable by default, hope that sudo-er doesn't
> modify the snapshot - do we need to implement a kind of read-only
> snapshot ?
>
> Thanks, -Anand
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[David Pottage]

> And a rough implementation design is here below. (As of now this does
> not include the GNOME integration since I have no idea how to do that).
>
> Further, implementation will contain 2 new files
> /etc/init.d/btrfs and /<path>/btrfs-auto-snapshot,
>
> any idea where does a file like btrfs-auto-snapshot should be placed
> (with its purpose as mentioned below).
[...]
> 2.
> Crontab: Upon start, crontab will be updated with something like the
> following.
>     every 15min `<path>/btrfs-auto-snapshot cleanup>`
>     every 15min `/<path>/btrfs-auto-snapshot 15min`
>     every hr   `/<path>/btrfs-auto-snapshot hr`
>     every day  `/<path>/btrfs-auto-snapshot day`
>     every month `/<path>/btrfs-auto-snapshot month`
>     every year  `/<path>/btrfs-auto-snapshot year`

I think that you need to be careful not to impose your idea of when to
take snapshots and how long to keep them onto the design. For example
why take snapshots every 15 minutes? Why not every 10 or every hour?

Why treat monthly snapshots as special when it does not fit into most
working weeks? would weekly be more logical? What about 2 weekly (When
I worked at Nokia, internal releases where done on Tuesday of each even
numbered week, so we would have wanted the snapshot taken on that day
to be retained longer than snapshots taken on other days, or Tuesdays
in odd numbered weeks.)

I think a more flexible design would be to allow the user to specify
(via a config file for each subvolume) a label for each type of snapshot
and how long to keep snapshots depending on when they are taken. This
can be done using syntax similar to crontab:

Example

# Format: <label> <crontab like fields> <retention time>

# Keep 15 min snapshots for 6 hours.
15_minute	*/15 * * * *	6 hour

# Keep hourly snapshots for 5 day
hourly		0 * * * *	5 day

# Keep daily snapshots for 21 days. Use the noon snapshot
daily		0 12 * * *	21 day

# Keep weekly snapshots for 6 months, Use the Tuesday snapshot
# TODO: we would need a week number field to specify even
#       numbered weeks only.
weekly		0 12 * * 2	6 mon

The btrfs-auto-snapshot script would need to make sure that the crontab
entry that takes snapshots runs frequently enough.

-- 
David Pottage

Error compiling committee.c To many arguments to function.

Re: [RFC] btrfs auto snapshot

[Matthias G. Eckermann]

Hello Anand and all,

On 2011-08-17 T 10:15 +0800 Anand Jain wrote:
=20
>  Appears that no one is working on the auto-snapshot feature for
>  btrfs, so here I am implementing the same.

thanks for bringing this up! The group of features you are listing is
indeed of high interest for people using btrfs.

That said, not only have other people though about this, but a lot of
your question already have been implemented in "snapper", and open
source infrastructure developed as part of openSUSE and SUSE Linux
Enterprise.

Please see:
	http://en.opensuse.org/Portal:Snapper
	http://en.opensuse.org/openSUSE:Snapper_install
	http://lizards.opensuse.org/2011/04/01/introducing-snapper/

Source code is here:
	http://gitorious.org/opensuse/snapper

"snapper" will be part of openSUSE 12.1 and SUSE Linux Enterprise 11
Service Pack 2, and is available as part of the respective Beta
releases and Milestones already.

snapper's concept in short:
- shared library to make the functionality available to
  other tools as well
- libsnapper is implemented on top of the btrfsprogs
- cmdline tool "snapper"
- global configuration file
	/etc/sysconfig/snapper
- one configuration file per subvolume to be snapshotted
	/etc/snapper/configs/<config-name>
  I call this a "single configuration" going forward.
  Here also policies for time based snapshotting and
  cleanup are to be configured.
- Integration into SUSE's management framework (YaST2/zypper),
  however, "snapper" should work independent of those,
  i.e. usable on other distributions easily.

>  Below is a draft on the feature list.  Any comments / questions /
>  suggestions are welcome, please do let me know.

Let me go through the single features quickly and list the matching
snapper functionality.
=20
>  btrfs auto snapshot feature will include:
>  Initially:
>      - configurable timely snapshots

Yes. Configured per single configuration

>      - uses services and crontab to schedule

Yes.

>      - Gnome integration

I more see a need for integration into systems management frameworks.

>      - snapshot rollback and cleanups

Yes. Rules for cleanups (time based, number of snapshots)
per single configuration.

>      - snapshot trashing based on available space

// not yet done.

>      - snapshot destination will be subvol/.btrfs/snapshot@<time> and

snapshot destination is "/.snapshots/<number>/",=20

>        snapshot/.btrfs/snapshot@<time> for subvolume and snapshot
>        respectively

Timestamp and Description of a snapshot are stored in a small XML
file /.snapshots/<number>/info.xml". One small file per snapshot.

[...]

>  Challenges:
>    - rollback per file or dir instead of entire snapshot-rollback ?

snapper implements  "rollback" on a FILE level only.

To differentiate this way of "rolling back" from jumping
into another snapshot, we call it
	"undochange"
for now. This keeps the option to also manage a full
per snapshot-rollback in a later point int time.

[...]
>      modify the snapshot - do we need to implement a kind of read-onl=
y
>      snapshot ?

snapper treats snapshots as read only snapshots, i.e. when doing a
rollback - aehem, I should say "undochange" - only the "master" volume
will be changed, not the single snapshots.  We are aware that this has
pros and cons. But that's another discussion.

I hope that this is a starting point for you.

Enjoy "snapper".

so long -
	MgE

--=20
Matthias G. Eckermann     Senior Product Manager   SUSE=C2=AE Linux Ent=
erprise
SUSE LINUX Products GmbH  Maxfeldstra=C3=9Fe 5          90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3=
=BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Dave]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

I've already done something similar.  I take hourly, daily, weekly, and monthly
snapshots of my /home subvolume.  Here's the script I've created for this:

#! /bin/bash

if [ "$#" -ne 2 ]; then
   echo Usage $0 SNAPSHOT_PREFIX NUM_SNAPSHOTS
   exit 1
fi

SNAPS=/var/lib/btrfs-root/__snapshot/home

btrfs subvolume snapshot -r /home $SNAPS/$1_$(date +%F_%T.%N)

num_snaps=$(ls -1d $SNAPS/$1_* | sort | wc -l)

if [ "$num_snaps" -gt "$2" ]; then
   let over=$num_snaps-$2
   ls -1d $SNAPS/$1_* | head -n $over | while read s; do
      btrfs subvolume delete $s
   done
fi


Here's my crontab:
0 * * * * /usr/local/bin/snapshot hourly 6
0 0 * * * /usr/local/bin/snapshot daily 7
0 0 * * 0 /usr/local/bin/snapshot weekly 4
-- 
-=[dave]=-

Entropy isn't what it used to be.

[-- Attachment #2: Type: application/pgp-signature, Size: 230 bytes --]

Re: [RFC] btrfs auto snapshot

[Ken A]

and much easier than trying to get snapper to compile on fedora
libblocxx ?
:-)
Ken


On 8/17/2011 9:04 AM, Dave wrote:
> I've already done something similar.  I take hourly, daily, weekly, and monthly
> snapshots of my /home subvolume.  Here's the script I've created for this:
>
> #! /bin/bash
>
> if [ "$#" -ne 2 ]; then
>     echo Usage $0 SNAPSHOT_PREFIX NUM_SNAPSHOTS
>     exit 1
> fi
>
> SNAPS=/var/lib/btrfs-root/__snapshot/home
>
> btrfs subvolume snapshot -r /home $SNAPS/$1_$(date +%F_%T.%N)
>
> num_snaps=$(ls -1d $SNAPS/$1_* | sort | wc -l)
>
> if [ "$num_snaps" -gt "$2" ]; then
>     let over=$num_snaps-$2
>     ls -1d $SNAPS/$1_* | head -n $over | while read s; do
>        btrfs subvolume delete $s
>     done
> fi
>
>
> Here's my crontab:
> 0 * * * * /usr/local/bin/snapshot hourly 6
> 0 0 * * * /usr/local/bin/snapshot daily 7
> 0 0 * * 0 /usr/local/bin/snapshot weekly 4

-- 
Ken Anderson

snapshot ctime // Re: [RFC] btrfs auto snapshot

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

On Wed, 17 Aug 2011 10:04:33 -0400
Dave <dave@thekilempire.com> wrote:

> I've already done something similar.  I take hourly, daily, weekly, and monthly
> snapshots of my /home subvolume.  Here's the script I've created for this:

On one machine I make hourly snapshots of my /home and of the root FS as well.

The tricky part is actually not the snapshotting, but the deletion of outdated snapshots.

That's due to the unfortunate fact (bug?), that snapshot-directories do not have their ctime set correctly at all, they have some totally bogus ctime instead.

.........../snaps/$ ls -la --time=ctime | tail
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-05@02-31-51
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-06@02-49-29
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-07@00-17-40
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-08@01-53-29
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-10@03-09-32
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-12@00-26-54
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-13@01-40-19
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-14@04-22-07
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-15@02-29-13
dr-xr-xr-x 1 root root  102 2011-06-28 12:29 2011-08-16@10-27-57

As you can see I have to store creation date/time in the snapshot name, and then parse it out to delete snapshots e.g. older than 3 months.

So until someone cares about snapshot ctime enough to fix this, btrfs will not be a convenient FS to work with timed snapshotting/cleanup.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: snapshot ctime // Re: [RFC] btrfs auto snapshot

[Jérôme Poulin]

On Wed, Aug 17, 2011 at 11:13 AM, Roman Mamedov <rm@romanrm.ru> wrote:
> So until someone cares about snapshot ctime enough to fix this, btrfs will not be a convenient FS to work with timed snapshotting/cleanup.

Isn't the ctime the creation date of the original folder?

Re: [RFC] btrfs auto snapshot

[Matthias G. Eckermann]

On 2011-08-17 T 09:50 -0500 Ken A wrote:

> and much easier

but less powerful:-)

> than trying to get snapper to compile on fedora libblocxx
> ?  :-)

Ah, sure. Sorry.  Packages for "blocxx" for:
	Fedora_14 	Fedora_15 =09
	RHEL-5		RHEL-6 =09
	SLE_11_SP1 =09
	openSUSE_11.4	openSUSE_Factory

are available in the openSUSE buildservice at:

	http://download.opensuse.org/repositories/home:/mge1512:/snapper/

See also the build project at:

	https://build.opensuse.org/project/show?project=3Dhome%3Amge1512%3Asna=
pper

If someone sends the packaging files for Debian/Ubuntu,
I can also add those.

so long -
	MgE

P.S.:	I also added "snapper" itself there. I am not sure though,
	if it will build out of the box. ... Stay tuned.

--=20
Matthias G. Eckermann     Senior Product Manager   SUSE=C2=AE Linux Ent=
erprise
SUSE LINUX Products GmbH  Maxfeldstra=C3=9Fe 5          90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3=
=BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Lenz Grimmer]

Hi Anand,

On Wed, Aug 17, 2011 at 11:24, Anand Jain <Anand.Jain@oracle.com> wrote=
:

> And a rough implementation design is here below. (As of now this does
> not include the GNOME integration since I have no idea how to do that=
).

Very cool idea! With regards to the Gnome integration, you might want t=
o take
a look at what the Solaris folks did for Nautilus - I really liked the
UI and the
integration with ZFS:

http://src.opensolaris.org/source/xref/jds/spec-files/branches/gnome-2-=
30/patches/nautilus-14-zfs-snapshot.diff

Bye,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 LenZ
--
=C2=A0Lenz Grimmer <lenz@grimmer.com> -=C2=A0 http://www.lenzg.net/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Matthias G. Eckermann]

Hello Ken and all,

On 2011-08-17 T 19:38 +0200 Matthias G. Eckermann wrote:
=20
> P.S.:	I also added "snapper" itself there. I am not sure
> though, if it will build out of the box. ... Stay tuned.
=20
A dinner later, the packages (.rpm/.src.rpm) for blocxx and
also snapper are available in the openSUSE Buildservice at:
	http://download.opensuse.org/repositories/home:/mge1512:/snapper/
for=20
	Fedora_15 RHEL-6 SLE_11_SP1 openSUSE_Factory

The download directory is a YUM repository and should easily
work with either of the distributions.

Enjoy!

so long -
 	MgE

P.S.: 	Again, if someone sends the packaging files for
	Debian/Ubuntu, I can also add those. Unfortunately
	my .deb experience is limited, ...

--=20
Matthias G. Eckermann     Senior Product Manager   SUSE=C2=AE Linux Ent=
erprise
SUSE LINUX Products GmbH  Maxfeldstra=C3=9Fe 5          90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3=
=BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Anand Jain]

  Thanks MgE. snapper is cool, does most the stuff required here.

  however the challenging part will be to keep the number of tools
  (to manage btrfs) at a limit 1 or 2 max. (too many tools to manage
  btrfs is most likely to confuse).

Cheers, Anand

On 08/17/2011 09:31 PM, Matthias G. Eckermann wrote:
> Hello Anand and all,
>
> On 2011-08-17 T 10:15 +0800 Anand Jain wrote:
>
>>   Appears that no one is working on the auto-snapshot feature for
>>   btrfs, so here I am implementing the same.
>
> thanks for bringing this up! The group of features you are listing is
> indeed of high interest for people using btrfs.
>
> That said, not only have other people though about this, but a lot of
> your question already have been implemented in "snapper", and open
> source infrastructure developed as part of openSUSE and SUSE Linux
> Enterprise.
>
> Please see:
> 	http://en.opensuse.org/Portal:Snapper
> 	http://en.opensuse.org/openSUSE:Snapper_install
> 	http://lizards.opensuse.org/2011/04/01/introducing-snapper/
>
> Source code is here:
> 	http://gitorious.org/opensuse/snapper
>
> "snapper" will be part of openSUSE 12.1 and SUSE Linux Enterprise 11
> Service Pack 2, and is available as part of the respective Beta
> releases and Milestones already.
>
> snapper's concept in short:
> - shared library to make the functionality available to
>    other tools as well
> - libsnapper is implemented on top of the btrfsprogs
> - cmdline tool "snapper"
> - global configuration file
> 	/etc/sysconfig/snapper
> - one configuration file per subvolume to be snapshotted
> 	/etc/snapper/configs/<config-name>
>    I call this a "single configuration" going forward.
>    Here also policies for time based snapshotting and
>    cleanup are to be configured.
> - Integration into SUSE's management framework (YaST2/zypper),
>    however, "snapper" should work independent of those,
>    i.e. usable on other distributions easily.
>
>>   Below is a draft on the feature list.  Any comments / questions /
>>   suggestions are welcome, please do let me know.
>
> Let me go through the single features quickly and list the matching
> snapper functionality.
>
>>   btrfs auto snapshot feature will include:
>>   Initially:
>>       - configurable timely snapshots
>
> Yes. Configured per single configuration
>
>>       - uses services and crontab to schedule
>
> Yes.
>
>>       - Gnome integration
>
> I more see a need for integration into systems management frameworks.
>
>>       - snapshot rollback and cleanups
>
> Yes. Rules for cleanups (time based, number of snapshots)
> per single configuration.
>
>>       - snapshot trashing based on available space
>
> // not yet done.
>
>>       - snapshot destination will be subvol/.btrfs/snapshot@<time>  and
>
> snapshot destination is "/.snapshots/<number>/",
>
>>         snapshot/.btrfs/snapshot@<time>  for subvolume and snapshot
>>         respectively
>
> Timestamp and Description of a snapshot are stored in a small XML
> file /.snapshots/<number>/info.xml". One small file per snapshot.
>
> [...]
>
>>   Challenges:
>>     - rollback per file or dir instead of entire snapshot-rollback ?
>
> snapper implements  "rollback" on a FILE level only.
>
> To differentiate this way of "rolling back" from jumping
> into another snapshot, we call it
> 	"undochange"
> for now. This keeps the option to also manage a full
> per snapshot-rollback in a later point int time.
>
> [...]
>>       modify the snapshot - do we need to implement a kind of read-only
>>       snapshot ?
>
> snapper treats snapshots as read only snapshots, i.e. when doing a
> rollback - aehem, I should say "undochange" - only the "master" volume
> will be changed, not the single snapshots.  We are aware that this has
> pros and cons. But that's another discussion.
>
> I hope that this is a starting point for you.
>
> Enjoy "snapper".
>
> so long -
> 	MgE
>

Re: [RFC] btrfs auto snapshot

[Anand Jain]

David,

> I think that you need to be careful not to impose your idea of when to
> take snapshots and how long to keep them onto the design. For example
> why take snapshots every 15 minutes? Why not every 10 or every hour?

  crontab is anyways changeable by the admin, I think we can have that
  flexibility.

> Why treat monthly snapshots as special when it does not fit into most
> working weeks? would weekly be more logical? What about 2 weekly (When
> I worked at Nokia, internal releases where done on Tuesday of each even
> numbered week, so we would have wanted the snapshot taken on that day
> to be retained longer than snapshots taken on other days, or Tuesdays
> in odd numbered weeks.)

  agreed. weekly is more important. (I had that in mind but missed it
  when writing, sorry for that).

> I think a more flexible design would be to allow the user to specify
> (via a config file for each subvolume) a label for each type of snapshot
> and how long to keep snapshots depending on when they are taken. This
> can be done using syntax similar to crontab:

  simple and nice idea. thanks for explaining, will try to get this
  in the initial release.

Cheers, Anand

Re: snapshot ctime // Re: [RFC] btrfs auto snapshot

[Anand Jain]

On 08/17/2011 11:56 PM, J=C3=A9r=C3=B4me Poulin wrote:
> On Wed, Aug 17, 2011 at 11:13 AM, Roman Mamedov<rm@romanrm.ru>  wrote=
:
>> So until someone cares about snapshot ctime enough to fix this, btrf=
s will not be a convenient FS to work with timed snapshotting/cleanup.
>
> Isn't the ctime the creation date of the original folder?

  It will be a nice thing to have the snapshot time returned by
  btrfs-prog.

  something like
   btrfs get stime <snapshot>

Cheers, Anand
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Fajar A. Nugraha]

On Thu, Aug 18, 2011 at 12:38 AM, Matthias G. Eckermann <mge@suse.com> =
wrote:
> Ah, sure. Sorry. =A0Packages for "blocxx" for:
> =A0 =A0 =A0 =A0Fedora_14 =A0 =A0 =A0 Fedora_15
> =A0 =A0 =A0 =A0RHEL-5 =A0 =A0 =A0 =A0 =A0RHEL-6
> =A0 =A0 =A0 =A0SLE_11_SP1
> =A0 =A0 =A0 =A0openSUSE_11.4 =A0 openSUSE_Factory
>
> are available in the openSUSE buildservice at:
>
> =A0 =A0 =A0 =A0http://download.opensuse.org/repositories/home:/mge151=
2:/snapper/
>

Hi Matthias,

I'm testing your packages on top of RHEL6 + kernel 3.2.7. A small
suggestion, you should include /etc/sysconfig/snapper in the package
(at least for RHEL6, haven't tested the other ones). Even if it just
contains

SNAPPER_CONFIGS=3D""

Thanks,

=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Hubert Kario]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Wednesday 17 of August 2011 10:15:46 Anand Jain wrote:
>   btrfs auto snapshot feature will include:
>   Initially:
<snip>
>       - snapshot destination will be subvol/.btrfs/snapshot@<time> and
>         snapshot/.btrfs/snapshot@<time> for subvolume and snapshot
>         respectively

Is there some reason not to use the format used by shadow_copy2 overlay for 
Samba? (The one providing Shadow Volume Copy functionality for Windows 
clients):

Current date in this format you get like this:

@GMT-`date -u '+%Y.%m.%d-%H.%M.%S'`

For example: @GMT-2012.02.23-10.34.32

This way, when the volume is exported using Samba, you can easily export 
past copies too, without creating links.

Regards,
-- 
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawerów 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2346 bytes --]

Re: [RFC] btrfs auto snapshot

[Anand Jain]

  autosnap code is available either end of this week or early
  next week and what you will notice is autosnap snapshots
  are named using uuid.

  Main reason to drop time-stamp based names is that,
     - test (clicking on Take-snapshot button) which took more
   than one snapshot per second was failing.
     - a more descriptive creation time is available using a
    command line option as in the example below.
  -----
  # btrfs su list -t tag=@minute,parent=/btrfs/sv1 /btrfs
  /btrfs/.autosnap/6c0dabfa-5ddb-11e1-a8c1-0800271feb99 Thu Feb 23 13:01:18 2012 /btrfs/sv1 @minute
  /btrfs/.autosnap/5669613e-5ddd-11e1-a644-0800271feb99 Thu Feb 23 13:15:01 2012 /btrfs/sv1 @minute
  -----
  
  As of now code for time-stamp as autosnap snapshot name is
  commented out, if more people wanted it to be a time-stamp
  based names, I don't mind having that way. Please do let me know.

Thanks, Anand
   

On Thursday 23,February,2012 06:37 PM, Hubert Kario wrote:
> On Wednesday 17 of August 2011 10:15:46 Anand Jain wrote:
>>    btrfs auto snapshot feature will include:
>>    Initially:
> <snip>
>>        - snapshot destination will be subvol/.btrfs/snapshot@<time>  and
>>          snapshot/.btrfs/snapshot@<time>  for subvolume and snapshot
>>          respectively
>
> Is there some reason not to use the format used by shadow_copy2 overlay for
> Samba? (The one providing Shadow Volume Copy functionality for Windows
> clients):
>
> Current date in this format you get like this:
>
> @GMT-`date -u '+%Y.%m.%d-%H.%M.%S'`
>
> For example: @GMT-2012.02.23-10.34.32
>
> This way, when the volume is exported using Samba, you can easily export
> past copies too, without creating links.
>
> Regards,

Re: [RFC] btrfs auto snapshot

[Fajar A. Nugraha]

On Thu, Feb 23, 2012 at 7:02 PM, Anand Jain <Anand.Jain@oracle.com> wro=
te:
>
>
> =A0autosnap code is available either end of this week or early
> =A0next week

I thought you stopped working on this :D

Alternatives are good though. Will test yours when it's out.

=46WIW, I also have another one, based on zfsonlinux's autosnapshot scr=
ipt :D

> and what you will notice is autosnap snapshots
> =A0are named using uuid.
>
> =A0Main reason to drop time-stamp based names is that,
> =A0 =A0- test (clicking on Take-snapshot button) which took more
> =A0than one snapshot per second was failing.
> =A0 =A0- a more descriptive creation time is available using a
> =A0 command line option as in the example below.
> =A0-----
> =A0# btrfs su list -t tag=3D@minute,parent=3D/btrfs/sv1 /btrfs
> =A0/btrfs/.autosnap/6c0dabfa-5ddb-11e1-a8c1-0800271feb99 Thu Feb 23 1=
3:01:18
> 2012 /btrfs/sv1 @minute
> =A0/btrfs/.autosnap/5669613e-5ddd-11e1-a644-0800271feb99 Thu Feb 23 1=
3:15:01
> 2012 /btrfs/sv1 @minute
> =A0-----
> =A0As of now code for time-stamp as autosnap snapshot name is
> =A0commented out, if more people wanted it to be a time-stamp
> =A0based names, I don't mind having that way. Please do let me know.

=46or me the main bonus point of having timestamp in names in the
abiility to sort it by creation date by simply using "ls". As for the
more-than-one-click-per-second problem, in my script I simply let it
fail and return informative-enough error message.

A workaround would be adding nanosecond timestamp, or put the UUID
AFTER the time stamp, e.g:
/btrfs/.autosnap/@minute_20120223_131501_123456_5669613e-5ddd-11e1-a644=
-0800271feb99

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Hubert Kario]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

On Thursday 23 of February 2012 20:02:38 Anand Jain wrote:
>   autosnap code is available either end of this week or early
>   next week and what you will notice is autosnap snapshots
>   are named using uuid.
> 
>   Main reason to drop time-stamp based names is that,
>      - test (clicking on Take-snapshot button) which took more
>    than one snapshot per second was failing.
>      - a more descriptive creation time is available using a
>     command line option as in the example below.
>   -----
>   # btrfs su list -t tag=@minute,parent=/btrfs/sv1 /btrfs
>   /btrfs/.autosnap/6c0dabfa-5ddb-11e1-a8c1-0800271feb99 Thu Feb 23 13:01:18
> 2012 /btrfs/sv1 @minute
> /btrfs/.autosnap/5669613e-5ddd-11e1-a644-0800271feb99 Thu Feb 23 13:15:01
> 2012 /btrfs/sv1 @minute -----
> 
>   As of now code for time-stamp as autosnap snapshot name is
>   commented out, if more people wanted it to be a time-stamp
>   based names, I don't mind having that way. Please do let me know.

I'd say, that having it as configure option (do Samba-style snapshot naming 
vs. uuid based) would be sufficient. The question remains what should be the 
default.

That being said, what use-case would require snapshots taken more often than 
every second? I doubt that you actually can do snapshots every second on a 
busy file system, let alone more often. On lightly-used one they will be 
identical and just clutter the name-space.

Regards,
-- 
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawerów 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2346 bytes --]

Re: [RFC] btrfs auto snapshot

[Matthias G. Eckermann]

Hello all,

On 2012-02-23 T 19:13 +0700 Fajar A. Nugraha wrote:
> On Thu, Feb 23, 2012 at 7:02 PM, Anand Jain wrote:
>=20
> > and what you will notice is autosnap snapshots =C2=A0are named
> > using uuid.
> >
> > =C2=A0Main reason to drop time-stamp based names is that,
> > =C2=A0 =C2=A0- test (clicking on Take-snapshot button) which took m=
ore
> > =C2=A0than one snapshot per second was failing.
> > =C2=A0 =C2=A0- a more descriptive creation time is available using =
a =C2=A0
> > command line option as in the example below.
> > =C2=A0-----
> > =C2=A0# btrfs su list -t tag=3D@minute,parent=3D/btrfs/sv1 /btrfs
> > =C2=A0/btrfs/.autosnap/6c0dabfa-5ddb-11e1-a8c1-0800271feb99 Thu Feb=
 23 13:01:18 2012 /btrfs/sv1 @minute
> > =C2=A0/btrfs/.autosnap/5669613e-5ddd-11e1-a644-0800271feb99 Thu Feb=
 23 13:15:01 2012 /btrfs/sv1 @minute
> > =C2=A0-----
> > =C2=A0As of now code for time-stamp as autosnap snapshot name is
> > =C2=A0commented out, if more people wanted it to be a time-stamp
> > =C2=A0based names, I don't mind having that way. Please do let me
> > know.
>=20
> For me the main bonus point of having timestamp in names in
> the abiility to sort it by creation date by simply using "ls".
> As for the more-than-one-click-per-second problem, in my
> script I simply let it fail and return informative-enough
> error message.
>=20
> A workaround would be adding nanosecond timestamp, or put the
> UUID AFTER the time stamp, e.g:
> /btrfs/.autosnap/@minute_20120223_131501_123456_5669613e-5ddd-11e1-a6=
44-0800271feb99

I wonder, if this is not mixing several aspects / requirements:
1. unique snapshot numbering
2. storing the time stamp of a snapshot
3. human visible order of snapshots

Thus we have chosen a different solution in our "snapper"
approach (as announced here in Aug 2011):
- the snapshot number is a simple integer
  -> human visible order of snapshots
- all metadata is stored in an XML file associated to
  this snapshot
- this requires that the btrfs snapshot itself is
  in a subdirectory of the snapshot-numbered directory.

Example:

----------------------------< snip >----------------------------

2012-02-23 15:02 (0) ~
naxos root (5)  # tree -x /.snapshots/6/
/.snapshots/6/
|-- filelist-5.txt
|-- info.xml
`-- snapshot

2012-02-23 15:02 (0) ~
naxos root (5)  # cat /.snapshots/6/info.xml
<?xml version=3D"1.0"?>
<snapshot>
  <type>post</type>
  <num>6</num>
  <date>2012-02-23 11:32:59</date>
  <pre_num>5</pre_num>
  <cleanup>number</cleanup>
</snapshot>

----------------------------< snap >----------------------------

With introducing a store for meta information, the requirements
listed above (1-3) can be fulfilled.

Shouldn't the autosnap implementation include a similar
option and/or functionality? Or does it already?

=46eel free to copy what snapper does:)

so long -
	MgE

P.S.: For those interested in the details: snapshots 5 and 6 are
a pre-post-pair, which enclose the real action: a timezone
change.  "filelist-5.txt" contains the file which differ between
5 and 6 for faster access from the management layer.

--=20
Matthias G. Eckermann     Senior Product Manager   SUSE=C2=AE Linux Ent=
erprise
Phone: +49 30 44315731    Mobile: +49 179 2949448        Twitter: mge15=
12
SUSE LINUX Products GmbH  Maxfeldstra=C3=9Fe 5          90409 N=C3=BCrn=
berg Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imend=C3=B6rffer, HRB 16746 (AG N=C3=
=BCrnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Fahrzin Hemmati]

I'd like to vote for timestamp/timestamp-uuid as a sysadmin. The 
timestamp allows for easy conversion from clients' wants to actual 
commands: "I need my data from two days ago" is easy when I have 
timestamps to use.

On 2/23/2012 10:05 PM, Anand Jain wrote:
>
>
>  Thanks for the inputs.  there is no clear winner as of now.
>  Let me keep the uuid for now, if more sysadmin feel timestamp
>  is better we could device it that way.
>
> -Anand
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Anand Jain]

  Thanks for the inputs.  there is no clear winner as of now.
  Let me keep the uuid for now, if more sysadmin feel timestamp
  is better we could device it that way.

-Anand

Re: [RFC] btrfs auto snapshot

[Arvin Schnell]

On Thu, Feb 23, 2012 at 04:54:06PM +0700, Fajar A. Nugraha wrote:
> On Thu, Aug 18, 2011 at 12:38 AM, Matthias G. Eckermann <mge@suse.com=
> wrote:

> > are available in the openSUSE buildservice at:
> >
> > =A0 =A0 =A0 =A0http://download.opensuse.org/repositories/home:/mge1=
512:/snapper/
> >
>=20
> Hi Matthias,
>=20
> I'm testing your packages on top of RHEL6 + kernel 3.2.7. A small
> suggestion, you should include /etc/sysconfig/snapper in the package
> (at least for RHEL6, haven't tested the other ones). Even if it just
> contains
>=20
> SNAPPER_CONFIGS=3D""

Hi Fajar,

thanks for reporting that issue, I have fixed it now.

We have now created a project in the openSUSE buildservice were
we provide snapper packages for various distributions, e.g. RHEL6
and Fedora 16. Please find the downloads at:

  http://download.opensuse.org/repositories/filesystems:/snapper/

I'll also add a link from the snapper home page:

  http://en.opensuse.org/Portal:Snapper.

I have tested snapper on Fedora 16 and found no problems.

Regards,
  Arvin

--=20
Arvin Schnell, <aschnell@suse.de>
Senior Software Engineer, Research & Development
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend=F6=
rffer, HRB 16746 (AG N=FCrnberg)
Maxfeldstra=DFe 5
90409 N=FCrnberg
Germany
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] btrfs auto snapshot

[Fajar A. Nugraha]

On Thu, Mar 1, 2012 at 8:48 PM, Arvin Schnell <aschnell@suse.de> wrote:
> On Thu, Feb 23, 2012 at 04:54:06PM +0700, Fajar A. Nugraha wrote:
>> On Thu, Aug 18, 2011 at 12:38 AM, Matthias G. Eckermann <mge@suse.co=
m> wrote:
>
>> > are available in the openSUSE buildservice at:
>> >
>> > =A0 =A0 =A0 =A0http://download.opensuse.org/repositories/home:/mge=
1512:/snapper/
>> >
>>
>> Hi Matthias,
>>
>> I'm testing your packages on top of RHEL6 + kernel 3.2.7. A small
>> suggestion, you should include /etc/sysconfig/snapper in the package
>> (at least for RHEL6, haven't tested the other ones). Even if it just
>> contains
>>
>> SNAPPER_CONFIGS=3D""
>
> Hi Fajar,
>
> thanks for reporting that issue, I have fixed it now.

Great! Thanks.

>
> We have now created a project in the openSUSE buildservice were
> we provide snapper packages for various distributions, e.g. RHEL6
> and Fedora 16. Please find the downloads at:
>
> =A0http://download.opensuse.org/repositories/filesystems:/snapper/
>
> I'll also add a link from the snapper home page:
>
> =A0http://en.opensuse.org/Portal:Snapper.
>
> I have tested snapper on Fedora 16 and found no problems.

When I installed it back then, the first thing that comes to mind was
"there's no documentation on how to get started".

http://en.opensuse.org/openSUSE:Snapper_Tutorial is good, but that' is
assuming root is btrfs, and snapper is already configured to snapshot
root. For other distros, you need to first create the config manually,
e.g. as shown for home in http://en.opensuse.org/openSUSE:Snapper_FAQ

Could you update the tutorial, or perhaps create a new "quickstart"
page? I'm kinda reluctant to do it myself since I don't use opensuse,
and some of my edits might not reflect the "correct" way to do it in
opensuse. If that's not possible, I'll put up the documentation
somewhere else (perhaps the semi-official http://btrfs.ipv5.de/ , or
my own wiki).

Two other things that I have't find is:
- how to add pre and post hooks, so (for example) snapper could create
the same pre-post snapshot whenever user runs "yum", similar to when a
user runs "yast" in opensuse,
- whether a rollback REALLY rolls back everyting (including binary and
new/missing files), or is it git-like behavior, or if it only process
text files.

=2E.. but those two aren't as important as the getting-started document=
ation.

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html