btrfs issue with mariadb incremental backup

btrfs issue with mariadb incremental backup

[siranee.ja]

Hi btrfs support team,

My name is siranee jaraswachirakul. I tested btrfs incremental send and receive and
I found something incorrect.

I posted detail in the url
http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238

Could you please help me to find the reason?


Best Regards,
Siranee Jaraswachirakul.

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Tue, Aug 8, 2017 at 10:32 PM,  <siranee.ja@tpc.co.th> wrote:
> Hi btrfs support team,
>
> My name is siranee jaraswachirakul. I tested btrfs incremental send and receive and
> I found something incorrect.
>
> I posted detail in the url
> http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238
>
> Could you please help me to find the reason?

The command "btrfs sub snap mysql_201707210830 mysql" is unexpected
for two reasons:

1. This is a rw snapshot. It needs to be ro using -r flag to make it
ro, and only ro snapshots can be used with btrfs send receive.
2. The naming convention seems reversed. The subvolume you're
snapshotting should be first, and the resulting snapshot is second,
and it is the second one that you'll send. So I think really what
you'll end up wanting is:

Machine A
btrfs sub snap -r mysql mysql_20170721
btrfs send mysql_20170721

Machine B
btrfs sub snap mysql_20170721 mysql

That will make a rw snapshot from the ro snapshot.

The next problem with the post is that you say "incremental" but I do
not see the exact send receive command you're using. An incremental
send requires -p option.



Machine A
btrfs sub snap -r mysql mysql_20170721
btrfs send mysql_20170721

Machine B
btrfs sub snap mysql_20170721 mysql
Test mysql
btrfs sub del mysql

Machine A (next day)

Machine A
btrfs sub snap -r mysql mysql_20170722
btrfs send -p mysql_20170721 mysql_20170722

Machine B
btrfs sub snap mysql_20170722 mysql
Test mysql
btrfs sub del mysql

And so on. You must have the -p snapshot on both file systems for the
incremental send/receive to work.



-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi Chris,

Sorry for the incompleted command that I used in the post.

This is the real command from Machine A

backup step :
in crontab
30 08 * * * root /root/script/backup/backupsnap.sh root password /var/lib/mariadb/mysql

[root@backuplogC7 ~]# cat  /root/script/backup/backupsnap.sh
#Backup
#
user=$1
password=$2
basepath=$3
datet=$(date +%Y%m%d%H%M)
snappath=${basepath}_${datet}
echo "Locking databases"
mysql -u$user -p$password << EOF
FLUSH TABLES WITH READ LOCK;
system btrfs sub snap -r $basepath $snappath
UNLOCK TABLES;
quit
EOF
echo "Databases unlocked"

The send incremental step :

 487  history|grep send
 488  btrfs send /var/lib/mariadb/mysql_201707210830 | ssh 192.168.10.29 btrfs
receive /var/lib/mariadb
  489  btrfs send -p /var/lib/mariadb/mysql_201707210830
/var/lib/mariadb/mysql_201707220830 |ssh 192.168.10.29 btrfs receive
/var/lib/mariadb
  490  btrfs send -p /var/lib/mariadb/mysql_201707220830
/var/lib/mariadb/mysql_201707230830 |pv| ssh 192.168.10.29 btrfs receive
/var/lib/mariadb

This is the real command from Machine B

 463  systemctl stop mariadb
  464  cd /var/lib/mariadb
  465  ls -l
  466  btrfs sub del mysql*
  467  btrfs sub sync .
  468  ip addr
  469  btrfs sub list
  470  btrfs sub list .
  471  btrfs sub snap mysql_201707210830 mysql
  472  systemctl start mariadb
  473  mysql -uroot -ppassword
  474  btrfs sub list
  475  btrfs sub list .
  476  mysql -uroot -ppassword
  477  systemctl stop mariadb
  478  btrfs sub list .
  479  btrfs sub del  mysql
  480  btrfs sub sync .
  481  btrfs sub snap mysql_201707220830 mysql
  482  systemctl start mariadb
  483  mysql -uroot -ppassword
  484  systemctl stop mariadb
  485  btrfs sub del  mysql
  486  btrfs sub sync .
  487  btrfs sub list .
  488  btrfs sub snap mysql_201707230830 mysql
  489  systemctl start mariadb
  490  btrfs sub list .
  491  cat /var/log/mariadb/mariadb.log

PS:  No any error in btrfs command or in /var/log/messages.

Best Regards
Siranee Jaraswachirakul.


> On Tue, Aug 8, 2017 at 10:32 PM,  <siranee.ja@tpc.co.th> wrote:
>> Hi btrfs support team,
>>
>> My name is siranee jaraswachirakul. I tested btrfs incremental send and receive
>> and
>> I found something incorrect.
>>
>> I posted detail in the url
>> http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238
>>
>> Could you please help me to find the reason?
>
> The command "btrfs sub snap mysql_201707210830 mysql" is unexpected
> for two reasons:
>
> 1. This is a rw snapshot. It needs to be ro using -r flag to make it
> ro, and only ro snapshots can be used with btrfs send receive.
> 2. The naming convention seems reversed. The subvolume you're
> snapshotting should be first, and the resulting snapshot is second,
> and it is the second one that you'll send. So I think really what
> you'll end up wanting is:
>
> Machine A
> btrfs sub snap -r mysql mysql_20170721
> btrfs send mysql_20170721
>
> Machine B
> btrfs sub snap mysql_20170721 mysql
>
> That will make a rw snapshot from the ro snapshot.
>
> The next problem with the post is that you say "incremental" but I do
> not see the exact send receive command you're using. An incremental
> send requires -p option.
>
>
>
> Machine A
> btrfs sub snap -r mysql mysql_20170721
> btrfs send mysql_20170721
>
> Machine B
> btrfs sub snap mysql_20170721 mysql
> Test mysql
> btrfs sub del mysql
>
> Machine A (next day)
>
> Machine A
> btrfs sub snap -r mysql mysql_20170722
> btrfs send -p mysql_20170721 mysql_20170722
>
> Machine B
> btrfs sub snap mysql_20170722 mysql
> Test mysql
> btrfs sub del mysql
>
> And so on. You must have the -p snapshot on both file systems for the
> incremental send/receive to work.
>
>
>
> --
> Chris Murphy
>

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Wed, Aug 9, 2017 at 12:36 AM,  <siranee.ja@tpc.co.th> wrote:

>   488  btrfs sub snap mysql_201707230830 mysql
>   489  systemctl start mariadb
>   490  btrfs sub list .
>   491  cat /var/log/mariadb/mariadb.log

OK so mysql_201707230830 once on machine B is inconsistent somehow. So
the questions I have are:

Is mysql_201707230830 on machine A really identical to
mysql_201707230830 on machine B? You can do an rsync -anc (double
check those options) which should independently check whether those
two subvolumes are in fact identical. The -n is a no op, which doesn't
really matter much because as read only subvolumes any attempt to sync
will just result in noisy messages. The -c causes rsync to do its own
checksum verification on both sides.

If the subvolumes are different, we need to find out why.

If the subvolumes are the same, then I wonder if you can reproduce the
mariadb complaint on machine A merely by making a rw snapshot of
mysql_201707230830 and trying to start it. If so, then it's not a send
receive problem, it sounds like the snapshot itself is inconsistent,
maybe mariadb hasn't actually completely closed out the database at
the time the read only snapshot was taken? I'm not sure.

If the subvolumes are different, I'm going to recommend updating at
least the btrfs-progs because 4.4 is kinda old at this point. The
kernel code is what's mainly responsible for the send stream, and the
user space code is mainly responsible for receiving. And I don't off
hand know or want to look up all the send receive changes between 4.4
and 4.12 to speculate on whether this is has already been fixed.

What's the kernel version?

-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

I'll test and response the result to you soon.

I have another question about the combination within Host lxd ubuntu and lxd
container centos

Host Machine X (16.04.3 LTS with lxd 2.0.10 and btrfs-progs v4.4
 kernel 4.4 ) set 1 lvm raw partitions and format to btrfs filesystem (on host)
Partition A  mount to /opt/partA

Container A (CentOS Linux release 7.3.1611 (Core) )  is in Host Machine X
mount /opt/partA to /var/lib/mariadb in container A

Container A making local readonly snapshot everyday

The Options are (online "btrfs send ||ssh btrfs receive" or batch "btrfs send >
file.btrfs" and scp from local to remote then "btrfs receive -f file.btrfs")
1. use btrfs send from Host Machine X to Host Machine Y
2. use btrfs send from Container A to Host Machine Y
3. use btrfs send from Container A to Container B
4. use btrfs send incremental from Host Machine X to Host Machine Y
5. use btrfs send incremental from Container A to Host Machine Y
6. use btrfs send incremental from Container A to Container B


Host Machine Y (16.04.3 LTS with lxd 2.0.10 and btrfs-progs v4.4
 kernel 4.4 ) set 1 lvm raw partitions and format to btrfs filesystem (on host)
Partition B  mount to /opt/partB

Container B (CentOS Linux release 7.3.1611 (Core) ) is in Host Machine Y
mount /opt/partB to /var/lib/mariadb in container B

The Options are (online "btrfs send ||ssh btrfs receive" or batch "btrfs send >
file.btrfs" and scp from local to remote then "btrfs receive -f file.btrfs")

1. use btrfs receive from Host Machine X to Host Machine Y
2. use btrfs receive from Container A to Host Machine Y
3. use btrfs receive from Container A to Container B
4. use btrfs receive incremental from Host Machine X to Host Machine Y
5. use btrfs receive incremental from Container A to Host Machine Y
6. use btrfs receive incremental from Container A to Container B

Could you please suggest me what is the combination that should work properly with
btrfs send and receive?

I plan to setup 2 Site (Production Site and DR site) each site has 1 box (Host
Machine X and Y) and want to test btrfs to send incremental.

Best Regards,

Siranee Jarwachirakul.


> On Wed, Aug 9, 2017 at 12:36 AM,  <siranee.ja@tpc.co.th> wrote:
>
>>   488  btrfs sub snap mysql_201707230830 mysql
>>   489  systemctl start mariadb
>>   490  btrfs sub list .
>>   491  cat /var/log/mariadb/mariadb.log
>
> OK so mysql_201707230830 once on machine B is inconsistent somehow. So
> the questions I have are:
>
> Is mysql_201707230830 on machine A really identical to
> mysql_201707230830 on machine B? You can do an rsync -anc (double
> check those options) which should independently check whether those
> two subvolumes are in fact identical. The -n is a no op, which doesn't
> really matter much because as read only subvolumes any attempt to sync
> will just result in noisy messages. The -c causes rsync to do its own
> checksum verification on both sides.
>
> If the subvolumes are different, we need to find out why.
>
> If the subvolumes are the same, then I wonder if you can reproduce the
> mariadb complaint on machine A merely by making a rw snapshot of
> mysql_201707230830 and trying to start it. If so, then it's not a send
> receive problem, it sounds like the snapshot itself is inconsistent,
> maybe mariadb hasn't actually completely closed out the database at
> the time the read only snapshot was taken? I'm not sure.
>
> If the subvolumes are different, I'm going to recommend updating at
> least the btrfs-progs because 4.4 is kinda old at this point. The
> kernel code is what's mainly responsible for the send stream, and the
> user space code is mainly responsible for receiving. And I don't off
> hand know or want to look up all the send receive changes between 4.4
> and 4.12 to speculate on whether this is has already been fixed.
>
> What's the kernel version?
>
> --
> Chris Murphy
>

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi Chris,
The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu lxd If I
want to change the kernel version I have to upgrade the host box.

As you suggest the rsync to  compare the subvolumes. I found the point.
the subvolumes are different only after I start to del old subvolumes on machine A 
the steps are

30 08 * * * root /root/script/backup/backupsnap.sh root password
/var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166
/var/lib/mariadb >> /var/log/btrfs_send.log
30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >>
/var/log/btrfs_del.log


The following script maintain snapshot to currently only 7 snapshots.
[root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh
basepath=$1
keepcount=$2
havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l`
delcount=$[$keepcount-$havecount];
datet=$(date +%Y%m%d%H%M)
echo "Start Delete ${datet}"
if [ $delcount -lt 0 ]; then
# list only snapshot subvolume
for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}`
do
        echo "btrfs sub delete ${basepath}/$i"
        btrfs sub delete ${basepath}/$i
        btrfs sub sync ${basepath}
done
else
        echo "$delcount -gt 0 nothing to delete"
fi
echo "Stop Delete ${datet}"

Does it mean my delete script is not the properly way of the btrfs purge old
snapshot on source?

Best Regards,

Siranee Jarwachirakul.

> On Wed, Aug 9, 2017 at 12:36 AM,  <siranee.ja@tpc.co.th> wrote:
>
>>   488  btrfs sub snap mysql_201707230830 mysql
>>   489  systemctl start mariadb
>>   490  btrfs sub list .
>>   491  cat /var/log/mariadb/mariadb.log
>
> OK so mysql_201707230830 once on machine B is inconsistent somehow. So
> the questions I have are:
>
> Is mysql_201707230830 on machine A really identical to
> mysql_201707230830 on machine B? You can do an rsync -anc (double
> check those options) which should independently check whether those
> two subvolumes are in fact identical. The -n is a no op, which doesn't
> really matter much because as read only subvolumes any attempt to sync
> will just result in noisy messages. The -c causes rsync to do its own
> checksum verification on both sides.
>
> If the subvolumes are different, we need to find out why.
>
> If the subvolumes are the same, then I wonder if you can reproduce the
> mariadb complaint on machine A merely by making a rw snapshot of
> mysql_201707230830 and trying to start it. If so, then it's not a send
> receive problem, it sounds like the snapshot itself is inconsistent,
> maybe mariadb hasn't actually completely closed out the database at
> the time the read only snapshot was taken? I'm not sure.
>
> If the subvolumes are different, I'm going to recommend updating at
> least the btrfs-progs because 4.4 is kinda old at this point. The
> kernel code is what's mainly responsible for the send stream, and the
> user space code is mainly responsible for receiving. And I don't off
> hand know or want to look up all the send receive changes between 4.4
> and 4.12 to speculate on whether this is has already been fixed.
>
> What's the kernel version?
>
> --
> Chris Murphy
>

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Sorry Chris,

I forgot to send the diff from rsync -anc result

the source container A start data as data on snapshot  mysql_201708040830

[root@backuplogC7 tmp]# ls -l /var/lib/mariadb
total 0
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql
drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708050830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708060830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708070830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708080830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708090830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708100830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708110830

the dest    container B start data as data on snapshot mysql_201708070830

[root@joytest tmp]# ls -l /var/lib/mariadb
total 0
drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql
drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708070830
drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708080830
drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708090830
drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708100830
drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708110830



tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt
dest_mysql_201708110830.txt
1c1
< drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708110830
---
> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708110830
5c5
< -rw-rwx---     5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1
---
> -rw-rwx---     5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1

Best Regards,
Siranee Jaraswachirakul.

> Hi Chris,
> The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu lxd If I
> want to change the kernel version I have to upgrade the host box.
>
> As you suggest the rsync to  compare the subvolumes. I found the point.
> the subvolumes are different only after I start to del old subvolumes on machine A
> the steps are
>
> 30 08 * * * root /root/script/backup/backupsnap.sh root password
> /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
> 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166
> /var/lib/mariadb >> /var/log/btrfs_send.log
> 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >>
> /var/log/btrfs_del.log
>
>
> The following script maintain snapshot to currently only 7 snapshots.
> [root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh
> basepath=$1
> keepcount=$2
> havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l`
> delcount=$[$keepcount-$havecount];
> datet=$(date +%Y%m%d%H%M)
> echo "Start Delete ${datet}"
> if [ $delcount -lt 0 ]; then
> # list only snapshot subvolume
> for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}`
> do
>         echo "btrfs sub delete ${basepath}/$i"
>         btrfs sub delete ${basepath}/$i
>         btrfs sub sync ${basepath}
> done
> else
>         echo "$delcount -gt 0 nothing to delete"
> fi
> echo "Stop Delete ${datet}"
>
> Does it mean my delete script is not the properly way of the btrfs purge old
> snapshot on source?
>
> Best Regards,
>
> Siranee Jarwachirakul.
>
>> On Wed, Aug 9, 2017 at 12:36 AM,  <siranee.ja@tpc.co.th> wrote:
>>
>>>   488  btrfs sub snap mysql_201707230830 mysql
>>>   489  systemctl start mariadb
>>>   490  btrfs sub list .
>>>   491  cat /var/log/mariadb/mariadb.log
>>
>> OK so mysql_201707230830 once on machine B is inconsistent somehow. So
>> the questions I have are:
>>
>> Is mysql_201707230830 on machine A really identical to
>> mysql_201707230830 on machine B? You can do an rsync -anc (double
>> check those options) which should independently check whether those
>> two subvolumes are in fact identical. The -n is a no op, which doesn't
>> really matter much because as read only subvolumes any attempt to sync
>> will just result in noisy messages. The -c causes rsync to do its own
>> checksum verification on both sides.
>>
>> If the subvolumes are different, we need to find out why.
>>
>> If the subvolumes are the same, then I wonder if you can reproduce the
>> mariadb complaint on machine A merely by making a rw snapshot of
>> mysql_201707230830 and trying to start it. If so, then it's not a send
>> receive problem, it sounds like the snapshot itself is inconsistent,
>> maybe mariadb hasn't actually completely closed out the database at
>> the time the read only snapshot was taken? I'm not sure.
>>
>> If the subvolumes are different, I'm going to recommend updating at
>> least the btrfs-progs because 4.4 is kinda old at this point. The
>> kernel code is what's mainly responsible for the send stream, and the
>> user space code is mainly responsible for receiving. And I don't off
>> hand know or want to look up all the send receive changes between 4.4
>> and 4.12 to speculate on whether this is has already been fixed.
>>
>> What's the kernel version?
>>
>> --
>> Chris Murphy
>>
>
>

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Thu, Aug 10, 2017 at 10:40 PM,  <siranee.ja@tpc.co.th> wrote:
> Hi Chris,
> The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu lxd If I
> want to change the kernel version I have to upgrade the host box.


I can't parse what kernel that really is compared to upstream, which
is at 4.4.80.



> As you suggest the rsync to  compare the subvolumes. I found the point.
> the subvolumes are different only after I start to del old subvolumes on machine A
> the steps are

You're saying a read only snapshot's contents are changing after
deleting other (child) snapshots?

If you have a subvolume "subvolume" and you make read only snapshots
of it over time "snapshot1" "snapshot2" "snapshot3" "snapshot4"
"snapshot5" "snapshot6" on both machine A and machine B.

And rsync -anc machineA/snapshot6 machineB/snapshot6 they are identical.

And then you delete "snapshot1" "snapshot2" "snapshot3" "snapshot4" on
machine A.

And rsync -anc machineA/snapshot6 machineB/snapshot6 they are no
longer identical?

I've never heard of this before.


>
> 30 08 * * * root /root/script/backup/backupsnap.sh root password
> /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
> 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166
> /var/lib/mariadb >> /var/log/btrfs_send.log
> 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >>
> /var/log/btrfs_del.log

I have no idea what this means or why it's relevant.


>
>
> The following script maintain snapshot to currently only 7 snapshots.
> [root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh
> basepath=$1
> keepcount=$2
> havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l`
> delcount=$[$keepcount-$havecount];
> datet=$(date +%Y%m%d%H%M)
> echo "Start Delete ${datet}"
> if [ $delcount -lt 0 ]; then
> # list only snapshot subvolume
> for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}`
> do
>         echo "btrfs sub delete ${basepath}/$i"
>         btrfs sub delete ${basepath}/$i
>         btrfs sub sync ${basepath}


Seems sane.


> Does it mean my delete script is not the properly way of the btrfs purge old
> snapshot on source?


I don't think so. But I still don't understand the problem, and what
exactly has changed and when, and it might be my confusion. But if you
have an ro snapshot that's correct at one moment, but then changes
when some other (child) snapshots are deleted, that's pretty
remarkable so I have to assume I'm confused.


-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Fri, Aug 11, 2017 at 12:00 AM,  <siranee.ja@tpc.co.th> wrote:
> Sorry Chris,
>
> I forgot to send the diff from rsync -anc result
>
> the source container A start data as data on snapshot  mysql_201708040830
>
> [root@backuplogC7 tmp]# ls -l /var/lib/mariadb
> total 0
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql
> drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708050830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708060830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708070830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708080830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708090830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708100830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708110830
>
> the dest    container B start data as data on snapshot mysql_201708070830
>
> [root@joytest tmp]# ls -l /var/lib/mariadb
> total 0
> drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql
> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708070830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708080830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708090830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708100830
> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708110830
>
>
>
> tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt
> dest_mysql_201708110830.txt
> 1c1
> < drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708110830
> ---
>> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708110830
> 5c5
> < -rw-rwx---     5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1
> ---
>> -rw-rwx---     5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1
>

I don't really understand this. I don't see the actual rsync -anc
command, what I see is a diff of two text files whose contents I also
don't understand. So I have to guess that ib_logfile1 is a file inside
of a snapshot on machine A, that was btrfs send -p / receive to
machine B and is now different for some reason?

If the problem is that ib_logfile1 is wrong only on machine B after
Btrfs send receive, that suggests it might be a network problem. The
Btrfs send receive stream only checksums btrfs metadata (the internal
commands in the stream). The data is not checksummed so it is possible
an uncaught network error can inject silent data corruption which
Btrfs will not catch - it's just the normal TCP/IP network
checksumming happening.

Anyway, I'm still confused whether the problem is a change only during
send/receive, or if there's a change happening on a machine in
isolation just when you delete other snapshots.


-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

[-- Attachment #1: Type: text/plain, Size: 3251 bytes --]

Hi Chris,

I explain what I have done in the attached file.

Please suggest me what I should do next?

I plan to reproduce the data mannually for the daily inserted data to check the
detail step by step to know exactly which steps make the btrfs send /receive result
diff on the Second box (Machine Y).  I will start with the first btrfs snapshot that
I already had (mysql_201707210830).


Do you think I should do others?

Best Regards,
Siranee Jaraswachirakul.

> On Fri, Aug 11, 2017 at 12:00 AM,  <siranee.ja@tpc.co.th> wrote:
>> Sorry Chris,
>>
>> I forgot to send the diff from rsync -anc result
>>
>> the source container A start data as data on snapshot  mysql_201708040830
>>
>> [root@backuplogC7 tmp]# ls -l /var/lib/mariadb
>> total 0
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql
>> drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708050830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708060830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708070830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708080830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708090830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708100830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708110830
>>
>> the dest    container B start data as data on snapshot mysql_201708070830
>>
>> [root@joytest tmp]# ls -l /var/lib/mariadb
>> total 0
>> drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708070830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708080830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708090830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708100830
>> drwxrwxr-x+ 1 mysql mysql 260 Aug  7 09:26 mysql_201708110830
>>
>>
>>
>> tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt
>> dest_mysql_201708110830.txt
>> 1c1
>> < drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708110830
>> ---
>>> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708110830
>> 5c5
>> < -rw-rwx---     5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1
>> ---
>>> -rw-rwx---     5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1
>>
>
> I don't really understand this. I don't see the actual rsync -anc
> command, what I see is a diff of two text files whose contents I also
> don't understand. So I have to guess that ib_logfile1 is a file inside
> of a snapshot on machine A, that was btrfs send -p / receive to
> machine B and is now different for some reason?
>
> If the problem is that ib_logfile1 is wrong only on machine B after
> Btrfs send receive, that suggests it might be a network problem. The
> Btrfs send receive stream only checksums btrfs metadata (the internal
> commands in the stream). The data is not checksummed so it is possible
> an uncaught network error can inject silent data corruption which
> Btrfs will not catch - it's just the normal TCP/IP network
> checksumming happening.
>
> Anyway, I'm still confused whether the problem is a change only during
> send/receive, or if there's a change happening on a machine in
> isolation just when you delete other snapshots.
>
>
> --
> Chris Murphy
>

[-- Attachment #2: detail_explain_more.txt --]
[-- Type: text/plain, Size: 53585 bytes --]

Hi Chris,

I would like to describe what I have done.

Exactly I have 3 boxes of ubuntu 16.04 LTS in mycompany environment and the other on hosting provider.

Overall steps
First -> started first box in company environment and send btrfs offline (batch) to receive with scp to remote box on hosting provider.
After incorrect result of incremental I had to send the new base to start over on remote box around 3 times I changed to the Second step.
Second -> Setup the second box in company environment on the different network segment of the first box and still send btrfs offline (batch) to receive with scp from first box to second box
Yes the result still the same as the First even different date but the incremental unusable vary (not only on the second incremental send)
Third -> Change from test with 2 boxes to be only one box with 2 containers and use online send and receive btrfs as you see the current result of btrfs sub volumes.
The result still incorrect some day which I had to start send the new base btrfs snapshot whenever it started to be incorrect.
I suspect the send/recieve machanism then I uses the snapshot on mysql_201708040830 to be the start base on the source btrfs in container A (backuplogC7) (As I said the local without send/receive it work) until now.
Since   mysql_201708040830 the incremental still work until mysql_201708110830 it start to incorrect.




1. I started with the first box (Machine X called "virtualtrust3") 
tpcorp@virtualtrust3:~$ uname -a
Linux virtualtrust3 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
tpcorp@virtualtrust3:~$ cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial


2. I have a container for the mariadb databases with have data inserted every day at 7:30 AM.

tpcorp@virtualtrust3:~$ lxc list backuplog
+-------------+---------+-----------------------+------+------------+-----------+
|    NAME     |  STATE  |         IPV4          | IPV6 |    TYPE    | SNAPSHOTS |
+-------------+---------+-----------------------+------+------------+-----------+
| backuplogC7 | RUNNING | 192.168.47.185 (eth0) |      | PERSISTENT | 0         |
+-------------+---------+-----------------------+------+------------+-----------+

inside container backuplogC7 (Container A) 
tpcorp@virtualtrust3:~$ lxc exec backuplogC7 -- bash
[root@backuplogC7 ~]# uname -a
Linux backuplogC7 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@backuplogC7 ~]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
[root@backuplogC7 ~]#

[root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
ID 257 gen 532 top level 5 path mysql
ID 315 gen 491 top level 5 path mysql_201708050830
ID 316 gen 498 top level 5 path mysql_201708060830
ID 317 gen 503 top level 5 path mysql_201708070830
ID 318 gen 507 top level 5 path mysql_201708080830
ID 319 gen 514 top level 5 path mysql_201708090830
ID 320 gen 524 top level 5 path mysql_201708100830
ID 321 gen 529 top level 5 path mysql_201708110830
[root@backuplogC7 ~]# ls -l /var/lib/mariadb
total 0
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708050830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708060830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708070830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708080830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708090830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708100830
drwxrwxr-x+ 1 mysql mysql 260 Aug  4 13:10 mysql_201708110830
[root@backuplogC7 ~]#

3. First time I created the container A I took the first snapshot at mysql_201707210830 

[root@backuplogC7 ~]# cat /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

30 08 * * * root /root/script/backup/backupsnap.sh root password /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166 /var/lib/mariadb >> /var/log/btrfs_send.log
0 19 * * * root /root/script/backup/doBackup_file_backuplog.sh
30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >> /var/log/btrfs_del.log

# temp
##00 09 * * * root /root/script/backupbtrfs_all.sh /var/lib/mariadb /backupbtrfs mysql_201707210830

You will see the /root/script/backupbtrfs_all.sh because I stated with batch send and receive to the box on hosting provider.
Daily Routine Steps to test still be the same except change from offline (batch) to online 
3.1 I had application to insert data to mariadb everyday around 7:30 AM
3.2 the snapshot has been took every day around 8:30 AM with the backupsnap.sh script/backup/backupsnap
3.3 First test I have to use the 09:00 AM /root/script/backupbtrfs_all.sh to btrfs send to files on /backupbtrfs with the starting snapshot mysql_201707210830
3.4 at 19:00 root/script/backup/doBackup_file_backuplog.sh for the ordinary online backup of mariadb just mysqldump
3.5 after backup ordinary I delete the old snapshot to keep only 7 snapshots


Now the offline btrfs send is commented out
and the Daily Routine Steps are
3.1 I had application to insert data to mariadb everyday around 7:30 AM
3.2 the snapshot has been took every day around 8:30 AM with the backupsnap.sh script/backup/backupsnap
3.3 09:05 AM /root/script/backupbtrfs_inc.sh to btrfs send to receive at destination  online
3.4 at 19:00 root/script/backup/doBackup_file_backuplog.sh for the ordinary online backup of mariadb just mysqldump
3.5 after backup ordinary I delete the old snapshot to keep only 7 snapshots



As you would like to see the rsync command

[root@backuplogC7 ~]# history|grep rsync
  436  rsync
  437  yum install rsync
  444  rsync -anc mysql_201708110830 > /tmp/source_mysql_201708110830.txt
  453  history|grep rsync
  454  rsync -anc mysql_201708100830 > /tmp/source_mysql_201708100830.txt
  455  rsync -anc /var/lib/mariadb/mysql_201708100830 > /tmp/source_mysql_201708100830.txt
  462  history|grep rsync
  463  rsync -anc /var/lib/mariadb/mysql_201708090830 > /tmp/source_mysql_201708090830.txt
  464  rsync -anc /var/lib/mariadb/mysql_201708080830 > /tmp/source_mysql_201708080830.txt
  465  rsync -anc /var/lib/mariadb/mysql_201708070830 > /tmp/source_mysql_201708070830.txt

[root@joytest ~]# history|grep rsync
  423  rsync -anc /var/lib/mariadb/mysql_201708090830
  424  yum install rsync
  425  rsync -anc /var/lib/mariadb/mysql_201708090830
  439  rsync -anc mysql_201708110830 > /tmp/dest_mysql_201708110830.txt
  448  hsitory|grep rsync
  449  history|grep rsync
  450  rsync -anc /var/lib/mariadb/mysql_201708100830  > /tmp/dest_mysql_201708100830.txt
  451  rsync -anc /var/lib/mariadb/mysql_201708090830  > /tmp/dest_mysql_201708090830.txt
  452  rsync -anc /var/lib/mariadb/mysql_201708080830  > /tmp/dest_mysql_201708080830.txt
  453  rsync -anc /var/lib/mariadb/mysql_201708070830  > /tmp/dest_mysql_201708070830.txt
  461  rsync -anc mysql_201708110830 > /tmp/dest_mysql_201708110830.txt2
  462  rsync -anc /var/lib/mariadb/mysql_201708110830 > /tmp/dest_mysql_201708110830.txt2
  485  history|grep rsync
  486  rsync -anc /var/lib/mariadb/mysql_201708110830 > /tmp/dest_mysql_201708110830.txt3
  501  history|grep rsync
  
tpcorp@virtualtrust3:~$ history|grep pull

1966  lxc file pull backuplogC7/tmp/source_/tmp/dest_mysql_201708110830.txt
 1967  lxc file pull backuplogC7/tmp/source_mysql_201708110830.txt .
 1968  lxc file pull joytest/tmp/dest_mysql_201708110830.txt .
 1975  history|grep pull
 1976  lxc file pull backuplogC7/tmp/source_mysql_201708100830.txt .
 1977  lxc file pull joytest/tmp/dest_mysql_201708100830.txt .
 1982  lxc file pull joytest/tmp/dest_mysql_*.txt .
 1983  history|grep pull
 1984  lxc file pull joytest/tmp/dest_mysql_201708090830.txt .
 1985  lxc file pull joytest/tmp/dest_mysql_201708080830.txt .
 1986  lxc file pull joytest/tmp/dest_mysql_201708070830.txt .
 1987  lxc file pull backuplogC7/tmp/source_mysql_201708090830.txt .
 1988  lxc file pull backuplogC7/tmp/source_mysql_201708080830.txt .
 1989  lxc file pull backuplogC7/tmp/source_mysql_201708070830.txt .


tpcorp@virtualtrust3:~$ history|grep diff
 1970  diff source_mysql_201708110830.txt  dest_mysql_201708110830.txt
 1972  diff source_mysql_201708110830.txt  dest_mysql_201708110830.txt
 1978  diff source_mysql_201708100830.txt dest_mysql_201708100830.txt
 1991  diff source_mysql_201708070830.txt dest_mysql_201708070830.txt
 1992  diff source_mysql_201708080830.txt dest_mysql_201708080830.txt
 1993  diff source_mysql_201708090830.txt dest_mysql_201708090830.txt
 1994  diff source_mysql_201708100830.txt dest_mysql_201708100830.txt
 1995  diff source_mysql_201708110830.txt dest_mysql_201708110830.txt

"The diff command just display only the difference between source and destination"

"The following means the source differs from destination not only subvolume "mysql_201708110830" diff at datetime but also file "ib_logfile1" which diff only one second. The mariadb can not start.

tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt  dest_mysql_201708110830.txt
1c1
< drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708110830
---
> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708110830
5c5
< -rw-rwx---     5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1
---
> -rw-rwx---     5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1


"The following means the source differs from destination only diff at datetime

tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708100830.txt dest_mysql_201708100830.txt
1c1
< drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708100830
---
> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708100830
tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708090830.txt dest_mysql_201708090830.txt
1c1
< drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708090830
---
> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708090830
tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708080830.txt dest_mysql_201708080830.txt
1c1
< drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708080830
---
> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708080830
tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708070830.txt dest_mysql_201708070830.txt
1c1
< drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708070830
---
> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708070830

The Last snapshot send /receive

tpcorp@virtualtrust3:/tmp$ lxc file pull backuplogC7/tmp/source_mysql_201708120830.txt .
tpcorp@virtualtrust3:/tmp$ lxc file pull joytest/tmp/dest_mysql_201708120830.txt .
tpcorp@virtualtrust3:/tmp$ diff /tmp/source_mysql_201708120830.txt /tmp/dest_mysql_201708120830.txt
1c1
< drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708120830
---
> drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708120830
5c5
< -rw-rwx---     5242880 2017/08/10 07:30:35 mysql_201708120830/ib_logfile1
---
> -rw-rwx---     5242880 2017/08/07 07:30:36 mysql_201708120830/ib_logfile1
tpcorp@virtualtrust3:/tmp$


-----------------
As you request for the real output on screen of source (container A) rsync
-----------------

[root@backuplogC7 ~]# rsync -anc /var/lib/mariadb/mysql_201708110830
drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708110830
-rw-rwx---       16384 2017/08/04 13:08:08 mysql_201708110830/aria_log.00000001
-rw-rwx---          52 2017/08/04 13:08:08 mysql_201708110830/aria_log_control
-rw-rwx---     5242880 2017/08/11 08:29:07 mysql_201708110830/ib_logfile0
-rw-rwx---     5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1
-rw-rwx---   706740224 2017/08/11 08:29:07 mysql_201708110830/ibdata1
-rw-rw----         264 2017/03/30 08:40:47 mysql_201708110830/mysql-bin.000001
-rw-rw----          19 2017/03/30 08:38:29 mysql_201708110830/mysql-bin.index
drwx------         644 2017/03/31 09:39:55 mysql_201708110830/backups_db
-rw-rw----          61 2017/03/28 15:14:28 mysql_201708110830/backups_db/db.opt
-rw-rw----        8554 2017/03/28 15:17:54 mysql_201708110830/backups_db/ints.frm
-rw-rw----        8766 2017/03/31 09:14:26 mysql_201708110830/backups_db/tpc_backup_detail.frm
-rw-rw----        8836 2017/03/31 09:25:41 mysql_201708110830/backups_db/tpc_backup_header.frm
-rw-rw----        9020 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_history.frm
-rw-rw----        8730 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_level_master.frm
-rw-rw----        8774 2017/03/31 09:39:55 mysql_201708110830/backups_db/tpc_backup_schedule_master.frm
-rw-rw----        8774 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_master_extra.frm
-rw-rw----        8774 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_schedule_old.frm
-rw-rw----        8672 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_server.frm
-rw-rw----        8888 2017/03/31 09:13:41 mysql_201708110830/backups_db/tpc_backup_step.frm
-rw-rw----        8722 2017/03/28 15:19:10 mysql_201708110830/backups_db/tpc_backup_user.frm
-rw-rw----        1851 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_backup_header_status.frm
-rw-rw----         716 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_calendar.frm
-rw-rw----        1611 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_dup_backup.frm
drwxrwx---        2094 2017/03/28 14:54:08 mysql_201708110830/mysql
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYI
-rw-rw----        8820 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.frm
-rw-rw----         440 2017/08/01 17:00:06 mysql_201708110830/mysql/db.MYD
-rw-rw----        5120 2017/08/02 08:30:01 mysql_201708110830/mysql/db.MYI
-rw-rw----        9582 2017/03/28 14:54:06 mysql_201708110830/mysql/db.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYD
-rw-rw----        2048 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYI
-rw-rw----       10239 2017/03/28 14:54:06 mysql_201708110830/mysql/event.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYD
-rw-rw----        1024 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYI
-rw-rw----        8665 2017/03/28 14:54:06 mysql_201708110830/mysql/func.frm
-rw-rwx---          35 2017/08/11 08:30:01 mysql_201708110830/mysql/general_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.CSV
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.frm
-rw-rw----        1092 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYD
-rw-rw----        3072 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYI
-rw-rw----        8700 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.frm
-rw-rw----       91408 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYD
-rw-rw----       16384 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYI
-rw-rw----        8612 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.frm
-rw-rw----        9252 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYD
-rw-rw----       17408 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYI
-rw-rw----        8630 2017/03/28 14:54:06 mysql_201708110830/mysql/help_relation.frm
-rw-rw----      450388 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYD
-rw-rw----       20480 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYI
-rw-rw----        8770 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYI
-rw-rw----        9510 2017/03/28 14:54:07 mysql_201708110830/mysql/host.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYI
-rw-rw----        8778 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYI
-rw-rw----        8586 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYI
-rw-rw----       10012 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYI
-rw-rw----        8875 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.frm
-rw-rw----        1386 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYD
-rw-rw----        5120 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYI
-rw-rw----        8800 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYI
-rw-rw----        8838 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.frm
-rw-rwx---          35 2017/08/11 08:30:01 mysql_201708110830/mysql/slow_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.CSV
-rw-rwx---        8976 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYI
-rw-rw----        8955 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYI
-rw-rw----        8636 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYI
-rw-rw----        8624 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYI
-rw-rw----        8606 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYI
-rw-rw----        8686 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYI
-rw-rw----        8748 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.frm
-rw-rw----         668 2017/08/01 17:00:06 mysql_201708110830/mysql/user.MYD
-rw-rw----        2048 2017/08/02 08:30:01 mysql_201708110830/mysql/user.MYI
-rw-rw----       10630 2017/03/28 14:54:07 mysql_201708110830/mysql/user.frm
drwxrwx---         868 2017/03/22 14:30:05 mysql_201708110830/performance_schema
-rw-rwx---        8624 2017/03/22 14:30:05 mysql_201708110830/performance_schema/cond_instances.frm
-rw-rwx---          61 2017/03/22 14:30:05 mysql_201708110830/performance_schema/db.opt
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_current.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history_long.frm
-rw-rwx---        8878 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_instance.frm
-rw-rwx---        8854 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
-rw-rwx---        8814 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_global_by_event_name.frm
-rw-rwx---        8654 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_instances.frm
-rw-rwx---        8800 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_event_name.frm
-rw-rwx---        8840 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_instance.frm
-rw-rwx---        8684 2017/03/22 14:30:05 mysql_201708110830/performance_schema/mutex_instances.frm
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708110830/performance_schema/performance_timers.frm
-rw-rwx---        8758 2017/03/22 14:30:05 mysql_201708110830/performance_schema/rwlock_instances.frm
-rw-rwx---        8605 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_consumers.frm
-rw-rwx---        8637 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_instruments.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_timers.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/threads.frm
drwx------          12 2017/03/29 13:29:07 mysql_201708110830/test
-rw-rw----          61 2017/03/29 13:11:26 mysql_201708110830/test/db.opt


[root@backuplogC7 ~]# rsync -anc /var/lib/mariadb/mysql_201708120830
drwxrwxr-x         260 2017/08/04 13:10:56 mysql_201708120830
-rw-rwx---       16384 2017/08/04 13:08:08 mysql_201708120830/aria_log.00000001
-rw-rwx---          52 2017/08/04 13:08:08 mysql_201708120830/aria_log_control
-rw-rwx---     5242880 2017/08/12 07:30:46 mysql_201708120830/ib_logfile0
-rw-rwx---     5242880 2017/08/10 07:30:35 mysql_201708120830/ib_logfile1
-rw-rwx---   706740224 2017/08/12 07:30:46 mysql_201708120830/ibdata1
-rw-rw----         264 2017/03/30 08:40:47 mysql_201708120830/mysql-bin.000001
-rw-rw----          19 2017/03/30 08:38:29 mysql_201708120830/mysql-bin.index
drwx------         644 2017/03/31 09:39:55 mysql_201708120830/backups_db
-rw-rw----          61 2017/03/28 15:14:28 mysql_201708120830/backups_db/db.opt
-rw-rw----        8554 2017/03/28 15:17:54 mysql_201708120830/backups_db/ints.frm
-rw-rw----        8766 2017/03/31 09:14:26 mysql_201708120830/backups_db/tpc_backup_detail.frm
-rw-rw----        8836 2017/03/31 09:25:41 mysql_201708120830/backups_db/tpc_backup_header.frm
-rw-rw----        9020 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_history.frm
-rw-rw----        8730 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_level_master.frm
-rw-rw----        8774 2017/03/31 09:39:55 mysql_201708120830/backups_db/tpc_backup_schedule_master.frm
-rw-rw----        8774 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_master_extra.frm
-rw-rw----        8774 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_schedule_old.frm
-rw-rw----        8672 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_server.frm
-rw-rw----        8888 2017/03/31 09:13:41 mysql_201708120830/backups_db/tpc_backup_step.frm
-rw-rw----        8722 2017/03/28 15:19:10 mysql_201708120830/backups_db/tpc_backup_user.frm
-rw-rw----        1851 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_backup_header_status.frm
-rw-rw----         716 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_calendar.frm
-rw-rw----        1611 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_dup_backup.frm
drwxrwx---        2094 2017/03/28 14:54:08 mysql_201708120830/mysql
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYI
-rw-rw----        8820 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.frm
-rw-rw----         440 2017/08/01 17:00:06 mysql_201708120830/mysql/db.MYD
-rw-rw----        5120 2017/08/02 08:30:01 mysql_201708120830/mysql/db.MYI
-rw-rw----        9582 2017/03/28 14:54:06 mysql_201708120830/mysql/db.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYD
-rw-rw----        2048 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYI
-rw-rw----       10239 2017/03/28 14:54:06 mysql_201708120830/mysql/event.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYD
-rw-rw----        1024 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYI
-rw-rw----        8665 2017/03/28 14:54:06 mysql_201708120830/mysql/func.frm
-rw-rwx---          35 2017/08/12 08:30:01 mysql_201708120830/mysql/general_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.CSV
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.frm
-rw-rw----        1092 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYD
-rw-rw----        3072 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYI
-rw-rw----        8700 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.frm
-rw-rw----       91408 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYD
-rw-rw----       16384 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYI
-rw-rw----        8612 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.frm
-rw-rw----        9252 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYD
-rw-rw----       17408 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYI
-rw-rw----        8630 2017/03/28 14:54:06 mysql_201708120830/mysql/help_relation.frm
-rw-rw----      450388 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYD
-rw-rw----       20480 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYI
-rw-rw----        8770 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYI
-rw-rw----        9510 2017/03/28 14:54:07 mysql_201708120830/mysql/host.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYI
-rw-rw----        8778 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYI
-rw-rw----        8586 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYI
-rw-rw----       10012 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYI
-rw-rw----        8875 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.frm
-rw-rw----        1386 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYD
-rw-rw----        5120 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYI
-rw-rw----        8800 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYI
-rw-rw----        8838 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.frm
-rw-rwx---          35 2017/08/12 08:30:01 mysql_201708120830/mysql/slow_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.CSV
-rw-rwx---        8976 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYI
-rw-rw----        8955 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYI
-rw-rw----        8636 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYI
-rw-rw----        8624 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYI
-rw-rw----        8606 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYI
-rw-rw----        8686 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYI
-rw-rw----        8748 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.frm
-rw-rw----         668 2017/08/01 17:00:06 mysql_201708120830/mysql/user.MYD
-rw-rw----        2048 2017/08/02 08:30:01 mysql_201708120830/mysql/user.MYI
-rw-rw----       10630 2017/03/28 14:54:07 mysql_201708120830/mysql/user.frm
drwxrwx---         868 2017/03/22 14:30:05 mysql_201708120830/performance_schema
-rw-rwx---        8624 2017/03/22 14:30:05 mysql_201708120830/performance_schema/cond_instances.frm
-rw-rwx---          61 2017/03/22 14:30:05 mysql_201708120830/performance_schema/db.opt
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_current.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history_long.frm
-rw-rwx---        8878 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_instance.frm
-rw-rwx---        8854 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
-rw-rwx---        8814 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_global_by_event_name.frm
-rw-rwx---        8654 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_instances.frm
-rw-rwx---        8800 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_event_name.frm
-rw-rwx---        8840 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_instance.frm
-rw-rwx---        8684 2017/03/22 14:30:05 mysql_201708120830/performance_schema/mutex_instances.frm
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708120830/performance_schema/performance_timers.frm
-rw-rwx---        8758 2017/03/22 14:30:05 mysql_201708120830/performance_schema/rwlock_instances.frm
-rw-rwx---        8605 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_consumers.frm
-rw-rwx---        8637 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_instruments.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_timers.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/threads.frm
drwx------          12 2017/03/29 13:29:07 mysql_201708120830/test
-rw-rw----          61 2017/03/29 13:11:26 mysql_201708120830/test/db.opt


-----------------
As you request for the real output on screen of source (container B) rsync
----------------- 

tpcorp@virtualtrust3:/tmp$ lxc exec joytest -- bash
[root@joytest ~]# rsync -anc /var/lib/mariadb/mysql_201708110830
drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708110830
-rw-rwx---       16384 2017/08/04 13:08:08 mysql_201708110830/aria_log.00000001
-rw-rwx---          52 2017/08/04 13:08:08 mysql_201708110830/aria_log_control
-rw-rwx---     5242880 2017/08/11 08:29:07 mysql_201708110830/ib_logfile0
-rw-rwx---     5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1
-rw-rwx---   706740224 2017/08/11 08:29:07 mysql_201708110830/ibdata1
-rw-rw----         264 2017/03/30 08:40:47 mysql_201708110830/mysql-bin.000001
-rw-rw----          19 2017/03/30 08:38:29 mysql_201708110830/mysql-bin.index
drwx------         644 2017/03/31 09:39:55 mysql_201708110830/backups_db
-rw-rw----          61 2017/03/28 15:14:28 mysql_201708110830/backups_db/db.opt
-rw-rw----        8554 2017/03/28 15:17:54 mysql_201708110830/backups_db/ints.frm
-rw-rw----        8766 2017/03/31 09:14:26 mysql_201708110830/backups_db/tpc_backup_detail.frm
-rw-rw----        8836 2017/03/31 09:25:41 mysql_201708110830/backups_db/tpc_backup_header.frm
-rw-rw----        9020 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_history.frm
-rw-rw----        8730 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_level_master.frm
-rw-rw----        8774 2017/03/31 09:39:55 mysql_201708110830/backups_db/tpc_backup_schedule_master.frm
-rw-rw----        8774 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_master_extra.frm
-rw-rw----        8774 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_schedule_old.frm
-rw-rw----        8672 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_server.frm
-rw-rw----        8888 2017/03/31 09:13:41 mysql_201708110830/backups_db/tpc_backup_step.frm
-rw-rw----        8722 2017/03/28 15:19:10 mysql_201708110830/backups_db/tpc_backup_user.frm
-rw-rw----        1851 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_backup_header_status.frm
-rw-rw----         716 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_calendar.frm
-rw-rw----        1611 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_dup_backup.frm
drwxrwx---        2094 2017/03/28 14:54:08 mysql_201708110830/mysql
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYI
-rw-rw----        8820 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.frm
-rw-rw----         440 2017/08/01 17:00:06 mysql_201708110830/mysql/db.MYD
-rw-rw----        5120 2017/08/02 08:30:01 mysql_201708110830/mysql/db.MYI
-rw-rw----        9582 2017/03/28 14:54:06 mysql_201708110830/mysql/db.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYD
-rw-rw----        2048 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYI
-rw-rw----       10239 2017/03/28 14:54:06 mysql_201708110830/mysql/event.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYD
-rw-rw----        1024 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYI
-rw-rw----        8665 2017/03/28 14:54:06 mysql_201708110830/mysql/func.frm
-rw-rwx---          35 2017/08/11 08:30:01 mysql_201708110830/mysql/general_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.CSV
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.frm
-rw-rw----        1092 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYD
-rw-rw----        3072 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYI
-rw-rw----        8700 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.frm
-rw-rw----       91408 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYD
-rw-rw----       16384 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYI
-rw-rw----        8612 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.frm
-rw-rw----        9252 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYD
-rw-rw----       17408 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYI
-rw-rw----        8630 2017/03/28 14:54:06 mysql_201708110830/mysql/help_relation.frm
-rw-rw----      450388 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYD
-rw-rw----       20480 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYI
-rw-rw----        8770 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYI
-rw-rw----        9510 2017/03/28 14:54:07 mysql_201708110830/mysql/host.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYI
-rw-rw----        8778 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYI
-rw-rw----        8586 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYI
-rw-rw----       10012 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYI
-rw-rw----        8875 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.frm
-rw-rw----        1386 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYD
-rw-rw----        5120 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYI
-rw-rw----        8800 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYI
-rw-rw----        8838 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.frm
-rw-rwx---          35 2017/08/11 08:30:01 mysql_201708110830/mysql/slow_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.CSV
-rw-rwx---        8976 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYI
-rw-rw----        8955 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYI
-rw-rw----        8636 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYI
-rw-rw----        8624 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYI
-rw-rw----        8606 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYI
-rw-rw----        8686 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYI
-rw-rw----        8748 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.frm
-rw-rw----         668 2017/08/01 17:00:06 mysql_201708110830/mysql/user.MYD
-rw-rw----        2048 2017/08/02 08:30:01 mysql_201708110830/mysql/user.MYI
-rw-rw----       10630 2017/03/28 14:54:07 mysql_201708110830/mysql/user.frm
drwxrwx---         868 2017/03/22 14:30:05 mysql_201708110830/performance_schema
-rw-rwx---        8624 2017/03/22 14:30:05 mysql_201708110830/performance_schema/cond_instances.frm
-rw-rwx---          61 2017/03/22 14:30:05 mysql_201708110830/performance_schema/db.opt
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_current.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history_long.frm
-rw-rwx---        8878 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_instance.frm
-rw-rwx---        8854 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
-rw-rwx---        8814 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_global_by_event_name.frm
-rw-rwx---        8654 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_instances.frm
-rw-rwx---        8800 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_event_name.frm
-rw-rwx---        8840 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_instance.frm
-rw-rwx---        8684 2017/03/22 14:30:05 mysql_201708110830/performance_schema/mutex_instances.frm
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708110830/performance_schema/performance_timers.frm
-rw-rwx---        8758 2017/03/22 14:30:05 mysql_201708110830/performance_schema/rwlock_instances.frm
-rw-rwx---        8605 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_consumers.frm
-rw-rwx---        8637 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_instruments.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_timers.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/threads.frm
drwx------          12 2017/03/29 13:29:07 mysql_201708110830/test
-rw-rw----          61 2017/03/29 13:11:26 mysql_201708110830/test/db.opt

[root@joytest ~]# rsync -anc /var/lib/mariadb/mysql_201708120830
drwxrwxr-x         260 2017/08/07 09:26:32 mysql_201708120830
-rw-rwx---       16384 2017/08/04 13:08:08 mysql_201708120830/aria_log.00000001
-rw-rwx---          52 2017/08/04 13:08:08 mysql_201708120830/aria_log_control
-rw-rwx---     5242880 2017/08/12 07:30:46 mysql_201708120830/ib_logfile0
-rw-rwx---     5242880 2017/08/07 07:30:36 mysql_201708120830/ib_logfile1
-rw-rwx---   706740224 2017/08/12 07:30:46 mysql_201708120830/ibdata1
-rw-rw----         264 2017/03/30 08:40:47 mysql_201708120830/mysql-bin.000001
-rw-rw----          19 2017/03/30 08:38:29 mysql_201708120830/mysql-bin.index
drwx------         644 2017/03/31 09:39:55 mysql_201708120830/backups_db
-rw-rw----          61 2017/03/28 15:14:28 mysql_201708120830/backups_db/db.opt
-rw-rw----        8554 2017/03/28 15:17:54 mysql_201708120830/backups_db/ints.frm
-rw-rw----        8766 2017/03/31 09:14:26 mysql_201708120830/backups_db/tpc_backup_detail.frm
-rw-rw----        8836 2017/03/31 09:25:41 mysql_201708120830/backups_db/tpc_backup_header.frm
-rw-rw----        9020 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_history.frm
-rw-rw----        8730 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_level_master.frm
-rw-rw----        8774 2017/03/31 09:39:55 mysql_201708120830/backups_db/tpc_backup_schedule_master.frm
-rw-rw----        8774 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_master_extra.frm
-rw-rw----        8774 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_schedule_old.frm
-rw-rw----        8672 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_server.frm
-rw-rw----        8888 2017/03/31 09:13:41 mysql_201708120830/backups_db/tpc_backup_step.frm
-rw-rw----        8722 2017/03/28 15:19:10 mysql_201708120830/backups_db/tpc_backup_user.frm
-rw-rw----        1851 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_backup_header_status.frm
-rw-rw----         716 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_calendar.frm
-rw-rw----        1611 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_dup_backup.frm
drwxrwx---        2094 2017/03/28 14:54:08 mysql_201708120830/mysql
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYI
-rw-rw----        8820 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.frm
-rw-rw----         440 2017/08/01 17:00:06 mysql_201708120830/mysql/db.MYD
-rw-rw----        5120 2017/08/02 08:30:01 mysql_201708120830/mysql/db.MYI
-rw-rw----        9582 2017/03/28 14:54:06 mysql_201708120830/mysql/db.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYD
-rw-rw----        2048 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYI
-rw-rw----       10239 2017/03/28 14:54:06 mysql_201708120830/mysql/event.frm
-rw-rw----           0 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYD
-rw-rw----        1024 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYI
-rw-rw----        8665 2017/03/28 14:54:06 mysql_201708120830/mysql/func.frm
-rw-rwx---          35 2017/08/12 08:30:01 mysql_201708120830/mysql/general_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.CSV
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.frm
-rw-rw----        1092 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYD
-rw-rw----        3072 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYI
-rw-rw----        8700 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.frm
-rw-rw----       91408 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYD
-rw-rw----       16384 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYI
-rw-rw----        8612 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.frm
-rw-rw----        9252 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYD
-rw-rw----       17408 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYI
-rw-rw----        8630 2017/03/28 14:54:06 mysql_201708120830/mysql/help_relation.frm
-rw-rw----      450388 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYD
-rw-rw----       20480 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYI
-rw-rw----        8770 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYI
-rw-rw----        9510 2017/03/28 14:54:07 mysql_201708120830/mysql/host.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYI
-rw-rw----        8778 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYI
-rw-rw----        8586 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYD
-rw-rw----        2048 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYI
-rw-rw----       10012 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYI
-rw-rw----        8875 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.frm
-rw-rw----        1386 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYD
-rw-rw----        5120 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYI
-rw-rw----        8800 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYI
-rw-rw----        8838 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.frm
-rw-rwx---          35 2017/08/12 08:30:01 mysql_201708120830/mysql/slow_log.CSM
-rw-rwx---           0 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.CSV
-rw-rwx---        8976 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYD
-rw-rw----        4096 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYI
-rw-rw----        8955 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYI
-rw-rw----        8636 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYI
-rw-rw----        8624 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYI
-rw-rw----        8606 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYI
-rw-rw----        8686 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.frm
-rw-rw----           0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYD
-rw-rw----        1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYI
-rw-rw----        8748 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.frm
-rw-rw----         668 2017/08/01 17:00:06 mysql_201708120830/mysql/user.MYD
-rw-rw----        2048 2017/08/02 08:30:01 mysql_201708120830/mysql/user.MYI
-rw-rw----       10630 2017/03/28 14:54:07 mysql_201708120830/mysql/user.frm
drwxrwx---         868 2017/03/22 14:30:05 mysql_201708120830/performance_schema
-rw-rwx---        8624 2017/03/22 14:30:05 mysql_201708120830/performance_schema/cond_instances.frm
-rw-rwx---          61 2017/03/22 14:30:05 mysql_201708120830/performance_schema/db.opt
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_current.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history.frm
-rw-rwx---        9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history_long.frm
-rw-rwx---        8878 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_instance.frm
-rw-rwx---        8854 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
-rw-rwx---        8814 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_global_by_event_name.frm
-rw-rwx---        8654 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_instances.frm
-rw-rwx---        8800 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_event_name.frm
-rw-rwx---        8840 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_instance.frm
-rw-rwx---        8684 2017/03/22 14:30:05 mysql_201708120830/performance_schema/mutex_instances.frm
-rw-rwx---        8776 2017/03/22 14:30:05 mysql_201708120830/performance_schema/performance_timers.frm
-rw-rwx---        8758 2017/03/22 14:30:05 mysql_201708120830/performance_schema/rwlock_instances.frm
-rw-rwx---        8605 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_consumers.frm
-rw-rwx---        8637 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_instruments.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_timers.frm
-rw-rwx---        8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/threads.frm
drwx------          12 2017/03/29 13:29:07 mysql_201708120830/test
-rw-rw----          61 2017/03/29 13:11:26 mysql_201708120830/test/db.opt

 

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Fri, Aug 11, 2017 at 8:38 PM,  <siranee.ja@tpc.co.th> wrote:
> Hi Chris,
>
> I explain what I have done in the attached file.

That is way too long. And please don't attach files directed at me,
that's not appropriate. Just message the list normally, so it can be
searched by others down the road.

There is a problem with your rsync command, you really don't
understand what I was asking. The whole point of rsync is to compare
the origin and destination subvolumes to see if they are different
from each other. You ran the command on a single subvolume on a single
machine which is pointless.

On machine A:

$ rsync -avnc /mnt/snapshot6/ chris@192.168.1.116:/mnt/snapshot6/
chris@192.168.1.116's password:
sending incremental file list

sent 125 bytes  received 12 bytes  39.14 bytes/sec
total size is 30,686  speedup is 223.99 (DRY RUN)
[chris@f26h ~]$

Because no files are listed, all are confirmed to be identical. If
files are listed, there's a difference.



> Please suggest me what I should do next?

Check this. I wonder if you're running into this obscure bug where
maybe your read only snapshot is not yet synced before it's being
sent. Hence it *is* changing on origin machine, but not because of
other snapshots being deleted. I have no idea if your scripting
handles errors like the bogus stale NFS handle error, maybe it's
possible it happens but you're not seeing it? Anyway, you might try
just adding a single sync right after taking the snapshot, change
nothing else and see if the problem reproduces itself.

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping

I'm pretty sure this is fixed in newer kernels, I haven't run into it
in a long time myself. But I don't know when it was fixed.



-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi Chris,

Sorry for the misunderstanding and inappropriate attached the file (I thought that
too long text in the mail message so I used the attached file).

The backup script has the btrfs sync command since Aug 3
I sent the following rsync command as you said (sorry for the misunderstanding with
the rsync command that I'd never used before)
The snapshot has been different since snapshot mysql_201708090830 but the mariadb
can recovery and start until mysql_201708110830 seems too much differences that can
not start.



[root@backuplogC7 ~]# more /root/script/backup/backupsnap.sh
#Backup
#
user=$1
password=$2
basepath=$3
datet=$(date +%Y%m%d%H%M)
snappath=${basepath}_${datet}
echo "Locking databases ${datet}"
mysql -u$user -p$password << EOF
FLUSH TABLES WITH READ LOCK;
system btrfs sub snap -r $basepath $snappath
system btrfs sub sync $basepath
UNLOCK TABLES;
quit
EOF
echo "Databases unlocked ${datet}"

[root@backuplogC7 ~]# ls -l /root/script/backup/backupsnap.sh
-rwxr--r-- 1 root root 333 Aug  3 09:11 /root/script/backup/backupsnap.sh


[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708070830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  1083.43 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708080830/
sending incremental file list
./

sent 3769 bytes  received 19 bytes  841.78 bytes/sec
total size is 718361496  speedup is 189641.37 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/
sending incremental file list
./
ib_logfile1
ibdata1

sent 3779 bytes  received 25 bytes  1086.86 bytes/sec
total size is 718361496  speedup is 188843.72 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708100830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708100830/
sending incremental file list
./
ib_logfile1
ibdata1

sent 3779 bytes  received 25 bytes  1086.86 bytes/sec
total size is 718361496  speedup is 188843.72 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708110830/
root@192.168.45.166:/var/lib/mariadb/mysql_20170811830/
sending incremental file list
created directory /var/lib/mariadb/mysql_20170811830
./
aria_log.00000001
aria_log_control
ib_logfile0
ib_logfile1
ibdata1
mysql-bin.000001
mysql-bin.index
backups_db/
backups_db/db.opt
backups_db/ints.frm
backups_db/tpc_backup_detail.frm
backups_db/tpc_backup_header.frm
backups_db/tpc_backup_history.frm
backups_db/tpc_backup_schedule_level_master.frm
backups_db/tpc_backup_schedule_master.frm
backups_db/tpc_backup_schedule_master_extra.frm
backups_db/tpc_backup_schedule_old.frm
backups_db/tpc_backup_server.frm
backups_db/tpc_backup_step.frm
backups_db/tpc_backup_user.frm
backups_db/v_backup_header_status.frm
backups_db/v_calendar.frm
backups_db/v_dup_backup.frm
mysql/
mysql/columns_priv.MYD
mysql/columns_priv.MYI
mysql/columns_priv.frm
mysql/db.MYD
mysql/db.MYI
mysql/db.frm
mysql/event.MYD
mysql/event.MYI
mysql/event.frm
mysql/func.MYD
mysql/func.MYI
mysql/func.frm
mysql/general_log.CSM
mysql/general_log.CSV
mysql/general_log.frm
mysql/help_category.MYD
mysql/help_category.MYI
mysql/help_category.frm
mysql/help_keyword.MYD
mysql/help_keyword.MYI
mysql/help_keyword.frm
mysql/help_relation.MYD
mysql/help_relation.MYI
mysql/help_relation.frm
mysql/help_topic.MYD
mysql/help_topic.MYI
mysql/help_topic.frm
mysql/host.MYD
mysql/host.MYI
mysql/host.frm
mysql/ndb_binlog_index.MYD
mysql/ndb_binlog_index.MYI
mysql/ndb_binlog_index.frm
mysql/plugin.MYD
mysql/plugin.MYI
mysql/plugin.frm
mysql/proc.MYD
mysql/proc.MYI
mysql/proc.frm
mysql/procs_priv.MYD
mysql/procs_priv.MYI
mysql/procs_priv.frm
mysql/proxies_priv.MYD
mysql/proxies_priv.MYI
mysql/proxies_priv.frm
mysql/servers.MYD
mysql/servers.MYI
mysql/servers.frm
mysql/slow_log.CSM
mysql/slow_log.CSV
mysql/slow_log.frm
mysql/tables_priv.MYD
mysql/tables_priv.MYI
mysql/tables_priv.frm
mysql/time_zone.MYD
mysql/time_zone.MYI
mysql/time_zone.frm
mysql/time_zone_leap_second.MYD
mysql/time_zone_leap_second.MYI
mysql/time_zone_leap_second.frm
mysql/time_zone_name.MYD
mysql/time_zone_name.MYI
mysql/time_zone_name.frm
mysql/time_zone_transition.MYD
mysql/time_zone_transition.MYI
mysql/time_zone_transition.frm
mysql/time_zone_transition_type.MYD
mysql/time_zone_transition_type.MYI
mysql/time_zone_transition_type.frm
mysql/user.MYD
mysql/user.MYI
mysql/user.frm
performance_schema/
performance_schema/cond_instances.frm
performance_schema/db.opt
performance_schema/events_waits_current.frm
performance_schema/events_waits_history.frm
performance_schema/events_waits_history_long.frm
performance_schema/events_waits_summary_by_instance.frm
performance_schema/events_waits_summary_by_thread_by_event_name.frm
performance_schema/events_waits_summary_global_by_event_name.frm
performance_schema/file_instances.frm
performance_schema/file_summary_by_event_name.frm
performance_schema/file_summary_by_instance.frm
performance_schema/mutex_instances.frm
performance_schema/performance_timers.frm
performance_schema/rwlock_instances.frm
performance_schema/setup_consumers.frm
performance_schema/setup_instruments.frm
performance_schema/setup_timers.frm
performance_schema/threads.frm
test/
test/db.opt

sent 4124 bytes  received 370 bytes  1797.60 bytes/sec
total size is 718361496  speedup is 159849.02 (DRY RUN)
[root@backuplogC7 ~]#
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708120830/
root@192.168.45.166:/var/lib/mariadb/mysql_20170812830/
sending incremental file list
created directory /var/lib/mariadb/mysql_20170812830
./
aria_log.00000001
aria_log_control
ib_logfile0
ib_logfile1
ibdata1
mysql-bin.000001
mysql-bin.index
backups_db/
backups_db/db.opt
backups_db/ints.frm
backups_db/tpc_backup_detail.frm
backups_db/tpc_backup_header.frm
backups_db/tpc_backup_history.frm
backups_db/tpc_backup_schedule_level_master.frm
backups_db/tpc_backup_schedule_master.frm
backups_db/tpc_backup_schedule_master_extra.frm
backups_db/tpc_backup_schedule_old.frm
backups_db/tpc_backup_server.frm
backups_db/tpc_backup_step.frm
backups_db/tpc_backup_user.frm
backups_db/v_backup_header_status.frm
backups_db/v_calendar.frm
backups_db/v_dup_backup.frm
mysql/
mysql/columns_priv.MYD
mysql/columns_priv.MYI
mysql/columns_priv.frm
mysql/db.MYD
mysql/db.MYI
mysql/db.frm
mysql/event.MYD
mysql/event.MYI
mysql/event.frm
mysql/func.MYD
mysql/func.MYI
mysql/func.frm
mysql/general_log.CSM
mysql/general_log.CSV
mysql/general_log.frm
mysql/help_category.MYD
mysql/help_category.MYI
mysql/help_category.frm
mysql/help_keyword.MYD
mysql/help_keyword.MYI
mysql/help_keyword.frm
mysql/help_relation.MYD
mysql/help_relation.MYI
mysql/help_relation.frm
mysql/help_topic.MYD
mysql/help_topic.MYI
mysql/help_topic.frm
mysql/host.MYD
mysql/host.MYI
mysql/host.frm
mysql/ndb_binlog_index.MYD
mysql/ndb_binlog_index.MYI
mysql/ndb_binlog_index.frm
mysql/plugin.MYD
mysql/plugin.MYI
mysql/plugin.frm
mysql/proc.MYD
mysql/proc.MYI
mysql/proc.frm
mysql/procs_priv.MYD
mysql/procs_priv.MYI
mysql/procs_priv.frm
mysql/proxies_priv.MYD
mysql/proxies_priv.MYI
mysql/proxies_priv.frm
mysql/servers.MYD
mysql/servers.MYI
mysql/servers.frm
mysql/slow_log.CSM
mysql/slow_log.CSV
mysql/slow_log.frm
mysql/tables_priv.MYD
mysql/tables_priv.MYI
mysql/tables_priv.frm
mysql/time_zone.MYD
mysql/time_zone.MYI
mysql/time_zone.frm
mysql/time_zone_leap_second.MYD
mysql/time_zone_leap_second.MYI
mysql/time_zone_leap_second.frm
mysql/time_zone_name.MYD
mysql/time_zone_name.MYI
mysql/time_zone_name.frm
mysql/time_zone_transition.MYD
mysql/time_zone_transition.MYI
mysql/time_zone_transition.frm
mysql/time_zone_transition_type.MYD
mysql/time_zone_transition_type.MYI
mysql/time_zone_transition_type.frm
mysql/user.MYD
mysql/user.MYI
mysql/user.frm
performance_schema/
performance_schema/cond_instances.frm
performance_schema/db.opt
performance_schema/events_waits_current.frm
performance_schema/events_waits_history.frm
performance_schema/events_waits_history_long.frm
performance_schema/events_waits_summary_by_instance.frm
performance_schema/events_waits_summary_by_thread_by_event_name.frm
performance_schema/events_waits_summary_global_by_event_name.frm
performance_schema/file_instances.frm
performance_schema/file_summary_by_event_name.frm
performance_schema/file_summary_by_instance.frm
performance_schema/mutex_instances.frm
performance_schema/performance_timers.frm
performance_schema/rwlock_instances.frm
performance_schema/setup_consumers.frm
performance_schema/setup_instruments.frm
performance_schema/setup_timers.frm
performance_schema/threads.frm
test/
test/db.opt

sent 4124 bytes  received 370 bytes  1797.60 bytes/sec
total size is 718361496  speedup is 159849.02 (DRY RUN)


Best Regards,
Siranee Jaraswachirakul

> On Fri, Aug 11, 2017 at 8:38 PM,  <siranee.ja@tpc.co.th> wrote:
>> Hi Chris,
>>
>> I explain what I have done in the attached file.
>
> That is way too long. And please don't attach files directed at me,
> that's not appropriate. Just message the list normally, so it can be
> searched by others down the road.
>
> There is a problem with your rsync command, you really don't
> understand what I was asking. The whole point of rsync is to compare
> the origin and destination subvolumes to see if they are different
> from each other. You ran the command on a single subvolume on a single
> machine which is pointless.
>
> On machine A:
>
> $ rsync -avnc /mnt/snapshot6/ chris@192.168.1.116:/mnt/snapshot6/
> chris@192.168.1.116's password:
> sending incremental file list
>
> sent 125 bytes  received 12 bytes  39.14 bytes/sec
> total size is 30,686  speedup is 223.99 (DRY RUN)
> [chris@f26h ~]$
>
> Because no files are listed, all are confirmed to be identical. If
> files are listed, there's a difference.
>
>
>
>> Please suggest me what I should do next?
>
> Check this. I wonder if you're running into this obscure bug where
> maybe your read only snapshot is not yet synced before it's being
> sent. Hence it *is* changing on origin machine, but not because of
> other snapshots being deleted. I have no idea if your scripting
> handles errors like the bogus stale NFS handle error, maybe it's
> possible it happens but you're not seeing it? Anyway, you might try
> just adding a single sync right after taking the snapshot, change
> nothing else and see if the problem reproduces itself.
>
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping
>
> I'm pretty sure this is fixed in newer kernels, I haven't run into it
> in a long time myself. But I don't know when it was fixed.
>
>
>
> --
> Chris Murphy
>

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Fri, Aug 11, 2017 at 11:08 PM,  <siranee.ja@tpc.co.th> wrote:


> The backup script has the btrfs sync command since Aug 3


>From your script:
> system btrfs sub snap -r $basepath $snappath
> system btrfs sub sync $basepath

>From the man page: sync <path> [subvolid...]
           Wait until given subvolume(s) are completely removed from the
           filesystem after deletion.


This 'subvolume sync' command, per the man page, is only about
subvolume deletion. I suggest replacing it with a regular sync
command.

I think the problem is that the script does things so fast that the
snapshot is not always consistent on disk before btrfs send starts.
It's just a guess though. If I'm right, this means the rsync mismaches
mean the destination snapshots are bad. Here's what I would do:


- delete all the bad/mismatching snapshots only on the destination computer.

- he most recent good snapshot pair, which rsync shows origin and
destination match, is mysql_201708080830 so you can keep that one on
both sides.

- manually do incremental send/receive, starting with
mysql_201708090830/, to make the destination current again with the
origin.

- confirm with rsync that the snapshot pairs on origin and destination
are the same

- now resume using the modified script, which will do snapshot -> sync -> send.

OPTIONAL, you could add to your script an rsync -avnc to double check
that the incremental send receive is working. This is admittedly
inefficient because it checks the *entire* contents of the snapshots
on both sides, it's not just checking the incremental data. But if it
doesn't take too long, it will help restore trust in send/receive, and
confirm that a regular sync is needed in between snapshot and send.




-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[Janos Toth F.]

On Sat, Aug 12, 2017 at 11:34 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Fri, Aug 11, 2017 at 11:08 PM,  <siranee.ja@tpc.co.th> wrote:
>
> I think the problem is that the script does things so fast that the
> snapshot is not always consistent on disk before btrfs send starts.
> It's just a guess though. If I'm right, this means the rsync mismaches
> mean the destination snapshots are bad.

Hmm. I was wondering about this exact issue the other day when I
fiddled with my own backup script.
- Should I issue a sync between creating a snapshot and starting to
rsync (or send/receive) the content of that fresh snapshot to an
external backup storage?
I dismissed the thought because I figured rsync should see the proper
state regardless if some data is still waiting in the system
write-back cache.
Now I am confused again. Is it only for send/receive? (I don't use
that feature yet but thought about switching to it from rsync.)

Re: btrfs issue with mariadb incremental backup

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

On Sat, Aug 12, 2017 at 03:34:01PM -0600, Chris Murphy wrote:
> On Fri, Aug 11, 2017 at 11:08 PM,  <siranee.ja@tpc.co.th> wrote:
> 
> 
> > The backup script has the btrfs sync command since Aug 3
> 
> 
> From your script:
> > system btrfs sub snap -r $basepath $snappath
> > system btrfs sub sync $basepath
> 
> From the man page: sync <path> [subvolid...]
>            Wait until given subvolume(s) are completely removed from the
>            filesystem after deletion.
> 
> 
> This 'subvolume sync' command, per the man page, is only about
> subvolume deletion. I suggest replacing it with a regular sync
> command.
> 
> I think the problem is that the script does things so fast that the
> snapshot is not always consistent on disk before btrfs send starts.
> It's just a guess though. If I'm right, this means the rsync mismaches
> mean the destination snapshots are bad. Here's what I would do:

   I don't see how that can happen. Snapshots are atomic -- they're
either there or not there. It's not a matter even of copying the
metadata part of the subvol. It's literally just adding a pointer to
point at an existing FS tree.

   Hugo.

-- 
Hugo Mills             | If it's December 1941 in Casablanca, what time is it
hugo@... carfax.org.uk | in New York?
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                               Rick Blaine, Casablanca

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Sat, Aug 12, 2017 at 4:41 PM, Janos Toth F. <toth.f.janos@gmail.com> wrote:
> On Sat, Aug 12, 2017 at 11:34 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> On Fri, Aug 11, 2017 at 11:08 PM,  <siranee.ja@tpc.co.th> wrote:
>>
>> I think the problem is that the script does things so fast that the
>> snapshot is not always consistent on disk before btrfs send starts.
>> It's just a guess though. If I'm right, this means the rsync mismaches
>> mean the destination snapshots are bad.
>
> Hmm. I was wondering about this exact issue the other day when I
> fiddled with my own backup script.
> - Should I issue a sync between creating a snapshot and starting to
> rsync (or send/receive) the content of that fresh snapshot to an
> external backup storage?

A sync won't hurt, but I think we need btrfs send to include sync if
this is still required as stated by the Wiki.

And if the problem stated in the Wiki has been fixed, then it'd be
useful to know when, but a git log search doesn't reveal anything
related to NFS and Btrfs.


-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Sat, Aug 12, 2017 at 4:49 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Sat, Aug 12, 2017 at 03:34:01PM -0600, Chris Murphy wrote:
>> On Fri, Aug 11, 2017 at 11:08 PM,  <siranee.ja@tpc.co.th> wrote:
>>
>>
>> > The backup script has the btrfs sync command since Aug 3
>>
>>
>> From your script:
>> > system btrfs sub snap -r $basepath $snappath
>> > system btrfs sub sync $basepath
>>
>> From the man page: sync <path> [subvolid...]
>>            Wait until given subvolume(s) are completely removed from the
>>            filesystem after deletion.
>>
>>
>> This 'subvolume sync' command, per the man page, is only about
>> subvolume deletion. I suggest replacing it with a regular sync
>> command.
>>
>> I think the problem is that the script does things so fast that the
>> snapshot is not always consistent on disk before btrfs send starts.
>> It's just a guess though. If I'm right, this means the rsync mismaches
>> mean the destination snapshots are bad. Here's what I would do:
>
>    I don't see how that can happen. Snapshots are atomic -- they're
> either there or not there. It's not a matter even of copying the
> metadata part of the subvol. It's literally just adding a pointer to
> point at an existing FS tree.

Do snapshots come with sync and flush to disk? Or does it just set a
checkpoint and the extents that are as yet uncommitted prior to the
snapshot aren't for sure on disk yet until the next commit time or
sync?

Nevertheless the wiki explicitly says to do sync after taking a
snapshot in the context of send/receive.
https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping

And people on the list have had this "stale NFS file handle" error .

I have no idea if it's still a problem with current kernels, or if it
applies to 4.4 kernel.

Anyway, the OP does appear to be having a real problem. rync is very
clearly showing a given snapshot with incremental send/receive differ
by *a lot* on source and destination machines.



-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi Chris,

I did as you suggest and the result was bad then I decided to start over with
snapshot mysql_201708070830 and manually send incremental the result rsync always
said "a log diff" but the dest can start mariadb until snapshot "mysql_201708100830"
it couldn't start mariadb.

The following are the result.

- delete all the bad/mismatching snapshots only on the destination computer.
tpcorp@virtualtrust3:~$ lxc exec joytest -- bash
[root@joytest ~]# btrfs sub list /var/lib/mariadb
ID 298 gen 177 top level 5 path mysql_201708070830
ID 301 gen 148 top level 5 path mysql_201708080830
ID 303 gen 156 top level 5 path mysql_201708090830
ID 305 gen 162 top level 5 path mysql_201708100830
ID 309 gen 175 top level 5 path mysql_201708110830
ID 310 gen 176 top level 5 path mysql
ID 311 gen 180 top level 5 path mysql_201708120830
[root@joytest ~]# cd /var/lib/mariadb
[root@joytest mariadb]# btrfs sub list .
ID 298 gen 177 top level 5 path mysql_201708070830
ID 301 gen 148 top level 5 path mysql_201708080830
ID 303 gen 156 top level 5 path mysql_201708090830
ID 305 gen 162 top level 5 path mysql_201708100830
ID 309 gen 175 top level 5 path mysql_201708110830
ID 310 gen 176 top level 5 path mysql
ID 311 gen 180 top level 5 path mysql_201708120830
[root@joytest mariadb]# btrfs sub del mysql_201708090830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
[root@joytest mariadb]# btrfs sub del mysql_201708100830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
[root@joytest mariadb]# btrfs sub del mysql_201708110830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
[root@joytest mariadb]# btrfs sub del mysql_201708120830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
[root@joytest mariadb]# btrfs sub sync .

- The most recent good snapshot pair, which rsync shows origin and
destination match, is mysql_201708080830 so you can keep that one on
both sides.
[root@joytest mariadb]# btrfs sub list .
ID 298 gen 177 top level 5 path mysql_201708070830
ID 301 gen 148 top level 5 path mysql_201708080830
ID 310 gen 176 top level 5 path mysql

- manually do incremental send/receive, starting with
mysql_201708090830/, to make the destination current again with the
origin.

tpcorp@virtualtrust3:~$ lxc exec backuplogC7 -- bash
[root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
ID 257 gen 537 top level 5 path mysql
ID 316 gen 498 top level 5 path mysql_201708060830
ID 317 gen 503 top level 5 path mysql_201708070830
ID 318 gen 507 top level 5 path mysql_201708080830
ID 319 gen 514 top level 5 path mysql_201708090830
ID 320 gen 524 top level 5 path mysql_201708100830
ID 321 gen 529 top level 5 path mysql_201708110830
ID 322 gen 533 top level 5 path mysql_201708120830


[root@backuplogC7 ~]# more /var/log/btrfs_send.log
Start Send 201708040905
btrfs send -p /var/lib/mariadb/mysql_201708030830
/var/lib/mariadb/mysql_201708040830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708040830
Stop Send 201708040905
Start Send 201708050905
btrfs send -p /var/lib/mariadb/mysql_201708040830
/var/lib/mariadb/mysql_201708050830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708050830
Stop Send 201708050905
Start Send 201708060905
btrfs send -p /var/lib/mariadb/mysql_201708050830
/var/lib/mariadb/mysql_201708060830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708060830
Stop Send 201708060905
Start Send 201708070905
btrfs send -p /var/lib/mariadb/mysql_201708060830
/var/lib/mariadb/mysql_201708070830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708070830
Stop Send 201708070905
Start Send 201708080905
btrfs send -p /var/lib/mariadb/mysql_201708070830
/var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708080830
Stop Send 201708080905
Start Send 201708090905
btrfs send -p /var/lib/mariadb/mysql_201708080830
/var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708090830
Stop Send 201708090905
Start Send 201708100905
btrfs send -p /var/lib/mariadb/mysql_201708090830
/var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708100830
Stop Send 201708100905
Start Send 201708110905
btrfs send -p /var/lib/mariadb/mysql_201708100830
/var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708110830
Stop Send 201708110905
Start Send 201708120905
btrfs send -p /var/lib/mariadb/mysql_201708110830
/var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708120830
Stop Send 201708120905
[root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
/var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708090830
At snapshot mysql_201708090830
[root@backuplogC7 ~]# history|grep rsync
  416  rsync
  417  yum install rsync
  424  rsync -anc mysql_201708110830 > /tmp/source_mysql_201708110830.txt
  433  history|grep rsync
  434  rsync -anc mysql_201708100830 > /tmp/source_mysql_201708100830.txt
  435  rsync -anc /var/lib/mariadb/mysql_201708100830 >
/tmp/source_mysql_201708100830.txt
  442  history|grep rsync
  443  rsync -anc /var/lib/mariadb/mysql_201708090830 >
/tmp/source_mysql_201708090830.txt
  444  rsync -anc /var/lib/mariadb/mysql_201708080830 >
/tmp/source_mysql_201708080830.txt
  445  rsync -anc /var/lib/mariadb/mysql_201708070830 >
/tmp/source_mysql_201708070830.txt
  486  history|grep rsync
  495  history|grep rsync
  496  rsync -anc /var/lib/mariadb/mysql_201708120830 >
/tmp/source_mysql_201708120830.txt
  498  rsync -anc /var/lib/mariadb/mysql_201708120830
  499  rsync -anc /var/lib/mariadb/mysql_201708110830
  504  history|grep rsync
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
root@192.168.45.166://var/lib/mariadb/mysql_201708090830
sending incremental file list
mysql_201708090830/
mysql_201708090830/aria_log.00000001
mysql_201708090830/aria_log_control
mysql_201708090830/ib_logfile0
mysql_201708090830/ib_logfile1
mysql_201708090830/ibdata1
mysql_201708090830/mysql-bin.000001
mysql_201708090830/mysql-bin.index
mysql_201708090830/backups_db/
mysql_201708090830/backups_db/db.opt
mysql_201708090830/backups_db/ints.frm
mysql_201708090830/backups_db/tpc_backup_detail.frm
mysql_201708090830/backups_db/tpc_backup_header.frm
mysql_201708090830/backups_db/tpc_backup_history.frm
mysql_201708090830/backups_db/tpc_backup_schedule_level_master.frm
mysql_201708090830/backups_db/tpc_backup_schedule_master.frm
mysql_201708090830/backups_db/tpc_backup_schedule_master_extra.frm
mysql_201708090830/backups_db/tpc_backup_schedule_old.frm
mysql_201708090830/backups_db/tpc_backup_server.frm
mysql_201708090830/backups_db/tpc_backup_step.frm
mysql_201708090830/backups_db/tpc_backup_user.frm
mysql_201708090830/backups_db/v_backup_header_status.frm
mysql_201708090830/backups_db/v_calendar.frm
mysql_201708090830/backups_db/v_dup_backup.frm
mysql_201708090830/mysql/
mysql_201708090830/mysql/columns_priv.MYD
mysql_201708090830/mysql/columns_priv.MYI
mysql_201708090830/mysql/columns_priv.frm
mysql_201708090830/mysql/db.MYD
mysql_201708090830/mysql/db.MYI
mysql_201708090830/mysql/db.frm
mysql_201708090830/mysql/event.MYD
mysql_201708090830/mysql/event.MYI
mysql_201708090830/mysql/event.frm
mysql_201708090830/mysql/func.MYD
mysql_201708090830/mysql/func.MYI
mysql_201708090830/mysql/func.frm
mysql_201708090830/mysql/general_log.CSM
mysql_201708090830/mysql/general_log.CSV
mysql_201708090830/mysql/general_log.frm
mysql_201708090830/mysql/help_category.MYD
mysql_201708090830/mysql/help_category.MYI
mysql_201708090830/mysql/help_category.frm
mysql_201708090830/mysql/help_keyword.MYD
mysql_201708090830/mysql/help_keyword.MYI
mysql_201708090830/mysql/help_keyword.frm
mysql_201708090830/mysql/help_relation.MYD
mysql_201708090830/mysql/help_relation.MYI
mysql_201708090830/mysql/help_relation.frm
mysql_201708090830/mysql/help_topic.MYD
mysql_201708090830/mysql/help_topic.MYI
mysql_201708090830/mysql/help_topic.frm
mysql_201708090830/mysql/host.MYD
mysql_201708090830/mysql/host.MYI
mysql_201708090830/mysql/host.frm
mysql_201708090830/mysql/ndb_binlog_index.MYD
mysql_201708090830/mysql/ndb_binlog_index.MYI
mysql_201708090830/mysql/ndb_binlog_index.frm
mysql_201708090830/mysql/plugin.MYD
mysql_201708090830/mysql/plugin.MYI
mysql_201708090830/mysql/plugin.frm
mysql_201708090830/mysql/proc.MYD
mysql_201708090830/mysql/proc.MYI
mysql_201708090830/mysql/proc.frm
mysql_201708090830/mysql/procs_priv.MYD
mysql_201708090830/mysql/procs_priv.MYI
mysql_201708090830/mysql/procs_priv.frm
mysql_201708090830/mysql/proxies_priv.MYD
mysql_201708090830/mysql/proxies_priv.MYI
mysql_201708090830/mysql/proxies_priv.frm
mysql_201708090830/mysql/servers.MYD
mysql_201708090830/mysql/servers.MYI
mysql_201708090830/mysql/servers.frm
mysql_201708090830/mysql/slow_log.CSM
mysql_201708090830/mysql/slow_log.CSV
mysql_201708090830/mysql/slow_log.frm
mysql_201708090830/mysql/tables_priv.MYD
mysql_201708090830/mysql/tables_priv.MYI
mysql_201708090830/mysql/tables_priv.frm
mysql_201708090830/mysql/time_zone.MYD
mysql_201708090830/mysql/time_zone.MYI
mysql_201708090830/mysql/time_zone.frm
mysql_201708090830/mysql/time_zone_leap_second.MYD
mysql_201708090830/mysql/time_zone_leap_second.MYI
mysql_201708090830/mysql/time_zone_leap_second.frm
mysql_201708090830/mysql/time_zone_name.MYD
mysql_201708090830/mysql/time_zone_name.MYI
mysql_201708090830/mysql/time_zone_name.frm
mysql_201708090830/mysql/time_zone_transition.MYD
mysql_201708090830/mysql/time_zone_transition.MYI
mysql_201708090830/mysql/time_zone_transition.frm
mysql_201708090830/mysql/time_zone_transition_type.MYD
mysql_201708090830/mysql/time_zone_transition_type.MYI
mysql_201708090830/mysql/time_zone_transition_type.frm
mysql_201708090830/mysql/user.MYD
mysql_201708090830/mysql/user.MYI
mysql_201708090830/mysql/user.frm
mysql_201708090830/performance_schema/
mysql_201708090830/performance_schema/cond_instances.frm
mysql_201708090830/performance_schema/db.opt
mysql_201708090830/performance_schema/events_waits_current.frm
mysql_201708090830/performance_schema/events_waits_history.frm
mysql_201708090830/performance_schema/events_waits_history_long.frm
mysql_201708090830/performance_schema/events_waits_summary_by_instance.frm
mysql_201708090830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
mysql_201708090830/performance_schema/events_waits_summary_global_by_event_name.frm
mysql_201708090830/performance_schema/file_instances.frm
mysql_201708090830/performance_schema/file_summary_by_event_name.frm
mysql_201708090830/performance_schema/file_summary_by_instance.frm
mysql_201708090830/performance_schema/mutex_instances.frm
mysql_201708090830/performance_schema/performance_timers.frm
mysql_201708090830/performance_schema/rwlock_instances.frm
mysql_201708090830/performance_schema/setup_consumers.frm
mysql_201708090830/performance_schema/setup_instruments.frm
mysql_201708090830/performance_schema/setup_timers.frm
mysql_201708090830/performance_schema/threads.frm
mysql_201708090830/test/
mysql_201708090830/test/db.opt

sent 4157 bytes  received 371 bytes  1811.20 bytes/sec
total size is 718361496  speedup is 158648.74 (DRY RUN)
[root@backuplogC7 ~]#

[root@joytest mariadb]# btrfs sub list .
ID 298 gen 182 top level 5 path mysql_201708070830
ID 301 gen 148 top level 5 path mysql_201708080830
ID 310 gen 176 top level 5 path mysql
ID 312 gen 185 top level 5 path mysql_201708090830

- now resume using the modified script, which will do snapshot -> sync -> send.

[root@backuplogC7 ~]# more /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

30 08 * * * root /root/script/backup/backupsnap.sh root password
/var/lib/mariadb/mysql >> /var/log/btrfs_snap.log
05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166
/var/lib/mariadb >> /var/log/btrfs_send.log
0 19 * * * root /root/script/backup/doBackup_file_backuplog.sh
30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >>
/var/log/btrfs_del.log

# temp
##00 09 * * * root /root/script/backupbtrfs_all.sh /var/lib/mariadb /backupbtrfs
mysql_201707210830

[root@backuplogC7 ~]# vi /root/script/backup/backupsnap.sh
[root@backuplogC7 ~]# ls -l /root/script/backup/backupsnap.sh
-rwxr--r-- 1 root root 314 Aug 13 08:12 /root/script/backup/backupsnap.sh
[root@backuplogC7 ~]# cat /root/script/backup/backupsnap.sh
#Backup
#
user=$1
password=$2
basepath=$3
datet=$(date +%Y%m%d%H%M)
snappath=${basepath}_${datet}
echo "Locking databases ${datet}"
mysql -u$user -p$password << EOF
FLUSH TABLES WITH READ LOCK;
system btrfs sub snap -r $basepath $snappath
system sync
UNLOCK TABLES;
quit
EOF
echo "Databases unlocked ${datet}"

[root@backuplogC7 ~]#

[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830 
root@192.168.45.166://var/lib/mariadb/mysql_201708080830
sending incremental file list
mysql_201708080830/
mysql_201708080830/aria_log.00000001
mysql_201708080830/aria_log_control
mysql_201708080830/ib_logfile0
mysql_201708080830/ib_logfile1
mysql_201708080830/ibdata1
mysql_201708080830/mysql-bin.000001
mysql_201708080830/mysql-bin.index
mysql_201708080830/backups_db/
mysql_201708080830/backups_db/db.opt
mysql_201708080830/backups_db/ints.frm
mysql_201708080830/backups_db/tpc_backup_detail.frm
mysql_201708080830/backups_db/tpc_backup_header.frm
mysql_201708080830/backups_db/tpc_backup_history.frm
mysql_201708080830/backups_db/tpc_backup_schedule_level_master.frm
mysql_201708080830/backups_db/tpc_backup_schedule_master.frm
mysql_201708080830/backups_db/tpc_backup_schedule_master_extra.frm
mysql_201708080830/backups_db/tpc_backup_schedule_old.frm
mysql_201708080830/backups_db/tpc_backup_server.frm
mysql_201708080830/backups_db/tpc_backup_step.frm
mysql_201708080830/backups_db/tpc_backup_user.frm
mysql_201708080830/backups_db/v_backup_header_status.frm
mysql_201708080830/backups_db/v_calendar.frm
mysql_201708080830/backups_db/v_dup_backup.frm
mysql_201708080830/mysql/
mysql_201708080830/mysql/columns_priv.MYD
mysql_201708080830/mysql/columns_priv.MYI
mysql_201708080830/mysql/columns_priv.frm
mysql_201708080830/mysql/db.MYD
mysql_201708080830/mysql/db.MYI
mysql_201708080830/mysql/db.frm
mysql_201708080830/mysql/event.MYD
mysql_201708080830/mysql/event.MYI
mysql_201708080830/mysql/event.frm
mysql_201708080830/mysql/func.MYD
mysql_201708080830/mysql/func.MYI
mysql_201708080830/mysql/func.frm
mysql_201708080830/mysql/general_log.CSM
mysql_201708080830/mysql/general_log.CSV
mysql_201708080830/mysql/general_log.frm
mysql_201708080830/mysql/help_category.MYD
mysql_201708080830/mysql/help_category.MYI
mysql_201708080830/mysql/help_category.frm
mysql_201708080830/mysql/help_keyword.MYD
mysql_201708080830/mysql/help_keyword.MYI
mysql_201708080830/mysql/help_keyword.frm
mysql_201708080830/mysql/help_relation.MYD
mysql_201708080830/mysql/help_relation.MYI
mysql_201708080830/mysql/help_relation.frm
mysql_201708080830/mysql/help_topic.MYD
mysql_201708080830/mysql/help_topic.MYI
mysql_201708080830/mysql/help_topic.frm
mysql_201708080830/mysql/host.MYD
mysql_201708080830/mysql/host.MYI
mysql_201708080830/mysql/host.frm
mysql_201708080830/mysql/ndb_binlog_index.MYD
mysql_201708080830/mysql/ndb_binlog_index.MYI
mysql_201708080830/mysql/ndb_binlog_index.frm
mysql_201708080830/mysql/plugin.MYD
mysql_201708080830/mysql/plugin.MYI
mysql_201708080830/mysql/plugin.frm
mysql_201708080830/mysql/proc.MYD
mysql_201708080830/mysql/proc.MYI
mysql_201708080830/mysql/proc.frm
mysql_201708080830/mysql/procs_priv.MYD
mysql_201708080830/mysql/procs_priv.MYI
mysql_201708080830/mysql/procs_priv.frm
mysql_201708080830/mysql/proxies_priv.MYD
mysql_201708080830/mysql/proxies_priv.MYI
mysql_201708080830/mysql/proxies_priv.frm
mysql_201708080830/mysql/servers.MYD
mysql_201708080830/mysql/servers.MYI
mysql_201708080830/mysql/servers.frm
mysql_201708080830/mysql/slow_log.CSM
mysql_201708080830/mysql/slow_log.CSV
mysql_201708080830/mysql/slow_log.frm
mysql_201708080830/mysql/tables_priv.MYD
mysql_201708080830/mysql/tables_priv.MYI
mysql_201708080830/mysql/tables_priv.frm
mysql_201708080830/mysql/time_zone.MYD
mysql_201708080830/mysql/time_zone.MYI
mysql_201708080830/mysql/time_zone.frm
mysql_201708080830/mysql/time_zone_leap_second.MYD
mysql_201708080830/mysql/time_zone_leap_second.MYI
mysql_201708080830/mysql/time_zone_leap_second.frm
mysql_201708080830/mysql/time_zone_name.MYD
mysql_201708080830/mysql/time_zone_name.MYI
mysql_201708080830/mysql/time_zone_name.frm
mysql_201708080830/mysql/time_zone_transition.MYD
mysql_201708080830/mysql/time_zone_transition.MYI
mysql_201708080830/mysql/time_zone_transition.frm
mysql_201708080830/mysql/time_zone_transition_type.MYD
mysql_201708080830/mysql/time_zone_transition_type.MYI
mysql_201708080830/mysql/time_zone_transition_type.frm
mysql_201708080830/mysql/user.MYD
mysql_201708080830/mysql/user.MYI
mysql_201708080830/mysql/user.frm
mysql_201708080830/performance_schema/
mysql_201708080830/performance_schema/cond_instances.frm
mysql_201708080830/performance_schema/db.opt
mysql_201708080830/performance_schema/events_waits_current.frm
mysql_201708080830/performance_schema/events_waits_history.frm
mysql_201708080830/performance_schema/events_waits_history_long.frm
mysql_201708080830/performance_schema/events_waits_summary_by_instance.frm
mysql_201708080830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
mysql_201708080830/performance_schema/events_waits_summary_global_by_event_name.frm
mysql_201708080830/performance_schema/file_instances.frm
mysql_201708080830/performance_schema/file_summary_by_event_name.frm
mysql_201708080830/performance_schema/file_summary_by_instance.frm
mysql_201708080830/performance_schema/mutex_instances.frm
mysql_201708080830/performance_schema/performance_timers.frm
mysql_201708080830/performance_schema/rwlock_instances.frm
mysql_201708080830/performance_schema/setup_consumers.frm
mysql_201708080830/performance_schema/setup_instruments.frm
mysql_201708080830/performance_schema/setup_timers.frm
mysql_201708080830/performance_schema/threads.frm
mysql_201708080830/test/
mysql_201708080830/test/db.opt

sent 4153 bytes  received 371 bytes  1809.60 bytes/sec
total size is 718361496  speedup is 158789.01 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830 
root@192.168.45.166://var/lib/mariadb/mysql_201708070830
sending incremental file list
mysql_201708070830/
mysql_201708070830/aria_log.00000001
mysql_201708070830/aria_log_control
mysql_201708070830/ib_logfile0
mysql_201708070830/ib_logfile1
mysql_201708070830/ibdata1
mysql_201708070830/mysql-bin.000001
mysql_201708070830/mysql-bin.index
mysql_201708070830/backups_db/
mysql_201708070830/backups_db/db.opt
mysql_201708070830/backups_db/ints.frm
mysql_201708070830/backups_db/tpc_backup_detail.frm
mysql_201708070830/backups_db/tpc_backup_header.frm
mysql_201708070830/backups_db/tpc_backup_history.frm
mysql_201708070830/backups_db/tpc_backup_schedule_level_master.frm
mysql_201708070830/backups_db/tpc_backup_schedule_master.frm
mysql_201708070830/backups_db/tpc_backup_schedule_master_extra.frm
mysql_201708070830/backups_db/tpc_backup_schedule_old.frm
mysql_201708070830/backups_db/tpc_backup_server.frm
mysql_201708070830/backups_db/tpc_backup_step.frm
mysql_201708070830/backups_db/tpc_backup_user.frm
mysql_201708070830/backups_db/v_backup_header_status.frm
mysql_201708070830/backups_db/v_calendar.frm
mysql_201708070830/backups_db/v_dup_backup.frm
mysql_201708070830/mysql/
mysql_201708070830/mysql/columns_priv.MYD
mysql_201708070830/mysql/columns_priv.MYI
mysql_201708070830/mysql/columns_priv.frm
mysql_201708070830/mysql/db.MYD
mysql_201708070830/mysql/db.MYI
mysql_201708070830/mysql/db.frm
mysql_201708070830/mysql/event.MYD
mysql_201708070830/mysql/event.MYI
mysql_201708070830/mysql/event.frm
mysql_201708070830/mysql/func.MYD
mysql_201708070830/mysql/func.MYI
mysql_201708070830/mysql/func.frm
mysql_201708070830/mysql/general_log.CSM
mysql_201708070830/mysql/general_log.CSV
mysql_201708070830/mysql/general_log.frm
mysql_201708070830/mysql/help_category.MYD
mysql_201708070830/mysql/help_category.MYI
mysql_201708070830/mysql/help_category.frm
mysql_201708070830/mysql/help_keyword.MYD
mysql_201708070830/mysql/help_keyword.MYI
mysql_201708070830/mysql/help_keyword.frm
mysql_201708070830/mysql/help_relation.MYD
mysql_201708070830/mysql/help_relation.MYI
mysql_201708070830/mysql/help_relation.frm
mysql_201708070830/mysql/help_topic.MYD
mysql_201708070830/mysql/help_topic.MYI
mysql_201708070830/mysql/help_topic.frm
mysql_201708070830/mysql/host.MYD
mysql_201708070830/mysql/host.MYI
mysql_201708070830/mysql/host.frm
mysql_201708070830/mysql/ndb_binlog_index.MYD
mysql_201708070830/mysql/ndb_binlog_index.MYI
mysql_201708070830/mysql/ndb_binlog_index.frm
mysql_201708070830/mysql/plugin.MYD
mysql_201708070830/mysql/plugin.MYI
mysql_201708070830/mysql/plugin.frm
mysql_201708070830/mysql/proc.MYD
mysql_201708070830/mysql/proc.MYI
mysql_201708070830/mysql/proc.frm
mysql_201708070830/mysql/procs_priv.MYD
mysql_201708070830/mysql/procs_priv.MYI
mysql_201708070830/mysql/procs_priv.frm
mysql_201708070830/mysql/proxies_priv.MYD
mysql_201708070830/mysql/proxies_priv.MYI
mysql_201708070830/mysql/proxies_priv.frm
mysql_201708070830/mysql/servers.MYD
mysql_201708070830/mysql/servers.MYI
mysql_201708070830/mysql/servers.frm
mysql_201708070830/mysql/slow_log.CSM
mysql_201708070830/mysql/slow_log.CSV
mysql_201708070830/mysql/slow_log.frm
mysql_201708070830/mysql/tables_priv.MYD
mysql_201708070830/mysql/tables_priv.MYI
mysql_201708070830/mysql/tables_priv.frm
mysql_201708070830/mysql/time_zone.MYD
mysql_201708070830/mysql/time_zone.MYI
mysql_201708070830/mysql/time_zone.frm
mysql_201708070830/mysql/time_zone_leap_second.MYD
mysql_201708070830/mysql/time_zone_leap_second.MYI
mysql_201708070830/mysql/time_zone_leap_second.frm
mysql_201708070830/mysql/time_zone_name.MYD
mysql_201708070830/mysql/time_zone_name.MYI
mysql_201708070830/mysql/time_zone_name.frm
mysql_201708070830/mysql/time_zone_transition.MYD
mysql_201708070830/mysql/time_zone_transition.MYI
mysql_201708070830/mysql/time_zone_transition.frm
mysql_201708070830/mysql/time_zone_transition_type.MYD
mysql_201708070830/mysql/time_zone_transition_type.MYI
mysql_201708070830/mysql/time_zone_transition_type.frm
mysql_201708070830/mysql/user.MYD
mysql_201708070830/mysql/user.MYI
mysql_201708070830/mysql/user.frm
mysql_201708070830/performance_schema/
mysql_201708070830/performance_schema/cond_instances.frm
mysql_201708070830/performance_schema/db.opt
mysql_201708070830/performance_schema/events_waits_current.frm
mysql_201708070830/performance_schema/events_waits_history.frm
mysql_201708070830/performance_schema/events_waits_history_long.frm
mysql_201708070830/performance_schema/events_waits_summary_by_instance.frm
mysql_201708070830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
mysql_201708070830/performance_schema/events_waits_summary_global_by_event_name.frm
mysql_201708070830/performance_schema/file_instances.frm
mysql_201708070830/performance_schema/file_summary_by_event_name.frm
mysql_201708070830/performance_schema/file_summary_by_instance.frm
mysql_201708070830/performance_schema/mutex_instances.frm
mysql_201708070830/performance_schema/performance_timers.frm
mysql_201708070830/performance_schema/rwlock_instances.frm
mysql_201708070830/performance_schema/setup_consumers.frm
mysql_201708070830/performance_schema/setup_instruments.frm
mysql_201708070830/performance_schema/setup_timers.frm
mysql_201708070830/performance_schema/threads.frm
mysql_201708070830/test/
mysql_201708070830/test/db.opt

sent 4157 bytes  received 371 bytes  1811.20 bytes/sec
total size is 718361496  speedup is 158648.74 (DRY RUN)
[root@backuplogC7 ~]#

- I decide to start with new send "mysql_201708070830"
[root@joytest mariadb]# btrfs sub del mysql*
Delete subvolume (no-commit): '/var/lib/mariadb/mysql'
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708070830'
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708080830'
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
[root@joytest mariadb]# btrfs sub sync .
^C
[root@joytest mariadb]# systemctl status mariadb
&#226;— mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset:
disabled)
   Active: failed (Result: exit-code) since Fri 2017-08-11 14:00:20 ICT; 1 day 18h ago
  Process: 15671 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
(code=exited, status=1/FAILURE)
  Process: 15670 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
status=0/SUCCESS)
  Process: 15642 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited,
status=0/SUCCESS)
 Main PID: 15670 (code=exited, status=0/SUCCESS)

Aug 11 14:00:19 joytest systemd[1]: Starting MariaDB database server...
Aug 11 14:00:19 joytest mysqld_safe[15670]: 170811 14:00:19 mysqld_safe Logging to
'/var/log/mariadb/mariadb.log'.
Aug 11 14:00:19 joytest mysqld_safe[15670]: 170811 14:00:19 mysqld_safe Starting
mysqld daemon with databases from /var/lib/mariadb/mysql
Aug 11 14:00:20 joytest systemd[1]: mariadb.service: control process exited,
code=exited status=1
Aug 11 14:00:20 joytest systemd[1]: Failed to start MariaDB database server.
Aug 11 14:00:20 joytest systemd[1]: Unit mariadb.service entered failed state.
Aug 11 14:00:20 joytest systemd[1]: mariadb.service failed.
[root@joytest mariadb]# btrfs sub sync .
[root@joytest mariadb]# btrfs sub list .

[root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708070830 | ssh
192.168.45.166 btrfs receive /var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708070830
At subvol mysql_201708070830
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830 
root@192.168.45.166://var/lib/mariadb/mysql_201708070830
sending incremental file list
mysql_201708070830/
mysql_201708070830/aria_log.00000001
mysql_201708070830/aria_log_control
mysql_201708070830/ib_logfile0
mysql_201708070830/ib_logfile1
mysql_201708070830/ibdata1
mysql_201708070830/mysql-bin.000001
mysql_201708070830/mysql-bin.index
mysql_201708070830/backups_db/
mysql_201708070830/backups_db/db.opt
mysql_201708070830/backups_db/ints.frm
mysql_201708070830/backups_db/tpc_backup_detail.frm
mysql_201708070830/backups_db/tpc_backup_header.frm
mysql_201708070830/backups_db/tpc_backup_history.frm
mysql_201708070830/backups_db/tpc_backup_schedule_level_master.frm
mysql_201708070830/backups_db/tpc_backup_schedule_master.frm
mysql_201708070830/backups_db/tpc_backup_schedule_master_extra.frm
mysql_201708070830/backups_db/tpc_backup_schedule_old.frm
mysql_201708070830/backups_db/tpc_backup_server.frm
mysql_201708070830/backups_db/tpc_backup_step.frm
mysql_201708070830/backups_db/tpc_backup_user.frm
mysql_201708070830/backups_db/v_backup_header_status.frm
mysql_201708070830/backups_db/v_calendar.frm
mysql_201708070830/backups_db/v_dup_backup.frm
mysql_201708070830/mysql/
mysql_201708070830/mysql/columns_priv.MYD
mysql_201708070830/mysql/columns_priv.MYI
mysql_201708070830/mysql/columns_priv.frm
mysql_201708070830/mysql/db.MYD
mysql_201708070830/mysql/db.MYI
mysql_201708070830/mysql/db.frm
mysql_201708070830/mysql/event.MYD
mysql_201708070830/mysql/event.MYI
mysql_201708070830/mysql/event.frm
mysql_201708070830/mysql/func.MYD
mysql_201708070830/mysql/func.MYI
mysql_201708070830/mysql/func.frm
mysql_201708070830/mysql/general_log.CSM
mysql_201708070830/mysql/general_log.CSV
mysql_201708070830/mysql/general_log.frm
mysql_201708070830/mysql/help_category.MYD
mysql_201708070830/mysql/help_category.MYI
mysql_201708070830/mysql/help_category.frm
mysql_201708070830/mysql/help_keyword.MYD
mysql_201708070830/mysql/help_keyword.MYI
mysql_201708070830/mysql/help_keyword.frm
mysql_201708070830/mysql/help_relation.MYD
mysql_201708070830/mysql/help_relation.MYI
mysql_201708070830/mysql/help_relation.frm
mysql_201708070830/mysql/help_topic.MYD
mysql_201708070830/mysql/help_topic.MYI
mysql_201708070830/mysql/help_topic.frm
mysql_201708070830/mysql/host.MYD
mysql_201708070830/mysql/host.MYI
mysql_201708070830/mysql/host.frm
mysql_201708070830/mysql/ndb_binlog_index.MYD
mysql_201708070830/mysql/ndb_binlog_index.MYI
mysql_201708070830/mysql/ndb_binlog_index.frm
mysql_201708070830/mysql/plugin.MYD
mysql_201708070830/mysql/plugin.MYI
mysql_201708070830/mysql/plugin.frm
mysql_201708070830/mysql/proc.MYD
mysql_201708070830/mysql/proc.MYI
mysql_201708070830/mysql/proc.frm
mysql_201708070830/mysql/procs_priv.MYD
mysql_201708070830/mysql/procs_priv.MYI
mysql_201708070830/mysql/procs_priv.frm
mysql_201708070830/mysql/proxies_priv.MYD
mysql_201708070830/mysql/proxies_priv.MYI
mysql_201708070830/mysql/proxies_priv.frm
mysql_201708070830/mysql/servers.MYD
mysql_201708070830/mysql/servers.MYI
mysql_201708070830/mysql/servers.frm
mysql_201708070830/mysql/slow_log.CSM
mysql_201708070830/mysql/slow_log.CSV
mysql_201708070830/mysql/slow_log.frm
mysql_201708070830/mysql/tables_priv.MYD
mysql_201708070830/mysql/tables_priv.MYI
mysql_201708070830/mysql/tables_priv.frm
mysql_201708070830/mysql/time_zone.MYD
mysql_201708070830/mysql/time_zone.MYI
mysql_201708070830/mysql/time_zone.frm
mysql_201708070830/mysql/time_zone_leap_second.MYD
mysql_201708070830/mysql/time_zone_leap_second.MYI
mysql_201708070830/mysql/time_zone_leap_second.frm
mysql_201708070830/mysql/time_zone_name.MYD
mysql_201708070830/mysql/time_zone_name.MYI
mysql_201708070830/mysql/time_zone_name.frm
mysql_201708070830/mysql/time_zone_transition.MYD
mysql_201708070830/mysql/time_zone_transition.MYI
mysql_201708070830/mysql/time_zone_transition.frm
mysql_201708070830/mysql/time_zone_transition_type.MYD
mysql_201708070830/mysql/time_zone_transition_type.MYI
mysql_201708070830/mysql/time_zone_transition_type.frm
mysql_201708070830/mysql/user.MYD
mysql_201708070830/mysql/user.MYI
mysql_201708070830/mysql/user.frm
mysql_201708070830/performance_schema/
mysql_201708070830/performance_schema/cond_instances.frm
mysql_201708070830/performance_schema/db.opt
mysql_201708070830/performance_schema/events_waits_current.frm
mysql_201708070830/performance_schema/events_waits_history.frm
mysql_201708070830/performance_schema/events_waits_history_long.frm
mysql_201708070830/performance_schema/events_waits_summary_by_instance.frm
mysql_201708070830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
mysql_201708070830/performance_schema/events_waits_summary_global_by_event_name.frm
mysql_201708070830/performance_schema/file_instances.frm
mysql_201708070830/performance_schema/file_summary_by_event_name.frm
mysql_201708070830/performance_schema/file_summary_by_instance.frm
mysql_201708070830/performance_schema/mutex_instances.frm
mysql_201708070830/performance_schema/performance_timers.frm
mysql_201708070830/performance_schema/rwlock_instances.frm
mysql_201708070830/performance_schema/setup_consumers.frm
mysql_201708070830/performance_schema/setup_instruments.frm
mysql_201708070830/performance_schema/setup_timers.frm
mysql_201708070830/performance_schema/threads.frm
mysql_201708070830/test/
mysql_201708070830/test/db.opt

sent 4157 bytes  received 371 bytes  3018.67 bytes/sec
total size is 718361496  speedup is 158648.74 (DRY RUN)

- "A lot diff" but dest mariadb can start and the data is correct.

[root@joytest mariadb]# btrfs sub list .
ID 313 gen 190 top level 5 path mysql_201708070830
[root@joytest mariadb]# btrfs sub snap mysql_201708070830 mysql
Create a snapshot of 'mysql_201708070830' in './mysql'
[root@joytest mariadb]# systemctl start mariadb
[root@joytest mariadb]# systemctl status mariadb
&#226;— mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset:
disabled)
   Active: active (running) since Sun 2017-08-13 08:30:17 ICT; 4min 36s ago
  Process: 17079 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
(code=exited, status=0/SUCCESS)
  Process: 17050 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited,
status=0/SUCCESS)
 Main PID: 17078 (mysqld_safe)
   CGroup: /system.slice/mariadb.service
           &#226;”&#339;&#226;”€17078 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
           &#226;””&#226;”€17630 /usr/libexec/mysqld --basedir=/usr
--datadir=/var/lib/mariadb/mysql --plugin-dir=/usr/lib64/mysql/plugin
--log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/m...

Aug 13 08:30:12 joytest systemd[1]: Starting MariaDB database server...
Aug 13 08:30:13 joytest mysqld_safe[17078]: 170813 08:30:13 mysqld_safe Logging to
'/var/log/mariadb/mariadb.log'.
Aug 13 08:30:13 joytest mysqld_safe[17078]: 170813 08:30:13 mysqld_safe Starting
mysqld daemon with databases from /var/lib/mariadb/mysql
Aug 13 08:30:17 joytest systemd[1]: Started MariaDB database server.
[root@joytest mariadb]# mysql -uroot -ppassword
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 5.5.52-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use backups_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

^[[ADatabase changed
MariaDB [backups_db]> select max(backup_start_date) from tpc_backup_header order by
backup_start_date desc;
+------------------------+
| max(backup_start_date) |
+------------------------+
| 2017-08-07 06:30:01    |
+------------------------+
1 row in set (0.09 sec)

MariaDB [backups_db]> Ctrl-C -- exit!
Aborted

- I continuously send the incremental and it work fine on dest (even rsync said "a
lot diff")  until the snapshot mysql_201708100830 mariadb can not start

[root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708090830
/var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708100830
At snapshot mysql_201708100830
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708100830
root@192.168.45.166:/var/lib/mariadb/mysql_201708100830
sending incremental file list
mysql_201708100830/
mysql_201708100830/aria_log.00000001
mysql_201708100830/aria_log_control
mysql_201708100830/ib_logfile0
mysql_201708100830/ib_logfile1
mysql_201708100830/ibdata1
mysql_201708100830/mysql-bin.000001
mysql_201708100830/mysql-bin.index
mysql_201708100830/backups_db/
mysql_201708100830/backups_db/db.opt
mysql_201708100830/backups_db/ints.frm
mysql_201708100830/backups_db/tpc_backup_detail.frm
mysql_201708100830/backups_db/tpc_backup_header.frm
mysql_201708100830/backups_db/tpc_backup_history.frm
mysql_201708100830/backups_db/tpc_backup_schedule_level_master.frm
mysql_201708100830/backups_db/tpc_backup_schedule_master.frm
mysql_201708100830/backups_db/tpc_backup_schedule_master_extra.frm
mysql_201708100830/backups_db/tpc_backup_schedule_old.frm
mysql_201708100830/backups_db/tpc_backup_server.frm
mysql_201708100830/backups_db/tpc_backup_step.frm
mysql_201708100830/backups_db/tpc_backup_user.frm
mysql_201708100830/backups_db/v_backup_header_status.frm
mysql_201708100830/backups_db/v_calendar.frm
mysql_201708100830/backups_db/v_dup_backup.frm
mysql_201708100830/mysql/
mysql_201708100830/mysql/columns_priv.MYD
mysql_201708100830/mysql/columns_priv.MYI
mysql_201708100830/mysql/columns_priv.frm
mysql_201708100830/mysql/db.MYD
mysql_201708100830/mysql/db.MYI
mysql_201708100830/mysql/db.frm
mysql_201708100830/mysql/event.MYD
mysql_201708100830/mysql/event.MYI
mysql_201708100830/mysql/event.frm
mysql_201708100830/mysql/func.MYD
mysql_201708100830/mysql/func.MYI
mysql_201708100830/mysql/func.frm
mysql_201708100830/mysql/general_log.CSM
mysql_201708100830/mysql/general_log.CSV
mysql_201708100830/mysql/general_log.frm
mysql_201708100830/mysql/help_category.MYD
mysql_201708100830/mysql/help_category.MYI
mysql_201708100830/mysql/help_category.frm
mysql_201708100830/mysql/help_keyword.MYD
mysql_201708100830/mysql/help_keyword.MYI
mysql_201708100830/mysql/help_keyword.frm
mysql_201708100830/mysql/help_relation.MYD
mysql_201708100830/mysql/help_relation.MYI
mysql_201708100830/mysql/help_relation.frm
mysql_201708100830/mysql/help_topic.MYD
mysql_201708100830/mysql/help_topic.MYI
mysql_201708100830/mysql/help_topic.frm
mysql_201708100830/mysql/host.MYD
mysql_201708100830/mysql/host.MYI
mysql_201708100830/mysql/host.frm
mysql_201708100830/mysql/ndb_binlog_index.MYD
mysql_201708100830/mysql/ndb_binlog_index.MYI
mysql_201708100830/mysql/ndb_binlog_index.frm
mysql_201708100830/mysql/plugin.MYD
mysql_201708100830/mysql/plugin.MYI
mysql_201708100830/mysql/plugin.frm
mysql_201708100830/mysql/proc.MYD
mysql_201708100830/mysql/proc.MYI
mysql_201708100830/mysql/proc.frm
mysql_201708100830/mysql/procs_priv.MYD
mysql_201708100830/mysql/procs_priv.MYI
mysql_201708100830/mysql/procs_priv.frm
mysql_201708100830/mysql/proxies_priv.MYD
mysql_201708100830/mysql/proxies_priv.MYI
mysql_201708100830/mysql/proxies_priv.frm
mysql_201708100830/mysql/servers.MYD
mysql_201708100830/mysql/servers.MYI
mysql_201708100830/mysql/servers.frm
mysql_201708100830/mysql/slow_log.CSM
mysql_201708100830/mysql/slow_log.CSV
mysql_201708100830/mysql/slow_log.frm
mysql_201708100830/mysql/tables_priv.MYD
mysql_201708100830/mysql/tables_priv.MYI
mysql_201708100830/mysql/tables_priv.frm
mysql_201708100830/mysql/time_zone.MYD
mysql_201708100830/mysql/time_zone.MYI
mysql_201708100830/mysql/time_zone.frm
mysql_201708100830/mysql/time_zone_leap_second.MYD
mysql_201708100830/mysql/time_zone_leap_second.MYI
mysql_201708100830/mysql/time_zone_leap_second.frm
mysql_201708100830/mysql/time_zone_name.MYD
mysql_201708100830/mysql/time_zone_name.MYI
mysql_201708100830/mysql/time_zone_name.frm
mysql_201708100830/mysql/time_zone_transition.MYD
mysql_201708100830/mysql/time_zone_transition.MYI
mysql_201708100830/mysql/time_zone_transition.frm
mysql_201708100830/mysql/time_zone_transition_type.MYD
mysql_201708100830/mysql/time_zone_transition_type.MYI
mysql_201708100830/mysql/time_zone_transition_type.frm
mysql_201708100830/mysql/user.MYD
mysql_201708100830/mysql/user.MYI
mysql_201708100830/mysql/user.frm
mysql_201708100830/performance_schema/
mysql_201708100830/performance_schema/cond_instances.frm
mysql_201708100830/performance_schema/db.opt
mysql_201708100830/performance_schema/events_waits_current.frm
mysql_201708100830/performance_schema/events_waits_history.frm
mysql_201708100830/performance_schema/events_waits_history_long.frm
mysql_201708100830/performance_schema/events_waits_summary_by_instance.frm
mysql_201708100830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
mysql_201708100830/performance_schema/events_waits_summary_global_by_event_name.frm
mysql_201708100830/performance_schema/file_instances.frm
mysql_201708100830/performance_schema/file_summary_by_event_name.frm
mysql_201708100830/performance_schema/file_summary_by_instance.frm
mysql_201708100830/performance_schema/mutex_instances.frm
mysql_201708100830/performance_schema/performance_timers.frm
mysql_201708100830/performance_schema/rwlock_instances.frm
mysql_201708100830/performance_schema/setup_consumers.frm
mysql_201708100830/performance_schema/setup_instruments.frm
mysql_201708100830/performance_schema/setup_timers.frm
mysql_201708100830/performance_schema/threads.frm
mysql_201708100830/test/
mysql_201708100830/test/db.opt

sent 4157 bytes  received 371 bytes  1811.20 bytes/sec
total size is 718361496  speedup is 158648.74 (DRY RUN)
[root@backuplogC7 ~]#

[root@joytest mariadb]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
57: eth0@if58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
qlen 1000
    link/ether 00:16:3e:e3:b3:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.45.166/21 brd 192.168.47.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fee3:b36f/64 scope link
       valid_lft forever preferred_lft forever
[root@joytest mariadb]# systemctl mariadb stop
Unknown operation 'mariadb'.
[root@joytest mariadb]# systemctl stop mariadb
[root@joytest mariadb]# btrfs sub list .
ID 313 gen 201 top level 5 path mysql_201708070830
ID 316 gen 199 top level 5 path mysql_201708080830
ID 317 gen 204 top level 5 path mysql
ID 318 gen 204 top level 5 path mysql_201708090830
[root@joytest mariadb]# btrfs sub del mysql
Delete subvolume (no-commit): '/var/lib/mariadb/mysql'
[root@joytest mariadb]# btrfs sub sync .
Subvolume id 317 is gone
[root@joytest mariadb]# btrfs sub snap mysql_201708090830 mysql
Create a snapshot of 'mysql_201708090830' in './mysql'
[root@joytest mariadb]# systemctl start mariadb
[root@joytest mariadb]# mysql -uroot -ppassword
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.52-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use backups_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [backups_db]> select max(backup_start_date) from tpc_backup_header order by
backup_start_date desc;
+------------------------+
| max(backup_start_date) |
+------------------------+
| 2017-08-09 06:30:01    |
+------------------------+
1 row in set (0.09 sec)

MariaDB [backups_db]> exit
Bye
[root@joytest mariadb]# systemctl stop mariadb
[root@joytest mariadb]# btrfs sub list .
ID 313 gen 207 top level 5 path mysql_201708070830
ID 316 gen 199 top level 5 path mysql_201708080830
ID 318 gen 205 top level 5 path mysql_201708090830
ID 319 gen 206 top level 5 path mysql
ID 320 gen 208 top level 5 path mysql_201708100830
[root@joytest mariadb]# btrfs sub del mysql
Delete subvolume (no-commit): '/var/lib/mariadb/mysql'
[root@joytest mariadb]# btrfs sub sync .
[root@joytest mariadb]# btrfs sub snap mysql_201708100830 mysql
Create a snapshot of 'mysql_201708100830' in './mysql'
[root@joytest mariadb]# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@joytest mariadb]#

- I still send the incremetal
[root@joytest mariadb]# btrfs sub list .
ID 313 gen 220 top level 5 path mysql_201708070830
ID 316 gen 199 top level 5 path mysql_201708080830
ID 318 gen 205 top level 5 path mysql_201708090830
ID 320 gen 211 top level 5 path mysql_201708100830
ID 321 gen 212 top level 5 path mysql
ID 322 gen 219 top level 5 path mysql_201708110830
ID 323 gen 219 top level 5 path mysql_201708120830
ID 324 gen 223 top level 5 path mysql_201708130830
[root@joytest mariadb]# btrfs sub del mysql
Delete subvolume (no-commit): '/var/lib/mariadb/mysql'
[root@joytest mariadb]# btrfs sub sync .
Subvolume id 321 is gone
[root@joytest mariadb]# btrfs sub snap mysql_201708130830 mysql
Create a snapshot of 'mysql_201708130830' in './mysql'
[root@joytest mariadb]# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@joytest mariadb]#

- The latest log from the source send tested at 201708130818 for comfirm that the
script run with correct syntax.
Start Send 201708130818
btrfs send -p /var/lib/mariadb/mysql_201708110830
/var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
rsync -avnc /var/lib/mariadb/mysql_201708120830
root@192.168.45.166://var/lib/mariadb/mysql_201708120830
Stop Send 201708130818

- the real send log after change include rsync in send script
Start Send 201708130905
btrfs send -p /var/lib/mariadb/mysql_201708120830
/var/lib/mariadb/mysql_201708130830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At snapshot mysql_201708130830
rsync -avnc /var/lib/mariadb/mysql_201708130830
root@192.168.45.166://var/lib/mariadb/mysql_201708130830
sending incremental file list
mysql_201708130830/
mysql_201708130830/aria_log.00000001
mysql_201708130830/aria_log_control
mysql_201708130830/ib_logfile0
mysql_201708130830/ib_logfile1
mysql_201708130830/ibdata1
mysql_201708130830/mysql-bin.000001
mysql_201708130830/mysql-bin.index
mysql_201708130830/backups_db/
mysql_201708130830/backups_db/db.opt
mysql_201708130830/backups_db/ints.frm
mysql_201708130830/backups_db/tpc_backup_detail.frm
mysql_201708130830/backups_db/tpc_backup_header.frm
mysql_201708130830/backups_db/tpc_backup_history.frm
mysql_201708130830/backups_db/tpc_backup_schedule_level_master.frm
mysql_201708130830/backups_db/tpc_backup_schedule_master.frm
mysql_201708130830/backups_db/tpc_backup_schedule_master_extra.frm
mysql_201708130830/backups_db/tpc_backup_schedule_old.frm
mysql_201708130830/backups_db/tpc_backup_server.frm
mysql_201708130830/backups_db/tpc_backup_step.frm
mysql_201708130830/backups_db/tpc_backup_user.frm
mysql_201708130830/backups_db/v_backup_header_status.frm
mysql_201708130830/backups_db/v_calendar.frm
mysql_201708130830/backups_db/v_dup_backup.frm
mysql_201708130830/mysql/
mysql_201708130830/mysql/columns_priv.MYD
mysql_201708130830/mysql/columns_priv.MYI
mysql_201708130830/mysql/columns_priv.frm
mysql_201708130830/mysql/db.MYD
mysql_201708130830/mysql/db.MYI
mysql_201708130830/mysql/db.frm
mysql_201708130830/mysql/event.MYD
mysql_201708130830/mysql/event.MYI
mysql_201708130830/mysql/event.frm
mysql_201708130830/mysql/func.MYD
mysql_201708130830/mysql/func.MYI
mysql_201708130830/mysql/func.frm
mysql_201708130830/mysql/general_log.CSM
mysql_201708130830/mysql/general_log.CSV
mysql_201708130830/mysql/general_log.frm
mysql_201708130830/mysql/help_category.MYD
mysql_201708130830/mysql/help_category.MYI
mysql_201708130830/mysql/help_category.frm
mysql_201708130830/mysql/help_keyword.MYD
mysql_201708130830/mysql/help_keyword.MYI
mysql_201708130830/mysql/help_keyword.frm
mysql_201708130830/mysql/help_relation.MYD
mysql_201708130830/mysql/help_relation.MYI
mysql_201708130830/mysql/help_relation.frm
mysql_201708130830/mysql/help_topic.MYD
mysql_201708130830/mysql/help_topic.MYI
mysql_201708130830/mysql/help_topic.frm
mysql_201708130830/mysql/host.MYD
mysql_201708130830/mysql/host.MYI
mysql_201708130830/mysql/host.frm
mysql_201708130830/mysql/ndb_binlog_index.MYD
mysql_201708130830/mysql/ndb_binlog_index.MYI
mysql_201708130830/mysql/ndb_binlog_index.frm
mysql_201708130830/mysql/plugin.MYD
mysql_201708130830/mysql/plugin.MYI
mysql_201708130830/mysql/plugin.frm
mysql_201708130830/mysql/proc.MYD
mysql_201708130830/mysql/proc.MYI
mysql_201708130830/mysql/proc.frm
mysql_201708130830/mysql/procs_priv.MYD
mysql_201708130830/mysql/procs_priv.MYI
mysql_201708130830/mysql/procs_priv.frm
mysql_201708130830/mysql/proxies_priv.MYD
mysql_201708130830/mysql/proxies_priv.MYI
mysql_201708130830/mysql/proxies_priv.frm
mysql_201708130830/mysql/servers.MYD
mysql_201708130830/mysql/servers.MYI
mysql_201708130830/mysql/servers.frm
mysql_201708130830/mysql/slow_log.CSM
mysql_201708130830/mysql/slow_log.CSV
mysql_201708130830/mysql/slow_log.frm
mysql_201708130830/mysql/tables_priv.MYD
mysql_201708130830/mysql/tables_priv.MYI
mysql_201708130830/mysql/tables_priv.frm
mysql_201708130830/mysql/time_zone.MYD
mysql_201708130830/mysql/time_zone.MYI
mysql_201708130830/mysql/time_zone.frm
mysql_201708130830/mysql/time_zone_leap_second.MYD
mysql_201708130830/mysql/time_zone_leap_second.MYI
mysql_201708130830/mysql/time_zone_leap_second.frm
mysql_201708130830/mysql/time_zone_name.MYD
mysql_201708130830/mysql/time_zone_name.MYI
mysql_201708130830/mysql/time_zone_name.frm
mysql_201708130830/mysql/time_zone_transition.MYD
mysql_201708130830/mysql/time_zone_transition.MYI
mysql_201708130830/mysql/time_zone_transition.frm
mysql_201708130830/mysql/time_zone_transition_type.MYD
mysql_201708130830/mysql/time_zone_transition_type.MYI
mysql_201708130830/mysql/time_zone_transition_type.frm
mysql_201708130830/mysql/user.MYD
mysql_201708130830/mysql/user.MYI
mysql_201708130830/mysql/user.frm
mysql_201708130830/performance_schema/
mysql_201708130830/performance_schema/cond_instances.frm
mysql_201708130830/performance_schema/db.opt
mysql_201708130830/performance_schema/events_waits_current.frm
mysql_201708130830/performance_schema/events_waits_history.frm
mysql_201708130830/performance_schema/events_waits_history_long.frm
mysql_201708130830/performance_schema/events_waits_summary_by_instance.frm
mysql_201708130830/performance_schema/events_waits_summary_by_thread_by_event_name.frm
mysql_201708130830/performance_schema/events_waits_summary_global_by_event_name.frm
mysql_201708130830/performance_schema/file_instances.frm
mysql_201708130830/performance_schema/file_summary_by_event_name.frm
mysql_201708130830/performance_schema/file_summary_by_instance.frm
mysql_201708130830/performance_schema/mutex_instances.frm
mysql_201708130830/performance_schema/performance_timers.frm
mysql_201708130830/performance_schema/rwlock_instances.frm
mysql_201708130830/performance_schema/setup_consumers.frm
mysql_201708130830/performance_schema/setup_instruments.frm
mysql_201708130830/performance_schema/setup_timers.frm
mysql_201708130830/performance_schema/threads.frm
mysql_201708130830/test/
mysql_201708130830/test/db.opt

sent 4157 bytes  received 371 bytes  823.27 bytes/sec
total size is 718361496  speedup is 158648.74 (DRY RUN)
Stop Send 201708130905
[root@backuplogC7 ~]# ^C
[root@backuplogC7 ~]#

Am I lost?

Best Regards,
Siranee Jaraswachirakul.


> On Fri, Aug 11, 2017 at 11:08 PM,  <siranee.ja@tpc.co.th> wrote:
>
>
>> The backup script has the btrfs sync command since Aug 3
>
>
> From your script:
>> system btrfs sub snap -r $basepath $snappath
>> system btrfs sub sync $basepath
>
> From the man page: sync <path> [subvolid...]
>            Wait until given subvolume(s) are completely removed from the
>            filesystem after deletion.
>
>
> This 'subvolume sync' command, per the man page, is only about
> subvolume deletion. I suggest replacing it with a regular sync
> command.
>
> I think the problem is that the script does things so fast that the
> snapshot is not always consistent on disk before btrfs send starts.
> It's just a guess though. If I'm right, this means the rsync mismaches
> mean the destination snapshots are bad. Here's what I would do:
>
>
> - delete all the bad/mismatching snapshots only on the destination computer.
>
> - he most recent good snapshot pair, which rsync shows origin and
> destination match, is mysql_201708080830 so you can keep that one on
> both sides.
>
> - manually do incremental send/receive, starting with
> mysql_201708090830/, to make the destination current again with the
> origin.
>
> - confirm with rsync that the snapshot pairs on origin and destination
> are the same
>
> - now resume using the modified script, which will do snapshot -> sync -> send.
>
> OPTIONAL, you could add to your script an rsync -avnc to double check
> that the incremental send receive is working. This is admittedly
> inefficient because it checks the *entire* contents of the snapshots
> on both sides, it's not just checking the incremental data. But if it
> doesn't take too long, it will help restore trust in send/receive, and
> confirm that a regular sync is needed in between snapshot and send.
>
>
>
>
> --
> Chris Murphy
>

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:

> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
> root@192.168.45.166://var/lib/mariadb/mysql_201708090830


You need trailing / for the first directory with -a option.

rsync -a dir dir

is not the same command as

rsync -a dir/ dir

It's confusing but your command is trying to create mysql_201708090830
directory on the source, in the mysql_201708090830 on the destination.
That is why everything mismatches. To make it mean "contents of" you
need trailing slash on at least the origin.



-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi Chris,

I started as your suggestion again. The diff occured since snapshot
mysql_201708090830 manually send. What should I do next?

- delete all the bad/mismatching snapshots only on the destination computer.
[root@joytest ~]# date
Sun Aug 13 10:27:23 ICT 2017
[root@joytest ~]# cd /var/lib/mariadb
[root@joytest mariadb]# btrfs sub list .
ID 313 gen 220 top level 5 path mysql_201708070830
ID 316 gen 199 top level 5 path mysql_201708080830
ID 318 gen 205 top level 5 path mysql_201708090830
ID 320 gen 211 top level 5 path mysql_201708100830
ID 322 gen 219 top level 5 path mysql_201708110830
ID 323 gen 219 top level 5 path mysql_201708120830
ID 324 gen 224 top level 5 path mysql_201708130830
ID 325 gen 225 top level 5 path mysql
[root@joytest mariadb]# btrfs sub del mysql_201708130830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
[root@joytest mariadb]# btrfs sub del mysql_201708120830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
[root@joytest mariadb]# btrfs sub del mysql_201708110830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
[root@joytest mariadb]# btrfs sub del mysql_201708100830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
[root@joytest mariadb]# btrfs sub del mysql_201708090830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
[root@joytest mariadb]# btrfs sub sync .
[root@joytest mariadb]# systemctl status mariadb
â— mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset:
disabled)
   Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago
  Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
(code=exited, status=1/FAILURE)
  Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
status=0/SUCCESS)
  Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited,
status=0/SUCCESS)
 Main PID: 19870 (code=exited, status=0/SUCCESS)

Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to
'/var/log/mariadb/mariadb.log'.
Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting
mysqld daemon with databases from /var/lib/mariadb/mysql
Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
code=exited status=1
Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
[root@joytest mariadb]# btrfs sub list .
ID 313 gen 220 top level 5 path mysql_201708070830
ID 316 gen 199 top level 5 path mysql_201708080830
ID 325 gen 225 top level 5 path mysql
[root@joytest mariadb]#

- The most recent good snapshot pair, which rsync shows origin and
destination match, is mysql_201708080830 so you can keep that one on
both sides.

[root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
ID 257 gen 538 top level 5 path mysql
ID 316 gen 498 top level 5 path mysql_201708060830
ID 317 gen 503 top level 5 path mysql_201708070830
ID 318 gen 507 top level 5 path mysql_201708080830
ID 319 gen 514 top level 5 path mysql_201708090830
ID 320 gen 524 top level 5 path mysql_201708100830
ID 321 gen 529 top level 5 path mysql_201708110830
ID 322 gen 533 top level 5 path mysql_201708120830
ID 323 gen 538 top level 5 path mysql_201708130830
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  842.67 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
sending incremental file list
./

sent 3769 bytes  received 19 bytes  841.78 bytes/sec
total size is 718361496  speedup is 189641.37 (DRY RUN)
[root@backuplogC7 ~]# date
Sun Aug 13 10:34:05 ICT 2017
[root@backuplogC7 ~]#

- manually do incremental send/receive, starting with
mysql_201708090830/, to make the destination current again with the
origin.

[root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
/var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708090830
At snapshot mysql_201708090830
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
sending incremental file list
./
ib_logfile1
ibdata1

sent 3779 bytes  received 25 bytes  507.20 bytes/sec
total size is 718361496  speedup is 188843.72 (DRY RUN)
[root@backuplogC7 ~]#

Best Regards,

Siranee Jaraswachirakul.

> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>
>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>
>
> You need trailing / for the first directory with -a option.
>
> rsync -a dir dir
>
> is not the same command as
>
> rsync -a dir/ dir
>
> It's confusing but your command is trying to create mysql_201708090830
> directory on the source, in the mysql_201708090830 on the destination.
> That is why everything mismatches. To make it mean "contents of" you
> need trailing slash on at least the origin.
>
>
>
> --
> Chris Murphy
>

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Sat, Aug 12, 2017 at 9:40 PM,  <siranee.ja@tpc.co.th> wrote:

> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
> sending incremental file list
> ./
> ib_logfile1
> ibdata1
>
> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
> total size is 718361496  speedup is 188843.72 (DRY RUN)


OK so I don't think this can be a sync related problem. That snapshot
has been committed to disk days ago. There's definitely something
wrong with the incremental send/receive, but it's unclear whether this
is a kernel bug (send side) or btrfs-progs (receive side), or if
there's any chance of file system corruption/confusion happening with
either of the two subvolumes on the origin or the subvolume (parent)
on the destination.

So that means you're really in the weeds on what to do next.

Try deleting mysql_201708090830/ snapshot on the destination. And
resend but this time do a full send of that snapshot, don't use -p. I
wonder if a full send, rather than incremental makes a difference.
Follow it up with the rsync command to compare origin and destination.



-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[A L]

Have you checked that there is no Received UUID on the source subvolume?

# btrfs sub show volume/mysql/
volume/mysql
         Name:                   mysql
         UUID:                   8a94524e-a956-c14b-bb8d-d453627f27d5
         Parent UUID:            -
         Received UUID:          -
         Creation time:          2017-04-17 11:46:20 +0200
         Subvolume ID:           1469
         Generation:             122934
         Gen at creation:        78671
         Parent ID:              5
         Top level ID:           5
         Flags:                  -
         Snapshot(s):

There is no Received UUID here. If it has, then btrfs send-receive will 
have problems, since all snapshots of the source subvolume will have the 
same Received UUID and it can't tell the differences between the snapshots.

On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote:
> Hi Chris,
>
> I started as your suggestion again. The diff occured since snapshot
> mysql_201708090830 manually send. What should I do next?
>
> - delete all the bad/mismatching snapshots only on the destination computer.
> [root@joytest ~]# date
> Sun Aug 13 10:27:23 ICT 2017
> [root@joytest ~]# cd /var/lib/mariadb
> [root@joytest mariadb]# btrfs sub list .
> ID 313 gen 220 top level 5 path mysql_201708070830
> ID 316 gen 199 top level 5 path mysql_201708080830
> ID 318 gen 205 top level 5 path mysql_201708090830
> ID 320 gen 211 top level 5 path mysql_201708100830
> ID 322 gen 219 top level 5 path mysql_201708110830
> ID 323 gen 219 top level 5 path mysql_201708120830
> ID 324 gen 224 top level 5 path mysql_201708130830
> ID 325 gen 225 top level 5 path mysql
> [root@joytest mariadb]# btrfs sub del mysql_201708130830
> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
> [root@joytest mariadb]# btrfs sub del mysql_201708120830
> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
> [root@joytest mariadb]# btrfs sub del mysql_201708110830
> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
> [root@joytest mariadb]# btrfs sub del mysql_201708100830
> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
> [root@joytest mariadb]# btrfs sub del mysql_201708090830
> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
> [root@joytest mariadb]# btrfs sub sync .
> [root@joytest mariadb]# systemctl status mariadb
> â— mariadb.service - MariaDB database server
>     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset:
> disabled)
>     Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago
>    Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
> (code=exited, status=1/FAILURE)
>    Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
> status=0/SUCCESS)
>    Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited,
> status=0/SUCCESS)
>   Main PID: 19870 (code=exited, status=0/SUCCESS)
>
> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to
> '/var/log/mariadb/mariadb.log'.
> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting
> mysqld daemon with databases from /var/lib/mariadb/mysql
> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
> code=exited status=1
> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
> [root@joytest mariadb]# btrfs sub list .
> ID 313 gen 220 top level 5 path mysql_201708070830
> ID 316 gen 199 top level 5 path mysql_201708080830
> ID 325 gen 225 top level 5 path mysql
> [root@joytest mariadb]#
>
> - The most recent good snapshot pair, which rsync shows origin and
> destination match, is mysql_201708080830 so you can keep that one on
> both sides.
>
> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
> ID 257 gen 538 top level 5 path mysql
> ID 316 gen 498 top level 5 path mysql_201708060830
> ID 317 gen 503 top level 5 path mysql_201708070830
> ID 318 gen 507 top level 5 path mysql_201708080830
> ID 319 gen 514 top level 5 path mysql_201708090830
> ID 320 gen 524 top level 5 path mysql_201708100830
> ID 321 gen 529 top level 5 path mysql_201708110830
> ID 322 gen 533 top level 5 path mysql_201708120830
> ID 323 gen 538 top level 5 path mysql_201708130830
> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
> sending incremental file list
> ./
>
> sent 3773 bytes  received 19 bytes  842.67 bytes/sec
> total size is 718361496  speedup is 189441.32 (DRY RUN)
> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
> sending incremental file list
> ./
>
> sent 3769 bytes  received 19 bytes  841.78 bytes/sec
> total size is 718361496  speedup is 189641.37 (DRY RUN)
> [root@backuplogC7 ~]# date
> Sun Aug 13 10:34:05 ICT 2017
> [root@backuplogC7 ~]#
>
> - manually do incremental send/receive, starting with
> mysql_201708090830/, to make the destination current again with the
> origin.
>
> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
> /var/lib/mariadb
> At subvol /var/lib/mariadb/mysql_201708090830
> At snapshot mysql_201708090830
> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
> sending incremental file list
> ./
> ib_logfile1
> ibdata1
>
> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
> total size is 718361496  speedup is 188843.72 (DRY RUN)
> [root@backuplogC7 ~]#
>
> Best Regards,
>
> Siranee Jaraswachirakul.
>
>> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>>
>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>>
>> You need trailing / for the first directory with -a option.
>>
>> rsync -a dir dir
>>
>> is not the same command as
>>
>> rsync -a dir/ dir
>>
>> It's confusing but your command is trying to create mysql_201708090830
>> directory on the source, in the mysql_201708090830 on the destination.
>> That is why everything mismatches. To make it mean "contents of" you
>> need trailing slash on at least the origin.
>>
>>
>>
>> --
>> Chris Murphy
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi Chris,

Try deleting mysql_201708090830/ snapshot on the destination. And
resend but this time do a full send of that snapshot, don't use -p. I
wonder if a full send, rather than incremental makes a difference.
Follow it up with the rsync command to compare origin and destination.


Yes, It's different.

[root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708090830 | ssh
192.168.45.166 btrfs receive /var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708090830
At subvol mysql_201708090830
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  842.67 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)

Best Regards,
Siranee Jaraswachirakul

> On Sat, Aug 12, 2017 at 9:40 PM,  <siranee.ja@tpc.co.th> wrote:
>
>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
>> sending incremental file list
>> ./
>> ib_logfile1
>> ibdata1
>>
>> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
>> total size is 718361496  speedup is 188843.72 (DRY RUN)
>
>
> OK so I don't think this can be a sync related problem. That snapshot
> has been committed to disk days ago. There's definitely something
> wrong with the incremental send/receive, but it's unclear whether this
> is a kernel bug (send side) or btrfs-progs (receive side), or if
> there's any chance of file system corruption/confusion happening with
> either of the two subvolumes on the origin or the subvolume (parent)
> on the destination.
>
> So that means you're really in the weeds on what to do next.
>
> Try deleting mysql_201708090830/ snapshot on the destination. And
> resend but this time do a full send of that snapshot, don't use -p. I
> wonder if a full send, rather than incremental makes a difference.
> Follow it up with the rsync command to compare origin and destination.
>
>
>
> --
> Chris Murphy
>

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi "A L",

[root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
/var/lib/mariadb/mysql
        Name:                   mysql
        UUID:                   92f319c5-e132-3249-9b13-d39ee77a2b44
        Parent UUID:            -
        Received UUID:          3ad0334a-4063-654c-add6-b1cbcdeaa639
        Creation time:          2017-06-21 13:27:41 +0700
        Subvolume ID:           257
        Generation:             539
        Gen at creation:        9
        Parent ID:              5
        Top level ID:           5
        Flags:                  -
        Snapshot(s):
                                mysql_201708060830
                                mysql_201708070830
                                mysql_201708080830
                                mysql_201708090830
                                mysql_201708100830
                                mysql_201708110830
                                mysql_201708120830
                                mysql_201708130830

yes I think it has Received UUID because I restored the source from snapshot
mysql_201708040830 for prove that the local snapshot was work.

How to clear the Received UUID ?
What to do next?

Best Regards,
Siranee Jaraswachirakul.

> Have you checked that there is no Received UUID on the source subvolume?
>
> # btrfs sub show volume/mysql/
> volume/mysql
>          Name:                   mysql
>          UUID:                   8a94524e-a956-c14b-bb8d-d453627f27d5
>          Parent UUID:            -
>          Received UUID:          -
>          Creation time:          2017-04-17 11:46:20 +0200
>          Subvolume ID:           1469
>          Generation:             122934
>          Gen at creation:        78671
>          Parent ID:              5
>          Top level ID:           5
>          Flags:                  -
>          Snapshot(s):
>
> There is no Received UUID here. If it has, then btrfs send-receive will
> have problems, since all snapshots of the source subvolume will have the
> same Received UUID and it can't tell the differences between the snapshots.
>
> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote:
>> Hi Chris,
>>
>> I started as your suggestion again. The diff occured since snapshot
>> mysql_201708090830 manually send. What should I do next?
>>
>> - delete all the bad/mismatching snapshots only on the destination computer.
>> [root@joytest ~]# date
>> Sun Aug 13 10:27:23 ICT 2017
>> [root@joytest ~]# cd /var/lib/mariadb
>> [root@joytest mariadb]# btrfs sub list .
>> ID 313 gen 220 top level 5 path mysql_201708070830
>> ID 316 gen 199 top level 5 path mysql_201708080830
>> ID 318 gen 205 top level 5 path mysql_201708090830
>> ID 320 gen 211 top level 5 path mysql_201708100830
>> ID 322 gen 219 top level 5 path mysql_201708110830
>> ID 323 gen 219 top level 5 path mysql_201708120830
>> ID 324 gen 224 top level 5 path mysql_201708130830
>> ID 325 gen 225 top level 5 path mysql
>> [root@joytest mariadb]# btrfs sub del mysql_201708130830
>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
>> [root@joytest mariadb]# btrfs sub del mysql_201708120830
>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
>> [root@joytest mariadb]# btrfs sub del mysql_201708110830
>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
>> [root@joytest mariadb]# btrfs sub del mysql_201708100830
>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
>> [root@joytest mariadb]# btrfs sub del mysql_201708090830
>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
>> [root@joytest mariadb]# btrfs sub sync .
>> [root@joytest mariadb]# systemctl status mariadb
>> â— mariadb.service - MariaDB database server
>>     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor
>> preset:
>> disabled)
>>     Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min
>> ago
>>    Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
>> (code=exited, status=1/FAILURE)
>>    Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
>> status=0/SUCCESS)
>>    Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n
>> (code=exited,
>> status=0/SUCCESS)
>>   Main PID: 19870 (code=exited, status=0/SUCCESS)
>>
>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to
>> '/var/log/mariadb/mariadb.log'.
>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting
>> mysqld daemon with databases from /var/lib/mariadb/mysql
>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
>> code=exited status=1
>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
>> [root@joytest mariadb]# btrfs sub list .
>> ID 313 gen 220 top level 5 path mysql_201708070830
>> ID 316 gen 199 top level 5 path mysql_201708080830
>> ID 325 gen 225 top level 5 path mysql
>> [root@joytest mariadb]#
>>
>> - The most recent good snapshot pair, which rsync shows origin and
>> destination match, is mysql_201708080830 so you can keep that one on
>> both sides.
>>
>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
>> ID 257 gen 538 top level 5 path mysql
>> ID 316 gen 498 top level 5 path mysql_201708060830
>> ID 317 gen 503 top level 5 path mysql_201708070830
>> ID 318 gen 507 top level 5 path mysql_201708080830
>> ID 319 gen 514 top level 5 path mysql_201708090830
>> ID 320 gen 524 top level 5 path mysql_201708100830
>> ID 321 gen 529 top level 5 path mysql_201708110830
>> ID 322 gen 533 top level 5 path mysql_201708120830
>> ID 323 gen 538 top level 5 path mysql_201708130830
>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
>> sending incremental file list
>> ./
>>
>> sent 3773 bytes  received 19 bytes  842.67 bytes/sec
>> total size is 718361496  speedup is 189441.32 (DRY RUN)
>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
>> sending incremental file list
>> ./
>>
>> sent 3769 bytes  received 19 bytes  841.78 bytes/sec
>> total size is 718361496  speedup is 189641.37 (DRY RUN)
>> [root@backuplogC7 ~]# date
>> Sun Aug 13 10:34:05 ICT 2017
>> [root@backuplogC7 ~]#
>>
>> - manually do incremental send/receive, starting with
>> mysql_201708090830/, to make the destination current again with the
>> origin.
>>
>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
>> /var/lib/mariadb
>> At subvol /var/lib/mariadb/mysql_201708090830
>> At snapshot mysql_201708090830
>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
>> sending incremental file list
>> ./
>> ib_logfile1
>> ibdata1
>>
>> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
>> total size is 718361496  speedup is 188843.72 (DRY RUN)
>> [root@backuplogC7 ~]#
>>
>> Best Regards,
>>
>> Siranee Jaraswachirakul.
>>
>>> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>>>
>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>>>
>>> You need trailing / for the first directory with -a option.
>>>
>>> rsync -a dir dir
>>>
>>> is not the same command as
>>>
>>> rsync -a dir/ dir
>>>
>>> It's confusing but your command is trying to create mysql_201708090830
>>> directory on the source, in the mysql_201708090830 on the destination.
>>> That is why everything mismatches. To make it mean "contents of" you
>>> need trailing slash on at least the origin.
>>>
>>>
>>>
>>> --
>>> Chris Murphy
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: btrfs issue with mariadb incremental backup

[A L]

On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote:
> Hi "A L",
>
> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
> /var/lib/mariadb/mysql
>          Name:                   mysql
>          UUID:                   92f319c5-e132-3249-9b13-d39ee77a2b44
>          Parent UUID:            -
>          Received UUID:          3ad0334a-4063-654c-add6-b1cbcdeaa639
>          Creation time:          2017-06-21 13:27:41 +0700
>          Subvolume ID:           257
>          Generation:             539
>          Gen at creation:        9
>          Parent ID:              5
>          Top level ID:           5
>          Flags:                  -
>          Snapshot(s):
>                                  mysql_201708060830
>                                  mysql_201708070830
>                                  mysql_201708080830
>                                  mysql_201708090830
>                                  mysql_201708100830
>                                  mysql_201708110830
>                                  mysql_201708120830
>                                  mysql_201708130830
>
> yes I think it has Received UUID because I restored the source from snapshot
> mysql_201708040830 for prove that the local snapshot was work.
>
> How to clear the Received UUID ?
> What to do next?
You need to make a read-write snapshot of  /var/lib/mariadb/mysql and 
then remove the old subvolume and all its snapshots.

Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md

# cd /mnt/btr_pool
# mv mysubvolume mysubvolume.broken
# btrfs subvolume snapshot mysubvolume.broken mysubvolume

You can do the same with each of your snapshots too, and send them as 
full snapshots (without -p).

~A

> Best Regards,
> Siranee Jaraswachirakul.
>
>> Have you checked that there is no Received UUID on the source subvolume?
>>
>> # btrfs sub show volume/mysql/
>> volume/mysql
>>           Name:                   mysql
>>           UUID:                   8a94524e-a956-c14b-bb8d-d453627f27d5
>>           Parent UUID:            -
>>           Received UUID:          -
>>           Creation time:          2017-04-17 11:46:20 +0200
>>           Subvolume ID:           1469
>>           Generation:             122934
>>           Gen at creation:        78671
>>           Parent ID:              5
>>           Top level ID:           5
>>           Flags:                  -
>>           Snapshot(s):
>>
>> There is no Received UUID here. If it has, then btrfs send-receive will
>> have problems, since all snapshots of the source subvolume will have the
>> same Received UUID and it can't tell the differences between the snapshots.
>>
>> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote:
>>> Hi Chris,
>>>
>>> I started as your suggestion again. The diff occured since snapshot
>>> mysql_201708090830 manually send. What should I do next?
>>>
>>> - delete all the bad/mismatching snapshots only on the destination computer.
>>> [root@joytest ~]# date
>>> Sun Aug 13 10:27:23 ICT 2017
>>> [root@joytest ~]# cd /var/lib/mariadb
>>> [root@joytest mariadb]# btrfs sub list .
>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>> ID 318 gen 205 top level 5 path mysql_201708090830
>>> ID 320 gen 211 top level 5 path mysql_201708100830
>>> ID 322 gen 219 top level 5 path mysql_201708110830
>>> ID 323 gen 219 top level 5 path mysql_201708120830
>>> ID 324 gen 224 top level 5 path mysql_201708130830
>>> ID 325 gen 225 top level 5 path mysql
>>> [root@joytest mariadb]# btrfs sub del mysql_201708130830
>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
>>> [root@joytest mariadb]# btrfs sub del mysql_201708120830
>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
>>> [root@joytest mariadb]# btrfs sub del mysql_201708110830
>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
>>> [root@joytest mariadb]# btrfs sub del mysql_201708100830
>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
>>> [root@joytest mariadb]# btrfs sub del mysql_201708090830
>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
>>> [root@joytest mariadb]# btrfs sub sync .
>>> [root@joytest mariadb]# systemctl status mariadb
>>> â— mariadb.service - MariaDB database server
>>>      Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor
>>> preset:
>>> disabled)
>>>      Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min
>>> ago
>>>     Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
>>> (code=exited, status=1/FAILURE)
>>>     Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
>>> status=0/SUCCESS)
>>>     Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n
>>> (code=exited,
>>> status=0/SUCCESS)
>>>    Main PID: 19870 (code=exited, status=0/SUCCESS)
>>>
>>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to
>>> '/var/log/mariadb/mariadb.log'.
>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting
>>> mysqld daemon with databases from /var/lib/mariadb/mysql
>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
>>> code=exited status=1
>>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
>>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
>>> [root@joytest mariadb]# btrfs sub list .
>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>> ID 325 gen 225 top level 5 path mysql
>>> [root@joytest mariadb]#
>>>
>>> - The most recent good snapshot pair, which rsync shows origin and
>>> destination match, is mysql_201708080830 so you can keep that one on
>>> both sides.
>>>
>>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
>>> ID 257 gen 538 top level 5 path mysql
>>> ID 316 gen 498 top level 5 path mysql_201708060830
>>> ID 317 gen 503 top level 5 path mysql_201708070830
>>> ID 318 gen 507 top level 5 path mysql_201708080830
>>> ID 319 gen 514 top level 5 path mysql_201708090830
>>> ID 320 gen 524 top level 5 path mysql_201708100830
>>> ID 321 gen 529 top level 5 path mysql_201708110830
>>> ID 322 gen 533 top level 5 path mysql_201708120830
>>> ID 323 gen 538 top level 5 path mysql_201708130830
>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
>>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
>>> sending incremental file list
>>> ./
>>>
>>> sent 3773 bytes  received 19 bytes  842.67 bytes/sec
>>> total size is 718361496  speedup is 189441.32 (DRY RUN)
>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
>>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
>>> sending incremental file list
>>> ./
>>>
>>> sent 3769 bytes  received 19 bytes  841.78 bytes/sec
>>> total size is 718361496  speedup is 189641.37 (DRY RUN)
>>> [root@backuplogC7 ~]# date
>>> Sun Aug 13 10:34:05 ICT 2017
>>> [root@backuplogC7 ~]#
>>>
>>> - manually do incremental send/receive, starting with
>>> mysql_201708090830/, to make the destination current again with the
>>> origin.
>>>
>>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
>>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
>>> /var/lib/mariadb
>>> At subvol /var/lib/mariadb/mysql_201708090830
>>> At snapshot mysql_201708090830
>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
>>> sending incremental file list
>>> ./
>>> ib_logfile1
>>> ibdata1
>>>
>>> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
>>> total size is 718361496  speedup is 188843.72 (DRY RUN)
>>> [root@backuplogC7 ~]#
>>>
>>> Best Regards,
>>>
>>> Siranee Jaraswachirakul.
>>>
>>>> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>>>>
>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>>>> You need trailing / for the first directory with -a option.
>>>>
>>>> rsync -a dir dir
>>>>
>>>> is not the same command as
>>>>
>>>> rsync -a dir/ dir
>>>>
>>>> It's confusing but your command is trying to create mysql_201708090830
>>>> directory on the source, in the mysql_201708090830 on the destination.
>>>> That is why everything mismatches. To make it mean "contents of" you
>>>> need trailing slash on at least the origin.
>>>>
>>>>
>>>>
>>>> --
>>>> Chris Murphy
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi "A L",

As your suggestion. Does it mean I have to make all my current subvolumes to clear
"Receive UUID" and send them with full snapshots?
>>                                  mysql
>>                                  mysql_201708060830
>>                                  mysql_201708070830
>>                                  mysql_201708080830
>>                                  mysql_201708090830
>>                                  mysql_201708100830
>>                                  mysql_201708110830
>>                                  mysql_201708120830
>>                                  mysql_201708130830

The next day that I take snapshot with the script will be
mysql_201708140830

which will be sent with incremental from mysql_201708130830 to mysql_201708140830

Is this correct?

I am not understand with the url that you gave me which said

"I'm getting an error: Aborted: "Received UUID" is set

You probably restored a backup with send-receive, and made it read/write using btrfs
property set. This is bad, as all snapshots and backups will inherit this identical
"Received UUID", which results in all these subvolumes will be treated as
"containing same data".

To fix this, create a "proper" snapshot:

- This is as your suggestion for the subvolume "mysql"

# cd /mnt/btr_pool
# mv mysubvolume mysubvolume.broken
# btrfs subvolume snapshot mysubvolume.broken mysubvolume

Now, mysubvolume should have an empty "Received UUID". Note that in order to have a
clean environment, you also need to fix all subvolumes (snapshots as well as
backups) that you created with the broken subvolume.

Check if there are more broken subvolumes:

# btrfs subvolume show mysubvolume.broken
# btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
# btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>

- This guide seem that I have to clear <"Received UUID" > only the subvolume "mysql"
and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r 
instead of btrfs subvolume snapshot. Is this correct?

Now clean all subvolume listed (same as above, but using btrfs subvolume snapshot -r
now). Then delete all the broken subvolumes:

# btrfs subvolume delete *.broken

Finally, you should have a clean environment, and btrbk will not complain any more.


Best Regards,
Siranee Jaraswachirakul.

>
>
> On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote:
>> Hi "A L",
>>
>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
>> /var/lib/mariadb/mysql
>>          Name:                   mysql
>>          UUID:                   92f319c5-e132-3249-9b13-d39ee77a2b44
>>          Parent UUID:            -
>>          Received UUID:          3ad0334a-4063-654c-add6-b1cbcdeaa639
>>          Creation time:          2017-06-21 13:27:41 +0700
>>          Subvolume ID:           257
>>          Generation:             539
>>          Gen at creation:        9
>>          Parent ID:              5
>>          Top level ID:           5
>>          Flags:                  -
>>          Snapshot(s):
>>                                  mysql_201708060830
>>                                  mysql_201708070830
>>                                  mysql_201708080830
>>                                  mysql_201708090830
>>                                  mysql_201708100830
>>                                  mysql_201708110830
>>                                  mysql_201708120830
>>                                  mysql_201708130830
>>
>> yes I think it has Received UUID because I restored the source from snapshot
>> mysql_201708040830 for prove that the local snapshot was work.
>>
>> How to clear the Received UUID ?
>> What to do next?
> You need to make a read-write snapshot of  /var/lib/mariadb/mysql and
> then remove the old subvolume and all its snapshots.
>
> Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md
>
> # cd /mnt/btr_pool
> # mv mysubvolume mysubvolume.broken
> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
>
> You can do the same with each of your snapshots too, and send them as
> full snapshots (without -p).
>
> ~A
>
>> Best Regards,
>> Siranee Jaraswachirakul.
>>
>>> Have you checked that there is no Received UUID on the source subvolume?
>>>
>>> # btrfs sub show volume/mysql/
>>> volume/mysql
>>>           Name:                   mysql
>>>           UUID:                   8a94524e-a956-c14b-bb8d-d453627f27d5
>>>           Parent UUID:            -
>>>           Received UUID:          -
>>>           Creation time:          2017-04-17 11:46:20 +0200
>>>           Subvolume ID:           1469
>>>           Generation:             122934
>>>           Gen at creation:        78671
>>>           Parent ID:              5
>>>           Top level ID:           5
>>>           Flags:                  -
>>>           Snapshot(s):
>>>
>>> There is no Received UUID here. If it has, then btrfs send-receive will
>>> have problems, since all snapshots of the source subvolume will have the
>>> same Received UUID and it can't tell the differences between the snapshots.
>>>
>>> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote:
>>>> Hi Chris,
>>>>
>>>> I started as your suggestion again. The diff occured since snapshot
>>>> mysql_201708090830 manually send. What should I do next?
>>>>
>>>> - delete all the bad/mismatching snapshots only on the destination computer.
>>>> [root@joytest ~]# date
>>>> Sun Aug 13 10:27:23 ICT 2017
>>>> [root@joytest ~]# cd /var/lib/mariadb
>>>> [root@joytest mariadb]# btrfs sub list .
>>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>>> ID 318 gen 205 top level 5 path mysql_201708090830
>>>> ID 320 gen 211 top level 5 path mysql_201708100830
>>>> ID 322 gen 219 top level 5 path mysql_201708110830
>>>> ID 323 gen 219 top level 5 path mysql_201708120830
>>>> ID 324 gen 224 top level 5 path mysql_201708130830
>>>> ID 325 gen 225 top level 5 path mysql
>>>> [root@joytest mariadb]# btrfs sub del mysql_201708130830
>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
>>>> [root@joytest mariadb]# btrfs sub del mysql_201708120830
>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
>>>> [root@joytest mariadb]# btrfs sub del mysql_201708110830
>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
>>>> [root@joytest mariadb]# btrfs sub del mysql_201708100830
>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
>>>> [root@joytest mariadb]# btrfs sub del mysql_201708090830
>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
>>>> [root@joytest mariadb]# btrfs sub sync .
>>>> [root@joytest mariadb]# systemctl status mariadb
>>>> &#226;— mariadb.service - MariaDB database server
>>>>      Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor
>>>> preset:
>>>> disabled)
>>>>      Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h
>>>> 24min
>>>> ago
>>>>     Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
>>>> (code=exited, status=1/FAILURE)
>>>>     Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
>>>> status=0/SUCCESS)
>>>>     Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n
>>>> (code=exited,
>>>> status=0/SUCCESS)
>>>>    Main PID: 19870 (code=exited, status=0/SUCCESS)
>>>>
>>>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging
>>>> to
>>>> '/var/log/mariadb/mariadb.log'.
>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting
>>>> mysqld daemon with databases from /var/lib/mariadb/mysql
>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
>>>> code=exited status=1
>>>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
>>>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
>>>> [root@joytest mariadb]# btrfs sub list .
>>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>>> ID 325 gen 225 top level 5 path mysql
>>>> [root@joytest mariadb]#
>>>>
>>>> - The most recent good snapshot pair, which rsync shows origin and
>>>> destination match, is mysql_201708080830 so you can keep that one on
>>>> both sides.
>>>>
>>>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
>>>> ID 257 gen 538 top level 5 path mysql
>>>> ID 316 gen 498 top level 5 path mysql_201708060830
>>>> ID 317 gen 503 top level 5 path mysql_201708070830
>>>> ID 318 gen 507 top level 5 path mysql_201708080830
>>>> ID 319 gen 514 top level 5 path mysql_201708090830
>>>> ID 320 gen 524 top level 5 path mysql_201708100830
>>>> ID 321 gen 529 top level 5 path mysql_201708110830
>>>> ID 322 gen 533 top level 5 path mysql_201708120830
>>>> ID 323 gen 538 top level 5 path mysql_201708130830
>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
>>>> sending incremental file list
>>>> ./
>>>>
>>>> sent 3773 bytes  received 19 bytes  842.67 bytes/sec
>>>> total size is 718361496  speedup is 189441.32 (DRY RUN)
>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
>>>> sending incremental file list
>>>> ./
>>>>
>>>> sent 3769 bytes  received 19 bytes  841.78 bytes/sec
>>>> total size is 718361496  speedup is 189641.37 (DRY RUN)
>>>> [root@backuplogC7 ~]# date
>>>> Sun Aug 13 10:34:05 ICT 2017
>>>> [root@backuplogC7 ~]#
>>>>
>>>> - manually do incremental send/receive, starting with
>>>> mysql_201708090830/, to make the destination current again with the
>>>> origin.
>>>>
>>>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
>>>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
>>>> /var/lib/mariadb
>>>> At subvol /var/lib/mariadb/mysql_201708090830
>>>> At snapshot mysql_201708090830
>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
>>>> sending incremental file list
>>>> ./
>>>> ib_logfile1
>>>> ibdata1
>>>>
>>>> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
>>>> total size is 718361496  speedup is 188843.72 (DRY RUN)
>>>> [root@backuplogC7 ~]#
>>>>
>>>> Best Regards,
>>>>
>>>> Siranee Jaraswachirakul.
>>>>
>>>>> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>>>>>
>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>>>>> You need trailing / for the first directory with -a option.
>>>>>
>>>>> rsync -a dir dir
>>>>>
>>>>> is not the same command as
>>>>>
>>>>> rsync -a dir/ dir
>>>>>
>>>>> It's confusing but your command is trying to create mysql_201708090830
>>>>> directory on the source, in the mysql_201708090830 on the destination.
>>>>> That is why everything mismatches. To make it mean "contents of" you
>>>>> need trailing slash on at least the origin.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chris Murphy
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>
>

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Sun, Aug 13, 2017 at 4:49 AM,  <siranee.ja@tpc.co.th> wrote:

> [root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708090830 | ssh
> 192.168.45.166 btrfs receive /var/lib/mariadb
> At subvol /var/lib/mariadb/mysql_201708090830
> At subvol mysql_201708090830
> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
> root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/
> sending incremental file list
> ./
>
> sent 3773 bytes  received 19 bytes  842.67 bytes/sec
> total size is 718361496  speedup is 189441.32 (DRY RUN)


OK so the full send is sane. That suggests the file systems on both
sides are OK, and that the problem is related strictly to incremental
send/receive. It suggests that in a critical way either an origin
parent or its copy on the destination are not identical. Why will be
hard to figure out, and I'm not even sure it's worth it.

Right now you basically have a work around: a known good full send,
you can start doing incremental send/receive again using this full
send/receive snapshot as the parent (with -p).

Where confusion can happen is if you have to restore a snapshot, I
think you must commit to a whole new canonical tree with the reverse
send/receive being a full one, NOT incremental. I'm asserting this, I
do not know if it is true, I don't know if the tools should prevent
reverse incremental send/receive, but the very idea of reversing an
incremental send receive to me just seems logically problematic. Any
time you reverse the send/receive direction it must be full. I think.

I did once get into trouble with this myself, having to do a restore
from backup, to a new file system. I then took a rw snapshot of it,
and made that the canonical live modify subvolume. I then ro
snapshotted it, and tried to do an incremental send *back* to backup
and it failed with an error I don't remember but I think it was later
send/receive versions than 4.4 where there's some extract sanity
checking to prevent this. And I was able to work around it with -c
(clone) option. Anyway, I got sufficiently confused myself after all
of that, that I pretty much gave up on such a strategy, blew away all
the ro snapshots on both sides, and started from scratch with a new
full send, and then resumed the incrementals unidirectionally. And
anytime they are reversed I assume it must be a full send.

Anyway, I think it's perilous to reverse directions while also trying
to do incremental send/receive without a lot of testing to thoroughly
understand what's going on. And I basically got to fuck this, not
worth the complexity.

-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[Chris Murphy]

On Sun, Aug 13, 2017 at 4:52 AM,  <siranee.ja@tpc.co.th> wrote:
> Hi "A L",
>
> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
> /var/lib/mariadb/mysql
>         Name:                   mysql
>         UUID:                   92f319c5-e132-3249-9b13-d39ee77a2b44
>         Parent UUID:            -
>         Received UUID:          3ad0334a-4063-654c-add6-b1cbcdeaa639
>         Creation time:          2017-06-21 13:27:41 +0700
>         Subvolume ID:           257
>         Generation:             539
>         Gen at creation:        9
>         Parent ID:              5
>         Top level ID:           5
>         Flags:                  -
>         Snapshot(s):
>                                 mysql_201708060830
>                                 mysql_201708070830
>                                 mysql_201708080830
>                                 mysql_201708090830
>                                 mysql_201708100830
>                                 mysql_201708110830
>                                 mysql_201708120830
>                                 mysql_201708130830
>
> yes I think it has Received UUID because I restored the source from snapshot
> mysql_201708040830 for prove that the local snapshot was work.
>
> How to clear the Received UUID ?
> What to do next?


I'm using btrfs-progs 4.12, and I just did a btrfs send/receive from
file system A to B; and then on B I took a rw snapshot of the ro
snapshot. The ro snapshot has Received UUID, but the rw snapshot made
from it does not have Received UUID. So now I'm confused how you have
a rw snapshot with Received UUID set.


Did you ever use 'btrfs property set' to change a ro snapshot to rw?
That's the only way I can think it's possible. The only other thing is
maybe the behavior has changed since btrfs-progs 4.4.

It is possible this behavior has changed since btrfs-progs 4.4. The
changelogs shows there are improvements in send/receive in particular
4.8.3 and 4.8.4, but also others. I have no idea which are related.
But anyway, this is one of the reasons why the expert users on this
almost always say use something newer because it's just too hard to
remember, even with changelogs, what's fixed and where.

My suggestion is to investigate moving to kernel 4.9 series; and it
should be safe to move to btrfs-progs 4.12 now. Any new features in
4.12 that can't be supported with an older kernel should fail
gracefully.

No matter what, you must have separate backups. You can't depend
exclusively on one backup, no matter the method. It's fine to use
send/receive for backup1. But backup2 should be a conventional backup,
e.g. XFS with rsync. Basically this is a game of risk and the more
independent backups you have, the more failure cases (especially user
error) you can recover from, at the expense of some complexity.



-- 
Chris Murphy

Re: btrfs issue with mariadb incremental backup

[A L]

---- From: siranee.ja@tpc.co.th -- Sent: 2017-08-13 - 16:00 ----

> Hi "A L",
> 
> As your suggestion. Does it mean I have to make all my current subvolumes to clear
> "Receive UUID" and send them with full snapshots?

You need to clear all subvolumes with 'Received UUID' in order to use send-receive safely. once you have done that you can start over with new incremental sends (after the initial full send).

I suggest to remove any existing snapshots of those volumes to avoid confusion.

>>>                                  mysql
>>>                                  mysql_201708060830
>>>                                  mysql_201708070830
>>>                                  mysql_201708080830
>>>                                  mysql_201708090830
>>>                                  mysql_201708100830
>>>                                  mysql_201708110830
>>>                                  mysql_201708120830
>>>                                  mysql_201708130830
> 
> The next day that I take snapshot with the script will be
> mysql_201708140830
> 
> which will be sent with incremental from mysql_201708130830 to mysql_201708140830
> 
> Is this correct?

yes, but your current snapshots can't be used because of the 'Received UUID'.
> 
> I am not understand with the url that you gave me which said

btrbk is a backup tool for btrfs. It detects the problem of the 'Received UUID'. The FAQ explains how to remove the 'Received UUID' from the source volumes and snapshots. 

 It is safer to remove all existing snapshots then trying to fix them and risking the integrity of the data. 


> 
> "I'm getting an error: Aborted: "Received UUID" is set
> 
> You probably restored a backup with send-receive, and made it read/write using btrfs
> property set. This is bad, as all snapshots and backups will inherit this identical
> "Received UUID", which results in all these subvolumes will be treated as
> "containing same data".
> 
> To fix this, create a "proper" snapshot:
> 
> - This is as your suggestion for the subvolume "mysql"
> 
> # cd /mnt/btr_pool
> # mv mysubvolume mysubvolume.broken
> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
> 
> Now, mysubvolume should have an empty "Received UUID". Note that in order to have a
> clean environment, you also need to fix all subvolumes (snapshots as well as
> backups) that you created with the broken subvolume.
> 
> Check if there are more broken subvolumes:
> 
> # btrfs subvolume show mysubvolume.broken
> # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
> # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>
> 
> - This guide seem that I have to clear <"Received UUID" > only the subvolume "mysql"
> and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r 
> instead of btrfs subvolume snapshot. Is this correct?
> 
> Now clean all subvolume listed (same as above, but using btrfs subvolume snapshot -r
> now). Then delete all the broken subvolumes:
> 
> # btrfs subvolume delete *.broken
> 
> Finally, you should have a clean environment, and btrbk will not complain any more.
> 
> 
> Best Regards,
> Siranee Jaraswachirakul.
> 
>>
>>
>> On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote:
>>> Hi "A L",
>>>
>>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
>>> /var/lib/mariadb/mysql
>>>          Name:                   mysql
>>>          UUID:                   92f319c5-e132-3249-9b13-d39ee77a2b44
>>>          Parent UUID:            -
>>>          Received UUID:          3ad0334a-4063-654c-add6-b1cbcdeaa639
>>>          Creation time:          2017-06-21 13:27:41 +0700
>>>          Subvolume ID:           257
>>>          Generation:             539
>>>          Gen at creation:        9
>>>          Parent ID:              5
>>>          Top level ID:           5
>>>          Flags:                  -
>>>          Snapshot(s):
>>>                                  mysql_201708060830
>>>                                  mysql_201708070830
>>>                                  mysql_201708080830
>>>                                  mysql_201708090830
>>>                                  mysql_201708100830
>>>                                  mysql_201708110830
>>>                                  mysql_201708120830
>>>                                  mysql_201708130830
>>>
>>> yes I think it has Received UUID because I restored the source from snapshot
>>> mysql_201708040830 for prove that the local snapshot was work.
>>>
>>> How to clear the Received UUID ?
>>> What to do next?
>> You need to make a read-write snapshot of  /var/lib/mariadb/mysql and
>> then remove the old subvolume and all its snapshots.
>>
>> Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md
>>
>> # cd /mnt/btr_pool
>> # mv mysubvolume mysubvolume.broken
>> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
>>
>> You can do the same with each of your snapshots too, and send them as
>> full snapshots (without -p).
>>
>> ~A
>>
>>> Best Regards,
>>> Siranee Jaraswachirakul.
>>>
>>>> Have you checked that there is no Received UUID on the source subvolume?
>>>>
>>>> # btrfs sub show volume/mysql/
>>>> volume/mysql
>>>>           Name:                   mysql
>>>>           UUID:                   8a94524e-a956-c14b-bb8d-d453627f27d5
>>>>           Parent UUID:            -
>>>>           Received UUID:          -
>>>>           Creation time:          2017-04-17 11:46:20 +0200
>>>>           Subvolume ID:           1469
>>>>           Generation:             122934
>>>>           Gen at creation:        78671
>>>>           Parent ID:              5
>>>>           Top level ID:           5
>>>>           Flags:                  -
>>>>           Snapshot(s):
>>>>
>>>> There is no Received UUID here. If it has, then btrfs send-receive will
>>>> have problems, since all snapshots of the source subvolume will have the
>>>> same Received UUID and it can't tell the differences between the snapshots.
>>>>
>>>> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote:
>>>>> Hi Chris,
>>>>>
>>>>> I started as your suggestion again. The diff occured since snapshot
>>>>> mysql_201708090830 manually send. What should I do next?
>>>>>
>>>>> - delete all the bad/mismatching snapshots only on the destination computer.
>>>>> [root@joytest ~]# date
>>>>> Sun Aug 13 10:27:23 ICT 2017
>>>>> [root@joytest ~]# cd /var/lib/mariadb
>>>>> [root@joytest mariadb]# btrfs sub list .
>>>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>>>> ID 318 gen 205 top level 5 path mysql_201708090830
>>>>> ID 320 gen 211 top level 5 path mysql_201708100830
>>>>> ID 322 gen 219 top level 5 path mysql_201708110830
>>>>> ID 323 gen 219 top level 5 path mysql_201708120830
>>>>> ID 324 gen 224 top level 5 path mysql_201708130830
>>>>> ID 325 gen 225 top level 5 path mysql
>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708130830
>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708120830
>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708110830
>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708100830
>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708090830
>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
>>>>> [root@joytest mariadb]# btrfs sub sync .
>>>>> [root@joytest mariadb]# systemctl status mariadb
>>>>> &#226;— mariadb.service - MariaDB database server
>>>>>      Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor
>>>>> preset:
>>>>> disabled)
>>>>>      Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h
>>>>> 24min
>>>>> ago
>>>>>     Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
>>>>> (code=exited, status=1/FAILURE)
>>>>>     Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
>>>>> status=0/SUCCESS)
>>>>>     Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n
>>>>> (code=exited,
>>>>> status=0/SUCCESS)
>>>>>    Main PID: 19870 (code=exited, status=0/SUCCESS)
>>>>>
>>>>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
>>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging
>>>>> to
>>>>> '/var/log/mariadb/mariadb.log'.
>>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting
>>>>> mysqld daemon with databases from /var/lib/mariadb/mysql
>>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
>>>>> code=exited status=1
>>>>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
>>>>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
>>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
>>>>> [root@joytest mariadb]# btrfs sub list .
>>>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>>>> ID 325 gen 225 top level 5 path mysql
>>>>> [root@joytest mariadb]#
>>>>>
>>>>> - The most recent good snapshot pair, which rsync shows origin and
>>>>> destination match, is mysql_201708080830 so you can keep that one on
>>>>> both sides.
>>>>>
>>>>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
>>>>> ID 257 gen 538 top level 5 path mysql
>>>>> ID 316 gen 498 top level 5 path mysql_201708060830
>>>>> ID 317 gen 503 top level 5 path mysql_201708070830
>>>>> ID 318 gen 507 top level 5 path mysql_201708080830
>>>>> ID 319 gen 514 top level 5 path mysql_201708090830
>>>>> ID 320 gen 524 top level 5 path mysql_201708100830
>>>>> ID 321 gen 529 top level 5 path mysql_201708110830
>>>>> ID 322 gen 533 top level 5 path mysql_201708120830
>>>>> ID 323 gen 538 top level 5 path mysql_201708130830
>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
>>>>> sending incremental file list
>>>>> ./
>>>>>
>>>>> sent 3773 bytes  received 19 bytes  842.67 bytes/sec
>>>>> total size is 718361496  speedup is 189441.32 (DRY RUN)
>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
>>>>> sending incremental file list
>>>>> ./
>>>>>
>>>>> sent 3769 bytes  received 19 bytes  841.78 bytes/sec
>>>>> total size is 718361496  speedup is 189641.37 (DRY RUN)
>>>>> [root@backuplogC7 ~]# date
>>>>> Sun Aug 13 10:34:05 ICT 2017
>>>>> [root@backuplogC7 ~]#
>>>>>
>>>>> - manually do incremental send/receive, starting with
>>>>> mysql_201708090830/, to make the destination current again with the
>>>>> origin.
>>>>>
>>>>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
>>>>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
>>>>> /var/lib/mariadb
>>>>> At subvol /var/lib/mariadb/mysql_201708090830
>>>>> At snapshot mysql_201708090830
>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
>>>>> sending incremental file list
>>>>> ./
>>>>> ib_logfile1
>>>>> ibdata1
>>>>>
>>>>> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
>>>>> total size is 718361496  speedup is 188843.72 (DRY RUN)
>>>>> [root@backuplogC7 ~]#
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Siranee Jaraswachirakul.
>>>>>
>>>>>> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>>>>>>
>>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>>>>>> You need trailing / for the first directory with -a option.
>>>>>>
>>>>>> rsync -a dir dir
>>>>>>
>>>>>> is not the same command as
>>>>>>
>>>>>> rsync -a dir/ dir
>>>>>>
>>>>>> It's confusing but your command is trying to create mysql_201708090830
>>>>>> directory on the source, in the mysql_201708090830 on the destination.
>>>>>> That is why everything mismatches. To make it mean "contents of" you
>>>>>> need trailing slash on at least the origin.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chris Murphy
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>
>>
> 
> 

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi "A L",

Thank you very much for your suggestion. I've got it.
It work properly right now.
what I have done are following.

mv all current mysql and snapshot mysql to mbroken and mbroken_yymmddhhmm
btrfs subvolume snapshot mbroken to mysql (to make it rw subvolume without Recieve
UUID)
btrfs subvolume snapshot mbroken_yymmddhhmm  to rw_yymmddhhmm (to make it rw
subvolume without Recieve UUID)
btrfs subvolume snap -r rw_yymmddhhmm  to mysql_yymmddhhmm ( to make ro snapshot)

after this just send full for the first ro snapshot (mysql_201708070830) and send
incremental until the end (mysql_201708130830).





[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708070830
rw_201708070830
Create a snapshot of 'mbroken_201708070830' in './rw_201708070830'
[root@backuplogC7 mariadb]# btrfs sub list .
ID 257 gen 542 top level 5 path mbroken
ID 317 gen 576 top level 5 path mbroken_201708070830
ID 318 gen 568 top level 5 path mbroken_201708080830
ID 319 gen 569 top level 5 path mbroken_201708090830
ID 320 gen 570 top level 5 path mbroken_201708100830
ID 321 gen 571 top level 5 path mbroken_201708110830
ID 322 gen 572 top level 5 path mbroken_201708120830
ID 323 gen 573 top level 5 path mbroken_201708130830
ID 324 gen 543 top level 5 path mysql
ID 348 gen 576 top level 5 path rw_201708070830
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708080830
rw_201708080830
Create a snapshot of 'mbroken_201708080830' in './rw_201708080830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708090830
rw_201708090830
Create a snapshot of 'mbroken_201708090830' in './rw_201708090830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708100830
rw_201708100830
Create a snapshot of 'mbroken_201708100830' in './rw_201708100830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708110830
rw_201708110830
Create a snapshot of 'mbroken_201708110830' in './rw_201708110830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708120830
rw_201708120830
Create a snapshot of 'mbroken_201708120830' in './rw_201708120830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708130830
rw_201708130830
Create a snapshot of 'mbroken_201708130830' in './rw_201708130830'
[root@backuplogC7 mariadb]# btrfs sub list .
ID 257 gen 542 top level 5 path mbroken
ID 317 gen 576 top level 5 path mbroken_201708070830
ID 318 gen 577 top level 5 path mbroken_201708080830
ID 319 gen 578 top level 5 path mbroken_201708090830
ID 320 gen 579 top level 5 path mbroken_201708100830
ID 321 gen 580 top level 5 path mbroken_201708110830
ID 322 gen 581 top level 5 path mbroken_201708120830
ID 323 gen 582 top level 5 path mbroken_201708130830
ID 324 gen 543 top level 5 path mysql
ID 348 gen 576 top level 5 path rw_201708070830
ID 349 gen 577 top level 5 path rw_201708080830
ID 350 gen 578 top level 5 path rw_201708090830
ID 351 gen 579 top level 5 path rw_201708100830
ID 352 gen 580 top level 5 path rw_201708110830
ID 353 gen 581 top level 5 path rw_201708120830
ID 354 gen 582 top level 5 path rw_201708130830
[root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep
"3ad0334a-4063-654c-add6-b1cbcdeaa639"
ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken
ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708070830
ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708080830
ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708090830
ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708100830
ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708110830
ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708120830
ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708130830
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708070830 mysql_201708070830
Create a readonly snapshot of 'rw_201708070830' in './mysql_201708070830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708080830 mysql_201708080830
Create a readonly snapshot of 'rw_201708080830' in './mysql_201708080830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708090830 mysql_201708090830
Create a readonly snapshot of 'rw_201708090830' in './mysql_201708090830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708100830 mysql_201708100830
Create a readonly snapshot of 'rw_201708100830' in './mysql_201708100830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708110830 mysql_201708110830
Create a readonly snapshot of 'rw_201708110830' in './mysql_201708110830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708120830 mysql_201708120830
Create a readonly snapshot of 'rw_201708120830' in './mysql_201708120830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708130830 mysql_201708130830
Create a readonly snapshot of 'rw_201708130830' in './mysql_201708130830'
[root@backuplogC7 mariadb]# btrfs sub list .
ID 257 gen 542 top level 5 path mbroken
ID 317 gen 576 top level 5 path mbroken_201708070830
ID 318 gen 577 top level 5 path mbroken_201708080830
ID 319 gen 578 top level 5 path mbroken_201708090830
ID 320 gen 579 top level 5 path mbroken_201708100830
ID 321 gen 580 top level 5 path mbroken_201708110830
ID 322 gen 581 top level 5 path mbroken_201708120830
ID 323 gen 582 top level 5 path mbroken_201708130830
ID 324 gen 584 top level 5 path mysql
ID 348 gen 583 top level 5 path rw_201708070830
ID 349 gen 584 top level 5 path rw_201708080830
ID 350 gen 585 top level 5 path rw_201708090830
ID 351 gen 586 top level 5 path rw_201708100830
ID 352 gen 587 top level 5 path rw_201708110830
ID 353 gen 588 top level 5 path rw_201708120830
ID 354 gen 589 top level 5 path rw_201708130830
ID 355 gen 583 top level 5 path mysql_201708070830
ID 356 gen 584 top level 5 path mysql_201708080830
ID 357 gen 585 top level 5 path mysql_201708090830
ID 358 gen 586 top level 5 path mysql_201708100830
ID 359 gen 587 top level 5 path mysql_201708110830
ID 360 gen 588 top level 5 path mysql_201708120830
ID 361 gen 589 top level 5 path mysql_201708130830

[root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep
"3ad0334a-4063-654c-add6-b1cbcdeaa639"
ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken
ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708070830
ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708080830
ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708090830
ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708100830
ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708110830
ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708120830
ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708130830
[root@backuplogC7 mariadb]# btrfs send /var/lib/mariadb/mysql_201708070830 | ssh
192.168.45.166 btrfs receive /var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708070830
At subvol mysql_201708070830
[root@backuplogC7 mariadb]# btrfs sub show mysql_201708070830
/var/lib/mariadb/mysql_201708070830
        Name:                   mysql_201708070830
        UUID:                   70ee3c31-126d-574a-814c-e3b4c81b414e
        Parent UUID:            1d5bb8eb-b0df-2549-8b62-552cfa517609
        Received UUID:          -
        Creation time:          2017-08-14 07:00:08 +0700
        Subvolume ID:           355
        Generation:             583
        Gen at creation:        583
        Parent ID:              5
        Top level ID:           5
        Flags:                  readonly
        Snapshot(s):
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708070830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  1083.43 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708070830
/var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708080830
At snapshot mysql_201708080830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708080830/
sending incremental file list
./

sent 3769 bytes  received 19 bytes  688.73 bytes/sec
total size is 718361496  speedup is 189641.37 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708080830
/var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708090830
At snapshot mysql_201708090830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  583.38 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708090830
/var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708100830
At snapshot mysql_201708100830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708100830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708100830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708100830
/var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708110830
At snapshot mysql_201708110830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708110830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708110830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708110830
/var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708120830
At snapshot mysql_201708120830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708120830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708120830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708120830
/var/lib/mariadb/mysql_201708130830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708130830
At snapshot mysql_201708130830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708130830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708130830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]#


>
>
> ---- From: siranee.ja@tpc.co.th -- Sent: 2017-08-13 - 16:00 ----
>
>> Hi "A L",
>>
>> As your suggestion. Does it mean I have to make all my current subvolumes to clear
>> "Receive UUID" and send them with full snapshots?
>
> You need to clear all subvolumes with 'Received UUID' in order to use send-receive
> safely. once you have done that you can start over with new incremental sends (after
> the initial full send).
>
> I suggest to remove any existing snapshots of those volumes to avoid confusion.
>
>>>>                                  mysql
>>>>                                  mysql_201708060830
>>>>                                  mysql_201708070830
>>>>                                  mysql_201708080830
>>>>                                  mysql_201708090830
>>>>                                  mysql_201708100830
>>>>                                  mysql_201708110830
>>>>                                  mysql_201708120830
>>>>                                  mysql_201708130830
>>
>> The next day that I take snapshot with the script will be
>> mysql_201708140830
>>
>> which will be sent with incremental from mysql_201708130830 to mysql_201708140830
>>
>> Is this correct?
>
> yes, but your current snapshots can't be used because of the 'Received UUID'.
>>
>> I am not understand with the url that you gave me which said
>
> btrbk is a backup tool for btrfs. It detects the problem of the 'Received UUID'. The
> FAQ explains how to remove the 'Received UUID' from the source volumes and
> snapshots.
>
>  It is safer to remove all existing snapshots then trying to fix them and risking
> the integrity of the data.
>
>
>>
>> "I'm getting an error: Aborted: "Received UUID" is set
>>
>> You probably restored a backup with send-receive, and made it read/write using
>> btrfs
>> property set. This is bad, as all snapshots and backups will inherit this
>> identical
>> "Received UUID", which results in all these subvolumes will be treated as
>> "containing same data".
>>
>> To fix this, create a "proper" snapshot:
>>
>> - This is as your suggestion for the subvolume "mysql"
>>
>> # cd /mnt/btr_pool
>> # mv mysubvolume mysubvolume.broken
>> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
>>
>> Now, mysubvolume should have an empty "Received UUID". Note that in order to have
>> a
>> clean environment, you also need to fix all subvolumes (snapshots as well as
>> backups) that you created with the broken subvolume.
>>
>> Check if there are more broken subvolumes:
>>
>> # btrfs subvolume show mysubvolume.broken
>> # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
>> # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>
>>
>> - This guide seem that I have to clear <"Received UUID" > only the subvolume
>> "mysql"
>> and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r
>> instead of btrfs subvolume snapshot. Is this correct?
>>
>> Now clean all subvolume listed (same as above, but using btrfs subvolume snapshot
>> -r
>> now). Then delete all the broken subvolumes:
>>
>> # btrfs subvolume delete *.broken
>>
>> Finally, you should have a clean environment, and btrbk will not complain any
>> more.
>>
>>
>> Best Regards,
>> Siranee Jaraswachirakul.
>>
>>>
>>>
>>> On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote:
>>>> Hi "A L",
>>>>
>>>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
>>>> /var/lib/mariadb/mysql
>>>>          Name:                   mysql
>>>>          UUID:                   92f319c5-e132-3249-9b13-d39ee77a2b44
>>>>          Parent UUID:            -
>>>>          Received UUID:          3ad0334a-4063-654c-add6-b1cbcdeaa639
>>>>          Creation time:          2017-06-21 13:27:41 +0700
>>>>          Subvolume ID:           257
>>>>          Generation:             539
>>>>          Gen at creation:        9
>>>>          Parent ID:              5
>>>>          Top level ID:           5
>>>>          Flags:                  -
>>>>          Snapshot(s):
>>>>                                  mysql_201708060830
>>>>                                  mysql_201708070830
>>>>                                  mysql_201708080830
>>>>                                  mysql_201708090830
>>>>                                  mysql_201708100830
>>>>                                  mysql_201708110830
>>>>                                  mysql_201708120830
>>>>                                  mysql_201708130830
>>>>
>>>> yes I think it has Received UUID because I restored the source from snapshot
>>>> mysql_201708040830 for prove that the local snapshot was work.
>>>>
>>>> How to clear the Received UUID ?
>>>> What to do next?
>>> You need to make a read-write snapshot of? /var/lib/mariadb/mysql and
>>> then remove the old subvolume and all its snapshots.
>>>
>>> Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md
>>>
>>> # cd /mnt/btr_pool
>>> # mv mysubvolume mysubvolume.broken
>>> # btrfs subvolume snapshot mysubvolume.broken mysubvolume
>>>
>>> You can do the same with each of your snapshots too, and send them as
>>> full snapshots (without -p).
>>>
>>> ~A
>>>
>>>> Best Regards,
>>>> Siranee Jaraswachirakul.
>>>>
>>>>> Have you checked that there is no Received UUID on the source subvolume?
>>>>>
>>>>> # btrfs sub show volume/mysql/
>>>>> volume/mysql
>>>>>   ??????? Name:?????????????????? mysql
>>>>>   ??????? UUID:?????????????????? 8a94524e-a956-c14b-bb8d-d453627f27d5
>>>>>   ??????? Parent UUID:??????????? -
>>>>>   ??????? Received UUID:????????? -
>>>>>   ??????? Creation time:????????? 2017-04-17 11:46:20 +0200
>>>>>   ??????? Subvolume ID:?????????? 1469
>>>>>   ??????? Generation:???????????? 122934
>>>>>   ??????? Gen at creation:??????? 78671
>>>>>   ??????? Parent ID:????????????? 5
>>>>>   ??????? Top level ID:?????????? 5
>>>>>   ??????? Flags:????????????????? -
>>>>>   ??????? Snapshot(s):
>>>>>
>>>>> There is no Received UUID here. If it has, then btrfs send-receive will
>>>>> have problems, since all snapshots of the source subvolume will have the
>>>>> same Received UUID and it can't tell the differences between the snapshots.
>>>>>
>>>>> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote:
>>>>>> Hi Chris,
>>>>>>
>>>>>> I started as your suggestion again. The diff occured since snapshot
>>>>>> mysql_201708090830 manually send. What should I do next?
>>>>>>
>>>>>> - delete all the bad/mismatching snapshots only on the destination computer.
>>>>>> [root@joytest ~]# date
>>>>>> Sun Aug 13 10:27:23 ICT 2017
>>>>>> [root@joytest ~]# cd /var/lib/mariadb
>>>>>> [root@joytest mariadb]# btrfs sub list .
>>>>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>>>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>>>>> ID 318 gen 205 top level 5 path mysql_201708090830
>>>>>> ID 320 gen 211 top level 5 path mysql_201708100830
>>>>>> ID 322 gen 219 top level 5 path mysql_201708110830
>>>>>> ID 323 gen 219 top level 5 path mysql_201708120830
>>>>>> ID 324 gen 224 top level 5 path mysql_201708130830
>>>>>> ID 325 gen 225 top level 5 path mysql
>>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708130830
>>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
>>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708120830
>>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
>>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708110830
>>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
>>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708100830
>>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
>>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708090830
>>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
>>>>>> [root@joytest mariadb]# btrfs sub sync .
>>>>>> [root@joytest mariadb]# systemctl status mariadb
>>>>>> &#226;? mariadb.service - MariaDB database server
>>>>>>      Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor
>>>>>> preset:
>>>>>> disabled)
>>>>>>      Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h
>>>>>> 24min
>>>>>> ago
>>>>>>     Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
>>>>>> (code=exited, status=1/FAILURE)
>>>>>>     Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
>>>>>> status=0/SUCCESS)
>>>>>>     Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n
>>>>>> (code=exited,
>>>>>> status=0/SUCCESS)
>>>>>>    Main PID: 19870 (code=exited, status=0/SUCCESS)
>>>>>>
>>>>>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
>>>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe
>>>>>> Logging
>>>>>> to
>>>>>> '/var/log/mariadb/mariadb.log'.
>>>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe
>>>>>> Starting
>>>>>> mysqld daemon with databases from /var/lib/mariadb/mysql
>>>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
>>>>>> code=exited status=1
>>>>>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
>>>>>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
>>>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
>>>>>> [root@joytest mariadb]# btrfs sub list .
>>>>>> ID 313 gen 220 top level 5 path mysql_201708070830
>>>>>> ID 316 gen 199 top level 5 path mysql_201708080830
>>>>>> ID 325 gen 225 top level 5 path mysql
>>>>>> [root@joytest mariadb]#
>>>>>>
>>>>>> - The most recent good snapshot pair, which rsync shows origin and
>>>>>> destination match, is mysql_201708080830 so you can keep that one on
>>>>>> both sides.
>>>>>>
>>>>>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
>>>>>> ID 257 gen 538 top level 5 path mysql
>>>>>> ID 316 gen 498 top level 5 path mysql_201708060830
>>>>>> ID 317 gen 503 top level 5 path mysql_201708070830
>>>>>> ID 318 gen 507 top level 5 path mysql_201708080830
>>>>>> ID 319 gen 514 top level 5 path mysql_201708090830
>>>>>> ID 320 gen 524 top level 5 path mysql_201708100830
>>>>>> ID 321 gen 529 top level 5 path mysql_201708110830
>>>>>> ID 322 gen 533 top level 5 path mysql_201708120830
>>>>>> ID 323 gen 538 top level 5 path mysql_201708130830
>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
>>>>>> sending incremental file list
>>>>>> ./
>>>>>>
>>>>>> sent 3773 bytes  received 19 bytes  842.67 bytes/sec
>>>>>> total size is 718361496  speedup is 189441.32 (DRY RUN)
>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
>>>>>> sending incremental file list
>>>>>> ./
>>>>>>
>>>>>> sent 3769 bytes  received 19 bytes  841.78 bytes/sec
>>>>>> total size is 718361496  speedup is 189641.37 (DRY RUN)
>>>>>> [root@backuplogC7 ~]# date
>>>>>> Sun Aug 13 10:34:05 ICT 2017
>>>>>> [root@backuplogC7 ~]#
>>>>>>
>>>>>> - manually do incremental send/receive, starting with
>>>>>> mysql_201708090830/, to make the destination current again with the
>>>>>> origin.
>>>>>>
>>>>>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
>>>>>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
>>>>>> /var/lib/mariadb
>>>>>> At subvol /var/lib/mariadb/mysql_201708090830
>>>>>> At snapshot mysql_201708090830
>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
>>>>>> sending incremental file list
>>>>>> ./
>>>>>> ib_logfile1
>>>>>> ibdata1
>>>>>>
>>>>>> sent 3779 bytes  received 25 bytes  507.20 bytes/sec
>>>>>> total size is 718361496  speedup is 188843.72 (DRY RUN)
>>>>>> [root@backuplogC7 ~]#
>>>>>>
>>>>>> Best Regards,
>>>>>>
>>>>>> Siranee Jaraswachirakul.
>>>>>>
>>>>>>> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>>>>>>>
>>>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>>>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>>>>>>> You need trailing / for the first directory with -a option.
>>>>>>>
>>>>>>> rsync -a dir dir
>>>>>>>
>>>>>>> is not the same command as
>>>>>>>
>>>>>>> rsync -a dir/ dir
>>>>>>>
>>>>>>> It's confusing but your command is trying to create mysql_201708090830
>>>>>>> directory on the source, in the mysql_201708090830 on the destination.
>>>>>>> That is why everything mismatches. To make it mean "contents of" you
>>>>>>> need trailing slash on at least the origin.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Chris Murphy
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>
>>>
>>>
>>
>>
>
>
>

Re: btrfs issue with mariadb incremental backup

[siranee.ja]

Hi Chris,

Thank you very much for your suggestion.

I didn't remember which steps that I made a mistake and made the mysql had Received
UUID.

I have done the following and it work as it should be right now.

[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708070830
rw_201708070830
Create a snapshot of 'mbroken_201708070830' in './rw_201708070830'
[root@backuplogC7 mariadb]# btrfs sub list .
ID 257 gen 542 top level 5 path mbroken
ID 317 gen 576 top level 5 path mbroken_201708070830
ID 318 gen 568 top level 5 path mbroken_201708080830
ID 319 gen 569 top level 5 path mbroken_201708090830
ID 320 gen 570 top level 5 path mbroken_201708100830
ID 321 gen 571 top level 5 path mbroken_201708110830
ID 322 gen 572 top level 5 path mbroken_201708120830
ID 323 gen 573 top level 5 path mbroken_201708130830
ID 324 gen 543 top level 5 path mysql
ID 348 gen 576 top level 5 path rw_201708070830
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708080830
rw_201708080830
Create a snapshot of 'mbroken_201708080830' in './rw_201708080830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708090830
rw_201708090830
Create a snapshot of 'mbroken_201708090830' in './rw_201708090830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708100830
rw_201708100830
Create a snapshot of 'mbroken_201708100830' in './rw_201708100830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708110830
rw_201708110830
Create a snapshot of 'mbroken_201708110830' in './rw_201708110830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708120830
rw_201708120830
Create a snapshot of 'mbroken_201708120830' in './rw_201708120830'
[root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708130830
rw_201708130830
Create a snapshot of 'mbroken_201708130830' in './rw_201708130830'
[root@backuplogC7 mariadb]# btrfs sub list .
ID 257 gen 542 top level 5 path mbroken
ID 317 gen 576 top level 5 path mbroken_201708070830
ID 318 gen 577 top level 5 path mbroken_201708080830
ID 319 gen 578 top level 5 path mbroken_201708090830
ID 320 gen 579 top level 5 path mbroken_201708100830
ID 321 gen 580 top level 5 path mbroken_201708110830
ID 322 gen 581 top level 5 path mbroken_201708120830
ID 323 gen 582 top level 5 path mbroken_201708130830
ID 324 gen 543 top level 5 path mysql
ID 348 gen 576 top level 5 path rw_201708070830
ID 349 gen 577 top level 5 path rw_201708080830
ID 350 gen 578 top level 5 path rw_201708090830
ID 351 gen 579 top level 5 path rw_201708100830
ID 352 gen 580 top level 5 path rw_201708110830
ID 353 gen 581 top level 5 path rw_201708120830
ID 354 gen 582 top level 5 path rw_201708130830
[root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep
"3ad0334a-4063-654c-add6-b1cbcdeaa639"
ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken
ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708070830
ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708080830
ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708090830
ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708100830
ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708110830
ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708120830
ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708130830
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708070830 mysql_201708070830
Create a readonly snapshot of 'rw_201708070830' in './mysql_201708070830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708080830 mysql_201708080830
Create a readonly snapshot of 'rw_201708080830' in './mysql_201708080830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708090830 mysql_201708090830
Create a readonly snapshot of 'rw_201708090830' in './mysql_201708090830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708100830 mysql_201708100830
Create a readonly snapshot of 'rw_201708100830' in './mysql_201708100830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708110830 mysql_201708110830
Create a readonly snapshot of 'rw_201708110830' in './mysql_201708110830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708120830 mysql_201708120830
Create a readonly snapshot of 'rw_201708120830' in './mysql_201708120830'
[root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708130830 mysql_201708130830
Create a readonly snapshot of 'rw_201708130830' in './mysql_201708130830'
[root@backuplogC7 mariadb]# btrfs sub list .
ID 257 gen 542 top level 5 path mbroken
ID 317 gen 576 top level 5 path mbroken_201708070830
ID 318 gen 577 top level 5 path mbroken_201708080830
ID 319 gen 578 top level 5 path mbroken_201708090830
ID 320 gen 579 top level 5 path mbroken_201708100830
ID 321 gen 580 top level 5 path mbroken_201708110830
ID 322 gen 581 top level 5 path mbroken_201708120830
ID 323 gen 582 top level 5 path mbroken_201708130830
ID 324 gen 584 top level 5 path mysql
ID 348 gen 583 top level 5 path rw_201708070830
ID 349 gen 584 top level 5 path rw_201708080830
ID 350 gen 585 top level 5 path rw_201708090830
ID 351 gen 586 top level 5 path rw_201708100830
ID 352 gen 587 top level 5 path rw_201708110830
ID 353 gen 588 top level 5 path rw_201708120830
ID 354 gen 589 top level 5 path rw_201708130830
ID 355 gen 583 top level 5 path mysql_201708070830
ID 356 gen 584 top level 5 path mysql_201708080830
ID 357 gen 585 top level 5 path mysql_201708090830
ID 358 gen 586 top level 5 path mysql_201708100830
ID 359 gen 587 top level 5 path mysql_201708110830
ID 360 gen 588 top level 5 path mysql_201708120830
ID 361 gen 589 top level 5 path mysql_201708130830

[root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep
"3ad0334a-4063-654c-add6-b1cbcdeaa639"
ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken
ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708070830
ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708080830
ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708090830
ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708100830
ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708110830
ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708120830
ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path
mbroken_201708130830
[root@backuplogC7 mariadb]# btrfs send /var/lib/mariadb/mysql_201708070830 | ssh
192.168.45.166 btrfs receive /var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708070830
At subvol mysql_201708070830
[root@backuplogC7 mariadb]# btrfs sub show mysql_201708070830
/var/lib/mariadb/mysql_201708070830
        Name:                   mysql_201708070830
        UUID:                   70ee3c31-126d-574a-814c-e3b4c81b414e
        Parent UUID:            1d5bb8eb-b0df-2549-8b62-552cfa517609
        Received UUID:          -
        Creation time:          2017-08-14 07:00:08 +0700
        Subvolume ID:           355
        Generation:             583
        Gen at creation:        583
        Parent ID:              5
        Top level ID:           5
        Flags:                  readonly
        Snapshot(s):
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708070830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  1083.43 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708070830
/var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708080830
At snapshot mysql_201708080830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708080830/
sending incremental file list
./

sent 3769 bytes  received 19 bytes  688.73 bytes/sec
total size is 718361496  speedup is 189641.37 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708080830
/var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708090830
At snapshot mysql_201708090830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  583.38 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708090830
/var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708100830
At snapshot mysql_201708100830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708100830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708100830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708100830
/var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708110830
At snapshot mysql_201708110830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708110830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708110830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708110830
/var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708120830
At snapshot mysql_201708120830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708120830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708120830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708120830
/var/lib/mariadb/mysql_201708130830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708130830
At snapshot mysql_201708130830
[root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708130830/
root@192.168.45.166:/var/lib/mariadb/mysql_201708130830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  689.45 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 mariadb]#


Best Regards,
Siranee Jaraswachirakul.






> On Sun, Aug 13, 2017 at 4:52 AM,  <siranee.ja@tpc.co.th> wrote:
>> Hi "A L",
>>
>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql
>> /var/lib/mariadb/mysql
>>         Name:                   mysql
>>         UUID:                   92f319c5-e132-3249-9b13-d39ee77a2b44
>>         Parent UUID:            -
>>         Received UUID:          3ad0334a-4063-654c-add6-b1cbcdeaa639
>>         Creation time:          2017-06-21 13:27:41 +0700
>>         Subvolume ID:           257
>>         Generation:             539
>>         Gen at creation:        9
>>         Parent ID:              5
>>         Top level ID:           5
>>         Flags:                  -
>>         Snapshot(s):
>>                                 mysql_201708060830
>>                                 mysql_201708070830
>>                                 mysql_201708080830
>>                                 mysql_201708090830
>>                                 mysql_201708100830
>>                                 mysql_201708110830
>>                                 mysql_201708120830
>>                                 mysql_201708130830
>>
>> yes I think it has Received UUID because I restored the source from snapshot
>> mysql_201708040830 for prove that the local snapshot was work.
>>
>> How to clear the Received UUID ?
>> What to do next?
>
>
> I'm using btrfs-progs 4.12, and I just did a btrfs send/receive from
> file system A to B; and then on B I took a rw snapshot of the ro
> snapshot. The ro snapshot has Received UUID, but the rw snapshot made
> from it does not have Received UUID. So now I'm confused how you have
> a rw snapshot with Received UUID set.
>
>
> Did you ever use 'btrfs property set' to change a ro snapshot to rw?
> That's the only way I can think it's possible. The only other thing is
> maybe the behavior has changed since btrfs-progs 4.4.
>
> It is possible this behavior has changed since btrfs-progs 4.4. The
> changelogs shows there are improvements in send/receive in particular
> 4.8.3 and 4.8.4, but also others. I have no idea which are related.
> But anyway, this is one of the reasons why the expert users on this
> almost always say use something newer because it's just too hard to
> remember, even with changelogs, what's fixed and where.
>
> My suggestion is to investigate moving to kernel 4.9 series; and it
> should be safe to move to btrfs-progs 4.12 now. Any new features in
> 4.12 that can't be supported with an older kernel should fail
> gracefully.
>
> No matter what, you must have separate backups. You can't depend
> exclusively on one backup, no matter the method. It's fine to use
> send/receive for backup1. But backup2 should be a conventional backup,
> e.g. XFS with rsync. Basically this is a game of risk and the more
> independent backups you have, the more failure cases (especially user
> error) you can recover from, at the expense of some complexity.
>
>
>
> --
> Chris Murphy
>