Re: enospc errors during balance — how to prevent out of space

[Leszek Dubiel <leszek@dubiel.pl>]

My system was making backups for about one week.

It was doing automatic "btrfs balance".



Yesterday it went through:

btrfs balance -dusage=0
btrfs balance -dusage=10
btrfs balance -dusage=20
btrfs balance -dusage=30
...
btrfs balance -dusage=100
...
btrfs balance -musage=0
btrfs balance -dusage=10
btrfs balance -dusage=20
...


Something went wrong when balancing musage (m, as metadata).
System got "read only".



While this happened btrfs was in a process of deleting four snapshots 
(btrfs sub list / -d  — not empty).




It had 450 GB of free space (shown for df -h).

It had almost no Unallocated space (btrfs dev usa /).



After reboot system is mounted read-only.
Kernel shows (Ctrl+D or give root password for maintenance).


Tried to run    btrfs balance -dusage /      on read only system failed.

Tried to mount -oremount,rw    hanged.


Reboot.




Started from USB key Finnix to repair.


Started to mount system.


Dmesg shows:

              bdev /dev/sdc3 errs: wr 0, rd 0, flush 0, corrupt 35967, gen 0


It mounts for a long time now.
Nothing more in dmesg.
Mount command seems stalled, but on iotop I see "btrfs-transaction" 
running — write about 10 M/s



I will leave the system over night and check tommorow or on monday if 
mount was successful.




PS. script that was balancing:


         findmnt --types btrfs --output SOURCE --nofsroot --noheadings | 
sort | uniq |
         while read dev; do
                 mnt=$(findmnt --source "$dev" --output TARGET 
--first-only --noheadings)
                 test -d "$mnt" || continue

                 # no balance if plenty of unallocated space
                 btrfs dev usage "$mnt" -g |
                 perl -ne '/Unallocated: +([0-9]+\.[0-9]{2})GiB/ and $1 
< 21 and print $1' |
                 grep -q . || continue

                 for typ in dusage musage; do
                         for usa in $(seq 0 10 100); do
                                 # if relocated, then get out of two 
loops for next "$dev"
                                 btrfs balance start -$typ=$usa,limit=3 
"$mnt" 2>&1 |
                                 grep -Eq "Done, had to relocate 
[1-9][0-9]* out of [0-9]+ chunks" &&
                                 break 2
                         done
                 done
         done