how to run balance successfully (No space left on device)?

how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

I'm trying to run balance on a 4.13.2 kernel without much luck:

# time btrfs balance start -v /var/lib/lxd -dusage=5 -musage=5
Dumping filters: flags 0x7, state 0x0, force is off
   DATA (flags 0x2): balancing, usage=5
   METADATA (flags 0x2): balancing, usage=5
   SYSTEM (flags 0x2): balancing, usage=5
Done, had to relocate 1 out of 353 chunks

real    0m2.356s
user    0m0.005s
sys     0m0.175s


# time btrfs balance start -v /var/lib/lxd -dusage=0 -musage=0
Dumping filters: flags 0x7, state 0x0, force is off
   DATA (flags 0x2): balancing, usage=0
   METADATA (flags 0x2): balancing, usage=0
   SYSTEM (flags 0x2): balancing, usage=0
Done, had to relocate 0 out of 353 chunks

real    0m0.076s
user    0m0.004s
sys     0m0.008s


# time btrfs balance start -v /var/lib/lxd
Dumping filters: flags 0x7, state 0x0, force is off
   DATA (flags 0x0): balancing
   METADATA (flags 0x0): balancing
   SYSTEM (flags 0x0): balancing
WARNING:

         Full balance without filters requested. This operation is very
         intense and takes potentially very long. It is recommended to
         use the balance filters to narrow down the balanced data.
         Use 'btrfs balance start --full-balance' option to skip this
         warning. The operation will start in 10 seconds.
         Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/var/lib/lxd': No space left on device
There may be more info in syslog - try dmesg | tail

real    284m58.541s
user    0m0.000s
sys     47m39.037s




# df -h /var/lib/lxd
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       424G  318G  105G  76% /var/lib/lxd


# btrfs fi df /var/lib/lxd
Data, RAID1: total=318.00GiB, used=313.82GiB
System, RAID1: total=32.00MiB, used=80.00KiB
Metadata, RAID1: total=5.00GiB, used=3.17GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# btrfs fi show /var/lib/lxd
Label: 'btrfs'  uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
         Total devices 2 FS bytes used 316.98GiB
         devid    1 size 423.13GiB used 323.03GiB path /dev/sda3
         devid    2 size 423.13GiB used 323.03GiB path /dev/sdb3


# btrfs fi usage /var/lib/lxd
Overall:
     Device size:                 846.25GiB
     Device allocated:            646.06GiB
     Device unallocated:          200.19GiB
     Device missing:                  0.00B
     Used:                        633.97GiB
     Free (estimated):            104.28GiB      (min: 104.28GiB)
     Data ratio:                       2.00
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID1: Size:318.00GiB, Used:313.82GiB
    /dev/sda3     318.00GiB
    /dev/sdb3     318.00GiB

Metadata,RAID1: Size:5.00GiB, Used:3.17GiB
    /dev/sda3       5.00GiB
    /dev/sdb3       5.00GiB

System,RAID1: Size:32.00MiB, Used:80.00KiB
    /dev/sda3      32.00MiB
    /dev/sdb3      32.00MiB

Unallocated:
    /dev/sda3     100.10GiB
    /dev/sdb3     100.10GiB


Mount flags in /etc/fstab are:

LABEL=btrfs /var/lib/lxd btrfs 
defaults,noatime,space_cache=v2,device=/dev/sda3,device=/dev/sdb3,discard 
0 0



Last pieces logged in dmesg:

[46867.225334] BTRFS info (device sda3): relocating block group 
2996254998528 flags data|raid1
[46874.563631] BTRFS info (device sda3): found 9250 extents
[46894.827895] BTRFS info (device sda3): found 9250 extents
[46898.463053] BTRFS info (device sda3): found 201 extents
[46898.562564] BTRFS info (device sda3): relocating block group 
2995181256704 flags data|raid1
[46903.555976] BTRFS info (device sda3): found 7299 extents
[46914.188044] BTRFS info (device sda3): found 7299 extents
[46914.303476] BTRFS info (device sda3): relocating block group 
2947936616448 flags metadata|raid1
[46939.570810] BTRFS info (device sda3): found 42022 extents
[46945.053488] BTRFS info (device sda3): 2 enospc errors during balance



Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Duncan]

Tomasz Chmielewski posted on Mon, 18 Sep 2017 00:02:46 +0900 as excerpted:

> I'm trying to run balance on a 4.13.2 kernel without much luck:
> 
> # time btrfs balance start -v /var/lib/lxd -dusage=5 -musage=5
> [works, but only 1 chunk balanced]

> # time btrfs balance start -v /var/lib/lxd -dusage=0 -musage=0
> [no chunks with 0 usage to balance]
> 
> 
> # time btrfs balance start -v /var/lib/lxd
> [...]
> ERROR: error during balancing '/var/lib/lxd': No space left on device

OK, that fails.  Let's see what your unallocated space looks like, 
below...

> # df -h /var/lib/lxd

FWIW, standard (aka util-linux) df is effectively useless in a situation 
such as this, as it really doesn't give you the information you need (it 
can say you have lots of space available, but if btrfs has all of it 
allocated into chunks, even if the chunks have space in them still, there 
can be problems).

And actually, (util-linux) df really doesn't give you a whole lot of 
useful information on a btrfs in enough cases that most list regulars 
tend to discount its output almost entirely.  The only thing it's really 
useful for is getting a reasonable idea as to whether your next major 
file operation can be expected to succeed or not -- if it says you have 
50 MB left and you're trying to put a new 1 GiB file on the btrfs, it's 
unlikely to work, but if it says you have 300 GiB left in a multi-TB 
multi-device filesystem, you might have 300, or 3000 (its estimates are 
deliberately on the pessimistic side).

For better numbers, always use the btrfs tools, btrfs fi usage is the one 
I tend to use most, but btrfs dev usage can be very useful if you're more 
interested in a per-device listing, and btrfs fi show combined with btrfs 
fi df provide much the same information, tho it needs a bit more 
interpreting.

But you do provide them too. =:^)

> # btrfs fi df /var/lib/lxd
> Data, RAID1: total=318.00GiB, used=313.82GiB
> System, RAID1: total=32.00MiB, used=80.00KiB
> Metadata, RAID1: total=5.00GiB, used=3.17GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B

Looks reasonably healthy.  No global reserve used, good as that's a major 
indicator of problems, and data and metadata usage is reasonably close to 
totals -- no huge number of mostly empty allocated chunks.

> # btrfs fi show /var/lib/lxd Label: 'btrfs'  uuid:
> f5f30428-ec5b-4497-82de-6e20065e6f61
>          Total devices 2 FS bytes used 316.98GiB
>          devid    1 size 423.13GiB used 323.03GiB path /dev/sda3
>          devid    2 size 423.13GiB used 323.03GiB path /dev/sdb3

OK, given the ENOSPC error on balance above, those device lines are the 
real interesting numbers, and...

Healthy here too.  Very much so, in fact, as only 323 gigs out of 423 is 
allocated on each device -- 100 gigs not chunk-allocated and therefore 
free for chunk allocation on each device. =:^)

The ENOSPC is therefore a bug -- it shouldn't be happening.

And as it happens, AFAIK from reading the list, there's a currently known 
bug with over-reservation under certain circumstances that among other 
things, can (wrongly) trigger ENOSPC on balances, when there's plenty of 
space.

Also AFAIK, there's a patch on-list and (I think) in 4.14-rc1, that is I 
believe marked for stable as well, that will very likely fix your 
problem.  If it doesn't, there's another bug triggering similar symptoms.

But I'm not a dev and haven't been tracking the specific patch, so you'll 
need to either track it down (or wait to see if a dev or someone else 
points you at it) and apply it on your 4.13.x, or wait until it hits 
stable backports and you can get it there, or try 4.14-rc1 or wait until 
later/safer rcs or full release.

Meanwhile...

> # btrfs fi usage /var/lib/lxd Overall:
>      Device size:                 846.25GiB
>      Device allocated:            646.06GiB
>      Device unallocated:          200.19GiB
>      Device missing:                  0.00B
>      Used:                        633.97GiB
>      Free (estimated):            104.28GiB      (min: 104.28GiB)
>      Data ratio:                       2.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
> 
> Data,RAID1: Size:318.00GiB, Used:313.82GiB
>     /dev/sda3     318.00GiB
>     /dev/sdb3     318.00GiB
> 
> Metadata,RAID1: Size:5.00GiB, Used:3.17GiB
>     /dev/sda3       5.00GiB
>     /dev/sdb3       5.00GiB
> 
> System,RAID1: Size:32.00MiB, Used:80.00KiB
>     /dev/sda3      32.00MiB
>     /dev/sdb3      32.00MiB
> 
> Unallocated:
>     /dev/sda3     100.10GiB
>     /dev/sdb3     100.10GiB

As I said above, btrfs fi usage output provides much of the same info, 
but in a much nicer format and with a bit more detail, than the 
combination of btrfs fi show and btrfs fi df.

This confirms the above 100 gigs per device unallocated, plenty for a 
balance if it's not bugging out, and data and metadata chunk usage in the 
same ball park as the totals, so as I said above, the ENOSPC during 
balance is very definitely a bug.  Everything looks healthy, which means 
an ENOSPC during balance /must/ be a bug, because it simply shouldn't be 
happening.

But chances are pretty good that one you get that patch integrated, 
whether by integrating it yourself to what you have currently, or by 
trying 4.14-rc1 or waiting until it hits release or stable, that bug will 
have been squashed! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

>> # df -h /var/lib/lxd
>> 
>> FWIW, standard (aka util-linux) df is effectively useless in a 
>> situation
>> such as this, as it really doesn't give you the information you need 
>> (it
>> can say you have lots of space available, but if btrfs has all of it
>> allocated into chunks, even if the chunks have space in them still, 
>> there
>> can be problems).

I see here on RAID-1, "df -h" it shows pretty much the same amount of 
free space as "btrfs fi show":

- "df -h" shows 105G free
- "btrfs fi show" says: Free (estimated):            104.28GiB      
(min: 104.28GiB)



> But chances are pretty good that one you get that patch integrated,
> whether by integrating it yourself to what you have currently, or by
> trying 4.14-rc1 or waiting until it hits release or stable, that bug 
> will
> have been squashed! =:^)

OK, will wait for 4.14.


Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Andrei Borzenkov]

On Mon, Sep 18, 2017 at 11:20 AM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>>> # df -h /var/lib/lxd
>>>
>>> FWIW, standard (aka util-linux) df is effectively useless in a situation
>>> such as this, as it really doesn't give you the information you need (it
>>> can say you have lots of space available, but if btrfs has all of it
>>> allocated into chunks, even if the chunks have space in them still, there
>>> can be problems).
>
>
> I see here on RAID-1, "df -h" it shows pretty much the same amount of free
> space as "btrfs fi show":
>
> - "df -h" shows 105G free
> - "btrfs fi show" says: Free (estimated):            104.28GiB      (min:
> 104.28GiB)
>

I think both use the same algorithm to compute free space (df at the
end just shows what kernel returns). The problem is that this
algorithm itself is just approximation in general case. For uniform
RAID1 profile it should be correct though.

Re: how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

On 2017-09-18 17:29, Andrei Borzenkov wrote:
> On Mon, Sep 18, 2017 at 11:20 AM, Tomasz Chmielewski <mangoo@wpkg.org> 
> wrote:
>>>> # df -h /var/lib/lxd
>>>> 
>>>> FWIW, standard (aka util-linux) df is effectively useless in a 
>>>> situation
>>>> such as this, as it really doesn't give you the information you need 
>>>> (it
>>>> can say you have lots of space available, but if btrfs has all of it
>>>> allocated into chunks, even if the chunks have space in them still, 
>>>> there
>>>> can be problems).
>> 
>> 
>> I see here on RAID-1, "df -h" it shows pretty much the same amount of 
>> free
>> space as "btrfs fi show":
>> 
>> - "df -h" shows 105G free
>> - "btrfs fi show" says: Free (estimated):            104.28GiB      
>> (min:
>> 104.28GiB)
>> 
> 
> I think both use the same algorithm to compute free space (df at the
> end just shows what kernel returns). The problem is that this
> algorithm itself is just approximation in general case. For uniform
> RAID1 profile it should be correct though.

And perhaps more important - can I assume that right now, with the 
latest stable kernel (4.13.2 right now), running "btrfs balance" is not 
safe and can lead to data corruption or loss?


Consider the following case:

- system admin runs btrfs balance on a filesystem with 100 GB free and 
assumes it is enough space to complete successfully

- btrfs balance fails due to some bug with "No space left on device"

- at the same time, a database using this filesystem will fail with "No 
space left on device", apt/rpm will fail a package upgrade, some program 
using temp space will fail, log collector will fail to catch some data, 
because of "No space left on device" and so on?



Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Peter Becker]

i'm not sure if it would help, but maybe you could try adding an 8GB
(or more) USB flash drive to the pool and try to start balance.
if it works out, you can throw him out of the pool after that.

Re: how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

On 2017-09-18 22:44, Peter Becker wrote:
> i'm not sure if it would help, but maybe you could try adding an 8GB
> (or more) USB flash drive to the pool and try to start balance.
> if it works out, you can throw him out of the pool after that.

I really can't, it's an "online server".

But I've removed some 65 GB data, so now it's 171 GB free, or, 60% used 
filesystem.

The balance still fails.


Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Duncan]

Tomasz Chmielewski posted on Mon, 18 Sep 2017 18:27:09 +0900 as excerpted:

> And perhaps more important - can I assume that right now, with the
> latest stable kernel (4.13.2 right now), running "btrfs balance" is not
> safe and can lead to data corruption or loss?
> 
> 
> Consider the following case:
> 
> - system admin runs btrfs balance on a filesystem with 100 GB free and
> assumes it is enough space to complete successfully
> 
> - btrfs balance fails due to some bug with "No space left on device"
> 
> - at the same time, a database using this filesystem will fail with "No
> space left on device", apt/rpm will fail a package upgrade, some program
> using temp space will fail, log collector will fail to catch some data,
> because of "No space left on device" and so on?

To the best of my knowledge that shouldn't be a problem, certainly not 
one I'd worry about if you're following the sysadmin's first rule of 
backups, the true value of data to you is defined not by any claims but 
by the number of backups you consider it worth having of that data, so it 
follows that no backups means you've defined the data as worth less than 
the time/trouble/resources it would take to create at least that one 
backup.

The ENOSPC is because the internal calculation for the reserved-space 
requirement is buggy ATM, but AFAIK it's just that, an /internal/ 
calculation, that goes waayyy wild, and stops any action it's going to 
stop before it goes anywhere -- it doesn't get to the point of affecting 
anything else because the reserve space calculation goes wild and stops 
it before it can actually reserve the space.

Talking about which... I've not seen it mentioned in the bug discussion, 
but I wonder if doing a btrfs balance start -d, followed by a another 
balance with -m replacing the -d, thus separating the data and metadata 
balances, might work around the problem.  At least you could know for 
sure which is causing it that way, and complete a balance of the other 
one.  And if that blocks on one or the other, you could split the job up 
further using the devid= and drange= filters (see the btrfs-balance 
manpage), doing only part of the filesystem at a time.  My speculation is 
that you should be able to divide the operation up enough so that even if 
the reserve space calculation is off, it'll still complete.

Meanwhile, I don't believe it's just balance that's affected, either, tho 
it's the most commonly reported.  By my understanding, any sufficiently 
large operation could trigger it, tho obviously a full btrfs balance is 
about the largest operation a btrfs is likely to have, so it stands to 
reason that would trigger it more reliably than common generic filesystem 
operations.

Of course if you're paranoid, you can refrain from doing balances until 
you know the bug is fixed, but then I'd have to ask, if you're that 
paranoid of a filesystem failure, why are you running the still 
stabilizing, not yet entirely stable and mature, btrfs, in the first 
place?  Seems a bit like the folks still running RHEL/CentOS 6 with their 
stable kernels because they want stability, yet choosing to run the still 
not entirely stable btrfs, definitely not entirely stable on that old a 
kernel, on top of them.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

On 2017-09-18 17:20, Tomasz Chmielewski wrote:
>>> # df -h /var/lib/lxd
>>> 
>>> FWIW, standard (aka util-linux) df is effectively useless in a 
>>> situation
>>> such as this, as it really doesn't give you the information you need 
>>> (it
>>> can say you have lots of space available, but if btrfs has all of it
>>> allocated into chunks, even if the chunks have space in them still, 
>>> there
>>> can be problems).
> 
> I see here on RAID-1, "df -h" it shows pretty much the same amount of
> free space as "btrfs fi show":
> 
> - "df -h" shows 105G free
> - "btrfs fi show" says: Free (estimated):            104.28GiB
> (min: 104.28GiB)
> 
> 
> 
>> But chances are pretty good that one you get that patch integrated,
>> whether by integrating it yourself to what you have currently, or by
>> trying 4.14-rc1 or waiting until it hits release or stable, that bug 
>> will
>> have been squashed! =:^)
> 
> OK, will wait for 4.14.

So I've tried to run balance with 4.14-rc6.

It succeeded on one server where it was failing with 4.13.x.


On a different server, however, it failed badly:

# time btrfs balance start /srv
WARNING:

         Full balance without filters requested. This operation is very
         intense and takes potentially very long. It is recommended to
         use the balance filters to narrow down the scope of balance.
         Use 'btrfs balance start --full-balance' option to skip this
         warning. The operation will start in 10 seconds.
         Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/srv': Read-only file system
There may be more info in syslog - try dmesg | tail

real    5194m41.749s
user    0m0.000s
sys     301m10.928s


[312304.050731] BTRFS info (device sda4): found 15073 extents
[313555.971253] BTRFS info (device sda4): relocating block group 
1208022466560 flags data|raid1
[314963.506580] BTRFS: Transaction aborted (error -28)
[314963.506608] ------------[ cut here ]------------
[314963.506639] WARNING: CPU: 2 PID: 27854 at 
/home/kernel/COD/linux/fs/btrfs/extent-tree.c:3089 
btrfs_run_delayed_refs+0x244/0x250 [btrfs]
[314963.506640] Modules linked in: vhost_net vhost tap xt_REDIRECT 
nf_nat_redirect xt_NFLOG nfnetlink_log nfnetlink xt_conntrack veth 
ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 
nf_nat_ipv6 ip6table_filter ip6_tables xt_comment xt_CHECKSUM 
binfmt_misc iptable_mangle nf_log_ipv4 nf_log_common xt_LOG 
ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 
xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc btrfs 
zstd_compress shpchp intel_rapl lpc_ich x86_pkg_temp_thermal 
intel_powerclamp input_leds tpm_infineon ie31200_edac serio_raw coretemp 
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel kvm_intel pcbc kvm 
aesni_intel irqbypass aes_x86_64 mac_hid crypto_simd glue_helper cryptd 
intel_cstate
[314963.506684]  eeepc_wmi asus_wmi sparse_keymap intel_rapl_perf 
wmi_bmof nfsd auth_rpcgss nfs_acl lockd grace sunrpc lp parport autofs4 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor 
async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 e1000e ahci 
libahci ptp pps_core wmi video
[314963.506710] CPU: 2 PID: 27854 Comm: sadc Tainted: G        W       
4.14.0-041400rc6-generic #201710230731
[314963.506711] Hardware name: System manufacturer System Product 
Name/P8B WS, BIOS 0904 10/24/2011
[314963.506713] task: ffff8bc0fd39ae00 task.stack: ffffb28d49490000
[314963.506732] RIP: 0010:btrfs_run_delayed_refs+0x244/0x250 [btrfs]
[314963.506734] RSP: 0018:ffffb28d49493d30 EFLAGS: 00010286
[314963.506736] RAX: 0000000000000026 RBX: 00000000ffffffe4 RCX: 
0000000000000000
[314963.506737] RDX: 0000000000000000 RSI: ffff8bc8afa8dc98 RDI: 
ffff8bc8afa8dc98
[314963.506738] RBP: ffffb28d49493d88 R08: 0000000000000001 R09: 
000000000000242b
[314963.506740] R10: ffffb28d49493c20 R11: 0000000000000000 R12: 
ffff8bc883a81078
[314963.506741] R13: ffff8bc887eb0000 R14: ffff8bc1876ec400 R15: 
000000000018ba90
[314963.506743] FS:  00007f62a12d9700(0000) GS:ffff8bc8afa80000(0000) 
knlGS:0000000000000000
[314963.506744] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[314963.506746] CR2: 00007f25f6f53880 CR3: 00000003cf4f7004 CR4: 
00000000000626e0
[314963.506747] Call Trace:
[314963.506773]  btrfs_commit_transaction+0x9b/0x8d0 [btrfs]
[314963.506799]  ? btrfs_wait_ordered_range+0x9c/0x110 [btrfs]
[314963.506821]  btrfs_sync_file+0x348/0x410 [btrfs]
[314963.506826]  vfs_fsync_range+0x4b/0xb0
[314963.506828]  do_fsync+0x3d/0x70
[314963.506831]  SyS_fdatasync+0x13/0x20
[314963.506834]  do_syscall_64+0x61/0x120
[314963.506838]  entry_SYSCALL64_slow_path+0x25/0x25
[314963.506840] RIP: 0033:0x7f62a0dfec30
[314963.506841] RSP: 002b:00007fffca89f288 EFLAGS: 00000246 ORIG_RAX: 
000000000000004b
[314963.506844] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 
00007f62a0dfec30
[314963.506845] RDX: 0000000000000000 RSI: 00007f62a10c47a0 RDI: 
0000000000000003
[314963.506846] RBP: 00007fffca89f400 R08: 00007f62a12d9700 R09: 
00007f62a12d9700
[314963.506847] R10: 00007fffca89f050 R11: 0000000000000246 R12: 
00000000ffffffff
[314963.506848] R13: 00007fffca89f440 R14: 00007fffca89f2a0 R15: 
00007fffca89f29c
[314963.506851] Code: fe ff 89 d9 ba 11 0c 00 00 48 c7 c6 40 48 67 c0 4c 
89 e7 e8 c5 bc 09 00 e9 b5 fe ff ff 89 de 48 c7 c7 f8 b4 67 c0 e8 2d 28 
51 d2 <0f> ff eb d3 e8 3a be 09 00 0f 1f 00 66 66 66 66 90 55 48 89 e5
[314963.506889] ---[ end trace b11381065314a695 ]---
[314963.506955] BTRFS: error (device sda4) in 
btrfs_run_delayed_refs:3089: errno=-28 No space left
[314963.507032] BTRFS info (device sda4): forced readonly
[314963.510570] BTRFS warning (device sda4): Skipping commit of aborted 
transaction.
[314963.510577] BTRFS: error (device sda4) in cleanup_transaction:1873: 
errno=-28 No space left
[314970.954768] mail[32290]: segfault at c0 ip 00007f6b507ae33b sp 
00007ffec4849ac0 error 4 in libmailutils.so.4.0.0[7f6b50724000+b0000]
[314983.475988] BTRFS error (device sda4): pending csums is 167936




# btrfs fi show /srv
Label: 'btrfs'  uuid: 105b2e0c-8af2-45ee-b4c8-14ff0a3ca899
         Total devices 2 FS bytes used 2.31TiB
         devid    1 size 2.63TiB used 2.32TiB path /dev/sda4
         devid    2 size 2.63TiB used 2.32TiB path /dev/sdb4


# btrfs fi df /srv
Data, RAID1: total=2.30TiB, used=2.29TiB
System, RAID1: total=32.00MiB, used=384.00KiB
Metadata, RAID1: total=22.00GiB, used=19.61GiB
GlobalReserve, single: total=512.00MiB, used=481.56MiB



# btrfs fi usage /srv
Overall:
     Device size:                   5.25TiB
     Device allocated:              4.63TiB
     Device unallocated:          633.97GiB
     Device missing:                  0.00B
     Used:                          4.62TiB
     Free (estimated):            319.11GiB      (min: 319.11GiB)
     Data ratio:                       2.00
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 481.56MiB)

Data,RAID1: Size:2.30TiB, Used:2.29TiB
    /dev/sda4       2.30TiB
    /dev/sdb4       2.30TiB

Metadata,RAID1: Size:22.00GiB, Used:19.61GiB
    /dev/sda4      22.00GiB
    /dev/sdb4      22.00GiB

System,RAID1: Size:32.00MiB, Used:384.00KiB
    /dev/sda4      32.00MiB
    /dev/sdb4      32.00MiB

Unallocated:
    /dev/sda4     316.99GiB
    /dev/sdb4     316.99GiB


Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

On 2017-10-31 23:18, Tomasz Chmielewski wrote:

> On a different server, however, it failed badly:
> 
> # time btrfs balance start /srv
> WARNING:
> 
>         Full balance without filters requested. This operation is very
>         intense and takes potentially very long. It is recommended to
>         use the balance filters to narrow down the scope of balance.
>         Use 'btrfs balance start --full-balance' option to skip this
>         warning. The operation will start in 10 seconds.
>         Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting balance without any filters.
> ERROR: error during balancing '/srv': Read-only file system
> There may be more info in syslog - try dmesg | tail
> 
> [312304.050731] BTRFS info (device sda4): found 15073 extents
> [313555.971253] BTRFS info (device sda4): relocating block group
> 1208022466560 flags data|raid1
> [314963.506580] BTRFS: Transaction aborted (error -28)
> [314963.506608] ------------[ cut here ]------------
> [314963.506639] WARNING: CPU: 2 PID: 27854 at
> /home/kernel/COD/linux/fs/btrfs/extent-tree.c:3089
> btrfs_run_delayed_refs+0x244/0x250 [btrfs]

(...)

> [314963.506955] BTRFS: error (device sda4) in
> btrfs_run_delayed_refs:3089: errno=-28 No space left
> [314963.507032] BTRFS info (device sda4): forced readonly
> [314963.510570] BTRFS warning (device sda4): Skipping commit of
> aborted transaction.
> [314963.510577] BTRFS: error (device sda4) in
> cleanup_transaction:1873: errno=-28 No space left
> [314970.954768] mail[32290]: segfault at c0 ip 00007f6b507ae33b sp
> 00007ffec4849ac0 error 4 in libmailutils.so.4.0.0[7f6b50724000+b0000]
> [314983.475988] BTRFS error (device sda4): pending csums is 167936


And btrfs balance can be a real database killer :(


root@backupslave01:/var/log/mysql# tail -f mysql-error.log
InnoDB: Doing recovery: scanned up to log sequence number 2206178343424
InnoDB: Doing recovery: scanned up to log sequence number 2206183586304
InnoDB: Doing recovery: scanned up to log sequence number 2206188829184
InnoDB: Doing recovery: scanned up to log sequence number 2206194072064
InnoDB: Doing recovery: scanned up to log sequence number 2206199314944
InnoDB: Doing recovery: scanned up to log sequence number 2206204557824
InnoDB: Doing recovery: scanned up to log sequence number 2206209800704
InnoDB: Doing recovery: scanned up to log sequence number 2206215043584
InnoDB: Doing recovery: scanned up to log sequence number 2206220286464
InnoDB: Doing recovery: scanned up to log sequence number 2206220752384

InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is 21145843968
2017-10-31 14:46:59 4359 [Note] InnoDB: Starting an apply batch of log 
records to the database...
InnoDB: Progress in percent: 14:46:59 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this 
binary
or one of the libraries it was linked against is corrupt, improperly 
built,
or misconfigured. This error can also be caused by malfunctioning 
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=0
max_threads=502
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 
232495 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x8d444b]
/usr/sbin/mysqld(handle_fatal_signal+0x49a)[0x649b0a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f74b90bf390]
/usr/sbin/mysqld[0x99fcae]
/usr/sbin/mysqld[0x9a17ed]
/usr/sbin/mysqld[0x9881ea]
/usr/sbin/mysqld[0x989fc7]
/usr/sbin/mysqld[0xa6dd87]
/usr/sbin/mysqld[0xab8cd8]
/usr/sbin/mysqld[0xa08300]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f74b90b56ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f74b854a3dd]



Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

On 2017-10-31 23:18, Tomasz Chmielewski wrote:
> On 2017-09-18 17:20, Tomasz Chmielewski wrote:
>>>> # df -h /var/lib/lxd
>>>> 
>>>> FWIW, standard (aka util-linux) df is effectively useless in a 
>>>> situation
>>>> such as this, as it really doesn't give you the information you need 
>>>> (it
>>>> can say you have lots of space available, but if btrfs has all of it
>>>> allocated into chunks, even if the chunks have space in them still, 
>>>> there
>>>> can be problems).
>> 
>> I see here on RAID-1, "df -h" it shows pretty much the same amount of
>> free space as "btrfs fi show":
>> 
>> - "df -h" shows 105G free
>> - "btrfs fi show" says: Free (estimated):            104.28GiB
>> (min: 104.28GiB)
>> 
>> 
>> 
>>> But chances are pretty good that one you get that patch integrated,
>>> whether by integrating it yourself to what you have currently, or by
>>> trying 4.14-rc1 or waiting until it hits release or stable, that bug 
>>> will
>>> have been squashed! =:^)
>> 
>> OK, will wait for 4.14.
> 
> So I've tried to run balance with 4.14-rc6.

I've also tried with 4.14-rc7 on a server which was failing with "no 
space left" - unfortunately, it's still failing:


# time btrfs balance start /srv
WARNING:

         Full balance without filters requested. This operation is very
         intense and takes potentially very long. It is recommended to
         use the balance filters to narrow down the scope of balance.
         Use 'btrfs balance start --full-balance' option to skip this
         warning. The operation will start in 10 seconds.
         Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/srv': No space left on device
There may be more info in syslog - try dmesg | tail

real    8731m13.424s
user    0m0.000s
sys     560m36.363s



# dmesg -c
(...)
[546228.496902] BTRFS info (device sda4): relocating block group 
297455845376 flags data|raid1
[546251.393541] BTRFS info (device sda4): found 107799 extents
[546512.346360] BTRFS info (device sda4): found 107799 extents
[546529.407077] BTRFS info (device sda4): relocating block group 
296382103552 flags metadata|raid1
[546692.465746] BTRFS info (device sda4): found 35202 extents
[546733.294172] BTRFS info (device sda4): found 2586 extents
[546738.487556] BTRFS info (device sda4): relocating block group 
295308361728 flags data|raid1
[546770.474409] BTRFS info (device sda4): found 140906 extents
[547037.744023] BTRFS info (device sda4): found 140906 extents
[547065.840993] BTRFS info (device sda4): 117 enospc errors during 
balance


# btrfs fi df /srv
Data, RAID1: total=2.46TiB, used=2.35TiB
System, RAID1: total=32.00MiB, used=416.00KiB
Metadata, RAID1: total=19.00GiB, used=12.92GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# btrfs fi show /srv
Label: 'btrfs'  uuid: 105b2e0c-8af2-45ee-b4c8-14ff0a3ca899
         Total devices 2 FS bytes used 2.36TiB
         devid    1 size 2.63TiB used 2.48TiB path /dev/sda4
         devid    2 size 2.63TiB used 2.48TiB path /dev/sdb4


# btrfs fi usage /srv
Overall:
     Device size:                   5.25TiB
     Device allocated:              4.96TiB
     Device unallocated:          302.00GiB
     Device missing:                  0.00B
     Used:                          4.72TiB
     Free (estimated):            268.66GiB      (min: 268.66GiB)
     Data ratio:                       2.00
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID1: Size:2.46TiB, Used:2.35TiB
    /dev/sda4       2.46TiB
    /dev/sdb4       2.46TiB

Metadata,RAID1: Size:19.00GiB, Used:12.92GiB
    /dev/sda4      19.00GiB
    /dev/sdb4      19.00GiB

System,RAID1: Size:32.00MiB, Used:416.00KiB
    /dev/sda4      32.00MiB
    /dev/sdb4      32.00MiB

Unallocated:
    /dev/sda4     151.00GiB
    /dev/sdb4     151.00GiB


Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Tomasz Chmielewski]

On 2017-11-07 23:49, E V wrote:

> Hmm, I used to see these phantom no space issues quite a bit on older
> 4.x kernels, and haven't seen them since switching to space_cache=v2.
> So it could be space cache corruption. You might try either clearing
> you space cache, or mounting with nospace_cache, or try converting to
> space_cache=v2 after reading up on it's caveats.

We have space_cache=v2.

Unfortunately yet one more system running 4.14-rc8 with "No space left" 
during balance:


[68443.535664] BTRFS info (device sdb3): relocating block group 
591771009024 flags data|raid1
[68463.203330] BTRFS info (device sdb3): found 8578 extents
[68492.238676] BTRFS info (device sdb3): found 8559 extents
[68500.751792] BTRFS info (device sdb3): 1 enospc errors during balance


# btrfs balance start /var/lib/lxd
WARNING:

         Full balance without filters requested. This operation is very
         intense and takes potentially very long. It is recommended to
         use the balance filters to narrow down the balanced data.
         Use 'btrfs balance start --full-balance' option to skip this
         warning. The operation will start in 10 seconds.
         Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/var/lib/lxd': No space left on device
There may be more info in syslog - try dmesg | tail


# btrfs fi usage /var/lib/lxd
Overall:
     Device size:                 846.26GiB
     Device allocated:            622.27GiB
     Device unallocated:          223.99GiB
     Device missing:                  0.00B
     Used:                        606.40GiB
     Free (estimated):            116.68GiB      (min: 116.68GiB)
     Data ratio:                       2.00
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID1: Size:306.00GiB, Used:301.31GiB
    /dev/sda3     306.00GiB
    /dev/sdb3     306.00GiB

Metadata,RAID1: Size:5.10GiB, Used:1.89GiB
    /dev/sda3       5.10GiB
    /dev/sdb3       5.10GiB

System,RAID1: Size:32.00MiB, Used:80.00KiB
    /dev/sda3      32.00MiB
    /dev/sdb3      32.00MiB

Unallocated:
    /dev/sda3     112.00GiB
    /dev/sdb3     112.00GiB


# btrfs fi show /var/lib/lxd
Label: 'btrfs'  uuid: 6340f5de-f635-4d09-bbb2-1e03b1e1b160
         Total devices 2 FS bytes used 303.20GiB
         devid    1 size 423.13GiB used 311.13GiB path /dev/sda3
         devid    2 size 423.13GiB used 311.13GiB path /dev/sdb3


# btrfs fi df /var/lib/lxd
Data, RAID1: total=306.00GiB, used=301.32GiB
System, RAID1: total=32.00MiB, used=80.00KiB
Metadata, RAID1: total=5.10GiB, used=1.89GiB
GlobalReserve, single: total=512.00MiB, used=0.00B



So far out of all systems which were giving us "No space left on device" 
with 4.13.x, all but one are still giving us "No space left on device" 
during balance with 4.14-rc7 and later.
We've seen it on a mix of servers with SSD or HDD disks, with 
filesystems ranging from 0.5 TB to 20 TB, and use % from 30% to 90%.

Combined with evidence that "No space left on device" during balance can 
lead to various file corruption (we've witnessed it with MySQL), I'd day 
btrfs balance is a dangerous operation and decision to use it should be 
considered very thoroughly.


Shouldn't "Balance" be marked as "mostly OK" or "Unstable" here? Giving 
it "OK" status is misleading.

https://btrfs.wiki.kernel.org/index.php/Status


Tomasz Chmielewski
https://lxadm.com

Re: how to run balance successfully (No space left on device)?

[Chris Murphy]

On Fri, Nov 10, 2017 at 12:42 AM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> On 2017-11-07 23:49, E V wrote:
>
>> Hmm, I used to see these phantom no space issues quite a bit on older
>> 4.x kernels, and haven't seen them since switching to space_cache=v2.
>> So it could be space cache corruption. You might try either clearing
>> you space cache, or mounting with nospace_cache, or try converting to
>> space_cache=v2 after reading up on it's caveats.
>
>
> We have space_cache=v2.


I have no idea if it's related or not, as this isn't a default mount
option and is still under testing.



>
> Unfortunately yet one more system running 4.14-rc8 with "No space left"
> during balance:
>
>
> [68443.535664] BTRFS info (device sdb3): relocating block group 591771009024
> flags data|raid1
> [68463.203330] BTRFS info (device sdb3): found 8578 extents
> [68492.238676] BTRFS info (device sdb3): found 8559 extents
> [68500.751792] BTRFS info (device sdb3): 1 enospc errors during balance
>
>
> # btrfs balance start /var/lib/lxd
> WARNING:
>
>         Full balance without filters requested. This operation is very
>         intense and takes potentially very long. It is recommended to
>         use the balance filters to narrow down the balanced data.
>         Use 'btrfs balance start --full-balance' option to skip this
>         warning. The operation will start in 10 seconds.
>         Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting balance without any filters.
> ERROR: error during balancing '/var/lib/lxd': No space left on device
> There may be more info in syslog - try dmesg | tail


OK I wonder if this is a bug in user space tool's error handling?
Because what you have in kernel messages is BTRFS info. It is not a
warning or an error. I interpret this as enospc error happened but it
recovered, so it was not an unhandled error condition, and definitely
non-fatal. But the user space tool is reporting a bogus "No space left
on device". It's plainly bogus because you have a lot of space on the
device, including unallocated space. So the user space tool needs to
either ignore this type of informational enospc or it needs a
different message to make it clear this is not a fatal error and was
properly handled.

Do you get any additional information when using enospc_debug mount
option and reproduce this problem?


> Unallocated:
>    /dev/sda3     112.00GiB
>    /dev/sdb3     112.00GiB

Metric shittons of space. The error is certainly bogus.



> Combined with evidence that "No space left on device" during balance can
> lead to various file corruption (we've witnessed it with MySQL), I'd day
> btrfs balance is a dangerous operation and decision to use it should be
> considered very thoroughly.

I've never heard of this. Balance is COW at the chunk level. The old
chunk is not dereferenced until it's written in the new location
correctly. Corruption during balance shouldn't be possible so if you
have a reproducer, the devs need to know about it.





-- 
Chris Murphy

Re: how to run balance successfully (No space left on device)?

[Martin Raiber]

On 10.11.2017 22:51 Chris Murphy wrote:
>> Combined with evidence that "No space left on device" during balance can
>> lead to various file corruption (we've witnessed it with MySQL), I'd day
>> btrfs balance is a dangerous operation and decision to use it should be
>> considered very thoroughly.
> I've never heard of this. Balance is COW at the chunk level. The old
> chunk is not dereferenced until it's written in the new location
> correctly. Corruption during balance shouldn't be possible so if you
> have a reproducer, the devs need to know about it.

I didn't say anything before, because I could not reproduce the problem.
I had (I guess) a corruption caused by balance as well. It had ENOSPC in
spite of enough free space (4.9.x), which made me balance it regularly
to keep unallocated space around. Corruption occured probably after or
shortly before power reset during a balance -- no skip_balance specified
so it continued directly after mount -- data was moved relatively fast
after the mount operation (copy file then delete old file). I think
space_cache=v2 was active at the time. I'm of course not completely sure
it was btrfs's fault and as usual not all the conditions may be
relevant. Could also be instead an upper layer error (Hyper-V storage),
memory issue or an application error.

Regards,
Martin Raiber