* Balance ENOSPC during balance despite additional storage added
@ 2019-09-20 14:29 Pete
2019-09-22 17:47 ` Chris Murphy
2019-09-24 13:22 ` Josef Bacik
0 siblings, 2 replies; 14+ messages in thread
From: Pete @ 2019-09-20 14:29 UTC (permalink / raw)
To: Btrfs BTRFS
I have a btrfs that is on top of an lvm logical volume on top of
dm-crypt on a single nvme drive (Samsung 870 Pro 512GB).
I added a second logical volume to give more space to get rid of ENOSPC
errors during balance, but to no avail. This was after I started
getting enospc during balance. Without this additional logical device,
before balance, I had run out of space owning to some unfortunate
scripting interacting with lxc snapshots (non btrfs backed in the
config, so a copy) and some copying. I was performing a balance,
following some deletions, when trying to get things back to a better state.
root@phoenix:/var/lib/lxc# btrfs balance start /var/lib/lxc
WARNING:
Full balance without filters requested. This operation is very
intense and takes potentially very long. It is recommended to
use the balance filters to narrow down the scope of balance.
Use 'btrfs balance start --full-balance' option to skip this
warning. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/var/lib/lxc': No space left on device
There may be more info in syslog - try dmesg | tail
root@phoenix:/var/lib/lxc#
I can still write to the filesystem.
Kernel 5.1.21 (downgraded from 5.2.12)
root@phoenix:/var/lib/lxc# btrfs --version
btrfs-progs v5.1
root@phoenix:/var/lib/lxc# btrfs fi show /var/lib/lxc
Label: 'LXC_BTRFS' uuid: 6b0245ec-bdd4-4076-b800-2243d466b174
Total devices 2 FS bytes used 79.74GiB
devid 1 size 250.00GiB used 93.03GiB path
/dev/mapper/nvme0_vg-lxc
devid 2 size 80.00GiB used 0.00B path
/dev/mapper/nvme0_vg-tempdel
root@phoenix:/var/lib/lxc# btrfs fi u /var/lib/lxc
Overall:
Device size: 330.00GiB
Device allocated: 93.03GiB
Device unallocated: 236.97GiB
Device missing: 0.00B
Used: 79.74GiB
Free (estimated): 237.70GiB (min: 237.70GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:71.00GiB, Used:70.26GiB
/dev/mapper/nvme0_vg-lxc 71.00GiB
Metadata,single: Size:22.00GiB, Used:9.48GiB
/dev/mapper/nvme0_vg-lxc 22.00GiB
System,single: Size:32.00MiB, Used:16.00KiB
/dev/mapper/nvme0_vg-lxc 32.00MiB
Unallocated:
/dev/mapper/nvme0_vg-lxc 156.97GiB
/dev/mapper/nvme0_vg-tempdel 80.00GiB
btrfs fi df /var/lib/lxc
Data, single: total=71.00GiB, used=70.26GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=22.00GiB, used=9.48GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
root@phoenix:/var/lib/lxc#
An unfiltered balance shows ENOSPC errors:
btrfs balance start /var/lib/lxc
Last bit of:
dmesg | tail -n 100
[ 920.915627] BTRFS info (device dm-4): found 67520 extents
[ 922.037071] BTRFS info (device dm-4): relocating block group
1703106576384 flags data
[ 924.742432] BTRFS info (device dm-4): found 57082 extents
[ 927.245236] BTRFS info (device dm-4): found 57082 extents
[ 928.371624] BTRFS info (device dm-4): relocating block group
1702032834560 flags data
[ 931.230841] BTRFS info (device dm-4): found 60454 extents
[ 933.373249] BTRFS info (device dm-4): found 60454 extents
[ 934.336628] BTRFS info (device dm-4): relocating block group
1700959092736 flags data
[ 937.330097] BTRFS info (device dm-4): found 67151 extents
[ 940.296250] BTRFS info (device dm-4): found 67151 extents
[ 941.524664] BTRFS info (device dm-4): relocating block group
1699885350912 flags data
[ 944.264618] BTRFS info (device dm-4): found 54931 extents
[ 945.910666] BTRFS info (device dm-4): found 54931 extents
[ 946.796308] BTRFS info (device dm-4): relocating block group
1698811609088 flags data
[ 949.426823] BTRFS info (device dm-4): found 55428 extents
[ 950.880553] BTRFS info (device dm-4): found 55428 extents
[ 951.622569] BTRFS info (device dm-4): relocating block group
1697737867264 flags data
[ 955.223382] BTRFS info (device dm-4): found 52897 extents
[ 956.544084] BTRFS info (device dm-4): found 52897 extents
[ 957.300021] BTRFS info (device dm-4): relocating block group
1696664125440 flags data
[ 959.936585] BTRFS info (device dm-4): found 48407 extents
[ 961.421771] BTRFS info (device dm-4): found 48407 extents
[ 962.203680] BTRFS info (device dm-4): relocating block group
1695590383616 flags data
[ 964.281128] BTRFS info (device dm-4): found 28238 extents
[ 965.325130] BTRFS info (device dm-4): found 28238 extents
[ 965.886794] BTRFS info (device dm-4): relocating block group
1694516641792 flags data
[ 968.999507] BTRFS info (device dm-4): found 46060 extents
[ 970.447815] BTRFS info (device dm-4): found 46060 extents
[ 971.276287] BTRFS info (device dm-4): relocating block group
1693442899968 flags data
[ 974.914746] BTRFS info (device dm-4): found 55159 extents
[ 976.914228] BTRFS info (device dm-4): found 55159 extents
[ 977.758643] BTRFS info (device dm-4): relocating block group
1692369158144 flags data
[ 980.081069] BTRFS info (device dm-4): found 36859 extents
[ 981.630065] BTRFS info (device dm-4): found 36859 extents
[ 982.498586] BTRFS info (device dm-4): relocating block group
1691295416320 flags data
[ 984.929101] BTRFS info (device dm-4): found 50062 extents
[ 986.440469] BTRFS info (device dm-4): found 50062 extents
[ 987.281364] BTRFS info (device dm-4): 11 enospc errors during balance
[ 987.281365] BTRFS info (device dm-4): balance: ended with status: -28
Unfortunately I don't seem to have any more info in dmesg of the enospc
errors:
root@phoenix:/var/lib/lxc# dmesg | grep enospc
[ 987.281364] BTRFS info (device dm-4): 11 enospc errors during balance
root@phoenix:/var/lib/lxc# dmesg | grep ENOSPC
root@phoenix:/var/lib/lxc#
I've seen something a little odd in /var/log/messages, not sure it is
related.:
Sep 20 13:05:07 phoenix named[1805]: zone 30.20.10.in-addr.arpa/IN: not
loaded due to errors.
Sep 20 13:05:08 phoenix kernel: [ 0.500023] PPR NX GT IA GA PC GA_vAPIC
Sep 20 13:05:09 phoenix kernel: [ 0.827723] Loading Adaptec I2O RAID:
Version 2.4 Build 5go
Sep 20 13:05:09 phoenix kernel: [ 0.829457] GDT-HA: Storage RAID
Controller Driver. Version: 3.05
Sep 20 13:05:09 phoenix kernel: [ 0.829615] 3ware Storage Controller
device driver for Linux v1.26.02.003.
Sep 20 13:05:09 phoenix kernel: [ 0.829766] 3ware 9000 Storage
Controller device driver for Linux v2.26.02.014.
Sep 20 13:05:09 phoenix kernel: [ 1.039111] nvme nvme0: missing or
invalid SUBNQN field.
Sep 20 13:05:09 phoenix kernel: [ 5.430984] ata5.00: supports DRM
functions and may not be fully accessible
Sep 20 13:05:09 phoenix kernel: [ 77.750030] RAX: ffffffffffffffda
RBX: 00005648284a1610 RCX: 00007fc76dd7736f
Sep 20 13:05:09 phoenix kernel: [ 77.750031] RDX: 0000000000000002
RSI: 0000000000000080 RDI: 00005648284a16ac
Sep 20 13:05:09 phoenix kernel: [ 77.750031] RBP: 00007fc7671e8e90
R08: 00005648284a1600 R09: 0000000000000000
Sep 20 13:05:09 phoenix kernel: [ 77.750032] R10: 0000000000000000
R11: 0000000000000246 R12: 00007fc7671e8e88
Sep 20 13:05:09 phoenix kernel: [ 77.750033] R13: 00007fc7671e8e80
R14: 00007fc7671e8e98 R15: 00005648284a1610
Sep 20 13:05:09 phoenix kernel: [ 77.750035] Call Trace:
Sep 20 13:05:09 phoenix kernel: [ 77.750036] ? __schedule+0x211/0x820
Sep 20 13:05:09 phoenix kernel: [ 77.750038] schedule+0x32/0x70
Sep 20 13:05:09 phoenix kernel: [ 77.750040]
futex_wait_queue_me+0xce/0x140
Sep 20 13:05:09 phoenix kernel: [ 77.750041] futex_wait+0xef/0x240
Sep 20 13:05:09 phoenix kernel: [ 77.750043] do_futex+0x17d/0xce0
Sep 20 13:05:09 phoenix kernel: [ 77.750045] ? __switch_to_asm+0x41/0x70
After no issues in quite a while I seem to be hitting a fair few at
present. No idea if I am doing something new.
Pete
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-20 14:29 Balance ENOSPC during balance despite additional storage added Pete
@ 2019-09-22 17:47 ` Chris Murphy
2019-09-22 19:06 ` Pete
2019-09-24 13:22 ` Josef Bacik
1 sibling, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2019-09-22 17:47 UTC (permalink / raw)
To: Pete; +Cc: Btrfs BTRFS
On Fri, Sep 20, 2019 at 8:31 AM Pete <pete@petezilla.co.uk> wrote:
>
> I have a btrfs that is on top of an lvm logical volume on top of
> dm-crypt on a single nvme drive (Samsung 870 Pro 512GB).
>
> I added a second logical volume to give more space to get rid of ENOSPC
> errors during balance, but to no avail. This was after I started
> getting enospc during balance. Without this additional logical device,
> before balance, I had run out of space owning to some unfortunate
> scripting interacting with lxc snapshots (non btrfs backed in the
> config, so a copy) and some copying. I was performing a balance,
> following some deletions, when trying to get things back to a better state.
>
> root@phoenix:/var/lib/lxc# btrfs balance start /var/lib/lxc
> WARNING:
>
> Full balance without filters requested. This operation is very
> intense and takes potentially very long. It is recommended to
> use the balance filters to narrow down the scope of balance.
> Use 'btrfs balance start --full-balance' option to skip this
> warning. The operation will start in 10 seconds.
> Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting balance without any filters.
> ERROR: error during balancing '/var/lib/lxc': No space left on device
> There may be more info in syslog - try dmesg | tail
> root@phoenix:/var/lib/lxc#
>
> I can still write to the filesystem.
>
>
> Kernel 5.1.21 (downgraded from 5.2.12)
>
> root@phoenix:/var/lib/lxc# btrfs --version
> btrfs-progs v5.1
>
> root@phoenix:/var/lib/lxc# btrfs fi show /var/lib/lxc
> Label: 'LXC_BTRFS' uuid: 6b0245ec-bdd4-4076-b800-2243d466b174
> Total devices 2 FS bytes used 79.74GiB
> devid 1 size 250.00GiB used 93.03GiB path
> /dev/mapper/nvme0_vg-lxc
> devid 2 size 80.00GiB used 0.00B path
> /dev/mapper/nvme0_vg-tempdel
>
> root@phoenix:/var/lib/lxc# btrfs fi u /var/lib/lxc
> Overall:
> Device size: 330.00GiB
> Device allocated: 93.03GiB
> Device unallocated: 236.97GiB
> Device missing: 0.00B
> Used: 79.74GiB
> Free (estimated): 237.70GiB (min: 237.70GiB)
> Data ratio: 1.00
> Metadata ratio: 1.00
> Global reserve: 512.00MiB (used: 0.00B)
>
> Data,single: Size:71.00GiB, Used:70.26GiB
> /dev/mapper/nvme0_vg-lxc 71.00GiB
>
> Metadata,single: Size:22.00GiB, Used:9.48GiB
> /dev/mapper/nvme0_vg-lxc 22.00GiB
>
> System,single: Size:32.00MiB, Used:16.00KiB
> /dev/mapper/nvme0_vg-lxc 32.00MiB
>
> Unallocated:
> /dev/mapper/nvme0_vg-lxc 156.97GiB
> /dev/mapper/nvme0_vg-tempdel 80.00GiB
>
> btrfs fi df /var/lib/lxc
> Data, single: total=71.00GiB, used=70.26GiB
> System, single: total=32.00MiB, used=16.00KiB
> Metadata, single: total=22.00GiB, used=9.48GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> root@phoenix:/var/lib/lxc#
>
>
>
> An unfiltered balance shows ENOSPC errors:
> btrfs balance start /var/lib/lxc
>
> Last bit of:
> dmesg | tail -n 100
>
>
> [ 920.915627] BTRFS info (device dm-4): found 67520 extents
> [ 922.037071] BTRFS info (device dm-4): relocating block group
> 1703106576384 flags data
> [ 924.742432] BTRFS info (device dm-4): found 57082 extents
> [ 927.245236] BTRFS info (device dm-4): found 57082 extents
> [ 928.371624] BTRFS info (device dm-4): relocating block group
> 1702032834560 flags data
> [ 931.230841] BTRFS info (device dm-4): found 60454 extents
> [ 933.373249] BTRFS info (device dm-4): found 60454 extents
> [ 934.336628] BTRFS info (device dm-4): relocating block group
> 1700959092736 flags data
> [ 937.330097] BTRFS info (device dm-4): found 67151 extents
> [ 940.296250] BTRFS info (device dm-4): found 67151 extents
> [ 941.524664] BTRFS info (device dm-4): relocating block group
> 1699885350912 flags data
> [ 944.264618] BTRFS info (device dm-4): found 54931 extents
> [ 945.910666] BTRFS info (device dm-4): found 54931 extents
> [ 946.796308] BTRFS info (device dm-4): relocating block group
> 1698811609088 flags data
> [ 949.426823] BTRFS info (device dm-4): found 55428 extents
> [ 950.880553] BTRFS info (device dm-4): found 55428 extents
> [ 951.622569] BTRFS info (device dm-4): relocating block group
> 1697737867264 flags data
> [ 955.223382] BTRFS info (device dm-4): found 52897 extents
> [ 956.544084] BTRFS info (device dm-4): found 52897 extents
> [ 957.300021] BTRFS info (device dm-4): relocating block group
> 1696664125440 flags data
> [ 959.936585] BTRFS info (device dm-4): found 48407 extents
> [ 961.421771] BTRFS info (device dm-4): found 48407 extents
> [ 962.203680] BTRFS info (device dm-4): relocating block group
> 1695590383616 flags data
> [ 964.281128] BTRFS info (device dm-4): found 28238 extents
> [ 965.325130] BTRFS info (device dm-4): found 28238 extents
> [ 965.886794] BTRFS info (device dm-4): relocating block group
> 1694516641792 flags data
> [ 968.999507] BTRFS info (device dm-4): found 46060 extents
> [ 970.447815] BTRFS info (device dm-4): found 46060 extents
> [ 971.276287] BTRFS info (device dm-4): relocating block group
> 1693442899968 flags data
> [ 974.914746] BTRFS info (device dm-4): found 55159 extents
> [ 976.914228] BTRFS info (device dm-4): found 55159 extents
> [ 977.758643] BTRFS info (device dm-4): relocating block group
> 1692369158144 flags data
> [ 980.081069] BTRFS info (device dm-4): found 36859 extents
> [ 981.630065] BTRFS info (device dm-4): found 36859 extents
> [ 982.498586] BTRFS info (device dm-4): relocating block group
> 1691295416320 flags data
> [ 984.929101] BTRFS info (device dm-4): found 50062 extents
> [ 986.440469] BTRFS info (device dm-4): found 50062 extents
> [ 987.281364] BTRFS info (device dm-4): 11 enospc errors during balance
> [ 987.281365] BTRFS info (device dm-4): balance: ended with status: -28
>
> Unfortunately I don't seem to have any more info in dmesg of the enospc
> errors:
You need to mount with enospc_debug to get more information, it might
be useful for a developer. This -28 error is one that has mostly gone
away, I don't know if the cause was ever discovered, but my
recollection is once you're hitting it, you're better off creating a
new file system rather than chasing it.
But you could use 5.2.15 or newer, mount with enospc_debug, and do
filtered balance. You could start with 1% increments, e.g. -dusage=1,
-dusage=2, up to 5. And then do it in 5% increments up to 70. The idea
of that is just to try and avoid enospc while picking off the low
hanging fruit first (the block groups with the most free space). At
that point I would then start a full balance, no filter. Maybe that'll
get it back on track. I haven't ever experienced this so this strategy
is totally a spitball method of trying to fix it. There is some degree
of metadata rewrites that happens as part of balance, and balance is
pretty complicated, and not entirely deterministic - meaning it's
plausible the filtered balance followed by a full balance could fix
it. But I don't understand it well enough.
Also I'd remove any snapshots you don't really need, it'll make the
balance less complicated and faster.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-22 17:47 ` Chris Murphy
@ 2019-09-22 19:06 ` Pete
2019-09-22 21:25 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Pete @ 2019-09-22 19:06 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 9/22/19 6:47 PM, Chris Murphy wrote:
>> Unfortunately I don't seem to have any more info in dmesg of the enospc
>> errors:
>
> You need to mount with enospc_debug to get more information, it might
> be useful for a developer. This -28 error is one that has mostly gone
> away, I don't know if the cause was ever discovered, but my
> recollection is once you're hitting it, you're better off creating a
> new file system rather than chasing it.
>
> But you could use 5.2.15 or newer, mount with enospc_debug, and do
> filtered balance. You could start with 1% increments, e.g. -dusage=1,
> -dusage=2, up to 5. And then do it in 5% increments up to 70. The idea
> of that is just to try and avoid enospc while picking off the low
> hanging fruit first (the block groups with the most free space). At
> that point I would then start a full balance, no filter. Maybe that'll
> get it back on track. I haven't ever experienced this so this strategy
> is totally a spitball method of trying to fix it. There is some degree
> of metadata rewrites that happens as part of balance, and balance is
> pretty complicated, and not entirely deterministic - meaning it's
> plausible the filtered balance followed by a full balance could fix
> it. But I don't understand it well enough.
OK, I'm building 5.2.17 now. Keen to avoid the corruption errors I was
hit by a few weeks back... May take a time as I'm in the middle of a
slow backup.
I note that a filtered balance, though not hitting enospc and not
reporting any errors did seemed to relocate a chunk/extent (sorry, I
forget the terminology) but running it a second, third and so on time
got the same result. As if the balance reported doing some work, but
did not actually do it. I also had to reboot at one point as it seemed
to get stuck in a loop but alas I can't repeat this. With the extra
logical volume added there is certainly no lack of space relative to the
size of the filesystem.
>
> Also I'd remove any snapshots you don't really need, it'll make the
> balance less complicated and faster.
>
>
There are not too many, but it does not do much harm to take a look.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-22 19:06 ` Pete
@ 2019-09-22 21:25 ` Chris Murphy
2019-09-23 18:55 ` Pete
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2019-09-22 21:25 UTC (permalink / raw)
To: Pete; +Cc: Chris Murphy, Btrfs BTRFS
On Sun, Sep 22, 2019 at 1:07 PM Pete <pete@petezilla.co.uk> wrote:
>
> On 9/22/19 6:47 PM, Chris Murphy wrote:
>
> >> Unfortunately I don't seem to have any more info in dmesg of the enospc
> >> errors:
> >
> > You need to mount with enospc_debug to get more information, it might
> > be useful for a developer. This -28 error is one that has mostly gone
> > away, I don't know if the cause was ever discovered, but my
> > recollection is once you're hitting it, you're better off creating a
> > new file system rather than chasing it.
> >
> > But you could use 5.2.15 or newer, mount with enospc_debug, and do
> > filtered balance. You could start with 1% increments, e.g. -dusage=1,
> > -dusage=2, up to 5. And then do it in 5% increments up to 70. The idea
> > of that is just to try and avoid enospc while picking off the low
> > hanging fruit first (the block groups with the most free space). At
> > that point I would then start a full balance, no filter. Maybe that'll
> > get it back on track. I haven't ever experienced this so this strategy
> > is totally a spitball method of trying to fix it. There is some degree
> > of metadata rewrites that happens as part of balance, and balance is
> > pretty complicated, and not entirely deterministic - meaning it's
> > plausible the filtered balance followed by a full balance could fix
> > it. But I don't understand it well enough.
>
> OK, I'm building 5.2.17 now. Keen to avoid the corruption errors I was
> hit by a few weeks back... May take a time as I'm in the middle of a
> slow backup.
Did you see corruption on this same file system? The 5.2 corruption
bug has been fixed in 5.2.15.
https://www.spinics.net/lists/stable-commits/msg129532.html
I'm not aware of a way to fix it though.
> I note that a filtered balance, though not hitting enospc and not
> reporting any errors did seemed to relocate a chunk/extent (sorry, I
> forget the terminology) but running it a second, third and so on time
> got the same result. As if the balance reported doing some work, but
> did not actually do it. I also had to reboot at one point as it seemed
> to get stuck in a loop but alas I can't repeat this. With the extra
> logical volume added there is certainly no lack of space relative to the
> size of the filesystem.
For sure you're running into some kind of bug.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-22 21:25 ` Chris Murphy
@ 2019-09-23 18:55 ` Pete
2019-09-23 21:33 ` Peter Chant
0 siblings, 1 reply; 14+ messages in thread
From: Pete @ 2019-09-23 18:55 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 9/22/19 10:25 PM, Chris Murphy wrote:
>> OK, I'm building 5.2.17 now. Keen to avoid the corruption errors I was
>> hit by a few weeks back... May take a time as I'm in the middle of a
>> slow backup.
>
> Did you see corruption on this same file system? The 5.2 corruption
> bug has been fixed in 5.2.15.
> https://www.spinics.net/lists/stable-commits/msg129532.html
>
> I'm not aware of a way to fix it though.
I had, what I believe is this issue, in late August/early Sept on my hdd
based filesystem. I was not aware that my nvme drive (/var/lib/lxc etc)
or my ssd, /, were affected. No obvious clues unlike the hdd filesystem
which threw checksum errors and went ro after a few minutes.
>
>
>> I note that a filtered balance, though not hitting enospc and not
>> reporting any errors did seemed to relocate a chunk/extent (sorry, I
>> forget the terminology) but running it a second, third and so on time
>> got the same result. As if the balance reported doing some work, but
>> did not actually do it. I also had to reboot at one point as it seemed
>> to get stuck in a loop but alas I can't repeat this. With the extra
>> logical volume added there is certainly no lack of space relative to the
>> size of the filesystem.
>
> For sure you're running into some kind of bug.
OK, I broadly did as you suggested, with the filesystem in question
mounted with enospc_debug and a 5.2.17 kernel. 'script' seemed to put a
few odd characters in, but you get my gist. Not sure what the first
line is?! Some garbage from somewhere.
Script started on Mon 23 Sep 2019 07:38:55 PM BST
root@phoenix:~# reboot\b\b\b\b\b\bmount | grep
lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bint-http-copy/umount_rootfs \b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b
\b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\b \b\a\abtrfs bal
staRT\b \b\b \brt -dusage=1 /var/lib/lxc
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=1 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b1\b^[[1P^[[1@1\b^[[1P^[[1@2
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start -dusage=2
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@3
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start -dusage=3
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@4
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start -dusage=4
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@5
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=5 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@1^[[1@0
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=10 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@2
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=20 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@3
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=30 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@4
Done, had to relocate 0 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=40 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@5
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=50 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@6
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-dusage=60 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1@7^[[1@0
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -dusage=70 /var/lib/lxc
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -dusage=70 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b70\b^[[1P\b^[[1P\b^[[1P\b^[[1P\b^[[1P\b^[[1P\b^[[1P\b^[[1P\b^[[1P^[[1@m^[[1@u^[[1@s^[[1@a^[[1@g^[[1@e^[[1@=^[[1@1
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=1 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b1\b^[[1P^[[1@2
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=2
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@3
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=3
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@4
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=4
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@5
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=5 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@1^[[1@0
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=10
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@5
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=15 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1@2^[[1@0
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=20
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@5
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal
start -musage=25 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1@3\b^[[1P^[[1@3^[[1@0
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=30
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@5
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=35 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1@4^[[1@0
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=40 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b0\b^[[1P^[[1@5
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=45 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1@5^[[1@0
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=50
/var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@5
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=55 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b5\b^[[1P\b^[[1P^[[1@6^[[1@0
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start
-musage=60 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1@6^[[1@5
Done, had to relocate 2 out of 89 chunks
root@phoenix:~# btrfs bal
start -musage=65 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1@5\b^[[1P^[[1@7^[[1@0
Done, had to relocate 2 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=70 /var/lib/lxc
Done, had to relocate 2 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=70 /var/lib/lxc
Done, had to relocate 2 out of 89 chunks
root@phoenix:~# btrfs bal start -musage=70 /var/lib/lxc
Done, had to relocate 2 out of 89 chunks
root@phoenix:~# btrfs
bal start -musage=70 /var/lib/lxc\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P^[[1@d
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# btrfs bal start -dusage=70 /var/lib/lxc\b
Done, had to relocate 1 out of 89 chunks
root@phoenix:~# exit
exit
Script done on Mon 23 Sep 2019 07:45:40 PM BST
dmesg showed this sort of thing:
root@phoenix:~# dmesg | tail -n 40
[ 724.387061] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 724.387062] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 724.387063] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 724.387064] BTRFS info (device dm-4): delayed_refs_rsv: size 4194304
reserved 1523712
[ 724.387093] BTRFS info (device dm-4): relocating block group
1942181904384 flags system
[ 724.412338] BTRFS info (device dm-4): found 1 extents
[ 724.438117] BTRFS info (device dm-4): balance: ended with status: 0
[ 725.144408] BTRFS info (device dm-4): balance: start -musage=70
-susage=70
[ 725.144453] BTRFS info (device dm-4): unable to make block group
1944362942464 ro
[ 725.144455] BTRFS info (device dm-4): sinfo_used=16384
bg_num_bytes=33538048 min_allocable=1048576
[ 725.144456] BTRFS info (device dm-4): space_info 2 has 33538048 free,
is not full
[ 725.144458] BTRFS info (device dm-4): space_info total=33554432,
used=16384, pinned=0, reserved=0, may_use=0, readonly=0
[ 725.144459] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536870912
[ 725.144460] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 725.144461] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 725.144462] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 725.144463] BTRFS info (device dm-4): delayed_refs_rsv: size 0 reserved 0
[ 725.144547] BTRFS info (device dm-4): relocating block group
1944362942464 flags system
[ 725.172398] BTRFS info (device dm-4): unable to make block group
1943289200640 ro
[ 725.172400] BTRFS info (device dm-4): sinfo_used=11051237376
bg_num_bytes=1073741824 min_allocable=1048576
[ 725.172402] BTRFS info (device dm-4): space_info 4 has 759922688
free, is not full
[ 725.172404] BTRFS info (device dm-4): space_info total=11811160064,
used=10512777216, pinned=0, reserved=114688, may_use=538345472, readonly=0
[ 725.172405] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536821760
[ 725.172406] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 725.172407] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 725.172408] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 725.172409] BTRFS info (device dm-4): delayed_refs_rsv: size 4194304
reserved 1523712
[ 725.172441] BTRFS info (device dm-4): relocating block group
1943289200640 flags metadata
[ 725.198714] BTRFS info (device dm-4): balance: ended with status: 0
[ 728.945125] BTRFS info (device dm-4): balance: start -dusage=70
[ 728.945261] BTRFS info (device dm-4): relocating block group
1938356699136 flags data
[ 729.822341] BTRFS info (device dm-4): found 9991 extents
[ 730.012601] BTRFS info (device dm-4): found 9990 extents
[ 730.160436] BTRFS info (device dm-4): balance: ended with status: 0
[ 732.697352] BTRFS info (device dm-4): balance: start -dusage=70
[ 732.697513] BTRFS info (device dm-4): relocating block group
1945503793152 flags data
[ 733.573416] BTRFS info (device dm-4): found 9820 extents
[ 733.738592] BTRFS info (device dm-4): found 9817 extents
[ 733.882864] BTRFS info (device dm-4): balance: ended with status: 0
[ 833.204449] radeon_dp_aux_transfer_native: 32 callbacks suppressed
root@phoenix:~#
I'm not sure the balance is resolving anything. The filesystem has not
gone read only. I'll try an unfiltered balance now to see how that goes.
Pete
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-23 18:55 ` Pete
@ 2019-09-23 21:33 ` Peter Chant
2019-09-23 21:46 ` Chris Murphy
2019-09-23 21:52 ` Chris Murphy
0 siblings, 2 replies; 14+ messages in thread
From: Peter Chant @ 2019-09-23 21:33 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 9/23/19 7:55 PM, Pete wrote:
> I'm not sure the balance is resolving anything. The filesystem has not
> gone read only. I'll try an unfiltered balance now to see how that goes.
>
OK, result of unfiltered balance:
root@phoenix:/var/lib/lxc# btrfs bal start /var/lib/lxc
WARNING:
Full balance without filters requested. This operation is very
intense and takes potentially very long. It is recommended to
use the balance filters to narrow down the scope of balance.
Use 'btrfs balance start --full-balance' option to skip this
warning. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/var/lib/lxc': No space left on device
There may be more info in syslog - try dmesg | tail
root@phoenix:/var/lib/lxc#
OK, here is the output in the last 100 lines of dmesg. The line 'unable
to make block group nnnnnnnnnnnnnnn ro' is repeated, I presume that is
relevant. I'm noting no other ill effects, if I'd not tried the balance
I would not know anything is amiss.
root@phoenix:/var/lib/lxc# dmesg | tail -n 100
[ 1880.155026] BTRFS info (device dm-4): sinfo_used=19522584576
bg_num_bytes=54394880 min_allocable=1048576
[ 1880.155027] BTRFS info (device dm-4): space_info 4 has
18446744065997733888 free, is not full
[ 1880.155029] BTRFS info (device dm-4): space_info total=11811160064,
used=10509664256, pinned=0, reserved=131072, may_use=9013182464, readonly=0
[ 1880.155029] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155030] BTRFS info (device dm-4): trans_block_rsv: size 262144
reserved 262144
[ 1880.155031] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 1880.155031] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155032] BTRFS info (device dm-4): delayed_refs_rsv: size
17514102784 reserved 8475983872
[ 1880.155049] BTRFS info (device dm-4): unable to make block group
1608583741440 ro
[ 1880.155050] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=54394880 min_allocable=1048576
[ 1880.155051] BTRFS info (device dm-4): space_info 4 has
18446744067071213568 free, is not full
[ 1880.155052] BTRFS info (device dm-4): space_info total=12884901888,
used=10509664256, pinned=0, reserved=131072, may_use=9013444608, readonly=0
[ 1880.155053] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155053] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155054] BTRFS info (device dm-4): chunk_block_rsv: size 393216
reserved 393216
[ 1880.155054] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155055] BTRFS info (device dm-4): delayed_refs_rsv: size
17515151360 reserved 8476639232
[ 1880.155080] BTRFS info (device dm-4): unable to make block group
1607509999616 ro
[ 1880.155081] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=78774272 min_allocable=1048576
[ 1880.155081] BTRFS info (device dm-4): space_info 4 has
18446744067071213568 free, is not full
[ 1880.155083] BTRFS info (device dm-4): space_info total=12884901888,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155083] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155084] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155085] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 1880.155086] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155087] BTRFS info (device dm-4): delayed_refs_rsv: size
17515937792 reserved 8476606464
[ 1880.155097] BTRFS info (device dm-4): unable to make block group
1607509999616 ro
[ 1880.155098] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=78774272 min_allocable=1048576
[ 1880.155098] BTRFS info (device dm-4): space_info 4 has
18446744068144955392 free, is not full
[ 1880.155099] BTRFS info (device dm-4): space_info total=13958643712,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155100] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155100] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155101] BTRFS info (device dm-4): chunk_block_rsv: size 393216
reserved 393216
[ 1880.155101] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155101] BTRFS info (device dm-4): delayed_refs_rsv: size
17516199936 reserved 8476606464
[ 1880.155106] BTRFS info (device dm-4): unable to make block group
1606436257792 ro
[ 1880.155106] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=183910400 min_allocable=1048576
[ 1880.155107] BTRFS info (device dm-4): space_info 4 has
18446744068144955392 free, is not full
[ 1880.155107] BTRFS info (device dm-4): space_info total=13958643712,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155108] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155108] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155109] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 1880.155109] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155110] BTRFS info (device dm-4): delayed_refs_rsv: size
17515937792 reserved 8476606464
[ 1880.155116] BTRFS info (device dm-4): unable to make block group
1606436257792 ro
[ 1880.155116] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=183910400 min_allocable=1048576
[ 1880.155117] BTRFS info (device dm-4): space_info 4 has
18446744069218697216 free, is not full
[ 1880.155118] BTRFS info (device dm-4): space_info total=15032385536,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155118] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155118] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155119] BTRFS info (device dm-4): chunk_block_rsv: size 393216
reserved 393216
[ 1880.155119] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155120] BTRFS info (device dm-4): delayed_refs_rsv: size
17516199936 reserved 8476606464
[ 1880.155123] BTRFS info (device dm-4): unable to make block group
1605362515968 ro
[ 1880.155124] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=14385152 min_allocable=1048576
[ 1880.155124] BTRFS info (device dm-4): space_info 4 has
18446744069218697216 free, is not full
[ 1880.155125] BTRFS info (device dm-4): space_info total=15032385536,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155126] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155126] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155126] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 1880.155127] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155127] BTRFS info (device dm-4): delayed_refs_rsv: size
17515937792 reserved 8476606464
[ 1880.155133] BTRFS info (device dm-4): unable to make block group
1605362515968 ro
[ 1880.155134] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=14385152 min_allocable=1048576
[ 1880.155134] BTRFS info (device dm-4): space_info 4 has
18446744070292439040 free, is not full
[ 1880.155135] BTRFS info (device dm-4): space_info total=16106127360,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155136] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155136] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155136] BTRFS info (device dm-4): chunk_block_rsv: size 393216
reserved 393216
[ 1880.155137] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155137] BTRFS info (device dm-4): delayed_refs_rsv: size
17516199936 reserved 8476606464
[ 1880.155141] BTRFS info (device dm-4): unable to make block group
1525872066560 ro
[ 1880.155141] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=303333376 min_allocable=1048576
[ 1880.155142] BTRFS info (device dm-4): space_info 4 has
18446744070292439040 free, is not full
[ 1880.155143] BTRFS info (device dm-4): space_info total=16106127360,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155143] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155144] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155144] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 1880.155144] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155145] BTRFS info (device dm-4): delayed_refs_rsv: size
17515937792 reserved 8476606464
[ 1880.155150] BTRFS info (device dm-4): unable to make block group
1525872066560 ro
[ 1880.155151] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=303333376 min_allocable=1048576
[ 1880.155151] BTRFS info (device dm-4): space_info 4 has
18446744071366180864 free, is not full
[ 1880.155152] BTRFS info (device dm-4): space_info total=17179869184,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155152] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155153] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155153] BTRFS info (device dm-4): chunk_block_rsv: size 393216
reserved 393216
[ 1880.155154] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155154] BTRFS info (device dm-4): delayed_refs_rsv: size
17516199936 reserved 8476606464
[ 1880.155157] BTRFS info (device dm-4): unable to make block group
1524798324736 ro
[ 1880.155158] BTRFS info (device dm-4): sinfo_used=19523239936
bg_num_bytes=105562112 min_allocable=1048576
[ 1880.155158] BTRFS info (device dm-4): space_info 4 has
18446744071366180864 free, is not full
[ 1880.155159] BTRFS info (device dm-4): space_info total=17179869184,
used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
[ 1880.155159] BTRFS info (device dm-4): global_block_rsv: size
536870912 reserved 536805376
[ 1880.155160] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
[ 1880.155160] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
[ 1880.155161] BTRFS info (device dm-4): delayed_block_rsv: size 0
reserved 0
[ 1880.155161] BTRFS info (device dm-4): delayed_refs_rsv: size
17515937792 reserved 8476606464
[ 1880.155167] BTRFS info (device dm-4): unable to make block group
1524798324736 ro
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-23 21:33 ` Peter Chant
@ 2019-09-23 21:46 ` Chris Murphy
2019-09-23 21:52 ` Chris Murphy
1 sibling, 0 replies; 14+ messages in thread
From: Chris Murphy @ 2019-09-23 21:46 UTC (permalink / raw)
To: Peter Chant; +Cc: Chris Murphy, Btrfs BTRFS
On Mon, Sep 23, 2019 at 3:34 PM Peter Chant <pete@petezilla.co.uk> wrote:
>
> On 9/23/19 7:55 PM, Pete wrote:
>
> > I'm not sure the balance is resolving anything. The filesystem has not
> > gone read only. I'll try an unfiltered balance now to see how that goes.
> >
>
> OK, result of unfiltered balance:
>
> root@phoenix:/var/lib/lxc# btrfs bal start /var/lib/lxc
> WARNING:
>
> Full balance without filters requested. This operation is very
> intense and takes potentially very long. It is recommended to
> use the balance filters to narrow down the scope of balance.
> Use 'btrfs balance start --full-balance' option to skip this
> warning. The operation will start in 10 seconds.
> Use Ctrl-C to stop it.
> 10 9 8 7 6 5 4 3 2 1
> Starting balance without any filters.
> ERROR: error during balancing '/var/lib/lxc': No space left on device
> There may be more info in syslog - try dmesg | tail
> root@phoenix:/var/lib/lxc#
>
> OK, here is the output in the last 100 lines of dmesg. The line 'unable
> to make block group nnnnnnnnnnnnnnn ro' is repeated, I presume that is
> relevant. I'm noting no other ill effects, if I'd not tried the balance
> I would not know anything is amiss.
>
> root@phoenix:/var/lib/lxc# dmesg | tail -n 100
> [ 1880.155026] BTRFS info (device dm-4): sinfo_used=19522584576
> bg_num_bytes=54394880 min_allocable=1048576
> [ 1880.155027] BTRFS info (device dm-4): space_info 4 has
> 18446744065997733888 free, is not full
> [ 1880.155029] BTRFS info (device dm-4): space_info total=11811160064,
> used=10509664256, pinned=0, reserved=131072, may_use=9013182464, readonly=0
> [ 1880.155029] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155030] BTRFS info (device dm-4): trans_block_rsv: size 262144
> reserved 262144
> [ 1880.155031] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
> [ 1880.155031] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155032] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17514102784 reserved 8475983872
> [ 1880.155049] BTRFS info (device dm-4): unable to make block group
> 1608583741440 ro
> [ 1880.155050] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=54394880 min_allocable=1048576
> [ 1880.155051] BTRFS info (device dm-4): space_info 4 has
> 18446744067071213568 free, is not full
> [ 1880.155052] BTRFS info (device dm-4): space_info total=12884901888,
> used=10509664256, pinned=0, reserved=131072, may_use=9013444608, readonly=0
> [ 1880.155053] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155053] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155054] BTRFS info (device dm-4): chunk_block_rsv: size 393216
> reserved 393216
> [ 1880.155054] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155055] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17515151360 reserved 8476639232
> [ 1880.155080] BTRFS info (device dm-4): unable to make block group
> 1607509999616 ro
> [ 1880.155081] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=78774272 min_allocable=1048576
> [ 1880.155081] BTRFS info (device dm-4): space_info 4 has
> 18446744067071213568 free, is not full
> [ 1880.155083] BTRFS info (device dm-4): space_info total=12884901888,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155083] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155084] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155085] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
> [ 1880.155086] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155087] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17515937792 reserved 8476606464
> [ 1880.155097] BTRFS info (device dm-4): unable to make block group
> 1607509999616 ro
> [ 1880.155098] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=78774272 min_allocable=1048576
> [ 1880.155098] BTRFS info (device dm-4): space_info 4 has
> 18446744068144955392 free, is not full
> [ 1880.155099] BTRFS info (device dm-4): space_info total=13958643712,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155100] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155100] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155101] BTRFS info (device dm-4): chunk_block_rsv: size 393216
> reserved 393216
> [ 1880.155101] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155101] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17516199936 reserved 8476606464
> [ 1880.155106] BTRFS info (device dm-4): unable to make block group
> 1606436257792 ro
> [ 1880.155106] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=183910400 min_allocable=1048576
> [ 1880.155107] BTRFS info (device dm-4): space_info 4 has
> 18446744068144955392 free, is not full
> [ 1880.155107] BTRFS info (device dm-4): space_info total=13958643712,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155108] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155108] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155109] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
> [ 1880.155109] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155110] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17515937792 reserved 8476606464
> [ 1880.155116] BTRFS info (device dm-4): unable to make block group
> 1606436257792 ro
> [ 1880.155116] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=183910400 min_allocable=1048576
> [ 1880.155117] BTRFS info (device dm-4): space_info 4 has
> 18446744069218697216 free, is not full
> [ 1880.155118] BTRFS info (device dm-4): space_info total=15032385536,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155118] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155118] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155119] BTRFS info (device dm-4): chunk_block_rsv: size 393216
> reserved 393216
> [ 1880.155119] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155120] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17516199936 reserved 8476606464
> [ 1880.155123] BTRFS info (device dm-4): unable to make block group
> 1605362515968 ro
> [ 1880.155124] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=14385152 min_allocable=1048576
> [ 1880.155124] BTRFS info (device dm-4): space_info 4 has
> 18446744069218697216 free, is not full
> [ 1880.155125] BTRFS info (device dm-4): space_info total=15032385536,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155126] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155126] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155126] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
> [ 1880.155127] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155127] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17515937792 reserved 8476606464
> [ 1880.155133] BTRFS info (device dm-4): unable to make block group
> 1605362515968 ro
> [ 1880.155134] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=14385152 min_allocable=1048576
> [ 1880.155134] BTRFS info (device dm-4): space_info 4 has
> 18446744070292439040 free, is not full
> [ 1880.155135] BTRFS info (device dm-4): space_info total=16106127360,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155136] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155136] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155136] BTRFS info (device dm-4): chunk_block_rsv: size 393216
> reserved 393216
> [ 1880.155137] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155137] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17516199936 reserved 8476606464
> [ 1880.155141] BTRFS info (device dm-4): unable to make block group
> 1525872066560 ro
> [ 1880.155141] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=303333376 min_allocable=1048576
> [ 1880.155142] BTRFS info (device dm-4): space_info 4 has
> 18446744070292439040 free, is not full
> [ 1880.155143] BTRFS info (device dm-4): space_info total=16106127360,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155143] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155144] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155144] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
> [ 1880.155144] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155145] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17515937792 reserved 8476606464
> [ 1880.155150] BTRFS info (device dm-4): unable to make block group
> 1525872066560 ro
> [ 1880.155151] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=303333376 min_allocable=1048576
> [ 1880.155151] BTRFS info (device dm-4): space_info 4 has
> 18446744071366180864 free, is not full
> [ 1880.155152] BTRFS info (device dm-4): space_info total=17179869184,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155152] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155153] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155153] BTRFS info (device dm-4): chunk_block_rsv: size 393216
> reserved 393216
> [ 1880.155154] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155154] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17516199936 reserved 8476606464
> [ 1880.155157] BTRFS info (device dm-4): unable to make block group
> 1524798324736 ro
> [ 1880.155158] BTRFS info (device dm-4): sinfo_used=19523239936
> bg_num_bytes=105562112 min_allocable=1048576
> [ 1880.155158] BTRFS info (device dm-4): space_info 4 has
> 18446744071366180864 free, is not full
> [ 1880.155159] BTRFS info (device dm-4): space_info total=17179869184,
> used=10509664256, pinned=0, reserved=163840, may_use=9013411840, readonly=0
> [ 1880.155159] BTRFS info (device dm-4): global_block_rsv: size
> 536870912 reserved 536805376
> [ 1880.155160] BTRFS info (device dm-4): trans_block_rsv: size 0 reserved 0
> [ 1880.155160] BTRFS info (device dm-4): chunk_block_rsv: size 0 reserved 0
> [ 1880.155161] BTRFS info (device dm-4): delayed_block_rsv: size 0
> reserved 0
> [ 1880.155161] BTRFS info (device dm-4): delayed_refs_rsv: size
> 17515937792 reserved 8476606464
> [ 1880.155167] BTRFS info (device dm-4): unable to make block group
> 1524798324736 ro
>
I've got the same problem on a brand new file system. This is a bug.
And it's not fixed in 5.3.0. It might even be a regression.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-23 21:33 ` Peter Chant
2019-09-23 21:46 ` Chris Murphy
@ 2019-09-23 21:52 ` Chris Murphy
2019-09-23 22:10 ` Pete
1 sibling, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2019-09-23 21:52 UTC (permalink / raw)
To: Peter Chant; +Cc: Btrfs BTRFS
What features do you have set?
# btrfs insp dump-s /dev/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-23 21:52 ` Chris Murphy
@ 2019-09-23 22:10 ` Pete
2019-09-23 23:10 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Pete @ 2019-09-23 22:10 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 9/23/19 10:52 PM, Chris Murphy wrote:
> What features do you have set?
>
> # btrfs insp dump-s /dev/
>
root@phoenix:/var/lib/lxc# btrfs insp dump-s /dev/nvme0_vg/lxc
superblock: bytenr=65536, device=/dev/nvme0_vg/lxc
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xae110928 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 6b0245ec-bdd4-4076-b800-2243d466b174
metadata_uuid 6b0245ec-bdd4-4076-b800-2243d466b174
label LXC_BTRFS
generation 548017
root 1957019598848
sys_array_size 97
chunk_root_generation 547983
root_level 1
chunk_root 1949798760448
chunk_root_level 0
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 354334801920
bytes_used 92528005120
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 2
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 548017
uuid_tree_generation 548017
dev_item.uuid 0766ea5f-6e0a-4946-a74c-aea3be78c087
dev_item.fsid 6b0245ec-bdd4-4076-b800-2243d466b174 [match]
dev_item.type 0
dev_item.total_bytes 268435456000
dev_item.bytes_used 100965285888
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
root@phoenix:/var/lib/lxc#
Just for good measure the device I added as a temporary measure to clear
enospc, I assume this would give the same result (but I am wrong, at
least in detail):
root@phoenix:/var/lib/lxc# btrfs insp dump-s /dev/nvme0_vg/tempdel
superblock: bytenr=65536, device=/dev/nvme0_vg/tempdel
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xb2dad359 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 6b0245ec-bdd4-4076-b800-2243d466b174
metadata_uuid 6b0245ec-bdd4-4076-b800-2243d466b174
label LXC_BTRFS
generation 548017
root 1957019598848
sys_array_size 97
chunk_root_generation 547983
root_level 1
chunk_root 1949798760448
chunk_root_level 0
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 354334801920
bytes_used 92528005120
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 2
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 548017
uuid_tree_generation 548017
dev_item.uuid 85b9c239-83ab-486b-b84c-f5d4336f913d
dev_item.fsid 6b0245ec-bdd4-4076-b800-2243d466b174 [match]
dev_item.type 0
dev_item.total_bytes 85899345920
dev_item.bytes_used 0
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 2
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
root@phoenix:/var/lib/lxc#
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-23 22:10 ` Pete
@ 2019-09-23 23:10 ` Chris Murphy
2019-09-23 23:21 ` Chris Murphy
2019-09-23 23:53 ` Pete
0 siblings, 2 replies; 14+ messages in thread
From: Chris Murphy @ 2019-09-23 23:10 UTC (permalink / raw)
To: Pete; +Cc: Btrfs BTRFS
On Mon, Sep 23, 2019 at 4:11 PM Pete <pete@petezilla.co.uk> wrote:
>
> On 9/23/19 10:52 PM, Chris Murphy wrote:
> > What features do you have set?
> >
> > # btrfs insp dump-s /dev/
> >
>
> root@phoenix:/var/lib/lxc# btrfs insp dump-s /dev/nvme0_vg/lxc
...
> sectorsize 4096
> nodesize 16384
...
> compat_ro_flags 0x0
> incompat_flags 0x161
> ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
Totally a default file system, looks like to me.
Bug 204973 - ERROR: error during balancing '/': No space left on device (edit)
https://bugzilla.kernel.org/show_bug.cgi?id=204973
Since I've reproduced it with all new progs and kernel I don't think
you need to add anything there.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-23 23:10 ` Chris Murphy
@ 2019-09-23 23:21 ` Chris Murphy
2019-09-23 23:53 ` Pete
1 sibling, 0 replies; 14+ messages in thread
From: Chris Murphy @ 2019-09-23 23:21 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS, David Sterba, Qu Wenruo, Josef Bacik
On Mon, Sep 23, 2019 at 5:10 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Mon, Sep 23, 2019 at 4:11 PM Pete <pete@petezilla.co.uk> wrote:
> >
> > On 9/23/19 10:52 PM, Chris Murphy wrote:
> > > What features do you have set?
> > >
> > > # btrfs insp dump-s /dev/
> > >
> >
> > root@phoenix:/var/lib/lxc# btrfs insp dump-s /dev/nvme0_vg/lxc
> ...
> > sectorsize 4096
> > nodesize 16384
> ...
> > compat_ro_flags 0x0
> > incompat_flags 0x161
> > ( MIXED_BACKREF |
> > BIG_METADATA |
> > EXTENDED_IREF |
> > SKINNY_METADATA )
>
> Totally a default file system, looks like to me.
>
> Bug 204973 - ERROR: error during balancing '/': No space left on device (edit)
> https://bugzilla.kernel.org/show_bug.cgi?id=204973
>
> Since I've reproduced it with all new progs and kernel I don't think
> you need to add anything there.
Guys, I'm not seeing anything queued up for 5.4 related to this so I'm
not sure whether it's a known problem or not. This enospc happens with
5.1.21 through 5.3.0. I reproduced it on a brand new file system about
2 weeks old that's only ever seen 5.2.9+ kernels, first hit the enospc
with 5.3.0. I've got three complicating features set compared to the
original poster above: space_cache=v2, no holes, compress=zstd but
since he's hitting this, I don't think it's related to those features.
It seems what's in common is this type of error:
[ 183.580410] BTRFS info (device sda4): unable to make block group 22020096 ro
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-23 23:10 ` Chris Murphy
2019-09-23 23:21 ` Chris Murphy
@ 2019-09-23 23:53 ` Pete
1 sibling, 0 replies; 14+ messages in thread
From: Pete @ 2019-09-23 23:53 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 9/24/19 12:10 AM, Chris Murphy wrote:
> Since I've reproduced it with all new progs and kernel I don't think
> you need to add anything there.
>
Thanks, appreciated.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-20 14:29 Balance ENOSPC during balance despite additional storage added Pete
2019-09-22 17:47 ` Chris Murphy
@ 2019-09-24 13:22 ` Josef Bacik
2019-09-24 19:42 ` Pete
1 sibling, 1 reply; 14+ messages in thread
From: Josef Bacik @ 2019-09-24 13:22 UTC (permalink / raw)
To: Pete; +Cc: Btrfs BTRFS
On Fri, Sep 20, 2019 at 03:29:32PM +0100, Pete wrote:
> I have a btrfs that is on top of an lvm logical volume on top of dm-crypt on
> a single nvme drive (Samsung 870 Pro 512GB).
>
> I added a second logical volume to give more space to get rid of ENOSPC
> errors during balance, but to no avail. This was after I started getting
> enospc during balance. Without this additional logical device, before
> balance, I had run out of space owning to some unfortunate scripting
> interacting with lxc snapshots (non btrfs backed in the config, so a copy)
> and some copying. I was performing a balance, following some deletions,
> when trying to get things back to a better state.
>
Just popping in to let you know I've been seeing this internally as well, I plan
to dig into it after we've run down the panic we're chasing currently. Thanks,
Josef
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Balance ENOSPC during balance despite additional storage added
2019-09-24 13:22 ` Josef Bacik
@ 2019-09-24 19:42 ` Pete
0 siblings, 0 replies; 14+ messages in thread
From: Pete @ 2019-09-24 19:42 UTC (permalink / raw)
To: Josef Bacik; +Cc: Btrfs BTRFS
On 9/24/19 2:22 PM, Josef Bacik wrote:
>
> Just popping in to let you know I've been seeing this internally as well, I plan
> to dig into it after we've run down the panic we're chasing currently. Thanks,
No problem. The only issue it seems to be causing is balance to fail.
Pete
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-09-24 19:43 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-20 14:29 Balance ENOSPC during balance despite additional storage added Pete
2019-09-22 17:47 ` Chris Murphy
2019-09-22 19:06 ` Pete
2019-09-22 21:25 ` Chris Murphy
2019-09-23 18:55 ` Pete
2019-09-23 21:33 ` Peter Chant
2019-09-23 21:46 ` Chris Murphy
2019-09-23 21:52 ` Chris Murphy
2019-09-23 22:10 ` Pete
2019-09-23 23:10 ` Chris Murphy
2019-09-23 23:21 ` Chris Murphy
2019-09-23 23:53 ` Pete
2019-09-24 13:22 ` Josef Bacik
2019-09-24 19:42 ` Pete
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).