* 100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour
@ 2019-12-06 2:04 Tomasz Chmielewski
2019-12-06 3:54 ` Chris Murphy
0 siblings, 1 reply; 5+ messages in thread
From: Tomasz Chmielewski @ 2019-12-06 2:04 UTC (permalink / raw)
To: linux-btrfs
I know that "df -h" is not to be trusted with btrfs, but some userspace
tools (i.e. apt, monitoring checks) rely on it.
Sometimes, "df -h" will report 100% disk usage on a btrfs filesystem,
while there is still space available and writes are possible.
Linux 5.4.1, LXD containers located in /data:
# df -h
/dev/nvme1n1 1000G 571G 0 100% /data
# btrfs fi usage /data
Overall:
Device size: 1000.00GiB
Device allocated: 580.02GiB
Device unallocated: 419.98GiB
Device missing: 0.00B
Used: 570.06GiB
Free (estimated): 429.45GiB (min: 429.45GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:578.01GiB, Used:568.54GiB
/dev/nvme1n1 578.01GiB
Metadata,single: Size:2.01GiB, Used:1.53GiB
/dev/nvme1n1 2.01GiB
System,single: Size:4.00MiB, Used:80.00KiB
/dev/nvme1n1 4.00MiB
Unallocated:
/dev/nvme1n1 419.98GiB
From container point of view, it looks like:
container# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1 1000G 571G 0 100% /
Let's see if we can still write? Yes we can:
container# dd if=/dev/urandom of=/bigfile.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.28306 s, 147 MB/s
container# sync
container# rm /bigfile.img
Let's try to use apt - unfortunately it says that there is not enough
space in /var/cache/apt/archives/:
container# apt install lzop
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
liblzo2-2
The following NEW packages will be installed:
liblzo2-2 lzop
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.5 kB of archives.
After this operation, 282 kB of additional disk space will be used.
E: You don't have enough free space in /var/cache/apt/archives/.
So let's try to write 1 GB to /var/cache/apt/archives/:
container# dd if=/dev/urandom of=/var/cache/apt/archives/bigfile.img
bs=1M count=1024
sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.66032 s, 140 MB/s
container# sync
container# rm /var/cache/apt/archives/bigfile.img
This IMO is a filesystem bug?
Some more info:
# btrfs fi df /data
Data, single: total=580.01GiB, used=569.78GiB
System, single: total=4.00MiB, used=80.00KiB
Metadata, single: total=2.01GiB, used=1.53GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /data
Label: 'data' uuid: 38fc3739-8695-49f1-bf14-946d0c0869c5
Total devices 1 FS bytes used 571.30GiB
devid 1 size 1000.00GiB used 582.02GiB path /dev/nvme1n1
Tomasz Chmielewski
https://lxadm.com
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour
2019-12-06 2:04 100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour Tomasz Chmielewski
@ 2019-12-06 3:54 ` Chris Murphy
2019-12-07 0:47 ` Tomasz Chmielewski
0 siblings, 1 reply; 5+ messages in thread
From: Chris Murphy @ 2019-12-06 3:54 UTC (permalink / raw)
To: Tomasz Chmielewski, Btrfs BTRFS
On Thu, Dec 5, 2019 at 7:04 PM Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>
> I know that "df -h" is not to be trusted with btrfs, but some userspace
> tools (i.e. apt, monitoring checks) rely on it.
>
> Sometimes, "df -h" will report 100% disk usage on a btrfs filesystem,
> while there is still space available and writes are possible.
>
> Linux 5.4.1, LXD containers located in /data:
>
> # df -h
> /dev/nvme1n1 1000G 571G 0 100% /data
>
> # btrfs fi usage /data
> Overall:
> Device size: 1000.00GiB
> Device allocated: 580.02GiB
> Device unallocated: 419.98GiB
> Device missing: 0.00B
> Used: 570.06GiB
> Free (estimated): 429.45GiB (min: 429.45GiB)
> Data ratio: 1.00
> Metadata ratio: 1.00
> Global reserve: 512.00MiB (used: 0.00B)
It does look like a bug. But I'm not sure if this is expected, I
thought such significant disagreements were long since fixed.
What version of coreutils?
Maybe attach a strace of df? (I'm not sure of the list attach size
limit but it's preferred, but something like a pastebin is OK also)
And maybe this will be useful for a dev?
# grep -r . /sys/fs/btrfs/<fs-uuid>/allocation/
You'll need to use the correct fs uuid for the btrfs in question.
> Data,single: Size:578.01GiB, Used:568.54GiB
> /dev/nvme1n1 578.01GiB
df seems to be treating only unallocated space as free, rather than
both unallocated and allocated that is unused. Color me surprised.
--
Chris Murphy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour
2019-12-06 3:54 ` Chris Murphy
@ 2019-12-07 0:47 ` Tomasz Chmielewski
2019-12-07 1:05 ` Chris Murphy
0 siblings, 1 reply; 5+ messages in thread
From: Tomasz Chmielewski @ 2019-12-07 0:47 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 2019-12-06 12:54, Chris Murphy wrote:
> What version of coreutils?
This is Ubuntu 18.04 LTS, with coreutils 8.28.
> Maybe attach a strace of df? (I'm not sure of the list attach size
> limit but it's preferred, but something like a pastebin is OK also)
"Unfortunately" I did run a balance, which "recovered" the space for df.
Tomasz
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour
2019-12-07 0:47 ` Tomasz Chmielewski
@ 2019-12-07 1:05 ` Chris Murphy
2019-12-07 1:54 ` Tomasz Chmielewski
0 siblings, 1 reply; 5+ messages in thread
From: Chris Murphy @ 2019-12-07 1:05 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: Chris Murphy, Btrfs BTRFS
On Fri, Dec 6, 2019 at 5:47 PM Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>
> On 2019-12-06 12:54, Chris Murphy wrote:
>
> > What version of coreutils?
>
> This is Ubuntu 18.04 LTS, with coreutils 8.28.
>
>
> > Maybe attach a strace of df? (I'm not sure of the list attach size
> > limit but it's preferred, but something like a pastebin is OK also)
>
> "Unfortunately" I did run a balance, which "recovered" the space for df.
Still an interesting data point.
--
Chris Murphy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour
2019-12-07 1:05 ` Chris Murphy
@ 2019-12-07 1:54 ` Tomasz Chmielewski
0 siblings, 0 replies; 5+ messages in thread
From: Tomasz Chmielewski @ 2019-12-07 1:54 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 2019-12-07 10:05, Chris Murphy wrote:
> On Fri, Dec 6, 2019 at 5:47 PM Tomasz Chmielewski <mangoo@wpkg.org>
> wrote:
>>
>> On 2019-12-06 12:54, Chris Murphy wrote:
>>
>> > What version of coreutils?
>>
>> This is Ubuntu 18.04 LTS, with coreutils 8.28.
>>
>>
>> > Maybe attach a strace of df? (I'm not sure of the list attach size
>> > limit but it's preferred, but something like a pastebin is OK also)
>>
>> "Unfortunately" I did run a balance, which "recovered" the space for
>> df.
>
> Still an interesting data point.
Well, there is some crap there... it went read-only.
[164625.770369] BTRFS info (device nvme1n1): found 683 extents
[164630.127647] BTRFS info (device nvme1n1): relocating block group
345766887424 flags data
[164650.186985] BTRFS info (device nvme1n1): found 4268 extents
[164658.075685] BTRFS info (device nvme1n1): found 4266 extents
[164663.979192] BTRFS info (device nvme1n1): relocating block group
344693145600 flags data
[164686.646073] BTRFS info (device nvme1n1): found 3721 extents
[164696.815655] BTRFS: error (device nvme1n1) in
btrfs_drop_snapshot:5402: errno=-4 unknown
[164696.821925] BTRFS info (device nvme1n1): forced readonly
[164696.834151] BTRFS warning (device nvme1n1): could not allocate space
for delete; will truncate on mount
[164696.834162] BTRFS info (device nvme1n1): balance: ended with status:
-4
[164697.468475] BTRFS info (device nvme1n1): delayed_refs has NO entry
Tomasz Chmielewski
https://lxadm.com
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-12-07 1:54 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-06 2:04 100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour Tomasz Chmielewski
2019-12-06 3:54 ` Chris Murphy
2019-12-07 0:47 ` Tomasz Chmielewski
2019-12-07 1:05 ` Chris Murphy
2019-12-07 1:54 ` Tomasz Chmielewski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).