* snapshot space available
@ 2013-08-26 13:35 Russell Coker
2013-08-26 14:47 ` Clemens Eisserer
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Russell Coker @ 2013-08-26 13:35 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 4889 bytes --]
Linux xev 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07) x86_64 GNU/Linux
I've attached a file of script output from a system running the above Debian
kernel on a system with an Intel 120G SSD. To get this working again I
extended the filesystem in question to also use a small USB flash device (4G
from memory) and then deleted some old snapshots to free some space.
The most intereting part of the script output is pasted below. The system
reports that the filesystem is 82% full but that there is no space left.
root@xev:~# df -h /
Filesystem Size Used Avail Use%
Mounted on
/dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82%
/
root@xev:~# btrfs filesystem df /
Data: total=101.57GB, used=81.50GB
System, DUP: total=8.00MB, used=20.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=3.00GB, used=2.50GB
Metadata: total=8.00MB, used=0.00
root@xev:~# ls -l > test
bash: test: No space left on device
root@xev:~# touch test
touch: cannot touch ‘test’: No space left on device
root@xev:~#
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
[-- Attachment #2: problem.txt --]
[-- Type: text/plain, Size: 4082 bytes --]
Script started on Fri 23 Aug 2013 00:19:03 EST
root@xev:~# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82% /
root@xev:~# btrfs df /\b\b\b\b^[[1@f^[[1@i^[[1@l^[[1@e^[[1@s^[[1@y^[[1@s^[[1@t^[[1@e^[[1@m^[[1@
Data: total=101.57GB, used=81.50GB
System, DUP: total=8.00MB, used=20.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=3.00GB, used=2.50GB
Metadata: total=8.00MB, used=0.00
root@xev:~# dmesg|tail -50
[ 15.128069] e1000e 0000:00:19.0: irq 43 for MSI/MSI-X
[ 15.128149] IPv6: ADDRCONF(NETDEV_UP): mb0: link is not ready
[ 15.129817] device eth0 entered promiscuous mode
[ 15.145722] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 15.147178] device eth1 entered promiscuous mode
[ 15.149120] 8139too 0000:04:01.0 eth1: link up, 100Mbps, full-duplex, lpa 0x41E1
[ 15.153676] xenbr0: port 3(eth1) entered listening state
[ 15.153690] xenbr0: port 3(eth1) entered listening state
[ 15.153697] xenbr0: port 2(eth0) entered listening state
[ 15.153703] xenbr0: port 2(eth0) entered listening state
[ 15.153735] IPv6: ADDRCONF(NETDEV_UP): xenbr0: link is not ready
[ 15.660053] xenbr0: port 2(eth0) entered disabled state
[ 17.372874] e1000e: mb0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 17.373017] IPv6: ADDRCONF(NETDEV_CHANGE): mb0: link becomes ready
[ 17.373047] xenbr0: port 1(mb0) entered listening state
[ 17.373054] xenbr0: port 1(mb0) entered listening state
[ 18.160017] xenbr0: port 3(eth1) entered learning state
[ 20.376017] xenbr0: port 1(mb0) entered learning state
[ 21.168010] xenbr0: topology change detected, sending tcn bpdu
[ 21.168035] xenbr0: port 3(eth1) entered forwarding state
[ 21.168062] IPv6: ADDRCONF(NETDEV_CHANGE): xenbr0: link becomes ready
[ 23.384013] xenbr0: topology change detected, sending tcn bpdu
[ 23.384033] xenbr0: port 1(mb0) entered forwarding state
[ 23.805317] RPC: Registered named UNIX socket transport module.
[ 23.805321] RPC: Registered udp transport module.
[ 23.805323] RPC: Registered tcp transport module.
[ 23.805324] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 23.811142] FS-Cache: Loaded
[ 23.812150] Key type dns_resolver registered
[ 23.822368] FS-Cache: Netfs 'nfs' registered for caching
[ 23.838200] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 23.843440] SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
[ 24.342794] SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
[ 24.423047] auditd (2584): /proc/2584/oom_adj is deprecated, please use /proc/2584/oom_score_adj instead.
[ 24.579080] lp0: using parport0 (interrupt-driven).
[ 24.586862] ppdev: user-space parallel port driver
[ 27.275382] BTRFS debug (device dm-0): truncated 2 orphans
[ 62.198125] SELinux: initialized (dev proc, type proc), uses genfs_contexts
[ 580.759753] NFS: Registering the id_resolver key type
[ 580.759772] Key type id_resolver registered
[ 580.759774] Key type id_legacy registered
[ 595.768019] RPC: AUTH_GSS upcall timed out.
[ 595.768019] Please check user daemon is running.
[ 595.773813] SELinux: initialized (dev 0:27, type nfs4), uses genfs_contexts
[ 595.779200] SELinux: initialized (dev 0:28, type nfs4), uses genfs_contexts
[ 595.782249] SELinux: initialized (dev 0:29, type nfs4), uses genfs_contexts
[ 606.227371] SELinux: initialized (dev 0:27, type nfs4), uses genfs_contexts
[ 606.230441] SELinux: initialized (dev 0:28, type nfs4), uses genfs_contexts
[ 606.233676] SELinux: initialized (dev 0:30, type nfs4), uses genfs_contexts
[ 658.902288] BTRFS debug (device dm-0): unlinked 1 orphans
root@xev:~# ls -l > test
bash: test: No space left on device
root@xev:~# echo =n \b^[[K\b^[[K\b^[[Kn\b^[[K\b^[[K\b^[[K\b^[[K\b^[[K\b^[[Ktouch test
touch: cannot touch ‘test’: No space left on device
root@xev:~# exit
Script done on Fri 23 Aug 2013 00:19:51 EST
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: snapshot space available
2013-08-26 13:35 snapshot space available Russell Coker
@ 2013-08-26 14:47 ` Clemens Eisserer
2013-08-26 15:27 ` Chris Murphy
2013-08-26 15:15 ` Chris Murphy
[not found] ` < 6FCC5036-CC36-4DA3-8594-1FA23F3BB256@colorremedies.com>
2 siblings, 1 reply; 8+ messages in thread
From: Clemens Eisserer @ 2013-08-26 14:47 UTC (permalink / raw)
To: russell, linux-btrfs
You are out of metadata, not "normal space".
However, the good question is why 0.5GB of metadata are unused and
btrfs reports no space left.
I have seen similar behaviour on a machine of mine, with exactly 0.5GB
of metadata unused.
Regards
2013/8/26 Russell Coker <russell@coker.com.au>:
> Linux xev 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07) x86_64 GNU/Linux
>
> I've attached a file of script output from a system running the above Debian
> kernel on a system with an Intel 120G SSD. To get this working again I
> extended the filesystem in question to also use a small USB flash device (4G
> from memory) and then deleted some old snapshots to free some space.
>
> The most intereting part of the script output is pasted below. The system
> reports that the filesystem is 82% full but that there is no space left.
>
> root@xev:~# df -h /
> Filesystem Size Used Avail Use%
> Mounted on
> /dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82%
> /
> root@xev:~# btrfs filesystem df /
> Data: total=101.57GB, used=81.50GB
> System, DUP: total=8.00MB, used=20.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=3.00GB, used=2.50GB
> Metadata: total=8.00MB, used=0.00
> root@xev:~# ls -l > test
> bash: test: No space left on device
> root@xev:~# touch test
> touch: cannot touch ‘test’: No space left on device
> root@xev:~#
>
> --
> My Main Blog http://etbe.coker.com.au/
> My Documents Blog http://doc.coker.com.au/
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: snapshot space available
2013-08-26 13:35 snapshot space available Russell Coker
2013-08-26 14:47 ` Clemens Eisserer
@ 2013-08-26 15:15 ` Chris Murphy
[not found] ` < 6FCC5036-CC36-4DA3-8594-1FA23F3BB256@colorremedies.com>
2 siblings, 0 replies; 8+ messages in thread
From: Chris Murphy @ 2013-08-26 15:15 UTC (permalink / raw)
To: russell; +Cc: linux-btrfs
On Aug 26, 2013, at 7:35 AM, Russell Coker <russell@coker.com.au> wrote:
>
> root@xev:~# btrfs filesystem df /
> Data: total=101.57GB, used=81.50GB
> System, DUP: total=8.00MB, used=20.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=3.00GB, used=2.50GB
> Metadata: total=8.00MB, used=0.00
> root@xev:~# ls -l > test
> bash: test: No space left on device
> root@xev:~# touch test
> touch: cannot touch ‘test’: No space left on device
> root@xev:~#
The issue is data chunks remaining allocated for data, and not enough free space to allocate metadata chunks. Ultimately, the volume needs to be balanced.
I'm uncertain if autodefrag avoids this problem. It does seem like in certain instances, like this, the file system needs to be able to prune itself somehow, like a partial balance to consolidate data chunks and then release their space so they can become metadata chunks.
Chris Murphy
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: snapshot space available
2013-08-26 14:47 ` Clemens Eisserer
@ 2013-08-26 15:27 ` Chris Murphy
2013-08-26 19:09 ` Chris Murphy
0 siblings, 1 reply; 8+ messages in thread
From: Chris Murphy @ 2013-08-26 15:27 UTC (permalink / raw)
To: Clemens Eisserer; +Cc: russell, linux-btrfs
On Aug 26, 2013, at 8:47 AM, Clemens Eisserer <linuxhippy@gmail.com> wrote:
> You are out of metadata, not "normal space".
> However, the good question is why 0.5GB of metadata are unused and
> btrfs reports no space left.
> I have seen similar behaviour on a machine of mine, with exactly 0.5GB
> of metadata unused.
Since metadata is set to DUP, the numbers are actually 6GB allocated and 5GB used, and I'm guessing that at the time it starts to use the last chunk of allocated space, it also wants to allocate a new one. But it can't allocate more for metadata because all free space in the fs is already allocated for data chunks.
Presently btrfs seems unable to unwind itself from this situation without a manual balance, which in many instances also requires more free space than is available once you're in this situation. So I called it a filesystem wedged into a corner. Fortunately so far I've found it to be rather docile in accepting a small device to give it a little extra free space, do a balance, and then delete the device from the volume.
I haven't tried doing a partial balance, deleting the extra device, then finishing the balance.
But this suggestion is certainly non-obvious to the mortal and sane. Especially if you do a df, which bases its values off the used portion of allocated chunks, the file system is distinctly not out of free space. So the confusion is understandable.
Chris Murphy
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: snapshot space available
2013-08-26 15:27 ` Chris Murphy
@ 2013-08-26 19:09 ` Chris Murphy
2013-08-31 9:04 ` Russell Coker
0 siblings, 1 reply; 8+ messages in thread
From: Chris Murphy @ 2013-08-26 19:09 UTC (permalink / raw)
To: Clemens Eisserer; +Cc: russell, linux-btrfs
On Aug 26, 2013, at 9:27 AM, Chris Murphy <lists@colorremedies.com> wrote:
> I haven't tried doing a partial balance, deleting the extra device, then finishing the balance.
Btrfs FAQ suggests a partial balance for this situation. So it's worth a try before adding a device.
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
Chris Murphy
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: snapshot space available
[not found] ` < 6FCC5036-CC36-4DA3-8594-1FA23F3BB256@colorremedies.com>
@ 2013-08-27 6:29 ` Duncan
0 siblings, 0 replies; 8+ messages in thread
From: Duncan @ 2013-08-27 6:29 UTC (permalink / raw)
To: linux-btrfs
Chris Murphy posted on Mon, 26 Aug 2013 09:15:46 -0600 as excerpted:
> I'm uncertain if autodefrag avoids this problem. It does seem like in
> certain instances, like this, the file system needs to be able to prune
> itself somehow, like a partial balance to consolidate data chunks and
> then release their space so they can become metadata chunks.
Intriguing question re defrag. I hadn't thought of the possibility until
you suggested it, but indeed, tracking hundreds of extents instead of one
or a dozen, does sound like it could reduce metadata usage, reducing the
chance of running into the issue in the first place as well as lessening
the chance of a "simple" delete temporarily requiring significant new
metadata resources in ordered to track all those extent frees before the
final atomic root entry update.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: snapshot space available
2013-08-26 19:09 ` Chris Murphy
@ 2013-08-31 9:04 ` Russell Coker
2013-08-31 17:27 ` Chris Murphy
0 siblings, 1 reply; 8+ messages in thread
From: Russell Coker @ 2013-08-31 9:04 UTC (permalink / raw)
To: Chris Murphy, Clemens Eisserer; +Cc: linux-btrfs
How would I determine when this situation is approaching? On all important systems I have cron jobs and nagios checks to determine via df when thw file system is about to run out of space. How do I do the same for this condition?
Chris Murphy <lists@colorremedies.com> wrote:
>
>On Aug 26, 2013, at 9:27 AM, Chris Murphy <lists@colorremedies.com>
>wrote:
>
>> I haven't tried doing a partial balance, deleting the extra device,
>then finishing the balance.
>
>Btrfs FAQ suggests a partial balance for this situation. So it's worth
>a try before adding a device.
>
>https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
>
>
>Chris Murphy
--
Sent from my Samsung Galaxy Note 2 with K-9 Mail.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: snapshot space available
2013-08-31 9:04 ` Russell Coker
@ 2013-08-31 17:27 ` Chris Murphy
0 siblings, 0 replies; 8+ messages in thread
From: Chris Murphy @ 2013-08-31 17:27 UTC (permalink / raw)
To: Russell Coker; +Cc: Clemens Eisserer, linux-btrfs
On Aug 31, 2013, at 3:04 AM, Russell Coker <russell@coker.com.au> wrote:
> How would I determine when this situation is approaching? On all important systems I have cron jobs and nagios checks to determine via df when thw file system is about to run out of space. How do I do the same for this condition?
I don't think it's possible to infer this condition from a single command.
However, if you look at only 'btrfs fi show', a small difference between size and used could be used as a trigger for a partial balance operation to avoid the condition.
What's a small difference? I'd say at least 2GB, or the size of the maximum expected file size + 2GB. That way a single write operation doesn't end up causing all free space to be allocated for data, leaving no more free space to be allocated for metadata.
Chris Murphy
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-08-31 17:27 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-08-26 13:35 snapshot space available Russell Coker
2013-08-26 14:47 ` Clemens Eisserer
2013-08-26 15:27 ` Chris Murphy
2013-08-26 19:09 ` Chris Murphy
2013-08-31 9:04 ` Russell Coker
2013-08-31 17:27 ` Chris Murphy
2013-08-26 15:15 ` Chris Murphy
[not found] ` < 6FCC5036-CC36-4DA3-8594-1FA23F3BB256@colorremedies.com>
2013-08-27 6:29 ` Duncan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.