Filesystem unable to recover from ENOSPC

Filesystem unable to recover from ENOSPC

[Chip Turner]

I have a filesystem that I can't seem to resolve ENOSPC issues.  No
write operation can succeed; I've tried the wiki's suggestions
(balancing, which fails because of ENOSPC, mounting with nodatacow,
clear_cache, nospace_cache, enospc_debug, truncating files, deleting
files, briefly microwaving the drive, etc).

btrfs show:
Label: none  uuid: 04283a32-b388-480b-9949-686675fad7df
Total devices 1 FS bytes used 135.58GiB
devid    1 size 238.22GiB used 238.22GiB path /dev/sdb2

btrfs fi df:
Data, single: total=234.21GiB, used=131.82GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=4.01GiB, used=3.76GiB

So, the filesystem is pretty much unusable, and I can find no way to
resuscitate it.  I ended up in this state by creating a snapshot of
the root of the fs into a read/write subvolume, which I wanted to
become my new root, then began deleting entries in the filesystem
itself outside of the new snapshot.  So nothing particularly weird or
crazy.  The only oddness is the file count -- I have a *lot* of
hardlinked files (this is an rsnapshot volume, so it has a large
number of files and many of them are hard linked).

It seems like the normal solution is btrfs balance, but that fails.
defragment also fails.  Kernel is 3.13.

Is there anything else I can or should do, or just wipe it and
recreate with perhaps better initial defaults?  If this kind of thing
is unavoidable, how might I have anticipated it and prevented it?
Fortunately this was a migration effort and so my original data is
safe inside of ext4 (ew).

-- 
Chip Turner - cturner@pattern.net

Re: Filesystem unable to recover from ENOSPC

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]

On Thu, Apr 10, 2014 at 01:00:35PM -0700, Chip Turner wrote:
> I have a filesystem that I can't seem to resolve ENOSPC issues.  No
> write operation can succeed; I've tried the wiki's suggestions
> (balancing, which fails because of ENOSPC, mounting with nodatacow,
> clear_cache, nospace_cache, enospc_debug, truncating files, deleting
> files, briefly microwaving the drive, etc).
> 
> btrfs show:
> Label: none  uuid: 04283a32-b388-480b-9949-686675fad7df
> Total devices 1 FS bytes used 135.58GiB
> devid    1 size 238.22GiB used 238.22GiB path /dev/sdb2
> 
> btrfs fi df:
> Data, single: total=234.21GiB, used=131.82GiB
> System, single: total=4.00MiB, used=48.00KiB
> Metadata, single: total=4.01GiB, used=3.76GiB

   I'm surprised it's managed to use that much of the metadata
allocation. The FS usually hits this problem with a much smaller
used-to-total ratio.

> So, the filesystem is pretty much unusable, and I can find no way to
> resuscitate it.  I ended up in this state by creating a snapshot of
> the root of the fs into a read/write subvolume, which I wanted to
> become my new root, then began deleting entries in the filesystem
> itself outside of the new snapshot.  So nothing particularly weird or
> crazy.  The only oddness is the file count -- I have a *lot* of
> hardlinked files (this is an rsnapshot volume, so it has a large
> number of files and many of them are hard linked).
> 
> It seems like the normal solution is btrfs balance, but that fails.
> defragment also fails.  Kernel is 3.13.
> 
> Is there anything else I can or should do, or just wipe it and
> recreate with perhaps better initial defaults?  If this kind of thing
> is unavoidable, how might I have anticipated it and prevented it?
> Fortunately this was a migration effort and so my original data is
> safe inside of ext4 (ew).

   One thing you could do is btrfs dev add a small new device to the
filesystem (say, a USB stick, or a 4 GiB loopback file mounted over
NBD or something). Then run the filtered balance. Then btrfs dev del
the spare device.

   The fact that this FS has ended up in this state should be
considered a bug. It used to be quite common, then josef fixed a load
of problems, and it's been rare for about a year. Only recently we've
been seeing more of this kind of problem, and I think there's been a
bit of a regression somewhere.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
            --- "Can I offer you anything? Tea? Seedcake? ---            
                         Glass of Amontillado?"                          

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Filesystem unable to recover from ENOSPC

[Chip Turner]

On Thu, Apr 10, 2014 at 1:34 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Thu, Apr 10, 2014 at 01:00:35PM -0700, Chip Turner wrote:
>> btrfs show:
>> Label: none  uuid: 04283a32-b388-480b-9949-686675fad7df
>> Total devices 1 FS bytes used 135.58GiB
>> devid    1 size 238.22GiB used 238.22GiB path /dev/sdb2
>>
>> btrfs fi df:
>> Data, single: total=234.21GiB, used=131.82GiB
>> System, single: total=4.00MiB, used=48.00KiB
>> Metadata, single: total=4.01GiB, used=3.76GiB
>
>    I'm surprised it's managed to use that much of the metadata
> allocation. The FS usually hits this problem with a much smaller
> used-to-total ratio.

It perhaps was related to how I build it -- basically I dumped many
files (around 8.5 million, 95% of which are hardlinks rather than
actual files) onto it at once, created a snapshot, and then began
deleting files.  The btrfs filesystem they were coming from was fine
with the files and its own original load, but something about the
snapshotting and deleting seemed to push it over the edge.

The issue happened over a couple of days so I don't recall exactly,
but I may also have run out of space and had to delete a lot of files
(basically, I left off the '-H' to rsync, so, when I copied to the
system, it made copies rather than hard links).  This occurred before
the original full copy and hence before the snapshot and deletion.

>    One thing you could do is btrfs dev add a small new device to the
> filesystem (say, a USB stick, or a 4 GiB loopback file mounted over
> NBD or something). Then run the filtered balance. Then btrfs dev del
> the spare device.

Ah, this worked great.  It fixed it in about ten seconds.

I'm curious about the space report; why doesn't Data+System+Metadata
add up to the total space used on the device?  Was the fs stuck in a
state where it couldn't clean up because it couldn't write more
metadata (and hence adding a few gb allowed it to balance)?  After the
balance, the used space dropped to around 150GB, roughly what I'd
expect.

>    The fact that this FS has ended up in this state should be
> considered a bug. It used to be quite common, then josef fixed a load
> of problems, and it's been rare for about a year. Only recently we've
> been seeing more of this kind of problem, and I think there's been a
> bit of a regression somewhere.

Is there any other data I can provide to help fix this?  I can see if
it's reproducible (I'm migrating disks so I can just keep doing it and
see if it happens again).

Chip

-- 
Chip Turner - cturner@pattern.net

Re: Filesystem unable to recover from ENOSPC

[Duncan]

Chip Turner posted on Thu, 10 Apr 2014 15:40:22 -0700 as excerpted:

> On Thu, Apr 10, 2014 at 1:34 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>> On Thu, Apr 10, 2014 at 01:00:35PM -0700, Chip Turner wrote:
>>> btrfs show:
>>> Label: none  uuid: 04283a32-b388-480b-9949-686675fad7df
>>> Total devices 1 FS bytes used 135.58GiB
>>> devid    1 size 238.22GiB used 238.22GiB path /dev/sdb2
>>>
>>> btrfs fi df:
>>> Data, single: total=234.21GiB, used=131.82GiB
>>> System, single: total=4.00MiB, used=48.00KiB
>>> Metadata, single: total=4.01GiB, used=3.76GiB

[Tried all the usual tricks, didn't work.]

>> One thing you could do is btrfs dev add a small new device to the
>> filesystem (say, a USB stick, or a 4 GiB loopback file mounted over NBD
>> or something). Then run the filtered balance. Then btrfs dev del the
>> spare device.
> 
> Ah, this worked great.  It fixed it in about ten seconds.
> 
> I'm curious about the space report; why doesn't Data+System+Metadata add
> up to the total space used on the device?

Actually, it does... *IF* you know how to read it.  Unfortunately that's 
a *BIG* *IF*, because btrfs show very confusingly reports two very 
different numbers using very similar wording, without making *AT* *ALL* 
clear what it's actually reporting.

Try this:

Add up the df totals (which is the space allocated for each category 
type). 234.21 gig, 4.01 gig, 4 meg.  238 gig and change, correct?  Look 
at the show output.  What number does that look like there?

Now do the same with the df used (which is the space used out of that 
allocated).  131.82 gig, 3.76 gig, (insubstantial).  135 gig and change.  
What number from btrfs show does /that/ look like?

Here's what's happening and how to read those numbers.  Btrfs uses space 
in two stages.

First it on-demand allocates chunks dedicated to the usage type.  Data 
chunks are 1 GiB in size.  Metadata chunks are 256 MiB in size, a quarter 
the size of a data chunk, altho by default on a single device they are 
allocated in pairs, dup mode, so 512 MiB at a time (half a data-chunk), 
tho I see your metadata is single mode so it's still only 256 MiB at a 
time.

The space used by these ALLOCATED chunks appears as the totals in btrfs 
filesystem df and as used in btrfs filesystem show for individual 
devices, but the show total line comes from somewhere else *ENTIRELY*, 
which is why the reported individual device used number sum up to (if 
there's more than one device, the individual device numbers can be added 
together, if it's just one, that's it) so much larger than the number 
reported by show as total used.

That metadata-single, BTW, probably explains Hugo's observation, that you 
were able to use more of your metadata than most, because you're running 
single metadata mode instead of the more usual dup.  (Either you set it 
up that way, or mkfs.btrfs detected SSD, in which case it defaults to 
single metadata for a single device filesystem.)  So you were able to get 
closer to full metadata usage.  (Btrfs reserves some metadata, typically 
about a chunk, which means about two chunks in dup mode, for its own 
usage.  That's never usable so it always looks like you have a bit more 
free metadata space than you actually do.  But as long as there's 
unallocated free space to allocate additional metadata blocks from, it 
doesn't matter.  Only when all space is allocated does it matter, since 
then it still looks like you have free metadata space to use when you 
don't.)

Anyway, once btrfs has a chunk of the appropriate type, it fills it up.  
When necessary, it'll try to allocate another chunk.

The actual usage of the already allocated chunks appears in btrfs 
filesystem df as used, with the total of all types for all devices also 
appearing in btrfs filesystem show on the total used line.

So data+metadata+system allocated as reported by df, adds up to the 
totals reported as used by show for all the individual devices, added 
together.

And data+metadata+system actually used (out of the allocated) as reported 
by df, adds up to the total reported by show as used, on the total used 
line.

But they are two very different numbers, one total chunks allocated, the 
other the total used OF those allocated chunks.  Makes sense *IF* *YOU* 
*KNOW* *HOW* *TO* *READ* *IT*, but otherwise, it's *ENTIRELY* misleading 
and confusing!

There has already been discussion and proposed patches for adding more 
detail to df and show, with the wording changed up as well, and I sort of 
expected to see that in btrfs-progs v3.14 when it came out altho I'm 
running it now and don't see a change, but FWIW, from the posted examples 
at least, I couldn't quite figure out the proposed new output either, so 
it might not be that much better than what we have.  Which might or might 
not have anything to do with it not appearing in v3.14 as I expected. 
Meanwhile, now that I actually know how to read the current output, it 
does provide the needed information, even if the output /is/ rather 
confusing to newbies.

Back to btrfs behavior and how it leads to nospc errors, however...

When btrfs deletes files, it frees space in the corresponding chunks, but 
since individual files normally use a lot more data space than metadata, 
data chunks get emptied faster than the corresponding metadata chunks.

But here's the problem.  Btrfs can automatically free space back to the 
allocated chunks as files get deleted, but it does *NOT* (yet) know how 
to automatically deallocate those now empty or mostly empty chunks, 
returning them to the unallocated pool so they can be reused as another 
chunk-type if necessary.

So btrfs uses space up in two stages, but can only automatically return 
unused space back in one stage, not the other.  Currently, to deallocate 
and free those unused blocks, you must run balance (which is where the 
filtered balance -dusage=20 or whatever, to balance, in that case only 
data chunks with 20% usage or less), which rewrites those blocks and 
consolidates any remaining usage as it goes, freeing up the blocks it 
empties back to the unallocated pool.

At some point the devs plan to automate the process, probably by 
automatically triggering a balance start -dusage=5 or balance start 
-musage=5, or whatever, as necessary.  But that hasn't happened yet.  
Which is why admins must currently keep an eye on things and run that 
balance manually (or hack up some sort of script to do it automatically, 
themselves) when necessary.

> Was the fs stuck in a state
> where it couldn't clean up because it couldn't write more metadata (and
> hence adding a few gb allowed it to balance)?

Basically, yes.  As you can see from the individual device line in the 
above show output, 238.22 gig used (that is, chunks allocated), of 238.22 
filesystem size.  There's no room left to allocate additional chunks, not 
even one in ordered to rewrite the remaining data from some of those 
mostly empty data chunks, in ordered to return them to the unallocated 
pool.

With a bit of luck, you would have had at least one entirely empty data 
chunk, in which case a balance start -dusage=0 would have freed it (since 
it was entirely empty and thus there was nothing to rewrite to a new 
chunk), thus giving you enough space to actually allocate a new chunk, to 
write into and free more of them.  But if you tried a balance start
-dusage=0 and it couldn't find even one entirely empty data chunk to 
free, as apparently you did, then you had a problem, since all available 
space was already allocated.

Temporarily adding another device gave it enough room to allocate a few 
new chunks, such that balance then had enough space to rewrite a few of 
the mostly empty chunks, thereby freeing enough space so you could then 
btrfs device delete the new device, rewriting those new chunks back to 
the newly deallocated space on the original device.

> After the balance, the
> used space dropped to around 150GB, roughly what I'd expect.

=:^)


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Filesystem unable to recover from ENOSPC

[Chip Turner]

Thank you for the extremely detailed and helpful reply.  I now
understand what was happening.  To me, when I read "total=" I guess I
thought that was capacity rather than allocated (but now "holey")
chunks.  I agree that perhaps adjusting the phrasing of the output of
df and show would be helpful in making this clearer (or perhaps some
adjustment to the wiki; maybe I will do that).

I assume metadata is where filesystem information (directory entries,
inodes, etc) are stored and not being able to delete was because there
wasn't room in the metadata chunks, and no room to allocate more
metadata chunks?  Or are those also data chunks and it was purely
about data chunks?

Does defrag do similar reallocation to balance such that it would help
combat this behavior?

Thanks again,
Chip

On Thu, Apr 10, 2014 at 7:09 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Chip Turner posted on Thu, 10 Apr 2014 15:40:22 -0700 as excerpted:
>
>> On Thu, Apr 10, 2014 at 1:34 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>>> On Thu, Apr 10, 2014 at 01:00:35PM -0700, Chip Turner wrote:
>>>> btrfs show:
>>>> Label: none  uuid: 04283a32-b388-480b-9949-686675fad7df
>>>> Total devices 1 FS bytes used 135.58GiB
>>>> devid    1 size 238.22GiB used 238.22GiB path /dev/sdb2
>>>>
>>>> btrfs fi df:
>>>> Data, single: total=234.21GiB, used=131.82GiB
>>>> System, single: total=4.00MiB, used=48.00KiB
>>>> Metadata, single: total=4.01GiB, used=3.76GiB
>
> [Tried all the usual tricks, didn't work.]
>
>>> One thing you could do is btrfs dev add a small new device to the
>>> filesystem (say, a USB stick, or a 4 GiB loopback file mounted over NBD
>>> or something). Then run the filtered balance. Then btrfs dev del the
>>> spare device.
>>
>> Ah, this worked great.  It fixed it in about ten seconds.
>>
>> I'm curious about the space report; why doesn't Data+System+Metadata add
>> up to the total space used on the device?
>
> Actually, it does... *IF* you know how to read it.  Unfortunately that's
> a *BIG* *IF*, because btrfs show very confusingly reports two very
> different numbers using very similar wording, without making *AT* *ALL*
> clear what it's actually reporting.
>
> Try this:
>
> Add up the df totals (which is the space allocated for each category
> type). 234.21 gig, 4.01 gig, 4 meg.  238 gig and change, correct?  Look
> at the show output.  What number does that look like there?
>
> Now do the same with the df used (which is the space used out of that
> allocated).  131.82 gig, 3.76 gig, (insubstantial).  135 gig and change.
> What number from btrfs show does /that/ look like?
>
> Here's what's happening and how to read those numbers.  Btrfs uses space
> in two stages.
>
> First it on-demand allocates chunks dedicated to the usage type.  Data
> chunks are 1 GiB in size.  Metadata chunks are 256 MiB in size, a quarter
> the size of a data chunk, altho by default on a single device they are
> allocated in pairs, dup mode, so 512 MiB at a time (half a data-chunk),
> tho I see your metadata is single mode so it's still only 256 MiB at a
> time.
>
> The space used by these ALLOCATED chunks appears as the totals in btrfs
> filesystem df and as used in btrfs filesystem show for individual
> devices, but the show total line comes from somewhere else *ENTIRELY*,
> which is why the reported individual device used number sum up to (if
> there's more than one device, the individual device numbers can be added
> together, if it's just one, that's it) so much larger than the number
> reported by show as total used.
>
> That metadata-single, BTW, probably explains Hugo's observation, that you
> were able to use more of your metadata than most, because you're running
> single metadata mode instead of the more usual dup.  (Either you set it
> up that way, or mkfs.btrfs detected SSD, in which case it defaults to
> single metadata for a single device filesystem.)  So you were able to get
> closer to full metadata usage.  (Btrfs reserves some metadata, typically
> about a chunk, which means about two chunks in dup mode, for its own
> usage.  That's never usable so it always looks like you have a bit more
> free metadata space than you actually do.  But as long as there's
> unallocated free space to allocate additional metadata blocks from, it
> doesn't matter.  Only when all space is allocated does it matter, since
> then it still looks like you have free metadata space to use when you
> don't.)
>
> Anyway, once btrfs has a chunk of the appropriate type, it fills it up.
> When necessary, it'll try to allocate another chunk.
>
> The actual usage of the already allocated chunks appears in btrfs
> filesystem df as used, with the total of all types for all devices also
> appearing in btrfs filesystem show on the total used line.
>
> So data+metadata+system allocated as reported by df, adds up to the
> totals reported as used by show for all the individual devices, added
> together.
>
> And data+metadata+system actually used (out of the allocated) as reported
> by df, adds up to the total reported by show as used, on the total used
> line.
>
> But they are two very different numbers, one total chunks allocated, the
> other the total used OF those allocated chunks.  Makes sense *IF* *YOU*
> *KNOW* *HOW* *TO* *READ* *IT*, but otherwise, it's *ENTIRELY* misleading
> and confusing!
>
> There has already been discussion and proposed patches for adding more
> detail to df and show, with the wording changed up as well, and I sort of
> expected to see that in btrfs-progs v3.14 when it came out altho I'm
> running it now and don't see a change, but FWIW, from the posted examples
> at least, I couldn't quite figure out the proposed new output either, so
> it might not be that much better than what we have.  Which might or might
> not have anything to do with it not appearing in v3.14 as I expected.
> Meanwhile, now that I actually know how to read the current output, it
> does provide the needed information, even if the output /is/ rather
> confusing to newbies.
>
> Back to btrfs behavior and how it leads to nospc errors, however...
>
> When btrfs deletes files, it frees space in the corresponding chunks, but
> since individual files normally use a lot more data space than metadata,
> data chunks get emptied faster than the corresponding metadata chunks.
>
> But here's the problem.  Btrfs can automatically free space back to the
> allocated chunks as files get deleted, but it does *NOT* (yet) know how
> to automatically deallocate those now empty or mostly empty chunks,
> returning them to the unallocated pool so they can be reused as another
> chunk-type if necessary.
>
> So btrfs uses space up in two stages, but can only automatically return
> unused space back in one stage, not the other.  Currently, to deallocate
> and free those unused blocks, you must run balance (which is where the
> filtered balance -dusage=20 or whatever, to balance, in that case only
> data chunks with 20% usage or less), which rewrites those blocks and
> consolidates any remaining usage as it goes, freeing up the blocks it
> empties back to the unallocated pool.
>
> At some point the devs plan to automate the process, probably by
> automatically triggering a balance start -dusage=5 or balance start
> -musage=5, or whatever, as necessary.  But that hasn't happened yet.
> Which is why admins must currently keep an eye on things and run that
> balance manually (or hack up some sort of script to do it automatically,
> themselves) when necessary.
>
>> Was the fs stuck in a state
>> where it couldn't clean up because it couldn't write more metadata (and
>> hence adding a few gb allowed it to balance)?
>
> Basically, yes.  As you can see from the individual device line in the
> above show output, 238.22 gig used (that is, chunks allocated), of 238.22
> filesystem size.  There's no room left to allocate additional chunks, not
> even one in ordered to rewrite the remaining data from some of those
> mostly empty data chunks, in ordered to return them to the unallocated
> pool.
>
> With a bit of luck, you would have had at least one entirely empty data
> chunk, in which case a balance start -dusage=0 would have freed it (since
> it was entirely empty and thus there was nothing to rewrite to a new
> chunk), thus giving you enough space to actually allocate a new chunk, to
> write into and free more of them.  But if you tried a balance start
> -dusage=0 and it couldn't find even one entirely empty data chunk to
> free, as apparently you did, then you had a problem, since all available
> space was already allocated.
>
> Temporarily adding another device gave it enough room to allocate a few
> new chunks, such that balance then had enough space to rewrite a few of
> the mostly empty chunks, thereby freeing enough space so you could then
> btrfs device delete the new device, rewriting those new chunks back to
> the newly deallocated space on the original device.
>
>> After the balance, the
>> used space dropped to around 150GB, roughly what I'd expect.
>
> =:^)
>
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Chip Turner - cturner@pattern.net

Re: Filesystem unable to recover from ENOSPC

[Liu Bo]

On Thu, Apr 10, 2014 at 11:10:33PM -0700, Chip Turner wrote:
> Thank you for the extremely detailed and helpful reply.  I now
> understand what was happening.  To me, when I read "total=" I guess I
> thought that was capacity rather than allocated (but now "holey")
> chunks.  I agree that perhaps adjusting the phrasing of the output of
> df and show would be helpful in making this clearer (or perhaps some
> adjustment to the wiki; maybe I will do that).
> 
> I assume metadata is where filesystem information (directory entries,
> inodes, etc) are stored and not being able to delete was because there
> wasn't room in the metadata chunks, and no room to allocate more
> metadata chunks?  Or are those also data chunks and it was purely
> about data chunks?

Yes and no.  Deleting metadata also needs metadata, and metadata chunk and data
chunk are different ones.

> 
> Does defrag do similar reallocation to balance such that it would help
> combat this behavior?

I'm afraid it won't, since you have that many hardlinks, defrag may not help on
reducing metadata.

thanks,
-liubo