Re: Filesystem goes readonly soon after mount, cannot free space or rebalance

[Chris Murphy <lists@colorremedies.com>]

On Sun, Jun 20, 2021 at 3:07 AM Asif Youssuff <yoasif@gmail.com> wrote:
>
> My complete mount options are:
> noatime,nodiratime,space_cache=v2,clear_cache,nospace_cache,skip_balance,commit=120,compress-force=zstd:9

space_cache=v2 is in conflict with nospace_cache and probably also
clear_cache; I'm not sure clear_cache can clear v2. Also once v2 is
used, there is a feature flag set on the super block so it always gets
used, and I'm not sure if nospace_cache means anything in the presence
of a free space tree feature flag.

Can you try to mount with just the skip_balance mount option, and show
dmesg from that point? I'd like to know how much time we have before
it goes read only and also what it's trying to do when it goes ro. And
make sure no user space programs write to this file system at all once
it's mounted.

Also, provide the output from "btrfs fi us".

There is a small chance it's possible to add four devices, and it
would need to be four, so that it can do what it's being very
insistent on, which is create a metadata block group. Since it's
raid1c4 profile, it needs to create one chunk on each of four devices.
If it works, that'd allow a filtered balance, to free up specific data
block groups and hopefully get enough space free to then remove those
four new devices.

The devices could be USB sticks, as terrible as that sounds. In theory
they can be RAM disks, but if you are put into a situation where you
*have to* reboot, or there's a crash or power failure, and can't
'btrfs dev remove' those devices first, you literally break the file
system, it'll be unfixable. So you do not want to use a ram disk. In a
bind you could use one disk containing four loop back mounted files,
but if you get really unlucky and that one disk has any issue
whatsoever, kaboom. Hence USB sticks, although if all four are in one
USB hub and the family turtle trips on its power cable...

There is also a small chance that freeing up space in metadata bg's by
clearing the v2 free space tree, but I'm reluctant to recommend it
right now. It's probably not going to free up that much space, and
currently it's pretty expensive to recreate on such a large file
system.

If you can mount with skip_balance and it does not go read only after
~5 minutes, try 'btrfs device add'. Just be aware that if the add
succeeds, that device is part of the file system now, until 'btrfs
device remove' succeeds. So they can't go missing or things get even
harder than they already are.

Also, consider backing up the most important data off this array in
case it's not repairable. Just in case. Right now it's going read-only
to prevent further confusion on disk. It's worth taking advantage of
it.

Last, consider updating btrfs-progs. Off hand I don't think you need
anything bug fix wise between 5.4 and 5.12, but 5.12.1 is current and
it'd be better to be closer to that, even though it's a low priority.

-- 
Chris Murphy