From: Giovanni Biscuolo <g@xelera.eu>
To: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: how to recover from "enospc errors during balance"
Date: Thu, 01 Oct 2020 10:56:15 +0200 [thread overview]
Message-ID: <878scq1g0g.fsf@roquette.i-did-not-set--mail-host-address--so-tickle-me> (raw)
In-Reply-To: <20200930000417.GH5890@hungrycats.org>
[-- Attachment #1: Type: text/plain, Size: 4223 bytes --]
Hello Zygo,
thank you for your help!
...but I still cannot mount the filesystem RW, see below.
Zygo Blaxell <ce3g8jdj@umail.furryterror.org> writes:
> On Tue, Sep 29, 2020 at 04:25:06PM +0200, Giovanni Biscuolo wrote:
[...]
>> [6928066.755704] BTRFS info (device sda3): balance: start -dusage=50 -musage=70 -susage=70
>
> Never balance metadata on a schedule. If it is done too often, and the
> disk fills up, it will eventually lead to ENOSPC errors that are hard
> to get out of...
OK I got it: I'll fix it as soon as I'll get to remount the (root)
filesystem.
I was using an option I did not fully understand and I was not able to
find such a warning in the documentation.
[...]
>> I tried to add a new device (I have 2 spare disks) but it does not work
>> with a read-only filesystem.
>>
>> Please how can I remount the filesystem read-write and free some space
>> deleting some files?
>
> Add 'skip_balance' to mount options so that the next mount will not
> attempt to resume balancing metadata. Keep mounting and umounting
> (not remounting) until it completes orphan and relocation cleanup (it
> may take more than one attempt, probably fewer than 20 attempts).
I try to mount with this command:
--8<---------------cut here---------------start------------->8---
~$ mount -o skip_balance,relatime,ssd,subvol=/ /dev/sda3 /
mount: /: wrong fs type, bad option, bad superblock on /dev/sda3, missing codepage or helper program, or other error.
--8<---------------cut here---------------end--------------->8---
dmesg says:
--8<---------------cut here---------------start------------->8---
[7484575.970136] BTRFS info (device sda3): disk space caching is enabled
[7484576.001375] BTRFS error (device sda3): Remounting read-write after error is not allowed
--8<---------------cut here---------------end--------------->8---
Am I doing something wrong?
It seems that the filesystem is not allowed to be remounted RW after the
error.
I don't think rebooting is a good option since it will be unbootable
(and it's a remote machine).
I fear the only option is to reboot from USB and revover :(
Do you have any other option in mind please?
> Once you have the filesystem mounted, run 'btrfs balance cancel' on
> the mount point. Then edit your maintenance scripts and remove the
> metadata balance (-m flag to 'btrfs balance start').
OK clear thanks.
>> Additional data:
>>
>> --8<---------------cut here---------------start------------->8---
[...]
>> ~$ sudo btrfs fi usage /
>> Overall:
>> Device size: 899.07GiB
>> Device allocated: 899.07GiB
>> Device unallocated: 2.01MiB
>> Device missing: 0.00B
>> Used: 897.05GiB
>> Free (estimated): 85.87MiB (min: 85.87MiB)
>> Data ratio: 2.00
>> Metadata ratio: 2.00
>> Global reserve: 512.00MiB (used: 5.53MiB)
>>
>> Data,RAID1: Size:446.50GiB, Used:446.42GiB (99.98%)
>> /dev/sda3 446.50GiB
>> /dev/sdb3 446.50GiB
>>
>> Metadata,RAID1: Size:3.00GiB, Used:2.11GiB (70.22%)
>> /dev/sda3 3.00GiB
>> /dev/sdb3 3.00GiB
>>
>> System,RAID1: Size:32.00MiB, Used:80.00KiB (0.24%)
>> /dev/sda3 32.00MiB
>> /dev/sdb3 32.00MiB
>>
>> Unallocated:
>> /dev/sda3 1.00MiB
>> /dev/sdb3 1.00MiB
[...]
> To avoid this, never run metadata balances from a scheduled job (or for
> any reason other than working around a kernel bug or adding disks to a
> RAID array) so that an appropriate number of metadata block groups is
> allocated and _stay_ allocated.
[...]
> Scheduled data balances (-d) are OK. They defragment free space and
> improve allocator performance, and make unallocated space available so
> that additional metadata block groups can be allocated when necessary.
OK got it: thank you for the clear and complete explanation.
No doubt I made a bad mistake with that scheduled job :-(
[...]
Thanks, Giovanni.
--
Giovanni Biscuolo
Xelera IT Infrastructures
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]
next prev parent reply other threads:[~2020-10-01 8:57 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-29 14:25 how to recover from "enospc errors during balance" Giovanni Biscuolo
2020-09-29 15:07 ` A L
2020-10-01 8:24 ` Giovanni Biscuolo
2020-09-30 0:04 ` Zygo Blaxell
2020-10-01 8:56 ` Giovanni Biscuolo [this message]
2020-10-01 15:28 ` A L
2020-10-02 9:32 ` Giovanni Biscuolo
2020-10-02 2:44 ` Zygo Blaxell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=878scq1g0g.fsf@roquette.i-did-not-set--mail-host-address--so-tickle-me \
--to=g@xelera.eu \
--cc=ce3g8jdj@umail.furryterror.org \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).