All of lore.kernel.org
 help / color / mirror / Atom feed
From: Eli V <eliventer@gmail.com>
To: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: 5.10 kernel/fs exhausting reserve running btrfs_delete_unused_bgs, going read-only
Date: Thu, 11 Nov 2021 12:33:43 -0500	[thread overview]
Message-ID: <CAJtFHURRaL-v60s_gk5+unRQtsU9wvZOTtLm5=c0ALqBF23kbQ@mail.gmail.com> (raw)
In-Reply-To: <CAJtFHUSN+RfZa2BitX9gH++M54uA7MTmn4Fn6Afx2RL4NPeaVQ@mail.gmail.com>

On Thu, Nov 11, 2021 at 10:17 AM Eli V <eliventer@gmail.com> wrote:
>
> Ideas requested on how to fix a large btrfs filesystem that goes
> read-only shortly after a fresh
> mount, with this in dmesg:
>
> BTRFS info (device sdb): left=1507328, need=1572864, flags=1
> BTRFS info (device sdb): space_info 2 has 1507328 free, is not full
> BTRFS info (device sdb): space_info total=75497472, used=72482816,
> pinned=0, reserved=1507328, may_use=0, readonly=0
> BTRFS info (device sdb): global_block_rsv: size 536870912 reserved 533856256
> BTRFS info (device sdb): trans_block_rsv: size 4194304 reserved 4194304
> BTRFS info (device sdb): chunk_block_rsv: size 0 reserved 0
> BTRFS info (device sdb): delayed_block_rsv: size 0 reserved 0
> BTRFS info (device sdb): delayed_refs_rsv: size 298844160 reserved 110231552
> BTRFS info (device sdb): left=1441792, need=1572864, flags=1
> BTRFS info (device sdb): space_info 2 has 1441792 free, is not full
> BTRFS info (device sdb): space_info total=75497472, used=72482816,
> pinned=0, reserved=1572864, may_use=0, readonly=0
> ...
> BTRFS info (device sdb): global_block_rsv: size 536870912 reserved 533266432
> BTRFS info (device sdb): trans_block_rsv: size 4194304 reserved 4194304
> BTRFS info (device sdb): chunk_block_rsv: size 0 reserved 0
> BTRFS info (device sdb): delayed_block_rsv: size 0 reserved 0
> BTRFS info (device sdb): delayed_refs_rsv: size 342884352 reserved 155910144
> BTRFS info (device sdb): left=1376256, need=1572864, flags=1
> BTRFS info (device sdb): space_info 2 has 1376256 free, is not full
> BTRFS info (device sdb): space_info total=75497472, used=72482816,
> pinned=0, reserved=1638400, may_use=0, readonly=0
> BTRFS info (device sdb): global_block_rsv: size 536870912 reserved 533266432
> BTRFS error (device sdb): allocation failed flags 18, wanted 65536
> BTRFS: Transaction aborted (error -28)
> WARNING: CPU: 1 PID: 1060882 at fs/btrfs/volumes.c:2989
> btrfs_remove_chunk+0x553/0x6b0 [btrfs]
> CPU: 1 PID: 1060882 Comm: btrfs-cleaner Tainted: G        W I
> 5.10.0-9-amd64 #1 Debian 5.10.70-1
> RIP: 0010:btrfs_remove_chunk+0x553/0x6b0 [btrfs]
> Code: 40 0a 00 00 02 72 28 83 f8 fb 0f 84 91 00 00 00 83 f8 e2 0f 84
> 88 00 00 00 89 c6 48 c7 c7 40 22 71 c0 89 04 24 e8 80 33 1f e0 <0f> 0b
> 8b 04 24 89 c1 ba ad 0b 00 00 4c 89 ef 89 04 24 48 c7 c6 d0
> RSP: 0018:ffffb82003983d90 EFLAGS: 00010286
> RAX: 0000000000000000 RBX: ffff95a324c2ef00 RCX: ffff959f5fc18a08
> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff959f5fc18a00
> RBP: ffff959d85f4a000 R08: 0000000000000000 R09: ffffb82003983bb0
> R10: ffffb82003983ba8 R11: ffffffffa16ba8f8 R12: ffff959d1acc4150
> R13: ffff95a1c807f138 R14: ffff959d85f4a380 R15: ffff95a095e90000
> FS:  0000000000000000(0000) GS:ffff959f5fc00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000557323588e70 CR3: 00000002dfa0a005 CR4: 00000000000206e0
> Call Trace:
>  btrfs_delete_unused_bgs+0x651/0x7a0 [btrfs]
>  cleaner_kthread+0xef/0x120 [btrfs]
>  ? btree_invalidatepage+0x40/0x40 [btrfs]
>  kthread+0x11b/0x140
>  ? __kthread_bind_mask+0x60/0x60
>  ret_from_fork+0x22/0x30
> ---[ end trace 6ad4eacda93c19a4 ]---
> BTRFS: error (device sdb) in btrfs_remove_chunk:2989: errno=-28 No space left
>
> After it goes read-only usage seems ok:
> $ btrfs filesystem usage -T /mirror
> Overall:
>     Device size:                 382.02TiB
>     Device allocated:            381.01TiB
>     Device unallocated:            1.01TiB
>     Device missing:                  0.00B
>     Used:                        337.69TiB
>     Free (estimated):             43.86TiB      (min: 43.86TiB)
>     Free (statfs, df):            42.97TiB
>     Data ratio:                       1.00
>     Metadata ratio:                   1.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
>
>             Data      Metadata  System
> Id Path     single    RAID1     RAID1    Unallocated
> -- -------- --------- --------- -------- -----------
>  1 /dev/sdb  27.22TiB  29.50GiB        -    33.51GiB
>  2 /dev/sdc  27.20TiB  39.00GiB        -    43.00GiB
>  3 /dev/sdd  36.28TiB  51.00GiB        -    57.00GiB
>  4 /dev/sde  36.24TiB  63.00GiB        -    78.00GiB
>  5 /dev/sdf  54.36TiB  98.02GiB 32.00MiB   121.05GiB
>  6 /dev/sdg  54.26TiB 148.50GiB  4.00MiB   172.50GiB
>  7 /dev/sdh  72.26TiB 247.52GiB 36.00MiB   269.55GiB
>  8 /dev/sdi  72.29TiB 234.50GiB        -   256.51GiB
> -- -------- --------- --------- -------- -----------
>    Total    380.12TiB 911.04GiB 72.00MiB     1.01TiB
>    Used     337.27TiB 432.68GiB 69.12MiB

Compiling 5.10.78 using 1GB for global reserve also went read-only
immediately, but using 2GB allowed things to get through and the fs
stayed rw after mount. It would be really nice if that could be a
kernel boot or sysfs param instead of a recompile. I'll run with 2GB
for now and see how things go:

btrfs filesystem usage -T /mirror
Overall:
    Device size:                 382.02TiB
    Device allocated:            381.13TiB
    Device unallocated:          913.12GiB
    Device missing:                  0.00B
    Used:                        337.69TiB
    Free (estimated):             43.86TiB      (min: 43.86TiB)
    Free (statfs, df):            42.97TiB
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:                2.00GiB      (used: 0.00B)
    Multiple profiles:                  no

            Data      Metadata  System
Id Path     single    RAID1     RAID1    Unallocated
-- -------- --------- --------- -------- -----------
 1 /dev/sdb  27.23TiB  29.50GiB        -    29.51GiB
 2 /dev/sdc  27.21TiB  39.00GiB        -    39.00GiB
 3 /dev/sdd  36.28TiB  51.00GiB        -    51.00GiB
 4 /dev/sde  36.26TiB  63.00GiB        -    63.00GiB
 5 /dev/sdf  54.38TiB  98.02GiB 32.00MiB    98.05GiB
 6 /dev/sdg  54.28TiB 148.50GiB  4.00MiB   148.50GiB
 7 /dev/sdh  72.28TiB 247.52GiB 36.00MiB   249.55GiB
 8 /dev/sdi  72.31TiB 234.50GiB        -   234.51GiB
-- -------- --------- --------- -------- -----------
   Total    380.24TiB 911.04GiB 72.00MiB   913.12GiB
   Used     337.27TiB 432.68GiB 69.12MiB

      reply	other threads:[~2021-11-11 17:33 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-11 15:17 5.10 kernel/fs exhausting reserve running btrfs_delete_unused_bgs, going read-only Eli V
2021-11-11 17:33 ` Eli V [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJtFHURRaL-v60s_gk5+unRQtsU9wvZOTtLm5=c0ALqBF23kbQ@mail.gmail.com' \
    --to=eliventer@gmail.com \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.