* ENOSPC on file system with nearly empty 12TB drive
@ 2022-01-14 20:24 Peter Chant
2022-01-14 22:12 ` Chris Murphy
0 siblings, 1 reply; 5+ messages in thread
From: Peter Chant @ 2022-01-14 20:24 UTC (permalink / raw)
To: Btrfs BTRFS
I have a file system that was _very_ full. Not ideal, but there were
reasons.
It comprised of 2x3TB drives and 2x4TB drives. I added a 12 TB drive
and tried to rebalance only to
hit ENOSPC. Surely adding a drive to a full file system fixes this?
Especially one that nearly doubles its capacity? Yet I kept hitting
this issue when
trying to rebalance and also remove a 4TB full drive.
Yet I have a drive with 11TB of free space. (24TB array)!?
The kernel was a 5.4.x series so I updated to 5.15.14.
I kept hitting the issue, however, after some filtered rebalancing I got
a little data
off the old drives and onto the new. I then decided, that since I had a
>90% empty 12TB drive
I could safely remove a 4TB drive, surely this would not be an issue?
Surely it would simply
simply it to the drive with most space, the new 12TB one? But despite a
nearly empty 12TB
drive it failed with ENOSPC.
I'm currently running a full rebalance (slow) to see if I can do this
without hitting ENOSPC and then remove a drive.
Some info, the kernel message:
[86348.556169] ------------[ cut here ]------------
[86348.556175] BTRFS: Transaction aborted (error -28)
[86348.556197] WARNING: CPU: 5 PID: 8736 at fs/btrfs/extent-tree.c:2150
btrfs_run_delayed_refs+0x1ab/0x1c0
[86348.556205] Modules linked in: nfsd bridge ipv6 8021q garp mrp stp
llc saa7134_alsa mt20xx tea5767 tda9887 tda8290 tuner uas usb_storage
edac_mce_amd saa7134 tveeprom kvm_amd wmi_bmof videobuf2_dma_sg
videobuf2_memops evdev videobuf2_v4l2 ccp videobuf2_common kvm radeon
videodev snd_hda_codec_via irqbypass snd_hda_codec_generic mc
ledtrig_audio drm_ttm_helper rc_core psmouse ttm snd_hda_codec_hdmi
snd_pcsp serio_raw via_rhine drm_kms_helper snd_hda_intel mii k10temp
snd_intel_dspcfg i2c_piix4 snd_intel_sdw_acpi snd_hda_codec drm
snd_hda_core agpgart snd_hwdep i2c_algo_bit snd_pcm fb_sys_fops
syscopyarea ohci_pci sysfillrect sysimgblt r8169 snd_timer realtek
i2c_core snd mdio_devres soundcore libphy atl1e ehci_pci ohci_hcd
ehci_hcd asus_atk0110 wmi button acpi_cpufreq loop
[86348.556252] CPU: 5 PID: 8736 Comm: btrfs Tainted: G W 5.15.14 #1
[86348.556255] Hardware name: System manufacturer System Product
Name/M4A78 PRO, BIOS 1701 08/16/2010
[86348.556257] RIP: 0010:btrfs_run_delayed_refs+0x1ab/0x1c0
[86348.556260] Code: 03 0f 82 44 74 6c 00 83 f8 fb 0f 84 3b 74 6c 00 83
f8 e2 0f 84 32 74 6c 00 89 c6 48 c7 c7 f8 12 5e a8 89 04 24 e8 4c 27 6b
00 <0f> 0b 8b 04 24 e9 17 74 6c 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f
[86348.556262] RSP: 0018:ffff9d5c03837b30 EFLAGS: 00010286
[86348.556264] RAX: 0000000000000000 RBX: ffff8e6493e2c800 RCX:
0000000000000000
[86348.556266] RDX: 0000000000000001 RSI: ffffffffa85a6db3 RDI:
00000000ffffffff
[86348.556267] RBP: ffff8e6594cb9c98 R08: 0000000000000000 R09:
ffff9d5c03837968
[86348.556268] R10: ffff9d5c03837960 R11: ffffffffa8912680 R12:
ffff8e65c3761178
[86348.556269] R13: ffff8e6494452000 R14: ffff8e6494452000 R15:
0000000000051dc3
[86348.556271] FS: 00007f1ccfeb48c0(0000) GS:ffff8e6698140000(0000)
knlGS:0000000000000000
[86348.556272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[86348.556273] CR2: 00007f4239364427 CR3: 000000017f41e000 CR4:
00000000000006e0
[86348.556275] Call Trace:
[86348.556278] <TASK>
[86348.556281] btrfs_commit_transaction+0x60/0xa00
[86348.556285] ? start_transaction+0xd2/0x600
[86348.556287] relocate_block_group+0x334/0x580
[86348.556290] btrfs_relocate_block_group+0x175/0x340
[86348.556292] btrfs_relocate_chunk+0x27/0xe0
[86348.556296] btrfs_shrink_device+0x260/0x530
[86348.556298] btrfs_rm_device+0x15b/0x4c0
[86348.556301] ? btrfs_ioctl+0xe91/0x2df0
[86348.556302] ? __check_object_size+0x136/0x150
[86348.556305] ? preempt_count_add+0x68/0xa0
[86348.556308] btrfs_ioctl+0xf27/0x2df0
[86348.556310] ? __handle_mm_fault+0xbf9/0x1450
[86348.556313] ? handle_mm_fault+0xc5/0x290
[86348.556315] ? __x64_sys_ioctl+0x83/0xb0
[86348.556318] __x64_sys_ioctl+0x83/0xb0
[86348.556320] do_syscall_64+0x3b/0xc0
[86348.556324] entry_SYSCALL_64_after_hwframe+0x44/0xae
[86348.556328] RIP: 0033:0x7f1ccffc54b7
[86348.556331] Code: 00 00 90 48 8b 05 d9 29 0d 00 64 c7 00 26 00 00 00
48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f
05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 29 0d 00 f7 d8 64 89 01 48
[86348.556333] RSP: 002b:00007ffe1f70abe8 EFLAGS: 00000206 ORIG_RAX:
0000000000000010
[86348.556335] RAX: ffffffffffffffda RBX: 00007ffe1f70cd90 RCX:
00007f1ccffc54b7
[86348.556336] RDX: 00007ffe1f70bc10 RSI: 000000005000943a RDI:
0000000000000003
[86348.556337] RBP: 0000000000000001 R08: 1999999999999999 R09:
0000000000000000
[86348.556338] R10: 000000000040838b R11: 0000000000000206 R12:
0000000000000000
[86348.556339] R13: 00007ffe1f70bc10 R14: 0000000000493660 R15:
0000000000000003
[86348.556341] </TASK>
[86348.556342] ---[ end trace c8c92460c79df13b ]---
[86348.556344] BTRFS: error (device dm-1) in
btrfs_run_delayed_refs:2150: errno=-28 No space left
[86348.556349] BTRFS info (device dm-1): forced readonly
root@dodo:/home/pete#
I was also getting ENOSPC errors when trying to rebalance, which I
corrected by adding a loop device on a temporary basis and using dusage
& musage filters with low values and progressively increasing them. So,
info from earlier on in this process:
btrfs balance start -dusage=94 -musage=94 . -v
root@dodo:/usr/local/sbin# btrfs fi u /mnt/backup-pool/
Overall:
Device size: 23.65TiB
Device allocated: 13.24TiB
Device unallocated: 10.41TiB
Device missing: 0.00B
Used: 13.19TiB
Free (estimated): 5.23TiB (min: 5.23TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID1: Size:6.60TiB, Used:6.58TiB (99.62%)
/dev/mapper/backup_disk_AC8F 3.59TiB
/dev/mapper/backup_disk_CCDZ 2.69TiB
/dev/mapper/backup_disk_XVN6 2.69TiB
/dev/mapper/backup_disk_E4U5 3.59TiB
/dev/mapper/backup_disk_4RNH 660.09GiB
Metadata,RAID1: Size:21.00GiB, Used:20.10GiB (95.74%)
/dev/mapper/backup_disk_AC8F 13.00GiB
/dev/mapper/backup_disk_CCDZ 9.00GiB
/dev/mapper/backup_disk_XVN6 9.00GiB
/dev/mapper/backup_disk_E4U5 10.00GiB
/dev/mapper/backup_disk_4RNH 1.00GiB
System,RAID1: Size:32.00MiB, Used:1.14MiB (3.56%)
/dev/mapper/backup_disk_E4U5 32.00MiB
/dev/mapper/backup_disk_4RNH 32.00MiB
Unallocated:
/dev/mapper/backup_disk_AC8F 37.00GiB
/dev/mapper/backup_disk_CCDZ 34.00GiB
/dev/mapper/backup_disk_XVN6 34.00GiB
/dev/mapper/backup_disk_E4U5 36.00GiB
/dev/mapper/backup_disk_4RNH 10.27TiB
/dev/mapper/backup-temp 84.00MiB
root@dodo:/usr/local/sbin#
root@dodo:/usr/local/sbin# btrfs fi show
Label: 'backup_store' uuid: 2e4162a8-ac38-4d1c-9a10-be873da6fbcc
Total devices 6 FS bytes used 6.59TiB
devid 1 size 3.64TiB used 3.60TiB path
/dev/mapper/backup_disk_AC8F
devid 2 size 2.73TiB used 2.70TiB path
/dev/mapper/backup_disk_CCDZ
devid 3 size 2.73TiB used 2.70TiB path
/dev/mapper/backup_disk_XVN6
devid 4 size 3.64TiB used 3.60TiB path
/dev/mapper/backup_disk_E4U5
devid 5 size 10.91TiB used 661.12GiB path
/dev/mapper/backup_disk_4RNH
devid 6 size 84.00MiB used 0.00B path /dev/mapper/backup-temp
root@dodo:/usr/local/sbin#
--
Pete
Peter Chant
07887 748492
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive
2022-01-14 20:24 ENOSPC on file system with nearly empty 12TB drive Peter Chant
@ 2022-01-14 22:12 ` Chris Murphy
2022-01-14 23:09 ` Pete
0 siblings, 1 reply; 5+ messages in thread
From: Chris Murphy @ 2022-01-14 22:12 UTC (permalink / raw)
To: Peter Chant; +Cc: Btrfs BTRFS
On Fri, Jan 14, 2022 at 1:30 PM Peter Chant <pete@petezilla.co.uk> wrote:
>
> I have a file system that was _very_ full. Not ideal, but there were
> reasons.
>
> It comprised of 2x3TB drives and 2x4TB drives. I added a 12 TB drive
> and tried to rebalance only to
>
> hit ENOSPC. Surely adding a drive to a full file system fixes this?
Not necessarily, in fact depending on the profile of the block group
type being balanced, e.g. if it's raid1 or raid10, you might need 2 or
4 drives added at once.
> Especially one that nearly doubles its capacity? Yet I kept hitting
> this issue when
>
> trying to rebalance and also remove a 4TB full drive.
>
> Yet I have a drive with 11TB of free space. (24TB array)!?
Sounds like metadata block groups are raid1. But it still can't add a
new metadata block group on *two* separate devices, so there's enospc.
>
>
> Some info, the kernel message:
>
> [86348.556169] ------------[ cut here ]------------
> [86348.556175] BTRFS: Transaction aborted (error -28)
> [86348.556197] WARNING: CPU: 5 PID: 8736 at fs/btrfs/extent-tree.c:2150
> btrfs_run_delayed_refs+0x1ab/0x1c0
> [86348.556205] Modules linked in: nfsd bridge ipv6 8021q garp mrp stp
> llc saa7134_alsa mt20xx tea5767 tda9887 tda8290 tuner uas usb_storage
> edac_mce_amd saa7134 tveeprom kvm_amd wmi_bmof videobuf2_dma_sg
> videobuf2_memops evdev videobuf2_v4l2 ccp videobuf2_common kvm radeon
> videodev snd_hda_codec_via irqbypass snd_hda_codec_generic mc
> ledtrig_audio drm_ttm_helper rc_core psmouse ttm snd_hda_codec_hdmi
> snd_pcsp serio_raw via_rhine drm_kms_helper snd_hda_intel mii k10temp
> snd_intel_dspcfg i2c_piix4 snd_intel_sdw_acpi snd_hda_codec drm
> snd_hda_core agpgart snd_hwdep i2c_algo_bit snd_pcm fb_sys_fops
> syscopyarea ohci_pci sysfillrect sysimgblt r8169 snd_timer realtek
> i2c_core snd mdio_devres soundcore libphy atl1e ehci_pci ohci_hcd
> ehci_hcd asus_atk0110 wmi button acpi_cpufreq loop
> [86348.556252] CPU: 5 PID: 8736 Comm: btrfs Tainted: G W 5.15.14 #1
> [86348.556255] Hardware name: System manufacturer System Product
> Name/M4A78 PRO, BIOS 1701 08/16/2010
> [86348.556257] RIP: 0010:btrfs_run_delayed_refs+0x1ab/0x1c0
> [86348.556260] Code: 03 0f 82 44 74 6c 00 83 f8 fb 0f 84 3b 74 6c 00 83
> f8 e2 0f 84 32 74 6c 00 89 c6 48 c7 c7 f8 12 5e a8 89 04 24 e8 4c 27 6b
> 00 <0f> 0b 8b 04 24 e9 17 74 6c 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f
> [86348.556262] RSP: 0018:ffff9d5c03837b30 EFLAGS: 00010286
> [86348.556264] RAX: 0000000000000000 RBX: ffff8e6493e2c800 RCX:
> 0000000000000000
> [86348.556266] RDX: 0000000000000001 RSI: ffffffffa85a6db3 RDI:
> 00000000ffffffff
> [86348.556267] RBP: ffff8e6594cb9c98 R08: 0000000000000000 R09:
> ffff9d5c03837968
> [86348.556268] R10: ffff9d5c03837960 R11: ffffffffa8912680 R12:
> ffff8e65c3761178
> [86348.556269] R13: ffff8e6494452000 R14: ffff8e6494452000 R15:
> 0000000000051dc3
> [86348.556271] FS: 00007f1ccfeb48c0(0000) GS:ffff8e6698140000(0000)
> knlGS:0000000000000000
> [86348.556272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [86348.556273] CR2: 00007f4239364427 CR3: 000000017f41e000 CR4:
> 00000000000006e0
> [86348.556275] Call Trace:
> [86348.556278] <TASK>
> [86348.556281] btrfs_commit_transaction+0x60/0xa00
> [86348.556285] ? start_transaction+0xd2/0x600
> [86348.556287] relocate_block_group+0x334/0x580
> [86348.556290] btrfs_relocate_block_group+0x175/0x340
> [86348.556292] btrfs_relocate_chunk+0x27/0xe0
> [86348.556296] btrfs_shrink_device+0x260/0x530
> [86348.556298] btrfs_rm_device+0x15b/0x4c0
Looks like a device in the process of being removed, the file system
is undergoing shrink, and a bg is being relocated...
> [86348.556301] ? btrfs_ioctl+0xe91/0x2df0
> [86348.556302] ? __check_object_size+0x136/0x150
> [86348.556305] ? preempt_count_add+0x68/0xa0
> [86348.556308] btrfs_ioctl+0xf27/0x2df0
> [86348.556310] ? __handle_mm_fault+0xbf9/0x1450
> [86348.556313] ? handle_mm_fault+0xc5/0x290
> [86348.556315] ? __x64_sys_ioctl+0x83/0xb0
> [86348.556318] __x64_sys_ioctl+0x83/0xb0
> [86348.556320] do_syscall_64+0x3b/0xc0
> [86348.556324] entry_SYSCALL_64_after_hwframe+0x44/0xae
> [86348.556328] RIP: 0033:0x7f1ccffc54b7
> [86348.556331] Code: 00 00 90 48 8b 05 d9 29 0d 00 64 c7 00 26 00 00 00
> 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 29 0d 00 f7 d8 64 89 01 48
> [86348.556333] RSP: 002b:00007ffe1f70abe8 EFLAGS: 00000206 ORIG_RAX:
> 0000000000000010
> [86348.556335] RAX: ffffffffffffffda RBX: 00007ffe1f70cd90 RCX:
> 00007f1ccffc54b7
> [86348.556336] RDX: 00007ffe1f70bc10 RSI: 000000005000943a RDI:
> 0000000000000003
> [86348.556337] RBP: 0000000000000001 R08: 1999999999999999 R09:
> 0000000000000000
> [86348.556338] R10: 000000000040838b R11: 0000000000000206 R12:
> 0000000000000000
> [86348.556339] R13: 00007ffe1f70bc10 R14: 0000000000493660 R15:
> 0000000000000003
> [86348.556341] </TASK>
> [86348.556342] ---[ end trace c8c92460c79df13b ]---
> [86348.556344] BTRFS: error (device dm-1) in
> btrfs_run_delayed_refs:2150: errno=-28 No space left
> [86348.556349] BTRFS info (device dm-1): forced readonly
And it gets confused because it can't successfully relocate the block
group, and it shuts down the file system to prevent confusion being
written to the file system.
>
> btrfs balance start -dusage=94 -musage=94 . -v
>
> root@dodo:/usr/local/sbin# btrfs fi u /mnt/backup-pool/
> Overall:
> Device size: 23.65TiB
> Device allocated: 13.24TiB
> Device unallocated: 10.41TiB
> Device missing: 0.00B
> Used: 13.19TiB
> Free (estimated): 5.23TiB (min: 5.23TiB)
> Data ratio: 2.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
>
> Data,RAID1: Size:6.60TiB, Used:6.58TiB (99.62%)
> /dev/mapper/backup_disk_AC8F 3.59TiB
> /dev/mapper/backup_disk_CCDZ 2.69TiB
> /dev/mapper/backup_disk_XVN6 2.69TiB
> /dev/mapper/backup_disk_E4U5 3.59TiB
> /dev/mapper/backup_disk_4RNH 660.09GiB
>
> Metadata,RAID1: Size:21.00GiB, Used:20.10GiB (95.74%)
> /dev/mapper/backup_disk_AC8F 13.00GiB
> /dev/mapper/backup_disk_CCDZ 9.00GiB
> /dev/mapper/backup_disk_XVN6 9.00GiB
> /dev/mapper/backup_disk_E4U5 10.00GiB
> /dev/mapper/backup_disk_4RNH 1.00GiB
>
> System,RAID1: Size:32.00MiB, Used:1.14MiB (3.56%)
> /dev/mapper/backup_disk_E4U5 32.00MiB
> /dev/mapper/backup_disk_4RNH 32.00MiB
>
> Unallocated:
> /dev/mapper/backup_disk_AC8F 37.00GiB
> /dev/mapper/backup_disk_CCDZ 34.00GiB
> /dev/mapper/backup_disk_XVN6 34.00GiB
> /dev/mapper/backup_disk_E4U5 36.00GiB
> /dev/mapper/backup_disk_4RNH 10.27TiB
> /dev/mapper/backup-temp 84.00MiB
> root@dodo:/usr/local/sbin#
OK metadata is raid1, and it's nearly full. But there's multiple
devices with at least a couple GiB unallocated such that at least 2
metadata block groups could be created.
Are you trying to do balance and device remove at the same time?
--
Chris Murphy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive
2022-01-14 22:12 ` Chris Murphy
@ 2022-01-14 23:09 ` Pete
2022-01-15 20:38 ` Chris Murphy
0 siblings, 1 reply; 5+ messages in thread
From: Pete @ 2022-01-14 23:09 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 14/01/2022 22:12, Chris Murphy wrote:
> On Fri, Jan 14, 2022 at 1:30 PM Peter Chant <pete@petezilla.co.uk> wrote:
A lot of snipping - I hope I don't cause confusion.
>> hit ENOSPC. Surely adding a drive to a full file system fixes this?
>
> Not necessarily, in fact depending on the profile of the block group
> type being balanced, e.g. if it's raid1 or raid10, you might need 2 or
> 4 drives added at once.
>
Raid1 data, raid1 metadata.
>
>> Especially one that nearly doubles its capacity? Yet I kept hitting
>> this issue when
>>
>> trying to rebalance and also remove a 4TB full drive.
>>
>> Yet I have a drive with 11TB of free space. (24TB array)!?
>
> Sounds like metadata block groups are raid1. But it still can't add a
> new metadata block group on *two* separate devices, so there's enospc.
There are five drives, excluding any (smallish) disk image I've added as
a loop device to help, but _four_ are painfully full.
...6281] btrfs_commit_transaction+0x60/0xa00
>> [86348.556285] ? start_transaction+0xd2/0x600
>> [86348.556287] relocate_block_group+0x334/0x580
>> [86348.556290] btrfs_relocate_block_group+0x175/0x340
>> [86348.556292] btrfs_relocate_chunk+0x27/0xe0
>> [86348.556296] btrfs_shrink_device+0x260/0x530
>> [86348.556298] btrfs_rm_device+0x15b/0x4c0
>
> Looks like a device in the process of being removed, the file system
> is undergoing shrink, and a bg is being relocated...
>
Well, I was trying to rebalance. But it was very slow. So I changed my
mind and though just removing one of the drives, which I intended to do
anyway, might help moving data onto the new device.
The drive I tried removing is a WD Red with Drive Managed SMR - so that
is helping with the speed...
...
>> btrfs_run_delayed_refs:2150: errno=-28 No space left
>> [86348.556349] BTRFS info (device dm-1): forced readonly
>
> And it gets confused because it can't successfully relocate the block
> group, and it shuts down the file system to prevent confusion being
> written to the file system.
>
Ok, that bit was a bit beyond my understanding.
...
>
> OK metadata is raid1, and it's nearly full. But there's multiple
> devices with at least a couple GiB unallocated such that at least 2
> metadata block groups could be created.
The latest (several hours later):
root@dodo:/home/pete# btrfs fi u /srv/nfs/backup-pull
Overall:
Device size: 23.65TiB
Device allocated: 13.51TiB
Device unallocated: 10.14TiB
Device missing: 0.00B
Used: 13.51TiB
Free (estimated): 5.07TiB (min: 5.07TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 256.00KiB)
Data,RAID1: Size:6.74TiB, Used:6.73TiB (99.98%)
/dev/mapper/backup_disk_AC8F 3.43TiB
/dev/mapper/backup_disk_CCDZ 2.60TiB
/dev/mapper/backup_disk_XVN6 2.60TiB
/dev/mapper/backup_disk_E4U5 2.42TiB
/dev/mapper/backup_disk_4RNH 2.41TiB
Metadata,RAID1: Size:21.00GiB, Used:20.49GiB (97.59%)
/dev/mapper/backup_disk_AC8F 12.00GiB
/dev/mapper/backup_disk_CCDZ 9.00GiB
/dev/mapper/backup_disk_XVN6 9.00GiB
/dev/mapper/backup_disk_E4U5 6.00GiB
/dev/mapper/backup_disk_4RNH 6.00GiB
System,RAID1: Size:32.00MiB, Used:1.28MiB (4.00%)
/dev/mapper/backup_disk_E4U5 32.00MiB
/dev/mapper/backup_disk_4RNH 32.00MiB
Unallocated:
/dev/mapper/backup_disk_AC8F 199.99GiB
/dev/mapper/backup_disk_CCDZ 120.00GiB
/dev/mapper/backup_disk_XVN6 124.00GiB
/dev/mapper/backup_disk_E4U5 1.21TiB
/dev/mapper/backup_disk_4RNH 8.50TiB
root@dodo:/home/pete#
>
> Are you trying to do balance and device remove at the same time?
>
>
Well, I cancelled a balance, tried a remove and since that ENOSPC'ed I'm
now running a full balance.
Any suggestions? Just be patient, and hope the balance finishes without
ENOSPC? Go for the remove again. I'd like to remove a 4TB drive if I
can without adding a 6th HD to the system. Still don't understand why I
might need more than one practically empty drive for raid1?
Or, given that I've got another 12TB drive on the way, just abandon this
file system (apart from reading any relevant data) and create a new file
system as single, and migrate the current 12TB disk over later on to go
back to raid1?
I'm wondering if part of the problem is a 3.6TB disk image that is
sitting on the file system (fixing someone else's external drive).
Deleting that might speed things up, but since I don't think that drive
is fully backed up I'd rather keep the disk image until I am sure that
all is well. (I don't need the backup lecture...)
Apart from a loop device mounted drive image I might be able to mount
another disk in the five disk file system - but I'd rather not. I'd
have to add a controller (have one spare) which might move the boot
device (not using initrd and UUIDs) - and shoehorning another drive in
is a little tricky but not impossible. Its an old midi tower cased
machine repurposed as my backup server.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive
2022-01-14 23:09 ` Pete
@ 2022-01-15 20:38 ` Chris Murphy
2022-01-16 12:27 ` Pete
0 siblings, 1 reply; 5+ messages in thread
From: Chris Murphy @ 2022-01-15 20:38 UTC (permalink / raw)
To: Pete; +Cc: Chris Murphy, Btrfs BTRFS
On Fri, Jan 14, 2022 at 4:09 PM Pete <pete@petezilla.co.uk> wrote:
> Any suggestions? Just be patient, and hope the balance finishes without
> ENOSPC? Go for the remove again. I'd like to remove a 4TB drive if I
> can without adding a 6th HD to the system. Still don't understand why I
> might need more than one practically empty drive for raid1?
When bg profile is raid1, any time the file system wants to add
another block group, it must create a chunk on 2 devices at the same
time for it to succeed or else you get ENOSPC. The question is why two
chunks can't be created given all the unallocated space you have even
without the empty drive.
Ordinarily you want to avoid doing metadata balance. Balancing data is
ok, it amounts to defragmenting free space, and returning excess into
unallocated space which can then be used to create any type of block
group.
>
> Or, given that I've got another 12TB drive on the way, just abandon this
> file system (apart from reading any relevant data) and create a new file
> system as single, and migrate the current 12TB disk over later on to go
> back to raid1?
>
> I'm wondering if part of the problem is a 3.6TB disk image that is
> sitting on the file system (fixing someone else's external drive).
> Deleting that might speed things up, but since I don't think that drive
> is fully backed up I'd rather keep the disk image until I am sure that
> all is well. (I don't need the backup lecture...)
I don't think the large image file is the problem.
In my opinion, you've hit a bug. There's plenty of unallocated space
on multiple drives. I think what's going on is an old bug that might
not be fixed yet where metadata overcommit is allowed to overestimate
due to the single large empty drive with a ton of space on it. But
since the overcommit can't be fulfilled on at least two drives, you
get ENOSPC even though it's not actually trying to create that many
block groups at once.
So what I suggest doing is removing the mostly empty device, and only
add devices in pairs when using raid1.
--
Chris Murphy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive
2022-01-15 20:38 ` Chris Murphy
@ 2022-01-16 12:27 ` Pete
0 siblings, 0 replies; 5+ messages in thread
From: Pete @ 2022-01-16 12:27 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
On 1/15/22 20:38, Chris Murphy wrote:
> On Fri, Jan 14, 2022 at 4:09 PM Pete <pete@petezilla.co.uk> wrote:
>
>> Any suggestions? Just be patient, and hope the balance finishes without
>> ENOSPC? Go for the remove again. I'd like to remove a 4TB drive if I
>> can without adding a 6th HD to the system. Still don't understand why I
>> might need more than one practically empty drive for raid1?
>
> When bg profile is raid1, any time the file system wants to add
> another block group, it must create a chunk on 2 devices at the same
> time for it to succeed or else you get ENOSPC. The question is why two
> chunks can't be created given all the unallocated space you have even
> without the empty drive.
So although in % terms the drives were very full the > 30GB of free
space on each ought to have been more than sufficient?
A little care is needed as the ENOSPC did not occur at exactly the time
I ran btrfs fi show and btrfs fi usage. So perhaps my post is the the
most rock solid evidence if there is an issue. However, I had multiple
instances of ENOSPC at the time, even after it had spent a few hours
balancing.
>
> Ordinarily you want to avoid doing metadata balance. Balancing data is
> ok, it amounts to defragmenting free space, and returning excess into
> unallocated space which can then be used to create any type of block
> group.
>
OK, but various resources show metadata balance being used when trying
to sort a ENOSPC issue, e.g.
https://www.suse.com/support/kb/doc/?id=000019789
But there are others that I don't seem to be able to find again. I
should have noted the URLs when I was trying to sort the issue...
>
> I don't think the large image file is the problem.
>
> In my opinion, you've hit a bug. There's plenty of unallocated space
> on multiple drives. I think what's going on is an old bug that might
> not be fixed yet where metadata overcommit is allowed to overestimate
> due to the single large empty drive with a ton of space on it. But
> since the overcommit can't be fulfilled on at least two drives, you
> get ENOSPC even though it's not actually trying to create that many
> block groups at once.
>
My naive but logical expectation was that balance would start
aggressively moving data to the empty drive. Most of the guidance I see
for recovering from an ENOSPC indicates that adding a single device
would be sufficient, I do note, however, that if you scroll down and
read it all, some sites do point this out:
https://wiki.tnonline.net/w/Btrfs/ENOSPC
I also note that the various guides / howtos online are not necessarily
within the control people who are on this mailing list - I'm not
implying that the active maintainers are/should be responsible about
everything written about btrfs online.
My strong recollection from reading this mailing list and when I have
searched on line was that adding one device only was mentioned for
managing an ENOSPC issue. However, from a limited amount of searching
I'm not sure that I can back the up with references. Perhaps that is
just my perception?
Adding the new drive plus a loop device allowed me to progressively
rebalance using dusage and musage filters until I could start a full
rebalance without the loop device.
Interestingly, though this array had been running, raid1, for a while
with four drives, without rebalancing, metadata was only stored on two
of the drives.
> So what I suggest doing is removing the mostly empty device, and only
> add devices in pairs when using raid1.
Should there be a caveat added to this "For very full btrfs raid1 file
systems only add devices in at least pairs."? Being able to add devices
in a fairly ad hoc manner is a great strength for btrfs.
It seems to be that I am past the point of removing the larger device
now as I have > 500 GB free on the smaller drives now, have removed the
loop device and am no longer hitting ENOSPC. I probably would have to
start deleting some of my backups to remove the new larger device just
to that the original 4 device array was not so cripplingly full.
Thank you for your help.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-01-16 12:28 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-14 20:24 ENOSPC on file system with nearly empty 12TB drive Peter Chant
2022-01-14 22:12 ` Chris Murphy
2022-01-14 23:09 ` Pete
2022-01-15 20:38 ` Chris Murphy
2022-01-16 12:27 ` Pete
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.