* ENOSPC on file system with nearly empty 12TB drive @ 2022-01-14 20:24 Peter Chant 2022-01-14 22:12 ` Chris Murphy 0 siblings, 1 reply; 5+ messages in thread From: Peter Chant @ 2022-01-14 20:24 UTC (permalink / raw) To: Btrfs BTRFS I have a file system that was _very_ full. Not ideal, but there were reasons. It comprised of 2x3TB drives and 2x4TB drives. I added a 12 TB drive and tried to rebalance only to hit ENOSPC. Surely adding a drive to a full file system fixes this? Especially one that nearly doubles its capacity? Yet I kept hitting this issue when trying to rebalance and also remove a 4TB full drive. Yet I have a drive with 11TB of free space. (24TB array)!? The kernel was a 5.4.x series so I updated to 5.15.14. I kept hitting the issue, however, after some filtered rebalancing I got a little data off the old drives and onto the new. I then decided, that since I had a >90% empty 12TB drive I could safely remove a 4TB drive, surely this would not be an issue? Surely it would simply simply it to the drive with most space, the new 12TB one? But despite a nearly empty 12TB drive it failed with ENOSPC. I'm currently running a full rebalance (slow) to see if I can do this without hitting ENOSPC and then remove a drive. Some info, the kernel message: [86348.556169] ------------[ cut here ]------------ [86348.556175] BTRFS: Transaction aborted (error -28) [86348.556197] WARNING: CPU: 5 PID: 8736 at fs/btrfs/extent-tree.c:2150 btrfs_run_delayed_refs+0x1ab/0x1c0 [86348.556205] Modules linked in: nfsd bridge ipv6 8021q garp mrp stp llc saa7134_alsa mt20xx tea5767 tda9887 tda8290 tuner uas usb_storage edac_mce_amd saa7134 tveeprom kvm_amd wmi_bmof videobuf2_dma_sg videobuf2_memops evdev videobuf2_v4l2 ccp videobuf2_common kvm radeon videodev snd_hda_codec_via irqbypass snd_hda_codec_generic mc ledtrig_audio drm_ttm_helper rc_core psmouse ttm snd_hda_codec_hdmi snd_pcsp serio_raw via_rhine drm_kms_helper snd_hda_intel mii k10temp snd_intel_dspcfg i2c_piix4 snd_intel_sdw_acpi snd_hda_codec drm snd_hda_core agpgart snd_hwdep i2c_algo_bit snd_pcm fb_sys_fops syscopyarea ohci_pci sysfillrect sysimgblt r8169 snd_timer realtek i2c_core snd mdio_devres soundcore libphy atl1e ehci_pci ohci_hcd ehci_hcd asus_atk0110 wmi button acpi_cpufreq loop [86348.556252] CPU: 5 PID: 8736 Comm: btrfs Tainted: G W 5.15.14 #1 [86348.556255] Hardware name: System manufacturer System Product Name/M4A78 PRO, BIOS 1701 08/16/2010 [86348.556257] RIP: 0010:btrfs_run_delayed_refs+0x1ab/0x1c0 [86348.556260] Code: 03 0f 82 44 74 6c 00 83 f8 fb 0f 84 3b 74 6c 00 83 f8 e2 0f 84 32 74 6c 00 89 c6 48 c7 c7 f8 12 5e a8 89 04 24 e8 4c 27 6b 00 <0f> 0b 8b 04 24 e9 17 74 6c 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f [86348.556262] RSP: 0018:ffff9d5c03837b30 EFLAGS: 00010286 [86348.556264] RAX: 0000000000000000 RBX: ffff8e6493e2c800 RCX: 0000000000000000 [86348.556266] RDX: 0000000000000001 RSI: ffffffffa85a6db3 RDI: 00000000ffffffff [86348.556267] RBP: ffff8e6594cb9c98 R08: 0000000000000000 R09: ffff9d5c03837968 [86348.556268] R10: ffff9d5c03837960 R11: ffffffffa8912680 R12: ffff8e65c3761178 [86348.556269] R13: ffff8e6494452000 R14: ffff8e6494452000 R15: 0000000000051dc3 [86348.556271] FS: 00007f1ccfeb48c0(0000) GS:ffff8e6698140000(0000) knlGS:0000000000000000 [86348.556272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86348.556273] CR2: 00007f4239364427 CR3: 000000017f41e000 CR4: 00000000000006e0 [86348.556275] Call Trace: [86348.556278] <TASK> [86348.556281] btrfs_commit_transaction+0x60/0xa00 [86348.556285] ? start_transaction+0xd2/0x600 [86348.556287] relocate_block_group+0x334/0x580 [86348.556290] btrfs_relocate_block_group+0x175/0x340 [86348.556292] btrfs_relocate_chunk+0x27/0xe0 [86348.556296] btrfs_shrink_device+0x260/0x530 [86348.556298] btrfs_rm_device+0x15b/0x4c0 [86348.556301] ? btrfs_ioctl+0xe91/0x2df0 [86348.556302] ? __check_object_size+0x136/0x150 [86348.556305] ? preempt_count_add+0x68/0xa0 [86348.556308] btrfs_ioctl+0xf27/0x2df0 [86348.556310] ? __handle_mm_fault+0xbf9/0x1450 [86348.556313] ? handle_mm_fault+0xc5/0x290 [86348.556315] ? __x64_sys_ioctl+0x83/0xb0 [86348.556318] __x64_sys_ioctl+0x83/0xb0 [86348.556320] do_syscall_64+0x3b/0xc0 [86348.556324] entry_SYSCALL_64_after_hwframe+0x44/0xae [86348.556328] RIP: 0033:0x7f1ccffc54b7 [86348.556331] Code: 00 00 90 48 8b 05 d9 29 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 29 0d 00 f7 d8 64 89 01 48 [86348.556333] RSP: 002b:00007ffe1f70abe8 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 [86348.556335] RAX: ffffffffffffffda RBX: 00007ffe1f70cd90 RCX: 00007f1ccffc54b7 [86348.556336] RDX: 00007ffe1f70bc10 RSI: 000000005000943a RDI: 0000000000000003 [86348.556337] RBP: 0000000000000001 R08: 1999999999999999 R09: 0000000000000000 [86348.556338] R10: 000000000040838b R11: 0000000000000206 R12: 0000000000000000 [86348.556339] R13: 00007ffe1f70bc10 R14: 0000000000493660 R15: 0000000000000003 [86348.556341] </TASK> [86348.556342] ---[ end trace c8c92460c79df13b ]--- [86348.556344] BTRFS: error (device dm-1) in btrfs_run_delayed_refs:2150: errno=-28 No space left [86348.556349] BTRFS info (device dm-1): forced readonly root@dodo:/home/pete# I was also getting ENOSPC errors when trying to rebalance, which I corrected by adding a loop device on a temporary basis and using dusage & musage filters with low values and progressively increasing them. So, info from earlier on in this process: btrfs balance start -dusage=94 -musage=94 . -v root@dodo:/usr/local/sbin# btrfs fi u /mnt/backup-pool/ Overall: Device size: 23.65TiB Device allocated: 13.24TiB Device unallocated: 10.41TiB Device missing: 0.00B Used: 13.19TiB Free (estimated): 5.23TiB (min: 5.23TiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data,RAID1: Size:6.60TiB, Used:6.58TiB (99.62%) /dev/mapper/backup_disk_AC8F 3.59TiB /dev/mapper/backup_disk_CCDZ 2.69TiB /dev/mapper/backup_disk_XVN6 2.69TiB /dev/mapper/backup_disk_E4U5 3.59TiB /dev/mapper/backup_disk_4RNH 660.09GiB Metadata,RAID1: Size:21.00GiB, Used:20.10GiB (95.74%) /dev/mapper/backup_disk_AC8F 13.00GiB /dev/mapper/backup_disk_CCDZ 9.00GiB /dev/mapper/backup_disk_XVN6 9.00GiB /dev/mapper/backup_disk_E4U5 10.00GiB /dev/mapper/backup_disk_4RNH 1.00GiB System,RAID1: Size:32.00MiB, Used:1.14MiB (3.56%) /dev/mapper/backup_disk_E4U5 32.00MiB /dev/mapper/backup_disk_4RNH 32.00MiB Unallocated: /dev/mapper/backup_disk_AC8F 37.00GiB /dev/mapper/backup_disk_CCDZ 34.00GiB /dev/mapper/backup_disk_XVN6 34.00GiB /dev/mapper/backup_disk_E4U5 36.00GiB /dev/mapper/backup_disk_4RNH 10.27TiB /dev/mapper/backup-temp 84.00MiB root@dodo:/usr/local/sbin# root@dodo:/usr/local/sbin# btrfs fi show Label: 'backup_store' uuid: 2e4162a8-ac38-4d1c-9a10-be873da6fbcc Total devices 6 FS bytes used 6.59TiB devid 1 size 3.64TiB used 3.60TiB path /dev/mapper/backup_disk_AC8F devid 2 size 2.73TiB used 2.70TiB path /dev/mapper/backup_disk_CCDZ devid 3 size 2.73TiB used 2.70TiB path /dev/mapper/backup_disk_XVN6 devid 4 size 3.64TiB used 3.60TiB path /dev/mapper/backup_disk_E4U5 devid 5 size 10.91TiB used 661.12GiB path /dev/mapper/backup_disk_4RNH devid 6 size 84.00MiB used 0.00B path /dev/mapper/backup-temp root@dodo:/usr/local/sbin# -- Pete Peter Chant 07887 748492 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive 2022-01-14 20:24 ENOSPC on file system with nearly empty 12TB drive Peter Chant @ 2022-01-14 22:12 ` Chris Murphy 2022-01-14 23:09 ` Pete 0 siblings, 1 reply; 5+ messages in thread From: Chris Murphy @ 2022-01-14 22:12 UTC (permalink / raw) To: Peter Chant; +Cc: Btrfs BTRFS On Fri, Jan 14, 2022 at 1:30 PM Peter Chant <pete@petezilla.co.uk> wrote: > > I have a file system that was _very_ full. Not ideal, but there were > reasons. > > It comprised of 2x3TB drives and 2x4TB drives. I added a 12 TB drive > and tried to rebalance only to > > hit ENOSPC. Surely adding a drive to a full file system fixes this? Not necessarily, in fact depending on the profile of the block group type being balanced, e.g. if it's raid1 or raid10, you might need 2 or 4 drives added at once. > Especially one that nearly doubles its capacity? Yet I kept hitting > this issue when > > trying to rebalance and also remove a 4TB full drive. > > Yet I have a drive with 11TB of free space. (24TB array)!? Sounds like metadata block groups are raid1. But it still can't add a new metadata block group on *two* separate devices, so there's enospc. > > > Some info, the kernel message: > > [86348.556169] ------------[ cut here ]------------ > [86348.556175] BTRFS: Transaction aborted (error -28) > [86348.556197] WARNING: CPU: 5 PID: 8736 at fs/btrfs/extent-tree.c:2150 > btrfs_run_delayed_refs+0x1ab/0x1c0 > [86348.556205] Modules linked in: nfsd bridge ipv6 8021q garp mrp stp > llc saa7134_alsa mt20xx tea5767 tda9887 tda8290 tuner uas usb_storage > edac_mce_amd saa7134 tveeprom kvm_amd wmi_bmof videobuf2_dma_sg > videobuf2_memops evdev videobuf2_v4l2 ccp videobuf2_common kvm radeon > videodev snd_hda_codec_via irqbypass snd_hda_codec_generic mc > ledtrig_audio drm_ttm_helper rc_core psmouse ttm snd_hda_codec_hdmi > snd_pcsp serio_raw via_rhine drm_kms_helper snd_hda_intel mii k10temp > snd_intel_dspcfg i2c_piix4 snd_intel_sdw_acpi snd_hda_codec drm > snd_hda_core agpgart snd_hwdep i2c_algo_bit snd_pcm fb_sys_fops > syscopyarea ohci_pci sysfillrect sysimgblt r8169 snd_timer realtek > i2c_core snd mdio_devres soundcore libphy atl1e ehci_pci ohci_hcd > ehci_hcd asus_atk0110 wmi button acpi_cpufreq loop > [86348.556252] CPU: 5 PID: 8736 Comm: btrfs Tainted: G W 5.15.14 #1 > [86348.556255] Hardware name: System manufacturer System Product > Name/M4A78 PRO, BIOS 1701 08/16/2010 > [86348.556257] RIP: 0010:btrfs_run_delayed_refs+0x1ab/0x1c0 > [86348.556260] Code: 03 0f 82 44 74 6c 00 83 f8 fb 0f 84 3b 74 6c 00 83 > f8 e2 0f 84 32 74 6c 00 89 c6 48 c7 c7 f8 12 5e a8 89 04 24 e8 4c 27 6b > 00 <0f> 0b 8b 04 24 e9 17 74 6c 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f > [86348.556262] RSP: 0018:ffff9d5c03837b30 EFLAGS: 00010286 > [86348.556264] RAX: 0000000000000000 RBX: ffff8e6493e2c800 RCX: > 0000000000000000 > [86348.556266] RDX: 0000000000000001 RSI: ffffffffa85a6db3 RDI: > 00000000ffffffff > [86348.556267] RBP: ffff8e6594cb9c98 R08: 0000000000000000 R09: > ffff9d5c03837968 > [86348.556268] R10: ffff9d5c03837960 R11: ffffffffa8912680 R12: > ffff8e65c3761178 > [86348.556269] R13: ffff8e6494452000 R14: ffff8e6494452000 R15: > 0000000000051dc3 > [86348.556271] FS: 00007f1ccfeb48c0(0000) GS:ffff8e6698140000(0000) > knlGS:0000000000000000 > [86348.556272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [86348.556273] CR2: 00007f4239364427 CR3: 000000017f41e000 CR4: > 00000000000006e0 > [86348.556275] Call Trace: > [86348.556278] <TASK> > [86348.556281] btrfs_commit_transaction+0x60/0xa00 > [86348.556285] ? start_transaction+0xd2/0x600 > [86348.556287] relocate_block_group+0x334/0x580 > [86348.556290] btrfs_relocate_block_group+0x175/0x340 > [86348.556292] btrfs_relocate_chunk+0x27/0xe0 > [86348.556296] btrfs_shrink_device+0x260/0x530 > [86348.556298] btrfs_rm_device+0x15b/0x4c0 Looks like a device in the process of being removed, the file system is undergoing shrink, and a bg is being relocated... > [86348.556301] ? btrfs_ioctl+0xe91/0x2df0 > [86348.556302] ? __check_object_size+0x136/0x150 > [86348.556305] ? preempt_count_add+0x68/0xa0 > [86348.556308] btrfs_ioctl+0xf27/0x2df0 > [86348.556310] ? __handle_mm_fault+0xbf9/0x1450 > [86348.556313] ? handle_mm_fault+0xc5/0x290 > [86348.556315] ? __x64_sys_ioctl+0x83/0xb0 > [86348.556318] __x64_sys_ioctl+0x83/0xb0 > [86348.556320] do_syscall_64+0x3b/0xc0 > [86348.556324] entry_SYSCALL_64_after_hwframe+0x44/0xae > [86348.556328] RIP: 0033:0x7f1ccffc54b7 > [86348.556331] Code: 00 00 90 48 8b 05 d9 29 0d 00 64 c7 00 26 00 00 00 > 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f > 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 29 0d 00 f7 d8 64 89 01 48 > [86348.556333] RSP: 002b:00007ffe1f70abe8 EFLAGS: 00000206 ORIG_RAX: > 0000000000000010 > [86348.556335] RAX: ffffffffffffffda RBX: 00007ffe1f70cd90 RCX: > 00007f1ccffc54b7 > [86348.556336] RDX: 00007ffe1f70bc10 RSI: 000000005000943a RDI: > 0000000000000003 > [86348.556337] RBP: 0000000000000001 R08: 1999999999999999 R09: > 0000000000000000 > [86348.556338] R10: 000000000040838b R11: 0000000000000206 R12: > 0000000000000000 > [86348.556339] R13: 00007ffe1f70bc10 R14: 0000000000493660 R15: > 0000000000000003 > [86348.556341] </TASK> > [86348.556342] ---[ end trace c8c92460c79df13b ]--- > [86348.556344] BTRFS: error (device dm-1) in > btrfs_run_delayed_refs:2150: errno=-28 No space left > [86348.556349] BTRFS info (device dm-1): forced readonly And it gets confused because it can't successfully relocate the block group, and it shuts down the file system to prevent confusion being written to the file system. > > btrfs balance start -dusage=94 -musage=94 . -v > > root@dodo:/usr/local/sbin# btrfs fi u /mnt/backup-pool/ > Overall: > Device size: 23.65TiB > Device allocated: 13.24TiB > Device unallocated: 10.41TiB > Device missing: 0.00B > Used: 13.19TiB > Free (estimated): 5.23TiB (min: 5.23TiB) > Data ratio: 2.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 0.00B) > > Data,RAID1: Size:6.60TiB, Used:6.58TiB (99.62%) > /dev/mapper/backup_disk_AC8F 3.59TiB > /dev/mapper/backup_disk_CCDZ 2.69TiB > /dev/mapper/backup_disk_XVN6 2.69TiB > /dev/mapper/backup_disk_E4U5 3.59TiB > /dev/mapper/backup_disk_4RNH 660.09GiB > > Metadata,RAID1: Size:21.00GiB, Used:20.10GiB (95.74%) > /dev/mapper/backup_disk_AC8F 13.00GiB > /dev/mapper/backup_disk_CCDZ 9.00GiB > /dev/mapper/backup_disk_XVN6 9.00GiB > /dev/mapper/backup_disk_E4U5 10.00GiB > /dev/mapper/backup_disk_4RNH 1.00GiB > > System,RAID1: Size:32.00MiB, Used:1.14MiB (3.56%) > /dev/mapper/backup_disk_E4U5 32.00MiB > /dev/mapper/backup_disk_4RNH 32.00MiB > > Unallocated: > /dev/mapper/backup_disk_AC8F 37.00GiB > /dev/mapper/backup_disk_CCDZ 34.00GiB > /dev/mapper/backup_disk_XVN6 34.00GiB > /dev/mapper/backup_disk_E4U5 36.00GiB > /dev/mapper/backup_disk_4RNH 10.27TiB > /dev/mapper/backup-temp 84.00MiB > root@dodo:/usr/local/sbin# OK metadata is raid1, and it's nearly full. But there's multiple devices with at least a couple GiB unallocated such that at least 2 metadata block groups could be created. Are you trying to do balance and device remove at the same time? -- Chris Murphy ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive 2022-01-14 22:12 ` Chris Murphy @ 2022-01-14 23:09 ` Pete 2022-01-15 20:38 ` Chris Murphy 0 siblings, 1 reply; 5+ messages in thread From: Pete @ 2022-01-14 23:09 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS On 14/01/2022 22:12, Chris Murphy wrote: > On Fri, Jan 14, 2022 at 1:30 PM Peter Chant <pete@petezilla.co.uk> wrote: A lot of snipping - I hope I don't cause confusion. >> hit ENOSPC. Surely adding a drive to a full file system fixes this? > > Not necessarily, in fact depending on the profile of the block group > type being balanced, e.g. if it's raid1 or raid10, you might need 2 or > 4 drives added at once. > Raid1 data, raid1 metadata. > >> Especially one that nearly doubles its capacity? Yet I kept hitting >> this issue when >> >> trying to rebalance and also remove a 4TB full drive. >> >> Yet I have a drive with 11TB of free space. (24TB array)!? > > Sounds like metadata block groups are raid1. But it still can't add a > new metadata block group on *two* separate devices, so there's enospc. There are five drives, excluding any (smallish) disk image I've added as a loop device to help, but _four_ are painfully full. ...6281] btrfs_commit_transaction+0x60/0xa00 >> [86348.556285] ? start_transaction+0xd2/0x600 >> [86348.556287] relocate_block_group+0x334/0x580 >> [86348.556290] btrfs_relocate_block_group+0x175/0x340 >> [86348.556292] btrfs_relocate_chunk+0x27/0xe0 >> [86348.556296] btrfs_shrink_device+0x260/0x530 >> [86348.556298] btrfs_rm_device+0x15b/0x4c0 > > Looks like a device in the process of being removed, the file system > is undergoing shrink, and a bg is being relocated... > Well, I was trying to rebalance. But it was very slow. So I changed my mind and though just removing one of the drives, which I intended to do anyway, might help moving data onto the new device. The drive I tried removing is a WD Red with Drive Managed SMR - so that is helping with the speed... ... >> btrfs_run_delayed_refs:2150: errno=-28 No space left >> [86348.556349] BTRFS info (device dm-1): forced readonly > > And it gets confused because it can't successfully relocate the block > group, and it shuts down the file system to prevent confusion being > written to the file system. > Ok, that bit was a bit beyond my understanding. ... > > OK metadata is raid1, and it's nearly full. But there's multiple > devices with at least a couple GiB unallocated such that at least 2 > metadata block groups could be created. The latest (several hours later): root@dodo:/home/pete# btrfs fi u /srv/nfs/backup-pull Overall: Device size: 23.65TiB Device allocated: 13.51TiB Device unallocated: 10.14TiB Device missing: 0.00B Used: 13.51TiB Free (estimated): 5.07TiB (min: 5.07TiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 256.00KiB) Data,RAID1: Size:6.74TiB, Used:6.73TiB (99.98%) /dev/mapper/backup_disk_AC8F 3.43TiB /dev/mapper/backup_disk_CCDZ 2.60TiB /dev/mapper/backup_disk_XVN6 2.60TiB /dev/mapper/backup_disk_E4U5 2.42TiB /dev/mapper/backup_disk_4RNH 2.41TiB Metadata,RAID1: Size:21.00GiB, Used:20.49GiB (97.59%) /dev/mapper/backup_disk_AC8F 12.00GiB /dev/mapper/backup_disk_CCDZ 9.00GiB /dev/mapper/backup_disk_XVN6 9.00GiB /dev/mapper/backup_disk_E4U5 6.00GiB /dev/mapper/backup_disk_4RNH 6.00GiB System,RAID1: Size:32.00MiB, Used:1.28MiB (4.00%) /dev/mapper/backup_disk_E4U5 32.00MiB /dev/mapper/backup_disk_4RNH 32.00MiB Unallocated: /dev/mapper/backup_disk_AC8F 199.99GiB /dev/mapper/backup_disk_CCDZ 120.00GiB /dev/mapper/backup_disk_XVN6 124.00GiB /dev/mapper/backup_disk_E4U5 1.21TiB /dev/mapper/backup_disk_4RNH 8.50TiB root@dodo:/home/pete# > > Are you trying to do balance and device remove at the same time? > > Well, I cancelled a balance, tried a remove and since that ENOSPC'ed I'm now running a full balance. Any suggestions? Just be patient, and hope the balance finishes without ENOSPC? Go for the remove again. I'd like to remove a 4TB drive if I can without adding a 6th HD to the system. Still don't understand why I might need more than one practically empty drive for raid1? Or, given that I've got another 12TB drive on the way, just abandon this file system (apart from reading any relevant data) and create a new file system as single, and migrate the current 12TB disk over later on to go back to raid1? I'm wondering if part of the problem is a 3.6TB disk image that is sitting on the file system (fixing someone else's external drive). Deleting that might speed things up, but since I don't think that drive is fully backed up I'd rather keep the disk image until I am sure that all is well. (I don't need the backup lecture...) Apart from a loop device mounted drive image I might be able to mount another disk in the five disk file system - but I'd rather not. I'd have to add a controller (have one spare) which might move the boot device (not using initrd and UUIDs) - and shoehorning another drive in is a little tricky but not impossible. Its an old midi tower cased machine repurposed as my backup server. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive 2022-01-14 23:09 ` Pete @ 2022-01-15 20:38 ` Chris Murphy 2022-01-16 12:27 ` Pete 0 siblings, 1 reply; 5+ messages in thread From: Chris Murphy @ 2022-01-15 20:38 UTC (permalink / raw) To: Pete; +Cc: Chris Murphy, Btrfs BTRFS On Fri, Jan 14, 2022 at 4:09 PM Pete <pete@petezilla.co.uk> wrote: > Any suggestions? Just be patient, and hope the balance finishes without > ENOSPC? Go for the remove again. I'd like to remove a 4TB drive if I > can without adding a 6th HD to the system. Still don't understand why I > might need more than one practically empty drive for raid1? When bg profile is raid1, any time the file system wants to add another block group, it must create a chunk on 2 devices at the same time for it to succeed or else you get ENOSPC. The question is why two chunks can't be created given all the unallocated space you have even without the empty drive. Ordinarily you want to avoid doing metadata balance. Balancing data is ok, it amounts to defragmenting free space, and returning excess into unallocated space which can then be used to create any type of block group. > > Or, given that I've got another 12TB drive on the way, just abandon this > file system (apart from reading any relevant data) and create a new file > system as single, and migrate the current 12TB disk over later on to go > back to raid1? > > I'm wondering if part of the problem is a 3.6TB disk image that is > sitting on the file system (fixing someone else's external drive). > Deleting that might speed things up, but since I don't think that drive > is fully backed up I'd rather keep the disk image until I am sure that > all is well. (I don't need the backup lecture...) I don't think the large image file is the problem. In my opinion, you've hit a bug. There's plenty of unallocated space on multiple drives. I think what's going on is an old bug that might not be fixed yet where metadata overcommit is allowed to overestimate due to the single large empty drive with a ton of space on it. But since the overcommit can't be fulfilled on at least two drives, you get ENOSPC even though it's not actually trying to create that many block groups at once. So what I suggest doing is removing the mostly empty device, and only add devices in pairs when using raid1. -- Chris Murphy ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ENOSPC on file system with nearly empty 12TB drive 2022-01-15 20:38 ` Chris Murphy @ 2022-01-16 12:27 ` Pete 0 siblings, 0 replies; 5+ messages in thread From: Pete @ 2022-01-16 12:27 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS On 1/15/22 20:38, Chris Murphy wrote: > On Fri, Jan 14, 2022 at 4:09 PM Pete <pete@petezilla.co.uk> wrote: > >> Any suggestions? Just be patient, and hope the balance finishes without >> ENOSPC? Go for the remove again. I'd like to remove a 4TB drive if I >> can without adding a 6th HD to the system. Still don't understand why I >> might need more than one practically empty drive for raid1? > > When bg profile is raid1, any time the file system wants to add > another block group, it must create a chunk on 2 devices at the same > time for it to succeed or else you get ENOSPC. The question is why two > chunks can't be created given all the unallocated space you have even > without the empty drive. So although in % terms the drives were very full the > 30GB of free space on each ought to have been more than sufficient? A little care is needed as the ENOSPC did not occur at exactly the time I ran btrfs fi show and btrfs fi usage. So perhaps my post is the the most rock solid evidence if there is an issue. However, I had multiple instances of ENOSPC at the time, even after it had spent a few hours balancing. > > Ordinarily you want to avoid doing metadata balance. Balancing data is > ok, it amounts to defragmenting free space, and returning excess into > unallocated space which can then be used to create any type of block > group. > OK, but various resources show metadata balance being used when trying to sort a ENOSPC issue, e.g. https://www.suse.com/support/kb/doc/?id=000019789 But there are others that I don't seem to be able to find again. I should have noted the URLs when I was trying to sort the issue... > > I don't think the large image file is the problem. > > In my opinion, you've hit a bug. There's plenty of unallocated space > on multiple drives. I think what's going on is an old bug that might > not be fixed yet where metadata overcommit is allowed to overestimate > due to the single large empty drive with a ton of space on it. But > since the overcommit can't be fulfilled on at least two drives, you > get ENOSPC even though it's not actually trying to create that many > block groups at once. > My naive but logical expectation was that balance would start aggressively moving data to the empty drive. Most of the guidance I see for recovering from an ENOSPC indicates that adding a single device would be sufficient, I do note, however, that if you scroll down and read it all, some sites do point this out: https://wiki.tnonline.net/w/Btrfs/ENOSPC I also note that the various guides / howtos online are not necessarily within the control people who are on this mailing list - I'm not implying that the active maintainers are/should be responsible about everything written about btrfs online. My strong recollection from reading this mailing list and when I have searched on line was that adding one device only was mentioned for managing an ENOSPC issue. However, from a limited amount of searching I'm not sure that I can back the up with references. Perhaps that is just my perception? Adding the new drive plus a loop device allowed me to progressively rebalance using dusage and musage filters until I could start a full rebalance without the loop device. Interestingly, though this array had been running, raid1, for a while with four drives, without rebalancing, metadata was only stored on two of the drives. > So what I suggest doing is removing the mostly empty device, and only > add devices in pairs when using raid1. Should there be a caveat added to this "For very full btrfs raid1 file systems only add devices in at least pairs."? Being able to add devices in a fairly ad hoc manner is a great strength for btrfs. It seems to be that I am past the point of removing the larger device now as I have > 500 GB free on the smaller drives now, have removed the loop device and am no longer hitting ENOSPC. I probably would have to start deleting some of my backups to remove the new larger device just to that the original 4 device array was not so cripplingly full. Thank you for your help. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-01-16 12:28 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-01-14 20:24 ENOSPC on file system with nearly empty 12TB drive Peter Chant 2022-01-14 22:12 ` Chris Murphy 2022-01-14 23:09 ` Pete 2022-01-15 20:38 ` Chris Murphy 2022-01-16 12:27 ` Pete
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).