* No space left errors on shutdown with systemd-homed /home dir @ 2022-01-25 17:46 Apostolos B. 2022-01-26 21:50 ` Boris Burkov 0 siblings, 1 reply; 20+ messages in thread From: Apostolos B. @ 2022-01-25 17:46 UTC (permalink / raw) To: linux-btrfs Hello. When i shut down my pc i get No space left errors -even though i have plenty of space in both / and home dirs- and this message on the journal: Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------ Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28) Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs] Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc encrypted_keys trusted asn1_e> Ιαν 25 14:34:32 mainland kernel: snd_pcm_dmaengine kvm snd_hda_intel iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul snd_intel_sdw_acpi hid_mul> Ιαν 25 14:34:32 mainland kernel: int340x_thermal_zone tpm_tis tpm_tis_core wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad ipmi_devint> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor Tainted: G W 5.16.2-arch1-1 #1 86fbf2c313cc37a553d65deb81d98e9dcc2a3659 Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO., LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP 04/15/2021 Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950 [btrfs] Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4 bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0 e8 24 6c 28 e> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246 Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000 R09: 0000000000000000 Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ffffffe4 Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0 R15: fffffffffffffff7 Ιαν 25 14:34:32 mainland kernel: FS: 00007f336b49ea80(0000) GS:ffff9823c3700000(0000) knlGS:0000000000000000 Ιαν 25 14:34:32 mainland kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002 CR4: 0000000000770ee0 Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554 Ιαν 25 14:34:32 mainland kernel: Call Trace: Ιαν 25 14:34:32 mainland kernel: <TASK> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: ? __reserve_bytes+0x164/0x7d0 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: relocate_block_group+0x6e/0x5a0 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_chunk+0x27/0x100 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: btrfs_shrink_device+0x277/0x5a0 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl_resize+0x449/0x470 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl+0x1fa8/0x2fc0 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: ? btrfs_statfs+0x418/0x570 [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] Ιαν 25 14:34:32 mainland kernel: ? _copy_to_user+0x1c/0x50 Ιαν 25 14:34:32 mainland kernel: ? do_statfs_native+0xaf/0xe0 Ιαν 25 14:34:32 mainland kernel: ? __seccomp_filter+0x39e/0x570 Ιαν 25 14:34:32 mainland kernel: ? __x64_sys_ioctl+0x8b/0xd0 Ιαν 25 14:34:32 mainland kernel: __x64_sys_ioctl+0x8b/0xd0 Ιαν 25 14:34:32 mainland kernel: do_syscall_64+0x59/0x90 Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 Ιαν 25 14:34:32 mainland kernel: ? exc_page_fault+0x72/0x180 Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 0> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000 RCX: 00007f336baa359b Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403 RDI: 0000000000000004 Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000 R09: 00007ffc945a0370 Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246 R12: 00007ffc945a0570 Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0 R15: 00007ffc945a1920 Ιαν 25 14:34:32 mainland kernel: </TASK> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]--- Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in __btrfs_free_extent:3066: errno=-28 No space left Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2149: errno=-28 No space left The dm-0 device is my /home directory and is set up using systemd-homed Kernel version: 5.16.2 Systemd version: 250.3 btrfs-progs version: 5.16 It seems to cause no issues thus far but a solution would be good to have. Thanks in advance. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-25 17:46 No space left errors on shutdown with systemd-homed /home dir Apostolos B. @ 2022-01-26 21:50 ` Boris Burkov 2022-01-26 22:07 ` Apostolos B. 0 siblings, 1 reply; 20+ messages in thread From: Boris Burkov @ 2022-01-26 21:50 UTC (permalink / raw) To: Apostolos B.; +Cc: linux-btrfs On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote: > Hello. > > When i shut down my pc i get No space left errors -even though i have plenty > of space in both / and home dirs- and this message on the journal: How did you conclude you have plenty of space? df can be misleading with btrfs, for example. Can you please post the output of 'btrfs filesystem usage /home' Thanks, Boris > > Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block > group 2177892352 flags data > Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block > group 1104150528 flags data > Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block > group 30408704 flags metadata|dup > Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------ > Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28) > Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at > fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs] > Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm > snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc > encrypted_keys trusted asn1_e> > Ιαν 25 14:34:32 mainland kernel: snd_pcm_dmaengine kvm snd_hda_intel > iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul > snd_intel_sdw_acpi hid_mul> > Ιαν 25 14:34:32 mainland kernel: int340x_thermal_zone tpm_tis tpm_tis_core > wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad > ipmi_devint> > Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor > Tainted: G W 5.16.2-arch1-1 #1 > 86fbf2c313cc37a553d65deb81d98e9dcc2a3659 > Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO., > LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP > 04/15/2021 > Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950 > [btrfs] > Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4 > bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0 > e8 24 6c 28 e> > Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246 > Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000 > RCX: 0000000000000000 > Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000 > RDI: 0000000000000000 > Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000 > R09: 0000000000000000 > Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000 > R12: 00000000ffffffe4 > Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0 > R15: fffffffffffffff7 > Ιαν 25 14:34:32 mainland kernel: FS: 00007f336b49ea80(0000) > GS:ffff9823c3700000(0000) knlGS:0000000000000000 > Ιαν 25 14:34:32 mainland kernel: CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002 > CR4: 0000000000770ee0 > Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554 > Ιαν 25 14:34:32 mainland kernel: Call Trace: > Ιαν 25 14:34:32 mainland kernel: <TASK> > Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0 > [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: ? __reserve_bytes+0x164/0x7d0 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: relocate_block_group+0x6e/0x5a0 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340 > [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_chunk+0x27/0x100 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: btrfs_shrink_device+0x277/0x5a0 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl_resize+0x449/0x470 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl+0x1fa8/0x2fc0 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: ? btrfs_statfs+0x418/0x570 [btrfs > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > Ιαν 25 14:34:32 mainland kernel: ? _copy_to_user+0x1c/0x50 > Ιαν 25 14:34:32 mainland kernel: ? do_statfs_native+0xaf/0xe0 > Ιαν 25 14:34:32 mainland kernel: ? __seccomp_filter+0x39e/0x570 > Ιαν 25 14:34:32 mainland kernel: ? __x64_sys_ioctl+0x8b/0xd0 > Ιαν 25 14:34:32 mainland kernel: __x64_sys_ioctl+0x8b/0xd0 > Ιαν 25 14:34:32 mainland kernel: do_syscall_64+0x59/0x90 > Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 > Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 > Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 > Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 > Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 > Ιαν 25 14:34:32 mainland kernel: ? exc_page_fault+0x72/0x180 > Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae > Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b > Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff > ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 > 00 00 00 0f 0> > Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246 > ORIG_RAX: 0000000000000010 > Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000 > RCX: 00007f336baa359b > Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403 > RDI: 0000000000000004 > Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000 > R09: 00007ffc945a0370 > Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246 > R12: 00007ffc945a0570 > Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0 > R15: 00007ffc945a1920 > Ιαν 25 14:34:32 mainland kernel: </TASK> > Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]--- > Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in > __btrfs_free_extent:3066: errno=-28 No space left > Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly > Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in > btrfs_run_delayed_refs:2149: errno=-28 No space left > > The dm-0 device is my /home directory and is set up using systemd-homed > > Kernel version: 5.16.2 > > Systemd version: 250.3 > > btrfs-progs version: 5.16 > > It seems to cause no issues thus far but a solution would be good to have. > > Thanks in advance. > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-26 21:50 ` Boris Burkov @ 2022-01-26 22:07 ` Apostolos B. 2022-01-26 23:19 ` Boris Burkov 0 siblings, 1 reply; 20+ messages in thread From: Apostolos B. @ 2022-01-26 22:07 UTC (permalink / raw) To: Boris Burkov; +Cc: linux-btrfs This is what homectl inspect user reports: Disk Size: 128.0G Disk Usage: 3.8G (= 3.1%) Disk Free: 124.0G (= 96.9%) and this is what btrfs usage reports: sudo btrfs filesystem usage /home/toliz Overall: Device size: 127.98GiB Device allocated: 4.02GiB Device unallocated: 123.96GiB Device missing: 0.00B Used: 1.89GiB Free (estimated): 124.10GiB (min: 62.12GiB) Free (statfs, df): 124.10GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 5.14MiB (used: 0.00B) Multiple profiles: no Data,single: Size:2.01GiB, Used:1.86GiB (92.73%) /dev/mapper/home-toliz 2.01GiB Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%) /dev/mapper/home-toliz 2.00GiB System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%) /dev/mapper/home-toliz 16.00MiB Unallocated: /dev/mapper/home-toliz 123.96GiB On 26/1/22 23:50, Boris Burkov wrote: > On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote: >> Hello. >> >> When i shut down my pc i get No space left errors -even though i have plenty >> of space in both / and home dirs- and this message on the journal: > How did you conclude you have plenty of space? df can be misleading with > btrfs, for example. Can you please post the output of > 'btrfs filesystem usage /home' > > Thanks, > Boris > >> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block >> group 2177892352 flags data >> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block >> group 1104150528 flags data >> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block >> group 30408704 flags metadata|dup >> Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------ >> Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28) >> Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at >> fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs] >> Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm >> snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc >> encrypted_keys trusted asn1_e> >> Ιαν 25 14:34:32 mainland kernel: snd_pcm_dmaengine kvm snd_hda_intel >> iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul >> snd_intel_sdw_acpi hid_mul> >> Ιαν 25 14:34:32 mainland kernel: int340x_thermal_zone tpm_tis tpm_tis_core >> wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad >> ipmi_devint> >> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor >> Tainted: G W 5.16.2-arch1-1 #1 >> 86fbf2c313cc37a553d65deb81d98e9dcc2a3659 >> Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO., >> LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP >> 04/15/2021 >> Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950 >> [btrfs] >> Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4 >> bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0 >> e8 24 6c 28 e> >> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246 >> Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000 >> RCX: 0000000000000000 >> Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000 >> RDI: 0000000000000000 >> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000 >> R09: 0000000000000000 >> Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000 >> R12: 00000000ffffffe4 >> Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0 >> R15: fffffffffffffff7 >> Ιαν 25 14:34:32 mainland kernel: FS: 00007f336b49ea80(0000) >> GS:ffff9823c3700000(0000) knlGS:0000000000000000 >> Ιαν 25 14:34:32 mainland kernel: CS: 0010 DS: 0000 ES: 0000 CR0: >> 0000000080050033 >> Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002 >> CR4: 0000000000770ee0 >> Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554 >> Ιαν 25 14:34:32 mainland kernel: Call Trace: >> Ιαν 25 14:34:32 mainland kernel: <TASK> >> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0 >> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: ? __reserve_bytes+0x164/0x7d0 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: relocate_block_group+0x6e/0x5a0 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340 >> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_chunk+0x27/0x100 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: btrfs_shrink_device+0x277/0x5a0 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl_resize+0x449/0x470 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl+0x1fa8/0x2fc0 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: ? btrfs_statfs+0x418/0x570 [btrfs >> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >> Ιαν 25 14:34:32 mainland kernel: ? _copy_to_user+0x1c/0x50 >> Ιαν 25 14:34:32 mainland kernel: ? do_statfs_native+0xaf/0xe0 >> Ιαν 25 14:34:32 mainland kernel: ? __seccomp_filter+0x39e/0x570 >> Ιαν 25 14:34:32 mainland kernel: ? __x64_sys_ioctl+0x8b/0xd0 >> Ιαν 25 14:34:32 mainland kernel: __x64_sys_ioctl+0x8b/0xd0 >> Ιαν 25 14:34:32 mainland kernel: do_syscall_64+0x59/0x90 >> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >> Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 >> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >> Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 >> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >> Ιαν 25 14:34:32 mainland kernel: ? exc_page_fault+0x72/0x180 >> Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae >> Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b >> Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff >> ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 >> 00 00 00 0f 0> >> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246 >> ORIG_RAX: 0000000000000010 >> Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000 >> RCX: 00007f336baa359b >> Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403 >> RDI: 0000000000000004 >> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000 >> R09: 00007ffc945a0370 >> Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246 >> R12: 00007ffc945a0570 >> Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0 >> R15: 00007ffc945a1920 >> Ιαν 25 14:34:32 mainland kernel: </TASK> >> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]--- >> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in >> __btrfs_free_extent:3066: errno=-28 No space left >> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly >> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in >> btrfs_run_delayed_refs:2149: errno=-28 No space left >> >> The dm-0 device is my /home directory and is set up using systemd-homed >> >> Kernel version: 5.16.2 >> >> Systemd version: 250.3 >> >> btrfs-progs version: 5.16 >> >> It seems to cause no issues thus far but a solution would be good to have. >> >> Thanks in advance. >> ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-26 22:07 ` Apostolos B. @ 2022-01-26 23:19 ` Boris Burkov 2022-01-26 23:29 ` Apostolos B. 2022-01-27 20:48 ` Chris Murphy 0 siblings, 2 replies; 20+ messages in thread From: Boris Burkov @ 2022-01-26 23:19 UTC (permalink / raw) To: Apostolos B.; +Cc: linux-btrfs On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote: > This is what homectl inspect user reports: > > Disk Size: 128.0G > Disk Usage: 3.8G (= 3.1%) > Disk Free: 124.0G (= 96.9%) > > and this is what btrfs usage reports: > > sudo btrfs filesystem usage /home/toliz > > Overall: > Device size: 127.98GiB > Device allocated: 4.02GiB > Device unallocated: 123.96GiB > Device missing: 0.00B > Used: 1.89GiB > Free (estimated): 124.10GiB (min: 62.12GiB) > Free (statfs, df): 124.10GiB > Data ratio: 1.00 > Metadata ratio: 2.00 > Global reserve: 5.14MiB (used: 0.00B) > Multiple profiles: no > > Data,single: Size:2.01GiB, Used:1.86GiB (92.73%) > /dev/mapper/home-toliz 2.01GiB > > Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%) > /dev/mapper/home-toliz 2.00GiB > > System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%) > /dev/mapper/home-toliz 16.00MiB > > Unallocated: > /dev/mapper/home-toliz 123.96GiB > OK, there is plenty of unallocated space, thanks for confirming. Looking at the stack trace a bit more, the only thing that really sticks out as suspicious to me is btrfs_shrink_device, I'm not sure who would want to do that or why. It might be interesting to trace it and see if we can catch the parameters/caller in the act. If you have bpftrace setup on your system, could you try to setup something like: bpftrace -e 'kprobe:btrfs_shrink_device { printf("%s %llu %s\n", comm, arg1, kstack); }' to write to a file during shutdown? > > On 26/1/22 23:50, Boris Burkov wrote: > > On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote: > > > Hello. > > > > > > When i shut down my pc i get No space left errors -even though i have plenty > > > of space in both / and home dirs- and this message on the journal: > > How did you conclude you have plenty of space? df can be misleading with > > btrfs, for example. Can you please post the output of > > 'btrfs filesystem usage /home' > > > > Thanks, > > Boris > > > > > Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block > > > group 2177892352 flags data > > > Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block > > > group 1104150528 flags data > > > Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block > > > group 30408704 flags metadata|dup > > > Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------ > > > Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28) > > > Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at > > > fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs] > > > Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm > > > snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc > > > encrypted_keys trusted asn1_e> > > > Ιαν 25 14:34:32 mainland kernel: snd_pcm_dmaengine kvm snd_hda_intel > > > iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul > > > snd_intel_sdw_acpi hid_mul> > > > Ιαν 25 14:34:32 mainland kernel: int340x_thermal_zone tpm_tis tpm_tis_core > > > wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad > > > ipmi_devint> > > > Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor > > > Tainted: G W 5.16.2-arch1-1 #1 > > > 86fbf2c313cc37a553d65deb81d98e9dcc2a3659 > > > Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO., > > > LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP > > > 04/15/2021 > > > Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950 > > > [btrfs] > > > Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4 > > > bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0 > > > e8 24 6c 28 e> > > > Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246 > > > Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000 > > > RCX: 0000000000000000 > > > Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000 > > > RDI: 0000000000000000 > > > Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000 > > > R09: 0000000000000000 > > > Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000 > > > R12: 00000000ffffffe4 > > > Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0 > > > R15: fffffffffffffff7 > > > Ιαν 25 14:34:32 mainland kernel: FS: 00007f336b49ea80(0000) > > > GS:ffff9823c3700000(0000) knlGS:0000000000000000 > > > Ιαν 25 14:34:32 mainland kernel: CS: 0010 DS: 0000 ES: 0000 CR0: > > > 0000000080050033 > > > Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002 > > > CR4: 0000000000770ee0 > > > Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554 > > > Ιαν 25 14:34:32 mainland kernel: Call Trace: > > > Ιαν 25 14:34:32 mainland kernel: <TASK> > > > Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0 > > > [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: ? __reserve_bytes+0x164/0x7d0 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: relocate_block_group+0x6e/0x5a0 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340 > > > [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_chunk+0x27/0x100 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: btrfs_shrink_device+0x277/0x5a0 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl_resize+0x449/0x470 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl+0x1fa8/0x2fc0 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: ? btrfs_statfs+0x418/0x570 [btrfs > > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b] > > > Ιαν 25 14:34:32 mainland kernel: ? _copy_to_user+0x1c/0x50 > > > Ιαν 25 14:34:32 mainland kernel: ? do_statfs_native+0xaf/0xe0 > > > Ιαν 25 14:34:32 mainland kernel: ? __seccomp_filter+0x39e/0x570 > > > Ιαν 25 14:34:32 mainland kernel: ? __x64_sys_ioctl+0x8b/0xd0 > > > Ιαν 25 14:34:32 mainland kernel: __x64_sys_ioctl+0x8b/0xd0 > > > Ιαν 25 14:34:32 mainland kernel: do_syscall_64+0x59/0x90 > > > Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 > > > Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 > > > Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 > > > Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 > > > Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 > > > Ιαν 25 14:34:32 mainland kernel: ? exc_page_fault+0x72/0x180 > > > Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae > > > Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b > > > Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff > > > ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 > > > 00 00 00 0f 0> > > > Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246 > > > ORIG_RAX: 0000000000000010 > > > Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000 > > > RCX: 00007f336baa359b > > > Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403 > > > RDI: 0000000000000004 > > > Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000 > > > R09: 00007ffc945a0370 > > > Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246 > > > R12: 00007ffc945a0570 > > > Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0 > > > R15: 00007ffc945a1920 > > > Ιαν 25 14:34:32 mainland kernel: </TASK> > > > Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]--- > > > Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in > > > __btrfs_free_extent:3066: errno=-28 No space left > > > Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly > > > Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in > > > btrfs_run_delayed_refs:2149: errno=-28 No space left > > > > > > The dm-0 device is my /home directory and is set up using systemd-homed > > > > > > Kernel version: 5.16.2 > > > > > > Systemd version: 250.3 > > > > > > btrfs-progs version: 5.16 > > > > > > It seems to cause no issues thus far but a solution would be good to have. > > > > > > Thanks in advance. > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-26 23:19 ` Boris Burkov @ 2022-01-26 23:29 ` Apostolos B. 2022-01-27 7:59 ` Wang Yugui 2022-01-27 19:13 ` Goffredo Baroncelli 2022-01-27 20:48 ` Chris Murphy 1 sibling, 2 replies; 20+ messages in thread From: Apostolos B. @ 2022-01-26 23:29 UTC (permalink / raw) To: Boris Burkov; +Cc: linux-btrfs I don't have a bpftrace setup and sadly i cant do much debugging on this machine. However i am sure its systemd that is involved in it. The few lines before the crash read: Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant. Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS device /dev/mapper/home-toliz, and validated password. Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully re-activated LUKS device. Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used loopback device /dev/loop0. Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size = 137436856320, image = 137438953472 Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 127.9G → 1.7G. Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 124.8G. Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file system: Read-only file system Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup On 27/1/22 01:19, Boris Burkov wrote: > On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote: >> This is what homectl inspect user reports: >> >> Disk Size: 128.0G >> Disk Usage: 3.8G (= 3.1%) >> Disk Free: 124.0G (= 96.9%) >> >> and this is what btrfs usage reports: >> >> sudo btrfs filesystem usage /home/toliz >> >> Overall: >> Device size: 127.98GiB >> Device allocated: 4.02GiB >> Device unallocated: 123.96GiB >> Device missing: 0.00B >> Used: 1.89GiB >> Free (estimated): 124.10GiB (min: 62.12GiB) >> Free (statfs, df): 124.10GiB >> Data ratio: 1.00 >> Metadata ratio: 2.00 >> Global reserve: 5.14MiB (used: 0.00B) >> Multiple profiles: no >> >> Data,single: Size:2.01GiB, Used:1.86GiB (92.73%) >> /dev/mapper/home-toliz 2.01GiB >> >> Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%) >> /dev/mapper/home-toliz 2.00GiB >> >> System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%) >> /dev/mapper/home-toliz 16.00MiB >> >> Unallocated: >> /dev/mapper/home-toliz 123.96GiB >> > OK, there is plenty of unallocated space, thanks for confirming. > > Looking at the stack trace a bit more, the only thing that really sticks > out as suspicious to me is btrfs_shrink_device, I'm not sure who would > want to do that or why. > > It might be interesting to trace it and see if we can catch the > parameters/caller in the act. If you have bpftrace setup on your system, > could you try to setup something like: > > bpftrace -e 'kprobe:btrfs_shrink_device { printf("%s %llu %s\n", comm, arg1, kstack); }' > > to write to a file during shutdown? > >> On 26/1/22 23:50, Boris Burkov wrote: >>> On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote: >>>> Hello. >>>> >>>> When i shut down my pc i get No space left errors -even though i have plenty >>>> of space in both / and home dirs- and this message on the journal: >>> How did you conclude you have plenty of space? df can be misleading with >>> btrfs, for example. Can you please post the output of >>> 'btrfs filesystem usage /home' >>> >>> Thanks, >>> Boris >>> >>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block >>>> group 2177892352 flags data >>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block >>>> group 1104150528 flags data >>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block >>>> group 30408704 flags metadata|dup >>>> Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------ >>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28) >>>> Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at >>>> fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs] >>>> Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm >>>> snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc >>>> encrypted_keys trusted asn1_e> >>>> Ιαν 25 14:34:32 mainland kernel: snd_pcm_dmaengine kvm snd_hda_intel >>>> iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul >>>> snd_intel_sdw_acpi hid_mul> >>>> Ιαν 25 14:34:32 mainland kernel: int340x_thermal_zone tpm_tis tpm_tis_core >>>> wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad >>>> ipmi_devint> >>>> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor >>>> Tainted: G W 5.16.2-arch1-1 #1 >>>> 86fbf2c313cc37a553d65deb81d98e9dcc2a3659 >>>> Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO., >>>> LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP >>>> 04/15/2021 >>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950 >>>> [btrfs] >>>> Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4 >>>> bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0 >>>> e8 24 6c 28 e> >>>> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246 >>>> Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000 >>>> RCX: 0000000000000000 >>>> Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000 >>>> RDI: 0000000000000000 >>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000 >>>> R09: 0000000000000000 >>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000 >>>> R12: 00000000ffffffe4 >>>> Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0 >>>> R15: fffffffffffffff7 >>>> Ιαν 25 14:34:32 mainland kernel: FS: 00007f336b49ea80(0000) >>>> GS:ffff9823c3700000(0000) knlGS:0000000000000000 >>>> Ιαν 25 14:34:32 mainland kernel: CS: 0010 DS: 0000 ES: 0000 CR0: >>>> 0000000080050033 >>>> Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002 >>>> CR4: 0000000000770ee0 >>>> Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554 >>>> Ιαν 25 14:34:32 mainland kernel: Call Trace: >>>> Ιαν 25 14:34:32 mainland kernel: <TASK> >>>> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0 >>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: ? __reserve_bytes+0x164/0x7d0 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: relocate_block_group+0x6e/0x5a0 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340 >>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_chunk+0x27/0x100 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: btrfs_shrink_device+0x277/0x5a0 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl_resize+0x449/0x470 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl+0x1fa8/0x2fc0 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: ? btrfs_statfs+0x418/0x570 [btrfs >>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>> Ιαν 25 14:34:32 mainland kernel: ? _copy_to_user+0x1c/0x50 >>>> Ιαν 25 14:34:32 mainland kernel: ? do_statfs_native+0xaf/0xe0 >>>> Ιαν 25 14:34:32 mainland kernel: ? __seccomp_filter+0x39e/0x570 >>>> Ιαν 25 14:34:32 mainland kernel: ? __x64_sys_ioctl+0x8b/0xd0 >>>> Ιαν 25 14:34:32 mainland kernel: __x64_sys_ioctl+0x8b/0xd0 >>>> Ιαν 25 14:34:32 mainland kernel: do_syscall_64+0x59/0x90 >>>> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >>>> Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 >>>> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >>>> Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 >>>> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >>>> Ιαν 25 14:34:32 mainland kernel: ? exc_page_fault+0x72/0x180 >>>> Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae >>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b >>>> Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff >>>> ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 >>>> 00 00 00 0f 0> >>>> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246 >>>> ORIG_RAX: 0000000000000010 >>>> Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000 >>>> RCX: 00007f336baa359b >>>> Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403 >>>> RDI: 0000000000000004 >>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000 >>>> R09: 00007ffc945a0370 >>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246 >>>> R12: 00007ffc945a0570 >>>> Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0 >>>> R15: 00007ffc945a1920 >>>> Ιαν 25 14:34:32 mainland kernel: </TASK> >>>> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]--- >>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in >>>> __btrfs_free_extent:3066: errno=-28 No space left >>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly >>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in >>>> btrfs_run_delayed_refs:2149: errno=-28 No space left >>>> >>>> The dm-0 device is my /home directory and is set up using systemd-homed >>>> >>>> Kernel version: 5.16.2 >>>> >>>> Systemd version: 250.3 >>>> >>>> btrfs-progs version: 5.16 >>>> >>>> It seems to cause no issues thus far but a solution would be good to have. >>>> >>>> Thanks in advance. >>>> ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-26 23:29 ` Apostolos B. @ 2022-01-27 7:59 ` Wang Yugui 2022-01-27 8:51 ` Wang Yugui 2022-01-27 19:13 ` Goffredo Baroncelli 1 sibling, 1 reply; 20+ messages in thread From: Wang Yugui @ 2022-01-27 7:59 UTC (permalink / raw) To: Apostolos B.; +Cc: Boris Burkov, linux-btrfs Hi, Why 'resize image size 128.0G → 1.8G'? This 'No space left errors ' happen when 'resize image size 128.0G → 1.8G'. and it will not happen when ''resize image size 128.0G → 2.0G'? Best Regards Wang Yugui (wangyugui@e16-tech.com) 2022/01/27 > I don't have a bpftrace setup and sadly i cant do much debugging on this machine. > > However i am sure its systemd that is involved in it. The few lines before the crash read: > > Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant. > Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS device /dev/mapper/home-toliz, and validated password. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully re-activated LUKS device. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used loopback device /dev/loop0. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size = 137436856320, image = 137438953472 > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 127.9G → 1.7G. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 124.8G. > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data > Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file system: Read-only file system > Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup > > On 27/1/22 01:19, Boris Burkov wrote: > > On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote: > >> ?This is what homectl inspect user reports: > >> > >> ? Disk Size: 128.0G > >> ? Disk Usage: 3.8G (= 3.1%) > >> ? Disk Free: 124.0G (= 96.9%) ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-27 7:59 ` Wang Yugui @ 2022-01-27 8:51 ` Wang Yugui 0 siblings, 0 replies; 20+ messages in thread From: Wang Yugui @ 2022-01-27 8:51 UTC (permalink / raw) To: Apostolos B., Boris Burkov, linux-btrfs Hi, > Why 'resize image size 128.0G → 1.8G'? > > This 'No space left errors ' happen when 'resize image size 128.0G → 1.8G'. > and it will not happen when ''resize image size 128.0G → 2.0G'? 'btrfs filesystem resize' and kernel btrfs_shrink_device() now does not check whether the new shrinked size is too small. if a too small shrinked size is specified to 'btrfs filesystem resize', a same call trace will happen. the min shrinkable size is in the unit of btrfs chunk, not the unit of btrfs blockt. This value is a little difficult to get correctly, so we need a new feature such as 'btrfs filesystem resize min' in addition to the exist 'btrfs filesystem resize max' Best Regards Wang Yugui (wangyugui@e16-tech.com) 2022/01/27 > > Best Regards > Wang Yugui (wangyugui@e16-tech.com) > 2022/01/27 > > > I don't have a bpftrace setup and sadly i cant do much debugging on this machine. > > > > However i am sure its systemd that is involved in it. The few lines before the crash read: > > > > Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant. > > Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS device /dev/mapper/home-toliz, and validated password. > > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully re-activated LUKS device. > > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used loopback device /dev/loop0. > > Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size = 137436856320, image = 137438953472 > > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 127.9G → 1.7G. > > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 124.8G. > > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data > > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data > > Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file system: Read-only file system > > Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup > > > > On 27/1/22 01:19, Boris Burkov wrote: > > > On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote: > > >> ?This is what homectl inspect user reports: > > >> > > >> ? Disk Size: 128.0G > > >> ? Disk Usage: 3.8G (= 3.1%) > > >> ? Disk Free: 124.0G (= 96.9%) > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-26 23:29 ` Apostolos B. 2022-01-27 7:59 ` Wang Yugui @ 2022-01-27 19:13 ` Goffredo Baroncelli 1 sibling, 0 replies; 20+ messages in thread From: Goffredo Baroncelli @ 2022-01-27 19:13 UTC (permalink / raw) To: Apostolos B., Boris Burkov; +Cc: linux-btrfs On 27/01/2022 00.29, Apostolos B. wrote: > I don't have a bpftrace setup and sadly i cant do much debugging on this machine. > > However i am sure its systemd that is involved in it. The few lines before the crash read: > > Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant. > Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS device /dev/mapper/home-toliz, and validated password. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully re-activated LUKS device. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used loopback device /dev/loop0. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size = 137436856320, image = 137438953472 > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 127.9G → 1.7G. > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 124.8G. > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data > Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file system: Read-only file system > Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup From the homectl manpage [...] --auto-resize-mode= Configures whether to automatically grow and/or shrink the backing file system on login and logout. [...] So it seems that systemd is doing a shrinking at logout time, because it was configured to do it. Looking at the code, it seems that the systemd function get_smallest_fs_size() is calculating the minimum needed storage size; it does that computing fstatfs(fd, &sfs) needed = sfs.f_blocks - sfs.f_bfree; needed *= sfs.f_bsize and adding a 'safety margin' of needed += HOME_MIN_FREE; (where HOME_MIN_FREE is 16M) I think that this is a too simplistic way to compute the needing space for BTRFS. But a comment of a developer with a more deeper knowledge of this topic may showed that I am wrong. Anyway, it seems that the problem can be easily solved disabling the systemd auto shrink feature. Anyway I suggest to open a bug in systemd. What I am not sure if BTRFS should accept shrinking value that can't satisfy. My guess, is that it is not simple for BTRFS to calculate which is the minimum value. Looking to the btrfs btrfs_shrink_device() function, I don't see where it is a check for an impossible shrinking value. Now I see that Wang comment say more or less the same thing. BR G.Baroncelli > > On 27/1/22 01:19, Boris Burkov wrote: >> On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote: >>> This is what homectl inspect user reports: >>> >>> Disk Size: 128.0G >>> Disk Usage: 3.8G (= 3.1%) >>> Disk Free: 124.0G (= 96.9%) >>> >>> and this is what btrfs usage reports: >>> >>> sudo btrfs filesystem usage /home/toliz >>> >>> Overall: >>> Device size: 127.98GiB >>> Device allocated: 4.02GiB >>> Device unallocated: 123.96GiB >>> Device missing: 0.00B >>> Used: 1.89GiB >>> Free (estimated): 124.10GiB (min: 62.12GiB) >>> Free (statfs, df): 124.10GiB >>> Data ratio: 1.00 >>> Metadata ratio: 2.00 >>> Global reserve: 5.14MiB (used: 0.00B) >>> Multiple profiles: no >>> >>> Data,single: Size:2.01GiB, Used:1.86GiB (92.73%) >>> /dev/mapper/home-toliz 2.01GiB >>> >>> Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%) >>> /dev/mapper/home-toliz 2.00GiB >>> >>> System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%) >>> /dev/mapper/home-toliz 16.00MiB >>> >>> Unallocated: >>> /dev/mapper/home-toliz 123.96GiB >>> >> OK, there is plenty of unallocated space, thanks for confirming. >> >> Looking at the stack trace a bit more, the only thing that really sticks >> out as suspicious to me is btrfs_shrink_device, I'm not sure who would >> want to do that or why. >> >> It might be interesting to trace it and see if we can catch the >> parameters/caller in the act. If you have bpftrace setup on your system, >> could you try to setup something like: >> >> bpftrace -e 'kprobe:btrfs_shrink_device { printf("%s %llu %s\n", comm, arg1, kstack); }' >> >> to write to a file during shutdown? >> >>> On 26/1/22 23:50, Boris Burkov wrote: >>>> On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote: >>>>> Hello. >>>>> >>>>> When i shut down my pc i get No space left errors -even though i have plenty >>>>> of space in both / and home dirs- and this message on the journal: >>>> How did you conclude you have plenty of space? df can be misleading with >>>> btrfs, for example. Can you please post the output of >>>> 'btrfs filesystem usage /home' >>>> >>>> Thanks, >>>> Boris >>>> >>>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block >>>>> group 2177892352 flags data >>>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block >>>>> group 1104150528 flags data >>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block >>>>> group 30408704 flags metadata|dup >>>>> Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------ >>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28) >>>>> Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at >>>>> fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs] >>>>> Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm >>>>> snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc >>>>> encrypted_keys trusted asn1_e> >>>>> Ιαν 25 14:34:32 mainland kernel: snd_pcm_dmaengine kvm snd_hda_intel >>>>> iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul >>>>> snd_intel_sdw_acpi hid_mul> >>>>> Ιαν 25 14:34:32 mainland kernel: int340x_thermal_zone tpm_tis tpm_tis_core >>>>> wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad >>>>> ipmi_devint> >>>>> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor >>>>> Tainted: G W 5.16.2-arch1-1 #1 >>>>> 86fbf2c313cc37a553d65deb81d98e9dcc2a3659 >>>>> Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO., >>>>> LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP >>>>> 04/15/2021 >>>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950 >>>>> [btrfs] >>>>> Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4 >>>>> bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0 >>>>> e8 24 6c 28 e> >>>>> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246 >>>>> Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000 >>>>> RCX: 0000000000000000 >>>>> Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000 >>>>> RDI: 0000000000000000 >>>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000 >>>>> R09: 0000000000000000 >>>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000 >>>>> R12: 00000000ffffffe4 >>>>> Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0 >>>>> R15: fffffffffffffff7 >>>>> Ιαν 25 14:34:32 mainland kernel: FS: 00007f336b49ea80(0000) >>>>> GS:ffff9823c3700000(0000) knlGS:0000000000000000 >>>>> Ιαν 25 14:34:32 mainland kernel: CS: 0010 DS: 0000 ES: 0000 CR0: >>>>> 0000000080050033 >>>>> Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002 >>>>> CR4: 0000000000770ee0 >>>>> Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554 >>>>> Ιαν 25 14:34:32 mainland kernel: Call Trace: >>>>> Ιαν 25 14:34:32 mainland kernel: <TASK> >>>>> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0 >>>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: ? __reserve_bytes+0x164/0x7d0 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: relocate_block_group+0x6e/0x5a0 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340 >>>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_chunk+0x27/0x100 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_shrink_device+0x277/0x5a0 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl_resize+0x449/0x470 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_ioctl+0x1fa8/0x2fc0 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: ? btrfs_statfs+0x418/0x570 [btrfs >>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b] >>>>> Ιαν 25 14:34:32 mainland kernel: ? _copy_to_user+0x1c/0x50 >>>>> Ιαν 25 14:34:32 mainland kernel: ? do_statfs_native+0xaf/0xe0 >>>>> Ιαν 25 14:34:32 mainland kernel: ? __seccomp_filter+0x39e/0x570 >>>>> Ιαν 25 14:34:32 mainland kernel: ? __x64_sys_ioctl+0x8b/0xd0 >>>>> Ιαν 25 14:34:32 mainland kernel: __x64_sys_ioctl+0x8b/0xd0 >>>>> Ιαν 25 14:34:32 mainland kernel: do_syscall_64+0x59/0x90 >>>>> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >>>>> Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 >>>>> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >>>>> Ιαν 25 14:34:32 mainland kernel: ? syscall_exit_to_user_mode+0x23/0x50 >>>>> Ιαν 25 14:34:32 mainland kernel: ? do_syscall_64+0x69/0x90 >>>>> Ιαν 25 14:34:32 mainland kernel: ? exc_page_fault+0x72/0x180 >>>>> Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae >>>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b >>>>> Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff >>>>> ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 >>>>> 00 00 00 0f 0> >>>>> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246 >>>>> ORIG_RAX: 0000000000000010 >>>>> Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000 >>>>> RCX: 00007f336baa359b >>>>> Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403 >>>>> RDI: 0000000000000004 >>>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000 >>>>> R09: 00007ffc945a0370 >>>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246 >>>>> R12: 00007ffc945a0570 >>>>> Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0 >>>>> R15: 00007ffc945a1920 >>>>> Ιαν 25 14:34:32 mainland kernel: </TASK> >>>>> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]--- >>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in >>>>> __btrfs_free_extent:3066: errno=-28 No space left >>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly >>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in >>>>> btrfs_run_delayed_refs:2149: errno=-28 No space left >>>>> >>>>> The dm-0 device is my /home directory and is set up using systemd-homed >>>>> >>>>> Kernel version: 5.16.2 >>>>> >>>>> Systemd version: 250.3 >>>>> >>>>> btrfs-progs version: 5.16 >>>>> >>>>> It seems to cause no issues thus far but a solution would be good to have. >>>>> >>>>> Thanks in advance. >>>>> -- gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-26 23:19 ` Boris Burkov 2022-01-26 23:29 ` Apostolos B. @ 2022-01-27 20:48 ` Chris Murphy 2022-01-29 9:53 ` Goffredo Baroncelli 1 sibling, 1 reply; 20+ messages in thread From: Chris Murphy @ 2022-01-27 20:48 UTC (permalink / raw) To: Boris Burkov; +Cc: Apostolos B., Btrfs BTRFS, systemd Mailing List On Wed, Jan 26, 2022 at 4:19 PM Boris Burkov <boris@bur.io> wrote: > > On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote: > > This is what homectl inspect user reports: > > > > Disk Size: 128.0G > > Disk Usage: 3.8G (= 3.1%) > > Disk Free: 124.0G (= 96.9%) > > > > and this is what btrfs usage reports: > > > > sudo btrfs filesystem usage /home/toliz > > > > Overall: > > Device size: 127.98GiB > > Device allocated: 4.02GiB > > Device unallocated: 123.96GiB > > Device missing: 0.00B > > Used: 1.89GiB > > Free (estimated): 124.10GiB (min: 62.12GiB) > > Free (statfs, df): 124.10GiB > > Data ratio: 1.00 > > Metadata ratio: 2.00 > > Global reserve: 5.14MiB (used: 0.00B) > > Multiple profiles: no > > > > Data,single: Size:2.01GiB, Used:1.86GiB (92.73%) > > /dev/mapper/home-toliz 2.01GiB > > > > Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%) > > /dev/mapper/home-toliz 2.00GiB > > > > System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%) > > /dev/mapper/home-toliz 16.00MiB > > > > Unallocated: > > /dev/mapper/home-toliz 123.96GiB > > > > OK, there is plenty of unallocated space, thanks for confirming. > > Looking at the stack trace a bit more, the only thing that really sticks > out as suspicious to me is btrfs_shrink_device, I'm not sure who would > want to do that or why. systemd-homed by default uses btrfs on LUKS on loop mount, with a backing file. On login, it grows the user home file system with some percentage (I think 80%) of the free space of the underlying file system. And on logout, it does both fstrim and shrinks the fs. I don't know why it does both, it seems adequate to do only fstrim on logout to return unused blocks to the underlying file system; and to do an fs resize on login to either grow or shrink the user home file system. But also, we don't really have a great estimator of the minimum size a file system can be. `btrfs inspect-internal min-dev-size` is pretty broken right now. https://github.com/kdave/btrfs-progs/issues/271 I'm not sure if systemd folks would use libbtrfsutil facility to determine the minimum device shrink size? But also even the kernel doesn't have a very good idea of how small a file system can be shrunk. Right now it basically has to just start trying, and does it one block group at a time. Adding systemd-devel@ -- Chris Murphy ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-27 20:48 ` Chris Murphy @ 2022-01-29 9:53 ` Goffredo Baroncelli 2022-01-29 18:01 ` Chris Murphy 2022-02-01 4:26 ` Zygo Blaxell 0 siblings, 2 replies; 20+ messages in thread From: Goffredo Baroncelli @ 2022-01-29 9:53 UTC (permalink / raw) To: Chris Murphy, Boris Burkov Cc: Apostolos B., Btrfs BTRFS, systemd Mailing List On 27/01/2022 21.48, Chris Murphy wrote: > On Wed, Jan 26, 2022 at 4:19 PM Boris Burkov <boris@bur.io> wrote: [...] > > systemd-homed by default uses btrfs on LUKS on loop mount, with a > backing file. On login, it grows the user home file system with some > percentage (I think 80%) of the free space of the underlying file > system. And on logout, it does both fstrim and shrinks the fs. I don't > know why it does both, it seems adequate to do only fstrim on logout > to return unused blocks to the underlying file system; and to do an fs > resize on login to either grow or shrink the user home file system. > > But also, we don't really have a great estimator of the minimum size a > file system can be. `btrfs inspect-internal min-dev-size` is pretty > broken right now. > https://github.com/kdave/btrfs-progs/issues/271 I tried the test case, but was unable to get a wrong value. However I think that this is due to the fact that btrfs improved the bg reclaiming. However tweaking the test case, I was able to trigger the problem (I reduced the filesize from 1GB to 256MB, so when some files are removed a BG is empty filled) > > I'm not sure if systemd folks would use libbtrfsutil facility to > determine the minimum device shrink size? But also even the kernel > doesn't have a very good idea of how small a file system can be > shrunk. Right now it basically has to just start trying, and does it > one block group at a time. I think that for the systemd uses cases (singled device FS), a simpler approach would be: fstatfs(fd, &sfs) needed = sfs.f_blocks - sfs.f_bavail; needed *= sfs.f_bsize needed = roundup_64(needed, 3*(1024*1024*1024)) Comparing the original systemd-homed code, I made the following changes - 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation) - 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP)) Comments ? > > Adding systemd-devel@ > > -- gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-29 9:53 ` Goffredo Baroncelli @ 2022-01-29 18:01 ` Chris Murphy 2022-01-30 9:27 ` Goffredo Baroncelli 2022-02-01 4:26 ` Zygo Blaxell 1 sibling, 1 reply; 20+ messages in thread From: Chris Murphy @ 2022-01-29 18:01 UTC (permalink / raw) To: Goffredo Baroncelli Cc: Chris Murphy, Boris Burkov, Apostolos B., Btrfs BTRFS, systemd Mailing List On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli <kreijack@libero.it> wrote: > > I think that for the systemd uses cases (singled device FS), a simpler > approach would be: > > fstatfs(fd, &sfs) > needed = sfs.f_blocks - sfs.f_bavail; > needed *= sfs.f_bsize > > needed = roundup_64(needed, 3*(1024*1024*1024)) > > Comparing the original systemd-homed code, I made the following changes > - 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation) > - 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP)) > > Comments ? I'm still wondering if such a significant shrink is even indicated, in lieu of trim. Isn't it sufficient to just trim on logout, thus returning unused blocks to the underlying filesystem? And then do an fs resize (shrink or grow) as needed on login, so that the user home shows ~80% of the free space in the underlying file system? homework-luks.c:3407: /* Before we shrink, let's trim the file system, so that we need less space on disk during the shrinking */ -- Chris Murphy ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-29 18:01 ` Chris Murphy @ 2022-01-30 9:27 ` Goffredo Baroncelli 2022-01-31 9:41 ` Colin Guthrie 0 siblings, 1 reply; 20+ messages in thread From: Goffredo Baroncelli @ 2022-01-30 9:27 UTC (permalink / raw) To: Chris Murphy Cc: Boris Burkov, Apostolos B., Btrfs BTRFS, systemd Mailing List On 29/01/2022 19.01, Chris Murphy wrote: > On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli <kreijack@libero.it> wrote: >> >> I think that for the systemd uses cases (singled device FS), a simpler >> approach would be: >> >> fstatfs(fd, &sfs) >> needed = sfs.f_blocks - sfs.f_bavail; >> needed *= sfs.f_bsize >> >> needed = roundup_64(needed, 3*(1024*1024*1024)) >> >> Comparing the original systemd-homed code, I made the following changes >> - 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation) >> - 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP)) >> >> Comments ? > > I'm still wondering if such a significant shrink is even indicated, in > lieu of trim. Isn't it sufficient to just trim on logout, thus > returning unused blocks to the underlying filesystem? I agree with you. In Fedora 35, and the default is ext4+luks+trim which provides the same results. However I remember that in the past the default was btrfs+luks+shrunk. I think that something is changed i. However, I want to provide do the systemd folks a suggestion ho change the code. Even a warning like: "it doesn't work that because this, please drop it" would be sufficient. > And then do an > fs resize (shrink or grow) as needed on login, so that the user home > shows ~80% of the free space in the underlying file system? > > homework-luks.c:3407: /* Before we > shrink, let's trim the file system, so that we need less space on disk > during the shrinking */ > > > -- gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-30 9:27 ` Goffredo Baroncelli @ 2022-01-31 9:41 ` Colin Guthrie 2022-02-01 19:55 ` Neal Gompa 0 siblings, 1 reply; 20+ messages in thread From: Colin Guthrie @ 2022-01-31 9:41 UTC (permalink / raw) To: kreijack, Chris Murphy Cc: Boris Burkov, Apostolos B., Btrfs BTRFS, systemd Mailing List Goffredo Baroncelli wrote on 30/01/2022 09:27: > On 29/01/2022 19.01, Chris Murphy wrote: >> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli >> <kreijack@libero.it> wrote: >>> >>> I think that for the systemd uses cases (singled device FS), a simpler >>> approach would be: >>> >>> fstatfs(fd, &sfs) >>> needed = sfs.f_blocks - sfs.f_bavail; >>> needed *= sfs.f_bsize >>> >>> needed = roundup_64(needed, 3*(1024*1024*1024)) >>> >>> Comparing the original systemd-homed code, I made the following changes >>> - 1) f_bfree is replaced by f_bavail (which seem to be more >>> consistent to the disk usage; to me it seems to consider also the >>> metadata chunk allocation) >>> - 2) the needing value is rounded up of 3GB in order to consider a >>> further 1 data chunk and 2 metadata chunk (DUP)) >>> >>> Comments ? >> >> I'm still wondering if such a significant shrink is even indicated, in >> lieu of trim. Isn't it sufficient to just trim on logout, thus >> returning unused blocks to the underlying filesystem? > > I agree with you. In Fedora 35, and the default is ext4+luks+trim > which provides the same results. However I remember that in the past the > default > was btrfs+luks+shrunk. I think that something is changed i. > > However, I want to provide do the systemd folks a suggestion ho change > the code. > Even a warning like: "it doesn't work that because this, please drop it" > would be sufficient. Out of curiosity (see other thread on the systemd list about this), what it the current recommendation (by systemd/btrfs folks rather then Fedora defaults) for homed machine partitioning? I have a new system to setup and the base filesystem is currently btrfs but I'm wondering if I should be putting a large luks file on that (can set nocow, but even still) or whether I should redo the base FS as ext4? Open to some degree of experimentation, especially if the next six months promises new features that will be useful etc. Col -- Colin Guthrie ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-31 9:41 ` Colin Guthrie @ 2022-02-01 19:55 ` Neal Gompa 2022-05-31 12:44 ` Colin Guthrie 0 siblings, 1 reply; 20+ messages in thread From: Neal Gompa @ 2022-02-01 19:55 UTC (permalink / raw) To: Colin Guthrie Cc: kreijack, Chris Murphy, Boris Burkov, Apostolos B., Btrfs BTRFS, systemd Mailing List On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> wrote: > > Goffredo Baroncelli wrote on 30/01/2022 09:27: > > On 29/01/2022 19.01, Chris Murphy wrote: > >> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli > >> <kreijack@libero.it> wrote: > >>> > >>> I think that for the systemd uses cases (singled device FS), a simpler > >>> approach would be: > >>> > >>> fstatfs(fd, &sfs) > >>> needed = sfs.f_blocks - sfs.f_bavail; > >>> needed *= sfs.f_bsize > >>> > >>> needed = roundup_64(needed, 3*(1024*1024*1024)) > >>> > >>> Comparing the original systemd-homed code, I made the following changes > >>> - 1) f_bfree is replaced by f_bavail (which seem to be more > >>> consistent to the disk usage; to me it seems to consider also the > >>> metadata chunk allocation) > >>> - 2) the needing value is rounded up of 3GB in order to consider a > >>> further 1 data chunk and 2 metadata chunk (DUP)) > >>> > >>> Comments ? > >> > >> I'm still wondering if such a significant shrink is even indicated, in > >> lieu of trim. Isn't it sufficient to just trim on logout, thus > >> returning unused blocks to the underlying filesystem? > > > > I agree with you. In Fedora 35, and the default is ext4+luks+trim > > which provides the same results. However I remember that in the past the > > default > > was btrfs+luks+shrunk. I think that something is changed i. > > > > However, I want to provide do the systemd folks a suggestion ho change > > the code. > > Even a warning like: "it doesn't work that because this, please drop it" > > would be sufficient. > > > Out of curiosity (see other thread on the systemd list about this), what > it the current recommendation (by systemd/btrfs folks rather then Fedora > defaults) for homed machine partitioning? > I'd probably recommend Btrfs with the /home subvolume set with nodatacow if you're going to use loops of LUKS backed Btrfs homedir images. The individual Btrfs loops will have their own COW anyway. Otherwise, the Fedora defaults for Btrfs should be sufficient. -- 真実はいつも一つ!/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-02-01 19:55 ` Neal Gompa @ 2022-05-31 12:44 ` Colin Guthrie 2022-05-31 18:12 ` Goffredo Baroncelli 0 siblings, 1 reply; 20+ messages in thread From: Colin Guthrie @ 2022-05-31 12:44 UTC (permalink / raw) To: linux-btrfs; +Cc: systemd-devel [-- Attachment #1: Type: text/plain, Size: 3398 bytes --] Hi, Neal Gompa wrote on 01/02/2022 19:55: > On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> wrote: >> >> Goffredo Baroncelli wrote on 30/01/2022 09:27: >>> On 29/01/2022 19.01, Chris Murphy wrote: >>>> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli >>>> <kreijack@libero.it> wrote: >>>>> >>>>> I think that for the systemd uses cases (singled device FS), a simpler >>>>> approach would be: >>>>> >>>>> fstatfs(fd, &sfs) >>>>> needed = sfs.f_blocks - sfs.f_bavail; >>>>> needed *= sfs.f_bsize >>>>> >>>>> needed = roundup_64(needed, 3*(1024*1024*1024)) >>>>> >>>>> Comparing the original systemd-homed code, I made the following changes >>>>> - 1) f_bfree is replaced by f_bavail (which seem to be more >>>>> consistent to the disk usage; to me it seems to consider also the >>>>> metadata chunk allocation) >>>>> - 2) the needing value is rounded up of 3GB in order to consider a >>>>> further 1 data chunk and 2 metadata chunk (DUP)) >>>>> >>>>> Comments ? >>>> >>>> I'm still wondering if such a significant shrink is even indicated, in >>>> lieu of trim. Isn't it sufficient to just trim on logout, thus >>>> returning unused blocks to the underlying filesystem? >>> >>> I agree with you. In Fedora 35, and the default is ext4+luks+trim >>> which provides the same results. However I remember that in the past the >>> default >>> was btrfs+luks+shrunk. I think that something is changed i. >>> >>> However, I want to provide do the systemd folks a suggestion ho change >>> the code. >>> Even a warning like: "it doesn't work that because this, please drop it" >>> would be sufficient. >> >> >> Out of curiosity (see other thread on the systemd list about this), what >> it the current recommendation (by systemd/btrfs folks rather then Fedora >> defaults) for homed machine partitioning? >> > > I'd probably recommend Btrfs with the /home subvolume set with > nodatacow if you're going to use loops of LUKS backed Btrfs homedir > images. The individual Btrfs loops will have their own COW anyway. > > Otherwise, the Fedora defaults for Btrfs should be sufficient. Thought I'd wait for Fedora 36 to be released with everything I need to test this setup. Fell at the first hurdle of transferring my data in! I transferred a subset of my data (240Gb) onto an external disk and used: homectl with colin -- rsync ... The transfer worked but the colin.home file grew to 394Gb. Only about 184Gb used (I presume due to compression). Ultimately, this was then unmounted and while it said it could shrink the filesystem with a "Ready to..." message this either didn't happen or the backing file wasn't shrunk to match it. I did receive a message later I'm not sure now where it's at with recovery but as nothing is strictly needed to make this work, I think I'll leave my playing with homed there for now and try again at some later date. I love the whole idea but it's still a bit to bleeding edge and quirky for my daily driver just yet! I've attached various logs in case they are useful (will post separately if the list removes this!) Cheers Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ [-- Attachment #2: 1.homectl-with-file-transfer.log --] [-- Type: text/x-log, Size: 4398 bytes --] May 31 09:30:11 colins-laptop systemd-homed[865]: colin: changing state inactive → activating-for-acquire May 31 09:30:11 colins-laptop systemd-homework[8653]: Provided password unlocks user record. May 31 09:30:11 colins-laptop systemd-homework[8653]: Successfully locked image file '/home/colin.home'. May 31 09:30:11 colins-laptop systemd-homework[8653]: Backing file is fully allocated already. May 31 09:30:11 colins-laptop systemd-homework[8653]: Setting up loopback device /dev/loop0 completed. May 31 09:30:12 colins-laptop systemd-homework[8653]: Setting up LUKS device /dev/mapper/home-colin completed. May 31 09:30:12 colins-laptop systemd-homework[8653]: Provided password unlocks user record. May 31 09:30:12 colins-laptop systemd-homework[8653]: Probing file system completed (found btrfs). May 31 09:30:12 colins-laptop systemd-homework[8653]: File system check completed. May 31 09:30:12 colins-laptop systemd-homework[8653]: Mounting file system completed. May 31 09:30:12 colins-laptop systemd-homework[8653]: Discovered used loopback device /dev/loop0. May 31 09:30:12 colins-laptop systemd-homework[8653]: offset = 1048576, size = 423003255296, image = 423004320768 May 31 09:30:12 colins-laptop systemd-homework[8653]: Ready to resize image size 393.9G → 458.1G, partition size 393.9G → 458.1G, file system size 393.9G → 458.0G. May 31 09:30:12 colins-laptop systemd-homework[8653]: Couldn't change image size. May 31 09:30:12 colins-laptop systemd-homework[8653]: Read embedded .identity file. May 31 09:30:12 colins-laptop systemd-homework[8653]: Provided password unlocks user record. May 31 09:30:12 colins-laptop systemd-homework[8653]: Reconciling user identities completed (host and header version were identical). May 31 09:30:12 colins-laptop systemd-homework[8653]: Reconciling embedded user identity completed (host and embedded version were identical). May 31 09:30:12 colins-laptop systemd-homework[8653]: Recursive changing of ownership not necessary, skipped. May 31 09:30:12 colins-laptop systemd-homework[8653]: Synchronized disk. May 31 09:30:12 colins-laptop systemd-homework[8653]: Moving to final mount point /home/colin completed. May 31 09:30:12 colins-laptop systemd-homework[8653]: Activation completed. May 31 09:30:12 colins-laptop systemd-homework[8653]: Image size is 393.9G, file system size is 393.9G, file system payload size is 393.9G, file system free is 393.5G. May 31 09:30:12 colins-laptop systemd-homed[865]: Home colin is signed exclusively by our key, accepting. May 31 09:30:12 colins-laptop systemd-homed[865]: colin: changing state activating-for-acquire → active May 31 09:30:12 colins-laptop systemd-homed[865]: Rebalancing complete. May 31 09:39:47 colins-laptop systemd-homed[865]: Rebalancing complete. May 31 09:49:23 colins-laptop systemd-homed[865]: Rebalancing complete. May 31 09:58:25 colins-laptop systemd-homed[865]: Got notification that all sessions of user colin ended, deactivating automatically. May 31 09:58:25 colins-laptop systemd-homed[865]: colin: changing state active → deactivating May 31 09:58:25 colins-laptop systemd-homework[9567]: Discarded unused 212.6G. May 31 09:58:26 colins-laptop systemd-homework[9567]: Syncing completed. May 31 09:58:27 colins-laptop systemd-homework[9567]: Discovered used LUKS device /dev/mapper/home-colin, and validated password. May 31 09:58:27 colins-laptop systemd-homework[9567]: Successfully re-activated LUKS device. May 31 09:58:27 colins-laptop systemd-homework[9567]: Discovered used loopback device /dev/loop0. May 31 09:58:27 colins-laptop systemd-homework[9567]: offset = 1048576, size = 423003255296, image = 423004320768 May 31 09:58:27 colins-laptop systemd-homework[9567]: Ready to resize image size 393.9G → 181.0G, partition size 393.9G → 181.0G, file system size 393.9G → 181.0G. May 31 09:58:28 colins-laptop systemd-homework[9567]: Unmounting completed. May 31 09:58:28 colins-laptop systemd-homework[9567]: Discovered used LUKS device /dev/mapper/home-colin. May 31 09:58:28 colins-laptop systemd-homework[9567]: Device home-colin is not active. May 31 09:58:28 colins-laptop systemd-homed[865]: block device /sys/devices/virtual/block/dm-0 has been removed. May 31 09:58:28 colins-laptop systemd-homework[9567]: Everything completed. May 31 09:58:28 colins-laptop systemd-homed[865]: colin: changing state deactivating → inactive [-- Attachment #3: 2.homectl-with-test-after-reboot.log --] [-- Type: text/x-log, Size: 2932 bytes --] May 31 10:48:36 colins-laptop systemd[1]: Starting systemd-homed.service - Home Area Manager... May 31 10:48:36 colins-laptop systemd-homed[861]: Successfully loaded private key pair. May 31 10:48:36 colins-laptop systemd-homed[861]: Watching /home. May 31 10:48:36 colins-laptop systemd-homed[861]: User record colin.identity is signed only by us, accepting. May 31 10:48:36 colins-laptop systemd-homed[861]: Added registered home for user colin. May 31 10:48:36 colins-laptop systemd[1]: Started systemd-homed.service - Home Area Manager. May 31 10:50:19 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire May 31 10:50:19 colins-laptop systemd-homework[2865]: None of the supplied plaintext passwords unlock the user record's hashed passwords. May 31 10:50:19 colins-laptop systemd-homed[861]: Activation failed: Required key not available May 31 10:50:19 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive May 31 10:50:19 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically. May 31 10:50:19 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed. May 31 10:50:22 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire May 31 10:50:22 colins-laptop systemd-homework[2873]: Provided password unlocks user record. May 31 10:50:22 colins-laptop systemd-homework[2873]: Successfully locked image file '/home/colin.home'. May 31 10:50:22 colins-laptop systemd-homed[861]: Activation failed: No space left on device May 31 10:50:22 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive May 31 10:50:22 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically. May 31 10:50:22 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed. May 31 11:25:45 colins-laptop systemd-homed[861]: colin: changing state inactive → authenticating May 31 11:25:45 colins-laptop systemd-homework[4330]: None of the supplied plaintext passwords unlock the user record's hashed passwords. May 31 11:25:45 colins-laptop systemd-homed[861]: Authentication failed: Required key not available May 31 11:25:45 colins-laptop systemd-homed[861]: colin: changing state authenticating → inactive May 31 11:25:48 colins-laptop systemd-homed[861]: colin: changing state inactive → authenticating May 31 11:25:48 colins-laptop systemd-homework[4335]: Provided password unlocks user record. May 31 11:25:48 colins-laptop systemd-homework[4335]: Successfully locked image file '/home/colin.home'. May 31 11:25:48 colins-laptop systemd-homed[861]: Authentication failed: No space left on device May 31 11:25:48 colins-laptop systemd-homed[861]: colin: changing state authenticating → inactive [-- Attachment #4: 3.homectl-with-test-debug.log --] [-- Type: text/x-log, Size: 10256 bytes --] May 31 11:31:46 colins-laptop systemd-homed[861]: Setting log level to debug. May 31 11:31:46 colins-laptop systemd-homed[861]: Sent message type=method_return sender=n/a destination=:1.144 path=n/a interface=n/a member=n/a cookie=41 reply_cookie=3 signature=n/a error-name=n/a error-message=n/a May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max) May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}} May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid"> May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max) May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}} May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid"> May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max) May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}} May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid"> May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected May 31 11:32:06 colins-laptop systemd-homed[861]: Got message type=method_call sender=:1.145 destination=org.freedesktop.home1 path=/org/freedesktop/home1 interface=org.freedesktop.home1.Manager member=AcquireHome cookie=2 reply_cookie=0> May 31 11:32:06 colins-laptop systemd-homed[861]: Sending to worker: {"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid":60242,"homeDirectory":"/home/coli> May 31 11:32:06 colins-laptop systemd-homed[861]: Successfully forked off '(sd-homework)' as PID 4678. May 31 11:32:06 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire May 31 11:32:06 colins-laptop systemd-homework[4678]: None of the supplied plaintext passwords unlock the user record's hashed passwords. May 31 11:32:06 colins-laptop systemd-homed[861]: Worker reported error code ENOKEY. May 31 11:32:06 colins-laptop systemd-homed[861]: Activation failed: Required key not available May 31 11:32:06 colins-laptop systemd-homed[861]: Sent message type=error sender=n/a destination=:1.145 path=n/a interface=n/a member=n/a cookie=42 reply_cookie=2 signature=s error-name=org.freedesktop.home1.BadPassword error-message=Pas> May 31 11:32:06 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive May 31 11:32:06 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically. May 31 11:32:06 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max) May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}} May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid"> May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket. May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected May 31 11:32:09 colins-laptop systemd-homed[861]: Got message type=method_call sender=:1.145 destination=org.freedesktop.home1 path=/org/freedesktop/home1 interface=org.freedesktop.home1.Manager member=AcquireHome cookie=3 reply_cookie=0> May 31 11:32:09 colins-laptop systemd-homed[861]: Sending to worker: {"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid":60242,"homeDirectory":"/home/coli> May 31 11:32:09 colins-laptop systemd-homed[861]: Successfully forked off '(sd-homework)' as PID 4681. May 31 11:32:09 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire May 31 11:32:09 colins-laptop systemd-homework[4681]: Provided password unlocks user record. May 31 11:32:09 colins-laptop systemd-homework[4681]: Successfully locked image file '/home/colin.home'. May 31 11:32:09 colins-laptop systemd-homed[861]: Successfully acquired LUKS lock fd from worker. May 31 11:32:09 colins-laptop systemd-homed[861]: Worker reported error code ENOSPC. May 31 11:32:09 colins-laptop systemd-homed[861]: Activation failed: No space left on device May 31 11:32:09 colins-laptop systemd-homed[861]: Sent message type=error sender=n/a destination=:1.145 path=n/a interface=n/a member=n/a cookie=43 reply_cookie=3 signature=s error-name=org.freedesktop.home1.NoDiskSpace error-message=Not> May 31 11:32:09 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive May 31 11:32:09 colins-laptop systemd-homed[861]: Successfully closed LUKS backing file lock for colin. May 31 11:32:09 colins-laptop systemd-homed[861]: /home/colin.home has been modified, having a look. May 31 11:32:09 colins-laptop systemd-homed[861]: Found an image for user colin which already has a record, skipping. May 31 11:32:09 colins-laptop systemd-homed[861]: /home/colin.home has been closed after writing, revalidating. May 31 11:32:09 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically. May 31 11:32:09 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed. May 31 11:32:09 colins-laptop systemd-homed[861]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/home1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=44 reply_cookie=0 signature=sa{sv> [-- Attachment #5: 4.homectl-inspect.log --] [-- Type: text/x-log, Size: 1120 bytes --] [root@colins-laptop home]# homectl inspect colin User name: colin State: inactive Disposition: regular Last Change: Sun 2022-05-29 21:00:25 BST Last Passw.: Wed 2022-01-05 16:57:13 GMT Login OK: yes Password OK: yes UID: 60242 GID: 60242 (colin) Aux. Groups: colin vboxusers wheel Real Name: Colin Guthrie Directory: /home/colin Storage: luks (strong encryption) Image Path: /home/colin.home Removable: no Shell: /bin/bash LUKS Discard: online=no offline=yes LUKS UUID: 387b9bc6-6b58-489b-a7a7-f350e5a37609 Part UUID: a394b420-012f-402e-8139-1d120180fd04 FS UUID: bc694cce-b5d7-4b25-99ff-df2be6ff9356 File System: btrfs LUKS Cipher: aes Cipher Mode: xts-plain64 Volume Key: 256bit Mount Flags: nosuid nodev exec Disk Size: 393.9G Disk Floor: 256.0M Disk Ceiling: 458.0G Good Auth.: 34 Last Good: Tue 2022-05-31 09:30:12 BST Bad Auth.: 39 Last Bad: Tue 2022-05-31 11:27:41 BST Next Try: anytime Auth. Limit: 30 attempts per 1min Passwords: 1 Local Sig.: yes Service: io.systemd.Home [-- Attachment #6: 5.btrfs-filesystem-usage.log --] [-- Type: text/x-log, Size: 684 bytes --] Overall: Device size: 468.14GiB Device allocated: 468.14GiB Device unallocated: 1.00MiB Device missing: 0.00B Used: 191.65GiB Free (estimated): 275.72GiB (min: 275.72GiB) Free (statfs, df): 275.72GiB Data ratio: 1.00 Metadata ratio: 1.00 Global reserve: 21.12MiB (used: 0.00B) Multiple profiles: no Data,single: Size:467.13GiB, Used:191.41GiB (40.97%) /dev/nvme0n1p3 467.13GiB Metadata,single: Size:1.01GiB, Used:249.56MiB (24.18%) /dev/nvme0n1p3 1.01GiB System,single: Size:4.00MiB, Used:80.00KiB (1.95%) /dev/nvme0n1p3 4.00MiB Unallocated: /dev/nvme0n1p3 1.00MiB [-- Attachment #7: 6.dot-home-file-stats.log --] [-- Type: text/x-log, Size: 825 bytes --] [root@colins-laptop home]# ls -l /home/colin.home -rw-------. 1 root root 423004320768 May 31 09:58 /home/colin.home [root@colins-laptop home]# ls -lh /home/colin.home -rw-------. 1 root root 394G May 31 09:58 /home/colin.home [root@colins-laptop home]# stat /home/colin.home File: /home/colin.home Size: 423004320768 Blocks: 382312560 IO Block: 4096 regular file Device: 0,41 Inode: 4708 Links: 1 Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-05-31 11:32:09.964909547 +0100 Modify: 2022-05-31 09:58:27.983134041 +0100 Change: 2022-05-31 11:32:09.976909412 +0100 Birth: -6024107610197374494.-683951100 [root@colins-laptop home]# du -sh /home/colin.home 183G /home/colin.home [root@colins-laptop home]# lsattr /home/colin.home ---------------C------ /home/colin.home ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-05-31 12:44 ` Colin Guthrie @ 2022-05-31 18:12 ` Goffredo Baroncelli 2022-06-01 9:36 ` Colin Guthrie 0 siblings, 1 reply; 20+ messages in thread From: Goffredo Baroncelli @ 2022-05-31 18:12 UTC (permalink / raw) To: Colin Guthrie, linux-btrfs; +Cc: systemd-devel On 31/05/2022 14.44, Colin Guthrie wrote: > Hi, > > Neal Gompa wrote on 01/02/2022 19:55: >> On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> wrote: >>> >>> Goffredo Baroncelli wrote on 30/01/2022 09:27: >>>> On 29/01/2022 19.01, Chris Murphy wrote: >>>>> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli >>>>> <kreijack@libero.it> wrote: >>>>>> >>>>>> I think that for the systemd uses cases (singled device FS), a simpler >>>>>> approach would be: >>>>>> >>>>>> fstatfs(fd, &sfs) >>>>>> needed = sfs.f_blocks - sfs.f_bavail; >>>>>> needed *= sfs.f_bsize >>>>>> >>>>>> needed = roundup_64(needed, 3*(1024*1024*1024)) >>>>>> >>>>>> Comparing the original systemd-homed code, I made the following changes >>>>>> - 1) f_bfree is replaced by f_bavail (which seem to be more >>>>>> consistent to the disk usage; to me it seems to consider also the >>>>>> metadata chunk allocation) >>>>>> - 2) the needing value is rounded up of 3GB in order to consider a >>>>>> further 1 data chunk and 2 metadata chunk (DUP)) >>>>>> >>>>>> Comments ? >>>>> >>>>> I'm still wondering if such a significant shrink is even indicated, in >>>>> lieu of trim. Isn't it sufficient to just trim on logout, thus >>>>> returning unused blocks to the underlying filesystem? >>>> >>>> I agree with you. In Fedora 35, and the default is ext4+luks+trim >>>> which provides the same results. However I remember that in the past the >>>> default >>>> was btrfs+luks+shrunk. I think that something is changed i. >>>> >>>> However, I want to provide do the systemd folks a suggestion ho change >>>> the code. >>>> Even a warning like: "it doesn't work that because this, please drop it" >>>> would be sufficient. >>> >>> >>> Out of curiosity (see other thread on the systemd list about this), what >>> it the current recommendation (by systemd/btrfs folks rather then Fedora >>> defaults) for homed machine partitioning? >>> >> >> I'd probably recommend Btrfs with the /home subvolume set with >> nodatacow if you're going to use loops of LUKS backed Btrfs homedir >> images. The individual Btrfs loops will have their own COW anyway. >> >> Otherwise, the Fedora defaults for Btrfs should be sufficient. > > Thought I'd wait for Fedora 36 to be released with everything I need to test this setup. > > Fell at the first hurdle of transferring my data in! > > I transferred a subset of my data (240Gb) onto an external disk and used: > > homectl with colin -- rsync ... > > > The transfer worked but the colin.home file grew to 394Gb. Only about 184Gb used (I presume due to compression). > > Ultimately, this was then unmounted and while it said it could shrink the filesystem with a "Ready to..." message this either didn't happen or the backing file wasn't shrunk to match it. I did receive a message later I suppose that colin.home is a sparse file, so even it has a length of 394GB, it consumes only 184GB. So to me these are valid values. It doesn't matter the length of the files. What does matter is the value returned by "du -sh". Below I create a file with a length of 1000GB. However being a sparse file, it doesn't consume any space and "du -sh" returns 0 $ truncate -s 1000GB foo $ du -sh foo 0 foo $ ls -l foo -rw-r--r-- 1 ghigo ghigo 1000000000000 May 31 19:29 foo > > I'm not sure now where it's at with recovery but as nothing is strictly needed to make this work, I think I'll leave my playing with homed there for now and try again at some later date. > > I love the whole idea but it's still a bit to bleeding edge and quirky for my daily driver just yet! > > > I've attached various logs in case they are useful (will post separately if the list removes this!) > > > Cheers > > Col > > -- gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-05-31 18:12 ` Goffredo Baroncelli @ 2022-06-01 9:36 ` Colin Guthrie 2022-07-23 19:09 ` Chris Murphy 0 siblings, 1 reply; 20+ messages in thread From: Colin Guthrie @ 2022-06-01 9:36 UTC (permalink / raw) To: linux-btrfs; +Cc: systemd-devel Goffredo Baroncelli wrote on 31/05/2022 19:12: > On 31/05/2022 14.44, Colin Guthrie wrote: >> Hi, >> >> Neal Gompa wrote on 01/02/2022 19:55: >>> On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> >>> wrote: >>>> >>>> Goffredo Baroncelli wrote on 30/01/2022 09:27: >>>>> On 29/01/2022 19.01, Chris Murphy wrote: >>>>>> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli >>>>>> <kreijack@libero.it> wrote: >>>>>>> >>>>>>> I think that for the systemd uses cases (singled device FS), a >>>>>>> simpler >>>>>>> approach would be: >>>>>>> >>>>>>> fstatfs(fd, &sfs) >>>>>>> needed = sfs.f_blocks - sfs.f_bavail; >>>>>>> needed *= sfs.f_bsize >>>>>>> >>>>>>> needed = roundup_64(needed, 3*(1024*1024*1024)) >>>>>>> >>>>>>> Comparing the original systemd-homed code, I made the following >>>>>>> changes >>>>>>> - 1) f_bfree is replaced by f_bavail (which seem to be more >>>>>>> consistent to the disk usage; to me it seems to consider also the >>>>>>> metadata chunk allocation) >>>>>>> - 2) the needing value is rounded up of 3GB in order to consider a >>>>>>> further 1 data chunk and 2 metadata chunk (DUP)) >>>>>>> >>>>>>> Comments ? >>>>>> >>>>>> I'm still wondering if such a significant shrink is even >>>>>> indicated, in >>>>>> lieu of trim. Isn't it sufficient to just trim on logout, thus >>>>>> returning unused blocks to the underlying filesystem? >>>>> >>>>> I agree with you. In Fedora 35, and the default is ext4+luks+trim >>>>> which provides the same results. However I remember that in the >>>>> past the >>>>> default >>>>> was btrfs+luks+shrunk. I think that something is changed i. >>>>> >>>>> However, I want to provide do the systemd folks a suggestion ho change >>>>> the code. >>>>> Even a warning like: "it doesn't work that because this, please >>>>> drop it" >>>>> would be sufficient. >>>> >>>> >>>> Out of curiosity (see other thread on the systemd list about this), >>>> what >>>> it the current recommendation (by systemd/btrfs folks rather then >>>> Fedora >>>> defaults) for homed machine partitioning? >>>> >>> >>> I'd probably recommend Btrfs with the /home subvolume set with >>> nodatacow if you're going to use loops of LUKS backed Btrfs homedir >>> images. The individual Btrfs loops will have their own COW anyway. >>> >>> Otherwise, the Fedora defaults for Btrfs should be sufficient. >> >> Thought I'd wait for Fedora 36 to be released with everything I need >> to test this setup. >> >> Fell at the first hurdle of transferring my data in! >> >> I transferred a subset of my data (240Gb) onto an external disk and used: >> >> homectl with colin -- rsync ... >> >> >> The transfer worked but the colin.home file grew to 394Gb. Only about >> 184Gb used (I presume due to compression). >> >> Ultimately, this was then unmounted and while it said it could shrink >> the filesystem with a "Ready to..." message this either didn't happen >> or the backing file wasn't shrunk to match it. I did receive a message >> later > > > I suppose that colin.home is a sparse file, so even it has a length of > 394GB, it consumes only 184GB. So to me these are valid values. It > doesn't matter the length of the files. What does matter is the value > returned by "du -sh". > > Below I create a file with a length of 1000GB. However being a sparse > file, it doesn't consume any space and "du -sh" returns 0 > > $ truncate -s 1000GB foo > $ du -sh foo > 0 foo > $ ls -l foo > -rw-r--r-- 1 ghigo ghigo 1000000000000 May 31 19:29 foo Yeah the file will be sparse. That's not really an issue, I'm not worried about the fact it's not consuming as much as it reports as that's all expected. The issue is that systemd-homed (or btrfs's fallocate) can't handle this situation and that user is effectively bricked unless migrated to a host with more storage space! Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-06-01 9:36 ` Colin Guthrie @ 2022-07-23 19:09 ` Chris Murphy 0 siblings, 0 replies; 20+ messages in thread From: Chris Murphy @ 2022-07-23 19:09 UTC (permalink / raw) To: Colin Guthrie, Btrfs BTRFS Cc: systemd List, Goffredo Baroncelli, Zygo Blaxell [sorry had to resend to get it on btrfs list, due to html in the original :\] On Wed, Jun 1, 2022, at 5:36 AM, Colin Guthrie wrote: > Goffredo Baroncelli wrote on 31/05/2022 19:12: > > > I suppose that colin.home is a sparse file, so even it has a length of > > 394GB, it consumes only 184GB. So to me these are valid values. It > > doesn't matter the length of the files. What does matter is the value > > returned by "du -sh". > > > > Below I create a file with a length of 1000GB. However being a sparse > > file, it doesn't consume any space and "du -sh" returns 0 > > > > $ truncate -s 1000GB foo > > $ du -sh foo > > 0 foo > > $ ls -l foo > > -rw-r--r-- 1 ghigo ghigo 1000000000000 May 31 19:29 foo > > Yeah the file will be sparse. > > That's not really an issue, I'm not worried about the fact it's not > consuming as much as it reports as that's all expected. > > The issue is that systemd-homed (or btrfs's fallocate) can't handle this > situation and that user is effectively bricked unless migrated to a host > with more storage space! Hopefully there's time for systemd-252 for a change still? That version is what I expect to ship in Fedora 37 [1] There's merit to sd-homed and I want it to be safe and reliable for users to keep using, in order to build momentum. I really think sd-homed must move the shrink on logout, to login. When the user logs out, they are decently likely to immediately close the laptop lid thus suspend-to-ram; or shutdown. I don't know if shrink can be cancelled. But regardless, there's going to be a period of time where the file system and storage stacks are busy, right at the time the user is expecting *imminent* suspend or shutdown, which no matter what has to be inhibited until the shrink is cancelled or completed, and all pending writes are flushed to stable media. Next, consider the low battery situation. Upon notification, anyone with an 18+ month old battery knows there may be no additional warnings, and you could in fact get a power failure next. In this scenario we have to depend on all storage stack layers, and the drive firmware, doing the exact correct thing in order for the file system to be in a consistent state to be mountable at next boot. I just think this is too much risk, and since sd-homed is targeted at laptop users primarily, all the more reason the fs resize operation should happen at login time, not logout. In fact, sd-homed might want to inhibit a resize shrink operation if (a) AC power is not plugged in and (b) battery remaining is less than 30%, or some other reasonable value. The resize grow operation is sufficiently cheap and fast that I don't think it needs inhibiting. Thoughts? I also just found a few bug reports with a non-exhaustive search that also make me nervous about fs shrink at logout (also implying restart and shutdown) time. On shutdown, homed resizes until it gets killed https://github.com/systemd/systemd/issues/22901 Getting "New partition doesn't fit into backing storage, refusing" https://github.com/systemd/systemd/issues/22255 fails to resize https://github.com/systemd/systemd/issues/22124 [1] Branch from Rawhide August 9, the earliest release date would be October 18. -- Chris Murphy ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-01-29 9:53 ` Goffredo Baroncelli 2022-01-29 18:01 ` Chris Murphy @ 2022-02-01 4:26 ` Zygo Blaxell 2022-07-23 19:26 ` Chris Murphy 1 sibling, 1 reply; 20+ messages in thread From: Zygo Blaxell @ 2022-02-01 4:26 UTC (permalink / raw) To: kreijack Cc: Chris Murphy, Boris Burkov, Apostolos B., Btrfs BTRFS, systemd Mailing List On Sat, Jan 29, 2022 at 10:53:00AM +0100, Goffredo Baroncelli wrote: > On 27/01/2022 21.48, Chris Murphy wrote: > > On Wed, Jan 26, 2022 at 4:19 PM Boris Burkov <boris@bur.io> wrote: > [...] > > > > systemd-homed by default uses btrfs on LUKS on loop mount, with a > > backing file. On login, it grows the user home file system with some > > percentage (I think 80%) of the free space of the underlying file > > system. And on logout, it does both fstrim and shrinks the fs. I don't > > know why it does both, it seems adequate to do only fstrim on logout > > to return unused blocks to the underlying file system; and to do an fs > > resize on login to either grow or shrink the user home file system. > > > > But also, we don't really have a great estimator of the minimum size a > > file system can be. `btrfs inspect-internal min-dev-size` is pretty > > broken right now. > > https://github.com/kdave/btrfs-progs/issues/271 > > I tried the test case, but was unable to get a wrong value. However > I think that this is due to the fact that btrfs improved the bg reclaiming. > > However tweaking the test case, I was able to trigger the problem (I > reduced the filesize from 1GB to 256MB, so when some files are > removed a BG is empty filled) > > > > > > > I'm not sure if systemd folks would use libbtrfsutil facility to > > determine the minimum device shrink size? But also even the kernel > > doesn't have a very good idea of how small a file system can be > > shrunk. Right now it basically has to just start trying, and does it > > one block group at a time. > > I think that for the systemd uses cases (singled device FS), a simpler > approach would be: > > fstatfs(fd, &sfs) > needed = sfs.f_blocks - sfs.f_bavail; > needed *= sfs.f_bsize > > needed = roundup_64(needed, 3*(1024*1024*1024)) > > Comparing the original systemd-homed code, I made the following changes > - 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation) > - 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP)) > > Comments ? This is closer to the right answer but not quite there yet. A summary of the issues: * Discard (called by systemd-homed in the form of trim) locks a block group (makes it read-only and removes it from available space) while it runs. * Relocation (balance or filesystem resize) locks a block group while it runs. * Scrub in the worst case locks one block group per disk (but we may never run scrub on a systemd-homed filesystem, so ignore this for now). * Large (>50GB) filesystems have larger block groups than smaller filesystems. Resizing from >50GB to <50GB can require rewriting the _entire_ filesystem to make a sensible number of smaller block groups (high enough to be able to lock all the above block groups and still have enough free space to run a transaction without them). * System chunks aren't the same size as other chunks, which will create unusable free space holes between block groups, and (after lots of balancing/resizing runs) possibly create a lot of unusable free space that existing extents cannot be relocated into without temporarily increasing the size of the filesystem. * Resize is a fairly dumb algorithm as algorithms go. It works in one pass, in a fixed order, and it can't fragment an extent or a block group. The minimum size of a filesystem depends not just on how much data there is, but how capable the resize algorithm is at arranging the data into the space given all the overlapping constraints btrfs has on allocation. Resize makes several size-speed tradeoffs in favor of speed (or at least not in favor of size). * The minimum filesystem size to store the data is different from the minimum filesystem size to run specific btrfs data modification operations. Some metadata operations can require significant amounts of space to complete. If the filesystem is resized too small with exactly the right amount of free space, it may become impossible to perform metadata-intensive operations like orphan inode cleanup or snapshot deletion on the next mount. It's not possible to predict these additional space requirements without doing equivalent work to performing the operations and measuring the space required. This means that in order to compute the minimum filesystem size, we need to be able to predict (or strongly control) the future of the filesystem, at least long enough to grow the filesystem back to its service size. These combine to make it especially challenging to resize a nearly empty filesystem from 128GB to smaller than somewhere around 8GB (1GB data + 1GB locked by discard + 1GB locked by relocation + 1GB metadata * 2 for dup + 2GB trapped free dev_extent space from earlier resize operations). That could be reduced to about 6GB with single metadata, but that's a significant resiliency trade-off if the host filesystem doesn't implement data integrity and self-healing. It might be doable in multiple resize passes--one to resize the filesystem to <50GB so that all the data can be moved into smaller block groups, then again to resize it to the next block group size. In the worst case this copies all of the data in the filesystem multiple times, so it would be faster for systemd-homed to mkfs a new filesystem at the target size (which would have smaller block groups from the beginning) and simply copy the data with 'cp -a' to the new filesystem instead of resizing the old one. Resizing a filesystem with 1GB of data on it from over 50GB to 16GB is probably reliable. 8GB is less likely to succeed, and I wouldn't expect any smaller number to work reliably except on very simple test cases. It does suck that the kernel handles resizing below the minimum size of the filesystem so badly; however, even if it rejected the resize request cleanly with an error, it's not necessarily a good idea to attempt it. Pushing the lower limits of what is possible in resize to save a handful of GB is asking for trouble. It's far better to overestimate generously than to underestimate the minimum size. > > > > Adding systemd-devel@ > > > > > > > -- > gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> > Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: No space left errors on shutdown with systemd-homed /home dir 2022-02-01 4:26 ` Zygo Blaxell @ 2022-07-23 19:26 ` Chris Murphy 0 siblings, 0 replies; 20+ messages in thread From: Chris Murphy @ 2022-07-23 19:26 UTC (permalink / raw) To: Zygo Blaxell, Goffredo Baroncelli Cc: Boris Burkov, Apostolos B., Btrfs BTRFS, systemd List On Mon, Jan 31, 2022, at 11:26 PM, Zygo Blaxell wrote: > On Sat, Jan 29, 2022 at 10:53:00AM +0100, Goffredo Baroncelli wrote: > It does suck that the kernel handles resizing below the minimum size of > the filesystem so badly; however, even if it rejected the resize request > cleanly with an error, it's not necessarily a good idea to attempt it. > Pushing the lower limits of what is possible in resize to save a handful > of GB is asking for trouble. It's far better to overestimate generously > than to underestimate the minimum size. Yeah there's an inherent conflict with online shrink: the longer the time needed to relocate bg's, the more unpredictable operations can occur during that time to thwart any original estimations made about the shrink operation. I wondered a bit ago about a shrink API that takes shrink size as a suggestion rather than as a definite, and then the file system does the best job it can. Either this API reports actual shrink size once it completes, or the requesting program needs to know to call BTRFS_IOC_FS_INFO and BTRFS_IOC_DEV_INFO to know the actual size. This hypothetical API could have boundaries outside of which if the kernel code estimates it's going to fall short of, could trigger a cancel of the shrink. This could be size or time based. e.g. BTRFS_IOC_RESIZE_BEST (effort). -- Chris Murphy ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2022-07-23 19:26 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-01-25 17:46 No space left errors on shutdown with systemd-homed /home dir Apostolos B. 2022-01-26 21:50 ` Boris Burkov 2022-01-26 22:07 ` Apostolos B. 2022-01-26 23:19 ` Boris Burkov 2022-01-26 23:29 ` Apostolos B. 2022-01-27 7:59 ` Wang Yugui 2022-01-27 8:51 ` Wang Yugui 2022-01-27 19:13 ` Goffredo Baroncelli 2022-01-27 20:48 ` Chris Murphy 2022-01-29 9:53 ` Goffredo Baroncelli 2022-01-29 18:01 ` Chris Murphy 2022-01-30 9:27 ` Goffredo Baroncelli 2022-01-31 9:41 ` Colin Guthrie 2022-02-01 19:55 ` Neal Gompa 2022-05-31 12:44 ` Colin Guthrie 2022-05-31 18:12 ` Goffredo Baroncelli 2022-06-01 9:36 ` Colin Guthrie 2022-07-23 19:09 ` Chris Murphy 2022-02-01 4:26 ` Zygo Blaxell 2022-07-23 19:26 ` Chris Murphy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).