No space left errors on shutdown with systemd-homed /home dir

No space left errors on shutdown with systemd-homed /home dir

[Apostolos B.]

Hello.

When i shut down my pc i get No space left errors -even though i have 
plenty of space in both / and home dirs- and this message on the journal:

Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating 
block group 2177892352 flags data
Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating 
block group 1104150528 flags data
Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating 
block group 30408704 flags metadata|dup
Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------
Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28)
Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at 
fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs]
Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm 
snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc 
encrypted_keys trusted asn1_e>
Ιαν 25 14:34:32 mainland kernel:  snd_pcm_dmaengine kvm snd_hda_intel 
iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul 
snd_intel_sdw_acpi hid_mul>
Ιαν 25 14:34:32 mainland kernel:  int340x_thermal_zone tpm_tis 
tpm_tis_core wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel 
acpi_tad acpi_pad ipmi_devint>
Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor 
Tainted: G        W         5.16.2-arch1-1 #1 
86fbf2c313cc37a553d65deb81d98e9dcc2a3659
Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO., 
LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP 
04/15/2021
Ιαν 25 14:34:32 mainland kernel: RIP: 
0010:__btrfs_free_extent+0x59c/0x950 [btrfs]
Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 
d4 bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 
bd c0 e8 24 6c 28 e>
Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246
Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 
0000000000000000 RCX: 0000000000000000
Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 
0000000000000000 RDI: 0000000000000000
Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 
0000000000000000 R09: 0000000000000000
Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 
0000000000000000 R12: 00000000ffffffe4
Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: 
ffff9823b62514d0 R15: fffffffffffffff7
Ιαν 25 14:34:32 mainland kernel: FS:  00007f336b49ea80(0000) 
GS:ffff9823c3700000(0000) knlGS:0000000000000000
Ιαν 25 14:34:32 mainland kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 
000000010cfd0002 CR4: 0000000000770ee0
Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554
Ιαν 25 14:34:32 mainland kernel: Call Trace:
Ιαν 25 14:34:32 mainland kernel:  <TASK>
Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0 
[btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 
[btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  ? __reserve_bytes+0x164/0x7d0 [btrfs 
c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 
[btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  relocate_block_group+0x6e/0x5a0 [btrfs 
c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340 
[btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  btrfs_relocate_chunk+0x27/0x100 [btrfs 
c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  btrfs_shrink_device+0x277/0x5a0 [btrfs 
c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl_resize+0x449/0x470 [btrfs 
c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl+0x1fa8/0x2fc0 [btrfs 
c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  ? btrfs_statfs+0x418/0x570 [btrfs 
c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
Ιαν 25 14:34:32 mainland kernel:  ? _copy_to_user+0x1c/0x50
Ιαν 25 14:34:32 mainland kernel:  ? do_statfs_native+0xaf/0xe0
Ιαν 25 14:34:32 mainland kernel:  ? __seccomp_filter+0x39e/0x570
Ιαν 25 14:34:32 mainland kernel:  ? __x64_sys_ioctl+0x8b/0xd0
Ιαν 25 14:34:32 mainland kernel:  __x64_sys_ioctl+0x8b/0xd0
Ιαν 25 14:34:32 mainland kernel:  do_syscall_64+0x59/0x90
Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
Ιαν 25 14:34:32 mainland kernel:  ? exc_page_fault+0x72/0x180
Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b
Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff 
ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 
b8 10 00 00 00 0f 0>
Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 
00000246 ORIG_RAX: 0000000000000010
Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 
0000000072184000 RCX: 00007f336baa359b
Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 
0000000050009403 RDI: 0000000000000004
Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 
0000000000000000 R09: 00007ffc945a0370
Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 
0000000000000246 R12: 00007ffc945a0570
Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 
000055c0fade8cc0 R15: 00007ffc945a1920
Ιαν 25 14:34:32 mainland kernel:  </TASK>
Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]---
Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in 
__btrfs_free_extent:3066: errno=-28 No space left
Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly
Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in 
btrfs_run_delayed_refs:2149: errno=-28 No space left

The dm-0 device is my /home directory and is set up using systemd-homed

Kernel version: 5.16.2

Systemd version: 250.3

btrfs-progs version: 5.16

It seems to cause no issues thus far but a solution would be good to have.

Thanks in advance.

Re: No space left errors on shutdown with systemd-homed /home dir

[Boris Burkov]

On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote:
> Hello.
> 
> When i shut down my pc i get No space left errors -even though i have plenty
> of space in both / and home dirs- and this message on the journal:

How did you conclude you have plenty of space? df can be misleading with
btrfs, for example. Can you please post the output of
'btrfs filesystem usage /home'

Thanks,
Boris

> 
> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
> group 2177892352 flags data
> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
> group 1104150528 flags data
> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block
> group 30408704 flags metadata|dup
> Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------
> Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28)
> Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at
> fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs]
> Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm
> snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc
> encrypted_keys trusted asn1_e>
> Ιαν 25 14:34:32 mainland kernel:  snd_pcm_dmaengine kvm snd_hda_intel
> iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul
> snd_intel_sdw_acpi hid_mul>
> Ιαν 25 14:34:32 mainland kernel:  int340x_thermal_zone tpm_tis tpm_tis_core
> wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad
> ipmi_devint>
> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor
> Tainted: G        W         5.16.2-arch1-1 #1
> 86fbf2c313cc37a553d65deb81d98e9dcc2a3659
> Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO.,
> LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP
> 04/15/2021
> Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950
> [btrfs]
> Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4
> bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0
> e8 24 6c 28 e>
> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246
> Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000
> RCX: 0000000000000000
> Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000
> RDI: 0000000000000000
> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000
> R09: 0000000000000000
> Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000
> R12: 00000000ffffffe4
> Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0
> R15: fffffffffffffff7
> Ιαν 25 14:34:32 mainland kernel: FS:  00007f336b49ea80(0000)
> GS:ffff9823c3700000(0000) knlGS:0000000000000000
> Ιαν 25 14:34:32 mainland kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002
> CR4: 0000000000770ee0
> Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554
> Ιαν 25 14:34:32 mainland kernel: Call Trace:
> Ιαν 25 14:34:32 mainland kernel:  <TASK>
> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0
> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  ? __reserve_bytes+0x164/0x7d0 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  relocate_block_group+0x6e/0x5a0 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340
> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  btrfs_relocate_chunk+0x27/0x100 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  btrfs_shrink_device+0x277/0x5a0 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl_resize+0x449/0x470 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl+0x1fa8/0x2fc0 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  ? btrfs_statfs+0x418/0x570 [btrfs
> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> Ιαν 25 14:34:32 mainland kernel:  ? _copy_to_user+0x1c/0x50
> Ιαν 25 14:34:32 mainland kernel:  ? do_statfs_native+0xaf/0xe0
> Ιαν 25 14:34:32 mainland kernel:  ? __seccomp_filter+0x39e/0x570
> Ιαν 25 14:34:32 mainland kernel:  ? __x64_sys_ioctl+0x8b/0xd0
> Ιαν 25 14:34:32 mainland kernel:  __x64_sys_ioctl+0x8b/0xd0
> Ιαν 25 14:34:32 mainland kernel:  do_syscall_64+0x59/0x90
> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
> Ιαν 25 14:34:32 mainland kernel:  ? exc_page_fault+0x72/0x180
> Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
> Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b
> Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff
> ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10
> 00 00 00 0f 0>
> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246
> ORIG_RAX: 0000000000000010
> Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000
> RCX: 00007f336baa359b
> Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403
> RDI: 0000000000000004
> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000
> R09: 00007ffc945a0370
> Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246
> R12: 00007ffc945a0570
> Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0
> R15: 00007ffc945a1920
> Ιαν 25 14:34:32 mainland kernel:  </TASK>
> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]---
> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
> __btrfs_free_extent:3066: errno=-28 No space left
> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly
> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
> btrfs_run_delayed_refs:2149: errno=-28 No space left
> 
> The dm-0 device is my /home directory and is set up using systemd-homed
> 
> Kernel version: 5.16.2
> 
> Systemd version: 250.3
> 
> btrfs-progs version: 5.16
> 
> It seems to cause no issues thus far but a solution would be good to have.
> 
> Thanks in advance.
> 

Re: No space left errors on shutdown with systemd-homed /home dir

[Apostolos B.]

  This is what homectl inspect user reports:

   Disk Size: 128.0G
   Disk Usage: 3.8G (= 3.1%)
   Disk Free: 124.0G (= 96.9%)

and this is what btrfs usage reports:

sudo btrfs filesystem usage /home/toliz

Overall:
     Device size:             127.98GiB
     Device allocated:               4.02GiB
     Device unallocated:     123.96GiB
     Device missing:                 0.00B
     Used:                           1.89GiB
     Free (estimated):             124.10GiB    (min: 62.12GiB)
     Free (statfs, df):             124.10GiB
     Data ratio:                  1.00
     Metadata ratio:                  2.00
     Global reserve:               5.14MiB    (used: 0.00B)
     Multiple profiles:                    no

Data,single: Size:2.01GiB, Used:1.86GiB (92.73%)
    /dev/mapper/home-toliz       2.01GiB

Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%)
    /dev/mapper/home-toliz       2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
    /dev/mapper/home-toliz      16.00MiB

Unallocated:
    /dev/mapper/home-toliz     123.96GiB


On 26/1/22 23:50, Boris Burkov wrote:
> On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote:
>> Hello.
>>
>> When i shut down my pc i get No space left errors -even though i have plenty
>> of space in both / and home dirs- and this message on the journal:
> How did you conclude you have plenty of space? df can be misleading with
> btrfs, for example. Can you please post the output of
> 'btrfs filesystem usage /home'
>
> Thanks,
> Boris
>
>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
>> group 2177892352 flags data
>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
>> group 1104150528 flags data
>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block
>> group 30408704 flags metadata|dup
>> Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------
>> Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28)
>> Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at
>> fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs]
>> Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm
>> snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc
>> encrypted_keys trusted asn1_e>
>> Ιαν 25 14:34:32 mainland kernel:  snd_pcm_dmaengine kvm snd_hda_intel
>> iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul
>> snd_intel_sdw_acpi hid_mul>
>> Ιαν 25 14:34:32 mainland kernel:  int340x_thermal_zone tpm_tis tpm_tis_core
>> wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad
>> ipmi_devint>
>> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor
>> Tainted: G        W         5.16.2-arch1-1 #1
>> 86fbf2c313cc37a553d65deb81d98e9dcc2a3659
>> Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO.,
>> LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP
>> 04/15/2021
>> Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950
>> [btrfs]
>> Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4
>> bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0
>> e8 24 6c 28 e>
>> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246
>> Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000
>> RCX: 0000000000000000
>> Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000
>> RDI: 0000000000000000
>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000
>> R09: 0000000000000000
>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000
>> R12: 00000000ffffffe4
>> Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0
>> R15: fffffffffffffff7
>> Ιαν 25 14:34:32 mainland kernel: FS:  00007f336b49ea80(0000)
>> GS:ffff9823c3700000(0000) knlGS:0000000000000000
>> Ιαν 25 14:34:32 mainland kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
>> 0000000080050033
>> Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002
>> CR4: 0000000000770ee0
>> Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554
>> Ιαν 25 14:34:32 mainland kernel: Call Trace:
>> Ιαν 25 14:34:32 mainland kernel:  <TASK>
>> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0
>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  ? __reserve_bytes+0x164/0x7d0 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  relocate_block_group+0x6e/0x5a0 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340
>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  btrfs_relocate_chunk+0x27/0x100 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  btrfs_shrink_device+0x277/0x5a0 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl_resize+0x449/0x470 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl+0x1fa8/0x2fc0 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  ? btrfs_statfs+0x418/0x570 [btrfs
>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>> Ιαν 25 14:34:32 mainland kernel:  ? _copy_to_user+0x1c/0x50
>> Ιαν 25 14:34:32 mainland kernel:  ? do_statfs_native+0xaf/0xe0
>> Ιαν 25 14:34:32 mainland kernel:  ? __seccomp_filter+0x39e/0x570
>> Ιαν 25 14:34:32 mainland kernel:  ? __x64_sys_ioctl+0x8b/0xd0
>> Ιαν 25 14:34:32 mainland kernel:  __x64_sys_ioctl+0x8b/0xd0
>> Ιαν 25 14:34:32 mainland kernel:  do_syscall_64+0x59/0x90
>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>> Ιαν 25 14:34:32 mainland kernel:  ? exc_page_fault+0x72/0x180
>> Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>> Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b
>> Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff
>> ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10
>> 00 00 00 0f 0>
>> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246
>> ORIG_RAX: 0000000000000010
>> Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000
>> RCX: 00007f336baa359b
>> Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403
>> RDI: 0000000000000004
>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000
>> R09: 00007ffc945a0370
>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246
>> R12: 00007ffc945a0570
>> Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0
>> R15: 00007ffc945a1920
>> Ιαν 25 14:34:32 mainland kernel:  </TASK>
>> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]---
>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
>> __btrfs_free_extent:3066: errno=-28 No space left
>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly
>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
>> btrfs_run_delayed_refs:2149: errno=-28 No space left
>>
>> The dm-0 device is my /home directory and is set up using systemd-homed
>>
>> Kernel version: 5.16.2
>>
>> Systemd version: 250.3
>>
>> btrfs-progs version: 5.16
>>
>> It seems to cause no issues thus far but a solution would be good to have.
>>
>> Thanks in advance.
>>

Re: No space left errors on shutdown with systemd-homed /home dir

[Boris Burkov]

On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote:
>  This is what homectl inspect user reports:
> 
>   Disk Size: 128.0G
>   Disk Usage: 3.8G (= 3.1%)
>   Disk Free: 124.0G (= 96.9%)
> 
> and this is what btrfs usage reports:
> 
> sudo btrfs filesystem usage /home/toliz
> 
> Overall:
>     Device size:             127.98GiB
>     Device allocated:               4.02GiB
>     Device unallocated:     123.96GiB
>     Device missing:                 0.00B
>     Used:                           1.89GiB
>     Free (estimated):             124.10GiB    (min: 62.12GiB)
>     Free (statfs, df):             124.10GiB
>     Data ratio:                  1.00
>     Metadata ratio:                  2.00
>     Global reserve:               5.14MiB    (used: 0.00B)
>     Multiple profiles:                    no
> 
> Data,single: Size:2.01GiB, Used:1.86GiB (92.73%)
>    /dev/mapper/home-toliz       2.01GiB
> 
> Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%)
>    /dev/mapper/home-toliz       2.00GiB
> 
> System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
>    /dev/mapper/home-toliz      16.00MiB
> 
> Unallocated:
>    /dev/mapper/home-toliz     123.96GiB
> 

OK, there is plenty of unallocated space, thanks for confirming.

Looking at the stack trace a bit more, the only thing that really sticks
out as suspicious to me is btrfs_shrink_device, I'm not sure who would
want to do that or why.

It might be interesting to trace it and see if we can catch the
parameters/caller in the act. If you have bpftrace setup on your system,
could you try to setup something like:

bpftrace -e 'kprobe:btrfs_shrink_device { printf("%s %llu %s\n", comm, arg1, kstack); }'

to write to a file during shutdown?

> 
> On 26/1/22 23:50, Boris Burkov wrote:
> > On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote:
> > > Hello.
> > > 
> > > When i shut down my pc i get No space left errors -even though i have plenty
> > > of space in both / and home dirs- and this message on the journal:
> > How did you conclude you have plenty of space? df can be misleading with
> > btrfs, for example. Can you please post the output of
> > 'btrfs filesystem usage /home'
> > 
> > Thanks,
> > Boris
> > 
> > > Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
> > > group 2177892352 flags data
> > > Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
> > > group 1104150528 flags data
> > > Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block
> > > group 30408704 flags metadata|dup
> > > Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------
> > > Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28)
> > > Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at
> > > fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs]
> > > Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm
> > > snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc
> > > encrypted_keys trusted asn1_e>
> > > Ιαν 25 14:34:32 mainland kernel:  snd_pcm_dmaengine kvm snd_hda_intel
> > > iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul
> > > snd_intel_sdw_acpi hid_mul>
> > > Ιαν 25 14:34:32 mainland kernel:  int340x_thermal_zone tpm_tis tpm_tis_core
> > > wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad
> > > ipmi_devint>
> > > Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor
> > > Tainted: G        W         5.16.2-arch1-1 #1
> > > 86fbf2c313cc37a553d65deb81d98e9dcc2a3659
> > > Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO.,
> > > LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP
> > > 04/15/2021
> > > Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950
> > > [btrfs]
> > > Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4
> > > bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0
> > > e8 24 6c 28 e>
> > > Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246
> > > Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000
> > > RCX: 0000000000000000
> > > Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000
> > > RDI: 0000000000000000
> > > Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000
> > > R09: 0000000000000000
> > > Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000
> > > R12: 00000000ffffffe4
> > > Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0
> > > R15: fffffffffffffff7
> > > Ιαν 25 14:34:32 mainland kernel: FS:  00007f336b49ea80(0000)
> > > GS:ffff9823c3700000(0000) knlGS:0000000000000000
> > > Ιαν 25 14:34:32 mainland kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> > > 0000000080050033
> > > Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002
> > > CR4: 0000000000770ee0
> > > Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554
> > > Ιαν 25 14:34:32 mainland kernel: Call Trace:
> > > Ιαν 25 14:34:32 mainland kernel:  <TASK>
> > > Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0
> > > [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  ? __reserve_bytes+0x164/0x7d0 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  relocate_block_group+0x6e/0x5a0 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340
> > > [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  btrfs_relocate_chunk+0x27/0x100 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  btrfs_shrink_device+0x277/0x5a0 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl_resize+0x449/0x470 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl+0x1fa8/0x2fc0 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  ? btrfs_statfs+0x418/0x570 [btrfs
> > > c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
> > > Ιαν 25 14:34:32 mainland kernel:  ? _copy_to_user+0x1c/0x50
> > > Ιαν 25 14:34:32 mainland kernel:  ? do_statfs_native+0xaf/0xe0
> > > Ιαν 25 14:34:32 mainland kernel:  ? __seccomp_filter+0x39e/0x570
> > > Ιαν 25 14:34:32 mainland kernel:  ? __x64_sys_ioctl+0x8b/0xd0
> > > Ιαν 25 14:34:32 mainland kernel:  __x64_sys_ioctl+0x8b/0xd0
> > > Ιαν 25 14:34:32 mainland kernel:  do_syscall_64+0x59/0x90
> > > Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
> > > Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
> > > Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
> > > Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
> > > Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
> > > Ιαν 25 14:34:32 mainland kernel:  ? exc_page_fault+0x72/0x180
> > > Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
> > > Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b
> > > Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff
> > > ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10
> > > 00 00 00 0f 0>
> > > Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246
> > > ORIG_RAX: 0000000000000010
> > > Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000
> > > RCX: 00007f336baa359b
> > > Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403
> > > RDI: 0000000000000004
> > > Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000
> > > R09: 00007ffc945a0370
> > > Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246
> > > R12: 00007ffc945a0570
> > > Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0
> > > R15: 00007ffc945a1920
> > > Ιαν 25 14:34:32 mainland kernel:  </TASK>
> > > Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]---
> > > Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
> > > __btrfs_free_extent:3066: errno=-28 No space left
> > > Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly
> > > Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
> > > btrfs_run_delayed_refs:2149: errno=-28 No space left
> > > 
> > > The dm-0 device is my /home directory and is set up using systemd-homed
> > > 
> > > Kernel version: 5.16.2
> > > 
> > > Systemd version: 250.3
> > > 
> > > btrfs-progs version: 5.16
> > > 
> > > It seems to cause no issues thus far but a solution would be good to have.
> > > 
> > > Thanks in advance.
> > > 

Re: No space left errors on shutdown with systemd-homed /home dir

[Apostolos B.]

I don't have a bpftrace setup and sadly i cant do much debugging on this 
machine.

However i am sure its systemd that is involved in it. The few lines 
before the crash read:

Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant.
Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS 
device /dev/mapper/home-toliz, and validated password.
Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully 
re-activated LUKS device.
Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used 
loopback device /dev/loop0.
Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size 
= 137436856320, image = 137438953472
Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image 
size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 
127.9G → 1.7G.
Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 
124.8G.
Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating 
block group 2177892352 flags data
Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating 
block group 1104150528 flags data
Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file 
system: Read-only file system
Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating 
block group 30408704 flags metadata|dup

On 27/1/22 01:19, Boris Burkov wrote:
> On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote:
>>   This is what homectl inspect user reports:
>>
>>    Disk Size: 128.0G
>>    Disk Usage: 3.8G (= 3.1%)
>>    Disk Free: 124.0G (= 96.9%)
>>
>> and this is what btrfs usage reports:
>>
>> sudo btrfs filesystem usage /home/toliz
>>
>> Overall:
>>      Device size:             127.98GiB
>>      Device allocated:               4.02GiB
>>      Device unallocated:     123.96GiB
>>      Device missing:                 0.00B
>>      Used:                           1.89GiB
>>      Free (estimated):             124.10GiB    (min: 62.12GiB)
>>      Free (statfs, df):             124.10GiB
>>      Data ratio:                  1.00
>>      Metadata ratio:                  2.00
>>      Global reserve:               5.14MiB    (used: 0.00B)
>>      Multiple profiles:                    no
>>
>> Data,single: Size:2.01GiB, Used:1.86GiB (92.73%)
>>     /dev/mapper/home-toliz       2.01GiB
>>
>> Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%)
>>     /dev/mapper/home-toliz       2.00GiB
>>
>> System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
>>     /dev/mapper/home-toliz      16.00MiB
>>
>> Unallocated:
>>     /dev/mapper/home-toliz     123.96GiB
>>
> OK, there is plenty of unallocated space, thanks for confirming.
>
> Looking at the stack trace a bit more, the only thing that really sticks
> out as suspicious to me is btrfs_shrink_device, I'm not sure who would
> want to do that or why.
>
> It might be interesting to trace it and see if we can catch the
> parameters/caller in the act. If you have bpftrace setup on your system,
> could you try to setup something like:
>
> bpftrace -e 'kprobe:btrfs_shrink_device { printf("%s %llu %s\n", comm, arg1, kstack); }'
>
> to write to a file during shutdown?
>
>> On 26/1/22 23:50, Boris Burkov wrote:
>>> On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote:
>>>> Hello.
>>>>
>>>> When i shut down my pc i get No space left errors -even though i have plenty
>>>> of space in both / and home dirs- and this message on the journal:
>>> How did you conclude you have plenty of space? df can be misleading with
>>> btrfs, for example. Can you please post the output of
>>> 'btrfs filesystem usage /home'
>>>
>>> Thanks,
>>> Boris
>>>
>>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
>>>> group 2177892352 flags data
>>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
>>>> group 1104150528 flags data
>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block
>>>> group 30408704 flags metadata|dup
>>>> Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------
>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28)
>>>> Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at
>>>> fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs]
>>>> Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm
>>>> snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc
>>>> encrypted_keys trusted asn1_e>
>>>> Ιαν 25 14:34:32 mainland kernel:  snd_pcm_dmaengine kvm snd_hda_intel
>>>> iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul
>>>> snd_intel_sdw_acpi hid_mul>
>>>> Ιαν 25 14:34:32 mainland kernel:  int340x_thermal_zone tpm_tis tpm_tis_core
>>>> wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad
>>>> ipmi_devint>
>>>> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor
>>>> Tainted: G        W         5.16.2-arch1-1 #1
>>>> 86fbf2c313cc37a553d65deb81d98e9dcc2a3659
>>>> Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO.,
>>>> LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP
>>>> 04/15/2021
>>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950
>>>> [btrfs]
>>>> Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4
>>>> bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0
>>>> e8 24 6c 28 e>
>>>> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246
>>>> Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000
>>>> RCX: 0000000000000000
>>>> Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000
>>>> RDI: 0000000000000000
>>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000
>>>> R09: 0000000000000000
>>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000
>>>> R12: 00000000ffffffe4
>>>> Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0
>>>> R15: fffffffffffffff7
>>>> Ιαν 25 14:34:32 mainland kernel: FS:  00007f336b49ea80(0000)
>>>> GS:ffff9823c3700000(0000) knlGS:0000000000000000
>>>> Ιαν 25 14:34:32 mainland kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002
>>>> CR4: 0000000000770ee0
>>>> Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554
>>>> Ιαν 25 14:34:32 mainland kernel: Call Trace:
>>>> Ιαν 25 14:34:32 mainland kernel:  <TASK>
>>>> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0
>>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  ? __reserve_bytes+0x164/0x7d0 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  relocate_block_group+0x6e/0x5a0 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340
>>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_relocate_chunk+0x27/0x100 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_shrink_device+0x277/0x5a0 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl_resize+0x449/0x470 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl+0x1fa8/0x2fc0 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  ? btrfs_statfs+0x418/0x570 [btrfs
>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>> Ιαν 25 14:34:32 mainland kernel:  ? _copy_to_user+0x1c/0x50
>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_statfs_native+0xaf/0xe0
>>>> Ιαν 25 14:34:32 mainland kernel:  ? __seccomp_filter+0x39e/0x570
>>>> Ιαν 25 14:34:32 mainland kernel:  ? __x64_sys_ioctl+0x8b/0xd0
>>>> Ιαν 25 14:34:32 mainland kernel:  __x64_sys_ioctl+0x8b/0xd0
>>>> Ιαν 25 14:34:32 mainland kernel:  do_syscall_64+0x59/0x90
>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>>>> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>>>> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>>>> Ιαν 25 14:34:32 mainland kernel:  ? exc_page_fault+0x72/0x180
>>>> Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b
>>>> Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff
>>>> ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10
>>>> 00 00 00 0f 0>
>>>> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246
>>>> ORIG_RAX: 0000000000000010
>>>> Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000
>>>> RCX: 00007f336baa359b
>>>> Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403
>>>> RDI: 0000000000000004
>>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000
>>>> R09: 00007ffc945a0370
>>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246
>>>> R12: 00007ffc945a0570
>>>> Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0
>>>> R15: 00007ffc945a1920
>>>> Ιαν 25 14:34:32 mainland kernel:  </TASK>
>>>> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]---
>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
>>>> __btrfs_free_extent:3066: errno=-28 No space left
>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly
>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
>>>> btrfs_run_delayed_refs:2149: errno=-28 No space left
>>>>
>>>> The dm-0 device is my /home directory and is set up using systemd-homed
>>>>
>>>> Kernel version: 5.16.2
>>>>
>>>> Systemd version: 250.3
>>>>
>>>> btrfs-progs version: 5.16
>>>>
>>>> It seems to cause no issues thus far but a solution would be good to have.
>>>>
>>>> Thanks in advance.
>>>>

Re: No space left errors on shutdown with systemd-homed /home dir

[Wang Yugui]

Hi,

Why 'resize image size 128.0G → 1.8G'?

This 'No space left errors ' happen when 'resize image size 128.0G → 1.8G'.
and it will not happen when ''resize image size 128.0G → 2.0G'?

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/01/27

> I don't have a bpftrace setup and sadly i cant do much debugging on this machine.
> 
> However i am sure its systemd that is involved in it. The few lines before the crash read:
> 
> Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant.
> Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS device /dev/mapper/home-toliz, and validated password.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully re-activated LUKS device.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used loopback device /dev/loop0.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size = 137436856320, image = 137438953472
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 127.9G → 1.7G.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 124.8G.
> Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data
> Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data
> Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file system: Read-only file system
> Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup
> 
> On 27/1/22 01:19, Boris Burkov wrote:
> > On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote:
> >>  ?This is what homectl inspect user reports:
> >>
> >>  ? Disk Size: 128.0G
> >>  ? Disk Usage: 3.8G (= 3.1%)
> >>  ? Disk Free: 124.0G (= 96.9%)

Re: No space left errors on shutdown with systemd-homed /home dir

[Wang Yugui]

Hi,

> Why 'resize image size 128.0G → 1.8G'?
> 
> This 'No space left errors ' happen when 'resize image size 128.0G → 1.8G'.
> and it will not happen when ''resize image size 128.0G → 2.0G'?

'btrfs filesystem resize' and kernel btrfs_shrink_device() now does not
check whether the new shrinked size is too small.

if a too small shrinked size is specified to 'btrfs filesystem resize',
a same call trace will happen.

the min shrinkable size is in the unit of btrfs chunk, not the unit of btrfs
blockt.
This value is a little difficult to get correctly, so we need a new
feature such as 'btrfs filesystem resize min' in addition to 
the exist 'btrfs filesystem resize max'

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/01/27

> 
> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2022/01/27
> 
> > I don't have a bpftrace setup and sadly i cant do much debugging on this machine.
> > 
> > However i am sure its systemd that is involved in it. The few lines before the crash read:
> > 
> > Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant.
> > Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS device /dev/mapper/home-toliz, and validated password.
> > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully re-activated LUKS device.
> > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used loopback device /dev/loop0.
> > Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size = 137436856320, image = 137438953472
> > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 127.9G → 1.7G.
> > Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 124.8G.
> > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data
> > Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data
> > Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file system: Read-only file system
> > Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup
> > 
> > On 27/1/22 01:19, Boris Burkov wrote:
> > > On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote:
> > >>  ?This is what homectl inspect user reports:
> > >>
> > >>  ? Disk Size: 128.0G
> > >>  ? Disk Usage: 3.8G (= 3.1%)
> > >>  ? Disk Free: 124.0G (= 96.9%)
> 

Re: No space left errors on shutdown with systemd-homed /home dir

[Goffredo Baroncelli]

On 27/01/2022 00.29, Apostolos B. wrote:
> I don't have a bpftrace setup and sadly i cant do much debugging on this machine.
> 
> However i am sure its systemd that is involved in it. The few lines before the crash read:
> 
> Ιαν 26 22:45:06 mainland systemd[1]: Stopped WPA supplicant.
> Ιαν 26 22:45:06 mainland systemd-homework[14696]: Discovered used LUKS device /dev/mapper/home-toliz, and validated password.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Successfully re-activated LUKS device.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Discovered used loopback device /dev/loop0.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: offset = 1048576, size = 137436856320, image = 137438953472
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Ready to resize image size 128.0G → 1.8G, partition size 127.9G → 1.8G, file system size 127.9G → 1.7G.
> Ιαν 26 22:45:07 mainland systemd-homework[14696]: Allocated additional 124.8G.
> Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 2177892352 flags data
> Ιαν 26 22:45:07 mainland kernel: BTRFS info (device dm-0): relocating block group 1104150528 flags data
> Ιαν 26 22:45:08 mainland systemd-homework[14696]: Failed to resize file system: Read-only file system
> Ιαν 26 22:45:08 mainland kernel: BTRFS info (device dm-0): relocating block group 30408704 flags metadata|dup

 From the homectl manpage
[...]
--auto-resize-mode=
     Configures whether to automatically grow and/or shrink the backing file system on login and logout.
[...]

So it seems that systemd is doing a shrinking at logout time, because it was configured to do it.

Looking at the code, it seems that the systemd function get_smallest_fs_size() is calculating the
minimum needed storage size; it does that computing

	fstatfs(fd, &sfs)
	needed = sfs.f_blocks - sfs.f_bfree;
	needed *= sfs.f_bsize

and adding a 'safety margin' of

	needed += HOME_MIN_FREE;

(where HOME_MIN_FREE is 16M)

I think that this is a too simplistic way to compute the needing space for BTRFS.
But a comment of a developer with a more deeper knowledge of this topic may
showed that I am wrong.

Anyway, it seems that the problem can be easily solved disabling the systemd auto
shrink feature. Anyway I suggest to open a bug in systemd.

What I am not sure if BTRFS should accept shrinking value that can't satisfy.
My guess, is that it is not simple for BTRFS to calculate which is the
minimum value. Looking to the btrfs btrfs_shrink_device() function, I don't see
where it is a check for an impossible shrinking value.

Now I see that Wang comment say more or less the same thing.

BR
G.Baroncelli

> 
> On 27/1/22 01:19, Boris Burkov wrote:
>> On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote:
>>>   This is what homectl inspect user reports:
>>>
>>>    Disk Size: 128.0G
>>>    Disk Usage: 3.8G (= 3.1%)
>>>    Disk Free: 124.0G (= 96.9%)
>>>
>>> and this is what btrfs usage reports:
>>>
>>> sudo btrfs filesystem usage /home/toliz
>>>
>>> Overall:
>>>      Device size:             127.98GiB
>>>      Device allocated:               4.02GiB
>>>      Device unallocated:     123.96GiB
>>>      Device missing:                 0.00B
>>>      Used:                           1.89GiB
>>>      Free (estimated):             124.10GiB    (min: 62.12GiB)
>>>      Free (statfs, df):             124.10GiB
>>>      Data ratio:                  1.00
>>>      Metadata ratio:                  2.00
>>>      Global reserve:               5.14MiB    (used: 0.00B)
>>>      Multiple profiles:                    no
>>>
>>> Data,single: Size:2.01GiB, Used:1.86GiB (92.73%)
>>>     /dev/mapper/home-toliz       2.01GiB
>>>
>>> Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%)
>>>     /dev/mapper/home-toliz       2.00GiB
>>>
>>> System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
>>>     /dev/mapper/home-toliz      16.00MiB
>>>
>>> Unallocated:
>>>     /dev/mapper/home-toliz     123.96GiB
>>>
>> OK, there is plenty of unallocated space, thanks for confirming.
>>
>> Looking at the stack trace a bit more, the only thing that really sticks
>> out as suspicious to me is btrfs_shrink_device, I'm not sure who would
>> want to do that or why.
>>
>> It might be interesting to trace it and see if we can catch the
>> parameters/caller in the act. If you have bpftrace setup on your system,
>> could you try to setup something like:
>>
>> bpftrace -e 'kprobe:btrfs_shrink_device { printf("%s %llu %s\n", comm, arg1, kstack); }'
>>
>> to write to a file during shutdown?
>>
>>> On 26/1/22 23:50, Boris Burkov wrote:
>>>> On Tue, Jan 25, 2022 at 07:46:51PM +0200, Apostolos B. wrote:
>>>>> Hello.
>>>>>
>>>>> When i shut down my pc i get No space left errors -even though i have plenty
>>>>> of space in both / and home dirs- and this message on the journal:
>>>> How did you conclude you have plenty of space? df can be misleading with
>>>> btrfs, for example. Can you please post the output of
>>>> 'btrfs filesystem usage /home'
>>>>
>>>> Thanks,
>>>> Boris
>>>>
>>>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
>>>>> group 2177892352 flags data
>>>>> Ιαν 25 14:34:31 mainland kernel: BTRFS info (device dm-0): relocating block
>>>>> group 1104150528 flags data
>>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): relocating block
>>>>> group 30408704 flags metadata|dup
>>>>> Ιαν 25 14:34:32 mainland kernel: ------------[ cut here ]------------
>>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: Transaction aborted (error -28)
>>>>> Ιαν 25 14:34:32 mainland kernel: WARNING: CPU: 4 PID: 18307 at
>>>>> fs/btrfs/extent-tree.c:3066 __btrfs_free_extent+0x59c/0x950 [btrfs]
>>>>> Ιαν 25 14:34:32 mainland kernel: Modules linked in: uhid rfcomm
>>>>> snd_seq_dummy snd_hrtimer snd_seq snd_seq_device i2c_dev dm_crypt cbc
>>>>> encrypted_keys trusted asn1_e>
>>>>> Ιαν 25 14:34:32 mainland kernel:  snd_pcm_dmaengine kvm snd_hda_intel
>>>>> iTCO_wdt irqbypass snd_intel_dspcfg intel_pmc_bxt crct10dif_pclmul
>>>>> snd_intel_sdw_acpi hid_mul>
>>>>> Ιαν 25 14:34:32 mainland kernel:  int340x_thermal_zone tpm_tis tpm_tis_core
>>>>> wmi int3400_thermal tpm mac_hid rng_core acpi_thermal_rel acpi_tad acpi_pad
>>>>> ipmi_devint>
>>>>> Ιαν 25 14:34:32 mainland kernel: CPU: 4 PID: 18307 Comm: systemd-homewor
>>>>> Tainted: G        W         5.16.2-arch1-1 #1
>>>>> 86fbf2c313cc37a553d65deb81d98e9dcc2a3659
>>>>> Ιαν 25 14:34:32 mainland kernel: Hardware name: SAMSUNG ELECTRONICS CO.,
>>>>> LTD. 930XDB/931XDB/930XDY/NP930XDB-KF1IT, BIOS P03RFX.055.210415.SP
>>>>> 04/15/2021
>>>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0010:__btrfs_free_extent+0x59c/0x950
>>>>> [btrfs]
>>>>> Ιαν 25 14:34:32 mainland kernel: Code: 24 14 ba 7e 0c 00 00 48 c7 c6 40 d4
>>>>> bc c0 4c 89 ef e8 44 25 0c 00 e9 99 fe ff ff 44 89 e6 48 c7 c7 a0 95 bd c0
>>>>> e8 24 6c 28 e>
>>>>> Ιαν 25 14:34:32 mainland kernel: RSP: 0018:ffffb1ab80f837a0 EFLAGS: 00010246
>>>>> Ιαν 25 14:34:32 mainland kernel: RAX: 0000000000000000 RBX: 0000000000000000
>>>>> RCX: 0000000000000000
>>>>> Ιαν 25 14:34:32 mainland kernel: RDX: 0000000000000000 RSI: 0000000000000000
>>>>> RDI: 0000000000000000
>>>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000d07000 R08: 0000000000000000
>>>>> R09: 0000000000000000
>>>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000000000000 R11: 0000000000000000
>>>>> R12: 00000000ffffffe4
>>>>> Ιαν 25 14:34:32 mainland kernel: R13: ffff982240648888 R14: ffff9823b62514d0
>>>>> R15: fffffffffffffff7
>>>>> Ιαν 25 14:34:32 mainland kernel: FS:  00007f336b49ea80(0000)
>>>>> GS:ffff9823c3700000(0000) knlGS:0000000000000000
>>>>> Ιαν 25 14:34:32 mainland kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
>>>>> 0000000080050033
>>>>> Ιαν 25 14:34:32 mainland kernel: CR2: 00007fa3c5637050 CR3: 000000010cfd0002
>>>>> CR4: 0000000000770ee0
>>>>> Ιαν 25 14:34:32 mainland kernel: PKRU: 55555554
>>>>> Ιαν 25 14:34:32 mainland kernel: Call Trace:
>>>>> Ιαν 25 14:34:32 mainland kernel:  <TASK>
>>>>> Ιαν 25 14:34:32 mainland kernel: __btrfs_run_delayed_refs+0x25c/0x10d0
>>>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_run_delayed_refs+0x73/0x200 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? __reserve_bytes+0x164/0x7d0 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_commit_transaction+0xf6/0xb20 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  relocate_block_group+0x6e/0x5a0 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel: btrfs_relocate_block_group+0x18b/0x340
>>>>> [btrfs c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_relocate_chunk+0x27/0x100 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_shrink_device+0x277/0x5a0 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl_resize+0x449/0x470 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  btrfs_ioctl+0x1fa8/0x2fc0 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? btrfs_statfs+0x418/0x570 [btrfs
>>>>> c10068e329b0dae5c9bb0cca4f6f33712f172b3b]
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? _copy_to_user+0x1c/0x50
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_statfs_native+0xaf/0xe0
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? __seccomp_filter+0x39e/0x570
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? __x64_sys_ioctl+0x8b/0xd0
>>>>> Ιαν 25 14:34:32 mainland kernel:  __x64_sys_ioctl+0x8b/0xd0
>>>>> Ιαν 25 14:34:32 mainland kernel:  do_syscall_64+0x59/0x90
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? syscall_exit_to_user_mode+0x23/0x50
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? do_syscall_64+0x69/0x90
>>>>> Ιαν 25 14:34:32 mainland kernel:  ? exc_page_fault+0x72/0x180
>>>>> Ιαν 25 14:34:32 mainland kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>>>>> Ιαν 25 14:34:32 mainland kernel: RIP: 0033:0x7f336baa359b
>>>>> Ιαν 25 14:34:32 mainland kernel: Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff
>>>>> ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10
>>>>> 00 00 00 0f 0>
>>>>> Ιαν 25 14:34:32 mainland kernel: RSP: 002b:00007ffc945a04d8 EFLAGS: 00000246
>>>>> ORIG_RAX: 0000000000000010
>>>>> Ιαν 25 14:34:32 mainland kernel: RAX: ffffffffffffffda RBX: 0000000072184000
>>>>> RCX: 00007f336baa359b
>>>>> Ιαν 25 14:34:32 mainland kernel: RDX: 00007ffc945a0570 RSI: 0000000050009403
>>>>> RDI: 0000000000000004
>>>>> Ιαν 25 14:34:32 mainland kernel: RBP: 0000000000000004 R08: 0000000000000000
>>>>> R09: 00007ffc945a0370
>>>>> Ιαν 25 14:34:32 mainland kernel: R10: 0000000072184000 R11: 0000000000000246
>>>>> R12: 00007ffc945a0570
>>>>> Ιαν 25 14:34:32 mainland kernel: R13: 0000000000000000 R14: 000055c0fade8cc0
>>>>> R15: 00007ffc945a1920
>>>>> Ιαν 25 14:34:32 mainland kernel:  </TASK>
>>>>> Ιαν 25 14:34:32 mainland kernel: ---[ end trace 81d5963d986040ee ]---
>>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
>>>>> __btrfs_free_extent:3066: errno=-28 No space left
>>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS info (device dm-0): forced readonly
>>>>> Ιαν 25 14:34:32 mainland kernel: BTRFS: error (device dm-0) in
>>>>> btrfs_run_delayed_refs:2149: errno=-28 No space left
>>>>>
>>>>> The dm-0 device is my /home directory and is set up using systemd-homed
>>>>>
>>>>> Kernel version: 5.16.2
>>>>>
>>>>> Systemd version: 250.3
>>>>>
>>>>> btrfs-progs version: 5.16
>>>>>
>>>>> It seems to cause no issues thus far but a solution would be good to have.
>>>>>
>>>>> Thanks in advance.
>>>>>


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: No space left errors on shutdown with systemd-homed /home dir

[Chris Murphy]

On Wed, Jan 26, 2022 at 4:19 PM Boris Burkov <boris@bur.io> wrote:
>
> On Thu, Jan 27, 2022 at 12:07:53AM +0200, Apostolos B. wrote:
> >  This is what homectl inspect user reports:
> >
> >   Disk Size: 128.0G
> >   Disk Usage: 3.8G (= 3.1%)
> >   Disk Free: 124.0G (= 96.9%)
> >
> > and this is what btrfs usage reports:
> >
> > sudo btrfs filesystem usage /home/toliz
> >
> > Overall:
> >     Device size:             127.98GiB
> >     Device allocated:               4.02GiB
> >     Device unallocated:     123.96GiB
> >     Device missing:                 0.00B
> >     Used:                           1.89GiB
> >     Free (estimated):             124.10GiB    (min: 62.12GiB)
> >     Free (statfs, df):             124.10GiB
> >     Data ratio:                  1.00
> >     Metadata ratio:                  2.00
> >     Global reserve:               5.14MiB    (used: 0.00B)
> >     Multiple profiles:                    no
> >
> > Data,single: Size:2.01GiB, Used:1.86GiB (92.73%)
> >    /dev/mapper/home-toliz       2.01GiB
> >
> > Metadata,DUP: Size:1.00GiB, Used:12.47MiB (1.22%)
> >    /dev/mapper/home-toliz       2.00GiB
> >
> > System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
> >    /dev/mapper/home-toliz      16.00MiB
> >
> > Unallocated:
> >    /dev/mapper/home-toliz     123.96GiB
> >
>
> OK, there is plenty of unallocated space, thanks for confirming.
>
> Looking at the stack trace a bit more, the only thing that really sticks
> out as suspicious to me is btrfs_shrink_device, I'm not sure who would
> want to do that or why.

systemd-homed by default uses btrfs on LUKS on loop mount, with a
backing file. On login, it grows the user home file system with some
percentage (I think 80%) of the free space of the underlying file
system. And on logout, it does both fstrim and shrinks the fs. I don't
know why it does both, it seems adequate to do only fstrim on logout
to return unused blocks to the underlying file system; and to do an fs
resize on login to either grow or shrink the user home file system.

But also, we don't really have a great estimator of the minimum size a
file system can be. `btrfs inspect-internal min-dev-size` is pretty
broken right now.
https://github.com/kdave/btrfs-progs/issues/271

I'm not sure if systemd folks would use libbtrfsutil facility to
determine the minimum device shrink size? But also even the kernel
doesn't have a very good idea of how small a file system can be
shrunk. Right now it basically has to just start trying, and does it
one block group at a time.

Adding systemd-devel@


-- 
Chris Murphy

Re: No space left errors on shutdown with systemd-homed /home dir

[Goffredo Baroncelli]

On 27/01/2022 21.48, Chris Murphy wrote:
> On Wed, Jan 26, 2022 at 4:19 PM Boris Burkov <boris@bur.io> wrote:
[...]
> 
> systemd-homed by default uses btrfs on LUKS on loop mount, with a
> backing file. On login, it grows the user home file system with some
> percentage (I think 80%) of the free space of the underlying file
> system. And on logout, it does both fstrim and shrinks the fs. I don't
> know why it does both, it seems adequate to do only fstrim on logout
> to return unused blocks to the underlying file system; and to do an fs
> resize on login to either grow or shrink the user home file system.
> 
> But also, we don't really have a great estimator of the minimum size a
> file system can be. `btrfs inspect-internal min-dev-size` is pretty
> broken right now.
> https://github.com/kdave/btrfs-progs/issues/271

I tried the test case, but was unable to get a wrong value. However
I think that this is due to the fact that btrfs improved the bg reclaiming.

However tweaking the test case, I was able to trigger the problem (I
reduced the filesize from 1GB to 256MB, so when some files are
removed a BG is empty filled)



> 
> I'm not sure if systemd folks would use libbtrfsutil facility to
> determine the minimum device shrink size? But also even the kernel
> doesn't have a very good idea of how small a file system can be
> shrunk. Right now it basically has to just start trying, and does it
> one block group at a time.

I think that for the systemd uses cases (singled device FS), a simpler
approach would be:

     fstatfs(fd, &sfs)
     needed = sfs.f_blocks - sfs.f_bavail;
     needed *= sfs.f_bsize

     needed = roundup_64(needed, 3*(1024*1024*1024))

Comparing the original systemd-homed code, I made the following changes
- 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation)
- 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP))

Comments ?

> 
> Adding systemd-devel@
> 
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: No space left errors on shutdown with systemd-homed /home dir

[Chris Murphy]

On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli <kreijack@libero.it> wrote:
>
> I think that for the systemd uses cases (singled device FS), a simpler
> approach would be:
>
>      fstatfs(fd, &sfs)
>      needed = sfs.f_blocks - sfs.f_bavail;
>      needed *= sfs.f_bsize
>
>      needed = roundup_64(needed, 3*(1024*1024*1024))
>
> Comparing the original systemd-homed code, I made the following changes
> - 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation)
> - 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP))
>
> Comments ?

I'm still wondering if such a significant shrink is even indicated, in
lieu of trim. Isn't it sufficient to just trim on logout, thus
returning unused blocks to the underlying filesystem? And then do an
fs resize (shrink or grow) as needed on login, so that the user home
shows ~80% of the free space in the underlying file system?

homework-luks.c:3407:                                /* Before we
shrink, let's trim the file system, so that we need less space on disk
during the shrinking */



-- 
Chris Murphy

Re: No space left errors on shutdown with systemd-homed /home dir

[Goffredo Baroncelli]

On 29/01/2022 19.01, Chris Murphy wrote:
> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli <kreijack@libero.it> wrote:
>>
>> I think that for the systemd uses cases (singled device FS), a simpler
>> approach would be:
>>
>>       fstatfs(fd, &sfs)
>>       needed = sfs.f_blocks - sfs.f_bavail;
>>       needed *= sfs.f_bsize
>>
>>       needed = roundup_64(needed, 3*(1024*1024*1024))
>>
>> Comparing the original systemd-homed code, I made the following changes
>> - 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation)
>> - 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP))
>>
>> Comments ?
> 
> I'm still wondering if such a significant shrink is even indicated, in
> lieu of trim. Isn't it sufficient to just trim on logout, thus
> returning unused blocks to the underlying filesystem? 

I agree with you. In Fedora 35, and the default is ext4+luks+trim
which provides the same results. However I remember that in the past the default
was btrfs+luks+shrunk. I think that something is changed i.

However, I want to provide do the systemd folks a suggestion ho change the code.
Even a warning like: "it doesn't work that because this, please drop it"
would be sufficient.

> And then do an
> fs resize (shrink or grow) as needed on login, so that the user home
> shows ~80% of the free space in the underlying file system?
> 
> homework-luks.c:3407:                                /* Before we
> shrink, let's trim the file system, so that we need less space on disk
> during the shrinking */
> 
> 
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: No space left errors on shutdown with systemd-homed /home dir

[Colin Guthrie]

Goffredo Baroncelli wrote on 30/01/2022 09:27:
> On 29/01/2022 19.01, Chris Murphy wrote:
>> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli 
>> <kreijack@libero.it> wrote:
>>>
>>> I think that for the systemd uses cases (singled device FS), a simpler
>>> approach would be:
>>>
>>>       fstatfs(fd, &sfs)
>>>       needed = sfs.f_blocks - sfs.f_bavail;
>>>       needed *= sfs.f_bsize
>>>
>>>       needed = roundup_64(needed, 3*(1024*1024*1024))
>>>
>>> Comparing the original systemd-homed code, I made the following changes
>>> - 1) f_bfree is replaced by f_bavail (which seem to be more 
>>> consistent to the disk usage; to me it seems to consider also the 
>>> metadata chunk allocation)
>>> - 2) the needing value is rounded up of 3GB in order to consider a 
>>> further 1 data chunk and 2 metadata chunk (DUP))
>>>
>>> Comments ?
>>
>> I'm still wondering if such a significant shrink is even indicated, in
>> lieu of trim. Isn't it sufficient to just trim on logout, thus
>> returning unused blocks to the underlying filesystem? 
> 
> I agree with you. In Fedora 35, and the default is ext4+luks+trim
> which provides the same results. However I remember that in the past the 
> default
> was btrfs+luks+shrunk. I think that something is changed i.
> 
> However, I want to provide do the systemd folks a suggestion ho change 
> the code.
> Even a warning like: "it doesn't work that because this, please drop it"
> would be sufficient.


Out of curiosity (see other thread on the systemd list about this), what 
it the current recommendation (by systemd/btrfs folks rather then Fedora 
defaults) for homed machine partitioning?

I have a new system to setup and the base filesystem is currently btrfs 
but I'm wondering if I should be putting a large luks file on that (can 
set nocow, but even still) or whether I should redo the base FS as ext4?

Open to some degree of experimentation, especially if the next six 
months promises new features that will be useful etc.

Col

-- 

Colin Guthrie

Re: No space left errors on shutdown with systemd-homed /home dir

[Neal Gompa]

On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> wrote:
>
> Goffredo Baroncelli wrote on 30/01/2022 09:27:
> > On 29/01/2022 19.01, Chris Murphy wrote:
> >> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli
> >> <kreijack@libero.it> wrote:
> >>>
> >>> I think that for the systemd uses cases (singled device FS), a simpler
> >>> approach would be:
> >>>
> >>>       fstatfs(fd, &sfs)
> >>>       needed = sfs.f_blocks - sfs.f_bavail;
> >>>       needed *= sfs.f_bsize
> >>>
> >>>       needed = roundup_64(needed, 3*(1024*1024*1024))
> >>>
> >>> Comparing the original systemd-homed code, I made the following changes
> >>> - 1) f_bfree is replaced by f_bavail (which seem to be more
> >>> consistent to the disk usage; to me it seems to consider also the
> >>> metadata chunk allocation)
> >>> - 2) the needing value is rounded up of 3GB in order to consider a
> >>> further 1 data chunk and 2 metadata chunk (DUP))
> >>>
> >>> Comments ?
> >>
> >> I'm still wondering if such a significant shrink is even indicated, in
> >> lieu of trim. Isn't it sufficient to just trim on logout, thus
> >> returning unused blocks to the underlying filesystem?
> >
> > I agree with you. In Fedora 35, and the default is ext4+luks+trim
> > which provides the same results. However I remember that in the past the
> > default
> > was btrfs+luks+shrunk. I think that something is changed i.
> >
> > However, I want to provide do the systemd folks a suggestion ho change
> > the code.
> > Even a warning like: "it doesn't work that because this, please drop it"
> > would be sufficient.
>
>
> Out of curiosity (see other thread on the systemd list about this), what
> it the current recommendation (by systemd/btrfs folks rather then Fedora
> defaults) for homed machine partitioning?
>

I'd probably recommend Btrfs with the /home subvolume set with
nodatacow if you're going to use loops of LUKS backed Btrfs homedir
images. The individual Btrfs loops will have their own COW anyway.

Otherwise, the Fedora defaults for Btrfs should be sufficient.



--
真実はいつも一つ！/ Always, there's only one truth!

Re: No space left errors on shutdown with systemd-homed /home dir

[Colin Guthrie]

[-- Attachment #1: Type: text/plain, Size: 3398 bytes --]

Hi,

Neal Gompa wrote on 01/02/2022 19:55:
> On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> wrote:
>>
>> Goffredo Baroncelli wrote on 30/01/2022 09:27:
>>> On 29/01/2022 19.01, Chris Murphy wrote:
>>>> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli
>>>> <kreijack@libero.it> wrote:
>>>>>
>>>>> I think that for the systemd uses cases (singled device FS), a simpler
>>>>> approach would be:
>>>>>
>>>>>        fstatfs(fd, &sfs)
>>>>>        needed = sfs.f_blocks - sfs.f_bavail;
>>>>>        needed *= sfs.f_bsize
>>>>>
>>>>>        needed = roundup_64(needed, 3*(1024*1024*1024))
>>>>>
>>>>> Comparing the original systemd-homed code, I made the following changes
>>>>> - 1) f_bfree is replaced by f_bavail (which seem to be more
>>>>> consistent to the disk usage; to me it seems to consider also the
>>>>> metadata chunk allocation)
>>>>> - 2) the needing value is rounded up of 3GB in order to consider a
>>>>> further 1 data chunk and 2 metadata chunk (DUP))
>>>>>
>>>>> Comments ?
>>>>
>>>> I'm still wondering if such a significant shrink is even indicated, in
>>>> lieu of trim. Isn't it sufficient to just trim on logout, thus
>>>> returning unused blocks to the underlying filesystem?
>>>
>>> I agree with you. In Fedora 35, and the default is ext4+luks+trim
>>> which provides the same results. However I remember that in the past the
>>> default
>>> was btrfs+luks+shrunk. I think that something is changed i.
>>>
>>> However, I want to provide do the systemd folks a suggestion ho change
>>> the code.
>>> Even a warning like: "it doesn't work that because this, please drop it"
>>> would be sufficient.
>>
>>
>> Out of curiosity (see other thread on the systemd list about this), what
>> it the current recommendation (by systemd/btrfs folks rather then Fedora
>> defaults) for homed machine partitioning?
>>
> 
> I'd probably recommend Btrfs with the /home subvolume set with
> nodatacow if you're going to use loops of LUKS backed Btrfs homedir
> images. The individual Btrfs loops will have their own COW anyway.
> 
> Otherwise, the Fedora defaults for Btrfs should be sufficient.

Thought I'd wait for Fedora 36 to be released with everything I need to 
test this setup.

Fell at the first hurdle of transferring my data in!

I transferred a subset of my data (240Gb) onto an external disk and used:

   homectl with colin -- rsync ...


The transfer worked but the colin.home file grew to 394Gb. Only about 
184Gb used (I presume due to compression).

Ultimately, this was then unmounted and while it said it could shrink 
the filesystem with a "Ready to..." message this either didn't happen or 
the backing file wasn't shrunk to match it. I did receive a message later

I'm not sure now where it's at with recovery but as nothing is strictly 
needed to make this work, I think I'll leave my playing with homed there 
for now and try again at some later date.

I love the whole idea but it's still a bit to bleeding edge and quirky 
for my daily driver just yet!


I've attached various logs in case they are useful (will post separately 
if the list removes this!)


Cheers

Col


-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
   Tribalogic Limited http://www.tribalogic.net/
Open Source:
   Mageia Contributor http://www.mageia.org/
   PulseAudio Hacker http://www.pulseaudio.org/
   Trac Hacker http://trac.edgewall.org/

[-- Attachment #2: 1.homectl-with-file-transfer.log --]
[-- Type: text/x-log, Size: 4398 bytes --]

May 31 09:30:11 colins-laptop systemd-homed[865]: colin: changing state inactive → activating-for-acquire
May 31 09:30:11 colins-laptop systemd-homework[8653]: Provided password unlocks user record.
May 31 09:30:11 colins-laptop systemd-homework[8653]: Successfully locked image file '/home/colin.home'.
May 31 09:30:11 colins-laptop systemd-homework[8653]: Backing file is fully allocated already.
May 31 09:30:11 colins-laptop systemd-homework[8653]: Setting up loopback device /dev/loop0 completed.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Setting up LUKS device /dev/mapper/home-colin completed.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Provided password unlocks user record.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Probing file system completed (found btrfs).
May 31 09:30:12 colins-laptop systemd-homework[8653]: File system check completed.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Mounting file system completed.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Discovered used loopback device /dev/loop0.
May 31 09:30:12 colins-laptop systemd-homework[8653]: offset = 1048576, size = 423003255296, image = 423004320768
May 31 09:30:12 colins-laptop systemd-homework[8653]: Ready to resize image size 393.9G → 458.1G, partition size 393.9G → 458.1G, file system size 393.9G → 458.0G.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Couldn't change image size.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Read embedded .identity file.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Provided password unlocks user record.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Reconciling user identities completed (host and header version were identical).
May 31 09:30:12 colins-laptop systemd-homework[8653]: Reconciling embedded user identity completed (host and embedded version were identical).
May 31 09:30:12 colins-laptop systemd-homework[8653]: Recursive changing of ownership not necessary, skipped.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Synchronized disk.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Moving to final mount point /home/colin completed.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Activation completed.
May 31 09:30:12 colins-laptop systemd-homework[8653]: Image size is 393.9G, file system size is 393.9G, file system payload size is 393.9G, file system free is 393.5G.
May 31 09:30:12 colins-laptop systemd-homed[865]: Home colin is signed exclusively by our key, accepting.
May 31 09:30:12 colins-laptop systemd-homed[865]: colin: changing state activating-for-acquire → active
May 31 09:30:12 colins-laptop systemd-homed[865]: Rebalancing complete.
May 31 09:39:47 colins-laptop systemd-homed[865]: Rebalancing complete.
May 31 09:49:23 colins-laptop systemd-homed[865]: Rebalancing complete.
May 31 09:58:25 colins-laptop systemd-homed[865]: Got notification that all sessions of user colin ended, deactivating automatically.
May 31 09:58:25 colins-laptop systemd-homed[865]: colin: changing state active → deactivating
May 31 09:58:25 colins-laptop systemd-homework[9567]: Discarded unused 212.6G.
May 31 09:58:26 colins-laptop systemd-homework[9567]: Syncing completed.
May 31 09:58:27 colins-laptop systemd-homework[9567]: Discovered used LUKS device /dev/mapper/home-colin, and validated password.
May 31 09:58:27 colins-laptop systemd-homework[9567]: Successfully re-activated LUKS device.
May 31 09:58:27 colins-laptop systemd-homework[9567]: Discovered used loopback device /dev/loop0.
May 31 09:58:27 colins-laptop systemd-homework[9567]: offset = 1048576, size = 423003255296, image = 423004320768
May 31 09:58:27 colins-laptop systemd-homework[9567]: Ready to resize image size 393.9G → 181.0G, partition size 393.9G → 181.0G, file system size 393.9G → 181.0G.
May 31 09:58:28 colins-laptop systemd-homework[9567]: Unmounting completed.
May 31 09:58:28 colins-laptop systemd-homework[9567]: Discovered used LUKS device /dev/mapper/home-colin.
May 31 09:58:28 colins-laptop systemd-homework[9567]: Device home-colin is not active.
May 31 09:58:28 colins-laptop systemd-homed[865]: block device /sys/devices/virtual/block/dm-0 has been removed.
May 31 09:58:28 colins-laptop systemd-homework[9567]: Everything completed.
May 31 09:58:28 colins-laptop systemd-homed[865]: colin: changing state deactivating → inactive


[-- Attachment #3: 2.homectl-with-test-after-reboot.log --]
[-- Type: text/x-log, Size: 2932 bytes --]

May 31 10:48:36 colins-laptop systemd[1]: Starting systemd-homed.service - Home Area Manager...
May 31 10:48:36 colins-laptop systemd-homed[861]: Successfully loaded private key pair.
May 31 10:48:36 colins-laptop systemd-homed[861]: Watching /home.
May 31 10:48:36 colins-laptop systemd-homed[861]: User record colin.identity is signed only by us, accepting.
May 31 10:48:36 colins-laptop systemd-homed[861]: Added registered home for user colin.
May 31 10:48:36 colins-laptop systemd[1]: Started systemd-homed.service - Home Area Manager.
May 31 10:50:19 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire
May 31 10:50:19 colins-laptop systemd-homework[2865]: None of the supplied plaintext passwords unlock the user record's hashed passwords.
May 31 10:50:19 colins-laptop systemd-homed[861]: Activation failed: Required key not available
May 31 10:50:19 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive
May 31 10:50:19 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically.
May 31 10:50:19 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed.
May 31 10:50:22 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire
May 31 10:50:22 colins-laptop systemd-homework[2873]: Provided password unlocks user record.
May 31 10:50:22 colins-laptop systemd-homework[2873]: Successfully locked image file '/home/colin.home'.
May 31 10:50:22 colins-laptop systemd-homed[861]: Activation failed: No space left on device
May 31 10:50:22 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive
May 31 10:50:22 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically.
May 31 10:50:22 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed.
May 31 11:25:45 colins-laptop systemd-homed[861]: colin: changing state inactive → authenticating
May 31 11:25:45 colins-laptop systemd-homework[4330]: None of the supplied plaintext passwords unlock the user record's hashed passwords.
May 31 11:25:45 colins-laptop systemd-homed[861]: Authentication failed: Required key not available
May 31 11:25:45 colins-laptop systemd-homed[861]: colin: changing state authenticating → inactive
May 31 11:25:48 colins-laptop systemd-homed[861]: colin: changing state inactive → authenticating
May 31 11:25:48 colins-laptop systemd-homework[4335]: Provided password unlocks user record.
May 31 11:25:48 colins-laptop systemd-homework[4335]: Successfully locked image file '/home/colin.home'.
May 31 11:25:48 colins-laptop systemd-homed[861]: Authentication failed: No space left on device
May 31 11:25:48 colins-laptop systemd-homed[861]: colin: changing state authenticating → inactive


[-- Attachment #4: 3.homectl-with-test-debug.log --]
[-- Type: text/x-log, Size: 10256 bytes --]

May 31 11:31:46 colins-laptop systemd-homed[861]: Setting log level to debug.
May 31 11:31:46 colins-laptop systemd-homed[861]: Sent message type=method_return sender=n/a destination=:1.144 path=n/a interface=n/a member=n/a cookie=41 reply_cookie=3 signature=n/a error-name=n/a error-message=n/a
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max)
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}}
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid">
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max)
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}}
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid">
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max)
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}}
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid">
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected
May 31 11:32:06 colins-laptop systemd-homed[861]: Got message type=method_call sender=:1.145 destination=org.freedesktop.home1 path=/org/freedesktop/home1 interface=org.freedesktop.home1.Manager member=AcquireHome cookie=2 reply_cookie=0>
May 31 11:32:06 colins-laptop systemd-homed[861]: Sending to worker: {"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid":60242,"homeDirectory":"/home/coli>
May 31 11:32:06 colins-laptop systemd-homed[861]: Successfully forked off '(sd-homework)' as PID 4678.
May 31 11:32:06 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire
May 31 11:32:06 colins-laptop systemd-homework[4678]: None of the supplied plaintext passwords unlock the user record's hashed passwords.
May 31 11:32:06 colins-laptop systemd-homed[861]: Worker reported error code ENOKEY.
May 31 11:32:06 colins-laptop systemd-homed[861]: Activation failed: Required key not available
May 31 11:32:06 colins-laptop systemd-homed[861]: Sent message type=error sender=n/a destination=:1.145 path=n/a interface=n/a member=n/a cookie=42 reply_cookie=2 signature=s error-name=org.freedesktop.home1.BadPassword error-message=Pas>
May 31 11:32:06 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive
May 31 11:32:06 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically.
May 31 11:32:06 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: New incoming connection.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink: Connections of user 0: 0 (of 1024 max)
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Setting state idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: New incoming message: {"method":"io.systemd.UserDatabase.GetUserRecord","parameters":{"userName":"colin","service":"io.systemd.Home"}}
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → processing-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Sending message: {"parameters":{"record":{"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid">
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-method → processed-method
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processed-method → idle-server
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Got POLLHUP from socket.
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state idle-server → pending-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state pending-disconnect → processing-disconnect
May 31 11:32:06 colins-laptop systemd-homed[861]: varlink-10: Changing state processing-disconnect → disconnected
May 31 11:32:09 colins-laptop systemd-homed[861]: Got message type=method_call sender=:1.145 destination=org.freedesktop.home1 path=/org/freedesktop/home1 interface=org.freedesktop.home1.Manager member=AcquireHome cookie=3 reply_cookie=0>
May 31 11:32:09 colins-laptop systemd-homed[861]: Sending to worker: {"binding":{"829fdf6bdee3486c90a4a34072f91754":{"fileSystemType":"btrfs","fileSystemUuid":"bc694cce-b5d7-4b25-99ff-df2be6ff9356","gid":60242,"homeDirectory":"/home/coli>
May 31 11:32:09 colins-laptop systemd-homed[861]: Successfully forked off '(sd-homework)' as PID 4681.
May 31 11:32:09 colins-laptop systemd-homed[861]: colin: changing state inactive → activating-for-acquire
May 31 11:32:09 colins-laptop systemd-homework[4681]: Provided password unlocks user record.
May 31 11:32:09 colins-laptop systemd-homework[4681]: Successfully locked image file '/home/colin.home'.
May 31 11:32:09 colins-laptop systemd-homed[861]: Successfully acquired LUKS lock fd from worker.
May 31 11:32:09 colins-laptop systemd-homed[861]: Worker reported error code ENOSPC.
May 31 11:32:09 colins-laptop systemd-homed[861]: Activation failed: No space left on device
May 31 11:32:09 colins-laptop systemd-homed[861]: Sent message type=error sender=n/a destination=:1.145 path=n/a interface=n/a member=n/a cookie=43 reply_cookie=3 signature=s error-name=org.freedesktop.home1.NoDiskSpace error-message=Not>
May 31 11:32:09 colins-laptop systemd-homed[861]: colin: changing state activating-for-acquire → inactive
May 31 11:32:09 colins-laptop systemd-homed[861]: Successfully closed LUKS backing file lock for colin.
May 31 11:32:09 colins-laptop systemd-homed[861]: /home/colin.home has been modified, having a look.
May 31 11:32:09 colins-laptop systemd-homed[861]: Found an image for user colin which already has a record, skipping.
May 31 11:32:09 colins-laptop systemd-homed[861]: /home/colin.home has been closed after writing, revalidating.
May 31 11:32:09 colins-laptop systemd-homed[861]: Got notification that all sessions of user colin ended, deactivating automatically.
May 31 11:32:09 colins-laptop systemd-homed[861]: Home colin already deactivated, no automatic deactivation needed.
May 31 11:32:09 colins-laptop systemd-homed[861]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/home1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=44 reply_cookie=0 signature=sa{sv>


[-- Attachment #5: 4.homectl-inspect.log --]
[-- Type: text/x-log, Size: 1120 bytes --]

[root@colins-laptop home]# homectl inspect colin
   User name: colin
       State: inactive
 Disposition: regular
 Last Change: Sun 2022-05-29 21:00:25 BST
 Last Passw.: Wed 2022-01-05 16:57:13 GMT
    Login OK: yes
 Password OK: yes
         UID: 60242
         GID: 60242 (colin)
 Aux. Groups: colin
              vboxusers
              wheel
   Real Name: Colin Guthrie
   Directory: /home/colin
     Storage: luks (strong encryption)
  Image Path: /home/colin.home
   Removable: no
       Shell: /bin/bash
LUKS Discard: online=no offline=yes
   LUKS UUID: 387b9bc6-6b58-489b-a7a7-f350e5a37609
   Part UUID: a394b420-012f-402e-8139-1d120180fd04
     FS UUID: bc694cce-b5d7-4b25-99ff-df2be6ff9356
 File System: btrfs
 LUKS Cipher: aes
 Cipher Mode: xts-plain64
  Volume Key: 256bit
 Mount Flags: nosuid nodev exec
   Disk Size: 393.9G
  Disk Floor: 256.0M
Disk Ceiling: 458.0G
  Good Auth.: 34
   Last Good: Tue 2022-05-31 09:30:12 BST
   Bad Auth.: 39
    Last Bad: Tue 2022-05-31 11:27:41 BST
    Next Try: anytime
 Auth. Limit: 30 attempts per 1min
   Passwords: 1
  Local Sig.: yes
     Service: io.systemd.Home


[-- Attachment #6: 5.btrfs-filesystem-usage.log --]
[-- Type: text/x-log, Size: 684 bytes --]

Overall:
    Device size:		 468.14GiB
    Device allocated:		 468.14GiB
    Device unallocated:		   1.00MiB
    Device missing:		     0.00B
    Used:			 191.65GiB
    Free (estimated):		 275.72GiB	(min: 275.72GiB)
    Free (statfs, df):		 275.72GiB
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		  21.12MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:467.13GiB, Used:191.41GiB (40.97%)
   /dev/nvme0n1p3	 467.13GiB

Metadata,single: Size:1.01GiB, Used:249.56MiB (24.18%)
   /dev/nvme0n1p3	   1.01GiB

System,single: Size:4.00MiB, Used:80.00KiB (1.95%)
   /dev/nvme0n1p3	   4.00MiB

Unallocated:
   /dev/nvme0n1p3	   1.00MiB


[-- Attachment #7: 6.dot-home-file-stats.log --]
[-- Type: text/x-log, Size: 825 bytes --]

[root@colins-laptop home]# ls -l /home/colin.home 
-rw-------. 1 root root 423004320768 May 31 09:58 /home/colin.home
[root@colins-laptop home]# ls -lh /home/colin.home 
-rw-------. 1 root root 394G May 31 09:58 /home/colin.home
[root@colins-laptop home]# stat /home/colin.home
  File: /home/colin.home
  Size: 423004320768	Blocks: 382312560  IO Block: 4096   regular file
Device: 0,41	Inode: 4708        Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-05-31 11:32:09.964909547 +0100
Modify: 2022-05-31 09:58:27.983134041 +0100
Change: 2022-05-31 11:32:09.976909412 +0100
 Birth: -6024107610197374494.-683951100
[root@colins-laptop home]# du -sh /home/colin.home
183G	/home/colin.home
[root@colins-laptop home]# lsattr /home/colin.home 
---------------C------ /home/colin.home

Re: No space left errors on shutdown with systemd-homed /home dir

[Goffredo Baroncelli]

On 31/05/2022 14.44, Colin Guthrie wrote:
> Hi,
> 
> Neal Gompa wrote on 01/02/2022 19:55:
>> On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> wrote:
>>>
>>> Goffredo Baroncelli wrote on 30/01/2022 09:27:
>>>> On 29/01/2022 19.01, Chris Murphy wrote:
>>>>> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli
>>>>> <kreijack@libero.it> wrote:
>>>>>>
>>>>>> I think that for the systemd uses cases (singled device FS), a simpler
>>>>>> approach would be:
>>>>>>
>>>>>>        fstatfs(fd, &sfs)
>>>>>>        needed = sfs.f_blocks - sfs.f_bavail;
>>>>>>        needed *= sfs.f_bsize
>>>>>>
>>>>>>        needed = roundup_64(needed, 3*(1024*1024*1024))
>>>>>>
>>>>>> Comparing the original systemd-homed code, I made the following changes
>>>>>> - 1) f_bfree is replaced by f_bavail (which seem to be more
>>>>>> consistent to the disk usage; to me it seems to consider also the
>>>>>> metadata chunk allocation)
>>>>>> - 2) the needing value is rounded up of 3GB in order to consider a
>>>>>> further 1 data chunk and 2 metadata chunk (DUP))
>>>>>>
>>>>>> Comments ?
>>>>>
>>>>> I'm still wondering if such a significant shrink is even indicated, in
>>>>> lieu of trim. Isn't it sufficient to just trim on logout, thus
>>>>> returning unused blocks to the underlying filesystem?
>>>>
>>>> I agree with you. In Fedora 35, and the default is ext4+luks+trim
>>>> which provides the same results. However I remember that in the past the
>>>> default
>>>> was btrfs+luks+shrunk. I think that something is changed i.
>>>>
>>>> However, I want to provide do the systemd folks a suggestion ho change
>>>> the code.
>>>> Even a warning like: "it doesn't work that because this, please drop it"
>>>> would be sufficient.
>>>
>>>
>>> Out of curiosity (see other thread on the systemd list about this), what
>>> it the current recommendation (by systemd/btrfs folks rather then Fedora
>>> defaults) for homed machine partitioning?
>>>
>>
>> I'd probably recommend Btrfs with the /home subvolume set with
>> nodatacow if you're going to use loops of LUKS backed Btrfs homedir
>> images. The individual Btrfs loops will have their own COW anyway.
>>
>> Otherwise, the Fedora defaults for Btrfs should be sufficient.
> 
> Thought I'd wait for Fedora 36 to be released with everything I need to test this setup.
> 
> Fell at the first hurdle of transferring my data in!
> 
> I transferred a subset of my data (240Gb) onto an external disk and used:
> 
>    homectl with colin -- rsync ...
> 
> 
> The transfer worked but the colin.home file grew to 394Gb. Only about 184Gb used (I presume due to compression).
> 
> Ultimately, this was then unmounted and while it said it could shrink the filesystem with a "Ready to..." message this either didn't happen or the backing file wasn't shrunk to match it. I did receive a message later


I suppose that colin.home is a sparse file, so even it has a length of 394GB, it consumes only 184GB. So to me these are valid values. It doesn't matter the length of the files. What does matter is the value returned by "du -sh".

Below I create a file with a length of 1000GB. However being a sparse file, it doesn't consume any space and "du -sh" returns 0

$ truncate -s 1000GB foo
$ du -sh foo
0	foo
$ ls -l foo
-rw-r--r-- 1 ghigo ghigo 1000000000000 May 31 19:29 foo



> 
> I'm not sure now where it's at with recovery but as nothing is strictly needed to make this work, I think I'll leave my playing with homed there for now and try again at some later date.
> 
> I love the whole idea but it's still a bit to bleeding edge and quirky for my daily driver just yet!
> 
> 
> I've attached various logs in case they are useful (will post separately if the list removes this!)
> 
> 
> Cheers
> 
> Col
> 
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: No space left errors on shutdown with systemd-homed /home dir

[Colin Guthrie]

Goffredo Baroncelli wrote on 31/05/2022 19:12:
> On 31/05/2022 14.44, Colin Guthrie wrote:
>> Hi,
>>
>> Neal Gompa wrote on 01/02/2022 19:55:
>>> On Tue, Feb 1, 2022 at 2:02 PM Colin Guthrie <colin@booksterhq.com> 
>>> wrote:
>>>>
>>>> Goffredo Baroncelli wrote on 30/01/2022 09:27:
>>>>> On 29/01/2022 19.01, Chris Murphy wrote:
>>>>>> On Sat, Jan 29, 2022 at 2:53 AM Goffredo Baroncelli
>>>>>> <kreijack@libero.it> wrote:
>>>>>>>
>>>>>>> I think that for the systemd uses cases (singled device FS), a 
>>>>>>> simpler
>>>>>>> approach would be:
>>>>>>>
>>>>>>>        fstatfs(fd, &sfs)
>>>>>>>        needed = sfs.f_blocks - sfs.f_bavail;
>>>>>>>        needed *= sfs.f_bsize
>>>>>>>
>>>>>>>        needed = roundup_64(needed, 3*(1024*1024*1024))
>>>>>>>
>>>>>>> Comparing the original systemd-homed code, I made the following 
>>>>>>> changes
>>>>>>> - 1) f_bfree is replaced by f_bavail (which seem to be more
>>>>>>> consistent to the disk usage; to me it seems to consider also the
>>>>>>> metadata chunk allocation)
>>>>>>> - 2) the needing value is rounded up of 3GB in order to consider a
>>>>>>> further 1 data chunk and 2 metadata chunk (DUP))
>>>>>>>
>>>>>>> Comments ?
>>>>>>
>>>>>> I'm still wondering if such a significant shrink is even 
>>>>>> indicated, in
>>>>>> lieu of trim. Isn't it sufficient to just trim on logout, thus
>>>>>> returning unused blocks to the underlying filesystem?
>>>>>
>>>>> I agree with you. In Fedora 35, and the default is ext4+luks+trim
>>>>> which provides the same results. However I remember that in the 
>>>>> past the
>>>>> default
>>>>> was btrfs+luks+shrunk. I think that something is changed i.
>>>>>
>>>>> However, I want to provide do the systemd folks a suggestion ho change
>>>>> the code.
>>>>> Even a warning like: "it doesn't work that because this, please 
>>>>> drop it"
>>>>> would be sufficient.
>>>>
>>>>
>>>> Out of curiosity (see other thread on the systemd list about this), 
>>>> what
>>>> it the current recommendation (by systemd/btrfs folks rather then 
>>>> Fedora
>>>> defaults) for homed machine partitioning?
>>>>
>>>
>>> I'd probably recommend Btrfs with the /home subvolume set with
>>> nodatacow if you're going to use loops of LUKS backed Btrfs homedir
>>> images. The individual Btrfs loops will have their own COW anyway.
>>>
>>> Otherwise, the Fedora defaults for Btrfs should be sufficient.
>>
>> Thought I'd wait for Fedora 36 to be released with everything I need 
>> to test this setup.
>>
>> Fell at the first hurdle of transferring my data in!
>>
>> I transferred a subset of my data (240Gb) onto an external disk and used:
>>
>>    homectl with colin -- rsync ...
>>
>>
>> The transfer worked but the colin.home file grew to 394Gb. Only about 
>> 184Gb used (I presume due to compression).
>>
>> Ultimately, this was then unmounted and while it said it could shrink 
>> the filesystem with a "Ready to..." message this either didn't happen 
>> or the backing file wasn't shrunk to match it. I did receive a message 
>> later
> 
> 
> I suppose that colin.home is a sparse file, so even it has a length of 
> 394GB, it consumes only 184GB. So to me these are valid values. It 
> doesn't matter the length of the files. What does matter is the value 
> returned by "du -sh".
> 
> Below I create a file with a length of 1000GB. However being a sparse 
> file, it doesn't consume any space and "du -sh" returns 0
> 
> $ truncate -s 1000GB foo
> $ du -sh foo
> 0    foo
> $ ls -l foo
> -rw-r--r-- 1 ghigo ghigo 1000000000000 May 31 19:29 foo

Yeah the file will be sparse.

That's not really an issue, I'm not worried about the fact it's not 
consuming as much as it reports as that's all expected.

The issue is that systemd-homed (or btrfs's fallocate) can't handle this 
situation and that user is effectively bricked unless migrated to a host 
with more storage space!

Col

-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
   Tribalogic Limited http://www.tribalogic.net/
Open Source:
   Mageia Contributor http://www.mageia.org/
   PulseAudio Hacker http://www.pulseaudio.org/
   Trac Hacker http://trac.edgewall.org/

Re: No space left errors on shutdown with systemd-homed /home dir

[Chris Murphy]

[sorry had to resend to get it on btrfs list, due to html in the original :\]

On Wed, Jun 1, 2022, at 5:36 AM, Colin Guthrie wrote:
> Goffredo Baroncelli wrote on 31/05/2022 19:12:
> 
> > I suppose that colin.home is a sparse file, so even it has a length of 
> > 394GB, it consumes only 184GB. So to me these are valid values. It 
> > doesn't matter the length of the files. What does matter is the value 
> > returned by "du -sh".
> > 
> > Below I create a file with a length of 1000GB. However being a sparse 
> > file, it doesn't consume any space and "du -sh" returns 0
> > 
> > $ truncate -s 1000GB foo
> > $ du -sh foo
> > 0    foo
> > $ ls -l foo
> > -rw-r--r-- 1 ghigo ghigo 1000000000000 May 31 19:29 foo
> 
> Yeah the file will be sparse.
> 
> That's not really an issue, I'm not worried about the fact it's not 
> consuming as much as it reports as that's all expected.
> 
> The issue is that systemd-homed (or btrfs's fallocate) can't handle this 
> situation and that user is effectively bricked unless migrated to a host 
> with more storage space!

Hopefully there's time for systemd-252 for a change still? That version is what I expect to ship in Fedora 37 [1] There's merit to sd-homed and I want it to be safe and reliable for users to keep using, in order to build momentum. 

I really think sd-homed must move the shrink on logout, to login.

When the user logs out, they are decently likely to immediately close the laptop lid thus suspend-to-ram; or shutdown. I don't know if shrink can be cancelled. But regardless, there's going to be a period of time where the file system and storage stacks are busy, right at the time the user is expecting *imminent* suspend or shutdown, which no matter what has to be inhibited until the shrink is cancelled or completed, and all pending writes are flushed to stable media.

Next, consider the low battery situation. Upon notification, anyone with an 18+ month old battery knows there may be no additional warnings, and you could in fact get a power failure next. In this scenario we have to depend on all storage stack layers, and the drive firmware, doing the exact correct thing in order for the file system to be in a consistent state to be mountable at next boot. I just think this is too much risk, and since sd-homed is targeted at laptop users primarily, all the more reason the fs resize operation should happen at login time, not logout.

In fact, sd-homed might want to inhibit a resize shrink operation if (a) AC power is not plugged in and (b) battery remaining is less than 30%, or some other reasonable value. The resize grow operation is sufficiently cheap and fast that I don't think it needs inhibiting.

Thoughts?

I also just found a few bug reports with a non-exhaustive search that also make me nervous about fs shrink at logout (also implying restart and shutdown) time.

On shutdown, homed resizes until it gets killed
https://github.com/systemd/systemd/issues/22901
Getting "New partition doesn't fit into backing storage, refusing"
https://github.com/systemd/systemd/issues/22255
fails to resize 
https://github.com/systemd/systemd/issues/22124



[1]
Branch from Rawhide August 9, the earliest release date would be October 18.



--
Chris Murphy

Re: No space left errors on shutdown with systemd-homed /home dir

[Zygo Blaxell]

On Sat, Jan 29, 2022 at 10:53:00AM +0100, Goffredo Baroncelli wrote:
> On 27/01/2022 21.48, Chris Murphy wrote:
> > On Wed, Jan 26, 2022 at 4:19 PM Boris Burkov <boris@bur.io> wrote:
> [...]
> > 
> > systemd-homed by default uses btrfs on LUKS on loop mount, with a
> > backing file. On login, it grows the user home file system with some
> > percentage (I think 80%) of the free space of the underlying file
> > system. And on logout, it does both fstrim and shrinks the fs. I don't
> > know why it does both, it seems adequate to do only fstrim on logout
> > to return unused blocks to the underlying file system; and to do an fs
> > resize on login to either grow or shrink the user home file system.
> > 
> > But also, we don't really have a great estimator of the minimum size a
> > file system can be. `btrfs inspect-internal min-dev-size` is pretty
> > broken right now.
> > https://github.com/kdave/btrfs-progs/issues/271
> 
> I tried the test case, but was unable to get a wrong value. However
> I think that this is due to the fact that btrfs improved the bg reclaiming.
> 
> However tweaking the test case, I was able to trigger the problem (I
> reduced the filesize from 1GB to 256MB, so when some files are
> removed a BG is empty filled)
> 
> 
> 
> > 
> > I'm not sure if systemd folks would use libbtrfsutil facility to
> > determine the minimum device shrink size? But also even the kernel
> > doesn't have a very good idea of how small a file system can be
> > shrunk. Right now it basically has to just start trying, and does it
> > one block group at a time.
> 
> I think that for the systemd uses cases (singled device FS), a simpler
> approach would be:
> 
>     fstatfs(fd, &sfs)
>     needed = sfs.f_blocks - sfs.f_bavail;
>     needed *= sfs.f_bsize
> 
>     needed = roundup_64(needed, 3*(1024*1024*1024))
> 
> Comparing the original systemd-homed code, I made the following changes
> - 1) f_bfree is replaced by f_bavail (which seem to be more consistent to the disk usage; to me it seems to consider also the metadata chunk allocation)
> - 2) the needing value is rounded up of 3GB in order to consider a further 1 data chunk and 2 metadata chunk (DUP))
> 
> Comments ?

This is closer to the right answer but not quite there yet.  A summary
of the issues:

 * Discard (called by systemd-homed in the form of trim) locks a block
 group (makes it read-only and removes it from available space) while
 it runs.

 * Relocation (balance or filesystem resize) locks a block group while
 it runs.

 * Scrub in the worst case locks one block group per disk (but we may
 never run scrub on a systemd-homed filesystem, so ignore this for now).

 * Large (>50GB) filesystems have larger block groups than smaller
 filesystems.  Resizing from >50GB to <50GB can require rewriting the
 _entire_ filesystem to make a sensible number of smaller block groups
 (high enough to be able to lock all the above block groups and still
 have enough free space to run a transaction without them).

 * System chunks aren't the same size as other chunks, which will create
 unusable free space holes between block groups, and (after lots of
 balancing/resizing runs) possibly create a lot of unusable free space
 that existing extents cannot be relocated into without temporarily
 increasing the size of the filesystem.

 * Resize is a fairly dumb algorithm as algorithms go.  It works in one
 pass, in a fixed order, and it can't fragment an extent or a block group.
 The minimum size of a filesystem depends not just on how much data there
 is, but how capable the resize algorithm is at arranging the data into
 the space given all the overlapping constraints btrfs has on allocation.
 Resize makes several size-speed tradeoffs in favor of speed (or at least
 not in favor of size).

 * The minimum filesystem size to store the data is different from
 the minimum filesystem size to run specific btrfs data modification
 operations.  Some metadata operations can require significant amounts
 of space to complete.  If the filesystem is resized too small with
 exactly the right amount of free space, it may become impossible to
 perform metadata-intensive operations like orphan inode cleanup or
 snapshot deletion on the next mount.  It's not possible to predict
 these additional space requirements without doing equivalent work to
 performing the operations and measuring the space required.  This means
 that in order to compute the minimum filesystem size, we need to be
 able to predict (or strongly control) the future of the filesystem,
 at least long enough to grow the filesystem back to its service size.

These combine to make it especially challenging to resize a nearly empty
filesystem from 128GB to smaller than somewhere around 8GB (1GB data +
1GB locked by discard + 1GB locked by relocation + 1GB metadata * 2 for
dup + 2GB trapped free dev_extent space from earlier resize operations).
That could be reduced to about 6GB with single metadata, but that's a
significant resiliency trade-off if the host filesystem doesn't implement
data integrity and self-healing.

It might be doable in multiple resize passes--one to resize the filesystem
to <50GB so that all the data can be moved into smaller block groups,
then again to resize it to the next block group size.  In the worst
case this copies all of the data in the filesystem multiple times, so
it would be faster for systemd-homed to mkfs a new filesystem at the
target size (which would have smaller block groups from the beginning)
and simply copy the data with 'cp -a' to the new filesystem instead of
resizing the old one.

Resizing a filesystem with 1GB of data on it from over 50GB to 16GB is
probably reliable.  8GB is less likely to succeed, and I wouldn't expect
any smaller number to work reliably except on very simple test cases.

It does suck that the kernel handles resizing below the minimum size of
the filesystem so badly; however, even if it rejected the resize request
cleanly with an error, it's not necessarily a good idea to attempt it.
Pushing the lower limits of what is possible in resize to save a handful
of GB is asking for trouble.  It's far better to overestimate generously
than to underestimate the minimum size.

> > 
> > Adding systemd-devel@
> > 
> > 
> 
> 
> -- 
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: No space left errors on shutdown with systemd-homed /home dir

[Chris Murphy]

On Mon, Jan 31, 2022, at 11:26 PM, Zygo Blaxell wrote:
> On Sat, Jan 29, 2022 at 10:53:00AM +0100, Goffredo Baroncelli wrote:

> It does suck that the kernel handles resizing below the minimum size of
> the filesystem so badly; however, even if it rejected the resize request
> cleanly with an error, it's not necessarily a good idea to attempt it.
> Pushing the lower limits of what is possible in resize to save a handful
> of GB is asking for trouble.  It's far better to overestimate generously
> than to underestimate the minimum size.

Yeah there's an inherent conflict with online shrink: the longer the time needed to relocate bg's, the more unpredictable operations can occur during that time to thwart any original estimations made about the shrink operation.

I wondered a bit ago about a shrink API that takes shrink size as a suggestion rather than as a definite, and then the file system does the best job it can. Either this API reports actual shrink size once it completes, or the requesting program needs to know to call BTRFS_IOC_FS_INFO and BTRFS_IOC_DEV_INFO to know the actual size. This hypothetical API could have boundaries outside of which if the kernel code estimates it's going to fall short of, could trigger a cancel of the shrink. This could be size or time based.  e.g. BTRFS_IOC_RESIZE_BEST (effort).


-- 
Chris Murphy