Re: No space left errors on shutdown with systemd-homed /home dir

From: "Chris Murphy" <lists@colorremedies.com>
To: "Colin Guthrie" <gmane@colin.guthr.ie>,
	"Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Cc: "systemd List" <systemd-devel@lists.freedesktop.org>,
	"Goffredo Baroncelli" <kreijack@libero.it>,
	"Zygo Blaxell" <ce3g8jdj@umail.furryterror.org>
Subject: Re: No space left errors on shutdown with systemd-homed /home dir
Date: Sat, 23 Jul 2022 15:09:59 -0400	[thread overview]
Message-ID: <79adde9b-0ef6-4845-ab5a-c10a3563c3de@www.fastmail.com> (raw)
In-Reply-To: <t77bub$vp5$1@ciao.gmane.io>

[sorry had to resend to get it on btrfs list, due to html in the original :\]

On Wed, Jun 1, 2022, at 5:36 AM, Colin Guthrie wrote:
> Goffredo Baroncelli wrote on 31/05/2022 19:12:
> 
> > I suppose that colin.home is a sparse file, so even it has a length of 
> > 394GB, it consumes only 184GB. So to me these are valid values. It 
> > doesn't matter the length of the files. What does matter is the value 
> > returned by "du -sh".
> > 
> > Below I create a file with a length of 1000GB. However being a sparse 
> > file, it doesn't consume any space and "du -sh" returns 0
> > 
> > $ truncate -s 1000GB foo
> > $ du -sh foo
> > 0    foo
> > $ ls -l foo
> > -rw-r--r-- 1 ghigo ghigo 1000000000000 May 31 19:29 foo
> 
> Yeah the file will be sparse.
> 
> That's not really an issue, I'm not worried about the fact it's not 
> consuming as much as it reports as that's all expected.
> 
> The issue is that systemd-homed (or btrfs's fallocate) can't handle this 
> situation and that user is effectively bricked unless migrated to a host 
> with more storage space!

Hopefully there's time for systemd-252 for a change still? That version is what I expect to ship in Fedora 37 [1] There's merit to sd-homed and I want it to be safe and reliable for users to keep using, in order to build momentum. 

I really think sd-homed must move the shrink on logout, to login.

When the user logs out, they are decently likely to immediately close the laptop lid thus suspend-to-ram; or shutdown. I don't know if shrink can be cancelled. But regardless, there's going to be a period of time where the file system and storage stacks are busy, right at the time the user is expecting *imminent* suspend or shutdown, which no matter what has to be inhibited until the shrink is cancelled or completed, and all pending writes are flushed to stable media.

Next, consider the low battery situation. Upon notification, anyone with an 18+ month old battery knows there may be no additional warnings, and you could in fact get a power failure next. In this scenario we have to depend on all storage stack layers, and the drive firmware, doing the exact correct thing in order for the file system to be in a consistent state to be mountable at next boot. I just think this is too much risk, and since sd-homed is targeted at laptop users primarily, all the more reason the fs resize operation should happen at login time, not logout.

In fact, sd-homed might want to inhibit a resize shrink operation if (a) AC power is not plugged in and (b) battery remaining is less than 30%, or some other reasonable value. The resize grow operation is sufficiently cheap and fast that I don't think it needs inhibiting.

Thoughts?

I also just found a few bug reports with a non-exhaustive search that also make me nervous about fs shrink at logout (also implying restart and shutdown) time.

On shutdown, homed resizes until it gets killed
https://github.com/systemd/systemd/issues/22901
Getting "New partition doesn't fit into backing storage, refusing"
https://github.com/systemd/systemd/issues/22255
fails to resize 
https://github.com/systemd/systemd/issues/22124

[1]
Branch from Rawhide August 9, the earliest release date would be October 18.

--
Chris Murphy