[Help] Can't login to my systemd-homed user account due to fallocate failure

[Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

Hello! I've lost the ability to log in to my systemd-homed user account, and after some investigation on Systemd mailing list I was directed here. I would be very grateful for any help!

My root partition is ~475GiB with BTRFS, my home partition is a ~400GiB LUKS-encrypted partition on a loopback file (also BTRFS) created by systemd-homed (residing at /home/azymohliad.home). Which leaves ~75GiB for the rest of the root FS.

Recently I've lost the ability to log in to that user account, because during authentication systemd does fallocate call for the image. CLI alternative for my case (suggested on systemd mailing list, I don't really know what is it) is:

    fallocate -l 403G -n /home/azymohliad.home

Which fails:

    fallocate: fallocate failed: No space left on device

My first idea was that I occupied all those ~75GiB on a root partition, but cleaning didn't help (I definitely released more space than I could occupy during the last working session).

Here are some details about my system:

uname -a

    Linux az-wolf-pc 5.8.3-arch1-1 #1 SMP PREEMPT Fri, 21 Aug 2020 16:54:16 +0000 x86_64 GNU/Linux

btrfs --version

    btrfs-progs v5.7


I can mount my home manually like this:

    losetup -fP /home/azymohliad.home
    cryptsetup open /dev/loop0p1
    mount /dev/mapper/home /mnt

and then,

btrfs fi show

    Label: none  uuid: b68411ce-702a-4259-9121-ac21c9119ddf
    	Total devices 1 FS bytes used 299.71GiB
    	devid    1 size 476.44GiB used 476.44GiB path /dev/nvme0n1p2

    Label: 'azymohliad'  uuid: 4ffae38b-42c9-4e53-89a1-3d21cd862938
    	Total devices 1 FS bytes used 221.92GiB
    	devid    1 size 402.72GiB used 258.02GiB path /dev/mapper/home


btrfs fi df /

    Data, single: total=475.43GiB, used=299.28GiB
    System, single: total=4.00MiB, used=80.00KiB
    Metadata, single: total=1.01GiB, used=437.05MiB
    GlobalReserve, single: total=61.03MiB, used=0.00B


btrfs fi df /mnt

    Data, single: total=256.01GiB, used=221.18GiB
    System, single: total=4.00MiB, used=48.00KiB
    Metadata, single: total=2.01GiB, used=749.92MiB
    GlobalReserve, single: total=297.11MiB, used=0.00B

dmesg.log: https://gitlab.com/-/snippets/2007155

What's interesting to me from above, the partition size on /home/azymohliad.home is 402.72GiB, but the file system size is 256.01GiB, and the image file size is 256.64GiB (from btrfs fi du /home, although ls -lh reports 403GiB). I'm not really sure, but iirc the fs and image sizes were around 403GiB too earlier. Could it be that it somehow got automatically reduced?

Could I do anything to make that fallocate call (with -l 403G) working? It will allow me to authenticate to homectl and resize the home partition from there.

If not, what is the safe way to shrink that LUKS-partition size? Maybe then systemd-homed would do fallocate for less space and it would work.

If from my assumptions you could tell that I'm looking in the wrong direction, please give me a hint. Thanks for taking time to read it!

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Hugo Mills]

On Sun, Aug 23, 2020 at 03:31:47PM +0000, Andrii Zymohliad wrote:
> Hello! I've lost the ability to log in to my systemd-homed user account, and after some investigation on Systemd mailing list I was directed here. I would be very grateful for any help!
> 
> My root partition is ~475GiB with BTRFS, my home partition is a ~400GiB LUKS-encrypted partition on a loopback file (also BTRFS) created by systemd-homed (residing at /home/azymohliad.home). Which leaves ~75GiB for the rest of the root FS.
> 
> Recently I've lost the ability to log in to that user account, because during authentication systemd does fallocate call for the image. CLI alternative for my case (suggested on systemd mailing list, I don't really know what is it) is:
> 
>     fallocate -l 403G -n /home/azymohliad.home
> 
> Which fails:
> 
>     fallocate: fallocate failed: No space left on device
> 
> My first idea was that I occupied all those ~75GiB on a root partition, but cleaning didn't help (I definitely released more space than I could occupy during the last working session).
> 
> Here are some details about my system:
> 
> uname -a
> 
>     Linux az-wolf-pc 5.8.3-arch1-1 #1 SMP PREEMPT Fri, 21 Aug 2020 16:54:16 +0000 x86_64 GNU/Linux
> 
> btrfs --version
> 
>     btrfs-progs v5.7
> 
> 
> I can mount my home manually like this:
> 
>     losetup -fP /home/azymohliad.home
>     cryptsetup open /dev/loop0p1
>     mount /dev/mapper/home /mnt
> 
> and then,
> 
> btrfs fi show
> 
>     Label: none  uuid: b68411ce-702a-4259-9121-ac21c9119ddf
>     	Total devices 1 FS bytes used 299.71GiB
>     	devid    1 size 476.44GiB used 476.44GiB path /dev/nvme0n1p2
> 
>     Label: 'azymohliad'  uuid: 4ffae38b-42c9-4e53-89a1-3d21cd862938
>     	Total devices 1 FS bytes used 221.92GiB
>     	devid    1 size 402.72GiB used 258.02GiB path /dev/mapper/home
> 
> 
> btrfs fi df /
> 
>     Data, single: total=475.43GiB, used=299.28GiB
>     System, single: total=4.00MiB, used=80.00KiB
>     Metadata, single: total=1.01GiB, used=437.05MiB
>     GlobalReserve, single: total=61.03MiB, used=0.00B
> 
> 
> btrfs fi df /mnt
> 
>     Data, single: total=256.01GiB, used=221.18GiB
>     System, single: total=4.00MiB, used=48.00KiB
>     Metadata, single: total=2.01GiB, used=749.92MiB
>     GlobalReserve, single: total=297.11MiB, used=0.00B
> 
> dmesg.log: https://gitlab.com/-/snippets/2007155

   The / filesystem is a clear case of needing a data balance. See
this link for what to do:

https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29

and see also this link for how to read the data you've pasted here:

https://btrfs.wiki.kernel.org/index.php/FAQ#or_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

> What's interesting to me from above, the partition size on /home/azymohliad.home is 402.72GiB, but the file system size is 256.01GiB, and the image file size is 256.64GiB (from btrfs fi du /home, although ls -lh reports 403GiB). I'm not really sure, but iirc the fs and image sizes were around 403GiB too earlier. Could it be that it somehow got automatically reduced?
> 
> Could I do anything to make that fallocate call (with -l 403G) working? It will allow me to authenticate to homectl and resize the home partition from there.
> 
> If not, what is the safe way to shrink that LUKS-partition size? Maybe then systemd-homed would do fallocate for less space and it would work.
> 
> If from my assumptions you could tell that I'm looking in the wrong direction, please give me a hint. Thanks for taking time to read it!

   The /mnt filesystem (LABEL=azymohliad) looks OK. Are you seeing
problems with that one at all?

   Hugo.

-- 
Hugo Mills             | Sometimes, when I'm alone, I Google myself.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrei Borzenkov]

23.08.2020 18:37, Hugo Mills пишет:
>    The /mnt filesystem (LABEL=azymohliad) looks OK. Are you seeing
> problems with that one at all?
> 


It's not about LUKS container. systemd tool attempts to reserve full
size of image with LUKS inside using fallocate which fails. It never
gets as far as actually unlocking it or mounting filesystem inside.

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

On Sunday, August 23, 2020 7:21 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> It's not about LUKS container. systemd tool attempts to reserve full
> size of image with LUKS inside using fallocate which fails. It never
> gets as far as actually unlocking it or mounting filesystem inside.

Yes, thanks Andrei for additional clarification. Here I unlocked and mounted home partition manually just to show what's the state of filesystem on it (I guess that's not important here). Usually, it should be unlocked and mounted by systemd, but it fails because of fallocate failure.


On Sunday, August 23, 2020 6:37 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> The / filesystem is a clear case of needing a data balance. See
> this link for what to do:
>
> https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29
>
> and see also this link for how to read the data you've pasted here:
>
> https://btrfs.wiki.kernel.org/index.php/FAQ#or_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21


Thanks for a very quick response and links, Hugo, those are definitely useful for me. But the issue here seems more complicated. I forgot to write about it, but the state of root filesystem differs before and after fallocate attempt (or trying to log in as systemd-homed user).

Before:

    # btrfs fi show

    Label: none  uuid: b68411ce-702a-4259-9121-ac21c9119ddf
    	Total devices 1 FS bytes used 299.71GiB
    	devid    1 size 476.44GiB used 301.03GiB path /dev/nvme0n1p2

    # btrfs fi df /

    Data, single: total=300.00GiB, used=299.29GiB
    System, single: total=32.00MiB, used=48.00KiB
    Metadata, single: total=1.00GiB, used=438.38MiB
    GlobalReserve, single: total=62.16MiB, used=0.00B

After:

    # btrfs fi show

    Label: none  uuid: b68411ce-702a-4259-9121-ac21c9119ddf
    	Total devices 1 FS bytes used 299.71GiB
    	devid    1 size 476.44GiB used 476.44GiB path /dev/nvme0n1p2

    btrfs fi df /

    Data, single: total=475.41GiB, used=299.29GiB
    System, single: total=32.00MiB, used=80.00KiB
    Metadata, single: total=1.00GiB, used=438.39MiB
    GlobalReserve, single: total=62.16MiB, used=0.00B

If I do "btrfs balance start /" it becomes like before fallocate attempt, but it still doesn't fix the issue for me. fallocate and hence "homectl authenticate" would still fail.

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrei Borzenkov]

23.08.2020 19:55, Andrii Zymohliad пишет:
> On Sunday, August 23, 2020 7:21 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>> It's not about LUKS container. systemd tool attempts to reserve full
>> size of image with LUKS inside using fallocate which fails. It never
>> gets as far as actually unlocking it or mounting filesystem inside.
> 
> Yes, thanks Andrei for additional clarification. Here I unlocked and mounted home partition manually just to show what's the state of filesystem on it (I guess that's not important here). Usually, it should be unlocked and mounted by systemd, but it fails because of fallocate failure.
> 
> 
> On Sunday, August 23, 2020 6:37 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>> The / filesystem is a clear case of needing a data balance. See
>> this link for what to do:
>>
>> https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29
>>
>> and see also this link for how to read the data you've pasted here:
>>
>> https://btrfs.wiki.kernel.org/index.php/FAQ#or_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21
> 
> 
> Thanks for a very quick response and links, Hugo, those are definitely useful for me. But the issue here seems more complicated. I forgot to write about it, but the state of root filesystem differs before and after fallocate attempt (or trying to log in as systemd-homed user).
> 
> Before:
> 
>     # btrfs fi show
> 
>     Label: none  uuid: b68411ce-702a-4259-9121-ac21c9119ddf
>     	Total devices 1 FS bytes used 299.71GiB
>     	devid    1 size 476.44GiB used 301.03GiB path /dev/nvme0n1p2
> 
>     # btrfs fi df /
> 
>     Data, single: total=300.00GiB, used=299.29GiB
>     System, single: total=32.00MiB, used=48.00KiB
>     Metadata, single: total=1.00GiB, used=438.38MiB
>     GlobalReserve, single: total=62.16MiB, used=0.00B
> 
> After:
> 
>     # btrfs fi show
> 
>     Label: none  uuid: b68411ce-702a-4259-9121-ac21c9119ddf
>     	Total devices 1 FS bytes used 299.71GiB
>     	devid    1 size 476.44GiB used 476.44GiB path /dev/nvme0n1p2
> 
>     btrfs fi df /
> 
>     Data, single: total=475.41GiB, used=299.29GiB
>     System, single: total=32.00MiB, used=80.00KiB
>     Metadata, single: total=1.00GiB, used=438.39MiB
>     GlobalReserve, single: total=62.16MiB, used=0.00B
> 
> If I do "btrfs balance start /" it becomes like before fallocate attempt, but it still doesn't fix the issue for me. fallocate and hence "homectl authenticate" would still fail.
> 


Output of "btrfs fi us /" before and after fallocate would probably give
more information. Also "btrfs sub li /" and "btrfs qgroup show -re /"
would be interesting. And "btrfs fi du /home/azymohliad.home".

Rough math so far - filesystem has 175GiB free space. Filesystem inside
container uses 258GiB so assuming container consumes exactly the same
space, in the worst case it needs 145GiB on containing filesystem for
full size. There should be enough free (even unallocated) space. I
thought there could be some snapshots that force btrfs to attempt to
allocate full 403GiB, bit in my testing even after creating read-only
snapshots fallocate does not attempt to duplicate space.

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> Output of "btrfs fi us /" before and after fallocate would probably give
> more information. Also "btrfs sub li /" and "btrfs qgroup show -re /"
> would be interesting. And "btrfs fi du /home/azymohliad.home".

Collected the outputs of all those commands here (in order of execution): https://gitlab.com/-/snippets/2007189

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Sun, Aug 23, 2020 at 9:37 AM Hugo Mills <hugo@carfax.org.uk> wrote:
>
> On Sun, Aug 23, 2020 at 03:31:47PM +0000, Andrii Zymohliad wrote:

> > btrfs fi df /
> >
> >     Data, single: total=475.43GiB, used=299.28GiB
> >     System, single: total=4.00MiB, used=80.00KiB
> >     Metadata, single: total=1.01GiB, used=437.05MiB
> >     GlobalReserve, single: total=61.03MiB, used=0.00B
> >
> >
>    The / filesystem is a clear case of needing a data balance. See
> this link for what to do:

If it really needs to be balanced, it's a bug. It's 2020, this is a
5.8.3 kernel, we can't keep telling people they should baby sit Btrfs
in such cases. And also it's not clear it needs a data balance.


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Sun, Aug 23, 2020 at 9:31 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:

> My root partition is ~475GiB with BTRFS, my home partition is a ~400GiB LUKS-encrypted partition on a loopback file (also BTRFS) created by systemd-homed (residing at /home/azymohliad.home). Which leaves ~75GiB for the rest of the root FS.
>
> Recently I've lost the ability to log in to that user account, because during authentication systemd does fallocate call for the image. CLI alternative for my case (suggested on systemd mailing list, I don't really know what is it) is:
>
>     fallocate -l 403G -n /home/azymohliad.home
>
> Which fails:
>
>     fallocate: fallocate failed: No space left on device

I don't understand why homectl is fallocating just to activate. I'd
expect homectl only uses fallocate for create and resize. Can you
confirm the exact homectl command you're using?

There is a homectl option that might help, when activating, --luks-discard=true

A consequence of this option, however, is it will make the backing
file sparse. i.e. it'll punch holes in it, freeing up space on the
underlying file system. You should read the warning for the
--allow-discards option in 'man cryptsetup' which is what 'homectl
--luks-discard' is using.

Also, this option will fairly quickly fstrim the user home upon
activation (last time I tested it on systemd-245, some things have
changed in 246). And that will erase all evidence about why you're
having this problem in the first place.

Preferably if you could post the results from:

ls -ls /home/azymohliad.home

Grab this debug file from upstream btrfs-progs
https://github.com/kdave/btrfs-progs/blob/master/btrfs-debugfs

sudo ./btrfs-debugfs -b /

Mount or remount the file system with mount option enospc_debug, and
then reproduce the problem, i.e. exact same homectl command you're
using that's failing with out of space error.


> I can mount my home manually like this:
>
>     losetup -fP /home/azymohliad.home
>     cryptsetup open /dev/loop0p1
>     mount /dev/mapper/home /mnt

Yeah, exactly. Therefore I'll argue this is a systemd-homed bug. If
fallocate fails, systemd-homed can't just lock users out of their user
home. It needs to have a fallback where it doesn't do the fallocate,
and warns the user about whatever dependent operation also can't
happen as a result. But the user home should activate.

> What's interesting to me from above, the partition size on /home/azymohliad.home is 402.72GiB, but the file system size is 256.01GiB, and the image file size is 256.64GiB (from btrfs fi du /home, although ls -lh reports 403GiB). I'm not really sure, but iirc the fs and image sizes were around 403GiB too earlier. Could it be that it somehow got automatically reduced?

I'm confused. But I'm gonna set this aside for now until you post back
with the other information.

> Could I do anything to make that fallocate call (with -l 403G) working? It will allow me to authenticate to homectl and resize the home partition from there.

At the expense of sounding like a smart ass, yes you can make the root
file system bigger (by adding another device) and then presumably the
fallocate will work. But the fallocate seems wrong to me - but I also
don't know what command you're using and why fallocate is even being
used.


>
> If not, what is the safe way to shrink that LUKS-partition size? Maybe then systemd-homed would do fallocate for less space and it would work.
>
> If from my assumptions you could tell that I'm looking in the wrong direction, please give me a hint. Thanks for taking time to read it!

For sure my top advice, is since you can manually mount this sd-homed
home, *freshen your backups now* while you still can. Later versions
of sd-homed create a subvolume on that LUKS Btrfs file system, so you
can snapshot it and use btrfs-send if you want or however else you're
doing backups.


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Sun, Aug 23, 2020 at 12:49 PM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> > Output of "btrfs fi us /" before and after fallocate would probably give
> > more information. Also "btrfs sub li /" and "btrfs qgroup show -re /"
> > would be interesting. And "btrfs fi du /home/azymohliad.home".
>
> Collected the outputs of all those commands here (in order of execution): https://gitlab.com/-/snippets/2007189
>
[pasted from above snippets url]
>># btrfs fi du /home/azymohliad.home
>>
>>    Total   Exclusive  Set shared  Filename
>>    256.64GiB   234.46GiB    22.18GiB  /home/azymohliad.home

When you manually mount this luks file, what do you get for 'btrfs fi
us /home/azymohliad' ?

Because if that file system is 403G, and sd-homed is trying to
fallocate a 256G file to be 403G, that's +147. And also in your
snippets from '/' btfs fi us

>>Free (estimated): 176.17GiB (min: 176.17GiB)

The fallocate should work. Has this home file been snapshot or reflink
copied or deduped?

filefrag -v /home/azymohliad.home | grep shared

Here's the thing. /home/azymohliad.home, the file, should be the same
size as the Btrfs on that file. Unless it's been subject to discards,
and is thus sparse already for some reason.

Is it possible the file is being trimmed then fallocated then trimmed
then fallocated? This could be new behavior in sd-246.

-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

Hello Chris! Thanks a lot for so many ideas, I updated this snippet with most of the commands you asked the output for: https://gitlab.com/-/snippets/2007189, more details inline.


> I don't understand why homectl is fallocating just to activate. I'd
> expect homectl only uses fallocate for create and resize. Can you
> confirm the exact homectl command you're using?

Any homectl command that requires authentication. For example:

    homectl authenticate azymohliad

After typing password, the output is the following:

    Operation on home azymohliad failed: Not enough disk space for home azymohliad

Here is the part about fallocate on Systemd mailing list: https://lists.freedesktop.org/archives/systemd-devel/2020-August/045074.html (from the same issue discussion). Andrei Borzenkov might give more insight about it (if you are reading this and when you have time).


> Preferably if you could post the results from:
>
> ls -ls /home/azymohliad.home

    269111212 -rw------- 1 root root 432439787520 сер 23 18:20 /home/azymohliad.home

Also, added to the gitlab snippet.


> When you manually mount this luks file, what do you get for 'btrfs fi
> us /home/azymohliad' ?

Added to the gitlab snippet.


> Because if that file system is 403G, and sd-homed is trying to
> fallocate a 256G file to be 403G, that's +147. And also in your
> snippets from '/' btfs fi us
>
> > > Free (estimated): 176.17GiB (min: 176.17GiB)
>
> The fallocate should work. Has this home file been snapshot or reflink
> copied or deduped?

I haven't made any explicitly, but I'm not very familiar with those concepts, maybe sd-homed did it implicitly.


> filefrag -v /home/azymohliad.home | grep shared

This one is 18k lines, uploaded here: https://gitlab.com/-/snippets/2007301


> Here's the thing. /home/azymohliad.home, the file, should be the same
> size as the Btrfs on that file. Unless it's been subject to discards,
> and is thus sparse already for some reason.
>
> Is it possible the file is being trimmed then fallocated then trimmed
> then fallocated? This could be new behavior in sd-246.

Idk, I guess it's better to go back to Systemd mailing list for that, but just in case I've added "homectl inspect azymohliad" output to the gitlab snippet. Here's what it says about discard:

    LUKS Discard: online=no offline=yes


> Grab this debug file from upstream btrfs-progs
> https://github.com/kdave/btrfs-progs/blob/master/btrfs-debugfs
>
> sudo ./btrfs-debugfs -b /
>
> Mount or remount the file system with mount option enospc_debug, and
> then reproduce the problem, i.e. exact same homectl command you're
> using that's failing with out of space error.

I didn't quite get what information I should extract from this.

Here is the btrfs-debugfs output: https://gitlab.com/-/snippets/2007302

I remounted root filesystem as with enospc_debug, reproduced the issue with "homectl authenticate azymohliad", but it didn't print any additional info, nor any additional logs in journalctl.


> For sure my top advice, is since you can manually mount this sd-homed
> home, freshen your backups now while you still can. Later versions
> of sd-homed create a subvolume on that LUKS Btrfs file system, so you
> can snapshot it and use btrfs-send if you want or however else you're
> doing backups.

Thanks! I'm unfamiliar with snapshots and btrfs-send yet, but it's the perfect time to learn.


> There is a homectl option that might help, when activating, --luks-discard=true
>
> A consequence of this option, however, is it will make the backing
> file sparse. i.e. it'll punch holes in it, freeing up space on the
> underlying file system. You should read the warning for the
> --allow-discards option in 'man cryptsetup' which is what 'homectl
> --luks-discard' is using.
>
> Also, this option will fairly quickly fstrim the user home upon
> activation (last time I tested it on systemd-245, some things have
> changed in 246). And that will erase all evidence about why you're
> having this problem in the first place.

Gonna read about this too, thanks!

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Mon, Aug 24, 2020 at 12:12 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> Hello Chris! Thanks a lot for so many ideas, I updated this snippet with most of the commands you asked the output for: https://gitlab.com/-/snippets/2007189, more details inline.

>> # ls -lhs /home/azymohliad.home
>>    257G -rw------- 1 root root 403G сер 23 18:20 /home/azymohliad.home

OK so it's a sparse file..


> This one is 18k lines, uploaded here: https://gitlab.com/-/snippets/2007301

There are many shared extents, so it has been subject to snapshot,
reflink copy, or dedup.

To my knowledge, sd-homed doesn't do any of these things. Something
else did it. Are you using snapper or timeshift? Have you used 'cp' to
duplicate the home backing file?

What do you get for

lsattr /home/azymohliad.home

> Idk, I guess it's better to go back to Systemd mailing list for that, but just in case I've added "homectl inspect azymohliad" output to the gitlab snippet. Here's what it says about discard:
>
>     LUKS Discard: online=no offline=yes

I'm not sure what it means, but I'll ask on systemd list.

> Here is the btrfs-debugfs output: https://gitlab.com/-/snippets/2007302

OK too late, it's already been balanced.

I think we've got a pretty good idea what's going on. The current work
around in your case will be to use the --luks-discard=true option when
activating. This will avoid the fallocate step. But the question
remains why the fallocate fails. I suspect it has something to do with
shared extents. Somewhere you have one or more snapshots containing
this home file. And that's causing some confusion.


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> Are you using snapper or timeshift?

No


> Have you used 'cp' to duplicate the home backing file?

The /home/azymohliad.home file itself - no


> What do you get for
>
> lsattr /home/azymohliad.home

    ---------------C---- /home/azymohliad.home


> I think we've got a pretty good idea what's going on. The current work
> around in your case will be to use the --luks-discard=true option when
> activating.

Thanks! I'm gonna finish reading this http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html
to have a better idea what discard implies, and then will try it out.

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> I think we've got a pretty good idea what's going on. The current work
> around in your case will be to use the --luks-discard=true option when
> activating.

Hm.. It doesn't help.

    # homectl activate --luks-discard=true azymohliad

or

    # homectl authenticate --luks-discard=true azymohliad

prints exactly the same error as without --luks-discard option:

    Operation on home azymohliad failed: Not enough disk space for home azymohliad

and the allocated space on / goes to 476G again (so it seems it still tries to do fallocate).

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

Sorry, my bad.

    # homectl update --luks-discard=on azymohliad

Does the job. I can log in again now. Thank you very very much!!!

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrei Borzenkov]

24.08.2020 10:13, Chris Murphy пишет:
> 
> I think we've got a pretty good idea what's going on. The current work
> around in your case will be to use the --luks-discard=true option when
> activating. This will avoid the fallocate step. But the question
> remains why the fallocate fails. I suspect it has something to do with
> shared extents.


Actually I see the same behavior without any shared extent at all.
fallocate attempts to allocate full file size, not just additional
unallocated space:

tw:/mnt # truncate -s $[112*5]M src/file
tw:/mnt # dd if=/dev/urandom of=src/file conv=notrunc,block,sync bs=112M
count=20+2 records in
2+0 records out
234881024 bytes (235 MB, 224 MiB) copied, 1.53186 s, 153 MB/s
tw:/mnt # sync
tw:/mnt # btrfs fi us /mnt
Overall:
    Device size:		   1.00GiB
    Device allocated:		 356.00MiB
    Device unallocated:		 668.00MiB
    Device missing:		     0.00B
    Used:			 224.59MiB
    Free (estimated):		 787.81MiB	(min: 787.81MiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		   3.25MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:344.00MiB, Used:224.19MiB (65.17%)
   /dev/vdc	 344.00MiB

Metadata,single: Size:8.00MiB, Used:400.00KiB (4.88%)
   /dev/vdc	   8.00MiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/vdc	   4.00MiB

Unallocated:
   /dev/vdc	 668.00MiB
tw:/mnt # fallocate -l $[112*5]M -n src/file
tw:/mnt # sync
tw:/mnt # btrfs fi us /mnt
Overall:
    Device size:		   1.00GiB
    Device allocated:		 916.00MiB
    Device unallocated:		 108.00MiB
    Device missing:		     0.00B
    Used:			 560.78MiB
    Free (estimated):		 451.62MiB	(min: 451.62MiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		   3.25MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:904.00MiB, Used:560.38MiB (61.99%)
   /dev/vdc	 904.00MiB

Metadata,single: Size:8.00MiB, Used:400.00KiB (4.88%)
   /dev/vdc	   8.00MiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/vdc	   4.00MiB

Unallocated:
   /dev/vdc	 108.00MiB
tw:/mnt # btrfs fi du src/file
     Total   Exclusive  Set shared  Filename
 560.00MiB   560.00MiB       0.00B  src/file
tw:/mnt # uname -a
Linux tw.0.2.15 5.8.0-1-default #1 SMP Tue Aug 4 07:30:59 UTC 2020
(9bc0044) x86_64 x86_64 x86_64 GNU/Linux
tw:/mnt #

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Goffredo Baroncelli]

On 8/24/20 5:23 PM, Andrei Borzenkov wrote:
> 24.08.2020 10:13, Chris Murphy пишет:
>>
>> I think we've got a pretty good idea what's going on. The current work
>> around in your case will be to use the --luks-discard=true option when
>> activating. This will avoid the fallocate step. But the question
>> remains why the fallocate fails. I suspect it has something to do with
>> shared extents.
> 
> 
> Actually I see the same behavior without any shared extent at all.
> fallocate attempts to allocate full file size, not just additional
> unallocated space:

I don't remember if at the time it was reached a conclusion, but this topic was already discussed in the past:

https://lore.kernel.org/linux-btrfs/798a9077-bcbd-076c-a458-3403010ce8ac@libero.it/

I think that this information should be forwarded back to the systemd mailing list...

Anyway IMHO a failing fallocate for a small file is a reasonable failure.
However I am not convinced that this is still true for a filesystem image (like the systemd-homed
case is). In this case it is reasonable to thinking the image as sparse image as a way of
over-provisioning.
However it is true that not all the stack (btrfs <-> loopback <-> luks <-> btrfs) is able to manage
a case where there is no space, so fallocat'ing may be the least worst scenario...

> 
> tw:/mnt # truncate -s $[112*5]M src/file
> tw:/mnt # dd if=/dev/urandom of=src/file conv=notrunc,block,sync bs=112M
> count=20+2 records in
> 2+0 records out
> 234881024 bytes (235 MB, 224 MiB) copied, 1.53186 s, 153 MB/s
> tw:/mnt # sync
> tw:/mnt # btrfs fi us /mnt
> Overall:
>      Device size:		   1.00GiB
>      Device allocated:		 356.00MiB
>      Device unallocated:		 668.00MiB
>      Device missing:		     0.00B
>      Used:			 224.59MiB
>      Free (estimated):		 787.81MiB	(min: 787.81MiB)
>      Data ratio:			      1.00
>      Metadata ratio:		      1.00
>      Global reserve:		   3.25MiB	(used: 0.00B)
>      Multiple profiles:		        no
> 
> Data,single: Size:344.00MiB, Used:224.19MiB (65.17%)
>     /dev/vdc	 344.00MiB
> 
> Metadata,single: Size:8.00MiB, Used:400.00KiB (4.88%)
>     /dev/vdc	   8.00MiB
> 
> System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
>     /dev/vdc	   4.00MiB
> 
> Unallocated:
>     /dev/vdc	 668.00MiB
> tw:/mnt # fallocate -l $[112*5]M -n src/file
> tw:/mnt # sync
> tw:/mnt # btrfs fi us /mnt
> Overall:
>      Device size:		   1.00GiB
>      Device allocated:		 916.00MiB
>      Device unallocated:		 108.00MiB
>      Device missing:		     0.00B
>      Used:			 560.78MiB
>      Free (estimated):		 451.62MiB	(min: 451.62MiB)
>      Data ratio:			      1.00
>      Metadata ratio:		      1.00
>      Global reserve:		   3.25MiB	(used: 0.00B)
>      Multiple profiles:		        no
> 
> Data,single: Size:904.00MiB, Used:560.38MiB (61.99%)
>     /dev/vdc	 904.00MiB
> 
> Metadata,single: Size:8.00MiB, Used:400.00KiB (4.88%)
>     /dev/vdc	   8.00MiB
> 
> System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
>     /dev/vdc	   4.00MiB
> 
> Unallocated:
>     /dev/vdc	 108.00MiB
> tw:/mnt # btrfs fi du src/file
>       Total   Exclusive  Set shared  Filename
>   560.00MiB   560.00MiB       0.00B  src/file
> tw:/mnt # uname -a
> Linux tw.0.2.15 5.8.0-1-default #1 SMP Tue Aug 4 07:30:59 UTC 2020
> (9bc0044) x86_64 x86_64 x86_64 GNU/Linux
> tw:/mnt #
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Mon, Aug 24, 2020 at 2:33 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> Sorry, my bad.
>
>     # homectl update --luks-discard=on azymohliad
>
> Does the job. I can log in again now. Thank you very very much!!!

Seems like bool should take either yes, true, on, or 1. But I'm an odd
duck. Anyway! Glad it works.

Next question is to find out why you have shared extents.

filefrag -v /home/azymohliad.home | grep -m 10 shared

Any of those shared lines that doesn't also include unwritten (ok
maybe unwritten is fine but I only tested shared without unwritten),
use the value in the fourth column which is the physical_offset start
(filefrag calls it physical but on btrfs it's actually a logical
extent).

The value for my case is 1001165..  so I can do the math in the command:

# btrfs inspect-internal logical-resolve $[1001165*4096] /mnt
/mnt/libvimages/fedora.raw
/mnt/libvimages/fedora.raw.bak

That found the two files that share that extent. From that we can find
out why you have shared extents for your home backing file.

Note: I think this command wants to be pointed at the top-level of the
file system. It failed otherwise.


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> Seems like bool should take either yes, true, on, or 1. But I'm an odd
> duck. Anyway! Glad it works.

Ah, sorry, I didn't notice I changed true to on. It doesn't matter. The thing is, it should be "homectl update", and not during activation/authorization itself.

> Next question is to find out why you have shared extents.

    # filefrag -v /home/azymohliad.home | grep -m 10 shared

    55082: 52756487..52756516:  291198215.. 291198244:     30:  291722503: shared
    55083: 52756518..52756522:  291198246.. 291198250:      5:             shared
    55084: 52756524..52756528:  291198252.. 291198256:      5:             shared
    55085: 52756530..52756530:  291198258.. 291198258:      1:             shared
    55086: 52756532..52756534:  291198260.. 291198262:      3:             shared
    55087: 52756536..52756546:  291198264.. 291198274:     11:             shared
    55088: 52756548..52756553:  291198276.. 291198281:      6:             shared
    55089: 52756555..52756557:  291198283.. 291198285:      3:             shared
    55090: 52756559..52756578:  291198287.. 291198306:     20:             shared
    55091: 52756582..52756595:  291198310.. 291198323:     14:             shared

And for all of these extents the output of "btrfs inspect-internal" is:

    //home/azymohliad.home

I call it on root / filesystem, e.g.:

    # btrfs inspect-internal logical-resolve $[291198246 * 4096] /

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Tue, Aug 25, 2020 at 1:22 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> > Seems like bool should take either yes, true, on, or 1. But I'm an odd
> > duck. Anyway! Glad it works.
>
> Ah, sorry, I didn't notice I changed true to on. It doesn't matter. The thing is, it should be "homectl update", and not during activation/authorization itself.
>
> > Next question is to find out why you have shared extents.
>
>     # filefrag -v /home/azymohliad.home | grep -m 10 shared
>
>     55082: 52756487..52756516:  291198215.. 291198244:     30:  291722503: shared
>     55083: 52756518..52756522:  291198246.. 291198250:      5:             shared
>     55084: 52756524..52756528:  291198252.. 291198256:      5:             shared
>     55085: 52756530..52756530:  291198258.. 291198258:      1:             shared
>     55086: 52756532..52756534:  291198260.. 291198262:      3:             shared
>     55087: 52756536..52756546:  291198264.. 291198274:     11:             shared
>     55088: 52756548..52756553:  291198276.. 291198281:      6:             shared
>     55089: 52756555..52756557:  291198283.. 291198285:      3:             shared
>     55090: 52756559..52756578:  291198287.. 291198306:     20:             shared
>     55091: 52756582..52756595:  291198310.. 291198323:     14:             shared
>
> And for all of these extents the output of "btrfs inspect-internal" is:
>
>     //home/azymohliad.home
>
> I call it on root / filesystem, e.g.:
>
>     # btrfs inspect-internal logical-resolve $[291198246 * 4096] /

Yeah but is / the top-level?

btrfs sub list -t /
mount | grep btrfs



-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> Yeah but is / the top-level?
>
> btrfs sub list -t /

ID	gen	top level	path
--	---	---------	----
258	94017	5		var/lib/portables
259	94017	5		var/lib/machines

> mount | grep btrfs

/dev/nvme0n1p2 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/mapper/home-azymohliad on /home/azymohliad type btrfs (rw,nosuid,nodev,relatime,ssd,discard,noacl,space_cache,subvolid=5,subvol=/)

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Tue, Aug 25, 2020 at 2:47 PM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> > Yeah but is / the top-level?
> >
> > btrfs sub list -t /
>
> ID      gen     top level       path
> --      ---     ---------       ----
> 258     94017   5               var/lib/portables
> 259     94017   5               var/lib/machines
>
> > mount | grep btrfs
>
> /dev/nvme0n1p2 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
> /dev/mapper/home-azymohliad on /home/azymohliad type btrfs (rw,nosuid,nodev,relatime,ssd,discard,noacl,space_cache,subvolid=5,subvol=/)
>

It might be that for each extent there's a mix of shared and unshared
extents so maybe you just got unlucky and hit a block that isn't
shared in that extent.

Try more of those shared addresses. If you can do python at all,
there's python-btrfs
https://github.com/knorrie/python-btrfs

He gave this example, which I don't follow because I don't do python
at all, but the vaddr is the filefrag -v address from before * 4096.
You could just iterate by incrementing by 4096 and stop as soon as you
get two files (or two inodes). And then us find -inum to track down
the inode numbers to files.


https://github.com/knorrie/python-btrfs



-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> Try more of those shared addresses. If you can do python at all,
> there's python-btrfs
> https://github.com/knorrie/python-btrfs
>
> He gave this example, which I don't follow because I don't do python
> at all, but the vaddr is the filefrag -v address from before * 4096.
> You could just iterate by incrementing by 4096 and stop as soon as you
> get two files (or two inodes). And then us find -inum to track down
> the inode numbers to files.
>
> https://github.com/knorrie/python-btrfs

Which example exactly? There are many in the repo.

I'm ok with python, but to save time learning the API of the library I've made a simple bash script:

    mkdir -p shared_extents
    addrs=$(filefrag -v /home/azymohliad.home | grep shared | awk '{print $3}' | tr -d '.')
    for addr in $addrs; do
        btrfs inspect-internal logical-resolve $[$addr * 4096] / > shared_extents/$addr.txt
        if [ $(wc -l shared_extents/$addr.txt | awk '{print $1}') -gt 1 ]; then
            cat shared_extents/$addr.txt
        fi
    done

It should print the outputs of "btrfs inspect-internal logical-resolve $[$addr * 4096]" if it's more than 1 line. And it prints nothing.
Every "btrfs inspect-internal ..." output here is just "//home/azymohliad.home" for each of these shared extents (I can double check it from "cat shared_extents/*").

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

Interestingly, I have much less shared extents now:

    # filefrag -v /home/azymohliad.home | grep shared | wc -l
    1679

It was over 18k the first time you asked me to check it (even before I enabled discard in homectl).

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Wed, Aug 26, 2020 at 7:40 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> Interestingly, I have much less shared extents now:
>
>     # filefrag -v /home/azymohliad.home | grep shared | wc -l
>     1679
>
> It was over 18k the first time you asked me to check it (even before I enabled discard in homectl).

OK so Josef found a bug. Filefrag is showing shared extents in certain
cases when there are none. So you can stop worrying there's some copy
of the file somewhere. And stop looking for it.
https://github.com/btrfs/btrfs-todo/issues/6

The fallocate problem is still real, and that might take some time for
Andrei or someone else to figure out why.


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Wed, Aug 26, 2020 at 1:09 PM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> Ok. Thanks a lot for your great help, Chris!

No problem.


> On Aug 26, 2020, 22:06, Chris Murphy < lists@colorremedies.com> wrote:
>
> The fallocate problem is still real, and that might take some time for
> Andrei or someone else to figure out why.


Actually, if you can reproduce the fallocate problem and strace it,
that would be helpful. For sure, homectl deactivate, for starters. And
then just 'strace fallocate' that file the same way you did
previously. I assume it still fails. Put that strace into a text file
or pastebin (month expiration should be enough) and link it.

And also paste into your reply:

grep -r . /sys/fs/btrfs/<uuidofbtrfs>/allocation/

You'll need to use the uuid for the btrfs that has this file on it.


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> Actually, if you can reproduce the fallocate problem and strace it,
> that would be helpful. For sure, homectl deactivate, for starters. And
> then just 'strace fallocate' that file the same way you did
> previously. I assume it still fails. Put that strace into a text file
> or pastebin (month expiration should be enough) and link it.

Yep, still fails. Here's the one with exactly the same arguments as before (-l 403G): https://gitlab.com/-/snippets/2008996

But I did "homectl resize azymohliad 300G" so now "ls -lhs" reports:

    228G -rw------- 1 root root 300G сер 26 22:50 /home/azymohliad.home

But even "fallocate -l 300G -n /home/azymohliad.home" fails the same way. Here's strace: https://gitlab.com/-/snippets/2008993

(I hope gitlab is ok too, you can add /raw to URLs to get raw txt file)


> And also paste into your reply:
>
> grep -r . /sys/fs/btrfs/<uuidofbtrfs>/allocation/

    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/disk_used:474251264
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_pinned:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_used:474251264
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/single/used_bytes:474251264
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/single/total_bytes:1073741824
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/total_bytes_pinned:163840
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/disk_total:1073741824
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/total_bytes:1073741824
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_reserved:163840
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_readonly:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_may_use:74252288
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/flags:4
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/disk_used:81920
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_pinned:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_used:81920
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/single/used_bytes:81920
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/single/total_bytes:33554432
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/total_bytes_pinned:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/disk_total:33554432
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/total_bytes:33554432
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_reserved:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_readonly:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_may_use:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/flags:2
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/global_rsv_reserved:68730880
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/disk_used:314346594304
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_pinned:0
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_used:314346594304
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/single/used_bytes:314346594304
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/single/total_bytes:510463901696
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/total_bytes_pinned:4096
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/disk_total:510463901696
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/total_bytes:510463901696
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_reserved:4096
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_readonly:131072
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_may_use:4096
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/flags:1
    /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/global_rsv_size:68730880

I also added this output to the collection here: https://gitlab.com/-/snippets/2007189
I'll keep all those gitlab snippets that I shared in this thread online for a while (some months at least).

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Wed, Aug 26, 2020 at 1:57 PM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> > Actually, if you can reproduce the fallocate problem and strace it,
> > that would be helpful. For sure, homectl deactivate, for starters. And
> > then just 'strace fallocate' that file the same way you did
> > previously. I assume it still fails. Put that strace into a text file
> > or pastebin (month expiration should be enough) and link it.
>
> Yep, still fails. Here's the one with exactly the same arguments as before (-l 403G): https://gitlab.com/-/snippets/2008996
>
> But I did "homectl resize azymohliad 300G" so now "ls -lhs" reports:
>
>     228G -rw------- 1 root root 300G сер 26 22:50 /home/azymohliad.home
>
> But even "fallocate -l 300G -n /home/azymohliad.home" fails the same way. Here's strace: https://gitlab.com/-/snippets/2008993
>
> (I hope gitlab is ok too, you can add /raw to URLs to get raw txt file)
>
>
> > And also paste into your reply:
> >
> > grep -r . /sys/fs/btrfs/<uuidofbtrfs>/allocation/
>
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/disk_used:474251264
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_pinned:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_used:474251264
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/single/used_bytes:474251264
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/single/total_bytes:1073741824
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/total_bytes_pinned:163840
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/disk_total:1073741824
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/total_bytes:1073741824
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_reserved:163840
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_readonly:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/bytes_may_use:74252288
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/metadata/flags:4
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/disk_used:81920
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_pinned:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_used:81920
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/single/used_bytes:81920
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/single/total_bytes:33554432
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/total_bytes_pinned:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/disk_total:33554432
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/total_bytes:33554432
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_reserved:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_readonly:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/bytes_may_use:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/system/flags:2
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/global_rsv_reserved:68730880
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/disk_used:314346594304
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_pinned:0
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_used:314346594304
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/single/used_bytes:314346594304
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/single/total_bytes:510463901696
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/total_bytes_pinned:4096
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/disk_total:510463901696
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/total_bytes:510463901696
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_reserved:4096
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_readonly:131072
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/bytes_may_use:4096
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/data/flags:1
>     /sys/fs/btrfs/b68411ce-702a-4259-9121-ac21c9119ddf/allocation/global_rsv_size:68730880
>
> I also added this output to the collection here: https://gitlab.com/-/snippets/2007189
> I'll keep all those gitlab snippets that I shared in this thread online for a while (some months at least).

OK is the sysfs output from before or after the homectl resize? And it
matches with the second strace fallocate -l 300g ?


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Wed, Aug 26, 2020 at 3:06 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> OK is the sysfs output from before or after the homectl resize? And it
> matches with the second strace fallocate -l 300g ?

Figured it out. The sysfs is the first strace before the resize.



-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Wed, Aug 26, 2020 at 3:08 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Wed, Aug 26, 2020 at 3:06 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > OK is the sysfs output from before or after the homectl resize? And it
> > matches with the second strace fallocate -l 300g ?
>
> Figured it out. The sysfs is the first strace before the resize.

Shoot. I confused myself and am not sure, so I need you to tell me
which strace that sysfs goes with, the 403g or 300g.


-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrei Borzenkov]

26.08.2020 22:06, Chris Murphy пишет:
> 
> The fallocate problem is still real, and that might take some time for
> Andrei or someone else to figure out why.
> 
> 

Sorry? It was already answered that btrfs by design fallocates full file
size no matter what.

https://lore.kernel.org/linux-btrfs/b0d0784a-03b5-c212-f4a1-f09ff487e355@libero.it/T/#meb706f2264e80029835b1d31d82c3a468b53dcda

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> > > OK is the sysfs output from before or after the homectl resize? And it
> > > matches with the second strace fallocate -l 300g ?
> >
> > Figured it out. The sysfs is the first strace before the resize.
>
> Shoot. I confused myself and am not sure, so I need you to tell me
> which strace that sysfs goes with, the 403g or 300g.

Sorry, I haven't put it clear. I've resized it right after I was able to log in a couple of days ago. So both straces and /sys/fs outputs are after resize. And /sys/fs output is after the second strace (-l 300G).

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Thu, Aug 27, 2020 at 12:48 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> > > > OK is the sysfs output from before or after the homectl resize? And it
> > > > matches with the second strace fallocate -l 300g ?
> > >
> > > Figured it out. The sysfs is the first strace before the resize.
> >
> > Shoot. I confused myself and am not sure, so I need you to tell me
> > which strace that sysfs goes with, the 403g or 300g.
>
> Sorry, I haven't put it clear. I've resized it right after I was able to log in a couple of days ago. So both straces and /sys/fs outputs are after resize. And /sys/fs output is after the second strace (-l 300G).

Andrii,

We're trying to reproduce this problem and need more information.
Could you provide the following:

1. ls -li  /home/azymohliad.home
2. filefrag -v /home/azymohliad.home | grep -m 10 shared
3. Pick any line, multiply the first extent (4th column) by 4096, this
is the bytenr
4. btrfs inspect-internal dump-tree -t 5 /dev/nvme0n1p2 | grep -C 10 <bytenr>

Thanks!

-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

Hi, sure!

> 1.  ls -li /home/azymohliad.home

43141 -rw------- 1 root root 322122547200 вер  1 19:57 /home/azymohliad.home

> 2.  filefrag -v /home/azymohliad.home | grep -m 10 shared

87954: 52756487..52756516:  291198215.. 291198244:     30:  291722503: shared
87955: 52756518..52756522:  291198246.. 291198250:      5:             shared
87956: 52756524..52756528:  291198252.. 291198256:      5:             shared
87957: 52756530..52756530:  291198258.. 291198258:      1:             shared
87958: 52756532..52756534:  291198260.. 291198262:      3:             shared
87959: 52756536..52756546:  291198264.. 291198274:     11:             shared
87960: 52756548..52756553:  291198276.. 291198281:      6:             shared
87961: 52756555..52756557:  291198283.. 291198285:      3:             shared
87962: 52756559..52756578:  291198287.. 291198306:     20:             shared
87963: 52756582..52756595:  291198310.. 291198323:     14:             shared


> 3.  Pick any line, multiply the first extent (4th column) by 4096, this
>     is the bytenr
>
> 4.  btrfs inspect-internal dump-tree -t 5 /dev/nvme0n1p2 | grep -C 10 <bytenr>

No output for any of the extents above. For example:

291198215 * 4096 = 1192747888640

Then the following produces no output:

    sudo btrfs inspect-internal dump-tree -t 5 /dev/nvme0n1p2 | grep -C 10 1192747888640

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Tue, Sep 1, 2020 at 11:06 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> Hi, sure!
>
> > 1.  ls -li /home/azymohliad.home
>
> 43141 -rw------- 1 root root 322122547200 вер  1 19:57 /home/azymohliad.home
>
> > 2.  filefrag -v /home/azymohliad.home | grep -m 10 shared
>
> 87954: 52756487..52756516:  291198215.. 291198244:     30:  291722503: shared
> 87955: 52756518..52756522:  291198246.. 291198250:      5:             shared
> 87956: 52756524..52756528:  291198252.. 291198256:      5:             shared
> 87957: 52756530..52756530:  291198258.. 291198258:      1:             shared
> 87958: 52756532..52756534:  291198260.. 291198262:      3:             shared
> 87959: 52756536..52756546:  291198264.. 291198274:     11:             shared
> 87960: 52756548..52756553:  291198276.. 291198281:      6:             shared
> 87961: 52756555..52756557:  291198283.. 291198285:      3:             shared
> 87962: 52756559..52756578:  291198287.. 291198306:     20:             shared
> 87963: 52756582..52756595:  291198310.. 291198323:     14:             shared
>
>
> > 3.  Pick any line, multiply the first extent (4th column) by 4096, this
> >     is the bytenr
> >
> > 4.  btrfs inspect-internal dump-tree -t 5 /dev/nvme0n1p2 | grep -C 10 <bytenr>
>
> No output for any of the extents above. For example:
>
> 291198215 * 4096 = 1192747888640
>
> Then the following produces no output:
>
>     sudo btrfs inspect-internal dump-tree -t 5 /dev/nvme0n1p2 | grep -C 10 1192747888640


OK it will take longer to search but omit '-t 5' and we'll just search
the whole tree dump.



-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

> OK it will take longer to search but omit '-t 5' and we'll just search
> the whole tree dump.

This one is also empty:

    sudo btrfs inspect-internal dump-tree /dev/nvme0n1p2 | grep -C 10 1192747888640

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Andrii Zymohliad]

I wonder how can "1192747888640" be a byte number if it's more than 1TiB, and I have only 476GiB partition. Is the 4096 multiplier surely correct for my filesystem?

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Tue, Sep 1, 2020 at 11:31 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> I wonder how can "1192747888640" be a byte number if it's more than 1TiB, and I have only 476GiB partition. Is the 4096 multiplier surely correct for my filesystem?
>

Yes, it's a Btrfs logical address so it can be a value larger than the
physical size of the file system.

Try command 2 again and see if that extent and block are still there.
It should work.

-- 
Chris Murphy

Re: [Help] Can't login to my systemd-homed user account due to fallocate failure

[Chris Murphy]

On Tue, Sep 1, 2020 at 11:31 AM Andrii Zymohliad
<azymohliad@protonmail.com> wrote:
>
> I wonder how can "1192747888640" be a byte number if it's more than 1TiB, and I have only 476GiB partition. Is the 4096 multiplier surely correct for my filesystem?
>

On second thought, it's probably easier if you just make an image
(metadata only). It'll need to be unmounted.

btrfs-image -ss -c4 -t4 /dev/nvme /path/to/file

-ss will scrub the file names. From prior output about this fs, it
looks like the resulting file will be around 100M. You can email me
the URL to the file offline.

Thanks,

-- 
Chris Murphy