Merging multiple erofs file systems on the same block device

Merging multiple erofs file systems on the same block device

[Daan De Meyer]

Hi,

I've been looking into erofs as an initramfs replacement by using
root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
However, by using a ramdisk instead of the usual compressed cpio, I
would lose the feature where the kernel merges multiple individual
cpios together into a single tmpfs filesystem. Looking at the
documentation for erofs, I noticed that erofs already seems to support
merging multiple erofs filesystems on separate block devices using the
device= cmdline option. Would it be possible to extend this so that
multiple erofs filesystems that follow each other on the same block
device can also be merged? This would allow me to pass multiple erofs
filesystems to the kernel via initrd=, which would get concatenated
together into a single buffer, which the kernel would write to a
ramdisk (using root=/dev/ram0) which the kernel would then have erofs
mount to /dev/root. erofs would notice that there's multiple erofs
filesystems on the ramdisk and overlay them together (perhaps only if
a cmdline option is enabled).

Does this make sense at all?

Cheers,

Daan De Meyer

Re: Merging multiple erofs file systems on the same block device

[Gao Xiang]

Hi,

On 2023/5/1 22:09, Daan De Meyer wrote:
> Hi,
> 
> I've been looking into erofs as an initramfs replacement by using
> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.

Sorry, I'm on vacation now.

May I ask what's your detailed use cases?  Sure, you could use
/dev/ram0 as a replacement, but currently it still takes double
memory compared with initramfs since ramdisk doesn't support FSDAX
for now (by enabling FSDAX, it won't take double memory at all.)

Actually I think ramdisk FSDAX is useful and I might sync up this on
the following LSF/MM/BPF 2023.

> However, by using a ramdisk instead of the usual compressed cpio, I
> would lose the feature where the kernel merges multiple individual
> cpios together into a single tmpfs filesystem. Looking at the
> documentation for erofs, I noticed that erofs already seems to support
> merging multiple erofs filesystems on separate block devices using the
> device= cmdline option. Would it be possible to extend this so that
Here `device=` is actually used to refer to seperate blobs with the
merged metadata.  For example, you could have

   device=/dev/ram1 original tar1
   device=/dev/ram2 original tar2
   /dev/ram0        merged metadata for tar1 + tar2.

which means, if you'd like to merge multiple EROFS filesystems, you
might need another step to build a merged metadata in advance in order
to merges multiple individual tarballs together, which could be built
when applying images or booting (by using a special bootloader with
such functionality.)

EROFS doesn't support stacking multiple fses runtimely since it seems
a duplicated feature of overlayfs (you could consider using overlayfs
honestly.)

> multiple erofs filesystems that follow each other on the same block
> device can also be merged? This would allow me to pass multiple erofs
> filesystems to the kernel via initrd=, which would get concatenated
> together into a single buffer, which the kernel would write to a
> ramdisk (using root=/dev/ram0) which the kernel would then have erofs
> mount to /dev/root. erofs would notice that there's multiple erofs
> filesystems on the ramdisk and overlay them together (perhaps only if
> a cmdline option is enabled).

Recently, A merged EROFS filesystem with a single block device can
be passed but you might still need a step to build a merged fs tree
before mounting.  Such feature is used to build a metadata and
reuse original tar blobs.

> 
> Does this make sense at all?

Hopefully it helps.

Thanks,
Gao Xiang

> 
> Cheers,
> 
> Daan De Meyer

Re: Merging multiple erofs file systems on the same block device

[Daan De Meyer]

> On 2023/5/1 22:09, Daan De Meyer wrote:
> > Hi,
> >
> > I've been looking into erofs as an initramfs replacement by using
> > root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
>
> Sorry, I'm on vacation now.
>
> May I ask what's your detailed use cases?  Sure, you could use
> /dev/ram0 as a replacement, but currently it still takes double
> memory compared with initramfs since ramdisk doesn't support FSDAX
> for now (by enabling FSDAX, it won't take double memory at all.)

I'm experimenting with larger initramfses and running into memory
bottlenecks since the entire compressed cpio has to be decompressed
into memory. I was hoping to use erofs as a replacement that could stay
compressed, where only the files that are actually accessed are
decompressed at runtime.

> Actually I think ramdisk FSDAX is useful and I might sync up this on
> the following LSF/MM/BPF 2023.
>
> > However, by using a ramdisk instead of the usual compressed cpio, I
> > would lose the feature where the kernel merges multiple individual
> > cpios together into a single tmpfs filesystem. Looking at the
> > documentation for erofs, I noticed that erofs already seems to support
> > merging multiple erofs filesystems on separate block devices using the
> > device= cmdline option. Would it be possible to extend this so that
> Here `device=` is actually used to refer to seperate blobs with the
> merged metadata.  For example, you could have
>
>    device=/dev/ram1 original tar1
>    device=/dev/ram2 original tar2
>    /dev/ram0        merged metadata for tar1 + tar2.
>
> which means, if you'd like to merge multiple EROFS filesystems, you
> might need another step to build a merged metadata in advance in order
> to merges multiple individual tarballs together, which could be built
> when applying images or booting (by using a special bootloader with
> such functionality.)

Ahh, I misunderstood the device= option then.

> EROFS doesn't support stacking multiple fses runtimely since it seems
> a duplicated feature of overlayfs (you could consider using overlayfs
> honestly.)

I would love to use overlayfs, but there's no way to specify to the kernel that
the initrd should be set up as an overlayfs of a set of ram disks. It would be
interesting if I could put multiple filesystems in the initrd and the
kernel would
notice and automatically set up an overlayfs of them.

> > multiple erofs filesystems that follow each other on the same block
> > device can also be merged? This would allow me to pass multiple erofs
> > filesystems to the kernel via initrd=, which would get concatenated
> > together into a single buffer, which the kernel would write to a
> > ramdisk (using root=/dev/ram0) which the kernel would then have erofs
> > mount to /dev/root. erofs would notice that there's multiple erofs
> > filesystems on the ramdisk and overlay them together (perhaps only if
> > a cmdline option is enabled).
>
> Recently, A merged EROFS filesystem with a single block device can
> be passed but you might still need a step to build a merged fs tree
> before mounting.  Such feature is used to build a metadata and
> reuse original tar blobs.
>
> >
> > Does this make sense at all?
>
> Hopefully it helps.

Re: Merging multiple erofs file systems on the same block device

[Gao Xiang]

On 2023/5/2 18:03, Daan De Meyer wrote:
>> On 2023/5/1 22:09, Daan De Meyer wrote:
>>> Hi,
>>>
>>> I've been looking into erofs as an initramfs replacement by using
>>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
>>
>> Sorry, I'm on vacation now.
>>
>> May I ask what's your detailed use cases?  Sure, you could use
>> /dev/ram0 as a replacement, but currently it still takes double
>> memory compared with initramfs since ramdisk doesn't support FSDAX
>> for now (by enabling FSDAX, it won't take double memory at all.)
> 
> I'm experimenting with larger initramfses and running into memory
> bottlenecks since the entire compressed cpio has to be decompressed
> into memory. I was hoping to use erofs as a replacement that could stay
> compressed, where only the files that are actually accessed are
> decompressed at runtime.

Sorry for late reply.

Okay, that makes sense, although FSDAX cannot be used as this way since
decompressed data is needed for mmapped accesses.

> 
>> Actually I think ramdisk FSDAX is useful and I might sync up this on
>> the following LSF/MM/BPF 2023.
>>
>>> However, by using a ramdisk instead of the usual compressed cpio, I
>>> would lose the feature where the kernel merges multiple individual
>>> cpios together into a single tmpfs filesystem. Looking at the
>>> documentation for erofs, I noticed that erofs already seems to support
>>> merging multiple erofs filesystems on separate block devices using the
>>> device= cmdline option. Would it be possible to extend this so that
>> Here `device=` is actually used to refer to seperate blobs with the
>> merged metadata.  For example, you could have
>>
>>     device=/dev/ram1 original tar1
>>     device=/dev/ram2 original tar2
>>     /dev/ram0        merged metadata for tar1 + tar2.
>>
>> which means, if you'd like to merge multiple EROFS filesystems, you
>> might need another step to build a merged metadata in advance in order
>> to merges multiple individual tarballs together, which could be built
>> when applying images or booting (by using a special bootloader with
>> such functionality.)
> 
> Ahh, I misunderstood the device= option then.
> 
>> EROFS doesn't support stacking multiple fses runtimely since it seems
>> a duplicated feature of overlayfs (you could consider using overlayfs
>> honestly.)
> 
> I would love to use overlayfs, but there's no way to specify to the kernel that
> the initrd should be set up as an overlayfs of a set of ram disks. It would be
> interesting if I could put multiple filesystems in the initrd and the
> kernel would
> notice and automatically set up an overlayfs of them.

I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
guess, you could specify a ramdisk with a customized init to stack
overlayfs like this in the userspace?  Not sure though...

Thanks,
Gao Xiang

Re: Merging multiple erofs file systems on the same block device

[Daan De Meyer]

> On 2023/5/2 18:03, Daan De Meyer wrote:
> >> On 2023/5/1 22:09, Daan De Meyer wrote:
> >>> Hi,
> >>>
> >>> I've been looking into erofs as an initramfs replacement by using
> >>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
> >>
> >> Sorry, I'm on vacation now.
> >>
> >> May I ask what's your detailed use cases?  Sure, you could use
> >> /dev/ram0 as a replacement, but currently it still takes double
> >> memory compared with initramfs since ramdisk doesn't support FSDAX
> >> for now (by enabling FSDAX, it won't take double memory at all.)
> >
> > I'm experimenting with larger initramfses and running into memory
> > bottlenecks since the entire compressed cpio has to be decompressed
> > into memory. I was hoping to use erofs as a replacement that could stay
> > compressed, where only the files that are actually accessed are
> > decompressed at runtime.
>
> Sorry for late reply.
>
> Okay, that makes sense, although FSDAX cannot be used as this way since
> decompressed data is needed for mmapped accesses.

Can you clarify why using a ramdisk would take double the memory if FSDAX
is not used?

> >
> >> Actually I think ramdisk FSDAX is useful and I might sync up this on
> >> the following LSF/MM/BPF 2023.
> >>
> >>> However, by using a ramdisk instead of the usual compressed cpio, I
> >>> would lose the feature where the kernel merges multiple individual
> >>> cpios together into a single tmpfs filesystem. Looking at the
> >>> documentation for erofs, I noticed that erofs already seems to support
> >>> merging multiple erofs filesystems on separate block devices using the
> >>> device= cmdline option. Would it be possible to extend this so that
> >> Here `device=` is actually used to refer to seperate blobs with the
> >> merged metadata.  For example, you could have
> >>
> >>     device=/dev/ram1 original tar1
> >>     device=/dev/ram2 original tar2
> >>     /dev/ram0        merged metadata for tar1 + tar2.
> >>
> >> which means, if you'd like to merge multiple EROFS filesystems, you
> >> might need another step to build a merged metadata in advance in order
> >> to merges multiple individual tarballs together, which could be built
> >> when applying images or booting (by using a special bootloader with
> >> such functionality.)
> >
> > Ahh, I misunderstood the device= option then.
> >
> >> EROFS doesn't support stacking multiple fses runtimely since it seems
> >> a duplicated feature of overlayfs (you could consider using overlayfs
> >> honestly.)
> >
> > I would love to use overlayfs, but there's no way to specify to the kernel that
> > the initrd should be set up as an overlayfs of a set of ram disks. It would be
> > interesting if I could put multiple filesystems in the initrd and the
> > kernel would
> > notice and automatically set up an overlayfs of them.
>
> I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
> guess, you could specify a ramdisk with a customized init to stack
> overlayfs like this in the userspace?  Not sure though...

This approach generally works, but if you want to enforce all mounted
filesystems to be dm-verity protected, you now need to add verity data
for all these filesystems to the initramfs. That's why we're looking
for a solution where the kernel sets up the filesystem mount of the
ramdisk which is special cased and doesn't require verity data to be
present. Anyway, the overlayfs issue is not something for erofs to
solve. If at all desirable, it should probably be filesystem
independent where the kernel just looks for multiple filesystems in
the initrd buffer and sets up an overlay mount using each found
filesystem.

Cheers,

Daan De Meyer


On Fri, 5 May 2023 at 07:05, Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>
>
>
> On 2023/5/2 18:03, Daan De Meyer wrote:
> >> On 2023/5/1 22:09, Daan De Meyer wrote:
> >>> Hi,
> >>>
> >>> I've been looking into erofs as an initramfs replacement by using
> >>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
> >>
> >> Sorry, I'm on vacation now.
> >>
> >> May I ask what's your detailed use cases?  Sure, you could use
> >> /dev/ram0 as a replacement, but currently it still takes double
> >> memory compared with initramfs since ramdisk doesn't support FSDAX
> >> for now (by enabling FSDAX, it won't take double memory at all.)
> >
> > I'm experimenting with larger initramfses and running into memory
> > bottlenecks since the entire compressed cpio has to be decompressed
> > into memory. I was hoping to use erofs as a replacement that could stay
> > compressed, where only the files that are actually accessed are
> > decompressed at runtime.
>
> Sorry for late reply.
>
> Okay, that makes sense, although FSDAX cannot be used as this way since
> decompressed data is needed for mmapped accesses.
>
> >
> >> Actually I think ramdisk FSDAX is useful and I might sync up this on
> >> the following LSF/MM/BPF 2023.
> >>
> >>> However, by using a ramdisk instead of the usual compressed cpio, I
> >>> would lose the feature where the kernel merges multiple individual
> >>> cpios together into a single tmpfs filesystem. Looking at the
> >>> documentation for erofs, I noticed that erofs already seems to support
> >>> merging multiple erofs filesystems on separate block devices using the
> >>> device= cmdline option. Would it be possible to extend this so that
> >> Here `device=` is actually used to refer to seperate blobs with the
> >> merged metadata.  For example, you could have
> >>
> >>     device=/dev/ram1 original tar1
> >>     device=/dev/ram2 original tar2
> >>     /dev/ram0        merged metadata for tar1 + tar2.
> >>
> >> which means, if you'd like to merge multiple EROFS filesystems, you
> >> might need another step to build a merged metadata in advance in order
> >> to merges multiple individual tarballs together, which could be built
> >> when applying images or booting (by using a special bootloader with
> >> such functionality.)
> >
> > Ahh, I misunderstood the device= option then.
> >
> >> EROFS doesn't support stacking multiple fses runtimely since it seems
> >> a duplicated feature of overlayfs (you could consider using overlayfs
> >> honestly.)
> >
> > I would love to use overlayfs, but there's no way to specify to the kernel that
> > the initrd should be set up as an overlayfs of a set of ram disks. It would be
> > interesting if I could put multiple filesystems in the initrd and the
> > kernel would
> > notice and automatically set up an overlayfs of them.
>
> I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
> guess, you could specify a ramdisk with a customized init to stack
> overlayfs like this in the userspace?  Not sure though...
>
> Thanks,
> Gao Xiang

Re: Merging multiple erofs file systems on the same block device

[Gao Xiang]

On 2023/5/5 16:19, Daan De Meyer wrote:
>> On 2023/5/2 18:03, Daan De Meyer wrote:
>>>> On 2023/5/1 22:09, Daan De Meyer wrote:
>>>>> Hi,
>>>>>
>>>>> I've been looking into erofs as an initramfs replacement by using
>>>>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
>>>>
>>>> Sorry, I'm on vacation now.
>>>>
>>>> May I ask what's your detailed use cases?  Sure, you could use
>>>> /dev/ram0 as a replacement, but currently it still takes double
>>>> memory compared with initramfs since ramdisk doesn't support FSDAX
>>>> for now (by enabling FSDAX, it won't take double memory at all.)
>>>
>>> I'm experimenting with larger initramfses and running into memory
>>> bottlenecks since the entire compressed cpio has to be decompressed
>>> into memory. I was hoping to use erofs as a replacement that could stay
>>> compressed, where only the files that are actually accessed are
>>> decompressed at runtime.
>>
>> Sorry for late reply.
>>
>> Okay, that makes sense, although FSDAX cannot be used as this way since
>> decompressed data is needed for mmapped accesses.
> 
> Can you clarify why using a ramdisk would take double the memory if FSDAX
> is not used?

You could take a look at
https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt

and section "ramfs and ramdisk" describes the details, but FSDAX could avoid
this problem.

> 
>>>
>>>> Actually I think ramdisk FSDAX is useful and I might sync up this on
>>>> the following LSF/MM/BPF 2023.
>>>>
>>>>> However, by using a ramdisk instead of the usual compressed cpio, I
>>>>> would lose the feature where the kernel merges multiple individual
>>>>> cpios together into a single tmpfs filesystem. Looking at the
>>>>> documentation for erofs, I noticed that erofs already seems to support
>>>>> merging multiple erofs filesystems on separate block devices using the
>>>>> device= cmdline option. Would it be possible to extend this so that
>>>> Here `device=` is actually used to refer to seperate blobs with the
>>>> merged metadata.  For example, you could have
>>>>
>>>>      device=/dev/ram1 original tar1
>>>>      device=/dev/ram2 original tar2
>>>>      /dev/ram0        merged metadata for tar1 + tar2.
>>>>
>>>> which means, if you'd like to merge multiple EROFS filesystems, you
>>>> might need another step to build a merged metadata in advance in order
>>>> to merges multiple individual tarballs together, which could be built
>>>> when applying images or booting (by using a special bootloader with
>>>> such functionality.)
>>>
>>> Ahh, I misunderstood the device= option then.
>>>
>>>> EROFS doesn't support stacking multiple fses runtimely since it seems
>>>> a duplicated feature of overlayfs (you could consider using overlayfs
>>>> honestly.)
>>>
>>> I would love to use overlayfs, but there's no way to specify to the kernel that
>>> the initrd should be set up as an overlayfs of a set of ram disks. It would be
>>> interesting if I could put multiple filesystems in the initrd and the
>>> kernel would
>>> notice and automatically set up an overlayfs of them.
>>
>> I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
>> guess, you could specify a ramdisk with a customized init to stack
>> overlayfs like this in the userspace?  Not sure though...
> 
> This approach generally works, but if you want to enforce all mounted
> filesystems to be dm-verity protected, you now need to add verity data
> for all these filesystems to the initramfs. That's why we're looking
> for a solution where the kernel sets up the filesystem mount of the
> ramdisk which is special cased and doesn't require verity data to be
> present. Anyway, the overlayfs issue is not something for erofs to
> solve. If at all desirable, it should probably be filesystem
> independent where the kernel just looks for multiple filesystems in
> the initrd buffer and sets up an overlay mount using each found
> filesystem.

I'm not sure if you could still specify one root filesystem with
dm-verity and it mounts filesystems of rest layers and then uses
overlayfs properly with a customized init.

Yes, anyway, you could enhance kernel init code as well.

Thanks,
Gao Xiang

> 
> Cheers,
> 
> Daan De Meyer