Some questions/thoughts on fs-verity

Some questions/thoughts on fs-verity

[Colin Walters]

It's clear that the Linux kernel is widely deployed with things like dm-verity devices where the user of the device isn't root.  There are great security properties from this in ensuring malicious code (compromised apps/OS) can't persist, and what Google is doing with ChromeOS/Android is a good example.

However, for cases where the user *is* root (or, like me as an OS vendor trying to support an OS where the user can be root), dm-verity comes with a whole host of restrictions and issues.  This was noted in the earlier fs-verity discussions.  Among other ones, simply trying to commit to a partition size beyond which the trusted OS cannot grow is seriously ugly.  Another example here is with ostree (or other filesystem-level tools) it's easy to have *three* images (or really N) so that while you're downloading updates you don't lose your rollback, etc.

I'm excited about the potential of fs-verity because it's so much more *flexible* - leaving aside the base OS case for a second - for example fs-verity is even available to unprivileged users by default, so if the admin has at least enabled the `verity` flag, if a user wanted to they could enable fs-verity for e.g. their `~/.bashrc`.  That's neat!

However, this gets into some questions I have around the security properties of fs-verity because - it only covers file contents.  There are many problems from this:

 - Verifying directories and symlinks is really desirable too; take e.g. /etc/systemd/system - I want to verify not just that the unit files there are valid, but also that there's no malicious ones.  
 - Being able to e.g. `chown root:root` `chmod u+s` a fs-verity protected binary is...not desired.
 - Finally, taking the scenario of a malicious code that has gained CAP_SYS_ADMIN and the ability to write to raw block devices, it seems to me that the discussions around "untrusted filesystems" (https://lwn.net/Articles/755593/) come to the fore.

In contrast because dm-verity is sealing up *everything* at the fs level, all of the above are avoided.  But it's obviously far less flexible...

(This discussion of course mirrors fs-crypt versus dm-crypt too)

I guess my concrete question here is: Are there any plans around extending fs-verity to address any of this?  Which I know given the current developers probably mostly boils down to a future Android/ChromeOS architecture question, but I think fs-verity has the potential to be used beyond just that if it isn't already.

Would love to discuss with any other distributions/update system developers etc. that are also looking at fs-verity!

Re: Some questions/thoughts on fs-verity

[Eric Biggers]

Hi Colin,

On Fri, Nov 08, 2019 at 02:18:32PM -0500, Colin Walters wrote:
> It's clear that the Linux kernel is widely deployed with things like dm-verity devices where the user of the device isn't root.  There are great security properties from this in ensuring malicious code (compromised apps/OS) can't persist, and what Google is doing with ChromeOS/Android is a good example.
> 
> However, for cases where the user *is* root (or, like me as an OS vendor trying to support an OS where the user can be root), dm-verity comes with a whole host of restrictions and issues.  This was noted in the earlier fs-verity discussions.  Among other ones, simply trying to commit to a partition size beyond which the trusted OS cannot grow is seriously ugly.  Another example here is with ostree (or other filesystem-level tools) it's easy to have *three* images (or really N) so that while you're downloading updates you don't lose your rollback, etc.
> 
> I'm excited about the potential of fs-verity because it's so much more *flexible* - leaving aside the base OS case for a second - for example fs-verity is even available to unprivileged users by default, so if the admin has at least enabled the `verity` flag, if a user wanted to they could enable fs-verity for e.g. their `~/.bashrc`.  That's neat!
> 
> However, this gets into some questions I have around the security properties of fs-verity because - it only covers file contents.  There are many problems from this:
> 
>  - Verifying directories and symlinks is really desirable too; take e.g. /etc/systemd/system - I want to verify not just that the unit files there are valid, but also that there's no malicious ones.  
>  - Being able to e.g. `chown root:root` `chmod u+s` a fs-verity protected binary is...not desired.
>  - Finally, taking the scenario of a malicious code that has gained CAP_SYS_ADMIN and the ability to write to raw block devices, it seems to me that the discussions around "untrusted filesystems" (https://lwn.net/Articles/755593/) come to the fore.
> 

Can you take a step back and consider what security properties you're actually
trying to achieve?  In particular, be aware that if you just enable fs-verity on
a file but never actually compare the hash to anything, it provides integrity
only (detection of accidental corruption), not any authenticity protection.

In your pull request to OSTree (https://github.com/ostreedev/ostree/pull/1959),
I see a call to FS_IOC_ENABLE_VERITY.  But there's no corresponding call to
FS_IOC_MEASURE_VERITY to actually authenticate the file(s), nor is fs-verity's
built-in signature verification feature being used.  So this is integrity-only;
there's no protection against malicious modifications to the data on-disk.

I.e., fs-verity is really meant to be used as part of a userspace-driven
authentication policy.  It's not something that magically increases security by
itself.  That's partly why the scope of fs-verity is limited to file contents:
userspace can still authenticate other metadata if needed.

As for an attacker exploiting a filesystem bug, yes that is a big problem
currently in Linux.  However, gaining code execution via such a bug is a *bug*
and is patchable and mitigable, whereas gaining code execution by modifying
unauthenticated code stored on-disk is simply working as intended.

- Eric

Re: Some questions/thoughts on fs-verity

[Colin Walters]

On Fri, Nov 8, 2019, at 10:41 PM, Eric Biggers wrote:

> Can you take a step back and consider what security properties you're actually
> trying to achieve? 

I'd like similar security properties to a full dm-verity bootable OS image - if someone gains CAP_SYS_ADMIN and the ability to subvert LSMs etc. and write arbitrary files (and most ideally protection against writes to underlying raw block devices), there should not be a "persistence vector" that allows them to regain CAP_SYS_ADMIN after a reboot (automatically, without re-doing an exploit).

This assumes that the userspace won't go off and execute unverified code from outside the root; see also
https://blog.verbum.org/2017/06/12/on-dm-verity-and-operating-systems/

> In particular, be aware that if you just enable fs-verity on
> a file but never actually compare the hash to anything, it provides integrity
> only (detection of accidental corruption), not any authenticity protection.

Right; as I note in the PR just to start
" Among other things, it does finally add an API that makes files immutable, which will help against some accidental damage."

Content immutable but still unlink() and link()-able being key here.

> In your pull request to OSTree (https://github.com/ostreedev/ostree/pull/1959),
> I see a call to FS_IOC_ENABLE_VERITY.  But there's no corresponding call to
> FS_IOC_MEASURE_VERITY to actually authenticate the file(s), nor is fs-verity's
> built-in signature verification feature being used.  So this is integrity-only;
> there's no protection against malicious modifications to the data on-disk.

Yes, I understand that, but I didn't investigate that too much exactly because of the issues I noted in my original mail; it doesn't help my use case much to measure files if e.g. one can gain code execution by other means.
 
> I.e., fs-verity is really meant to be used as part of a userspace-driven
> authentication policy.  It's not something that magically increases security by
> itself.  That's partly why the scope of fs-verity is limited to file contents:
> userspace can still authenticate other metadata if needed.

If I understand correctly, for the Android/ChromeOS use case, fs-verity will be applied to application ZIP files, and the base OS userspace will verify their signatures before launching.  That seems straightforward.  But that approach effectively requires abandoning a model of writing files directly to the filesystem and instead serializing to a tarball/zipfile.  Being able to support e.g. traditional OCI/Docker-style container images (where all of userspace expects to use the open() syscall rather than e.g. a special resource API[1]) on such a system would require something like a FUSE mount of a fs-verity tarball/zipfile (right?). It's *possible* but FUSE has all sorts of downsides.

It also means (if implemented naively at least) you have similar "write amplification" issues - if just a few files change in an app, you end up having to re-write a full zip/tarball.

Further, for the actual base OS case...hmm, I guess we could run a FUSE mount in the initramfs or so, but...eww.

> As for an attacker exploiting a filesystem bug, yes that is a big problem
> currently in Linux.  However, gaining code execution via such a bug is a *bug*
> and is patchable and mitigable, whereas gaining code execution by modifying
> unauthenticated code stored on-disk is simply working as intended.

Often when I use the term "secure" I mean "As a software vendor, we believe we can fix security issues with this over time, and there won't be too many to be embarrassing".   Canonical examples here being virtualization and container systems like Docker/podman/etc.

It sounds like you're saying you believe that (at least ext4?) could be secure in this sense - it's not clear to me (from previous discussions) that other filesystem developers agree (at least for their filesystems).

Anyways, to restate the goal - having the security properties of a dm-verity base OS, but with a lot more flexiblity (no fixed partition size, can easily have 3 or more images, etc.).

For example, I could imagine some API that allows userspace to also "seal" a directory and symlinks (inside the filesystem, this could end up stored similarly to a fs-verity file).  That'd be a powerful primitive, because one wouldn't need to store everything inside zip files, and it'd be much easier to e.g. have userspace verify the root hash of a sealed directory and have everything else trusted from that.  For example, without having to patch systemd to learn how to measure unit files.

Or really a bottom line here is - I could imagine reworking our userspace to do this FUSE mount of fs-verity tarball model, but if e.g. the kernel filesystems aren't really feasably made safe against malicious code, then...it may not be worth doing.

[1] https://developer.android.com/guide/topics/resources/providing-resources

Re: Some questions/thoughts on fs-verity

[Colin Walters]

On Sat, Nov 9, 2019, at 1:46 PM, Colin Walters wrote:
>
> Or really a bottom line here is - I could imagine reworking our 
> userspace to do this FUSE mount of fs-verity tarball model, but if e.g. 
> the kernel filesystems aren't really feasably made safe against 
> malicious code, then...it may not be worth doing.

On thinking about this sub-thread more, if one is shipping an OS with a persistent storage volume (whether the same as the fs-verity'd OS or separate) that's mounted as a regular Linux filesystem, all the concerns about corrupted FS images apply, regardless of whether one is using dm-verity or fs-verity for the OS.

Although perhaps in theory in the dm-verity case, if the persistent volume is separate one could e.g. do some userspace sanity checking (fsck) of the persistent volume before mounting it, or e.g. apply OS updates before mounting it and reboot (to address the scenario where an older kernel has an arbitrary code execution flaw that's fixed in a new update).

Hmm.  Maybe it's just not feasible to avoid effectively verifying a full filesystem image for the early boot OS anytime soon (whether that's an initramfs image baked into a signed kernel or actually dm-verity).  But probably for us going down the path of including the OS update system in the initramfs, and signing initramfs images will make more sense than dm-verity.

Either way, this gets us to the point of "can apply security updates and/or inspect filesystems on disk before mounting them" executing signed/trusted code; but leaves open the rest of the discussion around applications (Linux containers, Android/flatpak style apps) etc. that doesn't scale to bake into the initramfs (or generate dm-verity partitions), and like Android is doing we want to support potentially privileged applications that are distinct from the OS.  And basically whether fs-verity can be extended to support regular filesystem trees for those.