* IMA on remote file systems @ 2019-08-28 17:36 Chuck Lever 2019-09-13 14:50 ` Chuck Lever 2019-09-16 13:16 ` Janne Karhunen 0 siblings, 2 replies; 18+ messages in thread From: Chuck Lever @ 2019-08-28 17:36 UTC (permalink / raw) To: linux-integrity Last week I presented at the Linux Security Summit on a proposal for handling IMA metadata on NFS files. After the presentation, Mimi Zohar pointed out that although the proposal extends protection from an NFS file server to time-of-measurement on an NFS client, there is still a protection gap between time-of-measurement and time-of-use on that client. My proposal enables storage of per-file IMA metadata via the NFSv4 protocol. I have a prototype and an IETF nfsv4 Working Group document that specifies a small protocol extension. I would like to find a way to extend IMA protection all the way to time-of-use on NFS clients. The consensus is that a per-file Merkle tree would be the most desirable approach, as that is the same mechanism used for fs-verity protection. For a few important reasons, it will be challenging to plumb support for durable Merkle trees into NFS, although that is an eventual goal. My thought was to use an ephemeral Merkle tree for NFS (and possibly other remote filesystems, like FUSE, until these filesystems support durable per-file Merkle trees). A tree would be constructed when the client measures a file, but it would not saved to the filesystem. Instead of a hash of the file's contents, the tree's root signature is stored as the IMA metadata. Once a Merkle tree is available, it can be used in exactly the same way that a durable Merkle tree would, to verify the integrity of individual pages as they are used, evicted, and then read back from the server. If the client needs to evict part or all of an ephemeral tree, it can subsequently be reconstructed by measuring the file again and verifying its root signature against the stored IMA metadata. So the only difference here is that the latency-to-first-byte benefit of a durable Merkle tree would be absent. I'm interested in any thoughts or opinions about this approach. -- Chuck Lever ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-08-28 17:36 IMA on remote file systems Chuck Lever @ 2019-09-13 14:50 ` Chuck Lever 2019-09-15 21:42 ` Mimi Zohar 2019-09-16 13:16 ` Janne Karhunen 1 sibling, 1 reply; 18+ messages in thread From: Chuck Lever @ 2019-09-13 14:50 UTC (permalink / raw) To: linux-integrity, linux-fsdevel Cc: Mimi Zohar, Michael Halcrow, Theodore Y. Ts'o, Matthew Garrett, Eric Biggers Resending ... > On Aug 28, 2019, at 1:36 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > > Last week I presented at the Linux Security Summit on a proposal > for handling IMA metadata on NFS files. My proposal enables storage > of per-file IMA metadata via the NFSv4 protocol. I have a prototype > and an IETF nfsv4 Working Group document that specifies a small > protocol extension. > > After the presentation, Mimi Zohar pointed out that although the > proposal extends protection from an NFS file server to time-of- > measurement on an NFS client, there is still a protection gap between > time-of-measurement and time-of-use on that client. > > I would like to find a way to extend IMA protection all the way > to time-of-use on NFS clients. The consensus is that a per-file > Merkle tree would be the most desirable approach, as that is the > same mechanism used for fs-verity protection. > > For a few important reasons, it will be challenging to plumb > support for durable Merkle trees into NFS, although that is an > eventual goal. > > My thought was to use an ephemeral Merkle tree for NFS (and > possibly other remote filesystems, like FUSE, until these > filesystems support durable per-file Merkle trees). A tree would > be constructed when the client measures a file, but it would not > saved to the filesystem. Instead of a hash of the file's contents, > the tree's root signature is stored as the IMA metadata. > > Once a Merkle tree is available, it can be used in exactly the > same way that a durable Merkle tree would, to verify the integrity > of individual pages as they are used, evicted, and then read back > from the server. > > If the client needs to evict part or all of an ephemeral tree, it > can subsequently be reconstructed by measuring the file again and > verifying its root signature against the stored IMA metadata. > > So the only difference here is that the latency-to-first-byte > benefit of a durable Merkle tree would be absent. > > I'm interested in any thoughts or opinions about this approach. -- Chuck Lever ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-13 14:50 ` Chuck Lever @ 2019-09-15 21:42 ` Mimi Zohar 2019-09-16 16:10 ` Theodore Y. Ts'o 0 siblings, 1 reply; 18+ messages in thread From: Mimi Zohar @ 2019-09-15 21:42 UTC (permalink / raw) To: Chuck Lever, linux-integrity, linux-fsdevel Cc: Michael Halcrow, Theodore Y. Ts'o, Matthew Garrett, Eric Biggers On Fri, 2019-09-13 at 10:50 -0400, Chuck Lever wrote: > Resending ... > > > On Aug 28, 2019, at 1:36 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > > > > Last week I presented at the Linux Security Summit on a proposal > > for handling IMA metadata on NFS files. My proposal enables storage > > of per-file IMA metadata via the NFSv4 protocol. I have a prototype > > and an IETF nfsv4 Working Group document that specifies a small > > protocol extension. > > > > After the presentation, Mimi Zohar pointed out that although the > > proposal extends protection from an NFS file server to time-of- > > measurement on an NFS client, there is still a protection gap between > > time-of-measurement and time-of-use on that client. > > > > I would like to find a way to extend IMA protection all the way > > to time-of-use on NFS clients. The consensus is that a per-file > > Merkle tree would be the most desirable approach, as that is the > > same mechanism used for fs-verity protection. > > > > For a few important reasons, it will be challenging to plumb > > support for durable Merkle trees into NFS, although that is an > > eventual goal. > > > > My thought was to use an ephemeral Merkle tree for NFS (and > > possibly other remote filesystems, like FUSE, until these > > filesystems support durable per-file Merkle trees). A tree would > > be constructed when the client measures a file, but it would not > > saved to the filesystem. Instead of a hash of the file's contents, > > the tree's root signature is stored as the IMA metadata. > > > > Once a Merkle tree is available, it can be used in exactly the > > same way that a durable Merkle tree would, to verify the integrity > > of individual pages as they are used, evicted, and then read back > > from the server. > > > > If the client needs to evict part or all of an ephemeral tree, it > > can subsequently be reconstructed by measuring the file again and > > verifying its root signature against the stored IMA metadata. > > > > So the only difference here is that the latency-to-first-byte > > benefit of a durable Merkle tree would be absent. > > > > I'm interested in any thoughts or opinions about this approach. I like the idea, but there are a couple of things that need to happen first. Both fs-verity and IMA appended signatures need to be upstreamed. The IMA appended signature support simplifies ima_appraise_measurement(), paving the way for adding IMA support for other types of signature verification. How IMA will support fs-verity signatures still needs to be defined. That discussion will hopefully include NFS support. Mimi ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-15 21:42 ` Mimi Zohar @ 2019-09-16 16:10 ` Theodore Y. Ts'o 2019-09-16 18:16 ` Chuck Lever 0 siblings, 1 reply; 18+ messages in thread From: Theodore Y. Ts'o @ 2019-09-16 16:10 UTC (permalink / raw) To: Mimi Zohar Cc: Chuck Lever, linux-integrity, linux-fsdevel, Michael Halcrow, Theodore Y. Ts'o, Matthew Garrett, Eric Biggers On Sun, Sep 15, 2019 at 05:42:10PM -0400, Mimi Zohar wrote: > > > My thought was to use an ephemeral Merkle tree for NFS (and > > > possibly other remote filesystems, like FUSE, until these > > > filesystems support durable per-file Merkle trees). A tree would > > > be constructed when the client measures a file, but it would not > > > saved to the filesystem. Instead of a hash of the file's contents, > > > the tree's root signature is stored as the IMA metadata. > > > > > > Once a Merkle tree is available, it can be used in exactly the > > > same way that a durable Merkle tree would, to verify the integrity > > > of individual pages as they are used, evicted, and then read back > > > from the server. > > > > > > If the client needs to evict part or all of an ephemeral tree, it > > > can subsequently be reconstructed by measuring the file again and > > > verifying its root signature against the stored IMA metadata. Where would the client store the ephemeral tree? If you're thinking about storing in memory, calculating the emphemeral tree would require dragging the entire file across the network, which is going to be just as bad as using IMA --- plus the CPU cost of calculating the Merkle tree, and the memory cost of storing the ephemeral Merkle tree. I suspect that for most clients, it wouldn't be worth it unless the client can store the ephemeral tree *somewhere* on the client's local persistent storage, or maybe if it could store the Merkle tree on the NFS server (maybe via an xattr which contains the pathname to the Merkle tree relative to the NFS mount point?). > > > So the only difference here is that the latency-to-first-byte > > > benefit of a durable Merkle tree would be absent. What problem are you most interested in solving? And what cost do you think the user will be willing to pay in order to solve that problem? > I like the idea, but there are a couple of things that need to happen > first. Both fs-verity and IMA appended signatures need to be > upstreamed. Eric has sent the pull request fs-verity today. > The IMA appended signature support simplifies > ima_appraise_measurement(), paving the way for adding IMA support for > other types of signature verification. How IMA will support fs-verity > signatures still needs to be defined. That discussion will hopefully > include NFS support. As far as using the Merkle tree root hash for the IMA measurement, what sort of policy should be used for determining when the Merkle tree root hash should be used in preference to reading and checksuming the whole file when it is first opened? It could be as simple as, "if this is a fs-verity, use the fs-verity Merkle root". Is that OK? - Ted ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-16 16:10 ` Theodore Y. Ts'o @ 2019-09-16 18:16 ` Chuck Lever 0 siblings, 0 replies; 18+ messages in thread From: Chuck Lever @ 2019-09-16 18:16 UTC (permalink / raw) To: Theodore Y. Ts'o Cc: Mimi Zohar, linux-integrity, linux-fsdevel, Michael Halcrow, Theodore Y. Ts'o, Matthew Garrett, Eric Biggers > On Sep 16, 2019, at 12:10 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote: > > On Sun, Sep 15, 2019 at 05:42:10PM -0400, Mimi Zohar wrote: >>>> My thought was to use an ephemeral Merkle tree for NFS (and >>>> possibly other remote filesystems, like FUSE, until these >>>> filesystems support durable per-file Merkle trees). A tree would >>>> be constructed when the client measures a file, but it would not >>>> saved to the filesystem. Instead of a hash of the file's contents, >>>> the tree's root signature is stored as the IMA metadata. >>>> >>>> Once a Merkle tree is available, it can be used in exactly the >>>> same way that a durable Merkle tree would, to verify the integrity >>>> of individual pages as they are used, evicted, and then read back >>>> from the server. >>>> >>>> If the client needs to evict part or all of an ephemeral tree, it >>>> can subsequently be reconstructed by measuring the file again and >>>> verifying its root signature against the stored IMA metadata. > > Where would the client store the ephemeral tree? If you're thinking > about storing in memory, calculating the emphemeral tree would require > dragging the entire file across the network, which is going to be just > as bad as using IMA --- plus the CPU cost of calculating the Merkle > tree, and the memory cost of storing the ephemeral Merkle tree. A client would store ephemeral Merkle trees in memory. The most interesting use case to me is protecting executables and DLLs. These will tend to be limited in size, so the cost of Merkle tree construction should be nicely bounded in the typical case. An additional cost would arise if the in-memory tree were to be evicted. We hope that is an infrequent event. If the tree is partially evicted, only some of the file needs to be read back to re-construct it, since we would still have in-memory hashes stored in the interior nodes of the tree that enable the client to verify the portion of the tree that needs to be re-constructed. The short-term purpose of these trees is to add the value of better integrity protection for file systems that find it difficult to store per-file Merkle trees durably. We expect that situation will be temporary for many file systems, though not all. The price that is paid for this extra protection is that it will perform like traditional IMA, as you observed above. This is probably a different cost than reading from flash on a mobile device: a typical NFS client will be less memory- and CPU-constrained than a mobile device, and the cost of reading over NFS on a fast network from the server's cache is not high. The trade-offs here are going to be different. > I suspect that for most clients, it wouldn't be worth it unless the > client can store the ephemeral tree *somewhere* on the client's local > persistent storage, or maybe if it could store the Merkle tree on the > NFS server (maybe via an xattr which contains the pathname to the > Merkle tree relative to the NFS mount point?). The trees could be cached locally for exceptionally large files (eg files larger than the client's physical memory). For smaller files, which I expect will be the typical case, the cost of reading a file will be about the same as reading a Merkle tree. As mentioned in my proposal, the eventual goal is to extend the NFS protocol to store the Merkle tree durably on the server. We will get there eventually. Changing the protocol is a slow process, particularly because it involves consensus among NFS implementers who work on other operating systems besides Linux. >>>> So the only difference here is that the latency-to-first-byte >>>> benefit of a durable Merkle tree would be absent. > > What problem are you most interested in solving? And what cost do you > think the user will be willing to pay in order to solve that problem? NFS users would get full protection of their files from storage to point-of-use, at the same cost as IMA, until some point in the future when NFS can store the trees durably. The same would apply to other filesystems that find storing a full Merkle tree to be a challenge. >> I like the idea, but there are a couple of things that need to happen >> first. Both fs-verity and IMA appended signatures need to be >> upstreamed. > > Eric has sent the pull request fs-verity today. > >> The IMA appended signature support simplifies >> ima_appraise_measurement(), paving the way for adding IMA support for >> other types of signature verification. How IMA will support fs-verity >> signatures still needs to be defined. That discussion will hopefully >> include NFS support. > > As far as using the Merkle tree root hash for the IMA measurement, > what sort of policy should be used for determining when the Merkle > tree root hash should be used in preference to reading and checksuming > the whole file when it is first opened? It could be as simple as, "if > this is a fs-verity, use the fs-verity Merkle root". Is that OK? > > - Ted -- Chuck Lever ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-08-28 17:36 IMA on remote file systems Chuck Lever 2019-09-13 14:50 ` Chuck Lever @ 2019-09-16 13:16 ` Janne Karhunen 2019-09-16 14:47 ` Chuck Lever 1 sibling, 1 reply; 18+ messages in thread From: Janne Karhunen @ 2019-09-16 13:16 UTC (permalink / raw) To: Chuck Lever; +Cc: linux-integrity On Wed, Aug 28, 2019 at 8:36 PM Chuck Lever <chuck.lever@oracle.com> wrote: > My thought was to use an ephemeral Merkle tree for NFS (and > possibly other remote filesystems, like FUSE, until these > filesystems support durable per-file Merkle trees). A tree would > be constructed when the client measures a file, but it would not > saved to the filesystem. Instead of a hash of the file's contents, > the tree's root signature is stored as the IMA metadata. So the attack you are trying to guard against is that the pages that were evicted once and that are read back could still be integrity verified? Handling this properly would be awesome. I don't think we have anything against this now, the pages that were once evicted are really not checked when read back. -- Janne ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-16 13:16 ` Janne Karhunen @ 2019-09-16 14:47 ` Chuck Lever 2019-09-17 6:30 ` Janne Karhunen 0 siblings, 1 reply; 18+ messages in thread From: Chuck Lever @ 2019-09-16 14:47 UTC (permalink / raw) To: Janne Karhunen; +Cc: linux-integrity > On Sep 16, 2019, at 9:16 AM, Janne Karhunen <janne.karhunen@gmail.com> wrote: > > On Wed, Aug 28, 2019 at 8:36 PM Chuck Lever <chuck.lever@oracle.com> wrote: > >> My thought was to use an ephemeral Merkle tree for NFS (and >> possibly other remote filesystems, like FUSE, until these >> filesystems support durable per-file Merkle trees). A tree would >> be constructed when the client measures a file, but it would not >> saved to the filesystem. Instead of a hash of the file's contents, >> the tree's root signature is stored as the IMA metadata. > > So the attack you are trying to guard against is that the pages that > were evicted once and that are read back could still be integrity > verified? Yes, the idea would be to provide a generic mechanism for constructing ephemeral trees such that it can be used for the purpose you describe on behalf of file systems besides NFS; eg. FUSE, or other remote file systems such as SMB. In addition, I hope the mechanism would also be able to reconstruct a partially evicted Merkle tree as well (in the cases where there is no durable tree available). > Handling this properly would be awesome. I don't think we have > anything against this now, the pages that were once evicted are really > not checked when read back. -- Chuck Lever ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-16 14:47 ` Chuck Lever @ 2019-09-17 6:30 ` Janne Karhunen 2019-09-17 12:45 ` Theodore Y. Ts'o 0 siblings, 1 reply; 18+ messages in thread From: Janne Karhunen @ 2019-09-17 6:30 UTC (permalink / raw) To: Chuck Lever; +Cc: linux-integrity, Theodore Y. Ts'o On Mon, Sep 16, 2019 at 5:47 PM Chuck Lever <chuck.lever@oracle.com> wrote: > >> My thought was to use an ephemeral Merkle tree for NFS (and > >> possibly other remote filesystems, like FUSE, until these > >> filesystems support durable per-file Merkle trees). A tree would > >> be constructed when the client measures a file, but it would not > >> saved to the filesystem. Instead of a hash of the file's contents, > >> the tree's root signature is stored as the IMA metadata. > > > > So the attack you are trying to guard against is that the pages that > > were evicted once and that are read back could still be integrity > > verified? > > Yes, the idea would be to provide a generic mechanism for constructing > ephemeral trees such that it can be used for the purpose you describe > on behalf of file systems besides NFS; eg. FUSE, or other remote file > systems such as SMB. I might be wrong, but handling this properly would be good for the core IMA as well. Take an example of a memory mapped database file: this file will have generic write access for a group of processes. Now, if the attacker can create memory pressure on the host, we might eventually end up freeing pages from this particular file. Once this happens the attacker is free to modify the pages on the disk and they will all get eventually loaded back into the memory without no-one noticing. Could the fs-verity be plugged in as a measurement mechanism in the IMA? So rather than calling a hash function, call verity to measure and add new set of IMA hooks to report violations that arise after execution? IMA policy logic and functionality would be pretty much unchanged. -- Janne ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-17 6:30 ` Janne Karhunen @ 2019-09-17 12:45 ` Theodore Y. Ts'o 2019-09-17 14:18 ` Mimi Zohar 2019-09-17 14:56 ` James Bottomley 0 siblings, 2 replies; 18+ messages in thread From: Theodore Y. Ts'o @ 2019-09-17 12:45 UTC (permalink / raw) To: Janne Karhunen; +Cc: Chuck Lever, linux-integrity On Tue, Sep 17, 2019 at 09:30:31AM +0300, Janne Karhunen wrote: > I might be wrong, but handling this properly would be good for the > core IMA as well. Take an example of a memory mapped database file: > this file will have generic write access for a group of processes. > Now, if the attacker can create memory pressure on the host, we might > eventually end up freeing pages from this particular file. Once this > happens the attacker is free to modify the pages on the disk and they > will all get eventually loaded back into the memory without no-one > noticing. There seems to be a philosophical debate about this. Some IMA folks have claimed that you want to know at the time of the binary being executed, whether or not it is corrupt or not. Their concern is that if you can make a binary crash when it pages in some page of memory, you might be able to exploit that fact by being able to force some setuid binary to stop at some arbitrary point. My understanding was that they were pretty absolutist about that position, and that's one of the reason why the original fs-verity work was completely decoupled from IMA, and why fs-verity exists in the first place ---- if you have gargantuan binaries with ELF debugging sections which will never get read in normal use cases, or APK files with translation tables or video cut scenes which are only used at when the game is first started, or in between major sections of the game, you don't want to delay application startup for long periods of time just to checksum the whole darned file. The issue about using memory pressure to push pages out to disk and forcing a TOU/TOC security failure is the reason why Check is interested in fs-verity. In the original IMA design, the disk is considered part of the Trusted Computing Base (TCB), at least while it was spinning. So the asssumption is the attacker wouldn't be able to modify the disk blocks once the system was booted, and IMA merely protected against off-line attacks (aka, "the evil maid scenario" where the laptop or mobile device left in the hotel room gets a free upgrade courtesy of some state intelligence agency like the MSS). But for NFS, the NFS server is generally not considered part of the TCB, so the original IMA design very vulnerable to the concern you've described. > Could the fs-verity be plugged in as a measurement mechanism in the > IMA? So rather than calling a hash function, call verity to measure > and add new set of IMA hooks to report violations that arise after > execution? IMA policy logic and functionality would be pretty much > unchanged. That is the plan, and it's not hard to do. The question which I've raised is when should we do it, given that some people believe that pulling the entire file into memory and checksumming it at exec or open time is a feature, not a bug. Should we use the fs-verity merkel tree root hash as the measurement function unconditionally if it is present? Or does IMA want to have some kind of tuning knob; and if so, should it be on a per-file system basis, or globally, etc. etc. Those are IMA design questions, and I'll let the IMA folks decide what they want to do. - Ted ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-17 12:45 ` Theodore Y. Ts'o @ 2019-09-17 14:18 ` Mimi Zohar 2019-09-17 14:56 ` James Bottomley 1 sibling, 0 replies; 18+ messages in thread From: Mimi Zohar @ 2019-09-17 14:18 UTC (permalink / raw) To: Theodore Y. Ts'o, Janne Karhunen; +Cc: Chuck Lever, linux-integrity On Tue, 2019-09-17 at 08:45 -0400, Theodore Y. Ts'o wrote: > On Tue, Sep 17, 2019 at 09:30:31AM +0300, Janne Karhunen wrote: > > Could the fs-verity be plugged in as a measurement mechanism in the > > IMA? So rather than calling a hash function, call verity to measure > > and add new set of IMA hooks to report violations that arise after > > execution? IMA policy logic and functionality would be pretty much > > unchanged. > > That is the plan, and it's not hard to do. The question which I've > raised is when should we do it, given that some people believe that > pulling the entire file into memory and checksumming it at exec or > open time is a feature, not a bug. > > Should we use the fs-verity merkel tree root hash as the measurement > function unconditionally if it is present? Or does IMA want to have > some kind of tuning knob; and if so, should it be on a per-file system > basis, or globally, etc. etc. Those are IMA design questions, and > I'll let the IMA folks decide what they want to do. IMA doesn't hard code policy in the kernel, but is based on a single, centralized policy, which contains measurement, appraisal, and audit rules. Just as the new IMA appended signature support (kernel module signature format)[1] contains a new "appraise_type=imasig|modsig" option, there would be a similar option for fs-verity. Mimi [1] Included in the v5.4 pull request. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-17 12:45 ` Theodore Y. Ts'o 2019-09-17 14:18 ` Mimi Zohar @ 2019-09-17 14:56 ` James Bottomley 2019-09-18 5:27 ` Janne Karhunen 2019-09-18 12:37 ` Theodore Y. Ts'o 1 sibling, 2 replies; 18+ messages in thread From: James Bottomley @ 2019-09-17 14:56 UTC (permalink / raw) To: Theodore Y. Ts'o, Janne Karhunen; +Cc: Chuck Lever, linux-integrity On Tue, 2019-09-17 at 08:45 -0400, Theodore Y. Ts'o wrote: > On Tue, Sep 17, 2019 at 09:30:31AM +0300, Janne Karhunen wrote: > > I might be wrong, but handling this properly would be good for the > > core IMA as well. Take an example of a memory mapped database file: > > this file will have generic write access for a group of processes. > > Now, if the attacker can create memory pressure on the host, we > > might eventually end up freeing pages from this particular file. > > Once this happens the attacker is free to modify the pages on the > > disk and they will all get eventually loaded back into the memory > > without no-one noticing. > > There seems to be a philosophical debate about this. Some IMA folks > have claimed that you want to know at the time of the binary being > executed, whether or not it is corrupt or not. Their concern is that > if you can make a binary crash when it pages in some page of memory, > you might be able to exploit that fact by being able to force some > setuid binary to stop at some arbitrary point. My understanding was > that they were pretty absolutist about that position, and that's one > of the reason why the original fs-verity work was completely > decoupled from IMA, and why fs-verity exists in the first place ---- > if you have gargantuan binaries with ELF debugging sections which > will never get read in normal use cases, or APK files with > translation tables or video cut scenes which are only used at when > the game is first started, or in between major sections of the game, > you don't want to delay application startup for long periods of time > just to checksum the whole darned file. That's not my recollection of the IMA position. My recollection is that when IMA was created the idea was to use a single hash, gated in a variety of places dependent on policy, which would appraise the whole file when the gate was hit. The reason seems to be more pragmatic: single hashes and signed single hashes are small enough to store in xattrs, well understood in cryptographic situations and easy to deal with. The biggest problem with fs-verity has been where to store the merkel tree. However, what I've heard from IMA people is as long as the merkle tree storage problem gets solved satisfactorily, they're perfectly happy to have per page hash verification be an IMA mechanism because it's a simple extension of policy and an addition of a gate. > The issue about using memory pressure to push pages out to disk and > forcing a TOU/TOC security failure is the reason why Check is > interested in fs-verity. In the original IMA design, the disk is > considered part of the Trusted Computing Base (TCB), at least while > it was spinning. So the asssumption is the attacker wouldn't be able > to modify the disk blocks once the system was booted, and IMA merely > protected against off-line attacks (aka, "the evil maid scenario" > where the laptop or mobile device left in the hotel room gets a free > upgrade courtesy of some state intelligence agency like the MSS). I'd more say the Linux Kernel itself, at least for executables, has mechanisms to ensure open files aren't updated because that ends up causing potential I/D cache incoherence with resulting binary crashes, so for executables, IMA relies on this mechanism. For non executables, it tries to use the inode generation to see if a recheck is required. In order to corrupt the underlying file, you have to be able to write to it outside the VFS ... not impossible, but a well understood attack vector. I'd also be wary of pushing a merkle tree as a solution to the TOC/TOU problem because, while it's true you can appraise a page at a time using the lowest layer hash, unless you do a full merkle tree check on that hash every time, you're still vulnerable to the attacker corrupting both the hash and the page (especially as the merkle tree can be so huge, requiring that it be paged too). So I'd rather say the monolithic hash and merkle tree have different potential TOC/TOU failure modes and thus different potential security properties, so it's right a user gets to choose the properties they want by policy. > But for NFS, the NFS server is generally not considered part of the > TCB, so the original IMA design very vulnerable to the concern you've > described. A lot of work has gone into NFS security, so I wouldn't say you can never trust the NFS server (A lot of cloud storage still uses NFS, for instance, and they've sold it to their customers as secure). However, again, by policy, there might be situations where you have trust problems with the NFS server and want additional security. There it certainly seems that the merkle tree approach can provide useful additions to this situation provided you're careful how the hash is supplied and checked (this seems to me where we should be focussing the technical discussion). > > Could the fs-verity be plugged in as a measurement mechanism in the > > IMA? So rather than calling a hash function, call verity to measure > > and add new set of IMA hooks to report violations that arise after > > execution? IMA policy logic and functionality would be pretty much > > unchanged. > > That is the plan, and it's not hard to do. The question which I've > raised is when should we do it, given that some people believe that > pulling the entire file into memory and checksumming it at exec or > open time is a feature, not a bug. > > Should we use the fs-verity merkel tree root hash as the measurement > function unconditionally if it is present? Or does IMA want to have > some kind of tuning knob; and if so, should it be on a per-file > system basis, or globally, etc. etc. Those are IMA design questions, > and I'll let the IMA folks decide what they want to do. That's surely policy and IMA is very flexible with regard to policy, so you wouldn't really want to make a global prescription. I believe the current IMA plan is simply to let the policy decide the question, which seems right to me. James ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-17 14:56 ` James Bottomley @ 2019-09-18 5:27 ` Janne Karhunen 2019-09-18 12:50 ` Theodore Y. Ts'o 2019-09-18 15:52 ` James Bottomley 2019-09-18 12:37 ` Theodore Y. Ts'o 1 sibling, 2 replies; 18+ messages in thread From: Janne Karhunen @ 2019-09-18 5:27 UTC (permalink / raw) To: James Bottomley Cc: Theodore Y. Ts'o, Chuck Lever, linux-integrity, Mimi Zohar On Tue, Sep 17, 2019 at 5:57 PM James Bottomley <James.Bottomley@hansenpartnership.com> wrote: > with. The biggest problem with fs-verity has been where to store the > merkel tree. However, what I've heard from IMA people is as long as > the merkle tree storage problem gets solved satisfactorily, they're > perfectly happy to have per page hash verification be an IMA mechanism > because it's a simple extension of policy and an addition of a gate. The way I see this is that the greatest asset to protect on any device is the user data. The data security comes first, then the device security as a mechanism to protect that same data. You could even say that the device security is worthless when the device is empty. The user data is almost always mutable by nature. So, would be really great if the fs-verity metadata storage would take it into a consideration that one day someone will want to use it for the mutable data as well, even if Google does not want at this point in time. Things like photos, videos are ideal use cases for the verity like Ted pointed out. Heck, doubt we would even have the conspiracy over the moon landings anymore if the photos were taken with a device that could reliably identify the device, the device user, location and the time when the photos were taken ;) -- Janne ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-18 5:27 ` Janne Karhunen @ 2019-09-18 12:50 ` Theodore Y. Ts'o 2019-09-18 15:52 ` James Bottomley 1 sibling, 0 replies; 18+ messages in thread From: Theodore Y. Ts'o @ 2019-09-18 12:50 UTC (permalink / raw) To: Janne Karhunen; +Cc: James Bottomley, Chuck Lever, linux-integrity, Mimi Zohar On Wed, Sep 18, 2019 at 08:27:57AM +0300, Janne Karhunen wrote: > > The way I see this is that the greatest asset to protect on any device > is the user data. The data security comes first, then the device > security as a mechanism to protect that same data. You could even say > that the device security is worthless when the device is empty. The > user data is almost always mutable by nature. So, would be really > great if the fs-verity metadata storage would take it into a > consideration that one day someone will want to use it for the mutable > data as well, even if Google does not want at this point in time. > Things like photos, videos are ideal use cases for the verity like Ted > pointed out. Writable data is tricky, and has several problems. The first is performance; the block which gets modified and the cryptographic checksum has to be updated atomically; and if you use a Merkle tree, you have to update multiple blocks in the Merkle tree atomically. The journalling necessary to address this is really tricky. The other problem is if you want to update authenticated checksums while blocks are being updated, this raises the question about how do you secure the key used to sign the checksums? For read-only data, the private signing key can be stored off the device. For example, it can be stored on an off-line build server for a Red Hat package. Or Google can store the signing key for APK's on a secure server inside their data centers before the package is made available on Google Play Download servers. For mutable data, the signing key needs to be stored on the mobile device --- so the value is significantly decreased. After all, if a malicious attacker can take over the phone in order to modify a photo, the malicious attacker can also steal the signing key off the mobile device and then modify the photo. There are solutions for these problems. For example, one could use a storage device with 4128 byte sectors, with an inline encryption engine (ICE) which uses AES-GCM (an Authenticated Encryption with Associated Data mode), with the key stored in a secure enclave and where the host OS authenticates to the secure enclave, and then the secure enclave delivers the key to the ICE without the key ever touching the general purpose CPU. But in general, they all require a lot of custom hardware. And I'm not making any comments about any future product features in Android, but it's safe to say that security architects have been talking about such designs for quite some time. The challenge has always been balancing the cost / benefit tradeoffs, and whether customers are willing to pay what it costs, either in $$$ or performance, to get that level of security. Cheers, - Ted ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-18 5:27 ` Janne Karhunen 2019-09-18 12:50 ` Theodore Y. Ts'o @ 2019-09-18 15:52 ` James Bottomley 2019-09-19 6:47 ` Janne Karhunen 1 sibling, 1 reply; 18+ messages in thread From: James Bottomley @ 2019-09-18 15:52 UTC (permalink / raw) To: Janne Karhunen Cc: Theodore Y. Ts'o, Chuck Lever, linux-integrity, Mimi Zohar On Wed, 2019-09-18 at 08:27 +0300, Janne Karhunen wrote: > On Tue, Sep 17, 2019 at 5:57 PM James Bottomley > <James.Bottomley@hansenpartnership.com> wrote: > > > with. The biggest problem with fs-verity has been where to store > > the merkel tree. However, what I've heard from IMA people is as > > long as the merkle tree storage problem gets solved satisfactorily, > > they're perfectly happy to have per page hash verification be an > > IMA mechanism because it's a simple extension of policy and an > > addition of a gate. > > The way I see this is that the greatest asset to protect on any > device is the user data. The data security comes first, then the > device security as a mechanism to protect that same data. You could > even say that the device security is worthless when the device is > empty. The user data is almost always mutable by nature. So, would be > really great if the fs-verity metadata storage would take it into a > consideration that one day someone will want to use it for the > mutable data as well, even if Google does not want at this point in > time. Things like photos, videos are ideal use cases for the verity > like Ted pointed out. Mutability for integrity checked executables/data is problematic. With IMA you have to update the file and the xattr and make sure nothing touches it before you've completed all the updates otherwise you get an integrity check failure. This can work if your mutation is simply a distro update, but it's really hard to do if the file is constantly undergoing mutation because the window where the integrity check fails is huge ... thus it depends on your use case for mutability. James > Heck, doubt we would even have the conspiracy over the moon landings > anymore if the photos were taken with a device that could reliably > identify the device, the device user, location and the time when the > photos were taken ;) > > > -- > Janne > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-18 15:52 ` James Bottomley @ 2019-09-19 6:47 ` Janne Karhunen 0 siblings, 0 replies; 18+ messages in thread From: Janne Karhunen @ 2019-09-19 6:47 UTC (permalink / raw) To: James Bottomley Cc: Theodore Y. Ts'o, Chuck Lever, linux-integrity, Mimi Zohar On Wed, Sep 18, 2019 at 6:53 PM James Bottomley <James.Bottomley@hansenpartnership.com> wrote: > Mutability for integrity checked executables/data is problematic. With > IMA you have to update the file and the xattr and make sure nothing > touches it before you've completed all the updates otherwise you get an > integrity check failure. This can work if your mutation is simply a > distro update, but it's really hard to do if the file is constantly > undergoing mutation because the window where the integrity check fails > is huge ... thus it depends on your use case for mutability. Right, but try those patches I posted. They mostly address this, even android mmap'd databases seem to stay pretty well up to date and the performance is not entirely destroyed either if tuned a bit. -- Janne ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-17 14:56 ` James Bottomley 2019-09-18 5:27 ` Janne Karhunen @ 2019-09-18 12:37 ` Theodore Y. Ts'o 2019-09-18 14:40 ` Mimi Zohar 2019-09-18 15:49 ` James Bottomley 1 sibling, 2 replies; 18+ messages in thread From: Theodore Y. Ts'o @ 2019-09-18 12:37 UTC (permalink / raw) To: James Bottomley; +Cc: Janne Karhunen, Chuck Lever, linux-integrity On Tue, Sep 17, 2019 at 09:56:09AM -0500, James Bottomley wrote: > > There seems to be a philosophical debate about this. Some IMA folks > > have claimed that you want to know at the time of the binary being > > executed, whether or not it is corrupt or not. Their concern is that > > if you can make a binary crash when it pages in some page of memory,,,, > > That's not my recollection of the IMA position. I had *several* conversations with IMA folks, including Mimi, who very carefully explained to me why fs-verity was bad from a security perspective. We also had security experts inside Google who said it was fine, so it's fair to say that there is no unaminimity on this issue. (Put 4 security experts in a room, and watch 5 different positions develop). > I'd more say the Linux Kernel itself, at least for executables, has > mechanisms to ensure open files aren't updated because that ends up > causing potential I/D cache incoherence with resulting binary crashes, > so for executables, IMA relies on this mechanism. But that's only true because the disk is inside the TCB. If you can modify the disk while file is open --- for example, if the NSA has created trojaned HDD firmware --- then assuming that the kernel can ensure that the file can't be modified while it's being executed is completely false. Consider what happens if the file is stored on iSCSI, where either (a) there is no protection of the network connection, or (b) the iSCSI device is under the control of the malicious attacker. > I'd also be wary of pushing a merkle tree as a solution to the TOC/TOU > problem because, while it's true you can appraise a page at a time > using the lowest layer hash, unless you do a full merkle tree check on > that hash every time, you're still vulnerable to the attacker > corrupting both the hash and the page (especially as the merkle tree > can be so huge, requiring that it be paged too). If the page containing the lowest layer hash is in memory (and memory is considered part of the TCB), then we don't need to do the full merkle tree check. If the lowest layer hash has been pushed out of memory, then it has to be read from the storage device, but we might not still have to do a full merkle tree check. We only have to do a check up to the merkle tree block which is still in memory. In practice, both for dm-verity and fs-verity, the upper layers of the Merkle tree are will tend to be kept in memory because we can touch them as "accessed" even though we didn't need to actually recompute the hash of tree nodes which are still in memory. So the Merkle tree *does* solve the TOC/TOU issue, so long as we are careful and explicit about what are trust assumptions are. And certainly treating the memory as being in the TCB is fairly uncontroversial thing to do! Ted ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-18 12:37 ` Theodore Y. Ts'o @ 2019-09-18 14:40 ` Mimi Zohar 2019-09-18 15:49 ` James Bottomley 1 sibling, 0 replies; 18+ messages in thread From: Mimi Zohar @ 2019-09-18 14:40 UTC (permalink / raw) To: Theodore Y. Ts'o, James Bottomley Cc: Janne Karhunen, Chuck Lever, linux-integrity On Wed, 2019-09-18 at 08:37 -0400, Theodore Y. Ts'o wrote: > On Tue, Sep 17, 2019 at 09:56:09AM -0500, James Bottomley wrote: > > > There seems to be a philosophical debate about this. Some IMA folks > > > have claimed that you want to know at the time of the binary being > > > executed, whether or not it is corrupt or not. Their concern is that > > > if you can make a binary crash when it pages in some page of memory,,,, > > > > That's not my recollection of the IMA position. > > I had *several* conversations with IMA folks, including Mimi, who very > carefully explained to me why fs-verity was bad from a security > perspective. There are use cases where you want to fail immediately, but that is dependent on the use case (eg. critical industrial control systems). I'm not sure why you're bringing this up now, as we've already agreed there are different use cases with different requirements, even on the same system. IMA doesn't hard code policy in the kernel, but is based on a single, centralized policy, which contains measurement, appraisal, and audit rules. The same file hash, or in this case fs-verity's hash, could be included in the measurement list, used to extended TPM, and added to the audit log, as an IMA-audit record. Mimi ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: IMA on remote file systems 2019-09-18 12:37 ` Theodore Y. Ts'o 2019-09-18 14:40 ` Mimi Zohar @ 2019-09-18 15:49 ` James Bottomley 1 sibling, 0 replies; 18+ messages in thread From: James Bottomley @ 2019-09-18 15:49 UTC (permalink / raw) To: Theodore Y. Ts'o; +Cc: Janne Karhunen, Chuck Lever, linux-integrity On Wed, 2019-09-18 at 08:37 -0400, Theodore Y. Ts'o wrote: > On Tue, Sep 17, 2019 at 09:56:09AM -0500, James Bottomley wrote: > > > There seems to be a philosophical debate about this. Some IMA > > > folks have claimed that you want to know at the time of the > > > binary being executed, whether or not it is corrupt or > > > not. Their concern is that if you can make a binary crash when > > > it pages in some page of memory,,,, > > > > That's not my recollection of the IMA position. > > I had *several* conversations with IMA folks, including Mimi, who > very carefully explained to me why fs-verity was bad from a security > perspective. We also had security experts inside Google who said it > was fine, so it's fair to say that there is no unaminimity on this > issue. (Put 4 security experts in a room, and watch 5 different > positions develop). > > > I'd more say the Linux Kernel itself, at least for executables, has > > mechanisms to ensure open files aren't updated because that ends up > > causing potential I/D cache incoherence with resulting binary > > crashes, so for executables, IMA relies on this mechanism. > > But that's only true because the disk is inside the TCB. No, it's because Linux can't let an I/D cache inconsistency develop. IMA merely piggy backed off that. > If you can modify the disk while file is open --- for example, if > the NSA has created trojaned HDD firmware The NSA isn't necessary ... root can write to the underlying disk even if it has a mounted filesystem. If you can control the system such that you can trust root, then you can start on the NSA conspiracy theories. > --- then assuming that the kernel can ensure that the file can't be > modified while it's being executed is completely false. Consider > what happens if the file is stored on iSCSI, where either (a) there > is no protection of the network connection, or (b) the iSCSI device > is under the control of the malicious attacker. As I said, there are solutions which pull remote devices into the security envelope, but there are reasons why a user might have problems, with not having the security solution available topping the list. > > I'd also be wary of pushing a merkle tree as a solution to the > > TOC/TOU problem because, while it's true you can appraise a page at > > a time using the lowest layer hash, unless you do a full merkle > > tree check on that hash every time, you're still vulnerable to the > > attacker corrupting both the hash and the page (especially as the > > merkle tree can be so huge, requiring that it be paged too). > > If the page containing the lowest layer hash is in memory (and memory > is considered part of the TCB), then we don't need to do the full > merkle tree check. Root can modify the page cache, so this is, again, an implicit assumption you can trust root. If you're on the NSA conspiracy theory then the BMC can be trojan'd and can modify memory ... > If the lowest layer hash has been pushed out of memory, then it has > to be read from the storage device, but we might not still have to do > a full merkle tree check. We only have to do a check up to the > merkle tree block which is still in memory. In practice, both for > dm-verity and fs-verity, the upper layers of the Merkle tree are will > tend to be kept in memory because we can touch them as "accessed" > even though we didn't need to actually recompute the hash of tree > nodes which are still in memory. If I've managed to push out the lower nodes, I've likely managed to push out the upper nodes too along with most of the tree, so this is rather weak protection. As I said in the prior email, I think you can overcome this by forcing a check up to the top every time you read the tree, but that's an O(logN) effort. > So the Merkle tree *does* solve the TOC/TOU issue, so long as we are > careful and explicit about what are trust assumptions are. And > certainly treating the memory as being in the TCB is fairly > uncontroversial thing to do! In my mind, it's no more or less controversial than the disk being in it, which, as I said, you have to be careful about. The general point is there is no absolute trust in anything. However, customers are often willing to trust some things, with different customers willing to trust different things, so having a configurable policy here is really the required thing. James ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2019-09-19 6:48 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-28 17:36 IMA on remote file systems Chuck Lever 2019-09-13 14:50 ` Chuck Lever 2019-09-15 21:42 ` Mimi Zohar 2019-09-16 16:10 ` Theodore Y. Ts'o 2019-09-16 18:16 ` Chuck Lever 2019-09-16 13:16 ` Janne Karhunen 2019-09-16 14:47 ` Chuck Lever 2019-09-17 6:30 ` Janne Karhunen 2019-09-17 12:45 ` Theodore Y. Ts'o 2019-09-17 14:18 ` Mimi Zohar 2019-09-17 14:56 ` James Bottomley 2019-09-18 5:27 ` Janne Karhunen 2019-09-18 12:50 ` Theodore Y. Ts'o 2019-09-18 15:52 ` James Bottomley 2019-09-19 6:47 ` Janne Karhunen 2019-09-18 12:37 ` Theodore Y. Ts'o 2019-09-18 14:40 ` Mimi Zohar 2019-09-18 15:49 ` James Bottomley
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).