[LSF/MM TOPIC] Memory Encryption on top of filesystems

[LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dave Hansen]

Multi-Key Total Memory Encryption (MKTME) [1] is feature of a memory
controller that allows memory to be selectively encrypted with
user-controlled key, in hardware, at a very low runtime cost.  However,
it is implemented using AES-XTS which encrypts each block with a key
that is generated based on the physical address of the data being
encrypted.  This has nice security properties, making some replay and
substitution attacks harder, but it means that encrypted data can not be
naively relocated.

Combined with persistent memory, MKTME allows data to be unlocked at the
device (DIMM or namespace) level, but left encrypted until it actually
needs to be used.  However, if encrypted data were placed on a
filesystem, it might be in its encrypted state for long periods of time
and could not be moved by the filesystem during that time.

The “easy” solution to this is to just require that the encryption key
be present and programmed into the memory controller before data is
moved.  However, this means that filesystems would need to know when a
given block has been encrypted and can not be moved.

We would like to discuss an early proposal for the tooling, APIs and
on-disk changes necessary to implement this feature and ensure we have
not overlooked the interactions with complementary features like
existing software-driven encryption (eCryptfs and fscrypt).

1. https://patchwork.kernel.org/cover/10592621/

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dave Chinner]

On Tue, Feb 12, 2019 at 08:55:57AM -0800, Dave Hansen wrote:
> Multi-Key Total Memory Encryption (MKTME) [1] is feature of a memory
> controller that allows memory to be selectively encrypted with
> user-controlled key, in hardware, at a very low runtime cost.  However,
> it is implemented using AES-XTS which encrypts each block with a key
> that is generated based on the physical address of the data being
> encrypted.  This has nice security properties, making some replay and
> substitution attacks harder, but it means that encrypted data can not be
> naively relocated.

The subject is "Memory Encryption on top of filesystems", but really
what you are talking about is "physical memory encryption /below/
filesystems".

i.e. it's encryption of the physical storage the filesystem manages,
not encryption within the fileystem (like fscrypt) or or user data
on top of the filesystem (ecryptfs or userspace).

> Combined with persistent memory, MKTME allows data to be unlocked at the
> device (DIMM or namespace) level, but left encrypted until it actually
> needs to be used.

This sounds more like full disk encryption (either in the IO
path software by dm-crypt or in hardware itself), where the contents
are decrypted/encrypted in the IO path as the data is moved between
physical storage and the filesystem's memory (page/buffer caches).

Is there any finer granularity than a DIMM or pmem namespace for
specifying encrypted regions? Note that filesystems are not aware of
the physical layout of the memory address space (i.e. what DIMM
corresponds to which sector in the block device), so DIMM-level
granularity doesn't seem particularly useful right now....

Also, how many different hardware encryption keys are available for
use, and how many separate memory regions can a single key have
associated with it?

> However, if encrypted data were placed on a
> filesystem, it might be in its encrypted state for long periods of time
> and could not be moved by the filesystem during that time.

I'm not sure what you mean by "if encrypted data were placed on a
filesystem", given that the memory encryption is transparent to the
filesystem (i.e. happens in the memory controller on it's way
to/from the physical storage).

> The “easy” solution to this is to just require that the encryption key
> be present and programmed into the memory controller before data is
> moved.  However, this means that filesystems would need to know when a
> given block has been encrypted and can not be moved.

I'm missing something here - how does the filesystem even get
mounted if we haven't unlocked the device the filesystem is stored
on? i.e. we need to unlock the entire memory region containing the
filesystem so it can read and write it's metadata (which can be
randomly spread all over the block device).

And if we have to do that to mount the filesystem, then aren't we
also unlocking all the same memory regions that contain user data
and hence they can be moved?

At what point do we end up with a filesystem mounted and trying to
access a locked memory region?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dan Williams]

On Tue, Feb 12, 2019 at 3:51 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Tue, Feb 12, 2019 at 08:55:57AM -0800, Dave Hansen wrote:
> > Multi-Key Total Memory Encryption (MKTME) [1] is feature of a memory
> > controller that allows memory to be selectively encrypted with
> > user-controlled key, in hardware, at a very low runtime cost.  However,
> > it is implemented using AES-XTS which encrypts each block with a key
> > that is generated based on the physical address of the data being
> > encrypted.  This has nice security properties, making some replay and
> > substitution attacks harder, but it means that encrypted data can not be
> > naively relocated.
>
> The subject is "Memory Encryption on top of filesystems", but really
> what you are talking about is "physical memory encryption /below/
> filesystems".
>
> i.e. it's encryption of the physical storage the filesystem manages,
> not encryption within the fileystem (like fscrypt) or or user data
> on top of the filesystem (ecryptfs or userspace).
>
> > Combined with persistent memory, MKTME allows data to be unlocked at the
> > device (DIMM or namespace) level, but left encrypted until it actually
> > needs to be used.
>
> This sounds more like full disk encryption (either in the IO
> path software by dm-crypt or in hardware itself), where the contents
> are decrypted/encrypted in the IO path as the data is moved between
> physical storage and the filesystem's memory (page/buffer caches).
>
> Is there any finer granularity than a DIMM or pmem namespace for
> specifying encrypted regions? Note that filesystems are not aware of
> the physical layout of the memory address space (i.e. what DIMM
> corresponds to which sector in the block device), so DIMM-level
> granularity doesn't seem particularly useful right now....
>
> Also, how many different hardware encryption keys are available for
> use, and how many separate memory regions can a single key have
> associated with it?
>
> > However, if encrypted data were placed on a
> > filesystem, it might be in its encrypted state for long periods of time
> > and could not be moved by the filesystem during that time.
>
> I'm not sure what you mean by "if encrypted data were placed on a
> filesystem", given that the memory encryption is transparent to the
> filesystem (i.e. happens in the memory controller on it's way
> to/from the physical storage).
>
> > The “easy” solution to this is to just require that the encryption key
> > be present and programmed into the memory controller before data is
> > moved.  However, this means that filesystems would need to know when a
> > given block has been encrypted and can not be moved.
>
> I'm missing something here - how does the filesystem even get
> mounted if we haven't unlocked the device the filesystem is stored
> on? i.e. we need to unlock the entire memory region containing the
> filesystem so it can read and write it's metadata (which can be
> randomly spread all over the block device).
>
> And if we have to do that to mount the filesystem, then aren't we
> also unlocking all the same memory regions that contain user data
> and hence they can be moved?

Yes, and this is the most likely scenario for enabling MKTME with
persistent memory. The filesystem will not be able to mount until the
entire physical address range (namespace device) is unlocked, and the
filesystem is kept unaware of the encryption. One key per namespace
device.

> At what point do we end up with a filesystem mounted and trying to
> access a locked memory region?

Another option is to enable encryption to be specified at mmap time
with the motivation of being able to use the file system for
provisioning instead of managing multiple namespaces. The filesystem
would need to be careful to use the key for any physical block
management, and a decision would need to be made about when/whether
read(2)/write(2) access cipher text . The current thinking is that
this would be too invasive / restrictive for the filesystem, but it's
otherwise an interesting thought experiment for allowing the
filesystem to take on more physical-storage allocation
responsibilities.

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dave Chinner]

On Tue, Feb 12, 2019 at 04:27:20PM -0800, Dan Williams wrote:
> On Tue, Feb 12, 2019 at 3:51 PM Dave Chinner <david@fromorbit.com> wrote:
> >
> > On Tue, Feb 12, 2019 at 08:55:57AM -0800, Dave Hansen wrote:
> > > Multi-Key Total Memory Encryption (MKTME) [1] is feature of a memory
> > > controller that allows memory to be selectively encrypted with
> > > user-controlled key, in hardware, at a very low runtime cost.  However,
> > > it is implemented using AES-XTS which encrypts each block with a key
> > > that is generated based on the physical address of the data being
> > > encrypted.  This has nice security properties, making some replay and
> > > substitution attacks harder, but it means that encrypted data can not be
> > > naively relocated.
> >
> > The subject is "Memory Encryption on top of filesystems", but really
> > what you are talking about is "physical memory encryption /below/
> > filesystems".
> >
> > i.e. it's encryption of the physical storage the filesystem manages,
> > not encryption within the fileystem (like fscrypt) or or user data
> > on top of the filesystem (ecryptfs or userspace).
> >
> > > Combined with persistent memory, MKTME allows data to be unlocked at the
> > > device (DIMM or namespace) level, but left encrypted until it actually
> > > needs to be used.
> >
> > This sounds more like full disk encryption (either in the IO
> > path software by dm-crypt or in hardware itself), where the contents
> > are decrypted/encrypted in the IO path as the data is moved between
> > physical storage and the filesystem's memory (page/buffer caches).
> >
> > Is there any finer granularity than a DIMM or pmem namespace for
> > specifying encrypted regions? Note that filesystems are not aware of
> > the physical layout of the memory address space (i.e. what DIMM
> > corresponds to which sector in the block device), so DIMM-level
> > granularity doesn't seem particularly useful right now....
> >
> > Also, how many different hardware encryption keys are available for
> > use, and how many separate memory regions can a single key have
> > associated with it?
> >
> > > However, if encrypted data were placed on a
> > > filesystem, it might be in its encrypted state for long periods of time
> > > and could not be moved by the filesystem during that time.
> >
> > I'm not sure what you mean by "if encrypted data were placed on a
> > filesystem", given that the memory encryption is transparent to the
> > filesystem (i.e. happens in the memory controller on it's way
> > to/from the physical storage).
> >
> > > The “easy” solution to this is to just require that the encryption key
> > > be present and programmed into the memory controller before data is
> > > moved.  However, this means that filesystems would need to know when a
> > > given block has been encrypted and can not be moved.
> >
> > I'm missing something here - how does the filesystem even get
> > mounted if we haven't unlocked the device the filesystem is stored
> > on? i.e. we need to unlock the entire memory region containing the
> > filesystem so it can read and write it's metadata (which can be
> > randomly spread all over the block device).
> >
> > And if we have to do that to mount the filesystem, then aren't we
> > also unlocking all the same memory regions that contain user data
> > and hence they can be moved?
> 
> Yes, and this is the most likely scenario for enabling MKTME with
> persistent memory. The filesystem will not be able to mount until the
> entire physical address range (namespace device) is unlocked, and the
> filesystem is kept unaware of the encryption. One key per namespace
> device.
> 
> > At what point do we end up with a filesystem mounted and trying to
> > access a locked memory region?
> 
> Another option is to enable encryption to be specified at mmap time
> with the motivation of being able to use the file system for
> provisioning instead of managing multiple namespaces.

I'm assuming you are talking about DAX here, yes?

Because fscrypt....

> The filesystem
> would need to be careful to use the key for any physical block
> management, and a decision would need to be made about when/whether
> read(2)/write(2) access cipher text .

... already handles all this via page cache coherency for
mmap/read/write IO.

> The current thinking is that
> this would be too invasive / restrictive for the filesystem, but it's
> otherwise an interesting thought experiment for allowing the
> filesystem to take on more physical-storage allocation
> responsibilities.

Actually what we want in the filesystem world is /hardware offload/
abstractions in the filesystems, not "filesystem controls hardware
specific physical storage features" mechanisms.

i.e. if the filesystem/fscrypt can offload the encryption of the
data to the IO path by passing the fscrypt key/info with the IO,
then it works with everything, not just pmem.

In the case of pmem+DAX+mmap(), it needs to associate the correct
key with the memory region that is to be encrypted when it is
mmap()d. Then the DAX subsystem can associate the key with the
physical pages that are faulted during DAX access. If it's bio based
IO going to the DAX driver, then the keys should be attached to the
bio....

fscrypt encrypt/decrypt is already done at the filesystem/bio
interface layer via bounce buffers - it's not a great stretch to
push this down a layer so that it can be offloaded to the underlying
device if it is hardware encryption capable. fscrypt would really
only be used for key management (like needs work to support
arbitrary hardware keys) and in filesystem metadata encryption (e.g.
filenames) in that case....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dan Williams]

[ add Ted and Jaegeuk ]

On Tue, Feb 12, 2019 at 6:14 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Tue, Feb 12, 2019 at 04:27:20PM -0800, Dan Williams wrote:
> > On Tue, Feb 12, 2019 at 3:51 PM Dave Chinner <david@fromorbit.com> wrote:
> > >
> > > On Tue, Feb 12, 2019 at 08:55:57AM -0800, Dave Hansen wrote:
> > > > Multi-Key Total Memory Encryption (MKTME) [1] is feature of a memory
> > > > controller that allows memory to be selectively encrypted with
> > > > user-controlled key, in hardware, at a very low runtime cost.  However,
> > > > it is implemented using AES-XTS which encrypts each block with a key
> > > > that is generated based on the physical address of the data being
> > > > encrypted.  This has nice security properties, making some replay and
> > > > substitution attacks harder, but it means that encrypted data can not be
> > > > naively relocated.
> > >
> > > The subject is "Memory Encryption on top of filesystems", but really
> > > what you are talking about is "physical memory encryption /below/
> > > filesystems".
> > >
> > > i.e. it's encryption of the physical storage the filesystem manages,
> > > not encryption within the fileystem (like fscrypt) or or user data
> > > on top of the filesystem (ecryptfs or userspace).
> > >
> > > > Combined with persistent memory, MKTME allows data to be unlocked at the
> > > > device (DIMM or namespace) level, but left encrypted until it actually
> > > > needs to be used.
> > >
> > > This sounds more like full disk encryption (either in the IO
> > > path software by dm-crypt or in hardware itself), where the contents
> > > are decrypted/encrypted in the IO path as the data is moved between
> > > physical storage and the filesystem's memory (page/buffer caches).
> > >
> > > Is there any finer granularity than a DIMM or pmem namespace for
> > > specifying encrypted regions? Note that filesystems are not aware of
> > > the physical layout of the memory address space (i.e. what DIMM
> > > corresponds to which sector in the block device), so DIMM-level
> > > granularity doesn't seem particularly useful right now....
> > >
> > > Also, how many different hardware encryption keys are available for
> > > use, and how many separate memory regions can a single key have
> > > associated with it?
> > >
> > > > However, if encrypted data were placed on a
> > > > filesystem, it might be in its encrypted state for long periods of time
> > > > and could not be moved by the filesystem during that time.
> > >
> > > I'm not sure what you mean by "if encrypted data were placed on a
> > > filesystem", given that the memory encryption is transparent to the
> > > filesystem (i.e. happens in the memory controller on it's way
> > > to/from the physical storage).
> > >
> > > > The “easy” solution to this is to just require that the encryption key
> > > > be present and programmed into the memory controller before data is
> > > > moved.  However, this means that filesystems would need to know when a
> > > > given block has been encrypted and can not be moved.
> > >
> > > I'm missing something here - how does the filesystem even get
> > > mounted if we haven't unlocked the device the filesystem is stored
> > > on? i.e. we need to unlock the entire memory region containing the
> > > filesystem so it can read and write it's metadata (which can be
> > > randomly spread all over the block device).
> > >
> > > And if we have to do that to mount the filesystem, then aren't we
> > > also unlocking all the same memory regions that contain user data
> > > and hence they can be moved?
> >
> > Yes, and this is the most likely scenario for enabling MKTME with
> > persistent memory. The filesystem will not be able to mount until the
> > entire physical address range (namespace device) is unlocked, and the
> > filesystem is kept unaware of the encryption. One key per namespace
> > device.
> >
> > > At what point do we end up with a filesystem mounted and trying to
> > > access a locked memory region?
> >
> > Another option is to enable encryption to be specified at mmap time
> > with the motivation of being able to use the file system for
> > provisioning instead of managing multiple namespaces.
>
> I'm assuming you are talking about DAX here, yes?
>
> Because fscrypt....
>
> > The filesystem
> > would need to be careful to use the key for any physical block
> > management, and a decision would need to be made about when/whether
> > read(2)/write(2) access cipher text .
>
> ... already handles all this via page cache coherency for
> mmap/read/write IO.

Oh!

/me checks

It handles mmap coherency by making the page cache be clear text, but
perhaps in the DAX case we can make it be coherent cipher text through
both paths.

> > The current thinking is that
> > this would be too invasive / restrictive for the filesystem, but it's
> > otherwise an interesting thought experiment for allowing the
> > filesystem to take on more physical-storage allocation
> > responsibilities.
>
> Actually what we want in the filesystem world is /hardware offload/
> abstractions in the filesystems, not "filesystem controls hardware
> specific physical storage features" mechanisms.
>
> i.e. if the filesystem/fscrypt can offload the encryption of the
> data to the IO path by passing the fscrypt key/info with the IO,
> then it works with everything, not just pmem.
>
> In the case of pmem+DAX+mmap(), it needs to associate the correct
> key with the memory region that is to be encrypted when it is
> mmap()d. Then the DAX subsystem can associate the key with the
> physical pages that are faulted during DAX access. If it's bio based
> IO going to the DAX driver, then the keys should be attached to the
> bio....
>
> fscrypt encrypt/decrypt is already done at the filesystem/bio
> interface layer via bounce buffers - it's not a great stretch to
> push this down a layer so that it can be offloaded to the underlying
> device if it is hardware encryption capable. fscrypt would really
> only be used for key management (like needs work to support
> arbitrary hardware keys) and in filesystem metadata encryption (e.g.
> filenames) in that case....

Thanks, yes, fscrypt needs a closer look. As far I can see at a quick
glance fscrypt has the same physical block inputs for the encryption
algorithm as MKTME so it seems it could be crafted as a drop in
accelerator for fscrypt for pmem block devices.

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Theodore Y. Ts'o]

On Tue, Feb 12, 2019 at 07:31:37PM -0800, Dan Williams wrote:
> > fscrypt encrypt/decrypt is already done at the filesystem/bio
> > interface layer via bounce buffers - it's not a great stretch to
> > push this down a layer so that it can be offloaded to the underlying
> > device if it is hardware encryption capable. fscrypt would really
> > only be used for key management (like needs work to support
> > arbitrary hardware keys) and in filesystem metadata encryption (e.g.
> > filenames) in that case....
> 
> Thanks, yes, fscrypt needs a closer look. As far I can see at a quick
> glance fscrypt has the same physical block inputs for the encryption
> algorithm as MKTME so it seems it could be crafted as a drop in
> accelerator for fscrypt for pmem block devices.

Yes, and in fact this would solve another problem that is currently
being solved in an adhoc fashion, and that's where the
encryption/decryption engine is located between the storage device and
the DMA controller.  This is called an Inline Crypto Engine (ICE), and
it's been done on many mobile devices.

For fscrypt, we want to use to use a different key for each file,
where the per-file key is derived from the per-user key.  (This allows
mutually suspicious users to share a single Chrome OS device; so for
example, if Alice is logged in, she can access her files, but she
willt not be able to access Bob's file.)

So that means we need to pass a key selector (a small integer) into
the struct bio, so that it can be passed to encryption engine.  And we
will need an interface that can either be plumbed through the block
layer or through the memory co ntroller where we can upload a key to
the controller, and get back a key selector which can be passed to an
I/O request.   (Or mapped to a persistent memory range.)

The other important question is how to generate the per-block
Initialization Vector (IV).  There are two ways of doing this; one is
to base the IV on the logical block #, and the other is to base the IV
based on the physical block #.  Normally, fscrypt uses a logical block
number; this allows a file to be defragged, or for the block to be
moved (which is often required for log-structured file systems).  For
the ICE use case, we can't read or write the block without the data
passing through the IV, and it's painful to figure out how to get
logical block number to the ICE, so what the mobile handsets have been
doing is to let the ICE deal with generating the IV, since it has
access to the LBA number since it's located between the flash device
and the DMA engine.

I would imagine that the simple thing to do here for persistent memory
is to base the IV on the persistent memory address for the page.

Cheers,

						- Ted

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dave Hansen]

On 2/12/19 7:31 PM, Dan Williams wrote:
> Thanks, yes, fscrypt needs a closer look. As far I can see at a quick
> glance fscrypt has the same physical block inputs for the encryption
> algorithm as MKTME so it seems it could be crafted as a drop in
> accelerator for fscrypt for pmem block devices.

One bummer is that we have the platform tweak offsets to worry about.
As far as I know, those are opaque to software and practically prevent
us from replicating the MKTME hardware's encryption/decryption in software.

Unless we can get around that, I think it rules out being a drop-in
replacement for any software-driven encryption.

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dave Chinner]

On Wed, Feb 13, 2019 at 07:51:12AM -0800, Dave Hansen wrote:
> On 2/12/19 7:31 PM, Dan Williams wrote:
> > Thanks, yes, fscrypt needs a closer look. As far I can see at a quick
> > glance fscrypt has the same physical block inputs for the encryption
> > algorithm as MKTME so it seems it could be crafted as a drop in
> > accelerator for fscrypt for pmem block devices.
> 
> One bummer is that we have the platform tweak offsets to worry about.

What's a "platform tweak offset"?

> As far as I know, those are opaque to software and practically prevent
> us from replicating the MKTME hardware's encryption/decryption in software.

We're not trying to replicate the encryption in software, just use
the existing software to manage the keys that get fed to the
hardware so it can do the encrypt/decrypt operations as the data
passes through it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM TOPIC] Memory Encryption on top of filesystems

[Dave Hansen]

On 2/13/19 12:21 PM, Dave Chinner wrote:
> On Wed, Feb 13, 2019 at 07:51:12AM -0800, Dave Hansen wrote:
>> On 2/12/19 7:31 PM, Dan Williams wrote:
>>> Thanks, yes, fscrypt needs a closer look. As far I can see at a quick
>>> glance fscrypt has the same physical block inputs for the encryption
>>> algorithm as MKTME so it seems it could be crafted as a drop in
>>> accelerator for fscrypt for pmem block devices.
>>
>> One bummer is that we have the platform tweak offsets to worry about.
> 
> What's a "platform tweak offset"?

AES-XTS uses a "tweak key" that is typically generated from the physical
address of the data being encrypted.  This mitigates block-relocation
attacks.

However, in a real server, the physical address of an NVDIMM might
change due to a bunch of things, like a PCI card or memory getting added
or removed.  The platform tweak offsets allow the physical address that
actually goes into generating the tweak key to be adjusted.  This can
keep the tweak key for a physical block constant even if the block moves
around in the address space.

>> As far as I know, those are opaque to software and practically prevent
>> us from replicating the MKTME hardware's encryption/decryption in software.
> 
> We're not trying to replicate the encryption in software, just use
> the existing software to manage the keys that get fed to the
> hardware so it can do the encrypt/decrypt operations as the data
> passes through it.

OK, managing the keys alone sounds sane.  I really need to do some
fscrypt homework to see how it manages keys.