[dm-crypt] Some questions/clarifications around the LUKS spec

[dm-crypt] Some questions/clarifications around the LUKS spec

[Daniel P. Berrange]

I'm working on a QEMU native implementation of the LUKS specification
and in review of that, some questions came up about the LUKS spec.

Firstly in Appendix B of the LUKS on disk specification, there are
tables which list the valid cipher names, cipher modes and hash specs.
Although not explicitly said, it appears to imply that a compliant
implementation should not allow other unspecified cipher names/modes or
hashes to be used.

Looking at the dm-crypt kernel module and cryptsetup tools though, it
appears that in practice the reference implementation allows any cipher
name/mode and hash that exists in the Linux kernel crypto subsystem to
be used.

Assuming that is correct, should the spec just be saying that the Linux
crypto subsystem defines the canonical list of valid cipher names/modes
& hashes and not imply that it is restricted to a smaller whitelist ?


The second clarification is around alignment of key material and payload.
The LUKS spec gives an algorithm for calculating the offsets of the key
material and payload, and then goes to say these values are only written
/ cached into the header for safety when reading, implying that apps could
just calculate them from first principles and sanity check against the header.

The current cryptsetup code though no longer follows the approach shown in
the spec, instead ensuring each key material section is aligned to 4k and
the payload starts at 1 MB. So the offsets in the header now *must* be
treated as the canonical data source and never calculated again from first
principles using the method shown in the spec. The changes in cryptsetup
make sense, so there's no real problem here - just something in the spec
that should be clarified to be less misleading IMHO.


One final thing is a non-obvious aspect of the ESSIV usage in LUKS, in
that the key size used in the ESSIV encryption, is not neccessarily the
same as the key sized used for the payload encryption. The key size used
for ESSIV is indirectly determined by the size of the hash algorithm
digest. This is probably something that ought to be called out in the
spec as its not entirely obvious at first sight.


This all triggers the last question which is where is the source for the
spec document ? From the styling it appears to be written in Latex originally
and periodically updated by various people but I don't see any source for
the PDF in git. So how/where should people submit patches for it ?

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [dm-crypt] Some questions/clarifications around the LUKS spec

[Sven Eschenberg]

Hi Daniel,

Am 14.03.2016 um 16:21 schrieb Daniel P. Berrange:
> I'm working on a QEMU native implementation of the LUKS specification
> and in review of that, some questions came up about the LUKS spec.
>
> Firstly in Appendix B of the LUKS on disk specification, there are
> tables which list the valid cipher names, cipher modes and hash specs.
> Although not explicitly said, it appears to imply that a compliant
> implementation should not allow other unspecified cipher names/modes or
> hashes to be used.

Yes, that is what the 10 year old on disk specification suggests. It 
does even explain why this list is used/present. LUKS is not necessarily 
limited to the dm-crypt kernel ciphers and modes but can possibly use 
any crypto engine (which possibly has different naming schemes).

>
> Looking at the dm-crypt kernel module and cryptsetup tools though, it
> appears that in practice the reference implementation allows any cipher
> name/mode and hash that exists in the Linux kernel crypto subsystem to
> be used.

Indeed, even though 10 years passed, all that is supported by the 
current LUKS implementation (cryptsetup) is what is supported by 
kernel's dmcrypt. You could say it became 'best practise' to use 
kernel's naming scheme and all it's supported ciphers. However, if a 
different crypto engine (I.E. a HW device) is used, there's no guarantee 
any of these ciphers/modes work. Personally I'd consider app. B a list 
of absolut minimum requirements for every crypto engine, while the user 
should be aware that with EVERY cipher+mode not listed, he/she is up to 
his/her own devices when moving to a different crypto engine.

>
> Assuming that is correct, should the spec just be saying that the Linux
> crypto subsystem defines the canonical list of valid cipher names/modes
> & hashes and not imply that it is restricted to a smaller whitelist ?

That is a good question. Personally I'd say a more thorough list could 
be added to a v2 on disk format. For v1 assuming that all ciphers and 
modes from the kernel are supported seems a bad idea. Just imagine that 
some rather obscure cipher shows up and disappears at a later time - 
What is the canocical list then? And the list of ciphers/modes on older 
kernels is way smaller aswell.

On a sidenote: When cryptsetup switched it's defaults to aes-xts it 
stopped being compliant to the (original) LUKS specs. No doubt.

>
>
> The second clarification is around alignment of key material and payload.
> The LUKS spec gives an algorithm for calculating the offsets of the key
> material and payload, and then goes to say these values are only written
> / cached into the header for safety when reading, implying that apps could
> just calculate them from first principles and sanity check against the header.
>
> The current cryptsetup code though no longer follows the approach shown in
> the spec, instead ensuring each key material section is aligned to 4k and
> the payload starts at 1 MB. So the offsets in the header now *must* be
> treated as the canonical data source and never calculated again from first
> principles using the method shown in the spec. The changes in cryptsetup
> make sense, so there's no real problem here - just something in the spec
> that should be clarified to be less misleading IMHO.
>

Again cryptsetup gave up on the specs, as there was a severe need for 
these changes. It really had been wise to ditch the original on disk 
format at that time and create a new revised (sub)version of the on disk 
format. It did not happen though and currently the reference 
implementation is the 'de facto' specification, while the original spec 
is only the 'de jure' specification. But then again, 'industry' and 
'practice' define standards, don't they? *SCNR*

>
> One final thing is a non-obvious aspect of the ESSIV usage in LUKS, in
> that the key size used in the ESSIV encryption, is not neccessarily the
> same as the key sized used for the payload encryption. The key size used
> for ESSIV is indirectly determined by the size of the hash algorithm
> digest. This is probably something that ought to be called out in the
> spec as its not entirely obvious at first sight.
>
>
> This all triggers the last question which is where is the source for the
> spec document ? From the styling it appears to be written in Latex originally
> and periodically updated by various people but I don't see any source for
> the PDF in git. So how/where should people submit patches for it ?

While the original on disk spec was certainly typeset in LaTeX, I am not 
sure if Clemens ever released the source to the public and if Milan 
actually created the 'updated' PDF from the source. He should know though.

>
> Regards,
> Daniel
>

Regards

-Sven

Re: [dm-crypt] Some questions/clarifications around the LUKS spec

[Milan Broz]

On 03/14/2016 07:58 PM, Sven Eschenberg wrote:

...
> On a sidenote: When cryptsetup switched it's defaults to aes-xts it 
> stopped being compliant to the (original) LUKS specs. No doubt.
...
> Again cryptsetup gave up on the specs, as there was a severe need for 
> these changes. It really had been wise to ditch the original on disk 
> format at that time and create a new revised (sub)version of the on disk 
> format. It did not happen though and currently the reference 
> implementation is the 'de facto' specification, while the original spec 
> is only the 'de jure' specification. But then again, 'industry' and 
> 'practice' define standards, don't they? *SCNR*

Sorry? Spec is updated, it mentions XTS mode, and disk format remains still the same,
despite I did last update in 2011. (Simply because there was no need to update
it.) I'll mention more in another reply.

Cryptsetup never gave up specs, it is still compatible to the document.

Milan

Re: [dm-crypt] Some questions/clarifications around the LUKS spec

[Milan Broz]

Hi Dan,

On 03/14/2016 04:21 PM, Daniel P. Berrange wrote:

> Firstly in Appendix B of the LUKS on disk specification, there are
> tables which list the valid cipher names, cipher modes and hash specs.
> Although not explicitly said, it appears to imply that a compliant
> implementation should not allow other unspecified cipher names/modes or
> hashes to be used.

No, it should not imply that. These modes are modes that makes sense to use.
(In the time this was written these were the only usable modes.
So the wording is not the best probably there.)

Cryptsetup (upstream) always use "secure" defaults (from this list) but
will not limit user to use anything else (even if not secure).

(If you implement your block cipher or mode in kernel, cryptsetup will allow you
you use it and it is not against the specs.)

> Looking at the dm-crypt kernel module and cryptsetup tools though, it
> appears that in practice the reference implementation allows any cipher
> name/mode and hash that exists in the Linux kernel crypto subsystem to
> be used.

Yes.

> Assuming that is correct, should the spec just be saying that the Linux
> crypto subsystem defines the canonical list of valid cipher names/modes
> & hashes and not imply that it is restricted to a smaller whitelist ?

It is not restricted, specs just defines the-best use list.

The whole logic is that cryptsetup will never stop you to use "insecure",
combination if it is technically possible. There can be reasons to use it (a lot
of old systems are not secure - but we want to map/use them with cryptsetup).

> The second clarification is around alignment of key material and payload.
> The LUKS spec gives an algorithm for calculating the offsets of the key
> material and payload, and then goes to say these values are only written
> / cached into the header for safety when reading, implying that apps could
> just calculate them from first principles and sanity check against the header.
> 
> The current cryptsetup code though no longer follows the approach shown in
> the spec, instead ensuring each key material section is aligned to 4k and
> the payload starts at 1 MB. So the offsets in the header now *must* be
> treated as the canonical data source and never calculated again from first
> principles using the method shown in the spec. The changes in cryptsetup
> make sense, so there's no real problem here - just something in the spec
> that should be clarified to be less misleading IMHO.

IIRC LUKS header was always primary source of keyslot offset, from the first release
of cryptsetup I have ever seen (btw historic versions are in archive on kernel.org).

What exact algorithm in spec you mean (Figure 3? it is about LUKS format, not unlocking.)

Application should not compute offsets, just it should do some sanity checks.
(Unfortunately LUKS1 does not have any checksum to check that header is randomly corrupted,
this will be hopefully fixed in new format - but you have to expect even
intentionally malformed input anyway.)

Alignment to 4k is enforced, alignment to 1MB was changed later (and used can change it)
because of storage industry uses this as safe default (it is safe alignment in almost all storage
stacks including SSD, RAID chunks etc - the same applies for fdisk MBR, GPT etc)
I do not see problem here - it is payLoadeOffset in header.

> One final thing is a non-obvious aspect of the ESSIV usage in LUKS, in
> that the key size used in the ESSIV encryption, is not neccessarily the
> same as the key sized used for the payload encryption. The key size used
> for ESSIV is indirectly determined by the size of the hash algorithm
> digest. This is probably something that ought to be called out in the
> spec as its not entirely obvious at first sight.

Not sure if I understand - the key for ESSIV is hash of encryption key,
if you mean that, then yes.

I think ESSIV was not meant to be part of LUKS spec, but there was need to develop
safe IV for CBC mode.

All available / supported IVs in dmcrypt are here:
https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt
 
(I should add more formal description, I have it somewhere already.)

> This all triggers the last question which is where is the source for the
> spec document ? From the styling it appears to be written in Latex originally
> and periodically updated by various people but I don't see any source for
> the PDF in git. So how/where should people submit patches for it ?

It's hidden in layers of dust ;-)
Just joking, current version with BibTeX sources is here (just it is in wiki git): 
  https://gitlab.com/cryptsetup/cryptsetup/wikis/LUKS-standard/on-disk-format.tex
  https://gitlab.com/cryptsetup/cryptsetup/wikis/LUKS-standard/on-disk-format.bib

Output document (linked from homepage) is there as well
  https://gitlab.com/cryptsetup/cryptsetup/wikis/LUKS-standard/on-disk-format.pdf

If you have some patches, create issue on gitlab or github, send it to me or to the list,
whatever fits better.

Milan

Re: [dm-crypt] Some questions/clarifications around the LUKS spec

[Sven Eschenberg]

Am 14.03.2016 um 20:54 schrieb Milan Broz:
> On 03/14/2016 07:58 PM, Sven Eschenberg wrote:
>
> ...
>> On a sidenote: When cryptsetup switched it's defaults to aes-xts it
>> stopped being compliant to the (original) LUKS specs. No doubt.
> ...
>> Again cryptsetup gave up on the specs, as there was a severe need for
>> these changes. It really had been wise to ditch the original on disk
>> format at that time and create a new revised (sub)version of the on disk
>> format. It did not happen though and currently the reference
>> implementation is the 'de facto' specification, while the original spec
>> is only the 'de jure' specification. But then again, 'industry' and
>> 'practice' define standards, don't they? *SCNR*
>
> Sorry? Spec is updated, it mentions XTS mode, and disk format remains still the same,
> despite I did last update in 2011. (Simply because there was no need to update
> it.) I'll mention more in another reply.

Updating a spec needs more than just mentioning something. Esp. changes 
may not be incompatible to previous revisions. If changes are 
incompatible, a new version is needed (instead of a simple revision). A 
change to the list of valid values as well as the change in offset 
calculation to meet alignment requirements are indeed incompatible to 
the original specification for the v1 header, like it or not. Thus, by 
introducing these changes, a new version of the on disk format was 
introduced and this should have been reflected by reversioning the 
header as well. Having multiple possible specs for the same 
magic+version is something one really should not go for.

>
> Cryptsetup never gave up specs, it is still compatible to the document.
>
> Milan

Interesting *raising eyebrow*

-Sven

Re: [dm-crypt] Some questions/clarifications around the LUKS spec

[Milan Broz]

On 03/14/2016 10:24 PM, Sven Eschenberg wrote:

> Updating a spec needs more than just mentioning something. Esp. changes 
> may not be incompatible to previous revisions. If changes are 
> incompatible, a new version is needed (instead of a simple revision). A 
> change to the list of valid values as well as the change in offset 
> calculation to meet alignment requirements are indeed incompatible to 
> the original specification for the v1 header, like it or not. Thus, by 
> introducing these changes, a new version of the on disk format was 
> introduced and this should have been reflected by reversioning the 
> header as well. Having multiple possible specs for the same 
> magic+version is something one really should not go for.

On-disk format should be backward compatible since cryptsetup 1.0.1,
no change in version is needed.
(But there were bugs - so nobody should use such old versions.)

Algorithm support is always dynamic thing (you can blacklist kernel
module, run in FIPS mode that allows only NIST friendly algorithms...)
So "mandatory" list for LUKS does not make sense in reality.

Offset calculation for keyslot is the same as well ... but reading
that pseudo-algorithm in spec - the slot alignment to 4k diverged
in 1.0 -> 1.0.1 (2005). Clemens probably forgot to update spec here,
so this is IMHO bug in spec (and I missed this).

(Cryptsetup can still open old sector-aligned version - despite this version
was never in any distro.)
 
User data alignment was always read from header, it was never calculated
and I think spec expect it this way.

Milan

Re: [dm-crypt] Some questions/clarifications around the LUKS spec

[Daniel P. Berrange]

On Mon, Mar 14, 2016 at 09:31:36PM +0100, Milan Broz wrote:
> Hi Dan,
> 
> On 03/14/2016 04:21 PM, Daniel P. Berrange wrote:
> 
> > Firstly in Appendix B of the LUKS on disk specification, there are
> > tables which list the valid cipher names, cipher modes and hash specs.
> > Although not explicitly said, it appears to imply that a compliant
> > implementation should not allow other unspecified cipher names/modes or
> > hashes to be used.
> 
> No, it should not imply that. These modes are modes that makes sense to use.
> (In the time this was written these were the only usable modes.
> So the wording is not the best probably there.)
> 
> Cryptsetup (upstream) always use "secure" defaults (from this list) but
> will not limit user to use anything else (even if not secure).
> 
> (If you implement your block cipher or mode in kernel, cryptsetup will allow you
> you use it and it is not against the specs.)

Ok, thanks for clarifying that.

> Alignment to 4k is enforced, alignment to 1MB was changed later (and used can change it)
> because of storage industry uses this as safe default (it is safe alignment in almost all storage
> stacks including SSD, RAID chunks etc - the same applies for fdisk MBR, GPT etc)
> I do not see problem here - it is payLoadeOffset in header.

Yep, none of this is a problem - it is just query that some qemu reviewers
raised about impl vs spec, since neither the 4k alignment nor 1MB alignment
is mentioned in the spec.

> > One final thing is a non-obvious aspect of the ESSIV usage in LUKS, in
> > that the key size used in the ESSIV encryption, is not neccessarily the
> > same as the key sized used for the payload encryption. The key size used
> > for ESSIV is indirectly determined by the size of the hash algorithm
> > digest. This is probably something that ought to be called out in the
> > spec as its not entirely obvious at first sight.
> 
> Not sure if I understand - the key for ESSIV is hash of encryption key,
> if you mean that, then yes.

No, what I meant was that if you use AES-128 for payload encryption,
but have a SHA256 hash with ESSIV, then you'll need to use AES-256
in the ESSIV calculate, not AES-128, since dm-crypt picks keysize
based on the hash digest size, rather than using the same keysize as
the payload. Again not a bug, just an implementation choice by dm-crypt
for ESSIV.

> 
> I think ESSIV was not meant to be part of LUKS spec, but there was need to develop
> safe IV for CBC mode.
> 
> All available / supported IVs in dmcrypt are here:
> https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt
>  
> (I should add more formal description, I have it somewhere already.)

[snip]

> If you have some patches, create issue on gitlab or github, send it to me or to the list,
> whatever fits better.

Thanks, will look at doing that in future.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [dm-crypt] Some questions/clarifications around the LUKS spec

[Sven Eschenberg]

Am 14.03.2016 um 23:27 schrieb Milan Broz:
> On 03/14/2016 10:24 PM, Sven Eschenberg wrote:
>
>> Updating a spec needs more than just mentioning something. Esp. changes
>> may not be incompatible to previous revisions. If changes are
>> incompatible, a new version is needed (instead of a simple revision). A
>> change to the list of valid values as well as the change in offset
>> calculation to meet alignment requirements are indeed incompatible to
>> the original specification for the v1 header, like it or not. Thus, by
>> introducing these changes, a new version of the on disk format was
>> introduced and this should have been reflected by reversioning the
>> header as well. Having multiple possible specs for the same
>> magic+version is something one really should not go for.
>
> On-disk format should be backward compatible since cryptsetup 1.0.1,
> no change in version is needed.
> (But there were bugs - so nobody should use such old versions.)

Yes and no. It seems the spec purposefully decided to offer a fixed list 
(of ciphers/modes) to make alternative implementations possible. AFAIK 
none really exists (in the public domain) and since the old reference 
implementation already allowed diverging from the spec it did not really 
adhere it either. Any other implementationhonoring the spec before the 
xts updates could legitimately consider all headers with unlisted 
ciphers invalid and refuse to work (for safety). So there would no real 
harm be done, yet it would have been better to make this change clear by 
changing the version - afterall there's a 16-bit value and massive 
changes could still change the magic string for safety. (in example your 
plans for v2)

> Algorithm support is always dynamic thing (you can blacklist kernel
> module, run in FIPS mode that allows only NIST friendly algorithms...)
> So "mandatory" list for LUKS does not make sense in reality.

As I read the LUKS spec it is not limited to the stock kernel crypto 
modules, in practise no alternatives showed up so far though. Example: 
Imagine the crypto is done externally by a crypto device that is tied to 
your box by RDMA and offers a cipher which it names rijndael. So an 
implementation that sets up this device would map the name rijndael to 
aes and you could still use the encrypted device with any different 
kernel. I think limiting the list of ciphers and modes was for full 
portability, which makes sense.

>
> Offset calculation for keyslot is the same as well ... but reading
> that pseudo-algorithm in spec - the slot alignment to 4k diverged
> in 1.0 -> 1.0.1 (2005). Clemens probably forgot to update spec here,
> so this is IMHO bug in spec (and I missed this).
>
> (Cryptsetup can still open old sector-aligned version - despite this version
> was never in any distro.)

The question though is, would that still hold true the other way round 
for old cryptsetup versions (and alternative implementations) prior to 
the xts spec update?

>
> User data alignment was always read from header, it was never calculated
> and I think spec expect it this way.

I think the spec is not clear on this, it suggests the offsets are 
stored for safety. It does not explicitly state the stored values should 
take precedence, though this is one (sane) possible interpretation. It's 
never to late for an errata to clarify this ;-).

>
> Milan

Regards

-Sven

[dm-crypt] Cryptesetup and dm-crypt

[levent demir]

Hi,

I am a PhD student and works and I am working with cryptsetup and a hw
crypto engine. 

I am using an ATMEL board which is able to do AES-ECB mode for example. 

We have implemented an XTS version with all operations (XOR, GF
Multiplication, 2 AES : one for IV one for data...)

My question is can we use the page size instead of the block size of 512
bytes ?

I have read some thread it is written that it could be a security issue
but I don't agree because : 

Now to implement XTS I do as follows basically : 

I take the IV encrypt it with the second part of the key, then I compute
the 32 tweaks values in -for example- an array tweak_buf[512]. 
Then I just have to compute the XOR with plaintext. 
I do a DMA encryption with ECB of the 512 bytes. 
Again I xor the ciphered data with tweak_buf.

Then I have ciphered a whole block of 512 bytes.

Now my question, the hardware is supporting a DMA operation till 4096, 
is it not interesting to encrypt 8 blocks at a time ? 
It is exactly the same operation as encrypting each block
consecutively. 

Of course I need to pre-compute the 8 tweak_buf in a big
tweak_buf[4096].Then in only one DMA operation I will be able to encrypt
4096 bytes with ECB mode. 

I hope you understand what I mean.

Can you tell me if it is possible ? If yes can I use a previous patch, I
have seen that there are some works on it ?

Thank you. 

Re: [dm-crypt] Cryptesetup and dm-crypt

[Arno Wagner]

At the very least, it is a safety issue, as all documentation 
and all relevant help given here says block size is 512 bytes.
It is also a compatibility issue, as volumes done this way 
would only work on your specific system and could not be 
accessed and recovered anywhere else. 

My stance would be sure, shoot yourself in the foot, but do
not call the result "LUKS".

Regards,
Arno

On Wed, Apr 27, 2016 at 14:56:50 CEST, levent demir wrote:
> Hi,
> 
> I am a PhD student and works and I am working with cryptsetup and a hw
> crypto engine. 
> 
> I am using an ATMEL board which is able to do AES-ECB mode for example. 
> 
> We have implemented an XTS version with all operations (XOR, GF
> Multiplication, 2 AES : one for IV one for data...)
> 
> My question is can we use the page size instead of the block size of 512
> bytes ?
> 
> I have read some thread it is written that it could be a security issue
> but I don't agree because : 
> 
> Now to implement XTS I do as follows basically : 
> 
> I take the IV encrypt it with the second part of the key, then I compute
> the 32 tweaks values in -for example- an array tweak_buf[512]. 
> Then I just have to compute the XOR with plaintext. 
> I do a DMA encryption with ECB of the 512 bytes. 
> Again I xor the ciphered data with tweak_buf.
> 
> Then I have ciphered a whole block of 512 bytes.
> 
> Now my question, the hardware is supporting a DMA operation till 4096, 
> is it not interesting to encrypt 8 blocks at a time ? 
> It is exactly the same operation as encrypting each block
> consecutively. 
> 
> Of course I need to pre-compute the 8 tweak_buf in a big
> tweak_buf[4096].Then in only one DMA operation I will be able to encrypt
> 4096 bytes with ECB mode. 
> 
> I hope you understand what I mean.
> 
> Can you tell me if it is possible ? If yes can I use a previous patch, I
> have seen that there are some works on it ?
> 
> Thank you. 
> 
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier