[dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Laurence Darby]

Hello,

I have 2 disks currently as a RAID1, and want to encrypt it.  I've
done several tests with loopback devices, and found that I can either
create a raid1 with the raw devices, and then encrypt the /dev/md
device, eg:

losetup /dev/loop1 /tmp/a
losetup /dev/loop2 /tmp/b

mdadm --create /dev/md3 --level=1 /dev/loop1 /dev/loop2 --raid-devices=2

cryptsetup luksFormat /dev/md3
cryptsetup luksOpen /dev/md3 md3


or instead encrypt each device, and then make the 2 encrypted devices
into a raid, eg:

losetup /dev/loop3 /tmp/c
losetup /dev/loop4 /tmp/d

cryptsetup luksFormat /dev/loop3
cryptsetup luksFormat /dev/loop4

cryptsetup luksOpen /dev/loop3 data3
cryptsetup luksOpen /dev/loop4 data4

mdadm --create /dev/md4 --level=1 /dev/mapper/data3 /dev/mapper/data4
--raid-devices=2


Is there a recommended way to do this?

I've read the FAQ, especially the part about bad RAM, how single bit
flip errors get amplified - would that mean encrypting each disk
separately is safer, because a bit flip error would be only be copied
to only one device instead of both?  If that happened and a
verification test spotted it, I could remove each device in turn to
find which has the corruption, and repair it.  I am planning on running
the full set of RAM tests anyway.  Is there any reason not to encrypt
each device separately, or any other general advice here?

Thanks,
Laurence

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Jorge Fábregas]

On 07/11/2011 06:03 PM, Laurence Darby wrote:
> Is there a recommended way to do this?

Hello Laurence,

That's an interesting question:  encrypted raid1 or raid1 of encrypted
disks? That also could be phrased as "dm-crypt on top of dm-raid" or
"dm-raid on top of dm-crypt"?

I must admit  I would have never thought about a "raid1 of encrypted
disks" (seems awkward) but apparently it works.  I'm new here (and to
disk encryption at all) but here are my two cents:

# Performance
I guess from the point of view of performance (CPU-wise) , an "encrypted
RAID1" would be better as you would be only encrypting once and DM-raid
will take care of copying those bits as they are to the 2nd disk.  I
suggest you do some tests (copying large amount of data to the encrypted
disk) and measure it.

# Management
There's no doubt that an encrypted raid1 is much better (much less
commands: you just need to format once, luksOpen once, luksClose once.
one backup of the header)

# Reliability
I'm not sure about this part.  Let's see what others have to say
regarding this.

Regards,
Jorge

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Milan Broz]

On 07/12/2011 01:32 PM, Jorge Fábregas wrote:
> That's an interesting question:  encrypted raid1 or raid1 of encrypted
> disks? That also could be phrased as "dm-crypt on top of dm-raid" or
> "dm-raid on top of dm-crypt"?
> 
> I must admit  I would have never thought about a "raid1 of encrypted
> disks" (seems awkward) but apparently it works.  I'm new here (and to
> disk encryption at all) but here are my two cents:

Technically both works.

> # Performance
> I guess from the point of view of performance (CPU-wise) , an "encrypted
> RAID1" would be better as you would be only encrypting once and DM-raid
> will take care of copying those bits as they are to the 2nd disk.  I
> suggest you do some tests (copying large amount of data to the encrypted
> disk) and measure it.

This depends on kernel version and if the system is SMP/multi-cpu.
For <2.6.38 you may get better performance for raid over crypt,
for newer kernel it will be different.
(I am not saying better because there are still performance issues
with crypt over MD Raid. Depends on io pattern and if IO are issued
from different cpus or not. Like dd can be slower but threaded fs test
can have much more better performance.)

> # Management
> There's no doubt that an encrypted raid1 is much better (much less
> commands: you just need to format once, luksOpen once, luksClose once.
> one backup of the header)

yes, I would suggest crypt over MD always too.

> # Reliability
> I'm not sure about this part.  Let's see what others have to say
> regarding this.

IMHO both solutions are similar here. Some errors are propagated,
hw failure (RAM, disk) would have similar effect.

RAID is not backup. You should backup LUKS header and data anyway.

Milan

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Arno Wagner]

Hi Laurence,

for security reasons, encrypt the RAID1. Otherwise you a
re giving an attacker more data, namely everything encrypted
with two different keys.

For Performance reasons, the same. If you RAID1 encrypted disk, 
everything has to be encrypted twice, if you encrypot the 
RAID1, all data is encrypted only once, halving the effort.

On Mon, Jul 11, 2011 at 11:03:12PM +0100, Laurence Darby wrote:
> Hello,
> 
> I have 2 disks currently as a RAID1, and want to encrypt it.  I've
> done several tests with loopback devices, and found that I can either
> create a raid1 with the raw devices, and then encrypt the /dev/md
> device, eg:
> 
> losetup /dev/loop1 /tmp/a
> losetup /dev/loop2 /tmp/b
> 
> mdadm --create /dev/md3 --level=1 /dev/loop1 /dev/loop2 --raid-devices=2
> 
> cryptsetup luksFormat /dev/md3
> cryptsetup luksOpen /dev/md3 md3
> 
> 
> or instead encrypt each device, and then make the 2 encrypted devices
> into a raid, eg:
> 
> losetup /dev/loop3 /tmp/c
> losetup /dev/loop4 /tmp/d
> 
> cryptsetup luksFormat /dev/loop3
> cryptsetup luksFormat /dev/loop4
> 
> cryptsetup luksOpen /dev/loop3 data3
> cryptsetup luksOpen /dev/loop4 data4
> 
> mdadm --create /dev/md4 --level=1 /dev/mapper/data3 /dev/mapper/data4
> --raid-devices=2
> 
> 
> Is there a recommended way to do this?

See abofe, place encryption direcly below the filesystem and
above any RAID/LVM,... mechanism.
 
> I've read the FAQ, 

Very good!

> especially the part about bad RAM, how single bit
> flip errors get amplified - would that mean encrypting each disk
> separately is safer, because a bit flip error would be only be copied
> to only one device instead of both? 

Good thinking, but no. It has zero advantage having one good
and one corrupted copy, but no way to tell which is which.
If you have bit-errors, you have a najor problem. If they
get amplified, the main difference is that you may detect them
eralier, which is an advantage.

> If that happened and a
> verification test spotted it, I could remove each device in turn to
> find which has the corruption, and repair it.  

A RAID consistency check would spot it,but how would you identify
the good copy? And if the bit-error does hapen before encryption,
this does not help at all. In addition, as encryption does run mainly
in the CPU caches, bit errors are not very likely there anyways.

> I am planning on running
> the full set of RAM tests anyway.  Is there any reason not to encrypt
> each device separately, or any other general advice here?

See above. Don't worry about adapting your disk encryption to the
possibility of bit-errors. These are separate issues and should
not be mixed. 

Waht you should do with regard tobit-errors is whenever you copy 
something large or archive something large or do backups, always
do a verify that compares the data. This is the best way to
detect errors and ensure data integrity. For small writes,
bit errors are very, very unlikely, unless your system is
extremely instable.

I think I will add a section on encryption vs. RAID in the
FAQ.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Arno Wagner]

On Tue, Jul 12, 2011 at 02:10:03PM +0200, Milan Broz wrote:
> On 07/12/2011 01:32 PM, Jorge F?bregas wrote:
> > That's an interesting question:  encrypted raid1 or raid1 of encrypted
> > disks? That also could be phrased as "dm-crypt on top of dm-raid" or
> > "dm-raid on top of dm-crypt"?
> > 
> > I must admit  I would have never thought about a "raid1 of encrypted
> > disks" (seems awkward) but apparently it works.  I'm new here (and to
> > disk encryption at all) but here are my two cents:
> 
> Technically both works.

Technically, RAID and encryption are on the same layer of the
storage system and you can do arbitrary conbinations.
 
> > # Performance
> > I guess from the point of view of performance (CPU-wise) , an "encrypted
> > RAID1" would be better as you would be only encrypting once and DM-raid
> > will take care of copying those bits as they are to the 2nd disk.  I
> > suggest you do some tests (copying large amount of data to the encrypted
> > disk) and measure it.
> 
> This depends on kernel version and if the system is SMP/multi-cpu.
> For <2.6.38 you may get better performance for raid over crypt,
> for newer kernel it will be different.
> (I am not saying better because there are still performance issues
> with crypt over MD Raid. Depends on io pattern and if IO are issued
> from different cpus or not. Like dd can be slower but threaded fs test
> can have much more better performance.)
> 
> > # Management
> > There's no doubt that an encrypted raid1 is much better (much less
> > commands: you just need to format once, luksOpen once, luksClose once.
> > one backup of the header)
> 
> yes, I would suggest crypt over MD always too.

Good point. And you ned to enter your passphrase twice, exposing 
it more.

The main difference security-wise is that an attacker gets
the same data-set encrypted with two different keys when
RAIDing encrypted devices. That can be a concern. 


> > # Reliability
> > I'm not sure about this part.  Let's see what others have to say
> > regarding this.
> 
> IMHO both solutions are similar here. Some errors are propagated,
> hw failure (RAM, disk) would have similar effect.

Reliability with regard to data corryption really belongs into 
the filesystem-layer and above and into the hysocal device layer.

RAID seems to be reliability, but in fact it is redundancy 
only, and detection of errors (bit errors, unreadable sectors
and dead disks) is done below (or for bit-errors possibly
in the filesystem-layer with a checksumming FS). The RAID
cannot detect errors, it can just react to errors reported by
lower layers using the redundancy it provides. 
One exception: It can do consistency checks, but it can only 
do somethign about inconsistencioes if it is at least 3-way 
redundant by voting, i.e. >= 3-way RAID1 or RAID6. And consistency
checks are not a normal RAID operation, but rather an externally
triggered RAID maintenance operation.

> RAID is not backup. You should backup LUKS header and data anyway.

Indeed. See FAQ ;-)

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Laurence Darby]

Hi Arno, All,

Thanks very much for your comments, I'll definitely go with encrypting
the RAID.

Arno Wagner wrote:

> On Tue, Jul 12, 2011 at 02:10:03PM +0200, Milan Broz wrote:
> > On 07/12/2011 01:32 PM, Jorge F?bregas wrote:
> > > That's an interesting question:  encrypted raid1 or raid1 of
> > > encrypted disks? That also could be phrased as "dm-crypt on top
> > > of dm-raid" or "dm-raid on top of dm-crypt"?
> > > 
> > > I must admit  I would have never thought about a "raid1 of
> > > encrypted disks" (seems awkward) but apparently it works.  I'm
> > > new here (and to disk encryption at all) but here are my two
> > > cents:
> > 
> > Technically both works.
> 
> Technically, RAID and encryption are on the same layer of the
> storage system and you can do arbitrary conbinations.
>  
> > > # Performance
> > > I guess from the point of view of performance (CPU-wise) , an
> > > "encrypted RAID1" would be better as you would be only encrypting
> > > once and DM-raid will take care of copying those bits as they are
> > > to the 2nd disk.  I suggest you do some tests (copying large
> > > amount of data to the encrypted disk) and measure it.
> > 
> > This depends on kernel version and if the system is SMP/multi-cpu.
> > For <2.6.38 you may get better performance for raid over crypt,
> > for newer kernel it will be different.
> > (I am not saying better because there are still performance issues
> > with crypt over MD Raid. Depends on io pattern and if IO are issued
> > from different cpus or not. Like dd can be slower but threaded fs
> > test can have much more better performance.)
> > 

Yes, I noticed this - a single dd process has to wait for both the
encryption and then IO, so more processes are needed to fill the
pipeline.

There's not much of a performance difference anyway - I have a 2 core
machine, and found that both disks (loopback files) got encrypted at
the same time (top showed 2 kworker processes, each using 80% cpu, the
rest mostly waiting on IO.)


> > > # Management
> > > There's no doubt that an encrypted raid1 is much better (much less
> > > commands: you just need to format once, luksOpen once, luksClose
> > > once. one backup of the header)
> > 
> > yes, I would suggest crypt over MD always too.
> 
> Good point. And you ned to enter your passphrase twice, exposing 
> it more.
> 
> The main difference security-wise is that an attacker gets
> the same data-set encrypted with two different keys when
> RAIDing encrypted devices. That can be a concern. 
> 
> 


Thank you for pointing that out - it's exactly the kind of thing I
wasn't aware of but should be.


> > > # Reliability
> > > I'm not sure about this part.  Let's see what others have to say
> > > regarding this.
> > 
> > IMHO both solutions are similar here. Some errors are propagated,
> > hw failure (RAM, disk) would have similar effect.
> 
> Reliability with regard to data corryption really belongs into 
> the filesystem-layer and above and into the hysocal device layer.
> 
> RAID seems to be reliability, but in fact it is redundancy 
> only, and detection of errors (bit errors, unreadable sectors
> and dead disks) is done below (or for bit-errors possibly
> in the filesystem-layer with a checksumming FS). The RAID
> cannot detect errors, it can just react to errors reported by
> lower layers using the redundancy it provides. 
> One exception: It can do consistency checks, but it can only 
> do somethign about inconsistencioes if it is at least 3-way 
> redundant by voting, i.e. >= 3-way RAID1 or RAID6. And consistency
> checks are not a normal RAID operation, but rather an externally
> triggered RAID maintenance operation.
> 
> > RAID is not backup. You should backup LUKS header and data anyway.
> 


This RAID1 is my backup device, that I do daily rsync-snapshots to.
I'll admit, I'm not doing automated integrity checks yet, I know I
should be, but encryption is more fun :) 

I was thinking that with a RAID of encrypted devices, rather than the
encrypted RAID, that I wouldn't need a header backup - If I break one,
that mistake isn't immediately written to both drives, so I can just
reformat it, and rebuild the array.  Although, yes, just restoring a
header backup does sound a lot easier, and after what Arno said above,
it's not feasible anyway.

I'll reply to the other thread tomorrow, bedtime now.

Thanks,
Laurence

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Arno Wagner]

On Thu, Jul 14, 2011 at 09:41:58PM +1000, Roscoe wrote:
> On Thu, Jul 14, 2011 at 9:01 PM, Arno Wagner <arno@wagner.name> wrote:
> ...
> > I thing your risk model is wrong. Basically it covers attacks
> > were the attacker has access to only the storage and at the same
> > time can actually do something serious with data manipulation.
> > That is a rather unlikely scenario for disk encryption. Note that
> > for communication encryption, this is a real and valid scenario.
> 
> Given the prevalence of of iSCSI, FC and similar, I wouldn't say it's
> all that unlikely.

That would be transport security. If you have a real, externally
exposed to attackers transport device (e.g. a network cable), 
then you leave the area of storage encryption and need to do network
encryption. 
 
> I for one would quite like assurances that network block device
> providers couldn't impact my security. But, I imagine I'll have to
> wait a while longer...

If you, say, tunnel your network block device over SSH
(or some other VPN) and use, e.g. LUKS on the storage layer
in your local machine, you will be pretty secure.

But you need to realize that storage encryption and 
communication encryption are two different things and have
to be done separately. As one example, to illustrate this,
consider that for CBC, you have to have different 
upredictable, but not secret, IVs per connection, while
in disk encryption you need to have the same IV per
block and the IVs should not be publicly known. Entirely
different requirements, resulting from the different
nature of the problem.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Roscoe]

On Thu, Jul 14, 2011 at 9:01 PM, Arno Wagner <arno@wagner.name> wrote:
...
> I thing your risk model is wrong. Basically it covers attacks
> were the attacker has access to only the storage and at the same
> time can actually do something serious with data manipulation.
> That is a rather unlikely scenario for disk encryption. Note that
> for communication encryption, this is a real and valid scenario.

Given the prevalence of of iSCSI, FC and similar, I wouldn't say it's
all that unlikely.

I for one would quite like assurances that network block device
providers couldn't impact my security. But, I imagine I'll have to
wait a while longer...

-- Roscoe

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Arno Wagner]

On Thu, Jul 14, 2011 at 09:17:33AM +0300, Yaron Sheffer wrote:
> Hi Arno,
> 
> I agree that most practical considerations point towards encrypt-over-RAID.
> 
> But in fact from a security point of view, it seems to me the
> situation is reversed. Looking at RAID-over-encryption, I disagree
> that having the same plaintext encrypted over multiple keys is a
> concern with modern ciphers. 

You have to take into account that this is a non-moving target, 
i.e. disk encryption. Still, the security-loss should be small.

> The real concern with most full disk
> encryption (and dm-crypt in particular) is integrity protection: the
> ability of an attacker to change the ciphertext undetected. This
> ability is greatly hampered when the attacker needs to coordinate
> the attacks on two mirrored blocks, otherwise the two copies would
> not be consistent.

You possibly think that not having access to the RAID superblock
(as it is encrypted) will make this manipulation much harder. 
My take is that as soon as the attacker is on your device, he/she
can patch cryptsetup and then waut until you enter your passphrase.
Disk-encryption really only protects against attackers that 
only have access once and not while you are working on or opening
the device. This applies mostly to the case whre your device
is stolen. I don't agree that integrity protection has any
role in ordinary scenarios. And if you have a special scenario
where it plays a role, you should not use LUKS or dm-crypt anyways,
as it does not offer integrity protection, plain and simple.

> I haven't researched all figerprinting attacks and the interaction
> with various ways of generating IVs, so my intuition may still be
> proven wrong.

I thing your risk model is wrong. Basically it covers attacks
were the attacker has access to only the storage and at the same 
time can actually do something serious with data manipulation.
That is a rather unlikely scenario for disk encryption. Note that
for communication encryption, this is a real and valid scenario.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Encrypted Raid1 or Raid 1 of encrypted devices?

[Yaron Sheffer]

Hi Arno,

I agree that most practical considerations point towards encrypt-over-RAID.

But in fact from a security point of view, it seems to me the situation 
is reversed. Looking at RAID-over-encryption, I disagree that having the 
same plaintext encrypted over multiple keys is a concern with modern 
ciphers. The real concern with most full disk encryption (and dm-crypt 
in particular) is integrity protection: the ability of an attacker to 
change the ciphertext undetected. This ability is greatly hampered when 
the attacker needs to coordinate the attacks on two mirrored blocks, 
otherwise the two copies would not be consistent.

I haven't researched all figerprinting attacks and the interaction with 
various ways of generating IVs, so my intuition may still be proven wrong.

Thanks,
Yaron