[dm-crypt] Bad performance with software RAID5 and LUKS encryption

[dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Philipp Wendler]

Hi,

I have set up a Linux software RAID5 on three hard drives and want to
encrypt it with cryptsetup/LUKS. My tests showed that the encryption
leads to a massive performance decrease that I cannot explain.

The RAID5 is able to write 187 MB/s [1] without encryption. With
encryption on top of it, write speed is down to about 40 MB/s.

The RAID has a chunk size of 512K and a write intent bitmap. I used -c
aes-xts-plain -s 512 --align-payload=2048 as the parameters for
cryptsetup luksFormat, so the payload should be aligned to 2048 blocks
of 512 bytes (i.e., 1MB), if I understood this correctly. cryptsetup
luksDump shows a payload offset of 4096. So I think the alignment is
correct and fits to the RAID chunk size.

The CPU should not be the bottleneck here, as it has hardware support
for AES (aesni_intel). If I write on another drive (an SSD with LVM)
that is also encrypted, I do have a write speed of about 150 MB/s. top
shows that the CPU usage is indeed very low, only the RAID5 xor takes 14%.

I also tried putting a filesystem (ext4) directly on the unencrypted
RAID so see if the layering is problem. The filesystem decreases the
performance a little bit as expected, but by far not that much (write
speed varying, but > 100 MB/s).

Summary:
Disks + RAID5: good
Disks + RAID5 + ext4: good
Disks + RAID5 + encryption: bad
SSD + encryption + LVM + ext4: good

The read performance is not affected by the encryption, it is 207 MB/s
without and 205 MB/s with encryption (also showing that CPU power is not
the problem).

Is it possible to improve the write performance of the encrypted RAID?
What could be the reason of the bad performance?

Software versions:
Ubuntu 11.04
Linux 2.6.38
cryptsetup 1.1.3


Help / ideas / etc. would be very much appreciated.

Greetings,
Philipp


[1] All speed measurements were done with several runs of
dd if=/dev/zero of=DEV bs=100M count=100

Re: [dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Jens Wahnes]

[-- Attachment #1: Type: text/plain, Size: 357 bytes --]

Philipp Wendler wrote:

> [1] All speed measurements were done with several runs of
> dd if=/dev/zero of=DEV bs=100M count=100

Did you consider using another block size? I have noticed that this can
have considerable impact under the conditions I was testing once. Do
you see any changes if you try, e.g.

dd if=/dev/zero of=DEV bs=4K count=2500000


Jens

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Philipp Wendler]

Hi,

Am 05.07.2011 15:30, schrieb Jens Wahnes:
> Philipp Wendler wrote:
>
>> [1] All speed measurements were done with several runs of
>> dd if=/dev/zero of=DEV bs=100M count=100
>
> Did you consider using another block size?

So far I tested with a large block size only, because I was expecting a 
bad performance for small block sizes due to the raid5 anyway.

But I have now run a series of tests with 4K, 1M and 10M. Block size 
indeed affects the performance (although not as much as I would have 
expected), but the difference for with/without encryption still remains 
as large as before:

without encryption: 150-180 MB/s
with encryption: 30-40 MB/s

Greetings, Philipp

Re: [dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Philipp Wendler]

Hello,

Am 05.07.2011 08:45, schrieb Philipp Wendler:
> I have set up a Linux software RAID5 on three hard drives and want to
> encrypt it with cryptsetup/LUKS. My tests showed that the encryption
> leads to a massive performance decrease that I cannot explain.
>
> The RAID5 is able to write 187 MB/s [1] without encryption. With
> encryption on top of it, write speed is down to about 40 MB/s.

Sorry for answering to a mail in this very old thread,
but it seems I finally found a solution,
and I know that there are some other people interested in the solution
as well (so if you got this email directly from me, I BCC'ed you because
you contacted me about this).
If you want to read up the full story, here's the link:
http://www.saout.de/pipermail/dm-crypt/2011-July/001773.html

Today I read about the stripe_cache_size setting of md RAID, and tried
it out. With the default value of 256, the performance is slow as
described. With a value of 4096, I get a performance increase from about
40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.

Background: The stripe cache stores recently written blocks. If data is
written continuously, it might happen that during a first write only a
part of one stripe is written. This means, the RAID code has to read the
complete stripe from disk, update it, and write it completely again. If
a second write comes in for another part of the same stripe, all this
would have to be done again. Now, if the cache is used and still
contains the data written by the first write, the read that was
necessary before the second write can be omitted.


Now it seems that dm-crypt always writes with small block size to the
underlying disk, even when I write with a big block size.
Could this be true?
Could this perhaps be improved? While I have found a solution for me,
this could probably solve performance problems for many people.

Furthermore, dm-crypt write is still slower than unencrypted write,
although for reads the performance of encrypted and unencrypted are the
same. So I guess the small block size still has a performance penalty
(probably when first writing to a stripe and it is not yet in cache).

My current setup is Ubuntu 12.04 (Linux 3.2).
Nothing else has changed compared to when I first asked about this.

Greetings, Philipp

Re: [dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Arno Wagner]

Hi,

thanks for the info. I think I may add a "performance"
section to the FAQ, and this could be one of the items.
Do you have the reference where you found the info on the
stripe_cache_size?

Arno

On Tue, Feb 05, 2013 at 09:02:21PM +0100, Philipp Wendler wrote:
> Hello,
> 
> Am 05.07.2011 08:45, schrieb Philipp Wendler:
> > I have set up a Linux software RAID5 on three hard drives and want to
> > encrypt it with cryptsetup/LUKS. My tests showed that the encryption
> > leads to a massive performance decrease that I cannot explain.
> >
> > The RAID5 is able to write 187 MB/s [1] without encryption. With
> > encryption on top of it, write speed is down to about 40 MB/s.
> 
> Sorry for answering to a mail in this very old thread,
> but it seems I finally found a solution,
> and I know that there are some other people interested in the solution
> as well (so if you got this email directly from me, I BCC'ed you because
> you contacted me about this).
> If you want to read up the full story, here's the link:
> http://www.saout.de/pipermail/dm-crypt/2011-July/001773.html
> 
> Today I read about the stripe_cache_size setting of md RAID, and tried
> it out. With the default value of 256, the performance is slow as
> described. With a value of 4096, I get a performance increase from about
> 40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.
> 
> Background: The stripe cache stores recently written blocks. If data is
> written continuously, it might happen that during a first write only a
> part of one stripe is written. This means, the RAID code has to read the
> complete stripe from disk, update it, and write it completely again. If
> a second write comes in for another part of the same stripe, all this
> would have to be done again. Now, if the cache is used and still
> contains the data written by the first write, the read that was
> necessary before the second write can be omitted.
> 
> 
> Now it seems that dm-crypt always writes with small block size to the
> underlying disk, even when I write with a big block size.
> Could this be true?
> Could this perhaps be improved? While I have found a solution for me,
> this could probably solve performance problems for many people.
> 
> Furthermore, dm-crypt write is still slower than unencrypted write,
> although for reads the performance of encrypted and unencrypted are the
> same. So I guess the small block size still has a performance penalty
> (probably when first writing to a stripe and it is not yet in cache).
> 
> My current setup is Ubuntu 12.04 (Linux 3.2).
> Nothing else has changed compared to when I first asked about this.
> 
> Greetings, Philipp
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Philipp Wendler]

Hi,

Am 05.02.2013 21:45, schrieb Arno Wagner:
> thanks for the info. I think I may add a "performance"
> section to the FAQ, and this could be one of the items.
> Do you have the reference where you found the info on the
> stripe_cache_size?

Actually, I first read about the concept of a stripe cache in this btrfs
announcement post:
http://article.gmane.org/gmane.comp.file-systems.btrfs/23006
Then I googled it and found some usages:
http://h3x.no/2011/07/09/tuning-ubuntu-mdadm-raid56
http://ubuntuforums.org/showthread.php?t=1494846

Unfortunately, I didn't find a real citable source.
The kernel documentation (Documentation/md.txt) also only contains 3
lines saying nothing about what it actually is.

Greetings, Philipp


> On Tue, Feb 05, 2013 at 09:02:21PM +0100, Philipp Wendler wrote:
>> Hello,
>>
>> Am 05.07.2011 08:45, schrieb Philipp Wendler:
>>> I have set up a Linux software RAID5 on three hard drives and want to
>>> encrypt it with cryptsetup/LUKS. My tests showed that the encryption
>>> leads to a massive performance decrease that I cannot explain.
>>>
>>> The RAID5 is able to write 187 MB/s [1] without encryption. With
>>> encryption on top of it, write speed is down to about 40 MB/s.
>>
>> Sorry for answering to a mail in this very old thread,
>> but it seems I finally found a solution,
>> and I know that there are some other people interested in the solution
>> as well (so if you got this email directly from me, I BCC'ed you because
>> you contacted me about this).
>> If you want to read up the full story, here's the link:
>> http://www.saout.de/pipermail/dm-crypt/2011-July/001773.html
>>
>> Today I read about the stripe_cache_size setting of md RAID, and tried
>> it out. With the default value of 256, the performance is slow as
>> described. With a value of 4096, I get a performance increase from about
>> 40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.
>>
>> Background: The stripe cache stores recently written blocks. If data is
>> written continuously, it might happen that during a first write only a
>> part of one stripe is written. This means, the RAID code has to read the
>> complete stripe from disk, update it, and write it completely again. If
>> a second write comes in for another part of the same stripe, all this
>> would have to be done again. Now, if the cache is used and still
>> contains the data written by the first write, the read that was
>> necessary before the second write can be omitted.
>>
>>
>> Now it seems that dm-crypt always writes with small block size to the
>> underlying disk, even when I write with a big block size.
>> Could this be true?
>> Could this perhaps be improved? While I have found a solution for me,
>> this could probably solve performance problems for many people.
>>
>> Furthermore, dm-crypt write is still slower than unencrypted write,
>> although for reads the performance of encrypted and unencrypted are the
>> same. So I guess the small block size still has a performance penalty
>> (probably when first writing to a stripe and it is not yet in cache).
>>
>> My current setup is Ubuntu 12.04 (Linux 3.2).
>> Nothing else has changed compared to when I first asked about this.
>>
>> Greetings, Philipp
>> _______________________________________________
>> dm-crypt mailing list
>> dm-crypt@saout.de
>> http://www.saout.de/mailman/listinfo/dm-crypt
> 

Re: [dm-crypt] Bad performance with software RAID5 and LUKS encryption

[Arno Wagner]

Hi Philipp,

thanks, if that explains it reasonably well, I will just
paraphrase. But first I have to understand it myself ;-)

Arno

On Tue, Feb 05, 2013 at 10:09:36PM +0100, Philipp Wendler wrote:
> Hi,
> 
> Am 05.02.2013 21:45, schrieb Arno Wagner:
> > thanks for the info. I think I may add a "performance"
> > section to the FAQ, and this could be one of the items.
> > Do you have the reference where you found the info on the
> > stripe_cache_size?
> 
> Actually, I first read about the concept of a stripe cache in this btrfs
> announcement post:
> http://article.gmane.org/gmane.comp.file-systems.btrfs/23006
> Then I googled it and found some usages:
> http://h3x.no/2011/07/09/tuning-ubuntu-mdadm-raid56
> http://ubuntuforums.org/showthread.php?t=1494846
> 
> Unfortunately, I didn't find a real citable source.
> The kernel documentation (Documentation/md.txt) also only contains 3
> lines saying nothing about what it actually is.
> 
> Greetings, Philipp
> 
> 
> > On Tue, Feb 05, 2013 at 09:02:21PM +0100, Philipp Wendler wrote:
> >> Hello,
> >>
> >> Am 05.07.2011 08:45, schrieb Philipp Wendler:
> >>> I have set up a Linux software RAID5 on three hard drives and want to
> >>> encrypt it with cryptsetup/LUKS. My tests showed that the encryption
> >>> leads to a massive performance decrease that I cannot explain.
> >>>
> >>> The RAID5 is able to write 187 MB/s [1] without encryption. With
> >>> encryption on top of it, write speed is down to about 40 MB/s.
> >>
> >> Sorry for answering to a mail in this very old thread,
> >> but it seems I finally found a solution,
> >> and I know that there are some other people interested in the solution
> >> as well (so if you got this email directly from me, I BCC'ed you because
> >> you contacted me about this).
> >> If you want to read up the full story, here's the link:
> >> http://www.saout.de/pipermail/dm-crypt/2011-July/001773.html
> >>
> >> Today I read about the stripe_cache_size setting of md RAID, and tried
> >> it out. With the default value of 256, the performance is slow as
> >> described. With a value of 4096, I get a performance increase from about
> >> 40-50 MB/s to 123 MB/s. For values >= 8192, I get 140 MB/s out of it.
> >>
> >> Background: The stripe cache stores recently written blocks. If data is
> >> written continuously, it might happen that during a first write only a
> >> part of one stripe is written. This means, the RAID code has to read the
> >> complete stripe from disk, update it, and write it completely again. If
> >> a second write comes in for another part of the same stripe, all this
> >> would have to be done again. Now, if the cache is used and still
> >> contains the data written by the first write, the read that was
> >> necessary before the second write can be omitted.
> >>
> >>
> >> Now it seems that dm-crypt always writes with small block size to the
> >> underlying disk, even when I write with a big block size.
> >> Could this be true?
> >> Could this perhaps be improved? While I have found a solution for me,
> >> this could probably solve performance problems for many people.
> >>
> >> Furthermore, dm-crypt write is still slower than unencrypted write,
> >> although for reads the performance of encrypted and unencrypted are the
> >> same. So I guess the small block size still has a performance penalty
> >> (probably when first writing to a stripe and it is not yet in cache).
> >>
> >> My current setup is Ubuntu 12.04 (Linux 3.2).
> >> Nothing else has changed compared to when I first asked about this.
> >>
> >> Greetings, Philipp
> >> _______________________________________________
> >> dm-crypt mailing list
> >> dm-crypt@saout.de
> >> http://www.saout.de/mailman/listinfo/dm-crypt
> > 
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell