[dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

[dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

["C. Dominik Bódi"]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

(please CC me, I'm not subscribed to the list)

Hello,

I've discovered some strange behaviour on my Debian system, with
dm-crypt on raid 5 showing slow performance and I am seeking ways to
remedy that.

My setup:
Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI)
Intel X58 chipset
Debian sid/unstable
kernels tried: 3.12.9, 3.13.4 (Debian kernels)
4 x 3 TB SATA disks on a Intel X58 AHCI controller.
That controller has 6 SATA ports and connects directly to the chipsets
South Bridge, which connects to the North Bridge via a very fast DMI link.
Second PCIe-SATA controller on motherboard (ACHI driver, probably
JMicron chipset) - 2 SATA ports.

On that disk I have a RAID5 array (source media are the 4th partition on
each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS
volume. That LUKS volume is a PV for LVM.

4 disks => RAID5 => LUKS Volume => LVM PV

Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read
speed. Each single disk has about 170MB/s with hdparm, so that is ok.

However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s.
Performing dd tests on the mounted LVM filesystems gets me about 160MB/s.

Now I connected two of the four disks to the second SATA controller

hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the
RAID5 array dropped to ca. 400MB/s. That was to be expected, as that
second controller seems somewhat slower.

And now comes the funny part:
Now I did a hdparm -t /dev/mapper/encrypted_raid5
and that got me ca. 300MB/s

Running top whilst doing the hdparm tests showed me that, when all 4
disks were on the Intel X58 AHCI controller, only 1 kworker thread was
running on a single CPU core, utilizing 100% CPU time.

When 2 of those disks were connected to the second SATA controller, 2
kworker threads were running on two different CPU cores.

It seems that when the disks sit on two different controllers, dm-crypt
is using 2 kworker threads to handle the data, but uses only one kworker
thread when all disks sit on a single controller.

My guess is that one single cpu core can decrypt/encrypt the LUKS volume
at about 150MB/s, which seems to be reasonable for that kind of CPU.
Thus, dm-crypt using more than one kworker threads leads to doubling of
the decryption speed, even if the two disks are slightly slower when
connected to the second controller.

I'd like dm-crypt to use as many threads as possible (e.g. one kworker
for each disk in the RAID5 array). That would give me reasonable speeds
for the encrypted LUKS volume, close to that of the unencrypted RAID5
volume.

I've experimented with the RAID5 arrays group_thread_cnt sysfs setting.
That is set to zero by default, disabling multithreaded RAID5. However,
enabling that and setting it to 2 or 4 did not change anything speed-wise.

I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity
setting, yet. Would that help?

Is there a way to enforce dm-crypt using more than one kworker thread to
encrypt/decrypt data for a crypt-on-raid5 setup?

Regards,
Dominik
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJTDND9AAoJEH57oErWeAO1IHgP/jghJGdRS/mWZ0SU3goeWSB3
0Bii9xRb0Eo1tPS+TTXRad308pCL8BXloNdgLJXL9T1P/TLOaX0e7QxVRCC3j6Uj
siBZ7rMBp60bHQu6JNUyXPwssgQhyRhL7Web2tHm7RQYDUoNaVdIjL0p4KvCRAX6
4RE2FzMEze/DgumZtXYgSlBBxVb9Za87Y/iwH+SJQ87kloKPkD7kgX2hByYJ5D7h
NswCns1slj5KYyOSqikcoJM1Z9Sd1eJjMhRJ37/hH5ERMiP7m+LMO07ZsxtPCdE0
Fyi1J6Ra28YzKJPTcaBSa2oy+zeauennCIgLbA2px+7t3zEoaUiMSg4WrFbrV8i4
x1WWiHHrZV23dza2Hgy4BcE1lXAGxEKCuOSZr0Js/b20DfCeoV84oc4GIlMo3ZL4
LztRJ4yrtOUehH9KUwTqFLFk3r9xWDMC3ZlrbyyoxVAB5Nr+w1FnAzQRnvC7DOHb
wU2lj9zWloNT8hUsCwtdpLxQWSj9iWw1j70xuqSUGIP3SLBgBQdUfpZP0Xwwe0Bi
577yt6CbQcaqGgjhtlgetwL9e8Vj8rMXbKZUKbyimimwmHEAJ++pdOWnBTdz7NOQ
IKXP29mbI7+uIrzcgQvH6o5NS6psxdIcuxjfgYoR8CLPcNB87xKDcMXF9hU8JkZA
fVE8Z2X5LPDksJrs3eb4
=5Mww
-----END PGP SIGNATURE-----

Re: [dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

[shmick]

"C. Dominik Bódi":
> (please CC me, I'm not subscribed to the list)
> 
> Hello,
> 
> I've discovered some strange behaviour on my Debian system, with
> dm-crypt on raid 5 showing slow performance and I am seeking ways to
> remedy that.
> 
> My setup:
> Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI)
> Intel X58 chipset
> Debian sid/unstable
> kernels tried: 3.12.9, 3.13.4 (Debian kernels)
> 4 x 3 TB SATA disks on a Intel X58 AHCI controller.
> That controller has 6 SATA ports and connects directly to the chipsets
> South Bridge, which connects to the North Bridge via a very fast DMI link.
> Second PCIe-SATA controller on motherboard (ACHI driver, probably
> JMicron chipset) - 2 SATA ports.
> 
> On that disk I have a RAID5 array (source media are the 4th partition on
> each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS
> volume. That LUKS volume is a PV for LVM.
> 
> 4 disks => RAID5 => LUKS Volume => LVM PV
> 
> Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read
> speed. Each single disk has about 170MB/s with hdparm, so that is ok.
> 
> However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s.
> Performing dd tests on the mounted LVM filesystems gets me about 160MB/s.

*only* 150 ?
there's lots of numbers here but maybe we should work backwards and KIS
ask yourself what do you want to achieve ?
when you know the results you want and/or need (2 different things) you
can work backwards & figure out out how to get there
do you *need* to have a certain read/write speed for your application or
desired conclusion in mind ?

do you have a recent version of cryptsetup with the benchmark option ?
that can give you an idea of in memory performance and depends on your
CPU & chosen ciphers/hashes

i didn't quite get if your RAID is a hardware or software array ?

> 
> Now I connected two of the four disks to the second SATA controller
> 
> hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the
> RAID5 array dropped to ca. 400MB/s. That was to be expected, as that
> second controller seems somewhat slower.
> 
> And now comes the funny part:
> Now I did a hdparm -t /dev/mapper/encrypted_raid5
> and that got me ca. 300MB/s
> 
> Running top whilst doing the hdparm tests showed me that, when all 4
> disks were on the Intel X58 AHCI controller, only 1 kworker thread was
> running on a single CPU core, utilizing 100% CPU time.
> 
> When 2 of those disks were connected to the second SATA controller, 2
> kworker threads were running on two different CPU cores.
> 
> It seems that when the disks sit on two different controllers, dm-crypt
> is using 2 kworker threads to handle the data, but uses only one kworker
> thread when all disks sit on a single controller.

i'll leave that to the experts ;-)

> 
> My guess is that one single cpu core can decrypt/encrypt the LUKS volume
> at about 150MB/s, which seems to be reasonable for that kind of CPU.
> Thus, dm-crypt using more than one kworker threads leads to doubling of
> the decryption speed, even if the two disks are slightly slower when
> connected to the second controller.
> 
> I'd like dm-crypt to use as many threads as possible (e.g. one kworker
> for each disk in the RAID5 array). That would give me reasonable speeds
> for the encrypted LUKS volume, close to that of the unencrypted RAID5
> volume.
> 
> I've experimented with the RAID5 arrays group_thread_cnt sysfs setting.
> That is set to zero by default, disabling multithreaded RAID5. However,
> enabling that and setting it to 2 or 4 did not change anything speed-wise.
> 
> I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity
> setting, yet. Would that help?
> 
> Is there a way to enforce dm-crypt using more than one kworker thread to
> encrypt/decrypt data for a crypt-on-raid5 setup?
> 
> Regards,
> Dominik
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

Re: [dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

["C. Dominik Bódi"]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

Am 26.02.2014 00:58, schrieb shmick@riseup.net:
> *only* 150 ?
> there's lots of numbers here but maybe we should work backwards and KIS
> ask yourself what do you want to achieve ?
> when you know the results you want and/or need (2 different things) you
> can work backwards & figure out out how to get there
> do you *need* to have a certain read/write speed for your application or
> desired conclusion in mind ?
Well, yes, I'd like to have as much throughput as possible. The cpu is a
quad core with HT, so could dm-crypt not utilize 4 threads when there
are 4 disks in the raid5 array.

An the thing is:

4 disks on controller A => dm-crypt 150MB/s
2 disk on controller A, 2 disks on controller B => dm-crypt 300MB/s

I don't understand that behaviour. Why does it make a difference for
dm-crypt if the disks sit on different controllers or not? Why should I
waste speed when dm-crypt could easily utilize 4 worker threads and max
out the raid5 performance? Why doesn't it do so automatically. Is there
any means to make dm-crypt use 4 threads even if all 4 drives sit on a
single controller?

I know the simple answer would be to reverse the stack, encrypt each
raid drive singularly and then put the raid5 array on top of the
encrypted drives. That makes it really difficult to unlock the drives
remotely (via dropbear in the initramfs), though.

> do you have a recent version of cryptsetup with the benchmark option ?
> that can give you an idea of in memory performance and depends on your
> CPU & chosen ciphers/hashes
I'm using aes-xts-plain64 with 256 key size. The benchmark shows me a
throughput of 175 MB/s. I reckon the benchmark only runs a single
thread, so that seems to be ok.

> i didn't quite get if your RAID is a hardware or software array ?
Hmmm, sorry for that oversight. Yes, it is a software raid5 array set up
with mdadm.

> i'll leave that to the experts ;-)
Yes, indeed, I was looking for an answer from someone that knows the
dm-crypt kernel code quite well.

Regards,
Dominik


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

[Sven Eschenberg]

My best guess here is this:

There is one work queue (IO QUEUE) per controller. Which in turn means
that dm-crypt is fed by two queues and as such decides to have one kworker
instance per queue. I might be completely wrong though. (Actually there's
still mdraid, which seems to have accordingly 1/2 queues).

Regards

-Sven


On Wed, February 26, 2014 16:12, "C. Dominik Bódi" wrote:
> Am 26.02.2014 00:58, schrieb shmick@riseup.net:
>> *only* 150 ?
>> there's lots of numbers here but maybe we should work backwards and KIS
>> ask yourself what do you want to achieve ?
>> when you know the results you want and/or need (2 different things) you
>> can work backwards & figure out out how to get there
>> do you *need* to have a certain read/write speed for your application or
>> desired conclusion in mind ?
> Well, yes, I'd like to have as much throughput as possible. The cpu is a
> quad core with HT, so could dm-crypt not utilize 4 threads when there
> are 4 disks in the raid5 array.
>
> An the thing is:
>
> 4 disks on controller A => dm-crypt 150MB/s
> 2 disk on controller A, 2 disks on controller B => dm-crypt 300MB/s
>
> I don't understand that behaviour. Why does it make a difference for
> dm-crypt if the disks sit on different controllers or not? Why should I
> waste speed when dm-crypt could easily utilize 4 worker threads and max
> out the raid5 performance? Why doesn't it do so automatically. Is there
> any means to make dm-crypt use 4 threads even if all 4 drives sit on a
> single controller?
>

Re: [dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

[Arno Wagner]

Just on a meta-layer: This is a known issue, but it seems the
fix is not quite clear. If you are really concerned, put
the encryption below the RAID, but that has other issues.

Arno

On Wed, Feb 26, 2014 at 16:58:15 CET, Sven Eschenberg wrote:
> My best guess here is this:
> 
> There is one work queue (IO QUEUE) per controller. Which in turn means
> that dm-crypt is fed by two queues and as such decides to have one kworker
> instance per queue. I might be completely wrong though. (Actually there's
> still mdraid, which seems to have accordingly 1/2 queues).
> 
> Regards
> 
> -Sven
> 
> 
> On Wed, February 26, 2014 16:12, "C. Dominik Bódi" wrote:
> > Am 26.02.2014 00:58, schrieb shmick@riseup.net:
> >> *only* 150 ?
> >> there's lots of numbers here but maybe we should work backwards and KIS
> >> ask yourself what do you want to achieve ?
> >> when you know the results you want and/or need (2 different things) you
> >> can work backwards & figure out out how to get there
> >> do you *need* to have a certain read/write speed for your application or
> >> desired conclusion in mind ?
> > Well, yes, I'd like to have as much throughput as possible. The cpu is a
> > quad core with HT, so could dm-crypt not utilize 4 threads when there
> > are 4 disks in the raid5 array.
> >
> > An the thing is:
> >
> > 4 disks on controller A => dm-crypt 150MB/s
> > 2 disk on controller A, 2 disks on controller B => dm-crypt 300MB/s
> >
> > I don't understand that behaviour. Why does it make a difference for
> > dm-crypt if the disks sit on different controllers or not? Why should I
> > waste speed when dm-crypt could easily utilize 4 worker threads and max
> > out the raid5 performance? Why doesn't it do so automatically. Is there
> > any means to make dm-crypt use 4 threads even if all 4 drives sit on a
> > single controller?
> >
> 
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -  Plato