[dm-crypt] dm-crypt multithreaded yet?

[dm-crypt] dm-crypt multithreaded yet?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

Hi all.

Sorry for what is probably a very tiresome topic here, but just trying
to find out what the current status is...

I thought I knew the answer to this question (dm-crypt multithreaded
yet? no) but I had read in several places that dm-crypt had gained
multithreaded capabilities.  I'm still skeptical though.  I certainly
don't see it on Fedora 17's 3.3.4-5.fc17.x86_64 kernel.

Sadly, I don't have an AES-NI capable CPU, but I do have 4 (real, 8 with
HT enabled) cores and dm-crypting on my SSD reduces its performance down
from about 260MB/s (native) to 95MB/s.  :-(  I would like to think I
could get more performance out of it if more of the (usually available
anyway) CPU could be thrown at it.

Is this something being worked on currently?

I have taken note of
http://people.redhat.com/mpatocka/patches/kernel/dm-crypt-paralelizace/current/

but that work seems to be going back to Feb/2012.  And TBH, I'm not
entirely clear if that work is aimed at multi-threaded dm-crypt or not.

Cheers,
b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [dm-crypt] dm-crypt multithreaded yet?

[Thomas Bächler]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

Am 08.08.2012 16:29, schrieb Brian J. Murrell:
> I thought I knew the answer to this question (dm-crypt multithreaded
> yet? no) but I had read in several places that dm-crypt had gained
> multithreaded capabilities.  I'm still skeptical though.  I certainly
> don't see it on Fedora 17's 3.3.4-5.fc17.x86_64 kernel.
> 
> Sadly, I don't have an AES-NI capable CPU, but I do have 4 (real, 8 with
> HT enabled) cores and dm-crypting on my SSD reduces its performance down
> from about 260MB/s (native) to 95MB/s.  :-(  I would like to think I
> could get more performance out of it if more of the (usually available
> anyway) CPU could be thrown at it.

This is in 2.6.38 and later, I think:

https://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=c029772125594e31eb1a5ad9e0913724ed9891f2

The commit message also explains why you don't notice it.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [dm-crypt] dm-crypt multithreaded yet?

[Milan Broz]

On 08/08/2012 04:29 PM, Brian J. Murrell wrote:
> Is this something being worked on currently?

Sort of. I have no idea what are upstream DM priorities now.

Anyway, I have patches rebased for 3.5/3.6-rc here
http://mbroz.fedorapeople.org/dm-crypt/

If you can test it and prove it helps for your use case, its would
be nice. Just take it as experimental, there can be bugs still.

> but that work seems to be going back to Feb/2012.  And TBH, I'm not
> entirely clear if that work is aimed at multi-threaded dm-crypt or not.

Should be. Search archive of this list as well.

Milan

Re: [dm-crypt] dm-crypt multithreaded yet?

[Milan Broz]

On 08/08/2012 05:31 PM, Thomas Bächler wrote:
> This is in 2.6.38 and later, I think:

No, it is not. This is something different.

In current kernel is per-cpu changes but it will not use
parallel cpu cores if it is submitted from the same cpu.

Mikulas' patches tries to create parallel threads and run
operations on all cores (plus some other changes as sorting).

Milan

Re: [dm-crypt] dm-crypt multithreaded yet?

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 1112 bytes --]

Am 08.08.2012 16:29, schrieb Brian J. Murrell:
> I thought I knew the answer to this question (dm-crypt multithreaded
> yet? no) but I had read in several places that dm-crypt had gained
> multithreaded capabilities.  I'm still skeptical though.  I certainly
> don't see it on Fedora 17's 3.3.4-5.fc17.x86_64 kernel.
> 
> Sadly, I don't have an AES-NI capable CPU, but I do have 4 (real, 8 with
> HT enabled) cores and dm-crypting on my SSD reduces its performance down
> from about 260MB/s (native) to 95MB/s.  :-(  I would like to think I
> could get more performance out of it if more of the (usually available
> anyway) CPU could be thrown at it.

If that helps, my 500MB/s SSD is giving me 270MB/s when reading an encrypted
file (i.e. going through the block and the filesystem layer) on my laptop:
Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 308 bytes --]

Re: [dm-crypt] dm-crypt multithreaded yet?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

On 12-08-08 11:17 PM, Marc MERLIN wrote:
> 
> If that helps, my 500MB/s SSD is giving me 270MB/s when reading an encrypted
> file (i.e. going through the block and the filesystem layer) on my laptop:
> Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz

That would be AES-NI giving you that throughput.  Unfortunately not all
of us have AES-NI (and nor can we just throw away an entire laptop just
to get it) but do have lots of idle cores sitting around that could help
out.

b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [dm-crypt] dm-crypt multithreaded yet?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 505 bytes --]

On 12-08-08 11:39 AM, Milan Broz wrote:
> 
> Anyway, I have patches rebased for 3.5/3.6-rc here
> http://mbroz.fedorapeople.org/dm-crypt/

Unfortunately the files in http://mbroz.fedorapeople.org/dm-crypt/3.6-rc
all return 403 trying to fetch them.

> If you can test it and prove it helps for your use case, its would
> be nice. Just take it as experimental, there can be bugs still.

I was going to try this on an FC18 kernel.  Are you using these patches
by any chance?

Cheers,
b.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [dm-crypt] dm-crypt multithreaded yet?

[Milan Broz]

On 12/16/2012 01:10 PM, Brian J. Murrell wrote:
> On 12-08-08 11:39 AM, Milan Broz wrote:
>>
>> Anyway, I have patches rebased for 3.5/3.6-rc here
>> http://mbroz.fedorapeople.org/dm-crypt/
> 
> Unfortunately the files in http://mbroz.fedorapeople.org/dm-crypt/3.6-rc
> all return 403 trying to fetch them.

No idea why it turned world non-readable, fixed.
 
>> If you can test it and prove it helps for your use case, its would
>> be nice. Just take it as experimental, there can be bugs still.
> 
> I was going to try this on an FC18 kernel.  Are you using these patches
> by any chance?

But I am afraid I did not find any real situation where these patches
really helps (AND not cause problems with different pattern).
See also discussion on dm-devel list, I still think generic
kernel workqueues should be used to provide such parallelization and
we should not duplicate code in dm-crypt...

My suggestion is that using AES-NI extension helps much more with
the current upstream code than anything else (for AES, obviously).

Milan

Re: [dm-crypt] dm-crypt multithreaded yet?

[Heinz Diehl]

On 16.12.2012, Milan Broz wrote: 

> My suggestion is that using AES-NI extension helps much more with
> the current upstream code than anything else (for AES, obviously).

And not to forget the optimized modules in the kernel.
Here's an example with a WD Black Caviar using serpent
encryption on a test machine. 
This is stock cryptsetup. Wouldn't have had more speed
without encryption..


[htd@test2 ~]$ cat /proc/cpuinfo
model name : Intel(R) Core(TM) i5 CPU M 450  @ 2.40GHz

[root@test2 ~]# hdparm -i /dev/sda | grep Model
 Model=WDC WD7500BPKT-00PK4T0, FwRev=01.01A01,
 SerialNo=WD-XXXXXXXXX

[root@test2 ~]# cryptsetup luksDump /dev/sda1
LUKS header information for /dev/sda1

Version:       		1
Cipher name:   		serpent
Cipher mode:   		xts-plain64:whirlpool
Hash spec:     		whirlpool
Payload offset:		4096
MK bits:       		512
[....]

[root@test2 mapper]# hdparm -t /dev/mapper/luks-xxxxxxxxx 

/dev/mapper/luks-xxxxxxxxxxxx:
 Timing buffered disk reads: 386 MB in  3.01 seconds = 128.27 MB/sec

Modules:

serpent_sse2_x86_64    50364  6 
serpent_generic        25563  1 serpent_sse2_x86_64
lrw                    13145  1 serpent_sse2_x86_64
glue_helper            13172  1 serpent_sse2_x86_64
ablk_helper            13269  1 serpent_sse2_x86_64
xts                    12871  1 serpent_sse2_x86_64
gf128mul               14333  2 lrw,xts
dm_crypt               22893  3 

Re: [dm-crypt] dm-crypt multithreaded yet?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1530 bytes --]

On 12-12-16 01:15 PM, Milan Broz wrote:
> 
> No idea why it turned world non-readable, fixed.

Thanks.

> But I am afraid I did not find any real situation where these patches
> really helps

If successive block I/Os from a sequential I/O can all get queued up by
different CPU (cores) I can't imagine why that wouldn't help.

> See also discussion on dm-devel list, I still think generic
> kernel workqueues should be used to provide such parallelization and
> we should not duplicate code in dm-crypt...

Well, definitely, if the parallelization work is duplicating existing
kernel functionality, that needs to be factored out.  But regardless...

> My suggestion is that using AES-NI extension helps much more with
> the current upstream code than anything else (for AES, obviously).

But of course.  I am sure anyone that has a CPU with AES-NI instructions
wouldn't care so much (at all even) about multi-core dm-crypt.  But
unfortunately not everyone has AES-NI capable processors, yet they might
have 4 cores (8 if HT enabled) available, with 3 (7, again if HT is
enabled) all sitting around doing nothing while one is grunting away
with all of the workload.

I wish I could just trade in my laptop for one with AES-NI, but other
than the lack of AES-NI it's far better than what I would get in return
for trading it in.  Kind of like trading in one's Ferrari just because
it doesn't have a leather steering-wheel for a Fiat that does have a
leather steering wheel.  :-/

Cheers,
b.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]