LUKS2 - RAID0 - NVME : alignment issue

LUKS2 - RAID0 - NVME : alignment issue

[laurent cop]

[-- Attachment #1.1: Type: text/plain, Size: 1108 bytes --]

Hello,

I have trouble while in top of the stack :
mkfs.ext4 /dev/mapper/raid0luks2
/dev/mapper/raid0luks2 alignement is offset by 147456 bytes
This may result in very poor performance (re-) partitioning is suggested.
I have the same pb with mkfs.xfs

I am using following scheme:

LUKS2           => created with cryptsetup -v luksFormat --type luks2
/dev/md127 --cipher aes-gcm-random --integrity aeae
RAID 0           => created with mdadm chunk=32
3 Disks *NVME* => partition with gdisk (fist sector : 2048)

I got this information with lsblk -t for each 3 disks, i got the same
information.
nvmeXn1
       -> NvmeXn1p1                                 ALIGN = 0
                  -> md127                               ALIGN = 0
                           -> raid0luks2_dif           ALIGN = 147456
                                       -> raid0luks2     ALIGN = 147456
1) How can I solve my alignment issue?
2) Is it normal to have low performances while writing with dd. I was using
LUKS previously and I got only one dev mapper. Now I got 2. Does it have a
big impact on performances?

Kind regards.

[-- Attachment #1.2: Type: text/html, Size: 1689 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: LUKS2 - RAID0 - NVME : alignment issue

[Ondrej Kozina]

On 10/12/2018 01:56 PM, laurent cop wrote:
> Hello,
> 
> I have trouble while in top of the stack :
> mkfs.ext4 /dev/mapper/raid0luks2
> /dev/mapper/raid0luks2 alignement is offset by 147456 bytes
> This may result in very poor performance (re-) partitioning is suggested.
> I have the same pb with mkfs.xfs
> 
> I am using following scheme:
> 
> LUKS2           => created with cryptsetup -v luksFormat --type luks2 
> /dev/md127 --cipher aes-gcm-random --integrity aeae

There's a bug in align offset calculations in cryptsetup for LUKS2 
format. It'll get fixed with future 2.0.5 release.

Provided you don't have real data on the LUKS2 device already...could 
you provide us with --debug output from the luksFormat action run again? 
It'd help me to find out if it gets fixed with commit dd36d56d472e.

Regards
O.

> RAID 0           => created with mdadm chunk=32
> 3 Disks *NVME* => partition with gdisk (fist sector : 2048)
> 
> I got this information with lsblk -t for each 3 disks, i got the same 
> information.
> nvmeXn1
>         -> NvmeXn1p1                                 ALIGN = 0
>                    -> md127                               ALIGN = 0
>                             -> raid0luks2_dif           ALIGN = 147456
>                                         -> raid0luks2     ALIGN = 147456
> 1) How can I solve my alignment issue?
> 2) Is it normal to have low performances while writing with dd. I was 
> using LUKS previously and I got only one dev mapper. Now I got 2. Does 
> it have a big impact on performances?
> 
> Kind regards.
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: LUKS2 - RAID0 - NVME : alignment issue

[Mikulas Patocka]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1911 bytes --]



On Fri, 12 Oct 2018, laurent cop wrote:

> Hello,
> 
> I have trouble while in top of the stack :
> mkfs.ext4 /dev/mapper/raid0luks2
> /dev/mapper/raid0luks2 alignement is offset by 147456 bytes
> This may result in very poor performance (re-) partitioning is suggested.
> I have the same pb with mkfs.xfs

Just ignore this warning. dm-integrity interleaves data and metadata and 
consequently data are not aligned on raid stripe boundary.

> I am using following scheme:
> 
> LUKS2           => created with cryptsetup -v luksFormat --type luks2 /dev/md127 --cipher aes-gcm-random --integrity aeae
> RAID 0           => created with mdadm chunk=32
> 3 Disks NVME => partition with gdisk (fist sector : 2048)
> 
> I got this information with lsblk -t for each 3 disks, i got the same information.
> nvmeXn1
>        -> NvmeXn1p1                                 ALIGN = 0
>                   -> md127                               ALIGN = 0
>                            -> raid0luks2_dif           ALIGN = 147456
>                                        -> raid0luks2     ALIGN = 147456
> 1) How can I solve my alignment issue?
>
> 2) Is it normal to have low performances while writing with dd. I was 
> using LUKS previously and I got only one dev mapper. Now I got 2. Does 
> it have a big impact on performances?
> 
> Kind regards.

dm-integrity already slows down writes about 3 times. At this situation, 
trying to align accesses to raid stripe size doesn't make much sense.

If you want to improve performance, you may try to put two dm-ingerity 
images directly on the SSDs and create raid-0 on the top of them. It may 
perform better, but you'll have to benchmark it with the workload you are 
optimizing for.

Mikulas

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: LUKS2 - RAID0 - NVME : alignment issue

[Milan Broz]

On 18/10/2018 19:51, Mikulas Patocka wrote:
> On Fri, 12 Oct 2018, laurent cop wrote:
> 
>> Hello,
>>
>> I have trouble while in top of the stack :
>> mkfs.ext4 /dev/mapper/raid0luks2
>> /dev/mapper/raid0luks2 alignement is offset by 147456 bytes
>> This may result in very poor performance (re-) partitioning is suggested.
>> I have the same pb with mkfs.xfs
> 
> Just ignore this warning. dm-integrity interleaves data and metadata and 
> consequently data are not aligned on raid stripe boundary.
> 
>> I am using following scheme:
>>
>> LUKS2           => created with cryptsetup -v luksFormat --type luks2 /dev/md127 --cipher aes-gcm-random --integrity aeae
>> RAID 0           => created with mdadm chunk=32
>> 3 Disks NVME => partition with gdisk (fist sector : 2048)
>>
>> I got this information with lsblk -t for each 3 disks, i got the same information.
>> nvmeXn1
>>        -> NvmeXn1p1                                 ALIGN = 0
>>                   -> md127                               ALIGN = 0
>>                            -> raid0luks2_dif           ALIGN = 147456
>>                                        -> raid0luks2     ALIGN = 147456
>> 1) How can I solve my alignment issue?
>>
>> 2) Is it normal to have low performances while writing with dd. I was 
>> using LUKS previously and I got only one dev mapper. Now I got 2. Does 
>> it have a big impact on performances?
>>
>> Kind regards.
> 
> dm-integrity already slows down writes about 3 times. At this situation, 
> trying to align accesses to raid stripe size doesn't make much sense.

Actually, I think that the alignment reported there does not make any
sense for dm-integrity (here is not a simple offset, data and metadata are interleaved,
there is a journal... Dm-integrity here behaves more like a filesystem - what a single
offset means?

Anyway, Mikulas disagrees with me to simple remove it :-)

> If you want to improve performance, you may try to put two dm-ingerity 
> images directly on the SSDs and create raid-0 on the top of them. It may 
> perform better, but you'll have to benchmark it with the workload you are 
> optimizing for.

You can do this with LUKS2 authenticated encryption as well, but I am not sure
it is good idea - it will eat CPU time for encryption for each raid 0 member.

Milan

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: LUKS2 - RAID0 - NVME : alignment issue

[laurent cop]

[-- Attachment #1.1: Type: text/plain, Size: 4888 bytes --]

Thank you all for your advises.

Meanwhile, I have read  Practical Cryptographic Data Integrity Protection
with Full Disk Encryption Extended Version (1 jul 2018). Very interesting
publication. :-)

1) Performances seems pretty correct for linear access write 4k blocks NO
JOURNAL. Nevertheless, you don't indicate how you realise this test (cp,
dd, fio tool...?).

2) I will reconsider doing again my stack. (No datas on it) Have you any
advises related to fs, RAID0, luksformat (type luk2) creation?  Indeed, I
don't clearly understand this point :
*"dm-integrity interleaves data and metadata and consequently data are not
aligned on raid stripe boundary."*
According to what I inderstand this will kill my RAID0 main goal for
getting fast performance.

3) I want integrity features and I want to try the no journal option for
performance gain. How to use it  ?
cryptsetup -v luksFormat --type luks2 /dev/md127 --cipher aes-gcm-random
--integrity aead  *--integrity-no-journal*
4) I am not sure to understand the relation between sector size of each
layer of the stack as we introduce a layer (dif) with metadata under
dm-crypt layer and above storage devices. Does 4096 bytes must be choosen
for all layers?
*--sector-size <bytes>*Set sector size for use with disk encryption. It
must be power of two and in range 512 - 4096 bytes. The default is 512
bytes sectors. This option is available only in the LUKS2 mode. Note that
if sector size is higher than underlying device hardware sector and there
is not integrity protection that uses data journal, using this option can
increase risk on incomplete sector writes during a power fail. If used
together with *--integrity* option and dm-integrity journal, the atomicity
of writes is guaranteed in all cases (but it cost write performance - data
has to be written twice). Increasing sector size from 512 bytes to 4096
bytes can provide better performance on most of the modern storage devices
and also with some hw encryption accelerators
Thank you very much for explainations. Any links, articles, documents
answering to my interrogation are welcomed.(I have already read
https://manpages.debian.org/unstable/cryptsetup-bin/cryptsetup.8.en.html
and your publication)

I will try to make the stack again with --debug option.

King regards





Le ven. 19 oct. 2018 à 08:47, Milan Broz <gmazyland@gmail.com> a écrit :

> On 18/10/2018 19:51, Mikulas Patocka wrote:
> > On Fri, 12 Oct 2018, laurent cop wrote:
> >
> >> Hello,
> >>
> >> I have trouble while in top of the stack :
> >> mkfs.ext4 /dev/mapper/raid0luks2
> >> /dev/mapper/raid0luks2 alignement is offset by 147456 bytes
> >> This may result in very poor performance (re-) partitioning is
> suggested.
> >> I have the same pb with mkfs.xfs
> >
> > Just ignore this warning. dm-integrity interleaves data and metadata and
> > consequently data are not aligned on raid stripe boundary.
> >
> >> I am using following scheme:
> >>
> >> LUKS2           => created with cryptsetup -v luksFormat --type luks2
> /dev/md127 --cipher aes-gcm-random --integrity aeae
> >> RAID 0           => created with mdadm chunk=32
> >> 3 Disks NVME => partition with gdisk (fist sector : 2048)
> >>
> >> I got this information with lsblk -t for each 3 disks, i got the same
> information.
> >> nvmeXn1
> >>        -> NvmeXn1p1                                 ALIGN = 0
> >>                   -> md127                               ALIGN = 0
> >>                            -> raid0luks2_dif           ALIGN = 147456
> >>                                        -> raid0luks2     ALIGN = 147456
> >> 1) How can I solve my alignment issue?
> >>
> >> 2) Is it normal to have low performances while writing with dd. I was
> >> using LUKS previously and I got only one dev mapper. Now I got 2. Does
> >> it have a big impact on performances?
> >>
> >> Kind regards.
> >
> > dm-integrity already slows down writes about 3 times. At this situation,
> > trying to align accesses to raid stripe size doesn't make much sense.
>
> Actually, I think that the alignment reported there does not make any
> sense for dm-integrity (here is not a simple offset, data and metadata are
> interleaved,
> there is a journal... Dm-integrity here behaves more like a filesystem -
> what a single
> offset means?
>
> Anyway, Mikulas disagrees with me to simple remove it :-)
>
> > If you want to improve performance, you may try to put two dm-ingerity
> > images directly on the SSDs and create raid-0 on the top of them. It may
> > perform better, but you'll have to benchmark it with the workload you
> are
> > optimizing for.
>
> You can do this with LUKS2 authenticated encryption as well, but I am not
> sure
> it is good idea - it will eat CPU time for encryption for each raid 0
> member.
>
> Milan
>

[-- Attachment #1.2: Type: text/html, Size: 6296 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: LUKS2 - RAID0 - NVME : alignment issue

[Milan Broz]

Hi,

sorry for late reply...

On 28/10/2018 01:05, laurent cop wrote:
> Thank you all for your advises.
> 
> Meanwhile, I have read  Practical Cryptographic Data Integrity Protection with Full Disk Encryption Extended Version (1 jul 2018). Very interesting publication. :-)

Thanks. My major intention here is to demonstrate that we need authenticated encryption (AEAD) in future instead of current length
preserving encryption that is not able to provide even trivial integrity protections.

But as you already found, dm-integrity is far from ideal in some situations.
Anyway it is something to play with... and improve it in future.

> 1) Performances seems pretty correct for linear access write 4k blocks NO JOURNAL. Nevertheless, you don't indicate how you realise this test (cp, dd, fio tool...?).

It was just basic linear access that can be simulated with dd as well.
I used my code https://github.com/mbroz/dm_int_tools (that was later used in cryptsetup to wipe device).

Device was activated by cryptsetup (for no journal with option --integrity-no-journal) and then accessed
by the tool mentioned above.

In short, read test used for 4k and no-direct mode)
  # integrity_check check -n -b 8 <device>

and write test
  # integrity_check format -n -b 8 <device>

Time is measured in the test program using gettimeofday() (see code).
(The script also used /usr/bin/time, this way we found stupid mistakes like missing fsync() etc :-)

I think I removed more info from paper and forgot to put it back in extended version, sorry.

Anyway, the kernel evolves very quickly (specifically for NVME devices) so any comparison with older
version is misleading now, it need new tests.

Ondra (while working on his master thesis) had some benchmark scripts here
https://gitlab.com/omos/dm-crypt-test-scripts

(I would like to integrate dm-crypt/dm-integrity benchmarking to some better usable package...)

> 2) I will reconsider doing again my stack. (No datas on it) Have you
> any advises related to fs, RAID0, luksformat (type luk2) creation?
> Indeed, I don't clearly understand this point : /"dm-integrity
> interleaves data and metadata and consequently data are not aligned
> on raid stripe boundary."/ According to what I inderstand this will
> kill my RAID0 main goal for getting fast performance.
Unfortunately, yes. I would like to have some dm-integrity mode that is aligned though.
I added mention to release notes for cryptsetup 2.0.5 about performance problems.

We (DM developers) need to discuss what can be done with these configurations & dm-integrity in future...

TBH, do not expect super-fast performance with dm-integrity. The goal was
to prove it can be done, it can be used with common scenarios, but the performace
penalty for sector interleaving and journal will be always there.

> 3) I want integrity features and I want to try the no journal option for performance gain. How to use it  ?
> cryptsetup -v luksFormat --type luks2 /dev/md127 --cipher aes-gcm-random --integrity aead  /--integrity-no-journal/

Yes, --integrity-no-journal is the flag. Use it with open (luksOpen) command.

If open command is used with --persistent, it should be written to LUKS2 metadata and used later automatically.
Unfortunately, I just found a bug for persistent setting for this specific option...
So for now you have to use it on commandline always. I'll fix this.

> 4) I am not sure to understand the relation between sector size of
> each layer of the stack as we introduce a layer (dif) with metadata
> under dm-crypt layer and above storage devices. Does 4096 bytes must
> be choosen for all layers? *--sector-size <bytes>* Set sector size
> for use with disk encryption. It must be power of two and in range
> 512 - 4096 bytes. The default is 512 bytes sectors. This option is
> available only in the LUKS2 mode.
...

The sector size should be the same for all layers. 
For dm-crypt 4k sector should improve performance as well
(by default it uses 512 byte sector).

Anyway, thank you for testing and using it!

Regards,
Milan

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: LUKS2 - RAID0 - NVME : alignment issue

[Mike Snitzer]

On Fri, Oct 12 2018 at  7:56am -0400,
laurent cop <laurent.cop@gmail.com> wrote:

>    2) Is it normal to have low performances while writing with dd. I was
>    using LUKS previously and I got only one dev mapper. Now I got 2. Does it
>    have a big impact on performances?

This last question really makes me wonder: did you intend to enable the
use of dm-integrity?

Mike

Re: LUKS2 - RAID0 - NVME : alignment issue

[Milan Broz]

On 18/10/2018 20:43, Mike Snitzer wrote:
> On Fri, Oct 12 2018 at  7:56am -0400,
> laurent cop <laurent.cop@gmail.com> wrote:
> 
>>    2) Is it normal to have low performances while writing with dd. I was
>>    using LUKS previously and I got only one dev mapper. Now I got 2. Does it
>>    have a big impact on performances?
> 
> This last question really makes me wonder: did you intend to enable the
> use of dm-integrity?

If you use an authenticated encryption in LUKS2, it must use dm-integrity as a backend,
so you get two stacked devices. (Authentication tag and random IV must be stored together
with every sector, here dm-integrity just provides a space for it, it does not
calculate anything, all is up to dm-crypt.)


m.