[dm-crypt] Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[dm-crypt] Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[Johnny Dahlberg]

[-- Attachment #1.1: Type: text/plain, Size: 3085 bytes --]

Hello everyone!

I'd really like to check if I am missing any steps for setting up a fully
"4K sector" aligned storage stack. If there's anyone who knows the correct
answer, it would be here.

The goal is to have each Ext4 sector mapped exactly to a physical sector on
the SSD. For maximum read/write performance and to have perfectly aligned
TRIM support (yes I know about the minor security issue with encryption and
TRIM, but the SSD lifetime health is way more important to me).

My stack is SSD -> GPT Partition Table -> LUKS encryption -> LVM volume ->
ext4 filesystem, and the whole stack must be aligned to 4096 byte sectors
for this to work.

* Partition Table: Every partition MUST be completely aligned (START and
END) on MiB boundary (the ending alignment is important too since LUKS
can't encrypt partial sectors). Do this by setting GParted's "Align to:
MiB" when creating the partition. As the GParted manual explains, this
forces the partition to START and END on precise MiB boundaries: "Use MiB
alignment for modern operating systems. This setting aligns partitions to
start and end on precise mebibyte (1,048,576 byte) boundaries."

* Use GParted to create a "Filesystem: unformatted" partition. Again, it
must start and end on precise MiB alignment offsets via the "Align to: MiB"
setting.

* Terminal:

# Get device name for the "unformatted" partition, in my case
/dev/nvme0n1p3:
fdisk -l

# Ensure that the encrypted payload starts at 1 MiB (2048 * 512 byte
sectors):
cryptsetup --key-size 256 --type luks2 --sector-size 4096 --align-payload
2048 luksFormat /dev/nvme0n1p3

# Open the LUKS volume and permanently mark it as "allow discards (TRIM) so
that it always opens like that by default:
cryptsetup --allow-discards --persistent open /dev/nvme0n1p3 cryptdata

# Create the LVM volume metadata and enforce "start of payload" alignment
at the nearest 1 MiB boundary after the metadata (by default this means the
payload starts at 1 MiB):
pvcreate -ffy --metadatatype lvm2 --dataalignment 1m --dataalignmentoffset
0 /dev/mapper/cryptdata

# Create the LVM volume group and force it to divide the volume into
aligned 4 MiB chunks (this is the default, but enforcing it just to be
sure):
vgcreate -ffy --physicalextentsize 4m data /dev/mapper/cryptdata

# Create a LVM logical volume using all of the space:
lvcreate -y --name root --extents 100%FREE data

# Format as ext4 with 4096 byte (4 KiB) block size:
mkfs.ext4 -F -b 4096 /dev/mapper/data-root

If all of my research of all man pages and tons of old wikis and guides is
correct, this will create a stack that's completely aligned to 4 KiB
physical sectors.

Is there anything I am missing here, or anything that you would suggest
changing? For example, perhaps I should change "--align-payload 2048" to
"--align-payload 4096" (what would that do/improve?), which I've seen a few
mentions of?

Grateful for any help in clearing this up, or clarifying that my steps are
ready to use. It's been a very long research and I'd like to be sure that I
am doing everything correctly.


With best regards,

Johnny

[-- Attachment #1.2: Type: text/html, Size: 3401 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[Johnny Dahlberg]

On 19/03/2021, Maksim Fomin <maxim@fomin.one> wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
>> On Friday, 19 March 2021 г., 19:09, Johnny Dahlberg
>> <svartchimpans@gmail.com> wrote:
>>
>>> Hello everyone!
>>>
>>> Is there anything I am missing here, or anything that you would suggest
>>> changing? For example, perhaps I should change "--align-payload 2048" to
>>> "--align-payload 4096" (what would that do/improve?), which I've seen a
>>> few mentions of?
>>>
>>> Grateful for any help in clearing this up, or clarifying that my steps
>>> are ready to use. It's been a very long research and I'd like to be sure
>>> that I am doing everything correctly.
>>>
>>> With best regards,
>>>
>>> Johnny
>
> I think if LVM is partioned on 4k sectors, setting 4k at cryptsetup layer is
> not needed - all io requests will be done by lvm with 4k. However, I doubt
> there is a way to test this. Btw, I also have multilayer setup with
> traditional 512b at plain dm-crypt layer and 4k lvm.


Yeah. By default, LUKS (cryptsetup) uses 512 byte sectors. And indeed
it's possible to run 4K block size filesystems on top of LVM on top of
512 byte sector LUKS, which simply means that the filesystem will
always read/write 8X individual chunks of 512 byte LUKS sectors for
each 4K sector FS operation.

But LUKS has an inefficient legacy kernel queue design that re-queues
and copies queue items multiple times and wastes lots of time and
resources grabbing each LUKS chunk, so performance will be better if
LUKS is set to 4096 byte sectors simply because this means there's
less deep dm-crypt queues.

Cloudflare discovered these legacy problems in the dm-crypt code and
have upstreamed a semi-reliable patch that gets rid of the slow
queues. It has been merged a while ago. But since I don't want to run
their early and not yet 100% stable code (some issues with certain
hardware or filesystems were found), I decided to instead use the
plain old-school dm-crypt queues, and therefore 4K LUKS sectors is a
nice queue performance speed improvement compared to 8x512B.

Here's the earliest blog post which describes how they doubled the
performance of LUKS:
https://blog.cloudflare.com/speeding-up-linux-disk-encryption/

Here's their discussion about the ongoing work of improving the new
"queue-less" dm-crypt flags:
https://github.com/cloudflare/linux/issues/1

But the changes aren't mature yet. Picking 4K sector size in LUKS will
hopefully reduce the overhead of standard queue mode.

Anyway, I hope to hear from someone (perhaps even Milan :) if my
storage layer settings in the original message are okay or if they
could be improved. Looking forward to doing the OS installation with
the correct settings. :-)


-- Johnny
_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[Carlos E. R.]

[-- Attachment #1.1.1: Type: text/plain, Size: 831 bytes --]

On 19/03/2021 21.24, Johnny Dahlberg wrote:
> On 19/03/2021, Maksim Fomin <> wrote:
>>> On Friday, 19 March 2021 г., 19:09, Johnny Dahlberg <> wrote:

...

> But the changes aren't mature yet. Picking 4K sector size in LUKS will
> hopefully reduce the overhead of standard queue mode.
> 
> Anyway, I hope to hear from someone (perhaps even Milan :) if my
> storage layer settings in the original message are okay or if they
> could be improved. Looking forward to doing the OS installation with
> the correct settings. :-)

Related question:

How can one, on an exiting encrypted partition, find out what size is 
using each layer, and if they are aligned or not?

I am using encrypted partitions on SSD and I'm a bit worried.


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.2 x86_64 at Telcontar)


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[Johnny Dahlberg]

On 19/03/2021, Carlos E. R. <robin.listas@telefonica.net> wrote:
> On 19/03/2021 21.24, Johnny Dahlberg wrote:
>> On 19/03/2021, Maksim Fomin <> wrote:
>>>> On Friday, 19 March 2021 г., 19:09, Johnny Dahlberg <> wrote:
>
> ...
>
>> But the changes aren't mature yet. Picking 4K sector size in LUKS will
>> hopefully reduce the overhead of standard queue mode.
>>
>> Anyway, I hope to hear from someone (perhaps even Milan :) if my
>> storage layer settings in the original message are okay or if they
>> could be improved. Looking forward to doing the OS installation with
>> the correct settings. :-)
>
> Related question:
>
> How can one, on an exiting encrypted partition, find out what size is
> using each layer, and if they are aligned or not?
>
> I am using encrypted partitions on SSD and I'm a bit worried.
>
>
> --
> Cheers / Saludos,
>
> 		Carlos E. R.
> 		(from 15.2 x86_64 at Telcontar)
>
>

Hello Carlos!

First you would check the physical sector size of your SSD. It is most
likely 4096 bytes (4K). You can check this by doing:

cat /sys/block/(the parent device such as nvme0n1)/queue/physical_block_size

But sometimes the SSD lies and says 512 even though it actually uses
4096. So if you see 512, do an extra check online. Most likely it is
truly 4096 physically.

Next, check the start and end sectors of your partition and make sure
they are both on MiB boundaries, or at least on 4096 byte boundaries.

Next, check LUKS to make sure the "Payload offset" is 2048 or 4096
waning 1 MiB or 2 MiB respectively):

sudo cryptsetup luksDump /dev/(your luks device/partition)

Then check LVM to ensure your physical volume is starting at a
multiple of 1 MiB and that your volume "physical extents" size is a
multiple of 4 MiB:

sudo pvs -o +pe_start,vg_extent_size,pv_mda_size /dev/mapper/(name of
your LVM physical volume)

Lastly check your filsystem. If it is ext4, the command to check its
"block size" is:

sudo tune2fs -l /dev/(your ext4 device)

Good luck!

-- Johnny
_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[Carlos E. R.]

[-- Attachment #1.1.1: Type: text/plain, Size: 3952 bytes --]

On 19/03/2021 22.39, Johnny Dahlberg wrote:
> On 19/03/2021, Carlos E. R. <robin.listas@telefonica.net> wrote:
>> On 19/03/2021 21.24, Johnny Dahlberg wrote:
>>> On 19/03/2021, Maksim Fomin <> wrote:
>>>>> On Friday, 19 March 2021 г., 19:09, Johnny Dahlberg <> wrote:
>>
>> ...


>> Related question:
>>
>> How can one, on an exiting encrypted partition, find out what size is
>> using each layer, and if they are aligned or not?
>>
>> I am using encrypted partitions on SSD and I'm a bit worried.
>>
>>
> 
> Hello Carlos!
> 
> First you would check the physical sector size of your SSD. It is most
> likely 4096 bytes (4K). You can check this by doing:
> 
> cat /sys/block/(the parent device such as nvme0n1)/queue/physical_block_size

On the two machines I have running now, both have an nvme system disk, 
but only one has the encrypted partition on it; both use 512:

cer@Telcontar:~> cat /sys/block/nvme0n1/queue/physical_block_size
512
cer@Telcontar:~>

Isengard:~ # cat /sys/block/sda/queue/physical_block_size
512
Isengard:~ #

Hum!

Telcontar:~ # fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 970 EVO Plus 500GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt


Isengard:~ # fdisk -l /dev/sda
Disk /dev/sda: 111.8 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SMS200S
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

In both cases, the first partition starts on sector 2048.

I will look later in my laptop, one has a 1T SSD.



> But sometimes the SSD lies and says 512 even though it actually uses
> 4096. So if you see 512, do an extra check online. Most likely it is
> truly 4096 physically.

Could be... I'll check later.

> 
> Next, check the start and end sectors of your partition and make sure
> they are both on MiB boundaries, or at least on 4096 byte boundaries.

In both cases, the first partition starts on sector 2048.

80035840×512÷4096 = 10004480
80035840×512÷2²⁰ = 39080

so yes :-)

> 
> Next, check LUKS to make sure the "Payload offset" is 2048 or 4096
> waning 1 MiB or 2 MiB respectively):


> 
> sudo cryptsetup luksDump /dev/(your luks device/partition)

Isengard:~ # cryptsetup luksDump /dev/sda5
Payload offset:	4096


> 
> Then check LVM to ensure your physical volume is starting at a
> multiple of 1 MiB and that your volume "physical extents" size is a
> multiple of 4 MiB:

Oh, I'm not using LVM. one step less :-)


> sudo pvs -o +pe_start,vg_extent_size,pv_mda_size /dev/mapper/(name of
> your LVM physical volume)
> 
> Lastly check your filsystem. If it is ext4, the command to check its
> "block size" is:
> 
> sudo tune2fs -l /dev/(your ext4 device)

It is XFS.


Isengard:~ # xfs_info /dev/mapper/cr_hoard_2
meta-data=/dev/mapper/cr_hoard_2 isize=256    agcount=4, 
agsize=244172416 blks
          =                       sectsz=4096  attr=2, projid32bit=1
          =                       crc=0        finobt=0, sparse=0, rmapbt=0
          =                       reflink=0
data     =                       bsize=4096   blocks=976689664, imaxpct=5
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=476899, version=2
          =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Isengard:~ #

:-)

> Good luck!

I think I'm good -  no worries :-)

Thanks! :-)

-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.2 x86_64 at Telcontar)


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[Johnny Dahlberg]

[-- Attachment #1.1: Type: text/plain, Size: 2326 bytes --]

On Fri, 19 Mar 2021 at 23:21, Carlos E. R. <robin.listas@telefonica.net>
wrote:

> On 19/03/2021 22.39, Johnny Dahlberg wrote:
> > On 19/03/2021, Carlos E. R. <robin.listas@telefonica.net> wrote:
> >> On 19/03/2021 21.24, Johnny Dahlberg wrote:
> >>> On 19/03/2021, Maksim Fomin <> wrote:
> >>>>> On Friday, 19 March 2021 г., 19:09, Johnny Dahlberg <> wrote:
> >>
> >> ...
>
>
> >> Related question:
> >>
> >> How can one, on an exiting encrypted partition, find out what size is
> >> using each layer, and if they are aligned or not?
> >>
> >> I am using encrypted partitions on SSD and I'm a bit worried.
> >>
> >>
> >
> > Hello Carlos!
> >
> > First you would check the physical sector size of your SSD. It is most
> > likely 4096 bytes (4K). You can check this by doing:
> >
> > cat /sys/block/(the parent device such as
> nvme0n1)/queue/physical_block_size
>
> On the two machines I have running now, both have an nvme system disk,
> but only one has the encrypted partition on it; both use 512:
>
> cer@Telcontar:~> cat /sys/block/nvme0n1/queue/physical_block_size
> 512
> cer@Telcontar:~>
>
> Isengard:~ # cat /sys/block/sda/queue/physical_block_size
> 512
> Isengard:~ #
>
> Hum!
>

Since January 2011, all manufacturers began using Advanced Format (4096
bytes): https://en.m.wikipedia.org/wiki/Advanced_Format#History

From what I can find online, your Samsung 970 Evo Plus uses 8K sectors
physically.


> I think I'm good -  no worries :-)
>
> Thanks! :-)
>
> --
> Cheers / Saludos,
>
>                 Carlos E. R.
>                 (from 15.2 x86_64 at Telcontar)
>

If you want to check everything for sure, and learn everything important
that there is to know about these things, you can check out my new guide /
alignment checker tool at:

https://bananaman.github.io/friendly-guides/pages/storage_alignment.html

You just type in your current values in each field and it will tell you
whether they're aligned to your physical sectors, or better yet to MiB
boundaries (which is guaranteed to be aligned to all physical sector sizes
no matter what your drive uses).

Every step tells you which commands to run to check that particular value.

There's also plenty of interesting information to learn on there.

Take care!

-- Johnny

[-- Attachment #1.2: Type: text/html, Size: 3335 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Aligning entire LUKS/LVM/Ext4 filesystem stack to 4K sectors

[Erwin van Londen]

I think the quickest way to find out is:

"lsblk -o name,phy-sec,log-sec"

Output like this shows it nicely:

[ monster % ~ ] lsblk -o name,phy-sec,log-sec                         
[  5:57PM ]
NAME                   PHY-SEC LOG-SEC
sda                        512     512
├─sda1                     512     512
└─sda2                     512     512
  ├─vg_monster-lv_home     512     512
  ├─vg_monster-lv_swap     512     512
  └─vg_monster-lv_root     512     512
sdb                        512     512
├─sdb1                     512     512
└─sdb2                     512     512

Obviously playing around with the output parameters gives you plenty
flexibility.

Cheers
Erwin

On Wed, 2021-03-24 at 20:53 +0100, Johnny Dahlberg wrote:
> 
> 
> On Fri, 19 Mar 2021 at 23:21, Carlos E. R.
> <robin.listas@telefonica.net> wrote:
> > On 19/03/2021 22.39, Johnny Dahlberg wrote:
> > > On 19/03/2021, Carlos E. R. <robin.listas@telefonica.net> wrote:
> > > > On 19/03/2021 21.24, Johnny Dahlberg wrote:
> > > > > On 19/03/2021, Maksim Fomin <> wrote:
> > > > > > > On Friday, 19 March 2021 г., 19:09, Johnny Dahlberg <>
> > > > > > > wrote:
> > > > 
> > > > ...
> > 
> > 
> > > > Related question:
> > > > 
> > > > How can one, on an exiting encrypted partition, find out what
> > > > size is
> > > > using each layer, and if they are aligned or not?
> > > > 
> > > > I am using encrypted partitions on SSD and I'm a bit worried.
> > > > 
> > > > 
> > > 
> > > Hello Carlos!
> > > 
> > > First you would check the physical sector size of your SSD. It is
> > > most
> > > likely 4096 bytes (4K). You can check this by doing:
> > > 
> > > cat /sys/block/(the parent device such as
> > > nvme0n1)/queue/physical_block_size
> > 
> > On the two machines I have running now, both have an nvme system
> > disk, 
> > but only one has the encrypted partition on it; both use 512:
> > 
> > cer@Telcontar:~> cat /sys/block/nvme0n1/queue/physical_block_size
> > 512
> > cer@Telcontar:~>
> > 
> > Isengard:~ # cat /sys/block/sda/queue/physical_block_size
> > 512
> > Isengard:~ #
> > 
> > Hum!
> > 
> 
> 
> Since January 2011, all manufacturers began using Advanced Format
> (4096 bytes): https://en.m.wikipedia.org/wiki/Advanced_Format#History
> 
> From what I can find online, your Samsung 970 Evo Plus uses 8K
> sectors physically.
> 
> > 
> > I think I'm good -  no worries :-)
> > 
> > Thanks! :-)
> > 
> > _______________________________________________
> > dm-crypt mailing list -- dm-crypt@saout.de
> > To unsubscribe send an email to dm-crypt-leave@saout.de

_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de