[dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[Joe Hillenbrand]

I've been trying and trying to add a new key to one of my devices so
that I can automount,
but it consistently fails.

I run the following command. It hangs for ~85 minutes every time (I've
tried at least 5 times)
and then fails with the "Not compatible PBKDF2 options" message and exit code 1.

    $ sudo cryptsetup luksAddKey /dev/md1 /etc/.md1.key
    Enter any existing passphrase:
    Not compatible PBKDF2 options (using hash algorithm sha256).

I've tried generating several different keys of different sizes.
Exact same behavior for all of them.

    dd bs=512 count=4 if=/dev/urandom of=/etc/.md1.key iflag=fullblock
    dd bs=1 count=32 if=/dev/urandom of=/etc/.md1.key iflag=fullblock
    dd bs=8192KB count=1 if=/dev/urandom of=/etc/.md1.key iflag=fullblock
    head -c 2880 /dev/urandom | uuencode -m - | head -n 65 | tail -n
64 | sudo tee /etc/.md1.key

Here is some other information.

    $ uname -r
    4.6.2-1-ARCH

    $ cryptsetup --version
    cryptsetup 1.7.1

    $ sudo cryptsetup luksDump /dev/md1
    LUKS header information for /dev/md1

    Version:           1
    Cipher name:       aes
    Cipher mode:       xts-plain64
    Hash spec:         sha256
    Payload offset:    4096
    MK bits:           256
    MK digest:         xxxxx
    MK salt:           xxxxxx
    MK iterations:     333750
    UUID:

    Key Slot 0: ENABLED
        Iterations:             xxxxxxx
        Salt:                   xxxxxxx
        Key material offset:    8
        AF stripes:                4000
    Key Slot 1: DISABLED
    Key Slot 2: DISABLED
    Key Slot 3: DISABLED
    Key Slot 4: DISABLED
    Key Slot 5: DISABLED
    Key Slot 6: DISABLED
    Key Slot 7: DISABLED

    $ sudo mdadm --detail /dev/md1
    /dev/md1:
            Version : 1.2
      Creation Time : Thu Jun  9 14:01:50 2016
        Raid Level : raid5
        Array Size : 5860063232 (5588.59 GiB 6000.70 GB)
      Used Dev Size : 2930031616 (2794.30 GiB 3000.35 GB)
      Raid Devices : 3
      Total Devices : 3
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Fri Jun 17 13:13:00 2016
              State : clean
    Active Devices : 3
    Working Devices : 3
    Failed Devices : 0
      Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K


        Number   Major   Minor   RaidDevice State
          0       8       17        0      active sync   /dev/sdb1
          1       8       33        1      active sync   /dev/sdc1
          3       8       81        2      active sync   /dev/sdf1

Re: [dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[Milan Broz]

On 06/21/2016 12:15 AM, Joe Hillenbrand wrote:
> I've been trying and trying to add a new key to one of my devices so
> that I can automount,
> but it consistently fails.
> 
> I run the following command. It hangs for ~85 minutes every time (I've
> tried at least 5 times)

Does it work if you just open the device? For example try

cryptsetup luksOpen /dev/md1 --test-passphrase

> and then fails with the "Not compatible PBKDF2 options" message and exit code 1.
> 
>     $ sudo cryptsetup luksAddKey /dev/md1 /etc/.md1.key
>     Enter any existing passphrase:
>     Not compatible PBKDF2 options (using hash algorithm sha256).

Could you please send output with added --debug switch?

It is failing PBKDF2 benchmark here so please try and send output
of these commands as well:

cryptsetup benchmark -h sha256 --debug
cryptsetup benchmark -h sha1 --debug

What distro and version it is?
Do you compile anything (kernel, cryptsetup, library...) yourself?

> I've tried generating several different keys of different sizes.
> Exact same behavior for all of them.

Key format is not the problem here.

Milan

Re: [dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[Joe Hillenbrand]

I gathered the requested output. Unfortunately, it doesn't look very
helpful. Each command took about +95 minutes.

On Mon, Jun 20, 2016 at 8:24 PM, Milan Broz <gmazyland@gmail.com> wrote:
>
> Does it work if you just open the device? For example try
>
> cryptsetup luksOpen /dev/md1 --test-passphrase

Yes, I can open and mount it fine.

>
>> and then fails with the "Not compatible PBKDF2 options" message and exit code 1.
>>
>>     $ sudo cryptsetup luksAddKey /dev/md1 /etc/.md1.key
>>     Enter any existing passphrase:
>>     Not compatible PBKDF2 options (using hash algorithm sha256).
>
> Could you please send output with added --debug switch?

# cryptsetup 1.7.1 processing "cryptsetup luksAddKey --debug /dev/md1
/etc/.md1.key"
# Running command luksAddKey.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device /dev/md1 context.
# Trying to open and read device /dev/md1 with direct-io.
# Initialising device-mapper backend library.
# Trying to load LUKS1 crypt type from device /dev/md1.
# Crypto backend (gcrypt 1.7.1) initialized in cryptsetup library version 1.7.1.
# Detected kernel Linux 4.6.2-1-ARCH x86_64.
# Reading LUKS header of size 1024 from device /dev/md1
# Key length 32, device size 11720126464 sectors, header size 2050 sectors.
# Password verification disabled.
# Timeout set to 0 miliseconds.
# Iteration time set to 2000 miliseconds.
# Interactive passphrase entry requested.
Enter any existing passphrase:
# Checking volume  [keyslot -1] using passphrase.
# Trying to open key slot 0 [ACTIVE_LAST].
# Reading key slot 0 area.
# Using userspace crypto wrapper to access keyslot area.
Key slot 0 unlocked.
# File descriptor passphrase entry requested.
# Adding new keyslot, existing passphrase provided,new passphrase provided.
# Trying to open key slot 0 [ACTIVE_LAST].
# Reading key slot 0 area.
# Using userspace crypto wrapper to access keyslot area.
Key slot 0 unlocked.
# Calculating data for key slot 1
Not compatible PBKDF2 options (using hash algorithm sha256).
# Releasing crypt device /dev/md1 context.
# Releasing device-mapper backend.
# Unlocking memory.
Command failed with code 22: Invalid argument


> It is failing PBKDF2 benchmark here so please try and send output
> of these commands as well:
>
> cryptsetup benchmark -h sha256 --debug

# cryptsetup 1.7.1 processing "cryptsetup benchmark -h sha256 --debug"
# Running command benchmark.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Tests are approximate using memory only (no storage IO).
# Crypto backend (gcrypt 1.7.1) initialized in cryptsetup library version 1.7.1.
# Detected kernel Linux 4.6.2-1-ARCH x86_64.
PBKDF2-sha256        N/A
Command failed with code 22.

> cryptsetup benchmark -h sha1 --debug

# cryptsetup 1.7.1 processing "cryptsetup benchmark -h sha1 --debug"
# Running command benchmark.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Tests are approximate using memory only (no storage IO).
]# Crypto backend (gcrypt 1.7.1) initialized in cryptsetup library
version 1.7.1.
# Detected kernel Linux 4.6.2-1-ARCH x86_64.
PBKDF2-sha1          N/A
Command failed with code 22.

> What distro and version it is?

Arch

> Do you compile anything (kernel, cryptsetup, library...) yourself?

No. It's all stock.

Re: [dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[Milan Broz]

On 06/21/2016 05:51 PM, Joe Hillenbrand wrote:
> I gathered the requested output. Unfortunately, it doesn't look very
> helpful. Each command took about +95 minutes.

Thanks, there are versions of backend libraries I need, so it is helpful.
ALso it proves that for unlocking crypto works.

>> cryptsetup benchmark -h sha256 --debug
> 
> # cryptsetup 1.7.1 processing "cryptsetup benchmark -h sha256 --debug"
> # Running command benchmark.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Tests are approximate using memory only (no storage IO).
> # Crypto backend (gcrypt 1.7.1) initialized in cryptsetup library version 1.7.1.
> # Detected kernel Linux 4.6.2-1-ARCH x86_64.
> PBKDF2-sha256        N/A
> Command failed with code 22.

It is failing the benchmark here, so there is probably another bug in benchmarking.
What CPU is that? Is it some very fast machine? (paste lscpu output if possible).

What is number of iterations for the used slot? - you masked that out in the
first mail.

I need to either reproduce myself or I have to add some more debugging output
there.

If I add some debugging code, can you compile it and run it on your hw?
(Or I can provide debug binaries if you prefer that.)

Thanks,
Milan

Re: [dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[Joe Hillenbrand]

What luck! I was just about to respond with an update. Arch updated to
1.7.2, no change in behavior.

I was curious if gcrypt was the culprit so I built the source with
"--with-crypto-backend=openssl". It didn't help.

> It is failing the benchmark here, so there is probably another bug in benchmarking.
> What CPU is that? Is it some very fast machine? (paste lscpu output if possible).

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45
Model name:            Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz
Stepping:              7
CPU MHz:               3599.841
CPU max MHz:           3800.0000
CPU min MHz:           1200.0000
BogoMIPS:              7202.66
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              10240K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht
tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1
sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb
tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts


> What is number of iterations for the used slot?

2649416

> If I add some debugging code, can you compile it and run it on your hw?

I'd love to.

Re: [dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[Milan Broz]

On 06/21/2016 05:51 PM, Joe Hillenbrand wrote:
> I gathered the requested output. Unfortunately, it doesn't look very
> helpful. Each command took about +95 minutes.

Just for the archive, that particular system has apparently broken getrusage() call
that returns zeroed process/system time, so benchmark cannot work there properly.

I would like to know why but apparently this is not cryptsetup failure.

Milan


> 
> On Mon, Jun 20, 2016 at 8:24 PM, Milan Broz <gmazyland@gmail.com> wrote:
>>
>> Does it work if you just open the device? For example try
>>
>> cryptsetup luksOpen /dev/md1 --test-passphrase
> 
> Yes, I can open and mount it fine.
> 
>>
>>> and then fails with the "Not compatible PBKDF2 options" message and exit code 1.
>>>
>>>     $ sudo cryptsetup luksAddKey /dev/md1 /etc/.md1.key
>>>     Enter any existing passphrase:
>>>     Not compatible PBKDF2 options (using hash algorithm sha256).
>>
>> Could you please send output with added --debug switch?
> 
> # cryptsetup 1.7.1 processing "cryptsetup luksAddKey --debug /dev/md1
> /etc/.md1.key"
> # Running command luksAddKey.
> # Locking memory.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Allocating crypt device /dev/md1 context.
> # Trying to open and read device /dev/md1 with direct-io.
> # Initialising device-mapper backend library.
> # Trying to load LUKS1 crypt type from device /dev/md1.
> # Crypto backend (gcrypt 1.7.1) initialized in cryptsetup library version 1.7.1.
> # Detected kernel Linux 4.6.2-1-ARCH x86_64.
> # Reading LUKS header of size 1024 from device /dev/md1
> # Key length 32, device size 11720126464 sectors, header size 2050 sectors.
> # Password verification disabled.
> # Timeout set to 0 miliseconds.
> # Iteration time set to 2000 miliseconds.
> # Interactive passphrase entry requested.
> Enter any existing passphrase:
> # Checking volume  [keyslot -1] using passphrase.
> # Trying to open key slot 0 [ACTIVE_LAST].
> # Reading key slot 0 area.
> # Using userspace crypto wrapper to access keyslot area.
> Key slot 0 unlocked.
> # File descriptor passphrase entry requested.
> # Adding new keyslot, existing passphrase provided,new passphrase provided.
> # Trying to open key slot 0 [ACTIVE_LAST].
> # Reading key slot 0 area.
> # Using userspace crypto wrapper to access keyslot area.
> Key slot 0 unlocked.
> # Calculating data for key slot 1
> Not compatible PBKDF2 options (using hash algorithm sha256).
> # Releasing crypt device /dev/md1 context.
> # Releasing device-mapper backend.
> # Unlocking memory.
> Command failed with code 22: Invalid argument
> 
> 
>> It is failing PBKDF2 benchmark here so please try and send output
>> of these commands as well:
>>
>> cryptsetup benchmark -h sha256 --debug
> 
> # cryptsetup 1.7.1 processing "cryptsetup benchmark -h sha256 --debug"
> # Running command benchmark.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Tests are approximate using memory only (no storage IO).
> # Crypto backend (gcrypt 1.7.1) initialized in cryptsetup library version 1.7.1.
> # Detected kernel Linux 4.6.2-1-ARCH x86_64.
> PBKDF2-sha256        N/A
> Command failed with code 22.
> 
>> cryptsetup benchmark -h sha1 --debug
> 
> # cryptsetup 1.7.1 processing "cryptsetup benchmark -h sha1 --debug"
> # Running command benchmark.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Tests are approximate using memory only (no storage IO).
> ]# Crypto backend (gcrypt 1.7.1) initialized in cryptsetup library
> version 1.7.1.
> # Detected kernel Linux 4.6.2-1-ARCH x86_64.
> PBKDF2-sha1          N/A
> Command failed with code 22.
> 
>> What distro and version it is?
> 
> Arch
> 
>> Do you compile anything (kernel, cryptsetup, library...) yourself?
> 
> No. It's all stock.
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

Re: [dm-crypt] luksAddKey hangs for ~85 minutes and then fails

[Joe Hillenbrand]

In case someones stumbles across this thread, it turns out my
motherboard hadn't been updated since 2012 and there were some Intel
Microcode updates I was missing. I updated my BIOS firmware and
everything is working again.

Milan, Thank you so much for helping me dig into this.

On Wed, Jun 22, 2016 at 11:29 PM, Milan Broz <gmazyland@gmail.com> wrote:
> On 06/21/2016 05:51 PM, Joe Hillenbrand wrote:
>> I gathered the requested output. Unfortunately, it doesn't look very
>> helpful. Each command took about +95 minutes.
>
> Just for the archive, that particular system has apparently broken getrusage() call
> that returns zeroed process/system time, so benchmark cannot work there properly.
>
> I would like to know why but apparently this is not cryptsetup failure.
>
> Milan
>
>