ceph-disk and /dev/dm-* permissions - race condition?

[Wyllys Ingersoll <wyllys.ingersoll@keepertech.com>]

We are running 10.2.3 with encrypted OSDs and journals using the old
(i.e. non-Luks) keys and are seeing issues with the ceph-osd processes
after a reboot of a storage server.  Our data and journals are on
separate partitions on the same disk.

After a reboot, sometimes the OSDs fail to start because of
permissions problems.  The /dev/dm-* devices come back with
permissions set to "root:disk" sometimes instead of "ceph:ceph".
Weirder still is that sometimes the ceph-osd will start and work in
spite of the incorrect perrmissions (root:disk) and other times they
will fail and the logs show permissions errors when trying to access
the journals. Sometimes half of the /dev/dm- devices are "root:disk"
and others are "ceph:ceph".  There's no clear pattern, so that's what
leads me to think its a race condition in the ceph_disk "dmcrypt_map"
function.

Is there a known issue with ceph-disk and/or ceph-osd related to
timing of the encrypted devices being setup and the permissions
getting changed to the ceph processes can access them?

Wyllys Ingersoll
Keeper Technology, LLC