[dm-crypt] Device ... is still in use.

[dm-crypt] Device ... is still in use.

[Jean-Paul Calderone]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

Hello,

I have a luks device on removable media that has somehow become stuck on my
system.  The physical media has been disconnected but the luks device
continues to exist and I cannot remove it.

cryptosetup close fails:

$ sudo cryptsetup close luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> Device luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 is still in use.


I don't understand what is still using the device.  It is not mounted
anywhere:

$ mount | grep luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> $


How do I close this device?

Thanks,
Jean-Paul

[-- Attachment #2: Type: text/html, Size: 1018 bytes --]

Re: [dm-crypt] Device ... is still in use.

[Milan Broz]

On 19/08/2019 14:59, Jean-Paul Calderone wrote:
> Hello,
> 
> I have a luks device on removable media that has somehow become stuck on my system.  The physical media has been disconnected but the luks device continues to exist and I cannot remove it.
> 
> cryptosetup close fails:
> 
>     $ sudo cryptsetup close luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>     Device luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 is still in use.
> 
> 
> I don't understand what is still using the device.  It is not mounted anywhere:
> 
>     $ mount | grep luks-64ca6f08-82c9-4571-a976-f37a83aca3b1

See output of lsblk - it can have mapped partitions or lvm over it,
you need to deactivate it.

Also check lsof output if some process is not stuck keeping something open there.

Milan

Re: [dm-crypt] Device ... is still in use.

[Jean-Paul Calderone]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

On Mon, Aug 19, 2019 at 9:16 AM Milan Broz <gmazyland@gmail.com> wrote:

> On 19/08/2019 14:59, Jean-Paul Calderone wrote:
> > Hello,
> >
> > I have a luks device on removable media that has somehow become stuck on
> my system.  The physical media has been disconnected but the luks device
> continues to exist and I cannot remove it.
> >
> > cryptosetup close fails:
> >
> >     $ sudo cryptsetup close luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> >     Device luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 is still in use.
> >
> >
> > I don't understand what is still using the device.  It is not mounted
> anywhere:
> >
> >     $ mount | grep luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>
> See output of lsblk - it can have mapped partitions or lvm over it,
> you need to deactivate it.
>
> Also check lsof output if some process is not stuck keeping something open
> there.
>
>
Thanks for the suggestion.  `lsblk --all` doesn't mention the device -
either the mapped luks device or the underlying block device.  It mentions
some ram devices, some loopback devices (but `losetup --all` says none are
in use), and my internal storage devices.

lsof is similarly quiet:

$ sudo lsof /dev/mapper/luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> $


Poking further with dmsetup it looks like /dev/dm-0 might be what's keeping
the luks device busy?  I'm not completely sure I'm interpreting this output
right though:

$ sudo dmsetup ls
luks-64ca6f08-82c9-4571-a976-f37a83aca3b1       (254:0)
$ ls -l /dev/dm-0
brw-rw---- 1 root disk 254, 0 Aug 18 19:48 /dev/dm-0
$ sudo lsof /dev/dm-0
$ mount | grep dm-0
$


 Thanks again,
Jean-Paul

[-- Attachment #2: Type: text/html, Size: 2639 bytes --]

Re: [dm-crypt] Device ... is still in use.

[Milan Broz]

On 19/08/2019 15:23, Jean-Paul Calderone wrote:
> On Mon, Aug 19, 2019 at 9:16 AM Milan Broz <gmazyland@gmail.com <mailto:gmazyland@gmail.com>> wrote:
> 
>     On 19/08/2019 14:59, Jean-Paul Calderone wrote:
>     > Hello,
>     >
>     > I have a luks device on removable media that has somehow become stuck on my system.  The physical media has been disconnected but the luks device continues to exist and I cannot remove it.
>     >
>     > cryptosetup close fails:
>     >
>     >     $ sudo cryptsetup close luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>     >     Device luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 is still in use.
>     >
>     >
>     > I don't understand what is still using the device.  It is not mounted anywhere:
>     >
>     >     $ mount | grep luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> 
>     See output of lsblk - it can have mapped partitions or lvm over it,
>     you need to deactivate it.
> 
>     Also check lsof output if some process is not stuck keeping something open there.
> 
> 
> Thanks for the suggestion.  `lsblk --all` doesn't mention the device - either the mapped luks device or the underlying block device.  It mentions some ram devices, some loopback devices (but `losetup --all` says none are in use), and my internal storage devices.

If you can see it as /dev/dm-0 and in /dev/mapper but lsblk does not show it, your system seems
to be misconfigured (lsblk should read internal kernel state from sysfs).

What says "dmsetup info -c" ?

The /dev/dm-0 *is* the luks device itself (/dev/mapper/luks* should be symlink to it).

Anyway, if you just need to remove it by force, use
  dmsetup remove luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
if it still complains, add "--force" option - it will remap it to error target removing all references to underlying devices and keys.

Also check syslog if there is any related message (crash or something that could explain it).

m.

Re: [dm-crypt] Device ... is still in use.

[Jean-Paul Calderone]

[-- Attachment #1: Type: text/plain, Size: 2941 bytes --]

On Mon, Aug 19, 2019 at 9:46 AM Milan Broz <gmazyland@gmail.com> wrote:

> On 19/08/2019 15:23, Jean-Paul Calderone wrote:
> > On Mon, Aug 19, 2019 at 9:16 AM Milan Broz <gmazyland@gmail.com <mailto:
> gmazyland@gmail.com>> wrote:
> >
> >     On 19/08/2019 14:59, Jean-Paul Calderone wrote:
> >     > Hello,
> >     >
> >     > I have a luks device on removable media that has somehow become
> stuck on my system.  The physical media has been disconnected but the luks
> device continues to exist and I cannot remove it.
> >     >
> >     > cryptosetup close fails:
> >     >
> >     >     $ sudo cryptsetup close
> luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> >     >     Device luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 is still in
> use.
> >     >
> >     >
> >     > I don't understand what is still using the device.  It is not
> mounted anywhere:
> >     >
> >     >     $ mount | grep luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> >
> >     See output of lsblk - it can have mapped partitions or lvm over it,
> >     you need to deactivate it.
> >
> >     Also check lsof output if some process is not stuck keeping
> something open there.
> >
> >
> > Thanks for the suggestion.  `lsblk --all` doesn't mention the device -
> either the mapped luks device or the underlying block device.  It mentions
> some ram devices, some loopback devices (but `losetup --all` says none are
> in use), and my internal storage devices.
>
> If you can see it as /dev/dm-0 and in /dev/mapper but lsblk does not show
> it, your system seems
> to be misconfigured (lsblk should read internal kernel state from sysfs).
>
> What says "dmsetup info -c" ?
>

$ sudo dmsetup info -c
Name                                      Maj Min Stat Open Targ Event
 UUID
luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 254   0 L--w    1    1      0
CRYPT-LUKS1-64ca6f0882c94571a976f37a83aca3b1-luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
$



>
> The /dev/dm-0 *is* the luks device itself (/dev/mapper/luks* should be
> symlink to it).
>
> Anyway, if you just need to remove it by force, use
>   dmsetup remove luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> if it still complains, add "--force" option - it will remap it to error
> target removing all references to underlying devices and keys.
>
> Also check syslog if there is any related message (crash or something that
> could explain it).
>

Looks like something bad going on at a lower level:

$ sudo dmsetup remove /dev/dm-0
device-mapper: remove ioctl on luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
 failed: Device or resource busy
Command failed.
$ sudo dmsetup remove --force /dev/dm-0
device-mapper: remove ioctl on luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
 failed: Device or resource busy
Command failed.
$ dmesg | tail -n 1
[1323040.325203] Buffer I/O error on dev dm-0, logical block 1968624, async
page read
$


And every `dmsetup remove --force` adds another `Buffer I/O error ...` like
the above.

Thanks,
Jean-Paul

[-- Attachment #2: Type: text/html, Size: 4813 bytes --]

Re: [dm-crypt] Device ... is still in use.

[Jean-Paul Calderone]

[-- Attachment #1: Type: text/plain, Size: 3401 bytes --]

I went ahead and rebooted to clear the state.  I couldn't remount the luks
device while the old stale one was present and I couldn't get any work done
w/o the device. :/  After reboot, it's fine... for now, at least.

Thanks for the assistance,
Jean-Paul

On Mon, Aug 19, 2019 at 9:57 AM Jean-Paul Calderone <
exarkun@twistedmatrix.com> wrote:

> On Mon, Aug 19, 2019 at 9:46 AM Milan Broz <gmazyland@gmail.com> wrote:
>
>> On 19/08/2019 15:23, Jean-Paul Calderone wrote:
>> > On Mon, Aug 19, 2019 at 9:16 AM Milan Broz <gmazyland@gmail.com
>> <mailto:gmazyland@gmail.com>> wrote:
>> >
>> >     On 19/08/2019 14:59, Jean-Paul Calderone wrote:
>> >     > Hello,
>> >     >
>> >     > I have a luks device on removable media that has somehow become
>> stuck on my system.  The physical media has been disconnected but the luks
>> device continues to exist and I cannot remove it.
>> >     >
>> >     > cryptosetup close fails:
>> >     >
>> >     >     $ sudo cryptsetup close
>> luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>> >     >     Device luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 is still in
>> use.
>> >     >
>> >     >
>> >     > I don't understand what is still using the device.  It is not
>> mounted anywhere:
>> >     >
>> >     >     $ mount | grep luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>> >
>> >     See output of lsblk - it can have mapped partitions or lvm over it,
>> >     you need to deactivate it.
>> >
>> >     Also check lsof output if some process is not stuck keeping
>> something open there.
>> >
>> >
>> > Thanks for the suggestion.  `lsblk --all` doesn't mention the device -
>> either the mapped luks device or the underlying block device.  It mentions
>> some ram devices, some loopback devices (but `losetup --all` says none are
>> in use), and my internal storage devices.
>>
>> If you can see it as /dev/dm-0 and in /dev/mapper but lsblk does not show
>> it, your system seems
>> to be misconfigured (lsblk should read internal kernel state from sysfs).
>>
>> What says "dmsetup info -c" ?
>>
>
> $ sudo dmsetup info -c
> Name                                      Maj Min Stat Open Targ Event
>  UUID
> luks-64ca6f08-82c9-4571-a976-f37a83aca3b1 254   0 L--w    1    1      0
> CRYPT-LUKS1-64ca6f0882c94571a976f37a83aca3b1-luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
> $
>
>
>
>>
>> The /dev/dm-0 *is* the luks device itself (/dev/mapper/luks* should be
>> symlink to it).
>>
>> Anyway, if you just need to remove it by force, use
>>   dmsetup remove luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>> if it still complains, add "--force" option - it will remap it to error
>> target removing all references to underlying devices and keys.
>>
>> Also check syslog if there is any related message (crash or something
>> that could explain it).
>>
>
> Looks like something bad going on at a lower level:
>
> $ sudo dmsetup remove /dev/dm-0
> device-mapper: remove ioctl on luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>  failed: Device or resource busy
> Command failed.
> $ sudo dmsetup remove --force /dev/dm-0
> device-mapper: remove ioctl on luks-64ca6f08-82c9-4571-a976-f37a83aca3b1
>  failed: Device or resource busy
> Command failed.
> $ dmesg | tail -n 1
> [1323040.325203] Buffer I/O error on dev dm-0, logical block 1968624,
> async page read
> $
>
>
> And every `dmsetup remove --force` adds another `Buffer I/O error ...`
> like the above.
>
> Thanks,
> Jean-Paul
>
>

[-- Attachment #2: Type: text/html, Size: 5531 bytes --]

Re: [dm-crypt] Device ... is still in use.

[Milan Broz]

On 19/08/2019 16:22, Jean-Paul Calderone wrote:
> I went ahead and rebooted to clear the state.  I couldn't remount the
> luks device while the old stale one was present and I couldn't get
> any work done w/o the device. :/  After reboot, it's fine... for now,
> at least.
Yes, that was probably the simplest solution.
Just for info - what distro it is and what kernel and cryptsetup version?
Did you compile anything (kernel, cryptsetup) manually?

Anyway, if you cannot see active LUKS device in lsblk output,
something is quite strange.

Milan

Re: [dm-crypt] Device ... is still in use.

[Jean-Paul Calderone]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Mon, Aug 19, 2019 at 11:48 AM Milan Broz <gmazyland@gmail.com> wrote:

> On 19/08/2019 16:22, Jean-Paul Calderone wrote:
> > I went ahead and rebooted to clear the state.  I couldn't remount the
> > luks device while the old stale one was present and I couldn't get
> > any work done w/o the device. :/  After reboot, it's fine... for now,
> > at least.
> Yes, that was probably the simplest solution.
> Just for info - what distro it is and what kernel and cryptsetup version?
> Did you compile anything (kernel, cryptsetup) manually?
>

NixOS 19.03
Linux 4.19.57
cryptsetup 2.0.6
dmsetup 1.02.153

I didn't customize my kernel or cryptsetup though by nature of NixOS it's
possible Linux was compiled (using NixOS specified configuration) on my
system.


> Anyway, if you cannot see active LUKS device in lsblk output,
> something is quite strange.
>

It certainly seems so to me.  If there's more information I can gather to
try to make the cause clear, I can try to gather it if the problem
reoccurs.

Jean-Paul

[-- Attachment #2: Type: text/html, Size: 2063 bytes --]