Possible severe bug in the device mapper?

Possible severe bug in the device mapper?

[Andreas Heinlein]

Hello,

we observed the following behaviour:

1. Create a plain file on a removable drive, e.g. USB pendrive or
hotpluggable SATA drive (dd if=/dev/zero of=/media/somedrive/testfile
bs=1 count=1 seek=100000000). Don't make it bigger than ~50% of free memory.
2. Create a loopback device with this file (losetup -f
/media/somedrive/testfile)
3. Make this /dev/loopX a device mapper target. Tested with truecrypt,
dm_crypt (cryptsetup luksFormat ... and luksOpen ...) and lvm (pvcreate
/dev/loopX, vgcreate testvg /dev/loopX, lvcreate -N testlv testvg).
4. Create a filesystem on top of that mapping (mkfs.ext3 /dev/mapper/...)
5. Mount that filesystem
6. Copy some data to it
7. Remove the drive without unmounting anything

Expected behavior: Writing to the still mounted mapping should at least
give an error
Actual behavior: Reading from and writing to the mapping is still
possible, as long as the written data fits into the page cache. Even
unmounting works cleanly. When you remount the whole thing, data is
obviously lost.

This does not happen with whole block devices as targets, which is AFAIK
the case in most uses of the device mapper. We experienced this with
truecrypt volumes, where the user accidentally removed a drive
containing an encrypted TrueCrypt container. He plugged it back in, and
since everything worked, he continued to work. Only after unmounting and
re-opening the container the next day, he noticed that several hours of
work had been lost because nothing had actually been written. We are
using Ubuntu 10.04 LTS with linux 2.6.32.

Since this is reproducible with other device mapper targets, I am sure
this is not a truecrypt problem. It might be related to udev, which
AFAIK should notify higher levels when a device is removed.

IMHO, this is a rather severe problem, since loss of connection to a
device may also occur by bad cables, connections etc.

Do you have any explanation for that?

Thanks,
Andreas

Re: Possible severe bug in the device mapper?

[Milan Broz]

On 01/31/2011 10:30 AM, Andreas Heinlein wrote:

> Expected behavior: Writing to the still mounted mapping should at least
> give an error

It should, at least the first flush must return error.

Reading from page cache will probably not hit DM code at all, so it 
is possible that you can read some data without failure.

Can you reproduce this on recent kernel? 2.6.32 is quite old and I expect
it is some heavy patches distro kernel.

> This does not happen with whole block devices as targets, which is AFAIK
> the case in most uses of the device mapper.

What do you mean by "whole block devices" here? Like not partitioned disk?

> Since this is reproducible with other device mapper targets, I am sure
> this is not a truecrypt problem. It might be related to udev, which
> AFAIK should notify higher levels when a device is removed.

Yes, Truecrypt uses dm-crypt here. Udev should get remove event and remove
all symlinks, also there can be some higher level (udisks) which forcily
removes crypt mapping when underlying device disappears.

But it still must return error on write when underlying device disappeared.

Milan

Re: Possible severe bug in the device mapper?

[Joe Thornber]

On Mon, 2011-01-31 at 10:30 +0100, Andreas Heinlein wrote:
> 2. Create a loopback device with this file (losetup -f
> /media/somedrive/testfile) 

I'm just trying to understand your complaint here.  If device-mapper
wasn't in the equation here, and you were using the loop-back device
directly (i.e. by putting a filesystem on it and mounting).  Then you
pulled the removable device, what happens?

- Joe

Re: Possible severe bug in the device mapper?

[Andreas Heinlein]

Am 31.01.2011 12:20, schrieb Joe Thornber:
> On Mon, 2011-01-31 at 10:30 +0100, Andreas Heinlein wrote:
>> 2. Create a loopback device with this file (losetup -f
>> /media/somedrive/testfile) 
> I'm just trying to understand your complaint here.  If device-mapper
> wasn't in the equation here, and you were using the loop-back device
> directly (i.e. by putting a filesystem on it and mounting).  Then you
> pulled the removable device, what happens?
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
Thank you for your advice. Actually, it still happens without
device-mapper, so I'm at the wrong place here. Sorry for that. I did
some more tests with different distros and releases (Ubuntu 10.10 and
11.04alpha1, OpenSuse 11.3, Fedora 14), and almost all showed this
behaviour; only OpenSuse immediatley set the device to read-only on the
first write attempt, Ubuntu 11.04alpha1 crashed, but this may be because
of the alpha status. Still not quite what one would expect, though.

I will continue looking for help, can you point me in some direction?
'losetup' is part of util-linux, but I doubt this has actually anything
to do withit. The loopback device code itself is part of the kernel, right?

Thanks,
Andreas

Re: Possible severe bug in the device mapper?

[Joe Thornber]

On Tue, 2011-02-01 at 10:30 +0100, Andreas Heinlein wrote:
> I will continue looking for help, can you point me in some direction?
> 'losetup' is part of util-linux, but I doubt this has actually
> anything
> to do withit. The loopback device code itself is part of the kernel,
> right? 

Yes, you should try and find someone who admits to being the loopback
device maintainer and try and have a chat with them.  Also it would be
interesting to know how OpenSuse are doing a better job of handling
this.  For instance is it a userland solution or have they changed
something in kernel?

- Joe

Re: Possible severe bug in the device mapper?

[Zdenek Kabelac]

Dne 1.2.2011 10:30, Andreas Heinlein napsal(a):
> Am 31.01.2011 12:20, schrieb Joe Thornber:
>> On Mon, 2011-01-31 at 10:30 +0100, Andreas Heinlein wrote:
>>> 2. Create a loopback device with this file (losetup -f
>>> /media/somedrive/testfile) 
>> I'm just trying to understand your complaint here.  If device-mapper
>> wasn't in the equation here, and you were using the loop-back device
>> directly (i.e. by putting a filesystem on it and mounting).  Then you
>> pulled the removable device, what happens?
>>
>> - Joe
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
> Thank you for your advice. Actually, it still happens without
> device-mapper, so I'm at the wrong place here. Sorry for that. I did
> some more tests with different distros and releases (Ubuntu 10.10 and
> 11.04alpha1, OpenSuse 11.3, Fedora 14), and almost all showed this
> behaviour; only OpenSuse immediatley set the device to read-only on the
> first write attempt, Ubuntu 11.04alpha1 crashed, but this may be because
> of the alpha status. Still not quite what one would expect, though.

Do you mean 'device' or filesystem as read-only?
For filesystem you may easily switch error behavior so you will not destroy
your data when inconsistency is detected (tune2fs -e remount-ro)

Zdenek

Re: Possible severe bug in the device mapper?

[Andreas Heinlein]

> Do you mean 'device' or filesystem as read-only?
> For filesystem you may easily switch error behavior so you will not destroy
> your data when inconsistency is detected (tune2fs -e remount-ro)
>
> Zdenek
I mean filesystem. But the strange thing about my observation is that
obviously no inconsistency is ever detected, except for openSuSE. The
filesystem on the loopback device under Ubuntu and Fedora, backed by the
no-longer-existing usb drive, can be happily read from and written to
and even be fsck'ed without any problems. Only after completely
unmounting it, deleting the loopback device, then re-inserting the drive
and loop-mounting again you can see that data was lost. 'remount-ro' is
set in all cases.

Andreas