[dm-crypt] veritysetup forward error correction failure

[dm-crypt] veritysetup forward error correction failure

[Tom Eccles]

Hello,

I have a question about veritysetup using forward error correction.

I created my dm-verity volume using

veritysetup --check-at-most-once --fec-device /dev/vdc --fec-roots 24 format
/dev/vdb1 /dev/vdd

Then I introduce a single bit error into /dev/vdb1.

If I then open the dm-verity device using

veritysetup --fec-device /dev/vdc --fec-roots 24 open /dev/vdb1 vroot
/dev/vdd <hash>

When I mount the device and inspect the corrupted file, the single bit error
cannot be corrected:
[  108.994804] device-mapper: verity-fec: 254:17: FEC 19300352: failed to
correct: -74
[  108.994834] device-mapper: verity: 254:17: data block 273411 is corrupted

However, if I verify the whole device using veritysetup it succeeds:

veritysetup --fec-device /dev/vdc --fec-roots 24 verify /dev/vdb1 /dev/vdd
<hash>
Verification failed at position 1119891456.
Verification of data area failed.
Found 1 repairable errors with FEC device

So I suspect that I have the wrong flags to veritysetup open. Can anybody
see where I am going wrong?

Thanks,
Tom

Re: [dm-crypt] veritysetup forward error correction failure

[Michael Kjörling]

On 17 Jul 2019 18:06 +0100, from tom.eccles@codethink.co.uk (Tom Eccles):
> However, if I verify the whole device using veritysetup it succeeds:
> 
> veritysetup --fec-device /dev/vdc --fec-roots 24 verify /dev/vdb1 /dev/vdd
> <hash>
> Verification failed at position 1119891456.
> Verification of data area failed.
> Found 1 repairable errors with FEC device
> 
> So I suspect that I have the wrong flags to veritysetup open. Can anybody
> see where I am going wrong?

I don't have an answer for you, but can you re-run the hash
verification with `--verbose` and/or `--debug`, as appropriate?

Also, please provide the output of `uname -a` and `veritysetup --version`.

-- 
Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
  “The most dangerous thought that you can have as a creative person
              is to think you know what you’re doing.” (Bret Victor)

Re: [dm-crypt] veritysetup forward error correction failure

[Milan Broz]

On 17/07/2019 19:06, Tom Eccles wrote:
> Hello,
> 
> I have a question about veritysetup using forward error correction.
> 
> I created my dm-verity volume using
> 
> veritysetup --check-at-most-once --fec-device /dev/vdc --fec-roots 24 format
> /dev/vdb1 /dev/vdd
> 
> Then I introduce a single bit error into /dev/vdb1.
> 
> If I then open the dm-verity device using
> 
> veritysetup --fec-device /dev/vdc --fec-roots 24 open /dev/vdb1 vroot
> /dev/vdd <hash>
> 
> When I mount the device and inspect the corrupted file, the single bit error
> cannot be corrected:
> [  108.994804] device-mapper: verity-fec: 254:17: FEC 19300352: failed to
> correct: -74
> [  108.994834] device-mapper: verity: 254:17: data block 273411 is corrupted
> 
> However, if I verify the whole device using veritysetup it succeeds:
> 
> veritysetup --fec-device /dev/vdc --fec-roots 24 verify /dev/vdb1 /dev/vdd
> <hash>
> Verification failed at position 1119891456.
> Verification of data area failed.
> Found 1 repairable errors with FEC device
> 
> So I suspect that I have the wrong flags to veritysetup open. Can anybody
> see where I am going wrong?

Well, the FEC code in userspace and kernel differs, it is quite possible
that there could be some misconfiguration or a bug.

Please be sure you are testing access to the image and not
some cached data in meory - better flush all caches between runs with the command
"echo 3 > /proc/sys/vm/drop_caches".

If you can still reproduce it, please send version of the utility and
kernel (and --debug output as suggested in another mail) and if you have some
data/hash/fec images that can be used to reproduce it, let me know where I can find it.

Thanks,
Milan

Re: [dm-crypt] veritysetup forward error correction failure

[Tom Eccles]

Thanks all

On 17/07/2019 8:49 pm, Milan Broz wrote:
> Well, the FEC code in userspace and kernel differs, it is quite possible
> that there could be some misconfiguration or a bug.
> 
> Please be sure you are testing access to the image and not
> some cached data in meory - better flush all caches between runs with the command
> "echo 3 > /proc/sys/vm/drop_caches".

This made no difference. I have also tried rebooting.

> If you can still reproduce it, please send version of the utility and
> kernel (and --debug output as suggested in another mail) and if you have some
> data/hash/fec images that can be used to reproduce it, let me know where I can find it.

I am running Debian Buster (stable). I recompiled the kernel to add support
for FEC devices, and building dm-verity and virtioblk support into the
kernel image (not modules). Versions are:

tomeccles@debian-kernel-builder:~$ uname -a
Linux debian-kernel-builder 4.19.37-verity #6 SMP Wed Jul 17 11:47:38 BST
2019 x86_64 GNU/Linux
tomeccles@debian-kernel-builder:~$ /usr/sbin/veritysetup --version
veritysetup 2.1.0

I can also reproduce the issue using veritysetup 2.2.0-rc1.

Running veritysetup verify with --verbose and --debug:

tomeccles@debian-kernel-builder:~$ sudo veritysetup --verbose --debug
--fec-device /dev/vdc --fec-roots 24 verify /dev/vdb1 /dev/vdd
04c260f3595e80104ad5e389df12f12b17a70c6b5284841c7260f859767948bd
# cryptsetup 2.1.0 processing "veritysetup --verbose --debug --fec-device
/dev/vdc --fec-roots 24 verify /dev/vdb1 /dev/vdd
04c260f3595e80104ad5e389df12f12b17a70c6b5284841c7260f859767948bd"
# Running command verify.
# Allocating context for crypt device /dev/vdd.
# Trying to open and read device /dev/vdd with direct-io.
# Initialising device-mapper backend library.
# Setting ciphertext data device to /dev/vdb1.
# Trying to open and read device /dev/vdb1 with direct-io.
# Trying to load VERITY crypt type from device /dev/vdd.
# Crypto backend (OpenSSL 1.1.1c  28 May 2019) initialized in cryptsetup
library version 2.1.0.
# Detected kernel Linux 4.19.37-verity x86_64.
# Reading VERITY header of size 512 on device /dev/vdd, offset 0.
# Trying to open and read device /dev/vdc with direct-io.
# Checking volume  by volume key.
# Trying to activate VERITY device [none] using hash sha256.
# Verification of data in userspace required.
# Hash verification sha256, data device /dev/vdb1, data blocks 1310208,
hash_device /dev/vdd, offset 1.
# Using 3 hash levels.
# Data device size required: 5366611968 bytes.
# Hash device size required: 42262528 bytes.
Verification failed at position 1119891456.
Verification of data area failed.
# Verification failed, trying to repair with FEC device.
Found 1 repairable errors with FEC device.
# Releasing crypt device /dev/vdd context.
# Releasing device-mapper backend.
Command successful.

I will try to reproduce the issue with smaller images and provide these later.

Also, I tried adding some prints to veritysetup and the kernel, and on the
face of it, the reed-solomon state (struct rs in veritysetup) seems the
same. I have checked:
- mm = 8
- nn = 255
- nroots = 24
- fcr = 0
- prim = 1
- iprim = 1
The kernel does not have an equivalent of the pad field.

Thanks,
Tom

Re: [dm-crypt] veritysetup forward error correction failure

[Milan Broz]

On 18/07/2019 11:10, Tom Eccles wrote:
> tomeccles@debian-kernel-builder:~$ uname -a
> Linux debian-kernel-builder 4.19.37-verity #6 SMP Wed Jul 17 11:47:38 BST
> 2019 x86_64 GNU/Linux
> tomeccles@debian-kernel-builder:~$ /usr/sbin/veritysetup --version
> veritysetup 2.1.0
> 
> I can also reproduce the issue using veritysetup 2.2.0-rc1.

Thanks, the best would be to create an issue for cryptsetup project
https://gitlab.com/cryptsetup/cryptsetup/issues
(if it is a kernel bug we report it there later)

I will need some images (or a script to create them) to reproduce it though.

...

> Also, I tried adding some prints to veritysetup and the kernel, and on the
> face of it, the reed-solomon state (struct rs in veritysetup) seems the
> same.

AFAIK the kernel and our code was based on the same libfec code,
so I am quite interested to find out what is the difference here.

(It can take some time though, the userspace FEC image verification was
implemented by my student that already disappeared from the project :)

Thanks,
Milan

Re: [dm-crypt] veritysetup forward error correction failure

[Tom Eccles]

On 18/07/2019 10:50 am, Milan Broz wrote:
> On 18/07/2019 11:10, Tom Eccles wrote:
>> tomeccles@debian-kernel-builder:~$ uname -a
>> Linux debian-kernel-builder 4.19.37-verity #6 SMP Wed Jul 17 11:47:38 BST
>> 2019 x86_64 GNU/Linux
>> tomeccles@debian-kernel-builder:~$ /usr/sbin/veritysetup --version
>> veritysetup 2.1.0
>>
>> I can also reproduce the issue using veritysetup 2.2.0-rc1.
> 
> Thanks, the best would be to create an issue for cryptsetup project
> https://gitlab.com/cryptsetup/cryptsetup/issues
> (if it is a kernel bug we report it there later)

I will create an issue once I have images to reproduce the issue.

> I will need some images (or a script to create them) to reproduce it though.

I am working on this. I want to see if I can reproduce with much smaller
filesystem images so as to keep things simple (currently I am using a full
Debian rootfs).

> ...
> 
>> Also, I tried adding some prints to veritysetup and the kernel, and on the
>> face of it, the reed-solomon state (struct rs in veritysetup) seems the
>> same.
> 
> AFAIK the kernel and our code was based on the same libfec code,
> so I am quite interested to find out what is the difference here.

See struct rs_codec in kernel/include/linux/rslib.h:34
/**
 * struct rs_codec - rs codec data
 *
 * @mm:		Bits per symbol
 * @nn:		Symbols per block (= (1<<mm)-1)
 * @alpha_to:	log lookup table
 * @index_of:	Antilog lookup table
 * @genpoly:	Generator polynomial
 * @nroots:	Number of generator roots = number of parity symbols
 * @fcr:	First consecutive root, index form
 * @prim:	Primitive element, index form
 * @iprim:	prim-th root of 1, index form
 * @gfpoly:	The primitive generator polynominal
 * @gffunc:	Function to generate the field, if non-canonical representation
 * @users:	Users of this structure
 * @list:	List entry for the rs codec list
*/
struct rs_codec {
	int		mm;
	int		nn;
	uint16_t	*alpha_to;
	uint16_t	*index_of;
	uint16_t	*genpoly;
	int		nroots;
	int		fcr;
	int		prim;
	int		iprim;
	int		gfpoly;
	int		(*gffunc)(int);
	int		users;
	struct list_head list;
};

Vs cryptsetup/lib/verity/rs.h:34
/* Reed-Solomon codec control block */
struct rs {
	int mm;          /* Bits per symbol */
	int nn;          /* Symbols per block (= (1<<mm)-1) */
	data_t *alpha_to;/* log lookup table */
	data_t *index_of;/* Antilog lookup table */
	data_t *genpoly; /* Generator polynomial */
	int nroots;      /* Number of generator roots = number of parity symbols */
	int fcr;         /* First consecutive root, index form */
	int prim;        /* Primitive element, index form */
	int iprim;       /* prim-th root of 1, index form */
	int pad;         /* Padding bytes in shortened block */
};

> (It can take some time though, the userspace FEC image verification was
> implemented by my student that already disappeared from the project :)
> 
> Thanks,
> Milan
> 

Re: [dm-crypt] veritysetup forward error correction failure

[Tom Eccles]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On 17/07/2019 8:49 pm, Milan Broz wrote:
> If you can still reproduce it, please send version of the utility and
> kernel (and --debug output as suggested in another mail) and if you have some
> data/hash/fec images that can be used to reproduce it, let me know where I can find it.

Attached is a bash script gen_image.sh which should reproduce the issue.
find_and_corrupt.c should be in the working directory when gen_image.sh is
run. find_and_corrupt.c corrupts the disk image by simply searching for a
known string and introducing an error (see the diff outputted by gen_image.sh).

See in particular the error and dmesg sample when reading the corrupted file
(gen_image.sh:77) and the success when running veritysetup verify
(gen_image.sh:86).

I have created an issue at https://gitlab.com/cryptsetup/cryptsetup/issues/462

Thanks,
Tom

[-- Attachment #2: find_and_corrupt.c --]
[-- Type: text/x-csrc, Size: 3883 bytes --]

#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

// n <-> o is 0x6e <-> 0x6f so it is a single bit change
#define SEARCH_TEXT  "Can you spot this error?"
#define REPLACE_TEXT "Cao you spot this error?"
#define READ_SIZE    4096

// command line args
enum {
	PROG_NAME,
	PATH,
	MAX_ARGS
};

// so that we don't spend all our time in syscall overhead
struct read_buf {
	char *buffer;
	ssize_t len;
	ssize_t read_ptr;
	int fd;
};

static struct read_buf *read_buf_init(ssize_t size, int fd) {
	if (fd < 0) {
		return NULL;
	}

	struct read_buf *buf = malloc(sizeof(*buf));
	if (buf == NULL) {
		return NULL;
	}

	buf->buffer = malloc(size);
	if (buf->buffer == NULL) {
		free(buf);
		return NULL;
	}

	buf->len = size;
	buf->read_ptr = -1;
	buf->fd = fd;
	return buf;
}

static void read_buf_free(struct read_buf *buf) {
	free(buf->buffer);
	free(buf);
	// doesn't close fd
}

static inline int read_buf_getc(struct read_buf *buf, char *out) {
	// if we need to do another read
	if (buf->read_ptr == -1 || buf->read_ptr == buf->len) {
		// TODO partial read
		ssize_t count = read(buf->fd, buf->buffer, buf->len);
		if (count != buf->len) {
			fprintf(stderr, "Read %lli, wanted %lli: %s\n",
				(long long) count, (long long) buf->len,
				strerror(errno));
			return EXIT_FAILURE;
		}
		buf->read_ptr = 0;
	}

	*out = buf->buffer[buf->read_ptr++];
	return EXIT_SUCCESS;
}

// seek through fd until we find text.
// Position the file pointer at the start of the text
static int find_text(const int fd, const char *text)
{
	int ret = EXIT_FAILURE;
	const size_t len = strlen(text);
	size_t n_matching = 0;

	off_t offset = lseek(fd, 0, SEEK_CUR);;
	if (offset == -1) {
		fprintf(stderr, "Failed to get current offset: %s\n",
			strerror(errno));
		return EXIT_FAILURE;
	}

	struct read_buf *buf = read_buf_init(READ_SIZE, fd);
	if (buf == NULL) {
		fprintf(stderr, "Couldn't allocate read_buf\n");
		return EXIT_FAILURE;
	}

	printf("Searching for \"%s\"\n", text);

	while (n_matching < len) {
		char readc;

		if (read_buf_getc(buf, &readc) != EXIT_SUCCESS) {
			fprintf(stderr, "read_buf_getc failed\n");
			goto free_buffer;
		}
		++offset;

		if (readc == text[n_matching]) {
			++n_matching;
			if (n_matching == len) {
				break;
			}
		} else {
			// not a match
			if (n_matching) {
				n_matching = 0;
			}
		}
	}

	// go back to the beginning of the text
	off_t err = lseek(fd, offset - len, SEEK_SET);
	if (err == -1) {
		fprintf(stderr, "Lseek error: %s\n", strerror(errno));
		goto free_buffer;
	}

	printf("Found at %lli!\n", (long long) offset);
	ret = EXIT_SUCCESS;

free_buffer:
	read_buf_free(buf);
	return ret;
}

static int find_and_replace(const int fd, const char *search_text,
		     const char *replace_text)
{
	if (find_text(fd, search_text) != EXIT_SUCCESS) {
		return EXIT_FAILURE;
	}

	const ssize_t len = strlen(replace_text);

	printf("Overwriting with \"%s\"\n", replace_text);
	ssize_t count = write(fd, replace_text, len);
	// TODO partial write
	if (count != len) {
		fprintf(stderr, "write error: %s\n", strerror(errno));
		return EXIT_FAILURE;
	}

	return EXIT_SUCCESS;
}

static void print_help(const char *name)
{
	fprintf(stderr, "Usage: %s FILE\n", name);
}

int main(int argc, char **argv)
{
	int ret = EXIT_FAILURE;

	if (argc != MAX_ARGS) {
		print_help(argv[PROG_NAME]);
		return EXIT_FAILURE;
	}

	if (strcmp("--help", argv[1]) == 0 || strcmp("-h", argv[1]) == 0) {
		print_help(argv[PROG_NAME]);
		return EXIT_SUCCESS;
	}
	const char *path = argv[PATH];

	int fd = open(path, O_RDWR);
	if (fd == -1) {
		int err = errno;
		fprintf(stderr, "Can't open %s: %i: %s\n", path, err, strerror(err));
		return EXIT_FAILURE;
	}

	ret = find_and_replace(fd, SEARCH_TEXT, REPLACE_TEXT);

	close(fd);

	return ret;
}

[-- Attachment #3: gen_image.sh --]
[-- Type: application/x-shellscript, Size: 2154 bytes --]

Re: [dm-crypt] veritysetup forward error correction failure

[Milan Broz]

On 18/07/2019 13:41, Tom Eccles wrote:
> On 17/07/2019 8:49 pm, Milan Broz wrote:
>> If you can still reproduce it, please send version of the utility and
>> kernel (and --debug output as suggested in another mail) and if you have some
>> data/hash/fec images that can be used to reproduce it, let me know where I can find it.
> 
> Attached is a bash script gen_image.sh which should reproduce the issue.
> find_and_corrupt.c should be in the working directory when gen_image.sh is
> run. find_and_corrupt.c corrupts the disk image by simply searching for a
> known string and introducing an error (see the diff outputted by gen_image.sh).
> 
> See in particular the error and dmesg sample when reading the corrupted file
> (gen_image.sh:77) and the success when running veritysetup verify
> (gen_image.sh:86).
> 
> I have created an issue at https://gitlab.com/cryptsetup/cryptsetup/issues/462

Thanks.

I think I see the problem, and it is actually not FEC related.

Could you please try to manually allocate loop devices in RW mode (not read-only
as we have during automatic loop allocation if mapping image is in file)
over the provided images, and then use these loop device as arguments for
the veritysetup?

Like
 # losetup /dev/loop0 data8M.img.corrupt 
 # losetup /dev/loop1 hash-img 
 # losetup /dev/loop2 fec-img 

and

 # veritysetup open --fec-roots 24 --fec-device /dev/loop2 /dev/loop0 test /dev/loop1 15546e6799bb13608c77fa9cb840682a2dec3cfdb948d942ff3487f537966c01 --debug

Does it correct the data now? For me it prints

  kernel: device-mapper: verity: sha256 using implementation "sha256-generic"
  kernel: fec_decode_bufs: 13 callbacks suppressed
  kernel: device-mapper: verity-fec: 7:0: FEC 0: corrected 153 errors

Milan

Re: [dm-crypt] veritysetup forward error correction failure

[Tom Eccles]

[-- Attachment #1: Type: text/plain, Size: 8901 bytes --]

On 18/07/2019 2:09 pm, Milan Broz wrote:
> On 18/07/2019 13:41, Tom Eccles wrote:
>> On 17/07/2019 8:49 pm, Milan Broz wrote:
>>> If you can still reproduce it, please send version of the utility and
>>> kernel (and --debug output as suggested in another mail) and if you have some
>>> data/hash/fec images that can be used to reproduce it, let me know where I can find it.
>>
>> Attached is a bash script gen_image.sh which should reproduce the issue.
>> find_and_corrupt.c should be in the working directory when gen_image.sh is
>> run. find_and_corrupt.c corrupts the disk image by simply searching for a
>> known string and introducing an error (see the diff outputted by gen_image.sh).
>>
>> See in particular the error and dmesg sample when reading the corrupted file
>> (gen_image.sh:77) and the success when running veritysetup verify
>> (gen_image.sh:86).
>>
>> I have created an issue at https://gitlab.com/cryptsetup/cryptsetup/issues/462
> 
> Thanks.
> 
> I think I see the problem, and it is actually not FEC related.
> 
> Could you please try to manually allocate loop devices in RW mode (not read-only
> as we have during automatic loop allocation if mapping image is in file)
> over the provided images, and then use these loop device as arguments for
> the veritysetup?
> 
> Like
>  # losetup /dev/loop0 data8M.img.corrupt 
>  # losetup /dev/loop1 hash-img 
>  # losetup /dev/loop2 fec-img 
> 
> and
> 
>  # veritysetup open --fec-roots 24 --fec-device /dev/loop2 /dev/loop0 test /dev/loop1 15546e6799bb13608c77fa9cb840682a2dec3cfdb948d942ff3487f537966c01 --debug
> 
> Does it correct the data now? For me it prints
> 
>   kernel: device-mapper: verity: sha256 using implementation "sha256-generic"
>   kernel: fec_decode_bufs: 13 callbacks suppressed
>   kernel: device-mapper: verity-fec: 7:0: FEC 0: corrected 153 errors
> 
> Milan
> 

This is not working for me.

My modified version of the script is attached.

My output when running the script is

sudo dd if=/dev/zero of=data8M.img bs=8M count=1
[sudo] password for tomeccles:
1+0 records in
1+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0100743 s, 833 MB/s
sudo mkfs.ext4 -b 4096 -L verity-data data8M.img
mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: done
Creating filesystem with 2048 4k blocks and 2048 inodes

Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

sudo mount -t ext4 data8M.img mnt
echo "Can you spot this error?" | sudo tee "mnt/test_file.txt"
Can you spot this error?
sudo umount data8M.img
sudo veritysetup --verbose --debug --fec-roots 24 --fec-device fec-img
format "data8M.img" "hash-img" > "veritysetup_format.out"
cp data8M.img data8M.img.corrupt
./find_and_corrupt data8M.img.corrupt
Searching for "Can you spot this error?"
Found at 4497432!
Overwriting with "Cao you spot this error?"
--- /dev/fd/63	2019-07-18 14:45:50.983857200 +0100
+++ /dev/fd/62	2019-07-18 14:45:50.983857200 +0100
@@ -281086,7 +281086,7 @@
 00449fd0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00449fe0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00449ff0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
-0044a000: 4361 6e20 796f 7520 7370 6f74 2074 6869  Can you spot thi
+0044a000: 4361 6f20 796f 7520 7370 6f74 2074 6869  Cao you spot thi
 0044a010: 7320 6572 726f 723f 0a00 0000 0000 0000  s error?........
 0044a020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 0044a030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
sudo losetup /dev/loop0 data8M.img.corrupt
sudo losetup /dev/loop1 hash-img
sudo losetup /dev/loop2 fec-img
sudo veritysetup --verbose --debug --fec-roots 24 --fec-device /dev/loop2
open /dev/loop0 verity /dev/loop1
a963b029f27238283eae4b1cab58e94eebfaee29c9143e3716f992ea5c69811a
# cryptsetup 2.1.0 processing "veritysetup --verbose --debug --fec-roots 24
--fec-device /dev/loop2 open /dev/loop0 verity /dev/loop1
a963b029f27238283eae4b1cab58e94eebfaee29c9143e3716f992ea5c69811a"
# Running command open.
# Allocating context for crypt device /dev/loop1.
# Trying to open and read device /dev/loop1 with direct-io.
# Initialising device-mapper backend library.
# Setting ciphertext data device to /dev/loop0.
# Trying to open and read device /dev/loop0 with direct-io.
# Trying to load VERITY crypt type from device /dev/loop1.
# Crypto backend (OpenSSL 1.1.1c  28 May 2019) initialized in cryptsetup
library version 2.1.0.
# Detected kernel Linux 4.19.37-verity x86_64.
# Reading VERITY header of size 512 on device /dev/loop1, offset 0.
# Trying to open and read device /dev/loop2 with direct-io.
# Activating volume verity by volume key.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.39.0.
# Detected dm-verity version 1.4.0.
# Device-mapper backend running with UDEV support enabled.
# dm status verity  [ opencount noflush ]   [16384] (*1)
# Trying to activate VERITY device verity using hash sha256.
# Calculated device size is 16384 sectors (RW), offset 0.
# DM-UUID is CRYPT-VERITY-ae0850832ef64e1294ed2fd0a433f5d0-verity
# Udev cookie 0xd4dbd1e (semid 0) created
# Udev cookie 0xd4dbd1e (semid 0) incremented to 1
# Udev cookie 0xd4dbd1e (semid 0) incremented to 2
# Udev cookie 0xd4dbd1e (semid 0) assigned to CREATE task(0) with flags
DISABLE_LIBRARY_FALLBACK         (0x20)
# dm create verity CRYPT-VERITY-ae0850832ef64e1294ed2fd0a433f5d0-verity [
opencount flush ]   [16384] (*1)
# dm reload verity  [ opencount flush readonly securedata ]   [16384] (*1)
# dm resume verity  [ opencount flush readonly securedata ]   [16384] (*1)
# verity: Stacking NODE_ADD (253,0) 0:6 0660 [trust_udev]
# verity: Stacking NODE_READ_AHEAD 256 (flags=1)
# Udev cookie 0xd4dbd1e (semid 0) decremented to 1
# Udev cookie 0xd4dbd1e (semid 0) waiting for zero
# Udev cookie 0xd4dbd1e (semid 0) destroyed
# verity: Skipping NODE_ADD (253,0) 0:6 0660 [trust_udev]
# verity: Processing NODE_READ_AHEAD 256 (flags=1)
# verity (253:0): read ahead is 256
# verity: retaining kernel read ahead of 256 (requested 256)
# dm status verity  [ opencount noflush ]   [16384] (*1)
# Verity volume verity status is V.
# Releasing crypt device /dev/loop1 context.
# Releasing device-mapper backend.
Command successful.
sudo mount -t ext4 -o ro /dev/mapper/verity mnt
cat: mnt/test_file.txt: Input/output error
[    2.495571] snd_hda_codec_generic hdaudioC0D0:    inputs:
[    2.495572] snd_hda_codec_generic hdaudioC0D0:      Line=0x5
[    2.510317] [drm] Initialized qxl 0.1.0 20120117 for 0000:00:02.0 on minor 0
[   13.358563] loop: module loaded
[   13.379473] EXT4-fs (loop0): mounted filesystem with ordered data mode.
Opts: (null)
[   15.773706] EXT4-fs (dm-0): mounted filesystem with ordered data mode.
Opts: (null)
[   15.799744] device-mapper: verity-fec: 7:0: FEC 0: failed to correct: -74
[   15.799772] device-mapper: verity: 7:0: data block 1098 is corrupted
[   15.847392] device-mapper: verity-fec: 7:0: FEC 0: failed to correct: -74
[   15.847418] device-mapper: verity: 7:0: data block 1098 is corrupted
sudo umount mnt
sudo veritysetup close verity
sudo losetup -d  /dev/loop1 /dev/loop2
sudo veritysetup --verbose --debug --fec-roots 24 --fec-device fec-img
verify data8M.img.corrupt hash-img
a963b029f27238283eae4b1cab58e94eebfaee29c9143e3716f992ea5c69811a
# cryptsetup 2.1.0 processing "veritysetup --verbose --debug --fec-roots 24
--fec-device fec-img verify data8M.img.corrupt hash-img
a963b029f27238283eae4b1cab58e94eebfaee29c9143e3716f992ea5c69811a"
# Running command verify.
# Allocating context for crypt device hash-img.
# Trying to open and read device hash-img with direct-io.
# Initialising device-mapper backend library.
# Setting ciphertext data device to data8M.img.corrupt.
# Trying to open and read device data8M.img.corrupt with direct-io.
# Trying to load VERITY crypt type from device hash-img.
# Crypto backend (OpenSSL 1.1.1c  28 May 2019) initialized in cryptsetup
library version 2.1.0.
# Detected kernel Linux 4.19.37-verity x86_64.
# Reading VERITY header of size 512 on device hash-img, offset 0.
# Trying to open and read device fec-img with direct-io.
# Checking volume  by volume key.
# Trying to activate VERITY device [none] using hash sha256.
# Verification of data in userspace required.
# Hash verification sha256, data device data8M.img.corrupt, data blocks
2048, hash_device hash-img, offset 1.
# Using 2 hash levels.
# Data device size required: 8388608 bytes.
# Hash device size required: 73728 bytes.
Verification failed at position 4497408.
Verification of data area failed.
# Verification failed, trying to repair with FEC device.
Found 1 repairable errors with FEC device.
# Releasing crypt device hash-img context.
# Releasing device-mapper backend.
Command successful.
Success


[-- Attachment #2: gen_image.sh --]
[-- Type: application/x-shellscript, Size: 2598 bytes --]

Re: [dm-crypt] veritysetup forward error correction failure

[Tom Eccles]

On 18/07/2019 2:09 pm, Milan Broz wrote:
> On 18/07/2019 13:41, Tom Eccles wrote:
>> On 17/07/2019 8:49 pm, Milan Broz wrote:
>>> If you can still reproduce it, please send version of the utility and
>>> kernel (and --debug output as suggested in another mail) and if you have some
>>> data/hash/fec images that can be used to reproduce it, let me know where I can find it.
>>
>> Attached is a bash script gen_image.sh which should reproduce the issue.
>> find_and_corrupt.c should be in the working directory when gen_image.sh is
>> run. find_and_corrupt.c corrupts the disk image by simply searching for a
>> known string and introducing an error (see the diff outputted by gen_image.sh).
>>
>> See in particular the error and dmesg sample when reading the corrupted file
>> (gen_image.sh:77) and the success when running veritysetup verify
>> (gen_image.sh:86).
>>
>> I have created an issue at https://gitlab.com/cryptsetup/cryptsetup/issues/462
> 
> Thanks.
> 
> I think I see the problem, and it is actually not FEC related.
> 
> Could you please try to manually allocate loop devices in RW mode (not read-only
> as we have during automatic loop allocation if mapping image is in file)
> over the provided images, and then use these loop device as arguments for
> the veritysetup?
> 
> Like
>  # losetup /dev/loop0 data8M.img.corrupt 
>  # losetup /dev/loop1 hash-img 
>  # losetup /dev/loop2 fec-img 
> 
> and
> 
>  # veritysetup open --fec-roots 24 --fec-device /dev/loop2 /dev/loop0 test /dev/loop1 15546e6799bb13608c77fa9cb840682a2dec3cfdb948d942ff3487f537966c01 --debug
> 
> Does it correct the data now? For me it prints
> 
>   kernel: device-mapper: verity: sha256 using implementation "sha256-generic"
>   kernel: fec_decode_bufs: 13 callbacks suppressed
>   kernel: device-mapper: verity-fec: 7:0: FEC 0: corrected 153 errors

Should write access be required to the device be required? The kernel
doesn't allow writable verity devices:

drivers/md/dm-verity-target.c:954:

	if ((dm_table_get_mode(ti->table) & ~FMODE_READ)) {
		ti->error = "Device must be readonly";
		r = -EINVAL;
		goto bad;
	}

Thanks,
Tom

Re: [dm-crypt] veritysetup forward error correction failure

[Milan Broz]

On 18/07/2019 15:49, Tom Eccles wrote:
> On 18/07/2019 2:09 pm, Milan Broz wrote:
>> On 18/07/2019 13:41, Tom Eccles wrote:
>>> On 17/07/2019 8:49 pm, Milan Broz wrote:
>>>> If you can still reproduce it, please send version of the utility and
>>>> kernel (and --debug output as suggested in another mail) and if you have some
>>>> data/hash/fec images that can be used to reproduce it, let me know where I can find it.
>>>
>>> Attached is a bash script gen_image.sh which should reproduce the issue.
>>> find_and_corrupt.c should be in the working directory when gen_image.sh is
>>> run. find_and_corrupt.c corrupts the disk image by simply searching for a
>>> known string and introducing an error (see the diff outputted by gen_image.sh).
>>>
>>> See in particular the error and dmesg sample when reading the corrupted file
>>> (gen_image.sh:77) and the success when running veritysetup verify
>>> (gen_image.sh:86).
>>>
>>> I have created an issue at https://gitlab.com/cryptsetup/cryptsetup/issues/462
>>
>> Thanks.
>>
>> I think I see the problem, and it is actually not FEC related.
>>
>> Could you please try to manually allocate loop devices in RW mode (not read-only
>> as we have during automatic loop allocation if mapping image is in file)
>> over the provided images, and then use these loop device as arguments for
>> the veritysetup?
>>
>> Like
>>  # losetup /dev/loop0 data8M.img.corrupt 
>>  # losetup /dev/loop1 hash-img 
>>  # losetup /dev/loop2 fec-img 
>>
>> and
>>
>>  # veritysetup open --fec-roots 24 --fec-device /dev/loop2 /dev/loop0 test /dev/loop1 15546e6799bb13608c77fa9cb840682a2dec3cfdb948d942ff3487f537966c01 --debug
>>
>> Does it correct the data now? For me it prints
>>
>>   kernel: device-mapper: verity: sha256 using implementation "sha256-generic"
>>   kernel: fec_decode_bufs: 13 callbacks suppressed
>>   kernel: device-mapper: verity-fec: 7:0: FEC 0: corrected 153 errors
>>
>> Milan
>>
> 
> This is not working for me.

ok, I actually just tried dd the whole device, not through the mounted fs.
will try it later...

But anyway, it seems like a problem with some infrastructure inside dm/block layer
and not the FEC algorithm itself (errno -74 is EBADMSG, something that should not be here).

m.

Re: [dm-crypt] veritysetup forward error correction failure

[Tom Eccles]

On 18/07/2019 2:59 pm, Milan Broz wrote:
> On 18/07/2019 15:49, Tom Eccles wrote:
>> On 18/07/2019 2:09 pm, Milan Broz wrote:
>>> On 18/07/2019 13:41, Tom Eccles wrote:
>>>> On 17/07/2019 8:49 pm, Milan Broz wrote:
>>>>> If you can still reproduce it, please send version of the utility and
>>>>> kernel (and --debug output as suggested in another mail) and if you have some
>>>>> data/hash/fec images that can be used to reproduce it, let me know where I can find it.
>>>>
>>>> Attached is a bash script gen_image.sh which should reproduce the issue.
>>>> find_and_corrupt.c should be in the working directory when gen_image.sh is
>>>> run. find_and_corrupt.c corrupts the disk image by simply searching for a
>>>> known string and introducing an error (see the diff outputted by gen_image.sh).
>>>>
>>>> See in particular the error and dmesg sample when reading the corrupted file
>>>> (gen_image.sh:77) and the success when running veritysetup verify
>>>> (gen_image.sh:86).
>>>>
>>>> I have created an issue at https://gitlab.com/cryptsetup/cryptsetup/issues/462
>>>
>>> Thanks.
>>>
>>> I think I see the problem, and it is actually not FEC related.
>>>
>>> Could you please try to manually allocate loop devices in RW mode (not read-only
>>> as we have during automatic loop allocation if mapping image is in file)
>>> over the provided images, and then use these loop device as arguments for
>>> the veritysetup?
>>>
>>> Like
>>>  # losetup /dev/loop0 data8M.img.corrupt 
>>>  # losetup /dev/loop1 hash-img 
>>>  # losetup /dev/loop2 fec-img 
>>>
>>> and
>>>
>>>  # veritysetup open --fec-roots 24 --fec-device /dev/loop2 /dev/loop0 test /dev/loop1 15546e6799bb13608c77fa9cb840682a2dec3cfdb948d942ff3487f537966c01 --debug
>>>
>>> Does it correct the data now? For me it prints
>>>
>>>   kernel: device-mapper: verity: sha256 using implementation "sha256-generic"
>>>   kernel: fec_decode_bufs: 13 callbacks suppressed
>>>   kernel: device-mapper: verity-fec: 7:0: FEC 0: corrected 153 errors
>>>
>>> Milan
>>>
>>
>> This is not working for me.
> 
> ok, I actually just tried dd the whole device, not through the mounted fs.
> will try it later...
> 
> But anyway, it seems like a problem with some infrastructure inside dm/block layer
> and not the FEC algorithm itself (errno -74 is EBADMSG, something that should not be here).
> 
> m.
> 

Thanks. I will have a look around there.

Re: [dm-crypt] veritysetup forward error correction failure

[Milan Broz]

On 18/07/2019 15:59, Tom Eccles wrote:
> 
> Should write access be required to the device be required? The kernel
> doesn't allow writable verity devices:

This is correct, verity devices must be read-only.

Anyway, I tried to run it again and this time I sometimes get error
and sometimes it recovers the device properly.

So there is something wrong inside kernel verity target, it could
be even two years old problem with switch to asynchronous crypto calls...

My losetup "workaround" probably just changed some timining, so
on your system it does not help. I'll investigate it :)

Anyway, thanks for reporting this!

Milan