strange dm-crypt problem with qemu / LIO / vhost

strange dm-crypt problem with qemu / LIO / vhost

[Andreas Kinzler]

I am trying to use a /dev/mapper/luks (via cryptsetup/LUKS) on a 
backstorage/iblock from the target/SCSI framework and pass that through 
vhost (tcm_vhost) to qemu guests. It works with Linux guests but with 
Windows guests (using the virtio-scsi pv-driver) I get tons of errors in 
dmesg:

[ 3683.403467] bio error: 00000000f33a8dcd,  err: 10
[ 3683.403612] bio error: 00000000a508ea6b,  err: 10
[ 3683.414686] bio error: 00000000f06baf1d,  err: 10
[ 3683.414767] bio error: 0000000035714712,  err: 10
[ 3683.414829] bio error: 0000000062dd8892,  err: 10
[ 3683.414883] bio error: 000000005c303a2d,  err: 10
[ 3683.414936] bio error: 0000000024d02dea,  err: 10

This is from target_core_iblock.c function "iblock_bio_done".

Any ideas? Hints?

Regards Andreas

Re: strange dm-crypt problem with qemu / LIO / vhost

[David Disseldorp]

Hi Andreas,

It's normally a good idea to provide your kernel version with any
bug reports...

On Mon, 6 Apr 2020 20:21:55 +0200, Andreas Kinzler wrote:

> [ 3683.414936] bio error: 0000000024d02dea,  err: 10
> 
> This is from target_core_iblock.c function "iblock_bio_done".
> 
> Any ideas? Hints?

This looks like an I/O error from the block layer / underlying dm-crypt
block device. It's not much to go on, but I'd suggest tracing the I/O
further down the stack.

Cheers, David

Re: strange dm-crypt problem with qemu / LIO / vhost

[Andreas Kinzler]

Hello David,

> It's normally a good idea to provide your kernel version with any
> bug reports...

Sorry. Kernel is 5.3.18 vanilla from kernel.org

>> [ 3683.414936] bio error: 0000000024d02dea,  err: 10
>> This is from target_core_iblock.c function "iblock_bio_done".
>> Any ideas? Hints?
> This looks like an I/O error from the block layer / underlying dm-crypt
> block device. It's not much to go on, but I'd suggest tracing the I/O
> further down the stack.

In the meantime, I was able to debug further. Actually in Windows only 
the "format disk" operation fails. If you do that otherwise and later 
use the formatted disk, it seems to work. So I assume it is a special 
SCSI opcode that is only used during "format disk" that fails.

What would be a simple debug statement in iblock_bio_done to print the 
SCSI opcode to find out which operation failed?

> Cheers, David

Regards Andreas

Re: strange dm-crypt problem with qemu / LIO / vhost

[David Disseldorp]

On Tue, 7 Apr 2020 00:01:49 +0200, Andreas Kinzler wrote:

> > It's normally a good idea to provide your kernel version with any
> > bug reports...  
> 
> Sorry. Kernel is 5.3.18 vanilla from kernel.org

Thanks.

> >> [ 3683.414936] bio error: 0000000024d02dea,  err: 10
> >> This is from target_core_iblock.c function "iblock_bio_done".
> >> Any ideas? Hints?  
> > This looks like an I/O error from the block layer / underlying dm-crypt
> > block device. It's not much to go on, but I'd suggest tracing the I/O
> > further down the stack.  
> 
> In the meantime, I was able to debug further. Actually in Windows only 
> the "format disk" operation fails. If you do that otherwise and later 
> use the formatted disk, it seems to work. So I assume it is a special 
> SCSI opcode that is only used during "format disk" that fails.
> 
> What would be a simple debug statement in iblock_bio_done to print the 
> SCSI opcode to find out which operation failed?

You could print the SCSI opcode value with something like:

--- a/drivers/target/target_core_iblock.c
+++ b/drivers/target/target_core_iblock.c
@@ -290,7 +290,8 @@ static void iblock_bio_done(struct bio *bio)
        struct iblock_req *ibr = cmd->priv;
 
        if (bio->bi_status) {
-               pr_err("bio error: %p,  err: %d\n", bio, bio->bi_status);
+               pr_err("op: %#x, bio error: %p, err: %d\n", cmd->t_task_cdb[0],
+                      bio, bio->bi_status);
                /*
                 * Bump the ib_bio_err_cnt and release bio.
                 */


include/scsi/scsi_proto.h should allow you to lookup the name.

Cheers, David

Re: strange dm-crypt problem with qemu / LIO / vhost

[Andreas Kinzler]

Hello David,

>> In the meantime, I was able to debug further. Actually in Windows only
>> the "format disk" operation fails. If you do that otherwise and later
>> use the formatted disk, it seems to work. So I assume it is a special
>> SCSI opcode that is only used during "format disk" that fails.
> You could print the SCSI opcode value with something like:

Thanks for the help. I was able to find the root cause. To avoid 
duplication with the issue tracker of the kvm-guest drivers, here is the 
link to my findings:

https://github.com/virtio-win/kvm-guest-drivers-windows/issues/446

Please feel free to add more comments to remaining open questions there.
Especially about the question of alignment in BIOs.

Regards Andreas