[Qemu-devel] [Bug 1815993] [NEW] drive-backup with iscsi will cause vm disk no response

[Qemu-devel] [Bug 1815993] [NEW] drive-backup with iscsi will cause vm disk no response

[Cheng Chen]

Public bug reported:

virsh qemu-monitor-command ${DOMAIN} '{ "execute" : "drive-backup" ,
"arguments" : { "device" : "drive-virtio-disk0" , "sync" : "top" ,
"target" : "iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0" } }'

When the drive-backup is running, I manually crash the iscsi server（or
interrupt network, eg. iptables -j DROP）.

Then after less than 1 minute：
virsh qemu-monitor-command ${DOMAIN} --pretty '{ "execute": "query-block" }' will block and no any response, until timeout. This is still excusable.
But, the disk（drive-virtio-disk0）will occur the same situation：in vm os, the disk will block and no any response.

In other words, when qemu and iscsi-server lose contact, It will cause
the vm unable.

---
Host: centos 7.5
qemu version: ovirt-4.2（qemu-2.12.0）
qemu command line: qemu-system-x86_64 -name guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-190-test./master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -m 1024 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu/190-test -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1c8611c2-a18a-4b1c-b40b-9d82040eafa4 -smbios type=1,manufacturer=IaaS -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/opt/vol/sas/fb0c7c37-13e7-41fe-b3f8-f0fbaaeec7ce,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -drive file=/opt/vol/sas/bde66671-536d-49cd-8b46-a4f1ea7be513,format=qcow2,if=none,id=drive-virtio-disk1,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1,write-cache=on -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:85:45:3e:d4:3a,bus=pci.0,addr=0x6 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on

iscsi：
yum -y install targetcli python-rtslib
systemctl start target
systemctl enable target

targetcli /iscsi create iqn.2019-01.com.iaas

targetcli /iscsi/iqn.2019-01.com.iaas/tpg1 set attribute
authentication=0 demo_mode_write_protect=0 generate_node_acls=1

targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/portals create 192.168.1.100 3260
targetcli /backstores/fileio create testfile1 /backup/file1 2G
targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/luns create /backstores/fileio/testfile1

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1815993

Title:
  drive-backup with iscsi will cause vm disk no response

Status in QEMU:
  New

Bug description:
  virsh qemu-monitor-command ${DOMAIN} '{ "execute" : "drive-backup" ,
  "arguments" : { "device" : "drive-virtio-disk0" , "sync" : "top" ,
  "target" : "iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0" } }'

  When the drive-backup is running, I manually crash the iscsi server（or
  interrupt network, eg. iptables -j DROP）.

  Then after less than 1 minute：
  virsh qemu-monitor-command ${DOMAIN} --pretty '{ "execute": "query-block" }' will block and no any response, until timeout. This is still excusable.
  But, the disk（drive-virtio-disk0）will occur the same situation：in vm os, the disk will block and no any response.

  In other words, when qemu and iscsi-server lose contact, It will cause
  the vm unable.

  ---
  Host: centos 7.5
  qemu version: ovirt-4.2（qemu-2.12.0）
  qemu command line: qemu-system-x86_64 -name guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-190-test./master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -m 1024 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu/190-test -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1c8611c2-a18a-4b1c-b40b-9d82040eafa4 -smbios type=1,manufacturer=IaaS -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/opt/vol/sas/fb0c7c37-13e7-41fe-b3f8-f0fbaaeec7ce,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -drive file=/opt/vol/sas/bde66671-536d-49cd-8b46-a4f1ea7be513,format=qcow2,if=none,id=drive-virtio-disk1,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1,write-cache=on -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:85:45:3e:d4:3a,bus=pci.0,addr=0x6 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on

  iscsi：
  yum -y install targetcli python-rtslib
  systemctl start target
  systemctl enable target

  targetcli /iscsi create iqn.2019-01.com.iaas

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1 set attribute
  authentication=0 demo_mode_write_protect=0 generate_node_acls=1

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/portals create 192.168.1.100 3260
  targetcli /backstores/fileio create testfile1 /backup/file1 2G
  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/luns create /backstores/fileio/testfile1

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1815993/+subscriptions

Re: [Qemu-devel] [Bug 1815993] [NEW] drive-backup with iscsi will cause vm disk no response

[Stefan Hajnoczi]

On Fri, Feb 15, 2019 at 03:03:34AM -0000, Cheng Chen wrote:
> When the drive-backup is running, I manually crash the iscsi server（or
> interrupt network, eg. iptables -j DROP）.
> 
> Then after less than 1 minute：
> virsh qemu-monitor-command ${DOMAIN} --pretty '{ "execute": "query-block" }' will block and no any response, until timeout. This is still excusable.
> But, the disk（drive-virtio-disk0）will occur the same situation：in vm os, the disk will block and no any response.
> 
> In other words, when qemu and iscsi-server lose contact, It will cause
> the vm unable.

I haven't tried to reproduce this but I guess QEMU reaches a
synchronization point where it waits for all outstanding requests to
complete.  Since the iSCSI target is unresponsive QEMU gets stuck.

These issues can sometimes be fixed by avoiding the synchronization
point (a backtrace should reveal where the main loop thread is stuck)
but other times it really is necessary to wait for all requests and the
solution isn't as obvious.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1815993

Title:
  drive-backup with iscsi will cause vm disk no response

Status in QEMU:
  New

Bug description:
  virsh qemu-monitor-command ${DOMAIN} '{ "execute" : "drive-backup" ,
  "arguments" : { "device" : "drive-virtio-disk0" , "sync" : "top" ,
  "target" : "iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0" } }'

  When the drive-backup is running, I manually crash the iscsi server（or
  interrupt network, eg. iptables -j DROP）.

  Then after less than 1 minute：
  virsh qemu-monitor-command ${DOMAIN} --pretty '{ "execute": "query-block" }' will block and no any response, until timeout. This is still excusable.
  But, the disk（drive-virtio-disk0）will occur the same situation：in vm os, the disk will block and no any response.

  In other words, when qemu and iscsi-server lose contact, It will cause
  the vm unable.

  ---
  Host: centos 7.5
  qemu version: ovirt-4.2（qemu-2.12.0）
  qemu command line: qemu-system-x86_64 -name guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-190-test./master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -m 1024 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu/190-test -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1c8611c2-a18a-4b1c-b40b-9d82040eafa4 -smbios type=1,manufacturer=IaaS -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/opt/vol/sas/fb0c7c37-13e7-41fe-b3f8-f0fbaaeec7ce,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -drive file=/opt/vol/sas/bde66671-536d-49cd-8b46-a4f1ea7be513,format=qcow2,if=none,id=drive-virtio-disk1,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1,write-cache=on -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:85:45:3e:d4:3a,bus=pci.0,addr=0x6 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on

  iscsi：
  yum -y install targetcli python-rtslib
  systemctl start target
  systemctl enable target

  targetcli /iscsi create iqn.2019-01.com.iaas

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1 set attribute
  authentication=0 demo_mode_write_protect=0 generate_node_acls=1

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/portals create 192.168.1.100 3260
  targetcli /backstores/fileio create testfile1 /backup/file1 2G
  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/luns create /backstores/fileio/testfile1

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1815993/+subscriptions

[Bug 1815993] Re: drive-backup with iscsi will cause vm disk no response

[Thomas Huth]

The QEMU project is currently considering to move its bug tracking to another system. For this we need to know which bugs are still valid and which could be closed already. Thus we are setting older bugs to "Incomplete" now.
If you still think this bug report here is valid, then please switch the state back to "New" within the next 60 days, otherwise this report will be marked as "Expired". Or mark it as "Fix Released" if the problem has been solved with a newer version of QEMU already. Thank you and sorry for the inconvenience.

** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1815993

Title:
  drive-backup with iscsi will cause vm disk no response

Status in QEMU:
  Incomplete

Bug description:
  virsh qemu-monitor-command ${DOMAIN} '{ "execute" : "drive-backup" ,
  "arguments" : { "device" : "drive-virtio-disk0" , "sync" : "top" ,
  "target" : "iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0" } }'

  When the drive-backup is running, I manually crash the iscsi server（or
  interrupt network, eg. iptables -j DROP）.

  Then after less than 1 minute：
  virsh qemu-monitor-command ${DOMAIN} --pretty '{ "execute": "query-block" }' will block and no any response, until timeout. This is still excusable.
  But, the disk（drive-virtio-disk0）will occur the same situation：in vm os, the disk will block and no any response.

  In other words, when qemu and iscsi-server lose contact, It will cause
  the vm unable.

  ---
  Host: centos 7.5
  qemu version: ovirt-4.2（qemu-2.12.0）
  qemu command line: qemu-system-x86_64 -name guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-190-test./master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -m 1024 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu/190-test -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1c8611c2-a18a-4b1c-b40b-9d82040eafa4 -smbios type=1,manufacturer=IaaS -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/opt/vol/sas/fb0c7c37-13e7-41fe-b3f8-f0fbaaeec7ce,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -drive file=/opt/vol/sas/bde66671-536d-49cd-8b46-a4f1ea7be513,format=qcow2,if=none,id=drive-virtio-disk1,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1,write-cache=on -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:85:45:3e:d4:3a,bus=pci.0,addr=0x6 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on

  iscsi：
  yum -y install targetcli python-rtslib
  systemctl start target
  systemctl enable target

  targetcli /iscsi create iqn.2019-01.com.iaas

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1 set attribute
  authentication=0 demo_mode_write_protect=0 generate_node_acls=1

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/portals create 192.168.1.100 3260
  targetcli /backstores/fileio create testfile1 /backup/file1 2G
  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/luns create /backstores/fileio/testfile1

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1815993/+subscriptions

[Bug 1815993] Re: drive-backup with iscsi will cause vm disk no response

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1815993

Title:
  drive-backup with iscsi will cause vm disk no response

Status in QEMU:
  Expired

Bug description:
  virsh qemu-monitor-command ${DOMAIN} '{ "execute" : "drive-backup" ,
  "arguments" : { "device" : "drive-virtio-disk0" , "sync" : "top" ,
  "target" : "iscsi://192.168.1.100:3260/iqn.2019-01.com.iaas/0" } }'

  When the drive-backup is running, I manually crash the iscsi server（or
  interrupt network, eg. iptables -j DROP）.

  Then after less than 1 minute：
  virsh qemu-monitor-command ${DOMAIN} --pretty '{ "execute": "query-block" }' will block and no any response, until timeout. This is still excusable.
  But, the disk（drive-virtio-disk0）will occur the same situation：in vm os, the disk will block and no any response.

  In other words, when qemu and iscsi-server lose contact, It will cause
  the vm unable.

  ---
  Host: centos 7.5
  qemu version: ovirt-4.2（qemu-2.12.0）
  qemu command line: qemu-system-x86_64 -name guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-190-test./master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -m 1024 -mem-prealloc -mem-path /dev/hugepages1G/libvirt/qemu/190-test -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1c8611c2-a18a-4b1c-b40b-9d82040eafa4 -smbios type=1,manufacturer=IaaS -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/opt/vol/sas/fb0c7c37-13e7-41fe-b3f8-f0fbaaeec7ce,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -drive file=/opt/vol/sas/bde66671-536d-49cd-8b46-a4f1ea7be513,format=qcow2,if=none,id=drive-virtio-disk1,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1,write-cache=on -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:85:45:3e:d4:3a,bus=pci.0,addr=0x6 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on

  iscsi：
  yum -y install targetcli python-rtslib
  systemctl start target
  systemctl enable target

  targetcli /iscsi create iqn.2019-01.com.iaas

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1 set attribute
  authentication=0 demo_mode_write_protect=0 generate_node_acls=1

  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/portals create 192.168.1.100 3260
  targetcli /backstores/fileio create testfile1 /backup/file1 2G
  targetcli /iscsi/iqn.2019-01.com.iaas/tpg1/luns create /backstores/fileio/testfile1

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1815993/+subscriptions