[Bug 1912224] [NEW] qemu may freeze during drive-mirroring on fragmented FS

* [Bug 1912224] [NEW] qemu may freeze during drive-mirroring on fragmented FS
@ 2021-01-18 14:33 Alexandre arents
  2021-01-22 13:19 ` [Bug 1912224] " Alexandre arents
                   ` (10 more replies)
  0 siblings, 11 replies; 12+ messages in thread
From: Alexandre arents @ 2021-01-18 14:33 UTC (permalink / raw)
  To: qemu-devel

Public bug reported:

We have odd behavior in operation where qemu freeze during long
seconds, We started an thread about that issue here:
https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05623.html

It happens at least during openstack nova snapshot (qemu blockdev-mirror)
or live block migration(which include network copy of disk).

After further troubleshoots, it seems related to FS fragmentation on
host.

reproducible at least on:
Ubuntu 18.04.3/4.18.0-25-generic/qemu-4.0
Ubuntu 16.04.6/5.10.6/qemu-5.2.0-rc2

# Lets create a dedicated file system on a SSD/Nvme 60GB disk in my case:
$sudo mkfs.ext4 /dev/sda3
$sudo mount /dev/sda3 /mnt
$df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3         59G   53M   56G   1% /mnt

#Create a fragmented disk on it using 2MB Chunks (about 30min):
$sudo python3 create_fragged_disk.py /mnt 2
Filling up FS by Creating chunks files in:  /mnt/chunks
We are probably full as expected!!:  [Errno 28] No space left on device
Creating fragged disk file:  /mnt/disk

$ls -lhs 
59G -rw-r--r-- 1 root root 59G Jan 15 14:08 /mnt/disk

$ sudo e4defrag -c /mnt/disk
 Total/best extents                             41971/30
 Average size per extent                        1466 KB
 Fragmentation score                            2
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
 This file (/mnt/disk) does not need defragmentation.
 Done.

# the tool^^^ says it is not enough fragmented to be able to defrag.

#Inject an image on fragmented disk
sudo chown ubuntu /mnt/disk
wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
qemu-img convert -O raw  bionic-server-cloudimg-amd64.img \
                         bionic-server-cloudimg-amd64.img.raw
dd conv=notrunc iflag=fullblock if=bionic-server-cloudimg-amd64.img.raw \
                of=/mnt/disk bs=1M
virt-customize -a /mnt/disk --root-password password:xxxx

# logon run console activity ex: ping -i 0.3 127.0.0.1
$qemu-system-x86_64 -m 2G -enable-kvm  -nographic \
    -chardev socket,id=test,path=/tmp/qmp-monitor,server,nowait \
    -mon chardev=test,mode=control \
    -drive file=/mnt/disk,format=raw,if=none,id=drive-virtio-disk0,cache=none,discard\
    -device virtio-blk-pci,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on

$sync
$echo 3 | sudo tee -a /proc/sys/vm/drop_caches

#start drive-mirror via qmp on another SSD/nvme partition
nc -U /tmp/qmp-monitor
{"execute":"qmp_capabilities"}
{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","target":"/home/ubuntu/mirror","sync":"full","format":"qcow2"}}
^^^ qemu console may start to freeze at this step.

NOTE:
 - smaller chunk sz and bigger disk size the worst it is.
   In operation we also have issue on 400GB disk size with average 13MB/extent
 - Reproducible also on xfs

Expected behavior:
-------------------
QEMU should remain steady, eventually only have decrease storage Performance
or mirroring, because of fragmented fs.

Observed behavior:
-------------------
Perf of mirroring is still quite good even on fragmented FS,
but it breaks qemu.

######################  create_fragged_disk.py ############
import sys
import os
import tempfile
import glob
import errno

MNT_DIR = sys.argv[1]
CHUNK_SZ_MB = int(sys.argv[2])
CHUNKS_DIR = MNT_DIR + '/chunks'
DISK_FILE = MNT_DIR + '/disk'

if not os.path.exists(CHUNKS_DIR):
    os.makedirs(CHUNKS_DIR)

with open("/dev/urandom", "rb") as f_rand:
     mb_rand=f_rand.read(1024 * 1024)

print("Filling up FS by Creating chunks files in: ",CHUNKS_DIR)
try:
    while True:
        tp = tempfile.NamedTemporaryFile(dir=CHUNKS_DIR,delete=False)
        for x in range(CHUNK_SZ_MB):
            tp.write(mb_rand)
        os.fsync(tp)
        tp.close()
except Exception as ex:
    print("We are probably full as expected!!: ",ex)

chunks = glob.glob(CHUNKS_DIR + '/*')

print("Creating fragged disk file: ",DISK_FILE)
with open(DISK_FILE, "w+b") as f_disk:
    for chunk in chunks:
        try:
            os.unlink(chunk)
            for x in range(CHUNK_SZ_MB):
                f_disk.write(mb_rand)
            os.fsync(f_disk)
        except IOError as ex:
            if ex.errno != errno.ENOSPC:
                raise
###########################################################3

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1912224

Title:
  qemu may freeze during drive-mirroring on fragmented FS

Status in QEMU:
  New

Bug description:

  We have odd behavior in operation where qemu freeze during long
  seconds, We started an thread about that issue here:
  https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05623.html

  It happens at least during openstack nova snapshot (qemu blockdev-mirror)
  or live block migration(which include network copy of disk).

  After further troubleshoots, it seems related to FS fragmentation on
  host.

  reproducible at least on:
  Ubuntu 18.04.3/4.18.0-25-generic/qemu-4.0
  Ubuntu 16.04.6/5.10.6/qemu-5.2.0-rc2

  # Lets create a dedicated file system on a SSD/Nvme 60GB disk in my case:
  $sudo mkfs.ext4 /dev/sda3
  $sudo mount /dev/sda3 /mnt
  $df -h /mnt
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda3         59G   53M   56G   1% /mnt

  #Create a fragmented disk on it using 2MB Chunks (about 30min):
  $sudo python3 create_fragged_disk.py /mnt 2
  Filling up FS by Creating chunks files in:  /mnt/chunks
  We are probably full as expected!!:  [Errno 28] No space left on device
  Creating fragged disk file:  /mnt/disk

  $ls -lhs 
  59G -rw-r--r-- 1 root root 59G Jan 15 14:08 /mnt/disk

  $ sudo e4defrag -c /mnt/disk
   Total/best extents                             41971/30
   Average size per extent                        1466 KB
   Fragmentation score                            2
   [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
   This file (/mnt/disk) does not need defragmentation.
   Done.

  # the tool^^^ says it is not enough fragmented to be able to defrag.

  #Inject an image on fragmented disk
  sudo chown ubuntu /mnt/disk
  wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
  qemu-img convert -O raw  bionic-server-cloudimg-amd64.img \
                           bionic-server-cloudimg-amd64.img.raw
  dd conv=notrunc iflag=fullblock if=bionic-server-cloudimg-amd64.img.raw \
                  of=/mnt/disk bs=1M
  virt-customize -a /mnt/disk --root-password password:xxxx

  # logon run console activity ex: ping -i 0.3 127.0.0.1
  $qemu-system-x86_64 -m 2G -enable-kvm  -nographic \
      -chardev socket,id=test,path=/tmp/qmp-monitor,server,nowait \
      -mon chardev=test,mode=control \
      -drive file=/mnt/disk,format=raw,if=none,id=drive-virtio-disk0,cache=none,discard\
      -device virtio-blk-pci,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on

  $sync
  $echo 3 | sudo tee -a /proc/sys/vm/drop_caches

  #start drive-mirror via qmp on another SSD/nvme partition
  nc -U /tmp/qmp-monitor
  {"execute":"qmp_capabilities"}
  {"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","target":"/home/ubuntu/mirror","sync":"full","format":"qcow2"}}
  ^^^ qemu console may start to freeze at this step.

  NOTE:
   - smaller chunk sz and bigger disk size the worst it is.
     In operation we also have issue on 400GB disk size with average 13MB/extent
   - Reproducible also on xfs

  Expected behavior:
  -------------------
  QEMU should remain steady, eventually only have decrease storage Performance
  or mirroring, because of fragmented fs.

  Observed behavior:
  -------------------
  Perf of mirroring is still quite good even on fragmented FS,
  but it breaks qemu.

  ######################  create_fragged_disk.py ############
  import sys
  import os
  import tempfile
  import glob
  import errno

  MNT_DIR = sys.argv[1]
  CHUNK_SZ_MB = int(sys.argv[2])
  CHUNKS_DIR = MNT_DIR + '/chunks'
  DISK_FILE = MNT_DIR + '/disk'

  if not os.path.exists(CHUNKS_DIR):
      os.makedirs(CHUNKS_DIR)

  with open("/dev/urandom", "rb") as f_rand:
       mb_rand=f_rand.read(1024 * 1024)

  print("Filling up FS by Creating chunks files in: ",CHUNKS_DIR)
  try:
      while True:
          tp = tempfile.NamedTemporaryFile(dir=CHUNKS_DIR,delete=False)
          for x in range(CHUNK_SZ_MB):
              tp.write(mb_rand)
          os.fsync(tp)
          tp.close()
  except Exception as ex:
      print("We are probably full as expected!!: ",ex)

  chunks = glob.glob(CHUNKS_DIR + '/*')

  print("Creating fragged disk file: ",DISK_FILE)
  with open(DISK_FILE, "w+b") as f_disk:
      for chunk in chunks:
          try:
              os.unlink(chunk)
              for x in range(CHUNK_SZ_MB):
                  f_disk.write(mb_rand)
              os.fsync(f_disk)
          except IOError as ex:
              if ex.errno != errno.ENOSPC:
                  raise
  ###########################################################3

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1912224/+subscriptions

^ permalink raw reply	[flat|nested] 12+ messages in thread