From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,LOTS_OF_MONEY,MAILING_LIST_MULTI,MONEY_NOHTML, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F38DEC433E0 for ; Fri, 22 Jan 2021 13:28:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 733F823433 for ; Fri, 22 Jan 2021 13:28:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 733F823433 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=bugs.launchpad.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34110 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l2wTw-0001Qs-DY for qemu-devel@archiver.kernel.org; Fri, 22 Jan 2021 08:28:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60748) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l2wRh-00009n-El for qemu-devel@nongnu.org; Fri, 22 Jan 2021 08:25:53 -0500 Received: from indium.canonical.com ([91.189.90.7]:41408) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l2wRX-0002fD-CR for qemu-devel@nongnu.org; Fri, 22 Jan 2021 08:25:49 -0500 Received: from loganberry.canonical.com ([91.189.90.37]) by indium.canonical.com with esmtp (Exim 4.86_2 #2 (Debian)) id 1l2wRT-0004Ks-S9 for ; Fri, 22 Jan 2021 13:25:35 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id CF10E2E8137 for ; Fri, 22 Jan 2021 13:25:35 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Fri, 22 Jan 2021 13:19:22 -0000 From: Alexandre arents <1912224@bugs.launchpad.net> To: qemu-devel@nongnu.org X-Launchpad-Notification-Type: bug X-Launchpad-Bug: product=qemu; status=New; importance=Undecided; assignee=None; X-Launchpad-Bug-Information-Type: Public X-Launchpad-Bug-Private: no X-Launchpad-Bug-Security-Vulnerability: no X-Launchpad-Bug-Commenters: aarents X-Launchpad-Bug-Reporter: Alexandre arents (aarents) X-Launchpad-Bug-Modifier: Alexandre arents (aarents) References: <161098039664.6686.1246044899603761821.malonedeb@wampee.canonical.com> Message-Id: <161132156293.20216.13851368057585754104.malone@wampee.canonical.com> Subject: [Bug 1912224] Re: qemu may freeze during drive-mirroring on fragmented FS X-Launchpad-Message-Rationale: Subscriber (QEMU) @qemu-devel-ml X-Launchpad-Message-For: qemu-devel-ml Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="2d1d5e352f0d063d660df2300e31f66bed027fa5"; Instance="production" X-Launchpad-Hash: 5aa94fd1f631cc906a75c1e9821b54f0e948c5fa Received-SPF: none client-ip=91.189.90.7; envelope-from=bounces@canonical.com; helo=indium.canonical.com X-Spam_score_int: -46 X-Spam_score: -4.7 X-Spam_bar: ---- X-Spam_report: (-4.7 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, LOTS_OF_MONEY=0.001, MONEY_NOHTML=1.988, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bug 1912224 <1912224@bugs.launchpad.net> Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Seems a regression introduce in commit : (2.7.0) commit 0965a41e998ab820b5d660c8abfc8c819c97bc1b Author: Vladimir Sementsov-Ogievskiy Date: Thu Jul 14 20:19:01 2016 +0300 mirror: double performance of the bulk stage if the disc is full = Mirror can do up to 16 in-flight requests, but actually on full copy (the whole source disk is non-zero) in-flight is always 1. This happens as the request is not limited in size: the data occupies maximum available capacity of s->buf. = The patch limits the size of the request to some artificial constant (1 Mb here), which is not that big or small. This effectively enables back parallelism in mirror code as it was designed. = The result is important: the time to migrate 10 Gb disk is reduced from ~350 sec to 170 sec. If I revert this commit on master on top of (fef80ea073c48) (minor conflicts) issue disappears. on a fragmented file (average size per exent 2125 KB) of 58GB: upstream 207s (285 MB/s) instance is broken during mirroring upstream with revert 225s (262 MB/s) instance is fine. not fully tested, but the patch probably improve perf in other cases (as disccuss in git message), maybe just need to improve to avoid the freez= e? -- = You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1912224 Title: qemu may freeze during drive-mirroring on fragmented FS Status in QEMU: New Bug description: = We have odd behavior in operation where qemu freeze during long seconds, We started an thread about that issue here: https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05623.html It happens at least during openstack nova snapshot (qemu blockdev-mirror) or live block migration(which include network copy of disk). After further troubleshoots, it seems related to FS fragmentation on host. reproducible at least on: Ubuntu 18.04.3/4.18.0-25-generic/qemu-4.0 Ubuntu 16.04.6/5.10.6/qemu-5.2.0-rc2 # Lets create a dedicated file system on a SSD/Nvme 60GB disk in my case: $sudo mkfs.ext4 /dev/sda3 $sudo mount /dev/sda3 /mnt $df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/sda3 59G 53M 56G 1% /mnt #Create a fragmented disk on it using 2MB Chunks (about 30min): $sudo python3 create_fragged_disk.py /mnt 2 Filling up FS by Creating chunks files in: /mnt/chunks We are probably full as expected!!: [Errno 28] No space left on device Creating fragged disk file: /mnt/disk $ls -lhs = 59G -rw-r--r-- 1 root root 59G Jan 15 14:08 /mnt/disk $ sudo e4defrag -c /mnt/disk Total/best extents 41971/30 Average size per extent 1466 KB Fragmentation score 2 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag] This file (/mnt/disk) does not need defragmentation. Done. # the tool^^^ says it is not enough fragmented to be able to defrag. #Inject an image on fragmented disk sudo chown ubuntu /mnt/disk wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudim= g-amd64.img qemu-img convert -O raw bionic-server-cloudimg-amd64.img \ bionic-server-cloudimg-amd64.img.raw dd conv=3Dnotrunc iflag=3Dfullblock if=3Dbionic-server-cloudimg-amd64.img= .raw \ of=3D/mnt/disk bs=3D1M virt-customize -a /mnt/disk --root-password password:xxxx # logon run console activity ex: ping -i 0.3 127.0.0.1 $qemu-system-x86_64 -m 2G -enable-kvm -nographic \ -chardev socket,id=3Dtest,path=3D/tmp/qmp-monitor,server,nowait \ -mon chardev=3Dtest,mode=3Dcontrol \ -drive file=3D/mnt/disk,format=3Draw,if=3Dnone,id=3Ddrive-virtio-disk= 0,cache=3Dnone,discard\ -device virtio-blk-pci,scsi=3Doff,drive=3Ddrive-virtio-disk0,id=3Dvir= tio-disk0,bootindex=3D1,write-cache=3Don $sync $echo 3 | sudo tee -a /proc/sys/vm/drop_caches #start drive-mirror via qmp on another SSD/nvme partition nc -U /tmp/qmp-monitor {"execute":"qmp_capabilities"} {"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","tar= get":"/home/ubuntu/mirror","sync":"full","format":"qcow2"}} ^^^ qemu console may start to freeze at this step. NOTE: - smaller chunk sz and bigger disk size the worst it is. In operation we also have issue on 400GB disk size with average 13MB/e= xtent - Reproducible also on xfs = Expected behavior: ------------------- QEMU should remain steady, eventually only have decrease storage Performa= nce or mirroring, because of fragmented fs. Observed behavior: ------------------- Perf of mirroring is still quite good even on fragmented FS, but it breaks qemu. = ###################### create_fragged_disk.py ############ import sys import os import tempfile import glob import errno MNT_DIR =3D sys.argv[1] CHUNK_SZ_MB =3D int(sys.argv[2]) CHUNKS_DIR =3D MNT_DIR + '/chunks' DISK_FILE =3D MNT_DIR + '/disk' if not os.path.exists(CHUNKS_DIR): os.makedirs(CHUNKS_DIR) with open("/dev/urandom", "rb") as f_rand: mb_rand=3Df_rand.read(1024 * 1024) print("Filling up FS by Creating chunks files in: ",CHUNKS_DIR) try: while True: tp =3D tempfile.NamedTemporaryFile(dir=3DCHUNKS_DIR,delete=3DFals= e) for x in range(CHUNK_SZ_MB): tp.write(mb_rand) os.fsync(tp) tp.close() except Exception as ex: print("We are probably full as expected!!: ",ex) chunks =3D glob.glob(CHUNKS_DIR + '/*') print("Creating fragged disk file: ",DISK_FILE) with open(DISK_FILE, "w+b") as f_disk: for chunk in chunks: try: os.unlink(chunk) for x in range(CHUNK_SZ_MB): f_disk.write(mb_rand) os.fsync(f_disk) except IOError as ex: if ex.errno !=3D errno.ENOSPC: raise ###########################################################3 To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1912224/+subscriptions