From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EE97C433F5 for ; Thu, 9 Sep 2021 10:24:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3FB8B610A3 for ; Thu, 9 Sep 2021 10:24:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3FB8B610A3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=imatronix.cl Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:56028 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mOHEm-0003wr-1C for qemu-devel@archiver.kernel.org; Thu, 09 Sep 2021 06:24:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57172) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mOHE5-0003Gy-07; Thu, 09 Sep 2021 06:24:13 -0400 Received: from ip27.imatronix.com ([200.63.97.108]:37244) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mOHE1-0002ut-1P; Thu, 09 Sep 2021 06:24:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=superfactura.cl; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=mhaRn7KeKJ4+USzYE670oOmvI+9/ZCwnMQjkmI6hdJw=; b=EB8NL0izfpvhcRCRaPywTdse0s /7ZWzfkORVFJtGK4T87yWxdU3rEO4wSHw+ooKDlMoR1fotAqMAc4b0nVO3qG8IDnqudMw2xrNaDkp y2+syRgzFyRED7bY+a+yREVEe7FSwP+GHd/0GAfbpnDiFZTOf1dXYj5kJPTWBYgkTfNqg+vQdl1gm tv/oHIYju4F3K5JqtXEbRT5N5ynSZSXb/j3ot7sDxH8LNjbGv8/k2JzufowfHNmcGkdtmT65HVp8x 4Zz63EcANI2dgbZpeL8uOsyMvN4imaVGybSr2zKBbuez0tLWuekMtcag9oW0kAi6ooISYYvhVz5K2 bR2x8Z2Q==; Received: from [200.73.112.45] by cpanel.imatronix.com with esmtpsa (TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.94.2) (envelope-from ) id 1mOHDx-00011W-8w ; Thu, 09 Sep 2021 07:24:03 -0300 Subject: Re: qcow2 perfomance: read-only IO on the guest generates high write IO on the host To: Kevin Wolf References: <55980bc8-97ad-77a4-1bb7-a086f2712ea1@imatronix.cl> From: Christopher Pereira Organization: IMATRONIX S.A. Message-ID: Date: Thu, 9 Sep 2021 07:23:56 -0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------A19125F1FBA61FE84814F263" Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel.imatronix.com X-AntiAbuse: Original Domain - nongnu.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - imatronix.cl X-Get-Message-Sender-Via: cpanel.imatronix.com: authenticated_id: soporte@superfactura.cl X-Authenticated-Sender: cpanel.imatronix.com: soporte@superfactura.cl X-Source: X-Source-Args: X-Source-Dir: Received-SPF: pass client-ip=200.63.97.108; envelope-from=kripper@imatronix.cl; helo=ip27.imatronix.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-1.922, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is a multi-part message in MIME format. --------------A19125F1FBA61FE84814F263 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 24-08-2021 11:37, Kevin Wolf wrote: > [ Cc: qemu-block ] > > Am 11.08.2021 um 13:36 hat Christopher Pereira geschrieben: >> Hi, >> >> I'm reading a directory with 5.000.000 files (2,4 GB) inside a guest using >> "find | grep -c". >> >> On the host I saw high write IO (40 MB/s !) during over 1 hour using >> virt-top. >> >> I later repeated the read-only operation inside the guest and no additional >> data was written on the host. The operation took only some seconds. >> >> I believe QEMU was creating some kind of cache or metadata map the first >> time I accessed the inodes. > No, at least in theory, QEMU shouldn't allocate anything when you're > just reading. Hmm...interesting. > Are you sure that this isn't activity coming from your guest OS? Yes. iotop was showing only read IOs on the guest, and on the host iotop and virt-top where showing strong write IOs for hours. Stopping the "find" command on the guest also stopped the write IOs on the host. >> But I wonder why the cache or metadata map wasn't available the first time >> and why QEMU had to recreate it? >> >> The VM has "compressed base <- snap 1" and base was converted without >> prealloc. >> >> Is it because we created the base using convert without metadata prealloc >> and so the metadata map got lost? >> >> I will do some experiments soon using convert + metadata prealloc and >> probably find out myself, but I will happy to read your comments and gain >> some additional insights. >> If it the problem persists, I would try again without compression. > What were the results of your experiments? Is the behaviour related to > any of these options? I will do the experiments and report back. It's also strange that the second time I repeat the "find" command, I see no more write IOs and it takes only seconds instead of hours. I was assuming QEMU was creating some kind of map or cache on the snapshot for the content present in the base, but now I got more curious. --------------A19125F1FBA61FE84814F263 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
On 24-08-2021 11:37, Kevin Wolf wrote:
[ Cc: qemu-block ]

Am 11.08.2021 um 13:36 hat Christopher Pereira geschrieben:

Hi,

I'm reading a directory with 5.000.000 files (2,4 GB) inside a guest using
"find | grep -c".

On the host I saw high write IO (40 MB/s !) during over 1 hour using
virt-top.

I later repeated the read-only operation inside the guest and no additional
data was written on the host. The operation took only some seconds.

I believe QEMU was creating some kind of cache or metadata map the first
time I accessed the inodes.

No, at least in theory, QEMU shouldn't allocate anything when you're
just reading.
Hmm...interesting.
Are you sure that this isn't activity coming from your guest OS?

Yes. iotop was showing only read IOs on the guest, and on the host iotop and virt-top where showing strong write IOs for hours.
Stopping the "find" command on the guest also stopped the write IOs on the host. 

But I wonder why the cache or metadata map wasn't available the first time
and why QEMU had to recreate it?

The VM has "compressed base <- snap 1" and base was converted without
prealloc.

Is it because we created the base using convert without metadata prealloc
and so the metadata map got lost?

I will do some experiments soon using convert + metadata prealloc and
probably find out myself, but I will happy to read your comments and gain
some additional insights.
If it the problem persists, I would try again without compression.

What were the results of your experiments? Is the behaviour related to
any of these options?

I will do the experiments and report back.

It's also strange that the second time I repeat the "find" command, I see no more write IOs and it takes only seconds instead of hours.

I was assuming QEMU was creating some kind of map or cache on the snapshot for the content present in the base, but now I got more curious.

--------------A19125F1FBA61FE84814F263--