From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C343FC433F5 for ; Sat, 12 Mar 2022 00:15:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229733AbiCLAQp (ORCPT ); Fri, 11 Mar 2022 19:16:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbiCLAQp (ORCPT ); Fri, 11 Mar 2022 19:16:45 -0500 Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD3CF11C03 for ; Fri, 11 Mar 2022 16:15:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1647044135; bh=JglsZiyGqgI8L5eqK5Pee9tS0AUW3158kevLEAZPZSo=; h=X-UI-Sender-Class:Date:Subject:To:Cc:References:From:In-Reply-To; b=iSQe2GIVQSIAgF42CH4seezhbtHVo6B2BYn3mu3GMcDPYdxSWptGp5891z/L6Ui4F V39U5BcaXvhbk2I6nSg8F0OCGeGp2iURn+jGlwAoMhDdKn6Ss3ip3qBrl9gkC8lQJl 4iWUQ2WYbl99LNX2qIc1nXJFmwOb8dWDOf7KkGSU= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [0.0.0.0] ([149.28.201.231]) by mail.gmx.net (mrgmx104 [212.227.17.174]) with ESMTPSA (Nemesis) id 1MLi8m-1nkMIo355E-00HgEt; Sat, 12 Mar 2022 01:15:35 +0100 Message-ID: <71d5b72b-6a6a-8d66-f20b-6c8872545537@gmx.com> Date: Sat, 12 Mar 2022 08:15:31 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 Subject: Re: Btrfs autodefrag wrote 5TB in one day to a 0.5TB SSD without a measurable benefit Content-Language: en-US To: Jan Ziak <0xe2.0x9a.0x9b@gmail.com> Cc: linux-btrfs@vger.kernel.org References: <3c668ffe-edb0-bbbb-cfe0-e307bad79b1a@gmx.com> <70bc749c-4b85-f7e6-b5fd-23eb573aab70@gmx.com> <7fc9f5b4-ddb6-bd3b-bb02-2bd4af703e3b@gmx.com> <078f9f05-3f8f-eef1-8b0b-7d2a26bf1f97@gmx.com> <452af644-e871-20e3-60b2-f69a92dc406d@gmx.com> From: Qu Wenruo In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:pH+zlYVFXj/R0bnWGVRl8oTyaof8V8GS5NPnkJDxmBwmfK1NtVQ ehP87j48qqzu/NCA902PmmcqvYCGmsJZLQnkIUD1v4XhZbpvKuR9ozlvXbU7pskEflRH9Oy JWH4clSaGFiDaQhnmZUQy7PdjN+KTMUwNpwdpnMfJbjrwyuQWcTWmVJcywEZnuORsquixWX TjAL26KUd/DrAdcupyDHg== X-UI-Out-Filterresults: notjunk:1;V03:K0:/FQOIZ6m4PA=:DNj8NJgHauJanwSS/I+ixP ggHmLGSzc6VAgadJU+47MZsIXXuAkmCsPev4wTY8Jj5zBbr8OLMgQ42RuGzwa2sJvo12RguLl 72aoC3aW9uH70CRCwREhpEqSVnFwkx461FiqWs8zei5gnVbQLAf9mFATjBHq5X4KQeCa2uneT Z1u6r/J8zOgt6iDsijcaV6rPb4hAmz6ltUsAQYHSNUGpX7YJqhIldGYaxpixFnkgkuOaUDNH2 QmxP+3cIkQBmaSQQSW324nBkv9Wz/gSD5qyqCGO1vRoeEVW1qAtTZW3t0rcfsHrZLKzwMRq0N s3j7YDYpeGLtDMK/rbJgL28/T+8ZvIp22VFcT5Z0gxlUfFkawrWYCDq2pOYOxEW79Fl6QtTTC 7AUiVdL0aWGzJOkDlyZJ/pidicmirTIIdZ+Qk1bkPbyXmztfkeZ7IvXrweRUB9Trb9zM3uXNz qMz/sPrVfajoUwZ2xC2/27/blUR582fBOaH84WQqmD7awepKMWxgCdZSeUn4NMnIC/eEK+n3t YLyXsS2xKoOXIwoMGIju0Kyk9EA4erphJOwEm4G0nhL9kmYsvTrfT1E51bXNA8Gp+B+fLIMsF 1ara6IdOVzs6cmcJpbPhEMBpQ7RAtP+rwxyyMqW2nugm+lReLYlfkaw8zf9mH/pfx5nNxzz5W R6it0IY5AWEV8uskOusT51ko7utweLuizVA9A+SwgkiEt2pAm32Q9kFuhM82fzXmf2jk6ncB2 5Zd/U1mdFeOSGDru3Fv2ZaEyAg+Yf/X/sQRSD2NvVxo/+o85r6wrhb9l0WgCsWKNzomXJ8OXx oC7nql5BNElaY1800X9Zub2y3muqc6vaKdFAHEAZ47fsgVG7AO1i3LQsc5hSXB++joo7W/2hR rmlg7V3pQX03nxe71qIRJM2rNEw18xoeLfiUWQQNEyWNB8PofQj+/6pb31C+l2pA7osH3v4kC Kt00YAp/sNCLZyqCz7KBtS8y8JMHY4LkNp5RTZxxBUl4rY6A+0JtCjwfU4HTuHno44TwiyDec m5tm/NFCkz80VD1GSYMWI30rMxzMjyZbgieH7WqIPuBBermweIfVDqC61xqCoZxUnsnL5eo/J Fdx3wYx7wgNXMA= Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org On 2022/3/12 08:01, Jan Ziak wrote: > On Sat, Mar 12, 2022 at 12:39 AM Qu Wenruo wrot= e: >> On 2022/3/12 07:28, Jan Ziak wrote: >>> On Sat, Mar 12, 2022 at 12:04 AM Qu Wenruo wr= ote: >>>> As stated before, autodefrag is not really that useful for database. >>> >>> Do you realize that you are claiming that btrfs autodefrag should not >>> - by design - be effective in the case of high-fragmentation files? >> >> Unfortunately, that's exactly what I mean. >> >> We all know random writes would cause fragments, but autodefrag is not >> like regular defrag ioctl, as it only scan newer extents. >> >> For example: >> >> Our autodefrag is required to defrag writes newer than gen 100, and our >> inode has the following layout: >> >> |---Ext A---|--- Ext B---|---Ext C---|---Ext D---|---Ext E---| >> Gen 50 Gen 101 Gen 49 Gen 30 Gen 30 >> >> Then autodefrag will only try to defrag extent B and extent C. >> >> Extent B meets the generation requirement, and is mergable with the nex= t >> extent C. >> >> But all the remaining extents A, D, E will not be defragged as their >> generations don't meet the requirement. >> >> While for regular defrag ioctl, we don't have such generation >> requirement, and is able to defrag all extents from A to E. >> (But cause way more IO). >> >> Furthermore, autodefrag works by marking the target range dirty, and >> wait for writeback (and hopefully get more writes near it, so it can ge= t >> even larger) >> >> But if the application, like the database, is calling fsync() >> frequently, such re-dirtied range is going to writeback almost >> immediately, without any further chance to get merged larger. > > So, basically, what you are saying is that you are refusing to work > together towards fixing/improving the auto-defragmentation algorithm. I'm explaining how autodefrag works, and work to improve autodefrag to handle the worst case scenario. If it doesn't fit your workload, that's unfortunate. There are always cases btrfs can't handle well. > > Based on your decision in this matter, I am now forced either to find > a replacement filesystem with features similar to btrfs or to > implement a filesystem (where auto-defragmentation works correctly) > myself. > > Since I failed to persuade you that there are serious errors/mistakes > in the current btrfs-autodefrag implementation, this is my last email > in this whole forum thread. > > Sincerely > Jan