From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D89AC47083 for ; Wed, 2 Jun 2021 23:35:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E40BB61246 for ; Wed, 2 Jun 2021 23:35:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229844AbhFBXhg (ORCPT ); Wed, 2 Jun 2021 19:37:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229553AbhFBXhg (ORCPT ); Wed, 2 Jun 2021 19:37:36 -0400 Received: from bhuna.collabora.co.uk (bhuna.collabora.co.uk [IPv6:2a00:1098:0:82:1000:25:2eeb:e3e3]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 058F3C06174A for ; Wed, 2 Jun 2021 16:35:53 -0700 (PDT) Received: from localhost (unknown [IPv6:2a00:5f00:102:0:f4d2:afff:fe2b:18b5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: krisman) by bhuna.collabora.co.uk (Postfix) with ESMTPSA id 230681F40F47; Thu, 3 Jun 2021 00:35:51 +0100 (BST) From: Gabriel Krisman Bertazi To: "Theodore Ts'o" Cc: kernel@collabora.com, linux-ext4@vger.kernel.org Subject: Potential regression with iomap DIO for 4k writes Date: Wed, 02 Jun 2021 19:35:48 -0400 Message-ID: <87lf7rkffv.fsf@collabora.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Hi, While I've been exploring the performance of different DIO implementations, I've come across what seems a noticeable regression (22% slowdown) in 4k writes in Ext4 when comparing the original direct-io with the current iomap dio implementation, as existing on linus/master. Perhaps you already know about this, but I'm having a hard time understanding the root cause, in order to attempt to improve the situation. * Benchmark For starter, I'm comparing three kernels, built with same config and compiler (gcc-8.4.0 (locally built)). My DUT is a Samsung SSD 970 EVO Plus 250GB dedicated to this test (no concurrent IO). - Kernel 1: Commit immediately before iomap for ext4 is merged ("f112a2fd1f59"). On the data below, this kernel is identified as 5.4.0-original-dio. Available in a public branch at: - Kernel 2: tag 5.5 (first release with dio-iomap). In the data below, identified as 5.5.0-old-iomap. For completeness, it is available at: - Kernel 3: Kernel tag 5.13-rc3. In the data below, identified as 5.13-rc3-iomap. For completeness, it is available at: I ran the fio job below with the combinations: BS=4k,16k and RW=read,write fio --ioengine libaio --size=2G --direct=1 --iodepth=64 --time_based=1 \ --thread=1 --overwrite=1 --runtime=100 --output-format=terse For every kernel test, the file system was recreated, and the 2GB file was pre-allocated. In an attempt to further isolate the problem, I tested both xfs and ext4 in the same condition. The script I used is available at: * Results I obtained the following performance results, relative to the baseline 5.4.0-original-dio. | IOPS | | kernel | read-4k | read-16k | write-4k | write-16k | |------------------------+------------------+-------------------+-------------------+-------------------+ | 5.13.0-rc3-iomap-ext4 | 1.01192950082305 | 1.00026413252562 | 0.806377013901006 | 1.00020735846057 | | 5.5.0-old-iomap-ext4 | 1.01154156662508 | 0.998753983520427 | 0.777051125458035 | 0.999937792461829 | | 5.13.0-rc3-iomap-xfs | 1.00234888443008 | 1.00027645151444 | 1.00996172750095 | 1.00156349447934 | | 5.5.0-old-iomap-xfs | 1.00010412786902 | 1.00202731110586 | 1.01502846821264 | 1.00149431330769 | Total IO is the amount of data copied (relative to baseline). | TOTAL_IO | kernel | read-4k | read-16k | write-4k | write-16k | |------------------------+------------------+-------------------+-------------------+-------------------| | 5.13.0-rc3-iomap-ext4 | 1.01193023173156 | 1.00026332569559 | 0.806377530301477 | 1.00014686835205 | | 5.5.0-old-iomap-ext4 | 1.01154196621591 | 0.998758131673757 | 0.777050753425118 | 0.999902824986834 | | 5.13.0-rc3-iomap-xfs | 1.00234893734134 | 1.00027535318322 | 1.00996437458991 | 1.00156305646789 | | 5.5.0-old-iomap-xfs | 1.00010328564078 | 1.00202831801018 | 1.01503060595258 | 1.00149069402364 | With a visualization of the above data here: The only out of the ordinary result seems to be in write-4k for Ext4, which suggests around 20% less IOPS (and total IO) for iomap in comparison to the original DIO. This is not a one-off run, as it seems to be consistently reproducible with more test runs in my environment. The performance reduction also doesn't reproduce on XFS. I tried to limit the influence of other parts of the kernel that could affect the behavior by comparing the kernel immediately before the introduction of dio-iomap for ext4 with the first version with that feature. By also observing that xfs doesn't change, I believe it to be ext4 specific. I'm also publishing raw data and all related material to the link below, in case anyone wants to tinker with my data: https://people.collabora.com/~krisman/dio/ Perhaps I'm missing something obvious. But I can't pinpoint a specific problem with my analysis. Is this expected, given the way ext4 iomap work? Do you have any idea of the root cause or how it can be improved? I will keep looking to this issue, but I'd like to share this partial result, in case there is a problem with my analysis, or if you have any suggestion. Thanks, -- Gabriel Krisman Bertazi