From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3672C4338F for ; Sat, 24 Jul 2021 18:50:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 83F1260E96 for ; Sat, 24 Jul 2021 18:50:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 83F1260E96 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 245306B005D; Sat, 24 Jul 2021 14:50:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F5ED6B006C; Sat, 24 Jul 2021 14:50:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10BB46B0070; Sat, 24 Jul 2021 14:50:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0040.hostedemail.com [216.40.44.40]) by kanga.kvack.org (Postfix) with ESMTP id EC80B6B005D for ; Sat, 24 Jul 2021 14:50:34 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8E573180ACEE6 for ; Sat, 24 Jul 2021 18:50:34 +0000 (UTC) X-FDA: 78398372388.28.36EC41D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id A20793009970 for ; Sat, 24 Jul 2021 18:50:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=eTIqvGtmgSD1grJHtfLM6OPgrkrmdCF2vMDDkcCO9xM=; b=akqHBRwHpJ3ScR0Pn2WCaCdoTi pdA+NUUAPz92r+ZAn2WpHPkluxu7XqiuyiDmiOFgDkEepmQHi8WRmWwTMkQsLTrW0UACI7Vp24pIM S537piHtomKG30N9Tnc7L7szCRymrlfaNK+pS5OhxTJlZWuMPj+ihlr22KoYkSPgr/SU9vhDErNPm NNMDNVVnL+rD+8PZmqY0cue2xlkXJMtqcK7lEZTI7vkKoQ5tF9QZ9AMaqpQ5lyXtcAeT9AbXQF1jF fZ1EPdOl9qtDU+tRnykno/ZbapvYKr/KkIHaCbOzDUPTr5IhgPQj06/h7dZa4cTVHPXFn69kkG+/l qMmfq2Xw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1m7Mir-00CTiD-Iv; Sat, 24 Jul 2021 18:50:13 +0000 Date: Sat, 24 Jul 2021 19:50:05 +0100 From: Matthew Wilcox To: James Bottomley Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Linus Torvalds , Andrew Morton , "Darrick J. Wong" , Christoph Hellwig , Andres Freund , Michael Larabel Subject: Re: Folios give an 80% performance win Message-ID: References: <20210715033704.692967-1-willy@infradead.org> <1e48f7edcb6d9a67e8b78823660939007e14bae1.camel@HansenPartnership.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A20793009970 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=akqHBRwH; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: 9icgyhdgbdim355yyedpfydqz9g556tf X-HE-Tag: 1627152633-939068 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, Jul 24, 2021 at 11:23:25AM -0700, James Bottomley wrote: > On Sat, 2021-07-24 at 19:14 +0100, Matthew Wilcox wrote: > > On Sat, Jul 24, 2021 at 11:09:02AM -0700, James Bottomley wrote: > > > On Sat, 2021-07-24 at 18:27 +0100, Matthew Wilcox wrote: > > > > What blows me away is the 80% performance improvement for > > > > PostgreSQL. I know they use the page cache extensively, so it's > > > > plausibly real. I'm a bit surprised that it has such good > > > > locality, and the size of the win far exceeds my > > > > expectations. We should probably dive into it and figure out > > > > exactly what's going on. > > > > > > Since none of the other tested databases showed more than a 3% > > > improvement, this looks like an anomalous result specific to > > > something in postgres ... although the next biggest db: mariadb > > > wasn't part of the tests so I'm not sure that's > > > definitive. Perhaps the next step should be to t > > > est mariadb? Since they're fairly similar in domain (both full > > > SQL) if mariadb shows this type of improvement, you can > > > safely assume it's something in the way SQL databases handle paging > > > and if it doesn't, it's likely fixing a postgres inefficiency. > > > > I think the thing that's specific to PostgreSQL is that it's a heavy > > user of the page cache. My understanding is that most databases use > > direct IO and manage their own page cache, while PostgreSQL trusts > > the kernel to get it right. > > That's testable with mariadb, at least for the innodb engine since the > flush_method is settable. We're still not communicating well. I'm not talking about writes, I'm talking about reads. Postgres uses the page cache for reads. InnoDB uses O_DIRECT (afaict). See articles like this one: https://www.percona.com/blog/2018/02/08/fsync-performance-storage-devices/ : The first and most obvious type of IO are pages reads and writes from : the tablespaces. The pages are most often read one at a time, as 16KB : random read operations. Writes to the tablespaces are also typically : 16KB random operations, but they are done in batches. After every batch, : fsync is called on the tablespace file handle. (the current folio patch set does not create multi-page folios for writes, only for reads) I downloaded the mariadb source package that's in Debian, and from what I can glean, it does indeed set O_DIRECT on data files in Linux, through os_file_set_nocache(). > > Regardless of whether postgres is "doing something wrong" or not, > > do you not think that an 80% performance win would exert a certain > > amount of pressure on distros to do the backport? > > Well, I cut the previous question deliberately, but if you're going to > force me to answer, my experience with storage tells me that one test > being 10x different from all the others usually indicates a problem > with the benchmark test itself rather than a baseline improvement, so > I'd wait for more data. ... or the two benchmarks use Linux in completely different ways such that one sees a huge benefit while the other sees none. Which is what you'd expect for a patchset that improves the page cache and using a benchmark that doesn't use the page cache.