From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: ** X-Spam-Status: No, score=2.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57F3DC433C1 for ; Sat, 27 Mar 2021 01:59:40 +0000 (UTC) Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C603661A24 for ; Sat, 27 Mar 2021 01:59:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C603661A24 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernelnewbies-bounces+kernelnewbies=archiver.kernel.org@kernelnewbies.org Received: from localhost ([::1] helo=shelob.surriel.com) by shelob.surriel.com with esmtp (Exim 4.94) (envelope-from ) id 1lPyEk-0000Qo-Dd for kernelnewbies@archiver.kernel.org; Fri, 26 Mar 2021 21:59:38 -0400 Received: from mail-io1-xd2a.google.com ([2607:f8b0:4864:20::d2a]) by shelob.surriel.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94) (envelope-from ) id 1lPyD2-0007Qt-Mj for Kernelnewbies@kernelnewbies.org; Fri, 26 Mar 2021 21:57:52 -0400 Received: by mail-io1-xd2a.google.com with SMTP id j26so7250277iog.13 for ; Fri, 26 Mar 2021 18:57:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=jR6kG6X9ykkFJ2rnZydeHMcq6Ojf1yX2L9CaIPO4w4w=; b=BkDlcWbEg1j7n4c732cwkxfAOs1loU/HajgTrK7BCJIVK8rHTEwI9Umb5iqV67kcPS dpTJ89X4X72bywRsJLPFTKjs5/0CXf3IJnpCVwHF96+WG8UJvFriTd6lsZYaA4aGrWjJ XiMSfscWMFzwvnkRcl0jZl+XaNuVo5yzl/mhWiCa0PiII0ywjyrTE/+doz+U3B461q0m Nl9FBYV5ggQtXFW4+SCwdOf1nowcUS6CP5MOGHmDe3bYm6KW/IVmj784HEQ7qqIJqKb/ ZR4itGeYR/VuKzdhTOQ1inH9PyouyV0EUWwtQ1YOe0BbC8YqFu1vS5mnaPsLqWDLlMcg ZQJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=jR6kG6X9ykkFJ2rnZydeHMcq6Ojf1yX2L9CaIPO4w4w=; b=VX7WOfput5COddHU3v05E4VutDtl/4V6JDmU+eMnVzRWr/qQKOd3uOPvQB8A2796it Iu348oJlCbUhHbUxRmr8gR7TqeQhMP6pQmlxh68V2DxqiJ8OTK14dWZgDrPHRWz0pKLE 1zW9EtDzwkPZ8PrYSsUdeWdDkR3sb0ZvbiJyHBWfrFGrdzGQAH9Y+oWeQvVjNn0VTrTq XhzcUvTOe36JeaGdXcgdCO1DaMZ2fj7iBMr3dd+fuHQGVVXv1QJMB89oQaUbhDJRGQJb m+yPvtAum8L6bRB0IDmfD/0L9/tCbFlvX0e0npyrsEiEFnVCTV2wFsCyEkq3WCAXHSeH H6fQ== X-Gm-Message-State: AOAM533udrPTOyOeudgeTUhQLsEBmMdEqSv5v4HN7yEXwdTMNI6SWCAd JGtg0oNsz2a+ZT3m4DfB/2a7tC7jOXmgOWqQPDhMSuh5 X-Google-Smtp-Source: ABdhPJz/eSapSEL1umzKQqU5cexPCu3qcUEA8ZBI+aHZKzMusHYDxRDHvHx4cNDSRMhIptKzigXhkvhNbIUjXb+QwcQ= X-Received: by 2002:a5d:9ac4:: with SMTP id x4mr12247090ion.117.1616810268887; Fri, 26 Mar 2021 18:57:48 -0700 (PDT) MIME-Version: 1.0 References: <36a97042-528d-cda6-8817-c91ba73072b2@student.hpi.de> In-Reply-To: <36a97042-528d-cda6-8817-c91ba73072b2@student.hpi.de> From: Mulyadi Santosa Date: Sat, 27 Mar 2021 08:57:35 +0700 Message-ID: Subject: Re: Understanding the locking behavior of msync To: kernelnewbies X-BeenThere: kernelnewbies@kernelnewbies.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Learn about the Linux kernel List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============6676489538468699390==" Errors-To: kernelnewbies-bounces+kernelnewbies=archiver.kernel.org@kernelnewbies.org --===============6676489538468699390== Content-Type: multipart/alternative; boundary="000000000000bf0d9105be7af9ce" --000000000000bf0d9105be7af9ce Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Mar 26, 2021, 22:05 Maximilian B=C3=B6ther < maximilian.boether@student.hpi.de> wrote: > Hello! > > I am investigating an application that writes random data in fixed-size > chunks (e.g. 4k) to random locations in a large buffer file. I have > several processes (not threads) doing that, each process has its own > buffer file assigned. > > If I use mmap+msync to write and persist data to disk, I see a > performance spike for 16 processes, and a performance drop for more > threads (32 processes). The CPU has 32 logical cores in total, and we > are not CPU bound. > > If I use open+write+fsync, I do not see such a spike, instead a > performance plateau (and mmap is slower than open/write). > > I've read multiple times [1,2] that both mmap and msync can take locks. > With vtune, I analyzed that we are indeed spinlocking, and spending the > most time in clear_page_erms and xas_load functions. > > However, when reading the source code for msync [3], I cannot understand > whether these locks are global or per-file. The paper [2] states that > the locks are on radix-trees within the kernel that are per-file, > however, as I do observe some spinlocks in the kernel, I believe that > some locks may be global, as I have one file per process. > > Do you have an explanation on why we have such a spike at 16 processes > for mmap and input on the locking behavior of msync? > > Thank you! > > Best, > Maximilian B=C3=B6ther > > [1] > > https://kb.pmem.io/development/100000025-Why-msync-is-less-optimal-for-pe= rsistent-memory/ > - I know it's about PMem, but the lock argument is important > > [2] Optimizing Memory-mapped I/O for Fast Storage Devices, Papagiannis > et al., ATC '20 > > [3] https://elixir.bootlin.com/linux/latest/source/mm/msync.c > > _______________________________________________ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbi > es > Is it NUMA? --000000000000bf0d9105be7af9ce Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Mar 26, 2021, 22:05 Maximilian B=C3=B6ther <= ;maximilian.boether@st= udent.hpi.de> wrote:
Hello!<= br>
I am investigating an application that writes random data in fixed-size chunks (e.g. 4k) to random locations in a large buffer file. I have
several processes (not threads) doing that, each process has its own
buffer file assigned.

If I use mmap+msync to write and persist data to disk, I see a
performance spike for 16 processes, and a performance drop for more
threads (32 processes). The CPU has 32 logical cores in total, and we
are not CPU bound.

If I use open+write+fsync, I do not see such a spike, instead a
performance plateau (and mmap is slower than open/write).

I've read multiple times [1,2] that both mmap and msync can take locks.=
With vtune, I analyzed that we are indeed spinlocking, and spending the most time in clear_page_erms and xas_load functions.

However, when reading the source code for msync [3], I cannot understand whether these locks are global or per-file. The paper [2] states that
the locks are on radix-trees within the kernel that are per-file,
however, as I do observe some spinlocks in the kernel, I believe that
some locks may be global, as I have one file per process.

Do you have an explanation on why we have such a spike at 16 processes
for mmap and input on the locking behavior of msync?

Thank you!

Best,
Maximilian B=C3=B6ther

[1]
= https://kb.pmem.io/development/100000025-Why-msync-is-less-optimal-for-pers= istent-memory/
- I know it's about PMem, but the lock argument is important

[2] Optimizing Memory-mapped I/O for Fast Storage Devices, Papagiannis
et al., ATC '20

[3] https://elixir.bootlin.com/l= inux/latest/source/mm/msync.c

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies= .org/mailman/listinfo/kernelnewbies

Is it NUMA?
<= br>
--000000000000bf0d9105be7af9ce-- --===============6676489538468699390== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies --===============6676489538468699390==--