From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AD53C35257 for ; Fri, 2 Oct 2020 22:46:47 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5A93A206C9 for ; Fri, 2 Oct 2020 22:46:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5A93A206C9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=bugs.launchpad.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:60338 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kOTp7-00042M-AL for qemu-devel@archiver.kernel.org; Fri, 02 Oct 2020 18:46:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42716) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kOTo6-0003Pm-2A for qemu-devel@nongnu.org; Fri, 02 Oct 2020 18:45:42 -0400 Received: from indium.canonical.com ([91.189.90.7]:35230) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kOTo3-0003tT-AY for qemu-devel@nongnu.org; Fri, 02 Oct 2020 18:45:41 -0400 Received: from loganberry.canonical.com ([91.189.90.37]) by indium.canonical.com with esmtp (Exim 4.86_2 #2 (Debian)) id 1kOTo1-0002Ac-7L for ; Fri, 02 Oct 2020 22:45:37 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 350842E80DB for ; Fri, 2 Oct 2020 22:45:37 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Fri, 02 Oct 2020 22:34:00 -0000 From: Richard Henderson <1898011@bugs.launchpad.net> To: qemu-devel@nongnu.org X-Launchpad-Notification-Type: bug X-Launchpad-Bug: product=qemu; status=Confirmed; importance=Undecided; assignee=None; X-Launchpad-Bug-Information-Type: Public X-Launchpad-Bug-Private: no X-Launchpad-Bug-Security-Vulnerability: no X-Launchpad-Bug-Commenters: ksserebr rth X-Launchpad-Bug-Reporter: Kostya Serebryany (ksserebr) X-Launchpad-Bug-Modifier: Richard Henderson (rth) References: <160153380394.6201.10648910301442382269.malonedeb@soybean.canonical.com> Message-Id: <160167804013.13277.5302166593333035509.malone@gac.canonical.com> Subject: [Bug 1898011] Re: mmap MAP_NORESERVE of 2^42 bytes consumes 16Gb of actual RAM X-Launchpad-Message-Rationale: Subscriber (QEMU) @qemu-devel-ml X-Launchpad-Message-For: qemu-devel-ml Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="d50d1e75c500726862802414f880ee3e3bb759bf"; Instance="production" X-Launchpad-Hash: eeaa0bf14abc9cf1f73f680bb9dee5831ccf7c31 Received-SPF: none client-ip=91.189.90.7; envelope-from=bounces@canonical.com; helo=indium.canonical.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/02 16:00:49 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -65 X-Spam_score: -6.6 X-Spam_bar: ------ X-Spam_report: (-6.6 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bug 1898011 <1898011@bugs.launchpad.net> Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Without actually looking, an allocation of 2**42 (4PB) requires 2**30 (1G) pages, and thus 1G page table entries, so 16GB memory allocation sounds about right for qemu's internal page table allocation. We need to change data structures for representing guest memory, probably akin to the kernel's VMAs. ** Changed in: qemu Status: New =3D> Confirmed -- = You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1898011 Title: mmap MAP_NORESERVE of 2^42 bytes consumes 16Gb of actual RAM Status in QEMU: Confirmed Bug description: Run this program: #include #include int main() { for (int i =3D 30; i <=3D 44; i++) { fprintf(stderr, "trying 2**%d\n", i); mmap((void*)0x600000000000,1ULL << i, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED|MAP_NORESERVE= ,-1,0); } } (tried qemu-x86_64 and qemu-aarch64, 4.2.1 and trunk/5.1.50) On each iteration qemu will consume 2x more physical RAM, = e.g. when mapping 2^42 it will have RSS of 16Gb. On normal linux it works w/o consuming much RAM, due to MAP_NORESERVE. Also: qemu -strace prints 0 instead of the correct size starting from siz= e=3D2^32 and prints -2147483648 for size=3D2^31. = mmap(0x0000600000000000,1073741824,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS|MA= P_FIXED|MAP_NORESERVE,-1,0) =3D 0x0000600000000000 mmap(0x0000600000000000,-2147483648,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS|M= AP_FIXED|MAP_NORESERVE,-1,0) =3D 0x0000600000000000 mmap(0x0000600000000000,0,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED|M= AP_NORESERVE,-1,0) =3D 0x0000600000000000 To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1898011/+subscriptions