From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9775C433F5 for ; Sat, 9 Apr 2022 17:50:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7C3A6B0071; Sat, 9 Apr 2022 13:50:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2A166B0073; Sat, 9 Apr 2022 13:50:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCC486B0074; Sat, 9 Apr 2022 13:50:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id AA3A66B0071 for ; Sat, 9 Apr 2022 13:50:41 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 878F1207D4 for ; Sat, 9 Apr 2022 17:50:41 +0000 (UTC) X-FDA: 79338080682.14.7B9CBCF Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by imf05.hostedemail.com (Postfix) with ESMTP id 05B66100002 for ; Sat, 9 Apr 2022 17:50:40 +0000 (UTC) Received: by mail-lf1-f41.google.com with SMTP id bu29so20106725lfb.0 for ; Sat, 09 Apr 2022 10:50:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=2BJwi+4MUHVrKX308NfOg46ABCdrEGwFNuCfJTM4PEA=; b=MN50HswbicjFFL7K5mQ5CImChULXEkQz0FzkiaI3+yYKdPBrcrNzQMylJQdy8TjEQf 5wcmnqtlY6cIWr4Xwwhf9qfp8YDXgnNp89sliELF/wPXIOODX3w4yMpkdZo4HBh1nycu HS84zY9faBsIE5HwW8AqridmwGGbjA8ZCFMdSkmf8q+yET0bkKRsutZOyVTnJxI0XK0X zT8+sXdketeYf0lWkZ/JTEURmbHJhouPvYqBYKM/lXukoPLyzU99u0RB86e6XN0moGGi Y5ih3+OSgtBEuDPI8SUkl+4qC2exssdnliXZ71YfnkSnuCm5gudeyMUsppD01fj1gAeH 0xFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=2BJwi+4MUHVrKX308NfOg46ABCdrEGwFNuCfJTM4PEA=; b=nsEaY8A6PiQ78TMGVIpXrzAIUjBgDSz3WmSF1KZWd7wf8gLjuV83armBh6PbFun1g3 7Rn+NNq+r4x60F3K07F7CJQpnmM3yuc9iI59DvKyGqjdtzZgSupmN6yVeWZy1csiDJV+ 0Q0nLLHnpgLX0w81Qa6/5V7neeWncTfvg3I6JPyXZ1rRxeW3tyLOQxjr+4GtvkTP+97L hhAOU0j2tmwDNfgpdOcH8A0qPLBB3iixXk+ijYbxiKNjihO5yuD6gPUlucjtXgjJaUfi 1YNAW0GX+RNO5Pf+HSd1/b+IwtZbYTBPQHN/5JFGCPtZbSEk4zAKdk903r9G4JF3SPSz M9OA== X-Gm-Message-State: AOAM533JxD1QUXt5rJzgDEvQj+FXBOMBGZHTaY86iMDedxlPZbYYscWb +bQfp8nLza1KbQisQyCL7vyp+g== X-Google-Smtp-Source: ABdhPJyiKDi7TSfRV/vRqhjM4G5p2LRlKpj4VTzoGYJmpsDau3aBPPbEKGpuhXEfzoNTsoN8W7+JPw== X-Received: by 2002:a05:6512:ac1:b0:46b:8fd0:b030 with SMTP id n1-20020a0565120ac100b0046b8fd0b030mr2977704lfu.372.1649526639318; Sat, 09 Apr 2022 10:50:39 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id h24-20020a19ca58000000b0046b9dbfc809sm65098lfj.56.2022.04.09.10.50.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Apr 2022 10:50:38 -0700 (PDT) Received: by box.localdomain (Postfix, from userid 1000) id 1747B1039DB; Sat, 9 Apr 2022 20:52:10 +0300 (+03) Date: Sat, 9 Apr 2022 20:52:10 +0300 From: "Kirill A. Shutemov" To: Dave Hansen Cc: "Kirill A. Shutemov" , Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel , Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , Brijesh Singh , Mike Rapoport , David Hildenbrand , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, Mike Rapoport Subject: Re: [PATCHv4 1/8] mm: Add support for unaccepted memory Message-ID: <20220409175210.xik3ue3shpagskvi@box.shutemov.name> References: <20220405234343.74045-1-kirill.shutemov@linux.intel.com> <20220405234343.74045-2-kirill.shutemov@linux.intel.com> <93a7cfdf-02e6-6880-c563-76b01c9f41f5@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <93a7cfdf-02e6-6880-c563-76b01c9f41f5@intel.com> X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 05B66100002 X-Stat-Signature: 6k8u95w6t8mnh6tejo959shmgkie58zo Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=shutemov-name.20210112.gappssmtp.com header.s=20210112 header.b=MN50Hswb; dmarc=none; spf=none (imf05.hostedemail.com: domain of kirill@shutemov.name has no SPF policy when checking 209.85.167.41) smtp.mailfrom=kirill@shutemov.name X-Rspam-User: X-HE-Tag: 1649526640-451807 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Apr 08, 2022 at 12:11:58PM -0700, Dave Hansen wrote: > On 4/5/22 16:43, Kirill A. Shutemov wrote: > > Kernel only needs to accept memory once after boot, so during the boot > > and warm up phase there will be a lot of memory acceptance. After things > > are settled down the only price of the feature if couple of checks for > > PageUnaccepted() in allocate and free paths. The check refers a hot > > variable (that also encodes PageBuddy()), so it is cheap and not visible > > on profiles. > > Let's also not sugar-coat this. Page acceptance is hideously slow. > It's agonizingly slow. To boot, it's done holding a global spinlock > with interrupts disabled (see patch 6/8). At the very, very least, each > acceptance operation involves a couple of what are effectively ring > transitions, a 2MB memset(), and a bunch of cache flushing. > > The system is going to be downright unusable during this time, right? Well, yes. The CPU that doing accepting is completely blocked by it. But other CPUs may proceed until in in its turn steps onto memory accepting. > Sure, it's *temporary* and only happens once at boot. But, it's going > to suck. > > Am I over-stating this in any way? > > The ACCEPT_MEMORY vmstat is good to have around. Thanks for adding it. > But, I think we should also write down some guidance like: > > If your TDX system seems as slow as snail after boot, look at > the "accept_memory" counter in /proc/vmstat. If it is > incrementing, then TDX memory acceptance is likely to blame. Sure. Will add to commit message. > Do we need anything more discrete to tell users when acceptance is over? I can imagine setups that where acceptance is never over. A VM running a workload with fixed dataset can have planty of memory unaccepted. I don't think "make it over" should be the goal. > For instance, maybe they run something and it goes really slow, they > watch "accept_memory" until it stops. They rejoice at their good > fortune! Then, memory allocation starts falling over to a new node and > the agony beings anew. > > I can think of dealing with this in two ways: > > cat /sys/.../unaccepted_pages_left > > which just walks the bitmap and counts the amount of pages remaining. or > something like: > > echo 1 > /sys/devices/system/node/node0/make_the_pain_stop > > Which will, well, make the pain stop on node0. Sure we can add handles. But API is hard. Maybe we should wait and see what is actually needed. (Yes, I'm lazy.:) -- Kirill A. Shutemov