From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61FB310EC for ; Mon, 11 Apr 2022 06:41:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1649659314; x=1681195314; h=message-id:date:mime-version:to:cc:references:from: subject:in-reply-to:content-transfer-encoding; bh=qqbyajFBqHRHUrF7z93uzdgz9ZWtIREVY38IHJI0vvs=; b=HRmv+Wzo+zTtE59KfLTXBS6tBm+3P/mFn++KxF/lYPjrJRyEks3pOQg2 mMBuy2c3a3ajF9j04oDf+v761b1GlxLI5qBWhqOSu2l/bJOVa3OQBBfOE OENSo3S3l3t/yHQkdMfyjwEZbSvGvvpJqZvnhCeNVmwL6GulwyGhC20jq pm8eMbtQESnmWjj2S9FZ1yPYB3U1BlOYL9hd2R198fs4n0BN2w37Fjc9h 2eksxdPg3GTvbHRtS9RGFahSdnCiiFNhCYNCBdxqyvPZgix5nvMx7u9ff hXgwqdsc3ii85iX7guU06DXT7VC2UkEqFWnvLC3Z/gbVEN3QwU44D/Nxw Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10313"; a="322490012" X-IronPort-AV: E=Sophos;i="5.90,251,1643702400"; d="scan'208";a="322490012" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2022 23:41:53 -0700 X-IronPort-AV: E=Sophos;i="5.90,251,1643702400"; d="scan'208";a="525314042" Received: from srkondle-mobl.amr.corp.intel.com (HELO [10.212.113.6]) ([10.212.113.6]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2022 23:41:51 -0700 Message-ID: Date: Sun, 10 Apr 2022 23:41:57 -0700 Precedence: bulk X-Mailing-List: linux-coco@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Content-Language: en-US To: "Kirill A. Shutemov" Cc: "Kirill A. Shutemov" , Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel , Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , Brijesh Singh , Mike Rapoport , David Hildenbrand , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, Mike Rapoport References: <20220405234343.74045-1-kirill.shutemov@linux.intel.com> <20220405234343.74045-2-kirill.shutemov@linux.intel.com> <93a7cfdf-02e6-6880-c563-76b01c9f41f5@intel.com> <20220409175210.xik3ue3shpagskvi@box.shutemov.name> From: Dave Hansen Subject: Re: [PATCHv4 1/8] mm: Add support for unaccepted memory In-Reply-To: <20220409175210.xik3ue3shpagskvi@box.shutemov.name> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 4/9/22 10:52, Kirill A. Shutemov wrote: > On Fri, Apr 08, 2022 at 12:11:58PM -0700, Dave Hansen wrote: >> On 4/5/22 16:43, Kirill A. Shutemov wrote: >>> Kernel only needs to accept memory once after boot, so during the boot >>> and warm up phase there will be a lot of memory acceptance. After things >>> are settled down the only price of the feature if couple of checks for >>> PageUnaccepted() in allocate and free paths. The check refers a hot >>> variable (that also encodes PageBuddy()), so it is cheap and not visible >>> on profiles. >> >> Let's also not sugar-coat this. Page acceptance is hideously slow. >> It's agonizingly slow. To boot, it's done holding a global spinlock >> with interrupts disabled (see patch 6/8). At the very, very least, each >> acceptance operation involves a couple of what are effectively ring >> transitions, a 2MB memset(), and a bunch of cache flushing. >> >> The system is going to be downright unusable during this time, right? ... >> Do we need anything more discrete to tell users when acceptance is over? > > I can imagine setups that where acceptance is never over. A VM running > a workload with fixed dataset can have planty of memory unaccepted. > > I don't think "make it over" should be the goal. I agree, there will be users that don't care when acceptance is over. But, I'm also sure that there are users that will care deeply. >> For instance, maybe they run something and it goes really slow, they >> watch "accept_memory" until it stops. They rejoice at their good >> fortune! Then, memory allocation starts falling over to a new node and >> the agony beings anew. >> >> I can think of dealing with this in two ways: >> >> cat /sys/.../unaccepted_pages_left >> >> which just walks the bitmap and counts the amount of pages remaining. or >> something like: >> >> echo 1 > /sys/devices/system/node/node0/make_the_pain_stop >> >> Which will, well, make the pain stop on node0. > > Sure we can add handles. But API is hard. Maybe we should wait and see > what is actually needed. (Yes, I'm lazy.:) Let's just call out the possible (probable?) need for new ABI here. Maybe it will cue folks who care to speak up.