From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3682DC433DF for ; Fri, 16 Oct 2020 23:11:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BF0D922201 for ; Fri, 16 Oct 2020 23:11:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=mg.codeaurora.org header.i=@mg.codeaurora.org header.b="R+kKwoPo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2410669AbgJPXLW (ORCPT ); Fri, 16 Oct 2020 19:11:22 -0400 Received: from m42-4.mailgun.net ([69.72.42.4]:28418 "EHLO m42-4.mailgun.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2392711AbgJPXLW (ORCPT ); Fri, 16 Oct 2020 19:11:22 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1602889881; h=Message-ID: Subject: Cc: To: From: Date: Content-Transfer-Encoding: Content-Type: MIME-Version: Sender; bh=KP659FIJpORGhMvLQ93vED7u9wN2xczwCJIEuJ6VKqM=; b=R+kKwoPodvudtiATvOBHc0B1LQV6Z/+bn3aLINvclfqQcKQHY6QjBeHmQxEBDgqGBw1lIuuB Fb+IUhAUlNeMHLy1oCM4F/U3EZlCs/PjtwAh5S+Y0cLN24o8vJ2Q8kafFIYYqmXmWENH4dP+ sIFKKV1szXZr31dUj02Xyd9ieBY= X-Mailgun-Sending-Ip: 69.72.42.4 X-Mailgun-Sid: WyI0MWYwYSIsICJsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnIiwgImJlOWU0YSJd Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n06.prod.us-west-2.postgun.com with SMTP id 5f8a2897856d9308b5cdcb95 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Fri, 16 Oct 2020 23:11:19 GMT Sender: sudaraja=codeaurora.org@mg.codeaurora.org Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 88906C433FF; Fri, 16 Oct 2020 23:11:19 +0000 (UTC) Received: from mail.codeaurora.org (localhost.localdomain [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: sudaraja) by smtp.codeaurora.org (Postfix) with ESMTPSA id AC32CC433F1; Fri, 16 Oct 2020 23:11:18 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Fri, 16 Oct 2020 16:11:18 -0700 From: Sudarshan Rajagopalan To: Anshuman Khandual , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Suren Baghdasaryan , pratikp@codeaurora.org, Gavin Shan , Mark Rutland , Logan Gunthorpe , David Hildenbrand , Andrew Morton , Steven Price Subject: arm64: dropping prevent_bootmem_remove_notifier Message-ID: X-Sender: sudaraja@codeaurora.org User-Agent: Roundcube Webmail/1.3.9 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello Anshuman, In the patch that enables memory hot-remove (commit bbd6ec605c0f ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier put in place that prevents boot memory from being offlined and removed. Also commit text mentions that boot memory on arm64 cannot be removed. We wanted to understand more about the reasoning for this. X86 and other archs doesn’t seem to do this prevention. There’s also comment in the code that this notifier could be dropped in future if and when boot memory can be removed. The current logic is that only “new” memory blocks which are hot-added can later be offlined and removed. The memory that system booted up with cannot be offlined and removed. But there could be many usercases such as inter-VM memory sharing where a primary VM could offline and hot-remove a block/section of memory and lend it to secondary VM where it could hot-add it. And after usecase is done, the reverse happens where secondary VM hot-removes and gives it back to primary which can hot-add it back. In such cases, the present logic for arm64 doesn’t allow this hot-remove in primary to happen. Also, on systems with movable zone that sort of guarantees pages to be migrated and isolated so that blocks can be offlined, this logic also defeats the purpose of having a movable zone which system can rely on memory hot-plugging, which say virt-io mem also relies on for fully plugged memory blocks. I understand that some region of boot RAM shouldn’t be allowed to be removed, but such regions won’t be allowed to be offlined in first place since pages cannot be migrated and isolated, example reserved pages. So we’re trying to understand the reasoning for such a prevention put in place for arm64 arch alone. One possible way to solve this is by marking the required sections as “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. This puts these sections in the context of “memory hotpluggable” which can be offlined-removed and added-onlined which are part of boot RAM itself and doesn’t need any extra blocks to be hot added. This way of marking certain sections as “non-early” could be exported so that module drivers can set the required number of sections as “memory hotpluggable”. This could have certain checks put in place to see which sections are allowed, example only movable zone sections can be marked as “non-early”. Your thoughts on this? We are also looking for different ways to solve the problem without having to completely dropping this notifier, but just putting out the concern here about the notifier logic that is breaking our usecase which is a generic memory sharing usecase using memory hotplug feature. Sudarshan -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1334C43467 for ; Fri, 16 Oct 2020 23:13:33 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3B3E7212CC for ; Fri, 16 Oct 2020 23:13:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="x07vyEEj"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=mg.codeaurora.org header.i=@mg.codeaurora.org header.b="xYozm5EU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3B3E7212CC Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-ID:Subject:To:From:Date:MIME-Version: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=Hg0CzPrXpXNuPcQhJ8Lsmb7FPnWL8MbrdsLNdY6Wcvg=; b=x07vyEEjTBxVuWlCM8rBvyiHG1 JfpJJTsk+IGku4eFve5xC+OOgLlOSTBSjiuRU5/z3u23K3uo2yBJJ3hiM3B19lrLVqmJThhdV8oBG h1NKnc8YQDcdrFDTMLHQpsGIAs/enovkI+kzbBCo1Sf5xaCDQBRgyqK67fSMBaFdTyU8tTXY8pkG5 t9B31H0QjBPddp8+bYc1TjWXasEpmb8bpqkO4law1/oWYMEEDZegz1F2bSk5Tu2wEGSQpSte1aRM5 koN75dMPYBM0duau8AggayWYhSmnvX23jFF4aJDNyaUkZuAURUt4urbPf37ZG3uTPxskGD1YM79bq 2+HSARKA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kTYse-0007yf-SB; Fri, 16 Oct 2020 23:11:24 +0000 Received: from m42-4.mailgun.net ([69.72.42.4]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kTYsb-0007xu-UX for linux-arm-kernel@lists.infradead.org; Fri, 16 Oct 2020 23:11:23 +0000 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1602889880; h=Message-ID: Subject: Cc: To: From: Date: Content-Transfer-Encoding: Content-Type: MIME-Version: Sender; bh=KP659FIJpORGhMvLQ93vED7u9wN2xczwCJIEuJ6VKqM=; b=xYozm5EUsJDo4tJhU5Y2XVrfAMdNFm2e2VA56oUMBR75VZPF4aEZ9sbWRX5eMUelbFy2nbGN e6An7p/7hDqxv88RrJpDx/Ej13lGZQODJMsHE2Td5aekz3iYLbRA9DVtpaQH4cACVe/48PH5 D1D3yO3nu1RMZt2k2I2JAVuonb8= X-Mailgun-Sending-Ip: 69.72.42.4 X-Mailgun-Sid: WyJiYzAxZiIsICJsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmciLCAiYmU5ZTRhIl0= Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n01.prod.us-west-2.postgun.com with SMTP id 5f8a2897319d4e9cb53a5634 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Fri, 16 Oct 2020 23:11:19 GMT Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 7BBA8C433CB; Fri, 16 Oct 2020 23:11:19 +0000 (UTC) Received: from mail.codeaurora.org (localhost.localdomain [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: sudaraja) by smtp.codeaurora.org (Postfix) with ESMTPSA id AC32CC433F1; Fri, 16 Oct 2020 23:11:18 +0000 (UTC) MIME-Version: 1.0 Date: Fri, 16 Oct 2020 16:11:18 -0700 From: Sudarshan Rajagopalan To: Anshuman Khandual , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: arm64: dropping prevent_bootmem_remove_notifier Message-ID: X-Sender: sudaraja@codeaurora.org User-Agent: Roundcube Webmail/1.3.9 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201016_191122_116925_C3BB4AF4 X-CRM114-Status: GOOD ( 15.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Gavin Shan , David Hildenbrand , Logan Gunthorpe , Steven Price , Suren Baghdasaryan , Andrew Morton , pratikp@codeaurora.org Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CkhlbGxvIEFuc2h1bWFuLAoKSW4gdGhlIHBhdGNoIHRoYXQgZW5hYmxlcyBtZW1vcnkgaG90LXJl bW92ZSAoY29tbWl0IGJiZDZlYzYwNWMwZiAKKCJhcm02NC9tbTogRW5hYmxlIG1lbW9yeSBob3Qg cmVtb3ZlIikpIGZvciBhcm02NCwgdGhlcmXigJlzIGEgbm90aWZpZXIgCnB1dCBpbiBwbGFjZSB0 aGF0IHByZXZlbnRzIGJvb3QgbWVtb3J5IGZyb20gYmVpbmcgb2ZmbGluZWQgYW5kIHJlbW92ZWQu IApBbHNvIGNvbW1pdCB0ZXh0IG1lbnRpb25zIHRoYXQgYm9vdCBtZW1vcnkgb24gYXJtNjQgY2Fu bm90IGJlIHJlbW92ZWQuIApXZSB3YW50ZWQgdG8gdW5kZXJzdGFuZCBtb3JlIGFib3V0IHRoZSBy ZWFzb25pbmcgZm9yIHRoaXMuIFg4NiBhbmQgb3RoZXIgCmFyY2hzIGRvZXNu4oCZdCBzZWVtIHRv IGRvIHRoaXMgcHJldmVudGlvbi4gVGhlcmXigJlzIGFsc28gY29tbWVudCBpbiB0aGUgCmNvZGUg dGhhdCB0aGlzIG5vdGlmaWVyIGNvdWxkIGJlIGRyb3BwZWQgaW4gZnV0dXJlIGlmIGFuZCB3aGVu IGJvb3QgCm1lbW9yeSBjYW4gYmUgcmVtb3ZlZC4KClRoZSBjdXJyZW50IGxvZ2ljIGlzIHRoYXQg b25seSDigJxuZXfigJ0gbWVtb3J5IGJsb2NrcyB3aGljaCBhcmUgaG90LWFkZGVkIApjYW4gbGF0 ZXIgYmUgb2ZmbGluZWQgYW5kIHJlbW92ZWQuIFRoZSBtZW1vcnkgdGhhdCBzeXN0ZW0gYm9vdGVk IHVwIHdpdGggCmNhbm5vdCBiZSBvZmZsaW5lZCBhbmQgcmVtb3ZlZC4gQnV0IHRoZXJlIGNvdWxk IGJlIG1hbnkgdXNlcmNhc2VzIHN1Y2ggCmFzIGludGVyLVZNIG1lbW9yeSBzaGFyaW5nIHdoZXJl IGEgcHJpbWFyeSBWTSBjb3VsZCBvZmZsaW5lIGFuZCAKaG90LXJlbW92ZSBhIGJsb2NrL3NlY3Rp b24gb2YgbWVtb3J5IGFuZCBsZW5kIGl0IHRvIHNlY29uZGFyeSBWTSB3aGVyZSAKaXQgY291bGQg aG90LWFkZCBpdC4gQW5kIGFmdGVyIHVzZWNhc2UgaXMgZG9uZSwgdGhlIHJldmVyc2UgaGFwcGVu cyAKd2hlcmUgc2Vjb25kYXJ5IFZNIGhvdC1yZW1vdmVzIGFuZCBnaXZlcyBpdCBiYWNrIHRvIHBy aW1hcnkgd2hpY2ggY2FuIApob3QtYWRkIGl0IGJhY2suIEluIHN1Y2ggY2FzZXMsIHRoZSBwcmVz ZW50IGxvZ2ljIGZvciBhcm02NCBkb2VzbuKAmXQgCmFsbG93IHRoaXMgaG90LXJlbW92ZSBpbiBw cmltYXJ5IHRvIGhhcHBlbi4KCkFsc28sIG9uIHN5c3RlbXMgd2l0aCBtb3ZhYmxlIHpvbmUgdGhh dCBzb3J0IG9mIGd1YXJhbnRlZXMgcGFnZXMgdG8gYmUgCm1pZ3JhdGVkIGFuZCBpc29sYXRlZCBz byB0aGF0IGJsb2NrcyBjYW4gYmUgb2ZmbGluZWQsIHRoaXMgbG9naWMgYWxzbyAKZGVmZWF0cyB0 aGUgcHVycG9zZSBvZiBoYXZpbmcgYSBtb3ZhYmxlIHpvbmUgd2hpY2ggc3lzdGVtIGNhbiByZWx5 IG9uIAptZW1vcnkgaG90LXBsdWdnaW5nLCB3aGljaCBzYXkgdmlydC1pbyBtZW0gYWxzbyByZWxp ZXMgb24gZm9yIGZ1bGx5IApwbHVnZ2VkIG1lbW9yeSBibG9ja3MuCgpJIHVuZGVyc3RhbmQgdGhh dCBzb21lIHJlZ2lvbiBvZiBib290IFJBTSBzaG91bGRu4oCZdCBiZSBhbGxvd2VkIHRvIGJlIApy ZW1vdmVkLCBidXQgc3VjaCByZWdpb25zIHdvbuKAmXQgYmUgYWxsb3dlZCB0byBiZSBvZmZsaW5l ZCBpbiBmaXJzdCBwbGFjZSAKc2luY2UgcGFnZXMgY2Fubm90IGJlIG1pZ3JhdGVkIGFuZCBpc29s YXRlZCwgZXhhbXBsZSByZXNlcnZlZCBwYWdlcy4KClNvIHdl4oCZcmUgdHJ5aW5nIHRvIHVuZGVy c3RhbmQgdGhlIHJlYXNvbmluZyBmb3Igc3VjaCBhIHByZXZlbnRpb24gcHV0IGluIApwbGFjZSBm b3IgYXJtNjQgYXJjaCBhbG9uZS4KCk9uZSBwb3NzaWJsZSB3YXkgdG8gc29sdmUgdGhpcyBpcyBi eSBtYXJraW5nIHRoZSByZXF1aXJlZCBzZWN0aW9ucyBhcyAK4oCcbm9uLWVhcmx54oCdIGJ5IHJl bW92aW5nIHRoZSBTRUNUSU9OX0lTX0VBUkxZIGJpdCBpbiBpdHMgc2VjdGlvbl9tZW1fbWFwLiAK VGhpcyBwdXRzIHRoZXNlIHNlY3Rpb25zIGluIHRoZSBjb250ZXh0IG9mIOKAnG1lbW9yeSBob3Rw bHVnZ2FibGXigJ0gd2hpY2ggCmNhbiBiZSBvZmZsaW5lZC1yZW1vdmVkIGFuZCBhZGRlZC1vbmxp bmVkIHdoaWNoIGFyZSBwYXJ0IG9mIGJvb3QgUkFNIAppdHNlbGYgYW5kIGRvZXNu4oCZdCBuZWVk IGFueSBleHRyYSBibG9ja3MgdG8gYmUgaG90IGFkZGVkLiBUaGlzIHdheSBvZiAKbWFya2luZyBj ZXJ0YWluIHNlY3Rpb25zIGFzIOKAnG5vbi1lYXJseeKAnSBjb3VsZCBiZSBleHBvcnRlZCBzbyB0 aGF0IG1vZHVsZSAKZHJpdmVycyBjYW4gc2V0IHRoZSByZXF1aXJlZCBudW1iZXIgb2Ygc2VjdGlv bnMgYXMg4oCcbWVtb3J5IApob3RwbHVnZ2FibGXigJ0uIFRoaXMgY291bGQgaGF2ZSBjZXJ0YWlu IGNoZWNrcyBwdXQgaW4gcGxhY2UgdG8gc2VlIHdoaWNoIApzZWN0aW9ucyBhcmUgYWxsb3dlZCwg ZXhhbXBsZSBvbmx5IG1vdmFibGUgem9uZSBzZWN0aW9ucyBjYW4gYmUgbWFya2VkIAphcyDigJxu b24tZWFybHnigJ0uCgpZb3VyIHRob3VnaHRzIG9uIHRoaXM/IFdlIGFyZSBhbHNvIGxvb2tpbmcg Zm9yIGRpZmZlcmVudCB3YXlzIHRvIHNvbHZlIAp0aGUgcHJvYmxlbSB3aXRob3V0IGhhdmluZyB0 byBjb21wbGV0ZWx5IGRyb3BwaW5nIHRoaXMgbm90aWZpZXIsIGJ1dCAKanVzdCBwdXR0aW5nIG91 dCB0aGUgY29uY2VybiBoZXJlIGFib3V0IHRoZSBub3RpZmllciBsb2dpYyB0aGF0IGlzIApicmVh a2luZyBvdXIgdXNlY2FzZSB3aGljaCBpcyBhIGdlbmVyaWMgbWVtb3J5IHNoYXJpbmcgdXNlY2Fz ZSB1c2luZyAKbWVtb3J5IGhvdHBsdWcgZmVhdHVyZS4KCgpTdWRhcnNoYW4KCi0tClF1YWxjb21t IElubm92YXRpb24gQ2VudGVyLCBJbmMuIGlzIGEgbWVtYmVyIG9mIENvZGUgQXVyb3JhIEZvcnVt LCBhIApMaW51eCBGb3VuZGF0aW9uIENvbGxhYm9yYXRpdmUgUHJvamVjdAoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWls aW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0 cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=