From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3FEEC433E9 for ; Tue, 23 Feb 2021 10:27:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 29EA764E4B for ; Tue, 23 Feb 2021 10:27:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 29EA764E4B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4D3CF6B0005; Tue, 23 Feb 2021 05:27:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4816C6B0006; Tue, 23 Feb 2021 05:27:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3219B6B006E; Tue, 23 Feb 2021 05:27:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id 1D8DE6B0005 for ; Tue, 23 Feb 2021 05:27:49 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id DE1DA18332C51 for ; Tue, 23 Feb 2021 10:27:48 +0000 (UTC) X-FDA: 77849156616.05.FD898B3 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf09.hostedemail.com (Postfix) with ESMTP id 5290860024A2 for ; Tue, 23 Feb 2021 10:27:42 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id t9so1498668pjl.5 for ; Tue, 23 Feb 2021 02:27:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=PsAlBEioJWPlE/SfmIRHP8CqLqTdFtcrzccsM5cB+R0=; b=Ys/7GJocBf6sJ4nFA3vG4a8OX6mw+NdBaUqZ12DWzrV3fZccA3R4QYfl3pLfZPTFTc 36b47Oq/dFpsrhWFkqM1lnReccu6RUOyMP8EQSEA4RQW7hHWsVs5XExtJw4q2djMEBP7 oH2lkscq41McTnJskdNFbT3LcWlovKmlUnG55WZo1nX9InAPpRIRDewHJITEbGXxS+W8 QJ1Jl0LrGvEf4yht4eLQKVZooAiGg0HSpZJsQotql2FReOt2Ab1bY37Tuyt7x/pDPBnO P2XYO5w7gVCunLge1+dYnKJWht7vv4xAJLZiy6xSQc7dWTrcBw6cMAhH7q2R/deehLib 5i0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PsAlBEioJWPlE/SfmIRHP8CqLqTdFtcrzccsM5cB+R0=; b=ni4vdfx8s4fwMOXpLXt0mVH9ZQCrkMChLiUPNcMuK17Gr1yEORyS4gxS6P7IdyJf+1 Ij7gJrwdnKeM3Uwk20eYzxwT7GSgaWDVRQDvc50esJaaJqi04rYfoVBwZrBJlem79sOd hcEawaqkTwBNCuoPAxkxFGiEiYFJw9asN42IC6ucXJx3vGdKmQB6u4rKjuh4S+zOx4HL LUjzuY2ZE/AGyLWo0Ys0Xh7G1QNUR8SVyUX9QLoF2jNze4/vJUrGQ5bMUx7N7jPcaXb2 s7fQDyzXbQSfE0eQFFshR/PjEpYilQSoSXShXHucIOo17vnbcNfUXW3FOjRAUsDbIXYB cP1A== X-Gm-Message-State: AOAM533RntEYNV8rA0Zl+PkuSMuiAuk6jpqDm4V4/Z/5Vebvtce4AMEK 3gciqjvSkZx2PrmhXOOmriMHv7FPIFa03oJ3EjI97A== X-Google-Smtp-Source: ABdhPJzoTOm14qBpXTrTOy4tiMVrs/RUpxAE2Rzm0OLejTjzGP7Zp5z6bsLhSG9DlKlA1aw3h9BenJT+PO3lsTYt9hQ= X-Received: by 2002:a17:90a:f18d:: with SMTP id bv13mr7273299pjb.13.1614076064700; Tue, 23 Feb 2021 02:27:44 -0800 (PST) MIME-Version: 1.0 References: <20210219104954.67390-1-songmuchun@bytedance.com> <20210219104954.67390-5-songmuchun@bytedance.com> <13a5363c-6af4-1e1f-9a18-972ca18278b5@oracle.com> <20210223092740.GA1998@linux> In-Reply-To: <20210223092740.GA1998@linux> From: Muchun Song Date: Tue, 23 Feb 2021 18:27:07 +0800 Message-ID: Subject: Re: [External] Re: [PATCH v16 4/9] mm: hugetlb: alloc the vmemmap pages associated with each HugeTLB page To: Oscar Salvador Cc: Mike Kravetz , Jonathan Corbet , Thomas Gleixner , mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, Peter Zijlstra , viro@zeniv.linux.org.uk, Andrew Morton , paulmck@kernel.org, mchehab+huawei@kernel.org, pawan.kumar.gupta@linux.intel.com, Randy Dunlap , oneukum@suse.com, anshuman.khandual@arm.com, jroedel@suse.de, Mina Almasry , David Rientjes , Matthew Wilcox , Michal Hocko , "Song Bao Hua (Barry Song)" , David Hildenbrand , =?UTF-8?B?SE9SSUdVQ0hJIE5BT1lBKOWggOWPoyDnm7TkuZ8p?= , Joao Martins , Xiongchun duan , linux-doc@vger.kernel.org, LKML , Linux Memory Management List , linux-fsdevel Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: phf75g7shsh5mzsmny7fqi3du4o39d6k X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5290860024A2 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mail-pj1-f47.google.com; client-ip=209.85.216.47 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614076062-961346 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 23, 2021 at 5:28 PM Oscar Salvador wrote: > > On Mon, Feb 22, 2021 at 04:00:27PM -0800, Mike Kravetz wrote: > > > -static void update_and_free_page(struct hstate *h, struct page *page) > > > +static int update_and_free_page(struct hstate *h, struct page *page) > > > + __releases(&hugetlb_lock) __acquires(&hugetlb_lock) > > > { > > > int i; > > > + int nid = page_to_nid(page); > > > > > > if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) > > > - return; > > > + return 0; > > > > > > h->nr_huge_pages--; > > > - h->nr_huge_pages_node[page_to_nid(page)]--; > > > + h->nr_huge_pages_node[nid]--; > > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); > > > + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); > > > + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); > > > + set_page_refcounted(page); > > > > I think you added the set_page_refcounted() because the huge page will > > appear as just a compound page without a reference after dropping the > > hugetlb lock? It might be better to set the reference before modifying > > the destructor. Otherwise, page scanning code could find the non-hugetlb > > compound page with no reference. I could not find any code where this > > would be a problem, but I think it would be safer to set the reference > > first. > > But we already had set_page_refcounted() before this patchset there. > Are the worries only because we drop the lock? AFAICS, the "page-scanning" > problem could have happened before as well? > Although, what does page scanning mean in this context? > > I am not opposed to move it above, but I would like to understand the concern > here. > > > > > > + spin_unlock(&hugetlb_lock); > > > > I really like the way this code is structured. It is much simpler than > > previous versions with retries or workqueue. There is nothing wrong with > > always dropping the lock here. However, I wonder if we should think about > > optimizing for the case where this feature is not enabled and we are not > > freeing a 1G huge page. I suspect this will be the most common case for > > some time, and there is no need to drop the lock in this case. > > > > Please do not change the code based on my comment. I just wanted to bring > > this up for thought. > > > > Is it as simple as checking? > > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > > spin_unlock(&hugetlb_lock); > > > > /* before return */ > > if (free_vmemmap_pages_per_hpage(h) || hstate_is_gigantic(h)) > > spin_lock(&hugetlb_lock); > > AFAIK, we at least need the hstate_is_gigantic? Comment below says that > free_gigantic_page might block, so we need to drop the lock. > And I am fine with the change overall. > > Unless I am missing something, we should not need to drop the lock unless > we need to allocate vmemmap pages (apart from gigantic pages). > > > > > > + > > > + if (alloc_huge_page_vmemmap(h, page)) { > > > + int zeroed; > > > + > > > + spin_lock(&hugetlb_lock); > > > + INIT_LIST_HEAD(&page->lru); > > > + set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); > > > + h->nr_huge_pages++; > > > + h->nr_huge_pages_node[nid]++; > > I think prep_new_huge_page() does this for us? Actually, there are some differences. e.g. prep_new_huge_page() will reset hugetlb cgroup and ClearHPageFreed, but we do not need them here. And prep_new_huge_page will acquire and release the hugetlb_lock. But here we also need hold the lock to update the surplus counter and enqueue the page to the free list. So I do not think reuse prep_new_huge_page is a good idea. > > > > + > > > + /* > > > + * If we cannot allocate vmemmap pages, just refuse to free the > > > + * page and put the page back on the hugetlb free list and treat > > > + * as a surplus page. > > > + */ > > > + h->surplus_huge_pages++; > > > + h->surplus_huge_pages_node[nid]++; > > > + > > > + /* > > > + * This page is now managed by the hugetlb allocator and has > > > + * no users -- drop the last reference. > > > + */ > > > + zeroed = put_page_testzero(page); > > > + VM_BUG_ON_PAGE(!zeroed, page); > > Can this actually happen? AFAIK, page landed in update_and_free_page should be > zero refcounted, then we increase the reference, and I cannot see how the > reference might have changed in the meantime. I am not sure whether other modules get the page and then put the page. I see gather_surplus_pages does the same thing. So I copied from there. I try to look at the memory_failure routine. CPU0: CPU1: set_compound_page_dtor(HUGETLB_PAGE_DTOR); memory_failure_hugetlb get_hwpoison_page __get_hwpoison_page get_page_unless_zero put_page_testzero() Maybe this can happen. But it is a very corner case. If we want to deal with this. We can put_page_testzero() first and then set_compound_page_dtor(HUGETLB_PAGE_DTOR). > > I am all for catching corner cases, but not sure how realistic this is. > Moreover, if we __ever__ get there, things can get nasty. > > We basically will have an in-use page in the free hugetlb pool, so corruption > will happen. At that point, a plain BUG_ON might be better. > > But as I said, I do not think we need that. > > I yet need to look further, but what I have seen so far looks good. > > -- > Oscar Salvador > SUSE L3