From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48F0DC433DB for ; Fri, 29 Jan 2021 19:42:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A36B864E09 for ; Fri, 29 Jan 2021 19:42:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A36B864E09 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0D5C38D0002; Fri, 29 Jan 2021 14:42:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 086EA8D0001; Fri, 29 Jan 2021 14:42:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB7238D0002; Fri, 29 Jan 2021 14:42:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0079.hostedemail.com [216.40.44.79]) by kanga.kvack.org (Postfix) with ESMTP id D1E0A8D0001 for ; Fri, 29 Jan 2021 14:42:25 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8DECE363E for ; Fri, 29 Jan 2021 19:42:25 +0000 (UTC) X-FDA: 77759834250.16.juice43_3d181bf275aa Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 66411100E6903 for ; Fri, 29 Jan 2021 19:42:25 +0000 (UTC) X-HE-Tag: juice43_3d181bf275aa X-Filterd-Recvd-Size: 6017 Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Fri, 29 Jan 2021 19:42:24 +0000 (UTC) Received: by mail-ed1-f51.google.com with SMTP id d22so11902816edy.1 for ; Fri, 29 Jan 2021 11:42:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=bXc3CLlhoRRbiZlIKLzbmyPtbePCc73LLUj7gRG/Gys=; b=SNGgJMg5nX+wA1SntWFxktHPHK4RtZGL23wpCqPSjmBcHfgmUwzuWrrVZW7LRHnd0k lCed0sM96lX2IhRnEmCGbjxur17Hqoe2I2xNP5IPBrGNpkjDjoK6Q0uLr1Hx1Pc4pFpQ 9goatTdy/THUEDR8UcEPB/LDIdTJIQfQfjFB3ekNKKDN63BaPucilhVdE7MmxL46VHen Vs1AYsNt9FYJ0xyg2pElnkBcPM4dNYXOd9Nu87D4BdgjBAkD95/hBonABV4GnETiGxOo bPEsV/ZM9sJJTKTULvrPb594Ov6Rz2oggBzXAOD3cTzz2S9GJ4m1nZK4nU4GKINPum2K tVbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=bXc3CLlhoRRbiZlIKLzbmyPtbePCc73LLUj7gRG/Gys=; b=oaLukXlbGZ/qoNEX1WXcIFhCWj+u3WIToEKs7O89GEn4XLxRdrSFQxax/mxmrbAoP7 0uYPI56DUEsux3WZKPs/RioLcUTSJ7z1+rmYSpCY1sgz5TfYzdeXMU7dwZj8xgH2Gbwj QX9umiAsK/1bpNfY4FzUggHz7kBms4CDc8MoL+G5OLHl78KEgUap14WreoYfNpxhy1f4 H8wEiMs8qQNkis5Y89iUXUO1+lFZj08AJ0XbWOShM9dbqgJX26wqdGdotod/zJmzsskv zuM5cA/gvuVBwP5owNckDI8KPmvjdtniybEPcpxgsS4qCIBbHidEAtqXA29y4MdRRcg6 dOoQ== X-Gm-Message-State: AOAM530TDGblodHi5lv3cJ2daXmsAz3328+xs8geOdT04Cu6jYdX2mL3 yJ8dGWiBW4JxIKw9EIv4na3N+L9A5cDEPed9bex+ww== X-Google-Smtp-Source: ABdhPJx9WMbW0fOUtS/53VzqRSYRLFCLP3Qc1vMEHgKyHW5AEH76aAxkdjlvjI8EYGJ8I70iRrUpnYFg2qn/8imPHGA= X-Received: by 2002:a05:6402:402:: with SMTP id q2mr7098751edv.116.1611949343718; Fri, 29 Jan 2021 11:42:23 -0800 (PST) MIME-Version: 1.0 References: <8c2b75fe-a3e5-8eff-7f37-5d23c7ad9742@redhat.com> <94797c92-cd90-8a65-b879-0bb5f12b9fc5@redhat.com> <92912784-f3a3-b5a5-2d45-4c86ae26315f@redhat.com> In-Reply-To: From: Pavel Tatashin Date: Fri, 29 Jan 2021 14:41:47 -0500 Message-ID: Subject: Re: dax alignment problem on arm64 (and other achitectures) To: David Hildenbrand Cc: Anshuman Khandual , linux-mm , LKML , Sasha Levin , Tyler Hicks , Andrew Morton , Dan Williams , Michal Hocko , Oscar Salvador , Vlastimil Babka , Joonsoo Kim , Jason Gunthorpe , Marc Zyngier , Linux ARM , Will Deacon , James Morse , James Morris Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jan 29, 2021 at 2:12 PM Pavel Tatashin wrote: > > On Fri, Jan 29, 2021 at 2:06 PM Pavel Tatashin > wrote: > > > > > > Definitely, but we should try figuring out what's going on here. I > > > > assume on x86-64 it behaves differently? > > > > > > Yes, we should root cause. I highly suspect that there is somewhere > > > alignment miscalculations happen that cause this memory waste with the > > > offset 16M. I am also not sure why the 2M label size was increased, > > > and why 16M is now an alignment requirement. > > > > This appears to be because even if we set vmemmap to be outside of the > > dax device, the alignment calculates the maximum size of vmemmap for > > this device, and subtracts it from the devdax size. > > See [1], line 795 is where this offset is calculated. > > > > This also explains why with 64K pages, the 16M offset worked: because > > fewer struct pages were able to fit within 16M - label size. > > > > [1] https://soleen.com/source/xref/linux/drivers/nvdimm/pfn_devs.c?r=b7b3c01b&mo=18459&fi=718#795 > > Actually, strike the previous e-mail. The extra space is when we > reserve vmemmap from devdax. IFF we do it from mem, the extra space is > not added. Now, this alignment makes total sense. commit 2522afb86a8cceba0f67dbf05772d21b76d79f06 Author: Dan Williams Date: Thu Jan 30 12:06:23 2020 -0800 libnvdimm/region: Introduce an 'align' attribute This is the patch that introduced the 16M alignment. /* * PowerPC requires this alignment for memremap_pages(). All other archs * should be ok with SUBSECTION_SIZE (see memremap_compat_align()). */ #define MEMREMAP_COMPAT_ALIGN_MAX SZ_16M static unsigned long default_align(struct nd_region *nd_region) { unsigned long align; int i, mappings; u32 remainder; if (is_nd_blk(&nd_region->dev)) align = PAGE_SIZE; else align = MEMREMAP_COMPAT_ALIGN_MAX; Dan, is this logic correct? Why is_nd_pmem() cannot be set to SUBSECTION_SIZE alignment? Thank you, Pasha > > Pasha