From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2F91C433E0 for ; Fri, 29 Jan 2021 02:07:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 192C064DFB for ; Fri, 29 Jan 2021 02:07:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 192C064DFB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1707F6B0005; Thu, 28 Jan 2021 21:07:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 121246B0006; Thu, 28 Jan 2021 21:07:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0601D6B006C; Thu, 28 Jan 2021 21:07:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id E27156B0005 for ; Thu, 28 Jan 2021 21:07:01 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id ABE018249980 for ; Fri, 29 Jan 2021 02:07:01 +0000 (UTC) X-FDA: 77757174642.30.ducks32_3c17483275a4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 8590C180B3C8B for ; Fri, 29 Jan 2021 02:07:01 +0000 (UTC) X-HE-Tag: ducks32_3c17483275a4 X-Filterd-Recvd-Size: 6846 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Fri, 29 Jan 2021 02:07:00 +0000 (UTC) Received: by mail-ej1-f46.google.com with SMTP id kg20so10750708ejc.4 for ; Thu, 28 Jan 2021 18:07:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yDfksjTQWFj58x+SISTgpJag83AD3GcsJWGYuG1gaks=; b=WY2HzNjVYTlxpd3xGcsSp7nCBA/CEoErKuKCMSrlAWONlNwsl8YqB23Viw2KNecRgM YlKnEncBOIo3jOWRNxj2ahMnnksKcw+xEGiLbZUK8Td6N0ffwvScsOr2NDEJID0Xbk/d wX8F5NXIfOGaNz8y4ZHE7paB3RL2C5CuMExfoTZkBO3/0eBncnmUnDYADw3nWvdusSgW DdPlCLJd8chRzzaV1ThYuGKJ+XNfc3oZjQv1Jnnjiz7wCW6Y8Y1Az8Cz09BF/ZZYeI9j pF5edhpo459x/7qA6HLVoHFgun1Td8khHwH37DzRKiwzJFJer7Eloi0lA2f14dxJy3ID 6t5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yDfksjTQWFj58x+SISTgpJag83AD3GcsJWGYuG1gaks=; b=Blj1Fk1G37iAEnhZUT1qHytYmzMteXDkd92/xGrGkPMc1XO1pbpexZJ0Z9tE9FCitv +VhKoOoqpqhWSzWZMTTx9+q3Ueysc/QvE27IeSQ6nHJaSrpbjLEZIfnXjGVLMcrDeU79 pJcIFIhBLq5vAFXFZF5xXZKudojPga4t8n9hs9l3OI5RZAuHFPVH5hi2jwulEew3VbA3 8vdSPsx2aySUieD8S65h3QKwCMtdc1PU7gExNtF6rgonRNtsDDb893aUYIRCluJl2Z10 XpkNVs2ghb8IuF/fk/6xjCKT3jYt2eqgEEbUJAN2go1YgyBhDM1MGh9Cz2PiRv+Egv31 95Pg== X-Gm-Message-State: AOAM532pZGRTF+LmGABptricuUZrmLZfPNhON/+E3LAuat858D97onRr pEhx1uMp6FPYjcho1PnimGyGJKwNXlq08Aft/ie+ZA== X-Google-Smtp-Source: ABdhPJz5bNtZVcWPhnt9aV3bgD/xY7CMXjgvCMjaX/cDqiqhnAdXwZ5ttI04IzKW0v+FIrzR1NqNmcRxBOY5N+eWPXQ= X-Received: by 2002:a17:906:c5b:: with SMTP id t27mr2400380ejf.129.1611886019683; Thu, 28 Jan 2021 18:06:59 -0800 (PST) MIME-Version: 1.0 References: <8c2b75fe-a3e5-8eff-7f37-5d23c7ad9742@redhat.com> <94797c92-cd90-8a65-b879-0bb5f12b9fc5@redhat.com> In-Reply-To: From: Pavel Tatashin Date: Thu, 28 Jan 2021 21:06:23 -0500 Message-ID: Subject: Re: dax alignment problem on arm64 (and other achitectures) To: David Hildenbrand Cc: linux-mm , LKML , Sasha Levin , Tyler Hicks , Andrew Morton , Dan Williams , Michal Hocko , Oscar Salvador , Vlastimil Babka , Joonsoo Kim , Jason Gunthorpe , Marc Zyngier , Linux ARM , Will Deacon , James Morse , James Morris Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > > Might be related to the broken custom pfn_valid() implementation for > > ZONE_DEVICE. > > > > https://lkml.kernel.org/r/1608621144-4001-1-git-send-email-anshuman.khandual@arm.com > > > > And essentially ignoring sub-section data in there for now as well (but > > might not be that relevant yet). In addition, this might also be related to > > > > https://lkml.kernel.org/r/161058499000.1840162.702316708443239771.stgit@dwillia2-desk3.amr.corp.intel.com > > I will check it, and see what I find. I saw that panic almost a year > ago, things might have changed since then. Hi David, There is no panic anymore, but I also can't offset by 2M anymore, the minimum that works now is 16M, and if alignment is less than 16M creating devdax device fails. So, I tried the new ARM64 patch that reduces section sizes, and two alignments for pmem: regular 2G alignment, and 2G+16M alignment. (subtracted 16M from the bottom) ***** 4K page, 6G RAM, 2G PRAM ***** BOOT: 40000000-1bfffffff : System RAM 1c0000000-23fffffff : namespace0.0 DEVDAX: 40000000-1bfffffff : System RAM 1c0000000-1c21fffff : namespace0.0 1c2200000-23fffffff : dax0.0 HOTPLUG: 40000000-1bfffffff : System RAM 1c0000000-1c21fffff : namespace0.0 1c8000000-23fffffff : dax0.0 1c8000000-23fffffff : System RAM (kmem) 128M Wasted (Expected) ***** 4K page, 6G-16M RAM, 2G+16M PRAM ***** BOOT: 40000000-1beffffff : System RAM 1bf000000-23fffffff : namespace0.0 DEVDAX: 40000000-1beffffff : System RAM 1bf000000-1c11fffff : namespace0.0 1c1200000-23fffffff : dax0.0 HOTPLUG: 40000000-1beffffff : System RAM 1bf000000-1c11fffff : namespace0.0 1c8000000-23fffffff : dax0.0 1c8000000-23fffffff : System RAM (kmem) 144M Wasted (????) ***** 64K page, 6G RAM, 2G PRAM ***** BOOT: 40000000-1bfffffff : System RAM 1c0000000-23fffffff : namespace0.0 DEVDAX: 40000000-1bfffffff : System RAM 1c0000000-1dfffffff : namespace0.0 1e0000000-23fffffff : dax0.0 HOTPLUG: 40000000-1bfffffff : System RAM 1c0000000-1dfffffff : namespace0.0 1e0000000-23fffffff : dax0.0 1e0000000-23fffffff : System RAM (kmem) 512M Wasted (Expected) ***** 64K page, 6G-16M RAM, 2G+16M PRAM ***** BOOT: 40000000-1beffffff : System RAM 1bf000000-23fffffff : namespace0.0 DEVDAX: 40000000-1beffffff : System RAM 1bf000000-1bf3fffff : namespace0.0 1bf400000-23fffffff : dax0.0 HOTPLUG: 40000000-1beffffff : System RAM 1bf000000-1bf3fffff : namespace0.0 1c0000000-23fffffff : dax0.0 1c0000000-23fffffff : System RAM (kmem) 16M Wasted (Optimal) In all three cases only System RAM, namespace0.0, and dax0.0 were printed from /proc/iomem. BOOT content of iomem right after boot DEVDAX content of iomem after devdax is created ndctl create-namespace --mode devdax -e namespace0.0" HOTPLUG content of imem after dax0.0 is hotplugged: echo dax0.0 > /sys/bus/dax/drivers/device_dax/unbind echo dax0.0 > /sys/bus/dax/drivers/kmem/new_id The most surprising part is why with 4K pages and 16M offset 144M is wasted? For whatever reason, when devdax is created 34 goes wasted to the label? Something is wrong here.. However, I am happy with 64K pages result, and that only 16M is wasted, of course optimally, we should be using any memory here, but it is still much better than what we have now. Pasha