From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D1B8C433E0 for ; Wed, 27 Jan 2021 20:43:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF15F64DBD for ; Wed, 27 Jan 2021 20:43:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF15F64DBD Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 12BDD6B0005; Wed, 27 Jan 2021 15:43:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B58C6B0006; Wed, 27 Jan 2021 15:43:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE6E26B006E; Wed, 27 Jan 2021 15:43:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0064.hostedemail.com [216.40.44.64]) by kanga.kvack.org (Postfix) with ESMTP id D34946B0005 for ; Wed, 27 Jan 2021 15:43:56 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 97995363E for ; Wed, 27 Jan 2021 20:43:56 +0000 (UTC) X-FDA: 77752731672.05.car88_2803dbc2759a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 78FB81802EF11 for ; Wed, 27 Jan 2021 20:43:56 +0000 (UTC) X-HE-Tag: car88_2803dbc2759a X-Filterd-Recvd-Size: 5839 Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 20:43:55 +0000 (UTC) Received: by mail-ed1-f48.google.com with SMTP id c6so4129972ede.0 for ; Wed, 27 Jan 2021 12:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=8YqfyvXUUcZWe9IQfEwv1uN3EftJilGP7wEEowSADhg=; b=dYQh6vgGP4ZJ/OG6qVH+8jnnFmvM+HetBo7y4oFBFJuBZ/tK/QE0cZ8Evi0PgVCq9Y HQgk+0jYm4I6a8jngkSyzZOHPGw5WqsQ0T5UNaoCe0v1iSW3kK3PMWUUqMozL8lT2ZUV SvVyCyvx3MvBUUntT3HMaPa4tBbct+NKlD6vJ0qjGRldHPphtF2jpyloIPvHuyjAlRZs wHI9AiC4Q7Yvag5yQNjceqjWre1NZYS0ZUprZDgu/UCpgcJPvONx/kisXAnFpZpIbmpu 3lK0/BzInBaL98xzZJsrYh/ni6qTof4ss9VC45Xuk8pcPh9xKMhURqNVwkNq7BJRAEoh UDPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=8YqfyvXUUcZWe9IQfEwv1uN3EftJilGP7wEEowSADhg=; b=GGiZpO86CX+r7vVCBBPAGwZouRs4slvVeyzE9RbgnNU5mAR6Y+N+dB2XBhdQg07Efx C9AOrJRPcd2rNXPl1PwrglzJi6Yvc0R8TRDhBoqHmhGB2466Kqz5hktqa54V9FvCzqdK okJQKdirv3vzWYVWjNA16z7Wf1mRXgRij2OEXqllW5Kr3cQwwY1ooxdom0Ya5KkISkQ3 jgVxhK3OMKXjEzkm6uNWl4eZH9Ea4cBsST9Fcngz9huOSHd7U3oLs3MQCy2vGtIm5Gh5 I7VnF9TDlSZTIZ2kRw+wi2h/d1yQSRXhXiXDkyegA6wS3OrmV8P7omSOc+nth9x6D/sk 6MEw== X-Gm-Message-State: AOAM530uODf1sV3h8eN4dIu55UsPtP0i228ZA1ic+9pIaw5TGCxL/7bY KBzt3x/vZIbUCV6/vucH6Y8F29V7VabEVpjsNK8yIBVBENJ61q3k X-Google-Smtp-Source: ABdhPJxc8pyTwJ+yBOYx0lG6ZcKs+mHhMLS5/NAJbmCaRdz5fb7UOXiPY8lELRdOFNUFD/g12SvV63SAL5sCeXP7dck= X-Received: by 2002:aa7:d987:: with SMTP id u7mr5937436eds.62.1611780234243; Wed, 27 Jan 2021 12:43:54 -0800 (PST) MIME-Version: 1.0 From: Pavel Tatashin Date: Wed, 27 Jan 2021 15:43:18 -0500 Message-ID: Subject: dax alignment problem on arm64 (and other achitectures) To: linux-mm , LKML , Sasha Levin , Tyler Hicks , Andrew Morton , Dan Williams , David Hildenbrand , Michal Hocko , Oscar Salvador , Vlastimil Babka , Joonsoo Kim , Jason Gunthorpe , Marc Zyngier , Linux ARM , Will Deacon , James Morse , James Morris Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is something that Dan Williams and I discussed off the mailing list sometime ago, but I want to have a broader discussion about this problem so I could send out a fix that would be acceptable. We have a 2G pmem device that is carved out of regular memory that we use to pass data across reboots. After the machine is rebooted we hotplug that memory back, so we do not lose 2G of system memory (machine is small, only 8G of RAM total). In order to hotplug pmem memory it first must be converted to devdax. Devdax has a label 2M in size that is placed at the beginning of the pmem device memory which brings the problem. The section size is a hotplugging unit on Linux. Whatever gets hot-plugged or hot-removed must be section size aligned. On x86 section size is 128M on arm64 it is 1G (because arm64 supports 64K pages, and 128M does not work with 64K pages). Because the first 2M are subtracted from the pmem device to create devdax, that actual hot-pluggable memory is not 1G/128M aligned, and instead we lose 126M on x86 or 1022M on arm64 of memory that is getting hot-plugged, the whole first section is skipped when memory gets hot plugged because of 2M label. As a workaround, so we do not lose 1022M out of 8G of memory on arm64 we have section size reduced to 128M. We are using this patch [1]. This way we are losing 126M (which I still hate!) I would like to get rid of this workaround. First, because I would like us to switch to 64K pages to gain performance, and second so we do not depend on an unofficial patch which already has given us some headache with kdump support. Here are some solutions that I think we can do: 1. Instead of carving the memory at 1G aligned address, do it at 1G - 2M address, this way when devdax is created it is perfectly 1G aligned. On ARM64 it causes a panic because there is a 2M hole in memory. Even if panic is fixed, I do not think this is a proper fix. This is simply a workaround to the underlying problem. 2. Dan Williams introduced subsections [2]. They, however do not work with devdax, and hot-plugging in general. Those patches take care of __add_pages() side of things, and not add_memory(). Also, it is unclear what kind of user interface changes need to be made in order to enable subsection features to online/offline pages. 3. Allow to hot plug daxdev together with the label, but teach the kernel not to touch label (i.e. allocate its memory). IMO, kind of ugly solution, because when devdax is hot-plugged it is not even aware of label size. But, perhaps that can be changed. 4. Other ideas? (move dax label to the end? a special case without a label? label outside of data?) Thank you, Pasha [1] https://lore.kernel.org/lkml/20190423203843.2898-1-pasha.tatashin@soleen.com [2] https://lore.kernel.org/lkml/156092349300.979959.17603710711957735135.stgit@dwillia2-desk3.amr.corp.intel.com