Linux-mm Archive on lore.kernel.org
 help / color / Atom feed
From: Dan Williams <dan.j.williams@intel.com>
To: David Hildenbrand <david@redhat.com>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Linux MM <linux-mm@kvack.org>,  Michal Hocko <mhocko@kernel.org>,
	Michal Hocko <mhocko@suse.com>,
	 Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	 Andrew Morton <akpm@linux-foundation.org>,
	powerpc-utils-devel@googlegroups.com,
	 util-linux@vger.kernel.org,
	Badari Pulavarty <pbadari@us.ibm.com>,
	 Nathan Fontenot <nfont@linux.vnet.ibm.com>,
	Robert Jennings <rcj@linux.vnet.ibm.com>,
	 Heiko Carstens <heiko.carstens@de.ibm.com>,
	Karel Zak <kzak@redhat.com>,
	 "Scargall, Steve" <steve.scargall@intel.com>
Subject: Re: [PATCH v1] drivers/base/memory.c: indicate all memory blocks as removable
Date: Thu, 26 Mar 2020 23:24:08 -0700
Message-ID: <CAPcyv4jpdaNvJ67SkjyUJLBnBnXXQv686BiVW042g03FUmWLXw@mail.gmail.com> (raw)
In-Reply-To: <20200128093542.6908-1-david@redhat.com>

On Tue, Jan 28, 2020 at 1:44 AM David Hildenbrand <david@redhat.com> wrote:
>
> We see multiple issues with the implementation/interface to compute
> whether a memory block can be offlined (exposed via
> /sys/devices/system/memory/memoryX/removable) and would like to simplify
> it (remove the implementation).
>
> 1. It runs basically lockless. While this might be good for performance,
>    we see possible races with memory offlining that will require at least
>    some sort of locking to fix.
>
> 2. Nowadays, more false positives are possible. No arch-specific checks
>    are performed that validate if memory offlining will not be denied
>    right away (and such check will require locking). For example, arm64
>    won't allow to offline any memory block that was added during boot -
>    which will imply a very high error rate. Other archs have other
>    constraints.
>
> 3. The interface is inherently racy. E.g., if a memory block is
>    detected to be removable (and was not a false positive at that time),
>    there is still no guarantee that offlining will actually succeed. So
>    any caller already has to deal with false positives.
>
> 4. It is unclear which performance benefit this interface actually
>    provides. The introducing commit 5c755e9fd813 ("memory-hotplug: add
>    sysfs removable attribute for hotplug memory remove") mentioned
>         "A user-level agent must be able to identify which sections of
>          memory are likely to be removable before attempting the
>          potentially expensive operation."
>    However, no actual performance comparison was included.
>
> Known users:
> - lsmem: Will group memory blocks based on the "removable" property. [1]
> - chmem: Indirect user. It has a RANGE mode where one can specify
>          removable ranges identified via lsmem to be offlined. However, it
>          also has a "SIZE" mode, which allows a sysadmin to skip the manual
>          "identify removable blocks" step. [2]
> - powerpc-utils: Uses the "removable" attribute to skip some memory
>                  blocks right away when trying to find some to
>                  offline+remove. However, with ballooning enabled, it
>                  already skips this information completely (because it
>                  once resulted in many false negatives). Therefore, the
>                  implementation can deal with false positives properly
>                  already. [3]
>
> According to Nathan Fontenot, DLPAR on powerpc is nowadays no longer
> driven from userspace via the drmgr command (powerpc-utils). Nowadays
> it's managed in the kernel - including onlining/offlining of memory
> blocks - triggered by drmgr writing to /sys/kernel/dlpar. So the
> affected legacy userspace handling is only active on old kernels. Only very
> old versions of drmgr on a new kernel (unlikely) might execute slower -
> totally acceptable.
>
> With CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not
> break any user space tool. We implement a very bad heuristic now.  Without
> CONFIG_MEMORY_HOTREMOVE we cannot offline anything, so report
> "not removable" as before.
>
> Original discussion can be found in [4] ("[PATCH RFC v1] mm:
> is_mem_section_removable() overhaul").
>
> Other users of is_mem_section_removable() will be removed next, so that
> we can remove is_mem_section_removable() completely.
>
> [1] http://man7.org/linux/man-pages/man1/lsmem.1.html
> [2] http://man7.org/linux/man-pages/man8/chmem.8.html
> [3] https://github.com/ibm-power-utilities/powerpc-utils
> [4] https://lkml.kernel.org/r/20200117105759.27905-1-david@redhat.com
>
> Suggested-by: Michal Hocko <mhocko@kernel.org>
> Acked-by: Michal Hocko <mhocko@suse.com>
> Cc: Dan Williams <dan.j.williams@intel.com>

David, Andrew,

I'd like to recommend this patch for -stable as it likely (test
underway) solves this crash report from Steve:

[  148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p))
[  148.796074] ------------[ cut here ]------------
[  148.796098] kernel BUG at include/linux/mm.h:1087!
[  148.796126] invalid opcode: 0000 [#1] SMP NOPTI
[  148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8=
6_64+debug #1
[  148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5=
C620.86B.02.01.0010.010620200716 01/06/2020
[  148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0
[  148.796561] Call Trace:
[  148.796591]  removable_show+0x6e/0xa0
[  148.796608]  dev_attr_show+0x19/0x40
[  148.796625]  sysfs_kf_seq_show+0xa9/0x100
[  148.796640]  seq_read+0xd5/0x450
[  148.796657]  vfs_read+0xc5/0x180
[  148.796672]  ksys_read+0x68/0xe0
[  148.796688]  do_syscall_64+0x5c/0xa0
[  148.796704]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  148.796721] RIP: 0033:0x7f3ab1646412

...on a non-debug kernel it just crashes.

In this case lsmem is failing when reading memory96:

openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC) = 4
fcntl(4, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
read(4,  <unfinished ...>)              = ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)

...which is phys_index 0x60 => memory address 0x3000000000

On this platform that lands us here:

100000000-303fffffff : System RAM
  291f000000-291fe00f70 : Kernel code
  2920000000-292051efff : Kernel rodata
  2920600000-292093b0bf : Kernel data
  29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent Memory

...where the last memory block of System RAM is shared with persistent
memory. I.e. the block is only partially online which means that
page_to_nid() in is_mem_section_removable() will assert or crash for
some of the offline pages in that block.


  parent reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-28  9:35 David Hildenbrand
2020-01-31 13:41 ` Fontenot, Nathan
2020-03-27  6:24 ` Dan Williams [this message]
2020-03-27  7:47   ` Michal Hocko
2020-03-27  9:00     ` David Hildenbrand
2020-03-27 16:28       ` Dan Williams
2020-03-27 16:50         ` David Hildenbrand
2020-03-27 22:13           ` Dan Williams
2020-03-27 22:42             ` David Hildenbrand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAPcyv4jpdaNvJ67SkjyUJLBnBnXXQv686BiVW042g03FUmWLXw@mail.gmail.com \
    --to=dan.j.williams@intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=david@redhat.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=heiko.carstens@de.ibm.com \
    --cc=kzak@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    --cc=mhocko@suse.com \
    --cc=nfont@linux.vnet.ibm.com \
    --cc=pbadari@us.ibm.com \
    --cc=powerpc-utils-devel@googlegroups.com \
    --cc=rafael@kernel.org \
    --cc=rcj@linux.vnet.ibm.com \
    --cc=steve.scargall@intel.com \
    --cc=util-linux@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-mm Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-mm/0 linux-mm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-mm linux-mm/ https://lore.kernel.org/linux-mm \
		linux-mm@kvack.org
	public-inbox-index linux-mm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kvack.linux-mm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git