[PATCH RFC 0/2] s390/mm: don't set ARCH_KEEP_MEMBLOCK

[PATCH RFC 0/2] s390/mm: don't set ARCH_KEEP_MEMBLOCK

[David Hildenbrand]

Looking into why we still create memblocks for hotplugged memory (via
add_memory()), I discovered that we might not need ARCH_KEEP_MEMBLOCK on
s390x after all.

After [1] we will no longer create memblocks for hotplugged memory in
case of !CONFIG_ARCH_KEEP_MEMBLOCK. With this series, the same will apply
to standby memory on s390x, added via add_memory().

[1] https://lkml.kernel.org/r/20200416104707.20219-1-david@redhat.com

David Hildenbrand (2):
  s390/zcore: traverse resources instead of memblocks
  s390/mm: don't set ARCH_KEEP_MEMBLOCK

 arch/s390/Kconfig         |  1 -
 drivers/s390/char/zcore.c | 61 ++++++++++++++++++++++++++++++---------
 2 files changed, 48 insertions(+), 14 deletions(-)

-- 
2.25.1

[PATCH RFC 1/2] s390/zcore: traverse resources instead of memblocks

[David Hildenbrand]

The zcore memmap basically contains the first level of all system RAM from
/proc/iomem. We want to disable CONFIG_ARCH_KEEP_MEMBLOCK (e.g., to not
create memblocks for hotplugged/standby memory and save space), switch to
traversing system ram resources instead. During early boot, we create
resources for all early memblocks (including the crash kernel area). When
adding standby memory, we currently create both, memblocks and resources.

Note: As we don't have memory hotplug after boot (standby memory is added
via sclp during boot), we don't have to worry about races.

I am only able to test under KVM (where I hacked up zcore to still
create the memmap file)

root@vm0:~# cat /proc/iomem
00000000-2fffffff : System RAM
  10424000-10ec6fff : Kernel code
  10ec7000-1139a0e3 : Kernel data
  1177a000-11850fff : Kernel bss
30000000-3fffffff : Crash kernel

Result without this patch:
root@vm0:~# cat /sys/kernel/debug/zcore/memmap
0000000000000000 0000000040000000

Result with this patch:
root@vm0:~# cat /sys/kernel/debug/zcore/memmap
0000000000000000 0000000030000000 0000000030000000 0000000010000000

The difference is due to memblocks getting merged, resources (currently)
not. So we might have some more entries, but they describe the same
memory map.

Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Philipp Rudo <prudo@linux.ibm.com>
Cc: Kirill Smelkov <kirr@nexedi.com>
Cc: Michael Holzheu <holzheu@linux.vnet.ibm.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 drivers/s390/char/zcore.c | 61 ++++++++++++++++++++++++++++++---------
 1 file changed, 48 insertions(+), 13 deletions(-)

diff --git a/drivers/s390/char/zcore.c b/drivers/s390/char/zcore.c
index 08f812475f5e..c40ac7d548d8 100644
--- a/drivers/s390/char/zcore.c
+++ b/drivers/s390/char/zcore.c
@@ -16,7 +16,7 @@
 #include <linux/init.h>
 #include <linux/slab.h>
 #include <linux/debugfs.h>
-#include <linux/memblock.h>
+#include <linux/ioport.h>
 
 #include <asm/asm-offsets.h>
 #include <asm/ipl.h>
@@ -139,35 +139,70 @@ static void release_hsa(void)
 	hsa_available = 0;
 }
 
+struct zcore_memmap_info {
+	char *buf;
+	size_t length;
+};
+
 static ssize_t zcore_memmap_read(struct file *filp, char __user *buf,
 				 size_t count, loff_t *ppos)
 {
-	return simple_read_from_buffer(buf, count, ppos, filp->private_data,
-				       memblock.memory.cnt * CHUNK_INFO_SIZE);
+	struct zcore_memmap_info *info = filp->private_data;
+
+	return simple_read_from_buffer(buf, count, ppos, info->buf,
+				       info->length);
+}
+
+static int zcore_count_ram_resource(struct resource *res, void *arg)
+{
+	size_t *count = arg;
+
+	*count += 1;
+	return 0;
+}
+
+static int zcore_process_ram_resource(struct resource *res, void *arg)
+{
+	char **buf = arg;
+
+	sprintf(*buf, "%016lx %016lx ", (unsigned long) res->start,
+		(unsigned long) resource_size(res));
+
+	*buf += CHUNK_INFO_SIZE;
+	return 0;
 }
 
 static int zcore_memmap_open(struct inode *inode, struct file *filp)
 {
-	struct memblock_region *reg;
+	struct zcore_memmap_info *info;
+	size_t count = 0;
 	char *buf;
-	int i = 0;
 
-	buf = kcalloc(memblock.memory.cnt, CHUNK_INFO_SIZE, GFP_KERNEL);
+	walk_system_ram_res(0, ULONG_MAX, &count, zcore_count_ram_resource);
+
+	info = kmalloc(sizeof(*info), GFP_KERNEL);
+	if (!info)
+		return -ENOMEM;
+	buf = kcalloc(count, CHUNK_INFO_SIZE, GFP_KERNEL);
 	if (!buf) {
+		kfree(info);
 		return -ENOMEM;
 	}
-	for_each_memblock(memory, reg) {
-		sprintf(buf + (i++ * CHUNK_INFO_SIZE), "%016llx %016llx ",
-			(unsigned long long) reg->base,
-			(unsigned long long) reg->size);
-	}
-	filp->private_data = buf;
+	info->length = count * CHUNK_INFO_SIZE;
+	info->buf = buf;
+
+	walk_system_ram_res(0, ULONG_MAX, &buf, zcore_process_ram_resource);
+
+	filp->private_data = info;
 	return nonseekable_open(inode, filp);
 }
 
 static int zcore_memmap_release(struct inode *inode, struct file *filp)
 {
-	kfree(filp->private_data);
+	struct zcore_memmap_info *info = filp->private_data;
+
+	kfree(info->buf);
+	kfree(info);
 	return 0;
 }
 
-- 
2.25.1

Re: [PATCH RFC 1/2] s390/zcore: traverse resources instead of memblocks

[Heiko Carstens]

On Fri, Apr 17, 2020 at 05:01:50PM +0200, David Hildenbrand wrote:
> The zcore memmap basically contains the first level of all system RAM from
> /proc/iomem. We want to disable CONFIG_ARCH_KEEP_MEMBLOCK (e.g., to not
> create memblocks for hotplugged/standby memory and save space), switch to
> traversing system ram resources instead. During early boot, we create
> resources for all early memblocks (including the crash kernel area). When
> adding standby memory, we currently create both, memblocks and resources.
> 
> Note: As we don't have memory hotplug after boot (standby memory is added
> via sclp during boot), we don't have to worry about races.
> 
> I am only able to test under KVM (where I hacked up zcore to still
> create the memmap file)
> 
> root@vm0:~# cat /proc/iomem
> 00000000-2fffffff : System RAM
>   10424000-10ec6fff : Kernel code
>   10ec7000-1139a0e3 : Kernel data
>   1177a000-11850fff : Kernel bss
> 30000000-3fffffff : Crash kernel
> 
> Result without this patch:
> root@vm0:~# cat /sys/kernel/debug/zcore/memmap
> 0000000000000000 0000000040000000
> 
> Result with this patch:
> root@vm0:~# cat /sys/kernel/debug/zcore/memmap
> 0000000000000000 0000000030000000 0000000030000000 0000000010000000
> 
> The difference is due to memblocks getting merged, resources (currently)
> not. So we might have some more entries, but they describe the same
> memory map.
> 
> Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
> Cc: Vasily Gorbik <gor@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> Cc: Philipp Rudo <prudo@linux.ibm.com>
> Cc: Kirill Smelkov <kirr@nexedi.com>
> Cc: Michael Holzheu <holzheu@linux.vnet.ibm.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  drivers/s390/char/zcore.c | 61 ++++++++++++++++++++++++++++++---------
>  1 file changed, 48 insertions(+), 13 deletions(-)

I'm having a hard time to find any code that ever made use of this
file. After all this was only meant for our zfcp dumper, but as far as
I can tell there was never code out there that read the memmap file.

So the pragmatic approach would be to just change its contents, or a
more progressive variant would be to remove the file completely.
But maybe I'm entirely wrong...

I'm leaving this up to Philipp and Alexander :)

Re: [PATCH RFC 1/2] s390/zcore: traverse resources instead of memblocks

[Philipp Rudo]

Hi David,

zcore/memmap is no longer needed and Alexander is preparing a patch to remove
it. You can drop this patch.

Thanks
Philipp

On Wed, 10 Jun 2020 13:45:23 +0200
Heiko Carstens <heiko.carstens@de.ibm.com> wrote:

> On Fri, Apr 17, 2020 at 05:01:50PM +0200, David Hildenbrand wrote:
> > The zcore memmap basically contains the first level of all system RAM from
> > /proc/iomem. We want to disable CONFIG_ARCH_KEEP_MEMBLOCK (e.g., to not
> > create memblocks for hotplugged/standby memory and save space), switch to
> > traversing system ram resources instead. During early boot, we create
> > resources for all early memblocks (including the crash kernel area). When
> > adding standby memory, we currently create both, memblocks and resources.
> > 
> > Note: As we don't have memory hotplug after boot (standby memory is added
> > via sclp during boot), we don't have to worry about races.
> > 
> > I am only able to test under KVM (where I hacked up zcore to still
> > create the memmap file)
> > 
> > root@vm0:~# cat /proc/iomem
> > 00000000-2fffffff : System RAM
> >   10424000-10ec6fff : Kernel code
> >   10ec7000-1139a0e3 : Kernel data
> >   1177a000-11850fff : Kernel bss
> > 30000000-3fffffff : Crash kernel
> > 
> > Result without this patch:
> > root@vm0:~# cat /sys/kernel/debug/zcore/memmap
> > 0000000000000000 0000000040000000
> > 
> > Result with this patch:
> > root@vm0:~# cat /sys/kernel/debug/zcore/memmap
> > 0000000000000000 0000000030000000 0000000030000000 0000000010000000
> > 
> > The difference is due to memblocks getting merged, resources (currently)
> > not. So we might have some more entries, but they describe the same
> > memory map.
> > 
> > Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
> > Cc: Vasily Gorbik <gor@linux.ibm.com>
> > Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> > Cc: Philipp Rudo <prudo@linux.ibm.com>
> > Cc: Kirill Smelkov <kirr@nexedi.com>
> > Cc: Michael Holzheu <holzheu@linux.vnet.ibm.com>
> > Signed-off-by: David Hildenbrand <david@redhat.com>
> > ---
> >  drivers/s390/char/zcore.c | 61 ++++++++++++++++++++++++++++++---------
> >  1 file changed, 48 insertions(+), 13 deletions(-)  
> 
> I'm having a hard time to find any code that ever made use of this
> file. After all this was only meant for our zfcp dumper, but as far as
> I can tell there was never code out there that read the memmap file.
> 
> So the pragmatic approach would be to just change its contents, or a
> more progressive variant would be to remove the file completely.
> But maybe I'm entirely wrong...
> 
> I'm leaving this up to Philipp and Alexander :)

Re: [PATCH RFC 1/2] s390/zcore: traverse resources instead of memblocks

[David Hildenbrand]

On 19.06.20 14:47, Philipp Rudo wrote:
> Hi David,
> 
> zcore/memmap is no longer needed and Alexander is preparing a patch to remove
> it. You can drop this patch.

Awesome, once that's done, can you consider patch #2? Thanks!

-- 
Thanks,

David / dhildenb

[PATCH RFC 2/2] s390/mm: don't set ARCH_KEEP_MEMBLOCK

[David Hildenbrand]

Commit 50be63450728 ("s390/mm: Convert bootmem to memblock") mentions
	"The original bootmem allocator is getting replaced by memblock. To
	cover the needs of the s390 kdump implementation the physical
	memory list is used."

zcore was converted to use resources instead of memblocks.
memblock_discard() will *not* mess with "physmem", only with "memory" and
"reserved" memblocks. So, that data will stay after early boot, to be
used in arch/s390/kernel/crash_dump.c to create the ELF header from
inside the 2nd (a.k.a. dumping) kernel.

We no longer need ARCH_KEEP_MEMBLOCK.

Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Philipp Rudo <prudo@linux.ibm.com>
Cc: Michael Holzheu <holzheu@linux.vnet.ibm.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 arch/s390/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 8206b2c19aa8..122bf4cad8a5 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -102,7 +102,6 @@ config S390
 	select ARCH_INLINE_WRITE_UNLOCK_BH
 	select ARCH_INLINE_WRITE_UNLOCK_IRQ
 	select ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE
-	select ARCH_KEEP_MEMBLOCK
 	select ARCH_STACKWALK
 	select ARCH_SUPPORTS_ATOMIC_RMW
 	select ARCH_SUPPORTS_NUMA_BALANCING
-- 
2.25.1

Re: [PATCH RFC 2/2] s390/mm: don't set ARCH_KEEP_MEMBLOCK

[Heiko Carstens]

On Fri, Apr 17, 2020 at 05:01:51PM +0200, David Hildenbrand wrote:
> Commit 50be63450728 ("s390/mm: Convert bootmem to memblock") mentions
> 	"The original bootmem allocator is getting replaced by memblock. To
> 	cover the needs of the s390 kdump implementation the physical
> 	memory list is used."
> 
> zcore was converted to use resources instead of memblocks.
> memblock_discard() will *not* mess with "physmem", only with "memory" and
> "reserved" memblocks. So, that data will stay after early boot, to be
> used in arch/s390/kernel/crash_dump.c to create the ELF header from
> inside the 2nd (a.k.a. dumping) kernel.
> 
> We no longer need ARCH_KEEP_MEMBLOCK.
> 
> Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
> Cc: Vasily Gorbik <gor@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> Cc: Philipp Rudo <prudo@linux.ibm.com>
> Cc: Michael Holzheu <holzheu@linux.vnet.ibm.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  arch/s390/Kconfig | 1 -
>  1 file changed, 1 deletion(-)

Applied, thanks!

Re: [PATCH RFC 2/2] s390/mm: don't set ARCH_KEEP_MEMBLOCK

[Heiko Carstens]

On Fri, Jun 26, 2020 at 06:32:15PM +0200, Heiko Carstens wrote:
> On Fri, Apr 17, 2020 at 05:01:51PM +0200, David Hildenbrand wrote:
> > Commit 50be63450728 ("s390/mm: Convert bootmem to memblock") mentions
> > 	"The original bootmem allocator is getting replaced by memblock. To
> > 	cover the needs of the s390 kdump implementation the physical
> > 	memory list is used."
> > 
> > zcore was converted to use resources instead of memblocks.
> > memblock_discard() will *not* mess with "physmem", only with "memory" and
> > "reserved" memblocks. So, that data will stay after early boot, to be
> > used in arch/s390/kernel/crash_dump.c to create the ELF header from
> > inside the 2nd (a.k.a. dumping) kernel.
> > 
> > We no longer need ARCH_KEEP_MEMBLOCK.
> > 
> > Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
> > Cc: Vasily Gorbik <gor@linux.ibm.com>
> > Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> > Cc: Philipp Rudo <prudo@linux.ibm.com>
> > Cc: Michael Holzheu <holzheu@linux.vnet.ibm.com>
> > Signed-off-by: David Hildenbrand <david@redhat.com>
> > ---
> >  arch/s390/Kconfig | 1 -
> >  1 file changed, 1 deletion(-)
> 
> Applied, thanks!

Hmm, this triggers:

WARNING: modpost: vmlinux.o(.text+0x1e7c2): Section mismatch in reference from the function elfcorehdr_alloc() to the variable .meminit.data:memblock
The function elfcorehdr_alloc() references
the variable __meminitdata memblock.
This is often because elfcorehdr_alloc lacks a __meminitdata 
annotation or the annotation of memblock is wrong.

WARNING: modpost: vmlinux.o(.text+0x1e800): Section mismatch in reference from the function elfcorehdr_alloc() to the function .meminit.text:__next_mem_range()
The function elfcorehdr_alloc() references
the function __meminit __next_mem_range().
This is often because elfcorehdr_alloc lacks a __meminit 
annotation or the annotation of __next_mem_range is wrong.

WARNING: modpost: vmlinux.o(.text+0x1e844): Section mismatch in reference from the function elfcorehdr_alloc() to the function .meminit.text:__next_mem_range()
The function elfcorehdr_alloc() references
the function __meminit __next_mem_range().
This is often because elfcorehdr_alloc lacks a __meminit 
annotation or the annotation of __next_mem_range is wrong.

So, reverted again ;)

Re: [PATCH RFC 2/2] s390/mm: don't set ARCH_KEEP_MEMBLOCK

[David Hildenbrand]

On 29.06.20 14:44, Heiko Carstens wrote:
> On Fri, Jun 26, 2020 at 06:32:15PM +0200, Heiko Carstens wrote:
>> On Fri, Apr 17, 2020 at 05:01:51PM +0200, David Hildenbrand wrote:
>>> Commit 50be63450728 ("s390/mm: Convert bootmem to memblock") mentions
>>> 	"The original bootmem allocator is getting replaced by memblock. To
>>> 	cover the needs of the s390 kdump implementation the physical
>>> 	memory list is used."
>>>
>>> zcore was converted to use resources instead of memblocks.
>>> memblock_discard() will *not* mess with "physmem", only with "memory" and
>>> "reserved" memblocks. So, that data will stay after early boot, to be
>>> used in arch/s390/kernel/crash_dump.c to create the ELF header from
>>> inside the 2nd (a.k.a. dumping) kernel.
>>>
>>> We no longer need ARCH_KEEP_MEMBLOCK.
>>>
>>> Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
>>> Cc: Vasily Gorbik <gor@linux.ibm.com>
>>> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
>>> Cc: Philipp Rudo <prudo@linux.ibm.com>
>>> Cc: Michael Holzheu <holzheu@linux.vnet.ibm.com>
>>> Signed-off-by: David Hildenbrand <david@redhat.com>
>>> ---
>>>  arch/s390/Kconfig | 1 -
>>>  1 file changed, 1 deletion(-)
>>
>> Applied, thanks!
> 
> Hmm, this triggers:

Ah, I see, will have a look. Weird I didn't notice that ...

Thanks!



-- 
Thanks,

David / dhildenb

Re: [PATCH RFC 0/2] s390/mm: don't set ARCH_KEEP_MEMBLOCK

[David Hildenbrand]

On 17.04.20 17:01, David Hildenbrand wrote:
> Looking into why we still create memblocks for hotplugged memory (via
> add_memory()), I discovered that we might not need ARCH_KEEP_MEMBLOCK on
> s390x after all.
> 
> After [1] we will no longer create memblocks for hotplugged memory in
> case of !CONFIG_ARCH_KEEP_MEMBLOCK. With this series, the same will apply
> to standby memory on s390x, added via add_memory().
> 
> [1] https://lkml.kernel.org/r/20200416104707.20219-1-david@redhat.com
> 
> David Hildenbrand (2):
>   s390/zcore: traverse resources instead of memblocks
>   s390/mm: don't set ARCH_KEEP_MEMBLOCK
> 
>  arch/s390/Kconfig         |  1 -
>  drivers/s390/char/zcore.c | 61 ++++++++++++++++++++++++++++++---------
>  2 files changed, 48 insertions(+), 14 deletions(-)
> 

Ping.

-- 
Thanks,

David / dhildenb