Re: mlockall(MCL_CURRENT) blocking infinitely

linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed

* Re: mlockall(MCL_CURRENT) blocking infinitely
       [not found] <4576b336-66e6-e2bb-cd6a-51300ed74ab8@snazy.de>
@ 2019-10-24 23:34 ` Randy Dunlap
  2019-10-25  9:21   ` Michal Hocko
  0 siblings, 1 reply; 39+ messages in thread
From: Randy Dunlap @ 2019-10-24 23:34 UTC (permalink / raw)
  To: Robert Stupp, linux-kernel, Linux MM; +Cc: Andrew Morton, Potyra, Stefan

[adding linux-mm + people]

I see only one change in the last 4 years:

commit dedca63504a204dc8410d98883fdc16dffa8cb80
Author: Potyra, Stefan <Stefan.Potyra@elektrobit.com>
Date:   Thu Jun 13 15:55:55 2019 -0700

    mm/mlock.c: mlockall error for flag MCL_ONFAULT


On 10/24/19 12:36 AM, Robert Stupp wrote:
> Hi guys,
> 
> I've got an issue with `mlockall(MCL_CURRENT)` after upgrading Ubuntu 19.04 to 19.10 - i.e. kernel version change from 5.0.x to 5.3.x.
> 
> The following simple program hangs forever with one CPU running at 100% (kernel):
> 
> #include <stdio.h>
> #include <sys/mman.h>
> int main(char** argv) {
>   printf("Before mlockall(MCL_CURRENT)\n");
>   // works in 5.0
>   // hangs forever w/ 5.1 and newer
>   mlockall(MCL_CURRENT);
>   printf("After mlockall(MCL_CURRENT)\n");
> }
> 
> All kernel versions since 5.1 (tried 5.1.0, 5.1.21, 5.2.21, 5.3.0-19, 5.3.7, 5.4-rc4) show the same symptom (hanging in mlockall(MCL_CURRENT) with 100% kernel-CPU). 5.0 kernel versions (5.0.21) are fine.
> 
> First, I thought, that it's something generic, so I tried the above program in a fresh install of Ubuntu eoan (5.3.x) in a VM in virtualbox, but it works fine there. So I suspect, that it has to do with something that's specific to my machine.
> 
> My first suspicion was that some library "hijacks" mlockall(), but calling the test program with `LD_DEBUG=all` shows that glibc gets called directly:
>      12248:    symbol=mlockall;  lookup in file=./test [0]
>      12248:    symbol=mlockall;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
>      12248:    binding file ./test [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `mlockall' [GLIBC_2.2.5]
> An `strace` doesn't show anything meaningful (beside that mlockall's been called but never returns). dmesg and syslog don't show anything obvious (to me) as well.
> 
> Some information about the machine:
> - Intel(R) Core(TM) i7-6900K, Intel X99 chipset
> - NVMe 1.1b
> - 64GB RAM (4x 16GB)
> 
> I've also reverted all changes for sysctl and ld.conf and checked for other suspicious software without any luck.
> 
> I also tried a bunch of variations of the above program, but only `mlockall(MCL_CURRENT)` or `mlockall(MCL_FUTURE | MCL_CURRENT)` hang.
> 
> A `git diff v5.0..v5.1 mm/` doesn't show anything obvious (to me).
> 
> It seems, there's no debug/trace information that would help to find out what exactly it's doing.
> 
> I'm kinda lost at the moment.
> 
> 
> PS: Variations of the above test program:
> 
> #include <stdio.h>
> #include <sys/mman.h>
> char foo[65536];
> int main(char** argv) {
>   printf("Before mlock()\n");
>   int e = mlock(foo, 8192); // works in 5.0, 5.1, 5.2, 5.3, 5.4
>   printf("After mlock()=%d\n", e);
> }
> 
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <sys/mman.h>
> int main(char** argv) {
>   printf("Before mlockall(MCL_FUTURE)\n");
>   int e = mlockall(MCL_FUTURE); // works in 5.0, 5.1, 5.2, 5.3, 5.4
>   printf("After mlockall(MCL_FUTURE) = %d\n", e);
>   void* mem = malloc(1024 * 1024 * 1024);
>   printf("After malloc()\n");
>   mem = malloc(1024 * 1024 * 1024);
>   printf("After malloc()\n");
>   mem = malloc(1024 * 1024 * 1024);
>   printf("After malloc()\n");
>   // works in 5.0, 5.1, 5.2, 5.3, 5.4
> }
> 
> 
> #include <stdio.h>
> #include <sys/mman.h>
> int main(char** argv) {
>   printf("Before munlockall()\n");
>   int e = munlockall(); // works in 5.0, 5.1, 5.2, 5.3, 5.4
>   printf("After munlockall() = %d\n", e);
> }
> 
> 
> #include <stdio.h>
> #include <sys/mman.h>
> int main(char** argv) {
>   printf("Before mlockall(MCL_CURRENT|MCL_FUTURE)\n");
>   // works in 5.0
>   // hangs forever w/ 5.1 and newer
>   int e = mlockall(MCL_CURRENT|MCL_FUTURE);
>   printf("After mlockall(MCL_CURRENT|MCL_FUTURE) = %d\n", e);
> }
> 
> PPS: Kernel version images installed from https://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D
> 


-- 
~Randy



^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-24 23:34 ` mlockall(MCL_CURRENT) blocking infinitely Randy Dunlap
@ 2019-10-25  9:21   ` Michal Hocko
  2019-10-25 11:02     ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Michal Hocko @ 2019-10-25  9:21 UTC (permalink / raw)
  To: Robert Stupp
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Thu 24-10-19 16:34:46, Randy Dunlap wrote:
> [adding linux-mm + people]
> 
> I see only one change in the last 4 years:
> 
> commit dedca63504a204dc8410d98883fdc16dffa8cb80
> Author: Potyra, Stefan <Stefan.Potyra@elektrobit.com>
> Date:   Thu Jun 13 15:55:55 2019 -0700
> 
>     mm/mlock.c: mlockall error for flag MCL_ONFAULT
> 
> 
> On 10/24/19 12:36 AM, Robert Stupp wrote:
> > Hi guys,
> > 
> > I've got an issue with `mlockall(MCL_CURRENT)` after upgrading Ubuntu 19.04 to 19.10 - i.e. kernel version change from 5.0.x to 5.3.x.
> > 
> > The following simple program hangs forever with one CPU running at 100% (kernel):

Can you capture everal snapshots of proc/$(pidof $YOURTASK)/stack while
this is happening?
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25  9:21   ` Michal Hocko
@ 2019-10-25 11:02     ` Robert Stupp
  2019-10-25 11:46       ` Michal Hocko
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 11:02 UTC (permalink / raw)
  To: Michal Hocko, Robert Stupp
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 11:21 +0200, Michal Hocko wrote:
> On Thu 24-10-19 16:34:46, Randy Dunlap wrote:
> > [adding linux-mm + people]
> >
> > On 10/24/19 12:36 AM, Robert Stupp wrote:
> > > Hi guys,
> > >
> > > I've got an issue with `mlockall(MCL_CURRENT)` after upgrading
> > > Ubuntu 19.04 to 19.10 - i.e. kernel version change from 5.0.x to
> > > 5.3.x.
> > >
> > > The following simple program hangs forever with one CPU running
> > > at 100% (kernel):
>
> Can you capture everal snapshots of proc/$(pidof $YOURTASK)/stack
> while
> this is happening?

Sure,

Approach:
- one shell running
  while true; do cat /proc/$(pidof test)/stack; done
- starting ./test in another shell + ctrl-c quite some times

Vast majority of all ./test invocations return an empty 'stack' file.
Some tries, maybe 1 out of 20, returned these snapshots.
Was running 5.3.7 for this test.


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


[<0>] __do_fault+0x3c/0x130
[<0>] do_fault+0x248/0x640
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


// doubt this one is relevant
[<0>] __wake_up_common_lock+0x7c/0xc0
[<0>] __wake_up_sync_key+0x1e/0x30
[<0>] __wake_up_parent+0x26/0x30
[<0>] do_notify_parent+0x1cc/0x280
[<0>] do_exit+0x703/0xaf0
[<0>] do_group_exit+0x47/0xb0
[<0>] get_signal+0x165/0x880
[<0>] do_signal+0x34/0x280
[<0>] exit_to_usermode_loop+0xbf/0x160
[<0>] do_syscall_64+0x10f/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0


[<0>] __handle_mm_fault+0x4c5/0x7a0


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0xc0/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


[<0>] __do_fault+0x3c/0x130
[<0>] do_fault+0x248/0x640
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0xc0/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0x230/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] __handle_mm_fault+0x4c5/0x7a0
[<0>] handle_mm_fault+0xca/0x1f0
[<0>] __get_user_pages+0xc0/0x770
[<0>] populate_vma_page_range+0x74/0x80
[<0>] __mm_populate+0xb1/0x150
[<0>] __x64_sys_mlockall+0x11c/0x190
[<0>] do_syscall_64+0x5a/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9



Some snippets from `dmesg`:
[    0.000000] microcode: microcode updated early to revision
0xb000038, date = 2019-06-18
[    0.000000] Linux version 5.3.7-050307-generic (kernel@tangerine)
(gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)) #201910180652 SMP
Fri Oct 18 10:56:47 UTC 2019
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.3.7-050307-
generic root=UUID=4325fe2c-ba38-4a75-b4dc-896db1796495 ro
intel_idle.max_cstate=0 quiet splash vt.handoff=7
...
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff]
usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000038d35fff]
usable
[    0.000000] BIOS-e820: [mem 0x0000000038d36000-0x0000000039a94fff]
reserved
[    0.000000] BIOS-e820: [mem 0x0000000039a95000-0x0000000039d4dfff]
usable
[    0.000000] BIOS-e820: [mem 0x0000000039d4e000-0x000000003a8fafff]
ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000003a8fb000-0x000000003b159fff]
reserved
[    0.000000] BIOS-e820: [mem 0x000000003b15a000-0x000000003b1befff]
type 20
[    0.000000] BIOS-e820: [mem 0x000000003b1bf000-0x000000003b1bffff]
usable
[    0.000000] BIOS-e820: [mem 0x000000003b1c0000-0x000000003b245fff]
reserved
[    0.000000] BIOS-e820: [mem 0x000000003b246000-0x000000003bffffff]
usable
[    0.000000] BIOS-e820: [mem 0x000000003c000000-0x000000003dffffff]
reserved
[    0.000000] BIOS-e820: [mem 0x0000000040000000-0x000000004fffffff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000010bfffffff]
usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] efi: EFI v2.40 by American Megatrends
[    0.000000] efi:  ESRT=0x3b156218  ACPI=0x3a55e000  ACPI
2.0=0x3a55e000  SMBIOS=0xf05e0  SMBIOS 3.0=0x3b01e000  MPS=0xfca50
[    0.000000] SMBIOS 3.0.0 present.
[    0.000000] DMI: MSI MS-7A54/X99A TOMAHAWK (MS-7A54), BIOS 2.10
09/13/2016
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 3199.897 MHz processor
[    0.001428] e820: update [mem 0x00000000-0x00000fff] usable ==>
reserved
[    0.001429] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.001433] last_pfn = 0x10c0000 max_arch_pfn = 0x400000000
[    0.001436] MTRR default type: write-back
[    0.001437] MTRR fixed ranges enabled:
[    0.001438]   00000-9FFFF write-back
[    0.001439]   A0000-BFFFF uncachable
[    0.001439]   C0000-FFFFF write-protect
[    0.001440] MTRR variable ranges enabled:
[    0.001441]   0 base 000080000000 mask 3FFF80000000 uncachable
[    0.001441]   1 base 000040000000 mask 3FFFC0000000 uncachable
[    0.001442]   2 base 380000000000 mask 3FC000000000 uncachable
[    0.001443]   3 base 00003F000000 mask 3FFFFF000000 uncachable
[    0.001443]   4 base 0000E0000000 mask 3FFFF0000000 write-through
[    0.001444]   5 base 0000F0000000 mask 3FFFFE000000 write-through
[    0.001444]   6 disabled
[    0.001445]   7 disabled
[    0.001445]   8 disabled
[    0.001445]   9 disabled
[    0.001872] x86/PAT: Configuration [0-7]: WB  WC  UC-
UC  WB  WP  UC- WT
[    0.002037] last_pfn = 0x3c000 max_arch_pfn = 0x400000000
[    0.008082] found SMP MP-table at [mem 0x000fcd80-0x000fcd8f]
[    0.008091] esrt: Reserving ESRT space from 0x000000003b156218 to
0x000000003b156250.
[    0.008098] check: Scanning 1 areas for low memory corruption
[    0.008101] Using GB pages for direct mapping
[    0.008102] BRK [0xe77001000, 0xe77001fff] PGTABLE
[    0.008104] BRK [0xe77002000, 0xe77002fff] PGTABLE
[    0.008105] BRK [0xe77003000, 0xe77003fff] PGTABLE
[    0.008122] BRK [0xe77004000, 0xe77004fff] PGTABLE
[    0.008123] BRK [0xe77005000, 0xe77005fff] PGTABLE
[    0.008186] BRK [0xe77006000, 0xe77006fff] PGTABLE
[    0.008203] BRK [0xe77007000, 0xe77007fff] PGTABLE
[    0.008206] BRK [0xe77008000, 0xe77008fff] PGTABLE
[    0.008222] BRK [0xe77009000, 0xe77009fff] PGTABLE
[    0.008230] BRK [0xe7700a000, 0xe7700afff] PGTABLE
[    0.008265] Secure boot could not be determined
[    0.008266] RAMDISK: [mem 0x2e47c000-0x310e4fff]
[    0.008271] ACPI: Early table checksum verification disabled
[    0.008274] ACPI: RSDP 0x000000003A55E000 000024 (v02 ALASKA)
[    0.008276] ACPI: XSDT 0x000000003A55E090 0000A4 (v01 ALASKA A M
I    01072009 AMI  00010013)
[    0.008281] ACPI: FACP 0x000000003A591EB8 00010C (v05 ALASKA A M
I    01072009 AMI  00010013)
[    0.008285] ACPI: DSDT 0x000000003A55E1D0 033CE5 (v02 ALASKA A M
I    01072009 INTL 20091013)
[    0.008287] ACPI: FACS 0x000000003A8F9F80 000040
[    0.008289] ACPI: APIC 0x000000003A591FC8 000138 (v03 ALASKA A M
I    01072009 AMI  00010013)
[    0.008291] ACPI: FPDT 0x000000003A592100 000044 (v01 ALASKA A M
I    01072009 AMI  00010013)
[    0.008292] ACPI: FIDT 0x000000003A592148 00009C (v01 ALASKA A M
I    01072009 AMI  00010013)
[    0.008294] ACPI: MCFG 0x000000003A5921E8 00003C (v01 ALASKA A M
I    01072009 MSFT 00000097)
[    0.008296] ACPI: SSDT 0x000000003A592228 00036D (v01 SataRe
SataTabl 00001000 INTL 20120913)
[    0.008298] ACPI: UEFI 0x000000003A592598 000042 (v01 ALASKA A M
I    01072009      00000000)
[    0.008300] ACPI: HPET 0x000000003A5925E0 000038 (v01 ALASKA A M
I    00000001 INTL 20091013)
[    0.008302] ACPI: MSCT 0x000000003A592618 000090 (v01 ALASKA A M
I    00000001 INTL 20091013)
[    0.008304] ACPI: SLIT 0x000000003A5926A8 00002D (v01 ALASKA A M
I    00000001 INTL 20091013)
[    0.008306] ACPI: SRAT 0x000000003A5926D8 001158 (v03 ALASKA A M
I    00000001 INTL 20091013)
[    0.008308] ACPI: WDDT 0x000000003A593830 000040 (v01 ALASKA A M
I    00000000 INTL 20091013)
[    0.008310] ACPI: SSDT 0x000000003A593870 015307 (v02 ALASKA
PmMgt    00000001 INTL 20120913)
[    0.008312] ACPI: NITR 0x000000003A5A8B78 000071 (v02 ALASKA A M
I    00000001 INTL 20091013)
[    0.008314] ACPI: DMAR 0x000000003A5A8BF0 0000E4 (v01 ALASKA A M
I    00000001 INTL 20091013)
[    0.008316] ACPI: ASF! 0x000000003A5A8CD8 0000A0 (v32
INTEL   HCG     00000001 TFSM 000F4240)
[    0.008322] ACPI: Local APIC address 0xfee00000
[    0.008345] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.008345] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.008345] SRAT: PXM 0 -> APIC 0x04 -> Node 0
[    0.008346] SRAT: PXM 0 -> APIC 0x06 -> Node 0
[    0.008346] SRAT: PXM 0 -> APIC 0x08 -> Node 0
[    0.008347] SRAT: PXM 0 -> APIC 0x0a -> Node 0
[    0.008347] SRAT: PXM 0 -> APIC 0x0c -> Node 0
[    0.008347] SRAT: PXM 0 -> APIC 0x0e -> Node 0
[    0.008348] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.008348] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[    0.008349] SRAT: PXM 0 -> APIC 0x05 -> Node 0
[    0.008349] SRAT: PXM 0 -> APIC 0x07 -> Node 0
[    0.008349] SRAT: PXM 0 -> APIC 0x09 -> Node 0
[    0.008350] SRAT: PXM 0 -> APIC 0x0b -> Node 0
[    0.008350] SRAT: PXM 0 -> APIC 0x0d -> Node 0
[    0.008350] SRAT: PXM 0 -> APIC 0x0f -> Node 0
[    0.008357] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff]
[    0.008357] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x10bfffffff]
[    0.008362] NUMA: Initialized distance table, cnt=1
[    0.008363] NUMA: Node 0 [mem 0x00000000-0x3fffffff] + [mem
0x100000000-0x10bfffffff] -> [mem 0x00000000-0x10bfffffff]
[    0.008370] NODE_DATA(0) allocated [mem 0x10bffd4000-0x10bfffefff]
[    0.008559] Zone ranges:
[    0.008560]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.008561]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.008562]   Normal   [mem 0x0000000100000000-0x00000010bfffffff]
[    0.008563]   Device   empty
[    0.008563] Movable zone start for each node
[    0.008566] Early memory node ranges
[    0.008566]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
[    0.008567]   node   0: [mem 0x0000000000100000-0x0000000038d35fff]
[    0.008568]   node   0: [mem 0x0000000039a95000-0x0000000039d4dfff]
[    0.008568]   node   0: [mem 0x000000003b1bf000-0x000000003b1bffff]
[    0.008568]   node   0: [mem 0x000000003b246000-0x000000003bffffff]
[    0.008569]   node   0: [mem 0x0000000100000000-0x00000010bfffffff]
[    0.008788] Zeroed struct page in unavailable ranges: 25271 pages
[    0.008789] Initmem setup node 0 [mem 0x0000000000001000-
0x00000010bfffffff]
[    0.008790] On node 0 totalpages: 16751945
[    0.008791]   DMA zone: 64 pages used for memmap
[    0.008791]   DMA zone: 25 pages reserved
[    0.008792]   DMA zone: 3999 pages, LIFO batch:0
[    0.008833]   DMA32 zone: 3639 pages used for memmap
[    0.008834]   DMA32 zone: 232874 pages, LIFO batch:63
[    0.014058]   Normal zone: 258048 pages used for memmap
[    0.014058]   Normal zone: 16515072 pages, LIFO batch:63
[    0.168284] ACPI: PM-Timer IO Port: 0x408
[    0.168286] ACPI: Local APIC address 0xfee00000
[    0.168293] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.168293] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.168294] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.168294] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.168295] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
[    0.168295] ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
[    0.168295] ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
[    0.168296] ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
[    0.168296] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.168297] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.168297] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.168297] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.168298] ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
[    0.168298] ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
[    0.168298] ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
[    0.168299] ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
[    0.168309] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000,
GSI 0-23
[    0.168313] IOAPIC[1]: apic_id 2, version 32, address 0xfec01000,
GSI 24-47
[    0.168314] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.168315] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high
level)
[    0.168316] ACPI: IRQ0 used by override.
[    0.168317] ACPI: IRQ9 used by override.
[    0.168319] Using ACPI (MADT) for SMP configuration information
[    0.168320] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.168322] smpboot: Allowing 16 CPUs, 0 hotplug CPUs
[    0.168334] PM: Registered nosave memory: [mem 0x00000000-
0x00000fff]
[    0.168335] PM: Registered nosave memory: [mem 0x000a0000-
0x000fffff]
[    0.168336] PM: Registered nosave memory: [mem 0x38d36000-
0x39a94fff]
[    0.168337] PM: Registered nosave memory: [mem 0x39d4e000-
0x3a8fafff]
[    0.168337] PM: Registered nosave memory: [mem 0x3a8fb000-
0x3b159fff]
[    0.168338] PM: Registered nosave memory: [mem 0x3b15a000-
0x3b1befff]
[    0.168339] PM: Registered nosave memory: [mem 0x3b1c0000-
0x3b245fff]
[    0.168340] PM: Registered nosave memory: [mem 0x3c000000-
0x3dffffff]
[    0.168340] PM: Registered nosave memory: [mem 0x3e000000-
0x3fffffff]
[    0.168341] PM: Registered nosave memory: [mem 0x40000000-
0x4fffffff]
[    0.168341] PM: Registered nosave memory: [mem 0x50000000-
0xfed1bfff]
[    0.168341] PM: Registered nosave memory: [mem 0xfed1c000-
0xfed44fff]
[    0.168342] PM: Registered nosave memory: [mem 0xfed45000-
0xfeffffff]
[    0.168342] PM: Registered nosave memory: [mem 0xff000000-
0xffffffff]
[    0.168344] [mem 0x50000000-0xfed1bfff] available for PCI devices
[    0.168345] Booting paravirtualized kernel on bare hardware
[    0.168347] clocksource: refined-jiffies: mask: 0xffffffff
max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.168352] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:16
nr_cpu_ids:16 nr_node_ids:1
[    0.168718] percpu: Embedded 54 pages/cpu s184320 r8192 d28672
u262144
[    0.168724] pcpu-alloc: s184320 r8192 d28672 u262144 alloc=1*2097152
[    0.168724] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11
12 13 14 15
[    0.168747] Built 1 zonelists, mobility grouping on.  Total pages:
16490169
[    0.168748] Policy zone: Normal
[    0.168749] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.3.7-
050307-generic root=UUID=4325fe2c-ba38-4a75-b4dc-896db1796495 ro
intel_idle.max_cstate=0 quiet splash vt.handoff=7
[    0.172892] Dentry cache hash table entries: 8388608 (order: 14,
67108864 bytes, linear)
[    0.175045] Inode-cache hash table entries: 4194304 (order: 13,
33554432 bytes, linear)
[    0.175154] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.179532] Calgary: detecting Calgary via BIOS EBDA area
[    0.179533] Calgary: Unable to locate Rio Grande table in EBDA -
bailing!
[    0.318204] Memory: 65577656K/67007780K available (14339K kernel
code, 2387K rwdata, 4700K rodata, 2664K init, 5060K bss, 1430124K
reserved, 0K cma-reserved)
[    0.318334] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=16,
Nodes=1
[    0.318344] Kernel/User page tables isolation: enabled
[    0.318357] ftrace: allocating 43143 entries in 169 pages
[    0.330881] rcu: Hierarchical RCU implementation.
[    0.330883] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to
nr_cpu_ids=16.
[    0.330883] 	Tasks RCU enabled.
[    0.330884] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[    0.330884] rcu: Adjusting geometry for rcu_fanout_leaf=16,
nr_cpu_ids=16
[    0.333275] NR_IRQS: 524544, nr_irqs: 960, preallocated irqs: 16
[    0.333488] random: crng done (trusting CPU's manufacturer)
[    0.333512] Console: colour dummy device 80x25
[    0.333516] printk: console [tty0] enabled
[    0.333530] ACPI: Core revision 20190703
[    0.333738] clocksource: hpet: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 133484882848 ns
[    0.333747] APIC: Switch to symmetric I/O mode setup
[    0.333748] DMAR: Host address width 46
[    0.333748] DMAR: DRHD base: 0x000000fbffd000 flags: 0x0
[    0.333752] DMAR: dmar0: reg_base_addr fbffd000 ver 1:0 cap
8d2008c10ef0466 ecap f0205b
[    0.333753] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.333755] DMAR: dmar1: reg_base_addr fbffc000 ver 1:0 cap
8d2078c106f0466 ecap f020df
[    0.333756] DMAR: RMRR base: 0x0000003b030000 end: 0x0000003b03ffff
[    0.333757] DMAR: ATSR flags: 0x0
[    0.333757] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.333759] DMAR-IR: IOAPIC id 1 under DRHD base  0xfbffc000 IOMMU 1
[    0.333759] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 1
[    0.333760] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[    0.333760] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt
out bit.
[    0.333760] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the
BIOS setting.
[    0.334167] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.334168] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.334171] Switched APIC routing to physical flat.
[    0.334649] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.353749] clocksource: tsc-early: mask: 0xffffffffffffffff
max_cycles: 0x2e1fe890e9e, max_idle_ns: 440795268505 ns
[    0.353751] Calibrating delay loop (skipped), value calculated using
timer frequency.. 6399.79 BogoMIPS (lpj=12799588)
[    0.353752] pid_max: default: 32768 minimum: 301
[    0.356787] LSM: Security Framework initializing
[    0.356795] Yama: becoming mindful.
[    0.356848] AppArmor: AppArmor initialized
[    0.356976] Mount-cache hash table entries: 131072 (order: 8,
1048576 bytes, linear)
[    0.357078] Mountpoint-cache hash table entries: 131072 (order: 8,
1048576 bytes, linear)
[    0.357204] *** VALIDATE proc ***
[    0.357244] *** VALIDATE cgroup1 ***
[    0.357245] *** VALIDATE cgroup2 ***
[    0.357297] mce: CPU0: Thermal monitoring enabled (TM1)
[    0.357334] process: using mwait in idle threads
[    0.357336] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.357337] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.357338] Spectre V1 : Mitigation: usercopy/swapgs barriers and
__user pointer sanitization
[    0.357340] Spectre V2 : Mitigation: Full generic retpoline
[    0.357340] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling
RSB on context switch
[    0.357341] Spectre V2 : Enabling Restricted Speculation for
firmware calls
[    0.357342] Spectre V2 : mitigation: Enabling conditional Indirect
Branch Prediction Barrier
[    0.357342] Spectre V2 : User space: Mitigation: STIBP via seccomp
and prctl
[    0.357343] Speculative Store Bypass: Mitigation: Speculative Store
Bypass disabled via prctl and seccomp
[    0.357345] MDS: Mitigation: Clear CPU buffers
[    0.357526] Freeing SMP alternatives memory: 36K
[    0.361805] TSC deadline timer enabled
[    0.361806] smpboot: CPU0: Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz
(family: 0x6, model: 0x4f, stepping: 0x1)
[    0.361888] Performance Events: PEBS fmt2+, Broadwell events, 16-
deep LBR, full-width counters, Intel PMU driver.
[    0.361893] ... version:                3
[    0.361893] ... bit width:              48
[    0.361894] ... generic registers:      4
[    0.361894] ... value mask:             0000ffffffffffff
[    0.361895] ... max period:             00007fffffffffff
[    0.361895] ... fixed-purpose events:   3
[    0.361895] ... event mask:             000000070000000f
[    0.361921] rcu: Hierarchical SRCU implementation.
[    0.363125] NMI watchdog: Enabled. Permanently consumes one hw-PMU
counter.
[    0.363220] smp: Bringing up secondary CPUs ...
[    0.363284] x86: Booting SMP configuration:
[    0.363284] .... node  #0,
CPUs:        #1  #2  #3  #4  #5  #6  #7  #8
[    0.381823] MDS CPU bug present and SMT on, data leak possible. See
https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for
more details.
[    0.381856]   #9 #10 #11 #12 #13 #14 #15
[    0.386438] smp: Brought up 1 node, 16 CPUs
[    0.386438] smpboot: Max logical packages: 1
[    0.386438] smpboot: Total of 16 processors activated (102396.70
BogoMIPS)
[    0.390924] devtmpfs: initialized
[    0.390924] x86/mm: Memory block size: 1024MB
[    0.390924] PM: Registering ACPI NVS region [mem 0x39d4e000-
0x3a8fafff] (12242944 bytes)
[    0.390924] clocksource: jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.390924] futex hash table entries: 4096 (order: 6, 262144 bytes,
linear)
[    0.390924] pinctrl core: initialized pinctrl subsystem
[    0.390924] PM: RTC time: 08:17:57, date: 2019-10-25
[    0.390924] NET: Registered protocol family 16
[    0.390924] audit: initializing netlink subsys (disabled)
[    0.390924] audit: type=2000 audit(1571991477.056:1):
state=initialized audit_enabled=0 res=1
[    0.390924] EISA bus registered
[    0.390924] cpuidle: using governor ladder
[    0.390924] cpuidle: using governor menu
[    0.390924] ACPI FADT declares the system doesn't support PCIe ASPM,
so disable it
[    0.390924] ACPI: bus type PCI registered
[    0.390924] acpiphp: ACPI Hot Plug PCI Controller Driver version:
0.5
[    0.390924] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem
0x40000000-0x4fffffff] (base 0x40000000)
[    0.390924] PCI: MMCONFIG at [mem 0x40000000-0x4fffffff] reserved in
E820
[    0.390924] PCI: Using configuration type 1 for base access
[    0.393952] HugeTLB registered 1.00 GiB page size, pre-allocated 0
pages
[    0.393952] HugeTLB registered 2.00 MiB page size, pre-allocated 0
pages
[    0.393952] ACPI: Added _OSI(Module Device)
[    0.393952] ACPI: Added _OSI(Processor Device)
[    0.393952] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.393952] ACPI: Added _OSI(Processor Aggregator Device)
[    0.393952] ACPI: Added _OSI(Linux-Dell-Video)
[    0.393952] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.393952] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.444579] ACPI: 3 ACPI AML tables successfully acquired and loaded
[    0.448501] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.451495] ACPI: Dynamic OEM Table Load:
[    0.478469] ACPI: Interpreter enabled
[    0.478482] ACPI: (supports S0 S3 S4 S5)
[    0.478483] ACPI: Using IOAPIC for interrupt routing
[    0.478509] PCI: Using host bridge windows from ACPI; if necessary,
use "pci=nocrs" and report a bug
[    0.479312] ACPI: Enabled 4 GPEs in block 00 to 3F
[    0.514946] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus ff])
[    0.514949] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM
ClockPM Segments MSI HPX-Type3]
[    0.517147] acpi PNP0A03:03: _OSC: platform does not support
[SHPCHotplug PME AER LTR]
[    0.518203] acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug
PCIeCapability]
[    0.518204] acpi PNP0A03:03: FADT indicates ASPM is unsupported,
using BIOS configuration
[    0.518234] PCI host bridge to bus 0000:ff
...




^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 11:02     ` Robert Stupp
@ 2019-10-25 11:46       ` Michal Hocko
  2019-10-25 11:50         ` Michal Hocko
  2019-10-25 11:55         ` Robert Stupp
  0 siblings, 2 replies; 39+ messages in thread
From: Michal Hocko @ 2019-10-25 11:46 UTC (permalink / raw)
  To: snazy; +Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri 25-10-19 13:02:23, Robert Stupp wrote:
> On Fri, 2019-10-25 at 11:21 +0200, Michal Hocko wrote:
> > On Thu 24-10-19 16:34:46, Randy Dunlap wrote:
> > > [adding linux-mm + people]
> > >
> > > On 10/24/19 12:36 AM, Robert Stupp wrote:
> > > > Hi guys,
> > > >
> > > > I've got an issue with `mlockall(MCL_CURRENT)` after upgrading
> > > > Ubuntu 19.04 to 19.10 - i.e. kernel version change from 5.0.x to
> > > > 5.3.x.
> > > >
> > > > The following simple program hangs forever with one CPU running
> > > > at 100% (kernel):
> >
> > Can you capture everal snapshots of proc/$(pidof $YOURTASK)/stack
> > while
> > this is happening?
> 
> Sure,
> 
> Approach:
> - one shell running
>   while true; do cat /proc/$(pidof test)/stack; done
> - starting ./test in another shell + ctrl-c quite some times
> 
> Vast majority of all ./test invocations return an empty 'stack' file.
> Some tries, maybe 1 out of 20, returned these snapshots.
> Was running 5.3.7 for this test.
> 
> 
> [<0>] __handle_mm_fault+0x4c5/0x7a0
> [<0>] handle_mm_fault+0xca/0x1f0
> [<0>] __get_user_pages+0x230/0x770
> [<0>] populate_vma_page_range+0x74/0x80
> [<0>] __mm_populate+0xb1/0x150
> [<0>] __x64_sys_mlockall+0x11c/0x190
> [<0>] do_syscall_64+0x5a/0x130
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [<0>] __handle_mm_fault+0x4c5/0x7a0
> [<0>] handle_mm_fault+0xca/0x1f0
> [<0>] __get_user_pages+0x230/0x770
> [<0>] populate_vma_page_range+0x74/0x80
> [<0>] __mm_populate+0xb1/0x150
> [<0>] __x64_sys_mlockall+0x11c/0x190
> [<0>] do_syscall_64+0x5a/0x130
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> 
> [<0>] __handle_mm_fault+0x4c5/0x7a0
> [<0>] handle_mm_fault+0xca/0x1f0
> [<0>] __get_user_pages+0x230/0x770
> [<0>] populate_vma_page_range+0x74/0x80
> [<0>] __mm_populate+0xb1/0x150
> [<0>] __x64_sys_mlockall+0x11c/0x190
> [<0>] do_syscall_64+0x5a/0x130
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> 
> [<0>] __do_fault+0x3c/0x130
> [<0>] do_fault+0x248/0x640
> [<0>] __handle_mm_fault+0x4c5/0x7a0
> [<0>] handle_mm_fault+0xca/0x1f0
> [<0>] __get_user_pages+0x230/0x770
> [<0>] populate_vma_page_range+0x74/0x80
> [<0>] __mm_populate+0xb1/0x150
> [<0>] __x64_sys_mlockall+0x11c/0x190
> [<0>] do_syscall_64+0x5a/0x130
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 

This is expected.

> // doubt this one is relevant
> [<0>] __wake_up_common_lock+0x7c/0xc0
> [<0>] __wake_up_sync_key+0x1e/0x30
> [<0>] __wake_up_parent+0x26/0x30
> [<0>] do_notify_parent+0x1cc/0x280
> [<0>] do_exit+0x703/0xaf0
> [<0>] do_group_exit+0x47/0xb0
> [<0>] get_signal+0x165/0x880
> [<0>] do_signal+0x34/0x280
> [<0>] exit_to_usermode_loop+0xbf/0x160
> [<0>] do_syscall_64+0x10f/0x130
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Hmm, this means that the task has exited so how come there are 
other syscalls happening. Are you sure you are collecting stacks for the
correct task?
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 11:46       ` Michal Hocko
@ 2019-10-25 11:50         ` Michal Hocko
  2019-10-25 11:59           ` Robert Stupp
  2019-10-25 11:55         ` Robert Stupp
  1 sibling, 1 reply; 39+ messages in thread
From: Michal Hocko @ 2019-10-25 11:50 UTC (permalink / raw)
  To: snazy; +Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri 25-10-19 13:46:33, Michal Hocko wrote:
> On Fri 25-10-19 13:02:23, Robert Stupp wrote:
> > On Fri, 2019-10-25 at 11:21 +0200, Michal Hocko wrote:
> > > On Thu 24-10-19 16:34:46, Randy Dunlap wrote:
> > > > [adding linux-mm + people]
> > > >
> > > > On 10/24/19 12:36 AM, Robert Stupp wrote:
> > > > > Hi guys,
> > > > >
> > > > > I've got an issue with `mlockall(MCL_CURRENT)` after upgrading
> > > > > Ubuntu 19.04 to 19.10 - i.e. kernel version change from 5.0.x to
> > > > > 5.3.x.
> > > > >
> > > > > The following simple program hangs forever with one CPU running
> > > > > at 100% (kernel):
> > >
> > > Can you capture everal snapshots of proc/$(pidof $YOURTASK)/stack
> > > while
> > > this is happening?

Btw. I have tested
$ cat test_mlockall.c
#include <stdio.h>
#include <sys/mman.h>
int main(char** argv) {
	printf("Before mlockall(MCL_CURRENT|MCL_FUTURE)\n");
	// works in 5.0
	// hangs forever w/ 5.1 and newer
	int e = mlockall(MCL_CURRENT|MCL_FUTURE);
	printf("After mlockall(MCL_CURRENT|MCL_FUTURE) %d\n", e);
}

$./test_mlockall
Before mlockall(MCL_CURRENT|MCL_FUTURE)
After mlockall(MCL_CURRENT|MCL_FUTURE) 0
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 11:50         ` Michal Hocko
@ 2019-10-25 11:59           ` Robert Stupp
  2019-10-25 13:19             ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 11:59 UTC (permalink / raw)
  To: Michal Hocko, snazy
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 13:50 +0200, Michal Hocko wrote:
> On Fri 25-10-19 13:46:33, Michal Hocko wrote:
> > On Fri 25-10-19 13:02:23, Robert Stupp wrote:
> > > On Fri, 2019-10-25 at 11:21 +0200, Michal Hocko wrote:
> > > > On Thu 24-10-19 16:34:46, Randy Dunlap wrote:
> > > > > [adding linux-mm + people]
> > > > >
> > > > > On 10/24/19 12:36 AM, Robert Stupp wrote:
> > > > > > Hi guys,
> > > > > >
> > > > > > I've got an issue with `mlockall(MCL_CURRENT)` after
> > > > > > upgrading
> > > > > > Ubuntu 19.04 to 19.10 - i.e. kernel version change from
> > > > > > 5.0.x to
> > > > > > 5.3.x.
> > > > > >
> > > > > > The following simple program hangs forever with one CPU
> > > > > > running
> > > > > > at 100% (kernel):
> > > >
> > > > Can you capture everal snapshots of proc/$(pidof
> > > > $YOURTASK)/stack
> > > > while
> > > > this is happening?
>
> Btw. I have tested
> $ cat test_mlockall.c
> #include <stdio.h>
> #include <sys/mman.h>
> int main(char** argv) {
> 	printf("Before mlockall(MCL_CURRENT|MCL_FUTURE)\n");
> 	// works in 5.0
> 	// hangs forever w/ 5.1 and newer
> 	int e = mlockall(MCL_CURRENT|MCL_FUTURE);
> 	printf("After mlockall(MCL_CURRENT|MCL_FUTURE) %d\n", e);
> }
>
> $./test_mlockall
> Before mlockall(MCL_CURRENT|MCL_FUTURE)
> After mlockall(MCL_CURRENT|MCL_FUTURE) 0

I suspect, that it's something that's "special" on my machine. But I've
got no clue what that might be. Do you think it makes sense to try with
all the spectre/meltdown mitigations disabled? Or SMT disabled?



^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 11:59           ` Robert Stupp
@ 2019-10-25 13:19             ` Robert Stupp
  0 siblings, 0 replies; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 13:19 UTC (permalink / raw)
  To: snazy, Michal Hocko
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 13:59 +0200, Robert Stupp wrote:
> On Fri, 2019-10-25 at 13:50 +0200, Michal Hocko wrote:
> > On Fri 25-10-19 13:46:33, Michal Hocko wrote:
> > >
> I suspect, that it's something that's "special" on my machine. But
> I've
> got no clue what that might be. Do you think it makes sense to try
> with
> all the spectre/meltdown mitigations disabled? Or SMT disabled?
>

Bummer - booting the kernel with `mitigations=off` didn't help.




^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 11:46       ` Michal Hocko
  2019-10-25 11:50         ` Michal Hocko
@ 2019-10-25 11:55         ` Robert Stupp
  2019-10-25 12:05           ` Michal Hocko
  1 sibling, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 11:55 UTC (permalink / raw)
  To: Michal Hocko, snazy
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 13:46 +0200, Michal Hocko wrote:
> On Fri 25-10-19 13:02:23, Robert Stupp wrote:
> > On Fri, 2019-10-25 at 11:21 +0200, Michal Hocko wrote:
> > > On Thu 24-10-19 16:34:46, Randy Dunlap wrote:
> > > > [adding linux-mm + people]
> > > >
> > > > On 10/24/19 12:36 AM, Robert Stupp wrote:
> > > > > Hi guys,
> > > > >
> > > > > I've got an issue with `mlockall(MCL_CURRENT)` after
> > > > > upgrading
> > > > > Ubuntu 19.04 to 19.10 - i.e. kernel version change from 5.0.x
> > > > > to
> > > > > 5.3.x.
> > > > >
> > > > > The following simple program hangs forever with one CPU
> > > > > running
> > > > > at 100% (kernel):
> > >
> > > Can you capture everal snapshots of proc/$(pidof $YOURTASK)/stack
> > > while
> > > this is happening?
> >
> > Sure,
> >
> > Approach:
> > - one shell running
> >   while true; do cat /proc/$(pidof test)/stack; done
> > - starting ./test in another shell + ctrl-c quite some times
> >
> > Vast majority of all ./test invocations return an empty 'stack'
> > file.
> > Some tries, maybe 1 out of 20, returned these snapshots.
> > Was running 5.3.7 for this test.
> >
> >
> > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > [<0>] handle_mm_fault+0xca/0x1f0
> > [<0>] __get_user_pages+0x230/0x770
> > [<0>] populate_vma_page_range+0x74/0x80
> > [<0>] __mm_populate+0xb1/0x150
> > [<0>] __x64_sys_mlockall+0x11c/0x190
> > [<0>] do_syscall_64+0x5a/0x130
> > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > [<0>] handle_mm_fault+0xca/0x1f0
> > [<0>] __get_user_pages+0x230/0x770
> > [<0>] populate_vma_page_range+0x74/0x80
> > [<0>] __mm_populate+0xb1/0x150
> > [<0>] __x64_sys_mlockall+0x11c/0x190
> > [<0>] do_syscall_64+0x5a/0x130
> > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> >
> > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > [<0>] handle_mm_fault+0xca/0x1f0
> > [<0>] __get_user_pages+0x230/0x770
> > [<0>] populate_vma_page_range+0x74/0x80
> > [<0>] __mm_populate+0xb1/0x150
> > [<0>] __x64_sys_mlockall+0x11c/0x190
> > [<0>] do_syscall_64+0x5a/0x130
> > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> >
> > [<0>] __do_fault+0x3c/0x130
> > [<0>] do_fault+0x248/0x640
> > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > [<0>] handle_mm_fault+0xca/0x1f0
> > [<0>] __get_user_pages+0x230/0x770
> > [<0>] populate_vma_page_range+0x74/0x80
> > [<0>] __mm_populate+0xb1/0x150
> > [<0>] __x64_sys_mlockall+0x11c/0x190
> > [<0>] do_syscall_64+0x5a/0x130
> > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
>
> This is expected.
>
> > // doubt this one is relevant
> > [<0>] __wake_up_common_lock+0x7c/0xc0
> > [<0>] __wake_up_sync_key+0x1e/0x30
> > [<0>] __wake_up_parent+0x26/0x30
> > [<0>] do_notify_parent+0x1cc/0x280
> > [<0>] do_exit+0x703/0xaf0
> > [<0>] do_group_exit+0x47/0xb0
> > [<0>] get_signal+0x165/0x880
> > [<0>] do_signal+0x34/0x280
> > [<0>] exit_to_usermode_loop+0xbf/0x160
> > [<0>] do_syscall_64+0x10f/0x130
> > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Hmm, this means that the task has exited so how come there are
> other syscalls happening. Are you sure you are collecting stacks for
> the
> correct task?

I guess that the `cat /proc/$(pidof test)/stack` captured the stack
after I hit ctrl-c. Does that make sense?

Also tried `syscall(SYS_mlockall, MCL_CURRENT);` instead of
`mlockall(MCL_CURRENT)` - same behavior.



^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 11:55         ` Robert Stupp
@ 2019-10-25 12:05           ` Michal Hocko
  2019-10-25 12:11             ` Michal Hocko
  0 siblings, 1 reply; 39+ messages in thread
From: Michal Hocko @ 2019-10-25 12:05 UTC (permalink / raw)
  To: snazy; +Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri 25-10-19 13:55:13, Robert Stupp wrote:
> On Fri, 2019-10-25 at 13:46 +0200, Michal Hocko wrote:
> > On Fri 25-10-19 13:02:23, Robert Stupp wrote:
> > > On Fri, 2019-10-25 at 11:21 +0200, Michal Hocko wrote:
> > > > On Thu 24-10-19 16:34:46, Randy Dunlap wrote:
> > > > > [adding linux-mm + people]
> > > > >
> > > > > On 10/24/19 12:36 AM, Robert Stupp wrote:
> > > > > > Hi guys,
> > > > > >
> > > > > > I've got an issue with `mlockall(MCL_CURRENT)` after
> > > > > > upgrading
> > > > > > Ubuntu 19.04 to 19.10 - i.e. kernel version change from 5.0.x
> > > > > > to
> > > > > > 5.3.x.
> > > > > >
> > > > > > The following simple program hangs forever with one CPU
> > > > > > running
> > > > > > at 100% (kernel):
> > > >
> > > > Can you capture everal snapshots of proc/$(pidof $YOURTASK)/stack
> > > > while
> > > > this is happening?
> > >
> > > Sure,
> > >
> > > Approach:
> > > - one shell running
> > >   while true; do cat /proc/$(pidof test)/stack; done
> > > - starting ./test in another shell + ctrl-c quite some times
> > >
> > > Vast majority of all ./test invocations return an empty 'stack'
> > > file.
> > > Some tries, maybe 1 out of 20, returned these snapshots.
> > > Was running 5.3.7 for this test.
> > >
> > >
> > > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > > [<0>] handle_mm_fault+0xca/0x1f0
> > > [<0>] __get_user_pages+0x230/0x770
> > > [<0>] populate_vma_page_range+0x74/0x80
> > > [<0>] __mm_populate+0xb1/0x150
> > > [<0>] __x64_sys_mlockall+0x11c/0x190
> > > [<0>] do_syscall_64+0x5a/0x130
> > > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > > [<0>] handle_mm_fault+0xca/0x1f0
> > > [<0>] __get_user_pages+0x230/0x770
> > > [<0>] populate_vma_page_range+0x74/0x80
> > > [<0>] __mm_populate+0xb1/0x150
> > > [<0>] __x64_sys_mlockall+0x11c/0x190
> > > [<0>] do_syscall_64+0x5a/0x130
> > > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > >
> > > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > > [<0>] handle_mm_fault+0xca/0x1f0
> > > [<0>] __get_user_pages+0x230/0x770
> > > [<0>] populate_vma_page_range+0x74/0x80
> > > [<0>] __mm_populate+0xb1/0x150
> > > [<0>] __x64_sys_mlockall+0x11c/0x190
> > > [<0>] do_syscall_64+0x5a/0x130
> > > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > >
> > > [<0>] __do_fault+0x3c/0x130
> > > [<0>] do_fault+0x248/0x640
> > > [<0>] __handle_mm_fault+0x4c5/0x7a0
> > > [<0>] handle_mm_fault+0xca/0x1f0
> > > [<0>] __get_user_pages+0x230/0x770
> > > [<0>] populate_vma_page_range+0x74/0x80
> > > [<0>] __mm_populate+0xb1/0x150
> > > [<0>] __x64_sys_mlockall+0x11c/0x190
> > > [<0>] do_syscall_64+0x5a/0x130
> > > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> >
> > This is expected.
> >
> > > // doubt this one is relevant
> > > [<0>] __wake_up_common_lock+0x7c/0xc0
> > > [<0>] __wake_up_sync_key+0x1e/0x30
> > > [<0>] __wake_up_parent+0x26/0x30
> > > [<0>] do_notify_parent+0x1cc/0x280
> > > [<0>] do_exit+0x703/0xaf0
> > > [<0>] do_group_exit+0x47/0xb0
> > > [<0>] get_signal+0x165/0x880
> > > [<0>] do_signal+0x34/0x280
> > > [<0>] exit_to_usermode_loop+0xbf/0x160
> > > [<0>] do_syscall_64+0x10f/0x130
> > > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > Hmm, this means that the task has exited so how come there are
> > other syscalls happening. Are you sure you are collecting stacks for
> > the
> > correct task?
> 
> I guess that the `cat /proc/$(pidof test)/stack` captured the stack
> after I hit ctrl-c. Does that make sense?
> 
> Also tried `syscall(SYS_mlockall, MCL_CURRENT);` instead of
> `mlockall(MCL_CURRENT)` - same behavior.
 
This smells like something that could be runtime specific. Could you
post strace output of your testcase?

-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 12:05           ` Michal Hocko
@ 2019-10-25 12:11             ` Michal Hocko
  2019-10-25 13:10               ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Michal Hocko @ 2019-10-25 12:11 UTC (permalink / raw)
  To: snazy; +Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri 25-10-19 14:05:05, Michal Hocko wrote:
[...]
> This smells like something that could be runtime specific. Could you
> post strace output of your testcase?

Ohh and /proc/<pid>/smaps while the strace is hung on mlockall please.
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 12:11             ` Michal Hocko
@ 2019-10-25 13:10               ` Robert Stupp
  2019-10-25 13:27                 ` Michal Hocko
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 13:10 UTC (permalink / raw)
  To: Michal Hocko, snazy
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 14:11 +0200, Michal Hocko wrote:
> On Fri 25-10-19 14:05:05, Michal Hocko wrote:
> [...]
> > This smells like something that could be runtime specific. Could
> > you
> > post strace output of your testcase?
>
> Ohh and /proc/<pid>/smaps while the strace is hung on mlockall
> please.

Sure, see below.


strace ./test
execve("./test", ["./test"], 0x7ffd6ba0f970 /* 89 vars */) = 0
brk(NULL)                               = 0x564ee2c79000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd58f1d460) = -1 EINVAL (Invalid
argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=176037, ...}) = 0
mmap(NULL, 176037, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8be92bb000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC)
= 3
read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360r\2\0\0\0\0\0"...,
832) = 832
lseek(3, 64, SEEK_SET)                  = 64
read(3,
"\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"...,
784) = 784
lseek(3, 848, SEEK_SET)                 = 848
read(3,
"\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32)
= 32
lseek(3, 880, SEEK_SET)                 = 880
read(3,
"\4\0\0\0\24\0\0\0\3\0\0\0GNU\0!U\364U\255V\275\207\34\202%\274\312\205
\356%"..., 68) = 68
fstat(3, {st_mode=S_IFREG|0755, st_size=2025032, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f8be92b9000
lseek(3, 64, SEEK_SET)                  = 64
read(3,
"\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"...,
784) = 784
lseek(3, 848, SEEK_SET)                 = 848
read(3,
"\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32)
= 32
lseek(3, 880, SEEK_SET)                 = 880
read(3,
"\4\0\0\0\24\0\0\0\3\0\0\0GNU\0!U\364U\255V\275\207\34\202%\274\312\205
\356%"..., 68) = 68
mmap(NULL, 2032984, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f8be90c8000
mmap(0x7f8be90ed000, 1540096, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7f8be90ed000
mmap(0x7f8be9265000, 303104, PROT_READ,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19d000) = 0x7f8be9265000
mmap(0x7f8be92af000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e6000) = 0x7f8be92af000
mmap(0x7f8be92b5000, 13656, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8be92b5000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f8be92ba540) = 0
mprotect(0x7f8be92af000, 12288, PROT_READ) = 0
mprotect(0x564ee18c8000, 4096, PROT_READ) = 0
mprotect(0x7f8be9312000, 4096, PROT_READ) = 0
munmap(0x7f8be92bb000, 176037)          = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0xa), ...}) = 0
brk(NULL)                               = 0x564ee2c79000
brk(0x564ee2c9a000)                     = 0x564ee2c9a000
write(1, "Before mlockall(MCL_CURRENT)\n", 29Before
mlockall(MCL_CURRENT)
) = 29
mlockall(MCL_CURRENT

^ the strace stops there (no closing bracket)

Unlike non-strace runs, I've got to do a `kill -9 $(pidof test)`



cat /proc/$(pidof test)/smaps
564ee18c5000-564ee18c6000 r--p 00000000 103:02
49174659                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
564ee18c6000-564ee18c7000 r-xp 00001000 103:02
49174659                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd ex mr mw me dw lo sd
564ee18c7000-564ee18c8000 r--p 00002000 103:02
49174659                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
564ee18c8000-564ee18c9000 r--p 00002000 103:02
49174659                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo ac sd
564ee18c9000-564ee18ca000 rw-p 00003000 103:02
49174659                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me dw lo ac sd
564ee2c79000-564ee2c9a000 rw-p 00000000 00:00
0                          [heap]
Size:                132 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 132 kB
Pss:                 132 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:       132 kB
Referenced:          132 kB
Anonymous:           132 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:              132 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7f8be90c8000-7f8be90ed000 r--p 00000000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                148 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 144 kB
Pss:                   0 kB
Shared_Clean:        144 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          144 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me lo sd
7f8be90ed000-7f8be9265000 r-xp 00025000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:               1504 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 832 kB
Pss:                   5 kB
Shared_Clean:        832 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          832 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                5 kB
THPeligible:		0
VmFlags: rd ex mr mw me lo sd
7f8be9265000-7f8be92af000 r--p 0019d000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                296 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  64 kB
Pss:                   0 kB
Shared_Clean:         64 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           64 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me lo sd
7f8be92af000-7f8be92b2000 r--p 001e6000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               12 kB
THPeligible:		0
VmFlags: rd mr mw me lo ac sd
7f8be92b2000-7f8be92b5000 rw-p 001e9000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               12 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7f8be92b5000-7f8be92bb000 rw-p 00000000 00:00 0
Size:                 24 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  24 kB
Pss:                  24 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        24 kB
Referenced:           24 kB
Anonymous:            24 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               24 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7f8be92e6000-7f8be92e7000 r--p 00000000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
7f8be92e7000-7f8be9309000 r-xp 00001000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                136 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 136 kB
Pss:                   0 kB
Shared_Clean:        136 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          136 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me dw lo sd
7f8be9309000-7f8be9311000 r--p 00023000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                 32 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  32 kB
Pss:                   0 kB
Shared_Clean:         32 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           32 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
7f8be9312000-7f8be9313000 r--p 0002b000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo ac sd
7f8be9313000-7f8be9314000 rw-p 0002c000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me dw lo ac sd
7f8be9314000-7f8be9315000 rw-p 00000000 00:00 0
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7ffd58efe000-7ffd58f20000 rw-p 00000000 00:00
0                          [stack]
Size:                136 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  20 kB
Pss:                  20 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        20 kB
Referenced:           20 kB
Anonymous:            20 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               20 kB
THPeligible:		0
VmFlags: rd wr mr mw me gd lo ac
7ffd58f74000-7ffd58f77000 r--p 00000000 00:00
0                          [vvar]
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr pf io de dd sd
7ffd58f77000-7ffd58f78000 r-xp 00000000 00:00
0                          [vdso]
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me de sd
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00
0                  [vsyscall]
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: ex




^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 13:10               ` Robert Stupp
@ 2019-10-25 13:27                 ` Michal Hocko
  2019-10-25 13:45                   ` Robert Stupp
  2019-10-25 13:55                   ` Robert Stupp
  0 siblings, 2 replies; 39+ messages in thread
From: Michal Hocko @ 2019-10-25 13:27 UTC (permalink / raw)
  To: snazy; +Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri 25-10-19 15:10:39, Robert Stupp wrote:
[...]
> cat /proc/$(pidof test)/smaps

Nothing really unusual that would jump at me except for
> 7f8be90ed000-7f8be9265000 r-xp 00025000 103:02
> 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
> Size:               1504 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> Rss:                 832 kB
> Pss:                   5 kB
> Shared_Clean:        832 kB
> Shared_Dirty:          0 kB
> Private_Clean:         0 kB
> Private_Dirty:         0 kB
> Referenced:          832 kB
> Anonymous:             0 kB
> LazyFree:              0 kB
> AnonHugePages:         0 kB
> ShmemPmdMapped:        0 kB
> Shared_Hugetlb:        0 kB
> Private_Hugetlb:       0 kB
> Swap:                  0 kB
> SwapPss:               0 kB
> Locked:                5 kB

Huh, 5kB, is this really the case or some copy&paste error?
How can we end up with !pagesize multiple here?

> THPeligible:		0
> VmFlags: rd ex mr mw me lo sd
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 13:27                 ` Michal Hocko
@ 2019-10-25 13:45                   ` Robert Stupp
  2019-10-25 13:57                     ` Michal Hocko
  2019-10-25 13:55                   ` Robert Stupp
  1 sibling, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 13:45 UTC (permalink / raw)
  To: Michal Hocko, snazy
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 15:27 +0200, Michal Hocko wrote:
> On Fri 25-10-19 15:10:39, Robert Stupp wrote:
> [...]
> > cat /proc/$(pidof test)/smaps
>
> Nothing really unusual that would jump at me except for
> > 7f8be90ed000-7f8be9265000 r-xp 00025000 103:02
> > 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
> > Size:               1504 kB
> > KernelPageSize:        4 kB
> > MMUPageSize:           4 kB
> > Rss:                 832 kB
> > Pss:                   5 kB
> > Shared_Clean:        832 kB
> > Shared_Dirty:          0 kB
> > Private_Clean:         0 kB
> > Private_Dirty:         0 kB
> > Referenced:          832 kB
> > Anonymous:             0 kB
> > LazyFree:              0 kB
> > AnonHugePages:         0 kB
> > ShmemPmdMapped:        0 kB
> > Shared_Hugetlb:        0 kB
> > Private_Hugetlb:       0 kB
> > Swap:                  0 kB
> > SwapPss:               0 kB
> > Locked:                5 kB
>
> Huh, 5kB, is this really the case or some copy&paste error?
> How can we end up with !pagesize multiple here?
>
> > THPeligible:		0
> > VmFlags: rd ex mr mw me lo sd

mlockall() seems to lock everything though, it just never returns.

Pretty sure that it's not a copy&paste error. Got a couple more runs
that have an "odd size" - this time with 3kB...
All "Locked" values seem to be okay - except that one. And it's always
odd for the same one (the one with `Size: 1504 kB`).
It's not always odd (3 kB or 5 kB) though - sometimes it says 4 kB.
Seems it's a little breadcrumb?


560429953000-560429954000 r--p 00000000 103:02
49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
560429954000-560429955000 r-xp 00001000 103:02
49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd ex mr mw me dw lo sd
560429955000-560429956000 r--p 00002000 103:02
49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
560429956000-560429957000 r--p 00002000 103:02
49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo ac sd
560429957000-560429958000 rw-p 00003000 103:02
49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me dw lo ac sd
5604299e7000-560429a08000 rw-p 00000000 00:00
0                          [heap]
Size:                132 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 132 kB
Pss:                 132 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:       132 kB
Referenced:          132 kB
Anonymous:           132 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:              132 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7f243057c000-7f24305a1000 r--p 00000000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                148 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 144 kB
Pss:                   0 kB
Shared_Clean:        144 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          144 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me lo sd
7f24305a1000-7f2430719000 r-xp 00025000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:               1504 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 816 kB
Pss:                   3 kB
Shared_Clean:        816 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          816 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                3 kB
THPeligible:		0
VmFlags: rd ex mr mw me lo sd
7f2430719000-7f2430763000 r--p 0019d000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                296 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 128 kB
Pss:                   0 kB
Shared_Clean:        128 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          128 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me lo sd
7f2430763000-7f2430766000 r--p 001e6000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               12 kB
THPeligible:		0
VmFlags: rd mr mw me lo ac sd
7f2430766000-7f2430769000 rw-p 001e9000 103:02
44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               12 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7f2430769000-7f243076f000 rw-p 00000000 00:00 0
Size:                 24 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  24 kB
Pss:                  24 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        24 kB
Referenced:           24 kB
Anonymous:            24 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               24 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7f243079a000-7f243079b000 r--p 00000000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
7f243079b000-7f24307bd000 r-xp 00001000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                136 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 136 kB
Pss:                   0 kB
Shared_Clean:        136 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          136 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me dw lo sd
7f24307bd000-7f24307c5000 r--p 00023000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                 32 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  32 kB
Pss:                   0 kB
Shared_Clean:         32 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           32 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd
7f24307c6000-7f24307c7000 r--p 0002b000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo ac sd
7f24307c7000-7f24307c8000 rw-p 0002c000 103:02
44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me dw lo ac sd
7f24307c8000-7f24307c9000 rw-p 00000000 00:00 0
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd
7ffe7d4d0000-7ffe7d4f2000 rw-p 00000000 00:00
0                          [stack]
Size:                136 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  20 kB
Pss:                  20 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        20 kB
Referenced:           20 kB
Anonymous:            20 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               20 kB
THPeligible:		0
VmFlags: rd wr mr mw me gd lo ac
7ffe7d502000-7ffe7d505000 r--p 00000000 00:00
0                          [vvar]
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr pf io de dd sd
7ffe7d505000-7ffe7d506000 r-xp 00000000 00:00
0                          [vdso]
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me de sd
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00
0                  [vsyscall]
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: ex




^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 13:45                   ` Robert Stupp
@ 2019-10-25 13:57                     ` Michal Hocko
  2019-10-25 14:00                       ` Michal Hocko
  0 siblings, 1 reply; 39+ messages in thread
From: Michal Hocko @ 2019-10-25 13:57 UTC (permalink / raw)
  To: snazy; +Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri 25-10-19 15:45:37, Robert Stupp wrote:
> On Fri, 2019-10-25 at 15:27 +0200, Michal Hocko wrote:
> > On Fri 25-10-19 15:10:39, Robert Stupp wrote:
> > [...]
> > > cat /proc/$(pidof test)/smaps
> >
> > Nothing really unusual that would jump at me except for
> > > 7f8be90ed000-7f8be9265000 r-xp 00025000 103:02
> > > 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
> > > Size:               1504 kB
> > > KernelPageSize:        4 kB
> > > MMUPageSize:           4 kB
> > > Rss:                 832 kB
> > > Pss:                   5 kB
> > > Shared_Clean:        832 kB
> > > Shared_Dirty:          0 kB
> > > Private_Clean:         0 kB
> > > Private_Dirty:         0 kB
> > > Referenced:          832 kB
> > > Anonymous:             0 kB
> > > LazyFree:              0 kB
> > > AnonHugePages:         0 kB
> > > ShmemPmdMapped:        0 kB
> > > Shared_Hugetlb:        0 kB
> > > Private_Hugetlb:       0 kB
> > > Swap:                  0 kB
> > > SwapPss:               0 kB
> > > Locked:                5 kB
> >
> > Huh, 5kB, is this really the case or some copy&paste error?
> > How can we end up with !pagesize multiple here?

Ohh, I haven't noticed pss and didn't realize that Locked is pss like as
well.

> >
> > > THPeligible:		0
> > > VmFlags: rd ex mr mw me lo sd
> 
> mlockall() seems to lock everything though, it just never returns.
> 
> Pretty sure that it's not a copy&paste error. Got a couple more runs
> that have an "odd size" - this time with 3kB...
> All "Locked" values seem to be okay - except that one. And it's always
> odd for the same one (the one with `Size: 1504 kB`).
> It's not always odd (3 kB or 5 kB) though - sometimes it says 4 kB.
> Seems it's a little breadcrumb?

Please try to watch for stack of the syscall and see if there is any
pattern.
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 13:57                     ` Michal Hocko
@ 2019-10-25 14:00                       ` Michal Hocko
  2019-10-25 15:58                         ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Michal Hocko @ 2019-10-25 14:00 UTC (permalink / raw)
  To: snazy; +Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

And one more thing. Considering that you are able to reproduce and you
have a working kernel, could you try to bisect this?
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 14:00                       ` Michal Hocko
@ 2019-10-25 15:58                         ` Robert Stupp
  2019-11-05 13:23                           ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 15:58 UTC (permalink / raw)
  To: Michal Hocko, snazy
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 16:00 +0200, Michal Hocko wrote:
> And one more thing. Considering that you are able to reproduce and
> you
> have a working kernel, could you try to bisect this?

Yikes - running self-built kernels brings back a lot of memories ;)

Anyway, going this route (using the `config` from Ubuntu 5.1.x as a
base and accepting the defaults for `make oldconfig`):

git checkout v5.1-rc1
git bisect start
git bisect bad
git bisect good v5.0

... first try @ e266ca36da7de45b64b05698e98e04b578a88888 is a `git
bisect good`

Will report back, when I've got a result...




^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 15:58                         ` Robert Stupp
@ 2019-11-05 13:23                           ` Robert Stupp
  2019-11-05 15:28                             ` Vlastimil Babka
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-11-05 13:23 UTC (permalink / raw)
  To: Michal Hocko
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

"git bisect" led to a result.

The offending merge commit is f91f2ee54a21404fbc633550e99d69d14c2478f2
"Merge branch 'akpm' (rest of patches from Andrew)".

The first bad commit in the merged series of commits is
https://github.com/torvalds/linux/commit/6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
. a75d4c33377277b6034dd1e2663bce444f952c14, the commit before 6b4c9f44,
is good.

I've also verified 5.1.21 and 5.3.8 (from
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/)
without 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11 and both builds are
good.
(The 5.1.21 and 5.3.7 builds from Ubuntu were bad, so I haven't cross-
checked "vanilla" 5.1.21 and 5.3.8 kernel builds.)



Recap symptoms:
- `mlockall(MCL_CURRENT)` hangs
- shutdown/reboot hangs when it reaches "shutdown->reboot"
- `cat /proc/$(pidof test)/smaps` shows "Locked" w/ odd values, which
are equal to "Pss"

Affected:
- `cryptsetup luksOpen` hangs (when it tries to lock memory)
- Apache Cassandra hangs during startup (when it performs an
`mlockall(MCL_CURRENT)`)



git checkout v5.1.21
# revert the "comment-only" commit (no need to test this one)
# "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
# "filemap: drop the mmap_sem for all blocking operations"
git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
--> GOOD

git checkout v5.3.8
# revert the "comment-only" commit (no need to test this one)
# "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
# "filemap: drop the mmap_sem for all blocking operations"
git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
--> GOOD



Bisect log:
git bisect start
# bad: [9e98c678c2d6ae3a17cb2de55d17f69dddaa231b] Linux 5.1-rc1
git bisect bad 9e98c678c2d6ae3a17cb2de55d17f69dddaa231b
# good: [1c163f4c7b3f621efff9b28a47abb36f7378d783] Linux 5.0
git bisect good 1c163f4c7b3f621efff9b28a47abb36f7378d783
# good: [e266ca36da7de45b64b05698e98e04b578a88888] Merge tag 'staging-
5.1-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect good e266ca36da7de45b64b05698e98e04b578a88888
# good: [36011ddc78395b59a8a418c37f20bcc18828f1ef] Merge tag 'gfs2-
5.1.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-
gfs2
git bisect good 36011ddc78395b59a8a418c37f20bcc18828f1ef
# good: [6bc3fe8e7e172d5584e529a04cf9eec946428768] tools: mark
'test_vmalloc.sh' executable
git bisect good 6bc3fe8e7e172d5584e529a04cf9eec946428768
# good: [dc2535be1fd547fbd56aff091370280007b0a1af] Merge tag 'clk-for-
linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux
git bisect good dc2535be1fd547fbd56aff091370280007b0a1af
# bad: [2b9c272cf5cd81708e51b4ce3e432ce9566cfa47] Merge tag 'fbdev-
v5.1' of git://github.com/bzolnier/linux
git bisect bad 2b9c272cf5cd81708e51b4ce3e432ce9566cfa47
# good: [9bc446100334dbbc14eb3757274ef08746c3f9bd] Merge tag
'microblaze-v5.1-rc1' of git://git.monstr.eu/linux-2.6-microblaze
git bisect good 9bc446100334dbbc14eb3757274ef08746c3f9bd
# bad: [5160bcce5c3c80de7d8722511c144d3041409657] Merge tag 'f2fs-for-
5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
git bisect bad 5160bcce5c3c80de7d8722511c144d3041409657
# good: [240a59156d9bcfabceddb66be449e7b32fb5dc4a] f2fs: fix to add
refcount once page is tagged PG_private
git bisect good 240a59156d9bcfabceddb66be449e7b32fb5dc4a
# good: [9352ca585b2ac7b67d2119b9386573b2a4c0ef4b] Merge tag 'pm-5.1-
rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect good 9352ca585b2ac7b67d2119b9386573b2a4c0ef4b
# good: [f261c4e529dac5608a604d3dd3ae1cd2adf23c89] Merge branch 'akpm'
(patches from Andrew)
git bisect good f261c4e529dac5608a604d3dd3ae1cd2adf23c89
# good: [aadcef64b22f668c1a107b86d3521d9cac915c24] f2fs: fix to avoid
deadlock in f2fs_read_inline_dir()
git bisect good aadcef64b22f668c1a107b86d3521d9cac915c24
# bad: [8b0f9fa2e02dc95216577c3387b0707c5f60fbaf] filemap: add a
comment about FAULT_FLAG_RETRY_NOWAIT behavior
git bisect bad 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
# bad: [6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11] filemap: drop the
mmap_sem for all blocking operations
git bisect bad 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
# bad: [a75d4c33377277b6034dd1e2663bce444f952c14] filemap: kill
page_cache_read usage in filemap_fault
git bisect good a75d4c33377277b6034dd1e2663bce444f952c14


All kernels built with
make oldconfig # accept the defaults
make bindeb-pkg



On Fri, 2019-10-25 at 17:58 +0200, Robert Stupp wrote:
> On Fri, 2019-10-25 at 16:00 +0200, Michal Hocko wrote:
> > And one more thing. Considering that you are able to reproduce and
> > you
> > have a working kernel, could you try to bisect this?
>
> Yikes - running self-built kernels brings back a lot of memories ;)
>
> Anyway, going this route (using the `config` from Ubuntu 5.1.x as a
> base and accepting the defaults for `make oldconfig`):
>
> git checkout v5.1-rc1
> git bisect start
> git bisect bad
> git bisect good v5.0
>
> ... first try @ e266ca36da7de45b64b05698e98e04b578a88888 is a `git
> bisect good`
>
> Will report back, when I've got a result...
>



^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-05 13:23                           ` Robert Stupp
@ 2019-11-05 15:28                             ` Vlastimil Babka
  2019-11-05 18:22                               ` Johannes Weiner
  0 siblings, 1 reply; 39+ messages in thread
From: Vlastimil Babka @ 2019-11-05 15:28 UTC (permalink / raw)
  To: snazy, Michal Hocko, Josef Bacik, Johannes Weiner, Jan Kara,
	Kirill A. Shutemov
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On 11/5/19 2:23 PM, Robert Stupp wrote:
> "git bisect" led to a result.
> 
> The offending merge commit is f91f2ee54a21404fbc633550e99d69d14c2478f2
> "Merge branch 'akpm' (rest of patches from Andrew)".
> 
> The first bad commit in the merged series of commits is
> https://github.com/torvalds/linux/commit/6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> . a75d4c33377277b6034dd1e2663bce444f952c14, the commit before 6b4c9f44,
> is good.

Ah, great you could bisect this. CCing people from the commit
6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

First mail in thread:
https://lore.kernel.org/linux-mm/b8ff71f5-2d9c-7ebb-d621-017d4b9bc932@infradead.org/

> I've also verified 5.1.21 and 5.3.8 (from
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/)
> without 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11 and both builds are
> good.
> (The 5.1.21 and 5.3.7 builds from Ubuntu were bad, so I haven't cross-
> checked "vanilla" 5.1.21 and 5.3.8 kernel builds.)
> 
> 
> 
> Recap symptoms:
> - `mlockall(MCL_CURRENT)` hangs
> - shutdown/reboot hangs when it reaches "shutdown->reboot"
> - `cat /proc/$(pidof test)/smaps` shows "Locked" w/ odd values, which
> are equal to "Pss"
> 
> Affected:
> - `cryptsetup luksOpen` hangs (when it tries to lock memory)
> - Apache Cassandra hangs during startup (when it performs an
> `mlockall(MCL_CURRENT)`)
> 
> 
> 
> git checkout v5.1.21
> # revert the "comment-only" commit (no need to test this one)
> # "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
> git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
> # "filemap: drop the mmap_sem for all blocking operations"
> git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> --> GOOD
> 
> git checkout v5.3.8
> # revert the "comment-only" commit (no need to test this one)
> # "filemap: add a comment about FAULT_FLAG_RETRY_NOWAIT behavior"
> git revert 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
> # "filemap: drop the mmap_sem for all blocking operations"
> git revert 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> --> GOOD
> 
> 
> 
> Bisect log:
> git bisect start
> # bad: [9e98c678c2d6ae3a17cb2de55d17f69dddaa231b] Linux 5.1-rc1
> git bisect bad 9e98c678c2d6ae3a17cb2de55d17f69dddaa231b
> # good: [1c163f4c7b3f621efff9b28a47abb36f7378d783] Linux 5.0
> git bisect good 1c163f4c7b3f621efff9b28a47abb36f7378d783
> # good: [e266ca36da7de45b64b05698e98e04b578a88888] Merge tag 'staging-
> 5.1-rc1' of
> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect good e266ca36da7de45b64b05698e98e04b578a88888
> # good: [36011ddc78395b59a8a418c37f20bcc18828f1ef] Merge tag 'gfs2-
> 5.1.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-
> gfs2
> git bisect good 36011ddc78395b59a8a418c37f20bcc18828f1ef
> # good: [6bc3fe8e7e172d5584e529a04cf9eec946428768] tools: mark
> 'test_vmalloc.sh' executable
> git bisect good 6bc3fe8e7e172d5584e529a04cf9eec946428768
> # good: [dc2535be1fd547fbd56aff091370280007b0a1af] Merge tag 'clk-for-
> linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux
> git bisect good dc2535be1fd547fbd56aff091370280007b0a1af
> # bad: [2b9c272cf5cd81708e51b4ce3e432ce9566cfa47] Merge tag 'fbdev-
> v5.1' of git://github.com/bzolnier/linux
> git bisect bad 2b9c272cf5cd81708e51b4ce3e432ce9566cfa47
> # good: [9bc446100334dbbc14eb3757274ef08746c3f9bd] Merge tag
> 'microblaze-v5.1-rc1' of git://git.monstr.eu/linux-2.6-microblaze
> git bisect good 9bc446100334dbbc14eb3757274ef08746c3f9bd
> # bad: [5160bcce5c3c80de7d8722511c144d3041409657] Merge tag 'f2fs-for-
> 5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
> git bisect bad 5160bcce5c3c80de7d8722511c144d3041409657
> # good: [240a59156d9bcfabceddb66be449e7b32fb5dc4a] f2fs: fix to add
> refcount once page is tagged PG_private
> git bisect good 240a59156d9bcfabceddb66be449e7b32fb5dc4a
> # good: [9352ca585b2ac7b67d2119b9386573b2a4c0ef4b] Merge tag 'pm-5.1-
> rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
> git bisect good 9352ca585b2ac7b67d2119b9386573b2a4c0ef4b
> # good: [f261c4e529dac5608a604d3dd3ae1cd2adf23c89] Merge branch 'akpm'
> (patches from Andrew)
> git bisect good f261c4e529dac5608a604d3dd3ae1cd2adf23c89
> # good: [aadcef64b22f668c1a107b86d3521d9cac915c24] f2fs: fix to avoid
> deadlock in f2fs_read_inline_dir()
> git bisect good aadcef64b22f668c1a107b86d3521d9cac915c24
> # bad: [8b0f9fa2e02dc95216577c3387b0707c5f60fbaf] filemap: add a
> comment about FAULT_FLAG_RETRY_NOWAIT behavior
> git bisect bad 8b0f9fa2e02dc95216577c3387b0707c5f60fbaf
> # bad: [6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11] filemap: drop the
> mmap_sem for all blocking operations
> git bisect bad 6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> # bad: [a75d4c33377277b6034dd1e2663bce444f952c14] filemap: kill
> page_cache_read usage in filemap_fault
> git bisect good a75d4c33377277b6034dd1e2663bce444f952c14
> 
> 
> All kernels built with
> make oldconfig # accept the defaults
> make bindeb-pkg
> 
> 
> 
> On Fri, 2019-10-25 at 17:58 +0200, Robert Stupp wrote:
>> On Fri, 2019-10-25 at 16:00 +0200, Michal Hocko wrote:
>>> And one more thing. Considering that you are able to reproduce and
>>> you
>>> have a working kernel, could you try to bisect this?
>>
>> Yikes - running self-built kernels brings back a lot of memories ;)
>>
>> Anyway, going this route (using the `config` from Ubuntu 5.1.x as a
>> base and accepting the defaults for `make oldconfig`):
>>
>> git checkout v5.1-rc1
>> git bisect start
>> git bisect bad
>> git bisect good v5.0
>>
>> ... first try @ e266ca36da7de45b64b05698e98e04b578a88888 is a `git
>> bisect good`
>>
>> Will report back, when I've got a result...
>>
> 
> 



^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-05 15:28                             ` Vlastimil Babka
@ 2019-11-05 18:22                               ` Johannes Weiner
  2019-11-05 20:05                                 ` Robert Stupp
  2019-11-06 12:03                                 ` Jan Kara
  0 siblings, 2 replies; 39+ messages in thread
From: Johannes Weiner @ 2019-11-05 18:22 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: snazy, Michal Hocko, Josef Bacik, Jan Kara, Kirill A. Shutemov,
	Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra,
	Stefan

On Tue, Nov 05, 2019 at 04:28:21PM +0100, Vlastimil Babka wrote:
> On 11/5/19 2:23 PM, Robert Stupp wrote:
> > "git bisect" led to a result.
> > 
> > The offending merge commit is f91f2ee54a21404fbc633550e99d69d14c2478f2
> > "Merge branch 'akpm' (rest of patches from Andrew)".
> > 
> > The first bad commit in the merged series of commits is
> > https://github.com/torvalds/linux/commit/6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> > . a75d4c33377277b6034dd1e2663bce444f952c14, the commit before 6b4c9f44,
> > is good.
> 
> Ah, great you could bisect this. CCing people from the commit
> 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

Judging from Robert's stack captures, the task is not hung but
busy-looping in __mm_populate(). AFAICS, the only way this can occur
is if populate_vma_page_range() returns 0 and we don't advance the
iteration position (if it returned an error, we wouldn't reset nend
and move on to the next vma as ignore_errors is 1 for mlockall.)

populate_vma_page_range() returns 0 when the first page is not found
and faultin_page() returns -EBUSY (if it were processing pages, or if
the error from faultin_page() would be a different one, we would
return the number of pages processed or -error).

faultin_page() returns -EBUSY when VM_FAULT_RETRY is set, i.e. we
dropped the mmap_sem in order to initiate IO and require a retry. That
is consistent with the bisect result (new VM_FAULT_RETRY conditions).

At this point, regular page fault would retry with FAULT_FLAG_TRIED to
indicate that the mmap_sem cannot be dropped a second time. But this
mlock path doesn't set that flag and we can loop repeatedly. That is
something we probably need to fix with a FOLL_TRIED somewhere.

What I don't quite understand yet is why the fault path doesn't make
progress eventually. We must drop the mmap_sem without changing the
state in any way. How can we keep looping on the same page?

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-05 18:22                               ` Johannes Weiner
@ 2019-11-05 20:05                                 ` Robert Stupp
  2019-11-06 10:25                                   ` Robert Stupp
  2019-11-06 12:03                                 ` Jan Kara
  1 sibling, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-11-05 20:05 UTC (permalink / raw)
  To: Johannes Weiner, Vlastimil Babka
  Cc: Michal Hocko, Josef Bacik, Jan Kara, Kirill A. Shutemov,
	Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra,
	Stefan

[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]

On Tue, 2019-11-05 at 13:22 -0500, Johannes Weiner wrote:
> Judging from Robert's stack captures, the task is not hung but
> busy-looping in __mm_populate(). AFAICS, the only way this can occur
> is if populate_vma_page_range() returns 0 and we don't advance the
> iteration position (if it returned an error, we wouldn't reset nend
> and move on to the next vma as ignore_errors is 1 for mlockall.)
>
> populate_vma_page_range() returns 0 when the first page is not found
> and faultin_page() returns -EBUSY (if it were processing pages, or if
> the error from faultin_page() would be a different one, we would
> return the number of pages processed or -error).
>
> faultin_page() returns -EBUSY when VM_FAULT_RETRY is set, i.e. we
> dropped the mmap_sem in order to initiate IO and require a retry.
> That
> is consistent with the bisect result (new VM_FAULT_RETRY conditions).
>
> At this point, regular page fault would retry with FAULT_FLAG_TRIED
> to
> indicate that the mmap_sem cannot be dropped a second time. But this
> mlock path doesn't set that flag and we can loop repeatedly. That is
> something we probably need to fix with a FOLL_TRIED somewhere.
>
> What I don't quite understand yet is why the fault path doesn't make
> progress eventually. We must drop the mmap_sem without changing the
> state in any way. How can we keep looping on the same page?

I've played a bit around by adding some `printk` messages (see attached
patch) and found exactly what you describe: it's busy-looping in
__mm_populate(), because populate_vma_page_range returns 0.

However, there's a slightly interesting thing in there. Before it loops
forever, it processes
	nstart=5574d92e1000
	locked=1
	vma->vm_start=7f5e4bfec000
	vma->vm_end=  7f5e4c011000
	vma->vm_flags=8002071
for which populate_vma_page_range() returns 1, then it processes this
over and over again:
	nstart=7f5e4bfed000
	locked=0
	vma->vm_start=7f5e4bfec000  (same as before)
	vma->vm_end=  7f5e4c011000
	vma->vm_flags=8002071
These are the additional dmesg messages with timestamp 105.x. At
timestamp 106.x, I've hit ctrl-c (ret=-512).

dmesg output with the patch applied (on top of the v5.3.8 git tag)
attached.


[-- Attachment #2: gup-printk.txt --]
[-- Type: text/plain, Size: 3310 bytes --]

diff --git a/mm/gup.c b/mm/gup.c
index 98f13ab37bac..3bc25fd44433 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -3,6 +3,7 @@
 #include <linux/errno.h>
 #include <linux/err.h>
 #include <linux/spinlock.h>
+#include <linux/printk.h>
 
 #include <linux/mm.h>
 #include <linux/memremap.h>
@@ -1241,14 +1242,23 @@ long populate_vma_page_range(struct vm_area_struct *vma,
 int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 {
 	struct mm_struct *mm = current->mm;
-	unsigned long end, nstart, nend;
+	unsigned long end, nstart, nend = 0L;
 	struct vm_area_struct *vma = NULL;
 	int locked = 0;
 	long ret = 0;
+	unsigned long nstart_prev = 0L - 1L, nend_prev = 0L - 1L;
+	int ign;
 
 	end = start + len;
 
+	printk(KERN_WARNING "_mm_populate %lx %lx %lx %d ENTER\n", start, len, end, ignore_errors);
+
 	for (nstart = start; nstart < end; nstart = nend) {
+	    ign = nstart == nstart_prev && nend == nend_prev;
+	    nstart_prev = nstart;
+	    nend_prev = nend;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP %lx %d %ld\n", start, len, end, ignore_errors, nstart, locked, ret);
 		/*
 		 * We want to fault in pages for [nstart; end) address range.
 		 * Find first corresponding VMA.
@@ -1259,6 +1269,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			vma = find_vma(mm, nstart);
 		} else if (nstart >= vma->vm_end)
 			vma = vma->vm_next;
+		if (!ign && vma)
+		    printk(KERN_WARNING "_mm_populate %lx %lx %lx %d vma->vm_start=%lx vma->vm_end=%lx vma->vm_flags=%lx\n", start, len, end, ignore_errors, vma->vm_start, vma->vm_end, vma->vm_flags);
 		if (!vma || vma->vm_start >= end)
 			break;
 		/*
@@ -1266,8 +1278,13 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 * range with the first VMA. Also, skip undesirable VMA types.
 		 */
 		nend = min(end, vma->vm_end);
-		if (vma->vm_flags & (VM_IO | VM_PFNMAP))
-			continue;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d nend=%lx %lx %lx\n", start, len, end, ignore_errors, nend, end, vma->vm_end);
+		if (vma->vm_flags & (VM_IO | VM_PFNMAP)) {
+            if (!ign)
+                printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-1 %lx\n", start, len, end, ignore_errors, vma->vm_flags);
+            continue;
+        }
 		if (nstart < vma->vm_start)
 			nstart = vma->vm_start;
 		/*
@@ -1277,6 +1294,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 */
 		ret = populate_vma_page_range(vma, nstart, nend, &locked);
 		if (ret < 0) {
+            if (!ign)
+                printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-2 %ld\n", start, len, end, ignore_errors, ret);
 			if (ignore_errors) {
 				ret = 0;
 				continue;	/* continue at next VMA */
@@ -1284,8 +1303,11 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			break;
 		}
 		nend = nstart + ret * PAGE_SIZE;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-3 ret=%ld nend=%lx\n", start, len, end, ignore_errors, ret, nend);
 		ret = 0;
 	}
+    printk(KERN_WARNING "_mm_populate END %lu %lu %d\n", start, len, locked);
 	if (locked)
 		up_read(&mm->mmap_sem);
 	return ret;	/* 0 or negative error code */

[-- Attachment #3: dmesg-out.txt.gz --]
[-- Type: application/gzip, Size: 25208 bytes --]

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-05 20:05                                 ` Robert Stupp
@ 2019-11-06 10:25                                   ` Robert Stupp
  2019-11-06 11:26                                     ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-11-06 10:25 UTC (permalink / raw)
  To: Johannes Weiner, Vlastimil Babka
  Cc: Michal Hocko, Josef Bacik, Jan Kara, Kirill A. Shutemov,
	Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra,
	Stefan

[-- Attachment #1: Type: text/plain, Size: 2791 bytes --]

Here's one more dmesg output with more information captured in
__get_user_pages() as well. It basically confirms that
handle_mm_fault() returns VM_FAULT_RETRY.

I'm not sure where and what to change ("fix with a FOLL_TRIED
somewhere") to make it work. My (uneducated) impression is, that only
__get_user_pages() needs to be changed - but I might be wrong.

On Tue, 2019-11-05 at 21:05 +0100, Robert Stupp wrote:
> On Tue, 2019-11-05 at 13:22 -0500, Johannes Weiner wrote:
> > Judging from Robert's stack captures, the task is not hung but
> > busy-looping in __mm_populate(). AFAICS, the only way this can
> > occur
> > is if populate_vma_page_range() returns 0 and we don't advance the
> > iteration position (if it returned an error, we wouldn't reset nend
> > and move on to the next vma as ignore_errors is 1 for mlockall.)
> >
> > populate_vma_page_range() returns 0 when the first page is not
> > found
> > and faultin_page() returns -EBUSY (if it were processing pages, or
> > if
> > the error from faultin_page() would be a different one, we would
> > return the number of pages processed or -error).
> >
> > faultin_page() returns -EBUSY when VM_FAULT_RETRY is set, i.e. we
> > dropped the mmap_sem in order to initiate IO and require a retry.
> > That
> > is consistent with the bisect result (new VM_FAULT_RETRY
> > conditions).
> >
> > At this point, regular page fault would retry with FAULT_FLAG_TRIED
> > to
> > indicate that the mmap_sem cannot be dropped a second time. But
> > this
> > mlock path doesn't set that flag and we can loop repeatedly. That
> > is
> > something we probably need to fix with a FOLL_TRIED somewhere.
> >
> > What I don't quite understand yet is why the fault path doesn't
> > make
> > progress eventually. We must drop the mmap_sem without changing the
> > state in any way. How can we keep looping on the same page?
>
> I've played a bit around by adding some `printk` messages (see
> attached
> patch) and found exactly what you describe: it's busy-looping in
> __mm_populate(), because populate_vma_page_range returns 0.
>
> However, there's a slightly interesting thing in there. Before it
> loops
> forever, it processes
> 	nstart=5574d92e1000
> 	locked=1
> 	vma->vm_start=7f5e4bfec000
> 	vma->vm_end=  7f5e4c011000
> 	vma->vm_flags=8002071
> for which populate_vma_page_range() returns 1, then it processes this
> over and over again:
> 	nstart=7f5e4bfed000
> 	locked=0
> 	vma->vm_start=7f5e4bfec000  (same as before)
> 	vma->vm_end=  7f5e4c011000
> 	vma->vm_flags=8002071
> These are the additional dmesg messages with timestamp 105.x. At
> timestamp 106.x, I've hit ctrl-c (ret=-512).
>
> dmesg output with the patch applied (on top of the v5.3.8 git tag)
> attached.
>

[-- Attachment #2: dmesg-out.txt --]
[-- Type: text/plain, Size: 32005 bytes --]

[   84.533231] _mm_populate 0 7ffffffff000 7ffffffff000 1 ENTER
[   84.533235] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 0 0 0
[   84.533238] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5626b4b8f000 vma->vm_end=5626b4b90000 vma->vm_flags=8002871
[   84.533240] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5626b4b90000 7ffffffff000 5626b4b90000
[   84.533242] __get_user_pages start=5626b4b8f000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   84.533243] __get_user_pages @1 vma==NULL
[   84.533245] __get_user_pages @2 start=5626b4b8f000 vma->vm_start=5626b4b8f000 vma->vm_end=5626b4b90000 vma->vm_flags=8002871
[   84.533250] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533251] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533253] __get_user_pages @10 i=1 start=5626b4b90000 nr_pages=0
[   84.533254] __get_user_pages LEAVE i=1 ret=0
[   84.533256] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5626b4b90000
[   84.533258] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5626b4b90000 1 0
[   84.533260] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5626b4b90000 vma->vm_end=5626b4b91000 vma->vm_flags=8002875
[   84.533262] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5626b4b91000 7ffffffff000 5626b4b91000
[   84.533263] __get_user_pages start=5626b4b90000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   84.533264] __get_user_pages @1 vma==NULL
[   84.533266] __get_user_pages @2 start=5626b4b90000 vma->vm_start=5626b4b90000 vma->vm_end=5626b4b91000 vma->vm_flags=8002875
[   84.533268] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533269] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533270] __get_user_pages @10 i=1 start=5626b4b91000 nr_pages=0
[   84.533271] __get_user_pages LEAVE i=1 ret=0
[   84.533273] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5626b4b91000
[   84.533275] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5626b4b91000 1 0
[   84.533276] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5626b4b91000 vma->vm_end=5626b4b92000 vma->vm_flags=8002871
[   84.533278] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5626b4b92000 7ffffffff000 5626b4b92000
[   84.533280] __get_user_pages start=5626b4b91000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   84.533280] __get_user_pages @1 vma==NULL
[   84.533282] __get_user_pages @2 start=5626b4b91000 vma->vm_start=5626b4b91000 vma->vm_end=5626b4b92000 vma->vm_flags=8002871
[   84.533284] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533285] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533286] __get_user_pages @10 i=1 start=5626b4b92000 nr_pages=0
[   84.533287] __get_user_pages LEAVE i=1 ret=0
[   84.533289] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5626b4b92000
[   84.533290] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5626b4b92000 1 0
[   84.533292] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5626b4b92000 vma->vm_end=5626b4b93000 vma->vm_flags=8102871
[   84.533294] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5626b4b93000 7ffffffff000 5626b4b93000
[   84.533295] __get_user_pages start=5626b4b92000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   84.533296] __get_user_pages @1 vma==NULL
[   84.533298] __get_user_pages @2 start=5626b4b92000 vma->vm_start=5626b4b92000 vma->vm_end=5626b4b93000 vma->vm_flags=8102871
[   84.533300] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533301] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533302] __get_user_pages @10 i=1 start=5626b4b93000 nr_pages=0
[   84.533303] __get_user_pages LEAVE i=1 ret=0
[   84.533304] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5626b4b93000
[   84.533306] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5626b4b93000 1 0
[   84.533308] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5626b4b93000 vma->vm_end=5626b4b94000 vma->vm_flags=8102873
[   84.533310] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5626b4b94000 7ffffffff000 5626b4b94000
[   84.533311] __get_user_pages start=5626b4b93000 nr_pages=1 gup_flags=1053 ctx.page_mask=0
[   84.533312] __get_user_pages @1 vma==NULL
[   84.533313] __get_user_pages @2 start=5626b4b93000 vma->vm_start=5626b4b93000 vma->vm_end=5626b4b94000 vma->vm_flags=8102873
[   84.533315] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533316] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533317] __get_user_pages @10 i=1 start=5626b4b94000 nr_pages=0
[   84.533318] __get_user_pages LEAVE i=1 ret=0
[   84.533320] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5626b4b94000
[   84.533322] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5626b4b94000 1 0
[   84.533324] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5626b5a54000 vma->vm_end=5626b5a75000 vma->vm_flags=8102073
[   84.533325] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5626b5a75000 7ffffffff000 5626b5a75000
[   84.533327] __get_user_pages start=5626b5a54000 nr_pages=33 gup_flags=1053 ctx.page_mask=0
[   84.533328] __get_user_pages @1 vma==NULL
[   84.533329] __get_user_pages @2 start=5626b5a54000 vma->vm_start=5626b5a54000 vma->vm_end=5626b5a75000 vma->vm_flags=8102073
[   84.533331] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533332] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533333] __get_user_pages @10 i=1 start=5626b5a55000 nr_pages=32
[   84.533335] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533337] faultin_page handle_mm_fault --> ret = 0
[   84.533338] faultin_page-->0
[   84.533339] __get_user_pages @8 faultin_page --> 0
[   84.533341] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533342] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533343] __get_user_pages @10 i=2 start=5626b5a56000 nr_pages=31
[   84.533344] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533346] faultin_page handle_mm_fault --> ret = 0
[   84.533347] faultin_page-->0
[   84.533348] __get_user_pages @8 faultin_page --> 0
[   84.533349] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533350] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533351] __get_user_pages @10 i=3 start=5626b5a57000 nr_pages=30
[   84.533353] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533354] faultin_page handle_mm_fault --> ret = 0
[   84.533355] faultin_page-->0
[   84.533356] __get_user_pages @8 faultin_page --> 0
[   84.533357] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533358] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533360] __get_user_pages @10 i=4 start=5626b5a58000 nr_pages=29
[   84.533361] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533362] faultin_page handle_mm_fault --> ret = 0
[   84.533363] faultin_page-->0
[   84.533364] __get_user_pages @8 faultin_page --> 0
[   84.533366] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533366] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533368] __get_user_pages @10 i=5 start=5626b5a59000 nr_pages=28
[   84.533369] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533371] faultin_page handle_mm_fault --> ret = 0
[   84.533371] faultin_page-->0
[   84.533372] __get_user_pages @8 faultin_page --> 0
[   84.533374] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533374] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533376] __get_user_pages @10 i=6 start=5626b5a5a000 nr_pages=27
[   84.533377] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533379] faultin_page handle_mm_fault --> ret = 0
[   84.533379] faultin_page-->0
[   84.533380] __get_user_pages @8 faultin_page --> 0
[   84.533382] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533382] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533384] __get_user_pages @10 i=7 start=5626b5a5b000 nr_pages=26
[   84.533385] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533387] faultin_page handle_mm_fault --> ret = 0
[   84.533387] faultin_page-->0
[   84.533388] __get_user_pages @8 faultin_page --> 0
[   84.533390] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533391] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533392] __get_user_pages @10 i=8 start=5626b5a5c000 nr_pages=25
[   84.533393] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533395] faultin_page handle_mm_fault --> ret = 0
[   84.533396] faultin_page-->0
[   84.533397] __get_user_pages @8 faultin_page --> 0
[   84.533398] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533399] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533401] __get_user_pages @10 i=9 start=5626b5a5d000 nr_pages=24
[   84.533402] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533404] faultin_page handle_mm_fault --> ret = 0
[   84.533404] faultin_page-->0
[   84.533405] __get_user_pages @8 faultin_page --> 0
[   84.533407] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533408] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533409] __get_user_pages @10 i=10 start=5626b5a5e000 nr_pages=23
[   84.533410] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533412] faultin_page handle_mm_fault --> ret = 0
[   84.533413] faultin_page-->0
[   84.533414] __get_user_pages @8 faultin_page --> 0
[   84.533415] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533416] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533417] __get_user_pages @10 i=11 start=5626b5a5f000 nr_pages=22
[   84.533419] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533421] faultin_page handle_mm_fault --> ret = 0
[   84.533421] faultin_page-->0
[   84.533422] __get_user_pages @8 faultin_page --> 0
[   84.533424] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533425] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533426] __get_user_pages @10 i=12 start=5626b5a60000 nr_pages=21
[   84.533427] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533429] faultin_page handle_mm_fault --> ret = 0
[   84.533430] faultin_page-->0
[   84.533430] __get_user_pages @8 faultin_page --> 0
[   84.533432] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533433] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533434] __get_user_pages @10 i=13 start=5626b5a61000 nr_pages=20
[   84.533435] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533437] faultin_page handle_mm_fault --> ret = 0
[   84.533438] faultin_page-->0
[   84.533439] __get_user_pages @8 faultin_page --> 0
[   84.533440] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533441] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533442] __get_user_pages @10 i=14 start=5626b5a62000 nr_pages=19
[   84.533443] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533445] faultin_page handle_mm_fault --> ret = 0
[   84.533446] faultin_page-->0
[   84.533447] __get_user_pages @8 faultin_page --> 0
[   84.533448] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533449] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533451] __get_user_pages @10 i=15 start=5626b5a63000 nr_pages=18
[   84.533452] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533454] faultin_page handle_mm_fault --> ret = 0
[   84.533455] faultin_page-->0
[   84.533455] __get_user_pages @8 faultin_page --> 0
[   84.533457] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533458] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533459] __get_user_pages @10 i=16 start=5626b5a64000 nr_pages=17
[   84.533460] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533462] faultin_page handle_mm_fault --> ret = 0
[   84.533463] faultin_page-->0
[   84.533464] __get_user_pages @8 faultin_page --> 0
[   84.533465] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533466] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533467] __get_user_pages @10 i=17 start=5626b5a65000 nr_pages=16
[   84.533468] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533470] faultin_page handle_mm_fault --> ret = 0
[   84.533471] faultin_page-->0
[   84.533472] __get_user_pages @8 faultin_page --> 0
[   84.533473] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533474] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533476] __get_user_pages @10 i=18 start=5626b5a66000 nr_pages=15
[   84.533477] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533479] faultin_page handle_mm_fault --> ret = 0
[   84.533479] faultin_page-->0
[   84.533480] __get_user_pages @8 faultin_page --> 0
[   84.533482] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533483] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533484] __get_user_pages @10 i=19 start=5626b5a67000 nr_pages=14
[   84.533485] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533487] faultin_page handle_mm_fault --> ret = 0
[   84.533488] faultin_page-->0
[   84.533489] __get_user_pages @8 faultin_page --> 0
[   84.533490] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533491] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533492] __get_user_pages @10 i=20 start=5626b5a68000 nr_pages=13
[   84.533494] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533496] faultin_page handle_mm_fault --> ret = 0
[   84.533496] faultin_page-->0
[   84.533497] __get_user_pages @8 faultin_page --> 0
[   84.533499] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533500] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533501] __get_user_pages @10 i=21 start=5626b5a69000 nr_pages=12
[   84.533502] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533504] faultin_page handle_mm_fault --> ret = 0
[   84.533504] faultin_page-->0
[   84.533505] __get_user_pages @8 faultin_page --> 0
[   84.533507] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533508] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533509] __get_user_pages @10 i=22 start=5626b5a6a000 nr_pages=11
[   84.533510] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533512] faultin_page handle_mm_fault --> ret = 0
[   84.533512] faultin_page-->0
[   84.533513] __get_user_pages @8 faultin_page --> 0
[   84.533515] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533516] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533517] __get_user_pages @10 i=23 start=5626b5a6b000 nr_pages=10
[   84.533518] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533520] faultin_page handle_mm_fault --> ret = 0
[   84.533521] faultin_page-->0
[   84.533521] __get_user_pages @8 faultin_page --> 0
[   84.533523] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533524] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533525] __get_user_pages @10 i=24 start=5626b5a6c000 nr_pages=9
[   84.533526] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533528] faultin_page handle_mm_fault --> ret = 0
[   84.533529] faultin_page-->0
[   84.533530] __get_user_pages @8 faultin_page --> 0
[   84.533531] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533532] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533534] __get_user_pages @10 i=25 start=5626b5a6d000 nr_pages=8
[   84.533535] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533537] faultin_page handle_mm_fault --> ret = 0
[   84.533537] faultin_page-->0
[   84.533538] __get_user_pages @8 faultin_page --> 0
[   84.533540] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533541] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533542] __get_user_pages @10 i=26 start=5626b5a6e000 nr_pages=7
[   84.533543] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533545] faultin_page handle_mm_fault --> ret = 0
[   84.533545] faultin_page-->0
[   84.533546] __get_user_pages @8 faultin_page --> 0
[   84.533548] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533549] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533550] __get_user_pages @10 i=27 start=5626b5a6f000 nr_pages=6
[   84.533551] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533553] faultin_page handle_mm_fault --> ret = 0
[   84.533554] faultin_page-->0
[   84.533555] __get_user_pages @8 faultin_page --> 0
[   84.533557] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533558] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533559] __get_user_pages @10 i=28 start=5626b5a70000 nr_pages=5
[   84.533560] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533562] faultin_page handle_mm_fault --> ret = 0
[   84.533563] faultin_page-->0
[   84.533564] __get_user_pages @8 faultin_page --> 0
[   84.533565] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533566] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533567] __get_user_pages @10 i=29 start=5626b5a71000 nr_pages=4
[   84.533568] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533570] faultin_page handle_mm_fault --> ret = 0
[   84.533571] faultin_page-->0
[   84.533572] __get_user_pages @8 faultin_page --> 0
[   84.533573] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533574] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533576] __get_user_pages @10 i=30 start=5626b5a72000 nr_pages=3
[   84.533577] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533579] faultin_page handle_mm_fault --> ret = 0
[   84.533579] faultin_page-->0
[   84.533580] __get_user_pages @8 faultin_page --> 0
[   84.533582] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533583] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533584] __get_user_pages @10 i=31 start=5626b5a73000 nr_pages=2
[   84.533585] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533587] faultin_page handle_mm_fault --> ret = 0
[   84.533588] faultin_page-->0
[   84.533588] __get_user_pages @8 faultin_page --> 0
[   84.533590] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533591] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533592] __get_user_pages @10 i=32 start=5626b5a74000 nr_pages=1
[   84.533593] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533595] faultin_page handle_mm_fault --> ret = 0
[   84.533596] faultin_page-->0
[   84.533597] __get_user_pages @8 faultin_page --> 0
[   84.533598] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533599] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533600] __get_user_pages @10 i=33 start=5626b5a75000 nr_pages=0
[   84.533601] __get_user_pages LEAVE i=33 ret=0
[   84.533603] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=33 nend=5626b5a75000
[   84.533605] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5626b5a75000 1 0
[   84.533607] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e2be000 vma->vm_end=7f247e2e3000 vma->vm_flags=8002071
[   84.533608] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e2e3000 7ffffffff000 7f247e2e3000
[   84.533610] __get_user_pages start=7f247e2be000 nr_pages=37 gup_flags=1052 ctx.page_mask=0
[   84.533611] __get_user_pages @1 vma==NULL
[   84.533612] __get_user_pages @2 start=7f247e2be000 vma->vm_start=7f247e2be000 vma->vm_end=7f247e2e3000 vma->vm_flags=8002071
[   84.533613] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533617] faultin_page handle_mm_fault --> ret = 256
[   84.533618] faultin_page-->0
[   84.533619] __get_user_pages @8 faultin_page --> 0
[   84.533620] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   84.533621] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   84.533622] __get_user_pages @10 i=1 start=7f247e2bf000 nr_pages=36
[   84.533624] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533627] faultin_page handle_mm_fault --> ret = 1024
[   84.533629] faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?1 FAULT_FLAG_RETRY_NOWAIT?0
[   84.533630] __get_user_pages @8 faultin_page --> -16
[   84.533631] __get_user_pages LEAVE i=1 ret=0
[   84.533632] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=7f247e2bf000
[   84.533634] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e2bf000 0 0
[   84.533636] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e2be000 vma->vm_end=7f247e2e3000 vma->vm_flags=8002071
[   84.533638] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e2e3000 7ffffffff000 7f247e2e3000
[   84.533639] __get_user_pages start=7f247e2bf000 nr_pages=36 gup_flags=1052 ctx.page_mask=0
[   84.533640] __get_user_pages @1 vma==NULL
[   84.533642] __get_user_pages @2 start=7f247e2bf000 vma->vm_start=7f247e2be000 vma->vm_end=7f247e2e3000 vma->vm_flags=8002071
[   84.533643] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   84.533646] faultin_page handle_mm_fault --> ret = 1024
[   84.533647] faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?1 FAULT_FLAG_RETRY_NOWAIT?0
[   84.533648] __get_user_pages @8 faultin_page --> -16
[   84.533649] __get_user_pages LEAVE i=0 ret=0
[   84.533650] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=0 nend=7f247e2bf000
[   94.274156] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e2e3000 1 0
[   94.274157] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e2e3000 vma->vm_end=7f247e45b000 vma->vm_flags=8002075
[   94.274158] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e45b000 7ffffffff000 7f247e45b000
[   94.274159] __get_user_pages start=7f247e2e3000 nr_pages=376 gup_flags=1052 ctx.page_mask=0
[   94.274159] __get_user_pages @1 vma==NULL
[   94.274160] __get_user_pages @2 start=7f247e2e3000 vma->vm_start=7f247e2e3000 vma->vm_end=7f247e45b000 vma->vm_flags=8002075
[   94.274161] __get_user_pages @6 ERESTARTSYS
[   94.274161] __get_user_pages LEAVE i=0 ret=-512
[   94.274162] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274162] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e45b000 1 0
[   94.274163] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e45b000 vma->vm_end=7f247e4a5000 vma->vm_flags=8002071
[   94.274164] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e4a5000 7ffffffff000 7f247e4a5000
[   94.274164] __get_user_pages start=7f247e45b000 nr_pages=74 gup_flags=1052 ctx.page_mask=0
[   94.274164] __get_user_pages @1 vma==NULL
[   94.274165] __get_user_pages @2 start=7f247e45b000 vma->vm_start=7f247e45b000 vma->vm_end=7f247e4a5000 vma->vm_flags=8002071
[   94.274165] __get_user_pages @6 ERESTARTSYS
[   94.274166] __get_user_pages LEAVE i=0 ret=-512
[   94.274166] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274167] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e4a5000 1 0
[   94.274167] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e4a5000 vma->vm_end=7f247e4a8000 vma->vm_flags=8102071
[   94.274168] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e4a8000 7ffffffff000 7f247e4a8000
[   94.274168] __get_user_pages start=7f247e4a5000 nr_pages=3 gup_flags=1052 ctx.page_mask=0
[   94.274169] __get_user_pages @1 vma==NULL
[   94.274169] __get_user_pages @2 start=7f247e4a5000 vma->vm_start=7f247e4a5000 vma->vm_end=7f247e4a8000 vma->vm_flags=8102071
[   94.274170] __get_user_pages @6 ERESTARTSYS
[   94.274170] __get_user_pages LEAVE i=0 ret=-512
[   94.274170] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274171] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e4a8000 1 0
[   94.274171] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e4a8000 vma->vm_end=7f247e4ab000 vma->vm_flags=8102073
[   94.274172] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e4ab000 7ffffffff000 7f247e4ab000
[   94.274172] __get_user_pages start=7f247e4a8000 nr_pages=3 gup_flags=1053 ctx.page_mask=0
[   94.274173] __get_user_pages @1 vma==NULL
[   94.274173] __get_user_pages @2 start=7f247e4a8000 vma->vm_start=7f247e4a8000 vma->vm_end=7f247e4ab000 vma->vm_flags=8102073
[   94.274174] __get_user_pages @6 ERESTARTSYS
[   94.274174] __get_user_pages LEAVE i=0 ret=-512
[   94.274174] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274175] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e4ab000 1 0
[   94.274175] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e4ab000 vma->vm_end=7f247e4b1000 vma->vm_flags=8102073
[   94.274176] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e4b1000 7ffffffff000 7f247e4b1000
[   94.274176] __get_user_pages start=7f247e4ab000 nr_pages=6 gup_flags=1053 ctx.page_mask=0
[   94.274177] __get_user_pages @1 vma==NULL
[   94.274177] __get_user_pages @2 start=7f247e4ab000 vma->vm_start=7f247e4ab000 vma->vm_end=7f247e4b1000 vma->vm_flags=8102073
[   94.274177] __get_user_pages @6 ERESTARTSYS
[   94.274178] __get_user_pages LEAVE i=0 ret=-512
[   94.274178] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274179] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e4b1000 1 0
[   94.274179] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e4dc000 vma->vm_end=7f247e4dd000 vma->vm_flags=8002871
[   94.274180] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e4dd000 7ffffffff000 7f247e4dd000
[   94.274180] __get_user_pages start=7f247e4dc000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   94.274181] __get_user_pages @1 vma==NULL
[   94.274181] __get_user_pages @2 start=7f247e4dc000 vma->vm_start=7f247e4dc000 vma->vm_end=7f247e4dd000 vma->vm_flags=8002871
[   94.274181] __get_user_pages @6 ERESTARTSYS
[   94.274182] __get_user_pages LEAVE i=0 ret=-512
[   94.274182] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274183] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e4dd000 1 0
[   94.274183] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e4dd000 vma->vm_end=7f247e4ff000 vma->vm_flags=8002875
[   94.274184] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e4ff000 7ffffffff000 7f247e4ff000
[   94.274184] __get_user_pages start=7f247e4dd000 nr_pages=34 gup_flags=1052 ctx.page_mask=0
[   94.274185] __get_user_pages @1 vma==NULL
[   94.274185] __get_user_pages @2 start=7f247e4dd000 vma->vm_start=7f247e4dd000 vma->vm_end=7f247e4ff000 vma->vm_flags=8002875
[   94.274185] __get_user_pages @6 ERESTARTSYS
[   94.274186] __get_user_pages LEAVE i=0 ret=-512
[   94.274186] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274187] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e4ff000 1 0
[   94.274187] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e4ff000 vma->vm_end=7f247e507000 vma->vm_flags=8002871
[   94.274188] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e507000 7ffffffff000 7f247e507000
[   94.274188] __get_user_pages start=7f247e4ff000 nr_pages=8 gup_flags=1052 ctx.page_mask=0
[   94.274189] __get_user_pages @1 vma==NULL
[   94.274189] __get_user_pages @2 start=7f247e4ff000 vma->vm_start=7f247e4ff000 vma->vm_end=7f247e507000 vma->vm_flags=8002871
[   94.274189] __get_user_pages @6 ERESTARTSYS
[   94.274190] __get_user_pages LEAVE i=0 ret=-512
[   94.274190] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274191] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e507000 1 0
[   94.274191] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e508000 vma->vm_end=7f247e509000 vma->vm_flags=8102871
[   94.274192] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e509000 7ffffffff000 7f247e509000
[   94.274192] __get_user_pages start=7f247e508000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   94.274192] __get_user_pages @1 vma==NULL
[   94.274193] __get_user_pages @2 start=7f247e508000 vma->vm_start=7f247e508000 vma->vm_end=7f247e509000 vma->vm_flags=8102871
[   94.274193] __get_user_pages @6 ERESTARTSYS
[   94.274193] __get_user_pages LEAVE i=0 ret=-512
[   94.274194] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274194] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e509000 1 0
[   94.274195] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e509000 vma->vm_end=7f247e50a000 vma->vm_flags=8102873
[   94.274196] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e50a000 7ffffffff000 7f247e50a000
[   94.274196] __get_user_pages start=7f247e509000 nr_pages=1 gup_flags=1053 ctx.page_mask=0
[   94.274196] __get_user_pages @1 vma==NULL
[   94.274197] __get_user_pages @2 start=7f247e509000 vma->vm_start=7f247e509000 vma->vm_end=7f247e50a000 vma->vm_flags=8102873
[   94.274197] __get_user_pages @6 ERESTARTSYS
[   94.274198] __get_user_pages LEAVE i=0 ret=-512
[   94.274198] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274199] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e50a000 1 0
[   94.274199] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7f247e50a000 vma->vm_end=7f247e50b000 vma->vm_flags=8102073
[   94.274200] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7f247e50b000 7ffffffff000 7f247e50b000
[   94.274200] __get_user_pages start=7f247e50a000 nr_pages=1 gup_flags=1053 ctx.page_mask=0
[   94.274200] __get_user_pages @1 vma==NULL
[   94.274201] __get_user_pages @2 start=7f247e50a000 vma->vm_start=7f247e50a000 vma->vm_end=7f247e50b000 vma->vm_flags=8102073
[   94.274201] __get_user_pages @6 ERESTARTSYS
[   94.274202] __get_user_pages LEAVE i=0 ret=-512
[   94.274202] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274203] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7f247e50b000 1 0
[   94.274203] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7ffe95c8c000 vma->vm_end=7ffe95cae000 vma->vm_flags=102173
[   94.274204] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7ffe95cae000 7ffffffff000 7ffe95cae000
[   94.274204] __get_user_pages start=7ffe95c8c000 nr_pages=34 gup_flags=1053 ctx.page_mask=0
[   94.274204] __get_user_pages @1 vma==NULL
[   94.274205] __get_user_pages @2 start=7ffe95c8c000 vma->vm_start=7ffe95c8c000 vma->vm_end=7ffe95cae000 vma->vm_flags=102173
[   94.274205] __get_user_pages @6 ERESTARTSYS
[   94.274206] __get_user_pages LEAVE i=0 ret=-512
[   94.274206] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274207] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7ffe95cae000 1 0
[   94.274207] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7ffe95db1000 vma->vm_end=7ffe95db4000 vma->vm_flags=c044411
[   94.274208] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7ffe95db4000 7ffffffff000 7ffe95db4000
[   94.274208] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-1 c044411
[   94.274209] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7ffe95db4000 1 0
[   94.274209] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7ffe95db4000 vma->vm_end=7ffe95db5000 vma->vm_flags=8040075
[   94.274210] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7ffe95db5000 7ffffffff000 7ffe95db5000
[   94.274210] __get_user_pages start=7ffe95db4000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   94.274211] __get_user_pages @1 vma==NULL
[   94.274211] __get_user_pages @2 start=7ffe95db4000 vma->vm_start=7ffe95db4000 vma->vm_end=7ffe95db5000 vma->vm_flags=8040075
[   94.274211] __get_user_pages @6 ERESTARTSYS
[   94.274212] __get_user_pages LEAVE i=0 ret=-512
[   94.274212] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-2 -512
[   94.274213] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7ffe95db5000 1 0
[   94.274213] _mm_populate END 0 140737488351232 1

[-- Attachment #3: gup-printk.txt --]
[-- Type: text/plain, Size: 12272 bytes --]

diff --git a/mm/gup.c b/mm/gup.c
index 98f13ab37bac..5c9825745bb2 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -3,6 +3,7 @@
 #include <linux/errno.h>
 #include <linux/err.h>
 #include <linux/spinlock.h>
+#include <linux/printk.h>
 
 #include <linux/mm.h>
 #include <linux/memremap.h>
@@ -626,8 +627,8 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address,
  * *@flags does not include FOLL_NOWAIT, the mmap_sem may be released.
  * If it is, *@nonblocking will be set to 0 and -EBUSY returned.
  */
-static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
-		unsigned long address, unsigned int *flags, int *nonblocking)
+static int faultin_page_x(struct task_struct *tsk, struct vm_area_struct *vma,
+		unsigned long address, unsigned int *flags, int *nonblocking, int ign)
 {
 	unsigned int fault_flags = 0;
 	vm_fault_t ret;
@@ -649,8 +650,14 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 	}
 
 	ret = handle_mm_fault(vma, address, fault_flags);
+    if (!ign) {
+        printk(KERN_WARNING "faultin_page handle_mm_fault --> ret = %u\n", ret);
+    }
 	if (ret & VM_FAULT_ERROR) {
 		int err = vm_fault_to_errno(ret, *flags);
+        if (!ign) {
+            printk(KERN_WARNING "faultin_page handle_mm_fault --> err = %d\n", err);
+        }
 
 		if (err)
 			return err;
@@ -665,6 +672,9 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 	}
 
 	if (ret & VM_FAULT_RETRY) {
+	    if (!ign) {
+            printk(KERN_WARNING "faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?%d FAULT_FLAG_RETRY_NOWAIT?%d\n", nonblocking?1:0, (fault_flags & FAULT_FLAG_RETRY_NOWAIT)?1:0);
+        }
 		if (nonblocking && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
 			*nonblocking = 0;
 		return -EBUSY;
@@ -681,8 +691,16 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 	 */
 	if ((ret & VM_FAULT_WRITE) && !(vma->vm_flags & VM_WRITE))
 		*flags |= FOLL_COW;
+    if (!ign) {
+        printk(KERN_WARNING "faultin_page-->0\n");
+    }
 	return 0;
 }
+static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
+		unsigned long address, unsigned int *flags, int *nonblocking)
+{
+    return faultin_page_x(tsk, vma, address, flags, nonblocking, 1);
+}
 
 static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
 {
@@ -787,15 +805,18 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
  * instead of __get_user_pages. __get_user_pages should be used only if
  * you need some special @gup_flags.
  */
-static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
+static long __get_user_pages_x(struct task_struct *tsk, struct mm_struct *mm,
 		unsigned long start, unsigned long nr_pages,
 		unsigned int gup_flags, struct page **pages,
-		struct vm_area_struct **vmas, int *nonblocking)
+		struct vm_area_struct **vmas, int *nonblocking, int ign)
 {
 	long ret = 0, i = 0;
 	struct vm_area_struct *vma = NULL;
 	struct follow_page_context ctx = { NULL };
 
+	if (!ign)
+	    printk(KERN_WARNING "__get_user_pages start=%lx nr_pages=%lu gup_flags=%x ctx.page_mask=%u\n", start, nr_pages, gup_flags, ctx.page_mask);
+
 	if (!nr_pages)
 		return 0;
 
@@ -816,11 +837,25 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 
 		/* first iteration or cross vma bound */
 		if (!vma || start >= vma->vm_end) {
+            if (!ign) {
+                if (!vma)
+                    printk(KERN_WARNING "__get_user_pages @1 vma==NULL\n");
+                else
+                    printk(KERN_WARNING "__get_user_pages @1 start=%lx vma->vm_start=%lx vma->vm_end=%lx vma->vm_flags=%lx\n", start, vma->vm_start, vma->vm_end, vma->vm_flags);
+            }
 			vma = find_extend_vma(mm, start);
+            if (!ign) {
+                if (!vma)
+                    printk(KERN_WARNING "__get_user_pages @2 vma==NULL\n");
+                else
+                    printk(KERN_WARNING "__get_user_pages @2 start=%lx vma->vm_start=%lx vma->vm_end=%lx vma->vm_flags=%lx\n", start, vma->vm_start, vma->vm_end, vma->vm_flags);
+            }
 			if (!vma && in_gate_area(mm, start)) {
 				ret = get_gate_page(mm, start & PAGE_MASK,
 						gup_flags, &vma,
 						pages ? &pages[i] : NULL);
+                if (!ign)
+                    printk(KERN_WARNING "__get_user_pages @3 get_gate_page --> %ld\n", ret);
 				if (ret)
 					goto out;
 				ctx.page_mask = 0;
@@ -828,6 +863,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			}
 
 			if (!vma || check_vma_flags(vma, gup_flags)) {
+                if (!ign)
+                    printk(KERN_WARNING "__get_user_pages @4 EFAULT\n");
 				ret = -EFAULT;
 				goto out;
 			}
@@ -835,6 +872,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 				i = follow_hugetlb_page(mm, vma, pages, vmas,
 						&start, &nr_pages, i,
 						gup_flags, nonblocking);
+                if (!ign)
+                    printk(KERN_WARNING "__get_user_pages @5 follow_hugetlb_page --> %ld\n", i);
 				continue;
 			}
 		}
@@ -844,15 +883,21 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 		 * potentially allocating memory.
 		 */
 		if (fatal_signal_pending(current)) {
+            if (!ign)
+                printk(KERN_WARNING "__get_user_pages @6 ERESTARTSYS\n");
 			ret = -ERESTARTSYS;
 			goto out;
 		}
 		cond_resched();
 
 		page = follow_page_mask(vma, start, foll_flags, &ctx);
+        if (!ign)
+            printk(KERN_WARNING "__get_user_pages @7 follow_page_mask --> %d  ctx.page_mask=%u\n", page ? 1 : 0, ctx.page_mask);
 		if (!page) {
-			ret = faultin_page(tsk, vma, start, &foll_flags,
-					nonblocking);
+			ret = faultin_page_x(tsk, vma, start, &foll_flags,
+					nonblocking, ign);
+            if (!ign)
+                printk(KERN_WARNING "__get_user_pages @8 faultin_page --> %ld\n", ret);
 			switch (ret) {
 			case 0:
 				goto retry;
@@ -868,6 +913,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			}
 			BUG();
 		} else if (PTR_ERR(page) == -EEXIST) {
+            if (!ign)
+                printk(KERN_WARNING "__get_user_pages @8 EEXIST\n");
 			/*
 			 * Proper page table entry exists, but no corresponding
 			 * struct page.
@@ -875,6 +922,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			goto next_page;
 		} else if (IS_ERR(page)) {
 			ret = PTR_ERR(page);
+			if (!ign)
+                printk(KERN_WARNING "__get_user_pages @8 IS_ERR -> ret=%ld\n", ret);
 			goto out;
 		}
 		if (pages) {
@@ -889,17 +938,31 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			ctx.page_mask = 0;
 		}
 		page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask);
+        if (!ign)
+            printk(KERN_WARNING "__get_user_pages @9 page_increm=%u ctx.page_mask=%u\n", page_increm, ctx.page_mask);
 		if (page_increm > nr_pages)
 			page_increm = nr_pages;
 		i += page_increm;
 		start += page_increm * PAGE_SIZE;
 		nr_pages -= page_increm;
+        if (!ign)
+            printk(KERN_WARNING "__get_user_pages @10 i=%ld start=%lx nr_pages=%ld\n", i, start, nr_pages);
 	} while (nr_pages);
 out:
 	if (ctx.pgmap)
 		put_dev_pagemap(ctx.pgmap);
+    if (!ign) {
+        printk(KERN_WARNING "__get_user_pages LEAVE i=%ld ret=%ld\n", i, ret);
+    }
 	return i ? i : ret;
 }
+static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
+		unsigned long start, unsigned long nr_pages,
+		unsigned int gup_flags, struct page **pages,
+		struct vm_area_struct **vmas, int *nonblocking)
+{
+    return __get_user_pages_x(tsk, mm, start, nr_pages, gup_flags, pages, vmas, nonblocking, 1);
+}
 
 static bool vma_permits_fault(struct vm_area_struct *vma,
 			      unsigned int fault_flags)
@@ -1192,8 +1255,9 @@ EXPORT_SYMBOL(get_user_pages_remote);
  * If @nonblocking is non-NULL, it must held for read only and may be
  * released.  If it's released, *@nonblocking will be set to 0.
  */
-long populate_vma_page_range(struct vm_area_struct *vma,
-		unsigned long start, unsigned long end, int *nonblocking)
+long populate_vma_page_range_x(struct vm_area_struct *vma,
+		unsigned long start, unsigned long end, int *nonblocking,
+		int ign)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long nr_pages = (end - start) / PAGE_SIZE;
@@ -1227,8 +1291,13 @@ long populate_vma_page_range(struct vm_area_struct *vma,
 	 * We made sure addr is within a VMA, so the following will
 	 * not result in a stack expansion that recurses back here.
 	 */
-	return __get_user_pages(current, mm, start, nr_pages, gup_flags,
-				NULL, NULL, nonblocking);
+	return __get_user_pages_x(current, mm, start, nr_pages, gup_flags,
+				NULL, NULL, nonblocking, ign);
+}
+long populate_vma_page_range(struct vm_area_struct *vma,
+		unsigned long start, unsigned long end, int *nonblocking)
+{
+    return populate_vma_page_range_x(vma, start, end, nonblocking, 1);
 }
 
 /*
@@ -1241,14 +1310,23 @@ long populate_vma_page_range(struct vm_area_struct *vma,
 int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 {
 	struct mm_struct *mm = current->mm;
-	unsigned long end, nstart, nend;
+	unsigned long end, nstart, nend = 0L;
 	struct vm_area_struct *vma = NULL;
 	int locked = 0;
 	long ret = 0;
+	unsigned long nstart_prev = 0L - 1L, nend_prev = 0L - 1L;
+	int ign;
 
 	end = start + len;
 
+	printk(KERN_WARNING "_mm_populate %lx %lx %lx %d ENTER\n", start, len, end, ignore_errors);
+
 	for (nstart = start; nstart < end; nstart = nend) {
+	    ign = nstart == nstart_prev && nend == nend_prev;
+	    nstart_prev = nstart;
+	    nend_prev = nend;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP %lx %d %ld\n", start, len, end, ignore_errors, nstart, locked, ret);
 		/*
 		 * We want to fault in pages for [nstart; end) address range.
 		 * Find first corresponding VMA.
@@ -1259,6 +1337,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			vma = find_vma(mm, nstart);
 		} else if (nstart >= vma->vm_end)
 			vma = vma->vm_next;
+		if (!ign && vma)
+		    printk(KERN_WARNING "_mm_populate %lx %lx %lx %d vma->vm_start=%lx vma->vm_end=%lx vma->vm_flags=%lx\n", start, len, end, ignore_errors, vma->vm_start, vma->vm_end, vma->vm_flags);
 		if (!vma || vma->vm_start >= end)
 			break;
 		/*
@@ -1266,8 +1346,13 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 * range with the first VMA. Also, skip undesirable VMA types.
 		 */
 		nend = min(end, vma->vm_end);
-		if (vma->vm_flags & (VM_IO | VM_PFNMAP))
-			continue;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d nend=%lx %lx %lx\n", start, len, end, ignore_errors, nend, end, vma->vm_end);
+		if (vma->vm_flags & (VM_IO | VM_PFNMAP)) {
+            if (!ign)
+                printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-1 %lx\n", start, len, end, ignore_errors, vma->vm_flags);
+            continue;
+        }
 		if (nstart < vma->vm_start)
 			nstart = vma->vm_start;
 		/*
@@ -1275,8 +1360,10 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 * double checks the vma flags, so that it won't mlock pages
 		 * if the vma was already munlocked.
 		 */
-		ret = populate_vma_page_range(vma, nstart, nend, &locked);
+		ret = populate_vma_page_range_x(vma, nstart, nend, &locked, ign);
 		if (ret < 0) {
+            if (!ign)
+                printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-2 %ld\n", start, len, end, ignore_errors, ret);
 			if (ignore_errors) {
 				ret = 0;
 				continue;	/* continue at next VMA */
@@ -1284,8 +1371,11 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			break;
 		}
 		nend = nstart + ret * PAGE_SIZE;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-3 ret=%ld nend=%lx\n", start, len, end, ignore_errors, ret, nend);
 		ret = 0;
 	}
+    printk(KERN_WARNING "_mm_populate END %lu %lu %d\n", start, len, locked);
 	if (locked)
 		up_read(&mm->mmap_sem);
 	return ret;	/* 0 or negative error code */

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 10:25                                   ` Robert Stupp
@ 2019-11-06 11:26                                     ` Robert Stupp
  2019-11-06 12:04                                       ` Jan Kara
  2019-11-06 12:24                                       ` Robert Stupp
  0 siblings, 2 replies; 39+ messages in thread
From: Robert Stupp @ 2019-11-06 11:26 UTC (permalink / raw)
  To: Johannes Weiner, Vlastimil Babka
  Cc: Michal Hocko, Josef Bacik, Jan Kara, Kirill A. Shutemov,
	Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra,
	Stefan

Maybe a native and wrong idea, but would it work to call
__get_user_pages_locked() instead of __get_user_pages() in
populate_vma_page_range() ?

On Wed, 2019-11-06 at 11:25 +0100, Robert Stupp wrote:
> Here's one more dmesg output with more information captured in
> __get_user_pages() as well. It basically confirms that
> handle_mm_fault() returns VM_FAULT_RETRY.
>
> I'm not sure where and what to change ("fix with a FOLL_TRIED
> somewhere") to make it work. My (uneducated) impression is, that only
> __get_user_pages() needs to be changed - but I might be wrong.
>



^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 11:26                                     ` Robert Stupp
@ 2019-11-06 12:04                                       ` Jan Kara
  2019-11-06 12:24                                       ` Robert Stupp
  1 sibling, 0 replies; 39+ messages in thread
From: Jan Kara @ 2019-11-06 12:04 UTC (permalink / raw)
  To: snazy
  Cc: Johannes Weiner, Vlastimil Babka, Michal Hocko, Josef Bacik,
	Jan Kara, Kirill A. Shutemov, Randy Dunlap, linux-kernel,
	Linux MM, Andrew Morton, Potyra, Stefan

On Wed 06-11-19 12:26:24, Robert Stupp wrote:
> Maybe a native and wrong idea, but would it work to call
> __get_user_pages_locked() instead of __get_user_pages() in
> populate_vma_page_range() ?

See my reply to Johannes. It would work but it would be somewhat fragile.

								Honza

> On Wed, 2019-11-06 at 11:25 +0100, Robert Stupp wrote:
> > Here's one more dmesg output with more information captured in
> > __get_user_pages() as well. It basically confirms that
> > handle_mm_fault() returns VM_FAULT_RETRY.
> >
> > I'm not sure where and what to change ("fix with a FOLL_TRIED
> > somewhere") to make it work. My (uneducated) impression is, that only
> > __get_user_pages() needs to be changed - but I might be wrong.
> >
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 11:26                                     ` Robert Stupp
  2019-11-06 12:04                                       ` Jan Kara
@ 2019-11-06 12:24                                       ` Robert Stupp
  1 sibling, 0 replies; 39+ messages in thread
From: Robert Stupp @ 2019-11-06 12:24 UTC (permalink / raw)
  To: snazy, Johannes Weiner, Vlastimil Babka
  Cc: Michal Hocko, Josef Bacik, Jan Kara, Kirill A. Shutemov,
	Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra,
	Stefan

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Oh! That (s/__get_user_pages()/__get_user_pages_locked()/) seems to fix
it.

Attached a patch (gup-mlockall.txt, against v5.3.8) - although, I've
got no clue how patches work here.
gup-debug-mlockall.txt is the debug printk() I've added plus the code
change.
dmesg-out.txt.gz is the dmesg output of the latter for test.c


On Wed, 2019-11-06 at 12:26 +0100, Robert Stupp wrote:
> Maybe a native and wrong idea, but would it work to call
> __get_user_pages_locked() instead of __get_user_pages() in
> populate_vma_page_range() ?
>
> On Wed, 2019-11-06 at 11:25 +0100, Robert Stupp wrote:
> > Here's one more dmesg output with more information captured in
> > __get_user_pages() as well. It basically confirms that
> > handle_mm_fault() returns VM_FAULT_RETRY.
> >
> > I'm not sure where and what to change ("fix with a FOLL_TRIED
> > somewhere") to make it work. My (uneducated) impression is, that
> > only
> > __get_user_pages() needs to be changed - but I might be wrong.
> >


[-- Attachment #2: gup-mlockall.txt --]
[-- Type: text/plain, Size: 2313 bytes --]

commit d2a1ce44cd81b8ccf426a6c40a3a0d9b2b51e076
Author: Robert Stupp <snazy@snazy.de>
Date:   Wed Nov 6 12:58:12 2019 +0100

    make mlockall(MCL_CURRENT) aware of new VM_FAULT_RETRY conditions
    
    mlockall(MCL_CURRENT) can run into an busy/endless loop in
    __mm_populate(). This can happen since
    6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11 (introduced in Linux 5.1),
    "filemap: drop the mmap_sem for all blocking operations" / new
    VM_FAULT_RETRY conditions.
    
    The fix here is to replace the call to __get_user_pages() with a
    call to __get_user_pages_locked() in populate_vma_page_range().
    
    This change seems to fix the behavior for mlockall(MCL_CURRENT) but
    does not change the behavior for other usages of
    populate_vma_page_range() in find_extend_vma() and
    mprotect_fixup().
    
    (Text by Johannes Weiner hannes-at-cmpxchg.org):
    The only way this can occur is if populate_vma_page_range() returns
    0 and we don't advance the iteration position (if it returned an
    error, we wouldn't reset nend and move on to the next vma as
    ignore_errors is 1 for mlockall.)
    
    populate_vma_page_range() returns 0 when the first page is not
    found and faultin_page() returns -EBUSY (if it were processing
    pages, or if the error from faultin_page() would be a different
    one, we would return the number of pages processed or -error).
    
    faultin_page() returns -EBUSY when VM_FAULT_RETRY is set, i.e. we
    dropped the mmap_sem in order to initiate IO and require a retry.
    That is consistent with the bisect result (new VM_FAULT_RETRY
    conditions).
    
    At this point, regular page fault would retry with FAULT_FLAG_TRIED
    to indicate that the mmap_sem cannot be dropped a second time.

diff --git a/mm/gup.c b/mm/gup.c
index 98f13ab37bac..e319ffb625b1 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1227,8 +1227,8 @@ long populate_vma_page_range(struct vm_area_struct *vma,
 	 * We made sure addr is within a VMA, so the following will
 	 * not result in a stack expansion that recurses back here.
 	 */
-	return __get_user_pages(current, mm, start, nr_pages, gup_flags,
-				NULL, NULL, nonblocking);
+	return __get_user_pages_locked(current, mm, start, nr_pages,
+                   NULL, NULL, nonblocking, gup_flags);
 }
 
 /*

[-- Attachment #3: gup-debug-mlockall.txt --]
[-- Type: text/plain, Size: 15994 bytes --]

commit 8c2e4c28bde0b10ba488d32d43d644612b2df737
Author: Robert Stupp <snazy@snazy.de>
Date:   Wed Nov 6 12:29:53 2019 +0100

    Replace __get_user_pages() with __get_user_pages_locked() in populate_vma_page_range()

diff --git a/mm/gup.c b/mm/gup.c
index 5c9825745bb2..c0b61b3130a8 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1065,14 +1065,15 @@ int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm,
 }
 EXPORT_SYMBOL_GPL(fixup_user_fault);
 
-static __always_inline long __get_user_pages_locked(struct task_struct *tsk,
+static __always_inline long __get_user_pages_locked_x(struct task_struct *tsk,
 						struct mm_struct *mm,
 						unsigned long start,
 						unsigned long nr_pages,
 						struct page **pages,
 						struct vm_area_struct **vmas,
 						int *locked,
-						unsigned int flags)
+						unsigned int flags,
+						int ign)
 {
 	long ret, pages_done;
 	bool lock_dropped;
@@ -1090,8 +1091,8 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,
 	pages_done = 0;
 	lock_dropped = false;
 	for (;;) {
-		ret = __get_user_pages(tsk, mm, start, nr_pages, flags, pages,
-				       vmas, locked);
+		ret = __get_user_pages_x(tsk, mm, start, nr_pages, flags, pages,
+				       vmas, locked, ign);
 		if (!locked)
 			/* VM_FAULT_RETRY couldn't trigger, bypass */
 			return ret;
@@ -1133,8 +1134,8 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,
 		*locked = 1;
 		lock_dropped = true;
 		down_read(&mm->mmap_sem);
-		ret = __get_user_pages(tsk, mm, start, 1, flags | FOLL_TRIED,
-				       pages, NULL, NULL);
+		ret = __get_user_pages_x(tsk, mm, start, 1, flags | FOLL_TRIED,
+				       pages, NULL, NULL, ign);
 		if (ret != 1) {
 			BUG_ON(ret > 1);
 			if (!pages_done)
@@ -1159,6 +1160,17 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,
 	}
 	return pages_done;
 }
+static __always_inline long __get_user_pages_locked(struct task_struct *tsk,
+						struct mm_struct *mm,
+						unsigned long start,
+						unsigned long nr_pages,
+						struct page **pages,
+						struct vm_area_struct **vmas,
+						int *locked,
+						unsigned int flags)
+{
+    return __get_user_pages_locked_x(tsk, mm, start, nr_pages, pages, vmas, locked, flags, 1);
+}
 
 /*
  * get_user_pages_remote() - pin user pages in memory
@@ -1291,8 +1303,10 @@ long populate_vma_page_range_x(struct vm_area_struct *vma,
 	 * We made sure addr is within a VMA, so the following will
 	 * not result in a stack expansion that recurses back here.
 	 */
-	return __get_user_pages_x(current, mm, start, nr_pages, gup_flags,
-				NULL, NULL, nonblocking, ign);
+	return __get_user_pages_locked_x(current, mm, start, nr_pages,
+	            NULL, NULL, nonblocking, gup_flags, ign);
+//	return __get_user_pages_x(current, mm, start, nr_pages, gup_flags,
+//				NULL, NULL, nonblocking, ign);
 }
 long populate_vma_page_range(struct vm_area_struct *vma,
 		unsigned long start, unsigned long end, int *nonblocking)

commit 8b0bed7848a32c74512e8278c94de16eea390274
Author: Robert Stupp <snazy@snazy.de>
Date:   Wed Nov 6 09:50:32 2019 +0100

    More printk

diff --git a/mm/gup.c b/mm/gup.c
index 3bc25fd44433..5c9825745bb2 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -627,8 +627,8 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address,
  * *@flags does not include FOLL_NOWAIT, the mmap_sem may be released.
  * If it is, *@nonblocking will be set to 0 and -EBUSY returned.
  */
-static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
-		unsigned long address, unsigned int *flags, int *nonblocking)
+static int faultin_page_x(struct task_struct *tsk, struct vm_area_struct *vma,
+		unsigned long address, unsigned int *flags, int *nonblocking, int ign)
 {
 	unsigned int fault_flags = 0;
 	vm_fault_t ret;
@@ -650,8 +650,14 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 	}
 
 	ret = handle_mm_fault(vma, address, fault_flags);
+    if (!ign) {
+        printk(KERN_WARNING "faultin_page handle_mm_fault --> ret = %u\n", ret);
+    }
 	if (ret & VM_FAULT_ERROR) {
 		int err = vm_fault_to_errno(ret, *flags);
+        if (!ign) {
+            printk(KERN_WARNING "faultin_page handle_mm_fault --> err = %d\n", err);
+        }
 
 		if (err)
 			return err;
@@ -666,6 +672,9 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 	}
 
 	if (ret & VM_FAULT_RETRY) {
+	    if (!ign) {
+            printk(KERN_WARNING "faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?%d FAULT_FLAG_RETRY_NOWAIT?%d\n", nonblocking?1:0, (fault_flags & FAULT_FLAG_RETRY_NOWAIT)?1:0);
+        }
 		if (nonblocking && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
 			*nonblocking = 0;
 		return -EBUSY;
@@ -682,8 +691,16 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 	 */
 	if ((ret & VM_FAULT_WRITE) && !(vma->vm_flags & VM_WRITE))
 		*flags |= FOLL_COW;
+    if (!ign) {
+        printk(KERN_WARNING "faultin_page-->0\n");
+    }
 	return 0;
 }
+static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
+		unsigned long address, unsigned int *flags, int *nonblocking)
+{
+    return faultin_page_x(tsk, vma, address, flags, nonblocking, 1);
+}
 
 static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
 {
@@ -788,15 +805,18 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
  * instead of __get_user_pages. __get_user_pages should be used only if
  * you need some special @gup_flags.
  */
-static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
+static long __get_user_pages_x(struct task_struct *tsk, struct mm_struct *mm,
 		unsigned long start, unsigned long nr_pages,
 		unsigned int gup_flags, struct page **pages,
-		struct vm_area_struct **vmas, int *nonblocking)
+		struct vm_area_struct **vmas, int *nonblocking, int ign)
 {
 	long ret = 0, i = 0;
 	struct vm_area_struct *vma = NULL;
 	struct follow_page_context ctx = { NULL };
 
+	if (!ign)
+	    printk(KERN_WARNING "__get_user_pages start=%lx nr_pages=%lu gup_flags=%x ctx.page_mask=%u\n", start, nr_pages, gup_flags, ctx.page_mask);
+
 	if (!nr_pages)
 		return 0;
 
@@ -817,11 +837,25 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 
 		/* first iteration or cross vma bound */
 		if (!vma || start >= vma->vm_end) {
+            if (!ign) {
+                if (!vma)
+                    printk(KERN_WARNING "__get_user_pages @1 vma==NULL\n");
+                else
+                    printk(KERN_WARNING "__get_user_pages @1 start=%lx vma->vm_start=%lx vma->vm_end=%lx vma->vm_flags=%lx\n", start, vma->vm_start, vma->vm_end, vma->vm_flags);
+            }
 			vma = find_extend_vma(mm, start);
+            if (!ign) {
+                if (!vma)
+                    printk(KERN_WARNING "__get_user_pages @2 vma==NULL\n");
+                else
+                    printk(KERN_WARNING "__get_user_pages @2 start=%lx vma->vm_start=%lx vma->vm_end=%lx vma->vm_flags=%lx\n", start, vma->vm_start, vma->vm_end, vma->vm_flags);
+            }
 			if (!vma && in_gate_area(mm, start)) {
 				ret = get_gate_page(mm, start & PAGE_MASK,
 						gup_flags, &vma,
 						pages ? &pages[i] : NULL);
+                if (!ign)
+                    printk(KERN_WARNING "__get_user_pages @3 get_gate_page --> %ld\n", ret);
 				if (ret)
 					goto out;
 				ctx.page_mask = 0;
@@ -829,6 +863,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			}
 
 			if (!vma || check_vma_flags(vma, gup_flags)) {
+                if (!ign)
+                    printk(KERN_WARNING "__get_user_pages @4 EFAULT\n");
 				ret = -EFAULT;
 				goto out;
 			}
@@ -836,6 +872,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 				i = follow_hugetlb_page(mm, vma, pages, vmas,
 						&start, &nr_pages, i,
 						gup_flags, nonblocking);
+                if (!ign)
+                    printk(KERN_WARNING "__get_user_pages @5 follow_hugetlb_page --> %ld\n", i);
 				continue;
 			}
 		}
@@ -845,15 +883,21 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 		 * potentially allocating memory.
 		 */
 		if (fatal_signal_pending(current)) {
+            if (!ign)
+                printk(KERN_WARNING "__get_user_pages @6 ERESTARTSYS\n");
 			ret = -ERESTARTSYS;
 			goto out;
 		}
 		cond_resched();
 
 		page = follow_page_mask(vma, start, foll_flags, &ctx);
+        if (!ign)
+            printk(KERN_WARNING "__get_user_pages @7 follow_page_mask --> %d  ctx.page_mask=%u\n", page ? 1 : 0, ctx.page_mask);
 		if (!page) {
-			ret = faultin_page(tsk, vma, start, &foll_flags,
-					nonblocking);
+			ret = faultin_page_x(tsk, vma, start, &foll_flags,
+					nonblocking, ign);
+            if (!ign)
+                printk(KERN_WARNING "__get_user_pages @8 faultin_page --> %ld\n", ret);
 			switch (ret) {
 			case 0:
 				goto retry;
@@ -869,6 +913,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			}
 			BUG();
 		} else if (PTR_ERR(page) == -EEXIST) {
+            if (!ign)
+                printk(KERN_WARNING "__get_user_pages @8 EEXIST\n");
 			/*
 			 * Proper page table entry exists, but no corresponding
 			 * struct page.
@@ -876,6 +922,8 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			goto next_page;
 		} else if (IS_ERR(page)) {
 			ret = PTR_ERR(page);
+			if (!ign)
+                printk(KERN_WARNING "__get_user_pages @8 IS_ERR -> ret=%ld\n", ret);
 			goto out;
 		}
 		if (pages) {
@@ -890,17 +938,31 @@ static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 			ctx.page_mask = 0;
 		}
 		page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask);
+        if (!ign)
+            printk(KERN_WARNING "__get_user_pages @9 page_increm=%u ctx.page_mask=%u\n", page_increm, ctx.page_mask);
 		if (page_increm > nr_pages)
 			page_increm = nr_pages;
 		i += page_increm;
 		start += page_increm * PAGE_SIZE;
 		nr_pages -= page_increm;
+        if (!ign)
+            printk(KERN_WARNING "__get_user_pages @10 i=%ld start=%lx nr_pages=%ld\n", i, start, nr_pages);
 	} while (nr_pages);
 out:
 	if (ctx.pgmap)
 		put_dev_pagemap(ctx.pgmap);
+    if (!ign) {
+        printk(KERN_WARNING "__get_user_pages LEAVE i=%ld ret=%ld\n", i, ret);
+    }
 	return i ? i : ret;
 }
+static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
+		unsigned long start, unsigned long nr_pages,
+		unsigned int gup_flags, struct page **pages,
+		struct vm_area_struct **vmas, int *nonblocking)
+{
+    return __get_user_pages_x(tsk, mm, start, nr_pages, gup_flags, pages, vmas, nonblocking, 1);
+}
 
 static bool vma_permits_fault(struct vm_area_struct *vma,
 			      unsigned int fault_flags)
@@ -1193,8 +1255,9 @@ EXPORT_SYMBOL(get_user_pages_remote);
  * If @nonblocking is non-NULL, it must held for read only and may be
  * released.  If it's released, *@nonblocking will be set to 0.
  */
-long populate_vma_page_range(struct vm_area_struct *vma,
-		unsigned long start, unsigned long end, int *nonblocking)
+long populate_vma_page_range_x(struct vm_area_struct *vma,
+		unsigned long start, unsigned long end, int *nonblocking,
+		int ign)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long nr_pages = (end - start) / PAGE_SIZE;
@@ -1228,8 +1291,13 @@ long populate_vma_page_range(struct vm_area_struct *vma,
 	 * We made sure addr is within a VMA, so the following will
 	 * not result in a stack expansion that recurses back here.
 	 */
-	return __get_user_pages(current, mm, start, nr_pages, gup_flags,
-				NULL, NULL, nonblocking);
+	return __get_user_pages_x(current, mm, start, nr_pages, gup_flags,
+				NULL, NULL, nonblocking, ign);
+}
+long populate_vma_page_range(struct vm_area_struct *vma,
+		unsigned long start, unsigned long end, int *nonblocking)
+{
+    return populate_vma_page_range_x(vma, start, end, nonblocking, 1);
 }
 
 /*
@@ -1292,7 +1360,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 * double checks the vma flags, so that it won't mlock pages
 		 * if the vma was already munlocked.
 		 */
-		ret = populate_vma_page_range(vma, nstart, nend, &locked);
+		ret = populate_vma_page_range_x(vma, nstart, nend, &locked, ign);
 		if (ret < 0) {
             if (!ign)
                 printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-2 %ld\n", start, len, end, ignore_errors, ret);

commit 74da9b1c9bbaf66320d973ff7c8dc63962b6dc7d
Author: Robert Stupp <snazy@snazy.de>
Date:   Tue Nov 5 17:18:49 2019 +0100

    Add debug things

diff --git a/mm/gup.c b/mm/gup.c
index 98f13ab37bac..3bc25fd44433 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -3,6 +3,7 @@
 #include <linux/errno.h>
 #include <linux/err.h>
 #include <linux/spinlock.h>
+#include <linux/printk.h>
 
 #include <linux/mm.h>
 #include <linux/memremap.h>
@@ -1241,14 +1242,23 @@ long populate_vma_page_range(struct vm_area_struct *vma,
 int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 {
 	struct mm_struct *mm = current->mm;
-	unsigned long end, nstart, nend;
+	unsigned long end, nstart, nend = 0L;
 	struct vm_area_struct *vma = NULL;
 	int locked = 0;
 	long ret = 0;
+	unsigned long nstart_prev = 0L - 1L, nend_prev = 0L - 1L;
+	int ign;
 
 	end = start + len;
 
+	printk(KERN_WARNING "_mm_populate %lx %lx %lx %d ENTER\n", start, len, end, ignore_errors);
+
 	for (nstart = start; nstart < end; nstart = nend) {
+	    ign = nstart == nstart_prev && nend == nend_prev;
+	    nstart_prev = nstart;
+	    nend_prev = nend;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP %lx %d %ld\n", start, len, end, ignore_errors, nstart, locked, ret);
 		/*
 		 * We want to fault in pages for [nstart; end) address range.
 		 * Find first corresponding VMA.
@@ -1259,6 +1269,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			vma = find_vma(mm, nstart);
 		} else if (nstart >= vma->vm_end)
 			vma = vma->vm_next;
+		if (!ign && vma)
+		    printk(KERN_WARNING "_mm_populate %lx %lx %lx %d vma->vm_start=%lx vma->vm_end=%lx vma->vm_flags=%lx\n", start, len, end, ignore_errors, vma->vm_start, vma->vm_end, vma->vm_flags);
 		if (!vma || vma->vm_start >= end)
 			break;
 		/*
@@ -1266,8 +1278,13 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 * range with the first VMA. Also, skip undesirable VMA types.
 		 */
 		nend = min(end, vma->vm_end);
-		if (vma->vm_flags & (VM_IO | VM_PFNMAP))
-			continue;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d nend=%lx %lx %lx\n", start, len, end, ignore_errors, nend, end, vma->vm_end);
+		if (vma->vm_flags & (VM_IO | VM_PFNMAP)) {
+            if (!ign)
+                printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-1 %lx\n", start, len, end, ignore_errors, vma->vm_flags);
+            continue;
+        }
 		if (nstart < vma->vm_start)
 			nstart = vma->vm_start;
 		/*
@@ -1277,6 +1294,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 */
 		ret = populate_vma_page_range(vma, nstart, nend, &locked);
 		if (ret < 0) {
+            if (!ign)
+                printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-2 %ld\n", start, len, end, ignore_errors, ret);
 			if (ignore_errors) {
 				ret = 0;
 				continue;	/* continue at next VMA */
@@ -1284,8 +1303,11 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			break;
 		}
 		nend = nstart + ret * PAGE_SIZE;
+        if (!ign)
+            printk(KERN_WARNING "_mm_populate %lx %lx %lx %d LOOP-3 ret=%ld nend=%lx\n", start, len, end, ignore_errors, ret, nend);
 		ret = 0;
 	}
+    printk(KERN_WARNING "_mm_populate END %lu %lu %d\n", start, len, locked);
 	if (locked)
 		up_read(&mm->mmap_sem);
 	return ret;	/* 0 or negative error code */

[-- Attachment #4: dmesg-out.txt.gz --]
[-- Type: application/gzip, Size: 17577 bytes --]

[-- Attachment #5: test.c --]
[-- Type: text/x-csrc, Size: 319 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>

int main(int argc, char** argv) {
  if (argc > 1) {
    printf("Sleeping... %s\n", argv[1]);
    sleep(atoi(argv[1]));
  }
  printf("Before mlockall(MCL_CURRENT)\n");
  mlockall(MCL_CURRENT);
  printf("After mlockall(MCL_CURRENT)\n");
}

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-05 18:22                               ` Johannes Weiner
  2019-11-05 20:05                                 ` Robert Stupp
@ 2019-11-06 12:03                                 ` Jan Kara
  2019-11-06 13:45                                   ` Robert Stupp
  1 sibling, 1 reply; 39+ messages in thread
From: Jan Kara @ 2019-11-06 12:03 UTC (permalink / raw)
  To: Johannes Weiner
  Cc: Vlastimil Babka, snazy, Michal Hocko, Josef Bacik, Jan Kara,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> On Tue, Nov 05, 2019 at 04:28:21PM +0100, Vlastimil Babka wrote:
> > On 11/5/19 2:23 PM, Robert Stupp wrote:
> > > "git bisect" led to a result.
> > > 
> > > The offending merge commit is f91f2ee54a21404fbc633550e99d69d14c2478f2
> > > "Merge branch 'akpm' (rest of patches from Andrew)".
> > > 
> > > The first bad commit in the merged series of commits is
> > > https://github.com/torvalds/linux/commit/6b4c9f4469819a0c1a38a0a4541337e0f9bf6c11
> > > . a75d4c33377277b6034dd1e2663bce444f952c14, the commit before 6b4c9f44,
> > > is good.
> > 
> > Ah, great you could bisect this. CCing people from the commit
> > 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")
> 
> Judging from Robert's stack captures, the task is not hung but
> busy-looping in __mm_populate(). AFAICS, the only way this can occur
> is if populate_vma_page_range() returns 0 and we don't advance the
> iteration position (if it returned an error, we wouldn't reset nend
> and move on to the next vma as ignore_errors is 1 for mlockall.)
> 
> populate_vma_page_range() returns 0 when the first page is not found
> and faultin_page() returns -EBUSY (if it were processing pages, or if
> the error from faultin_page() would be a different one, we would
> return the number of pages processed or -error).
> 
> faultin_page() returns -EBUSY when VM_FAULT_RETRY is set, i.e. we
> dropped the mmap_sem in order to initiate IO and require a retry. That
> is consistent with the bisect result (new VM_FAULT_RETRY conditions).
> 
> At this point, regular page fault would retry with FAULT_FLAG_TRIED to
> indicate that the mmap_sem cannot be dropped a second time. But this
> mlock path doesn't set that flag and we can loop repeatedly. That is
> something we probably need to fix with a FOLL_TRIED somewhere.

It seems we could use __get_user_pages_locked() for that in
populate_vma_page_range() if we were guaranteed that mm stays alive.  This
is guaranteed for current->mm cases but there seem to be some callers to
populate_vma_page_range() where mm could indeed go away once we drop
mmap_sem. These luckily pass NULL for the 'nonblocking' parameter though so
all call sites seem to be fine but it would be fragile...

> What I don't quite understand yet is why the fault path doesn't make
> progress eventually. We must drop the mmap_sem without changing the
> state in any way. How can we keep looping on the same page?

That may be a slight suboptimality with Josef's patches. If the page
is marked as PageReadahead, we always drop mmap_sem if we can and start
readahead without checking whether that makes sense or not in
do_async_mmap_readahead(). OTOH page_cache_async_readahead() then clears
PageReadahead so the only way how I can see we could loop like this is when
file->ra->ra_pages is 0. Not sure if that's what's happening through. We'd
need to find which of the paths in filemap_fault() calls
maybe_unlock_mmap_for_io() to tell more.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 12:03                                 ` Jan Kara
@ 2019-11-06 13:45                                   ` Robert Stupp
  2019-11-06 14:35                                     ` Jan Kara
                                                       ` (2 more replies)
  0 siblings, 3 replies; 39+ messages in thread
From: Robert Stupp @ 2019-11-06 13:45 UTC (permalink / raw)
  To: Jan Kara, Johannes Weiner
  Cc: Vlastimil Babka, snazy, Michal Hocko, Josef Bacik,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

[-- Attachment #1: Type: text/plain, Size: 3504 bytes --]

On Wed, 2019-11-06 at 13:03 +0100, Jan Kara wrote:
> On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> > What I don't quite understand yet is why the fault path doesn't
> > make
> > progress eventually. We must drop the mmap_sem without changing the
> > state in any way. How can we keep looping on the same page?
>
> That may be a slight suboptimality with Josef's patches. If the page
> is marked as PageReadahead, we always drop mmap_sem if we can and
> start
> readahead without checking whether that makes sense or not in
> do_async_mmap_readahead(). OTOH page_cache_async_readahead() then
> clears
> PageReadahead so the only way how I can see we could loop like this
> is when
> file->ra->ra_pages is 0. Not sure if that's what's happening through.
> We'd
> need to find which of the paths in filemap_fault() calls
> maybe_unlock_mmap_for_io() to tell more.

Yes, ra_pages==0
Attached the dmesg + smaps outputs


[   91.803976] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP
5637e3a2b000 1 0
[   91.803978] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma-
>vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.803980] _mm_populate 0 7ffffffff000 7ffffffff000 1
nend=7fa32b00a000 7ffffffff000 7fa32b00a000
[   91.803981] __get_user_pages start=7fa32afe5000 nr_pages=37
gup_flags=1052 ctx.page_mask=0
[   91.803982] __get_user_pages @1 vma==NULL
[   91.803984] __get_user_pages @2 start=7fa32afe5000 vma-
>vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.803986] __get_user_pages @2 vma->vm_file->f_mode=80a801d
ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0
ra.prev_pos=948
[   91.803988] __get_user_pages @7 follow_page_mask -->
1  ctx.page_mask=0
[   91.803989] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803990] __get_user_pages @10 i=1 start=7fa32afe6000 nr_pages=36
[   91.803991] __get_user_pages @7 follow_page_mask -->
0  ctx.page_mask=0
[   91.803996] faultin_page handle_mm_fault --> ret = 1024
[   91.803997] faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?1
FAULT_FLAG_RETRY_NOWAIT?0
[   91.803998] __get_user_pages @8 faultin_page --> -16
[   91.803999] __get_user_pages LEAVE i=1 ret=0
[   91.804001] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1
nend=7fa32afe6000
[   91.804002] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP
7fa32afe6000 0 0
[   91.804004] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma-
>vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.804006] _mm_populate 0 7ffffffff000 7ffffffff000 1
nend=7fa32b00a000 7ffffffff000 7fa32b00a000
[   91.804007] __get_user_pages start=7fa32afe6000 nr_pages=36
gup_flags=1052 ctx.page_mask=0
[   91.804008] __get_user_pages @1 vma==NULL
[   91.804010] __get_user_pages @2 start=7fa32afe6000 vma-
>vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.804012] __get_user_pages @2 vma->vm_file->f_mode=80a801d
ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0
ra.prev_pos=948
[   91.804013] __get_user_pages @7 follow_page_mask -->
0  ctx.page_mask=0
[   91.804016] faultin_page handle_mm_fault --> ret = 1024
[   91.804017] faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?1
FAULT_FLAG_RETRY_NOWAIT?0
[   91.804018] __get_user_pages @8 faultin_page --> -16
[   91.804019] __get_user_pages LEAVE i=0 ret=0
[   91.804020] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=0
nend=7fa32afe6000


[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 22616 bytes --]

[   91.803590] _mm_populate 0 7ffffffff000 7ffffffff000 1 ENTER
[   91.803594] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 0 0 0
[   91.803596] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5637e22a2000 vma->vm_end=5637e22a3000 vma->vm_flags=8002871
[   91.803598] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5637e22a3000 7ffffffff000 5637e22a3000
[   91.803600] __get_user_pages start=5637e22a2000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   91.803601] __get_user_pages @1 vma==NULL
[   91.803603] __get_user_pages @2 start=5637e22a2000 vma->vm_start=5637e22a2000 vma->vm_end=5637e22a3000 vma->vm_flags=8002871
[   91.803606] __get_user_pages @2 vma->vm_file->f_mode=80a801d ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0 ra.prev_pos=820
[   91.803611] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803612] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803614] __get_user_pages @10 i=1 start=5637e22a3000 nr_pages=0
[   91.803615] __get_user_pages LEAVE i=1 ret=0
[   91.803617] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5637e22a3000
[   91.803619] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5637e22a3000 1 0
[   91.803621] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5637e22a3000 vma->vm_end=5637e22a4000 vma->vm_flags=8002875
[   91.803623] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5637e22a4000 7ffffffff000 5637e22a4000
[   91.803625] __get_user_pages start=5637e22a3000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   91.803625] __get_user_pages @1 vma==NULL
[   91.803627] __get_user_pages @2 start=5637e22a3000 vma->vm_start=5637e22a3000 vma->vm_end=5637e22a4000 vma->vm_flags=8002875
[   91.803629] __get_user_pages @2 vma->vm_file->f_mode=80a801d ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0 ra.prev_pos=820
[   91.803631] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803632] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803634] __get_user_pages @10 i=1 start=5637e22a4000 nr_pages=0
[   91.803635] __get_user_pages LEAVE i=1 ret=0
[   91.803636] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5637e22a4000
[   91.803638] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5637e22a4000 1 0
[   91.803640] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5637e22a4000 vma->vm_end=5637e22a5000 vma->vm_flags=8002871
[   91.803642] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5637e22a5000 7ffffffff000 5637e22a5000
[   91.803643] __get_user_pages start=5637e22a4000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   91.803644] __get_user_pages @1 vma==NULL
[   91.803646] __get_user_pages @2 start=5637e22a4000 vma->vm_start=5637e22a4000 vma->vm_end=5637e22a5000 vma->vm_flags=8002871
[   91.803647] __get_user_pages @2 vma->vm_file->f_mode=80a801d ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0 ra.prev_pos=820
[   91.803649] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803650] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803651] __get_user_pages @10 i=1 start=5637e22a5000 nr_pages=0
[   91.803652] __get_user_pages LEAVE i=1 ret=0
[   91.803654] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5637e22a5000
[   91.803656] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5637e22a5000 1 0
[   91.803658] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5637e22a5000 vma->vm_end=5637e22a6000 vma->vm_flags=8102871
[   91.803659] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5637e22a6000 7ffffffff000 5637e22a6000
[   91.803661] __get_user_pages start=5637e22a5000 nr_pages=1 gup_flags=1052 ctx.page_mask=0
[   91.803661] __get_user_pages @1 vma==NULL
[   91.803663] __get_user_pages @2 start=5637e22a5000 vma->vm_start=5637e22a5000 vma->vm_end=5637e22a6000 vma->vm_flags=8102871
[   91.803665] __get_user_pages @2 vma->vm_file->f_mode=80a801d ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0 ra.prev_pos=820
[   91.803667] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803668] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803669] __get_user_pages @10 i=1 start=5637e22a6000 nr_pages=0
[   91.803670] __get_user_pages LEAVE i=1 ret=0
[   91.803672] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5637e22a6000
[   91.803673] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5637e22a6000 1 0
[   91.803675] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5637e22a6000 vma->vm_end=5637e22a7000 vma->vm_flags=8102873
[   91.803677] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5637e22a7000 7ffffffff000 5637e22a7000
[   91.803678] __get_user_pages start=5637e22a6000 nr_pages=1 gup_flags=1053 ctx.page_mask=0
[   91.803679] __get_user_pages @1 vma==NULL
[   91.803681] __get_user_pages @2 start=5637e22a6000 vma->vm_start=5637e22a6000 vma->vm_end=5637e22a7000 vma->vm_flags=8102873
[   91.803682] __get_user_pages @2 vma->vm_file->f_mode=80a801d ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0 ra.prev_pos=820
[   91.803684] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803685] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803686] __get_user_pages @10 i=1 start=5637e22a7000 nr_pages=0
[   91.803687] __get_user_pages LEAVE i=1 ret=0
[   91.803689] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=5637e22a7000
[   91.803691] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5637e22a7000 1 0
[   91.803693] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=5637e3a0a000 vma->vm_end=5637e3a2b000 vma->vm_flags=8102073
[   91.803694] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=5637e3a2b000 7ffffffff000 5637e3a2b000
[   91.803696] __get_user_pages start=5637e3a0a000 nr_pages=33 gup_flags=1053 ctx.page_mask=0
[   91.803697] __get_user_pages @1 vma==NULL
[   91.803698] __get_user_pages @2 start=5637e3a0a000 vma->vm_start=5637e3a0a000 vma->vm_end=5637e3a2b000 vma->vm_flags=8102073
[   91.803699] __get_user_pages @2 vma->vm_file==NULL
[   91.803701] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803702] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803703] __get_user_pages @10 i=1 start=5637e3a0b000 nr_pages=32
[   91.803705] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803707] faultin_page handle_mm_fault --> ret = 0
[   91.803708] faultin_page-->0
[   91.803709] __get_user_pages @8 faultin_page --> 0
[   91.803711] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803712] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803713] __get_user_pages @10 i=2 start=5637e3a0c000 nr_pages=31
[   91.803714] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803716] faultin_page handle_mm_fault --> ret = 0
[   91.803717] faultin_page-->0
[   91.803718] __get_user_pages @8 faultin_page --> 0
[   91.803720] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803721] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803722] __get_user_pages @10 i=3 start=5637e3a0d000 nr_pages=30
[   91.803723] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803725] faultin_page handle_mm_fault --> ret = 0
[   91.803725] faultin_page-->0
[   91.803726] __get_user_pages @8 faultin_page --> 0
[   91.803728] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803729] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803730] __get_user_pages @10 i=4 start=5637e3a0e000 nr_pages=29
[   91.803731] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803733] faultin_page handle_mm_fault --> ret = 0
[   91.803733] faultin_page-->0
[   91.803734] __get_user_pages @8 faultin_page --> 0
[   91.803736] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803737] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803738] __get_user_pages @10 i=5 start=5637e3a0f000 nr_pages=28
[   91.803739] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803741] faultin_page handle_mm_fault --> ret = 0
[   91.803742] faultin_page-->0
[   91.803742] __get_user_pages @8 faultin_page --> 0
[   91.803744] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803745] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803746] __get_user_pages @10 i=6 start=5637e3a10000 nr_pages=27
[   91.803747] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803749] faultin_page handle_mm_fault --> ret = 0
[   91.803750] faultin_page-->0
[   91.803750] __get_user_pages @8 faultin_page --> 0
[   91.803752] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803753] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803754] __get_user_pages @10 i=7 start=5637e3a11000 nr_pages=26
[   91.803755] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803757] faultin_page handle_mm_fault --> ret = 0
[   91.803758] faultin_page-->0
[   91.803759] __get_user_pages @8 faultin_page --> 0
[   91.803760] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803761] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803762] __get_user_pages @10 i=8 start=5637e3a12000 nr_pages=25
[   91.803764] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803766] faultin_page handle_mm_fault --> ret = 0
[   91.803766] faultin_page-->0
[   91.803767] __get_user_pages @8 faultin_page --> 0
[   91.803769] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803770] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803771] __get_user_pages @10 i=9 start=5637e3a13000 nr_pages=24
[   91.803772] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803774] faultin_page handle_mm_fault --> ret = 0
[   91.803775] faultin_page-->0
[   91.803776] __get_user_pages @8 faultin_page --> 0
[   91.803777] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803778] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803779] __get_user_pages @10 i=10 start=5637e3a14000 nr_pages=23
[   91.803780] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803783] faultin_page handle_mm_fault --> ret = 0
[   91.803783] faultin_page-->0
[   91.803784] __get_user_pages @8 faultin_page --> 0
[   91.803786] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803786] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803788] __get_user_pages @10 i=11 start=5637e3a15000 nr_pages=22
[   91.803789] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803791] faultin_page handle_mm_fault --> ret = 0
[   91.803791] faultin_page-->0
[   91.803792] __get_user_pages @8 faultin_page --> 0
[   91.803794] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803795] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803796] __get_user_pages @10 i=12 start=5637e3a16000 nr_pages=21
[   91.803797] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803799] faultin_page handle_mm_fault --> ret = 0
[   91.803800] faultin_page-->0
[   91.803800] __get_user_pages @8 faultin_page --> 0
[   91.803802] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803803] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803804] __get_user_pages @10 i=13 start=5637e3a17000 nr_pages=20
[   91.803805] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803807] faultin_page handle_mm_fault --> ret = 0
[   91.803808] faultin_page-->0
[   91.803809] __get_user_pages @8 faultin_page --> 0
[   91.803810] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803811] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803812] __get_user_pages @10 i=14 start=5637e3a18000 nr_pages=19
[   91.803814] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803816] faultin_page handle_mm_fault --> ret = 0
[   91.803816] faultin_page-->0
[   91.803817] __get_user_pages @8 faultin_page --> 0
[   91.803819] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803820] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803821] __get_user_pages @10 i=15 start=5637e3a19000 nr_pages=18
[   91.803822] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803824] faultin_page handle_mm_fault --> ret = 0
[   91.803825] faultin_page-->0
[   91.803826] __get_user_pages @8 faultin_page --> 0
[   91.803827] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803828] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803829] __get_user_pages @10 i=16 start=5637e3a1a000 nr_pages=17
[   91.803830] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803833] faultin_page handle_mm_fault --> ret = 0
[   91.803833] faultin_page-->0
[   91.803834] __get_user_pages @8 faultin_page --> 0
[   91.803836] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803837] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803838] __get_user_pages @10 i=17 start=5637e3a1b000 nr_pages=16
[   91.803839] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803841] faultin_page handle_mm_fault --> ret = 0
[   91.803842] faultin_page-->0
[   91.803842] __get_user_pages @8 faultin_page --> 0
[   91.803844] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803845] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803846] __get_user_pages @10 i=18 start=5637e3a1c000 nr_pages=15
[   91.803847] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803849] faultin_page handle_mm_fault --> ret = 0
[   91.803850] faultin_page-->0
[   91.803850] __get_user_pages @8 faultin_page --> 0
[   91.803852] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803853] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803854] __get_user_pages @10 i=19 start=5637e3a1d000 nr_pages=14
[   91.803855] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803858] faultin_page handle_mm_fault --> ret = 0
[   91.803858] faultin_page-->0
[   91.803859] __get_user_pages @8 faultin_page --> 0
[   91.803861] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803861] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803863] __get_user_pages @10 i=20 start=5637e3a1e000 nr_pages=13
[   91.803864] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803866] faultin_page handle_mm_fault --> ret = 0
[   91.803866] faultin_page-->0
[   91.803867] __get_user_pages @8 faultin_page --> 0
[   91.803869] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803870] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803871] __get_user_pages @10 i=21 start=5637e3a1f000 nr_pages=12
[   91.803872] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803874] faultin_page handle_mm_fault --> ret = 0
[   91.803874] faultin_page-->0
[   91.803875] __get_user_pages @8 faultin_page --> 0
[   91.803877] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803878] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803879] __get_user_pages @10 i=22 start=5637e3a20000 nr_pages=11
[   91.803880] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803882] faultin_page handle_mm_fault --> ret = 0
[   91.803883] faultin_page-->0
[   91.803884] __get_user_pages @8 faultin_page --> 0
[   91.803885] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803886] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803888] __get_user_pages @10 i=23 start=5637e3a21000 nr_pages=10
[   91.803889] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803891] faultin_page handle_mm_fault --> ret = 0
[   91.803891] faultin_page-->0
[   91.803892] __get_user_pages @8 faultin_page --> 0
[   91.803893] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803894] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803896] __get_user_pages @10 i=24 start=5637e3a22000 nr_pages=9
[   91.803897] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803899] faultin_page handle_mm_fault --> ret = 0
[   91.803899] faultin_page-->0
[   91.803900] __get_user_pages @8 faultin_page --> 0
[   91.803902] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803903] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803904] __get_user_pages @10 i=25 start=5637e3a23000 nr_pages=8
[   91.803905] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803907] faultin_page handle_mm_fault --> ret = 0
[   91.803908] faultin_page-->0
[   91.803908] __get_user_pages @8 faultin_page --> 0
[   91.803910] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803911] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803912] __get_user_pages @10 i=26 start=5637e3a24000 nr_pages=7
[   91.803913] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803915] faultin_page handle_mm_fault --> ret = 0
[   91.803916] faultin_page-->0
[   91.803917] __get_user_pages @8 faultin_page --> 0
[   91.803918] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803919] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803921] __get_user_pages @10 i=27 start=5637e3a25000 nr_pages=6
[   91.803922] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803924] faultin_page handle_mm_fault --> ret = 0
[   91.803924] faultin_page-->0
[   91.803925] __get_user_pages @8 faultin_page --> 0
[   91.803928] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803929] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803930] __get_user_pages @10 i=28 start=5637e3a26000 nr_pages=5
[   91.803931] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803933] faultin_page handle_mm_fault --> ret = 0
[   91.803934] faultin_page-->0
[   91.803934] __get_user_pages @8 faultin_page --> 0
[   91.803936] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803937] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803938] __get_user_pages @10 i=29 start=5637e3a27000 nr_pages=4
[   91.803939] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803941] faultin_page handle_mm_fault --> ret = 0
[   91.803942] faultin_page-->0
[   91.803943] __get_user_pages @8 faultin_page --> 0
[   91.803944] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803945] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803947] __get_user_pages @10 i=30 start=5637e3a28000 nr_pages=3
[   91.803948] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803950] faultin_page handle_mm_fault --> ret = 0
[   91.803951] faultin_page-->0
[   91.803951] __get_user_pages @8 faultin_page --> 0
[   91.803953] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803954] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803955] __get_user_pages @10 i=31 start=5637e3a29000 nr_pages=2
[   91.803956] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803958] faultin_page handle_mm_fault --> ret = 0
[   91.803959] faultin_page-->0
[   91.803960] __get_user_pages @8 faultin_page --> 0
[   91.803961] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803962] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803963] __get_user_pages @10 i=32 start=5637e3a2a000 nr_pages=1
[   91.803964] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803967] faultin_page handle_mm_fault --> ret = 0
[   91.803967] faultin_page-->0
[   91.803968] __get_user_pages @8 faultin_page --> 0
[   91.803970] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803971] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803972] __get_user_pages @10 i=33 start=5637e3a2b000 nr_pages=0
[   91.803973] __get_user_pages LEAVE i=33 ret=0
[   91.803975] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=33 nend=5637e3a2b000
[   91.803976] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 5637e3a2b000 1 0
[   91.803978] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.803980] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7fa32b00a000 7ffffffff000 7fa32b00a000
[   91.803981] __get_user_pages start=7fa32afe5000 nr_pages=37 gup_flags=1052 ctx.page_mask=0
[   91.803982] __get_user_pages @1 vma==NULL
[   91.803984] __get_user_pages @2 start=7fa32afe5000 vma->vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.803986] __get_user_pages @2 vma->vm_file->f_mode=80a801d ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0 ra.prev_pos=948
[   91.803988] __get_user_pages @7 follow_page_mask --> 1  ctx.page_mask=0
[   91.803989] __get_user_pages @9 page_increm=1 ctx.page_mask=0
[   91.803990] __get_user_pages @10 i=1 start=7fa32afe6000 nr_pages=36
[   91.803991] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.803996] faultin_page handle_mm_fault --> ret = 1024
[   91.803997] faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?1 FAULT_FLAG_RETRY_NOWAIT?0
[   91.803998] __get_user_pages @8 faultin_page --> -16
[   91.803999] __get_user_pages LEAVE i=1 ret=0
[   91.804001] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=1 nend=7fa32afe6000
[   91.804002] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP 7fa32afe6000 0 0
[   91.804004] _mm_populate 0 7ffffffff000 7ffffffff000 1 vma->vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.804006] _mm_populate 0 7ffffffff000 7ffffffff000 1 nend=7fa32b00a000 7ffffffff000 7fa32b00a000
[   91.804007] __get_user_pages start=7fa32afe6000 nr_pages=36 gup_flags=1052 ctx.page_mask=0
[   91.804008] __get_user_pages @1 vma==NULL
[   91.804010] __get_user_pages @2 start=7fa32afe6000 vma->vm_start=7fa32afe5000 vma->vm_end=7fa32b00a000 vma->vm_flags=8002071
[   91.804012] __get_user_pages @2 vma->vm_file->f_mode=80a801d ra.async_size=0 ra.ra_pages=0 ra.mmap_miss=0 ra.size=0 ra.start=0 ra.prev_pos=948
[   91.804013] __get_user_pages @7 follow_page_mask --> 0  ctx.page_mask=0
[   91.804016] faultin_page handle_mm_fault --> ret = 1024
[   91.804017] faultin_page-->EBUSY VM_FAULT_RETRY non-blocking?1 FAULT_FLAG_RETRY_NOWAIT?0
[   91.804018] __get_user_pages @8 faultin_page --> -16
[   91.804019] __get_user_pages LEAVE i=0 ret=0
[   91.804020] _mm_populate 0 7ffffffff000 7ffffffff000 1 LOOP-3 ret=0 nend=7fa32afe6000

[-- Attachment #3: smaps.txt --]
[-- Type: text/plain, Size: 14851 bytes --]

5637e22a2000-5637e22a3000 r--p 00000000 103:02 49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd 
5637e22a3000-5637e22a4000 r-xp 00001000 103:02 49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd ex mr mw me dw lo sd 
5637e22a4000-5637e22a5000 r--p 00002000 103:02 49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd 
5637e22a5000-5637e22a6000 r--p 00002000 103:02 49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo ac sd 
5637e22a6000-5637e22a7000 rw-p 00003000 103:02 49172550                  /home/snazy/devel/misc/zzz/test
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me dw lo ac sd 
5637e3a0a000-5637e3a2b000 rw-p 00000000 00:00 0                          [heap]
Size:                132 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 132 kB
Pss:                 132 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:       132 kB
Referenced:          132 kB
Anonymous:           132 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:              132 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd 
7fa32afe5000-7fa32b00a000 r--p 00000000 103:02 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                148 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 144 kB
Pss:                   0 kB
Shared_Clean:        144 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          144 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me lo sd 
7fa32b00a000-7fa32b182000 r-xp 00025000 103:02 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:               1504 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 784 kB
Pss:                   4 kB
Shared_Clean:        784 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd ex mr mw me lo sd 
7fa32b182000-7fa32b1cc000 r--p 0019d000 103:02 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                296 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  64 kB
Pss:                   0 kB
Shared_Clean:         64 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           64 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me lo sd 
7fa32b1cc000-7fa32b1cf000 r--p 001e6000 103:02 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               12 kB
THPeligible:		0
VmFlags: rd mr mw me lo ac sd 
7fa32b1cf000-7fa32b1d2000 rw-p 001e9000 103:02 44307431                  /lib/x86_64-linux-gnu/libc-2.30.so
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               12 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd 
7fa32b1d2000-7fa32b1d8000 rw-p 00000000 00:00 0 
Size:                 24 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  24 kB
Pss:                  24 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        24 kB
Referenced:           24 kB
Anonymous:            24 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               24 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd 
7fa32b203000-7fa32b204000 r--p 00000000 103:02 44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd 
7fa32b204000-7fa32b226000 r-xp 00001000 103:02 44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                136 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                 136 kB
Pss:                   0 kB
Shared_Clean:        136 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          136 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me dw lo sd 
7fa32b226000-7fa32b22e000 r--p 00023000 103:02 44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                 32 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  32 kB
Pss:                   0 kB
Shared_Clean:         32 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           32 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo sd 
7fa32b22f000-7fa32b230000 r--p 0002b000 103:02 44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd mr mw me dw lo ac sd 
7fa32b230000-7fa32b231000 rw-p 0002c000 103:02 44302414                  /lib/x86_64-linux-gnu/ld-2.30.so
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me dw lo ac sd 
7fa32b231000-7fa32b232000 rw-p 00000000 00:00 0 
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                4 kB
THPeligible:		0
VmFlags: rd wr mr mw me lo ac sd 
7ffd4fdcc000-7ffd4fdee000 rw-p 00000000 00:00 0                          [stack]
Size:                136 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  20 kB
Pss:                  20 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        20 kB
Referenced:           20 kB
Anonymous:            20 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:               20 kB
THPeligible:		0
VmFlags: rd wr mr mw me gd lo ac 
7ffd4fdf8000-7ffd4fdfb000 r--p 00000000 00:00 0                          [vvar]
Size:                 12 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd mr pf io de dd sd 
7ffd4fdfb000-7ffd4fdfc000 r-xp 00000000 00:00 0                          [vdso]
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me de sd 
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: ex 

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 13:45                                   ` Robert Stupp
@ 2019-11-06 14:35                                     ` Jan Kara
  2019-11-06 15:32                                       ` Robert Stupp
  2019-11-06 14:38                                     ` Jan Kara
  2019-11-06 14:56                                     ` Josef Bacik
  2 siblings, 1 reply; 39+ messages in thread
From: Jan Kara @ 2019-11-06 14:35 UTC (permalink / raw)
  To: snazy
  Cc: Jan Kara, Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Josef Bacik, Kirill A. Shutemov, Randy Dunlap, linux-kernel,
	Linux MM, Andrew Morton, Potyra, Stefan

On Wed 06-11-19 14:45:43, Robert Stupp wrote:
> On Wed, 2019-11-06 at 13:03 +0100, Jan Kara wrote:
> > On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> > > What I don't quite understand yet is why the fault path doesn't
> > > make
> > > progress eventually. We must drop the mmap_sem without changing the
> > > state in any way. How can we keep looping on the same page?
> >
> > That may be a slight suboptimality with Josef's patches. If the page
> > is marked as PageReadahead, we always drop mmap_sem if we can and
> > start
> > readahead without checking whether that makes sense or not in
> > do_async_mmap_readahead(). OTOH page_cache_async_readahead() then
> > clears
> > PageReadahead so the only way how I can see we could loop like this
> > is when
> > file->ra->ra_pages is 0. Not sure if that's what's happening through.
> > We'd
> > need to find which of the paths in filemap_fault() calls
> > maybe_unlock_mmap_for_io() to tell more.
> 
> Yes, ra_pages==0

OK, thanks for confirmation!

> 5637e22a2000-5637e22a3000 r--p 00000000 103:02 49172550                  /home/snazy/devel/misc/zzz/test

What kind of device & fs does your /home stay on? I don't recognize the major
number...

							Honza

-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 14:35                                     ` Jan Kara
@ 2019-11-06 15:32                                       ` Robert Stupp
  0 siblings, 0 replies; 39+ messages in thread
From: Robert Stupp @ 2019-11-06 15:32 UTC (permalink / raw)
  To: Jan Kara
  Cc: Johannes Weiner, Vlastimil Babka, Michal Hocko, Josef Bacik,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

On Wed, 2019-11-06 at 15:35 +0100, Jan Kara wrote:
> On Wed 06-11-19 14:45:43, Robert Stupp wrote:
> >
> > Yes, ra_pages==0
>
> OK, thanks for confirmation!
>
> > 5637e22a2000-5637e22a3000 r--p 00000000 103:02
> > 49172550                  /home/snazy/devel/misc/zzz/test
>
> What kind of device & fs does your /home stay on? I don't recognize
> the major
> number...
>

It's ext4 on an NVMe SSD
Maybe it's of interest as well: my /tmp is on tmpfs




^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 13:45                                   ` Robert Stupp
  2019-11-06 14:35                                     ` Jan Kara
@ 2019-11-06 14:38                                     ` Jan Kara
  2019-11-06 14:56                                     ` Josef Bacik
  2 siblings, 0 replies; 39+ messages in thread
From: Jan Kara @ 2019-11-06 14:38 UTC (permalink / raw)
  To: snazy
  Cc: Jan Kara, Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Josef Bacik, Kirill A. Shutemov, Randy Dunlap, linux-kernel,
	Linux MM, Andrew Morton, Potyra, Stefan

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Wed 06-11-19 14:45:43, Robert Stupp wrote:
> On Wed, 2019-11-06 at 13:03 +0100, Jan Kara wrote:
> > On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> > > What I don't quite understand yet is why the fault path doesn't
> > > make
> > > progress eventually. We must drop the mmap_sem without changing the
> > > state in any way. How can we keep looping on the same page?
> >
> > That may be a slight suboptimality with Josef's patches. If the page
> > is marked as PageReadahead, we always drop mmap_sem if we can and
> > start
> > readahead without checking whether that makes sense or not in
> > do_async_mmap_readahead(). OTOH page_cache_async_readahead() then
> > clears
> > PageReadahead so the only way how I can see we could loop like this
> > is when
> > file->ra->ra_pages is 0. Not sure if that's what's happening through.
> > We'd
> > need to find which of the paths in filemap_fault() calls
> > maybe_unlock_mmap_for_io() to tell more.
> 
> Yes, ra_pages==0

BTW, attached patch should workaround your problem as well. But that's just
a performance optimization that happens to paper over your problem. Real
fix is the proper handling of fault retry as you did it.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[-- Attachment #2: 0001-mm-Don-t-bother-dropping-mmap_sem-for-zero-size-read.patch --]
[-- Type: text/x-patch, Size: 1079 bytes --]

From e12c861e687364ff5a891f0ae90283b384d74197 Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Wed, 6 Nov 2019 15:30:26 +0100
Subject: [PATCH] mm: Don't bother dropping mmap_sem for zero size readahead

When handling a page fault, we drop mmap_sem to start async readahead so
that we don't block on IO submission with mmap_sem held.  However
there's no point to drop mmap_sem in case readahead is disabled. Handle
that case to avoid pointless dropping of mmap_sem and retrying the
fault.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 mm/filemap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 1146fcfa3215..3d39c437b07e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2458,7 +2458,7 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
 	pgoff_t offset = vmf->pgoff;
 
 	/* If we don't want any read-ahead, don't bother */
-	if (vmf->vma->vm_flags & VM_RAND_READ)
+	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
 		return fpin;
 	if (ra->mmap_miss > 0)
 		ra->mmap_miss--;
-- 
2.16.4


^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 13:45                                   ` Robert Stupp
  2019-11-06 14:35                                     ` Jan Kara
  2019-11-06 14:38                                     ` Jan Kara
@ 2019-11-06 14:56                                     ` Josef Bacik
  2019-11-06 15:05                                       ` Jan Kara
  2 siblings, 1 reply; 39+ messages in thread
From: Josef Bacik @ 2019-11-06 14:56 UTC (permalink / raw)
  To: snazy
  Cc: Jan Kara, Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Josef Bacik, Kirill A. Shutemov, Randy Dunlap, linux-kernel,
	Linux MM, Andrew Morton, Potyra, Stefan

On Wed, Nov 06, 2019 at 02:45:43PM +0100, Robert Stupp wrote:
> On Wed, 2019-11-06 at 13:03 +0100, Jan Kara wrote:
> > On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> > > What I don't quite understand yet is why the fault path doesn't
> > > make
> > > progress eventually. We must drop the mmap_sem without changing the
> > > state in any way. How can we keep looping on the same page?
> >
> > That may be a slight suboptimality with Josef's patches. If the page
> > is marked as PageReadahead, we always drop mmap_sem if we can and
> > start
> > readahead without checking whether that makes sense or not in
> > do_async_mmap_readahead(). OTOH page_cache_async_readahead() then
> > clears
> > PageReadahead so the only way how I can see we could loop like this
> > is when
> > file->ra->ra_pages is 0. Not sure if that's what's happening through.
> > We'd
> > need to find which of the paths in filemap_fault() calls
> > maybe_unlock_mmap_for_io() to tell more.
> 
> Yes, ra_pages==0
> Attached the dmesg + smaps outputs
> 
> 

Ah ok I see what's happening, __get_user_pages() returns 0 if we get an EBUSY
from faultin_page, and then __mm_populate does nend = nstart + ret * PAGE_SIZE,
which just leaves us where we are.

We need to handle the non-blocking and the locking separately in __mm_populate
so we know what's going on.  Jan's fix for the readahead thing is definitely
valid as well, but this will keep us from looping forever in other retry cases.

diff --git a/mm/gup.c b/mm/gup.c
index 8f236a335ae9..ac625805d569 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1237,6 +1237,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 	unsigned long end, nstart, nend;
 	struct vm_area_struct *vma = NULL;
 	int locked = 0;
+	int nonblocking = 1;
 	long ret = 0;
 
 	end = start + len;
@@ -1268,7 +1269,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 * double checks the vma flags, so that it won't mlock pages
 		 * if the vma was already munlocked.
 		 */
-		ret = populate_vma_page_range(vma, nstart, nend, &locked);
+		ret = populate_vma_page_range(vma, nstart, nend, &nonblocking);
 		if (ret < 0) {
 			if (ignore_errors) {
 				ret = 0;
@@ -1276,6 +1277,14 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			}
 			break;
 		}
+
+		/*
+		 * We dropped the mmap_sem, so we need to re-lock, and the next
+		 * loop around we won't drop because nonblocking is now 0.
+		 */
+		if (!nonblocking)
+			locked = 0;
+
 		nend = nstart + ret * PAGE_SIZE;
 		ret = 0;
 	}


^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 14:56                                     ` Josef Bacik
@ 2019-11-06 15:05                                       ` Jan Kara
  2019-11-06 15:14                                         ` Josef Bacik
  0 siblings, 1 reply; 39+ messages in thread
From: Jan Kara @ 2019-11-06 15:05 UTC (permalink / raw)
  To: Josef Bacik
  Cc: snazy, Jan Kara, Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

On Wed 06-11-19 09:56:09, Josef Bacik wrote:
> On Wed, Nov 06, 2019 at 02:45:43PM +0100, Robert Stupp wrote:
> > On Wed, 2019-11-06 at 13:03 +0100, Jan Kara wrote:
> > > On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> > > > What I don't quite understand yet is why the fault path doesn't
> > > > make
> > > > progress eventually. We must drop the mmap_sem without changing the
> > > > state in any way. How can we keep looping on the same page?
> > >
> > > That may be a slight suboptimality with Josef's patches. If the page
> > > is marked as PageReadahead, we always drop mmap_sem if we can and
> > > start
> > > readahead without checking whether that makes sense or not in
> > > do_async_mmap_readahead(). OTOH page_cache_async_readahead() then
> > > clears
> > > PageReadahead so the only way how I can see we could loop like this
> > > is when
> > > file->ra->ra_pages is 0. Not sure if that's what's happening through.
> > > We'd
> > > need to find which of the paths in filemap_fault() calls
> > > maybe_unlock_mmap_for_io() to tell more.
> > 
> > Yes, ra_pages==0
> > Attached the dmesg + smaps outputs
> > 
> > 
> 
> Ah ok I see what's happening, __get_user_pages() returns 0 if we get an EBUSY
> from faultin_page, and then __mm_populate does nend = nstart + ret * PAGE_SIZE,
> which just leaves us where we are.
> 
> We need to handle the non-blocking and the locking separately in __mm_populate
> so we know what's going on.  Jan's fix for the readahead thing is definitely
> valid as well, but this will keep us from looping forever in other retry cases.

I don't think this will work. AFAICS faultin_page() just checks whether
'nonblocking' is != NULL but doesn't ever look at its value... Honestly the
whole interface is rather weird like lots of things around gup().

								Honza

> 
> diff --git a/mm/gup.c b/mm/gup.c
> index 8f236a335ae9..ac625805d569 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -1237,6 +1237,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
>  	unsigned long end, nstart, nend;
>  	struct vm_area_struct *vma = NULL;
>  	int locked = 0;
> +	int nonblocking = 1;
>  	long ret = 0;
>  
>  	end = start + len;
> @@ -1268,7 +1269,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
>  		 * double checks the vma flags, so that it won't mlock pages
>  		 * if the vma was already munlocked.
>  		 */
> -		ret = populate_vma_page_range(vma, nstart, nend, &locked);
> +		ret = populate_vma_page_range(vma, nstart, nend, &nonblocking);
>  		if (ret < 0) {
>  			if (ignore_errors) {
>  				ret = 0;
> @@ -1276,6 +1277,14 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
>  			}
>  			break;
>  		}
> +
> +		/*
> +		 * We dropped the mmap_sem, so we need to re-lock, and the next
> +		 * loop around we won't drop because nonblocking is now 0.
> +		 */
> +		if (!nonblocking)
> +			locked = 0;
> +
>  		nend = nstart + ret * PAGE_SIZE;
>  		ret = 0;
>  	}
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 15:05                                       ` Jan Kara
@ 2019-11-06 15:14                                         ` Josef Bacik
  2019-11-06 15:25                                           ` Jan Kara
  2019-11-06 16:39                                           ` Robert Stupp
  0 siblings, 2 replies; 39+ messages in thread
From: Josef Bacik @ 2019-11-06 15:14 UTC (permalink / raw)
  To: Jan Kara
  Cc: Josef Bacik, snazy, Johannes Weiner, Vlastimil Babka,
	Michal Hocko, Kirill A. Shutemov, Randy Dunlap, linux-kernel,
	Linux MM, Andrew Morton, Potyra, Stefan

On Wed, Nov 06, 2019 at 04:05:24PM +0100, Jan Kara wrote:
> On Wed 06-11-19 09:56:09, Josef Bacik wrote:
> > On Wed, Nov 06, 2019 at 02:45:43PM +0100, Robert Stupp wrote:
> > > On Wed, 2019-11-06 at 13:03 +0100, Jan Kara wrote:
> > > > On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> > > > > What I don't quite understand yet is why the fault path doesn't
> > > > > make
> > > > > progress eventually. We must drop the mmap_sem without changing the
> > > > > state in any way. How can we keep looping on the same page?
> > > >
> > > > That may be a slight suboptimality with Josef's patches. If the page
> > > > is marked as PageReadahead, we always drop mmap_sem if we can and
> > > > start
> > > > readahead without checking whether that makes sense or not in
> > > > do_async_mmap_readahead(). OTOH page_cache_async_readahead() then
> > > > clears
> > > > PageReadahead so the only way how I can see we could loop like this
> > > > is when
> > > > file->ra->ra_pages is 0. Not sure if that's what's happening through.
> > > > We'd
> > > > need to find which of the paths in filemap_fault() calls
> > > > maybe_unlock_mmap_for_io() to tell more.
> > > 
> > > Yes, ra_pages==0
> > > Attached the dmesg + smaps outputs
> > > 
> > > 
> > 
> > Ah ok I see what's happening, __get_user_pages() returns 0 if we get an EBUSY
> > from faultin_page, and then __mm_populate does nend = nstart + ret * PAGE_SIZE,
> > which just leaves us where we are.
> > 
> > We need to handle the non-blocking and the locking separately in __mm_populate
> > so we know what's going on.  Jan's fix for the readahead thing is definitely
> > valid as well, but this will keep us from looping forever in other retry cases.
> 
> I don't think this will work. AFAICS faultin_page() just checks whether
> 'nonblocking' is != NULL but doesn't ever look at its value... Honestly the
> whole interface is rather weird like lots of things around gup().
> 

Oh what the hell, yeah this is super bonkers.  The whole fault path probably
should be cleaned up to handle retry better.  This will do the trick I think?

Josef

diff --git a/mm/gup.c b/mm/gup.c
index 8f236a335ae9..2468789298e6 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -628,7 +628,7 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
 		fault_flags |= FAULT_FLAG_WRITE;
 	if (*flags & FOLL_REMOTE)
 		fault_flags |= FAULT_FLAG_REMOTE;
-	if (nonblocking)
+	if (nonblocking && *nonblocking != 0)
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY;
 	if (*flags & FOLL_NOWAIT)
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT;
@@ -1237,6 +1237,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 	unsigned long end, nstart, nend;
 	struct vm_area_struct *vma = NULL;
 	int locked = 0;
+	int nonblocking = 1;
 	long ret = 0;
 
 	end = start + len;
@@ -1268,7 +1269,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 		 * double checks the vma flags, so that it won't mlock pages
 		 * if the vma was already munlocked.
 		 */
-		ret = populate_vma_page_range(vma, nstart, nend, &locked);
+		ret = populate_vma_page_range(vma, nstart, nend, &nonblocking);
 		if (ret < 0) {
 			if (ignore_errors) {
 				ret = 0;
@@ -1276,6 +1277,14 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
 			}
 			break;
 		}
+
+		/*
+		 * We dropped the mmap_sem, so we need to re-lock, and the next
+		 * loop around we won't drop because nonblocking is now 0.
+		 */
+		if (!nonblocking)
+			locked = 0;
+
 		nend = nstart + ret * PAGE_SIZE;
 		ret = 0;
 	}


^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 15:14                                         ` Josef Bacik
@ 2019-11-06 15:25                                           ` Jan Kara
  2019-11-06 16:39                                           ` Robert Stupp
  1 sibling, 0 replies; 39+ messages in thread
From: Jan Kara @ 2019-11-06 15:25 UTC (permalink / raw)
  To: Josef Bacik
  Cc: Jan Kara, snazy, Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

On Wed 06-11-19 10:14:31, Josef Bacik wrote:
> On Wed, Nov 06, 2019 at 04:05:24PM +0100, Jan Kara wrote:
> > On Wed 06-11-19 09:56:09, Josef Bacik wrote:
> > > On Wed, Nov 06, 2019 at 02:45:43PM +0100, Robert Stupp wrote:
> > > > On Wed, 2019-11-06 at 13:03 +0100, Jan Kara wrote:
> > > > > On Tue 05-11-19 13:22:11, Johannes Weiner wrote:
> > > > > > What I don't quite understand yet is why the fault path doesn't
> > > > > > make
> > > > > > progress eventually. We must drop the mmap_sem without changing the
> > > > > > state in any way. How can we keep looping on the same page?
> > > > >
> > > > > That may be a slight suboptimality with Josef's patches. If the page
> > > > > is marked as PageReadahead, we always drop mmap_sem if we can and
> > > > > start
> > > > > readahead without checking whether that makes sense or not in
> > > > > do_async_mmap_readahead(). OTOH page_cache_async_readahead() then
> > > > > clears
> > > > > PageReadahead so the only way how I can see we could loop like this
> > > > > is when
> > > > > file->ra->ra_pages is 0. Not sure if that's what's happening through.
> > > > > We'd
> > > > > need to find which of the paths in filemap_fault() calls
> > > > > maybe_unlock_mmap_for_io() to tell more.
> > > > 
> > > > Yes, ra_pages==0
> > > > Attached the dmesg + smaps outputs
> > > > 
> > > > 
> > > 
> > > Ah ok I see what's happening, __get_user_pages() returns 0 if we get an EBUSY
> > > from faultin_page, and then __mm_populate does nend = nstart + ret * PAGE_SIZE,
> > > which just leaves us where we are.
> > > 
> > > We need to handle the non-blocking and the locking separately in __mm_populate
> > > so we know what's going on.  Jan's fix for the readahead thing is definitely
> > > valid as well, but this will keep us from looping forever in other retry cases.
> > 
> > I don't think this will work. AFAICS faultin_page() just checks whether
> > 'nonblocking' is != NULL but doesn't ever look at its value... Honestly the
> > whole interface is rather weird like lots of things around gup().
> > 
> 
> Oh what the hell, yeah this is super bonkers.  The whole fault path probably
> should be cleaned up to handle retry better.  This will do the trick I think?

Yes, this should do it. But then I'm not sure if using
__get_user_pages_locked() instead of __get_user_pages() in
populate_vma_page_range() isn't a better option. I guess I'll defer that
decision to mm people to pick their poison ;)...

								Honza

> diff --git a/mm/gup.c b/mm/gup.c
> index 8f236a335ae9..2468789298e6 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -628,7 +628,7 @@ static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
>  		fault_flags |= FAULT_FLAG_WRITE;
>  	if (*flags & FOLL_REMOTE)
>  		fault_flags |= FAULT_FLAG_REMOTE;
> -	if (nonblocking)
> +	if (nonblocking && *nonblocking != 0)
>  		fault_flags |= FAULT_FLAG_ALLOW_RETRY;
>  	if (*flags & FOLL_NOWAIT)
>  		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT;
> @@ -1237,6 +1237,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
>  	unsigned long end, nstart, nend;
>  	struct vm_area_struct *vma = NULL;
>  	int locked = 0;
> +	int nonblocking = 1;
>  	long ret = 0;
>  
>  	end = start + len;
> @@ -1268,7 +1269,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
>  		 * double checks the vma flags, so that it won't mlock pages
>  		 * if the vma was already munlocked.
>  		 */
> -		ret = populate_vma_page_range(vma, nstart, nend, &locked);
> +		ret = populate_vma_page_range(vma, nstart, nend, &nonblocking);
>  		if (ret < 0) {
>  			if (ignore_errors) {
>  				ret = 0;
> @@ -1276,6 +1277,14 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
>  			}
>  			break;
>  		}
> +
> +		/*
> +		 * We dropped the mmap_sem, so we need to re-lock, and the next
> +		 * loop around we won't drop because nonblocking is now 0.
> +		 */
> +		if (!nonblocking)
> +			locked = 0;
> +
>  		nend = nstart + ret * PAGE_SIZE;
>  		ret = 0;
>  	}
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 15:14                                         ` Josef Bacik
  2019-11-06 15:25                                           ` Jan Kara
@ 2019-11-06 16:39                                           ` Robert Stupp
  2019-11-06 17:03                                             ` Robert Stupp
  1 sibling, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-11-06 16:39 UTC (permalink / raw)
  To: Josef Bacik, Jan Kara
  Cc: Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

[-- Attachment #1: Type: text/plain, Size: 2880 bytes --]

On Wed, 2019-11-06 at 10:14 -0500, Josef Bacik wrote:
> On Wed, Nov 06, 2019 at 04:05:24PM +0100, Jan Kara wrote:
> > On Wed 06-11-19 09:56:09, Josef Bacik wrote:
> > I don't think this will work. AFAICS faultin_page() just checks
> > whether
> > 'nonblocking' is != NULL but doesn't ever look at its value...
> > Honestly the
> > whole interface is rather weird like lots of things around gup().
> >
>
> Oh what the hell, yeah this is super bonkers.  The whole fault path
> probably
> should be cleaned up to handle retry better.  This will do the trick
> I think?

I tried the patch, and it seems to fix the `mlockall(MCL_CURRENT)`
issue for "my test.c".
However, shutdown & reboot are still broken - i.e. the console says
"Reached target reboot" and "hangs forever". Shutdown & reboot work
with __get_user_pages_locked(). No clue what the difference there is.

Anyway, I've captured three smaps outputs: from 5.0.21, from
5.3.9+"nonblocking patch", from 5.3.9+"__get_user_pages_locked". All
three look okay to me - although I don't completely understand why some
areas are not locked (-> "Locked: 0kB") - but "Locked" is always equal
to "Pss", so I assume that's fine?

>
> Josef
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 8f236a335ae9..2468789298e6 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -628,7 +628,7 @@ static int faultin_page(struct task_struct *tsk,
> struct vm_area_struct *vma,
>  		fault_flags |= FAULT_FLAG_WRITE;
>  	if (*flags & FOLL_REMOTE)
>  		fault_flags |= FAULT_FLAG_REMOTE;
> -	if (nonblocking)
> +	if (nonblocking && *nonblocking != 0)
>  		fault_flags |= FAULT_FLAG_ALLOW_RETRY;
>  	if (*flags & FOLL_NOWAIT)
>  		fault_flags |= FAULT_FLAG_ALLOW_RETRY |
> FAULT_FLAG_RETRY_NOWAIT;
> @@ -1237,6 +1237,7 @@ int __mm_populate(unsigned long start, unsigned
> long len, int ignore_errors)
>  	unsigned long end, nstart, nend;
>  	struct vm_area_struct *vma = NULL;
>  	int locked = 0;
> +	int nonblocking = 1;
>  	long ret = 0;
>
>  	end = start + len;
> @@ -1268,7 +1269,7 @@ int __mm_populate(unsigned long start, unsigned
> long len, int ignore_errors)
>  		 * double checks the vma flags, so that it won't mlock
> pages
>  		 * if the vma was already munlocked.
>  		 */
> -		ret = populate_vma_page_range(vma, nstart, nend,
> &locked);
> +		ret = populate_vma_page_range(vma, nstart, nend,
> &nonblocking);
>  		if (ret < 0) {
>  			if (ignore_errors) {
>  				ret = 0;
> @@ -1276,6 +1277,14 @@ int __mm_populate(unsigned long start,
> unsigned long len, int ignore_errors)
>  			}
>  			break;
>  		}
> +
> +		/*
> +		 * We dropped the mmap_sem, so we need to re-lock, and
> the next
> +		 * loop around we won't drop because nonblocking is now
> 0.
> +		 */
> +		if (!nonblocking)
> +			locked = 0;
> +
>  		nend = nstart + ret * PAGE_SIZE;
>  		ret = 0;
>  	}

[-- Attachment #2: smaps-5.0.21.txt.gz --]
[-- Type: application/gzip, Size: 1049 bytes --]

[-- Attachment #3: smaps-add-nonblocking.txt.gz --]
[-- Type: application/gzip, Size: 1068 bytes --]

[-- Attachment #4: smaps-get_user_pages_locked.txt.gz --]
[-- Type: application/gzip, Size: 1077 bytes --]

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 16:39                                           ` Robert Stupp
@ 2019-11-06 17:03                                             ` Robert Stupp
  2019-11-06 17:25                                               ` Jan Kara
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-11-06 17:03 UTC (permalink / raw)
  To: Josef Bacik, Jan Kara
  Cc: Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

Oh, and I just realized, that I have two custom settings in my
/etc/rc.local - guess, I configured that when I installed that machine
years ago. Sorry, that I haven't mentioned that earlier.

echo 0 > /sys/block/nvme0n1/queue/read_ahead_kb
echo never >
/sys/kernel/mm/transparent_hugepage/defrag

That setting is probably not quite standard, but I assume there are
some database server setups out there, that set RA=0 as well.

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 17:03                                             ` Robert Stupp
@ 2019-11-06 17:25                                               ` Jan Kara
  2019-11-07  8:08                                                 ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Jan Kara @ 2019-11-06 17:25 UTC (permalink / raw)
  To: snazy
  Cc: Josef Bacik, Jan Kara, Johannes Weiner, Vlastimil Babka,
	Michal Hocko, Kirill A. Shutemov, Randy Dunlap, linux-kernel,
	Linux MM, Andrew Morton, Potyra, Stefan

On Wed 06-11-19 18:03:10, Robert Stupp wrote:
> Oh, and I just realized, that I have two custom settings in my
> /etc/rc.local - guess, I configured that when I installed that machine
> years ago. Sorry, that I haven't mentioned that earlier.
> 
> echo 0 > /sys/block/nvme0n1/queue/read_ahead_kb
> echo never >
> /sys/kernel/mm/transparent_hugepage/defrag
> 
> That setting is probably not quite standard, but I assume there are
> some database server setups out there, that set RA=0 as well.

Ok, yes, that explains ra_pages == 0. Thanks for the details.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR


^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-06 17:25                                               ` Jan Kara
@ 2019-11-07  8:08                                                 ` Robert Stupp
  2019-11-20 12:42                                                   ` Robert Stupp
  0 siblings, 1 reply; 39+ messages in thread
From: Robert Stupp @ 2019-11-07  8:08 UTC (permalink / raw)
  To: Jan Kara
  Cc: Josef Bacik, Johannes Weiner, Vlastimil Babka, Michal Hocko,
	Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

On Wed, 2019-11-06 at 18:25 +0100, Jan Kara wrote:
> On Wed 06-11-19 18:03:10, Robert Stupp wrote:
> > Oh, and I just realized, that I have two custom settings in my
> > /etc/rc.local - guess, I configured that when I installed that
> > machine
> > years ago. Sorry, that I haven't mentioned that earlier.
> >
> > echo 0 > /sys/block/nvme0n1/queue/read_ahead_kb
> > echo never >
> > /sys/kernel/mm/transparent_hugepage/defrag
> >
> > That setting is probably not quite standard, but I assume there are
> > some database server setups out there, that set RA=0 as well.
>
> Ok, yes, that explains ra_pages == 0. Thanks for the details.

Ah! And with these settings, a fresh & clean installation of Ubuntu
eoan (5.3.x kernel), hangs during reboot.
And if the VM's booted with these settings in /etc/rc.local, the
mlockall(MCL_CURRENT) is reproducible as well.




^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-11-07  8:08                                                 ` Robert Stupp
@ 2019-11-20 12:42                                                   ` Robert Stupp
  0 siblings, 0 replies; 39+ messages in thread
From: Robert Stupp @ 2019-11-20 12:42 UTC (permalink / raw)
  To: Jan Kara, Josef Bacik, Johannes Weiner, Vlastimil Babka, Michal Hocko
  Cc: Kirill A. Shutemov, Randy Dunlap, linux-kernel, Linux MM,
	Andrew Morton, Potyra, Stefan

Is there any chance to get a patch merged?

On Thu, 2019-11-07 at 09:08 +0100, Robert Stupp wrote:
> On Wed, 2019-11-06 at 18:25 +0100, Jan Kara wrote:
> > On Wed 06-11-19 18:03:10, Robert Stupp wrote:
> > > Oh, and I just realized, that I have two custom settings in my
> > > /etc/rc.local - guess, I configured that when I installed that
> > > machine
> > > years ago. Sorry, that I haven't mentioned that earlier.
> > >
> > > echo 0 > /sys/block/nvme0n1/queue/read_ahead_kb
> > > echo never >
> > > /sys/kernel/mm/transparent_hugepage/defrag
> > >
> > > That setting is probably not quite standard, but I assume there
> > > are
> > > some database server setups out there, that set RA=0 as well.
> >
> > Ok, yes, that explains ra_pages == 0. Thanks for the details.
>
> Ah! And with these settings, a fresh & clean installation of Ubuntu
> eoan (5.3.x kernel), hangs during reboot.
> And if the VM's booted with these settings in /etc/rc.local, the
> mlockall(MCL_CURRENT) is reproducible as well.
>
>



^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: mlockall(MCL_CURRENT) blocking infinitely
  2019-10-25 13:27                 ` Michal Hocko
  2019-10-25 13:45                   ` Robert Stupp
@ 2019-10-25 13:55                   ` Robert Stupp
  1 sibling, 0 replies; 39+ messages in thread
From: Robert Stupp @ 2019-10-25 13:55 UTC (permalink / raw)
  To: Michal Hocko, snazy
  Cc: Randy Dunlap, linux-kernel, Linux MM, Andrew Morton, Potyra, Stefan

On Fri, 2019-10-25 at 15:27 +0200, Michal Hocko wrote:
> > Pss:                   5 kB

Seems Pss==Locked is wrong here - or better: it doesn't look right that
Locked==Pss?



^ permalink raw reply	[flat|nested] 39+ messages in thread

end of thread, other threads:[~2019-11-20 12:42 UTC | newest]

Thread overview: 39+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <4576b336-66e6-e2bb-cd6a-51300ed74ab8@snazy.de>
2019-10-24 23:34 ` mlockall(MCL_CURRENT) blocking infinitely Randy Dunlap
2019-10-25  9:21   ` Michal Hocko
2019-10-25 11:02     ` Robert Stupp
2019-10-25 11:46       ` Michal Hocko
2019-10-25 11:50         ` Michal Hocko
2019-10-25 11:59           ` Robert Stupp
2019-10-25 13:19             ` Robert Stupp
2019-10-25 11:55         ` Robert Stupp
2019-10-25 12:05           ` Michal Hocko
2019-10-25 12:11             ` Michal Hocko
2019-10-25 13:10               ` Robert Stupp
2019-10-25 13:27                 ` Michal Hocko
2019-10-25 13:45                   ` Robert Stupp
2019-10-25 13:57                     ` Michal Hocko
2019-10-25 14:00                       ` Michal Hocko
2019-10-25 15:58                         ` Robert Stupp
2019-11-05 13:23                           ` Robert Stupp
2019-11-05 15:28                             ` Vlastimil Babka
2019-11-05 18:22                               ` Johannes Weiner
2019-11-05 20:05                                 ` Robert Stupp
2019-11-06 10:25                                   ` Robert Stupp
2019-11-06 11:26                                     ` Robert Stupp
2019-11-06 12:04                                       ` Jan Kara
2019-11-06 12:24                                       ` Robert Stupp
2019-11-06 12:03                                 ` Jan Kara
2019-11-06 13:45                                   ` Robert Stupp
2019-11-06 14:35                                     ` Jan Kara
2019-11-06 15:32                                       ` Robert Stupp
2019-11-06 14:38                                     ` Jan Kara
2019-11-06 14:56                                     ` Josef Bacik
2019-11-06 15:05                                       ` Jan Kara
2019-11-06 15:14                                         ` Josef Bacik
2019-11-06 15:25                                           ` Jan Kara
2019-11-06 16:39                                           ` Robert Stupp
2019-11-06 17:03                                             ` Robert Stupp
2019-11-06 17:25                                               ` Jan Kara
2019-11-07  8:08                                                 ` Robert Stupp
2019-11-20 12:42                                                   ` Robert Stupp
2019-10-25 13:55                   ` Robert Stupp

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).