Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Pratyush Anand]

Hi All,

One of the hugetlbfs test[1] fails on ARM64 kernel with Segmentation 
fault. Segmentation fault (unhandled level 2 translation fault [3]) 
happens just after kernel returns to user space from mmap() system call. 
See the tailored version of failing test case [2] (it reproduces in 
maximum 10 trials).

When it fails lr shows the address of the instruction which follow 
mmap() call in main. Also, if test is run with strace, then it says that 
mmap() system call was fine. (so,I believe lr is correct). Cross checked 
from /proc/pid/maps and PC is also correct.It is the address of 
instruction where kernel will return after executing mmap() system call. 
So, when the first user space instruction is executed after returning 
from kernel, it gives segmentation fault. Not sure what went wrong..Just 
a guess, probably this page was swapped out (core dump does not have 
this page), but kernel expect it to be there. Any pointer to debug will 
be helpful.

Following is one failing scenario, which leads to above assertions.

 From /proc/pid/maps:
ffffa1760000-ffffa18c0000 r-xp 00000000 fd:00 33718606 
/usr/lib64/libc-2.17.so

from dmesg:
[34928.473302] PC is at 0xffffa1835a44
[34928.476771] LR is at 0x400a3c

offset of failing address in libc-2.17.so =   0xffffa1835a44 - 
0xffffa1760000 = 0xD5A44

from objdump of  libc-2.17.so:
00000000000d5a30 <mmap>:
    d5a30:       93407c42        sxtw    x2, w2
    d5a34:       93407c63        sxtw    x3, w3
    d5a38:       93407c84        sxtw    x4, w4
    d5a3c:       d2801bc8        mov     x8, #0xde 
// #222
    d5a40:       d4000001        svc     #0x0
    d5a44:       b140041f        cmn     x0, #0x1, lsl #12
    d5a48:       54000048        b.hi    d5a50 <mmap+0x20>

and from objdump of test (hugetlb_test_stack):
00000000004008dc <main>:
   [...]
   400a30:       b940f7a4        ldr     w4, [x29,#244]
   400a34:       d2800005        mov     x5, #0x0 
// #0
   400a38:       97ffff3e        bl      400730 <mmap@plt>
   400a3c:       f9006fa0        str     x0, [x29,#216]

from core dump:
Program terminated with signal 11, Segmentation fault.
#0  0x0000ffffa1835a44 in ?? ()

(gdb) x/g 0x0000ffffa1835a44
0xffffa1835a44: Cannot access memory at address 0xffffa1835a44

~Pratyush


[1] 
https://github.com/libhugetlbfs/libhugetlbfs/blob/master/tests/stack_grow_into_huge.c

[2]
---------------------------------------------------------------------------
# cat hugetlb_test_stack.c

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/types.h>

#define ALIGN(x, a)     (((x) + (a) - 1) & ~((a) - 1))
#define PALIGN(p, a)    ((void *)ALIGN((unsigned long)(p), (a)))

int main(int argc, char *argv[])
{
         long hpage_size;;
         void *stack_address, *mmap_address, *mmap_ret_address;
         struct rlimit r;
         int fd;

         if (argc < 3) {
                 printf("Pass hugetlb page size as 1st argument and
path of a file in hugetlbfs as second argument\n");
                 exit(0);
         }
         hpage_size = atol(argv[1]);

         printf("hpage_size is %lx\n", hpage_size);
         printf("file path is %s\n", argv[2]);
         r.rlim_cur = RLIM_INFINITY;
         r.rlim_max = RLIM_INFINITY;
         setrlimit(RLIMIT_STACK, &r);

         fd = open(argv[2], O_RDWR);

         if (fd < 0) {
                 printf("open() failed: %s\n", strerror(errno));
                 return -1;
         }

         stack_address = alloca(0);
         mmap_address = PALIGN(stack_address - 2 * hpage_size, hpage_size);

         printf("Address to be mapped is %p\n", mmap_address);
         mmap_ret_address = mmap(mmap_address, hpage_size, 
PROT_READ|PROT_WRITE,
                                 MAP_FIXED|MAP_SHARED, fd, 0);
         printf("mmap_ret_address is %p\n", mmap_ret_address);
}

# gcc -o hugetlb_test_stack hugetlb_test_stack.c
# ls /sys/kernel/mm/hugepages/
hugepages-2048kB  hugepages-524288kB

I used 524288KB page size file for test. It did not reproduces with 
2048K page size.

# echo 5 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
#  mount -t hugetlbfs none /mnt/hugetlbfs -o pagesize=524288K
# touch /mnt/hugetlbfs/test
# ./hugetlb_test_stack 536870912  /mnt/hugetlbfs/test
---------------------------------------------------------------------------

[3]
[34928.425865] hugetlb_test_st[10566]: unhandled level 2 translation 
fault (11) at 0xffffa1835a44, esr 0x82000006
[34928.435848] pgd = ffff8003c89eec00
[34928.439231] [ffffa1835a44] *pgd=00000043cfc70003, 
*pud=00000043cfc70003, *pmd=0000000000000000
[34928.447818]
[34928.449303] CPU: 1 PID: 10566 Comm: hugetlb_test_st Not tainted 
4.10.0-XXXXXX.aarch64 #1
[34928.457706] Hardware name: AppliedMicro X-Gene Mustang Board/X-Gene 
Mustang Board, BIOS 3.06.25 Oct 17 2016
[34928.467405] task: ffff8003c89b8000 task.stack: ffff800370294000
[34928.473302] PC is at 0xffffa1835a44
[34928.476771] LR is at 0x400a3c
[34928.479721] pc : [<0000ffffa1835a44>] lr : [<0000000000400a3c>] 
pstate: 80000000
[34928.487095] sp : 0000ffffc7384fd0
[34928.490400] x29: 0000ffffc7384fd0 x28: 0000000000000000
[34928.495685] x27: 0000000000000000 x26: 0000000000000000
[34928.500980] x25: 0000000000000000 x24: 0000000000000000
[34928.506265] x23: 0000000000000000 x22: 0000000000000000
[34928.511557] x21: 0000000000400770 x20: 0000000000000000
[34928.516841] x19: 0000000000000000 x18: 0000ffffc7384da0
[34928.522130] x17: 0000000000420048 x16: 0000ffffa1835a30
[34928.527414] x15: 00000000001815e7 x14: 0000ffffa194ffb8
[34928.532702] x13: ffffffffffffffff x12: 0000000000000013
[34928.537986] x11: 0000ffffc738f6de x10: 00000000ffffffff
[34928.543276] x9 : 0000000000400bd9 x8 : 00000000000000de
[34928.548559] x7 : 0000000000000000 x6 : 0000000000000000
[34928.553848] x5 : 0000000000000000 x4 : 0000000000000003
[34928.559131] x3 : 0000000000000011 x2 : 0000000000000003
[34928.564431] x1 : 0000000020000000 x0 : 0000ffffa0000000
[34928.569718]

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Mark Rutland]

Hi,

On Fri, Mar 24, 2017 at 07:51:34PM +0530, Pratyush Anand wrote:
> # cat hugetlb_test_stack.c
> 
> #include <errno.h>
> #include <fcntl.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <sys/mman.h>
> #include <sys/resource.h>
> #include <sys/stat.h>
> #include <sys/time.h>
> #include <sys/types.h>
> 
> #define ALIGN(x, a)     (((x) + (a) - 1) & ~((a) - 1))
> #define PALIGN(p, a)    ((void *)ALIGN((unsigned long)(p), (a)))
> 
> int main(int argc, char *argv[])
> {
>         long hpage_size;;
>         void *stack_address, *mmap_address, *mmap_ret_address;
>         struct rlimit r;
>         int fd;
> 
>         if (argc < 3) {
>                 printf("Pass hugetlb page size as 1st argument and
> path of a file in hugetlbfs as second argument\n");
>                 exit(0);
>         }
>         hpage_size = atol(argv[1]);
> 
>         printf("hpage_size is %lx\n", hpage_size);
>         printf("file path is %s\n", argv[2]);
>         r.rlim_cur = RLIM_INFINITY;
>         r.rlim_max = RLIM_INFINITY;
>         setrlimit(RLIMIT_STACK, &r);
> 
>         fd = open(argv[2], O_RDWR);
> 
>         if (fd < 0) {
>                 printf("open() failed: %s\n", strerror(errno));
>                 return -1;
>         }
> 
>         stack_address = alloca(0);
>         mmap_address = PALIGN(stack_address - 2 * hpage_size, hpage_size);
> 
>         printf("Address to be mapped is %p\n", mmap_address);
>         mmap_ret_address = mmap(mmap_address, hpage_size,
> PROT_READ|PROT_WRITE,
>                                 MAP_FIXED|MAP_SHARED, fd, 0);
>         printf("mmap_ret_address is %p\n", mmap_ret_address);
> }
> 
> # gcc -o hugetlb_test_stack hugetlb_test_stack.c
> # ls /sys/kernel/mm/hugepages/
> hugepages-2048kB  hugepages-524288kB
> 
> I used 524288KB page size file for test. It did not reproduces with
> 2048K page size.
> 
> # echo 5 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
> #  mount -t hugetlbfs none /mnt/hugetlbfs -o pagesize=524288K
> # touch /mnt/hugetlbfs/test
> # ./hugetlb_test_stack 536870912  /mnt/hugetlbfs/test

I modified the test to print out some additional information, and ran it under
GDB, with a breakpoint on main, and ASLR enabled with:

	set disable-randomization off

It's clear from the log that the test is simply blatting a number of
important mappings including libc, so I think this is simply a broken
test.

(gdb) run               
Starting program: /host/home/nanook/htfs 536870912 /mnt/hugetlbfs/test

Breakpoint 1, 0x0000000000400848 in main ()
(gdb) info proc mappings
process 1301
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
            0x400000           0x410000    0x10000        0x0 /host/home/nanook/htfs
            0x410000           0x420000    0x10000        0x0 /host/home/nanook/htfs
      0xffffa76f0000     0xffffa7700000    0x10000        0x0 
      0xffffa7700000     0xffffa7830000   0x130000        0x0 /host/lib/aarch64-linux-gnu/libc-2.19.so
      0xffffa7830000     0xffffa7840000    0x10000   0x130000 /host/lib/aarch64-linux-gnu/libc-2.19.so
      0xffffa7840000     0xffffa7850000    0x10000        0x0 [vvar]
      0xffffa7850000     0xffffa7860000    0x10000        0x0 [vdso]
      0xffffa7860000     0xffffa7880000    0x20000        0x0 /host/lib/aarch64-linux-gnu/ld-2.19.so
      0xffffa7880000     0xffffa7890000    0x10000    0x10000 /host/lib/aarch64-linux-gnu/ld-2.19.so
      0xffffd90f0000     0xffffd9120000    0x30000        0x0 [stack]
(gdb) continue          
Continuing.
hpage_size is 20000000
file path is /mnt/hugetlbfs/test
Range to be mapped is 0xffffa0000000-0xffffbfffffff
Currently main at 0x400840

Program received signal SIGILL, Illegal instruction.
0x0000ffffa77c4c04 in __mmap (addr=<optimized out>, len=536870912, prot=3, 
    flags=17, fd=3, offset=0)
    at ../ports/sysdeps/unix/sysv/linux/aarch64/mmap.c:29
29      ../ports/sysdeps/unix/sysv/linux/aarch64/mmap.c: No such file or directory.

Thanks,
Mark.

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Pratyush Anand]

Hi Mark,

On Friday 24 March 2017 09:45 PM, Mark Rutland wrote:
> It's clear from the log that the test is simply blatting a number of
> important mappings including libc, so I think this is simply a broken
> test.

Thanks a lot for taking out time and investigating it so quickly. Yes, I 
noticed that as well. Frankly, I do not understand that why that 
upstream libhugetlbfs test is like that. But that test has been passing 
on different architecture for quite long time.

Moreover, even if mmap() in test routine crosses over many other 
text/rd/rw area mappings, should it fail? We are not writing anything to 
mmaped area. So, why should just a creation of another map result in 
segmentation fault?

~Pratyush

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Mark Rutland]

On Fri, Mar 24, 2017 at 10:11:10PM +0530, Pratyush Anand wrote:
> Hi Mark,
> 
> On Friday 24 March 2017 09:45 PM, Mark Rutland wrote:
> >It's clear from the log that the test is simply blatting a number of
> >important mappings including libc, so I think this is simply a broken
> >test.
> 
> Thanks a lot for taking out time and investigating it so quickly.
> Yes, I noticed that as well. Frankly, I do not understand that why
> that upstream libhugetlbfs test is like that. But that test has been
> passing on different architecture for quite long time.

I guess that on other architectures, the test is run with a smaller
hugepage size, which happens to not clobber impoertant data.

> Moreover, even if mmap() in test routine crosses over many other
> text/rd/rw area mappings, should it fail? We are not writing
> anything to mmaped area. So, why should just a creation of another
> map result in segmentation fault?

The new mapping replaces the old mappings that it clobbers, so all the
old data/code is gone. Loads or instruction fetches will see data from
the new mapping.

In my case, since the libc code was clobbered, the CPU tried to execute
the zeroes it was clobbered with, and took an illegal instruction abort.

For your report, it's not clear to me what's going on. Did you take the
/proc/pid/maps data from teh exact same process that the segfault
occurred in? and/or did you disable ASLR?

Thanks,
Mark.

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Pratyush Anand]

On Friday 24 March 2017 10:55 PM, Mark Rutland wrote:
>> Moreover, even if mmap() in test routine crosses over many other
>> text/rd/rw area mappings, should it fail? We are not writing
>> anything to mmaped area. So, why should just a creation of another
>> map result in segmentation fault?
> The new mapping replaces the old mappings that it clobbers, so all the
> old data/code is gone. Loads or instruction fetches will see data from
> the new mapping.
>

Humm..I see..But in that case mmap() should have failed and return 
MAP_FAILED instead of remapping and which could cause a segfault.
I see that upstream test case, has mmap() in a loop and it goes further 
only with a successful mapping. So, if it would have failed then 
probably test would have tried next address for mmaping until it would 
have touched heap area.

> In my case, since the libc code was clobbered, the CPU tried to execute
> the zeroes it was clobbered with, and took an illegal instruction abort.
>
> For your report, it's not clear to me what's going on. Did you take the
> /proc/pid/maps data from teh exact same process that the segfault
> occurred in? and/or did you disable ASLR?

Yes, it is from the same process. Since, I was not able to reproduce 
with gdb so, I had inserted a scanf() just before mmap() and then had 
read /proc/pid/maps.

~Pratyush

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Mark Rutland]

On Fri, Mar 24, 2017 at 11:32:58PM +0530, Pratyush Anand wrote:
> 
> 
> On Friday 24 March 2017 10:55 PM, Mark Rutland wrote:
> >>Moreover, even if mmap() in test routine crosses over many other
> >>text/rd/rw area mappings, should it fail? We are not writing
> >>anything to mmaped area. So, why should just a creation of another
> >>map result in segmentation fault?
> >The new mapping replaces the old mappings that it clobbers, so all the
> >old data/code is gone. Loads or instruction fetches will see data from
> >the new mapping.
> 
> Humm..I see..But in that case mmap() should have failed and return
> MAP_FAILED instead of remapping and which could cause a segfault.

That does nto appear to be the case. As per the mmap man page:

  MAP_FIXED
      Don't  interpret addr as a hint: place the mapping at exactly that
      address.  addr must be a multiple of the page size.  If the memory
      region specified by addr and len overlaps pages of any existing
      mapping(s), then the overlapped part of the existing mapping(s)
      will  be discarded.  If the specified address cannot be used,
      mmap() will fail.  Because requiring a fixed address for a mapping
      is less portable, the use of this option is discouraged.

[...]

> >For your report, it's not clear to me what's going on. Did you take the
> >/proc/pid/maps data from teh exact same process that the segfault
> >occurred in? and/or did you disable ASLR?
> 
> Yes, it is from the same process.

That is troubling; I cannot explain that.

> Since, I was not able to reproduce with gdb so, I had inserted a
> scanf() just before mmap() and then had read /proc/pid/maps.

That might be because GDB disables ASLR by default. Did you re-enable
ASLR within GDB with:

	set disable-randomization off

If not, could you give that a go?

Thanks,
Mark.

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Pratyush Anand]

On Friday 24 March 2017 11:46 PM, Mark Rutland wrote:
>>> For your report, it's not clear to me what's going on. Did you take the
>>> /proc/pid/maps data from teh exact same process that the segfault
>>> occurred in? and/or did you disable ASLR?
>> Yes, it is from the same process.
> That is troubling; I cannot explain that.

Can you pl try in an infinite loop for some time and see if "SIGSEGV" is 
received in any of the run at your end.

# while [[ 1 ]]; do  ./hugetlb_test_stack 536870912 
/mnt/hugetlbfs/test;done

>
>> Since, I was not able to reproduce with gdb so, I had inserted a
>> scanf() just before mmap() and then had read /proc/pid/maps.
> That might be because GDB disables ASLR by default. Did you re-enable
> ASLR within GDB with:
>
> 	set disable-randomization off
>
> If not, could you give that a go?

Yes, with ASLR enabled, it reproduced in GDB as well. I do not see 
SIGILL, it is SIGSEGV there too.

(gdb) set disable-randomization off
(gdb) b main
Breakpoint 1 at 0x400884
(gdb) r
Starting program: /home/panand/work/hugetlb/./hugetlb_test_stack 
536870912 /mnt/hugetlbfs/test

Breakpoint 1, 0x0000000000400884 in main ()
(gdb) info proc mappings
process 2949
Mapped address spaces:

           Start Addr           End Addr       Size     Offset objfile
             0x400000           0x410000    0x10000        0x0 
/home/panand/work/hugetlb/hugetlb_test_stack
             0x410000           0x420000    0x10000        0x0 
/home/panand/work/hugetlb/hugetlb_test_stack
             0x420000           0x430000    0x10000    0x10000 
/home/panand/work/hugetlb/hugetlb_test_stack
       0xffffada70000     0xffffadbd0000   0x160000        0x0 
/usr/lib64/libc-2.17.so
       0xffffadbd0000     0xffffadbe0000    0x10000   0x150000 
/usr/lib64/libc-2.17.so
       0xffffadbe0000     0xffffadbf0000    0x10000   0x160000 
/usr/lib64/libc-2.17.so
       0xffffadc10000     0xffffadc20000    0x10000        0x0 [vvar]
       0xffffadc20000     0xffffadc30000    0x10000        0x0 [vdso]
       0xffffadc30000     0xffffadc50000    0x20000        0x0 
/usr/lib64/ld-2.17.so
       0xffffadc50000     0xffffadc60000    0x10000    0x10000 
/usr/lib64/ld-2.17.so
       0xffffadc60000     0xffffadc70000    0x10000    0x20000 
/usr/lib64/ld-2.17.so
       0xffffcb1d0000     0xffffcb200000    0x30000        0x0 [stack]
(gdb) c
Continuing.
hpage_size is 20000000
file path is /mnt/hugetlbfs/test
stack_address is 0xffffcb1facc0
Address to be mapped is 0xffffa0000000

Program received signal SIGSEGV, Segmentation fault.
0x0000ffffadb45a44 in __mmap (addr=<optimized out>, len=536870912, 
prot=3, flags=17, fd=7, offset=0)
     at ../ports/sysdeps/unix/sysv/linux/aarch64/mmap.c:29
29        return (__ptr_t) INLINE_SYSCALL (mmap, 6, addr, len, prot, 
flags, fd, offset);
(gdb)

~Pratyush

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Mark Rutland]

On Sat, Mar 25, 2017 at 05:44:58PM +0530, Pratyush Anand wrote:
> On Friday 24 March 2017 11:46 PM, Mark Rutland wrote:
> >>>For your report, it's not clear to me what's going on. Did you take the
> >>>/proc/pid/maps data from teh exact same process that the segfault
> >>>occurred in? and/or did you disable ASLR?
> >>Yes, it is from the same process.
> >That is troubling; I cannot explain that.
> 
> Can you pl try in an infinite loop for some time and see if
> "SIGSEGV" is received in any of the run at your end.

After several thousand runs, I see a few unhandled translations faults
(all for address 0) in dmesg. 

I suspect that in this case, the hugepage has clobbered some
datastructure used shortly after the return from the syscall, and we end
up dereferencing a pointer that's been replaced with zeroes.

> >>Since, I was not able to reproduce with gdb so, I had inserted a
> >>scanf() just before mmap() and then had read /proc/pid/maps.
> >That might be because GDB disables ASLR by default. Did you re-enable
> >ASLR within GDB with:
> >
> >	set disable-randomization off
> >
> >If not, could you give that a go?
> 
> Yes, with ASLR enabled, it reproduced in GDB as well. I do not see
> SIGILL, it is SIGSEGV there too.

So far, I have not managed to trigger a single SIGSEGV while running
under GDB.

However, I have a theory that could explain that. I suspect that my
toolchain has built the binary with an executable stack, while yours has
not. Linux automatically sets READ_IMPLIES_EXEC for binaries with
executable stacks, which IIUC would implicitly make the mmap RWX rather
than RW.

So in my case, the huge page is executable, and I get a SIGILL when
trying to execute from it. In your case, the huge page is not
executable, so you get a SIGSEGV.

Looking at your report below:

> Mapped address spaces:
> 
>           Start Addr           End Addr       Size     Offset objfile
>             0x400000           0x410000    0x10000        0x0
> /home/panand/work/hugetlb/hugetlb_test_stack
>             0x410000           0x420000    0x10000        0x0
> /home/panand/work/hugetlb/hugetlb_test_stack
>             0x420000           0x430000    0x10000    0x10000
> /home/panand/work/hugetlb/hugetlb_test_stack

All the entries from here ...

>       0xffffada70000     0xffffadbd0000   0x160000        0x0
> /usr/lib64/libc-2.17.so
>       0xffffadbd0000     0xffffadbe0000    0x10000   0x150000
> /usr/lib64/libc-2.17.so
>       0xffffadbe0000     0xffffadbf0000    0x10000   0x160000
> /usr/lib64/libc-2.17.so
>       0xffffadc10000     0xffffadc20000    0x10000        0x0 [vvar]
>       0xffffadc20000     0xffffadc30000    0x10000        0x0 [vdso]
>       0xffffadc30000     0xffffadc50000    0x20000        0x0
> /usr/lib64/ld-2.17.so
>       0xffffadc50000     0xffffadc60000    0x10000    0x10000
> /usr/lib64/ld-2.17.so
>       0xffffadc60000     0xffffadc70000    0x10000    0x20000

... to here ...

> /usr/lib64/ld-2.17.so
>       0xffffcb1d0000     0xffffcb200000    0x30000        0x0 [stack]
> (gdb) c
> Continuing.
> hpage_size is 20000000
> file path is /mnt/hugetlbfs/test
> stack_address is 0xffffcb1facc0
> Address to be mapped is 0xffffa0000000

... are clobbered by this map, which will cover the range:
	
	0xffffa0000000-0xFFFFC0000000

> Program received signal SIGSEGV, Segmentation fault.
> 0x0000ffffadb45a44 in __mmap (addr=<optimized out>, len=536870912,
> prot=3, flags=17, fd=7, offset=0)

That address falls within libc-2.17.so, which is clobbered by the mmap.

Do you happen to know how to parse that 'prot=3' in the SEGV report? I'm
guessing that means RW, !X.

Thanks,
Mark.

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Pratyush Anand]

Hi Mark,

Thanks a lot for your explanations!!

On Monday 27 March 2017 05:48 PM, Mark Rutland wrote:
> So far, I have not managed to trigger a single SIGSEGV while running
> under GDB.
>
> However, I have a theory that could explain that. I suspect that my
> toolchain has built the binary with an executable stack, while yours has
> not. Linux automatically sets READ_IMPLIES_EXEC for binaries with
> executable stacks, which IIUC would implicitly make the mmap RWX rather
> than RW.
>
> So in my case, the huge page is executable, and I get a SIGILL when
> trying to execute from it. In your case, the huge page is not
> executable, so you get a SIGSEGV.

Yes, your theory seems  convincing.
I passed PROT_EXEC as well along with PROT_READ|PROT_WRITE to mmap(),and 
then I received SIGILL.

[...]

>> Program received signal SIGSEGV, Segmentation fault.
>> 0x0000ffffadb45a44 in __mmap (addr=<optimized out>, len=536870912,
>> prot=3, flags=17, fd=7, offset=0)
> That address falls within libc-2.17.so, which is clobbered by the mmap.
>
> Do you happen to know how to parse that 'prot=3' in the SEGV report? I'm
> guessing that means RW, !X.

Yes, it seems like that.

https://sourceware.org/git/?p=glibc.git;a=blob;f=bits/mman-linux.h;h=8126ce836947718576f09cd8f874a25b2c1340f7;hb=HEAD

~Pratyush

Query: ARM64: A random failure with hugetlbfs linked mmap() of a stack area

[Mark Rutland]

On Mon, Mar 27, 2017 at 06:50:53PM +0530, Pratyush Anand wrote:
> Hi Mark,
> 
> Thanks a lot for your explanations!!
> 
> On Monday 27 March 2017 05:48 PM, Mark Rutland wrote:
> >So far, I have not managed to trigger a single SIGSEGV while running
> >under GDB.
> >
> >However, I have a theory that could explain that. I suspect that my
> >toolchain has built the binary with an executable stack, while yours has
> >not. Linux automatically sets READ_IMPLIES_EXEC for binaries with
> >executable stacks, which IIUC would implicitly make the mmap RWX rather
> >than RW.
> >
> >So in my case, the huge page is executable, and I get a SIGILL when
> >trying to execute from it. In your case, the huge page is not
> >executable, so you get a SIGSEGV.
> 
> Yes, your theory seems  convincing.
> I passed PROT_EXEC as well along with PROT_READ|PROT_WRITE to
> mmap(),and then I received SIGILL.

Ok, I think that explains it, then.

I was a little confused by the PMD being NULL in your initial splat, but
from a look at the mmap() codepaths, we lazily fault in hugetlbfs
mappings unless explicitly asked to populate them.

So mmap would have cleared any existing mappings (leaving the PMD NULL), 
set up the VMA as RW, then returned without setting up the new mapping
inthe page tables. Since that clobbered libc, when we get back to
userspace we immediately take an instruction abort, and since we're not
supposed to have execute permission, we get a SEGV.

Thanks,
Mark.