* brk() system call, RLIMIT_AS, and RLIMIT_DATA in 32-bit compatibility mode
[not found] <ae7baec9-7075-4eca-b4ec-969b072fcd9d@zmail15.collab.prod.int.phx2.redhat.com>
@ 2011-12-21 15:58 ` Ramon de C Valle
2011-12-22 9:46 ` Ramon de C Valle
0 siblings, 1 reply; 2+ messages in thread
From: Ramon de C Valle @ 2011-12-21 15:58 UTC (permalink / raw)
To: linux-kernel
Hi,
Today I came across an interesting behaviour and would like some enlightenment of what I'm probably missing.
Considering the following program:
.globl _start
_start:
xorl %ebx,%ebx
pushl $0xc0000000
popl %ebx
continue:
addl $0x01000000,%ebx
xorl %eax,%eax
movb $0x2d,%al
int $0x80
jmp continue
This is the strace output of the above program in a 32-bit machine:
[...]
brk(0xb9000000) = 0xb9000000
brk(0xba000000) = 0xba000000
brk(0xbb000000) = 0xbb000000
brk(0xbc000000) = 0xbc000000
brk(0xbd000000) = 0xbd000000
brk(0xbe000000) = 0xbe000000
brk(0xbf000000) = 0xbf000000
brk(0xc0000000) = 0xbf000000
brk(0xc1000000) = 0xbf000000
brk(0xc2000000) = 0xbf000000
brk(0xc3000000) = 0xbf000000
brk(0xc4000000) = 0xbf000000
brk(0xc5000000) = 0xbf000000
[...]
brk(0xf9000000) = 0xbf000000
brk(0xfa000000) = 0xbf000000
brk(0xfb000000) = 0xbf000000
brk(0xfc000000) = 0xbf000000
brk(0xfd000000) = 0xbf000000
brk(0xfe000000) = 0xbf000000
brk(0xff000000) = 0xbf000000
brk(0) = 0xbf000000
brk(0x1000000) = 0xbf000000
brk(0x2000000) = 0xbf000000
brk(0x3000000) = 0xbf000000
brk(0x4000000) = 0xbf000000
brk(0x5000000) = 0xbf000000
brk(0x6000000) = 0xbf000000
brk(0x7000000) = 0xbf000000
brk(0x8000000) = 0xbf000000
brk(0x9000000) = 0xbf000000
brk(0xa000000) = 0xa000000
[...]
Notice that from 0xc0000000 upwards it fails and returns the current break as expected. However, the following is the strace output of the same program in a 64-bit machine running in 32-bit compatibility mode:
[...]
brk(0xb9000000) = 0xffffffffb9000000
brk(0xba000000) = 0xffffffffba000000
brk(0xbb000000) = 0xffffffffbb000000
brk(0xbc000000) = 0xffffffffbc000000
brk(0xbd000000) = 0xffffffffbd000000
brk(0xbe000000) = 0xffffffffbe000000
brk(0xbf000000) = 0xffffffffbf000000
brk(0xc0000000) = 0xffffffffc0000000
brk(0xc1000000) = 0xffffffffc1000000
brk(0xc2000000) = 0xffffffffc2000000
brk(0xc3000000) = 0xffffffffc3000000
brk(0xc4000000) = 0xffffffffc4000000
brk(0xc5000000) = 0xffffffffc5000000
[...]
brk(0xf9000000) = 0xfffffffff9000000
brk(0xfa000000) = 0xfffffffffa000000
brk(0xfb000000) = 0xfffffffffb000000
brk(0xfc000000) = 0xfffffffffc000000
brk(0xfd000000) = 0xfffffffffd000000
brk(0xfe000000) = 0xfffffffffe000000
brk(0xff000000) = 0xffffffffff000000
brk(0) = 0xffffffffff000000
brk(0x1000000) = 0xffffffffff000000
brk(0x2000000) = 0xffffffffff000000
brk(0x3000000) = 0xffffffffff000000
brk(0x4000000) = 0xffffffffff000000
brk(0x5000000) = 0xffffffffff000000
brk(0x6000000) = 0xffffffffff000000
brk(0x7000000) = 0xffffffffff000000
brk(0x8000000) = 0xffffffffff000000
brk(0x9000000) = 0x9000000
brk(0xa000000) = 0xa000000
[...]
Notice that from 0xc0000000 upwards it continues expanding the program break above 0xc0000000 up to 0xff000000 where it also starts failing and returning the current break. Is this an expected behaviour when running an application in 64-bit machine in 32-bit compatibility mode? Is RLIMIT_AS and RLIMIT_DATA being honoured?
Thanks,
--
Ramon de C Valle / Red Hat Security Response Team
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: brk() system call, RLIMIT_AS, and RLIMIT_DATA in 32-bit compatibility mode
2011-12-21 15:58 ` brk() system call, RLIMIT_AS, and RLIMIT_DATA in 32-bit compatibility mode Ramon de C Valle
@ 2011-12-22 9:46 ` Ramon de C Valle
0 siblings, 0 replies; 2+ messages in thread
From: Ramon de C Valle @ 2011-12-22 9:46 UTC (permalink / raw)
To: linux-kernel
> Notice that from 0xc0000000 upwards it continues expanding the
> program break above 0xc0000000 up to 0xff000000 where it also starts
> failing and returning the current break. Is this an expected
> behaviour when running an application in 64-bit machine in 32-bit
> compatibility mode? Is RLIMIT_AS and RLIMIT_DATA being honoured?
I completely ignored the PER_LINUX32_3GB personality and thought this would be enforced by default in this case.
Thanks,
--
Ramon de C Valle / Red Hat Security Response Team
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2011-12-22 9:46 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <ae7baec9-7075-4eca-b4ec-969b072fcd9d@zmail15.collab.prod.int.phx2.redhat.com>
2011-12-21 15:58 ` brk() system call, RLIMIT_AS, and RLIMIT_DATA in 32-bit compatibility mode Ramon de C Valle
2011-12-22 9:46 ` Ramon de C Valle
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).