mlock() confusing 1 half of system RAM limit

mlock() confusing 1 half of system RAM limit

[Alex Richman]

Hi,

I'm seeing a 1/2 of system RAM limit when calling mlock().  mlock(2) says:
 > ENOMEM (Linux 2.4 and earlier) the calling process tried to lock more 
than half of RAM.
Which implies to me that the 1 half of system RAM limit should be only 
Linux 2.4 and earlier, but I'm running linux 4.4.127 and still seeing 
the limit.

This is on a system with 66007168 KB total memory, locking 33794621440 
bytes works, locking 33795653632 bytes does not work.

Anyone know if this is the man pages or the kernel or something unrelated?

More details here: 
https://stackoverflow.com/questions/50639709/mlock-of-large-shm-address-range-fails-with-enomem-even-with-lots-of-free-memo

Thanks,
- Alex.

-- 
Alex Richman
alex.r@gblabs.co.uk
Engineering
GB Labs
2 Orpheus House,
Calleva park,
Reading
RG7 8TA
Tel:+44 (0)118 455 5000
www.gblabs.com



The information contained in this message and any attachment may be proprietary, confidential and privileged.
If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering
this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you received this communication in error, please contact me immediately,
and delete the communication (including attachments, if applicable) from any computer or network system.

Re: mlock() confusing 1 half of system RAM limit

[Michal Hocko]

On Fri 01-06-18 10:53:57, Alex Richman wrote:
> Hi,
> 
> I'm seeing a 1/2 of system RAM limit when calling mlock().  mlock(2) says:
> > ENOMEM (Linux 2.4 and earlier) the calling process tried to lock more than
> half of RAM.
> Which implies to me that the 1 half of system RAM limit should be only Linux
> 2.4 and earlier, but I'm running linux 4.4.127 and still seeing the limit.
> 
> This is on a system with 66007168 KB total memory, locking 33794621440 bytes
> works, locking 33795653632 bytes does not work.

I assume you are doing mlock on MAP_SHARED resp. on mmap of shmem
file. You are likely to hit the shmem limit. This is half of the
available memory by default.
-- 
Michal Hocko
SUSE Labs

Re: mlock() confusing 1 half of system RAM limit

[Alex Richman]

I am using a shm MAP_SHARED, along these lines:
 > shm_fd = shm_open(handle, (O_RDWR | O_CREAT), (S_IRWXU | S_IRWXG | 
S_IRWXO));
 > ftruncate(shm_fd, channel->sled_size)
 > channel->sled = mmap(NULL, channel->sled_size, (PROT_READ | 
PROT_WRITE), (MAP_SHARED | MAP_NORESERVE), shm_fd, 0);
 > mlock(channel->sled, channel->sled_size) /* Fails with ENOMEM. */

But shmmax is unlimited on my box:
# sysctl -a | grep shm
kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096

Any ideas?

Many thanks,
- Alex.


On 01/06/18 12:44, Michal Hocko wrote:
> On Fri 01-06-18 10:53:57, Alex Richman wrote:
>> Hi,
>>
>> I'm seeing a 1/2 of system RAM limit when calling mlock().  mlock(2) says:
>>> ENOMEM (Linux 2.4 and earlier) the calling process tried to lock more than
>> half of RAM.
>> Which implies to me that the 1 half of system RAM limit should be only Linux
>> 2.4 and earlier, but I'm running linux 4.4.127 and still seeing the limit.
>>
>> This is on a system with 66007168 KB total memory, locking 33794621440 bytes
>> works, locking 33795653632 bytes does not work.
> I assume you are doing mlock on MAP_SHARED resp. on mmap of shmem
> file. You are likely to hit the shmem limit. This is half of the
> available memory by default.

-- 
Alex Richman
alex.r@gblabs.co.uk
Engineering
GB Labs
2 Orpheus House,
Calleva park,
Reading
RG7 8TA
Tel:+44 (0)118 455 5000
www.gblabs.com



The information contained in this message and any attachment may be proprietary, confidential and privileged.
If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering
this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you received this communication in error, please contact me immediately,
and delete the communication (including attachments, if applicable) from any computer or network system.

Re: mlock() confusing 1 half of system RAM limit

[Michal Hocko]

On Fri 01-06-18 13:26:59, Alex Richman wrote:
> I am using a shm MAP_SHARED, along these lines:
> > shm_fd = shm_open(handle, (O_RDWR | O_CREAT), (S_IRWXU | S_IRWXG |
> S_IRWXO));
> > ftruncate(shm_fd, channel->sled_size)
> > channel->sled = mmap(NULL, channel->sled_size, (PROT_READ | PROT_WRITE),
> > (MAP_SHARED | MAP_NORESERVE), shm_fd, 0);
> > mlock(channel->sled, channel->sled_size) /* Fails with ENOMEM. */
> 
> But shmmax is unlimited on my box:
> # sysctl -a | grep shm
> kernel.shm_next_id = -1
> kernel.shm_rmid_forced = 0
> kernel.shmall = 18446744073692774399
> kernel.shmmax = 18446744073692774399
> kernel.shmmni = 4096
> 
> Any ideas?

shm_open uses tmpfs/shmem under the cover and that has the internal
limit as explained above.
-- 
Michal Hocko
SUSE Labs

Re: mlock() confusing 1 half of system RAM limit

[Alex Richman]

Ah, that's it.  Increased the limit on the mount and it works fine now.

Thanks!
- Alex.

On 01/06/18 14:05, Michal Hocko wrote:
> On Fri 01-06-18 13:26:59, Alex Richman wrote:
>> I am using a shm MAP_SHARED, along these lines:
>>> shm_fd = shm_open(handle, (O_RDWR | O_CREAT), (S_IRWXU | S_IRWXG |
>> S_IRWXO));
>>> ftruncate(shm_fd, channel->sled_size)
>>> channel->sled = mmap(NULL, channel->sled_size, (PROT_READ | PROT_WRITE),
>>> (MAP_SHARED | MAP_NORESERVE), shm_fd, 0);
>>> mlock(channel->sled, channel->sled_size) /* Fails with ENOMEM. */
>> But shmmax is unlimited on my box:
>> # sysctl -a | grep shm
>> kernel.shm_next_id = -1
>> kernel.shm_rmid_forced = 0
>> kernel.shmall = 18446744073692774399
>> kernel.shmmax = 18446744073692774399
>> kernel.shmmni = 4096
>>
>> Any ideas?
> shm_open uses tmpfs/shmem under the cover and that has the internal
> limit as explained above.

-- 
Alex Richman
alex.r@gblabs.co.uk
Engineering
GB Labs
2 Orpheus House,
Calleva park,
Reading
RG7 8TA
Tel:+44 (0)118 455 5000
www.gblabs.com



The information contained in this message and any attachment may be proprietary, confidential and privileged.
If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering
this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you received this communication in error, please contact me immediately,
and delete the communication (including attachments, if applicable) from any computer or network system.

Re: mlock() confusing 1 half of system RAM limit

[Michal Hocko]

On Fri 01-06-18 14:24:23, Alex Richman wrote:
> Ah, that's it.  Increased the limit on the mount and it works fine now.

Well, it is hidden very well. It is not until the page fault time (or
get_user_pages aka __mm_populate) when the operation fails and remasks
ENOSPC -> VM_FAULT_SIGBUS -> EFAULT -> ENOMEM on the way.
-- 
Michal Hocko
SUSE Labs