Re: "Cannot allocate memory" on ring creation (not RLIMIT_MEMLOCK)

[Dmitry Kadashev <dkadashev@gmail.com>]

On Thu, Dec 17, 2020 at 5:38 PM Josef <josef.grieb@gmail.com> wrote:
>
> > > That is curious. This ticket mentions Shmem though, and in our case it does
>  > not look suspicious at all. E.g. on a box that has the problem at the moment:
>  > Shmem:  41856 kB. The box has 256GB of RAM.
>  >
>  > But I'd (given my lack of knowledge) expect the issues to be related anyway.
>
> what about mapped? mapped is pretty high 1GB on my machine, I'm still
> reproduce that in C...however the user process is killed but not the
> io_wq_worker kernel processes, that's also the reason why the server
> socket still listening(even if the user process is killed), the bug
> only occurs(in netty) with a high number of operations and using
> eventfd_write to unblock io_uring_enter(IORING_ENTER_GETEVENTS)
>
> (tested on kernel 5.9 and 5.10)

Stats from another box with this problem (still 256G of RAM):

Mlocked:           17096 kB
Mapped:           171480 kB
Shmem:             41880 kB

Does not look suspicious at a glance. Number of io_wq* processes is 23-31.

Uptime is 27 days, 24 rings per process, process was restarted 4 times, 3 out of
these four the old instance was killed with SIGKILL. On the last process start
18 rings failed to initialize, but after that 6 more were initialized
successfully. It was before the old instance was killed. Maybe it's related to
the load and number of io-wq processes, e.g. some of them exited and a few more
rings were initialized successfully.

-- 
Dmitry Kadashev