Re: "Cannot allocate memory" on ring creation (not RLIMIT_MEMLOCK)

[Dmitry Kadashev <dkadashev@gmail.com>]

On Tue, Dec 22, 2020 at 11:37 PM Pavel Begunkov <asml.silence@gmail.com> wrote:
>
> On 22/12/2020 11:04, Dmitry Kadashev wrote:
> > On Tue, Dec 22, 2020 at 11:11 AM Pavel Begunkov <asml.silence@gmail.com> wrote:
> [...]
> >>> What about smaller rings? Can you check io_uring of what SQ size it can allocate?
> >>> That can be a different program, e.g. modify a bit liburing/test/nop.
> > Unfortunately I've rebooted the box I've used for tests yesterday, so I can't
> > try this there. Also I was not able to come up with an isolated reproducer for
> > this yet.
> >
> > The good news is I've found a relatively easy way to provoke this on a test VM
> > using our software. Our app runs with "admin" user perms (plus some
> > capabilities), it bumps RLIMIT_MEMLOCK to infinity on start. I've also created
> > an user called 'ioutest' to run the check for ring sizes using a different user.
> >
> > I've modified the test program slightly, to show the number of rings
> > successfully
> > created on each iteration and the actual error message (to debug a problem I was
> > having with it, but I've kept this after that). Here is the output:
> >
> > # sudo -u admin bash -c 'ulimit -a' | grep locked
> > max locked memory       (kbytes, -l) 1024
> >
> > # sudo -u ioutest bash -c 'ulimit -a' | grep locked
> > max locked memory       (kbytes, -l) 1024
> >
> > # sudo -u admin ./iou-test1
> > Failed after 0 rings with 1024 size: Cannot allocate memory
> > Failed after 0 rings with 512 size: Cannot allocate memory
> > Failed after 0 rings with 256 size: Cannot allocate memory
> > Failed after 0 rings with 128 size: Cannot allocate memory
> > Failed after 0 rings with 64 size: Cannot allocate memory
> > Failed after 0 rings with 32 size: Cannot allocate memory
> > Failed after 0 rings with 16 size: Cannot allocate memory
> > Failed after 0 rings with 8 size: Cannot allocate memory
> > Failed after 0 rings with 4 size: Cannot allocate memory
> > Failed after 0 rings with 2 size: Cannot allocate memory
> > can't allocate 1
> >
> > # sudo -u ioutest ./iou-test1
> > max size 1024
>
> Then we screw that specific user. Interestingly, if it has CAP_IPC_LOCK
> capability we don't even account locked memory.

We do have some capabilities, but not CAP_IPC_LOCK. Ours are:

CAP_NET_ADMIN, CAP_NET_BIND_SERVICE, CAP_SYS_RESOURCE, CAP_KILL,
CAP_DAC_READ_SEARCH.

The latter was necessary for integration with some third-party thing that we do
not really use anymore, so we can try building without it, but it'd require some
time, mostly because I'm not sure how quickly I'd be able to provoke the issue.

> btw, do you use registered buffers?

No, we do not use neither registered buffers nor registered files (nor anything
else).

Also, I just tried the test program on a real box (this time one instance of our
program is still running - can repeat the check with it dead, but I expect the
results to be pretty much the same, at least after a few more restarts). This
box runs 5.9.5.

# sudo -u admin bash -c 'ulimit -l'
1024

# sudo -u admin ./iou-test1
Failed after 0 rings with 1024 size: Cannot allocate memory
Failed after 0 rings with 512 size: Cannot allocate memory
Failed after 0 rings with 256 size: Cannot allocate memory
Failed after 0 rings with 128 size: Cannot allocate memory
Failed after 0 rings with 64 size: Cannot allocate memory
Failed after 0 rings with 32 size: Cannot allocate memory
Failed after 0 rings with 16 size: Cannot allocate memory
Failed after 0 rings with 8 size: Cannot allocate memory
Failed after 0 rings with 4 size: Cannot allocate memory
Failed after 0 rings with 2 size: Cannot allocate memory
can't allocate 1

# sudo -u dmitry bash -c 'ulimit -l'
1024

# sudo -u dmitry ./iou-test1
max size 1024

-- 
Dmitry Kadashev