VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[Ben Greear]

Hello,

We see a problem on a particular system when trying to run 'ip vrf exec _vrf1 ping 1.1.1.1'.
This system reproduces the problem all the time, but other systems with exact same (as far as
we can tell) software may fail occasionally, but then it will work again.

Here is an strace output.  I changed to the "/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1"
directory as root user, and could view the files in that directory, so I'm not sure why the strace shows error 5.

Any idea what could be the problem and/or how to fix it or debug further?


This command was run as root user.

....

openat(AT_FDCWD, "/proc/15650/cgroup", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(5, "10:pids:/user.slice/user-1000.sl"..., 1024) = 274
close(5)                               = 0
mkdir("/sys", 0755)                     = -1 EEXIST (File exists)
mkdir("/sys", 0755)                     = -1 EEXIST (File exists)
mkdir("/sys/fs", 0755)                  = -1 EEXIST (File exists)
mkdir("/sys/fs", 0755)                  = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup", 0755)           = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup", 0755)           = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified", 0755)   = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified", 0755)   = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1", 0755) = -1 EEXIST (File exists)
openat(AT_FDCWD, "/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1", O_RDONLY|O_DIRECTORY) = 5
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SOCK, insn_cnt=6, insns=0x7ffc8e5d1e00, license="GPL", log_level=1, log_size=262144, log_buf="", 
kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, 
func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 112) = -1 EPERM (Operation not permitted)
write(2, "Failed to load BPF prog: 'Operat"..., 51Failed to load BPF prog: 'Operation not permitted'


Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[David Ahern]

On 11/22/19 5:03 PM, Ben Greear wrote:
> Hello,
> 
> We see a problem on a particular system when trying to run 'ip vrf exec
> _vrf1 ping 1.1.1.1'.
> This system reproduces the problem all the time, but other systems with
> exact same (as far as
> we can tell) software may fail occasionally, but then it will work again.
> 
> Here is an strace output.  I changed to the
> "/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1"
> 
> directory as root user, and could view the files in that directory, so
> I'm not sure why the strace shows error 5.
> 
> Any idea what could be the problem and/or how to fix it or debug further?
> 
> 
> This command was run as root user.

check 'ulimit -l'. BPF is used to set the VRF and it requires locked memory.

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[Ben Greear]

On 11/22/19 4:06 PM, David Ahern wrote:
> On 11/22/19 5:03 PM, Ben Greear wrote:
>> Hello,
>>
>> We see a problem on a particular system when trying to run 'ip vrf exec
>> _vrf1 ping 1.1.1.1'.
>> This system reproduces the problem all the time, but other systems with
>> exact same (as far as
>> we can tell) software may fail occasionally, but then it will work again.
>>
>> Here is an strace output.  I changed to the
>> "/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1"
>>
>> directory as root user, and could view the files in that directory, so
>> I'm not sure why the strace shows error 5.
>>
>> Any idea what could be the problem and/or how to fix it or debug further?
>>
>>
>> This command was run as root user.
> 
> check 'ulimit -l'. BPF is used to set the VRF and it requires locked memory.

It is set to '64'.  What is a good value to use?

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[David Ahern]

On 11/22/19 5:14 PM, Ben Greear wrote:
> On 11/22/19 4:06 PM, David Ahern wrote:
>> On 11/22/19 5:03 PM, Ben Greear wrote:
>>> Hello,
>>>
>>> We see a problem on a particular system when trying to run 'ip vrf exec
>>> _vrf1 ping 1.1.1.1'.
>>> This system reproduces the problem all the time, but other systems with
>>> exact same (as far as
>>> we can tell) software may fail occasionally, but then it will work
>>> again.
>>>
>>> Here is an strace output.  I changed to the
>>> "/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1"
>>>
>>>
>>> directory as root user, and could view the files in that directory, so
>>> I'm not sure why the strace shows error 5.
>>>
>>> Any idea what could be the problem and/or how to fix it or debug
>>> further?
>>>
>>>
>>> This command was run as root user.
>>
>> check 'ulimit -l'. BPF is used to set the VRF and it requires locked
>> memory.
> 
> It is set to '64'.  What is a good value to use?
> 

This is a pain point in using BPF for this. It's really use case
dependent. 128kB, 256kB.

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[Ben Greear]

On 11/22/19 4:17 PM, David Ahern wrote:
> On 11/22/19 5:14 PM, Ben Greear wrote:
>> On 11/22/19 4:06 PM, David Ahern wrote:
>>> On 11/22/19 5:03 PM, Ben Greear wrote:
>>>> Hello,
>>>>
>>>> We see a problem on a particular system when trying to run 'ip vrf exec
>>>> _vrf1 ping 1.1.1.1'.
>>>> This system reproduces the problem all the time, but other systems with
>>>> exact same (as far as
>>>> we can tell) software may fail occasionally, but then it will work
>>>> again.
>>>>
>>>> Here is an strace output.  I changed to the
>>>> "/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-2.scope/vrf/_vrf1"
>>>>
>>>>
>>>> directory as root user, and could view the files in that directory, so
>>>> I'm not sure why the strace shows error 5.
>>>>
>>>> Any idea what could be the problem and/or how to fix it or debug
>>>> further?
>>>>
>>>>
>>>> This command was run as root user.
>>>
>>> check 'ulimit -l'. BPF is used to set the VRF and it requires locked
>>> memory.
>>
>> It is set to '64'.  What is a good value to use?
>>
> 
> This is a pain point in using BPF for this. It's really use case
> dependent. 128kB, 256kB.
> 

Setting:  ulimit -l 1024

'fixed' the problem.

I'd rather waste a bit of memory and not have any of my users hit such an esoteric
bug, so I'll set it to at least 1024 going forward.

Would large numbers of vrf and/or network devices mean you need more locked memory?

And surely 'ip' could output a better error than just 'permission denied' for
this error case?  Or even something that would show up in dmesg to give a clue?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[David Ahern]

On 11/22/19 5:23 PM, Ben Greear wrote:
>>
> 
> Setting:  ulimit -l 1024
> 
> 'fixed' the problem.
> 
> I'd rather waste a bit of memory and not have any of my users hit such
> an esoteric
> bug, so I'll set it to at least 1024 going forward.

agreed.

> 
> Would large numbers of vrf and/or network devices mean you need more
> locked memory?

I have seen this problem way too much, but not taken the time to track
down all of the locked memory use. A rough estimate is that each 'ip vrf
exec' uses 1 page (4kB) of locked memory until the command exits. If you
use that as a rule you would be on the high end. Commands in the same
cgroup hierarchy should all be using the same program.

> 
> And surely 'ip' could output a better error than just 'permission
> denied' for
> this error case?  Or even something that would show up in dmesg to give
> a clue?

That error comes from the bpf syscall:

bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SOCK, insn_cnt=6,
insns=0x7ffc8e5d1e00, license="GPL", log_level=1, log_size=262144,
log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
prog_name="", prog_ifindex=0,
expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 112) = -1 EPERM
(Operation not permitted)

Yes it is odd and unhelpful for a memory limit to cause the failure and
then return EPERM.

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[Ben Greear]

On 11/23/19 10:10 AM, David Ahern wrote:
> On 11/22/19 5:23 PM, Ben Greear wrote:
>>>
>>
>> Setting:  ulimit -l 1024
>>
>> 'fixed' the problem.
>>
>> I'd rather waste a bit of memory and not have any of my users hit such
>> an esoteric
>> bug, so I'll set it to at least 1024 going forward.
> 
> agreed.
> 
>>
>> Would large numbers of vrf and/or network devices mean you need more
>> locked memory?
> 
> I have seen this problem way too much, but not taken the time to track
> down all of the locked memory use. A rough estimate is that each 'ip vrf
> exec' uses 1 page (4kB) of locked memory until the command exits. If you
> use that as a rule you would be on the high end. Commands in the same
> cgroup hierarchy should all be using the same program.

I am not sure if it is a coincidence or not, but we saw the problem when
connected with rdesktop more often than we connected with VNC.

>> And surely 'ip' could output a better error than just 'permission
>> denied' for
>> this error case?  Or even something that would show up in dmesg to give
>> a clue?
> 
> That error comes from the bpf syscall:
> 
> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SOCK, insn_cnt=6,
> insns=0x7ffc8e5d1e00, license="GPL", log_level=1, log_size=262144,
> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
> prog_name="", prog_ifindex=0,
> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 112) = -1 EPERM
> (Operation not permitted)

So, we can change iproute/lib/bpf.c to print a suggestion to increase locked memory
if this returns EPERM?

Thanks,
Ben

> 
> Yes it is odd and unhelpful for a memory limit to cause the failure and
> then return EPERM.
> 


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[David Ahern]

On 11/25/19 10:35 AM, Ben Greear wrote:
>>> And surely 'ip' could output a better error than just 'permission
>>> denied' for
>>> this error case?  Or even something that would show up in dmesg to give
>>> a clue?
>>
>> That error comes from the bpf syscall:
>>
>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SOCK, insn_cnt=6,
>> insns=0x7ffc8e5d1e00, license="GPL", log_level=1, log_size=262144,
>> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>> prog_name="", prog_ifindex=0,
>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 112) = -1 EPERM
>> (Operation not permitted)
> 
> So, we can change iproute/lib/bpf.c to print a suggestion to increase
> locked memory
> if this returns EPERM?
> 

looks like SYS_ADMIN and locked memory are the -EPERM failures.

I do not see any API that returns user->locked_vm, only per-task
locked_vm. Knowing that number would help a lot in understanding proper
system settings.

Running 'perf record' while trying to do 'ip vrf exec' is an easy way to
hit the locked memory exceeded error. We could add a hint to iproute2.
Something like:

diff --git a/ip/ipvrf.c b/ip/ipvrf.c
index b9a43675cbd6..15637924f31a 100644
--- a/ip/ipvrf.c
+++ b/ip/ipvrf.c
@@ -281,9 +281,16 @@ static int vrf_configure_cgroup(const char *path,
int ifindex)
                fprintf(stderr, "Failed to load BPF prog: '%s'\n",
                        strerror(errno));

-               if (errno != EPERM) {
+               if (errno == EPERM) {
+                       if (geteuid() != 0)
+                               fprintf(stderr,
+                                       "Hint: Must run as root to set
VRF.\n");
+                       else
+                               fprintf(stderr,
+                                       "Hint: Most likely locked memory
threshold exceeded. Increase 'ulimit -l'\n");
+               } else {
                        fprintf(stderr,
-                               "Kernel compiled with CGROUP_BPF
enabled?\n");
+                               "Hint: Kernel compiled with CGROUP_BPF
enabled?\n");
                }
                goto out;
        }

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[Toke Høiland-Jørgensen]

David Ahern <dsahern@gmail.com> writes:

> On 11/25/19 10:35 AM, Ben Greear wrote:
>>>> And surely 'ip' could output a better error than just 'permission
>>>> denied' for
>>>> this error case?  Or even something that would show up in dmesg to give
>>>> a clue?
>>>
>>> That error comes from the bpf syscall:
>>>
>>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SOCK, insn_cnt=6,
>>> insns=0x7ffc8e5d1e00, license="GPL", log_level=1, log_size=262144,
>>> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>>> prog_name="", prog_ifindex=0,
>>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 112) = -1 EPERM
>>> (Operation not permitted)
>> 
>> So, we can change iproute/lib/bpf.c to print a suggestion to increase
>> locked memory
>> if this returns EPERM?
>> 
>
> looks like SYS_ADMIN and locked memory are the -EPERM failures.
>
> I do not see any API that returns user->locked_vm, only per-task
> locked_vm. Knowing that number would help a lot in understanding proper
> system settings.

Yes! Having a way to see the current amount of locked memory is sorely
needed. Absent this, the application basically has to do one of:

- Throw up its hands and tell the user to increase ulimit (not terribly
  user-friendly)
- Just set the limit to infinity (this is what iproute2 does; works, but
  defeats the whole purpose of having a limit in the first place)
- Keep retrying while gradually increasing the limit (inefficient, and
  annoying to have to implement everywhere)

-Toke

Re: VRF and/or cgroups problem on Fedora-30, 5.2.21+ kernel

[Ben Greear]

On 11/25/2019 12:53 PM, David Ahern wrote:
> On 11/25/19 10:35 AM, Ben Greear wrote:
>>>> And surely 'ip' could output a better error than just 'permission
>>>> denied' for
>>>> this error case?  Or even something that would show up in dmesg to give
>>>> a clue?
>>>
>>> That error comes from the bpf syscall:
>>>
>>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SOCK, insn_cnt=6,
>>> insns=0x7ffc8e5d1e00, license="GPL", log_level=1, log_size=262144,
>>> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>>> prog_name="", prog_ifindex=0,
>>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 112) = -1 EPERM
>>> (Operation not permitted)
>>
>> So, we can change iproute/lib/bpf.c to print a suggestion to increase
>> locked memory
>> if this returns EPERM?
>>
>
> looks like SYS_ADMIN and locked memory are the -EPERM failures.
>
> I do not see any API that returns user->locked_vm, only per-task
> locked_vm. Knowing that number would help a lot in understanding proper
> system settings.
>
> Running 'perf record' while trying to do 'ip vrf exec' is an easy way to
> hit the locked memory exceeded error. We could add a hint to iproute2.
> Something like:
>
> diff --git a/ip/ipvrf.c b/ip/ipvrf.c
> index b9a43675cbd6..15637924f31a 100644
> --- a/ip/ipvrf.c
> +++ b/ip/ipvrf.c
> @@ -281,9 +281,16 @@ static int vrf_configure_cgroup(const char *path,
> int ifindex)
>                  fprintf(stderr, "Failed to load BPF prog: '%s'\n",
>                          strerror(errno));
>
> -               if (errno != EPERM) {
> +               if (errno == EPERM) {
> +                       if (geteuid() != 0)
> +                               fprintf(stderr,
> +                                       "Hint: Must run as root to set
> VRF.\n");
> +                       else
> +                               fprintf(stderr,
> +                                       "Hint: Most likely locked memory
> threshold exceeded. Increase 'ulimit -l'\n");

I think I would suggest 'ulimit -l 1024'.  Advanced users can try different
values if they care or have super tight memory constraints, but most users could
probably get benefit from the suggestion.

Thanks,
Ben

> +               } else {
>                          fprintf(stderr,
> -                               "Kernel compiled with CGROUP_BPF
> enabled?\n");
> +                               "Hint: Kernel compiled with CGROUP_BPF
> enabled?\n");
>                  }
>                  goto out;
>          }
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com