pthread_setschedparam returns 1 (Operation Not Permitted) after including docker in yocto build

pthread_setschedparam returns 1 (Operation Not Permitted) after including docker in yocto build

[Hans-Jacob Enemark]

Hi
In a yocto build for a multicore system, pulling in docker through the
meta-virtualization layer from open-embedded, a multi-threaded app
suddenly starts failing.
For some reason it is no longer possible to set the priority and the
scheduling policy of a running pthread inside the thread function
itself - even with the app running as root..
The kernel configuration changed with the addition of the
meta-virtualization layer.
I hope to get docker running on the build to be able to build on that.
So I hope you can help as this is beyond me
Best regards
Hans-Jacob S. Enemark

Appendices:
1) The minimal code that fails on the yocto build with docker but runs
fine on the old build without the docker modifications.
2) Kernel config diffs. Left has docker included. Right is the one
without. Several cgroup configs has been enabled and the reason why I
post to this mailing list.
3) A small snippet of the strace output that shows the issue I am facing.


1) ---------------------------------------------------------
#include <stdio.h>
#include <pthread.h>
#include <string.h>
#include <sys/resource.h>

#define POLICY SCHED_FIFO
static void *thread_routine(void *arg)
{
  int ret;
  struct sched_param param;
  param.sched_priority = sched_get_priority_max(POLICY) - 2;

  ret = pthread_setschedparam(pthread_self(), POLICY, &param);

  printf("%s: pthread_setschedparam result: [%d] %s\n", __func__, ret,
strerror(ret));

  return NULL;
}

int main(int argc, char* argv[])
{
  pthread_t threadId;

  printf("Starting main\n");
  if(pthread_create(&threadId, NULL, thread_routine, NULL)){
    printf("%s: Thread creation failed\n", __func__);
  }
  pthread_join(threadId, NULL);
  printf("Done with main\n");
  return 0;
}

2) ---------------------------------------------------------
140,143c140
< CONFIG_PAGE_COUNTER=y
< CONFIG_MEMCG=y
< CONFIG_MEMCG_SWAP=y
< CONFIG_MEMCG_SWAP_ENABLED=y
---
> # CONFIG_MEMCG is not set
149c146
< CONFIG_CGROUP_PIDS=y
---
> # CONFIG_CGROUP_PIDS is not set
153c150
< CONFIG_CGROUP_DEVICE=y
---
> # CONFIG_CGROUP_DEVICE is not set
155c152
< CONFIG_CGROUP_PERF=y
---
> # CONFIG_CGROUP_PERF is not set
157c154
< CONFIG_SOCK_CGROUP_DATA=y
---
> # CONFIG_SOCK_CGROUP_DATA is not set
163c160
< CONFIG_USER_NS=y
---
> # CONFIG_USER_NS is not set
227d223
< # CONFIG_SLUB_MEMCG_SYSFS_ON is not set
791c787
< CONFIG_BRIDGE_NETFILTER=y
---
> CONFIG_BRIDGE_NETFILTER=m
877c873
< CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
---
> # CONFIG_NETFILTER_XT_MATCH_ADDRTYPE is not set
898d893
< CONFIG_NETFILTER_XT_MATCH_IPVS=y
923,962c918
< CONFIG_IP_VS=y
< # CONFIG_IP_VS_IPV6 is not set
< # CONFIG_IP_VS_DEBUG is not set
< CONFIG_IP_VS_TAB_BITS=12
<
< #
< # IPVS transport protocol load balancing support
< #
< # CONFIG_IP_VS_PROTO_TCP is not set
< # CONFIG_IP_VS_PROTO_UDP is not set
< # CONFIG_IP_VS_PROTO_AH_ESP is not set
< # CONFIG_IP_VS_PROTO_ESP is not set
< # CONFIG_IP_VS_PROTO_AH is not set
< # CONFIG_IP_VS_PROTO_SCTP is not set
<
< #
< # IPVS scheduler
< #
< # CONFIG_IP_VS_RR is not set
< # CONFIG_IP_VS_WRR is not set
< # CONFIG_IP_VS_LC is not set
< # CONFIG_IP_VS_WLC is not set
< # CONFIG_IP_VS_FO is not set
< # CONFIG_IP_VS_OVF is not set
< # CONFIG_IP_VS_LBLC is not set
< # CONFIG_IP_VS_LBLCR is not set
< # CONFIG_IP_VS_DH is not set
< # CONFIG_IP_VS_SH is not set
< # CONFIG_IP_VS_SED is not set
< # CONFIG_IP_VS_NQ is not set
<
< #
< # IPVS SH scheduler
< #
< CONFIG_IP_VS_SH_TAB_BITS=8
<
< #
< # IPVS application helper
< #
< # CONFIG_IP_VS_NFCT is not set
---
> # CONFIG_IP_VS is not set
1115c1071
< CONFIG_NET_CLS_CGROUP=y
---
> # CONFIG_NET_CLS_CGROUP is not set
1164,1165c1120,1121
< CONFIG_CGROUP_NET_PRIO=y
< CONFIG_CGROUP_NET_CLASSID=y
---
> # CONFIG_CGROUP_NET_PRIO is not set
> # CONFIG_CGROUP_NET_CLASSID is not set
1632c1588
< CONFIG_VETH=y
---
> # CONFIG_VETH is not set
4347,4352c4303
< CONFIG_BTRFS_FS=y
< CONFIG_BTRFS_FS_POSIX_ACL=y
< # CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
< # CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
< # CONFIG_BTRFS_DEBUG is not set
< # CONFIG_BTRFS_ASSERT is not set
---
> # CONFIG_BTRFS_FS is not set
4376,4378c4327
< CONFIG_OVERLAY_FS=y
< # CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
< # CONFIG_OVERLAY_FS_INDEX is not set
---
> # CONFIG_OVERLAY_FS is not set
4776d4724
< CONFIG_XOR_BLOCKS=y

4944d4891
< CONFIG_RAID6_PQ=y
4968d4914
< CONFIG_XXHASH=y
4977,4978d4922
< CONFIG_ZSTD_COMPRESS=y
< CONFIG_ZSTD_DECOMPRESS=y

3) ---------------------------------------------------------
write(1, "Starting main\n", 14)         = 14
mmap2(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK,
-1, 0) = 0xb65cb000
mprotect(0xb65cc000, 8388608, PROT_READ|PROT_WRITE) = 0
clone(child_stack=0xb6dcaf98,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0xb6dcb4c8, tls=0xb6dcb920, child_tidptr=0xb6dcb4c8) =
2779
futex(0xb6dcb4c8, FUTEX_WAIT, 2779, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
write(1, "Done with main\n", 15)        = 15

Re: pthread_setschedparam returns 1 (Operation Not Permitted) after including docker in yocto build

[Michal Koutný]

Hello.

On Sat, Sep 25, 2021 at 10:45:43PM +0200, Hans-Jacob Enemark <hjenemark-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> For some reason it is no longer possible to set the priority and the
> scheduling policy of a running pthread inside the thread function
> itself - even with the app running as root..
> #define POLICY SCHED_FIFO

You use an RT scheduling policy.

It's not in your diff dump but my bet is on CONFIG_RT_GROUP_SCHED.
The non-root RT groups have zero RT runtime allocation by default,
therefore sched_setpriority would fail in such a cgroup.

(The changes in your setup might have modified whether any CPU cgroups
are created or only root cpu cgroup is present.)

HTH,
Michal