* AVC denied for docker while trying to set labels for tmpfs mounts
@ 2021-07-21 18:46 Sujithra P
2021-07-21 21:01 ` Paul Moore
0 siblings, 1 reply; 5+ messages in thread
From: Sujithra P @ 2021-07-21 18:46 UTC (permalink / raw)
To: linux-security-module, selinux
Hi SELinux Experts,
The following issue is described in the below post as well.
https://github.com/containers/container-selinux/issues/141
Occasionally running into the following selinux denials for docker
type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
scontext=system_u:object_r:container_file_t:s0:c263,c914
tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
scontext=system_u:object_r:container_file_t:s0:c578,c672
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
level=error msg="Handler for POST
/v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
returned error: error setting label on mount source
'/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
failed to set file label on
/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
permission denied"
Docker is not able to set labels for these tmpfs mounts because they
end up having wrong labels when they are created (sometimes
"locale_t", sometimes "lib_t" which of course is not the
default/correct context for tmpfs fs).
Apparently semodule -R and deleting these tmps files or reboot of the
node fixes the problem.
Not sure what is causing the tmpfs mounts to get wrong labels in the
first place.
Everything seems to be fine to begin with, but as the system keeps
scheduling pods on the node, this behavior is observed sometimes (not
consistent always).
OS Details:
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
Docker Version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false
Kubernetes Version*
v1.20.8-gke.1500
Any help on how to debug this issue would be greatly appreciated.
Thanks
Sujithra.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AVC denied for docker while trying to set labels for tmpfs mounts
2021-07-21 18:46 AVC denied for docker while trying to set labels for tmpfs mounts Sujithra P
@ 2021-07-21 21:01 ` Paul Moore
2021-07-21 22:17 ` Sujithra P
0 siblings, 1 reply; 5+ messages in thread
From: Paul Moore @ 2021-07-21 21:01 UTC (permalink / raw)
To: Sujithra P; +Cc: linux-security-module, selinux
On Wed, Jul 21, 2021 at 2:46 PM Sujithra P <sujithrap@gmail.com> wrote:
>
> Hi SELinux Experts,
>
> The following issue is described in the below post as well.
> https://github.com/containers/container-selinux/issues/141
>
> Occasionally running into the following selinux denials for docker
>
> type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
> for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
> scontext=system_u:object_r:container_file_t:s0:c263,c914
> tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
>
> type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
> for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
> scontext=system_u:object_r:container_file_t:s0:c578,c672
> tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
>
>
> level=error msg="Handler for POST
> /v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
> returned error: error setting label on mount source
> '/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
> failed to set file label on
> /var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
> permission denied"
>
>
> Docker is not able to set labels for these tmpfs mounts because they
> end up having wrong labels when they are created (sometimes
> "locale_t", sometimes "lib_t" which of course is not the
> default/correct context for tmpfs fs).
> Apparently semodule -R and deleting these tmps files or reboot of the
> node fixes the problem.
> Not sure what is causing the tmpfs mounts to get wrong labels in the
> first place.
>
> Everything seems to be fine to begin with, but as the system keeps
> scheduling pods on the node, this behavior is observed sometimes (not
> consistent always).
>
>
> OS Details:
>
> NAME="CentOS Linux"
> VERSION="8 (Core)"
> ID="centos"
> ID_LIKE="rhel fedora"
> VERSION_ID="8"
> PLATFORM_ID="platform:el8"
> PRETTY_NAME="CentOS Linux 8 (Core)"
>
> Docker Version:
> Client: Docker Engine - Community
> Version: 19.03.13
> API version: 1.40
> Go version: go1.13.15
> Git commit: 4484c46d9d
> Built: Wed Sep 16 17:02:36 2020
> OS/Arch: linux/amd64
> Experimental: false
>
> Kubernetes Version*
> v1.20.8-gke.1500
>
>
> Any help on how to debug this issue would be greatly appreciated.
This sounds like it might be a problem with CentOS and/or your Docker
install, have you tried talking with the RH/CentOS folks about this
problem? We focus mostly on upstream issues here and it isn't clear
to me at this moment that this is an upstream issue.
--
paul moore
www.paul-moore.com
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AVC denied for docker while trying to set labels for tmpfs mounts
2021-07-21 21:01 ` Paul Moore
@ 2021-07-21 22:17 ` Sujithra P
2021-07-22 9:38 ` Daniel Walsh
0 siblings, 1 reply; 5+ messages in thread
From: Sujithra P @ 2021-07-21 22:17 UTC (permalink / raw)
To: Paul Moore, dwalsh; +Cc: linux-security-module, selinux
Thanks Paul!
Is there any specific centos/RH mailing list that I can ask? Not sure
whether it is a problem with kernel/docker/kubelet.
semodule -R seems to fix the problem, but not sure what is causing the
loaded policy to get corrupt.
Any insight on how to figure this out would be very much appreciated.
Thanks
Sujithra.
On Wed, Jul 21, 2021 at 2:01 PM Paul Moore <paul@paul-moore.com> wrote:
>
> On Wed, Jul 21, 2021 at 2:46 PM Sujithra P <sujithrap@gmail.com> wrote:
> >
> > Hi SELinux Experts,
> >
> > The following issue is described in the below post as well.
> > https://github.com/containers/container-selinux/issues/141
> >
> > Occasionally running into the following selinux denials for docker
> >
> > type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
> > for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
> > scontext=system_u:object_r:container_file_t:s0:c263,c914
> > tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
> >
> > type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
> > for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
> > scontext=system_u:object_r:container_file_t:s0:c578,c672
> > tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
> >
> >
> > level=error msg="Handler for POST
> > /v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
> > returned error: error setting label on mount source
> > '/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
> > failed to set file label on
> > /var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
> > permission denied"
> >
> >
> > Docker is not able to set labels for these tmpfs mounts because they
> > end up having wrong labels when they are created (sometimes
> > "locale_t", sometimes "lib_t" which of course is not the
> > default/correct context for tmpfs fs).
> > Apparently semodule -R and deleting these tmps files or reboot of the
> > node fixes the problem.
> > Not sure what is causing the tmpfs mounts to get wrong labels in the
> > first place.
> >
> > Everything seems to be fine to begin with, but as the system keeps
> > scheduling pods on the node, this behavior is observed sometimes (not
> > consistent always).
> >
> >
> > OS Details:
> >
> > NAME="CentOS Linux"
> > VERSION="8 (Core)"
> > ID="centos"
> > ID_LIKE="rhel fedora"
> > VERSION_ID="8"
> > PLATFORM_ID="platform:el8"
> > PRETTY_NAME="CentOS Linux 8 (Core)"
> >
> > Docker Version:
> > Client: Docker Engine - Community
> > Version: 19.03.13
> > API version: 1.40
> > Go version: go1.13.15
> > Git commit: 4484c46d9d
> > Built: Wed Sep 16 17:02:36 2020
> > OS/Arch: linux/amd64
> > Experimental: false
> >
> > Kubernetes Version*
> > v1.20.8-gke.1500
> >
> >
> > Any help on how to debug this issue would be greatly appreciated.
>
> This sounds like it might be a problem with CentOS and/or your Docker
> install, have you tried talking with the RH/CentOS folks about this
> problem? We focus mostly on upstream issues here and it isn't clear
> to me at this moment that this is an upstream issue.
>
> --
> paul moore
> www.paul-moore.com
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AVC denied for docker while trying to set labels for tmpfs mounts
2021-07-21 22:17 ` Sujithra P
@ 2021-07-22 9:38 ` Daniel Walsh
2021-07-23 6:06 ` Sujithra P
0 siblings, 1 reply; 5+ messages in thread
From: Daniel Walsh @ 2021-07-22 9:38 UTC (permalink / raw)
To: Sujithra P, Paul Moore; +Cc: linux-security-module, selinux
On 7/21/21 18:17, Sujithra P wrote:
> Thanks Paul!
>
> Is there any specific centos/RH mailing list that I can ask? Not sure
> whether it is a problem with kernel/docker/kubelet.
> semodule -R seems to fix the problem, but not sure what is causing the
> loaded policy to get corrupt.
> Any insight on how to figure this out would be very much appreciated.
>
> Thanks
> Sujithra.
I am guessing that one of the containers is loading policy. You should
be able to see something in the auditlog, about a policy load.
> On Wed, Jul 21, 2021 at 2:01 PM Paul Moore <paul@paul-moore.com> wrote:
>> On Wed, Jul 21, 2021 at 2:46 PM Sujithra P <sujithrap@gmail.com> wrote:
>>> Hi SELinux Experts,
>>>
>>> The following issue is described in the below post as well.
>>> https://github.com/containers/container-selinux/issues/141
>>>
>>> Occasionally running into the following selinux denials for docker
>>>
>>> type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
>>> for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
>>> scontext=system_u:object_r:container_file_t:s0:c263,c914
>>> tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
>>>
>>> type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
>>> for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
>>> scontext=system_u:object_r:container_file_t:s0:c578,c672
>>> tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
>>>
>>>
>>> level=error msg="Handler for POST
>>> /v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
>>> returned error: error setting label on mount source
>>> '/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
>>> failed to set file label on
>>> /var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
>>> permission denied"
>>>
>>>
>>> Docker is not able to set labels for these tmpfs mounts because they
>>> end up having wrong labels when they are created (sometimes
>>> "locale_t", sometimes "lib_t" which of course is not the
>>> default/correct context for tmpfs fs).
>>> Apparently semodule -R and deleting these tmps files or reboot of the
>>> node fixes the problem.
>>> Not sure what is causing the tmpfs mounts to get wrong labels in the
>>> first place.
>>>
>>> Everything seems to be fine to begin with, but as the system keeps
>>> scheduling pods on the node, this behavior is observed sometimes (not
>>> consistent always).
>>>
>>>
>>> OS Details:
>>>
>>> NAME="CentOS Linux"
>>> VERSION="8 (Core)"
>>> ID="centos"
>>> ID_LIKE="rhel fedora"
>>> VERSION_ID="8"
>>> PLATFORM_ID="platform:el8"
>>> PRETTY_NAME="CentOS Linux 8 (Core)"
>>>
>>> Docker Version:
>>> Client: Docker Engine - Community
>>> Version: 19.03.13
>>> API version: 1.40
>>> Go version: go1.13.15
>>> Git commit: 4484c46d9d
>>> Built: Wed Sep 16 17:02:36 2020
>>> OS/Arch: linux/amd64
>>> Experimental: false
>>>
>>> Kubernetes Version*
>>> v1.20.8-gke.1500
>>>
>>>
>>> Any help on how to debug this issue would be greatly appreciated.
>> This sounds like it might be a problem with CentOS and/or your Docker
>> install, have you tried talking with the RH/CentOS folks about this
>> problem? We focus mostly on upstream issues here and it isn't clear
>> to me at this moment that this is an upstream issue.
>>
>> --
>> paul moore
>> www.paul-moore.com
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: AVC denied for docker while trying to set labels for tmpfs mounts
2021-07-22 9:38 ` Daniel Walsh
@ 2021-07-23 6:06 ` Sujithra P
0 siblings, 0 replies; 5+ messages in thread
From: Sujithra P @ 2021-07-23 6:06 UTC (permalink / raw)
To: dwalsh; +Cc: Paul Moore, linux-security-module, selinux
Thanks Daniel.
I'm not seeing anything suspicious in the audit logs, the. following
are the MAC_POLICY_LOAD events
type=SYSCALL msg=audit(1622991228.547:183): arch=c000003e syscall=1
success=yes exit=8577048 a0=4 a1=7fd682c61000 a2=82e018 a3=0 items=0
ppid=62178 pid=62186 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="load_policy"
exe="/usr/sbin/load_policy"
subj=unconfined_u:unconfined_r:load_policy_t:s0-s0:c0.c1023 key=(null)
type=MAC_POLICY_LOAD msg=audit(1627002776.825:6075): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
type=MAC_POLICY_LOAD msg=audit(1627008064.852:7615): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
type=MAC_POLICY_LOAD msg=audit(1627008273.029:7617): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
type=MAC_POLICY_LOAD msg=audit(1627009159.383:8392): auid=4294967295
ses=4294967295 lsm=selinux res=1
----
On Thu, Jul 22, 2021 at 2:38 AM Daniel Walsh <dwalsh@redhat.com> wrote:
>
> On 7/21/21 18:17, Sujithra P wrote:
> > Thanks Paul!
> >
> > Is there any specific centos/RH mailing list that I can ask? Not sure
> > whether it is a problem with kernel/docker/kubelet.
> > semodule -R seems to fix the problem, but not sure what is causing the
> > loaded policy to get corrupt.
> > Any insight on how to figure this out would be very much appreciated.
> >
> > Thanks
> > Sujithra.
> I am guessing that one of the containers is loading policy. You should
> be able to see something in the auditlog, about a policy load.
> > On Wed, Jul 21, 2021 at 2:01 PM Paul Moore <paul@paul-moore.com> wrote:
> >> On Wed, Jul 21, 2021 at 2:46 PM Sujithra P <sujithrap@gmail.com> wrote:
> >>> Hi SELinux Experts,
> >>>
> >>> The following issue is described in the below post as well.
> >>> https://github.com/containers/container-selinux/issues/141
> >>>
> >>> Occasionally running into the following selinux denials for docker
> >>>
> >>> type=AVC msg=audit(1626732057.636:4583): avc: denied { associate }
> >>> for pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
> >>> scontext=system_u:object_r:container_file_t:s0:c263,c914
> >>> tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
> >>>
> >>> type=AVC msg=audit(1626812823.170:9434): avc: denied { associate }
> >>> for pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
> >>> scontext=system_u:object_r:container_file_t:s0:c578,c672
> >>> tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
> >>>
> >>>
> >>> level=error msg="Handler for POST
> >>> /v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
> >>> returned error: error setting label on mount source
> >>> '/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
> >>> failed to set file label on
> >>> /var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
> >>> permission denied"
> >>>
> >>>
> >>> Docker is not able to set labels for these tmpfs mounts because they
> >>> end up having wrong labels when they are created (sometimes
> >>> "locale_t", sometimes "lib_t" which of course is not the
> >>> default/correct context for tmpfs fs).
> >>> Apparently semodule -R and deleting these tmps files or reboot of the
> >>> node fixes the problem.
> >>> Not sure what is causing the tmpfs mounts to get wrong labels in the
> >>> first place.
> >>>
> >>> Everything seems to be fine to begin with, but as the system keeps
> >>> scheduling pods on the node, this behavior is observed sometimes (not
> >>> consistent always).
> >>>
> >>>
> >>> OS Details:
> >>>
> >>> NAME="CentOS Linux"
> >>> VERSION="8 (Core)"
> >>> ID="centos"
> >>> ID_LIKE="rhel fedora"
> >>> VERSION_ID="8"
> >>> PLATFORM_ID="platform:el8"
> >>> PRETTY_NAME="CentOS Linux 8 (Core)"
> >>>
> >>> Docker Version:
> >>> Client: Docker Engine - Community
> >>> Version: 19.03.13
> >>> API version: 1.40
> >>> Go version: go1.13.15
> >>> Git commit: 4484c46d9d
> >>> Built: Wed Sep 16 17:02:36 2020
> >>> OS/Arch: linux/amd64
> >>> Experimental: false
> >>>
> >>> Kubernetes Version*
> >>> v1.20.8-gke.1500
> >>>
> >>>
> >>> Any help on how to debug this issue would be greatly appreciated.
> >> This sounds like it might be a problem with CentOS and/or your Docker
> >> install, have you tried talking with the RH/CentOS folks about this
> >> problem? We focus mostly on upstream issues here and it isn't clear
> >> to me at this moment that this is an upstream issue.
> >>
> >> --
> >> paul moore
> >> www.paul-moore.com
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-07-23 6:06 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-21 18:46 AVC denied for docker while trying to set labels for tmpfs mounts Sujithra P
2021-07-21 21:01 ` Paul Moore
2021-07-21 22:17 ` Sujithra P
2021-07-22 9:38 ` Daniel Walsh
2021-07-23 6:06 ` Sujithra P
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.