All of lore.kernel.org
 help / color / mirror / Atom feed
* Performance issues - huge amount of AVC misses
@ 2015-12-08 10:25 Michal Marciniszyn
  2015-12-08 10:44 ` Dominick Grift
                   ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Michal Marciniszyn @ 2015-12-08 10:25 UTC (permalink / raw)
  To: selinux

[-- Attachment #1: Type: text/plain, Size: 3750 bytes --]

Hello,

we are heavy SELinux shop and we recently run into AVC related performance
issue. I was trying to find an answer on freenode IRC chat but I was sent
here by multiple guys. We're running on Scientific Linux 6.6 (upgrade to
6.7 ongoing) and we see this on some of our nodes:

# cat /selinux/avc/cache_stats
lookups hits misses allocations reclaims frees
3976846641 3626568307 350278334 350303465 344833264 346344169
3474274460 3092218096 382056364 382081270 381170512 382671551
2037181411 1655679702 381501709 381527148 380680320 382162477
1943162363 1651603455 291558908 291584892 288099840 289631602
829213467 406079951 423133516 423158604 422311024 423847681
1963015875 1555848944 407166931 407192104 406718592 408227742
3490131033 3117047653 373083380 373108386 372270880 373862706
940880689 549698684 391182005 391207388 390339328 391888374
4098417807 3712068859 386348948 386373592 385604096 387172806
3931378773 3549502965 381875808 381901074 381059904 382628308

Also we see

# cat /selinux/avc/hash_stats
entries: 499
buckets used: 257/512
longest chain: 6

Some times under load we see SELinux consuming about 30% of CPU time. There
is about 16% of cache misses on these nodes (and sometimes it goes as high
as 30%). The lates article about the issue is from RHEL 5 times -
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
. We do not feel this to be too relevant in this case.

Are there any recommendations on cache sizing for SELinux? We can resize
cache to 1024 or 2048 entries, but would this help to resolve the issue?

I'm attaching seinfo from node with our policy and then for comparison from
node without any policy.

With policy:
# seinfo

Statistics for policy file: /etc/selinux/targeted/policy/policy.24
Policy Version & Type: v.24 (binary, mls)

   Classes:            81    Permissions:       238
   Sensitivities:       1    Categories:       1024
   Types:            4273    Attributes:        295
   Users:               9    Roles:              12
   Booleans:          234    Cond. Expr.:       274
   Allow:          352554    Neverallow:          0
   Auditallow:        140    Dontaudit:      321786
   Type_trans:      42813    Type_change:        38
   Type_member:        48    Role allow:         19
   Role_trans:        409    Range_trans:      6421
   Constraints:        90    Validatetrans:       0
   Initial SIDs:       27    Fs_use:             23
   Genfscon:           84    Portcon:           505
   Netifcon:            0    Nodecon:             0
   Permissives:        91    Polcap:              2



Without policy:

seinfo

Statistics for policy file: /etc/selinux/targeted/policy/policy.24
Policy Version & Type: v.24 (binary, mls)

   Classes:            81    Permissions:       238
   Sensitivities:       1    Categories:       1024
   Types:            3926    Attributes:        295
   Users:               9    Roles:              12
   Booleans:          234    Cond. Expr.:       274
   Allow:          320969    Neverallow:          0
   Auditallow:        140    Dontaudit:      273256
   Type_trans:      41915    Type_change:        38
   Type_member:        48    Role allow:         19
   Role_trans:        386    Range_trans:      6069
   Constraints:        90    Validatetrans:       0
   Initial SIDs:       27    Fs_use:             23
   Genfscon:           84    Portcon:           479
   Netifcon:            0    Nodecon:             0
   Permissives:        91    Polcap:              2


Any help or guidance would be very much appreciated, if there is more
in-depth info needed I'll be more than happy to provide it.

Yours sincerely,

Michal Marciniszyn
Manager - SW Engineering
GoodData

[-- Attachment #2: Type: text/html, Size: 5300 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 10:25 Performance issues - huge amount of AVC misses Michal Marciniszyn
@ 2015-12-08 10:44 ` Dominick Grift
  2015-12-08 14:56   ` Michal Marciniszyn
  2015-12-08 15:29 ` Stephen Smalley
  2015-12-09 10:07 ` Milos Malik
  2 siblings, 1 reply; 17+ messages in thread
From: Dominick Grift @ 2015-12-08 10:44 UTC (permalink / raw)
  To: Michal Marciniszyn; +Cc: selinux

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Tue, Dec 08, 2015 at 11:25:40AM +0100, Michal Marciniszyn wrote:
> Hello,
> 
> we are heavy SELinux shop and we recently run into AVC related performance
> issue. I was trying to find an answer on freenode IRC chat but I was sent
> here by multiple guys. We're running on Scientific Linux 6.6 (upgrade to
> 6.7 ongoing) and we see this on some of our nodes:
> 
> # cat /selinux/avc/cache_stats
> lookups hits misses allocations reclaims frees
> 3976846641 3626568307 350278334 350303465 344833264 346344169
> 3474274460 3092218096 382056364 382081270 381170512 382671551
> 2037181411 1655679702 381501709 381527148 380680320 382162477
> 1943162363 1651603455 291558908 291584892 288099840 289631602
> 829213467 406079951 423133516 423158604 422311024 423847681
> 1963015875 1555848944 407166931 407192104 406718592 408227742
> 3490131033 3117047653 373083380 373108386 372270880 373862706
> 940880689 549698684 391182005 391207388 390339328 391888374
> 4098417807 3712068859 386348948 386373592 385604096 387172806
> 3931378773 3549502965 381875808 381901074 381059904 382628308
> 
> Also we see
> 
> # cat /selinux/avc/hash_stats
> entries: 499
> buckets used: 257/512
> longest chain: 6
> 
> Some times under load we see SELinux consuming about 30% of CPU time. There
> is about 16% of cache misses on these nodes (and sometimes it goes as high
> as 30%). The lates article about the issue is from RHEL 5 times -
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
> . We do not feel this to be too relevant in this case.
> 
> Are there any recommendations on cache sizing for SELinux? We can resize
> cache to 1024 or 2048 entries, but would this help to resolve the issue?
> 
> I'm attaching seinfo from node with our policy and then for comparison from
> node without any policy.
> 
> With policy:
> # seinfo
> 
> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> Policy Version & Type: v.24 (binary, mls)
> 
>    Classes:            81    Permissions:       238
>    Sensitivities:       1    Categories:       1024
>    Types:            4273    Attributes:        295
>    Users:               9    Roles:              12
>    Booleans:          234    Cond. Expr.:       274
>    Allow:          352554    Neverallow:          0
>    Auditallow:        140    Dontaudit:      321786
>    Type_trans:      42813    Type_change:        38
>    Type_member:        48    Role allow:         19
>    Role_trans:        409    Range_trans:      6421
>    Constraints:        90    Validatetrans:       0
>    Initial SIDs:       27    Fs_use:             23
>    Genfscon:           84    Portcon:           505
>    Netifcon:            0    Nodecon:             0
>    Permissives:        91    Polcap:              2

I don't have any useful input but just an unrelated observation: you
almost have as many dontaudit rules as you have allow rules. I would not
be surprised if that were to be somehow related.

> 
> 
> 
> Without policy:
> 
> seinfo
> 
> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> Policy Version & Type: v.24 (binary, mls)
> 
>    Classes:            81    Permissions:       238
>    Sensitivities:       1    Categories:       1024
>    Types:            3926    Attributes:        295
>    Users:               9    Roles:              12
>    Booleans:          234    Cond. Expr.:       274
>    Allow:          320969    Neverallow:          0
>    Auditallow:        140    Dontaudit:      273256
>    Type_trans:      41915    Type_change:        38
>    Type_member:        48    Role allow:         19
>    Role_trans:        386    Range_trans:      6069
>    Constraints:        90    Validatetrans:       0
>    Initial SIDs:       27    Fs_use:             23
>    Genfscon:           84    Portcon:           479
>    Netifcon:            0    Nodecon:             0
>    Permissives:        91    Polcap:              2
> 
> 
> Any help or guidance would be very much appreciated, if there is more
> in-depth info needed I'll be more than happy to provide it.
> 
> Yours sincerely,
> 
> Michal Marciniszyn
> Manager - SW Engineering
> GoodData

> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.


- -- 
02DFF788
4D30 903A 1CF3 B756 FB48  1514 3148 83A2 02DF F788
https://sks-keyservers.net/pks/lookup?op=get&search=0x314883A202DFF788
Dominick Grift
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQGcBAEBCgAGBQJWZrSWAAoJENAR6kfG5xmctzkMALax9f+yHvM9hiH/RFgf4JMH
2avyWCkJggce+DilkHLGuhAZe0yMJW/h4WryF/a93y52q/09l/vYpa4oEShhrasD
dsOmCOINVW77E6TyWMuv80hYywPoXft+h3XIIgLO9FrURCJoCNlY7WGEpuVIy9PF
fxk6dxSov4yxxVGnEFW43X8SZ9haypuTiq3DkfvCVTbfeEs1xYu5j2vQ2Ghi0BN0
N9JdiLiPBBjAZo4O6VFkfgJ3Jt+EfyYuImcL3EhKmOc7c+vTtggc3VEamaSRXnhY
oXYUnKEqDraaJ7kizgODntPw79YRVpVqpaRipArZq96Qjq9loH/3RsG9DEyRTBgR
f3VH63L0URGeA7O/qWQmjiHro8ZgZvmKdfnRWtnwtUCfHmaGU8r8rDgWHReC42HD
FeRn+ymouSp0JDfq9wg3Nbk8R5z/FF4qIk4NpUNIm4KWRREbYQnkTjhMwN3hepg4
ikMHBdfUP/coPw1kPJtCwYNtwcv+z1D1XbRBiU/icQ==
=41nB
-----END PGP SIGNATURE-----

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 10:44 ` Dominick Grift
@ 2015-12-08 14:56   ` Michal Marciniszyn
  2015-12-08 15:05     ` Daniel J Walsh
                       ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Michal Marciniszyn @ 2015-12-08 14:56 UTC (permalink / raw)
  To: Michal Marciniszyn, selinux

[-- Attachment #1: Type: text/plain, Size: 6332 bytes --]

Hi Dominic,

while there is quite a lot of dontaudit rules around, the amount for
domains running on this node is not high. Is there any way how to monitor
which rules are loaded and released from the cache? Anything better than
plain aggregated stats? We would bot care about performance of such
monitoring tool if it provides some useful answer. For instance, is there a
way how to use system tap or similar kernel profiling to get the data?

I'll do a profiling on how many rules actually apply for the domains on the
node (i.e. use sesearch to find out). If doing so, does the rule in cache
hold whole vector (i.e. A is allowed to do X, Y, Z on B or is one cache
entry A can do X on B)?

Thanks,

--michal

On Tue, Dec 8, 2015 at 11:44 AM, Dominick Grift <dac.override@gmail.com>
wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On Tue, Dec 08, 2015 at 11:25:40AM +0100, Michal Marciniszyn wrote:
> > Hello,
> >
> > we are heavy SELinux shop and we recently run into AVC related
> performance
> > issue. I was trying to find an answer on freenode IRC chat but I was sent
> > here by multiple guys. We're running on Scientific Linux 6.6 (upgrade to
> > 6.7 ongoing) and we see this on some of our nodes:
> >
> > # cat /selinux/avc/cache_stats
> > lookups hits misses allocations reclaims frees
> > 3976846641 3626568307 350278334 350303465 344833264 346344169
> > 3474274460 3092218096 382056364 382081270 381170512 382671551
> > 2037181411 1655679702 381501709 381527148 380680320 382162477
> > 1943162363 1651603455 291558908 291584892 288099840 289631602
> > 829213467 406079951 423133516 423158604 422311024 423847681
> > 1963015875 1555848944 407166931 407192104 406718592 408227742
> > 3490131033 3117047653 373083380 373108386 372270880 373862706
> > 940880689 549698684 391182005 391207388 390339328 391888374
> > 4098417807 3712068859 386348948 386373592 385604096 387172806
> > 3931378773 3549502965 381875808 381901074 381059904 382628308
> >
> > Also we see
> >
> > # cat /selinux/avc/hash_stats
> > entries: 499
> > buckets used: 257/512
> > longest chain: 6
> >
> > Some times under load we see SELinux consuming about 30% of CPU time.
> There
> > is about 16% of cache misses on these nodes (and sometimes it goes as
> high
> > as 30%). The lates article about the issue is from RHEL 5 times -
> >
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
> > . We do not feel this to be too relevant in this case.
> >
> > Are there any recommendations on cache sizing for SELinux? We can resize
> > cache to 1024 or 2048 entries, but would this help to resolve the issue?
> >
> > I'm attaching seinfo from node with our policy and then for comparison
> from
> > node without any policy.
> >
> > With policy:
> > # seinfo
> >
> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > Policy Version & Type: v.24 (binary, mls)
> >
> >    Classes:            81    Permissions:       238
> >    Sensitivities:       1    Categories:       1024
> >    Types:            4273    Attributes:        295
> >    Users:               9    Roles:              12
> >    Booleans:          234    Cond. Expr.:       274
> >    Allow:          352554    Neverallow:          0
> >    Auditallow:        140    Dontaudit:      321786
> >    Type_trans:      42813    Type_change:        38
> >    Type_member:        48    Role allow:         19
> >    Role_trans:        409    Range_trans:      6421
> >    Constraints:        90    Validatetrans:       0
> >    Initial SIDs:       27    Fs_use:             23
> >    Genfscon:           84    Portcon:           505
> >    Netifcon:            0    Nodecon:             0
> >    Permissives:        91    Polcap:              2
>
> I don't have any useful input but just an unrelated observation: you
> almost have as many dontaudit rules as you have allow rules. I would not
> be surprised if that were to be somehow related.
>
> >
> >
> >
> > Without policy:
> >
> > seinfo
> >
> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > Policy Version & Type: v.24 (binary, mls)
> >
> >    Classes:            81    Permissions:       238
> >    Sensitivities:       1    Categories:       1024
> >    Types:            3926    Attributes:        295
> >    Users:               9    Roles:              12
> >    Booleans:          234    Cond. Expr.:       274
> >    Allow:          320969    Neverallow:          0
> >    Auditallow:        140    Dontaudit:      273256
> >    Type_trans:      41915    Type_change:        38
> >    Type_member:        48    Role allow:         19
> >    Role_trans:        386    Range_trans:      6069
> >    Constraints:        90    Validatetrans:       0
> >    Initial SIDs:       27    Fs_use:             23
> >    Genfscon:           84    Portcon:           479
> >    Netifcon:            0    Nodecon:             0
> >    Permissives:        91    Polcap:              2
> >
> >
> > Any help or guidance would be very much appreciated, if there is more
> > in-depth info needed I'll be more than happy to provide it.
> >
> > Yours sincerely,
> >
> > Michal Marciniszyn
> > Manager - SW Engineering
> > GoodData
>
> > _______________________________________________
> > Selinux mailing list
> > Selinux@tycho.nsa.gov
> > To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> > To get help, send an email containing "help" to
> Selinux-request@tycho.nsa.gov.
>
>
> - --
> 02DFF788
> 4D30 903A 1CF3 B756 FB48  1514 3148 83A2 02DF F788
> https://sks-keyservers.net/pks/lookup?op=get&search=0x314883A202DFF788
> Dominick Grift
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQGcBAEBCgAGBQJWZrSWAAoJENAR6kfG5xmctzkMALax9f+yHvM9hiH/RFgf4JMH
> 2avyWCkJggce+DilkHLGuhAZe0yMJW/h4WryF/a93y52q/09l/vYpa4oEShhrasD
> dsOmCOINVW77E6TyWMuv80hYywPoXft+h3XIIgLO9FrURCJoCNlY7WGEpuVIy9PF
> fxk6dxSov4yxxVGnEFW43X8SZ9haypuTiq3DkfvCVTbfeEs1xYu5j2vQ2Ghi0BN0
> N9JdiLiPBBjAZo4O6VFkfgJ3Jt+EfyYuImcL3EhKmOc7c+vTtggc3VEamaSRXnhY
> oXYUnKEqDraaJ7kizgODntPw79YRVpVqpaRipArZq96Qjq9loH/3RsG9DEyRTBgR
> f3VH63L0URGeA7O/qWQmjiHro8ZgZvmKdfnRWtnwtUCfHmaGU8r8rDgWHReC42HD
> FeRn+ymouSp0JDfq9wg3Nbk8R5z/FF4qIk4NpUNIm4KWRREbYQnkTjhMwN3hepg4
> ikMHBdfUP/coPw1kPJtCwYNtwcv+z1D1XbRBiU/icQ==
> =41nB
> -----END PGP SIGNATURE-----
>

[-- Attachment #2: Type: text/html, Size: 8366 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 14:56   ` Michal Marciniszyn
@ 2015-12-08 15:05     ` Daniel J Walsh
  2015-12-08 15:10     ` Dominick Grift
  2015-12-08 15:35     ` Stephen Smalley
  2 siblings, 0 replies; 17+ messages in thread
From: Daniel J Walsh @ 2015-12-08 15:05 UTC (permalink / raw)
  To: Michal Marciniszyn, selinux

[-- Attachment #1: Type: text/plain, Size: 6198 bytes --]

How are you defining your dontaudit rules?  It is a bad idea to use the
"-" sign.

dontaudit {domain -nissdomain} self:process kill;

Or something like that causes hundreds (Thousands) of rules to be added. 

If you could somehow clean this up to use an attribute like

dontaudit domainswithoutnis self:process kill

Would add one rule.

On 12/08/2015 09:56 AM, Michal Marciniszyn wrote:
> Hi Dominic, > > while there is quite a lot of dontaudit rules around, the amount
for domains running on this node is not high. Is there any way how to
monitor which rules are loaded and released from the cache? Anything
better than plain aggregated stats? We would bot care about performance
of such monitoring tool if it provides some useful answer. For instance,
is there a way how to use system tap or similar kernel profiling to get
the data? > > I'll do a profiling on how many rules actually apply for
the domains on the node (i.e. use sesearch to find out). If doing so,
does the rule in cache hold whole vector (i.e. A is allowed to do X, Y,
Z on B or is one cache entry A can do X on B)? > > Thanks, > > --michal
> > On Tue, Dec 8, 2015 at 11:44 AM, Dominick Grift
<dac.override@gmail.com <mailto:dac.override@gmail.com>> wrote: >
> On Tue, Dec 08, 2015 at 11:25:40AM +0100, Michal Marciniszyn wrote:
> > Hello,
>
> > we are heavy SELinux shop and we recently run into AVC related
> performance
> > issue. I was trying to find an answer on freenode IRC chat but I was
> sent
> > here by multiple guys. We're running on Scientific Linux 6.6 (upgrade to
> > 6.7 ongoing) and we see this on some of our nodes:
>
> > # cat /selinux/avc/cache_stats
> > lookups hits misses allocations reclaims frees
> > 3976846641 3626568307 350278334 350303465 344833264 346344169
> > 3474274460 3092218096 382056364 382081270 381170512 382671551
> > 2037181411 1655679702 381501709 381527148 380680320 382162477
> > 1943162363 1651603455 291558908 291584892 288099840 289631602
> > 829213467 406079951 423133516 423158604 422311024 423847681
> > 1963015875 1555848944 407166931 407192104 406718592 408227742
> > 3490131033 3117047653 373083380 373108386 372270880 373862706
> > 940880689 549698684 391182005 391207388 390339328 391888374
> > 4098417807 3712068859 386348948 386373592 385604096 387172806
> > 3931378773 3549502965 381875808 381901074 381059904 382628308
>
> > Also we see
>
> > # cat /selinux/avc/hash_stats
> > entries: 499
> > buckets used: 257/512
> > longest chain: 6
>
> > Some times under load we see SELinux consuming about 30% of CPU
> time. There
> > is about 16% of cache misses on these nodes (and sometimes it goes
> as high
> > as 30%). The lates article about the issue is from RHEL 5 times -
> >
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
> > . We do not feel this to be too relevant in this case.
>
> > Are there any recommendations on cache sizing for SELinux? We can resize
> > cache to 1024 or 2048 entries, but would this help to resolve the issue?
>
> > I'm attaching seinfo from node with our policy and then for
> comparison from
> > node without any policy.
>
> > With policy:
> > # seinfo
>
> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > Policy Version & Type: v.24 (binary, mls)
>
> >    Classes:            81    Permissions:       238
> >    Sensitivities:       1    Categories:       1024
> >    Types:            4273    Attributes:        295
> >    Users:               9    Roles:              12
> >    Booleans:          234    Cond. Expr.:       274
> >    Allow:          352554    Neverallow:          0
> >    Auditallow:        140    Dontaudit:      321786
> >    Type_trans:      42813    Type_change:        38
> >    Type_member:        48    Role allow:         19
> >    Role_trans:        409    Range_trans:      6421
> >    Constraints:        90    Validatetrans:       0
> >    Initial SIDs:       27    Fs_use:             23
> >    Genfscon:           84    Portcon:           505
> >    Netifcon:            0    Nodecon:             0
> >    Permissives:        91    Polcap:              2
>
> I don't have any useful input but just an unrelated observation: you
> almost have as many dontaudit rules as you have allow rules. I would not
> be surprised if that were to be somehow related.
>
>
>
>
> > Without policy:
>
> > seinfo
>
> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > Policy Version & Type: v.24 (binary, mls)
>
> >    Classes:            81    Permissions:       238
> >    Sensitivities:       1    Categories:       1024
> >    Types:            3926    Attributes:        295
> >    Users:               9    Roles:              12
> >    Booleans:          234    Cond. Expr.:       274
> >    Allow:          320969    Neverallow:          0
> >    Auditallow:        140    Dontaudit:      273256
> >    Type_trans:      41915    Type_change:        38
> >    Type_member:        48    Role allow:         19
> >    Role_trans:        386    Range_trans:      6069
> >    Constraints:        90    Validatetrans:       0
> >    Initial SIDs:       27    Fs_use:             23
> >    Genfscon:           84    Portcon:           479
> >    Netifcon:            0    Nodecon:             0
> >    Permissives:        91    Polcap:              2
>
>
> > Any help or guidance would be very much appreciated, if there is more
> > in-depth info needed I'll be more than happy to provide it.
>
> > Yours sincerely,
>
> > Michal Marciniszyn
> > Manager - SW Engineering
> > GoodData
>
> > _______________________________________________
> > Selinux mailing list
> > Selinux@tycho.nsa.gov <mailto:Selinux@tycho.nsa.gov>
> > To unsubscribe, send email to Selinux-leave@tycho.nsa.gov
> <mailto:Selinux-leave@tycho.nsa.gov>.
> > To get help, send an email containing "help" to
> Selinux-request@tycho.nsa.gov <mailto:Selinux-request@tycho.nsa.gov>.
>
>
> > > > > _______________________________________________ > Selinux
mailing list > Selinux@tycho.nsa.gov > To unsubscribe, send email to
Selinux-leave@tycho.nsa.gov. > To get help, send an email containing
"help" to Selinux-request@tycho.nsa.gov.



[-- Attachment #2: Type: text/html, Size: 9169 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 14:56   ` Michal Marciniszyn
  2015-12-08 15:05     ` Daniel J Walsh
@ 2015-12-08 15:10     ` Dominick Grift
  2015-12-08 15:35     ` Stephen Smalley
  2 siblings, 0 replies; 17+ messages in thread
From: Dominick Grift @ 2015-12-08 15:10 UTC (permalink / raw)
  To: Michal Marciniszyn; +Cc: selinux

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Tue, Dec 08, 2015 at 03:56:38PM +0100, Michal Marciniszyn wrote:
> Hi Dominic,
> 
> while there is quite a lot of dontaudit rules around, the amount for
> domains running on this node is not high. Is there any way how to monitor
> which rules are loaded and released from the cache? Anything better than
> plain aggregated stats? We would bot care about performance of such
> monitoring tool if it provides some useful answer. For instance, is there a
> way how to use system tap or similar kernel profiling to get the data?
> 
> I'll do a profiling on how many rules actually apply for the domains on the
> node (i.e. use sesearch to find out). If doing so, does the rule in cache
> hold whole vector (i.e. A is allowed to do X, Y, Z on B or is one cache
> entry A can do X on B)?

Hi,

I will let other people comment on these technical issues since i do
not feel that i am qualified to do so.

Let me instead suggest a different approach to the dontaudit challenge.

You might consider dontauditting on a different level using the base
attributes. This will allow you to catch everything with very little
rules.

So in a development phase you start by just ignoring things you want to
dontaudit in production. Then once you feel that your policy is ready
for production you add a few dontaudit rules that catch everything.

for example common access to files:

dontaudit domain file_type:file { manage_file_perms relabel_file_perms
};

any bind access to port objects:

dontaudit domain port_type:{ tcp_socket udp_socket dccp_socket }
name_bind;

etc etc

The point is that you use high level type attributes to catch
remaining attempts. It is not a perfect solution and it has its
drawbacks but atleast you will not have 300k dontaudit rules.

> 
> Thanks,
> 
> --michal
> 
> On Tue, Dec 8, 2015 at 11:44 AM, Dominick Grift <dac.override@gmail.com>
> wrote:
> 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > On Tue, Dec 08, 2015 at 11:25:40AM +0100, Michal Marciniszyn wrote:
> > > Hello,
> > >
> > > we are heavy SELinux shop and we recently run into AVC related
> > performance
> > > issue. I was trying to find an answer on freenode IRC chat but I was sent
> > > here by multiple guys. We're running on Scientific Linux 6.6 (upgrade to
> > > 6.7 ongoing) and we see this on some of our nodes:
> > >
> > > # cat /selinux/avc/cache_stats
> > > lookups hits misses allocations reclaims frees
> > > 3976846641 3626568307 350278334 350303465 344833264 346344169
> > > 3474274460 3092218096 382056364 382081270 381170512 382671551
> > > 2037181411 1655679702 381501709 381527148 380680320 382162477
> > > 1943162363 1651603455 291558908 291584892 288099840 289631602
> > > 829213467 406079951 423133516 423158604 422311024 423847681
> > > 1963015875 1555848944 407166931 407192104 406718592 408227742
> > > 3490131033 3117047653 373083380 373108386 372270880 373862706
> > > 940880689 549698684 391182005 391207388 390339328 391888374
> > > 4098417807 3712068859 386348948 386373592 385604096 387172806
> > > 3931378773 3549502965 381875808 381901074 381059904 382628308
> > >
> > > Also we see
> > >
> > > # cat /selinux/avc/hash_stats
> > > entries: 499
> > > buckets used: 257/512
> > > longest chain: 6
> > >
> > > Some times under load we see SELinux consuming about 30% of CPU time.
> > There
> > > is about 16% of cache misses on these nodes (and sometimes it goes as
> > high
> > > as 30%). The lates article about the issue is from RHEL 5 times -
> > >
> > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
> > > . We do not feel this to be too relevant in this case.
> > >
> > > Are there any recommendations on cache sizing for SELinux? We can resize
> > > cache to 1024 or 2048 entries, but would this help to resolve the issue?
> > >
> > > I'm attaching seinfo from node with our policy and then for comparison
> > from
> > > node without any policy.
> > >
> > > With policy:
> > > # seinfo
> > >
> > > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > > Policy Version & Type: v.24 (binary, mls)
> > >
> > >    Classes:            81    Permissions:       238
> > >    Sensitivities:       1    Categories:       1024
> > >    Types:            4273    Attributes:        295
> > >    Users:               9    Roles:              12
> > >    Booleans:          234    Cond. Expr.:       274
> > >    Allow:          352554    Neverallow:          0
> > >    Auditallow:        140    Dontaudit:      321786
> > >    Type_trans:      42813    Type_change:        38
> > >    Type_member:        48    Role allow:         19
> > >    Role_trans:        409    Range_trans:      6421
> > >    Constraints:        90    Validatetrans:       0
> > >    Initial SIDs:       27    Fs_use:             23
> > >    Genfscon:           84    Portcon:           505
> > >    Netifcon:            0    Nodecon:             0
> > >    Permissives:        91    Polcap:              2
> >
> > I don't have any useful input but just an unrelated observation: you
> > almost have as many dontaudit rules as you have allow rules. I would not
> > be surprised if that were to be somehow related.
> >
> > >
> > >
> > >
> > > Without policy:
> > >
> > > seinfo
> > >
> > > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > > Policy Version & Type: v.24 (binary, mls)
> > >
> > >    Classes:            81    Permissions:       238
> > >    Sensitivities:       1    Categories:       1024
> > >    Types:            3926    Attributes:        295
> > >    Users:               9    Roles:              12
> > >    Booleans:          234    Cond. Expr.:       274
> > >    Allow:          320969    Neverallow:          0
> > >    Auditallow:        140    Dontaudit:      273256
> > >    Type_trans:      41915    Type_change:        38
> > >    Type_member:        48    Role allow:         19
> > >    Role_trans:        386    Range_trans:      6069
> > >    Constraints:        90    Validatetrans:       0
> > >    Initial SIDs:       27    Fs_use:             23
> > >    Genfscon:           84    Portcon:           479
> > >    Netifcon:            0    Nodecon:             0
> > >    Permissives:        91    Polcap:              2
> > >
> > >
> > > Any help or guidance would be very much appreciated, if there is more
> > > in-depth info needed I'll be more than happy to provide it.
> > >
> > > Yours sincerely,
> > >
> > > Michal Marciniszyn
> > > Manager - SW Engineering
> > > GoodData
> >
> > > _______________________________________________
> > > Selinux mailing list
> > > Selinux@tycho.nsa.gov
> > > To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> > > To get help, send an email containing "help" to
> > Selinux-request@tycho.nsa.gov.
> >
> >
> > - --
> > 02DFF788
> > 4D30 903A 1CF3 B756 FB48  1514 3148 83A2 02DF F788
> > https://sks-keyservers.net/pks/lookup?op=get&search=0x314883A202DFF788
> > Dominick Grift
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2
> >
> > iQGcBAEBCgAGBQJWZrSWAAoJENAR6kfG5xmctzkMALax9f+yHvM9hiH/RFgf4JMH
> > 2avyWCkJggce+DilkHLGuhAZe0yMJW/h4WryF/a93y52q/09l/vYpa4oEShhrasD
> > dsOmCOINVW77E6TyWMuv80hYywPoXft+h3XIIgLO9FrURCJoCNlY7WGEpuVIy9PF
> > fxk6dxSov4yxxVGnEFW43X8SZ9haypuTiq3DkfvCVTbfeEs1xYu5j2vQ2Ghi0BN0
> > N9JdiLiPBBjAZo4O6VFkfgJ3Jt+EfyYuImcL3EhKmOc7c+vTtggc3VEamaSRXnhY
> > oXYUnKEqDraaJ7kizgODntPw79YRVpVqpaRipArZq96Qjq9loH/3RsG9DEyRTBgR
> > f3VH63L0URGeA7O/qWQmjiHro8ZgZvmKdfnRWtnwtUCfHmaGU8r8rDgWHReC42HD
> > FeRn+ymouSp0JDfq9wg3Nbk8R5z/FF4qIk4NpUNIm4KWRREbYQnkTjhMwN3hepg4
> > ikMHBdfUP/coPw1kPJtCwYNtwcv+z1D1XbRBiU/icQ==
> > =41nB
> > -----END PGP SIGNATURE-----
> >

> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.


- -- 
02DFF788
4D30 903A 1CF3 B756 FB48  1514 3148 83A2 02DF F788
https://sks-keyservers.net/pks/lookup?op=get&search=0x314883A202DFF788
Dominick Grift
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQGcBAEBCgAGBQJWZvLHAAoJENAR6kfG5xmcGxEMAIWpCUl/BWQFTYMgPG5t85bB
NHMaIAKdRPw0IdxWOEzSRWHS4bmOgx0c6ChMlAxBAIAm0ncOpOoZgY5JO6PlOYp4
cvcXUC88b+QVa/BfReemZCb5jeFelW8o0aZYkQVl3IaMvXJLgXVE/O9xwJH5n459
TZGGD08GEVRykM8qn9fgcKlp02RGtZcFCqjgXPeZum291k8cdo7e+ejwTRmssQfs
tPtL3MsGWsIxaQq+G4u0OJVJmASNV0zjje8jLWFxUoV33vbwPCrLPqCsoAtVAPkJ
2Vjd+U1dw8ncL1R8dXWt3Y0finXBnI7n7QCiHCg3UIBoRPSZhAQ4DevNWbrYvZjl
RYRf+fAOIWHmgzxlQJAqKnPu0rc6QQgk/0XhvjpQb/jPbSpF2UEWD1aKrYmoNmkJ
ij2g7/UMm6RLNnncKXL2ff6Zatl8z7ObDl0+mrCG2cgDBwLGRapzVtVBQCMlbXa8
JKR31LXE1CgRRfhDhG19mY/KHh3GTAV7SLF/4dHrWQ==
=p9u8
-----END PGP SIGNATURE-----

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 10:25 Performance issues - huge amount of AVC misses Michal Marciniszyn
  2015-12-08 10:44 ` Dominick Grift
@ 2015-12-08 15:29 ` Stephen Smalley
  2015-12-08 16:16   ` Michal Marciniszyn
  2015-12-09 10:07 ` Milos Malik
  2 siblings, 1 reply; 17+ messages in thread
From: Stephen Smalley @ 2015-12-08 15:29 UTC (permalink / raw)
  To: Michal Marciniszyn, selinux

On 12/08/2015 05:25 AM, Michal Marciniszyn wrote:
> Hello,
>
> we are heavy SELinux shop and we recently run into AVC related
> performance issue. I was trying to find an answer on freenode IRC chat
> but I was sent here by multiple guys. We're running on Scientific Linux
> 6.6 (upgrade to 6.7 ongoing) and we see this on some of our nodes:
>
> # cat /selinux/avc/cache_stats
> lookups hits misses allocations reclaims frees
> 3976846641 3626568307 350278334 350303465 344833264 346344169
> 3474274460 3092218096 382056364 382081270 381170512 382671551
> 2037181411 1655679702 381501709 381527148 380680320 382162477
> 1943162363 1651603455 291558908 291584892 288099840 289631602
> 829213467 406079951 423133516 423158604 422311024 423847681
> 1963015875 1555848944 407166931 407192104 406718592 408227742
> 3490131033 3117047653 373083380 373108386 372270880 373862706
> 940880689 549698684 391182005 391207388 390339328 391888374
> 4098417807 3712068859 386348948 386373592 385604096 387172806
> 3931378773 3549502965 381875808 381901074 381059904 382628308

FWIW, avcstat would summarize that for you.
Those stats seem very unusual.  You said you see this on some nodes. 
Anything to distinguish these nodes from the others that don't exhibit 
this behavior?

>
> Also we see
>
> # cat /selinux/avc/hash_stats
> entries: 499
> buckets used: 257/512
> longest chain: 6
>
> Some times under load we see SELinux consuming about 30% of CPU time.
> There is about 16% of cache misses on these nodes (and sometimes it goes
> as high as 30%). The lates article about the issue is from RHEL 5 times
> -
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
> . We do not feel this to be too relevant in this case.
>
> Are there any recommendations on cache sizing for SELinux? We can resize
> cache to 1024 or 2048 entries, but would this help to resolve the issue?

Yes, increasing the cache threshold should help as you are evidently 
thrashing the cache.

> I'm attaching seinfo from node with our policy and then for comparison
> from node without any policy.

What do you mean by "our policy" versus "without any policy"?  Do you 
mean that the former has some local policy modules that you have added 
and the latter is the stock SL6.6 policy?

> With policy:
> # seinfo
>
> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> Policy Version & Type: v.24 (binary, mls)
>
>     Classes:            81    Permissions:       238
>     Sensitivities:       1    Categories:       1024
>     Types:            4273    Attributes:        295
>     Users:               9    Roles:              12
>     Booleans:          234    Cond. Expr.:       274
>     Allow:          352554    Neverallow:          0
>     Auditallow:        140    Dontaudit:      321786
>     Type_trans:      42813    Type_change:        38
>     Type_member:        48    Role allow:         19
>     Role_trans:        409    Range_trans:      6421
>     Constraints:        90    Validatetrans:       0
>     Initial SIDs:       27    Fs_use:             23
>     Genfscon:           84    Portcon:           505
>     Netifcon:            0    Nodecon:             0
>     Permissives:        91    Polcap:              2
>
>
>
> Without policy:
>
> seinfo
>
> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> Policy Version & Type: v.24 (binary, mls)
>
>     Classes:            81    Permissions:       238
>     Sensitivities:       1    Categories:       1024
>     Types:            3926    Attributes:        295
>     Users:               9    Roles:              12
>     Booleans:          234    Cond. Expr.:       274
>     Allow:          320969    Neverallow:          0
>     Auditallow:        140    Dontaudit:      273256
>     Type_trans:      41915    Type_change:        38
>     Type_member:        48    Role allow:         19
>     Role_trans:        386    Range_trans:      6069
>     Constraints:        90    Validatetrans:       0
>     Initial SIDs:       27    Fs_use:             23
>     Genfscon:           84    Portcon:           479
>     Netifcon:            0    Nodecon:             0
>     Permissives:        91    Polcap:              2
>
>
> Any help or guidance would be very much appreciated, if there is more
> in-depth info needed I'll be more than happy to provide it.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 14:56   ` Michal Marciniszyn
  2015-12-08 15:05     ` Daniel J Walsh
  2015-12-08 15:10     ` Dominick Grift
@ 2015-12-08 15:35     ` Stephen Smalley
  2015-12-08 16:21       ` Michal Marciniszyn
  2 siblings, 1 reply; 17+ messages in thread
From: Stephen Smalley @ 2015-12-08 15:35 UTC (permalink / raw)
  To: Michal Marciniszyn, selinux, Paul Moore

On 12/08/2015 09:56 AM, Michal Marciniszyn wrote:
> Hi Dominic,
>
> while there is quite a lot of dontaudit rules around, the amount for
> domains running on this node is not high. Is there any way how to
> monitor which rules are loaded and released from the cache? Anything
> better than plain aggregated stats? We would bot care about performance
> of such monitoring tool if it provides some useful answer. For instance,
> is there a way how to use system tap or similar kernel profiling to get
> the data?
>
> I'll do a profiling on how many rules actually apply for the domains on
> the node (i.e. use sesearch to find out). If doing so, does the rule in
> cache hold whole vector (i.e. A is allowed to do X, Y, Z on B or is one
> cache entry A can do X on B)?

One cache entry holds the entire access vector.  However, they are 
unique per (source context, target context, target class) triple.

Are you using categories on this system (i.e. running processes in 
specific category sets, assigning specific categories to files), or just 
types?

How many unique domains are running in your workload?  How many file 
types are typically accessed by your workload?  How many different kinds 
of files (regular, directory, symbolic link, block device, ...) are part 
of your workload?

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 15:29 ` Stephen Smalley
@ 2015-12-08 16:16   ` Michal Marciniszyn
  0 siblings, 0 replies; 17+ messages in thread
From: Michal Marciniszyn @ 2015-12-08 16:16 UTC (permalink / raw)
  To: Stephen Smalley; +Cc: selinux, Paul Moore

[-- Attachment #1: Type: text/plain, Size: 4814 bytes --]

On Tue, Dec 8, 2015 at 4:29 PM, Stephen Smalley <sds@tycho.nsa.gov> wrote:

> On 12/08/2015 05:25 AM, Michal Marciniszyn wrote:
>
>> Hello,
>>
>> we are heavy SELinux shop and we recently run into AVC related
>> performance issue. I was trying to find an answer on freenode IRC chat
>> but I was sent here by multiple guys. We're running on Scientific Linux
>> 6.6 (upgrade to 6.7 ongoing) and we see this on some of our nodes:
>>
>> # cat /selinux/avc/cache_stats
>> lookups hits misses allocations reclaims frees
>> 3976846641 3626568307 350278334 350303465 344833264 346344169
>> 3474274460 3092218096 382056364 382081270 381170512 382671551
>> 2037181411 1655679702 381501709 381527148 380680320 382162477
>> 1943162363 1651603455 291558908 291584892 288099840 289631602
>> 829213467 406079951 423133516 423158604 422311024 423847681
>> 1963015875 1555848944 407166931 407192104 406718592 408227742
>> 3490131033 3117047653 373083380 373108386 372270880 373862706
>> 940880689 549698684 391182005 391207388 390339328 391888374
>> 4098417807 3712068859 386348948 386373592 385604096 387172806
>> 3931378773 3549502965 381875808 381901074 381059904 382628308
>>
>
> FWIW, avcstat would summarize that for you.
> Those stats seem very unusual.  You said you see this on some nodes.
> Anything to distinguish these nodes from the others that don't exhibit this
> behavior?

Yes, these nodes are nodes with clustered vertica DB, which probably has
the biggest amount of rules overall.


>
>
>
>> Also we see
>>
>> # cat /selinux/avc/hash_stats
>> entries: 499
>> buckets used: 257/512
>> longest chain: 6
>>
>> Some times under load we see SELinux consuming about 30% of CPU time.
>> There is about 16% of cache misses on these nodes (and sometimes it goes
>> as high as 30%). The lates article about the issue is from RHEL 5 times
>> -
>>
>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
>> . We do not feel this to be too relevant in this case.
>>
>> Are there any recommendations on cache sizing for SELinux? We can resize
>> cache to 1024 or 2048 entries, but would this help to resolve the issue?
>>
>
> Yes, increasing the cache threshold should help as you are evidently
> thrashing the cache.
>
> I'm attaching seinfo from node with our policy and then for comparison
>> from node without any policy.
>>
>
> What do you mean by "our policy" versus "without any policy"?  Do you mean
> that the former has some local policy modules that you have added and the
> latter is the stock SL6.6 policy?

Yes, exacly, the 1st has stock SL 6.6 + our module, the latter one is just
with stock 6.6.

>
>
> With policy:
>> # seinfo
>>
>> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
>> Policy Version & Type: v.24 (binary, mls)
>>
>>     Classes:            81    Permissions:       238
>>     Sensitivities:       1    Categories:       1024
>>     Types:            4273    Attributes:        295
>>     Users:               9    Roles:              12
>>     Booleans:          234    Cond. Expr.:       274
>>     Allow:          352554    Neverallow:          0
>>     Auditallow:        140    Dontaudit:      321786
>>     Type_trans:      42813    Type_change:        38
>>     Type_member:        48    Role allow:         19
>>     Role_trans:        409    Range_trans:      6421
>>     Constraints:        90    Validatetrans:       0
>>     Initial SIDs:       27    Fs_use:             23
>>     Genfscon:           84    Portcon:           505
>>     Netifcon:            0    Nodecon:             0
>>     Permissives:        91    Polcap:              2
>>
>>
>>
>> Without policy:
>>
>> seinfo
>>
>> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
>> Policy Version & Type: v.24 (binary, mls)
>>
>>     Classes:            81    Permissions:       238
>>     Sensitivities:       1    Categories:       1024
>>     Types:            3926    Attributes:        295
>>     Users:               9    Roles:              12
>>     Booleans:          234    Cond. Expr.:       274
>>     Allow:          320969    Neverallow:          0
>>     Auditallow:        140    Dontaudit:      273256
>>     Type_trans:      41915    Type_change:        38
>>     Type_member:        48    Role allow:         19
>>     Role_trans:        386    Range_trans:      6069
>>     Constraints:        90    Validatetrans:       0
>>     Initial SIDs:       27    Fs_use:             23
>>     Genfscon:           84    Portcon:           479
>>     Netifcon:            0    Nodecon:             0
>>     Permissives:        91    Polcap:              2
>>
>>
>> Any help or guidance would be very much appreciated, if there is more
>> in-depth info needed I'll be more than happy to provide it.
>>
>
>
>

[-- Attachment #2: Type: text/html, Size: 7065 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 15:35     ` Stephen Smalley
@ 2015-12-08 16:21       ` Michal Marciniszyn
  2015-12-08 16:29         ` Dominick Grift
  2015-12-08 17:06         ` Stephen Smalley
  0 siblings, 2 replies; 17+ messages in thread
From: Michal Marciniszyn @ 2015-12-08 16:21 UTC (permalink / raw)
  To: Stephen Smalley; +Cc: selinux, Paul Moore

[-- Attachment #1: Type: text/plain, Size: 2074 bytes --]

Hi,

there are neither categories nor MLS used on the system. I'll get the
amount of different types used by the system (I need to do some digging,
will get the data tomorrow). Most of classes will be regular file,
directories and some symbolic links. There will be a lots of files as AFAIK
vertica uses lots of smaller files.

I'll try to reduce amount of dontaudit rules and I'll see how much this
reduces cache misses. The hard truth is, that vertica is looking at many
places during the run, most of which it does not need. Maybe the way we
have rules defined is creating a lot of stress on the amount of rules in
the policy, I'll try to get the data on that.

On Tue, Dec 8, 2015 at 4:35 PM, Stephen Smalley <sds@tycho.nsa.gov> wrote:

> On 12/08/2015 09:56 AM, Michal Marciniszyn wrote:
>
>> Hi Dominic,
>>
>> while there is quite a lot of dontaudit rules around, the amount for
>> domains running on this node is not high. Is there any way how to
>> monitor which rules are loaded and released from the cache? Anything
>> better than plain aggregated stats? We would bot care about performance
>> of such monitoring tool if it provides some useful answer. For instance,
>> is there a way how to use system tap or similar kernel profiling to get
>> the data?
>>
>> I'll do a profiling on how many rules actually apply for the domains on
>> the node (i.e. use sesearch to find out). If doing so, does the rule in
>> cache hold whole vector (i.e. A is allowed to do X, Y, Z on B or is one
>> cache entry A can do X on B)?
>>
>
> One cache entry holds the entire access vector.  However, they are unique
> per (source context, target context, target class) triple.
>
> Are you using categories on this system (i.e. running processes in
> specific category sets, assigning specific categories to files), or just
> types?
>
> How many unique domains are running in your workload?  How many file types
> are typically accessed by your workload?  How many different kinds of files
> (regular, directory, symbolic link, block device, ...) are part of your
> workload?
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2647 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 16:21       ` Michal Marciniszyn
@ 2015-12-08 16:29         ` Dominick Grift
  2015-12-08 17:06         ` Stephen Smalley
  1 sibling, 0 replies; 17+ messages in thread
From: Dominick Grift @ 2015-12-08 16:29 UTC (permalink / raw)
  To: Michal Marciniszyn; +Cc: Stephen Smalley, selinux

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Tue, Dec 08, 2015 at 05:21:17PM +0100, Michal Marciniszyn wrote:
> 
> I'll try to reduce amount of dontaudit rules and I'll see how much this
> reduces cache misses. The hard truth is, that vertica is looking at many
> places during the run, most of which it does not need. Maybe the way we
> have rules defined is creating a lot of stress on the amount of rules in
> the policy, I'll try to get the data on that.
> 

Yes, no after second thought I now believe it is totally unrelated and
not an issue. The amount of dontaudit rules is huge in stock 6.6 as well
(You are adding like 10% (?) so that is pretty insignificant)

Also there little you can do about the majority of dontaudit rules.

So stock SL6.6 comes with 91 permissive domains? wow, just wow.

- -- 
02DFF788
4D30 903A 1CF3 B756 FB48  1514 3148 83A2 02DF F788
https://sks-keyservers.net/pks/lookup?op=get&search=0x314883A202DFF788
Dominick Grift
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQGcBAEBCgAGBQJWZwWBAAoJENAR6kfG5xmccvUMAJmXVcoBvqrXzN5kTHpWnHBM
wPrGdx18tEyHeokE0U6FSlqXSh8/Hl9Fn2VSZLUyYO5mYm2dMoCFHkw45zs9svAB
2ugG9hEmoNgaX8KPxSm1LwWn23zTxnngeq8HU4n+ZSblQiW+EAeLPTtSHhqtA2OC
sXIBm6B3lfp5OPinQTsZ5xvpfTNe8eyswhEej3DCzr02tw5rheYzk3KvPKXKP6wV
OpQH7CwZ5Fi/7Ik298lU4tR321qtvLwxMUGcSMGT3Nkakul/GhH/RQOis2SFKlAy
HZGr4z/eLtAiwTgKFt+TuEgS+auFyZIeu4rlnky8qUhcc+j4fAVzDTNPtRV6LDHG
+Z2kbjgvR0Qk7QI7szuHiFYUfV/8ts6uzGMLEaQtBNEH0K7X1d0wk5qLOBiKrZOa
Zp0Sjnsv/ADhlRMD4WnqJ4R5NvU/p7rhYq5Xlh9/NadBXOon9Q4KBFzGUa+ZDvpy
YH7hgaRMQoQuPW/3FPlU47v2o1lMusuyYXqgGsZZyA==
=W08P
-----END PGP SIGNATURE-----

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 16:21       ` Michal Marciniszyn
  2015-12-08 16:29         ` Dominick Grift
@ 2015-12-08 17:06         ` Stephen Smalley
  1 sibling, 0 replies; 17+ messages in thread
From: Stephen Smalley @ 2015-12-08 17:06 UTC (permalink / raw)
  To: Michal Marciniszyn; +Cc: selinux, Paul Moore

On 12/08/2015 11:21 AM, Michal Marciniszyn wrote:
> Hi,
>
> there are neither categories nor MLS used on the system. I'll get the
> amount of different types used by the system (I need to do some digging,
> will get the data tomorrow). Most of classes will be regular file,
> directories and some symbolic links. There will be a lots of files as
> AFAIK vertica uses lots of smaller files.
>
> I'll try to reduce amount of dontaudit rules and I'll see how much this
> reduces cache misses. The hard truth is, that vertica is looking at many
> places during the run, most of which it does not need. Maybe the way we
> have rules defined is creating a lot of stress on the amount of rules in
> the policy, I'll try to get the data on that.

Cache misses aren't related to your number of dontaudit rules (or your 
number of rules at all, for that matter).  The optimal AVC size is 
driven by the number of unique (source security context, target security 
context, target security class) triples being accessed during the 
workload.  Each entry holds a complete access vector decision structure, 
including permissions that are allowed, permissions that are audited 
when denied, and permissions that are audited when allowed.

I would recommend trying different values for the cache threshold and 
see how it performs. Collecting information on the number of domains, 
types, and classes involved in your workload may be helpful in 
determining the optimal value, but some experimentation will likely be 
required regardless.

Reducing the number of rules may help with the performance overhead when 
there is an AVC miss, but the first step is to reduce the AVC misses.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-08 10:25 Performance issues - huge amount of AVC misses Michal Marciniszyn
  2015-12-08 10:44 ` Dominick Grift
  2015-12-08 15:29 ` Stephen Smalley
@ 2015-12-09 10:07 ` Milos Malik
  2015-12-09 10:19   ` Michal Marciniszyn
  2 siblings, 1 reply; 17+ messages in thread
From: Milos Malik @ 2015-12-09 10:07 UTC (permalink / raw)
  To: Michal Marciniszyn; +Cc: selinux

Hi Michal,

which process (from the "top -d1" output) is consuming almost 30% of CPU? Is it setroubleshootd or auditd or sedispatch or kernel? Thanks for the answer.

Milos Malik
SELinux QE person
BaseOS QE Security team
Red Hat Czech

----- Original Message -----
> Hello,
> 
> we are heavy SELinux shop and we recently run into AVC related performance
> issue. I was trying to find an answer on freenode IRC chat but I was sent
> here by multiple guys. We're running on Scientific Linux 6.6 (upgrade to 6.7
> ongoing) and we see this on some of our nodes:
> 
> # cat /selinux/avc/cache_stats
> lookups hits misses allocations reclaims frees
> 3976846641 3626568307 350278334 350303465 344833264 346344169
> 3474274460 3092218096 382056364 382081270 381170512 382671551
> 2037181411 1655679702 381501709 381527148 380680320 382162477
> 1943162363 1651603455 291558908 291584892 288099840 289631602
> 829213467 406079951 423133516 423158604 422311024 423847681
> 1963015875 1555848944 407166931 407192104 406718592 408227742
> 3490131033 3117047653 373083380 373108386 372270880 373862706
> 940880689 549698684 391182005 391207388 390339328 391888374
> 4098417807 3712068859 386348948 386373592 385604096 387172806
> 3931378773 3549502965 381875808 381901074 381059904 382628308
> 
> Also we see
> 
> # cat /selinux/avc/hash_stats
> entries: 499
> buckets used: 257/512
> longest chain: 6
> 
> Some times under load we see SELinux consuming about 30% of CPU time. There
> is about 16% of cache misses on these nodes (and sometimes it goes as high
> as 30%). The lates article about the issue is from RHEL 5 times -
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
> . We do not feel this to be too relevant in this case.
> 
> Are there any recommendations on cache sizing for SELinux? We can resize
> cache to 1024 or 2048 entries, but would this help to resolve the issue?
> 
> I'm attaching seinfo from node with our policy and then for comparison from
> node without any policy.
> 
> With policy:
> # seinfo
> 
> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> Policy Version & Type: v.24 (binary, mls)
> 
> Classes: 81 Permissions: 238
> Sensitivities: 1 Categories: 1024
> Types: 4273 Attributes: 295
> Users: 9 Roles: 12
> Booleans: 234 Cond. Expr.: 274
> Allow: 352554 Neverallow: 0
> Auditallow: 140 Dontaudit: 321786
> Type_trans: 42813 Type_change: 38
> Type_member: 48 Role allow: 19
> Role_trans: 409 Range_trans: 6421
> Constraints: 90 Validatetrans: 0
> Initial SIDs: 27 Fs_use: 23
> Genfscon: 84 Portcon: 505
> Netifcon: 0 Nodecon: 0
> Permissives: 91 Polcap: 2
> 
> 
> 
> Without policy:
> 
> seinfo
> 
> Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> Policy Version & Type: v.24 (binary, mls)
> 
> Classes: 81 Permissions: 238
> Sensitivities: 1 Categories: 1024
> Types: 3926 Attributes: 295
> Users: 9 Roles: 12
> Booleans: 234 Cond. Expr.: 274
> Allow: 320969 Neverallow: 0
> Auditallow: 140 Dontaudit: 273256
> Type_trans: 41915 Type_change: 38
> Type_member: 48 Role allow: 19
> Role_trans: 386 Range_trans: 6069
> Constraints: 90 Validatetrans: 0
> Initial SIDs: 27 Fs_use: 23
> Genfscon: 84 Portcon: 479
> Netifcon: 0 Nodecon: 0
> Permissives: 91 Polcap: 2
> 
> 
> Any help or guidance would be very much appreciated, if there is more
> in-depth info needed I'll be more than happy to provide it.
> 
> Yours sincerely,
> 
> Michal Marciniszyn
> Manager - SW Engineering
> GoodData
> 
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to
> Selinux-request@tycho.nsa.gov.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-09 10:07 ` Milos Malik
@ 2015-12-09 10:19   ` Michal Marciniszyn
  2015-12-09 13:15     ` Michal Marciniszyn
  0 siblings, 1 reply; 17+ messages in thread
From: Michal Marciniszyn @ 2015-12-09 10:19 UTC (permalink / raw)
  To: Milos Malik; +Cc: selinux

[-- Attachment #1: Type: text/plain, Size: 4233 bytes --]

Hi,

regarding the process, we see that in perf top. Under heavier load we see
following
21.53% [kernel] [k] avtab_search_node

sometimes even with higher percentage.

--blesk

On Wed, Dec 9, 2015 at 11:07 AM, Milos Malik <mmalik@redhat.com> wrote:

> Hi Michal,
>
> which process (from the "top -d1" output) is consuming almost 30% of CPU?
> Is it setroubleshootd or auditd or sedispatch or kernel? Thanks for the
> answer.
>
> Milos Malik
> SELinux QE person
> BaseOS QE Security team
> Red Hat Czech
>
> ----- Original Message -----
> > Hello,
> >
> > we are heavy SELinux shop and we recently run into AVC related
> performance
> > issue. I was trying to find an answer on freenode IRC chat but I was sent
> > here by multiple guys. We're running on Scientific Linux 6.6 (upgrade to
> 6.7
> > ongoing) and we see this on some of our nodes:
> >
> > # cat /selinux/avc/cache_stats
> > lookups hits misses allocations reclaims frees
> > 3976846641 3626568307 350278334 350303465 344833264 346344169
> > 3474274460 3092218096 382056364 382081270 381170512 382671551
> > 2037181411 1655679702 381501709 381527148 380680320 382162477
> > 1943162363 1651603455 291558908 291584892 288099840 289631602
> > 829213467 406079951 423133516 423158604 422311024 423847681
> > 1963015875 1555848944 407166931 407192104 406718592 408227742
> > 3490131033 3117047653 373083380 373108386 372270880 373862706
> > 940880689 549698684 391182005 391207388 390339328 391888374
> > 4098417807 3712068859 386348948 386373592 385604096 387172806
> > 3931378773 3549502965 381875808 381901074 381059904 382628308
> >
> > Also we see
> >
> > # cat /selinux/avc/hash_stats
> > entries: 499
> > buckets used: 257/512
> > longest chain: 6
> >
> > Some times under load we see SELinux consuming about 30% of CPU time.
> There
> > is about 16% of cache misses on these nodes (and sometimes it goes as
> high
> > as 30%). The lates article about the issue is from RHEL 5 times -
> >
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
> > . We do not feel this to be too relevant in this case.
> >
> > Are there any recommendations on cache sizing for SELinux? We can resize
> > cache to 1024 or 2048 entries, but would this help to resolve the issue?
> >
> > I'm attaching seinfo from node with our policy and then for comparison
> from
> > node without any policy.
> >
> > With policy:
> > # seinfo
> >
> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > Policy Version & Type: v.24 (binary, mls)
> >
> > Classes: 81 Permissions: 238
> > Sensitivities: 1 Categories: 1024
> > Types: 4273 Attributes: 295
> > Users: 9 Roles: 12
> > Booleans: 234 Cond. Expr.: 274
> > Allow: 352554 Neverallow: 0
> > Auditallow: 140 Dontaudit: 321786
> > Type_trans: 42813 Type_change: 38
> > Type_member: 48 Role allow: 19
> > Role_trans: 409 Range_trans: 6421
> > Constraints: 90 Validatetrans: 0
> > Initial SIDs: 27 Fs_use: 23
> > Genfscon: 84 Portcon: 505
> > Netifcon: 0 Nodecon: 0
> > Permissives: 91 Polcap: 2
> >
> >
> >
> > Without policy:
> >
> > seinfo
> >
> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
> > Policy Version & Type: v.24 (binary, mls)
> >
> > Classes: 81 Permissions: 238
> > Sensitivities: 1 Categories: 1024
> > Types: 3926 Attributes: 295
> > Users: 9 Roles: 12
> > Booleans: 234 Cond. Expr.: 274
> > Allow: 320969 Neverallow: 0
> > Auditallow: 140 Dontaudit: 273256
> > Type_trans: 41915 Type_change: 38
> > Type_member: 48 Role allow: 19
> > Role_trans: 386 Range_trans: 6069
> > Constraints: 90 Validatetrans: 0
> > Initial SIDs: 27 Fs_use: 23
> > Genfscon: 84 Portcon: 479
> > Netifcon: 0 Nodecon: 0
> > Permissives: 91 Polcap: 2
> >
> >
> > Any help or guidance would be very much appreciated, if there is more
> > in-depth info needed I'll be more than happy to provide it.
> >
> > Yours sincerely,
> >
> > Michal Marciniszyn
> > Manager - SW Engineering
> > GoodData
> >
> > _______________________________________________
> > Selinux mailing list
> > Selinux@tycho.nsa.gov
> > To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> > To get help, send an email containing "help" to
> > Selinux-request@tycho.nsa.gov.
>

[-- Attachment #2: Type: text/html, Size: 5748 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-09 10:19   ` Michal Marciniszyn
@ 2015-12-09 13:15     ` Michal Marciniszyn
  2015-12-09 15:05       ` Stephen Smalley
  0 siblings, 1 reply; 17+ messages in thread
From: Michal Marciniszyn @ 2015-12-09 13:15 UTC (permalink / raw)
  To: Milos Malik; +Cc: selinux

[-- Attachment #1: Type: text/plain, Size: 5645 bytes --]

Hi,

after increasing the cache, I do not see many reclaims, like couple of them
here and there. The cache size had to be increased to 2048 to get ti this
state.

# avcstat 15

    537645     537623         22         22         32         32
    942916     942912          4          4          0          0
    604466     604457          9          9          0          0
    451737     451730          7          7         16         16
    457669     457669          0          0          0          0
    519135     519133          2          2          0          0
    517288     517288          0          0          0          0
    380376     380376          0          0          0          0
    464272     464269          3          3          0          0
    531484     531482          2          2          0          0
   1422422    1422421          1          1          0          0
   1380932    1380932          0          0          0          0
    512999     512999          0          0          0          0


Is it ok if I get longest chain length 13 in hash stats (It was higher in
the beginning - 19, but got to 13 after 2 hours)?

Michal

On Wed, Dec 9, 2015 at 11:19 AM, Michal Marciniszyn <
michal.marciniszyn@gooddata.com> wrote:

> Hi,
>
> regarding the process, we see that in perf top. Under heavier load we see
> following
> 21.53% [kernel] [k] avtab_search_node
>
> sometimes even with higher percentage.
>
> --blesk
>
> On Wed, Dec 9, 2015 at 11:07 AM, Milos Malik <mmalik@redhat.com> wrote:
>
>> Hi Michal,
>>
>> which process (from the "top -d1" output) is consuming almost 30% of CPU?
>> Is it setroubleshootd or auditd or sedispatch or kernel? Thanks for the
>> answer.
>>
>> Milos Malik
>> SELinux QE person
>> BaseOS QE Security team
>> Red Hat Czech
>>
>> ----- Original Message -----
>> > Hello,
>> >
>> > we are heavy SELinux shop and we recently run into AVC related
>> performance
>> > issue. I was trying to find an answer on freenode IRC chat but I was
>> sent
>> > here by multiple guys. We're running on Scientific Linux 6.6 (upgrade
>> to 6.7
>> > ongoing) and we see this on some of our nodes:
>> >
>> > # cat /selinux/avc/cache_stats
>> > lookups hits misses allocations reclaims frees
>> > 3976846641 3626568307 350278334 350303465 344833264 346344169
>> > 3474274460 3092218096 382056364 382081270 381170512 382671551
>> > 2037181411 1655679702 381501709 381527148 380680320 382162477
>> > 1943162363 1651603455 291558908 291584892 288099840 289631602
>> > 829213467 406079951 423133516 423158604 422311024 423847681
>> > 1963015875 1555848944 407166931 407192104 406718592 408227742
>> > 3490131033 3117047653 373083380 373108386 372270880 373862706
>> > 940880689 549698684 391182005 391207388 390339328 391888374
>> > 4098417807 3712068859 386348948 386373592 385604096 387172806
>> > 3931378773 3549502965 381875808 381901074 381059904 382628308
>> >
>> > Also we see
>> >
>> > # cat /selinux/avc/hash_stats
>> > entries: 499
>> > buckets used: 257/512
>> > longest chain: 6
>> >
>> > Some times under load we see SELinux consuming about 30% of CPU time.
>> There
>> > is about 16% of cache misses on these nodes (and sometimes it goes as
>> high
>> > as 30%). The lates article about the issue is from RHEL 5 times -
>> >
>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/rhlcommon-section-0102.html
>> > . We do not feel this to be too relevant in this case.
>> >
>> > Are there any recommendations on cache sizing for SELinux? We can resize
>> > cache to 1024 or 2048 entries, but would this help to resolve the issue?
>> >
>> > I'm attaching seinfo from node with our policy and then for comparison
>> from
>> > node without any policy.
>> >
>> > With policy:
>> > # seinfo
>> >
>> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
>> > Policy Version & Type: v.24 (binary, mls)
>> >
>> > Classes: 81 Permissions: 238
>> > Sensitivities: 1 Categories: 1024
>> > Types: 4273 Attributes: 295
>> > Users: 9 Roles: 12
>> > Booleans: 234 Cond. Expr.: 274
>> > Allow: 352554 Neverallow: 0
>> > Auditallow: 140 Dontaudit: 321786
>> > Type_trans: 42813 Type_change: 38
>> > Type_member: 48 Role allow: 19
>> > Role_trans: 409 Range_trans: 6421
>> > Constraints: 90 Validatetrans: 0
>> > Initial SIDs: 27 Fs_use: 23
>> > Genfscon: 84 Portcon: 505
>> > Netifcon: 0 Nodecon: 0
>> > Permissives: 91 Polcap: 2
>> >
>> >
>> >
>> > Without policy:
>> >
>> > seinfo
>> >
>> > Statistics for policy file: /etc/selinux/targeted/policy/policy.24
>> > Policy Version & Type: v.24 (binary, mls)
>> >
>> > Classes: 81 Permissions: 238
>> > Sensitivities: 1 Categories: 1024
>> > Types: 3926 Attributes: 295
>> > Users: 9 Roles: 12
>> > Booleans: 234 Cond. Expr.: 274
>> > Allow: 320969 Neverallow: 0
>> > Auditallow: 140 Dontaudit: 273256
>> > Type_trans: 41915 Type_change: 38
>> > Type_member: 48 Role allow: 19
>> > Role_trans: 386 Range_trans: 6069
>> > Constraints: 90 Validatetrans: 0
>> > Initial SIDs: 27 Fs_use: 23
>> > Genfscon: 84 Portcon: 479
>> > Netifcon: 0 Nodecon: 0
>> > Permissives: 91 Polcap: 2
>> >
>> >
>> > Any help or guidance would be very much appreciated, if there is more
>> > in-depth info needed I'll be more than happy to provide it.
>> >
>> > Yours sincerely,
>> >
>> > Michal Marciniszyn
>> > Manager - SW Engineering
>> > GoodData
>> >
>> > _______________________________________________
>> > Selinux mailing list
>> > Selinux@tycho.nsa.gov
>> > To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
>> > To get help, send an email containing "help" to
>> > Selinux-request@tycho.nsa.gov.
>>
>
>

[-- Attachment #2: Type: text/html, Size: 8182 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-09 13:15     ` Michal Marciniszyn
@ 2015-12-09 15:05       ` Stephen Smalley
  2015-12-09 16:07         ` Joe Nall
  0 siblings, 1 reply; 17+ messages in thread
From: Stephen Smalley @ 2015-12-09 15:05 UTC (permalink / raw)
  To: Michal Marciniszyn, Milos Malik; +Cc: selinux

On 12/09/2015 08:15 AM, Michal Marciniszyn wrote:
> Hi,
>
> after increasing the cache, I do not see many reclaims, like couple of
> them here and there. The cache size had to be increased to 2048 to get
> ti this state.
>
> # avcstat 15
>
>      537645     537623         22         22         32         32
>      942916     942912          4          4          0          0
>      604466     604457          9          9          0          0
>      451737     451730          7          7         16         16
>      457669     457669          0          0          0          0
>      519135     519133          2          2          0          0
>      517288     517288          0          0          0          0
>      380376     380376          0          0          0          0
>      464272     464269          3          3          0          0
>      531484     531482          2          2          0          0
>     1422422    1422421          1          1          0          0
>     1380932    1380932          0          0          0          0
>      512999     512999          0          0          0          0
>
>
> Is it ok if I get longest chain length 13 in hash stats (It was higher
> in the beginning - 19, but got to 13 after 2 hours)?

I wouldn't worry about that; it's insignificant compared to the cost of 
an AVC miss.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-09 15:05       ` Stephen Smalley
@ 2015-12-09 16:07         ` Joe Nall
  2015-12-09 17:07           ` Stephen Smalley
  0 siblings, 1 reply; 17+ messages in thread
From: Joe Nall @ 2015-12-09 16:07 UTC (permalink / raw)
  To: Stephen Smalley; +Cc: Michal Marciniszyn, Milos Malik, selinux

This thread motivated me to look at some test boxes. One is seeing about 2k misses per second under high load. Raising the cache_threshold to 1024 lowered that to 600 misses per second and raising it to 2048 lowered it to 0 with occasional bounces to 20-50.

Are there any negatives to raising the cache_threshold?

What is the approximate cost of a miss?

Is there a persistent mechanism to set the cache_threshold? The system is RHEL 6.6 with custom MLS policy.

joe


> On Dec 9, 2015, at 9:05 AM, Stephen Smalley <sds@tycho.nsa.gov> wrote:
> 
> On 12/09/2015 08:15 AM, Michal Marciniszyn wrote:
>> Hi,
>> 
>> after increasing the cache, I do not see many reclaims, like couple of
>> them here and there. The cache size had to be increased to 2048 to get
>> ti this state.
>> 
>> # avcstat 15
>> 
>>     537645     537623         22         22         32         32
>>     942916     942912          4          4          0          0
>>     604466     604457          9          9          0          0
>>     451737     451730          7          7         16         16
>>     457669     457669          0          0          0          0
>>     519135     519133          2          2          0          0
>>     517288     517288          0          0          0          0
>>     380376     380376          0          0          0          0
>>     464272     464269          3          3          0          0
>>     531484     531482          2          2          0          0
>>    1422422    1422421          1          1          0          0
>>    1380932    1380932          0          0          0          0
>>     512999     512999          0          0          0          0
>> 
>> 
>> Is it ok if I get longest chain length 13 in hash stats (It was higher
>> in the beginning - 19, but got to 13 after 2 hours)?
> 
> I wouldn't worry about that; it's insignificant compared to the cost of an AVC miss.
> 
> 
> 
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: Performance issues - huge amount of AVC misses
  2015-12-09 16:07         ` Joe Nall
@ 2015-12-09 17:07           ` Stephen Smalley
  0 siblings, 0 replies; 17+ messages in thread
From: Stephen Smalley @ 2015-12-09 17:07 UTC (permalink / raw)
  To: Joe Nall; +Cc: Michal Marciniszyn, Milos Malik, selinux

On 12/09/2015 11:07 AM, Joe Nall wrote:
> This thread motivated me to look at some test boxes. One is seeing about 2k misses per second under high load. Raising the cache_threshold to 1024 lowered that to 600 misses per second and raising it to 2048 lowered it to 0 with occasional bounces to 20-50.
>
> Are there any negatives to raising the cache_threshold?

Could waste memory and degrade the AVC hash chain lengths, but worth it 
if it makes AVC misses rare.

> What is the approximate cost of a miss?

On a miss, you're talking about a full security server access vector 
computation.  Cost will depend on your policy (number of rules, type 
attribute density, number and complexity of constraints) but with the 
SL6 policy stats he was showing I imagine it is quite high.

> Is there a persistent mechanism to set the cache_threshold? The system is RHEL 6.6 with custom MLS policy.

Not without patching your kernel.
Just write the value to selinuxfs from an init script or set it via 
tmpfiles.d if using systemd.

^ permalink raw reply	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2015-12-09 17:07 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-08 10:25 Performance issues - huge amount of AVC misses Michal Marciniszyn
2015-12-08 10:44 ` Dominick Grift
2015-12-08 14:56   ` Michal Marciniszyn
2015-12-08 15:05     ` Daniel J Walsh
2015-12-08 15:10     ` Dominick Grift
2015-12-08 15:35     ` Stephen Smalley
2015-12-08 16:21       ` Michal Marciniszyn
2015-12-08 16:29         ` Dominick Grift
2015-12-08 17:06         ` Stephen Smalley
2015-12-08 15:29 ` Stephen Smalley
2015-12-08 16:16   ` Michal Marciniszyn
2015-12-09 10:07 ` Milos Malik
2015-12-09 10:19   ` Michal Marciniszyn
2015-12-09 13:15     ` Michal Marciniszyn
2015-12-09 15:05       ` Stephen Smalley
2015-12-09 16:07         ` Joe Nall
2015-12-09 17:07           ` Stephen Smalley

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.