[RFC] Source Policy, CIL, and High Level Languages

[RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

In January, we sent an RFC [1] to update userspace to integrate CIL
[2] and source policy. And in April, we sent an updated RFC [3] which
added support for high level languages and a tool to convert policy
package (pp) files to CIL. After getting some good feedback, we have
made some more changes, mostly to maintain ABI compatibility. The
major changes made since the last patchset are:

- Change how semanage_set_root was re-added to use the source policy
  infrastructure. Fixes were made so that semanage.conf was looked for
  inside the root. Also adds an semanage_root() function to get the
  current root.
- In previous patchsets, the semanage_module_upgrade* and
  semanage_module_install_base* functions were removed from the API,
  and semanage_module_install* had modified parameters. However, these
  changes broke the API and ABI. To maintain ABI compatibility, we've
  now added symbolic versioning to support the old version of the
  functions, which now just call the new install functions. semodule
  is updated to support --base and --upgrade, but with the addition of
  a deprecation message. API compatability is not maintained.
- Likewise, symbolic versioning was added to support the old module
  enable/disable functions, which call the new enable/disable
  functions.
- Modify the libsepol Makefile to now make including CIL optional via
  the DISABLE_CIL build flag. This only affects libsepol (not
  libsemanage), primarily so that SE for Android does not need to
  include unused CIL cruft.

With these changes, ABI compatibility is maintained. Additionally, we
have tested these changes with the userspace tests and against the
kernel test suite, and no new failures were discovered. We have
also tested this patchset with both Fedora 20 policy and with reference
policy and found no errors.

Because of the size of the patchset (67 file changes, ~8300
insertions, ~1800 deletions), all the changes have been pushed to the
selinux git repository to the 'integration' branch for
comments/review. Unlike the previous RFCs, for simplicity there is now
only a single branch, containing three types of changes:

Reverts
   Reverts changes made to master that conflict with the new source
   policy infrastructure (e.g. how paths are handled,
   enabled/disable modules). Rather than dealing with a large amount
   of conflicts with the source policy work, it was easier to just
   remove the commits that added conflicting features, rebase the old
   source policy work on top of that, and add back any features in a
   manner consistent with source policy. The only conflicts were
   related to enabling/disabling of modules, and semanage_set_root.

Source Policy
   This is a rebase of the old src-policy branch on top of the
   reverted commits.  The goal of these changes is to improve the API
   for module handling, add support for source policies, module
   priorities, enabling/disabling of modules, and moving the policy
   store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.

CIL Integration
   These changes build CIL into libsepol, and updates libsepol,
   libsemanage, semodule, and semanage to work with and understand CIL
   files and manage /var/lib/selinux and /etc/selinux. Switching to
   CIL has a few side effects, such as removing base modules,
   versions, and upgrades.

   This also adds a new tool (installed to
   /usr/libexec/selinux/hll/pp), which is an HLL compiler that
   converts binary pp modules to CIL. The infrastructure to use this
   compiler (or any other HLL compiler) was added to compile HLL
   modules to CIL, which is accomplished by writing the HLL data to
   the stdin of the compiler and reading the equivilent CIL from
   stdout. The resulting CIL is then cached in the policy store so
   this compilation does not need to take place during future store
   updates. Cached CIL modules can be ignored using a new semodule
   flag (-C/--ignore-cache) or a new configuration option in
   semanage.conf (ignore-cache). Other configuration options were
   added to semanage.conf to manage the path to HLL compilers
   (compiler-directory) and the policy store (store-root). Semodule
   was also modified to support changing the policy store with the
   -S/--store-root option.

   Lastly, the CIL integration changes required changes to the API,
   but symbolic versioning was used to maintain ABI compatibility.
   Because of this, the .so version is no longer incremented like in
   the previous version of this RFC.

With these changes, it is possible to build and manage SELinux
policy using pp and CIL modules and the familiar semodule/semanage
tools.

To make this easier to experiment with and test, below are the steps
needed to install the updated userspace and migrate a minimal Fedora 20
installation to the new policy store.

Thanks, and we look forward to any questions/comments.

- Steve

[1] http://marc.info/?l=selinux&m=138921403805934&w=2
[2] https://github.com/SELinuxProject/cil/wiki
[3] http://marc.info/?l=selinux&m=139878606630921&w=2


Steps to Install SELinux Userspace with source policy, CIL, and HLL

# Start with a fresh Fedora 20-x86_64 Mimimal Installation

# Install SELinux userspace dependencies
$ yum install audit-libs-devel bison bzip2-devel dbus-devel
dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
setools-devel swig ustr-devel

# Update to the latest targeted policy
$ yum update selinux-policy-targeted

# Clone the repos and checkout branches
$ git clone -b integration https://github.com/SELinuxProject/selinux.git
$ git clone -b master https://github.com/SELinuxProject/cil.git

# Create a symlink to the cil repo so CIL can be built into libsepol
$ ln -s ~/cil/ selinux/libsepol/cil

# Install SELinux userspace with CIL integration and HLL support
$ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap

# Migrate to the new source policy infrastructure
$ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py

# List the installed modules, showing priority and HLL
$ semodule --list=full

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
> In January, we sent an RFC [1] to update userspace to integrate CIL
> [2] and source policy. And in April, we sent an updated RFC [3] which
> added support for high level languages and a tool to convert policy
> package (pp) files to CIL. After getting some good feedback, we have
> made some more changes, mostly to maintain ABI compatibility. The
> major changes made since the last patchset are:

<snip>

I just spent a few hours playing with this and i am impressed.

Everything i tested just works.

What did i test?

1. disabling/enabling existing modules
2. toggling booleans with semanage
3. adding and removing port and file contexts with semanage
4. adding/removing a policy module with semodule, checkmodule,
semodule_package
5. adding/removing a (cil) policy module with semodule
6. associating a (new) user with staff_t identity

Comments?

if i do restorecon -R -v -F /home it resets contexts *every* time (from
s0 to s0-s0). No noticable side effects because of this

After associating user john with staff_u, johns home directory is
properly labeled (staff_u associated with /home/john). However, what is
strange here is that i cannot see staff_u home dir context specs
in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs

Am i looking in the wrong place? How does SELinux know that staff_u
needs to be associated with /home/john

When you remove a custom module (semodule -r mycustmodule) semodule is a
little verbose. (one line gets printed)

Other than that it looks flawless. Ofcourse i only tested it for a few
hours but on the surface everything looks ok

I recorded the whole testing session for reference and submitted the
video to youtube under the name of cil testday

Thanks

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 02:51 AM, Dominick Grift wrote:
> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
> 
> <snip>
> 
> I just spent a few hours playing with this and i am impressed.
> 
> Everything i tested just works.
> 
> What did i test?
> 
> 1. disabling/enabling existing modules
> 2. toggling booleans with semanage
> 3. adding and removing port and file contexts with semanage
> 4. adding/removing a policy module with semodule, checkmodule,
> semodule_package
> 5. adding/removing a (cil) policy module with semodule
> 6. associating a (new) user with staff_t identity
> 
> Comments?
> 
> if i do restorecon -R -v -F /home it resets contexts *every* time (from
> s0 to s0-s0). No noticable side effects because of this

I think this is related to how right now when generated file contexts,
CIL doesn't remove the high level if it is the same as the low level. We
just always include both levels separated by a hyphen. Shouldn't be too
hard to fix.

We'll fix it in CIL, though, this might also be considered a bug in
restorecon. It appears it doesn't doesn't realize that the u:r:t:s0 is
the same as u:r:t:s0-s0, so it relabels the file. And then when it does
relabel, the duplicate level isn't included.

> After associating user john with staff_u, johns home directory is
> properly labeled (staff_u associated with /home/john). However, what is
> strange here is that i cannot see staff_u home dir context specs
> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
> 
> Am i looking in the wrong place? How does SELinux know that staff_u
> needs to be associated with /home/john

Interesting. Looking into this now.

> When you remove a custom module (semodule -r mycustmodule) semodule is a
> little verbose. (one line gets printed)

The source policy work added a bit more logging output. This message is
kindof helpful in that it lets you know that a module with a lower
priority hasn't become active, and that this module is completely gone.
But perhaps that is assumed, and we should only output messages when
there are priority changes or something similar. We'll take a look at
the new messages and reevaluate their usefulness.

> Other than that it looks flawless. Ofcourse i only tested it for a few
> hours but on the surface everything looks ok
> 

Great to hear!

> I recorded the whole testing session for reference and submitted the
> video to youtube under the name of cil testday
> 
> Thanks
> 

Thanks so much!

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 02:51 AM, Dominick Grift wrote:
> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
> 
> <snip>
> 
> I just spent a few hours playing with this and i am impressed.
> 
> Everything i tested just works.
> 
> What did i test?
> 
> 1. disabling/enabling existing modules
> 2. toggling booleans with semanage
> 3. adding and removing port and file contexts with semanage
> 4. adding/removing a policy module with semodule, checkmodule,
> semodule_package
> 5. adding/removing a (cil) policy module with semodule
> 6. associating a (new) user with staff_t identity
> 
> Comments?
> 
> if i do restorecon -R -v -F /home it resets contexts *every* time (from
> s0 to s0-s0). No noticable side effects because of this
> 
> After associating user john with staff_u, johns home directory is
> properly labeled (staff_u associated with /home/john). However, what is
> strange here is that i cannot see staff_u home dir context specs
> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
> 
> Am i looking in the wrong place? How does SELinux know that staff_u
> needs to be associated with /home/john
> 
> When you remove a custom module (semodule -r mycustmodule) semodule is a
> little verbose. (one line gets printed)
> 
> Other than that it looks flawless. Ofcourse i only tested it for a few
> hours but on the surface everything looks ok
> 
> I recorded the whole testing session for reference and submitted the
> video to youtube under the name of cil testday

Thanks for testing it.  How did it look from a performance POV, wrt
memory use and runtime?

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:

<snip>

> Thanks for testing it.  How did it look from a performance POV, wrt
> memory use and runtime?
> 

I have not (yet) really focused on that but i suppose there was no real
noticeable slow down or speed up.

Any tips on how i could provide useful benchmarks?

I suppose i could enable the neverallow check
in /etc/selinux/semanage.conf and i would bet it is now much faster than
it used to be (in fact ill try that)

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
> 
> <snip>
> 
> > Thanks for testing it.  How did it look from a performance POV, wrt
> > memory use and runtime?
> > 
> 
> I have not (yet) really focused on that but i suppose there was no real
> noticeable slow down or speed up.
> 
> Any tips on how i could provide useful benchmarks?
> 
> I suppose i could enable the neverallow check
> in /etc/selinux/semanage.conf and i would bet it is now much faster than
> it used to be (in fact ill try that)
> 
> 

I suspect i was lying.

I am installing a guest with similar specs now and same software except
the cil mods and then do some comparison.

i suppose stuff like time semodule -B
and looking at top

I did do a semodule -B with checking for neverallow rules but that found
a violation really fast (thanks fedora). So although i cant really say
how much faster that is , it is pretty safe to assume its much faster
now

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 09:09 AM, Dominick Grift wrote:
> On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
>> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
>>
>> <snip>
>>
>>> Thanks for testing it.  How did it look from a performance POV, wrt
>>> memory use and runtime?
>>>
>>
>> I have not (yet) really focused on that but i suppose there was no real
>> noticeable slow down or speed up.
>>
>> Any tips on how i could provide useful benchmarks?
>>
>> I suppose i could enable the neverallow check
>> in /etc/selinux/semanage.conf and i would bet it is now much faster than
>> it used to be (in fact ill try that)
>>
>>
> 
> I suspect i was lying.
> 
> I am installing a guest with similar specs now and same software except
> the cil mods and then do some comparison.
> 
> i suppose stuff like time semodule -B
> and looking at top
> 
> I did do a semodule -B with checking for neverallow rules but that found
> a violation really fast (thanks fedora). So although i cant really say
> how much faster that is , it is pretty safe to assume its much faster
> now

/usr/bin/time setsebool -P httpd_can_network_connect=1
valgrind --tool=massif setsebool -P httpd_can_network_connect=1
ms_print massif.out.<pid>

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 09:12 -0400, Stephen Smalley wrote:
> On 07/10/2014 09:09 AM, Dominick Grift wrote:
> > On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
> >> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
> >>
> >> <snip>
> >>
> >>> Thanks for testing it.  How did it look from a performance POV, wrt
> >>> memory use and runtime?
> >>>
> >>
> >> I have not (yet) really focused on that but i suppose there was no real
> >> noticeable slow down or speed up.
> >>
> >> Any tips on how i could provide useful benchmarks?
> >>
> >> I suppose i could enable the neverallow check
> >> in /etc/selinux/semanage.conf and i would bet it is now much faster than
> >> it used to be (in fact ill try that)
> >>
> >>
> > 
> > I suspect i was lying.
> > 
> > I am installing a guest with similar specs now and same software except
> > the cil mods and then do some comparison.
> > 
> > i suppose stuff like time semodule -B
> > and looking at top
> > 
> > I did do a semodule -B with checking for neverallow rules but that found
> > a violation really fast (thanks fedora). So although i cant really say
> > how much faster that is , it is pretty safe to assume its much faster
> > now
> 
> /usr/bin/time setsebool -P httpd_can_network_connect=1
> valgrind --tool=massif setsebool -P httpd_can_network_connect=1
> ms_print massif.out.<pid>
> 
> 
> 

Will do that next.

I did a time semodule -B on similar configs (2 cores/2GB ram):

Result: cil seems faster but seems to take more memory:

CIL: real 0m13.XXXs (23% mem (of 2 GB)
REGULAR: real 0m21.XXXs (15% mem (of 2 GB)

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 09:26 AM, Dominick Grift wrote:
> On Thu, 2014-07-10 at 09:12 -0400, Stephen Smalley wrote:
>> On 07/10/2014 09:09 AM, Dominick Grift wrote:
>>> On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
>>>> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
>>>>
>>>> <snip>
>>>>
>>>>> Thanks for testing it.  How did it look from a performance POV, wrt
>>>>> memory use and runtime?
>>>>>
>>>>
>>>> I have not (yet) really focused on that but i suppose there was no real
>>>> noticeable slow down or speed up.
>>>>
>>>> Any tips on how i could provide useful benchmarks?
>>>>
>>>> I suppose i could enable the neverallow check
>>>> in /etc/selinux/semanage.conf and i would bet it is now much faster than
>>>> it used to be (in fact ill try that)
>>>>
>>>>
>>>
>>> I suspect i was lying.
>>>
>>> I am installing a guest with similar specs now and same software except
>>> the cil mods and then do some comparison.
>>>
>>> i suppose stuff like time semodule -B
>>> and looking at top
>>>
>>> I did do a semodule -B with checking for neverallow rules but that found
>>> a violation really fast (thanks fedora). So although i cant really say
>>> how much faster that is , it is pretty safe to assume its much faster
>>> now
>>
>> /usr/bin/time setsebool -P httpd_can_network_connect=1
>> valgrind --tool=massif setsebool -P httpd_can_network_connect=1
>> ms_print massif.out.<pid>
>>
>>
>>
> 
> Will do that next.
> 
> I did a time semodule -B on similar configs (2 cores/2GB ram):
> 
> Result: cil seems faster but seems to take more memory:
> 
> CIL: real 0m13.XXXs (23% mem (of 2 GB)
> REGULAR: real 0m21.XXXs (15% mem (of 2 GB)

So, that's a concern, as we already have various bug reports on semodule
and setsebool being killed by the OOM killer, e.g.
https://bugzilla.redhat.com/show_bug.cgi?id=1098446

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 09:38 -0400, Stephen Smalley wrote:
> On 07/10/2014 09:26 AM, Dominick Grift wrote:
> > On Thu, 2014-07-10 at 09:12 -0400, Stephen Smalley wrote:
> >> On 07/10/2014 09:09 AM, Dominick Grift wrote:
> >>> On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
> >>>> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
> >>>>
> >>>> <snip>
> >>>>
> >>>>> Thanks for testing it.  How did it look from a performance POV, wrt
> >>>>> memory use and runtime?
> >>>>>
> >>>>
> >>>> I have not (yet) really focused on that but i suppose there was no real
> >>>> noticeable slow down or speed up.
> >>>>
> >>>> Any tips on how i could provide useful benchmarks?
> >>>>
> >>>> I suppose i could enable the neverallow check
> >>>> in /etc/selinux/semanage.conf and i would bet it is now much faster than
> >>>> it used to be (in fact ill try that)
> >>>>
> >>>>
> >>>
> >>> I suspect i was lying.
> >>>
> >>> I am installing a guest with similar specs now and same software except
> >>> the cil mods and then do some comparison.
> >>>
> >>> i suppose stuff like time semodule -B
> >>> and looking at top
> >>>
> >>> I did do a semodule -B with checking for neverallow rules but that found
> >>> a violation really fast (thanks fedora). So although i cant really say
> >>> how much faster that is , it is pretty safe to assume its much faster
> >>> now
> >>
> >> /usr/bin/time setsebool -P httpd_can_network_connect=1
> >> valgrind --tool=massif setsebool -P httpd_can_network_connect=1
> >> ms_print massif.out.<pid>
> >>
> >>
> >>
> > 
> > Will do that next.
> > 
> > I did a time semodule -B on similar configs (2 cores/2GB ram):
> > 
> > Result: cil seems faster but seems to take more memory:
> > 
> > CIL: real 0m13.XXXs (23% mem (of 2 GB)
> > REGULAR: real 0m21.XXXs (15% mem (of 2 GB)
> 
> So, that's a concern, as we already have various bug reports on semodule
> and setsebool being killed by the OOM killer, e.g.
> https://bugzilla.redhat.com/show_bug.cgi?id=1098446
> 
> 
valgrind output:

http://paste.fedoraproject.org/116966/4999755/

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 09:45 AM, Dominick Grift wrote:
> On Thu, 2014-07-10 at 09:38 -0400, Stephen Smalley wrote:
>> On 07/10/2014 09:26 AM, Dominick Grift wrote:
>>> On Thu, 2014-07-10 at 09:12 -0400, Stephen Smalley wrote:
>>>> On 07/10/2014 09:09 AM, Dominick Grift wrote:
>>>>> On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
>>>>>> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
>>>>>>
>>>>>> <snip>
>>>>>>
>>>>>>> Thanks for testing it.  How did it look from a performance POV, wrt
>>>>>>> memory use and runtime?
>>>>>>>
>>>>>>
>>>>>> I have not (yet) really focused on that but i suppose there was no real
>>>>>> noticeable slow down or speed up.
>>>>>>
>>>>>> Any tips on how i could provide useful benchmarks?
>>>>>>
>>>>>> I suppose i could enable the neverallow check
>>>>>> in /etc/selinux/semanage.conf and i would bet it is now much faster than
>>>>>> it used to be (in fact ill try that)
>>>>>>
>>>>>>
>>>>>
>>>>> I suspect i was lying.
>>>>>
>>>>> I am installing a guest with similar specs now and same software except
>>>>> the cil mods and then do some comparison.
>>>>>
>>>>> i suppose stuff like time semodule -B
>>>>> and looking at top
>>>>>
>>>>> I did do a semodule -B with checking for neverallow rules but that found
>>>>> a violation really fast (thanks fedora). So although i cant really say
>>>>> how much faster that is , it is pretty safe to assume its much faster
>>>>> now
>>>>
>>>> /usr/bin/time setsebool -P httpd_can_network_connect=1
>>>> valgrind --tool=massif setsebool -P httpd_can_network_connect=1
>>>> ms_print massif.out.<pid>
>>>>
>>>>
>>>>
>>>
>>> Will do that next.
>>>
>>> I did a time semodule -B on similar configs (2 cores/2GB ram):
>>>
>>> Result: cil seems faster but seems to take more memory:
>>>
>>> CIL: real 0m13.XXXs (23% mem (of 2 GB)
>>> REGULAR: real 0m21.XXXs (15% mem (of 2 GB)
>>
>> So, that's a concern, as we already have various bug reports on semodule
>> and setsebool being killed by the OOM killer, e.g.
>> https://bugzilla.redhat.com/show_bug.cgi?id=1098446
>>
>>
> valgrind output:
> 
> http://paste.fedoraproject.org/116966/4999755/
> 

We just pushed a commit to CIL that greatly reduces peak memory usage.
Some quick testing brings the peak memory usage of compiling Fedora's
policy from around 460MB down to around 260MB. So I think its now about
on par with current userspace. We're also working on some other changes,
but those require a bit more work and so might take a little longer. But
I don't think those changes will get memory usage down significantly
more (maybe down another 10-20MB), so I doubt this will do a whole lot
in solving the OOM killer issue.

- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/11/2014 11:02 AM, Steve Lawrence wrote:
> On 07/10/2014 09:45 AM, Dominick Grift wrote:
>> On Thu, 2014-07-10 at 09:38 -0400, Stephen Smalley wrote:
>>> On 07/10/2014 09:26 AM, Dominick Grift wrote:
>>>> On Thu, 2014-07-10 at 09:12 -0400, Stephen Smalley wrote:
>>>>> On 07/10/2014 09:09 AM, Dominick Grift wrote:
>>>>>> On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
>>>>>>> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
>>>>>>>
>>>>>>> <snip>
>>>>>>>
>>>>>>>> Thanks for testing it.  How did it look from a performance POV, wrt
>>>>>>>> memory use and runtime?
>>>>>>>>
>>>>>>>
>>>>>>> I have not (yet) really focused on that but i suppose there was no real
>>>>>>> noticeable slow down or speed up.
>>>>>>>
>>>>>>> Any tips on how i could provide useful benchmarks?
>>>>>>>
>>>>>>> I suppose i could enable the neverallow check
>>>>>>> in /etc/selinux/semanage.conf and i would bet it is now much faster than
>>>>>>> it used to be (in fact ill try that)
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I suspect i was lying.
>>>>>>
>>>>>> I am installing a guest with similar specs now and same software except
>>>>>> the cil mods and then do some comparison.
>>>>>>
>>>>>> i suppose stuff like time semodule -B
>>>>>> and looking at top
>>>>>>
>>>>>> I did do a semodule -B with checking for neverallow rules but that found
>>>>>> a violation really fast (thanks fedora). So although i cant really say
>>>>>> how much faster that is , it is pretty safe to assume its much faster
>>>>>> now
>>>>>
>>>>> /usr/bin/time setsebool -P httpd_can_network_connect=1
>>>>> valgrind --tool=massif setsebool -P httpd_can_network_connect=1
>>>>> ms_print massif.out.<pid>
>>>>>
>>>>>
>>>>>
>>>>
>>>> Will do that next.
>>>>
>>>> I did a time semodule -B on similar configs (2 cores/2GB ram):
>>>>
>>>> Result: cil seems faster but seems to take more memory:
>>>>
>>>> CIL: real 0m13.XXXs (23% mem (of 2 GB)
>>>> REGULAR: real 0m21.XXXs (15% mem (of 2 GB)
>>>
>>> So, that's a concern, as we already have various bug reports on semodule
>>> and setsebool being killed by the OOM killer, e.g.
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1098446
>>>
>>>
>> valgrind output:
>>
>> http://paste.fedoraproject.org/116966/4999755/
>>
> 
> We just pushed a commit to CIL that greatly reduces peak memory usage.
> Some quick testing brings the peak memory usage of compiling Fedora's
> policy from around 460MB down to around 260MB. So I think its now about
> on par with current userspace. We're also working on some other changes,
> but those require a bit more work and so might take a little longer. But
> I don't think those changes will get memory usage down significantly
> more (maybe down another 10-20MB), so I doubt this will do a whole lot
> in solving the OOM killer issue.
> 

We recently pushed the second group of changes to reduce peak memory
usage, and reduced it much more than we thought it would, reducing about
50MB. It now takes about 210MB to build current fedora policy, which is
quite a bit less than the 15% of 2GB (~300MB) that Dominick saw.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 09:26 AM, Dominick Grift wrote:
> On Thu, 2014-07-10 at 09:12 -0400, Stephen Smalley wrote:
>> On 07/10/2014 09:09 AM, Dominick Grift wrote:
>>> On Thu, 2014-07-10 at 14:52 +0200, Dominick Grift wrote:
>>>> On Thu, 2014-07-10 at 08:35 -0400, Stephen Smalley wrote:
>>>>
>>>> <snip>
>>>>
>>>>> Thanks for testing it.  How did it look from a performance POV, wrt
>>>>> memory use and runtime?
>>>>>
>>>>
>>>> I have not (yet) really focused on that but i suppose there was no real
>>>> noticeable slow down or speed up.
>>>>
>>>> Any tips on how i could provide useful benchmarks?
>>>>
>>>> I suppose i could enable the neverallow check
>>>> in /etc/selinux/semanage.conf and i would bet it is now much faster than
>>>> it used to be (in fact ill try that)
>>>>
>>>>
>>>
>>> I suspect i was lying.
>>>
>>> I am installing a guest with similar specs now and same software except
>>> the cil mods and then do some comparison.
>>>
>>> i suppose stuff like time semodule -B
>>> and looking at top
>>>
>>> I did do a semodule -B with checking for neverallow rules but that found
>>> a violation really fast (thanks fedora). So although i cant really say
>>> how much faster that is , it is pretty safe to assume its much faster
>>> now
>>
>> /usr/bin/time setsebool -P httpd_can_network_connect=1
>> valgrind --tool=massif setsebool -P httpd_can_network_connect=1
>> ms_print massif.out.<pid>
>>
>>
>>
> 
> Will do that next.
> 
> I did a time semodule -B on similar configs (2 cores/2GB ram):
> 
> Result: cil seems faster but seems to take more memory:
> 
> CIL: real 0m13.XXXs (23% mem (of 2 GB)
> REGULAR: real 0m21.XXXs (15% mem (of 2 GB)

For /usr/bin/time semodule -i test_policy.pp (from selinux-testsuite), I
see:
#integration (i.e. CIL): 6.24user 1.00system 0:07.30elapsed 99%CPU
(0avgtext+0avgdata 464480maxresident)k

#next (i.e. regular/upstream): 24.31user 0.88system 0:25.37elapsed
99%CPU (0avgtext+0avgdata 300792maxresident)k

That seems like a significant speed-up but it is more expensive in
memory use.

This is with up-to-date Fedora 20 policy, no customization of
semanage.conf.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 02:51 AM, Dominick Grift wrote:
> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
> 
> <snip>
> 
> I just spent a few hours playing with this and i am impressed.
> 
> Everything i tested just works.
> 
> What did i test?
> 
> 1. disabling/enabling existing modules
> 2. toggling booleans with semanage
> 3. adding and removing port and file contexts with semanage
> 4. adding/removing a policy module with semodule, checkmodule,
> semodule_package
> 5. adding/removing a (cil) policy module with semodule
> 6. associating a (new) user with staff_t identity
> 
> Comments?
> 
> if i do restorecon -R -v -F /home it resets contexts *every* time (from
> s0 to s0-s0). No noticable side effects because of this
> 

We recently pushed a fix to CIL that fixes the issue with how CIL
generates file contexts. It now removes the high level if it is the same
as the low level.

- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/11/2014 01:20 PM, Steve Lawrence wrote:
> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>> added support for high level languages and a tool to convert policy
>>> package (pp) files to CIL. After getting some good feedback, we have
>>> made some more changes, mostly to maintain ABI compatibility. The
>>> major changes made since the last patchset are:
>>
>> <snip>
>>
>> I just spent a few hours playing with this and i am impressed.
>>
>> Everything i tested just works.
>>
>> What did i test?
>>
>> 1. disabling/enabling existing modules
>> 2. toggling booleans with semanage
>> 3. adding and removing port and file contexts with semanage
>> 4. adding/removing a policy module with semodule, checkmodule,
>> semodule_package
>> 5. adding/removing a (cil) policy module with semodule
>> 6. associating a (new) user with staff_t identity
>>
>> Comments?
>>
>> if i do restorecon -R -v -F /home it resets contexts *every* time (from
>> s0 to s0-s0). No noticable side effects because of this
>>
> 
> We recently pushed a fix to CIL that fixes the issue with how CIL
> generates file contexts. It now removes the high level if it is the same
> as the low level.

So, if I revert my system to stock F20 (yum reinstall checkpolicy*
libsepol* libsemanage* libselinux* policycoreutils*
selinux-policy-targeted) , then re-install from the integration branch
as per your instructions and run the migration script, then attempt to
ssh into the system, sshd says "Unable to get valid context for sds" and
the connection is closed.  dmesg shows:
systemd[1]: SELinux policy denies access.

Can you merge #next to #integration so we get the more detailed
information on unknown classes/perms?

I'm guessing a reboot will clear the problem again (since systemd will
then remap the class from name to value at boot against the current
policy).  But ideally this wouldn't be necessary.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/14/2014 12:48 PM, Stephen Smalley wrote:
> On 07/11/2014 01:20 PM, Steve Lawrence wrote:
>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>> added support for high level languages and a tool to convert policy
>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>> major changes made since the last patchset are:
>>>
>>> <snip>
>>>
>>> I just spent a few hours playing with this and i am impressed.
>>>
>>> Everything i tested just works.
>>>
>>> What did i test?
>>>
>>> 1. disabling/enabling existing modules
>>> 2. toggling booleans with semanage
>>> 3. adding and removing port and file contexts with semanage
>>> 4. adding/removing a policy module with semodule, checkmodule,
>>> semodule_package
>>> 5. adding/removing a (cil) policy module with semodule
>>> 6. associating a (new) user with staff_t identity
>>>
>>> Comments?
>>>
>>> if i do restorecon -R -v -F /home it resets contexts *every* time (from
>>> s0 to s0-s0). No noticable side effects because of this
>>>
>>
>> We recently pushed a fix to CIL that fixes the issue with how CIL
>> generates file contexts. It now removes the high level if it is the same
>> as the low level.
> 
> So, if I revert my system to stock F20 (yum reinstall checkpolicy*
> libsepol* libsemanage* libselinux* policycoreutils*
> selinux-policy-targeted) , then re-install from the integration branch
> as per your instructions and run the migration script, then attempt to
> ssh into the system, sshd says "Unable to get valid context for sds" and
> the connection is closed.  dmesg shows:
> systemd[1]: SELinux policy denies access.
> 
> Can you merge #next to #integration so we get the more detailed
> information on unknown classes/perms?
> 
> I'm guessing a reboot will clear the problem again (since systemd will
> then remap the class from name to value at boot against the current
> policy).  But ideally this wouldn't be necessary.

Hmmm....a reboot did not clear it.  All logins, local or remote,
disabled for non-root.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/14/2014 12:53 PM, Stephen Smalley wrote:
> On 07/14/2014 12:48 PM, Stephen Smalley wrote:
>> On 07/11/2014 01:20 PM, Steve Lawrence wrote:
>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>> added support for high level languages and a tool to convert policy
>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>> major changes made since the last patchset are:
>>>>
>>>> <snip>
>>>>
>>>> I just spent a few hours playing with this and i am impressed.
>>>>
>>>> Everything i tested just works.
>>>>
>>>> What did i test?
>>>>
>>>> 1. disabling/enabling existing modules
>>>> 2. toggling booleans with semanage
>>>> 3. adding and removing port and file contexts with semanage
>>>> 4. adding/removing a policy module with semodule, checkmodule,
>>>> semodule_package
>>>> 5. adding/removing a (cil) policy module with semodule
>>>> 6. associating a (new) user with staff_t identity
>>>>
>>>> Comments?
>>>>
>>>> if i do restorecon -R -v -F /home it resets contexts *every* time (from
>>>> s0 to s0-s0). No noticable side effects because of this
>>>>
>>>
>>> We recently pushed a fix to CIL that fixes the issue with how CIL
>>> generates file contexts. It now removes the high level if it is the same
>>> as the low level.
>>
>> So, if I revert my system to stock F20 (yum reinstall checkpolicy*
>> libsepol* libsemanage* libselinux* policycoreutils*
>> selinux-policy-targeted) , then re-install from the integration branch
>> as per your instructions and run the migration script, then attempt to
>> ssh into the system, sshd says "Unable to get valid context for sds" and
>> the connection is closed.  dmesg shows:
>> systemd[1]: SELinux policy denies access.
>>
>> Can you merge #next to #integration so we get the more detailed
>> information on unknown classes/perms?
>>
>> I'm guessing a reboot will clear the problem again (since systemd will
>> then remap the class from name to value at boot against the current
>> policy).  But ideally this wouldn't be necessary.
> 
> Hmmm....a reboot did not clear it.  All logins, local or remote,
> disabled for non-root.

This seems to be a labeling problem; saw denials on attempts to read
seusers with semanage_store_t.  But restorecon -Rv /etc/selinux/targeted
restored it to selinux_config_t.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/14/2014 01:08 PM, Stephen Smalley wrote:
> On 07/14/2014 12:53 PM, Stephen Smalley wrote:
>> On 07/14/2014 12:48 PM, Stephen Smalley wrote:
>>> On 07/11/2014 01:20 PM, Steve Lawrence wrote:
>>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>>> added support for high level languages and a tool to convert policy
>>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>>> major changes made since the last patchset are:
>>>>>
>>>>> <snip>
>>>>>
>>>>> I just spent a few hours playing with this and i am impressed.
>>>>>
>>>>> Everything i tested just works.
>>>>>
>>>>> What did i test?
>>>>>
>>>>> 1. disabling/enabling existing modules
>>>>> 2. toggling booleans with semanage
>>>>> 3. adding and removing port and file contexts with semanage
>>>>> 4. adding/removing a policy module with semodule, checkmodule,
>>>>> semodule_package
>>>>> 5. adding/removing a (cil) policy module with semodule
>>>>> 6. associating a (new) user with staff_t identity
>>>>>
>>>>> Comments?
>>>>>
>>>>> if i do restorecon -R -v -F /home it resets contexts *every* time (from
>>>>> s0 to s0-s0). No noticable side effects because of this
>>>>>
>>>>
>>>> We recently pushed a fix to CIL that fixes the issue with how CIL
>>>> generates file contexts. It now removes the high level if it is the same
>>>> as the low level.
>>>
>>> So, if I revert my system to stock F20 (yum reinstall checkpolicy*
>>> libsepol* libsemanage* libselinux* policycoreutils*
>>> selinux-policy-targeted) , then re-install from the integration branch
>>> as per your instructions and run the migration script, then attempt to
>>> ssh into the system, sshd says "Unable to get valid context for sds" and
>>> the connection is closed.  dmesg shows:
>>> systemd[1]: SELinux policy denies access.
>>>
>>> Can you merge #next to #integration so we get the more detailed
>>> information on unknown classes/perms?
>>>
>>> I'm guessing a reboot will clear the problem again (since systemd will
>>> then remap the class from name to value at boot against the current
>>> policy).  But ideally this wouldn't be necessary.
>>
>> Hmmm....a reboot did not clear it.  All logins, local or remote,
>> disabled for non-root.
> 
> This seems to be a labeling problem; saw denials on attempts to read
> seusers with semanage_store_t.  But restorecon -Rv /etc/selinux/targeted
> restored it to selinux_config_t.
> 

Ah, interesting. We saw that problem a long time ago, but couldn't
reproduce it and it disappeared. Though I'm still unable to reproduce it
following your steps. I can still login and seusers is labeled
selinux_config_t. I'll keep looking into this.

I've also rebased/pushed #integration onto #next.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/14/2014 01:12 PM, Steve Lawrence wrote:
> Ah, interesting. We saw that problem a long time ago, but couldn't
> reproduce it and it disappeared. Though I'm still unable to reproduce it
> following your steps. I can still login and seusers is labeled
> selinux_config_t. I'll keep looking into this.
> 
> I've also rebased/pushed #integration onto #next.

# Revert to stock F20 SELinux userspace and policy.
yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
policycoreutils* selinux-policy-targeted

# Clear prior source/CIL policy store.
rm -rf /var/lib/selinux

# Reboot to ensure systemd and friends are using the new policy.
reboot

# Reset selinux and cil to latest sources
cd selinux
git clean -fdx
git fetch origin
git reset --hard origin/integration
cd ../cil
git clean -fdx
git fetch origin
git reset --hard origin/master

# Build and install new userspace
cd ..
ln -sf ../../cil selinux/libsepol/cil
make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
relabel

# Convert
 ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py

Try to login on console or via ssh:  Unable to get valid context for sds.

dmesg | grep systemd
[  343.739985] systemd[1]: SELinux policy denies access.
[  348.256030] systemd[1]: SELinux policy denies access.
[  376.335248] systemd[1]: SELinux policy denies access.
[  376.515343] systemd[1]: SELinux policy denies access.

restorecon -R /etc/selinux/targeted

Try to login again, hangs for a long time before finally succeeding.

reboot

Everything is happy.

100% reproducible, every time.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/14/2014 01:49 PM, Stephen Smalley wrote:
> On 07/14/2014 01:12 PM, Steve Lawrence wrote:
>> Ah, interesting. We saw that problem a long time ago, but couldn't
>> reproduce it and it disappeared. Though I'm still unable to reproduce it
>> following your steps. I can still login and seusers is labeled
>> selinux_config_t. I'll keep looking into this.
>>
>> I've also rebased/pushed #integration onto #next.
> 
> # Revert to stock F20 SELinux userspace and policy.
> yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
> policycoreutils* selinux-policy-targeted
> 
> # Clear prior source/CIL policy store.
> rm -rf /var/lib/selinux
> 
> # Reboot to ensure systemd and friends are using the new policy.
> reboot
> 
> # Reset selinux and cil to latest sources
> cd selinux
> git clean -fdx
> git fetch origin
> git reset --hard origin/integration
> cd ../cil
> git clean -fdx
> git fetch origin
> git reset --hard origin/master
> 
> # Build and install new userspace
> cd ..
> ln -sf ../../cil selinux/libsepol/cil
> make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
> relabel
> 
> # Convert
>  ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
> 
> Try to login on console or via ssh:  Unable to get valid context for sds.
> 
> dmesg | grep systemd
> [  343.739985] systemd[1]: SELinux policy denies access.
> [  348.256030] systemd[1]: SELinux policy denies access.
> [  376.335248] systemd[1]: SELinux policy denies access.
> [  376.515343] systemd[1]: SELinux policy denies access.
> 
> restorecon -R /etc/selinux/targeted
> 
> Try to login again, hangs for a long time before finally succeeding.
> 
> reboot
> 
> Everything is happy.
> 
> 100% reproducible, every time.
> 
> 

Thanks for the steps. I think I found what causes the labeling problem,
but I'm not yet sure why. It looks like it has something to do with how
the migration script rebuilds policy. If you run the migration script
with the --norebuild flag, and then run semodule -B, everything is
rebuilt and the files in /etc/selinux are labeled correctly.

I'm not yet convinced this is the same problem as the systemd issue (I
do get "unable to get valid context", but I still don't get "SELinux
policy denies access"). Let me know if you still see that after using
--norebuild/semodule -B.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/15/2014 03:56 PM, Steve Lawrence wrote:
> On 07/14/2014 01:49 PM, Stephen Smalley wrote:
>> On 07/14/2014 01:12 PM, Steve Lawrence wrote:
>>> Ah, interesting. We saw that problem a long time ago, but couldn't
>>> reproduce it and it disappeared. Though I'm still unable to reproduce it
>>> following your steps. I can still login and seusers is labeled
>>> selinux_config_t. I'll keep looking into this.
>>>
>>> I've also rebased/pushed #integration onto #next.
>>
>> # Revert to stock F20 SELinux userspace and policy.
>> yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>> policycoreutils* selinux-policy-targeted
>>
>> # Clear prior source/CIL policy store.
>> rm -rf /var/lib/selinux
>>
>> # Reboot to ensure systemd and friends are using the new policy.
>> reboot
>>
>> # Reset selinux and cil to latest sources
>> cd selinux
>> git clean -fdx
>> git fetch origin
>> git reset --hard origin/integration
>> cd ../cil
>> git clean -fdx
>> git fetch origin
>> git reset --hard origin/master
>>
>> # Build and install new userspace
>> cd ..
>> ln -sf ../../cil selinux/libsepol/cil
>> make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>> relabel
>>
>> # Convert
>>  ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>
>> Try to login on console or via ssh:  Unable to get valid context for sds.
>>
>> dmesg | grep systemd
>> [  343.739985] systemd[1]: SELinux policy denies access.
>> [  348.256030] systemd[1]: SELinux policy denies access.
>> [  376.335248] systemd[1]: SELinux policy denies access.
>> [  376.515343] systemd[1]: SELinux policy denies access.
>>
>> restorecon -R /etc/selinux/targeted
>>
>> Try to login again, hangs for a long time before finally succeeding.
>>
>> reboot
>>
>> Everything is happy.
>>
>> 100% reproducible, every time.
>>
>>
> 
> Thanks for the steps. I think I found what causes the labeling problem,
> but I'm not yet sure why. It looks like it has something to do with how
> the migration script rebuilds policy. If you run the migration script
> with the --norebuild flag, and then run semodule -B, everything is
> rebuilt and the files in /etc/selinux are labeled correctly.
> 
> I'm not yet convinced this is the same problem as the systemd issue (I
> do get "unable to get valid context", but I still don't get "SELinux
> policy denies access"). Let me know if you still see that after using
> --norebuild/semodule -B.

Ok, if I do:
./libsemanage/utils/semanage_migrate_etc_to_var.py -n
semodule -B

then /etc/selinux is labeled correctly and I can still login.
However, running su hangs and I get the systemd error message again.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/16/2014 10:16 AM, Stephen Smalley wrote:
> On 07/15/2014 03:56 PM, Steve Lawrence wrote:
>> On 07/14/2014 01:49 PM, Stephen Smalley wrote:
>>> On 07/14/2014 01:12 PM, Steve Lawrence wrote:
>>>> Ah, interesting. We saw that problem a long time ago, but couldn't
>>>> reproduce it and it disappeared. Though I'm still unable to reproduce it
>>>> following your steps. I can still login and seusers is labeled
>>>> selinux_config_t. I'll keep looking into this.
>>>>
>>>> I've also rebased/pushed #integration onto #next.
>>>
>>> # Revert to stock F20 SELinux userspace and policy.
>>> yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>>> policycoreutils* selinux-policy-targeted
>>>
>>> # Clear prior source/CIL policy store.
>>> rm -rf /var/lib/selinux
>>>
>>> # Reboot to ensure systemd and friends are using the new policy.
>>> reboot
>>>
>>> # Reset selinux and cil to latest sources
>>> cd selinux
>>> git clean -fdx
>>> git fetch origin
>>> git reset --hard origin/integration
>>> cd ../cil
>>> git clean -fdx
>>> git fetch origin
>>> git reset --hard origin/master
>>>
>>> # Build and install new userspace
>>> cd ..
>>> ln -sf ../../cil selinux/libsepol/cil
>>> make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>> relabel
>>>
>>> # Convert
>>>  ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>>
>>> Try to login on console or via ssh:  Unable to get valid context for sds.
>>>
>>> dmesg | grep systemd
>>> [  343.739985] systemd[1]: SELinux policy denies access.
>>> [  348.256030] systemd[1]: SELinux policy denies access.
>>> [  376.335248] systemd[1]: SELinux policy denies access.
>>> [  376.515343] systemd[1]: SELinux policy denies access.
>>>
>>> restorecon -R /etc/selinux/targeted
>>>
>>> Try to login again, hangs for a long time before finally succeeding.
>>>
>>> reboot
>>>
>>> Everything is happy.
>>>
>>> 100% reproducible, every time.
>>>
>>>
>>
>> Thanks for the steps. I think I found what causes the labeling problem,
>> but I'm not yet sure why. It looks like it has something to do with how
>> the migration script rebuilds policy. If you run the migration script
>> with the --norebuild flag, and then run semodule -B, everything is
>> rebuilt and the files in /etc/selinux are labeled correctly.
>>
>> I'm not yet convinced this is the same problem as the systemd issue (I
>> do get "unable to get valid context", but I still don't get "SELinux
>> policy denies access"). Let me know if you still see that after using
>> --norebuild/semodule -B.
> 
> Ok, if I do:
> ./libsemanage/utils/semanage_migrate_etc_to_var.py -n
> semodule -B
> 
> then /etc/selinux is labeled correctly and I can still login.
> However, running su hangs and I get the systemd error message again.

Also get this:
ERROR:dbus.proxies:Introspect error on :1.4:/org/freedesktop/login1:
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did
not receive a reply. Possible causes include: the remote application did
not send a reply, the message bus security policy blocked the reply, the
reply timeout expired, or the network connection was broken.

when trying to re-install the original selinux userspace packages.
With more errors from systemd in dmesg,
[14421.297045] systemd[1]: SELinux policy denies access.
[14451.711732] systemd[1]: SELinux policy denies access.
[14451.743787] systemd[1]: SELinux policy denies access.
[14512.551210] systemd[1]: SELinux policy denies access.
[14512.584126] systemd[1]: SELinux policy denies access.
[14543.063872] systemd[1]: SELinux policy denies access.
[14634.451680] systemd[1]: SELinux policy denies access.
[14634.469813] systemd[1]: SELinux policy denies access.
[14659.346867] systemd[1]: SELinux policy denies access.
[14659.665108] systemd[1]: SELinux policy denies access.
[14695.602174] systemd[1]: SELinux policy denies access.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/16/2014 10:21 AM, Stephen Smalley wrote:
> On 07/16/2014 10:16 AM, Stephen Smalley wrote:
>> On 07/15/2014 03:56 PM, Steve Lawrence wrote:
>>> On 07/14/2014 01:49 PM, Stephen Smalley wrote:
>>>> On 07/14/2014 01:12 PM, Steve Lawrence wrote:
>>>>> Ah, interesting. We saw that problem a long time ago, but couldn't
>>>>> reproduce it and it disappeared. Though I'm still unable to reproduce it
>>>>> following your steps. I can still login and seusers is labeled
>>>>> selinux_config_t. I'll keep looking into this.
>>>>>
>>>>> I've also rebased/pushed #integration onto #next.
>>>>
>>>> # Revert to stock F20 SELinux userspace and policy.
>>>> yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>>>> policycoreutils* selinux-policy-targeted
>>>>
>>>> # Clear prior source/CIL policy store.
>>>> rm -rf /var/lib/selinux
>>>>
>>>> # Reboot to ensure systemd and friends are using the new policy.
>>>> reboot
>>>>
>>>> # Reset selinux and cil to latest sources
>>>> cd selinux
>>>> git clean -fdx
>>>> git fetch origin
>>>> git reset --hard origin/integration
>>>> cd ../cil
>>>> git clean -fdx
>>>> git fetch origin
>>>> git reset --hard origin/master
>>>>
>>>> # Build and install new userspace
>>>> cd ..
>>>> ln -sf ../../cil selinux/libsepol/cil
>>>> make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>>> relabel
>>>>
>>>> # Convert
>>>>  ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>>>
>>>> Try to login on console or via ssh:  Unable to get valid context for sds.
>>>>
>>>> dmesg | grep systemd
>>>> [  343.739985] systemd[1]: SELinux policy denies access.
>>>> [  348.256030] systemd[1]: SELinux policy denies access.
>>>> [  376.335248] systemd[1]: SELinux policy denies access.
>>>> [  376.515343] systemd[1]: SELinux policy denies access.
>>>>
>>>> restorecon -R /etc/selinux/targeted
>>>>
>>>> Try to login again, hangs for a long time before finally succeeding.
>>>>
>>>> reboot
>>>>
>>>> Everything is happy.
>>>>
>>>> 100% reproducible, every time.
>>>>
>>>>
>>>
>>> Thanks for the steps. I think I found what causes the labeling problem,
>>> but I'm not yet sure why. It looks like it has something to do with how
>>> the migration script rebuilds policy. If you run the migration script
>>> with the --norebuild flag, and then run semodule -B, everything is
>>> rebuilt and the files in /etc/selinux are labeled correctly.
>>>
>>> I'm not yet convinced this is the same problem as the systemd issue (I
>>> do get "unable to get valid context", but I still don't get "SELinux
>>> policy denies access"). Let me know if you still see that after using
>>> --norebuild/semodule -B.
>>
>> Ok, if I do:
>> ./libsemanage/utils/semanage_migrate_etc_to_var.py -n
>> semodule -B
>>
>> then /etc/selinux is labeled correctly and I can still login.
>> However, running su hangs and I get the systemd error message again.
> 
> Also get this:
> ERROR:dbus.proxies:Introspect error on :1.4:/org/freedesktop/login1:
> dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did
> not receive a reply. Possible causes include: the remote application did
> not send a reply, the message bus security policy blocked the reply, the
> reply timeout expired, or the network connection was broken.
> 
> when trying to re-install the original selinux userspace packages.
> With more errors from systemd in dmesg,
> [14421.297045] systemd[1]: SELinux policy denies access.
> [14451.711732] systemd[1]: SELinux policy denies access.
> [14451.743787] systemd[1]: SELinux policy denies access.
> [14512.551210] systemd[1]: SELinux policy denies access.
> [14512.584126] systemd[1]: SELinux policy denies access.
> [14543.063872] systemd[1]: SELinux policy denies access.
> [14634.451680] systemd[1]: SELinux policy denies access.
> [14634.469813] systemd[1]: SELinux policy denies access.
> [14659.346867] systemd[1]: SELinux policy denies access.
> [14659.665108] systemd[1]: SELinux policy denies access.
> [14695.602174] systemd[1]: SELinux policy denies access.

journalctl -r shows an additional error message for each of these denials:
systemd-logind[913]: Failed to query ActiveState: Access denied

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/16/2014 10:26 AM, Stephen Smalley wrote:
> On 07/16/2014 10:21 AM, Stephen Smalley wrote:
>> On 07/16/2014 10:16 AM, Stephen Smalley wrote:
>>> On 07/15/2014 03:56 PM, Steve Lawrence wrote:
>>>> On 07/14/2014 01:49 PM, Stephen Smalley wrote:
>>>>> On 07/14/2014 01:12 PM, Steve Lawrence wrote:
>>>>>> Ah, interesting. We saw that problem a long time ago, but couldn't
>>>>>> reproduce it and it disappeared. Though I'm still unable to reproduce it
>>>>>> following your steps. I can still login and seusers is labeled
>>>>>> selinux_config_t. I'll keep looking into this.
>>>>>>
>>>>>> I've also rebased/pushed #integration onto #next.
>>>>>
>>>>> # Revert to stock F20 SELinux userspace and policy.
>>>>> yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>>>>> policycoreutils* selinux-policy-targeted
>>>>>
>>>>> # Clear prior source/CIL policy store.
>>>>> rm -rf /var/lib/selinux
>>>>>
>>>>> # Reboot to ensure systemd and friends are using the new policy.
>>>>> reboot
>>>>>
>>>>> # Reset selinux and cil to latest sources
>>>>> cd selinux
>>>>> git clean -fdx
>>>>> git fetch origin
>>>>> git reset --hard origin/integration
>>>>> cd ../cil
>>>>> git clean -fdx
>>>>> git fetch origin
>>>>> git reset --hard origin/master
>>>>>
>>>>> # Build and install new userspace
>>>>> cd ..
>>>>> ln -sf ../../cil selinux/libsepol/cil
>>>>> make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>>>> relabel
>>>>>
>>>>> # Convert
>>>>>  ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>>>>
>>>>> Try to login on console or via ssh:  Unable to get valid context for sds.
>>>>>
>>>>> dmesg | grep systemd
>>>>> [  343.739985] systemd[1]: SELinux policy denies access.
>>>>> [  348.256030] systemd[1]: SELinux policy denies access.
>>>>> [  376.335248] systemd[1]: SELinux policy denies access.
>>>>> [  376.515343] systemd[1]: SELinux policy denies access.
>>>>>
>>>>> restorecon -R /etc/selinux/targeted
>>>>>
>>>>> Try to login again, hangs for a long time before finally succeeding.
>>>>>
>>>>> reboot
>>>>>
>>>>> Everything is happy.
>>>>>
>>>>> 100% reproducible, every time.
>>>>>
>>>>>
>>>>
>>>> Thanks for the steps. I think I found what causes the labeling problem,
>>>> but I'm not yet sure why. It looks like it has something to do with how
>>>> the migration script rebuilds policy. If you run the migration script
>>>> with the --norebuild flag, and then run semodule -B, everything is
>>>> rebuilt and the files in /etc/selinux are labeled correctly.
>>>>
>>>> I'm not yet convinced this is the same problem as the systemd issue (I
>>>> do get "unable to get valid context", but I still don't get "SELinux
>>>> policy denies access"). Let me know if you still see that after using
>>>> --norebuild/semodule -B.
>>>
>>> Ok, if I do:
>>> ./libsemanage/utils/semanage_migrate_etc_to_var.py -n
>>> semodule -B
>>>
>>> then /etc/selinux is labeled correctly and I can still login.
>>> However, running su hangs and I get the systemd error message again.
>>
>> Also get this:
>> ERROR:dbus.proxies:Introspect error on :1.4:/org/freedesktop/login1:
>> dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did
>> not receive a reply. Possible causes include: the remote application did
>> not send a reply, the message bus security policy blocked the reply, the
>> reply timeout expired, or the network connection was broken.
>>
>> when trying to re-install the original selinux userspace packages.
>> With more errors from systemd in dmesg,
>> [14421.297045] systemd[1]: SELinux policy denies access.
>> [14451.711732] systemd[1]: SELinux policy denies access.
>> [14451.743787] systemd[1]: SELinux policy denies access.
>> [14512.551210] systemd[1]: SELinux policy denies access.
>> [14512.584126] systemd[1]: SELinux policy denies access.
>> [14543.063872] systemd[1]: SELinux policy denies access.
>> [14634.451680] systemd[1]: SELinux policy denies access.
>> [14634.469813] systemd[1]: SELinux policy denies access.
>> [14659.346867] systemd[1]: SELinux policy denies access.
>> [14659.665108] systemd[1]: SELinux policy denies access.
>> [14695.602174] systemd[1]: SELinux policy denies access.
> 
> journalctl -r shows an additional error message for each of these denials:
> systemd-logind[913]: Failed to query ActiveState: Access denied

ausearch -m USER_AVC reports:
type=USER_AVC msg=audit(1405520109.191:1031): pid=1 uid=0
auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
msg='avc:  denied  { status } for auid=-1 uid=-1 gid=-1
scontext=system_u:system_r:systemd_logind_t:s0
tcontext=system_u:system_r:init_t:s0 tclass=service
exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/16/2014 10:33 AM, Stephen Smalley wrote:
> On 07/16/2014 10:26 AM, Stephen Smalley wrote:
>> On 07/16/2014 10:21 AM, Stephen Smalley wrote:
>>> On 07/16/2014 10:16 AM, Stephen Smalley wrote:
>>>> On 07/15/2014 03:56 PM, Steve Lawrence wrote:
>>>>> On 07/14/2014 01:49 PM, Stephen Smalley wrote:
>>>>>> On 07/14/2014 01:12 PM, Steve Lawrence wrote:
>>>>>>> Ah, interesting. We saw that problem a long time ago, but couldn't
>>>>>>> reproduce it and it disappeared. Though I'm still unable to reproduce it
>>>>>>> following your steps. I can still login and seusers is labeled
>>>>>>> selinux_config_t. I'll keep looking into this.
>>>>>>>
>>>>>>> I've also rebased/pushed #integration onto #next.
>>>>>>
>>>>>> # Revert to stock F20 SELinux userspace and policy.
>>>>>> yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>>>>>> policycoreutils* selinux-policy-targeted
>>>>>>
>>>>>> # Clear prior source/CIL policy store.
>>>>>> rm -rf /var/lib/selinux
>>>>>>
>>>>>> # Reboot to ensure systemd and friends are using the new policy.
>>>>>> reboot
>>>>>>
>>>>>> # Reset selinux and cil to latest sources
>>>>>> cd selinux
>>>>>> git clean -fdx
>>>>>> git fetch origin
>>>>>> git reset --hard origin/integration
>>>>>> cd ../cil
>>>>>> git clean -fdx
>>>>>> git fetch origin
>>>>>> git reset --hard origin/master
>>>>>>
>>>>>> # Build and install new userspace
>>>>>> cd ..
>>>>>> ln -sf ../../cil selinux/libsepol/cil
>>>>>> make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>>>>> relabel
>>>>>>
>>>>>> # Convert
>>>>>>  ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>>>>>
>>>>>> Try to login on console or via ssh:  Unable to get valid context for sds.
>>>>>>
>>>>>> dmesg | grep systemd
>>>>>> [  343.739985] systemd[1]: SELinux policy denies access.
>>>>>> [  348.256030] systemd[1]: SELinux policy denies access.
>>>>>> [  376.335248] systemd[1]: SELinux policy denies access.
>>>>>> [  376.515343] systemd[1]: SELinux policy denies access.
>>>>>>
>>>>>> restorecon -R /etc/selinux/targeted
>>>>>>
>>>>>> Try to login again, hangs for a long time before finally succeeding.
>>>>>>
>>>>>> reboot
>>>>>>
>>>>>> Everything is happy.
>>>>>>
>>>>>> 100% reproducible, every time.
>>>>>>
>>>>>>
>>>>>
>>>>> Thanks for the steps. I think I found what causes the labeling problem,
>>>>> but I'm not yet sure why. It looks like it has something to do with how
>>>>> the migration script rebuilds policy. If you run the migration script
>>>>> with the --norebuild flag, and then run semodule -B, everything is
>>>>> rebuilt and the files in /etc/selinux are labeled correctly.
>>>>>
>>>>> I'm not yet convinced this is the same problem as the systemd issue (I
>>>>> do get "unable to get valid context", but I still don't get "SELinux
>>>>> policy denies access"). Let me know if you still see that after using
>>>>> --norebuild/semodule -B.
>>>>
>>>> Ok, if I do:
>>>> ./libsemanage/utils/semanage_migrate_etc_to_var.py -n
>>>> semodule -B
>>>>
>>>> then /etc/selinux is labeled correctly and I can still login.
>>>> However, running su hangs and I get the systemd error message again.
>>>
>>> Also get this:
>>> ERROR:dbus.proxies:Introspect error on :1.4:/org/freedesktop/login1:
>>> dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did
>>> not receive a reply. Possible causes include: the remote application did
>>> not send a reply, the message bus security policy blocked the reply, the
>>> reply timeout expired, or the network connection was broken.
>>>
>>> when trying to re-install the original selinux userspace packages.
>>> With more errors from systemd in dmesg,
>>> [14421.297045] systemd[1]: SELinux policy denies access.
>>> [14451.711732] systemd[1]: SELinux policy denies access.
>>> [14451.743787] systemd[1]: SELinux policy denies access.
>>> [14512.551210] systemd[1]: SELinux policy denies access.
>>> [14512.584126] systemd[1]: SELinux policy denies access.
>>> [14543.063872] systemd[1]: SELinux policy denies access.
>>> [14634.451680] systemd[1]: SELinux policy denies access.
>>> [14634.469813] systemd[1]: SELinux policy denies access.
>>> [14659.346867] systemd[1]: SELinux policy denies access.
>>> [14659.665108] systemd[1]: SELinux policy denies access.
>>> [14695.602174] systemd[1]: SELinux policy denies access.
>>
>> journalctl -r shows an additional error message for each of these denials:
>> systemd-logind[913]: Failed to query ActiveState: Access denied
> 
> ausearch -m USER_AVC reports:
> type=USER_AVC msg=audit(1405520109.191:1031): pid=1 uid=0
> auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
> msg='avc:  denied  { status } for auid=-1 uid=-1 gid=-1
> scontext=system_u:system_r:systemd_logind_t:s0
> tcontext=system_u:system_r:init_t:s0 tclass=service
> exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
> 

Hmm. I still can't get this error. The only thing I get with ausearch is

type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
addr=? terminal=?'

Which looks correct. Fedora's latest policy does not include start in
the system class:

$ seinfo -csystem -x
   system
      status
      module_request
      reboot
      disable
      enable
      undefined
      ipc_info
      syslog_read
      halt
      reload
      syslog_console
      syslog_mod

Also, the policy built with CIL on my machine allows the USER_AVC you're
seeing:

$ sesearch -A -s systemd_logind_t -t init_t -c service
Found 2 semantic av rules:
   allow systemd_domain init_t : service { stop status reload start } ;
   allow systemd_logind_t init_t : service status ;



Not sure if this would help, but it looks like you can set the boot
parameter systemd.log_level=debug, and it should print all the selinux
access checks, including which ones cause the "SELinux policy denies
access" message. Unfortunately, I think the extra debug messages
prevents my VM from booting, but you might have better luck.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
<snip>
> Hmm. I still can't get this error. The only thing I get with ausearch is
> 
> type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
> ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
> start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
> addr=? terminal=?'
> 
> Which looks correct. Fedora's latest policy does not include start in
> the system class:
> 
> $ seinfo -csystem -x
>    system
>       status
>       module_request
>       reboot
>       disable
>       enable
>       undefined
>       ipc_info
>       syslog_read
>       halt
>       reload
>       syslog_console
>       syslog_mod
> 
> Also, the policy built with CIL on my machine allows the USER_AVC you're
> seeing:
> 
> $ sesearch -A -s systemd_logind_t -t init_t -c service
> Found 2 semantic av rules:
>    allow systemd_domain init_t : service { stop status reload start } ;
>    allow systemd_logind_t init_t : service status ;
> 
> 
> 
> Not sure if this would help, but it looks like you can set the boot
> parameter systemd.log_level=debug, and it should print all the selinux
> access checks, including which ones cause the "SELinux policy denies
> access" message. Unfortunately, I think the extra debug messages
> prevents my VM from booting, but you might have better luck.
> 

The same symptoms as with the classorder issue except that this time it
only happens once after the upgrade. Rebooting fixes the issue (?)

That was not the case with the classorder issue.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Wed, 2014-07-16 at 17:53 +0200, Dominick Grift wrote:
> On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
> <snip>
> > Hmm. I still can't get this error. The only thing I get with ausearch is
> > 
> > type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
> > ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
> > start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
> > addr=? terminal=?'
> > 
> > Which looks correct. Fedora's latest policy does not include start in
> > the system class:
> > 
> > $ seinfo -csystem -x
> >    system
> >       status
> >       module_request
> >       reboot
> >       disable
> >       enable
> >       undefined
> >       ipc_info
> >       syslog_read
> >       halt
> >       reload
> >       syslog_console
> >       syslog_mod
> > 
> > Also, the policy built with CIL on my machine allows the USER_AVC you're
> > seeing:
> > 
> > $ sesearch -A -s systemd_logind_t -t init_t -c service
> > Found 2 semantic av rules:
> >    allow systemd_domain init_t : service { stop status reload start } ;
> >    allow systemd_logind_t init_t : service status ;
> > 
> > 
> > 
> > Not sure if this would help, but it looks like you can set the boot
> > parameter systemd.log_level=debug, and it should print all the selinux
> > access checks, including which ones cause the "SELinux policy denies
> > access" message. Unfortunately, I think the extra debug messages
> > prevents my VM from booting, but you might have better luck.
> > 
> 
> The same symptoms as with the classorder issue except that this time it
> only happens once after the upgrade. Rebooting fixes the issue (?)
> 
> That was not the case with the classorder issue.
> 
> 

I meant the boolean issue. (access denied whilst rules to allow are
present)

I have a feeling that this is caused by the migration script since i
have no issues on my systems with plain secilc, load_policy.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/16/2014 11:53 AM, Dominick Grift wrote:
> On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
> <snip>
>> Hmm. I still can't get this error. The only thing I get with ausearch is
>>
>> type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
>> ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
>> start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
>> addr=? terminal=?'
>>
>> Which looks correct. Fedora's latest policy does not include start in
>> the system class:
>>
>> $ seinfo -csystem -x
>>    system
>>       status
>>       module_request
>>       reboot
>>       disable
>>       enable
>>       undefined
>>       ipc_info
>>       syslog_read
>>       halt
>>       reload
>>       syslog_console
>>       syslog_mod
>>
>> Also, the policy built with CIL on my machine allows the USER_AVC you're
>> seeing:
>>
>> $ sesearch -A -s systemd_logind_t -t init_t -c service
>> Found 2 semantic av rules:
>>    allow systemd_domain init_t : service { stop status reload start } ;
>>    allow systemd_logind_t init_t : service status ;
>>
>>
>>
>> Not sure if this would help, but it looks like you can set the boot
>> parameter systemd.log_level=debug, and it should print all the selinux
>> access checks, including which ones cause the "SELinux policy denies
>> access" message. Unfortunately, I think the extra debug messages
>> prevents my VM from booting, but you might have better luck.
>>
> 
> The same symptoms as with the classorder issue except that this time it
> only happens once after the upgrade. Rebooting fixes the issue (?)
> 
> That was not the case with the classorder issue.

$ yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
policycoreutils* selinux-policy-targeted
$ cp -a /sys/fs/selinux/class class.orig
$ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install
install-pywrap relabel
$ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
$ cp -a /sys/fs/selinux/class class.new
$ diff -ru class.orig class.new

Lots of differences in permission values.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/16/2014 03:00 PM, Stephen Smalley wrote:
> On 07/16/2014 11:53 AM, Dominick Grift wrote:
>> On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
>> <snip>
>>> Hmm. I still can't get this error. The only thing I get with ausearch is
>>>
>>> type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
>>> ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
>>> start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
>>> addr=? terminal=?'
>>>
>>> Which looks correct. Fedora's latest policy does not include start in
>>> the system class:
>>>
>>> $ seinfo -csystem -x
>>>    system
>>>       status
>>>       module_request
>>>       reboot
>>>       disable
>>>       enable
>>>       undefined
>>>       ipc_info
>>>       syslog_read
>>>       halt
>>>       reload
>>>       syslog_console
>>>       syslog_mod
>>>
>>> Also, the policy built with CIL on my machine allows the USER_AVC you're
>>> seeing:
>>>
>>> $ sesearch -A -s systemd_logind_t -t init_t -c service
>>> Found 2 semantic av rules:
>>>    allow systemd_domain init_t : service { stop status reload start } ;
>>>    allow systemd_logind_t init_t : service status ;
>>>
>>>
>>>
>>> Not sure if this would help, but it looks like you can set the boot
>>> parameter systemd.log_level=debug, and it should print all the selinux
>>> access checks, including which ones cause the "SELinux policy denies
>>> access" message. Unfortunately, I think the extra debug messages
>>> prevents my VM from booting, but you might have better luck.
>>>
>>
>> The same symptoms as with the classorder issue except that this time it
>> only happens once after the upgrade. Rebooting fixes the issue (?)
>>
>> That was not the case with the classorder issue.
> 
> $ yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
> policycoreutils* selinux-policy-targeted
> $ cp -a /sys/fs/selinux/class class.orig
> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install
> install-pywrap relabel
> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
> $ cp -a /sys/fs/selinux/class class.new
> $ diff -ru class.orig class.new
> 
> Lots of differences in permission values.
> 

Nice find! This looks like it's the problem. When we convert the base pp
to CIL, the permissions are output to CIL in the order that hashtab_map
receives them, which is is not in any particular order. So the
permission orders are all wrong, causing the values in the binary to be
different. I guess the kernel doesn't like when permission values change
when loading a new policy. And this explains why things work following a
reboot.

We've updated the pp2cil compiler to sort the permissions and output
them in the correct order. This change has been rebased and pushed to
the integration branch on github.

Thanks,
- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/17/2014 09:49 AM, Steve Lawrence wrote:
> On 07/16/2014 03:00 PM, Stephen Smalley wrote:
>> On 07/16/2014 11:53 AM, Dominick Grift wrote:
>>> On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
>>> <snip>
>>>> Hmm. I still can't get this error. The only thing I get with ausearch is
>>>>
>>>> type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
>>>> ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
>>>> start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
>>>> addr=? terminal=?'
>>>>
>>>> Which looks correct. Fedora's latest policy does not include start in
>>>> the system class:
>>>>
>>>> $ seinfo -csystem -x
>>>>    system
>>>>       status
>>>>       module_request
>>>>       reboot
>>>>       disable
>>>>       enable
>>>>       undefined
>>>>       ipc_info
>>>>       syslog_read
>>>>       halt
>>>>       reload
>>>>       syslog_console
>>>>       syslog_mod
>>>>
>>>> Also, the policy built with CIL on my machine allows the USER_AVC you're
>>>> seeing:
>>>>
>>>> $ sesearch -A -s systemd_logind_t -t init_t -c service
>>>> Found 2 semantic av rules:
>>>>    allow systemd_domain init_t : service { stop status reload start } ;
>>>>    allow systemd_logind_t init_t : service status ;
>>>>
>>>>
>>>>
>>>> Not sure if this would help, but it looks like you can set the boot
>>>> parameter systemd.log_level=debug, and it should print all the selinux
>>>> access checks, including which ones cause the "SELinux policy denies
>>>> access" message. Unfortunately, I think the extra debug messages
>>>> prevents my VM from booting, but you might have better luck.
>>>>
>>>
>>> The same symptoms as with the classorder issue except that this time it
>>> only happens once after the upgrade. Rebooting fixes the issue (?)
>>>
>>> That was not the case with the classorder issue.
>>
>> $ yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>> policycoreutils* selinux-policy-targeted
>> $ cp -a /sys/fs/selinux/class class.orig
>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install
>> install-pywrap relabel
>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>> $ cp -a /sys/fs/selinux/class class.new
>> $ diff -ru class.orig class.new
>>
>> Lots of differences in permission values.
>>
> 
> Nice find! This looks like it's the problem. When we convert the base pp
> to CIL, the permissions are output to CIL in the order that hashtab_map
> receives them, which is is not in any particular order. So the
> permission orders are all wrong, causing the values in the binary to be
> different. I guess the kernel doesn't like when permission values change
> when loading a new policy. And this explains why things work following a
> reboot.

I think the kernel is fine with it, but not all userspace object
managers will be.  Legacy userspace object managers may still be using
fixed permission values from selinux/av_permissions.h, and even modern
ones may be only mapping classes/permissions once at startup (via
selinux_set_mapping) and then assuming that this mapping is stable at
runtime.  Even for userspace object managers that are using
selinux_check_access(), libselinux/src/stringrep.c currently caches the
class/perm values on first lookup and does not flush this cache from
avc_reset upon policy reload notification - that's likely a bug that
should be fixed.  But even if we fix it there, there is no current
atomicity provided within selinux_check_access() around the class/perm
lookup and the AVC lookup so they could end up being inconsistent.  In
short, we don't currently guarantee that changing class/perm values at
runtime without a reboot will work.

> We've updated the pp2cil compiler to sort the permissions and output
> them in the correct order. This change has been rebased and pushed to
> the integration branch on github.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/17/2014 09:49 AM, Steve Lawrence wrote:
> On 07/16/2014 03:00 PM, Stephen Smalley wrote:
>> On 07/16/2014 11:53 AM, Dominick Grift wrote:
>>> On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
>>> <snip>
>>>> Hmm. I still can't get this error. The only thing I get with ausearch is
>>>>
>>>> type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
>>>> ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
>>>> start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
>>>> addr=? terminal=?'
>>>>
>>>> Which looks correct. Fedora's latest policy does not include start in
>>>> the system class:
>>>>
>>>> $ seinfo -csystem -x
>>>>    system
>>>>       status
>>>>       module_request
>>>>       reboot
>>>>       disable
>>>>       enable
>>>>       undefined
>>>>       ipc_info
>>>>       syslog_read
>>>>       halt
>>>>       reload
>>>>       syslog_console
>>>>       syslog_mod
>>>>
>>>> Also, the policy built with CIL on my machine allows the USER_AVC you're
>>>> seeing:
>>>>
>>>> $ sesearch -A -s systemd_logind_t -t init_t -c service
>>>> Found 2 semantic av rules:
>>>>    allow systemd_domain init_t : service { stop status reload start } ;
>>>>    allow systemd_logind_t init_t : service status ;
>>>>
>>>>
>>>>
>>>> Not sure if this would help, but it looks like you can set the boot
>>>> parameter systemd.log_level=debug, and it should print all the selinux
>>>> access checks, including which ones cause the "SELinux policy denies
>>>> access" message. Unfortunately, I think the extra debug messages
>>>> prevents my VM from booting, but you might have better luck.
>>>>
>>>
>>> The same symptoms as with the classorder issue except that this time it
>>> only happens once after the upgrade. Rebooting fixes the issue (?)
>>>
>>> That was not the case with the classorder issue.
>>
>> $ yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>> policycoreutils* selinux-policy-targeted
>> $ cp -a /sys/fs/selinux/class class.orig
>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install
>> install-pywrap relabel
>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>> $ cp -a /sys/fs/selinux/class class.new
>> $ diff -ru class.orig class.new
>>
>> Lots of differences in permission values.
>>
> 
> Nice find! This looks like it's the problem. When we convert the base pp
> to CIL, the permissions are output to CIL in the order that hashtab_map
> receives them, which is is not in any particular order. So the
> permission orders are all wrong, causing the values in the binary to be
> different. I guess the kernel doesn't like when permission values change
> when loading a new policy. And this explains why things work following a
> reboot.
> 
> We've updated the pp2cil compiler to sort the permissions and output
> them in the correct order. This change has been rebased and pushed to
> the integration branch on github.

Ok, so I think this resolves the issues I noticed.

What other known issues remain unresolved?  Is there any functionality
known to be missing that was supported by the old policy toolchain?

What new functionality is included here that was not previously
supported by the old policy toolchain?

Any chance of getting a hll compiler for refpolicy source modules, i.e.
in .if/.te/.fc form?

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/17/2014 02:02 PM, Stephen Smalley wrote:
> On 07/17/2014 09:49 AM, Steve Lawrence wrote:
>> On 07/16/2014 03:00 PM, Stephen Smalley wrote:
>>> On 07/16/2014 11:53 AM, Dominick Grift wrote:
>>>> On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
>>>> <snip>
>>>>> Hmm. I still can't get this error. The only thing I get with ausearch is
>>>>>
>>>>> type=USER_AVC msg=audit(1405522202.264:463): pid=1 uid=0 auid=4294967295
>>>>> ses=4294967295  subj=system_u:system_r:init_t:s0 msg='Unknown permission
>>>>> start for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=?
>>>>> addr=? terminal=?'
>>>>>
>>>>> Which looks correct. Fedora's latest policy does not include start in
>>>>> the system class:
>>>>>
>>>>> $ seinfo -csystem -x
>>>>>    system
>>>>>       status
>>>>>       module_request
>>>>>       reboot
>>>>>       disable
>>>>>       enable
>>>>>       undefined
>>>>>       ipc_info
>>>>>       syslog_read
>>>>>       halt
>>>>>       reload
>>>>>       syslog_console
>>>>>       syslog_mod
>>>>>
>>>>> Also, the policy built with CIL on my machine allows the USER_AVC you're
>>>>> seeing:
>>>>>
>>>>> $ sesearch -A -s systemd_logind_t -t init_t -c service
>>>>> Found 2 semantic av rules:
>>>>>    allow systemd_domain init_t : service { stop status reload start } ;
>>>>>    allow systemd_logind_t init_t : service status ;
>>>>>
>>>>>
>>>>>
>>>>> Not sure if this would help, but it looks like you can set the boot
>>>>> parameter systemd.log_level=debug, and it should print all the selinux
>>>>> access checks, including which ones cause the "SELinux policy denies
>>>>> access" message. Unfortunately, I think the extra debug messages
>>>>> prevents my VM from booting, but you might have better luck.
>>>>>
>>>>
>>>> The same symptoms as with the classorder issue except that this time it
>>>> only happens once after the upgrade. Rebooting fixes the issue (?)
>>>>
>>>> That was not the case with the classorder issue.
>>>
>>> $ yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>>> policycoreutils* selinux-policy-targeted
>>> $ cp -a /sys/fs/selinux/class class.orig
>>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install
>>> install-pywrap relabel
>>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>> $ cp -a /sys/fs/selinux/class class.new
>>> $ diff -ru class.orig class.new
>>>
>>> Lots of differences in permission values.
>>>
>>
>> Nice find! This looks like it's the problem. When we convert the base pp
>> to CIL, the permissions are output to CIL in the order that hashtab_map
>> receives them, which is is not in any particular order. So the
>> permission orders are all wrong, causing the values in the binary to be
>> different. I guess the kernel doesn't like when permission values change
>> when loading a new policy. And this explains why things work following a
>> reboot.
>>
>> We've updated the pp2cil compiler to sort the permissions and output
>> them in the correct order. This change has been rebased and pushed to
>> the integration branch on github.
> 
> Ok, so I think this resolves the issues I noticed.

Great!

> What other known issues remain unresolved?  Is there any functionality
> known to be missing that was supported by the old policy toolchain?

I think the only remaining issue is the one Dominick mentioned in his
first email regarding file_contexts.homedirs. I don't think this is an
actual bug, just the migration script migrating things that don't need
to be migrated. Still investigating it. We should have an update
sometime tomorrow.

> What new functionality is included here that was not previously
> supported by the old policy toolchain?

In terms a user would see, the most visible change is support for CIL
policies and HLLs, of which there's only one right now (pp2cil). There
are also some new semanage.conf options (target-platform, compiler-dir,
ignore-module-cache, store-root) but I imagine the vast majority of
people could just use the defaults. Similarly, we've added
--ignore-module-cache and --store-root to the semodule command. We've
also moved the store to /var/lib/selinux, but this is more behind the
scenes and should really only affect distributions.

Though, there are two things we just realized have a different behavior.

1) verify_modules is now performed on the CIL modules, rather than pp
(or HLL) modules. So if someone is using verify_modules, things will
probably break. I'm not sure if anyone uses this feature or how
important it is that we maintain backwards compatibility.

2) verify_linked is no longer called, since there isn't any concept of a
linked base module with CIL

Aside from that, I think all functionality should remain the same.

> Any chance of getting a hll compiler for refpolicy source modules, i.e.
> in .if/.te/.fc form?

That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
but the current HLL infrastructure may need some changes before that can
be supported. I think the main problem is that Jim's tool needs
knowledge of all modules to be able to convert them to CIL, but the
current HLL infrastructure compiles each module separately. We have
various ideas on how we can update the HLL infrastructure to support
this, but we've primarily been focused on getting the core CIL/HLL
functionality complete and upstreamed before focusing on the more
complicated HLL patterns.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/17/2014 02:58 PM, Steve Lawrence wrote:
> I think the only remaining issue is the one Dominick mentioned in his
> first email regarding file_contexts.homedirs. I don't think this is an
> actual bug, just the migration script migrating things that don't need
> to be migrated. Still investigating it. We should have an update
> sometime tomorrow.

So everything you reverted you restored in equivalent form?

>> What new functionality is included here that was not previously
>> supported by the old policy toolchain?
> 
> In terms a user would see, the most visible change is support for CIL
> policies and HLLs, of which there's only one right now (pp2cil). There
> are also some new semanage.conf options (target-platform, compiler-dir,
> ignore-module-cache, store-root) but I imagine the vast majority of
> people could just use the defaults. Similarly, we've added
> --ignore-module-cache and --store-root to the semodule command. We've
> also moved the store to /var/lib/selinux, but this is more behind the
> scenes and should really only affect distributions.

What about new features/options of the user-facing commands?  I know
some features were copied from earlier source/CIL releases into the main
selinux userspace (e.g. enabled/disabled modules), but aren't some
things like module priorities new?

> Though, there are two things we just realized have a different behavior.
> 
> 1) verify_modules is now performed on the CIL modules, rather than pp
> (or HLL) modules. So if someone is using verify_modules, things will
> probably break. I'm not sure if anyone uses this feature or how
> important it is that we maintain backwards compatibility.
> 
> 2) verify_linked is no longer called, since there isn't any concept of a
> linked base module with CIL
> 
> Aside from that, I think all functionality should remain the same.

I'm not aware of anyone using anything other than verify kernel.

>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>> in .if/.te/.fc form?
> 
> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
> but the current HLL infrastructure may need some changes before that can
> be supported. I think the main problem is that Jim's tool needs
> knowledge of all modules to be able to convert them to CIL, but the
> current HLL infrastructure compiles each module separately. We have
> various ideas on how we can update the HLL infrastructure to support
> this, but we've primarily been focused on getting the core CIL/HLL
> functionality complete and upstreamed before focusing on the more
> complicated HLL patterns.

Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
re-tooled to generate source modules, and we'll essentially need a
workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
mymodule.pp; semodule -i mymodule.pp.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/17/2014 03:10 PM, Stephen Smalley wrote:
> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>> I think the only remaining issue is the one Dominick mentioned in his
>> first email regarding file_contexts.homedirs. I don't think this is an
>> actual bug, just the migration script migrating things that don't need
>> to be migrated. Still investigating it. We should have an update
>> sometime tomorrow.
> 
> So everything you reverted you restored in equivalent form?
> 
>>> What new functionality is included here that was not previously
>>> supported by the old policy toolchain?
>>
>> In terms a user would see, the most visible change is support for CIL
>> policies and HLLs, of which there's only one right now (pp2cil). There
>> are also some new semanage.conf options (target-platform, compiler-dir,
>> ignore-module-cache, store-root) but I imagine the vast majority of
>> people could just use the defaults. Similarly, we've added
>> --ignore-module-cache and --store-root to the semodule command. We've
>> also moved the store to /var/lib/selinux, but this is more behind the
>> scenes and should really only affect distributions.
> 
> What about new features/options of the user-facing commands?  I know
> some features were copied from earlier source/CIL releases into the main
> selinux userspace (e.g. enabled/disabled modules), but aren't some
> things like module priorities new?
> 
>> Though, there are two things we just realized have a different behavior.
>>
>> 1) verify_modules is now performed on the CIL modules, rather than pp
>> (or HLL) modules. So if someone is using verify_modules, things will
>> probably break. I'm not sure if anyone uses this feature or how
>> important it is that we maintain backwards compatibility.
>>
>> 2) verify_linked is no longer called, since there isn't any concept of a
>> linked base module with CIL
>>
>> Aside from that, I think all functionality should remain the same.
> 
> I'm not aware of anyone using anything other than verify kernel.
> 
>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>> in .if/.te/.fc form?
>>
>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>> but the current HLL infrastructure may need some changes before that can
>> be supported. I think the main problem is that Jim's tool needs
>> knowledge of all modules to be able to convert them to CIL, but the
>> current HLL infrastructure compiles each module separately. We have
>> various ideas on how we can update the HLL infrastructure to support
>> this, but we've primarily been focused on getting the core CIL/HLL
>> functionality complete and upstreamed before focusing on the more
>> complicated HLL patterns.
> 
> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
> re-tooled to generate source modules, and we'll essentially need a
> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
> mymodule.pp; semodule -i mymodule.pp.

I guess one other possible concern might be storage:

$ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
5.4M	/etc/selinux/targeted/modules
11M	/var/lib/selinux/

I'm guessing that is just the cost of storing each module in both binary
and cil form?

Is there an option to discard the .pp files altogether and only retain
the cil files?

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/17/2014 03:48 PM, Stephen Smalley wrote:
> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>> I think the only remaining issue is the one Dominick mentioned in his
>>> first email regarding file_contexts.homedirs. I don't think this is an
>>> actual bug, just the migration script migrating things that don't need
>>> to be migrated. Still investigating it. We should have an update
>>> sometime tomorrow.
>>
>> So everything you reverted you restored in equivalent form?
>>
>>>> What new functionality is included here that was not previously
>>>> supported by the old policy toolchain?
>>>
>>> In terms a user would see, the most visible change is support for CIL
>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>> people could just use the defaults. Similarly, we've added
>>> --ignore-module-cache and --store-root to the semodule command. We've
>>> also moved the store to /var/lib/selinux, but this is more behind the
>>> scenes and should really only affect distributions.
>>
>> What about new features/options of the user-facing commands?  I know
>> some features were copied from earlier source/CIL releases into the main
>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>> things like module priorities new?
>>
>>> Though, there are two things we just realized have a different behavior.
>>>
>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>> (or HLL) modules. So if someone is using verify_modules, things will
>>> probably break. I'm not sure if anyone uses this feature or how
>>> important it is that we maintain backwards compatibility.
>>>
>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>> linked base module with CIL
>>>
>>> Aside from that, I think all functionality should remain the same.
>>
>> I'm not aware of anyone using anything other than verify kernel.
>>
>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>> in .if/.te/.fc form?
>>>
>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>> but the current HLL infrastructure may need some changes before that can
>>> be supported. I think the main problem is that Jim's tool needs
>>> knowledge of all modules to be able to convert them to CIL, but the
>>> current HLL infrastructure compiles each module separately. We have
>>> various ideas on how we can update the HLL infrastructure to support
>>> this, but we've primarily been focused on getting the core CIL/HLL
>>> functionality complete and upstreamed before focusing on the more
>>> complicated HLL patterns.
>>
>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>> re-tooled to generate source modules, and we'll essentially need a
>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>> mymodule.pp; semodule -i mymodule.pp.
> 
> I guess one other possible concern might be storage:
> 
> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
> 5.4M	/etc/selinux/targeted/modules
> 11M	/var/lib/selinux/
> 
> I'm guessing that is just the cost of storing each module in both binary
> and cil form?

Yep, we store both HLL and CIL. They are both compressed and CIL
compresses decently since it's just text, but it's still a lot of text.

> Is there an option to discard the .pp files altogether and only retain
> the cil files?

Not at the moment, but it wouldn't be hard to accomplish. Just need to
delete all the hll files and change the contents of lang_ext to 'cil'.
Something we could add if storage is an issue.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/17/2014 04:04 PM, Steve Lawrence wrote:
> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>>> I think the only remaining issue is the one Dominick mentioned in his
>>>> first email regarding file_contexts.homedirs. I don't think this is an
>>>> actual bug, just the migration script migrating things that don't need
>>>> to be migrated. Still investigating it. We should have an update
>>>> sometime tomorrow.
>>>
>>> So everything you reverted you restored in equivalent form?
>>>
>>>>> What new functionality is included here that was not previously
>>>>> supported by the old policy toolchain?
>>>>
>>>> In terms a user would see, the most visible change is support for CIL
>>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>>> people could just use the defaults. Similarly, we've added
>>>> --ignore-module-cache and --store-root to the semodule command. We've
>>>> also moved the store to /var/lib/selinux, but this is more behind the
>>>> scenes and should really only affect distributions.
>>>
>>> What about new features/options of the user-facing commands?  I know
>>> some features were copied from earlier source/CIL releases into the main
>>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>>> things like module priorities new?
>>>
>>>> Though, there are two things we just realized have a different behavior.
>>>>
>>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>>> (or HLL) modules. So if someone is using verify_modules, things will
>>>> probably break. I'm not sure if anyone uses this feature or how
>>>> important it is that we maintain backwards compatibility.
>>>>
>>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>>> linked base module with CIL
>>>>
>>>> Aside from that, I think all functionality should remain the same.
>>>
>>> I'm not aware of anyone using anything other than verify kernel.
>>>
>>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>>> in .if/.te/.fc form?
>>>>
>>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>>> but the current HLL infrastructure may need some changes before that can
>>>> be supported. I think the main problem is that Jim's tool needs
>>>> knowledge of all modules to be able to convert them to CIL, but the
>>>> current HLL infrastructure compiles each module separately. We have
>>>> various ideas on how we can update the HLL infrastructure to support
>>>> this, but we've primarily been focused on getting the core CIL/HLL
>>>> functionality complete and upstreamed before focusing on the more
>>>> complicated HLL patterns.
>>>
>>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>>> re-tooled to generate source modules, and we'll essentially need a
>>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>>> mymodule.pp; semodule -i mymodule.pp.
>>
>> I guess one other possible concern might be storage:
>>
>> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
>> 5.4M	/etc/selinux/targeted/modules
>> 11M	/var/lib/selinux/
>>
>> I'm guessing that is just the cost of storing each module in both binary
>> and cil form?
> 
> Yep, we store both HLL and CIL. They are both compressed and CIL
> compresses decently since it's just text, but it's still a lot of text.
> 
>> Is there an option to discard the .pp files altogether and only retain
>> the cil files?
> 
> Not at the moment, but it wouldn't be hard to accomplish. Just need to
> delete all the hll files and change the contents of lang_ext to 'cil'.
> Something we could add if storage is an issue.

That worked, albeit I had to learn that lang_ext must not include a
newline or libsemanage won't accept it.  Took it down to 5.8M.  I
suspect we could also stop retaining a copy of certain generated files
like file_contexts although that is no different than the current code.

Not sure what benefit there is in retaining the pp files, since they
carry no additional information AFAIK and they aren't human viewable or
editable.  Is there even an option for exporting modules from the policy
store currently that would allow extracting them except via direct file
access to the policy store?

More generally, if the user knows that the hll module is going to be
saved elsewhere, then there is no reason to retain a copy in the policy
store, so having the option of dropping the hll version, either for all
modules or for specific modules, seems useful.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Daniel J Walsh]

On 07/17/2014 04:37 PM, Stephen Smalley wrote:
> On 07/17/2014 04:04 PM, Steve Lawrence wrote:
>> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>>>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>>>> I think the only remaining issue is the one Dominick mentioned in his
>>>>> first email regarding file_contexts.homedirs. I don't think this is an
>>>>> actual bug, just the migration script migrating things that don't need
>>>>> to be migrated. Still investigating it. We should have an update
>>>>> sometime tomorrow.
>>>> So everything you reverted you restored in equivalent form?
>>>>
>>>>>> What new functionality is included here that was not previously
>>>>>> supported by the old policy toolchain?
>>>>> In terms a user would see, the most visible change is support for CIL
>>>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>>>> people could just use the defaults. Similarly, we've added
>>>>> --ignore-module-cache and --store-root to the semodule command. We've
>>>>> also moved the store to /var/lib/selinux, but this is more behind the
>>>>> scenes and should really only affect distributions.
>>>> What about new features/options of the user-facing commands?  I know
>>>> some features were copied from earlier source/CIL releases into the main
>>>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>>>> things like module priorities new?
>>>>
>>>>> Though, there are two things we just realized have a different behavior.
>>>>>
>>>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>>>> (or HLL) modules. So if someone is using verify_modules, things will
>>>>> probably break. I'm not sure if anyone uses this feature or how
>>>>> important it is that we maintain backwards compatibility.
>>>>>
>>>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>>>> linked base module with CIL
>>>>>
>>>>> Aside from that, I think all functionality should remain the same.
>>>> I'm not aware of anyone using anything other than verify kernel.
>>>>
>>>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>>>> in .if/.te/.fc form?
>>>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>>>> but the current HLL infrastructure may need some changes before that can
>>>>> be supported. I think the main problem is that Jim's tool needs
>>>>> knowledge of all modules to be able to convert them to CIL, but the
>>>>> current HLL infrastructure compiles each module separately. We have
>>>>> various ideas on how we can update the HLL infrastructure to support
>>>>> this, but we've primarily been focused on getting the core CIL/HLL
>>>>> functionality complete and upstreamed before focusing on the more
>>>>> complicated HLL patterns.
>>>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>>>> re-tooled to generate source modules, and we'll essentially need a
>>>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>>>> mymodule.pp; semodule -i mymodule.pp.
>>> I guess one other possible concern might be storage:
>>>
>>> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
>>> 5.4M	/etc/selinux/targeted/modules
>>> 11M	/var/lib/selinux/
>>>
>>> I'm guessing that is just the cost of storing each module in both binary
>>> and cil form?
>> Yep, we store both HLL and CIL. They are both compressed and CIL
>> compresses decently since it's just text, but it's still a lot of text.
>>
>>> Is there an option to discard the .pp files altogether and only retain
>>> the cil files?
>> Not at the moment, but it wouldn't be hard to accomplish. Just need to
>> delete all the hll files and change the contents of lang_ext to 'cil'.
>> Something we could add if storage is an issue.
> That worked, albeit I had to learn that lang_ext must not include a
> newline or libsemanage won't accept it.  Took it down to 5.8M.  I
> suspect we could also stop retaining a copy of certain generated files
> like file_contexts although that is no different than the current code.
>
> Not sure what benefit there is in retaining the pp files, since they
> carry no additional information AFAIK and they aren't human viewable or
> editable.  Is there even an option for exporting modules from the policy
> store currently that would allow extracting them except via direct file
> access to the policy store?
>
> More generally, if the user knows that the hll module is going to be
> saved elsewhere, then there is no reason to retain a copy in the policy
> store, so having the option of dropping the hll version, either for all
> modules or for specific modules, seems useful.
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>
>
I would also like to see the policy file itself compressed. I have some
patches I have played with to
compress the kernel.  The current libselinux on Fedora/Rawhide supports
readding a compressed policy
file.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Daniel J Walsh]

On 07/17/2014 04:37 PM, Stephen Smalley wrote:
> On 07/17/2014 04:04 PM, Steve Lawrence wrote:
>> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>>>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>>>> I think the only remaining issue is the one Dominick mentioned in his
>>>>> first email regarding file_contexts.homedirs. I don't think this is an
>>>>> actual bug, just the migration script migrating things that don't need
>>>>> to be migrated. Still investigating it. We should have an update
>>>>> sometime tomorrow.
>>>> So everything you reverted you restored in equivalent form?
>>>>
>>>>>> What new functionality is included here that was not previously
>>>>>> supported by the old policy toolchain?
>>>>> In terms a user would see, the most visible change is support for CIL
>>>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>>>> people could just use the defaults. Similarly, we've added
>>>>> --ignore-module-cache and --store-root to the semodule command. We've
>>>>> also moved the store to /var/lib/selinux, but this is more behind the
>>>>> scenes and should really only affect distributions.
>>>> What about new features/options of the user-facing commands?  I know
>>>> some features were copied from earlier source/CIL releases into the main
>>>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>>>> things like module priorities new?
>>>>
>>>>> Though, there are two things we just realized have a different behavior.
>>>>>
>>>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>>>> (or HLL) modules. So if someone is using verify_modules, things will
>>>>> probably break. I'm not sure if anyone uses this feature or how
>>>>> important it is that we maintain backwards compatibility.
>>>>>
>>>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>>>> linked base module with CIL
>>>>>
>>>>> Aside from that, I think all functionality should remain the same.
>>>> I'm not aware of anyone using anything other than verify kernel.
>>>>
>>>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>>>> in .if/.te/.fc form?
>>>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>>>> but the current HLL infrastructure may need some changes before that can
>>>>> be supported. I think the main problem is that Jim's tool needs
>>>>> knowledge of all modules to be able to convert them to CIL, but the
>>>>> current HLL infrastructure compiles each module separately. We have
>>>>> various ideas on how we can update the HLL infrastructure to support
>>>>> this, but we've primarily been focused on getting the core CIL/HLL
>>>>> functionality complete and upstreamed before focusing on the more
>>>>> complicated HLL patterns.
>>>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>>>> re-tooled to generate source modules, and we'll essentially need a
>>>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>>>> mymodule.pp; semodule -i mymodule.pp.
>>> I guess one other possible concern might be storage:
>>>
>>> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
>>> 5.4M	/etc/selinux/targeted/modules
>>> 11M	/var/lib/selinux/
>>>
>>> I'm guessing that is just the cost of storing each module in both binary
>>> and cil form?
>> Yep, we store both HLL and CIL. They are both compressed and CIL
>> compresses decently since it's just text, but it's still a lot of text.
>>
>>> Is there an option to discard the .pp files altogether and only retain
>>> the cil files?
>> Not at the moment, but it wouldn't be hard to accomplish. Just need to
>> delete all the hll files and change the contents of lang_ext to 'cil'.
>> Something we could add if storage is an issue.
> That worked, albeit I had to learn that lang_ext must not include a
> newline or libsemanage won't accept it.  Took it down to 5.8M.  I
> suspect we could also stop retaining a copy of certain generated files
> like file_contexts although that is no different than the current code.
>
> Not sure what benefit there is in retaining the pp files, since they
> carry no additional information AFAIK and they aren't human viewable or
> editable.  Is there even an option for exporting modules from the policy
> store currently that would allow extracting them except via direct file
> access to the policy store?
>
> More generally, if the user knows that the hll module is going to be
> saved elsewhere, then there is no reason to retain a copy in the policy
> store, so having the option of dropping the hll version, either for all
> modules or for specific modules, seems useful.
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>
>
BTW Moving the policy out of /etc is also great.  I would like to see
policy loaded from
/var/lib/selinux/ if it exists and then look in /usr/lib/selinux/ if it
does not.  Then distributions could
ship their own policies, and if a user wanted to get back to Factory
install he could just rm -rf /var/lib/selinux

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/17/2014 04:52 PM, Daniel J Walsh wrote:
> 
> On 07/17/2014 04:37 PM, Stephen Smalley wrote:
>> On 07/17/2014 04:04 PM, Steve Lawrence wrote:
>>> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>>>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>>>>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>>>>> I think the only remaining issue is the one Dominick mentioned in his
>>>>>> first email regarding file_contexts.homedirs. I don't think this is an
>>>>>> actual bug, just the migration script migrating things that don't need
>>>>>> to be migrated. Still investigating it. We should have an update
>>>>>> sometime tomorrow.
>>>>> So everything you reverted you restored in equivalent form?
>>>>>
>>>>>>> What new functionality is included here that was not previously
>>>>>>> supported by the old policy toolchain?
>>>>>> In terms a user would see, the most visible change is support for CIL
>>>>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>>>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>>>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>>>>> people could just use the defaults. Similarly, we've added
>>>>>> --ignore-module-cache and --store-root to the semodule command. We've
>>>>>> also moved the store to /var/lib/selinux, but this is more behind the
>>>>>> scenes and should really only affect distributions.
>>>>> What about new features/options of the user-facing commands?  I know
>>>>> some features were copied from earlier source/CIL releases into the main
>>>>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>>>>> things like module priorities new?
>>>>>
>>>>>> Though, there are two things we just realized have a different behavior.
>>>>>>
>>>>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>>>>> (or HLL) modules. So if someone is using verify_modules, things will
>>>>>> probably break. I'm not sure if anyone uses this feature or how
>>>>>> important it is that we maintain backwards compatibility.
>>>>>>
>>>>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>>>>> linked base module with CIL
>>>>>>
>>>>>> Aside from that, I think all functionality should remain the same.
>>>>> I'm not aware of anyone using anything other than verify kernel.
>>>>>
>>>>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>>>>> in .if/.te/.fc form?
>>>>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>>>>> but the current HLL infrastructure may need some changes before that can
>>>>>> be supported. I think the main problem is that Jim's tool needs
>>>>>> knowledge of all modules to be able to convert them to CIL, but the
>>>>>> current HLL infrastructure compiles each module separately. We have
>>>>>> various ideas on how we can update the HLL infrastructure to support
>>>>>> this, but we've primarily been focused on getting the core CIL/HLL
>>>>>> functionality complete and upstreamed before focusing on the more
>>>>>> complicated HLL patterns.
>>>>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>>>>> re-tooled to generate source modules, and we'll essentially need a
>>>>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>>>>> mymodule.pp; semodule -i mymodule.pp.
>>>> I guess one other possible concern might be storage:
>>>>
>>>> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
>>>> 5.4M	/etc/selinux/targeted/modules
>>>> 11M	/var/lib/selinux/
>>>>
>>>> I'm guessing that is just the cost of storing each module in both binary
>>>> and cil form?
>>> Yep, we store both HLL and CIL. They are both compressed and CIL
>>> compresses decently since it's just text, but it's still a lot of text.
>>>
>>>> Is there an option to discard the .pp files altogether and only retain
>>>> the cil files?
>>> Not at the moment, but it wouldn't be hard to accomplish. Just need to
>>> delete all the hll files and change the contents of lang_ext to 'cil'.
>>> Something we could add if storage is an issue.
>> That worked, albeit I had to learn that lang_ext must not include a
>> newline or libsemanage won't accept it.  Took it down to 5.8M.  I
>> suspect we could also stop retaining a copy of certain generated files
>> like file_contexts although that is no different than the current code.
>>
>> Not sure what benefit there is in retaining the pp files, since they
>> carry no additional information AFAIK and they aren't human viewable or
>> editable.  Is there even an option for exporting modules from the policy
>> store currently that would allow extracting them except via direct file
>> access to the policy store?
>>
>> More generally, if the user knows that the hll module is going to be
>> saved elsewhere, then there is no reason to retain a copy in the policy
>> store, so having the option of dropping the hll version, either for all
>> modules or for specific modules, seems useful.
>> _______________________________________________
>> Selinux mailing list
>> Selinux@tycho.nsa.gov
>> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
>> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>>
>>
> BTW Moving the policy out of /etc is also great.  I would like to see
> policy loaded from
> /var/lib/selinux/ if it exists and then look in /usr/lib/selinux/ if it
> does not.  Then distributions could
> ship their own policies, and if a user wanted to get back to Factory
> install he could just rm -rf /var/lib/selinux

At present, they still install the final policy files to
/etc/selinux/targeted since that is where libselinux expects to find
them for loading; only the policy module store managed by libsemanage is
moved from /etc/selinux/targeted/modules to /var/lib/selinux/targeted.
If you wanted to relocate even the final policy files, we'd have to
update libselinux to also look under /var/lib/selinux.  Also, there
might be a problem with relocating to /var for the final policy files
since it can be a separate partition and might not always be mounted
when systemd tries to load policy.

In the case of Android, we have the initial set of policy files in /
(part of the rootfs from the initramfs), and optionally you can place
your own policy under /data/security.  init always loads first from the
policy files in the rootfs and then later reloads from the
/data/security policy files if present after mounting /data.  Factory
reset wipes /data and restores to the original rootfs policy.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Daniel J Walsh]

On 07/23/2014 03:24 PM, Stephen Smalley wrote:
> On 07/17/2014 04:52 PM, Daniel J Walsh wrote:
>> On 07/17/2014 04:37 PM, Stephen Smalley wrote:
>>> On 07/17/2014 04:04 PM, Steve Lawrence wrote:
>>>> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>>>>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>>>>>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>>>>>> I think the only remaining issue is the one Dominick mentioned in his
>>>>>>> first email regarding file_contexts.homedirs. I don't think this is an
>>>>>>> actual bug, just the migration script migrating things that don't need
>>>>>>> to be migrated. Still investigating it. We should have an update
>>>>>>> sometime tomorrow.
>>>>>> So everything you reverted you restored in equivalent form?
>>>>>>
>>>>>>>> What new functionality is included here that was not previously
>>>>>>>> supported by the old policy toolchain?
>>>>>>> In terms a user would see, the most visible change is support for CIL
>>>>>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>>>>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>>>>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>>>>>> people could just use the defaults. Similarly, we've added
>>>>>>> --ignore-module-cache and --store-root to the semodule command. We've
>>>>>>> also moved the store to /var/lib/selinux, but this is more behind the
>>>>>>> scenes and should really only affect distributions.
>>>>>> What about new features/options of the user-facing commands?  I know
>>>>>> some features were copied from earlier source/CIL releases into the main
>>>>>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>>>>>> things like module priorities new?
>>>>>>
>>>>>>> Though, there are two things we just realized have a different behavior.
>>>>>>>
>>>>>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>>>>>> (or HLL) modules. So if someone is using verify_modules, things will
>>>>>>> probably break. I'm not sure if anyone uses this feature or how
>>>>>>> important it is that we maintain backwards compatibility.
>>>>>>>
>>>>>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>>>>>> linked base module with CIL
>>>>>>>
>>>>>>> Aside from that, I think all functionality should remain the same.
>>>>>> I'm not aware of anyone using anything other than verify kernel.
>>>>>>
>>>>>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>>>>>> in .if/.te/.fc form?
>>>>>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>>>>>> but the current HLL infrastructure may need some changes before that can
>>>>>>> be supported. I think the main problem is that Jim's tool needs
>>>>>>> knowledge of all modules to be able to convert them to CIL, but the
>>>>>>> current HLL infrastructure compiles each module separately. We have
>>>>>>> various ideas on how we can update the HLL infrastructure to support
>>>>>>> this, but we've primarily been focused on getting the core CIL/HLL
>>>>>>> functionality complete and upstreamed before focusing on the more
>>>>>>> complicated HLL patterns.
>>>>>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>>>>>> re-tooled to generate source modules, and we'll essentially need a
>>>>>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>>>>>> mymodule.pp; semodule -i mymodule.pp.
>>>>> I guess one other possible concern might be storage:
>>>>>
>>>>> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
>>>>> 5.4M	/etc/selinux/targeted/modules
>>>>> 11M	/var/lib/selinux/
>>>>>
>>>>> I'm guessing that is just the cost of storing each module in both binary
>>>>> and cil form?
>>>> Yep, we store both HLL and CIL. They are both compressed and CIL
>>>> compresses decently since it's just text, but it's still a lot of text.
>>>>
>>>>> Is there an option to discard the .pp files altogether and only retain
>>>>> the cil files?
>>>> Not at the moment, but it wouldn't be hard to accomplish. Just need to
>>>> delete all the hll files and change the contents of lang_ext to 'cil'.
>>>> Something we could add if storage is an issue.
>>> That worked, albeit I had to learn that lang_ext must not include a
>>> newline or libsemanage won't accept it.  Took it down to 5.8M.  I
>>> suspect we could also stop retaining a copy of certain generated files
>>> like file_contexts although that is no different than the current code.
>>>
>>> Not sure what benefit there is in retaining the pp files, since they
>>> carry no additional information AFAIK and they aren't human viewable or
>>> editable.  Is there even an option for exporting modules from the policy
>>> store currently that would allow extracting them except via direct file
>>> access to the policy store?
>>>
>>> More generally, if the user knows that the hll module is going to be
>>> saved elsewhere, then there is no reason to retain a copy in the policy
>>> store, so having the option of dropping the hll version, either for all
>>> modules or for specific modules, seems useful.
>>> _______________________________________________
>>> Selinux mailing list
>>> Selinux@tycho.nsa.gov
>>> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
>>> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>>>
>>>
>> BTW Moving the policy out of /etc is also great.  I would like to see
>> policy loaded from
>> /var/lib/selinux/ if it exists and then look in /usr/lib/selinux/ if it
>> does not.  Then distributions could
>> ship their own policies, and if a user wanted to get back to Factory
>> install he could just rm -rf /var/lib/selinux
> At present, they still install the final policy files to
> /etc/selinux/targeted since that is where libselinux expects to find
> them for loading; only the policy module store managed by libsemanage is
> moved from /etc/selinux/targeted/modules to /var/lib/selinux/targeted.
> If you wanted to relocate even the final policy files, we'd have to
> update libselinux to also look under /var/lib/selinux.  Also, there
> might be a problem with relocating to /var for the final policy files
> since it can be a separate partition and might not always be mounted
> when systemd tries to load policy.
>
> In the case of Android, we have the initial set of policy files in /
> (part of the rootfs from the initramfs), and optionally you can place
> your own policy under /data/security.  init always loads first from the
> policy files in the rootfs and then later reloads from the
> /data/security policy files if present after mounting /data.  Factory
> reset wipes /data and restores to the original rootfs policy.
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>
>
Ok lets stick with /etc and /usr/lib/selinux.  Distributions could ship
policy and config in /usr/lib/selinux and then any customization of 
policy should continue to go into /etc/selinux.  Then libselinux should
look in /etc/selinux/ first and fail over to /usr/lib/selinux.

One of the things the systemd guys are working on is a "Reset" button
where a  user could remove /var and /etc and still boot with system
defaults.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/17/2014 04:37 PM, Stephen Smalley wrote:
> On 07/17/2014 04:04 PM, Steve Lawrence wrote:
>> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>>>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>>>> I think the only remaining issue is the one Dominick mentioned in his
>>>>> first email regarding file_contexts.homedirs. I don't think this is an
>>>>> actual bug, just the migration script migrating things that don't need
>>>>> to be migrated. Still investigating it. We should have an update
>>>>> sometime tomorrow.
>>>>
>>>> So everything you reverted you restored in equivalent form?
>>>>
>>>>>> What new functionality is included here that was not previously
>>>>>> supported by the old policy toolchain?
>>>>>
>>>>> In terms a user would see, the most visible change is support for CIL
>>>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>>>> people could just use the defaults. Similarly, we've added
>>>>> --ignore-module-cache and --store-root to the semodule command. We've
>>>>> also moved the store to /var/lib/selinux, but this is more behind the
>>>>> scenes and should really only affect distributions.
>>>>
>>>> What about new features/options of the user-facing commands?  I know
>>>> some features were copied from earlier source/CIL releases into the main
>>>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>>>> things like module priorities new?
>>>>
>>>>> Though, there are two things we just realized have a different behavior.
>>>>>
>>>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>>>> (or HLL) modules. So if someone is using verify_modules, things will
>>>>> probably break. I'm not sure if anyone uses this feature or how
>>>>> important it is that we maintain backwards compatibility.
>>>>>
>>>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>>>> linked base module with CIL
>>>>>
>>>>> Aside from that, I think all functionality should remain the same.
>>>>
>>>> I'm not aware of anyone using anything other than verify kernel.
>>>>
>>>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>>>> in .if/.te/.fc form?
>>>>>
>>>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>>>> but the current HLL infrastructure may need some changes before that can
>>>>> be supported. I think the main problem is that Jim's tool needs
>>>>> knowledge of all modules to be able to convert them to CIL, but the
>>>>> current HLL infrastructure compiles each module separately. We have
>>>>> various ideas on how we can update the HLL infrastructure to support
>>>>> this, but we've primarily been focused on getting the core CIL/HLL
>>>>> functionality complete and upstreamed before focusing on the more
>>>>> complicated HLL patterns.
>>>>
>>>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>>>> re-tooled to generate source modules, and we'll essentially need a
>>>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>>>> mymodule.pp; semodule -i mymodule.pp.
>>>
>>> I guess one other possible concern might be storage:
>>>
>>> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
>>> 5.4M	/etc/selinux/targeted/modules
>>> 11M	/var/lib/selinux/
>>>
>>> I'm guessing that is just the cost of storing each module in both binary
>>> and cil form?
>>
>> Yep, we store both HLL and CIL. They are both compressed and CIL
>> compresses decently since it's just text, but it's still a lot of text.
>>
>>> Is there an option to discard the .pp files altogether and only retain
>>> the cil files?
>>
>> Not at the moment, but it wouldn't be hard to accomplish. Just need to
>> delete all the hll files and change the contents of lang_ext to 'cil'.
>> Something we could add if storage is an issue.
> 
> That worked, albeit I had to learn that lang_ext must not include a
> newline or libsemanage won't accept it.  Took it down to 5.8M.  I
> suspect we could also stop retaining a copy of certain generated files
> like file_contexts although that is no different than the current code.

I think the new policy store infrastructure actually doesn't store a
copy of generated files, like file_contexts. It just doesn't appear that
way because the migration script copies things from the old store that
it doesn't need to. But they are never actually used. Once I verify this
is correct, I'll have a patch that should fix the migration script.

> Not sure what benefit there is in retaining the pp files, since they
> carry no additional information AFAIK and they aren't human viewable or
> editable.  Is there even an option for exporting modules from the policy
> store currently that would allow extracting them except via direct file
> access to the policy store?

There isn't currently a way to extract policy from the store, but that
has been a use case that has been discussed in the past. Something like
the following could be useful:

semodule --priority 400 --extract foo # outputs to foo.<hll>
edit foo.<hll>
semodule --priority 500 --install foo.<hll>

So there could be cases in the future where it could be convenient to
keep the HLL files around.

Another option to reduce disk usage could be to disable caching of the
CIL files (right now, we only have an option to ignore the cached
files). This way, the user could still do the above and edit hll files
without having to rely on them being accessible from somewhere else.
Though, this would incur a penalty of having to recompile HLL files for
every change (which in my tests, about doubles compilation time).

> More generally, if the user knows that the hll module is going to be
> saved elsewhere, then there is no reason to retain a copy in the policy
> store, so having the option of dropping the hll version, either for all
> modules or for specific modules, seems useful.

Do you see this feature as necessary for this patchset to be upstreamed,
or is this something we could add as a later update?

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/18/2014 08:59 AM, Steve Lawrence wrote:
> There isn't currently a way to extract policy from the store, but that
> has been a use case that has been discussed in the past. Something like
> the following could be useful:
> 
> semodule --priority 400 --extract foo # outputs to foo.<hll>
> edit foo.<hll>
> semodule --priority 500 --install foo.<hll>
> 
> So there could be cases in the future where it could be convenient to
> keep the HLL files around.

Sure, for source modules.  For pp files, though?

> Another option to reduce disk usage could be to disable caching of the
> CIL files (right now, we only have an option to ignore the cached
> files). This way, the user could still do the above and edit hll files
> without having to rely on them being accessible from somewhere else.
> Though, this would incur a penalty of having to recompile HLL files for
> every change (which in my tests, about doubles compilation time).

Seems less useful than being able to disable storage of the HLL files.
No penalty incurred, you just have to know that you won't need them
again or that they will remain available externally.

>> More generally, if the user knows that the hll module is going to be
>> saved elsewhere, then there is no reason to retain a copy in the policy
>> store, so having the option of dropping the hll version, either for all
>> modules or for specific modules, seems useful.
> 
> Do you see this feature as necessary for this patchset to be upstreamed,
> or is this something we could add as a later update?

Not absolutely required, just wondering if we might encounter issues
with migration on some systems due to insufficient space.  We'll
actually have 3 copies of each module, 2 pp files (one still in
/etc/selinux/targeted/modules, not deleted by the migration script) and
1 cil file.  Plus all of the copying that occurs during the transaction
and installation of the files (-> /var/lib/selinux/targeted/tmp, ->
/var/lib/selinux/tmp/targeted?, -> /etc/selinux/targeted).  Any idea
what the max storage requirement for successful migration relative to
the original size of the policy store?

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/18/2014 10:30 AM, Stephen Smalley wrote:
> On 07/18/2014 08:59 AM, Steve Lawrence wrote:
>> There isn't currently a way to extract policy from the store, but that
>> has been a use case that has been discussed in the past. Something like
>> the following could be useful:
>>
>> semodule --priority 400 --extract foo # outputs to foo.<hll>
>> edit foo.<hll>
>> semodule --priority 500 --install foo.<hll>
>>
>> So there could be cases in the future where it could be convenient to
>> keep the HLL files around.
> 
> Sure, for source modules.  For pp files, though?
> 

Yeah, not useful for pp files. But I'd like to see pp files eventually
go away, with the source policies (which are likely much smaller) to be
the primary HLL.

Though, as Chris pointed out, extracting policies from the store could
potentially be useful for analysis.

>> Another option to reduce disk usage could be to disable caching of the
>> CIL files (right now, we only have an option to ignore the cached
>> files). This way, the user could still do the above and edit hll files
>> without having to rely on them being accessible from somewhere else.
>> Though, this would incur a penalty of having to recompile HLL files for
>> every change (which in my tests, about doubles compilation time).
> 
> Seems less useful than being able to disable storage of the HLL files.
> No penalty incurred, you just have to know that you won't need them
> again or that they will remain available externally.

Agreed, just throwing it out as another option.

>>> More generally, if the user knows that the hll module is going to be
>>> saved elsewhere, then there is no reason to retain a copy in the policy
>>> store, so having the option of dropping the hll version, either for all
>>> modules or for specific modules, seems useful.
>>
>> Do you see this feature as necessary for this patchset to be upstreamed,
>> or is this something we could add as a later update?
> 
> Not absolutely required, just wondering if we might encounter issues
> with migration on some systems due to insufficient space.  We'll
> actually have 3 copies of each module, 2 pp files (one still in
> /etc/selinux/targeted/modules, not deleted by the migration script) and
> 1 cil file.  Plus all of the copying that occurs during the transaction
> and installation of the files (-> /var/lib/selinux/targeted/tmp, ->
> /var/lib/selinux/tmp/targeted?, -> /etc/selinux/targeted).  Any idea
> what the max storage requirement for successful migration relative to
> the original size of the policy store?

We could delete the old store post migration but pre-rebuild. So we'd
only ever have two copies of the modules. We held off on doing this in
case something went wrong. Maybe after the migration script is run
through its paces we could be more confident and not require a backup.
Or we could delete the old store after a successful migration, so you
only take the extra storage hit on migration.

As far as max storage requirement, the current policy store is about
5MB. The CIL versions also required about 5MB total. So /var/lib/selinux
needs about 10MB to store all the policies (CIL + pp), and grows to
about 25MB during building. Makes sense since it makes a copy of the
active store (doubling from 10MB to 20MB), then builds into the sandbox
(~5MB). Then it deletes the sandbox and the old module store.

So you need about 5x the current policy to successfully migrate.

So we could probably save about 5MB if we deleted the old store after
migration, and save another 10MB during the build process if pp modules
were deleted after converted to CIL.

Re: [RFC] Source Policy, CIL, and High Level Languages

[James Carter]

On 07/18/2014 08:59 AM, Steve Lawrence wrote:
> On 07/17/2014 04:37 PM, Stephen Smalley wrote:
>> On 07/17/2014 04:04 PM, Steve Lawrence wrote:
>>> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>>>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:
>>>>> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>>>>>> I think the only remaining issue is the one Dominick mentioned in his
>>>>>> first email regarding file_contexts.homedirs. I don't think this is an
>>>>>> actual bug, just the migration script migrating things that don't need
>>>>>> to be migrated. Still investigating it. We should have an update
>>>>>> sometime tomorrow.
>>>>>
>>>>> So everything you reverted you restored in equivalent form?
>>>>>
>>>>>>> What new functionality is included here that was not previously
>>>>>>> supported by the old policy toolchain?
>>>>>>
>>>>>> In terms a user would see, the most visible change is support for CIL
>>>>>> policies and HLLs, of which there's only one right now (pp2cil). There
>>>>>> are also some new semanage.conf options (target-platform, compiler-dir,
>>>>>> ignore-module-cache, store-root) but I imagine the vast majority of
>>>>>> people could just use the defaults. Similarly, we've added
>>>>>> --ignore-module-cache and --store-root to the semodule command. We've
>>>>>> also moved the store to /var/lib/selinux, but this is more behind the
>>>>>> scenes and should really only affect distributions.
>>>>>
>>>>> What about new features/options of the user-facing commands?  I know
>>>>> some features were copied from earlier source/CIL releases into the main
>>>>> selinux userspace (e.g. enabled/disabled modules), but aren't some
>>>>> things like module priorities new?
>>>>>
>>>>>> Though, there are two things we just realized have a different behavior.
>>>>>>
>>>>>> 1) verify_modules is now performed on the CIL modules, rather than pp
>>>>>> (or HLL) modules. So if someone is using verify_modules, things will
>>>>>> probably break. I'm not sure if anyone uses this feature or how
>>>>>> important it is that we maintain backwards compatibility.
>>>>>>
>>>>>> 2) verify_linked is no longer called, since there isn't any concept of a
>>>>>> linked base module with CIL
>>>>>>
>>>>>> Aside from that, I think all functionality should remain the same.
>>>>>
>>>>> I'm not aware of anyone using anything other than verify kernel.
>>>>>
>>>>>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>>>>>> in .if/.te/.fc form?
>>>>>>
>>>>>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>>>>>> but the current HLL infrastructure may need some changes before that can
>>>>>> be supported. I think the main problem is that Jim's tool needs
>>>>>> knowledge of all modules to be able to convert them to CIL, but the
>>>>>> current HLL infrastructure compiles each module separately. We have
>>>>>> various ideas on how we can update the HLL infrastructure to support
>>>>>> this, but we've primarily been focused on getting the core CIL/HLL
>>>>>> functionality complete and upstreamed before focusing on the more
>>>>>> complicated HLL patterns.
>>>>>
>>>>> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
>>>>> re-tooled to generate source modules, and we'll essentially need a
>>>>> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
>>>>> mymodule.pp; semodule -i mymodule.pp.
>>>>
>>>> I guess one other possible concern might be storage:
>>>>
>>>> $ du -sh /etc/selinux/targeted/modules /var/lib/selinux/
>>>> 5.4M	/etc/selinux/targeted/modules
>>>> 11M	/var/lib/selinux/
>>>>
>>>> I'm guessing that is just the cost of storing each module in both binary
>>>> and cil form?
>>>
>>> Yep, we store both HLL and CIL. They are both compressed and CIL
>>> compresses decently since it's just text, but it's still a lot of text.
>>>
>>>> Is there an option to discard the .pp files altogether and only retain
>>>> the cil files?
>>>
>>> Not at the moment, but it wouldn't be hard to accomplish. Just need to
>>> delete all the hll files and change the contents of lang_ext to 'cil'.
>>> Something we could add if storage is an issue.
>>
>> That worked, albeit I had to learn that lang_ext must not include a
>> newline or libsemanage won't accept it.  Took it down to 5.8M.  I
>> suspect we could also stop retaining a copy of certain generated files
>> like file_contexts although that is no different than the current code.
>
> I think the new policy store infrastructure actually doesn't store a
> copy of generated files, like file_contexts. It just doesn't appear that
> way because the migration script copies things from the old store that
> it doesn't need to. But they are never actually used. Once I verify this
> is correct, I'll have a patch that should fix the migration script.
>
>> Not sure what benefit there is in retaining the pp files, since they
>> carry no additional information AFAIK and they aren't human viewable or
>> editable.  Is there even an option for exporting modules from the policy
>> store currently that would allow extracting them except via direct file
>> access to the policy store?
>
> There isn't currently a way to extract policy from the store, but that
> has been a use case that has been discussed in the past. Something like
> the following could be useful:
>
> semodule --priority 400 --extract foo # outputs to foo.<hll>
> edit foo.<hll>
> semodule --priority 500 --install foo.<hll>
>

This is something that we will definitely want to support. This is how I see 
people modifying the distro's policy.

A couple of other nice to have operations would be:
1) Extract all policy higher than a certain priority into a tarball, so that an 
admin can easily move their customizations to other machines.
2) Getting the diff between two priority levels of a module.

Of course, once you can extract the source policy these other operations aren't 
too hard to do.

> So there could be cases in the future where it could be convenient to
> keep the HLL files around.
>
> Another option to reduce disk usage could be to disable caching of the
> CIL files (right now, we only have an option to ignore the cached
> files). This way, the user could still do the above and edit hll files
> without having to rely on them being accessible from somewhere else.
> Though, this would incur a penalty of having to recompile HLL files for
> every change (which in my tests, about doubles compilation time).
>
>> More generally, if the user knows that the hll module is going to be
>> saved elsewhere, then there is no reason to retain a copy in the policy
>> store, so having the option of dropping the hll version, either for all
>> modules or for specific modules, seems useful.
>
> Do you see this feature as necessary for this patchset to be upstreamed,
> or is this something we could add as a later update?
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>


-- 
James Carter <jwcart2@tycho.nsa.gov>
National Security Agency

Re: [RFC] Source Policy, CIL, and High Level Languages

[Christopher J. PeBenito]

On 7/17/2014 4:37 PM, Stephen Smalley wrote:
> On 07/17/2014 04:04 PM, Steve Lawrence wrote:
>> On 07/17/2014 03:48 PM, Stephen Smalley wrote:
>>> On 07/17/2014 03:10 PM, Stephen Smalley wrote:

>> Yep, we store both HLL and CIL. They are both compressed and CIL
>> compresses decently since it's just text, but it's still a lot of text.
>>
>>> Is there an option to discard the .pp files altogether and only retain
>>> the cil files?
>>
>> Not at the moment, but it wouldn't be hard to accomplish. Just need to
>> delete all the hll files and change the contents of lang_ext to 'cil'.
>> Something we could add if storage is an issue.
[...]
> Not sure what benefit there is in retaining the pp files, since they
> carry no additional information AFAIK and they aren't human viewable or
> editable.  Is there even an option for exporting modules from the policy
> store currently that would allow extracting them except via direct file
> access to the policy store?

If we eventually have an upstream policy query library, then it could be
useful to export modules from the store for analysis tools.  Then you
could analyze the policy.29 but potentially have backtracing to the HLL
policy file and ideally source line where possible (obviously not for .pp).

-- 
Chris PeBenito
Tresys Technology, LLC
www.tresys.com | oss.tresys.com

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/17/2014 03:10 PM, Stephen Smalley wrote:
> On 07/17/2014 02:58 PM, Steve Lawrence wrote:
>> I think the only remaining issue is the one Dominick mentioned in his
>> first email regarding file_contexts.homedirs. I don't think this is an
>> actual bug, just the migration script migrating things that don't need
>> to be migrated. Still investigating it. We should have an update
>> sometime tomorrow.
> 
> So everything you reverted you restored in equivalent form?
> 

Yep. The only features reverted were enable/disable and
semanage_set_root (multiple commits added these features, which is why
there are 8 commits). Enable/disable as added back by the source policy
work, and setmanage_set_root was manually added back.

>>> What new functionality is included here that was not previously
>>> supported by the old policy toolchain?
>>
>> In terms a user would see, the most visible change is support for CIL
>> policies and HLLs, of which there's only one right now (pp2cil). There
>> are also some new semanage.conf options (target-platform, compiler-dir,
>> ignore-module-cache, store-root) but I imagine the vast majority of
>> people could just use the defaults. Similarly, we've added
>> --ignore-module-cache and --store-root to the semodule command. We've
>> also moved the store to /var/lib/selinux, but this is more behind the
>> scenes and should really only affect distributions.
> 
> What about new features/options of the user-facing commands?  I know
> some features were copied from earlier source/CIL releases into the main
> selinux userspace (e.g. enabled/disabled modules), but aren't some
> things like module priorities new?

Yes. The changes to semodule were:

New option -X, --priority to set the priority, defaults to 400.
New option -C, --ignore_module_cache to ignore cached CIL modules to
force recompilation.
Deprecated --base and --upgrade, which are both equivalent to --install.
Versions are no longer output in semodule --list (they don't exist in CIL).
The -l option can now accept a parameter (either "full" or "standard").
"standard" lists just active modules. This is the default if a parameter
is not provided. "full" lists all modules, priorities, and high level
language.

Changes to semanage:
- Add support for priorities with the module subcommand with the -X option
- Removes version references
- Modifies list to also output priority and hll
- Making a module permissive uses CIL instead of building a pp module



>> Though, there are two things we just realized have a different behavior.
>>
>> 1) verify_modules is now performed on the CIL modules, rather than pp
>> (or HLL) modules. So if someone is using verify_modules, things will
>> probably break. I'm not sure if anyone uses this feature or how
>> important it is that we maintain backwards compatibility.
>>
>> 2) verify_linked is no longer called, since there isn't any concept of a
>> linked base module with CIL
>>
>> Aside from that, I think all functionality should remain the same.
> 
> I'm not aware of anyone using anything other than verify kernel.
> 
>>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>>> in .if/.te/.fc form?
>>
>> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
>> but the current HLL infrastructure may need some changes before that can
>> be supported. I think the main problem is that Jim's tool needs
>> knowledge of all modules to be able to convert them to CIL, but the
>> current HLL infrastructure compiles each module separately. We have
>> various ideas on how we can update the HLL infrastructure to support
>> this, but we've primarily been focused on getting the core CIL/HLL
>> functionality complete and upstreamed before focusing on the more
>> complicated HLL patterns.
> 
> Ok.  Ultimately audit2allow -M i.e. sepolgen module compiler should be
> re-tooled to generate source modules, and we'll essentially need a
> workflow that replaces the old make -f /usr/share/selinux/devel/Makefile
> mymodule.pp; semodule -i mymodule.pp.

Re: [RFC] Source Policy, CIL, and High Level Languages

[James Carter]

On 07/17/2014 02:58 PM, Steve Lawrence wrote:
> On 07/17/2014 02:02 PM, Stephen Smalley wrote:
>> On 07/17/2014 09:49 AM, Steve Lawrence wrote:
>>> On 07/16/2014 03:00 PM, Stephen Smalley wrote:
>>>> On 07/16/2014 11:53 AM, Dominick Grift wrote:
>>>>> On Wed, 2014-07-16 at 11:11 -0400, Steve Lawrence wrote:
>>>>> <snip>

>
>> Any chance of getting a hll compiler for refpolicy source modules, i.e.
>> in .if/.te/.fc form?
>
> That's in the plan. Jim has a tool that will compile .if/.te/.fc to CIL,
> but the current HLL infrastructure may need some changes before that can
> be supported. I think the main problem is that Jim's tool needs
> knowledge of all modules to be able to convert them to CIL, but the
> current HLL infrastructure compiles each module separately. We have
> various ideas on how we can update the HLL infrastructure to support
> this, but we've primarily been focused on getting the core CIL/HLL
> functionality complete and upstreamed before focusing on the more
> complicated HLL patterns.

My tool currently does need to have knowledge of all modules, but I think that I 
can get it to work with the information in /usr/share/selinux/devel/include. At 
least as long as the module is not doing anything crazy with m4.

-- 
James Carter <jwcart2@tycho.nsa.gov>
National Security Agency

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/15/2014 03:56 PM, Steve Lawrence wrote:
> On 07/14/2014 01:49 PM, Stephen Smalley wrote:
>> On 07/14/2014 01:12 PM, Steve Lawrence wrote:
>>> Ah, interesting. We saw that problem a long time ago, but couldn't
>>> reproduce it and it disappeared. Though I'm still unable to reproduce it
>>> following your steps. I can still login and seusers is labeled
>>> selinux_config_t. I'll keep looking into this.
>>>
>>> I've also rebased/pushed #integration onto #next.
>>
>> # Revert to stock F20 SELinux userspace and policy.
>> yum reinstall checkpolicy* libsepol* libsemanage* libselinux*
>> policycoreutils* selinux-policy-targeted
>>
>> # Clear prior source/CIL policy store.
>> rm -rf /var/lib/selinux
>>
>> # Reboot to ensure systemd and friends are using the new policy.
>> reboot
>>
>> # Reset selinux and cil to latest sources
>> cd selinux
>> git clean -fdx
>> git fetch origin
>> git reset --hard origin/integration
>> cd ../cil
>> git clean -fdx
>> git fetch origin
>> git reset --hard origin/master
>>
>> # Build and install new userspace
>> cd ..
>> ln -sf ../../cil selinux/libsepol/cil
>> make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>> relabel
>>
>> # Convert
>>  ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>
>> Try to login on console or via ssh:  Unable to get valid context for sds.
>>
>> dmesg | grep systemd
>> [  343.739985] systemd[1]: SELinux policy denies access.
>> [  348.256030] systemd[1]: SELinux policy denies access.
>> [  376.335248] systemd[1]: SELinux policy denies access.
>> [  376.515343] systemd[1]: SELinux policy denies access.
>>
>> restorecon -R /etc/selinux/targeted
>>
>> Try to login again, hangs for a long time before finally succeeding.
>>
>> reboot
>>
>> Everything is happy.
>>
>> 100% reproducible, every time.
>>
>>
> 
> Thanks for the steps. I think I found what causes the labeling problem,
> but I'm not yet sure why. It looks like it has something to do with how
> the migration script rebuilds policy. If you run the migration script
> with the --norebuild flag, and then run semodule -B, everything is
> rebuilt and the files in /etc/selinux are labeled correctly.
> 
> I'm not yet convinced this is the same problem as the systemd issue (I
> do get "unable to get valid context", but I still don't get "SELinux
> policy denies access"). Let me know if you still see that after using
> --norebuild/semodule -B.

This issue has been fixed and rebased/pushed to the #integration branch.
The problem was that the migration script called setfscreatecon to
ensure file contexts in the new store were correct. However, it never
reset the fs create context, so when it rebuilt policy after migration,
all files were created with semanage_store_t instead of using default
labeling rules/transitions.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Mon, 2014-07-14 at 13:08 -0400, Stephen Smalley wrote:
> On 07/14/2014 12:53 PM, Stephen Smalley wrote:
> > On 07/14/2014 12:48 PM, Stephen Smalley wrote:
> >> On 07/11/2014 01:20 PM, Steve Lawrence wrote:
> >>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
> >>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
> >>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
> >>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
> >>>>> added support for high level languages and a tool to convert policy
> >>>>> package (pp) files to CIL. After getting some good feedback, we have
> >>>>> made some more changes, mostly to maintain ABI compatibility. The
> >>>>> major changes made since the last patchset are:
> >>>>
> >>>> <snip>
> >>>>
> >>>> I just spent a few hours playing with this and i am impressed.
> >>>>
> >>>> Everything i tested just works.
> >>>>
> >>>> What did i test?
> >>>>
> >>>> 1. disabling/enabling existing modules
> >>>> 2. toggling booleans with semanage
> >>>> 3. adding and removing port and file contexts with semanage
> >>>> 4. adding/removing a policy module with semodule, checkmodule,
> >>>> semodule_package
> >>>> 5. adding/removing a (cil) policy module with semodule
> >>>> 6. associating a (new) user with staff_t identity
> >>>>
> >>>> Comments?
> >>>>
> >>>> if i do restorecon -R -v -F /home it resets contexts *every* time (from
> >>>> s0 to s0-s0). No noticable side effects because of this
> >>>>
> >>>
> >>> We recently pushed a fix to CIL that fixes the issue with how CIL
> >>> generates file contexts. It now removes the high level if it is the same
> >>> as the low level.
> >>
> >> So, if I revert my system to stock F20 (yum reinstall checkpolicy*
> >> libsepol* libsemanage* libselinux* policycoreutils*
> >> selinux-policy-targeted) , then re-install from the integration branch
> >> as per your instructions and run the migration script, then attempt to
> >> ssh into the system, sshd says "Unable to get valid context for sds" and
> >> the connection is closed.  dmesg shows:
> >> systemd[1]: SELinux policy denies access.
> >>
> >> Can you merge #next to #integration so we get the more detailed
> >> information on unknown classes/perms?
> >>
> >> I'm guessing a reboot will clear the problem again (since systemd will
> >> then remap the class from name to value at boot against the current
> >> policy).  But ideally this wouldn't be necessary.
> > 
> > Hmmm....a reboot did not clear it.  All logins, local or remote,
> > disabled for non-root.
> 
> This seems to be a labeling problem; saw denials on attempts to read
> seusers with semanage_store_t.  But restorecon -Rv /etc/selinux/targeted
> restored it to selinux_config_t.
> 

In my experience this is probably not related to the event "systemd[1]:
SELinux policy denied access."

I may be wrong but I do not believe systemd prints those messages if it
is not directly related to systemd.

SSHD (and other login programs) might be (trying to) dbus chat to
systemd logind, and systemd logind needs to be able to "start" the
session manager. (although that should not prevent the login)

So i think there are two issue here. The labeling issue, which prevents
the login, and the issue that causes systemd to print "systemd[1]:
SELinux denied access."

I might be wrong...

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 02:51 AM, Dominick Grift wrote:
> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
> 
> <snip>
> 
> 
> After associating user john with staff_u, johns home directory is
> properly labeled (staff_u associated with /home/john). However, what is
> strange here is that i cannot see staff_u home dir context specs
> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>  
> Am i looking in the wrong place? How does SELinux know that staff_u
> needs to be associated with /home/john
> 

In the current upatream, file_contexts.homedirs is autogenerated and
created in /etc/selinux/targeted/modules/active/ before it is copied to
/etc/selinux/targeted/contexts/files. This file is not removed from the
store, so it actually exists in two places.

However, with the new source policy work, file_contexts.homedirs is
generated in a temporary sandbox (not the policy store). The contents of
the sandbox are copied to /etc/selinux, and then deleted at the end of
the transaction. So the new source policy infrastructure no longer
stores intermediate/final build files in the policy store.

However, the migration script copies all the files from the old store to
the new store, even including autogenerated files that the new source
policy infrastructure will never look at or touch. This is just a bug in
the migration script. We've updated the migration script to only migrate
the files that actually need to be migrated (mostly *.local files). This
has been rebased/pushed to github #integration branch.

Aside from the discussions regarding disk requirements, I think this is
the last known issue.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/18/2014 12:00 PM, Steve Lawrence wrote:
> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>> added support for high level languages and a tool to convert policy
>>> package (pp) files to CIL. After getting some good feedback, we have
>>> made some more changes, mostly to maintain ABI compatibility. The
>>> major changes made since the last patchset are:
>>
>> <snip>
>>
>>
>> After associating user john with staff_u, johns home directory is
>> properly labeled (staff_u associated with /home/john). However, what is
>> strange here is that i cannot see staff_u home dir context specs
>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>  
>> Am i looking in the wrong place? How does SELinux know that staff_u
>> needs to be associated with /home/john
>>
> 
> In the current upatream, file_contexts.homedirs is autogenerated and
> created in /etc/selinux/targeted/modules/active/ before it is copied to
> /etc/selinux/targeted/contexts/files. This file is not removed from the
> store, so it actually exists in two places.
> 
> However, with the new source policy work, file_contexts.homedirs is
> generated in a temporary sandbox (not the policy store). The contents of
> the sandbox are copied to /etc/selinux, and then deleted at the end of
> the transaction. So the new source policy infrastructure no longer
> stores intermediate/final build files in the policy store.
> 
> However, the migration script copies all the files from the old store to
> the new store, even including autogenerated files that the new source
> policy infrastructure will never look at or touch. This is just a bug in
> the migration script. We've updated the migration script to only migrate
> the files that actually need to be migrated (mostly *.local files). This
> has been rebased/pushed to github #integration branch.

If I run semanage_migrate_etc_to_var.py -n on a clean (no
/var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
directory contains a homedir_template and a netfilter_contexts file in
addition to the modules (and commit_num).  The first file is
automatically extracted from all of the file contexts during build and
the second is unused these days.  If I then run semodule -B (or omit the
-n option on migration), I further have file_contexts.template and
users_extra files under active, both of which are also generated.  I can
delete all four files and regenerate all but netfilter_contexts via
semodule -B.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/18/2014 02:10 PM, Stephen Smalley wrote:
> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>> added support for high level languages and a tool to convert policy
>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>> major changes made since the last patchset are:
>>>
>>> <snip>
>>>
>>>
>>> After associating user john with staff_u, johns home directory is
>>> properly labeled (staff_u associated with /home/john). However, what is
>>> strange here is that i cannot see staff_u home dir context specs
>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>  
>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>> needs to be associated with /home/john
>>>
>>
>> In the current upatream, file_contexts.homedirs is autogenerated and
>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>> store, so it actually exists in two places.
>>
>> However, with the new source policy work, file_contexts.homedirs is
>> generated in a temporary sandbox (not the policy store). The contents of
>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>> the transaction. So the new source policy infrastructure no longer
>> stores intermediate/final build files in the policy store.
>>
>> However, the migration script copies all the files from the old store to
>> the new store, even including autogenerated files that the new source
>> policy infrastructure will never look at or touch. This is just a bug in
>> the migration script. We've updated the migration script to only migrate
>> the files that actually need to be migrated (mostly *.local files). This
>> has been rebased/pushed to github #integration branch.
> 
> If I run semanage_migrate_etc_to_var.py -n on a clean (no
> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
> directory contains a homedir_template and a netfilter_contexts file in
> addition to the modules (and commit_num).  The first file is
> automatically extracted from all of the file contexts during build and
> the second is unused these days.  If I then run semodule -B (or omit the
> -n option on migration), I further have file_contexts.template and
> users_extra files under active, both of which are also generated.  I can
> delete all four files and regenerate all but netfilter_contexts via
> semodule -B.
> 

This has been fixed. Just needed to remove a couple more paths from the
migration script and add a couple of unlink()'s in direct_commit().

- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/21/2014 10:34 AM, Steve Lawrence wrote:
> On 07/18/2014 02:10 PM, Stephen Smalley wrote:
>> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>> added support for high level languages and a tool to convert policy
>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>> major changes made since the last patchset are:
>>>>
>>>> <snip>
>>>>
>>>>
>>>> After associating user john with staff_u, johns home directory is
>>>> properly labeled (staff_u associated with /home/john). However, what is
>>>> strange here is that i cannot see staff_u home dir context specs
>>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>>  
>>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>>> needs to be associated with /home/john
>>>>
>>>
>>> In the current upatream, file_contexts.homedirs is autogenerated and
>>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>>> store, so it actually exists in two places.
>>>
>>> However, with the new source policy work, file_contexts.homedirs is
>>> generated in a temporary sandbox (not the policy store). The contents of
>>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>>> the transaction. So the new source policy infrastructure no longer
>>> stores intermediate/final build files in the policy store.
>>>
>>> However, the migration script copies all the files from the old store to
>>> the new store, even including autogenerated files that the new source
>>> policy infrastructure will never look at or touch. This is just a bug in
>>> the migration script. We've updated the migration script to only migrate
>>> the files that actually need to be migrated (mostly *.local files). This
>>> has been rebased/pushed to github #integration branch.
>>
>> If I run semanage_migrate_etc_to_var.py -n on a clean (no
>> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
>> directory contains a homedir_template and a netfilter_contexts file in
>> addition to the modules (and commit_num).  The first file is
>> automatically extracted from all of the file contexts during build and
>> the second is unused these days.  If I then run semodule -B (or omit the
>> -n option on migration), I further have file_contexts.template and
>> users_extra files under active, both of which are also generated.  I can
>> delete all four files and regenerate all but netfilter_contexts via
>> semodule -B.
>>
> 
> This has been fixed. Just needed to remove a couple more paths from the
> migration script and add a couple of unlink()'s in direct_commit().

Thanks.  I had a question though about the approach being used in the
new libsemanage for populating /var/lib/selinux/tmp/targeted initially
(not to be confused with /var/lib/selinux/targeted/tmp, which frankly I
do find rather confusing even though I understand it now).  libsemanage
copies the entire /etc/selinux/targeted directory layout (but not all
the files) under /var/lib/selinux/tmp/targeted, even directories that it
will never use (including e.g. the old modules directory), and then
copies the files it manages from /etc/selinux/targeted to the
corresponding /var/lib/selinux/tmp/targeted location before generating
the new policy.  This raises a few questions in my mind:

- Is it a good idea to depend on an already existing and populated
/etc/selinux/targeted directory tree?

- Why copy the entire directory structure rather than just creating the
directories we know we will need?

- What exactly is the dependency here?  If there is a mismatch between
the built files in /etc/selinux/targeted and the
/var/lib/selinux/targeted/active tree, can bad things happen?  Is
libsemanage depending on those prebuilt files being identical to what
would be generated if it did a rebuild from
/var/lib/selinux/targeted/active?  If not, why does it need them at all?

It looks to me as if possibly the old libsemanage relied on having
policy.kern locally available in the sandbox, then RH switched it to a
symlink to the installed file (wrongly, I think, as this would seemingly
break a revert to prior policy), and this made it also depend on the
installed policy file.  And it appears to assume that the directory
structure at least already exists under /etc/selinux/targeted.  But this
seems to take it a step further.  How do I bootstrap a policy install
with no prior /etc/selinux/targeted directory?

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/21/2014 10:51 AM, Stephen Smalley wrote:
> On 07/21/2014 10:34 AM, Steve Lawrence wrote:
>> On 07/18/2014 02:10 PM, Stephen Smalley wrote:
>>> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>>> added support for high level languages and a tool to convert policy
>>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>>> major changes made since the last patchset are:
>>>>>
>>>>> <snip>
>>>>>
>>>>>
>>>>> After associating user john with staff_u, johns home directory is
>>>>> properly labeled (staff_u associated with /home/john). However, what is
>>>>> strange here is that i cannot see staff_u home dir context specs
>>>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>>>  
>>>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>>>> needs to be associated with /home/john
>>>>>
>>>>
>>>> In the current upatream, file_contexts.homedirs is autogenerated and
>>>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>>>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>>>> store, so it actually exists in two places.
>>>>
>>>> However, with the new source policy work, file_contexts.homedirs is
>>>> generated in a temporary sandbox (not the policy store). The contents of
>>>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>>>> the transaction. So the new source policy infrastructure no longer
>>>> stores intermediate/final build files in the policy store.
>>>>
>>>> However, the migration script copies all the files from the old store to
>>>> the new store, even including autogenerated files that the new source
>>>> policy infrastructure will never look at or touch. This is just a bug in
>>>> the migration script. We've updated the migration script to only migrate
>>>> the files that actually need to be migrated (mostly *.local files). This
>>>> has been rebased/pushed to github #integration branch.
>>>
>>> If I run semanage_migrate_etc_to_var.py -n on a clean (no
>>> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
>>> directory contains a homedir_template and a netfilter_contexts file in
>>> addition to the modules (and commit_num).  The first file is
>>> automatically extracted from all of the file contexts during build and
>>> the second is unused these days.  If I then run semodule -B (or omit the
>>> -n option on migration), I further have file_contexts.template and
>>> users_extra files under active, both of which are also generated.  I can
>>> delete all four files and regenerate all but netfilter_contexts via
>>> semodule -B.
>>>
>>
>> This has been fixed. Just needed to remove a couple more paths from the
>> migration script and add a couple of unlink()'s in direct_commit().
> 
> Thanks.  I had a question though about the approach being used in the
> new libsemanage for populating /var/lib/selinux/tmp/targeted initially
> (not to be confused with /var/lib/selinux/targeted/tmp, which frankly I
> do find rather confusing even though I understand it now).  libsemanage
> copies the entire /etc/selinux/targeted directory layout (but not all
> the files) under /var/lib/selinux/tmp/targeted, even directories that it
> will never use (including e.g. the old modules directory), and then
> copies the files it manages from /etc/selinux/targeted to the
> corresponding /var/lib/selinux/tmp/targeted location before generating
> the new policy.  This raises a few questions in my mind:
> 
> - Is it a good idea to depend on an already existing and populated
> /etc/selinux/targeted directory tree?

Wouldn't hurt to just create the directories we need.

> - Why copy the entire directory structure rather than just creating the
> directories we know we will need?

Yeah, seems unnecessary.

> - What exactly is the dependency here?  If there is a mismatch between
> the built files in /etc/selinux/targeted and the
> /var/lib/selinux/targeted/active tree, can bad things happen?  Is
> libsemanage depending on those prebuilt files being identical to what
> would be generated if it did a rebuild from
> /var/lib/selinux/targeted/active?  If not, why does it need them at all?

>From what I can tell, I think the files in /etc/selinux/targeted are
copied over for the cases where it doesn't rebuild policies (e.g.
changes to local fcontexts, seusers, ports, etc), but needs the current
policy files for validation. We'll look into how how drastic this change
would be.

Another thing that I think is incorrect is how file_context.local is
handle. That is stored in /etc/selinux/targeted/contexts/files, but it
seems like that should just be stored in
/var/lib/selinux/targeted/active/ just like other local modification
files, and merged into the file_contexts file. Not sure the reasoning
for keeping it separate.

> It looks to me as if possibly the old libsemanage relied on having
> policy.kern locally available in the sandbox, then RH switched it to a
> symlink to the installed file (wrongly, I think, as this would seemingly
> break a revert to prior policy), and this made it also depend on the
> installed policy file.  And it appears to assume that the directory
> structure at least already exists under /etc/selinux/targeted.  But this
> seems to take it a step further.  How do I bootstrap a policy install
> with no prior /etc/selinux/targeted directory?

I think all you really need is the directory structure set up, but it
should be a pretty simple fix so even that isn't required.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/21/2014 01:50 PM, Steve Lawrence wrote:
> On 07/21/2014 10:51 AM, Stephen Smalley wrote:
>> On 07/21/2014 10:34 AM, Steve Lawrence wrote:
>>> On 07/18/2014 02:10 PM, Stephen Smalley wrote:
>>>> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>>>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>>>> added support for high level languages and a tool to convert policy
>>>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>>>> major changes made since the last patchset are:
>>>>>>
>>>>>> <snip>
>>>>>>
>>>>>>
>>>>>> After associating user john with staff_u, johns home directory is
>>>>>> properly labeled (staff_u associated with /home/john). However, what is
>>>>>> strange here is that i cannot see staff_u home dir context specs
>>>>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>>>>  
>>>>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>>>>> needs to be associated with /home/john
>>>>>>
>>>>>
>>>>> In the current upatream, file_contexts.homedirs is autogenerated and
>>>>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>>>>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>>>>> store, so it actually exists in two places.
>>>>>
>>>>> However, with the new source policy work, file_contexts.homedirs is
>>>>> generated in a temporary sandbox (not the policy store). The contents of
>>>>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>>>>> the transaction. So the new source policy infrastructure no longer
>>>>> stores intermediate/final build files in the policy store.
>>>>>
>>>>> However, the migration script copies all the files from the old store to
>>>>> the new store, even including autogenerated files that the new source
>>>>> policy infrastructure will never look at or touch. This is just a bug in
>>>>> the migration script. We've updated the migration script to only migrate
>>>>> the files that actually need to be migrated (mostly *.local files). This
>>>>> has been rebased/pushed to github #integration branch.
>>>>
>>>> If I run semanage_migrate_etc_to_var.py -n on a clean (no
>>>> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
>>>> directory contains a homedir_template and a netfilter_contexts file in
>>>> addition to the modules (and commit_num).  The first file is
>>>> automatically extracted from all of the file contexts during build and
>>>> the second is unused these days.  If I then run semodule -B (or omit the
>>>> -n option on migration), I further have file_contexts.template and
>>>> users_extra files under active, both of which are also generated.  I can
>>>> delete all four files and regenerate all but netfilter_contexts via
>>>> semodule -B.
>>>>
>>>
>>> This has been fixed. Just needed to remove a couple more paths from the
>>> migration script and add a couple of unlink()'s in direct_commit().
>>
>> Thanks.  I had a question though about the approach being used in the
>> new libsemanage for populating /var/lib/selinux/tmp/targeted initially
>> (not to be confused with /var/lib/selinux/targeted/tmp, which frankly I
>> do find rather confusing even though I understand it now).  libsemanage
>> copies the entire /etc/selinux/targeted directory layout (but not all
>> the files) under /var/lib/selinux/tmp/targeted, even directories that it
>> will never use (including e.g. the old modules directory), and then
>> copies the files it manages from /etc/selinux/targeted to the
>> corresponding /var/lib/selinux/tmp/targeted location before generating
>> the new policy.  This raises a few questions in my mind:
>>
>> - Is it a good idea to depend on an already existing and populated
>> /etc/selinux/targeted directory tree?
> 
> Wouldn't hurt to just create the directories we need.
> 
>> - Why copy the entire directory structure rather than just creating the
>> directories we know we will need?
> 
> Yeah, seems unnecessary.
> 

Sorry for the delay in getting back to these issues. Took a little time
to look into. We've fixed the creation of the directory structure. It
now creates the required directories rather than copying the entire
directory structure.

>> - What exactly is the dependency here?  If there is a mismatch between
>> the built files in /etc/selinux/targeted and the
>> /var/lib/selinux/targeted/active tree, can bad things happen?  Is
>> libsemanage depending on those prebuilt files being identical to what
>> would be generated if it did a rebuild from
>> /var/lib/selinux/targeted/active?  If not, why does it need them at all?
> 
> From what I can tell, I think the files in /etc/selinux/targeted are
> copied over for the cases where it doesn't rebuild policies (e.g.
> changes to local fcontexts, seusers, ports, etc), but needs the current
> policy files for validation. We'll look into how how drastic this change
> would be.

This is actually not a simple fix. I think we would either need to make
changes to how dbase structures work (which isn't trivial considering
all the function pointers). We could alternatively add symlinks in
/var/lib/selinux/tmp/targeted that point to the correct locations in
/etc/selinux/targeted, and if we need to change any files in
/var/lib/selinux/tmp/targeted then we delete the symlinks and write the
new content. Though, this seems kindof hacky to me.

Unless you have any other ideas, or the symlink seems reasonable, maybe
we could hold of on this issue for now, and work on the properly
solution once everything is merged?

> Another thing that I think is incorrect is how file_context.local is
> handle. That is stored in /etc/selinux/targeted/contexts/files, but it
> seems like that should just be stored in
> /var/lib/selinux/targeted/active/ just like other local modification
> files, and merged into the file_contexts file. Not sure the reasoning
> for keeping it separate.
> 
>> It looks to me as if possibly the old libsemanage relied on having
>> policy.kern locally available in the sandbox, then RH switched it to a
>> symlink to the installed file (wrongly, I think, as this would seemingly
>> break a revert to prior policy), and this made it also depend on the
>> installed policy file.  And it appears to assume that the directory
>> structure at least already exists under /etc/selinux/targeted.  But this
>> seems to take it a step further.  How do I bootstrap a policy install
>> with no prior /etc/selinux/targeted directory?
> 
> I think all you really need is the directory structure set up, but it
> should be a pretty simple fix so even that isn't required.
> 

This has been fixed. The integration branch now creates all the
directories it needs, so nothing should need to be done to bootstrap the
system. In addition to these changes, we've also fixed the name of the
semodule --ignore-module-cache option (it used to be
--ignore_module_cache). We have rebased the #integration branch onto
github/next (fixing conflicts with the recent boolean changes).

I think that fixes all the known issues, aside from the two that I think
would best be dealt with after upstreaming the integration branch (the
two issues being 1) copying files from /etc/selinux/targeted to
/var/lib/selinux/tmp/targeted and 2) a way to delete hll files after
converted to cil to save space if needed).

Unless there are any more questions/concerns, perhaps it's time to start
talks about merging into upstream and what needs to be done to help
distributions integrate the changes?

- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 08/01/2014 10:51 AM, Steve Lawrence wrote:
> On 07/21/2014 01:50 PM, Steve Lawrence wrote:
>> On 07/21/2014 10:51 AM, Stephen Smalley wrote:
>>> On 07/21/2014 10:34 AM, Steve Lawrence wrote:
>>>> On 07/18/2014 02:10 PM, Stephen Smalley wrote:
>>>>> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>>>>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>>>>> added support for high level languages and a tool to convert policy
>>>>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>>>>> major changes made since the last patchset are:
>>>>>>>
>>>>>>> <snip>
>>>>>>>
>>>>>>>
>>>>>>> After associating user john with staff_u, johns home directory is
>>>>>>> properly labeled (staff_u associated with /home/john). However, what is
>>>>>>> strange here is that i cannot see staff_u home dir context specs
>>>>>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>>>>>  
>>>>>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>>>>>> needs to be associated with /home/john
>>>>>>>
>>>>>>
>>>>>> In the current upatream, file_contexts.homedirs is autogenerated and
>>>>>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>>>>>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>>>>>> store, so it actually exists in two places.
>>>>>>
>>>>>> However, with the new source policy work, file_contexts.homedirs is
>>>>>> generated in a temporary sandbox (not the policy store). The contents of
>>>>>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>>>>>> the transaction. So the new source policy infrastructure no longer
>>>>>> stores intermediate/final build files in the policy store.
>>>>>>
>>>>>> However, the migration script copies all the files from the old store to
>>>>>> the new store, even including autogenerated files that the new source
>>>>>> policy infrastructure will never look at or touch. This is just a bug in
>>>>>> the migration script. We've updated the migration script to only migrate
>>>>>> the files that actually need to be migrated (mostly *.local files). This
>>>>>> has been rebased/pushed to github #integration branch.
>>>>>
>>>>> If I run semanage_migrate_etc_to_var.py -n on a clean (no
>>>>> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
>>>>> directory contains a homedir_template and a netfilter_contexts file in
>>>>> addition to the modules (and commit_num).  The first file is
>>>>> automatically extracted from all of the file contexts during build and
>>>>> the second is unused these days.  If I then run semodule -B (or omit the
>>>>> -n option on migration), I further have file_contexts.template and
>>>>> users_extra files under active, both of which are also generated.  I can
>>>>> delete all four files and regenerate all but netfilter_contexts via
>>>>> semodule -B.
>>>>>
>>>>
>>>> This has been fixed. Just needed to remove a couple more paths from the
>>>> migration script and add a couple of unlink()'s in direct_commit().
>>>
>>> Thanks.  I had a question though about the approach being used in the
>>> new libsemanage for populating /var/lib/selinux/tmp/targeted initially
>>> (not to be confused with /var/lib/selinux/targeted/tmp, which frankly I
>>> do find rather confusing even though I understand it now).  libsemanage
>>> copies the entire /etc/selinux/targeted directory layout (but not all
>>> the files) under /var/lib/selinux/tmp/targeted, even directories that it
>>> will never use (including e.g. the old modules directory), and then
>>> copies the files it manages from /etc/selinux/targeted to the
>>> corresponding /var/lib/selinux/tmp/targeted location before generating
>>> the new policy.  This raises a few questions in my mind:
>>>
>>> - Is it a good idea to depend on an already existing and populated
>>> /etc/selinux/targeted directory tree?
>>
>> Wouldn't hurt to just create the directories we need.
>>
>>> - Why copy the entire directory structure rather than just creating the
>>> directories we know we will need?
>>
>> Yeah, seems unnecessary.
>>
> 
> Sorry for the delay in getting back to these issues. Took a little time
> to look into. We've fixed the creation of the directory structure. It
> now creates the required directories rather than copying the entire
> directory structure.
> 
>>> - What exactly is the dependency here?  If there is a mismatch between
>>> the built files in /etc/selinux/targeted and the
>>> /var/lib/selinux/targeted/active tree, can bad things happen?  Is
>>> libsemanage depending on those prebuilt files being identical to what
>>> would be generated if it did a rebuild from
>>> /var/lib/selinux/targeted/active?  If not, why does it need them at all?
>>
>> From what I can tell, I think the files in /etc/selinux/targeted are
>> copied over for the cases where it doesn't rebuild policies (e.g.
>> changes to local fcontexts, seusers, ports, etc), but needs the current
>> policy files for validation. We'll look into how how drastic this change
>> would be.
> 
> This is actually not a simple fix. I think we would either need to make
> changes to how dbase structures work (which isn't trivial considering
> all the function pointers). We could alternatively add symlinks in
> /var/lib/selinux/tmp/targeted that point to the correct locations in
> /etc/selinux/targeted, and if we need to change any files in
> /var/lib/selinux/tmp/targeted then we delete the symlinks and write the
> new content. Though, this seems kindof hacky to me.
> 
> Unless you have any other ideas, or the symlink seems reasonable, maybe
> we could hold of on this issue for now, and work on the properly
> solution once everything is merged?
> 
>> Another thing that I think is incorrect is how file_context.local is
>> handle. That is stored in /etc/selinux/targeted/contexts/files, but it
>> seems like that should just be stored in
>> /var/lib/selinux/targeted/active/ just like other local modification
>> files, and merged into the file_contexts file. Not sure the reasoning
>> for keeping it separate.
>>
>>> It looks to me as if possibly the old libsemanage relied on having
>>> policy.kern locally available in the sandbox, then RH switched it to a
>>> symlink to the installed file (wrongly, I think, as this would seemingly
>>> break a revert to prior policy), and this made it also depend on the
>>> installed policy file.  And it appears to assume that the directory
>>> structure at least already exists under /etc/selinux/targeted.  But this
>>> seems to take it a step further.  How do I bootstrap a policy install
>>> with no prior /etc/selinux/targeted directory?
>>
>> I think all you really need is the directory structure set up, but it
>> should be a pretty simple fix so even that isn't required.
>>
> 
> This has been fixed. The integration branch now creates all the
> directories it needs, so nothing should need to be done to bootstrap the
> system. In addition to these changes, we've also fixed the name of the
> semodule --ignore-module-cache option (it used to be
> --ignore_module_cache). We have rebased the #integration branch onto
> github/next (fixing conflicts with the recent boolean changes).
> 
> I think that fixes all the known issues, aside from the two that I think
> would best be dealt with after upstreaming the integration branch (the
> two issues being 1) copying files from /etc/selinux/targeted to
> /var/lib/selinux/tmp/targeted and 2) a way to delete hll files after
> converted to cil to save space if needed).
> 
> Unless there are any more questions/concerns, perhaps it's time to start
> talks about merging into upstream and what needs to be done to help
> distributions integrate the changes?

So we need to link cil into the tree for building.  I haven't tried it,
but git subtree (not submodule) looks promising, both for adding cil to
the tree and possibly to split each of the other subdirectories into its
own repository if we want to do that.

If we are going to keep maintaining */{ChangeLog,VERSION}, then we
should update them in all affected subdirectories (including the changes
on #next as well as on #integration).  If not, then we should get rid of
them and decide what if any comparable files we want to add at
top-level, e.g. NEWS.  Need to capture all of the user-visible changes
and migration instructions somewhere.  I expect the distributions will
continue to maintain separate packages for libsepol, checkpolicy,
libselinux, libsemanage and policycoreutils, so we should try to keep
things as simple as possible for them.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 08/01/2014 01:46 PM, Stephen Smalley wrote:
> On 08/01/2014 10:51 AM, Steve Lawrence wrote:
>> On 07/21/2014 01:50 PM, Steve Lawrence wrote:
>>> On 07/21/2014 10:51 AM, Stephen Smalley wrote:
>>>> On 07/21/2014 10:34 AM, Steve Lawrence wrote:
>>>>> On 07/18/2014 02:10 PM, Stephen Smalley wrote:
>>>>>> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>>>>>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>>>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>>>>>> added support for high level languages and a tool to convert policy
>>>>>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>>>>>> major changes made since the last patchset are:
>>>>>>>>
>>>>>>>> <snip>
>>>>>>>>
>>>>>>>>
>>>>>>>> After associating user john with staff_u, johns home directory is
>>>>>>>> properly labeled (staff_u associated with /home/john). However, what is
>>>>>>>> strange here is that i cannot see staff_u home dir context specs
>>>>>>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>>>>>>  
>>>>>>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>>>>>>> needs to be associated with /home/john
>>>>>>>>
>>>>>>>
>>>>>>> In the current upatream, file_contexts.homedirs is autogenerated and
>>>>>>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>>>>>>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>>>>>>> store, so it actually exists in two places.
>>>>>>>
>>>>>>> However, with the new source policy work, file_contexts.homedirs is
>>>>>>> generated in a temporary sandbox (not the policy store). The contents of
>>>>>>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>>>>>>> the transaction. So the new source policy infrastructure no longer
>>>>>>> stores intermediate/final build files in the policy store.
>>>>>>>
>>>>>>> However, the migration script copies all the files from the old store to
>>>>>>> the new store, even including autogenerated files that the new source
>>>>>>> policy infrastructure will never look at or touch. This is just a bug in
>>>>>>> the migration script. We've updated the migration script to only migrate
>>>>>>> the files that actually need to be migrated (mostly *.local files). This
>>>>>>> has been rebased/pushed to github #integration branch.
>>>>>>
>>>>>> If I run semanage_migrate_etc_to_var.py -n on a clean (no
>>>>>> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
>>>>>> directory contains a homedir_template and a netfilter_contexts file in
>>>>>> addition to the modules (and commit_num).  The first file is
>>>>>> automatically extracted from all of the file contexts during build and
>>>>>> the second is unused these days.  If I then run semodule -B (or omit the
>>>>>> -n option on migration), I further have file_contexts.template and
>>>>>> users_extra files under active, both of which are also generated.  I can
>>>>>> delete all four files and regenerate all but netfilter_contexts via
>>>>>> semodule -B.
>>>>>>
>>>>>
>>>>> This has been fixed. Just needed to remove a couple more paths from the
>>>>> migration script and add a couple of unlink()'s in direct_commit().
>>>>
>>>> Thanks.  I had a question though about the approach being used in the
>>>> new libsemanage for populating /var/lib/selinux/tmp/targeted initially
>>>> (not to be confused with /var/lib/selinux/targeted/tmp, which frankly I
>>>> do find rather confusing even though I understand it now).  libsemanage
>>>> copies the entire /etc/selinux/targeted directory layout (but not all
>>>> the files) under /var/lib/selinux/tmp/targeted, even directories that it
>>>> will never use (including e.g. the old modules directory), and then
>>>> copies the files it manages from /etc/selinux/targeted to the
>>>> corresponding /var/lib/selinux/tmp/targeted location before generating
>>>> the new policy.  This raises a few questions in my mind:
>>>>
>>>> - Is it a good idea to depend on an already existing and populated
>>>> /etc/selinux/targeted directory tree?
>>>
>>> Wouldn't hurt to just create the directories we need.
>>>
>>>> - Why copy the entire directory structure rather than just creating the
>>>> directories we know we will need?
>>>
>>> Yeah, seems unnecessary.
>>>
>>
>> Sorry for the delay in getting back to these issues. Took a little time
>> to look into. We've fixed the creation of the directory structure. It
>> now creates the required directories rather than copying the entire
>> directory structure.
>>
>>>> - What exactly is the dependency here?  If there is a mismatch between
>>>> the built files in /etc/selinux/targeted and the
>>>> /var/lib/selinux/targeted/active tree, can bad things happen?  Is
>>>> libsemanage depending on those prebuilt files being identical to what
>>>> would be generated if it did a rebuild from
>>>> /var/lib/selinux/targeted/active?  If not, why does it need them at all?
>>>
>>> From what I can tell, I think the files in /etc/selinux/targeted are
>>> copied over for the cases where it doesn't rebuild policies (e.g.
>>> changes to local fcontexts, seusers, ports, etc), but needs the current
>>> policy files for validation. We'll look into how how drastic this change
>>> would be.
>>
>> This is actually not a simple fix. I think we would either need to make
>> changes to how dbase structures work (which isn't trivial considering
>> all the function pointers). We could alternatively add symlinks in
>> /var/lib/selinux/tmp/targeted that point to the correct locations in
>> /etc/selinux/targeted, and if we need to change any files in
>> /var/lib/selinux/tmp/targeted then we delete the symlinks and write the
>> new content. Though, this seems kindof hacky to me.
>>
>> Unless you have any other ideas, or the symlink seems reasonable, maybe
>> we could hold of on this issue for now, and work on the properly
>> solution once everything is merged?
>>
>>> Another thing that I think is incorrect is how file_context.local is
>>> handle. That is stored in /etc/selinux/targeted/contexts/files, but it
>>> seems like that should just be stored in
>>> /var/lib/selinux/targeted/active/ just like other local modification
>>> files, and merged into the file_contexts file. Not sure the reasoning
>>> for keeping it separate.
>>>
>>>> It looks to me as if possibly the old libsemanage relied on having
>>>> policy.kern locally available in the sandbox, then RH switched it to a
>>>> symlink to the installed file (wrongly, I think, as this would seemingly
>>>> break a revert to prior policy), and this made it also depend on the
>>>> installed policy file.  And it appears to assume that the directory
>>>> structure at least already exists under /etc/selinux/targeted.  But this
>>>> seems to take it a step further.  How do I bootstrap a policy install
>>>> with no prior /etc/selinux/targeted directory?
>>>
>>> I think all you really need is the directory structure set up, but it
>>> should be a pretty simple fix so even that isn't required.
>>>
>>
>> This has been fixed. The integration branch now creates all the
>> directories it needs, so nothing should need to be done to bootstrap the
>> system. In addition to these changes, we've also fixed the name of the
>> semodule --ignore-module-cache option (it used to be
>> --ignore_module_cache). We have rebased the #integration branch onto
>> github/next (fixing conflicts with the recent boolean changes).
>>
>> I think that fixes all the known issues, aside from the two that I think
>> would best be dealt with after upstreaming the integration branch (the
>> two issues being 1) copying files from /etc/selinux/targeted to
>> /var/lib/selinux/tmp/targeted and 2) a way to delete hll files after
>> converted to cil to save space if needed).
>>
>> Unless there are any more questions/concerns, perhaps it's time to start
>> talks about merging into upstream and what needs to be done to help
>> distributions integrate the changes?
> 
> So we need to link cil into the tree for building.  I haven't tried it,
> but git subtree (not submodule) looks promising, both for adding cil to
> the tree and possibly to split each of the other subdirectories into its
> own repository if we want to do that.

I have tested merging the cil.git into selinux.git using git subtree,
and it works fine. It does make it a little messy when looking at the
history, since it contains histories of both selinux and cil, but git
log and other tools can filter out unwanted directories pretty easily.

A while ago I did look into splitting the subdirectories into separate
repos with subtree, and it didn't work as nicely as I would have liked.
It resulted in a lot of branches and tags that had nothing to do with
the subdirectory. I'm sure it could be fixed (and I didn't spend too
much time on it), but it might require a little bit of work.

> If we are going to keep maintaining */{ChangeLog,VERSION}, then we
> should update them in all affected subdirectories (including the changes
> on #next as well as on #integration).  If not, then we should get rid of
> them and decide what if any comparable files we want to add at
> top-level, e.g. NEWS.  Need to capture all of the user-visible changes
> and migration instructions somewhere.  I expect the distributions will
> continue to maintain separate packages for libsepol, checkpolicy,
> libselinux, libsemanage and policycoreutils, so we should try to keep
> things as simple as possible for them.

We'll add some patches to update the ChangeLog/VERSIONs for affected
subdirs, and create a wiki page that talks about migration and the new
CIL changes.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 08/01/2014 12:46 PM, Stephen Smalley wrote:
> On 08/01/2014 10:51 AM, Steve Lawrence wrote:
>> On 07/21/2014 01:50 PM, Steve Lawrence wrote:
>>> On 07/21/2014 10:51 AM, Stephen Smalley wrote:
>>>> On 07/21/2014 10:34 AM, Steve Lawrence wrote:
>>>>> On 07/18/2014 02:10 PM, Stephen Smalley wrote:
>>>>>> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>>>>>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>>>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>>>>>> added support for high level languages and a tool to convert policy
>>>>>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>>>>>> major changes made since the last patchset are:
>>>>>>>>
>>>>>>>> <snip>
>>>>>>>>
>>>>>>>>
>>>>>>>> After associating user john with staff_u, johns home directory is
>>>>>>>> properly labeled (staff_u associated with /home/john). However, what is
>>>>>>>> strange here is that i cannot see staff_u home dir context specs
>>>>>>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>>>>>>  
>>>>>>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>>>>>>> needs to be associated with /home/john
>>>>>>>>
>>>>>>>
>>>>>>> In the current upatream, file_contexts.homedirs is autogenerated and
>>>>>>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>>>>>>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>>>>>>> store, so it actually exists in two places.
>>>>>>>
>>>>>>> However, with the new source policy work, file_contexts.homedirs is
>>>>>>> generated in a temporary sandbox (not the policy store). The contents of
>>>>>>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>>>>>>> the transaction. So the new source policy infrastructure no longer
>>>>>>> stores intermediate/final build files in the policy store.
>>>>>>>
>>>>>>> However, the migration script copies all the files from the old store to
>>>>>>> the new store, even including autogenerated files that the new source
>>>>>>> policy infrastructure will never look at or touch. This is just a bug in
>>>>>>> the migration script. We've updated the migration script to only migrate
>>>>>>> the files that actually need to be migrated (mostly *.local files). This
>>>>>>> has been rebased/pushed to github #integration branch.
>>>>>>
>>>>>> If I run semanage_migrate_etc_to_var.py -n on a clean (no
>>>>>> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
>>>>>> directory contains a homedir_template and a netfilter_contexts file in
>>>>>> addition to the modules (and commit_num).  The first file is
>>>>>> automatically extracted from all of the file contexts during build and
>>>>>> the second is unused these days.  If I then run semodule -B (or omit the
>>>>>> -n option on migration), I further have file_contexts.template and
>>>>>> users_extra files under active, both of which are also generated.  I can
>>>>>> delete all four files and regenerate all but netfilter_contexts via
>>>>>> semodule -B.
>>>>>>
>>>>>
>>>>> This has been fixed. Just needed to remove a couple more paths from the
>>>>> migration script and add a couple of unlink()'s in direct_commit().
>>>>
>>>> Thanks.  I had a question though about the approach being used in the
>>>> new libsemanage for populating /var/lib/selinux/tmp/targeted initially
>>>> (not to be confused with /var/lib/selinux/targeted/tmp, which frankly I
>>>> do find rather confusing even though I understand it now).  libsemanage
>>>> copies the entire /etc/selinux/targeted directory layout (but not all
>>>> the files) under /var/lib/selinux/tmp/targeted, even directories that it
>>>> will never use (including e.g. the old modules directory), and then
>>>> copies the files it manages from /etc/selinux/targeted to the
>>>> corresponding /var/lib/selinux/tmp/targeted location before generating
>>>> the new policy.  This raises a few questions in my mind:
>>>>
>>>> - Is it a good idea to depend on an already existing and populated
>>>> /etc/selinux/targeted directory tree?
>>>
>>> Wouldn't hurt to just create the directories we need.
>>>
>>>> - Why copy the entire directory structure rather than just creating the
>>>> directories we know we will need?
>>>
>>> Yeah, seems unnecessary.
>>>
>>
>> Sorry for the delay in getting back to these issues. Took a little time
>> to look into. We've fixed the creation of the directory structure. It
>> now creates the required directories rather than copying the entire
>> directory structure.
>>
>>>> - What exactly is the dependency here?  If there is a mismatch between
>>>> the built files in /etc/selinux/targeted and the
>>>> /var/lib/selinux/targeted/active tree, can bad things happen?  Is
>>>> libsemanage depending on those prebuilt files being identical to what
>>>> would be generated if it did a rebuild from
>>>> /var/lib/selinux/targeted/active?  If not, why does it need them at all?
>>>
>>> From what I can tell, I think the files in /etc/selinux/targeted are
>>> copied over for the cases where it doesn't rebuild policies (e.g.
>>> changes to local fcontexts, seusers, ports, etc), but needs the current
>>> policy files for validation. We'll look into how how drastic this change
>>> would be.
>>
>> This is actually not a simple fix. I think we would either need to make
>> changes to how dbase structures work (which isn't trivial considering
>> all the function pointers). We could alternatively add symlinks in
>> /var/lib/selinux/tmp/targeted that point to the correct locations in
>> /etc/selinux/targeted, and if we need to change any files in
>> /var/lib/selinux/tmp/targeted then we delete the symlinks and write the
>> new content. Though, this seems kindof hacky to me.
>>
>> Unless you have any other ideas, or the symlink seems reasonable, maybe
>> we could hold of on this issue for now, and work on the properly
>> solution once everything is merged?
>>
>>> Another thing that I think is incorrect is how file_context.local is
>>> handle. That is stored in /etc/selinux/targeted/contexts/files, but it
>>> seems like that should just be stored in
>>> /var/lib/selinux/targeted/active/ just like other local modification
>>> files, and merged into the file_contexts file. Not sure the reasoning
>>> for keeping it separate.
>>>
>>>> It looks to me as if possibly the old libsemanage relied on having
>>>> policy.kern locally available in the sandbox, then RH switched it to a
>>>> symlink to the installed file (wrongly, I think, as this would seemingly
>>>> break a revert to prior policy), and this made it also depend on the
>>>> installed policy file.  And it appears to assume that the directory
>>>> structure at least already exists under /etc/selinux/targeted.  But this
>>>> seems to take it a step further.  How do I bootstrap a policy install
>>>> with no prior /etc/selinux/targeted directory?
>>>
>>> I think all you really need is the directory structure set up, but it
>>> should be a pretty simple fix so even that isn't required.
>>>
>>
>> This has been fixed. The integration branch now creates all the
>> directories it needs, so nothing should need to be done to bootstrap the
>> system. In addition to these changes, we've also fixed the name of the
>> semodule --ignore-module-cache option (it used to be
>> --ignore_module_cache). We have rebased the #integration branch onto
>> github/next (fixing conflicts with the recent boolean changes).
>>
>> I think that fixes all the known issues, aside from the two that I think
>> would best be dealt with after upstreaming the integration branch (the
>> two issues being 1) copying files from /etc/selinux/targeted to
>> /var/lib/selinux/tmp/targeted and 2) a way to delete hll files after
>> converted to cil to save space if needed).
>>
>> Unless there are any more questions/concerns, perhaps it's time to start
>> talks about merging into upstream and what needs to be done to help
>> distributions integrate the changes?
> 
> So we need to link cil into the tree for building.  I haven't tried it,
> but git subtree (not submodule) looks promising, both for adding cil to
> the tree and possibly to split each of the other subdirectories into its
> own repository if we want to do that.
> 
> If we are going to keep maintaining */{ChangeLog,VERSION}, then we
> should update them in all affected subdirectories (including the changes
> on #next as well as on #integration).  If not, then we should get rid of
> them and decide what if any comparable files we want to add at
> top-level, e.g. NEWS.  Need to capture all of the user-visible changes
> and migration instructions somewhere.  I expect the distributions will
> continue to maintain separate packages for libsepol, checkpolicy,
> libselinux, libsemanage and policycoreutils, so we should try to keep
> things as simple as possible for them.
> 

I've just pushed an updated integration branch that has the following
changes:

- Rename the migration script to semanage_migrate_store
- Add a Makefile to install semanage_migrate_store into
/usr/libexec/selinux/. This didn't seem like a tool used often enough to
install into /usr/bin
- Used git subtree to merge CIL into the selinux repo (squashed so as to
not muddy the selinux repo history)
- Updated the Changelogs to include the changes made for the source
policy/CIL integration changes
- Bumped libsemanage, libsemanage, and policycoreutils version numbers

There have also been a few bugs fixed in CIL recently, which are
included in the subtree merge.

We've also added two github wiki pages to describe the HLL
infrastructure [1] and how to migrate from the old store to the new
store [2].

[1]
https://github.com/SELinuxProject/selinux/wiki/High-Level-Language-Infrastructure
[2] https://github.com/SELinuxProject/selinux/wiki/Policy-Store-Migration

I think that's the last remaining issues.

- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/09/2014 03:21 PM, Steve Lawrence wrote:
> In January, we sent an RFC [1] to update userspace to integrate CIL
> [2] and source policy. And in April, we sent an updated RFC [3] which
> added support for high level languages and a tool to convert policy
> package (pp) files to CIL. After getting some good feedback, we have
> made some more changes, mostly to maintain ABI compatibility. The
> major changes made since the last patchset are:
> 
> - Change how semanage_set_root was re-added to use the source policy
>   infrastructure. Fixes were made so that semanage.conf was looked for
>   inside the root. Also adds an semanage_root() function to get the
>   current root.
> - In previous patchsets, the semanage_module_upgrade* and
>   semanage_module_install_base* functions were removed from the API,
>   and semanage_module_install* had modified parameters. However, these
>   changes broke the API and ABI. To maintain ABI compatibility, we've
>   now added symbolic versioning to support the old version of the
>   functions, which now just call the new install functions. semodule
>   is updated to support --base and --upgrade, but with the addition of
>   a deprecation message. API compatability is not maintained.
> - Likewise, symbolic versioning was added to support the old module
>   enable/disable functions, which call the new enable/disable
>   functions.
> - Modify the libsepol Makefile to now make including CIL optional via
>   the DISABLE_CIL build flag. This only affects libsepol (not
>   libsemanage), primarily so that SE for Android does not need to
>   include unused CIL cruft.
> 
> With these changes, ABI compatibility is maintained. Additionally, we
> have tested these changes with the userspace tests and against the
> kernel test suite, and no new failures were discovered. We have
> also tested this patchset with both Fedora 20 policy and with reference
> policy and found no errors.
> 
> Because of the size of the patchset (67 file changes, ~8300
> insertions, ~1800 deletions), all the changes have been pushed to the
> selinux git repository to the 'integration' branch for
> comments/review. Unlike the previous RFCs, for simplicity there is now
> only a single branch, containing three types of changes:
> 
> Reverts
>    Reverts changes made to master that conflict with the new source
>    policy infrastructure (e.g. how paths are handled,
>    enabled/disable modules). Rather than dealing with a large amount
>    of conflicts with the source policy work, it was easier to just
>    remove the commits that added conflicting features, rebase the old
>    source policy work on top of that, and add back any features in a
>    manner consistent with source policy. The only conflicts were
>    related to enabling/disabling of modules, and semanage_set_root.
> 
> Source Policy
>    This is a rebase of the old src-policy branch on top of the
>    reverted commits.  The goal of these changes is to improve the API
>    for module handling, add support for source policies, module
>    priorities, enabling/disabling of modules, and moving the policy
>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
> 
> CIL Integration
>    These changes build CIL into libsepol, and updates libsepol,
>    libsemanage, semodule, and semanage to work with and understand CIL
>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>    CIL has a few side effects, such as removing base modules,
>    versions, and upgrades.
> 
>    This also adds a new tool (installed to
>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>    converts binary pp modules to CIL. The infrastructure to use this
>    compiler (or any other HLL compiler) was added to compile HLL
>    modules to CIL, which is accomplished by writing the HLL data to
>    the stdin of the compiler and reading the equivilent CIL from
>    stdout. The resulting CIL is then cached in the policy store so
>    this compilation does not need to take place during future store
>    updates. Cached CIL modules can be ignored using a new semodule
>    flag (-C/--ignore-cache) or a new configuration option in
>    semanage.conf (ignore-cache). Other configuration options were
>    added to semanage.conf to manage the path to HLL compilers
>    (compiler-directory) and the policy store (store-root). Semodule
>    was also modified to support changing the policy store with the
>    -S/--store-root option.
> 
>    Lastly, the CIL integration changes required changes to the API,
>    but symbolic versioning was used to maintain ABI compatibility.
>    Because of this, the .so version is no longer incremented like in
>    the previous version of this RFC.
> 
> With these changes, it is possible to build and manage SELinux
> policy using pp and CIL modules and the familiar semodule/semanage
> tools.
> 
> To make this easier to experiment with and test, below are the steps
> needed to install the updated userspace and migrate a minimal Fedora 20
> installation to the new policy store.
> 
> Thanks, and we look forward to any questions/comments.
> 
> - Steve
> 
> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
> [2] https://github.com/SELinuxProject/cil/wiki
> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
> 
> 
> Steps to Install SELinux Userspace with source policy, CIL, and HLL
> 
> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
> 
> # Install SELinux userspace dependencies
> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
> setools-devel swig ustr-devel
> 
> # Update to the latest targeted policy
> $ yum update selinux-policy-targeted
> 
> # Clone the repos and checkout branches
> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
> $ git clone -b master https://github.com/SELinuxProject/cil.git
> 
> # Create a symlink to the cil repo so CIL can be built into libsepol
> $ ln -s ~/cil/ selinux/libsepol/cil
> 
> # Install SELinux userspace with CIL integration and HLL support
> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
> 
> # Migrate to the new source policy infrastructure
> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py

Is this expected:
# ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
Migrating from /etc/selinux/targeted/modules/active to
/var/lib/selinux/targeted/active
Attempting to rebuild policy from /var/lib/selinux
sysnetwork: Warning: 'else' blocks in optional statements are
unsupported in CIL. Dropping from output.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 09:52 -0400, Stephen Smalley wrote:

> Is this expected:
> # ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
> Migrating from /etc/selinux/targeted/modules/active to
> /var/lib/selinux/targeted/active
> Attempting to rebuild policy from /var/lib/selinux
> sysnetwork: Warning: 'else' blocks in optional statements are
> unsupported in CIL. Dropping from output.
> 

Maybe, but this particular one (in sysnetwork) seem to me a bug in
policy or at least not optimal, maybe even redundant. I encountered the
issue earlier, looked at it and i could not make sense out of it.

I concluded that similar result could be achieved without that "else"

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 09:52 AM, Stephen Smalley wrote:
> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
>>
>> - Change how semanage_set_root was re-added to use the source policy
>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>   inside the root. Also adds an semanage_root() function to get the
>>   current root.
>> - In previous patchsets, the semanage_module_upgrade* and
>>   semanage_module_install_base* functions were removed from the API,
>>   and semanage_module_install* had modified parameters. However, these
>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>   now added symbolic versioning to support the old version of the
>>   functions, which now just call the new install functions. semodule
>>   is updated to support --base and --upgrade, but with the addition of
>>   a deprecation message. API compatability is not maintained.
>> - Likewise, symbolic versioning was added to support the old module
>>   enable/disable functions, which call the new enable/disable
>>   functions.
>> - Modify the libsepol Makefile to now make including CIL optional via
>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>   libsemanage), primarily so that SE for Android does not need to
>>   include unused CIL cruft.
>>
>> With these changes, ABI compatibility is maintained. Additionally, we
>> have tested these changes with the userspace tests and against the
>> kernel test suite, and no new failures were discovered. We have
>> also tested this patchset with both Fedora 20 policy and with reference
>> policy and found no errors.
>>
>> Because of the size of the patchset (67 file changes, ~8300
>> insertions, ~1800 deletions), all the changes have been pushed to the
>> selinux git repository to the 'integration' branch for
>> comments/review. Unlike the previous RFCs, for simplicity there is now
>> only a single branch, containing three types of changes:
>>
>> Reverts
>>    Reverts changes made to master that conflict with the new source
>>    policy infrastructure (e.g. how paths are handled,
>>    enabled/disable modules). Rather than dealing with a large amount
>>    of conflicts with the source policy work, it was easier to just
>>    remove the commits that added conflicting features, rebase the old
>>    source policy work on top of that, and add back any features in a
>>    manner consistent with source policy. The only conflicts were
>>    related to enabling/disabling of modules, and semanage_set_root.
>>
>> Source Policy
>>    This is a rebase of the old src-policy branch on top of the
>>    reverted commits.  The goal of these changes is to improve the API
>>    for module handling, add support for source policies, module
>>    priorities, enabling/disabling of modules, and moving the policy
>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>
>> CIL Integration
>>    These changes build CIL into libsepol, and updates libsepol,
>>    libsemanage, semodule, and semanage to work with and understand CIL
>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>    CIL has a few side effects, such as removing base modules,
>>    versions, and upgrades.
>>
>>    This also adds a new tool (installed to
>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>    converts binary pp modules to CIL. The infrastructure to use this
>>    compiler (or any other HLL compiler) was added to compile HLL
>>    modules to CIL, which is accomplished by writing the HLL data to
>>    the stdin of the compiler and reading the equivilent CIL from
>>    stdout. The resulting CIL is then cached in the policy store so
>>    this compilation does not need to take place during future store
>>    updates. Cached CIL modules can be ignored using a new semodule
>>    flag (-C/--ignore-cache) or a new configuration option in
>>    semanage.conf (ignore-cache). Other configuration options were
>>    added to semanage.conf to manage the path to HLL compilers
>>    (compiler-directory) and the policy store (store-root). Semodule
>>    was also modified to support changing the policy store with the
>>    -S/--store-root option.
>>
>>    Lastly, the CIL integration changes required changes to the API,
>>    but symbolic versioning was used to maintain ABI compatibility.
>>    Because of this, the .so version is no longer incremented like in
>>    the previous version of this RFC.
>>
>> With these changes, it is possible to build and manage SELinux
>> policy using pp and CIL modules and the familiar semodule/semanage
>> tools.
>>
>> To make this easier to experiment with and test, below are the steps
>> needed to install the updated userspace and migrate a minimal Fedora 20
>> installation to the new policy store.
>>
>> Thanks, and we look forward to any questions/comments.
>>
>> - Steve
>>
>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>> [2] https://github.com/SELinuxProject/cil/wiki
>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>
>>
>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>
>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>
>> # Install SELinux userspace dependencies
>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>> setools-devel swig ustr-devel
>>
>> # Update to the latest targeted policy
>> $ yum update selinux-policy-targeted
>>
>> # Clone the repos and checkout branches
>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>
>> # Create a symlink to the cil repo so CIL can be built into libsepol
>> $ ln -s ~/cil/ selinux/libsepol/cil
>>
>> # Install SELinux userspace with CIL integration and HLL support
>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>
>> # Migrate to the new source policy infrastructure
>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
> 
> Is this expected:
> # ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
> Migrating from /etc/selinux/targeted/modules/active to
> /var/lib/selinux/targeted/active
> Attempting to rebuild policy from /var/lib/selinux
> sysnetwork: Warning: 'else' blocks in optional statements are
> unsupported in CIL. Dropping from output.
> 

Yes. The sysnetwork module is the only module that uses 'else' in an
optional statement. The relevant refpolicy is:

  optional_policy(`
      netutils_run_ping(dhcpc_t, dhcpc_roles)
      netutils_run(dhcpc_t, dhcpc_roles)
  ',`
      allow dhcpc_t self:capability setuid;
      allow dhcpc_t self:rawip_socket create_socket_perms;
  ')

So the 'else' block is only hit if the netutils module isn't installed
(which it is by default).

CIL does not support else blocks in optional statements, so the decision
was made to just drop the else block with a warning.

Re: [RFC] Source Policy, CIL, and High Level Languages

[James Carter]

On 07/10/2014 10:09 AM, Steve Lawrence wrote:
> On 07/10/2014 09:52 AM, Stephen Smalley wrote:
>> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>> added support for high level languages and a tool to convert policy
>>> package (pp) files to CIL. After getting some good feedback, we have
>>> made some more changes, mostly to maintain ABI compatibility. The
>>> major changes made since the last patchset are:
>>>
>>> - Change how semanage_set_root was re-added to use the source policy
>>>    infrastructure. Fixes were made so that semanage.conf was looked for
>>>    inside the root. Also adds an semanage_root() function to get the
>>>    current root.
>>> - In previous patchsets, the semanage_module_upgrade* and
>>>    semanage_module_install_base* functions were removed from the API,
>>>    and semanage_module_install* had modified parameters. However, these
>>>    changes broke the API and ABI. To maintain ABI compatibility, we've
>>>    now added symbolic versioning to support the old version of the
>>>    functions, which now just call the new install functions. semodule
>>>    is updated to support --base and --upgrade, but with the addition of
>>>    a deprecation message. API compatability is not maintained.
>>> - Likewise, symbolic versioning was added to support the old module
>>>    enable/disable functions, which call the new enable/disable
>>>    functions.
>>> - Modify the libsepol Makefile to now make including CIL optional via
>>>    the DISABLE_CIL build flag. This only affects libsepol (not
>>>    libsemanage), primarily so that SE for Android does not need to
>>>    include unused CIL cruft.
>>>
>>> With these changes, ABI compatibility is maintained. Additionally, we
>>> have tested these changes with the userspace tests and against the
>>> kernel test suite, and no new failures were discovered. We have
>>> also tested this patchset with both Fedora 20 policy and with reference
>>> policy and found no errors.
>>>
>>> Because of the size of the patchset (67 file changes, ~8300
>>> insertions, ~1800 deletions), all the changes have been pushed to the
>>> selinux git repository to the 'integration' branch for
>>> comments/review. Unlike the previous RFCs, for simplicity there is now
>>> only a single branch, containing three types of changes:
>>>
>>> Reverts
>>>     Reverts changes made to master that conflict with the new source
>>>     policy infrastructure (e.g. how paths are handled,
>>>     enabled/disable modules). Rather than dealing with a large amount
>>>     of conflicts with the source policy work, it was easier to just
>>>     remove the commits that added conflicting features, rebase the old
>>>     source policy work on top of that, and add back any features in a
>>>     manner consistent with source policy. The only conflicts were
>>>     related to enabling/disabling of modules, and semanage_set_root.
>>>
>>> Source Policy
>>>     This is a rebase of the old src-policy branch on top of the
>>>     reverted commits.  The goal of these changes is to improve the API
>>>     for module handling, add support for source policies, module
>>>     priorities, enabling/disabling of modules, and moving the policy
>>>     store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>>
>>> CIL Integration
>>>     These changes build CIL into libsepol, and updates libsepol,
>>>     libsemanage, semodule, and semanage to work with and understand CIL
>>>     files and manage /var/lib/selinux and /etc/selinux. Switching to
>>>     CIL has a few side effects, such as removing base modules,
>>>     versions, and upgrades.
>>>
>>>     This also adds a new tool (installed to
>>>     /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>>     converts binary pp modules to CIL. The infrastructure to use this
>>>     compiler (or any other HLL compiler) was added to compile HLL
>>>     modules to CIL, which is accomplished by writing the HLL data to
>>>     the stdin of the compiler and reading the equivilent CIL from
>>>     stdout. The resulting CIL is then cached in the policy store so
>>>     this compilation does not need to take place during future store
>>>     updates. Cached CIL modules can be ignored using a new semodule
>>>     flag (-C/--ignore-cache) or a new configuration option in
>>>     semanage.conf (ignore-cache). Other configuration options were
>>>     added to semanage.conf to manage the path to HLL compilers
>>>     (compiler-directory) and the policy store (store-root). Semodule
>>>     was also modified to support changing the policy store with the
>>>     -S/--store-root option.
>>>
>>>     Lastly, the CIL integration changes required changes to the API,
>>>     but symbolic versioning was used to maintain ABI compatibility.
>>>     Because of this, the .so version is no longer incremented like in
>>>     the previous version of this RFC.
>>>
>>> With these changes, it is possible to build and manage SELinux
>>> policy using pp and CIL modules and the familiar semodule/semanage
>>> tools.
>>>
>>> To make this easier to experiment with and test, below are the steps
>>> needed to install the updated userspace and migrate a minimal Fedora 20
>>> installation to the new policy store.
>>>
>>> Thanks, and we look forward to any questions/comments.
>>>
>>> - Steve
>>>
>>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>>> [2] https://github.com/SELinuxProject/cil/wiki
>>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>>
>>>
>>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>>
>>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>>
>>> # Install SELinux userspace dependencies
>>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>>> setools-devel swig ustr-devel
>>>
>>> # Update to the latest targeted policy
>>> $ yum update selinux-policy-targeted
>>>
>>> # Clone the repos and checkout branches
>>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>>
>>> # Create a symlink to the cil repo so CIL can be built into libsepol
>>> $ ln -s ~/cil/ selinux/libsepol/cil
>>>
>>> # Install SELinux userspace with CIL integration and HLL support
>>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>>
>>> # Migrate to the new source policy infrastructure
>>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>
>> Is this expected:
>> # ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>> Migrating from /etc/selinux/targeted/modules/active to
>> /var/lib/selinux/targeted/active
>> Attempting to rebuild policy from /var/lib/selinux
>> sysnetwork: Warning: 'else' blocks in optional statements are
>> unsupported in CIL. Dropping from output.
>>
>
> Yes. The sysnetwork module is the only module that uses 'else' in an
> optional statement. The relevant refpolicy is:
>
>    optional_policy(`
>        netutils_run_ping(dhcpc_t, dhcpc_roles)
>        netutils_run(dhcpc_t, dhcpc_roles)
>    ',`
>        allow dhcpc_t self:capability setuid;
>        allow dhcpc_t self:rawip_socket create_socket_perms;
>    ')
>
> So the 'else' block is only hit if the netutils module isn't installed
> (which it is by default).
>
> CIL does not support else blocks in optional statements, so the decision
> was made to just drop the else block with a warning.


It would be easy to get rid of this. I use the following patch when I convert 
Refpolicy.

diff --git a/policy/modules/admin/netutils.te b/policy/modules/admin/netutils.te
index 557da97..7aa88e0 100644
--- a/policy/modules/admin/netutils.te
+++ b/policy/modules/admin/netutils.te
@@ -1,5 +1,7 @@
  policy_module(netutils, 1.12.0)

+define(`use_netutils')
+
  ########################################
  #
  # Declarations
diff --git a/policy/modules/system/sysnetwork.te 
b/policy/modules/system/sysnetwork.te
index 11247e2..3390a59 100644
--- a/policy/modules/system/sysnetwork.te
+++ b/policy/modules/system/sysnetwork.te
@@ -190,7 +190,9 @@ optional_policy(`
  optional_policy(`
  	netutils_run_ping(dhcpc_t, dhcpc_roles)
  	netutils_run(dhcpc_t, dhcpc_roles)
-',`
+')
+
+ifndef(`use_netutils',`
  	allow dhcpc_t self:capability setuid;
  	allow dhcpc_t self:rawip_socket create_socket_perms;
  ')

-- 
James Carter <jwcart2@tycho.nsa.gov>
National Security Agency

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/09/2014 03:21 PM, Steve Lawrence wrote:
> In January, we sent an RFC [1] to update userspace to integrate CIL
> [2] and source policy. And in April, we sent an updated RFC [3] which
> added support for high level languages and a tool to convert policy
> package (pp) files to CIL. After getting some good feedback, we have
> made some more changes, mostly to maintain ABI compatibility. The
> major changes made since the last patchset are:
> 
> - Change how semanage_set_root was re-added to use the source policy
>   infrastructure. Fixes were made so that semanage.conf was looked for
>   inside the root. Also adds an semanage_root() function to get the
>   current root.
> - In previous patchsets, the semanage_module_upgrade* and
>   semanage_module_install_base* functions were removed from the API,
>   and semanage_module_install* had modified parameters. However, these
>   changes broke the API and ABI. To maintain ABI compatibility, we've
>   now added symbolic versioning to support the old version of the
>   functions, which now just call the new install functions. semodule
>   is updated to support --base and --upgrade, but with the addition of
>   a deprecation message. API compatability is not maintained.
> - Likewise, symbolic versioning was added to support the old module
>   enable/disable functions, which call the new enable/disable
>   functions.
> - Modify the libsepol Makefile to now make including CIL optional via
>   the DISABLE_CIL build flag. This only affects libsepol (not
>   libsemanage), primarily so that SE for Android does not need to
>   include unused CIL cruft.
> 
> With these changes, ABI compatibility is maintained. Additionally, we
> have tested these changes with the userspace tests and against the
> kernel test suite, and no new failures were discovered. We have
> also tested this patchset with both Fedora 20 policy and with reference
> policy and found no errors.
> 
> Because of the size of the patchset (67 file changes, ~8300
> insertions, ~1800 deletions), all the changes have been pushed to the
> selinux git repository to the 'integration' branch for
> comments/review. Unlike the previous RFCs, for simplicity there is now
> only a single branch, containing three types of changes:
> 
> Reverts
>    Reverts changes made to master that conflict with the new source
>    policy infrastructure (e.g. how paths are handled,
>    enabled/disable modules). Rather than dealing with a large amount
>    of conflicts with the source policy work, it was easier to just
>    remove the commits that added conflicting features, rebase the old
>    source policy work on top of that, and add back any features in a
>    manner consistent with source policy. The only conflicts were
>    related to enabling/disabling of modules, and semanage_set_root.
> 
> Source Policy
>    This is a rebase of the old src-policy branch on top of the
>    reverted commits.  The goal of these changes is to improve the API
>    for module handling, add support for source policies, module
>    priorities, enabling/disabling of modules, and moving the policy
>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
> 
> CIL Integration
>    These changes build CIL into libsepol, and updates libsepol,
>    libsemanage, semodule, and semanage to work with and understand CIL
>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>    CIL has a few side effects, such as removing base modules,
>    versions, and upgrades.
> 
>    This also adds a new tool (installed to
>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>    converts binary pp modules to CIL. The infrastructure to use this
>    compiler (or any other HLL compiler) was added to compile HLL
>    modules to CIL, which is accomplished by writing the HLL data to
>    the stdin of the compiler and reading the equivilent CIL from
>    stdout. The resulting CIL is then cached in the policy store so
>    this compilation does not need to take place during future store
>    updates. Cached CIL modules can be ignored using a new semodule
>    flag (-C/--ignore-cache) or a new configuration option in
>    semanage.conf (ignore-cache). Other configuration options were
>    added to semanage.conf to manage the path to HLL compilers
>    (compiler-directory) and the policy store (store-root). Semodule
>    was also modified to support changing the policy store with the
>    -S/--store-root option.
> 
>    Lastly, the CIL integration changes required changes to the API,
>    but symbolic versioning was used to maintain ABI compatibility.
>    Because of this, the .so version is no longer incremented like in
>    the previous version of this RFC.
> 
> With these changes, it is possible to build and manage SELinux
> policy using pp and CIL modules and the familiar semodule/semanage
> tools.
> 
> To make this easier to experiment with and test, below are the steps
> needed to install the updated userspace and migrate a minimal Fedora 20
> installation to the new policy store.
> 
> Thanks, and we look forward to any questions/comments.
> 
> - Steve
> 
> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
> [2] https://github.com/SELinuxProject/cil/wiki
> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
> 
> 
> Steps to Install SELinux Userspace with source policy, CIL, and HLL
> 
> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
> 
> # Install SELinux userspace dependencies
> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
> setools-devel swig ustr-devel
> 
> # Update to the latest targeted policy
> $ yum update selinux-policy-targeted
> 
> # Clone the repos and checkout branches
> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
> $ git clone -b master https://github.com/SELinuxProject/cil.git
> 
> # Create a symlink to the cil repo so CIL can be built into libsepol
> $ ln -s ~/cil/ selinux/libsepol/cil
> 
> # Install SELinux userspace with CIL integration and HLL support
> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
> 
> # Migrate to the new source policy infrastructure
> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
> 
> # List the installed modules, showing priority and HLL
> $ semodule --list=full

valgrind memcheck reports some issues:
# valgrind --leak-check=full setsebool -P httpd_can_network_connect=1
==10089== Memcheck, a memory error detector
==10089== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10089== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==10089== Command: setsebool -P httpd_can_network_connect=1
==10089==
==10089== Conditional jump or move depends on uninitialised value(s)
==10089==    at 0x511F50A: semanage_compile_hll (direct_api.c:937)
==10089==    by 0x511FD97: semanage_direct_commit (direct_api.c:1071)
==10089==    by 0x512DF59: semanage_commit (handle.c:426)
==10089==    by 0x4019C2: semanage_set_boolean_list (setsebool.c:206)
==10089==    by 0x401C48: setbool (setsebool.c:271)
==10089==    by 0x40161A: main (setsebool.c:94)
==10089==
(still running, may be more...)

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 09:59 AM, Stephen Smalley wrote:
> 
> valgrind memcheck reports some issues:
> # valgrind --leak-check=full setsebool -P httpd_can_network_connect=1
> ==10089== Memcheck, a memory error detector
> ==10089== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
> ==10089== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
> ==10089== Command: setsebool -P httpd_can_network_connect=1
> ==10089==
> ==10089== Conditional jump or move depends on uninitialised value(s)
> ==10089==    at 0x511F50A: semanage_compile_hll (direct_api.c:937)
> ==10089==    by 0x511FD97: semanage_direct_commit (direct_api.c:1071)
> ==10089==    by 0x512DF59: semanage_commit (handle.c:426)
> ==10089==    by 0x4019C2: semanage_set_boolean_list (setsebool.c:206)
> ==10089==    by 0x401C48: setbool (setsebool.c:271)
> ==10089==    by 0x40161A: main (setsebool.c:94)
> ==10089==
> (still running, may be more...)
> 

This issue has been fixed and squashed/pushed into the integration
branch. We were missing a variable initiation, which affected cases
where no HLL modules needed to be compiled (i.e. all modules had been
previously compiled and cached).

Thanks,
- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/09/2014 03:21 PM, Steve Lawrence wrote:
> In January, we sent an RFC [1] to update userspace to integrate CIL
> [2] and source policy. And in April, we sent an updated RFC [3] which
> added support for high level languages and a tool to convert policy
> package (pp) files to CIL. After getting some good feedback, we have
> made some more changes, mostly to maintain ABI compatibility. The
> major changes made since the last patchset are:
> 
> - Change how semanage_set_root was re-added to use the source policy
>   infrastructure. Fixes were made so that semanage.conf was looked for
>   inside the root. Also adds an semanage_root() function to get the
>   current root.
> - In previous patchsets, the semanage_module_upgrade* and
>   semanage_module_install_base* functions were removed from the API,
>   and semanage_module_install* had modified parameters. However, these
>   changes broke the API and ABI. To maintain ABI compatibility, we've
>   now added symbolic versioning to support the old version of the
>   functions, which now just call the new install functions. semodule
>   is updated to support --base and --upgrade, but with the addition of
>   a deprecation message. API compatability is not maintained.
> - Likewise, symbolic versioning was added to support the old module
>   enable/disable functions, which call the new enable/disable
>   functions.
> - Modify the libsepol Makefile to now make including CIL optional via
>   the DISABLE_CIL build flag. This only affects libsepol (not
>   libsemanage), primarily so that SE for Android does not need to
>   include unused CIL cruft.
> 
> With these changes, ABI compatibility is maintained. Additionally, we
> have tested these changes with the userspace tests and against the
> kernel test suite, and no new failures were discovered. We have
> also tested this patchset with both Fedora 20 policy and with reference
> policy and found no errors.
> 
> Because of the size of the patchset (67 file changes, ~8300
> insertions, ~1800 deletions), all the changes have been pushed to the
> selinux git repository to the 'integration' branch for
> comments/review. Unlike the previous RFCs, for simplicity there is now
> only a single branch, containing three types of changes:
> 
> Reverts
>    Reverts changes made to master that conflict with the new source
>    policy infrastructure (e.g. how paths are handled,
>    enabled/disable modules). Rather than dealing with a large amount
>    of conflicts with the source policy work, it was easier to just
>    remove the commits that added conflicting features, rebase the old
>    source policy work on top of that, and add back any features in a
>    manner consistent with source policy. The only conflicts were
>    related to enabling/disabling of modules, and semanage_set_root.
> 
> Source Policy
>    This is a rebase of the old src-policy branch on top of the
>    reverted commits.  The goal of these changes is to improve the API
>    for module handling, add support for source policies, module
>    priorities, enabling/disabling of modules, and moving the policy
>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
> 
> CIL Integration
>    These changes build CIL into libsepol, and updates libsepol,
>    libsemanage, semodule, and semanage to work with and understand CIL
>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>    CIL has a few side effects, such as removing base modules,
>    versions, and upgrades.
> 
>    This also adds a new tool (installed to
>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>    converts binary pp modules to CIL. The infrastructure to use this
>    compiler (or any other HLL compiler) was added to compile HLL
>    modules to CIL, which is accomplished by writing the HLL data to
>    the stdin of the compiler and reading the equivilent CIL from
>    stdout. The resulting CIL is then cached in the policy store so
>    this compilation does not need to take place during future store
>    updates. Cached CIL modules can be ignored using a new semodule
>    flag (-C/--ignore-cache) or a new configuration option in
>    semanage.conf (ignore-cache). Other configuration options were
>    added to semanage.conf to manage the path to HLL compilers
>    (compiler-directory) and the policy store (store-root). Semodule
>    was also modified to support changing the policy store with the
>    -S/--store-root option.
> 
>    Lastly, the CIL integration changes required changes to the API,
>    but symbolic versioning was used to maintain ABI compatibility.
>    Because of this, the .so version is no longer incremented like in
>    the previous version of this RFC.
> 
> With these changes, it is possible to build and manage SELinux
> policy using pp and CIL modules and the familiar semodule/semanage
> tools.
> 
> To make this easier to experiment with and test, below are the steps
> needed to install the updated userspace and migrate a minimal Fedora 20
> installation to the new policy store.
> 
> Thanks, and we look forward to any questions/comments.
> 
> - Steve
> 
> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
> [2] https://github.com/SELinuxProject/cil/wiki
> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
> 
> 
> Steps to Install SELinux Userspace with source policy, CIL, and HLL
> 
> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
> 
> # Install SELinux userspace dependencies
> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
> setools-devel swig ustr-devel
> 
> # Update to the latest targeted policy
> $ yum update selinux-policy-targeted
> 
> # Clone the repos and checkout branches
> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
> $ git clone -b master https://github.com/SELinuxProject/cil.git
> 
> # Create a symlink to the cil repo so CIL can be built into libsepol
> $ ln -s ~/cil/ selinux/libsepol/cil
> 
> # Install SELinux userspace with CIL integration and HLL support
> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
> 
> # Migrate to the new source policy infrastructure
> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
> 
> # List the installed modules, showing priority and HLL
> $ semodule --list=full

Is the classorder bug?
$ su <hangs forever>
$ dmesg
 systemd[1]: SELinux policy denies access.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 10:11 AM, Stephen Smalley wrote:
> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
>>
>> - Change how semanage_set_root was re-added to use the source policy
>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>   inside the root. Also adds an semanage_root() function to get the
>>   current root.
>> - In previous patchsets, the semanage_module_upgrade* and
>>   semanage_module_install_base* functions were removed from the API,
>>   and semanage_module_install* had modified parameters. However, these
>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>   now added symbolic versioning to support the old version of the
>>   functions, which now just call the new install functions. semodule
>>   is updated to support --base and --upgrade, but with the addition of
>>   a deprecation message. API compatability is not maintained.
>> - Likewise, symbolic versioning was added to support the old module
>>   enable/disable functions, which call the new enable/disable
>>   functions.
>> - Modify the libsepol Makefile to now make including CIL optional via
>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>   libsemanage), primarily so that SE for Android does not need to
>>   include unused CIL cruft.
>>
>> With these changes, ABI compatibility is maintained. Additionally, we
>> have tested these changes with the userspace tests and against the
>> kernel test suite, and no new failures were discovered. We have
>> also tested this patchset with both Fedora 20 policy and with reference
>> policy and found no errors.
>>
>> Because of the size of the patchset (67 file changes, ~8300
>> insertions, ~1800 deletions), all the changes have been pushed to the
>> selinux git repository to the 'integration' branch for
>> comments/review. Unlike the previous RFCs, for simplicity there is now
>> only a single branch, containing three types of changes:
>>
>> Reverts
>>    Reverts changes made to master that conflict with the new source
>>    policy infrastructure (e.g. how paths are handled,
>>    enabled/disable modules). Rather than dealing with a large amount
>>    of conflicts with the source policy work, it was easier to just
>>    remove the commits that added conflicting features, rebase the old
>>    source policy work on top of that, and add back any features in a
>>    manner consistent with source policy. The only conflicts were
>>    related to enabling/disabling of modules, and semanage_set_root.
>>
>> Source Policy
>>    This is a rebase of the old src-policy branch on top of the
>>    reverted commits.  The goal of these changes is to improve the API
>>    for module handling, add support for source policies, module
>>    priorities, enabling/disabling of modules, and moving the policy
>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>
>> CIL Integration
>>    These changes build CIL into libsepol, and updates libsepol,
>>    libsemanage, semodule, and semanage to work with and understand CIL
>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>    CIL has a few side effects, such as removing base modules,
>>    versions, and upgrades.
>>
>>    This also adds a new tool (installed to
>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>    converts binary pp modules to CIL. The infrastructure to use this
>>    compiler (or any other HLL compiler) was added to compile HLL
>>    modules to CIL, which is accomplished by writing the HLL data to
>>    the stdin of the compiler and reading the equivilent CIL from
>>    stdout. The resulting CIL is then cached in the policy store so
>>    this compilation does not need to take place during future store
>>    updates. Cached CIL modules can be ignored using a new semodule
>>    flag (-C/--ignore-cache) or a new configuration option in
>>    semanage.conf (ignore-cache). Other configuration options were
>>    added to semanage.conf to manage the path to HLL compilers
>>    (compiler-directory) and the policy store (store-root). Semodule
>>    was also modified to support changing the policy store with the
>>    -S/--store-root option.
>>
>>    Lastly, the CIL integration changes required changes to the API,
>>    but symbolic versioning was used to maintain ABI compatibility.
>>    Because of this, the .so version is no longer incremented like in
>>    the previous version of this RFC.
>>
>> With these changes, it is possible to build and manage SELinux
>> policy using pp and CIL modules and the familiar semodule/semanage
>> tools.
>>
>> To make this easier to experiment with and test, below are the steps
>> needed to install the updated userspace and migrate a minimal Fedora 20
>> installation to the new policy store.
>>
>> Thanks, and we look forward to any questions/comments.
>>
>> - Steve
>>
>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>> [2] https://github.com/SELinuxProject/cil/wiki
>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>
>>
>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>
>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>
>> # Install SELinux userspace dependencies
>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>> setools-devel swig ustr-devel
>>
>> # Update to the latest targeted policy
>> $ yum update selinux-policy-targeted
>>
>> # Clone the repos and checkout branches
>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>
>> # Create a symlink to the cil repo so CIL can be built into libsepol
>> $ ln -s ~/cil/ selinux/libsepol/cil
>>
>> # Install SELinux userspace with CIL integration and HLL support
>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>
>> # Migrate to the new source policy infrastructure
>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>
>> # List the installed modules, showing priority and HLL
>> $ semodule --list=full
> 
> Is the classorder bug?
> $ su <hangs forever>
> $ dmesg
>  systemd[1]: SELinux policy denies access.

Ok, not forever - but it does take a while to timeout.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 10:11 AM, Stephen Smalley wrote:
> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
>>
>> - Change how semanage_set_root was re-added to use the source policy
>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>   inside the root. Also adds an semanage_root() function to get the
>>   current root.
>> - In previous patchsets, the semanage_module_upgrade* and
>>   semanage_module_install_base* functions were removed from the API,
>>   and semanage_module_install* had modified parameters. However, these
>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>   now added symbolic versioning to support the old version of the
>>   functions, which now just call the new install functions. semodule
>>   is updated to support --base and --upgrade, but with the addition of
>>   a deprecation message. API compatability is not maintained.
>> - Likewise, symbolic versioning was added to support the old module
>>   enable/disable functions, which call the new enable/disable
>>   functions.
>> - Modify the libsepol Makefile to now make including CIL optional via
>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>   libsemanage), primarily so that SE for Android does not need to
>>   include unused CIL cruft.
>>
>> With these changes, ABI compatibility is maintained. Additionally, we
>> have tested these changes with the userspace tests and against the
>> kernel test suite, and no new failures were discovered. We have
>> also tested this patchset with both Fedora 20 policy and with reference
>> policy and found no errors.
>>
>> Because of the size of the patchset (67 file changes, ~8300
>> insertions, ~1800 deletions), all the changes have been pushed to the
>> selinux git repository to the 'integration' branch for
>> comments/review. Unlike the previous RFCs, for simplicity there is now
>> only a single branch, containing three types of changes:
>>
>> Reverts
>>    Reverts changes made to master that conflict with the new source
>>    policy infrastructure (e.g. how paths are handled,
>>    enabled/disable modules). Rather than dealing with a large amount
>>    of conflicts with the source policy work, it was easier to just
>>    remove the commits that added conflicting features, rebase the old
>>    source policy work on top of that, and add back any features in a
>>    manner consistent with source policy. The only conflicts were
>>    related to enabling/disabling of modules, and semanage_set_root.
>>
>> Source Policy
>>    This is a rebase of the old src-policy branch on top of the
>>    reverted commits.  The goal of these changes is to improve the API
>>    for module handling, add support for source policies, module
>>    priorities, enabling/disabling of modules, and moving the policy
>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>
>> CIL Integration
>>    These changes build CIL into libsepol, and updates libsepol,
>>    libsemanage, semodule, and semanage to work with and understand CIL
>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>    CIL has a few side effects, such as removing base modules,
>>    versions, and upgrades.
>>
>>    This also adds a new tool (installed to
>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>    converts binary pp modules to CIL. The infrastructure to use this
>>    compiler (or any other HLL compiler) was added to compile HLL
>>    modules to CIL, which is accomplished by writing the HLL data to
>>    the stdin of the compiler and reading the equivilent CIL from
>>    stdout. The resulting CIL is then cached in the policy store so
>>    this compilation does not need to take place during future store
>>    updates. Cached CIL modules can be ignored using a new semodule
>>    flag (-C/--ignore-cache) or a new configuration option in
>>    semanage.conf (ignore-cache). Other configuration options were
>>    added to semanage.conf to manage the path to HLL compilers
>>    (compiler-directory) and the policy store (store-root). Semodule
>>    was also modified to support changing the policy store with the
>>    -S/--store-root option.
>>
>>    Lastly, the CIL integration changes required changes to the API,
>>    but symbolic versioning was used to maintain ABI compatibility.
>>    Because of this, the .so version is no longer incremented like in
>>    the previous version of this RFC.
>>
>> With these changes, it is possible to build and manage SELinux
>> policy using pp and CIL modules and the familiar semodule/semanage
>> tools.
>>
>> To make this easier to experiment with and test, below are the steps
>> needed to install the updated userspace and migrate a minimal Fedora 20
>> installation to the new policy store.
>>
>> Thanks, and we look forward to any questions/comments.
>>
>> - Steve
>>
>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>> [2] https://github.com/SELinuxProject/cil/wiki
>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>
>>
>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>
>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>
>> # Install SELinux userspace dependencies
>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>> setools-devel swig ustr-devel
>>
>> # Update to the latest targeted policy
>> $ yum update selinux-policy-targeted
>>
>> # Clone the repos and checkout branches
>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>
>> # Create a symlink to the cil repo so CIL can be built into libsepol
>> $ ln -s ~/cil/ selinux/libsepol/cil
>>
>> # Install SELinux userspace with CIL integration and HLL support
>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>
>> # Migrate to the new source policy infrastructure
>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>
>> # List the installed modules, showing priority and HLL
>> $ semodule --list=full
> 
> Is the classorder bug?
> $ su <hangs forever>
> $ dmesg
>  systemd[1]: SELinux policy denies access.
> 
> 

The classorder bug has been fixed with the latest cil & integration
branch, and shouldn't have been a problem with fedora's policy since the
pp2cil tool should have output the classes in the correct order.

I'm not able to reproduce this problem. su works correctly and
immediately for me. What commands did you run before this?

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 10:17 AM, Steve Lawrence wrote:
> On 07/10/2014 10:11 AM, Stephen Smalley wrote:
>> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>> added support for high level languages and a tool to convert policy
>>> package (pp) files to CIL. After getting some good feedback, we have
>>> made some more changes, mostly to maintain ABI compatibility. The
>>> major changes made since the last patchset are:
>>>
>>> - Change how semanage_set_root was re-added to use the source policy
>>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>>   inside the root. Also adds an semanage_root() function to get the
>>>   current root.
>>> - In previous patchsets, the semanage_module_upgrade* and
>>>   semanage_module_install_base* functions were removed from the API,
>>>   and semanage_module_install* had modified parameters. However, these
>>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>>   now added symbolic versioning to support the old version of the
>>>   functions, which now just call the new install functions. semodule
>>>   is updated to support --base and --upgrade, but with the addition of
>>>   a deprecation message. API compatability is not maintained.
>>> - Likewise, symbolic versioning was added to support the old module
>>>   enable/disable functions, which call the new enable/disable
>>>   functions.
>>> - Modify the libsepol Makefile to now make including CIL optional via
>>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>>   libsemanage), primarily so that SE for Android does not need to
>>>   include unused CIL cruft.
>>>
>>> With these changes, ABI compatibility is maintained. Additionally, we
>>> have tested these changes with the userspace tests and against the
>>> kernel test suite, and no new failures were discovered. We have
>>> also tested this patchset with both Fedora 20 policy and with reference
>>> policy and found no errors.
>>>
>>> Because of the size of the patchset (67 file changes, ~8300
>>> insertions, ~1800 deletions), all the changes have been pushed to the
>>> selinux git repository to the 'integration' branch for
>>> comments/review. Unlike the previous RFCs, for simplicity there is now
>>> only a single branch, containing three types of changes:
>>>
>>> Reverts
>>>    Reverts changes made to master that conflict with the new source
>>>    policy infrastructure (e.g. how paths are handled,
>>>    enabled/disable modules). Rather than dealing with a large amount
>>>    of conflicts with the source policy work, it was easier to just
>>>    remove the commits that added conflicting features, rebase the old
>>>    source policy work on top of that, and add back any features in a
>>>    manner consistent with source policy. The only conflicts were
>>>    related to enabling/disabling of modules, and semanage_set_root.
>>>
>>> Source Policy
>>>    This is a rebase of the old src-policy branch on top of the
>>>    reverted commits.  The goal of these changes is to improve the API
>>>    for module handling, add support for source policies, module
>>>    priorities, enabling/disabling of modules, and moving the policy
>>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>>
>>> CIL Integration
>>>    These changes build CIL into libsepol, and updates libsepol,
>>>    libsemanage, semodule, and semanage to work with and understand CIL
>>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>>    CIL has a few side effects, such as removing base modules,
>>>    versions, and upgrades.
>>>
>>>    This also adds a new tool (installed to
>>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>>    converts binary pp modules to CIL. The infrastructure to use this
>>>    compiler (or any other HLL compiler) was added to compile HLL
>>>    modules to CIL, which is accomplished by writing the HLL data to
>>>    the stdin of the compiler and reading the equivilent CIL from
>>>    stdout. The resulting CIL is then cached in the policy store so
>>>    this compilation does not need to take place during future store
>>>    updates. Cached CIL modules can be ignored using a new semodule
>>>    flag (-C/--ignore-cache) or a new configuration option in
>>>    semanage.conf (ignore-cache). Other configuration options were
>>>    added to semanage.conf to manage the path to HLL compilers
>>>    (compiler-directory) and the policy store (store-root). Semodule
>>>    was also modified to support changing the policy store with the
>>>    -S/--store-root option.
>>>
>>>    Lastly, the CIL integration changes required changes to the API,
>>>    but symbolic versioning was used to maintain ABI compatibility.
>>>    Because of this, the .so version is no longer incremented like in
>>>    the previous version of this RFC.
>>>
>>> With these changes, it is possible to build and manage SELinux
>>> policy using pp and CIL modules and the familiar semodule/semanage
>>> tools.
>>>
>>> To make this easier to experiment with and test, below are the steps
>>> needed to install the updated userspace and migrate a minimal Fedora 20
>>> installation to the new policy store.
>>>
>>> Thanks, and we look forward to any questions/comments.
>>>
>>> - Steve
>>>
>>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>>> [2] https://github.com/SELinuxProject/cil/wiki
>>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>>
>>>
>>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>>
>>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>>
>>> # Install SELinux userspace dependencies
>>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>>> setools-devel swig ustr-devel
>>>
>>> # Update to the latest targeted policy
>>> $ yum update selinux-policy-targeted
>>>
>>> # Clone the repos and checkout branches
>>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>>
>>> # Create a symlink to the cil repo so CIL can be built into libsepol
>>> $ ln -s ~/cil/ selinux/libsepol/cil
>>>
>>> # Install SELinux userspace with CIL integration and HLL support
>>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>>
>>> # Migrate to the new source policy infrastructure
>>> $ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py
>>>
>>> # List the installed modules, showing priority and HLL
>>> $ semodule --list=full
>>
>> Is the classorder bug?
>> $ su <hangs forever>
>> $ dmesg
>>  systemd[1]: SELinux policy denies access.
>>
>>
> 
> The classorder bug has been fixed with the latest cil & integration
> branch, and shouldn't have been a problem with fedora's policy since the
> pp2cil tool should have output the classes in the correct order.
> 
> I'm not able to reproduce this problem. su works correctly and
> immediately for me. What commands did you run before this?

I tried various setsebool and semodule commands, and ran the
selinux-testsuite. Not sure exactly when it started doing this but it
seems to be consistent right now.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 10:11 -0400, Stephen Smalley wrote:

> Is the classorder bug?
> $ su <hangs forever>
> $ dmesg
>  systemd[1]: SELinux policy denies access.
> 

Is that with handle-unknown set to deny?

if so then this is due to a missing av permission for the system class
in the fedora policy

Else it may be indeed related to classorder but i think its the former

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 10:23 AM, Dominick Grift wrote:
> On Thu, 2014-07-10 at 10:11 -0400, Stephen Smalley wrote:
> 
>> Is the classorder bug?
>> $ su <hangs forever>
>> $ dmesg
>>  systemd[1]: SELinux policy denies access.
>>
> 
> Is that with handle-unknown set to deny?
> 
> if so then this is due to a missing av permission for the system class
> in the fedora policy
> 
> Else it may be indeed related to classorder but i think its the former

No, this is a stock system, so semanage.conf has the defaults, i.e. no
expand-check and no handle-unknown.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 10:25 AM, Stephen Smalley wrote:
> On 07/10/2014 10:23 AM, Dominick Grift wrote:
>> On Thu, 2014-07-10 at 10:11 -0400, Stephen Smalley wrote:
>>
>>> Is the classorder bug?
>>> $ su <hangs forever>
>>> $ dmesg
>>>  systemd[1]: SELinux policy denies access.
>>>
>>
>> Is that with handle-unknown set to deny?
>>
>> if so then this is due to a missing av permission for the system class
>> in the fedora policy
>>
>> Else it may be indeed related to classorder but i think its the former
> 
> No, this is a stock system, so semanage.conf has the defaults, i.e. no
> expand-check and no handle-unknown.

Hmmm...but rebooting "cleared" it and now I can su without delay and no
systemd error message.

Merged #next to #integration locally to try to pick up the improved
error reporting on unknown class/perms but can't reproduce it now...

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 10:34 -0400, Stephen Smalley wrote:
\
> 
> Hmmm...but rebooting "cleared" it and now I can su without delay and no
> systemd error message.
> 

Yes , sound like the issue i hit yesterday (when i updated my e145
policy)

basically i loaded the updated policy, rebooted (system was hanging),
hard reset, looking at avc denials, system_t self:dbus send_msg was
denied. (even though i had that rule both before and after i updated my
policy. was not able to reproduce it after that, but looks like we will
be seeing more from this issue soon.

> Merged #next to #integration locally to try to pick up the improved
> error reporting on unknown class/perms but can't reproduce it now...
> 
> 
> 

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-07-10 at 10:25 -0400, Stephen Smalley wrote:

> 
> No, this is a stock system, so semanage.conf has the defaults, i.e. no
> expand-check and no handle-unknown.
> 
> 

I see, If this only happened right after policy was (re) loaded and
after that it was gone. Then i might have hit the same issue (not with
this test though but yesterday with my e145 policy built using latest
secilc) when i loaded my policy with the classorder added (i am not
saying that classorder is causing this though)

I started seeing dbus denials for a short period (even though the rules
were in the policy). Soon after they disappeared and everything was
fine.

It probably going to take a little time for me to be able to reproduce
this if i can reproduce it at all.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/09/2014 03:21 PM, Steve Lawrence wrote:
> In January, we sent an RFC [1] to update userspace to integrate CIL
> [2] and source policy. And in April, we sent an updated RFC [3] which
> added support for high level languages and a tool to convert policy
> package (pp) files to CIL. After getting some good feedback, we have
> made some more changes, mostly to maintain ABI compatibility. The
> major changes made since the last patchset are:
> 
> - Change how semanage_set_root was re-added to use the source policy
>   infrastructure. Fixes were made so that semanage.conf was looked for
>   inside the root. Also adds an semanage_root() function to get the
>   current root.
> - In previous patchsets, the semanage_module_upgrade* and
>   semanage_module_install_base* functions were removed from the API,
>   and semanage_module_install* had modified parameters. However, these
>   changes broke the API and ABI. To maintain ABI compatibility, we've
>   now added symbolic versioning to support the old version of the
>   functions, which now just call the new install functions. semodule
>   is updated to support --base and --upgrade, but with the addition of
>   a deprecation message. API compatability is not maintained.
> - Likewise, symbolic versioning was added to support the old module
>   enable/disable functions, which call the new enable/disable
>   functions.
> - Modify the libsepol Makefile to now make including CIL optional via
>   the DISABLE_CIL build flag. This only affects libsepol (not
>   libsemanage), primarily so that SE for Android does not need to
>   include unused CIL cruft.
> 
> With these changes, ABI compatibility is maintained. Additionally, we
> have tested these changes with the userspace tests and against the
> kernel test suite, and no new failures were discovered. We have
> also tested this patchset with both Fedora 20 policy and with reference
> policy and found no errors.
> 
> Because of the size of the patchset (67 file changes, ~8300
> insertions, ~1800 deletions), all the changes have been pushed to the
> selinux git repository to the 'integration' branch for
> comments/review. Unlike the previous RFCs, for simplicity there is now
> only a single branch, containing three types of changes:
> 
> Reverts
>    Reverts changes made to master that conflict with the new source
>    policy infrastructure (e.g. how paths are handled,
>    enabled/disable modules). Rather than dealing with a large amount
>    of conflicts with the source policy work, it was easier to just
>    remove the commits that added conflicting features, rebase the old
>    source policy work on top of that, and add back any features in a
>    manner consistent with source policy. The only conflicts were
>    related to enabling/disabling of modules, and semanage_set_root.
> 
> Source Policy
>    This is a rebase of the old src-policy branch on top of the
>    reverted commits.  The goal of these changes is to improve the API
>    for module handling, add support for source policies, module
>    priorities, enabling/disabling of modules, and moving the policy
>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
> 
> CIL Integration
>    These changes build CIL into libsepol, and updates libsepol,
>    libsemanage, semodule, and semanage to work with and understand CIL
>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>    CIL has a few side effects, such as removing base modules,
>    versions, and upgrades.
> 
>    This also adds a new tool (installed to
>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>    converts binary pp modules to CIL. The infrastructure to use this
>    compiler (or any other HLL compiler) was added to compile HLL
>    modules to CIL, which is accomplished by writing the HLL data to
>    the stdin of the compiler and reading the equivilent CIL from
>    stdout. The resulting CIL is then cached in the policy store so
>    this compilation does not need to take place during future store
>    updates. Cached CIL modules can be ignored using a new semodule
>    flag (-C/--ignore-cache) or a new configuration option in
>    semanage.conf (ignore-cache). Other configuration options were
>    added to semanage.conf to manage the path to HLL compilers
>    (compiler-directory) and the policy store (store-root). Semodule
>    was also modified to support changing the policy store with the
>    -S/--store-root option.
> 
>    Lastly, the CIL integration changes required changes to the API,
>    but symbolic versioning was used to maintain ABI compatibility.
>    Because of this, the .so version is no longer incremented like in
>    the previous version of this RFC.
> 
> With these changes, it is possible to build and manage SELinux
> policy using pp and CIL modules and the familiar semodule/semanage
> tools.
> 
> To make this easier to experiment with and test, below are the steps
> needed to install the updated userspace and migrate a minimal Fedora 20
> installation to the new policy store.
> 
> Thanks, and we look forward to any questions/comments.
> 
> - Steve
> 
> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
> [2] https://github.com/SELinuxProject/cil/wiki
> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
> 
> 
> Steps to Install SELinux Userspace with source policy, CIL, and HLL
> 
> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
> 
> # Install SELinux userspace dependencies
> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
> setools-devel swig ustr-devel
> 
> # Update to the latest targeted policy
> $ yum update selinux-policy-targeted
> 
> # Clone the repos and checkout branches
> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
> $ git clone -b master https://github.com/SELinuxProject/cil.git
> 
> # Create a symlink to the cil repo so CIL can be built into libsepol
> $ ln -s ~/cil/ selinux/libsepol/cil
> 
> # Install SELinux userspace with CIL integration and HLL support
> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap

There is something weird about your build system with cil; you can't
build twice without first doing a complete clean.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 10:30 AM, Stephen Smalley wrote:
> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
>>
>> - Change how semanage_set_root was re-added to use the source policy
>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>   inside the root. Also adds an semanage_root() function to get the
>>   current root.
>> - In previous patchsets, the semanage_module_upgrade* and
>>   semanage_module_install_base* functions were removed from the API,
>>   and semanage_module_install* had modified parameters. However, these
>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>   now added symbolic versioning to support the old version of the
>>   functions, which now just call the new install functions. semodule
>>   is updated to support --base and --upgrade, but with the addition of
>>   a deprecation message. API compatability is not maintained.
>> - Likewise, symbolic versioning was added to support the old module
>>   enable/disable functions, which call the new enable/disable
>>   functions.
>> - Modify the libsepol Makefile to now make including CIL optional via
>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>   libsemanage), primarily so that SE for Android does not need to
>>   include unused CIL cruft.
>>
>> With these changes, ABI compatibility is maintained. Additionally, we
>> have tested these changes with the userspace tests and against the
>> kernel test suite, and no new failures were discovered. We have
>> also tested this patchset with both Fedora 20 policy and with reference
>> policy and found no errors.
>>
>> Because of the size of the patchset (67 file changes, ~8300
>> insertions, ~1800 deletions), all the changes have been pushed to the
>> selinux git repository to the 'integration' branch for
>> comments/review. Unlike the previous RFCs, for simplicity there is now
>> only a single branch, containing three types of changes:
>>
>> Reverts
>>    Reverts changes made to master that conflict with the new source
>>    policy infrastructure (e.g. how paths are handled,
>>    enabled/disable modules). Rather than dealing with a large amount
>>    of conflicts with the source policy work, it was easier to just
>>    remove the commits that added conflicting features, rebase the old
>>    source policy work on top of that, and add back any features in a
>>    manner consistent with source policy. The only conflicts were
>>    related to enabling/disabling of modules, and semanage_set_root.
>>
>> Source Policy
>>    This is a rebase of the old src-policy branch on top of the
>>    reverted commits.  The goal of these changes is to improve the API
>>    for module handling, add support for source policies, module
>>    priorities, enabling/disabling of modules, and moving the policy
>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>
>> CIL Integration
>>    These changes build CIL into libsepol, and updates libsepol,
>>    libsemanage, semodule, and semanage to work with and understand CIL
>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>    CIL has a few side effects, such as removing base modules,
>>    versions, and upgrades.
>>
>>    This also adds a new tool (installed to
>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>    converts binary pp modules to CIL. The infrastructure to use this
>>    compiler (or any other HLL compiler) was added to compile HLL
>>    modules to CIL, which is accomplished by writing the HLL data to
>>    the stdin of the compiler and reading the equivilent CIL from
>>    stdout. The resulting CIL is then cached in the policy store so
>>    this compilation does not need to take place during future store
>>    updates. Cached CIL modules can be ignored using a new semodule
>>    flag (-C/--ignore-cache) or a new configuration option in
>>    semanage.conf (ignore-cache). Other configuration options were
>>    added to semanage.conf to manage the path to HLL compilers
>>    (compiler-directory) and the policy store (store-root). Semodule
>>    was also modified to support changing the policy store with the
>>    -S/--store-root option.
>>
>>    Lastly, the CIL integration changes required changes to the API,
>>    but symbolic versioning was used to maintain ABI compatibility.
>>    Because of this, the .so version is no longer incremented like in
>>    the previous version of this RFC.
>>
>> With these changes, it is possible to build and manage SELinux
>> policy using pp and CIL modules and the familiar semodule/semanage
>> tools.
>>
>> To make this easier to experiment with and test, below are the steps
>> needed to install the updated userspace and migrate a minimal Fedora 20
>> installation to the new policy store.
>>
>> Thanks, and we look forward to any questions/comments.
>>
>> - Steve
>>
>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>> [2] https://github.com/SELinuxProject/cil/wiki
>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>
>>
>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>
>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>
>> # Install SELinux userspace dependencies
>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>> setools-devel swig ustr-devel
>>
>> # Update to the latest targeted policy
>> $ yum update selinux-policy-targeted
>>
>> # Clone the repos and checkout branches
>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>
>> # Create a symlink to the cil repo so CIL can be built into libsepol
>> $ ln -s ~/cil/ selinux/libsepol/cil
>>
>> # Install SELinux userspace with CIL integration and HLL support
>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
> 
> There is something weird about your build system with cil; you can't
> build twice without first doing a complete clean.

So, to be precise, it is some sequence where I build, do a git clean
-fdx in selinux, try to rebuild, fails due to missing cil symlink (why
doesn't it test for its existence explicitly?), re-create the symlink,
and try to build again.  Ends up with duplicated symbols in cil.  Have
to clean the cil directory to get it to build again.

Also, what's the plan for actual merging of cil into libsepol?  If
libsepol depends on cil for building, then you can't really keep it
separate and require cloning two projects and setting up a symlink by
hand like this in the long term.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 10:50 AM, Stephen Smalley wrote:
> On 07/10/2014 10:30 AM, Stephen Smalley wrote:
>> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>> added support for high level languages and a tool to convert policy
>>> package (pp) files to CIL. After getting some good feedback, we have
>>> made some more changes, mostly to maintain ABI compatibility. The
>>> major changes made since the last patchset are:
>>>
>>> - Change how semanage_set_root was re-added to use the source policy
>>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>>   inside the root. Also adds an semanage_root() function to get the
>>>   current root.
>>> - In previous patchsets, the semanage_module_upgrade* and
>>>   semanage_module_install_base* functions were removed from the API,
>>>   and semanage_module_install* had modified parameters. However, these
>>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>>   now added symbolic versioning to support the old version of the
>>>   functions, which now just call the new install functions. semodule
>>>   is updated to support --base and --upgrade, but with the addition of
>>>   a deprecation message. API compatability is not maintained.
>>> - Likewise, symbolic versioning was added to support the old module
>>>   enable/disable functions, which call the new enable/disable
>>>   functions.
>>> - Modify the libsepol Makefile to now make including CIL optional via
>>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>>   libsemanage), primarily so that SE for Android does not need to
>>>   include unused CIL cruft.
>>>
>>> With these changes, ABI compatibility is maintained. Additionally, we
>>> have tested these changes with the userspace tests and against the
>>> kernel test suite, and no new failures were discovered. We have
>>> also tested this patchset with both Fedora 20 policy and with reference
>>> policy and found no errors.
>>>
>>> Because of the size of the patchset (67 file changes, ~8300
>>> insertions, ~1800 deletions), all the changes have been pushed to the
>>> selinux git repository to the 'integration' branch for
>>> comments/review. Unlike the previous RFCs, for simplicity there is now
>>> only a single branch, containing three types of changes:
>>>
>>> Reverts
>>>    Reverts changes made to master that conflict with the new source
>>>    policy infrastructure (e.g. how paths are handled,
>>>    enabled/disable modules). Rather than dealing with a large amount
>>>    of conflicts with the source policy work, it was easier to just
>>>    remove the commits that added conflicting features, rebase the old
>>>    source policy work on top of that, and add back any features in a
>>>    manner consistent with source policy. The only conflicts were
>>>    related to enabling/disabling of modules, and semanage_set_root.
>>>
>>> Source Policy
>>>    This is a rebase of the old src-policy branch on top of the
>>>    reverted commits.  The goal of these changes is to improve the API
>>>    for module handling, add support for source policies, module
>>>    priorities, enabling/disabling of modules, and moving the policy
>>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>>
>>> CIL Integration
>>>    These changes build CIL into libsepol, and updates libsepol,
>>>    libsemanage, semodule, and semanage to work with and understand CIL
>>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>>    CIL has a few side effects, such as removing base modules,
>>>    versions, and upgrades.
>>>
>>>    This also adds a new tool (installed to
>>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>>    converts binary pp modules to CIL. The infrastructure to use this
>>>    compiler (or any other HLL compiler) was added to compile HLL
>>>    modules to CIL, which is accomplished by writing the HLL data to
>>>    the stdin of the compiler and reading the equivilent CIL from
>>>    stdout. The resulting CIL is then cached in the policy store so
>>>    this compilation does not need to take place during future store
>>>    updates. Cached CIL modules can be ignored using a new semodule
>>>    flag (-C/--ignore-cache) or a new configuration option in
>>>    semanage.conf (ignore-cache). Other configuration options were
>>>    added to semanage.conf to manage the path to HLL compilers
>>>    (compiler-directory) and the policy store (store-root). Semodule
>>>    was also modified to support changing the policy store with the
>>>    -S/--store-root option.
>>>
>>>    Lastly, the CIL integration changes required changes to the API,
>>>    but symbolic versioning was used to maintain ABI compatibility.
>>>    Because of this, the .so version is no longer incremented like in
>>>    the previous version of this RFC.
>>>
>>> With these changes, it is possible to build and manage SELinux
>>> policy using pp and CIL modules and the familiar semodule/semanage
>>> tools.
>>>
>>> To make this easier to experiment with and test, below are the steps
>>> needed to install the updated userspace and migrate a minimal Fedora 20
>>> installation to the new policy store.
>>>
>>> Thanks, and we look forward to any questions/comments.
>>>
>>> - Steve
>>>
>>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>>> [2] https://github.com/SELinuxProject/cil/wiki
>>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>>
>>>
>>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>>
>>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>>
>>> # Install SELinux userspace dependencies
>>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>>> setools-devel swig ustr-devel
>>>
>>> # Update to the latest targeted policy
>>> $ yum update selinux-policy-targeted
>>>
>>> # Clone the repos and checkout branches
>>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>>
>>> # Create a symlink to the cil repo so CIL can be built into libsepol
>>> $ ln -s ~/cil/ selinux/libsepol/cil
>>>
>>> # Install SELinux userspace with CIL integration and HLL support
>>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>
>> There is something weird about your build system with cil; you can't
>> build twice without first doing a complete clean.
> 
> So, to be precise, it is some sequence where I build, do a git clean
> -fdx in selinux, try to rebuild, fails due to missing cil symlink (why
> doesn't it test for its existence explicitly?), re-create the symlink,
> and try to build again.  Ends up with duplicated symbols in cil.  Have
> to clean the cil directory to get it to build again.

Thanks for the detailed steps to reproduce. Looking into this now.

> Also, what's the plan for actual merging of cil into libsepol?  If
> libsepol depends on cil for building, then you can't really keep it
> separate and require cloning two projects and setting up a symlink by
> hand like this in the long term.

I'm all open for other ideas, but I had always assumed that the CIL repo
would would be merged (via git subtree merge) into the SELinux userspace
repo right where the symlink goes, and it just becomes part of libsepol.

Licensing might be a problem, since CIL is currently BSD 2-Clause and
libsepol is LGPLv2. Not sure exactly how that works, or if relicensing
is in order. Relicesning CIL shouldn't be a problem if necessary.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Stephen Smalley]

On 07/10/2014 11:05 AM, Steve Lawrence wrote:
> On 07/10/2014 10:50 AM, Stephen Smalley wrote:
>> On 07/10/2014 10:30 AM, Stephen Smalley wrote:
>>> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>> added support for high level languages and a tool to convert policy
>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>> major changes made since the last patchset are:
>>>>
>>>> - Change how semanage_set_root was re-added to use the source policy
>>>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>>>   inside the root. Also adds an semanage_root() function to get the
>>>>   current root.
>>>> - In previous patchsets, the semanage_module_upgrade* and
>>>>   semanage_module_install_base* functions were removed from the API,
>>>>   and semanage_module_install* had modified parameters. However, these
>>>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>>>   now added symbolic versioning to support the old version of the
>>>>   functions, which now just call the new install functions. semodule
>>>>   is updated to support --base and --upgrade, but with the addition of
>>>>   a deprecation message. API compatability is not maintained.
>>>> - Likewise, symbolic versioning was added to support the old module
>>>>   enable/disable functions, which call the new enable/disable
>>>>   functions.
>>>> - Modify the libsepol Makefile to now make including CIL optional via
>>>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>>>   libsemanage), primarily so that SE for Android does not need to
>>>>   include unused CIL cruft.
>>>>
>>>> With these changes, ABI compatibility is maintained. Additionally, we
>>>> have tested these changes with the userspace tests and against the
>>>> kernel test suite, and no new failures were discovered. We have
>>>> also tested this patchset with both Fedora 20 policy and with reference
>>>> policy and found no errors.
>>>>
>>>> Because of the size of the patchset (67 file changes, ~8300
>>>> insertions, ~1800 deletions), all the changes have been pushed to the
>>>> selinux git repository to the 'integration' branch for
>>>> comments/review. Unlike the previous RFCs, for simplicity there is now
>>>> only a single branch, containing three types of changes:
>>>>
>>>> Reverts
>>>>    Reverts changes made to master that conflict with the new source
>>>>    policy infrastructure (e.g. how paths are handled,
>>>>    enabled/disable modules). Rather than dealing with a large amount
>>>>    of conflicts with the source policy work, it was easier to just
>>>>    remove the commits that added conflicting features, rebase the old
>>>>    source policy work on top of that, and add back any features in a
>>>>    manner consistent with source policy. The only conflicts were
>>>>    related to enabling/disabling of modules, and semanage_set_root.
>>>>
>>>> Source Policy
>>>>    This is a rebase of the old src-policy branch on top of the
>>>>    reverted commits.  The goal of these changes is to improve the API
>>>>    for module handling, add support for source policies, module
>>>>    priorities, enabling/disabling of modules, and moving the policy
>>>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>>>
>>>> CIL Integration
>>>>    These changes build CIL into libsepol, and updates libsepol,
>>>>    libsemanage, semodule, and semanage to work with and understand CIL
>>>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>>>    CIL has a few side effects, such as removing base modules,
>>>>    versions, and upgrades.
>>>>
>>>>    This also adds a new tool (installed to
>>>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>>>    converts binary pp modules to CIL. The infrastructure to use this
>>>>    compiler (or any other HLL compiler) was added to compile HLL
>>>>    modules to CIL, which is accomplished by writing the HLL data to
>>>>    the stdin of the compiler and reading the equivilent CIL from
>>>>    stdout. The resulting CIL is then cached in the policy store so
>>>>    this compilation does not need to take place during future store
>>>>    updates. Cached CIL modules can be ignored using a new semodule
>>>>    flag (-C/--ignore-cache) or a new configuration option in
>>>>    semanage.conf (ignore-cache). Other configuration options were
>>>>    added to semanage.conf to manage the path to HLL compilers
>>>>    (compiler-directory) and the policy store (store-root). Semodule
>>>>    was also modified to support changing the policy store with the
>>>>    -S/--store-root option.
>>>>
>>>>    Lastly, the CIL integration changes required changes to the API,
>>>>    but symbolic versioning was used to maintain ABI compatibility.
>>>>    Because of this, the .so version is no longer incremented like in
>>>>    the previous version of this RFC.
>>>>
>>>> With these changes, it is possible to build and manage SELinux
>>>> policy using pp and CIL modules and the familiar semodule/semanage
>>>> tools.
>>>>
>>>> To make this easier to experiment with and test, below are the steps
>>>> needed to install the updated userspace and migrate a minimal Fedora 20
>>>> installation to the new policy store.
>>>>
>>>> Thanks, and we look forward to any questions/comments.
>>>>
>>>> - Steve
>>>>
>>>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>>>> [2] https://github.com/SELinuxProject/cil/wiki
>>>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>>>
>>>>
>>>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>>>
>>>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>>>
>>>> # Install SELinux userspace dependencies
>>>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>>>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>>>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>>>> setools-devel swig ustr-devel
>>>>
>>>> # Update to the latest targeted policy
>>>> $ yum update selinux-policy-targeted
>>>>
>>>> # Clone the repos and checkout branches
>>>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>>>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>>>
>>>> # Create a symlink to the cil repo so CIL can be built into libsepol
>>>> $ ln -s ~/cil/ selinux/libsepol/cil
>>>>
>>>> # Install SELinux userspace with CIL integration and HLL support
>>>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
>>>
>>> There is something weird about your build system with cil; you can't
>>> build twice without first doing a complete clean.
>>
>> So, to be precise, it is some sequence where I build, do a git clean
>> -fdx in selinux, try to rebuild, fails due to missing cil symlink (why
>> doesn't it test for its existence explicitly?), re-create the symlink,
>> and try to build again.  Ends up with duplicated symbols in cil.  Have
>> to clean the cil directory to get it to build again.
> 
> Thanks for the detailed steps to reproduce. Looking into this now.
> 
>> Also, what's the plan for actual merging of cil into libsepol?  If
>> libsepol depends on cil for building, then you can't really keep it
>> separate and require cloning two projects and setting up a symlink by
>> hand like this in the long term.
> 
> I'm all open for other ideas, but I had always assumed that the CIL repo
> would would be merged (via git subtree merge) into the SELinux userspace
> repo right where the symlink goes, and it just becomes part of libsepol.
> 
> Licensing might be a problem, since CIL is currently BSD 2-Clause and
> libsepol is LGPLv2. Not sure exactly how that works, or if relicensing
> is in order. Relicesning CIL shouldn't be a problem if necessary.

I don't think re-licensing is required there (i.e. BSD 2-clause is
compatible with GPL/LGPL, and inclusion of BSD code within a LGPL
library shouldn't be a problem).

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 07/10/2014 10:30 AM, Stephen Smalley wrote:
> On 07/09/2014 03:21 PM, Steve Lawrence wrote:
>> In January, we sent an RFC [1] to update userspace to integrate CIL
>> [2] and source policy. And in April, we sent an updated RFC [3] which
>> added support for high level languages and a tool to convert policy
>> package (pp) files to CIL. After getting some good feedback, we have
>> made some more changes, mostly to maintain ABI compatibility. The
>> major changes made since the last patchset are:
>>
>> - Change how semanage_set_root was re-added to use the source policy
>>   infrastructure. Fixes were made so that semanage.conf was looked for
>>   inside the root. Also adds an semanage_root() function to get the
>>   current root.
>> - In previous patchsets, the semanage_module_upgrade* and
>>   semanage_module_install_base* functions were removed from the API,
>>   and semanage_module_install* had modified parameters. However, these
>>   changes broke the API and ABI. To maintain ABI compatibility, we've
>>   now added symbolic versioning to support the old version of the
>>   functions, which now just call the new install functions. semodule
>>   is updated to support --base and --upgrade, but with the addition of
>>   a deprecation message. API compatability is not maintained.
>> - Likewise, symbolic versioning was added to support the old module
>>   enable/disable functions, which call the new enable/disable
>>   functions.
>> - Modify the libsepol Makefile to now make including CIL optional via
>>   the DISABLE_CIL build flag. This only affects libsepol (not
>>   libsemanage), primarily so that SE for Android does not need to
>>   include unused CIL cruft.
>>
>> With these changes, ABI compatibility is maintained. Additionally, we
>> have tested these changes with the userspace tests and against the
>> kernel test suite, and no new failures were discovered. We have
>> also tested this patchset with both Fedora 20 policy and with reference
>> policy and found no errors.
>>
>> Because of the size of the patchset (67 file changes, ~8300
>> insertions, ~1800 deletions), all the changes have been pushed to the
>> selinux git repository to the 'integration' branch for
>> comments/review. Unlike the previous RFCs, for simplicity there is now
>> only a single branch, containing three types of changes:
>>
>> Reverts
>>    Reverts changes made to master that conflict with the new source
>>    policy infrastructure (e.g. how paths are handled,
>>    enabled/disable modules). Rather than dealing with a large amount
>>    of conflicts with the source policy work, it was easier to just
>>    remove the commits that added conflicting features, rebase the old
>>    source policy work on top of that, and add back any features in a
>>    manner consistent with source policy. The only conflicts were
>>    related to enabling/disabling of modules, and semanage_set_root.
>>
>> Source Policy
>>    This is a rebase of the old src-policy branch on top of the
>>    reverted commits.  The goal of these changes is to improve the API
>>    for module handling, add support for source policies, module
>>    priorities, enabling/disabling of modules, and moving the policy
>>    store from /etc/selinux/<store>/ to /var/lib/selinux/<store>/.
>>
>> CIL Integration
>>    These changes build CIL into libsepol, and updates libsepol,
>>    libsemanage, semodule, and semanage to work with and understand CIL
>>    files and manage /var/lib/selinux and /etc/selinux. Switching to
>>    CIL has a few side effects, such as removing base modules,
>>    versions, and upgrades.
>>
>>    This also adds a new tool (installed to
>>    /usr/libexec/selinux/hll/pp), which is an HLL compiler that
>>    converts binary pp modules to CIL. The infrastructure to use this
>>    compiler (or any other HLL compiler) was added to compile HLL
>>    modules to CIL, which is accomplished by writing the HLL data to
>>    the stdin of the compiler and reading the equivilent CIL from
>>    stdout. The resulting CIL is then cached in the policy store so
>>    this compilation does not need to take place during future store
>>    updates. Cached CIL modules can be ignored using a new semodule
>>    flag (-C/--ignore-cache) or a new configuration option in
>>    semanage.conf (ignore-cache). Other configuration options were
>>    added to semanage.conf to manage the path to HLL compilers
>>    (compiler-directory) and the policy store (store-root). Semodule
>>    was also modified to support changing the policy store with the
>>    -S/--store-root option.
>>
>>    Lastly, the CIL integration changes required changes to the API,
>>    but symbolic versioning was used to maintain ABI compatibility.
>>    Because of this, the .so version is no longer incremented like in
>>    the previous version of this RFC.
>>
>> With these changes, it is possible to build and manage SELinux
>> policy using pp and CIL modules and the familiar semodule/semanage
>> tools.
>>
>> To make this easier to experiment with and test, below are the steps
>> needed to install the updated userspace and migrate a minimal Fedora 20
>> installation to the new policy store.
>>
>> Thanks, and we look forward to any questions/comments.
>>
>> - Steve
>>
>> [1] http://marc.info/?l=selinux&m=138921403805934&w=2
>> [2] https://github.com/SELinuxProject/cil/wiki
>> [3] http://marc.info/?l=selinux&m=139878606630921&w=2
>>
>>
>> Steps to Install SELinux Userspace with source policy, CIL, and HLL
>>
>> # Start with a fresh Fedora 20-x86_64 Mimimal Installation
>>
>> # Install SELinux userspace dependencies
>> $ yum install audit-libs-devel bison bzip2-devel dbus-devel
>> dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
>> libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
>> setools-devel swig ustr-devel
>>
>> # Update to the latest targeted policy
>> $ yum update selinux-policy-targeted
>>
>> # Clone the repos and checkout branches
>> $ git clone -b integration https://github.com/SELinuxProject/selinux.git
>> $ git clone -b master https://github.com/SELinuxProject/cil.git
>>
>> # Create a symlink to the cil repo so CIL can be built into libsepol
>> $ ln -s ~/cil/ selinux/libsepol/cil
>>
>> # Install SELinux userspace with CIL integration and HLL support
>> $ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap
> 
> There is something weird about your build system with cil; you can't
> build twice without first doing a complete clean.
> 
> 

I've just squashed/pushed a fix for this in the integration branch.

The problem was that we defined OBJS as CIL_DIR/*.c + CIL_GENERATED,
where CIL_GENERATED is a .c file created by flex. So if the
CIL_GENERATED .c file already existed (like would happen second time
building), it would be matched by the wildcard, so it would appear twice
in OBJS. This resulted in it trying to link in the same object twice,
causing duplicate definitions. Adding a $(sort ...) to the Makefile
seems to be the standard way to remove these duplicates, and fixes the
problem.

Thanks,
- Steve

[RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

[-- Attachment #1: Type: text/plain, Size: 5511 bytes --]

In January, we sent an RFC [1] for updating userspace to integrate CIL
[2] and source policy. We received a lot of great feedback (thanks to
Richard Haines, Dominick Grift, Stephen Smalley, and Dan Walsh) and are
back with some changes, and again looking for more comments.

This keeps most of the changes as the previous RFC, including moving the
policy store to /var/lib/selinux and replacing binary modules with
support for CIL. Based on feedback from the previous email we have made
changes and added new features to this patchset, specifically:

- Support for preserve tunables
- Infrastructure to support High Level Languages (HLLs)
- A new HLL compiler which converts .pp files to CIL

With the new HLL infrastructure and the pp to CIL compiler, we have
successfully migrated a Fedora 20 system to the new /var/lib/selinux
layout, treating the existing .pp files as high level languages,
compiling them to CIL, and generating a binary policy file that is
semantically equivalent. This infrastructure allows for one to
transition to a full CIL policy, but still allows for backwards
compatiability with systems that rely either fully or partially on pp
modules.

Because of the size of the patchset (66 file changes, ~8000 insertions,
~2000 deletions), all the changes have been pushed to the selinux git
repository in three branches for comments/review. Each branch builds on
the other, with their purposes described below:

src-revert:
   Reverts changes made to master that conflict with the src-policy
   branch (e.g. how paths are handled, enabled/disable modules). Rather
   than dealing with a large amount of conflicts with the source policy
   work, it was easier to just remove the commits which add conflicting
   features, rebase the old source policy work on top of that, and add
   back any features in a manner consistent with source policy.

src-policy:
   This is a rebase of the old src-policy branch onto the src-revert
   branch, with some minor bug fixes squashed in. The goal of this
   patchset is to improve the API for module handling, add support for
   source policies, module priorities, enabling/disabling of modules,
   and moving the policy store from /etc/selinux/<store>/ to
   /var/lib/selinux/<store>/. Due to the use of -p/--path and
   -P/--preserve-tunables, the --priority short option was (pretty
   arbitrarily) changed to -X.

integration:
   This branch includes most of the more recent work. This builds CIL
   into libsepol, and updates libsepol, libsemanage, semodule, and
   semanage to work with and understand CIL files and manage
   /var/lib/selinux and /etc/selinux. Switching to CIL has a few side
   effects, such as removing base modules, versions, and upgrades. This
   also adds a new tool (installed to /usr/libexec/selinux/hll/pp),
   which is an HLL compiler that converts binary pp modules to CIL. The
   infrasture to use this compiler (or any other HLL compiler) was added
   to compile HLL modules to CIL, which is accomplished by writing the
   HLL data to the stdin of the compiler and reading the equivilent CIL
   from stdout. The resulting CIL is then cached in the policy store so
   this compilation does not need to take place during future store
   updates. Cached CIL modules can be ignored using a new semodule flag
   (-C/--ignore-cache) or a new configuration option in semanage.conf
   (ignore-cache). Other configuration options were added to
   semanage.conf to manage the path to HLL compilers
   (compiler-directory) and the policy store (store-root). Semodule was
   also modified to support changing the policy store with the with
   -S/--store-root option. Lastly, because much of the API is changed,
   the .so version numbers of libsemanage and libsepol were bumped to
   version 2.

With these three branches, it is possible to build and manage SELinux
policy using pp and CIL modules and the familiar semodule/semanage
tools.

To make this easier to experiment with and test, below are the steps
needed to install the updated userspace and migrate a minimal Fedora 20
installation to the new policy store.

Thanks, and we look forward to receive your feedback.

- Steve

[1] http://marc.info/?l=selinux&m=138921403805934&w=2
[2] http://userspace.selinuxproject.org/trac/wiki/CilDesign



Steps to Install SELinux Userspace with source policy, CIL, and HLL

# Start with a fresh Fedora 20-x86_64 Mimimal Installation

# Install SELinux userspace dependencies
$ yum install audit-libs-devel bison bzip2-devel dbus-devel
dbus-glib-devel flex flex-static gcc git glib2-devel libcap-ng-devel
libcgroup-devel libsepol-static pcre-devel python-devel python-IPy
setools-devel swig ustr-devel

# Disable the IRC module (this is due to an invalid filecon that has
since been fixed)
semodule --disable irc

# Clone the repos and checkout branches
$ git clone -b integration http://oss.tresys.com/git/selinux.git
$ git clone https://bitbucket.org/jwcarter/secilc.git

# Create a symlink to secilc so CIL can be built into libsepol
$ ln -s ~/secilc/ selinux/libsepol/cil

# Install SElinux userspace with CIL integration and HLL support
$ make -C selinux LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap

# Migrate to the new source policy infrastructure
$ ./selinux/libsemanage/utils/semanage_migrate_etc_to_var.py --norebuild

# Install gnome module that removes a duplicate filename transition (see
attached)
$ semodule --install gnome.cil

# List the installed modules, showing priority and HLL
$ semodule --list=full

[-- Attachment #2: gnome.cil --]
[-- Type: application/vnd.ms-artgalry, Size: 79226 bytes --]

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Tue, 2014-04-29 at 10:59 -0400, Steve Lawrence wrote:

I have not yet had time to try this out but i think i may have found
another bug in secilc.

dontaudit rules are not included in the policy it seems.

Today i called a terms_dontaudit_use_console()

which basically has a rule like:

(dontaudit ARG1 console_device_t rw_term_perms)

But the rule is not ending up in the resulting policy (in no dontaudit
rules at all)

secilc is looking mighty good overall though.

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 05/01/2014 08:38 AM, Dominick Grift wrote:
> On Tue, 2014-04-29 at 10:59 -0400, Steve Lawrence wrote:
> 
> I have not yet had time to try this out but i think i may have found
> another bug in secilc.
> 
> dontaudit rules are not included in the policy it seems.
> 
> Today i called a terms_dontaudit_use_console()
> 
> which basically has a rule like:
> 
> (dontaudit ARG1 console_device_t rw_term_perms)
> 
> But the rule is not ending up in the resulting policy (in no dontaudit
> rules at all)
> 
> secilc is looking mighty good overall though.
> 

I've tested with the pp to CIL method, Jim's cilpolicy.git, and a very
bare bones cil policy in test/policy.cil and I cannot reproduce the
issue you describe where dontaudit rules don't end up in the policy. The
only thing I can think of is that you're giving the -D flag, which will
disable dontaudits. If that's not the case, would it be possible to
provide us your CIL files?

Thanks,
- Steve

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-05-01 at 08:57 -0400, Steve Lawrence wrote:

> 
> I've tested with the pp to CIL method, Jim's cilpolicy.git, and a very
> bare bones cil policy in test/policy.cil and I cannot reproduce the
> issue you describe where dontaudit rules don't end up in the policy. The
> only thing I can think of is that you're giving the -D flag, which will
> disable dontaudits. If that's not the case, would it be possible to
> provide us your CIL files?

Sure, I can give the whole thing:

Here is the spec i use to build a secilc package:

https://github.com/doverride/secilc-spec

Here is my "work in progress" policy written in CIL:

https://github.com/doverride/monogam

( it has a script in "support/" that i most of the time use to
"build/load" policy )

Here is the spec i use to build a "monogam" policy package:

https://github.com/doverride/monogam-spec

Side note:

I am also using a custom installation of policycoreutils (without
semanage/semodule

The spec for that is here:

https://github.com/doverride/policycoreutils-spec

If you look in systemd policy module ( i believe ) then youll see that i
call the term_dontaudit_use_console() which for some reason does not
make it to the policy

seinfo shows no dontaudit rules and neither does sesearch

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-05-01 at 15:24 +0200, Dominick Grift wrote:

Whoops sorry... Its called by an unused type attribute

Sorry for the noise

> On Thu, 2014-05-01 at 08:57 -0400, Steve Lawrence wrote:
> 
> > 
> > I've tested with the pp to CIL method, Jim's cilpolicy.git, and a very
> > bare bones cil policy in test/policy.cil and I cannot reproduce the
> > issue you describe where dontaudit rules don't end up in the policy. The
> > only thing I can think of is that you're giving the -D flag, which will
> > disable dontaudits. If that's not the case, would it be possible to
> > provide us your CIL files?
> 
> Sure, I can give the whole thing:
> 
> Here is the spec i use to build a secilc package:
> 
> https://github.com/doverride/secilc-spec
> 
> Here is my "work in progress" policy written in CIL:
> 
> https://github.com/doverride/monogam
> 
> ( it has a script in "support/" that i most of the time use to
> "build/load" policy )
> 
> Here is the spec i use to build a "monogam" policy package:
> 
> https://github.com/doverride/monogam-spec
> 
> Side note:
> 
> I am also using a custom installation of policycoreutils (without
> semanage/semodule
> 
> The spec for that is here:
> 
> https://github.com/doverride/policycoreutils-spec
> 
> If you look in systemd policy module ( i believe ) then youll see that i
> call the term_dontaudit_use_console() which for some reason does not
> make it to the policy
> 
> seinfo shows no dontaudit rules and neither does sesearch
> 
> 
> 

Re: [RFC] Source Policy, CIL, and High Level Languages

[Dominick Grift]

On Thu, 2014-05-01 at 15:27 +0200, Dominick Grift wrote:
> On Thu, 2014-05-01 at 15:24 +0200, Dominick Grift wrote:
> 
> Whoops sorry... Its called by an unused type attribute
> 
> Sorry for the noise

hmm.. no i think its really a bug... it should be there

Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence]

On 05/01/2014 09:31 AM, Dominick Grift wrote:
> On Thu, 2014-05-01 at 15:27 +0200, Dominick Grift wrote:
>> On Thu, 2014-05-01 at 15:24 +0200, Dominick Grift wrote:
>>
>> Whoops sorry... Its called by an unused type attribute
>>
>> Sorry for the noise
> 
> hmm.. no i think its really a bug... it should be there
> 

Ugh, we were missing a break statement in the --handle-unknown case of
secilc, so it fell through to the dontaudit case. So using the -U option
caused dontaudits to be disabled. Below is a patch that should fix it.

Thanks,
- Steve

diff --git a/secilc.c b/secilc.c
index 88a9cfa..2b95875 100644
--- a/secilc.c
+++ b/secilc.c
@@ -156,6 +156,7 @@ int main(int argc, char *argv[])
                } else {
                    usage(argv[0]);
                }
+               break;
            case 'D':
                disable_dontaudit = 1;
                break;