Re: [RFC] Source Policy, CIL, and High Level Languages

[Steve Lawrence <slawrence@tresys.com>]

On 08/01/2014 01:46 PM, Stephen Smalley wrote:
> On 08/01/2014 10:51 AM, Steve Lawrence wrote:
>> On 07/21/2014 01:50 PM, Steve Lawrence wrote:
>>> On 07/21/2014 10:51 AM, Stephen Smalley wrote:
>>>> On 07/21/2014 10:34 AM, Steve Lawrence wrote:
>>>>> On 07/18/2014 02:10 PM, Stephen Smalley wrote:
>>>>>> On 07/18/2014 12:00 PM, Steve Lawrence wrote:
>>>>>>> On 07/10/2014 02:51 AM, Dominick Grift wrote:
>>>>>>>> On Wed, 2014-07-09 at 15:21 -0400, Steve Lawrence wrote:
>>>>>>>>> In January, we sent an RFC [1] to update userspace to integrate CIL
>>>>>>>>> [2] and source policy. And in April, we sent an updated RFC [3] which
>>>>>>>>> added support for high level languages and a tool to convert policy
>>>>>>>>> package (pp) files to CIL. After getting some good feedback, we have
>>>>>>>>> made some more changes, mostly to maintain ABI compatibility. The
>>>>>>>>> major changes made since the last patchset are:
>>>>>>>>
>>>>>>>> <snip>
>>>>>>>>
>>>>>>>>
>>>>>>>> After associating user john with staff_u, johns home directory is
>>>>>>>> properly labeled (staff_u associated with /home/john). However, what is
>>>>>>>> strange here is that i cannot see staff_u home dir context specs
>>>>>>>> in /var/lib/selinux/targeted/active/modules/file_contexts.homedirs
>>>>>>>>  
>>>>>>>> Am i looking in the wrong place? How does SELinux know that staff_u
>>>>>>>> needs to be associated with /home/john
>>>>>>>>
>>>>>>>
>>>>>>> In the current upatream, file_contexts.homedirs is autogenerated and
>>>>>>> created in /etc/selinux/targeted/modules/active/ before it is copied to
>>>>>>> /etc/selinux/targeted/contexts/files. This file is not removed from the
>>>>>>> store, so it actually exists in two places.
>>>>>>>
>>>>>>> However, with the new source policy work, file_contexts.homedirs is
>>>>>>> generated in a temporary sandbox (not the policy store). The contents of
>>>>>>> the sandbox are copied to /etc/selinux, and then deleted at the end of
>>>>>>> the transaction. So the new source policy infrastructure no longer
>>>>>>> stores intermediate/final build files in the policy store.
>>>>>>>
>>>>>>> However, the migration script copies all the files from the old store to
>>>>>>> the new store, even including autogenerated files that the new source
>>>>>>> policy infrastructure will never look at or touch. This is just a bug in
>>>>>>> the migration script. We've updated the migration script to only migrate
>>>>>>> the files that actually need to be migrated (mostly *.local files). This
>>>>>>> has been rebased/pushed to github #integration branch.
>>>>>>
>>>>>> If I run semanage_migrate_etc_to_var.py -n on a clean (no
>>>>>> /var/lib/selinux at all) system, the /var/lib/selinux/targeted/active
>>>>>> directory contains a homedir_template and a netfilter_contexts file in
>>>>>> addition to the modules (and commit_num).  The first file is
>>>>>> automatically extracted from all of the file contexts during build and
>>>>>> the second is unused these days.  If I then run semodule -B (or omit the
>>>>>> -n option on migration), I further have file_contexts.template and
>>>>>> users_extra files under active, both of which are also generated.  I can
>>>>>> delete all four files and regenerate all but netfilter_contexts via
>>>>>> semodule -B.
>>>>>>
>>>>>
>>>>> This has been fixed. Just needed to remove a couple more paths from the
>>>>> migration script and add a couple of unlink()'s in direct_commit().
>>>>
>>>> Thanks.  I had a question though about the approach being used in the
>>>> new libsemanage for populating /var/lib/selinux/tmp/targeted initially
>>>> (not to be confused with /var/lib/selinux/targeted/tmp, which frankly I
>>>> do find rather confusing even though I understand it now).  libsemanage
>>>> copies the entire /etc/selinux/targeted directory layout (but not all
>>>> the files) under /var/lib/selinux/tmp/targeted, even directories that it
>>>> will never use (including e.g. the old modules directory), and then
>>>> copies the files it manages from /etc/selinux/targeted to the
>>>> corresponding /var/lib/selinux/tmp/targeted location before generating
>>>> the new policy.  This raises a few questions in my mind:
>>>>
>>>> - Is it a good idea to depend on an already existing and populated
>>>> /etc/selinux/targeted directory tree?
>>>
>>> Wouldn't hurt to just create the directories we need.
>>>
>>>> - Why copy the entire directory structure rather than just creating the
>>>> directories we know we will need?
>>>
>>> Yeah, seems unnecessary.
>>>
>>
>> Sorry for the delay in getting back to these issues. Took a little time
>> to look into. We've fixed the creation of the directory structure. It
>> now creates the required directories rather than copying the entire
>> directory structure.
>>
>>>> - What exactly is the dependency here?  If there is a mismatch between
>>>> the built files in /etc/selinux/targeted and the
>>>> /var/lib/selinux/targeted/active tree, can bad things happen?  Is
>>>> libsemanage depending on those prebuilt files being identical to what
>>>> would be generated if it did a rebuild from
>>>> /var/lib/selinux/targeted/active?  If not, why does it need them at all?
>>>
>>> From what I can tell, I think the files in /etc/selinux/targeted are
>>> copied over for the cases where it doesn't rebuild policies (e.g.
>>> changes to local fcontexts, seusers, ports, etc), but needs the current
>>> policy files for validation. We'll look into how how drastic this change
>>> would be.
>>
>> This is actually not a simple fix. I think we would either need to make
>> changes to how dbase structures work (which isn't trivial considering
>> all the function pointers). We could alternatively add symlinks in
>> /var/lib/selinux/tmp/targeted that point to the correct locations in
>> /etc/selinux/targeted, and if we need to change any files in
>> /var/lib/selinux/tmp/targeted then we delete the symlinks and write the
>> new content. Though, this seems kindof hacky to me.
>>
>> Unless you have any other ideas, or the symlink seems reasonable, maybe
>> we could hold of on this issue for now, and work on the properly
>> solution once everything is merged?
>>
>>> Another thing that I think is incorrect is how file_context.local is
>>> handle. That is stored in /etc/selinux/targeted/contexts/files, but it
>>> seems like that should just be stored in
>>> /var/lib/selinux/targeted/active/ just like other local modification
>>> files, and merged into the file_contexts file. Not sure the reasoning
>>> for keeping it separate.
>>>
>>>> It looks to me as if possibly the old libsemanage relied on having
>>>> policy.kern locally available in the sandbox, then RH switched it to a
>>>> symlink to the installed file (wrongly, I think, as this would seemingly
>>>> break a revert to prior policy), and this made it also depend on the
>>>> installed policy file.  And it appears to assume that the directory
>>>> structure at least already exists under /etc/selinux/targeted.  But this
>>>> seems to take it a step further.  How do I bootstrap a policy install
>>>> with no prior /etc/selinux/targeted directory?
>>>
>>> I think all you really need is the directory structure set up, but it
>>> should be a pretty simple fix so even that isn't required.
>>>
>>
>> This has been fixed. The integration branch now creates all the
>> directories it needs, so nothing should need to be done to bootstrap the
>> system. In addition to these changes, we've also fixed the name of the
>> semodule --ignore-module-cache option (it used to be
>> --ignore_module_cache). We have rebased the #integration branch onto
>> github/next (fixing conflicts with the recent boolean changes).
>>
>> I think that fixes all the known issues, aside from the two that I think
>> would best be dealt with after upstreaming the integration branch (the
>> two issues being 1) copying files from /etc/selinux/targeted to
>> /var/lib/selinux/tmp/targeted and 2) a way to delete hll files after
>> converted to cil to save space if needed).
>>
>> Unless there are any more questions/concerns, perhaps it's time to start
>> talks about merging into upstream and what needs to be done to help
>> distributions integrate the changes?
> 
> So we need to link cil into the tree for building.  I haven't tried it,
> but git subtree (not submodule) looks promising, both for adding cil to
> the tree and possibly to split each of the other subdirectories into its
> own repository if we want to do that.

I have tested merging the cil.git into selinux.git using git subtree,
and it works fine. It does make it a little messy when looking at the
history, since it contains histories of both selinux and cil, but git
log and other tools can filter out unwanted directories pretty easily.

A while ago I did look into splitting the subdirectories into separate
repos with subtree, and it didn't work as nicely as I would have liked.
It resulted in a lot of branches and tags that had nothing to do with
the subdirectory. I'm sure it could be fixed (and I didn't spend too
much time on it), but it might require a little bit of work.

> If we are going to keep maintaining */{ChangeLog,VERSION}, then we
> should update them in all affected subdirectories (including the changes
> on #next as well as on #integration).  If not, then we should get rid of
> them and decide what if any comparable files we want to add at
> top-level, e.g. NEWS.  Need to capture all of the user-visible changes
> and migration instructions somewhere.  I expect the distributions will
> continue to maintain separate packages for libsepol, checkpolicy,
> libselinux, libsemanage and policycoreutils, so we should try to keep
> things as simple as possible for them.

We'll add some patches to update the ChangeLog/VERSIONs for affected
subdirs, and create a wiki page that talks about migration and the new
CIL changes.