All of lore.kernel.org
 help / color / mirror / Atom feed
* New Blog on how SELinux blocked Docker container escape.
@ 2017-01-13 19:48 Daniel J Walsh
  2017-01-13 20:43 ` Dominick Grift
  2017-01-14  8:23 ` Tracy Reed
  0 siblings, 2 replies; 15+ messages in thread
From: Daniel J Walsh @ 2017-01-13 19:48 UTC (permalink / raw)
  To: SELinux, Fedora SELinux Users

http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-13 19:48 New Blog on how SELinux blocked Docker container escape Daniel J Walsh
@ 2017-01-13 20:43 ` Dominick Grift
  2017-01-14  8:23 ` Tracy Reed
  1 sibling, 0 replies; 15+ messages in thread
From: Dominick Grift @ 2017-01-13 20:43 UTC (permalink / raw)
  To: selinux


[-- Attachment #1.1: Type: text/plain, Size: 944 bytes --]

On 01/13/2017 08:48 PM, Daniel J Walsh wrote:
> http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/

good job, but a minor suggestion. you raise the impression that SELinux
did this, and even though SELinux made this possible, your policy is
what actually achieved this by telling SELinux what to do.

I think it is important to make this distinction, not for us, but for
the crowd that is not so familiar with the matter. People often
blame/praise SELinux for the policy it enforces.


> 
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
> 


-- 
Key fingerprint = 5F4D 3CDB D3F8 3652 FBD8  02D5 3B6C 5F1D 2C7B 6B02
https://sks-keyservers.net/pks/lookup?op=get&search=0x3B6C5F1D2C7B6B02
Dominick Grift


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 648 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-13 19:48 New Blog on how SELinux blocked Docker container escape Daniel J Walsh
  2017-01-13 20:43 ` Dominick Grift
@ 2017-01-14  8:23 ` Tracy Reed
  2017-01-14  9:35   ` Marko Rauhamaa
  1 sibling, 1 reply; 15+ messages in thread
From: Tracy Reed @ 2017-01-14  8:23 UTC (permalink / raw)
  To: Daniel J Walsh; +Cc: SELinux, Fedora SELinux Users

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On Fri, Jan 13, 2017 at 11:48:20AM PST, Daniel J Walsh spake thusly:
> http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/

I have long been of the opinion that it is this sort of thing which best
advocates the use of SELinux. We need more examples like this. I posted
this to reddit a couple years ago:

https://www.reddit.com/r/linux/comments/1xdokz/selinux_saved_our_asses_xpost_rselinux/

And just a few months ago we had another SELinux save. I would say that
in the years I've been using SELinux I've probably seen 5 cases where
SELinux successfully contained an attack. I wish I would have written
them all up at the time. Those 5 saves definitely made it worth it.

Check out the comments on that reddit post, Daniel. :)

-- 
Tracy Reed

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-14  8:23 ` Tracy Reed
@ 2017-01-14  9:35   ` Marko Rauhamaa
  2017-01-17 13:48     ` Daniel J Walsh
  0 siblings, 1 reply; 15+ messages in thread
From: Marko Rauhamaa @ 2017-01-14  9:35 UTC (permalink / raw)
  To: Tracy Reed; +Cc: Daniel J Walsh, SELinux, Fedora SELinux Users

Tracy Reed <treed@ultraviolet.org>:

> On Fri, Jan 13, 2017 at 11:48:20AM PST, Daniel J Walsh spake thusly:
>> http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/
>
> I have long been of the opinion that it is this sort of thing which best
> advocates the use of SELinux. We need more examples like this.

The threats are obvious to anyone by now. What SELinux needs is a clear
methodology. For example, this is *not* a methodology:

   <URL: https://wiki.gentoo.org/wiki/SELinux/Tutorials/Creating_your_o
   wn_policy_module_file>


As a software developer, what am I expect to do wrt SELinux? Should I
ship my product with an SELinux policy module? Or should I simply make
it SELinux agnostic and supply information for the sysadmin so they can
add a policy module for my product? If so, what information should I
provide?

As a sysadmin, should I accept RedHat's policy collection or come up
with my own? If I need another boolean not supplied by RedHat, what am I
to do? How do I make sure my policy is sound? How do I find out what
legitimate access I need to permit for a random service apart from
monitoring the audit log?

It's much easier to understand sandboxes, namespaces, containers,
virtual machines and such. What happens in Vegas stays in Vegas.

Take Daniel Walsh's link above. I didn't get any smarter reading it.
Look at <URL: https://bugzilla.redhat.com/show_bug.cgi?id=1409531#c8>:

   The proposed exploit scenario [...] is *not* possible under the
   default SELinux configuration.

Would it be possible under an SELinux configuration defined by me?


Marko

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-14  9:35   ` Marko Rauhamaa
@ 2017-01-17 13:48     ` Daniel J Walsh
  2017-01-18  5:05       ` 面和毅
  0 siblings, 1 reply; 15+ messages in thread
From: Daniel J Walsh @ 2017-01-17 13:48 UTC (permalink / raw)
  To: Marko Rauhamaa, Tracy Reed; +Cc: Fedora SELinux Users, SELinux



On 01/14/2017 04:35 AM, Marko Rauhamaa wrote:
> Tracy Reed <treed@ultraviolet.org>:
>
>> On Fri, Jan 13, 2017 at 11:48:20AM PST, Daniel J Walsh spake thusly:
>>> http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/
>> I have long been of the opinion that it is this sort of thing which best
>> advocates the use of SELinux. We need more examples like this.
> The threats are obvious to anyone by now. What SELinux needs is a clear
> methodology. For example, this is *not* a methodology:
>
>    <URL: https://wiki.gentoo.org/wiki/SELinux/Tutorials/Creating_your_o
>    wn_policy_module_file>
>
>
> As a software developer, what am I expect to do wrt SELinux? Should I
> ship my product with an SELinux policy module? Or should I simply make
> it SELinux agnostic and supply information for the sysadmin so they can
> add a policy module for my product? If so, what information should I
> provide?
Either. You should definitely not advise to turn it off.  You should
either understand how SELinux would interact with
your product and tell admins with Booleans to set, Which labels to set,
andy modification of configuration required.
If you really want to push it you should ship additional policy with
your application to confine it.

In the container world this becomes much easier, since SELinux can treat
all applications as a black box and just make
sure the application stays in the box, which is what the BLOG is about,
how SELinux keeps the container processes contained.
> As a sysadmin, should I accept RedHat's policy collection or come up
> with my own? 
Yes you should most likely accept Red hat's Policy and customize it as
needed.
> If I need another boolean not supplied by RedHat, what am I
> to do? 
You most likely would just add a policy module and not another boolean. 
You can add allow rules
on the fly using semodule. This is what most admins do.  You would also
need to fix the labeling, which
is usually the problem.
> How do I make sure my policy is sound? 
Probably best to have other people more expert review it.  But Policy is
not rocket science.  You are
basically defining labels which a process label can write to, and read
from. 
> How do I find out what
> legitimate access I need to permit for a random service apart from
> monitoring the audit log?
That is the only way.  sepolicy generate does attempt to help you get
started but other the code analysys, we have
found that running an app through a test suite and collecting the logs
and processing them is the best way to figure
out what an application does.
> It's much easier to understand sandboxes, namespaces, containers,
> virtual machines and such. What happens in Vegas stays in Vegas.
I agree.  This is why we should be pushing towards containers.  90 % or
Android Cell Phones now run with SELinux (SEANdroid) in enforcing mode
and no one knows, because the applications are running in a form of
container.
> Take Daniel Walsh's link above. I didn't get any smarter reading it.
> Look at <URL: https://bugzilla.redhat.com/show_bug.cgi?id=1409531#c8>:
>
>    The proposed exploit scenario [...] is *not* possible under the
>    default SELinux configuration.
>
> Would it be possible under an SELinux configuration defined by me?
Does this help?
http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/

Yes it would be possible if you disabled SELinux on the host. Or if you
disabled it inside of the container, or you ran
the container privileged.  Of if you added custom policy to allow a
process running as container_t to write elsewhere on
the system.
>
> Marko
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>
>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-17 13:48     ` Daniel J Walsh
@ 2017-01-18  5:05       ` 面和毅
  2017-01-18 14:12         ` Daniel J Walsh
  2017-01-18 14:14         ` Daniel J Walsh
  0 siblings, 2 replies; 15+ messages in thread
From: 面和毅 @ 2017-01-18  5:05 UTC (permalink / raw)
  To: Daniel J Walsh; +Cc: Marko Rauhamaa, Tracy Reed, SELinux, Fedora SELinux Users

Dear Sir,

I'm member of Japan-SOSS SIG(Secure OSS Special
Interest Group).
We love SELinux(12years user) and we are promoting SELinux in Japan.

>From technical interesting(we are promoting Docker
with SELinux), we did PoC for CVE-2016-9962 on Fedora25.

Then we found current SELinux(maybe policy) does not
mitigate that vulnerability.

We could reproduce that vulnerability with
- add CAP_SYS_PTRACE to container
- modified runc because there’s not so much race window on runc.
then we think it's not so easy in usual situation.
Also we couldn't reproduce it on CentOS7(latest).

We posted that PoC result on our community blog.
https://jsosug.github.io/post/omok-selinux-docker-20170118/

Also we wish to argue how can we protect this kind of
vulnerability by using SELinux.

Kind Regards,

OMO
-- 
Kazuki Omo: ka-omo@sios.com
OSS &Security Evangelist
Secure OSS SIG
https://jsosug.github.io/
CISSP #366942
Tel: +81364015149

2017-01-17 22:48 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>
>
> On 01/14/2017 04:35 AM, Marko Rauhamaa wrote:
>> Tracy Reed <treed@ultraviolet.org>:
>>
>>> On Fri, Jan 13, 2017 at 11:48:20AM PST, Daniel J Walsh spake thusly:
>>>> http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/
>>> I have long been of the opinion that it is this sort of thing which best
>>> advocates the use of SELinux. We need more examples like this.
>> The threats are obvious to anyone by now. What SELinux needs is a clear
>> methodology. For example, this is *not* a methodology:
>>
>>    <URL: https://wiki.gentoo.org/wiki/SELinux/Tutorials/Creating_your_o
>>    wn_policy_module_file>
>>
>>
>> As a software developer, what am I expect to do wrt SELinux? Should I
>> ship my product with an SELinux policy module? Or should I simply make
>> it SELinux agnostic and supply information for the sysadmin so they can
>> add a policy module for my product? If so, what information should I
>> provide?
> Either. You should definitely not advise to turn it off.  You should
> either understand how SELinux would interact with
> your product and tell admins with Booleans to set, Which labels to set,
> andy modification of configuration required.
> If you really want to push it you should ship additional policy with
> your application to confine it.
>
> In the container world this becomes much easier, since SELinux can treat
> all applications as a black box and just make
> sure the application stays in the box, which is what the BLOG is about,
> how SELinux keeps the container processes contained.
>> As a sysadmin, should I accept RedHat's policy collection or come up
>> with my own?
> Yes you should most likely accept Red hat's Policy and customize it as
> needed.
>> If I need another boolean not supplied by RedHat, what am I
>> to do?
> You most likely would just add a policy module and not another boolean.
> You can add allow rules
> on the fly using semodule. This is what most admins do.  You would also
> need to fix the labeling, which
> is usually the problem.
>> How do I make sure my policy is sound?
> Probably best to have other people more expert review it.  But Policy is
> not rocket science.  You are
> basically defining labels which a process label can write to, and read
> from.
>> How do I find out what
>> legitimate access I need to permit for a random service apart from
>> monitoring the audit log?
> That is the only way.  sepolicy generate does attempt to help you get
> started but other the code analysys, we have
> found that running an app through a test suite and collecting the logs
> and processing them is the best way to figure
> out what an application does.
>> It's much easier to understand sandboxes, namespaces, containers,
>> virtual machines and such. What happens in Vegas stays in Vegas.
> I agree.  This is why we should be pushing towards containers.  90 % or
> Android Cell Phones now run with SELinux (SEANdroid) in enforcing mode
> and no one knows, because the applications are running in a form of
> container.
>> Take Daniel Walsh's link above. I didn't get any smarter reading it.
>> Look at <URL: https://bugzilla.redhat.com/show_bug.cgi?id=1409531#c8>:
>>
>>    The proposed exploit scenario [...] is *not* possible under the
>>    default SELinux configuration.
>>
>> Would it be possible under an SELinux configuration defined by me?
> Does this help?
> http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux/
>
> Yes it would be possible if you disabled SELinux on the host. Or if you
> disabled it inside of the container, or you ran
> the container privileged.  Of if you added custom policy to allow a
> process running as container_t to write elsewhere on
> the system.
>>
>> Marko
>> _______________________________________________
>> Selinux mailing list
>> Selinux@tycho.nsa.gov
>> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
>> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.
>>
>>
>
> _______________________________________________
> Selinux mailing list
> Selinux@tycho.nsa.gov
> To unsubscribe, send email to Selinux-leave@tycho.nsa.gov.
> To get help, send an email containing "help" to Selinux-request@tycho.nsa.gov.



-- 
Kazuki Omo: ka-omo@sios.com
OSS &Security Evangelist
OSS Business Planning Dept.
CISSP #366942
Tel: +81364015149

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-18  5:05       ` 面和毅
@ 2017-01-18 14:12         ` Daniel J Walsh
  2017-01-18 14:14         ` Daniel J Walsh
  1 sibling, 0 replies; 15+ messages in thread
From: Daniel J Walsh @ 2017-01-18 14:12 UTC (permalink / raw)
  To: 面和毅; +Cc: Fedora SELinux Users, SELinux, Marko Rauhamaa



On 01/18/2017 12:05 AM, 面和毅 wrote:
> Dear Sir,
>
> I'm member of Japan-SOSS SIG(Secure OSS Special
> Interest Group).
> We love SELinux(12years user) and we are promoting SELinux in Japan.
>
> >From technical interesting(we are promoting Docker
> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>
> Then we found current SELinux(maybe policy) does not
> mitigate that vulnerability.
>
> We could reproduce that vulnerability with
> - add CAP_SYS_PTRACE to container
> - modified runc because there’s not so much race window on runc.
> then we think it's not so easy in usual situation.
> Also we couldn't reproduce it on CentOS7(latest).
>
> We posted that PoC result on our community blog.
> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>
> Also we wish to argue how can we protect this kind of
> vulnerability by using SELinux.
>
> Kind Regards,
>
> OMO
It mitigates the escape by controlling what can be written and most of
what can be read.
Currently policy allows for some information to be read off of the host
system since the idea
was to allow things like

docker run -ti -v /etc/passwd:/etc/passwd fedora sh

Content in /usr and some content in /etc allowed to be read from inside
of the container.  Content in /var, /home, /root or most other places
where people would store data are blocked.

To work.  I am tightening up the policy to remove the ability to read
content in /etc

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-18  5:05       ` 面和毅
  2017-01-18 14:12         ` Daniel J Walsh
@ 2017-01-18 14:14         ` Daniel J Walsh
  2017-01-19  7:56           ` 面和毅
  1 sibling, 1 reply; 15+ messages in thread
From: Daniel J Walsh @ 2017-01-18 14:14 UTC (permalink / raw)
  To: 面和毅; +Cc: Fedora SELinux Users, SELinux, Marko Rauhamaa



On 01/18/2017 12:05 AM, 面和毅 wrote:
> Dear Sir,
>
> I'm member of Japan-SOSS SIG(Secure OSS Special
> Interest Group).
> We love SELinux(12years user) and we are promoting SELinux in Japan.
>
> >From technical interesting(we are promoting Docker
> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>
> Then we found current SELinux(maybe policy) does not
> mitigate that vulnerability.
>
> We could reproduce that vulnerability with
> - add CAP_SYS_PTRACE to container
> - modified runc because there’s not so much race window on runc.
> then we think it's not so easy in usual situation.
> Also we couldn't reproduce it on CentOS7(latest).
>
> We posted that PoC result on our community blog.
> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>
> Also we wish to argue how can we protect this kind of
> vulnerability by using SELinux.
>
> Kind Regards,
>
> OMO
Attempt to cat /etc/shadow in your test to see the blockage.

Here is a blog I wrote on the topic.

http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-18 14:14         ` Daniel J Walsh
@ 2017-01-19  7:56           ` 面和毅
  2017-01-19 14:06             ` Daniel J Walsh
  0 siblings, 1 reply; 15+ messages in thread
From: 面和毅 @ 2017-01-19  7:56 UTC (permalink / raw)
  To: Daniel J Walsh; +Cc: Fedora SELinux Users, SELinux, Marko Rauhamaa

Dear Sir,

Thanks. I was checking can we cat /etc/shadow in my testing environment.
It seems that is protected because that file's permission is set to "000".

Here is my test result;
--------------------------------------------------------------
----------. 1 root root system_u:object_r:shadow_t:s0
1268 Oct 13 07:55 /etc/shadow

SELinux Enforcing  -> Permission Denied
SELinux Permissive -> Permission Denied
SELinux Disabled -> Permission Denied

When I changed that permission to "755";

SELinux Enforcing  -> Could cat /etc/shadow
SELinux Permissive -> Could cat /etc/shadow
SELinux Disabled -> Could cat /etc/shadow

Then in this case that escaped user could
have read access to shadow_t label.
--------------------------------------------------------------

That "runc" process seems to be working as unconfined_t domain;

[root@fedora25 ~]# ps axZ|grep runc
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 1578 pts/0 Sl+
0:00 runc run ctr

So, I'm not sure but I guess we would better to assign
other domain to "runc" program (no unconfined_t).

Let me check if we will run "runc" in other domain.

Kind Regards,

OMO

2017-01-18 23:14 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>
>
> On 01/18/2017 12:05 AM, 面和毅 wrote:
>> Dear Sir,
>>
>> I'm member of Japan-SOSS SIG(Secure OSS Special
>> Interest Group).
>> We love SELinux(12years user) and we are promoting SELinux in Japan.
>>
>> >From technical interesting(we are promoting Docker
>> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>>
>> Then we found current SELinux(maybe policy) does not
>> mitigate that vulnerability.
>>
>> We could reproduce that vulnerability with
>> - add CAP_SYS_PTRACE to container
>> - modified runc because there’s not so much race window on runc.
>> then we think it's not so easy in usual situation.
>> Also we couldn't reproduce it on CentOS7(latest).
>>
>> We posted that PoC result on our community blog.
>> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>>
>> Also we wish to argue how can we protect this kind of
>> vulnerability by using SELinux.
>>
>> Kind Regards,
>>
>> OMO
> Attempt to cat /etc/shadow in your test to see the blockage.
>
> Here is a blog I wrote on the topic.
>
> http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/
>



-- 
Kazuki Omo: ka-omo@sios.com
OSS &Security Evangelist
OSS Business Planning Dept.
CISSP #366942
Tel: +81364015149

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-19  7:56           ` 面和毅
@ 2017-01-19 14:06             ` Daniel J Walsh
  2017-01-20 12:02               ` 面和毅
  0 siblings, 1 reply; 15+ messages in thread
From: Daniel J Walsh @ 2017-01-19 14:06 UTC (permalink / raw)
  To: 面和毅; +Cc: SELinux, Fedora SELinux Users, Marko Rauhamaa

You are not testing with SELinux if it can read /etc/shadow.  The
process should be running as container_t or svirt_lxc_net_t if it is an
older version.


We currently label runc as container_runtime_exec_t.

dnf reinstall container-selinux

ls -lZ /usr/sbin/runc



On 01/19/2017 02:56 AM, 面和毅 wrote:
> Dear Sir,
>
> Thanks. I was checking can we cat /etc/shadow in my testing environment.
> It seems that is protected because that file's permission is set to "000".
>
> Here is my test result;
> --------------------------------------------------------------
> ----------. 1 root root system_u:object_r:shadow_t:s0
> 1268 Oct 13 07:55 /etc/shadow
>
> SELinux Enforcing  -> Permission Denied
> SELinux Permissive -> Permission Denied
> SELinux Disabled -> Permission Denied
>
> When I changed that permission to "755";
>
> SELinux Enforcing  -> Could cat /etc/shadow
> SELinux Permissive -> Could cat /etc/shadow
> SELinux Disabled -> Could cat /etc/shadow
>
> Then in this case that escaped user could
> have read access to shadow_t label.
> --------------------------------------------------------------
>
> That "runc" process seems to be working as unconfined_t domain;
>
> [root@fedora25 ~]# ps axZ|grep runc
> unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 1578 pts/0 Sl+
> 0:00 runc run ctr
>
> So, I'm not sure but I guess we would better to assign
> other domain to "runc" program (no unconfined_t).
>
> Let me check if we will run "runc" in other domain.
>
> Kind Regards,
>
> OMO
>
> 2017-01-18 23:14 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>
>> On 01/18/2017 12:05 AM, 面和毅 wrote:
>>> Dear Sir,
>>>
>>> I'm member of Japan-SOSS SIG(Secure OSS Special
>>> Interest Group).
>>> We love SELinux(12years user) and we are promoting SELinux in Japan.
>>>
>>> >From technical interesting(we are promoting Docker
>>> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>>>
>>> Then we found current SELinux(maybe policy) does not
>>> mitigate that vulnerability.
>>>
>>> We could reproduce that vulnerability with
>>> - add CAP_SYS_PTRACE to container
>>> - modified runc because there’s not so much race window on runc.
>>> then we think it's not so easy in usual situation.
>>> Also we couldn't reproduce it on CentOS7(latest).
>>>
>>> We posted that PoC result on our community blog.
>>> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>>>
>>> Also we wish to argue how can we protect this kind of
>>> vulnerability by using SELinux.
>>>
>>> Kind Regards,
>>>
>>> OMO
>> Attempt to cat /etc/shadow in your test to see the blockage.
>>
>> Here is a blog I wrote on the topic.
>>
>> http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/
>>
>
>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-19 14:06             ` Daniel J Walsh
@ 2017-01-20 12:02               ` 面和毅
  2017-01-23  4:37                 ` 面和毅
  0 siblings, 1 reply; 15+ messages in thread
From: 面和毅 @ 2017-01-20 12:02 UTC (permalink / raw)
  To: Daniel J Walsh; +Cc: SELinux, Fedora SELinux Users, Marko Rauhamaa

Dear Sir,

Finally I found SELinux could mitigate that vulnerability. Good!! :-)

I checked my PoC system status(actually re-installed fedora25 again),
then I found that problem caused from selinux policy.

policy version: selinux-policy-targeted-3.13.1-225.6.fc25.noarch

I ran "runc" from shell, but it seems the policy is focusing to
run "runc" from systemd, etc (I checked from CIL polocy).

For my PoC, we need to run "runc" from shell.
Then I needed to add localpolicy for typetransition from
unconfined_t to container_t.
Finally I found SELinux could prevent to cat /etc/shadow file. :-)

Here is my result;
-----------------------------------------------
-rwxr-xr-x. 1 root root system_u:object_r: container_runtime_exec_t:s0
5016704 Jan 20 19:26 /usr/bin/runc

----------. 1 root root system_u:object_r:shadow_t:s0 1268 Oct 13
07:55 /etc/shadow

unconfined_u:system_r:container_t:s0-s0:c0.c1023 10721 pts/1 Sl+
0:00 runc run ctr

/etc/shadow permission is "000(Default)";
SELinux Enforcing  -> Permission Denied
SELinux Permissive -> Permission Denied
SELinux Disabled -> Permission Denied

/etc/shadow permission is "755(Modified)";
SELinux Enforcing  -> Permission Denied
SELinux Permissive -> Could cat /etc/shadow
SELinux Disabled -> Could cat /etc/shadow

On /var/log/audit/audit.log I found denied log;
type=AVC msg=audit(1484911003.065:1299): avc:  denied  { read } for
pid=10131 comm="cat" name="shadow" dev="dm-0" ino=785423 scontext=
unconfined_u:system_r:container_t:s0-s0:c0.c1023
tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=0
-----------------------------------------------

Below is additional policy(because this is just
for PoC, I only added denied permssion
to container_t domain. Also I ran that container
by "runc" in /tmp directory);

-----------------------------------------------
(typetransition unconfined_usertype container_runtime_exec_t process
container_t)
(roletransition unconfined_r container_runtime_exec_t process system_r)

(allow container_t user_tmp_t (file (open read execute execute_no_trans)))
(allow container_t var_run_t (dir (write add_name create setattr
remove_name rmdir)))
(allow container_t var_run_t (fifo_file (create setattr unlink read open)))
(allow container_t ptmx_t (chr_file (read write open ioctl)))
(allow container_t devpts_t (chr_file (setattr read write open ioctl getattr)))
(allow container_t root_t (dir (mounton)))
(allow container_t user_tmp_t (dir (mounton write add_name create
remove_name rmdir)))
(allow container_t user_tmp_t (lnk_file (read)))
(allow container_t proc_t (filesystem (mount remount)))
(allow container_t tmpfs_t (filesystem (mount remount)))
(allow container_t tmpfs_t (dir (setattr write add_name create mounton)))
(allow container_t devpts_t (filesystem (mount)))
(allow container_t sysfs_t (filesystem (mount)))
(allow container_t cgroup_t (filesystem (remount)))
(allow container_t tmpfs_t (lnk_file (create)))
(allow container_t tmpfs_t (chr_file (create setattr read write open
getattr ioctl append)))
(allow container_t tmpfs_t (file (open create mounton)))
(allow container_t proc_t (dir (mounton)))
(allow container_t proc_t (file (mounton)))
(allow container_t sysctl_irq_t (dir (mounton)))
(allow container_t sysctl_t (dir (mounton)))
(allow container_t sysctl_t (file (mounton)))
(allow container_t proc_kcore_t (file (mounton)))
(allow container_t nsfs_t (file (getattr read open)))
(allow container_t var_run_t (file (create read write open unlink)))
(allow container_t sysfs_t (dir (mounton)))
(allow container_t kernel_t (unix_stream_socket (read write)))
(allow init_t kernel_t (unix_stream_socket (read write)))
(allow container_t init_t (unix_stream_socket (read write)))
-----------------------------------------------

I'll also post this result on our community blog.

Kind Regards,

OMO

2017-01-19 23:06 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
> You are not testing with SELinux if it can read /etc/shadow.  The
> process should be running as container_t or svirt_lxc_net_t if it is an
> older version.
>
>
> We currently label runc as container_runtime_exec_t.
>
> dnf reinstall container-selinux
>
> ls -lZ /usr/sbin/runc
>
>
>
> On 01/19/2017 02:56 AM, 面和毅 wrote:
>> Dear Sir,
>>
>> Thanks. I was checking can we cat /etc/shadow in my testing environment.
>> It seems that is protected because that file's permission is set to "000".
>>
>> Here is my test result;
>> --------------------------------------------------------------
>> ----------. 1 root root system_u:object_r:shadow_t:s0
>> 1268 Oct 13 07:55 /etc/shadow
>>
>> SELinux Enforcing  -> Permission Denied
>> SELinux Permissive -> Permission Denied
>> SELinux Disabled -> Permission Denied
>>
>> When I changed that permission to "755";
>>
>> SELinux Enforcing  -> Could cat /etc/shadow
>> SELinux Permissive -> Could cat /etc/shadow
>> SELinux Disabled -> Could cat /etc/shadow
>>
>> Then in this case that escaped user could
>> have read access to shadow_t label.
>> --------------------------------------------------------------
>>
>> That "runc" process seems to be working as unconfined_t domain;
>>
>> [root@fedora25 ~]# ps axZ|grep runc
>> unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 1578 pts/0 Sl+
>> 0:00 runc run ctr
>>
>> So, I'm not sure but I guess we would better to assign
>> other domain to "runc" program (no unconfined_t).
>>
>> Let me check if we will run "runc" in other domain.
>>
>> Kind Regards,
>>
>> OMO
>>
>> 2017-01-18 23:14 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>>
>>> On 01/18/2017 12:05 AM, 面和毅 wrote:
>>>> Dear Sir,
>>>>
>>>> I'm member of Japan-SOSS SIG(Secure OSS Special
>>>> Interest Group).
>>>> We love SELinux(12years user) and we are promoting SELinux in Japan.
>>>>
>>>> >From technical interesting(we are promoting Docker
>>>> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>>>>
>>>> Then we found current SELinux(maybe policy) does not
>>>> mitigate that vulnerability.
>>>>
>>>> We could reproduce that vulnerability with
>>>> - add CAP_SYS_PTRACE to container
>>>> - modified runc because there’s not so much race window on runc.
>>>> then we think it's not so easy in usual situation.
>>>> Also we couldn't reproduce it on CentOS7(latest).
>>>>
>>>> We posted that PoC result on our community blog.
>>>> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>>>>
>>>> Also we wish to argue how can we protect this kind of
>>>> vulnerability by using SELinux.
>>>>
>>>> Kind Regards,
>>>>
>>>> OMO
>>> Attempt to cat /etc/shadow in your test to see the blockage.
>>>
>>> Here is a blog I wrote on the topic.
>>>
>>> http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/
>>>
>>
>>
>



-- 
Kazuki Omo: ka-omo@sios.com
OSS &Security Evangelist
OSS Business Planning Dept.
CISSP #366942
Tel: +81364015149

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-20 12:02               ` 面和毅
@ 2017-01-23  4:37                 ` 面和毅
  2017-01-25 20:26                   ` Daniel J Walsh
  0 siblings, 1 reply; 15+ messages in thread
From: 面和毅 @ 2017-01-23  4:37 UTC (permalink / raw)
  To: Daniel J Walsh; +Cc: SELinux, Fedora SELinux Users, Marko Rauhamaa

Dear, Sir,

I just post detailed information and result on our community blog.

https://jsosug.github.io/post/omok-selinux-20170123/

Kind Regards,

OMO

2017-01-20 21:02 GMT+09:00 面和毅 <ka-omo@sios.com>:
> Dear Sir,
>
> Finally I found SELinux could mitigate that vulnerability. Good!! :-)
>
> I checked my PoC system status(actually re-installed fedora25 again),
> then I found that problem caused from selinux policy.
>
> policy version: selinux-policy-targeted-3.13.1-225.6.fc25.noarch
>
> I ran "runc" from shell, but it seems the policy is focusing to
> run "runc" from systemd, etc (I checked from CIL polocy).
>
> For my PoC, we need to run "runc" from shell.
> Then I needed to add localpolicy for typetransition from
> unconfined_t to container_t.
> Finally I found SELinux could prevent to cat /etc/shadow file. :-)
>
> Here is my result;
> -----------------------------------------------
> -rwxr-xr-x. 1 root root system_u:object_r: container_runtime_exec_t:s0
> 5016704 Jan 20 19:26 /usr/bin/runc
>
> ----------. 1 root root system_u:object_r:shadow_t:s0 1268 Oct 13
> 07:55 /etc/shadow
>
> unconfined_u:system_r:container_t:s0-s0:c0.c1023 10721 pts/1 Sl+
> 0:00 runc run ctr
>
> /etc/shadow permission is "000(Default)";
> SELinux Enforcing  -> Permission Denied
> SELinux Permissive -> Permission Denied
> SELinux Disabled -> Permission Denied
>
> /etc/shadow permission is "755(Modified)";
> SELinux Enforcing  -> Permission Denied
> SELinux Permissive -> Could cat /etc/shadow
> SELinux Disabled -> Could cat /etc/shadow
>
> On /var/log/audit/audit.log I found denied log;
> type=AVC msg=audit(1484911003.065:1299): avc:  denied  { read } for
> pid=10131 comm="cat" name="shadow" dev="dm-0" ino=785423 scontext=
> unconfined_u:system_r:container_t:s0-s0:c0.c1023
> tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=0
> -----------------------------------------------
>
> Below is additional policy(because this is just
> for PoC, I only added denied permssion
> to container_t domain. Also I ran that container
> by "runc" in /tmp directory);
>
> -----------------------------------------------
> (typetransition unconfined_usertype container_runtime_exec_t process
> container_t)
> (roletransition unconfined_r container_runtime_exec_t process system_r)
>
> (allow container_t user_tmp_t (file (open read execute execute_no_trans)))
> (allow container_t var_run_t (dir (write add_name create setattr
> remove_name rmdir)))
> (allow container_t var_run_t (fifo_file (create setattr unlink read open)))
> (allow container_t ptmx_t (chr_file (read write open ioctl)))
> (allow container_t devpts_t (chr_file (setattr read write open ioctl getattr)))
> (allow container_t root_t (dir (mounton)))
> (allow container_t user_tmp_t (dir (mounton write add_name create
> remove_name rmdir)))
> (allow container_t user_tmp_t (lnk_file (read)))
> (allow container_t proc_t (filesystem (mount remount)))
> (allow container_t tmpfs_t (filesystem (mount remount)))
> (allow container_t tmpfs_t (dir (setattr write add_name create mounton)))
> (allow container_t devpts_t (filesystem (mount)))
> (allow container_t sysfs_t (filesystem (mount)))
> (allow container_t cgroup_t (filesystem (remount)))
> (allow container_t tmpfs_t (lnk_file (create)))
> (allow container_t tmpfs_t (chr_file (create setattr read write open
> getattr ioctl append)))
> (allow container_t tmpfs_t (file (open create mounton)))
> (allow container_t proc_t (dir (mounton)))
> (allow container_t proc_t (file (mounton)))
> (allow container_t sysctl_irq_t (dir (mounton)))
> (allow container_t sysctl_t (dir (mounton)))
> (allow container_t sysctl_t (file (mounton)))
> (allow container_t proc_kcore_t (file (mounton)))
> (allow container_t nsfs_t (file (getattr read open)))
> (allow container_t var_run_t (file (create read write open unlink)))
> (allow container_t sysfs_t (dir (mounton)))
> (allow container_t kernel_t (unix_stream_socket (read write)))
> (allow init_t kernel_t (unix_stream_socket (read write)))
> (allow container_t init_t (unix_stream_socket (read write)))
> -----------------------------------------------
>
> I'll also post this result on our community blog.
>
> Kind Regards,
>
> OMO
>
> 2017-01-19 23:06 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>> You are not testing with SELinux if it can read /etc/shadow.  The
>> process should be running as container_t or svirt_lxc_net_t if it is an
>> older version.
>>
>>
>> We currently label runc as container_runtime_exec_t.
>>
>> dnf reinstall container-selinux
>>
>> ls -lZ /usr/sbin/runc
>>
>>
>>
>> On 01/19/2017 02:56 AM, 面和毅 wrote:
>>> Dear Sir,
>>>
>>> Thanks. I was checking can we cat /etc/shadow in my testing environment.
>>> It seems that is protected because that file's permission is set to "000".
>>>
>>> Here is my test result;
>>> --------------------------------------------------------------
>>> ----------. 1 root root system_u:object_r:shadow_t:s0
>>> 1268 Oct 13 07:55 /etc/shadow
>>>
>>> SELinux Enforcing  -> Permission Denied
>>> SELinux Permissive -> Permission Denied
>>> SELinux Disabled -> Permission Denied
>>>
>>> When I changed that permission to "755";
>>>
>>> SELinux Enforcing  -> Could cat /etc/shadow
>>> SELinux Permissive -> Could cat /etc/shadow
>>> SELinux Disabled -> Could cat /etc/shadow
>>>
>>> Then in this case that escaped user could
>>> have read access to shadow_t label.
>>> --------------------------------------------------------------
>>>
>>> That "runc" process seems to be working as unconfined_t domain;
>>>
>>> [root@fedora25 ~]# ps axZ|grep runc
>>> unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 1578 pts/0 Sl+
>>> 0:00 runc run ctr
>>>
>>> So, I'm not sure but I guess we would better to assign
>>> other domain to "runc" program (no unconfined_t).
>>>
>>> Let me check if we will run "runc" in other domain.
>>>
>>> Kind Regards,
>>>
>>> OMO
>>>
>>> 2017-01-18 23:14 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>>>
>>>> On 01/18/2017 12:05 AM, 面和毅 wrote:
>>>>> Dear Sir,
>>>>>
>>>>> I'm member of Japan-SOSS SIG(Secure OSS Special
>>>>> Interest Group).
>>>>> We love SELinux(12years user) and we are promoting SELinux in Japan.
>>>>>
>>>>> >From technical interesting(we are promoting Docker
>>>>> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>>>>>
>>>>> Then we found current SELinux(maybe policy) does not
>>>>> mitigate that vulnerability.
>>>>>
>>>>> We could reproduce that vulnerability with
>>>>> - add CAP_SYS_PTRACE to container
>>>>> - modified runc because there’s not so much race window on runc.
>>>>> then we think it's not so easy in usual situation.
>>>>> Also we couldn't reproduce it on CentOS7(latest).
>>>>>
>>>>> We posted that PoC result on our community blog.
>>>>> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>>>>>
>>>>> Also we wish to argue how can we protect this kind of
>>>>> vulnerability by using SELinux.
>>>>>
>>>>> Kind Regards,
>>>>>
>>>>> OMO
>>>> Attempt to cat /etc/shadow in your test to see the blockage.
>>>>
>>>> Here is a blog I wrote on the topic.
>>>>
>>>> http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/
>>>>
>>>
>>>
>>
>
>
>
> --
> Kazuki Omo: ka-omo@sios.com
> OSS &Security Evangelist
> OSS Business Planning Dept.
> CISSP #366942
> Tel: +81364015149



-- 
Kazuki Omo: ka-omo@sios.com
OSS &Security Evangelist
OSS Business Planning Dept.
CISSP #366942
Tel: +81364015149

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-23  4:37                 ` 面和毅
@ 2017-01-25 20:26                   ` Daniel J Walsh
  2017-01-25 20:51                     ` 面和毅
  0 siblings, 1 reply; 15+ messages in thread
From: Daniel J Walsh @ 2017-01-25 20:26 UTC (permalink / raw)
  To: 面和毅; +Cc: Fedora SELinux Users, SELinux, Marko Rauhamaa

The link does not work.


On 01/23/2017 05:37 AM, 面和毅 wrote:
> Dear, Sir,
>
> I just post detailed information and result on our community blog.
>
> https://jsosug.github.io/post/omok-selinux-20170123/
>
> Kind Regards,
>
> OMO
>
> 2017-01-20 21:02 GMT+09:00 面和毅 <ka-omo@sios.com>:
>> Dear Sir,
>>
>> Finally I found SELinux could mitigate that vulnerability. Good!! :-)
>>
>> I checked my PoC system status(actually re-installed fedora25 again),
>> then I found that problem caused from selinux policy.
>>
>> policy version: selinux-policy-targeted-3.13.1-225.6.fc25.noarch
>>
>> I ran "runc" from shell, but it seems the policy is focusing to
>> run "runc" from systemd, etc (I checked from CIL polocy).
>>
>> For my PoC, we need to run "runc" from shell.
>> Then I needed to add localpolicy for typetransition from
>> unconfined_t to container_t.
>> Finally I found SELinux could prevent to cat /etc/shadow file. :-)
>>
>> Here is my result;
>> -----------------------------------------------
>> -rwxr-xr-x. 1 root root system_u:object_r: container_runtime_exec_t:s0
>> 5016704 Jan 20 19:26 /usr/bin/runc
>>
>> ----------. 1 root root system_u:object_r:shadow_t:s0 1268 Oct 13
>> 07:55 /etc/shadow
>>
>> unconfined_u:system_r:container_t:s0-s0:c0.c1023 10721 pts/1 Sl+
>> 0:00 runc run ctr
>>
>> /etc/shadow permission is "000(Default)";
>> SELinux Enforcing  -> Permission Denied
>> SELinux Permissive -> Permission Denied
>> SELinux Disabled -> Permission Denied
>>
>> /etc/shadow permission is "755(Modified)";
>> SELinux Enforcing  -> Permission Denied
>> SELinux Permissive -> Could cat /etc/shadow
>> SELinux Disabled -> Could cat /etc/shadow
>>
>> On /var/log/audit/audit.log I found denied log;
>> type=AVC msg=audit(1484911003.065:1299): avc:  denied  { read } for
>> pid=10131 comm="cat" name="shadow" dev="dm-0" ino=785423 scontext=
>> unconfined_u:system_r:container_t:s0-s0:c0.c1023
>> tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=0
>> -----------------------------------------------
>>
>> Below is additional policy(because this is just
>> for PoC, I only added denied permssion
>> to container_t domain. Also I ran that container
>> by "runc" in /tmp directory);
>>
>> -----------------------------------------------
>> (typetransition unconfined_usertype container_runtime_exec_t process
>> container_t)
>> (roletransition unconfined_r container_runtime_exec_t process system_r)
>>
>> (allow container_t user_tmp_t (file (open read execute execute_no_trans)))
>> (allow container_t var_run_t (dir (write add_name create setattr
>> remove_name rmdir)))
>> (allow container_t var_run_t (fifo_file (create setattr unlink read open)))
>> (allow container_t ptmx_t (chr_file (read write open ioctl)))
>> (allow container_t devpts_t (chr_file (setattr read write open ioctl getattr)))
>> (allow container_t root_t (dir (mounton)))
>> (allow container_t user_tmp_t (dir (mounton write add_name create
>> remove_name rmdir)))
>> (allow container_t user_tmp_t (lnk_file (read)))
>> (allow container_t proc_t (filesystem (mount remount)))
>> (allow container_t tmpfs_t (filesystem (mount remount)))
>> (allow container_t tmpfs_t (dir (setattr write add_name create mounton)))
>> (allow container_t devpts_t (filesystem (mount)))
>> (allow container_t sysfs_t (filesystem (mount)))
>> (allow container_t cgroup_t (filesystem (remount)))
>> (allow container_t tmpfs_t (lnk_file (create)))
>> (allow container_t tmpfs_t (chr_file (create setattr read write open
>> getattr ioctl append)))
>> (allow container_t tmpfs_t (file (open create mounton)))
>> (allow container_t proc_t (dir (mounton)))
>> (allow container_t proc_t (file (mounton)))
>> (allow container_t sysctl_irq_t (dir (mounton)))
>> (allow container_t sysctl_t (dir (mounton)))
>> (allow container_t sysctl_t (file (mounton)))
>> (allow container_t proc_kcore_t (file (mounton)))
>> (allow container_t nsfs_t (file (getattr read open)))
>> (allow container_t var_run_t (file (create read write open unlink)))
>> (allow container_t sysfs_t (dir (mounton)))
>> (allow container_t kernel_t (unix_stream_socket (read write)))
>> (allow init_t kernel_t (unix_stream_socket (read write)))
>> (allow container_t init_t (unix_stream_socket (read write)))
>> -----------------------------------------------
>>
>> I'll also post this result on our community blog.
>>
>> Kind Regards,
>>
>> OMO
>>
>> 2017-01-19 23:06 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>> You are not testing with SELinux if it can read /etc/shadow.  The
>>> process should be running as container_t or svirt_lxc_net_t if it is an
>>> older version.
>>>
>>>
>>> We currently label runc as container_runtime_exec_t.
>>>
>>> dnf reinstall container-selinux
>>>
>>> ls -lZ /usr/sbin/runc
>>>
>>>
>>>
>>> On 01/19/2017 02:56 AM, 面和毅 wrote:
>>>> Dear Sir,
>>>>
>>>> Thanks. I was checking can we cat /etc/shadow in my testing environment.
>>>> It seems that is protected because that file's permission is set to "000".
>>>>
>>>> Here is my test result;
>>>> --------------------------------------------------------------
>>>> ----------. 1 root root system_u:object_r:shadow_t:s0
>>>> 1268 Oct 13 07:55 /etc/shadow
>>>>
>>>> SELinux Enforcing  -> Permission Denied
>>>> SELinux Permissive -> Permission Denied
>>>> SELinux Disabled -> Permission Denied
>>>>
>>>> When I changed that permission to "755";
>>>>
>>>> SELinux Enforcing  -> Could cat /etc/shadow
>>>> SELinux Permissive -> Could cat /etc/shadow
>>>> SELinux Disabled -> Could cat /etc/shadow
>>>>
>>>> Then in this case that escaped user could
>>>> have read access to shadow_t label.
>>>> --------------------------------------------------------------
>>>>
>>>> That "runc" process seems to be working as unconfined_t domain;
>>>>
>>>> [root@fedora25 ~]# ps axZ|grep runc
>>>> unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 1578 pts/0 Sl+
>>>> 0:00 runc run ctr
>>>>
>>>> So, I'm not sure but I guess we would better to assign
>>>> other domain to "runc" program (no unconfined_t).
>>>>
>>>> Let me check if we will run "runc" in other domain.
>>>>
>>>> Kind Regards,
>>>>
>>>> OMO
>>>>
>>>> 2017-01-18 23:14 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>>>> On 01/18/2017 12:05 AM, 面和毅 wrote:
>>>>>> Dear Sir,
>>>>>>
>>>>>> I'm member of Japan-SOSS SIG(Secure OSS Special
>>>>>> Interest Group).
>>>>>> We love SELinux(12years user) and we are promoting SELinux in Japan.
>>>>>>
>>>>>> >From technical interesting(we are promoting Docker
>>>>>> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>>>>>>
>>>>>> Then we found current SELinux(maybe policy) does not
>>>>>> mitigate that vulnerability.
>>>>>>
>>>>>> We could reproduce that vulnerability with
>>>>>> - add CAP_SYS_PTRACE to container
>>>>>> - modified runc because there’s not so much race window on runc.
>>>>>> then we think it's not so easy in usual situation.
>>>>>> Also we couldn't reproduce it on CentOS7(latest).
>>>>>>
>>>>>> We posted that PoC result on our community blog.
>>>>>> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>>>>>>
>>>>>> Also we wish to argue how can we protect this kind of
>>>>>> vulnerability by using SELinux.
>>>>>>
>>>>>> Kind Regards,
>>>>>>
>>>>>> OMO
>>>>> Attempt to cat /etc/shadow in your test to see the blockage.
>>>>>
>>>>> Here is a blog I wrote on the topic.
>>>>>
>>>>> http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/
>>>>>
>>>>
>>
>>
>> --
>> Kazuki Omo: ka-omo@sios.com
>> OSS &Security Evangelist
>> OSS Business Planning Dept.
>> CISSP #366942
>> Tel: +81364015149
>
>

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-25 20:26                   ` Daniel J Walsh
@ 2017-01-25 20:51                     ` 面和毅
  2017-01-25 23:19                       ` 面和毅
  0 siblings, 1 reply; 15+ messages in thread
From: 面和毅 @ 2017-01-25 20:51 UTC (permalink / raw)
  To: Daniel J Walsh; +Cc: Fedora SELinux Users, SELinux, Marko Rauhamaa

Dear Sir,

Thanks. We had several site maintenance yesterday, so I guess
something is corrupted.
We'll fix them and let everyone know again.

Kind Regards,

OMO

2017-01-26 5:26 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
> The link does not work.
>
>
> On 01/23/2017 05:37 AM, 面和毅 wrote:
>> Dear, Sir,
>>
>> I just post detailed information and result on our community blog.
>>
>> https://jsosug.github.io/post/omok-selinux-20170123/
>>
>> Kind Regards,
>>
>> OMO
>>
>> 2017-01-20 21:02 GMT+09:00 面和毅 <ka-omo@sios.com>:
>>> Dear Sir,
>>>
>>> Finally I found SELinux could mitigate that vulnerability. Good!! :-)
>>>
>>> I checked my PoC system status(actually re-installed fedora25 again),
>>> then I found that problem caused from selinux policy.
>>>
>>> policy version: selinux-policy-targeted-3.13.1-225.6.fc25.noarch
>>>
>>> I ran "runc" from shell, but it seems the policy is focusing to
>>> run "runc" from systemd, etc (I checked from CIL polocy).
>>>
>>> For my PoC, we need to run "runc" from shell.
>>> Then I needed to add localpolicy for typetransition from
>>> unconfined_t to container_t.
>>> Finally I found SELinux could prevent to cat /etc/shadow file. :-)
>>>
>>> Here is my result;
>>> -----------------------------------------------
>>> -rwxr-xr-x. 1 root root system_u:object_r: container_runtime_exec_t:s0
>>> 5016704 Jan 20 19:26 /usr/bin/runc
>>>
>>> ----------. 1 root root system_u:object_r:shadow_t:s0 1268 Oct 13
>>> 07:55 /etc/shadow
>>>
>>> unconfined_u:system_r:container_t:s0-s0:c0.c1023 10721 pts/1 Sl+
>>> 0:00 runc run ctr
>>>
>>> /etc/shadow permission is "000(Default)";
>>> SELinux Enforcing  -> Permission Denied
>>> SELinux Permissive -> Permission Denied
>>> SELinux Disabled -> Permission Denied
>>>
>>> /etc/shadow permission is "755(Modified)";
>>> SELinux Enforcing  -> Permission Denied
>>> SELinux Permissive -> Could cat /etc/shadow
>>> SELinux Disabled -> Could cat /etc/shadow
>>>
>>> On /var/log/audit/audit.log I found denied log;
>>> type=AVC msg=audit(1484911003.065:1299): avc:  denied  { read } for
>>> pid=10131 comm="cat" name="shadow" dev="dm-0" ino=785423 scontext=
>>> unconfined_u:system_r:container_t:s0-s0:c0.c1023
>>> tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=0
>>> -----------------------------------------------
>>>
>>> Below is additional policy(because this is just
>>> for PoC, I only added denied permssion
>>> to container_t domain. Also I ran that container
>>> by "runc" in /tmp directory);
>>>
>>> -----------------------------------------------
>>> (typetransition unconfined_usertype container_runtime_exec_t process
>>> container_t)
>>> (roletransition unconfined_r container_runtime_exec_t process system_r)
>>>
>>> (allow container_t user_tmp_t (file (open read execute execute_no_trans)))
>>> (allow container_t var_run_t (dir (write add_name create setattr
>>> remove_name rmdir)))
>>> (allow container_t var_run_t (fifo_file (create setattr unlink read open)))
>>> (allow container_t ptmx_t (chr_file (read write open ioctl)))
>>> (allow container_t devpts_t (chr_file (setattr read write open ioctl getattr)))
>>> (allow container_t root_t (dir (mounton)))
>>> (allow container_t user_tmp_t (dir (mounton write add_name create
>>> remove_name rmdir)))
>>> (allow container_t user_tmp_t (lnk_file (read)))
>>> (allow container_t proc_t (filesystem (mount remount)))
>>> (allow container_t tmpfs_t (filesystem (mount remount)))
>>> (allow container_t tmpfs_t (dir (setattr write add_name create mounton)))
>>> (allow container_t devpts_t (filesystem (mount)))
>>> (allow container_t sysfs_t (filesystem (mount)))
>>> (allow container_t cgroup_t (filesystem (remount)))
>>> (allow container_t tmpfs_t (lnk_file (create)))
>>> (allow container_t tmpfs_t (chr_file (create setattr read write open
>>> getattr ioctl append)))
>>> (allow container_t tmpfs_t (file (open create mounton)))
>>> (allow container_t proc_t (dir (mounton)))
>>> (allow container_t proc_t (file (mounton)))
>>> (allow container_t sysctl_irq_t (dir (mounton)))
>>> (allow container_t sysctl_t (dir (mounton)))
>>> (allow container_t sysctl_t (file (mounton)))
>>> (allow container_t proc_kcore_t (file (mounton)))
>>> (allow container_t nsfs_t (file (getattr read open)))
>>> (allow container_t var_run_t (file (create read write open unlink)))
>>> (allow container_t sysfs_t (dir (mounton)))
>>> (allow container_t kernel_t (unix_stream_socket (read write)))
>>> (allow init_t kernel_t (unix_stream_socket (read write)))
>>> (allow container_t init_t (unix_stream_socket (read write)))
>>> -----------------------------------------------
>>>
>>> I'll also post this result on our community blog.
>>>
>>> Kind Regards,
>>>
>>> OMO
>>>
>>> 2017-01-19 23:06 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>>> You are not testing with SELinux if it can read /etc/shadow.  The
>>>> process should be running as container_t or svirt_lxc_net_t if it is an
>>>> older version.
>>>>
>>>>
>>>> We currently label runc as container_runtime_exec_t.
>>>>
>>>> dnf reinstall container-selinux
>>>>
>>>> ls -lZ /usr/sbin/runc
>>>>
>>>>
>>>>
>>>> On 01/19/2017 02:56 AM, 面和毅 wrote:
>>>>> Dear Sir,
>>>>>
>>>>> Thanks. I was checking can we cat /etc/shadow in my testing environment.
>>>>> It seems that is protected because that file's permission is set to "000".
>>>>>
>>>>> Here is my test result;
>>>>> --------------------------------------------------------------
>>>>> ----------. 1 root root system_u:object_r:shadow_t:s0
>>>>> 1268 Oct 13 07:55 /etc/shadow
>>>>>
>>>>> SELinux Enforcing  -> Permission Denied
>>>>> SELinux Permissive -> Permission Denied
>>>>> SELinux Disabled -> Permission Denied
>>>>>
>>>>> When I changed that permission to "755";
>>>>>
>>>>> SELinux Enforcing  -> Could cat /etc/shadow
>>>>> SELinux Permissive -> Could cat /etc/shadow
>>>>> SELinux Disabled -> Could cat /etc/shadow
>>>>>
>>>>> Then in this case that escaped user could
>>>>> have read access to shadow_t label.
>>>>> --------------------------------------------------------------
>>>>>
>>>>> That "runc" process seems to be working as unconfined_t domain;
>>>>>
>>>>> [root@fedora25 ~]# ps axZ|grep runc
>>>>> unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 1578 pts/0 Sl+
>>>>> 0:00 runc run ctr
>>>>>
>>>>> So, I'm not sure but I guess we would better to assign
>>>>> other domain to "runc" program (no unconfined_t).
>>>>>
>>>>> Let me check if we will run "runc" in other domain.
>>>>>
>>>>> Kind Regards,
>>>>>
>>>>> OMO
>>>>>
>>>>> 2017-01-18 23:14 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>>>>> On 01/18/2017 12:05 AM, 面和毅 wrote:
>>>>>>> Dear Sir,
>>>>>>>
>>>>>>> I'm member of Japan-SOSS SIG(Secure OSS Special
>>>>>>> Interest Group).
>>>>>>> We love SELinux(12years user) and we are promoting SELinux in Japan.
>>>>>>>
>>>>>>> >From technical interesting(we are promoting Docker
>>>>>>> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>>>>>>>
>>>>>>> Then we found current SELinux(maybe policy) does not
>>>>>>> mitigate that vulnerability.
>>>>>>>
>>>>>>> We could reproduce that vulnerability with
>>>>>>> - add CAP_SYS_PTRACE to container
>>>>>>> - modified runc because there’s not so much race window on runc.
>>>>>>> then we think it's not so easy in usual situation.
>>>>>>> Also we couldn't reproduce it on CentOS7(latest).
>>>>>>>
>>>>>>> We posted that PoC result on our community blog.
>>>>>>> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>>>>>>>
>>>>>>> Also we wish to argue how can we protect this kind of
>>>>>>> vulnerability by using SELinux.
>>>>>>>
>>>>>>> Kind Regards,
>>>>>>>
>>>>>>> OMO
>>>>>> Attempt to cat /etc/shadow in your test to see the blockage.
>>>>>>
>>>>>> Here is a blog I wrote on the topic.
>>>>>>
>>>>>> http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/
>>>>>>
>>>>>
>>>
>>>
>>> --
>>> Kazuki Omo: ka-omo@sios.com
>>> OSS &Security Evangelist
>>> OSS Business Planning Dept.
>>> CISSP #366942
>>> Tel: +81364015149
>>
>>
>



-- 
Kazuki Omo: ka-omo@sios.com
OSS &Security Evangelist
OSS Business Planning Dept.
CISSP #366942
Tel: +81364015149

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: New Blog on how SELinux blocked Docker container escape.
  2017-01-25 20:51                     ` 面和毅
@ 2017-01-25 23:19                       ` 面和毅
  0 siblings, 0 replies; 15+ messages in thread
From: 面和毅 @ 2017-01-25 23:19 UTC (permalink / raw)
  To: Daniel J Walsh; +Cc: Fedora SELinux Users, SELinux, Marko Rauhamaa

Folks,

Due to site maintenance trouble, we updated that site/blog to

Site:
http://www.secureoss.jp

Blog:
http://www.secureoss.jp/post/omok-selinux-docker-20170123/
http://www.secureoss.jp/post/omok-selinux-docker-20170118/

Kind Regards,

OMO

2017-01-26 5:51 GMT+09:00 面和毅 <ka-omo@sios.com>:
> Dear Sir,
>
> Thanks. We had several site maintenance yesterday, so I guess
> something is corrupted.
> We'll fix them and let everyone know again.
>
> Kind Regards,
>
> OMO
>
> 2017-01-26 5:26 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>> The link does not work.
>>
>>
>> On 01/23/2017 05:37 AM, 面和毅 wrote:
>>> Dear, Sir,
>>>
>>> I just post detailed information and result on our community blog.
>>>
>>> https://jsosug.github.io/post/omok-selinux-20170123/
>>>
>>> Kind Regards,
>>>
>>> OMO
>>>
>>> 2017-01-20 21:02 GMT+09:00 面和毅 <ka-omo@sios.com>:
>>>> Dear Sir,
>>>>
>>>> Finally I found SELinux could mitigate that vulnerability. Good!! :-)
>>>>
>>>> I checked my PoC system status(actually re-installed fedora25 again),
>>>> then I found that problem caused from selinux policy.
>>>>
>>>> policy version: selinux-policy-targeted-3.13.1-225.6.fc25.noarch
>>>>
>>>> I ran "runc" from shell, but it seems the policy is focusing to
>>>> run "runc" from systemd, etc (I checked from CIL polocy).
>>>>
>>>> For my PoC, we need to run "runc" from shell.
>>>> Then I needed to add localpolicy for typetransition from
>>>> unconfined_t to container_t.
>>>> Finally I found SELinux could prevent to cat /etc/shadow file. :-)
>>>>
>>>> Here is my result;
>>>> -----------------------------------------------
>>>> -rwxr-xr-x. 1 root root system_u:object_r: container_runtime_exec_t:s0
>>>> 5016704 Jan 20 19:26 /usr/bin/runc
>>>>
>>>> ----------. 1 root root system_u:object_r:shadow_t:s0 1268 Oct 13
>>>> 07:55 /etc/shadow
>>>>
>>>> unconfined_u:system_r:container_t:s0-s0:c0.c1023 10721 pts/1 Sl+
>>>> 0:00 runc run ctr
>>>>
>>>> /etc/shadow permission is "000(Default)";
>>>> SELinux Enforcing  -> Permission Denied
>>>> SELinux Permissive -> Permission Denied
>>>> SELinux Disabled -> Permission Denied
>>>>
>>>> /etc/shadow permission is "755(Modified)";
>>>> SELinux Enforcing  -> Permission Denied
>>>> SELinux Permissive -> Could cat /etc/shadow
>>>> SELinux Disabled -> Could cat /etc/shadow
>>>>
>>>> On /var/log/audit/audit.log I found denied log;
>>>> type=AVC msg=audit(1484911003.065:1299): avc:  denied  { read } for
>>>> pid=10131 comm="cat" name="shadow" dev="dm-0" ino=785423 scontext=
>>>> unconfined_u:system_r:container_t:s0-s0:c0.c1023
>>>> tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=0
>>>> -----------------------------------------------
>>>>
>>>> Below is additional policy(because this is just
>>>> for PoC, I only added denied permssion
>>>> to container_t domain. Also I ran that container
>>>> by "runc" in /tmp directory);
>>>>
>>>> -----------------------------------------------
>>>> (typetransition unconfined_usertype container_runtime_exec_t process
>>>> container_t)
>>>> (roletransition unconfined_r container_runtime_exec_t process system_r)
>>>>
>>>> (allow container_t user_tmp_t (file (open read execute execute_no_trans)))
>>>> (allow container_t var_run_t (dir (write add_name create setattr
>>>> remove_name rmdir)))
>>>> (allow container_t var_run_t (fifo_file (create setattr unlink read open)))
>>>> (allow container_t ptmx_t (chr_file (read write open ioctl)))
>>>> (allow container_t devpts_t (chr_file (setattr read write open ioctl getattr)))
>>>> (allow container_t root_t (dir (mounton)))
>>>> (allow container_t user_tmp_t (dir (mounton write add_name create
>>>> remove_name rmdir)))
>>>> (allow container_t user_tmp_t (lnk_file (read)))
>>>> (allow container_t proc_t (filesystem (mount remount)))
>>>> (allow container_t tmpfs_t (filesystem (mount remount)))
>>>> (allow container_t tmpfs_t (dir (setattr write add_name create mounton)))
>>>> (allow container_t devpts_t (filesystem (mount)))
>>>> (allow container_t sysfs_t (filesystem (mount)))
>>>> (allow container_t cgroup_t (filesystem (remount)))
>>>> (allow container_t tmpfs_t (lnk_file (create)))
>>>> (allow container_t tmpfs_t (chr_file (create setattr read write open
>>>> getattr ioctl append)))
>>>> (allow container_t tmpfs_t (file (open create mounton)))
>>>> (allow container_t proc_t (dir (mounton)))
>>>> (allow container_t proc_t (file (mounton)))
>>>> (allow container_t sysctl_irq_t (dir (mounton)))
>>>> (allow container_t sysctl_t (dir (mounton)))
>>>> (allow container_t sysctl_t (file (mounton)))
>>>> (allow container_t proc_kcore_t (file (mounton)))
>>>> (allow container_t nsfs_t (file (getattr read open)))
>>>> (allow container_t var_run_t (file (create read write open unlink)))
>>>> (allow container_t sysfs_t (dir (mounton)))
>>>> (allow container_t kernel_t (unix_stream_socket (read write)))
>>>> (allow init_t kernel_t (unix_stream_socket (read write)))
>>>> (allow container_t init_t (unix_stream_socket (read write)))
>>>> -----------------------------------------------
>>>>
>>>> I'll also post this result on our community blog.
>>>>
>>>> Kind Regards,
>>>>
>>>> OMO
>>>>
>>>> 2017-01-19 23:06 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>>>> You are not testing with SELinux if it can read /etc/shadow.  The
>>>>> process should be running as container_t or svirt_lxc_net_t if it is an
>>>>> older version.
>>>>>
>>>>>
>>>>> We currently label runc as container_runtime_exec_t.
>>>>>
>>>>> dnf reinstall container-selinux
>>>>>
>>>>> ls -lZ /usr/sbin/runc
>>>>>
>>>>>
>>>>>
>>>>> On 01/19/2017 02:56 AM, 面和毅 wrote:
>>>>>> Dear Sir,
>>>>>>
>>>>>> Thanks. I was checking can we cat /etc/shadow in my testing environment.
>>>>>> It seems that is protected because that file's permission is set to "000".
>>>>>>
>>>>>> Here is my test result;
>>>>>> --------------------------------------------------------------
>>>>>> ----------. 1 root root system_u:object_r:shadow_t:s0
>>>>>> 1268 Oct 13 07:55 /etc/shadow
>>>>>>
>>>>>> SELinux Enforcing  -> Permission Denied
>>>>>> SELinux Permissive -> Permission Denied
>>>>>> SELinux Disabled -> Permission Denied
>>>>>>
>>>>>> When I changed that permission to "755";
>>>>>>
>>>>>> SELinux Enforcing  -> Could cat /etc/shadow
>>>>>> SELinux Permissive -> Could cat /etc/shadow
>>>>>> SELinux Disabled -> Could cat /etc/shadow
>>>>>>
>>>>>> Then in this case that escaped user could
>>>>>> have read access to shadow_t label.
>>>>>> --------------------------------------------------------------
>>>>>>
>>>>>> That "runc" process seems to be working as unconfined_t domain;
>>>>>>
>>>>>> [root@fedora25 ~]# ps axZ|grep runc
>>>>>> unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 1578 pts/0 Sl+
>>>>>> 0:00 runc run ctr
>>>>>>
>>>>>> So, I'm not sure but I guess we would better to assign
>>>>>> other domain to "runc" program (no unconfined_t).
>>>>>>
>>>>>> Let me check if we will run "runc" in other domain.
>>>>>>
>>>>>> Kind Regards,
>>>>>>
>>>>>> OMO
>>>>>>
>>>>>> 2017-01-18 23:14 GMT+09:00 Daniel J Walsh <dwalsh@redhat.com>:
>>>>>>> On 01/18/2017 12:05 AM, 面和毅 wrote:
>>>>>>>> Dear Sir,
>>>>>>>>
>>>>>>>> I'm member of Japan-SOSS SIG(Secure OSS Special
>>>>>>>> Interest Group).
>>>>>>>> We love SELinux(12years user) and we are promoting SELinux in Japan.
>>>>>>>>
>>>>>>>> >From technical interesting(we are promoting Docker
>>>>>>>> with SELinux), we did PoC for CVE-2016-9962 on Fedora25.
>>>>>>>>
>>>>>>>> Then we found current SELinux(maybe policy) does not
>>>>>>>> mitigate that vulnerability.
>>>>>>>>
>>>>>>>> We could reproduce that vulnerability with
>>>>>>>> - add CAP_SYS_PTRACE to container
>>>>>>>> - modified runc because there’s not so much race window on runc.
>>>>>>>> then we think it's not so easy in usual situation.
>>>>>>>> Also we couldn't reproduce it on CentOS7(latest).
>>>>>>>>
>>>>>>>> We posted that PoC result on our community blog.
>>>>>>>> https://jsosug.github.io/post/omok-selinux-docker-20170118/
>>>>>>>>
>>>>>>>> Also we wish to argue how can we protect this kind of
>>>>>>>> vulnerability by using SELinux.
>>>>>>>>
>>>>>>>> Kind Regards,
>>>>>>>>
>>>>>>>> OMO
>>>>>>> Attempt to cat /etc/shadow in your test to see the blockage.
>>>>>>>
>>>>>>> Here is a blog I wrote on the topic.
>>>>>>>
>>>>>>> http://rhelblog.redhat.com/2017/01/13/selinux-mitigates-container-vulnerability/
>>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Kazuki Omo: ka-omo@sios.com
>>>> OSS &Security Evangelist
>>>> OSS Business Planning Dept.
>>>> CISSP #366942
>>>> Tel: +81364015149
>>>
>>>
>>
>
>
>
> --
> Kazuki Omo: ka-omo@sios.com
> OSS &Security Evangelist
> OSS Business Planning Dept.
> CISSP #366942
> Tel: +81364015149



-- 
Kazuki Omo: ka-omo@sios.com
OSS &Security Evangelist
OSS Business Planning Dept.
CISSP #366942
Tel: +81364015149

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2017-01-25 23:19 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-13 19:48 New Blog on how SELinux blocked Docker container escape Daniel J Walsh
2017-01-13 20:43 ` Dominick Grift
2017-01-14  8:23 ` Tracy Reed
2017-01-14  9:35   ` Marko Rauhamaa
2017-01-17 13:48     ` Daniel J Walsh
2017-01-18  5:05       ` 面和毅
2017-01-18 14:12         ` Daniel J Walsh
2017-01-18 14:14         ` Daniel J Walsh
2017-01-19  7:56           ` 面和毅
2017-01-19 14:06             ` Daniel J Walsh
2017-01-20 12:02               ` 面和毅
2017-01-23  4:37                 ` 面和毅
2017-01-25 20:26                   ` Daniel J Walsh
2017-01-25 20:51                     ` 面和毅
2017-01-25 23:19                       ` 面和毅

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.