container userspace tools

container userspace tools

[Daniel Lezcano]

Hi all,

I saw different emails from people asking for user space tools for the 
containers, so I made a library and a set of command line tools.
This component is called "lxc" which stands for "LinuX Container".

The package can be found in the download area of:
    http://sourceforge.net/projects/lxc

There is a cvs repository at:
    http://lxc.cvs.sourceforge.net/lxc

The API is exported in the "lxc.h" header file.

It is not perfect, but it is very light and I hope that will make the 
life easier for the kernel hackers.
It can be used to create virtual private server and to "boot" a minimal 
distro like "debian minimal".

It takes into account the control groups and the checkpoint/restart API 
... in a very optimistic way :)
It should be used with the kernel coming from the git tree, 
git://git.kernel.org/pub/scm/linux/kernel/git/daveh/linux-2.6-lxc.git
But the latest -mm should be ok, some commands will just fail (eg. 
checkpoint/restart).

How to use it ?
--------------------

(for checkpointr/restart see the end of the explanation for this very 
specific, experimental use case).

A container should be created before with (or without) a configuration file.

The configuration file has the following format (all fields are optional):

# set the hostname for the container
lxc.utsname = myhostname
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.hwaddr = 4a:49:43:49:79:bf
lxc.network.ipv4 = 1.2.3.5/24
lxc.network.ipv6 = 2003:db8:1:0:214:1234:fe0b:3597
lxc.mount = fstab
lxc.rootfs = rootfs
lxc.cgroup.cpuset.cpus = 1
lxc.cgroup.cpu.shares = 512

This is an example, but the configuration file can be much more simple 
or complex. In this example the configuration assume the host is 
configured with a bridge, the liblxc does not configure anything not 
related to the container, so it is up to the user/administrator to setup 
a bridge in this case. If the bridge is a problem, different network 
configurations exist to have a container to communicate with the outside 
world.
The network can be specified with several ip addresses, or either if 
there is only one physical network device, it is possible to specify 
multiple network devices for the container. To do that just replicate 
the network sections as follow:

lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.ipv4 = 192.168.9.100/24
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.name = eth1
lxc.network.ipv4 = 192.168.10.100/24

The more there are informations in the configuration file, the more the 
container is isolated. eg. if the network section is removed from the 
configuration file, the network won't be virtualized, the same for the 
ustname.

Step 1 : create the container named 'foo' with the configuration file 
'lxc.conf'

        lxc-create -n foo -f lxc.conf

If no configuration file is specified, the container will be created 
with default configuration, that is to say, pid namespace, mount 
namespace and ipc namespace. The other resources,  network, file system, 
utsname, won't be virtualized.

Step 2 : launch a command inside

        lxc-execute -n foo /bin/bash
            or
        lxc-start -n foo /bin/bash

The differences between lxc-execute and lxc-start are, lxc-execute 
creates an intermediate process for the pid namespace in order to 
support daemons and it mounts /proc and /sys.

Step 3 : Stop the container (from outside of the container)

        lxc-stop -n foo

Step 4 : Destroy the container

        lxc-destroy -n foo

ps: it is not mandatory to destroy the container, you can keep the 
configuration and execute/start the command again. That means after a 
reboot, the container configuration is still available.

These are the minimal steps to use a container, but there are some other 
useful commands:
    lxc-freeze / lxc-unfreeze => stop / resume all processes running 
inside the container
    lxc-monitor                    => follow the different states of the 
container
    lxc-wait                          => wait for a specific state 
before exiting (useful for scripting)
    lxc-ps                              => show all non-virtualized pids 
of a container
    lxc-info                           => show the state of the container
    lxc-cgroup                      => get/set cgroup info for this 
container

Checkpoint/Restart:
--------------------------

    The checkpoint / restart is currently *very* experimental, and takes 
into account only one task. For this reason the **lxc-start**  command 
*must* be used.

    The easiest way is specifying a container with the minimal isolation 
for the minimal CR :)

       lxc-create -n ckpt

       lxc-start -n ckpt ./myverysimpleprogram

       lxc-checkpoint -n ckpt -s > myckptfile

       lxc-restart -n ckpt < myckptfile

ps: you have just to create one time the container.

The lxc-checkpoint command will do freeze - checkpoint - unfreeze, if 
the '-s' option is set, the container is stopped after the checkpoint.

I hope that will make people to have fun with the containers and the new 
kernel features.

Thanks.
  -- Daniel

Re: container userspace tools

[Ian jonhson]

Dear Daniel,

I have tested the container tools in my VM.
It runs very well. But I can not see any highlight
show via container. I believe that two group of
user's processes (with same uid) can be isolated
via container when both of them access files or
are enforced with different quota. But I don't know
how to get the functionalities run.

Any help?

Thanks in advance.

On Wed, Oct 15, 2008 at 9:11 PM, Daniel Lezcano <dlezcano-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org> wrote:
> Ian jonhson wrote:
>>>
>>> The problem is related to the kernel version you have and a missing
>>> functionality.
>>>
>>> You should use the kernel coming from:
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/daveh/linux-2.6-lxc.git
>>>
>>
>> It needs not to set "Container support"  if I run with above kernel,
>> right?
>> I have downloaded the source codes.
>
> The container support will enable at the compile time the different features
> in the kernel. I added this kernel option because I was tired to activate
> the different features one by one as described in the README file. If you
> don't specify this option, the container code will be there in the kernel
> but the not active. So you have to set it.
>

OK, I have activated the "container support" step by step according to
README, but I met a mismatch setting:

Re: container userspace tools

[Daniel Lezcano]

Ian jonhson wrote:
> Dear Daniel,
> 
> I have tested the container tools in my VM.
> It runs very well. But I can not see any highlight
> show via container. 

The container will be more or less isolated depending of what you 
specify in the configuration file.

Without any configuration file, you will have pid, ipc and mount points 
isolated. If you specify the utsname, it will be isolated and if you 
specify the network you will have a new network stack allowing to run 
for example a new sshd server.

In the other side, the cgroup are tied with the container, so you can 
freeze/unfreeze all processes belonging to the container, change the 
priority or assign an amount of physical memory to be used by the container.

> I believe that two group of
> user's processes (with same uid) can be isolated
> via container when both of them access files or
> are enforced with different quota. But I don't know
> how to get the functionalities run.

Allowing to assign quota per container is a good idea, but I don't think 
it is supported by the kernel right now. Perhaps there is a trick to do 
that but I don't know it :)

The rootfs option allows you to specify the root file system to be used 
by the container, so if you specify it, your container will be chrooted 
inside. This feature is at a very early stage and will be improved in 
the future, allowing to example to specify a iso image of a file system 
tree and make use of it.

There are two contributions which are good examples on how to setup a 
container, I added them to:

http://sourceforge.net/projects/lxc/

The first one is a chroot of a sshd server and the second one is a 
minimalist debian showing a full distro booting.

> Any help?
> 
> Thanks in advance.
> 
> On Wed, Oct 15, 2008 at 9:11 PM, Daniel Lezcano <dlezcano-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org> wrote:
>> Ian jonhson wrote:
>>>> The problem is related to the kernel version you have and a missing
>>>> functionality.
>>>>
>>>> You should use the kernel coming from:
>>>>
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/daveh/linux-2.6-lxc.git
>>>>
>>> It needs not to set "Container support"  if I run with above kernel,
>>> right?
>>> I have downloaded the source codes.
>> The container support will enable at the compile time the different features
>> in the kernel. I added this kernel option because I was tired to activate
>> the different features one by one as described in the README file. If you
>> don't specify this option, the container code will be there in the kernel
>> but the not active. So you have to set it.
>>
> 
> OK, I have activated the "container support" step by step according to
> README, but I met a mismatch setting:

It looks like there is a piece missing at the end of the email.

Re: container userspace tools

[Ian jonhson]

> The container will be more or less isolated depending of what you specify in
> the configuration file.
>
yes

> Without any configuration file, you will have pid, ipc and mount points
> isolated. If you specify the utsname, it will be isolated and if you specify
> the network you will have a new network stack allowing to run for example a
> new sshd server.
>

hmm.... then, how to configure the container to get the isolation of
pid, ipc and
mount points? I have looked through the whole configuration example in README
and fond the following settings(which are not tried in my test because
I don't understand
them very well)

 * lxc.mount: is it to mount a file (or volume) so that all the
processes in this container can
    access the content in file?
 * lxc.utsname: I don't know what exact functionlity of this setting is.
 * lxc.network.xxx: are there other options expect
"link","hwaddr","ipv4" and "ipv6"?
 * lxc.rootfs:for application writing data in it?

I am now looking into the codes and find some information about what the
container can do maximally. It seems all the configuration settings are
binding with a container and all the schedules are based on container unit.
Configuration settings only can be setup before container creation and
can not be changed in container lifecycle, right?


> In the other side, the cgroup are tied with the container, so you can
> freeze/unfreeze all processes belonging to the container, change the
> priority or assign an amount of physical memory to be used by the container.
>

In my brain, the cgroup is mainly used in multiple CPUs. In traditional single
CPU machine, can the container separate or determine how much CPU cycle
are used by its processes now? Also, administrator has to configure cgroup
before it takes action. I have no idea whether both of cgroup and container are
totally integrate with each other, or both of them have to be handle
in some cases.


> Allowing to assign quota per container is a good idea, but I don't think it
> is supported by the kernel right now. Perhaps there is a trick to do that
> but I don't know it :)
>
I would like to do this part of job. Also, I need to control several groups of
processes (belonged to same user) in the same time, isolating them,
enforced them with given resource quota, and freeze/unfreeze some of
them.

BTW, as for checkpointing of container, is it easy to checkpoint/restart
given group of processes in above example?

> The rootfs option allows you to specify the root file system to be used by
> the container, so if you specify it, your container will be chrooted inside.
> This feature is at a very early stage and will be improved in the future,
> allowing to example to specify a iso image of a file system tree and make
> use of it.
>

Good. what kinds of rootfs are supported now?  I fond there are a debian file
in sourceforge, is it a rootfs image?


> There are two contributions which are good examples on how to setup a
> container, I added them to:
>
> http://sourceforge.net/projects/lxc/
>
> The first one is a chroot of a sshd server and the second one is a
> minimalist debian showing a full distro booting.
>

In sourceforge, there are

*  	sshd.tar.gz
*  	debian.tar.gz
*       utsname (actually, utstest.c)

I wonder what is the utstest.c?


Thanks again,

Ian

Re: container userspace tools

[Daniel Lezcano]

Ian jonhson wrote:
>> The container will be more or less isolated depending of what you specify in
>> the configuration file.
>>
> yes
> 
>> Without any configuration file, you will have pid, ipc and mount points
>> isolated. If you specify the utsname, it will be isolated and if you specify
>> the network you will have a new network stack allowing to run for example a
>> new sshd server.
>>
> 
> hmm.... then, how to configure the container to get the isolation of
> pid, ipc and
> mount points? 

This is done automatically, with or without configuration.

For examples:

	lxc-execute -n foo -- ps -ef --forest

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 16:55 ?        00:00:00 lxc-execute -n foo -- ps 
-ef --forest
root         2     1  0 16:55 pts/6    00:00:00 ps -ef --forest


	lxc-execute -n foo ls /proc

will only show process 1 and 2, showing the /proc fs has been remount 
inside the container without interfering with your own /proc.

You can do the same check by looking at the ipcs inside and outside the 
container  (assuming they are different).

> I have looked through the whole configuration example in README
> and fond the following settings(which are not tried in my test because
> I don't understand
> them very well)

>  * lxc.mount: is it to mount a file (or volume) so that all the
> processes in this container can access the content in file?

It is a file location in the fstab format. All the mounts point 
specified in this file will be mounted when launching the container.
You have a good example in the sshd contrib package.

>  * lxc.utsname: I don't know what exact functionlity of this setting is.

This option will set the hostname inside the container, so the hostname 
command will be private to the container.

>  * lxc.network.xxx: are there other options expect
> "link","hwaddr","ipv4" and "ipv6"?

This are the most complicated options:

lxc.network.type:
=================
That will specify the type of configuration, there are:
	* empty : new network stack but only with the loopback
	* veth : a bridge + veth pair device configuration, your system should 
be configured with a bridge before this kind of configuration
	* macvlan : virtualize using a macvlan

lxc.network.hwaddr:
===================
That will set a mac address for the virtualized network device (if not 
specified, a random value is used).

lxc.network.link:
=================
That specify the physical network device to be linked with the 
virtualized network device: eg. usually eth0 for macvlan configuration 
or br0 for bridge configuration.

lxc.network.ipv4:
=================
That sets the ipv4 address for the virtualized network device, it is in 
the form 1.2.3.4 and the prefix can be specified by adding "/24"

lxc.network.ipv6:
=================
Idem but for ipv6

There is a documentation about the network virtualization at 
http://lxc.sourceforge.net/network/configuration.php
Please forget Method 1 , it is pointless.

>  * lxc.rootfs:for application writing data in it?

This is the location of where the container will be chrooted, read the 
application will read/write inside.

> I am now looking into the codes and find some information about what the
> container can do maximally. It seems all the configuration settings are
> binding with a container and all the schedules are based on container unit.
> Configuration settings only can be setup before container creation and
> can not be changed in container lifecycle, right?

Yes, that's right.

>> In the other side, the cgroup are tied with the container, so you can
>> freeze/unfreeze all processes belonging to the container, change the
>> priority or assign an amount of physical memory to be used by the container.
>>
> 
> In my brain, the cgroup is mainly used in multiple CPUs. In traditional single
> CPU machine, can the container separate or determine how much CPU cycle
> are used by its processes now? Also, administrator has to configure cgroup
> before it takes action. I have no idea whether both of cgroup and container are
> totally integrate with each other, or both of them have to be handle
> in some cases.

Yes, the cpuset was integrated into the cgroup. But people is adding 
more subsystem to the cgroup. At present, there are the cpuset, the cpu 
accounting and the dev whitelist. There are the memory controller and 
cgroup fair scheduler too. Some other subsystems are not already in the 
mainline but -mm or in a specific patchset, this is the case of the freezer.

The lxc acts as a proxy for the cgroup. So if you mount the cgroup file 
system, you can see there are several subsystem. I have these ones for 
examples for my kernel:

cpuacct.usage			cpuset.sched_relax_domain_level
cpu.rt_period_us		cpu.shares
cpu.rt_runtime_us		devices.allow
cpuset.cpu_exclusive		devices.deny
cpuset.cpus			devices.list
cpuset.mem_exclusive		memory.failcnt
cpuset.mem_hardwall		memory.force_empty
cpuset.memory_migrate		memory.limit_in_bytes
cpuset.memory_pressure		memory.max_usage_in_bytes
cpuset.memory_pressure_enabled	memory.stat
cpuset.memory_spread_page	memory.usage_in_bytes
cpuset.memory_spread_slab	notify_on_release
cpuset.mems			release_agent
cpuset.sched_load_balance	tasks

For example, if I want to assign cpu 1 to my container 'foo', I will 
specify in the configuration file:

lxc.cgroup.cpuset.cpus = 1

If the container is already running, I can change this value by doing:

lxc-cgroup -n foo cpuset.cpu "0,1"


>> Allowing to assign quota per container is a good idea, but I don't think it
>> is supported by the kernel right now. Perhaps there is a trick to do that
>> but I don't know it :)
>>
> I would like to do this part of job. Also, I need to control several groups of
> processes (belonged to same user) in the same time, isolating them,
> enforced them with given resource quota, and freeze/unfreeze some of
> them.

Yeah, it is a good idea, I am not expert in fs/quota but perhaps someone 
in this mailing list can give some info.

Concerning the freeze, this is already part of lxc via 
lxc-freeze/lxc-unfreeze but that relies on the freezer cgroup subsystem 
which should be in mainline soon.

> BTW, as for checkpointing of container, is it easy to checkpoint/restart
> given group of processes in above example?

This is the objective. You should be able to checkpoint at any time the 
container. For example, you launched the container with the command 
lxc-execute -n foo, and later you want to checkpoint it. You can do 
lxc-checkpoint -n foo > my_checkpoint_file.

But the checkpoint / restart is actually under development. The lxc 
checkpoint/restart commands are experimental and the kernel code is at 
the beginning, just a single process can be checkpointed / restarted. 
Before being able to checkpoint multiple processes that will take 
awhile, especially to have it in the kernel mainline. I guess the quota 
do not need to be checkpointed as it is part of the file system, so it 
is always saved.

>> The rootfs option allows you to specify the root file system to be used by
>> the container, so if you specify it, your container will be chrooted inside.
>> This feature is at a very early stage and will be improved in the future,
>> allowing to example to specify a iso image of a file system tree and make
>> use of it.
>>
> 
> Good. what kinds of rootfs are supported now?  I fond there are a debian file
> in sourceforge, is it a rootfs image?

Right now, this is only a directory entry. I plan to change that to 
something more powerful, for example use the union mount, iso image and 
more.

The debian file is a rootfs image.

>> There are two contributions which are good examples on how to setup a
>> container, I added them to:
>>
>> http://sourceforge.net/projects/lxc/
>>
>> The first one is a chroot of a sshd server and the second one is a
>> minimalist debian showing a full distro booting.
>>
> 
> In sourceforge, there are
> 
> *  	sshd.tar.gz
> *  	debian.tar.gz
> *       utsname (actually, utstest.c)
> 
> I wonder what is the utstest.c?

This is a very old testcase, it checks the utsname is private.

> Thanks again,

Thanks.
   -- Daniel

Re: container userspace tools

[Serge E. Hallyn]

Quoting Daniel Lezcano (dlezcano-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org):
> Ian jonhson wrote:
> >> The container will be more or less isolated depending of what you specify in
> >> the configuration file.
> >>
> > yes
> > 
> >> Without any configuration file, you will have pid, ipc and mount points
> >> isolated. If you specify the utsname, it will be isolated and if you specify
> >> the network you will have a new network stack allowing to run for example a
> >> new sshd server.
> >>
> > 
> > hmm.... then, how to configure the container to get the isolation of
> > pid, ipc and
> > mount points? 
> 
> This is done automatically, with or without configuration.

I assume Ian has a kernel compiled with CONFIG_PID_NS=n?

-serge

Re: container userspace tools

[Ian jonhson]

> I assume Ian has a kernel compiled with CONFIG_PID_NS=n?
>


The PID_NS is set with "yes", according to lxc's README.

Re: container userspace tools

[Serge E. Hallyn]

Quoting Ian jonhson (jonhson.ian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org):
> > I assume Ian has a kernel compiled with CONFIG_PID_NS=n?
> >
> 
> 
> The PID_NS is set with "yes", according to lxc's README.

And you say that 'ps -ef' in the container shows all processes
on the system?

If so, that's odd.  What is the distro on your vm?  You're
starting the container as root I assume...  Is there any output
indicating that either the clone(CLONE_NEWPID) or the remount
of /proc failed?

-serge

Re: container userspace tools

[Ian jonhson]

>>
>> hmm.... then, how to configure the container to get the isolation of
>> pid, ipc and
>> mount points?
>
> This is done automatically, with or without configuration.
>
> For examples:
>
>        lxc-execute -n foo -- ps -ef --forest
>
> UID        PID  PPID  C STIME TTY          TIME CMD
> root         1     0  0 16:55 ?        00:00:00 lxc-execute -n foo -- ps -ef
> --forest
> root         2     1  0 16:55 pts/6    00:00:00 ps -ef --forest
>
>
>        lxc-execute -n foo ls /proc
>
> will only show process 1 and 2, showing the /proc fs has been remount inside
> the container without interfering with your own /proc.
>
> You can do the same check by looking at the ipcs inside and outside the
> container  (assuming they are different).
>

Is it possible to isolate processes in two different containers when they access
given local file. For example, I run a process_A in container_A to create a
file named "shared_file". Then, another process (for example, process_B)
in container_A can access the "shared_file", but another process, named
process_C in container_B can not access the same file. The process_A,
process_B, and process_C are run with same uid/gid. how to set the
configurations of container_A and container_B to achieve the isolation?
Is is possible to do this?


>
> This are the most complicated options:
>
> lxc.network.type:
> =================
> That will specify the type of configuration, there are:
>        * empty : new network stack but only with the loopback
>        * veth : a bridge + veth pair device configuration, your system
> should be configured with a bridge before this kind of configuration
>        * macvlan : virtualize using a macvlan
>
> lxc.network.hwaddr:
> lxc.network.link:
> lxc.network.ipv4:
> lxc.network.ipv6:
>
> There is a documentation about the network virtualization at
> http://lxc.sourceforge.net/network/configuration.php
> Please forget Method 1 , it is pointless.
>

It seems that all the network settings just tell how the container
uses network devices. Is it equipped with the functionalities
in somewhere (or in kernel ) to limit which container can connect
outside in a given time or how to schedule  multiple containers
access network via only one network device? Or, further, how much
bandwidth can each container use?


>>> In the other side, the cgroup are tied with the container, so you can
>>> freeze/unfreeze all processes belonging to the container, change the
> Yes, the cpuset was integrated into the cgroup. But people is adding more
> subsystem to the cgroup. At present, there are the cpuset, the cpu
> accounting and the dev whitelist. There are the memory controller and cgroup
> fair scheduler too. Some other subsystems are not already in the mainline
> but -mm or in a specific patchset, this is the case of the freezer.
>
> The lxc acts as a proxy for the cgroup. So if you mount the cgroup file
> system, you can see there are several subsystem. I have these ones for
> examples for my kernel:
>

I wonder here whether cgroup can efficiently isolate two containers access
given memory spaces. In my previous experiment about cgroup, I can not
achieve a ideal result.


> Concerning the freeze, this is already part of lxc via
> lxc-freeze/lxc-unfreeze but that relies on the freezer cgroup subsystem
> which should be in mainline soon.
>

The image of frozen container may be more easy to be migrated to other
homogeneous system.

>> BTW, as for checkpointing of container, is it easy to checkpoint/restart
>> given group of processes in above example?
>
> This is the objective. You should be able to checkpoint at any time the
> container. For example, you launched the container with the command
> lxc-execute -n foo, and later you want to checkpoint it. You can do
> lxc-checkpoint -n foo > my_checkpoint_file.
>
> But the checkpoint / restart is actually under development. The lxc
> checkpoint/restart commands are experimental and the kernel code is at the
> beginning, just a single process can be checkpointed / restarted. Before
> being able to checkpoint multiple processes that will take awhile,
> especially to have it in the kernel mainline.

good

> I guess the quota do not need
> to be checkpointed as it is part of the file system, so it is always saved.
>

Not just concerning with file system, the quota I meant also includes how many
CPU cycle, memory, bandwidth are allowed to use for a running container.
Ideally,  it should also be changed dynamically.



> Right now, this is only a directory entry. I plan to change that to
> something more powerful, for example use the union mount, iso image and
> more.
>

I agree.


Best Regards,

Ian

Re: container userspace tools

[Ian jonhson]

> And you say that 'ps -ef' in the container shows all processes
> on the system?
>

No, just two processes. similar as Denial's:

----------------------------------------------------------------------------------------------
$ sudo  lxc-execute -n foo -- ps -ef --forest
[sudo] password:
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 23:24 ?        00:00:00 lxc-execute -n foo -- ps -ef --f
root         2     1  0 23:24 pts/0    00:00:00 ps -ef --forest
---------------------------------------------------------------------------------------------

>  What is the distro on your vm?

Ubuntu 7.10

> You're
> starting the container as root I assume...

I have to run above commands with root, or it doesn't work.


Best Regards,

Ian

Re: container userspace tools

[Serge E. Hallyn]

Quoting Ian jonhson (jonhson.ian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org):
> > And you say that 'ps -ef' in the container shows all processes
> > on the system?
> >
> 
> No, just two processes. similar as Denial's:
> 
> ----------------------------------------------------------------------------------------------
> $ sudo  lxc-execute -n foo -- ps -ef --forest
> [sudo] password:
> UID        PID  PPID  C STIME TTY          TIME CMD
> root         1     0  0 23:24 ?        00:00:00 lxc-execute -n foo -- ps -ef --f
> root         2     1  0 23:24 pts/0    00:00:00 ps -ef --forest
> ---------------------------------------------------------------------------------------------

Oh, so in fact pids are being properly isolated.

-serge

Re: container userspace tools

[Serge E. Hallyn]

Quoting Ian jonhson (jonhson.ian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org):
> >>
> >> hmm.... then, how to configure the container to get the isolation of
> >> pid, ipc and
> >> mount points?
> >
> > This is done automatically, with or without configuration.
> >
> > For examples:
> >
> >        lxc-execute -n foo -- ps -ef --forest
> >
> > UID        PID  PPID  C STIME TTY          TIME CMD
> > root         1     0  0 16:55 ?        00:00:00 lxc-execute -n foo -- ps -ef
> > --forest
> > root         2     1  0 16:55 pts/6    00:00:00 ps -ef --forest
> >
> >
> >        lxc-execute -n foo ls /proc
> >
> > will only show process 1 and 2, showing the /proc fs has been remount inside
> > the container without interfering with your own /proc.
> >
> > You can do the same check by looking at the ipcs inside and outside the
> > container  (assuming they are different).
> >
> 
> Is it possible to isolate processes in two different containers when they access
> given local file. For example, I run a process_A in container_A to create a
> file named "shared_file". Then, another process (for example, process_B)
> in container_A can access the "shared_file", but another process, named
> process_C in container_B can not access the same file. The process_A,
> process_B, and process_C are run with same uid/gid. how to set the
> configurations of container_A and container_B to achieve the isolation?
> Is is possible to do this?

1. You can put them in each a different chroot.  For instance, extract
the debian.tar.gz which Daniel has pointed you to before into both
/vs1 and /vs2.  Put container_A in /vs1 and container_B in /vs2.

2. You can use SELinux or Smack.  Details for now left as an excercise
to the reader.

3. One day you'll be able to use user namespaces.  In that case, even
though process_{A,B,C} are all running with the same integer uid/gid,
process_C will be in a different different user namespace than
process_{A,B}.  So the file created by process_A will be accessed by
process_C as though they were different userids.  Which means if
process_A set permissions on the file as rwxr-xr-x, then process_c
will get r-x permissions.  If process_A set the permissions to
rwxr-xr-x, then process_c will have no permissions.

-serge

Re: container userspace tools

[Daniel Lezcano]

Ian jonhson wrote:
>>> hmm.... then, how to configure the container to get the isolation of
>>> pid, ipc and
>>> mount points?
>> This is done automatically, with or without configuration.
>>
>> For examples:
>>
>>        lxc-execute -n foo -- ps -ef --forest
>>
>> UID        PID  PPID  C STIME TTY          TIME CMD
>> root         1     0  0 16:55 ?        00:00:00 lxc-execute -n foo -- ps -ef
>> --forest
>> root         2     1  0 16:55 pts/6    00:00:00 ps -ef --forest
>>
>>
>>        lxc-execute -n foo ls /proc
>>
>> will only show process 1 and 2, showing the /proc fs has been remount inside
>> the container without interfering with your own /proc.
>>
>> You can do the same check by looking at the ipcs inside and outside the
>> container  (assuming they are different).
>>
> 
> Is it possible to isolate processes in two different containers when they access
> given local file. For example, I run a process_A in container_A to create a
> file named "shared_file". Then, another process (for example, process_B)
> in container_A can access the "shared_file", but another process, named
> process_C in container_B can not access the same file. The process_A,
> process_B, and process_C are run with same uid/gid. how to set the
> configurations of container_A and container_B to achieve the isolation?
> Is is possible to do this?

Yes, you can do this in different ways. The first one is as Serge 
suggested with chroot, sshd contrib is a good example on how a directory 
tree with a set of mount --bind can be chrooted to ensure file isolation 
(cf the fstab file for this configuration).

This first solution provides isolation and security. Isolation because 
in a container A, the processes running inside can access a file at the 
same location than the processes running inside a container B without 
interacting. Security because processes from container A can not access 
the file in container B.

But, if someone wants to run several instances of the same application 
and wants to avoid the file to be overwritten, the security is pointless 
in this case and a simpler configuration can be used with just a bind 
mount specified in the fstab file lxc option.

For example, we have a computing program writing its results in the 
directory /tmp/results and we want to launch multiple instances of this 
program in parallel. We can do:

mkdir -p /tmp/container_A/results
mkdir -p /tmp/container_B/results

One fstab per container

for container_A:
/tmp/container_A/results /tmp/results none ro,bind 0 0

for container_B:
/tmp/container_B/results /tmp/results none ro,bind 0 0

And one configuration per container with:

for container_A:
lxc.mount = fstab.container_A

for container_B:
lxc.mount = fstab.container_B

And finally:

lxc-execute -n container_A -f ./container_A.conf
lxc-execute -n container_B -f ./configure_B.conf

(if the container was not created before lxc-execute will automatically 
create it and autodestroy it after at exit).

So, the application is unmodified and will continue writing the results 
in /tmp/results which is in fact a private location for the container.

>> This are the most complicated options:
>>
>> lxc.network.type:
>> =================
>> That will specify the type of configuration, there are:
>>        * empty : new network stack but only with the loopback
>>        * veth : a bridge + veth pair device configuration, your system
>> should be configured with a bridge before this kind of configuration
>>        * macvlan : virtualize using a macvlan
>>
>> lxc.network.hwaddr:
>> lxc.network.link:
>> lxc.network.ipv4:
>> lxc.network.ipv6:
>>
>> There is a documentation about the network virtualization at
>> http://lxc.sourceforge.net/network/configuration.php
>> Please forget Method 1 , it is pointless.
>>
> 
> It seems that all the network settings just tell how the container
> uses network devices. Is it equipped with the functionalities
> in somewhere (or in kernel ) to limit which container can connect
> outside in a given time or how to schedule  multiple containers
> access network via only one network device? Or, further, how much
> bandwidth can each container use?

I didn't think about the scheduling. I guess this is certainly a job for 
an upper layer build on top of the liblxc. I wrote a small network 
namespace freezer patch I will post soon. It blocks all the network 
traffic for a given network namespace, so having some kind of resource 
manager making use of this feature can do easily the trick.

Concerning the bandwidth, this is part of the traffic control job and it 
can be plugged with the container. This is something I have in mind, 
that is specify the bandwidth per container. We have all the tools to do 
that now, network isolation and traffic control :)

>>>> In the other side, the cgroup are tied with the container, so you can
>>>> freeze/unfreeze all processes belonging to the container, change the
>> Yes, the cpuset was integrated into the cgroup. But people is adding more
>> subsystem to the cgroup. At present, there are the cpuset, the cpu
>> accounting and the dev whitelist. There are the memory controller and cgroup
>> fair scheduler too. Some other subsystems are not already in the mainline
>> but -mm or in a specific patchset, this is the case of the freezer.
>>
>> The lxc acts as a proxy for the cgroup. So if you mount the cgroup file
>> system, you can see there are several subsystem. I have these ones for
>> examples for my kernel:
>>
> 
> I wonder here whether cgroup can efficiently isolate two containers access
> given memory spaces. In my previous experiment about cgroup, I can not
> achieve a ideal result.

The cgroup does not aim to do isolation, it only provides a mechanism to 
group the processes under the same identifier. Each subsystem, written 
as plugins, make use of this to assign cpu, identify cpu consumption...

But in the other hand, if we use both cgroup and namespaces to provide a 
container with resource management, some isolation should be provided. 
For example, on a 8 cpus system, we assign 2 cpus to a container via 
cgroup/cpuset, because the cpus are not isolated, inside the container 
browsing /proc/cpuinfo will show all the cpus of the system and that 
can, in some cases, lead to an error (eg. sched_setaffinity), IMO.

>> Concerning the freeze, this is already part of lxc via
>> lxc-freeze/lxc-unfreeze but that relies on the freezer cgroup subsystem
>> which should be in mainline soon.
>>
> 
> The image of frozen container may be more easy to be migrated to other
> homogeneous system.
> 
>>> BTW, as for checkpointing of container, is it easy to checkpoint/restart
>>> given group of processes in above example?
>> This is the objective. You should be able to checkpoint at any time the
>> container. For example, you launched the container with the command
>> lxc-execute -n foo, and later you want to checkpoint it. You can do
>> lxc-checkpoint -n foo > my_checkpoint_file.
>>
>> But the checkpoint / restart is actually under development. The lxc
>> checkpoint/restart commands are experimental and the kernel code is at the
>> beginning, just a single process can be checkpointed / restarted. Before
>> being able to checkpoint multiple processes that will take awhile,
>> especially to have it in the kernel mainline.
> 
> good
> 
>> I guess the quota do not need
>> to be checkpointed as it is part of the file system, so it is always saved.
>>
> 
> Not just concerning with file system, the quota I meant also includes how many
> CPU cycle, memory, bandwidth are allowed to use for a running container.
> Ideally,  it should also be changed dynamically.

Ah ok, I see, quota + accounting. I think there is some work around that 
in this mailing list :)

>> Right now, this is only a directory entry. I plan to change that to
>> something more powerful, for example use the union mount, iso image and
>> more.
>>
> 
> I agree.
> 
> 
> Best Regards,

Thanks
   -- Daniel

Re: container userspace tools

[Ian jonhson]

Thank Daniel, Serge, and all the friends in the maillist.
Now, I have given a bird view about current status of container.
The finial question is that how to do contribution:
* what matters/fashion should I obey to develop codes?
* how to upload my patch or my extension codes?
* where should I upload the codes? sourceforge, or gforge?

Are there nay policies I should pay attention?

Thanks again,

Ian


>
> Thanks
>  -- Daniel
>

Re: container userspace tools

[Daniel Lezcano]

Ian jonhson wrote:
> Thank Daniel, Serge, and all the friends in the maillist.
> Now, I have given a bird view about current status of container.
> The finial question is that how to do contribution:
> * what matters/fashion should I obey to develop codes?
> * how to upload my patch or my extension codes?
> * where should I upload the codes? sourceforge, or gforge?
> 
> Are there nay policies I should pay attention?

Hi Ian,

I added the CONTRIBUTING file in the source tree directory, that gives 
the guidelines to submit contribution to 'lxc'.

Regards.
  -- Daniel.

Re: container userspace tools

[Ian jonhson]

Hi again,

> The container will be more or less isolated depending of what you specify in
> the configuration file.
>
> Without any configuration file, you will have pid, ipc and mount points
> isolated. If you specify the utsname, it will be isolated and if you specify
> the network you will have a new network stack allowing to run for example a

I have played with lxc-0.3.0 for several days, trying to transplant
existing applications
on the base of container.

The default configuration can present a view of isolated pid, ipc (e.g. run the
ps or top), however the memory seems not to be isolated. The process in
a container still can see the whole physical memory. Also, my test routine
can still apply more memory than the quota defined in "memory.limit_in_bytes".

Are there anything missing?


Best Regards,

Ian

Re: container userspace tools

[Balbir Singh]

* Ian jonhson <jonhson.ian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> [2008-12-12 15:33:15]:

> Hi again,
> 
> > The container will be more or less isolated depending of what you specify in
> > the configuration file.
> >
> > Without any configuration file, you will have pid, ipc and mount points
> > isolated. If you specify the utsname, it will be isolated and if you specify
> > the network you will have a new network stack allowing to run for example a
> 
> I have played with lxc-0.3.0 for several days, trying to transplant
> existing applications
> on the base of container.
> 
> The default configuration can present a view of isolated pid, ipc (e.g. run the
> ps or top), however the memory seems not to be isolated. The process in
> a container still can see the whole physical memory. Also, my test routine
> can still apply more memory than the quota defined in "memory.limit_in_bytes".
> 
> Are there anything missing?
>

Hi, Ian,

With what version of the kernel due you see this problem. The memory
controller is undergoing a churn and I think we found something that
potentially breaks the memory controller in the current -mm. To verify

1. Can you see the memory.usage_in_bytes corresponding to the control
group where you run the container
2. Check tasks file to see that the container tasks are indeed in the
container.


-- 
	Balbir

Re: container userspace tools

[Ian jonhson]

> With what version of the kernel due you see this problem. The memory
> controller is undergoing a churn and I think we found something that
> potentially breaks the memory controller in the current -mm. To verify
>
I am not sure whether the -mm is opened. I just opened the options as
said in REAEME of lxc-0.3.0. And, the kernel version in my VM is:
2.6.27-rc8.


> 1. Can you see the memory.usage_in_bytes corresponding to the control
> group where you run the container

How can I know the number correspond to the control group?

> 2. Check tasks file to see that the container tasks are indeed in the
> container.
>

My program forks a child and its child (grandson of previous process) call the
the /bin/bash to open a shell. I just can find the parent process is
in container
tasks file, but its child and grandson process are not.


Any help?

>

Re: container userspace tools

[Balbir Singh]

* Ian jonhson <jonhson.ian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> [2008-12-12 17:06:21]:

> > With what version of the kernel due you see this problem. The memory
> > controller is undergoing a churn and I think we found something that
> > potentially breaks the memory controller in the current -mm. To verify
> >
> I am not sure whether the -mm is opened. I just opened the options as
> said in REAEME of lxc-0.3.0. And, the kernel version in my VM is:
> 2.6.27-rc8.
>

OK, so this is not an lxc kernel, but a mainline kernel.
 
> 
> > 1. Can you see the memory.usage_in_bytes corresponding to the control
> > group where you run the container
> 
> How can I know the number correspond to the control group?

From outside the container, figure out the process id of init and then
do a cat /proc/<pid>/cgroup. Then go to that directory and cat
memory.usage_in_bytes, memory.stat and memory.limit_in_bytes to verify
the setup

> 
> > 2. Check tasks file to see that the container tasks are indeed in the
> > container.
> >
> 
> My program forks a child and its child (grandson of previous process) call the
> the /bin/bash to open a shell. I just can find the parent process is
> in container
> tasks file, but its child and grandson process are not.
>

Hmmm.. but the child and grandson are alive right? Do a cat
/proc/<pid>/cgroup on them to see where they went or where they are. 

-- 
	Balbir

Re: container userspace tools

[Daniel Lezcano]

Ian jonhson wrote:
>> With what version of the kernel due you see this problem. The memory
>> controller is undergoing a churn and I think we found something that
>> potentially breaks the memory controller in the current -mm. To verify
>>
> I am not sure whether the -mm is opened. I just opened the options as
> said in REAEME of lxc-0.3.0. And, the kernel version in my VM is:
> 2.6.27-rc8.

2.6.27.8 is out, perhaps you should try this version which contains the 
latest bug fixes.

>> 1. Can you see the memory.usage_in_bytes corresponding to the control
>> group where you run the container
> 
> How can I know the number correspond to the control group?

Assuming you created a container named "foo", you should be able 
retrieve these values with the following commands:

lxc-cgroup -n foo tasks
lxc-cgroup -n foo memory.usage_in_bytes

The lxc-0.5.1 has been released and it fixes some bugs related with the 
cgroup management.

>> 2. Check tasks file to see that the container tasks are indeed in the
>> container.
>>
> 
> My program forks a child and its child (grandson of previous process) call the
> the /bin/bash to open a shell. I just can find the parent process is
> in container
> tasks file, but its child and grandson process are not.
> 
> 
> Any help?
> 

Re: container userspace tools

[Ian jonhson]

>
> OK, so this is not an lxc kernel, but a mainline kernel.
>

So, what is the lxc kernel you meat? Is it different from the
2.6.27-rc8. I use that version  recommended by Daniel.

>>
>> > 1. Can you see the memory.usage_in_bytes corresponding to the control
>> > group where you run the container
>>
>> How can I know the number correspond to the control group?
>
> From outside the container, figure out the process id of init and then
> do a cat /proc/<pid>/cgroup. Then go to that directory and cat
> memory.usage_in_bytes, memory.stat and memory.limit_in_bytes to verify
> the setup
>

OK, I have done the checking, but I don't understand what is meaning of
"corresponding". The content is shown as follows:

$ cat /proc/4506/cgroup
255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/
$ cat memory.usage_in_bytes
223932416


>>
>> > 2. Check tasks file to see that the container tasks are indeed in the
>> > container.
>> >
>>
>> My program forks a child and its child (grandson of previous process) call the
>> the /bin/bash to open a shell. I just can find the parent process is
>> in container
>> tasks file, but its child and grandson process are not.
>>
>
> Hmmm.. but the child and grandson are alive right?
yes... I can see them by a container-outside bash running with ps

> Do a cat
> /proc/<pid>/cgroup on them to see where they went or where they are.
>

They are all there, similar with pid=4506.

$ cat /proc/4506/cgroup
255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/
$ cat /proc/4507/cgroup
255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4507
$ cat /proc/4508/cgroup
255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4506


Any idea?

Re: container userspace tools

[Balbir Singh]

* Ian jonhson <jonhson.ian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> [2008-12-12 18:02:22]:

> >
> > OK, so this is not an lxc kernel, but a mainline kernel.
> >
> 
> So, what is the lxc kernel you meat? Is it different from the
> 2.6.27-rc8. I use that version  recommended by Daniel.
> 
> >>
> >> > 1. Can you see the memory.usage_in_bytes corresponding to the control
> >> > group where you run the container
> >>
> >> How can I know the number correspond to the control group?
> >
> > From outside the container, figure out the process id of init and then
> > do a cat /proc/<pid>/cgroup. Then go to that directory and cat
> > memory.usage_in_bytes, memory.stat and memory.limit_in_bytes to verify
> > the setup
> >
> 
> OK, I have done the checking, but I don't understand what is meaning of
> "corresponding". The content is shown as follows:
> 
> $ cat /proc/4506/cgroup
> 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/
> $ cat memory.usage_in_bytes
> 223932416

OK, so memory.usage_in_bytes is 223932416 for the root cgroup. You
process 4506, which is (I assume) a container process is running in
"/" or the root cgroup. By default root cgroups are unlimited. You can
verify by reading memory.limit_in_bytes

> 
> 
> >>
> >> > 2. Check tasks file to see that the container tasks are indeed in the
> >> > container.
> >> >
> >>
> >> My program forks a child and its child (grandson of previous process) call the
> >> the /bin/bash to open a shell. I just can find the parent process is
> >> in container
> >> tasks file, but its child and grandson process are not.
> >>
> >
> > Hmmm.. but the child and grandson are alive right?
> yes... I can see them by a container-outside bash running with ps
> 
> > Do a cat
> > /proc/<pid>/cgroup on them to see where they went or where they are.
> >
> 
> They are all there, similar with pid=4506.
> 
> $ cat /proc/4506/cgroup
> 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/
> $ cat /proc/4507/cgroup
> 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4507
> $ cat /proc/4508/cgroup
> 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4506
>

Are there three containers on the system

Can you send me the output of <cgroup mount point>/4507/memory.* and
4506/memory.*. cgroup mount point can be obtained from /proc/mounts.
 

-- 
	Balbir

Re: container userspace tools

[Ian jonhson]

>> They are all there, similar with pid=4506.
>>
>> $ cat /proc/4506/cgroup
>> 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/
>> $ cat /proc/4507/cgroup
>> 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4507
>> $ cat /proc/4508/cgroup
>> 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4506
>>
>
> Are there three containers on the system
>
No. one container on system. They are parent-child relationship.


> Can you send me the output of <cgroup mount point>/4507/memory.* and
> 4506/memory.*. cgroup mount point can be obtained from /proc/mounts.
>
>
There is only the directory of 4507 in <cgroup mount point>, not exist
the 4506. I print the stuff as follows:

$ ls memory.*
memory.failcnt      memory.limit_in_bytes      memory.stat
memory.force_empty  memory.max_usage_in_bytes  memory.usage_in_bytes

$ cat memory.failcnt
0

$ cat memory.force_empty
cat: memory.force_empty: Invalid argument

$ cat memory.limit_in_bytes
9223372036854775807

$ cat memory.max_usage_in_bytes
1236992

$ cat memory.stat
cache 0
rss 851968
pgpgin 879
pgpgout 671
active 851968
inactive 0


I don't know why memory.force_empty can not be printed the content.


Any idea?

Re: container userspace tools

[Ian jonhson]

>
> There is only the directory of 4507 in <cgroup mount point>, not exist
> the 4506. I print the stuff as follows:
>
> $ ls memory.*
> memory.failcnt      memory.limit_in_bytes      memory.stat
> memory.force_empty  memory.max_usage_in_bytes  memory.usage_in_bytes
>
> $ cat memory.failcnt
> 0
>
> $ cat memory.force_empty
> cat: memory.force_empty: Invalid argument
>
> $ cat memory.limit_in_bytes
> 9223372036854775807
>
> $ cat memory.max_usage_in_bytes
> 1236992
>
> $ cat memory.stat
> cache 0
> rss 851968
> pgpgin 879
> pgpgout 671
> active 851968
> inactive 0
>
>

Sorry for missing the last one:

$ cat memory.usage_in_bytes
856064