C/R and stdio redirection

C/R and stdio redirection

[Sukadev Bhattiprolu]

Suppose we create a container and redirect its stdout/stderr as follows:

	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1

If we attempt to checkpoint the container 'foo', we fail bc one of the
fds in the application refers to /tmp/xyz.out, which is also in use
outside the container (specifically sys_checkpoint() fails due to the
"alien mount ns" check in ckpt_fill_fname()).

It can be argued, 'foo' is not a strict container (since it shares the
fd with another container).  For this reason, we currently need the
CHECKPOINT_SUBTREE flag in lxc-checkpoint.

We initially thought that solving mount-namespaces will solve this, but
realized that they are both separate problems. Mount-namespace C/R addresses
preserving the mounts within the container and /tmp/xyz.out is outside
the container.

So if an application container needs to redirect stdio as above, we should
either 
	a) disable/ignore the alien-mount-ns check or 

	b) try and start the application something like:

		$ cat /tmp/wrapper
		/path/to/app > /tmp/xyz.out 2>&1

		$ lxc-execute --name foo --  /tmp/wrapper

with the difference being /tmp/xyz.out is now inside the container's /tmp
filesystem rather than in the parent container.

Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
set - we had discussed this before and considered it hacky.

Or are there other solutions to this stdio redirection issue ?

Thanks,

Sukadev

Re: C/R and stdio redirection

[Louis Rilling]

[-- Attachment #1.1: Type: text/plain, Size: 2490 bytes --]

On 07/09/10 13:03 -0700, Sukadev Bhattiprolu wrote:
> 
> Suppose we create a container and redirect its stdout/stderr as follows:
> 
> 	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1
> 
> If we attempt to checkpoint the container 'foo', we fail bc one of the
> fds in the application refers to /tmp/xyz.out, which is also in use
> outside the container (specifically sys_checkpoint() fails due to the
> "alien mount ns" check in ckpt_fill_fname()).
> 
> It can be argued, 'foo' is not a strict container (since it shares the
> fd with another container).  For this reason, we currently need the
> CHECKPOINT_SUBTREE flag in lxc-checkpoint.
> 
> We initially thought that solving mount-namespaces will solve this, but
> realized that they are both separate problems. Mount-namespace C/R addresses
> preserving the mounts within the container and /tmp/xyz.out is outside
> the container.
> 
> So if an application container needs to redirect stdio as above, we should
> either 
> 	a) disable/ignore the alien-mount-ns check or 
> 
> 	b) try and start the application something like:
> 
> 		$ cat /tmp/wrapper
> 		/path/to/app > /tmp/xyz.out 2>&1
> 
> 		$ lxc-execute --name foo --  /tmp/wrapper
> 
> with the difference being /tmp/xyz.out is now inside the container's /tmp
> filesystem rather than in the parent container.
> 
> Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
> set - we had discussed this before and considered it hacky.
> 
> Or are there other solutions to this stdio redirection issue ?

I think that this should be solved by a file substitution facility combined with
an ability to ignore "unsupported files" at checkpoint.
File substitution is the ability to tell sys_restart() to substitute some of
caller's file descriptors to checkpointed file descriptors.
Ignoring "unsupported files" means that the checkpoint could record that a given
file descriptor was shared by some files_struct, but the file descriptor itself
is not checkpointed because of various reasons, like file descriptor outside
the container, unsupported file systems, etc.

Both of those mechanisms have been developed in Kerrighed, and Matthieu (in Cc)
should start a discussion about this in the next weeks.

Thanks,

Louis

-- 
Dr Louis Rilling			Kerlabs
Skype: louis.rilling			Batiment Germanium
Phone: (+33|0) 6 80 89 08 23		80 avenue des Buttes de Coesmes
http://www.kerlabs.com/			35700 Rennes

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 206 bytes --]

_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Re: C/R and stdio redirection

[Greg Kurz]

On Tue, 2010-09-07 at 13:03 -0700, Sukadev Bhattiprolu wrote:
> Suppose we create a container and redirect its stdout/stderr as follows:
> 
> 	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1
> 
> If we attempt to checkpoint the container 'foo', we fail bc one of the
> fds in the application refers to /tmp/xyz.out, which is also in use
> outside the container (specifically sys_checkpoint() fails due to the
> "alien mount ns" check in ckpt_fill_fname()).
> 
> It can be argued, 'foo' is not a strict container (since it shares the
> fd with another container).  For this reason, we currently need the
> CHECKPOINT_SUBTREE flag in lxc-checkpoint.
> 
> We initially thought that solving mount-namespaces will solve this, but
> realized that they are both separate problems. Mount-namespace C/R addresses
> preserving the mounts within the container and /tmp/xyz.out is outside
> the container.
> 
> So if an application container needs to redirect stdio as above, we should
> either 
> 	a) disable/ignore the alien-mount-ns check or 
> 
> 	b) try and start the application something like:
> 
> 		$ cat /tmp/wrapper
> 		/path/to/app > /tmp/xyz.out 2>&1
> 
> 		$ lxc-execute --name foo --  /tmp/wrapper
> 
> with the difference being /tmp/xyz.out is now inside the container's /tmp
> filesystem rather than in the parent container.
> 
> Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
> set - we had discussed this before and considered it hacky.
> 
> Or are there other solutions to this stdio redirection issue ?
> 

To be more accurate, this issue is about fd leaking from a parent
container to its descendants. The fd numbers may be anything else than
0,1 or 2 and the underlying files may be regular files, pipes,
sockets... For example, in the HPC world, stdio are often sockets
inheritated from a rshd like daemon.

-- 
Gregory Kurz                                     gkurz-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)534 638 479                           Fax +33 (0)561 400 420

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.

Re: C/R and stdio redirection

[Greg Kurz]

On Wed, 2010-09-08 at 10:41 +0200, Louis Rilling wrote:
> I think that this should be solved by a file substitution facility combined with
> an ability to ignore "unsupported files" at checkpoint.
> File substitution is the ability to tell sys_restart() to substitute some of
> caller's file descriptors to checkpointed file descriptors.
> Ignoring "unsupported files" means that the checkpoint could record that a given
> file descriptor was shared by some files_struct, but the file descriptor itself
> is not checkpointed because of various reasons, like file descriptor outside
> the container, unsupported file systems, etc.
> 

Metacluster (old IBM proprietary C/R solution) had also a similar
mechanism. This is the only sensible way to C/R a container with
inherited descriptors.

> Both of those mechanisms have been developed in Kerrighed, and Matthieu (in Cc)
> should start a discussion about this in the next weeks.
> 

Matthieu doesn't appear on the Cc line...

> Thanks,
> 
> Louis
> 

Regards.

-- 
Gregory Kurz                                     gkurz-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)534 638 479                           Fax +33 (0)561 400 420

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.

Re: C/R and stdio redirection

[Louis Rilling]

[-- Attachment #1.1: Type: text/plain, Size: 665 bytes --]

On 08/09/10 12:00 +0200, Greg Kurz wrote:
> On Wed, 2010-09-08 at 10:41 +0200, Louis Rilling wrote:
> > Both of those mechanisms have been developed in Kerrighed, and Matthieu (in Cc)
> > should start a discussion about this in the next weeks.
> > 
> 
> Matthieu doesn't appear on the Cc line...

He appeared in the Mail-Followup-To line, and received a copy, but it looks like
the mailing list didn't take his address in Cc. Maybe because of the 'é' in his
name?

Louis

-- 
Dr Louis Rilling			Kerlabs
Skype: louis.rilling			Batiment Germanium
Phone: (+33|0) 6 80 89 08 23		80 avenue des Buttes de Coesmes
http://www.kerlabs.com/			35700 Rennes

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 206 bytes --]

_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Re: C/R and stdio redirection

[Sukadev Bhattiprolu]

Greg Kurz [gkurz-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org] wrote:
| On Tue, 2010-09-07 at 13:03 -0700, Sukadev Bhattiprolu wrote:
| > Suppose we create a container and redirect its stdout/stderr as follows:
| > 
| > 	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1
| > 
| > If we attempt to checkpoint the container 'foo', we fail bc one of the
| > fds in the application refers to /tmp/xyz.out, which is also in use
| > outside the container (specifically sys_checkpoint() fails due to the
| > "alien mount ns" check in ckpt_fill_fname()).
| > 
| > It can be argued, 'foo' is not a strict container (since it shares the
| > fd with another container).  For this reason, we currently need the
| > CHECKPOINT_SUBTREE flag in lxc-checkpoint.
| > 
| > We initially thought that solving mount-namespaces will solve this, but
| > realized that they are both separate problems. Mount-namespace C/R addresses
| > preserving the mounts within the container and /tmp/xyz.out is outside
| > the container.
| > 
| > So if an application container needs to redirect stdio as above, we should
| > either 
| > 	a) disable/ignore the alien-mount-ns check or 
| > 
| > 	b) try and start the application something like:
| > 
| > 		$ cat /tmp/wrapper
| > 		/path/to/app > /tmp/xyz.out 2>&1
| > 
| > 		$ lxc-execute --name foo --  /tmp/wrapper
| > 
| > with the difference being /tmp/xyz.out is now inside the container's /tmp
| > filesystem rather than in the parent container.
| > 
| > Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
| > set - we had discussed this before and considered it hacky.
| > 
| > Or are there other solutions to this stdio redirection issue ?
| > 
| 
| To be more accurate, this issue is about fd leaking from a parent
| container to its descendants. The fd numbers may be anything else than
| 0,1 or 2 and the underlying files may be regular files, pipes,
| sockets... For example, in the HPC world, stdio are often sockets
| inheritated from a rshd like daemon.

I agree that fd substitution is the right way to go.

However, Matt Helsley and I were discussing this and wondered if we should
ignore the redirection and expect to user to specify it during restart.

i.e if container was created like this:

	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1

and checkpointed, can we expect the user to restart it like this ?

	lxc-restart --name foo --statefile ckpt.img >> /tmp/xyz.out 

i.e user has to redo the redirection or the output would go to stdout.

Doing this would somehow seem to match a (bogus container) like:

	lxc-execute --name foo -- /path/to/app | sort

If this container is checkpointed/restarted, we can't really redirect
the output of the app to 'sort' right ? So expecting the user to
redo the redirection on restart would treat both redirections ('>'
and '|') in a consistent way ?

Sukadev

Re: C/R and stdio redirection

[Louis Rilling]

[-- Attachment #1.1: Type: text/plain, Size: 3650 bytes --]

On 05/10/10 22:50 -0700, Sukadev Bhattiprolu wrote:
> Greg Kurz [gkurz-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org] wrote:
> | On Tue, 2010-09-07 at 13:03 -0700, Sukadev Bhattiprolu wrote:
> | > Suppose we create a container and redirect its stdout/stderr as follows:
> | > 
> | > 	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1
> | > 
> | > If we attempt to checkpoint the container 'foo', we fail bc one of the
> | > fds in the application refers to /tmp/xyz.out, which is also in use
> | > outside the container (specifically sys_checkpoint() fails due to the
> | > "alien mount ns" check in ckpt_fill_fname()).
> | > 
> | > It can be argued, 'foo' is not a strict container (since it shares the
> | > fd with another container).  For this reason, we currently need the
> | > CHECKPOINT_SUBTREE flag in lxc-checkpoint.
> | > 
> | > We initially thought that solving mount-namespaces will solve this, but
> | > realized that they are both separate problems. Mount-namespace C/R addresses
> | > preserving the mounts within the container and /tmp/xyz.out is outside
> | > the container.
> | > 
> | > So if an application container needs to redirect stdio as above, we should
> | > either 
> | > 	a) disable/ignore the alien-mount-ns check or 
> | > 
> | > 	b) try and start the application something like:
> | > 
> | > 		$ cat /tmp/wrapper
> | > 		/path/to/app > /tmp/xyz.out 2>&1
> | > 
> | > 		$ lxc-execute --name foo --  /tmp/wrapper
> | > 
> | > with the difference being /tmp/xyz.out is now inside the container's /tmp
> | > filesystem rather than in the parent container.
> | > 
> | > Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
> | > set - we had discussed this before and considered it hacky.
> | > 
> | > Or are there other solutions to this stdio redirection issue ?
> | > 
> | 
> | To be more accurate, this issue is about fd leaking from a parent
> | container to its descendants. The fd numbers may be anything else than
> | 0,1 or 2 and the underlying files may be regular files, pipes,
> | sockets... For example, in the HPC world, stdio are often sockets
> | inheritated from a rshd like daemon.
> 
> I agree that fd substitution is the right way to go.
> 
> However, Matt Helsley and I were discussing this and wondered if we should
> ignore the redirection and expect to user to specify it during restart.
> 
> i.e if container was created like this:
> 
> 	lxc-execute -name foo -- /path/to/app > /tmp/xyz.out 2>&1
> 
> and checkpointed, can we expect the user to restart it like this ?
> 
> 	lxc-restart --name foo --statefile ckpt.img >> /tmp/xyz.out 
> 
> i.e user has to redo the redirection or the output would go to stdout.
> 
> Doing this would somehow seem to match a (bogus container) like:
> 
> 	lxc-execute --name foo -- /path/to/app | sort
> 
> If this container is checkpointed/restarted, we can't really redirect
> the output of the app to 'sort' right ? So expecting the user to
> redo the redirection on restart would treat both redirections ('>'
> and '|') in a consistent way ?

From the fd substitution point of view, this means that lxc-restart would
automatically request the substitution of its stdout to the checkpointed
container init's stdout?

This sounds reasonable to me at least. Especially since the container is usually
not supposed to know where the host is redirecting its stdout.

Thanks,

Louis

-- 
Dr Louis Rilling			Kerlabs
Skype: louis.rilling			Batiment Germanium
Phone: (+33|0) 6 80 89 08 23		80 avenue des Buttes de Coesmes
http://www.kerlabs.com/			35700 Rennes

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 206 bytes --]

_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Re: C/R and stdio redirection

[Greg Kurz]

  On 10/06/2010 11:58 AM, Louis Rilling wrote:
> On 05/10/10 22:50 -0700, Sukadev Bhattiprolu wrote:
>> Greg Kurz [gkurz-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org] wrote:
>> | On Tue, 2010-09-07 at 13:03 -0700, Sukadev Bhattiprolu wrote:
>> |>  Suppose we create a container and redirect its stdout/stderr as follows:
>> |>
>> |>  	lxc-execute -name foo -- /path/to/app>  /tmp/xyz.out 2>&1
>> |>
>> |>  If we attempt to checkpoint the container 'foo', we fail bc one of the
>> |>  fds in the application refers to /tmp/xyz.out, which is also in use
>> |>  outside the container (specifically sys_checkpoint() fails due to the
>> |>  "alien mount ns" check in ckpt_fill_fname()).
>> |>
>> |>  It can be argued, 'foo' is not a strict container (since it shares the
>> |>  fd with another container).  For this reason, we currently need the
>> |>  CHECKPOINT_SUBTREE flag in lxc-checkpoint.
>> |>
>> |>  We initially thought that solving mount-namespaces will solve this, but
>> |>  realized that they are both separate problems. Mount-namespace C/R addresses
>> |>  preserving the mounts within the container and /tmp/xyz.out is outside
>> |>  the container.
>> |>
>> |>  So if an application container needs to redirect stdio as above, we should
>> |>  either
>> |>  	a) disable/ignore the alien-mount-ns check or
>> |>
>> |>  	b) try and start the application something like:
>> |>
>> |>  		$ cat /tmp/wrapper
>> |>  		/path/to/app>  /tmp/xyz.out 2>&1
>> |>
>> |>  		$ lxc-execute --name foo --  /tmp/wrapper
>> |>
>> |>  with the difference being /tmp/xyz.out is now inside the container's /tmp
>> |>  filesystem rather than in the parent container.
>> |>
>> |>  Maybe we can go with approach 'a' above only if CHECKPOINT_SUBTREE is also
>> |>  set - we had discussed this before and considered it hacky.
>> |>
>> |>  Or are there other solutions to this stdio redirection issue ?
>> |>
>> |
>> | To be more accurate, this issue is about fd leaking from a parent
>> | container to its descendants. The fd numbers may be anything else than
>> | 0,1 or 2 and the underlying files may be regular files, pipes,
>> | sockets... For example, in the HPC world, stdio are often sockets
>> | inheritated from a rshd like daemon.
>>
>> I agree that fd substitution is the right way to go.
>>
>> However, Matt Helsley and I were discussing this and wondered if we should
>> ignore the redirection and expect to user to specify it during restart.
>>
>> i.e if container was created like this:
>>
>> 	lxc-execute -name foo -- /path/to/app>  /tmp/xyz.out 2>&1
>>
>> and checkpointed, can we expect the user to restart it like this ?
>>
>> 	lxc-restart --name foo --statefile ckpt.img>>  /tmp/xyz.out
>>
>> i.e user has to redo the redirection or the output would go to stdout.
>>
>> Doing this would somehow seem to match a (bogus container) like:
>>
>> 	lxc-execute --name foo -- /path/to/app | sort
>>
>> If this container is checkpointed/restarted, we can't really redirect
>> the output of the app to 'sort' right ? So expecting the user to
>> redo the redirection on restart would treat both redirections ('>'
>> and '|') in a consistent way ?
>  From the fd substitution point of view, this means that lxc-restart would
> automatically request the substitution of its stdout to the checkpointed
> container init's stdout?
>

Yes, and this should apply to any inherited file descriptor (not only 
0,1 and 2).

> This sounds reasonable to me at least. Especially since the container is usually
> not supposed to know where the host is redirecting its stdout.
>
> Thanks,
>
> Louis
>
--
Greg