procfs reports unexpected mnt_id and path for overlayfs files

procfs reports unexpected mnt_id and path for overlayfs files

[Andrew Vagin]

Hello Miklos and Everyone else,

We have a report that a container with overlayfs can't be checkpointed.

To dump file descriptors we get information from /proc/pid/fd/ and /proc/pid/fdinfo.

But in case of overlayfs we see unexpected values there.

/proc/PID/fdinfo/FD contains mnt_id which isn't presented in /proc/pid/mountinfo.
/proc/PID/fd/FD does not contain an absolute path to the opened file.

Can someone help to fix these issues?

Here is a sequance of commands to reproduce the problem:

[root@ ]# mount -t overlay overlay -olowerdir=a,upperdir=b,workdir=c z
[root@ ]# cat /proc/self/mountinfo | grep over
163 77 0:42 / /home/avagin/tmp/z rw,relatime shared:109 - overlay overlay rw,lowerdir=a,upperdir=b,workdir=c
[root@ ]# ls -l /home/avagin/tmp/z/a
-rw-r--r-- 1 root root 0 Apr 14 09:43 /home/avagin/tmp/z/a
[root@ ]# ipython
In [1]: fd = open("/home/avagin/tmp/z/a")

In [2]: 
[1]+  Stopped                 ipython
[root@ ]# ps -C ipython
  PID TTY          TIME CMD
 8868 pts/6    00:00:00 ipython
[root@ ]# cat /proc/8868/fdinfo/5 
pos:	0
flags:	0100000
mnt_id:	164

# we expect to see mnt_id: 163

[root@ ]# ls -l /proc/8868/fd/5
lr-x------ 1 root root 64 Apr 15 11:21 /proc/8868/fd/5 -> /a

# we expect to see /proc/8868/fd/5 -> /home/avagin/tmp/z/a

Thanks,
Andrew

Re: procfs reports unexpected mnt_id and path for overlayfs files

[Miklos Szeredi]

On Wed, Apr 15, 2015 at 10:50 AM, Andrew Vagin <avagin@odin.com> wrote:
> Hello Miklos and Everyone else,
>
> We have a report that a container with overlayfs can't be checkpointed.
>
> To dump file descriptors we get information from /proc/pid/fd/ and /proc/pid/fdinfo.
>
> But in case of overlayfs we see unexpected values there.
>
> /proc/PID/fdinfo/FD contains mnt_id which isn't presented in /proc/pid/mountinfo.
> /proc/PID/fd/FD does not contain an absolute path to the opened file.
>
> Can someone help to fix these issues?

See this:

https://lkml.org/lkml/2015/3/20/372

Al didn't like that patch.  But it definitely will be fixed, one way
or the other, in the near future.

Thanks,
Miklos

Re: procfs reports unexpected mnt_id and path for overlayfs files

[Saied Kazemi]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Andrew, thank you for reporting and providing an easy way to reproduce the
problem.

Miklos, thanks for the quick reply.  If you'd like to see the issue with
checkpointing and restoring Docker containers using OverlayFS, you can use
the docker binary here  https://github.com/SaiedKazemi/docker/releases.

--Saied




On Wed, Apr 15, 2015 at 8:07 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:

> On Wed, Apr 15, 2015 at 10:50 AM, Andrew Vagin <avagin@odin.com> wrote:
> > Hello Miklos and Everyone else,
> >
> > We have a report that a container with overlayfs can't be checkpointed.
> >
> > To dump file descriptors we get information from /proc/pid/fd/ and
> /proc/pid/fdinfo.
> >
> > But in case of overlayfs we see unexpected values there.
> >
> > /proc/PID/fdinfo/FD contains mnt_id which isn't presented in
> /proc/pid/mountinfo.
> > /proc/PID/fd/FD does not contain an absolute path to the opened file.
> >
> > Can someone help to fix these issues?
>
> See this:
>
> https://lkml.org/lkml/2015/3/20/372
>
> Al didn't like that patch.  But it definitely will be fixed, one way
> or the other, in the near future.
>
> Thanks,
> Miklos
>

[-- Attachment #2: Type: text/html, Size: 1838 bytes --]

Re: procfs reports unexpected mnt_id and path for overlayfs files

[Saied Kazemi]

Hi Miklos,

Is there an update on this patch?  I checked OverlayFS in 4.1-rc5 and
noticed that the issue is still there.

I also observed something interesting/confusing applying this patch to
3.18.  When Docker 1.5 sets up a container, the path into OverlayFS
for a bind mounted file is not visible in /proc/<pid>/mountinfo which
means that the patch has been effective.  But when using Docker 1.7,
the path is visible again.  See below as an example for /etc/hosts:

Using Docker 1.5
# docker run -d busybox:latest ...
# cat /proc/<pid>/mountinfo
...
123 114 8:2 /docker/containers/<container-id>/hosts /etc/hosts
rw,relatime - ext4 /dev/sda2 rw,data=ordered

Using Docker 1.7
# docker run -d busybox:latest ...
# cat /proc/<pid>/mountinfo
...
77 68 8:2 /docker/containers/<container-id>/hosts
/sda2/docker/overlay/<container-id>/merged/etc/hosts rw,relatime -
ext4 /dev/sda2 rw,data=ordered

I verified that the mount system call to set up the container's
filesystem is identical between Docker 1.5 and 1.7.  Any idea as to
what may be causing this problem?

--Saied


On Wed, Apr 15, 2015 at 8:07 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Wed, Apr 15, 2015 at 10:50 AM, Andrew Vagin <avagin@odin.com> wrote:
>> Hello Miklos and Everyone else,
>>
>> We have a report that a container with overlayfs can't be checkpointed.
>>
>> To dump file descriptors we get information from /proc/pid/fd/ and /proc/pid/fdinfo.
>>
>> But in case of overlayfs we see unexpected values there.
>>
>> /proc/PID/fdinfo/FD contains mnt_id which isn't presented in /proc/pid/mountinfo.
>> /proc/PID/fd/FD does not contain an absolute path to the opened file.
>>
>> Can someone help to fix these issues?
>
> See this:
>
> https://lkml.org/lkml/2015/3/20/372
>
> Al didn't like that patch.  But it definitely will be fixed, one way
> or the other, in the near future.
>
> Thanks,
> Miklos

Re: procfs reports unexpected mnt_id and path for overlayfs files

[Saied Kazemi]

Any updates on this?

Thanks,
--Saied


On Wed, Apr 15, 2015 at 8:07 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Wed, Apr 15, 2015 at 10:50 AM, Andrew Vagin <avagin@odin.com> wrote:
>> Hello Miklos and Everyone else,
>>
>> We have a report that a container with overlayfs can't be checkpointed.
>>
>> To dump file descriptors we get information from /proc/pid/fd/ and /proc/pid/fdinfo.
>>
>> But in case of overlayfs we see unexpected values there.
>>
>> /proc/PID/fdinfo/FD contains mnt_id which isn't presented in /proc/pid/mountinfo.
>> /proc/PID/fd/FD does not contain an absolute path to the opened file.
>>
>> Can someone help to fix these issues?
>
> See this:
>
> https://lkml.org/lkml/2015/3/20/372
>
> Al didn't like that patch.  But it definitely will be fixed, one way
> or the other, in the near future.
>
> Thanks,
> Miklos

Re: procfs reports unexpected mnt_id and path for overlayfs files

[Miklos Szeredi]

On Mon, Jul 6, 2015 at 7:45 PM, Saied Kazemi <saied@google.com> wrote:
> On Wed, Apr 15, 2015 at 8:07 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:
>> On Wed, Apr 15, 2015 at 10:50 AM, Andrew Vagin <avagin@odin.com> wrote:
>>> Hello Miklos and Everyone else,
>>>
>>> We have a report that a container with overlayfs can't be checkpointed.
>>>
>>> To dump file descriptors we get information from /proc/pid/fd/ and /proc/pid/fdinfo.
>>>
>>> But in case of overlayfs we see unexpected values there.
>>>
>>> /proc/PID/fdinfo/FD contains mnt_id which isn't presented in /proc/pid/mountinfo.
>>> /proc/PID/fd/FD does not contain an absolute path to the opened file.
>>>
>>> Can someone help to fix these issues?
>>
>> See this:
>>
>> https://lkml.org/lkml/2015/3/20/372
>>
>> Al didn't like that patch.  But it definitely will be fixed, one way
>> or the other, in the near future.

Should be fixed in 4.2.

Thanks,
Miklos