45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Daniel Axtens]

Hi,

I installed a fresh 4.6.0 kernel on my ppc64le system, and tried to run
a docker container. It failed.

Docker gave me errors like this:

docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay/2bc07cedad2770da2f384d5c1e81a6f45fa3c44a7658f10d02e5973ef76620fc-init/merged: invalid argument.

In dmesg, I'm seeing:

overlayfs: upper fs needs to support d_type.

Reverting 45aebeaf4f67 ("ovl: Ensure upper filesystem supports d_type")
fixes the issue for me. I haven't investigated the root cause yet - at a
guess I'd say either Docker's layering system, or some weird interaction
with namespacing, maybe? I'll have a look when I get a spare moment.

For reference, I'm using docker 1.11.0-dev.

Regards,
Daniel Axtens

Re: 45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Vivek Goyal]

On Mon, May 16, 2016 at 04:45:09PM +1000, Daniel Axtens wrote:
> Hi,
> 
> I installed a fresh 4.6.0 kernel on my ppc64le system, and tried to run
> a docker container. It failed.
> 
> Docker gave me errors like this:
> 
> docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay/2bc07cedad2770da2f384d5c1e81a6f45fa3c44a7658f10d02e5973ef76620fc-init/merged: invalid argument.
> 
> In dmesg, I'm seeing:
> 
> overlayfs: upper fs needs to support d_type.
> 

Hi Daniel,

What't the underlying fs you are using. overlayfs requires underlying
filesystem to support d_type and there were cases where xfs was
built with ftype=0 and in that case xfs does not support d_type. That
means it led to issues like whiteouts not being recognized and being
left behind during various operations.

So it became clear that we need a check at mount time to make sure
d_type is supported otherwise error out. This will require users to
do mkfs.xfs with ftype=1 to make progress.

I think new defaults for mkfs.xfs are such that ftype=1 is set. I am
not sure which version that change was made in.

Thanks
Vivek

> Reverting 45aebeaf4f67 ("ovl: Ensure upper filesystem supports d_type")
> fixes the issue for me. I haven't investigated the root cause yet - at a
> guess I'd say either Docker's layering system, or some weird interaction
> with namespacing, maybe? I'll have a look when I get a spare moment.
> 
> For reference, I'm using docker 1.11.0-dev.
> 
> Regards,
> Daniel Axtens

Re: 45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Al Viro]

On Mon, May 16, 2016 at 09:07:27AM -0400, Vivek Goyal wrote:
> So it became clear that we need a check at mount time to make sure
> d_type is supported otherwise error out. This will require users to
> do mkfs.xfs with ftype=1 to make progress.
> 
> I think new defaults for mkfs.xfs are such that ftype=1 is set. I am
> not sure which version that change was made in.

Dumb question - can we end up with empty workdir at that point?  Because
if we do, the check would appear to return a false negative, no matter
what fs supports...

Re: 45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Miklos Szeredi]

On Tue, May 17, 2016 at 8:28 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Mon, May 16, 2016 at 09:07:27AM -0400, Vivek Goyal wrote:
>> So it became clear that we need a check at mount time to make sure
>> d_type is supported otherwise error out. This will require users to
>> do mkfs.xfs with ftype=1 to make progress.
>>
>> I think new defaults for mkfs.xfs are such that ftype=1 is set. I am
>> not sure which version that change was made in.
>
> Dumb question - can we end up with empty workdir at that point?  Because
> if we do, the check would appear to return a false negative, no matter
> what fs supports...

ovl_workdir_create() creates a subdirectory of workdir ("work") so
workdir itself won't be empty after that.  If somebody else messes
with workdir, then we are screwed anyway.

Thanks,
Miklos

Re: 45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Vivek Goyal]

On Tue, May 17, 2016 at 10:15:21AM +0200, Miklos Szeredi wrote:
> On Tue, May 17, 2016 at 8:28 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> > On Mon, May 16, 2016 at 09:07:27AM -0400, Vivek Goyal wrote:
> >> So it became clear that we need a check at mount time to make sure
> >> d_type is supported otherwise error out. This will require users to
> >> do mkfs.xfs with ftype=1 to make progress.
> >>
> >> I think new defaults for mkfs.xfs are such that ftype=1 is set. I am
> >> not sure which version that change was made in.
> >
> > Dumb question - can we end up with empty workdir at that point?  Because
> > if we do, the check would appear to return a false negative, no matter
> > what fs supports...
> 
> ovl_workdir_create() creates a subdirectory of workdir ("work") so
> workdir itself won't be empty after that.  If somebody else messes
> with workdir, then we are screwed anyway.

Right. Initially I was creating a directory of my own and later realized
that ovl_workdir_create() already creates one.

Having said that, what happens when ovl_workdir_create() fails and we
mount overlayfs read only. In that case I think we will conclude that
underlying fs does not support d_type and mounting will fail.

Any thoughts, on how to handle this failure path better?

Daniel,

Yesterday Eric Sandeen told me that I can run "xfs_info <mount-point>" to
figure out if ftype is 0 or 1. You might want to run "xfs_info /" and 
ensure ftype=0 in your case and overlay is not detecting it wrong.

Thanks
Vivek

Re: 45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Miklos Szeredi]

On Tue, May 17, 2016 at 2:10 PM, Vivek Goyal <vgoyal@redhat.com> wrote:

> Having said that, what happens when ovl_workdir_create() fails and we
> mount overlayfs read only. In that case I think we will conclude that
> underlying fs does not support d_type and mounting will fail.

Good point, I missed that.

> Any thoughts, on how to handle this failure path better?

Just skip the d_type test.  We won't create whiteouts that way (and
whiteouts would only have been created with a previous kernel that
didn't have this test, which I don't think is a very interesting
case).

Thanks,
Miklos

Re: 45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Daniel Axtens]

Hi Vivek,

My sincere apologies - it turns out I *was* running on xfs with
ftype=0. Someone in the office had moved docker's storage without
me noticing.

Apologies to all whose time I wasted.

Regards,
Daniel

Vivek Goyal <vgoyal@redhat.com> writes:

> On Tue, May 17, 2016 at 10:15:21AM +0200, Miklos Szeredi wrote:
>> On Tue, May 17, 2016 at 8:28 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
>> > On Mon, May 16, 2016 at 09:07:27AM -0400, Vivek Goyal wrote:
>> >> So it became clear that we need a check at mount time to make sure
>> >> d_type is supported otherwise error out. This will require users to
>> >> do mkfs.xfs with ftype=1 to make progress.
>> >>
>> >> I think new defaults for mkfs.xfs are such that ftype=1 is set. I am
>> >> not sure which version that change was made in.
>> >
>> > Dumb question - can we end up with empty workdir at that point?  Because
>> > if we do, the check would appear to return a false negative, no matter
>> > what fs supports...
>> 
>> ovl_workdir_create() creates a subdirectory of workdir ("work") so
>> workdir itself won't be empty after that.  If somebody else messes
>> with workdir, then we are screwed anyway.
>
> Right. Initially I was creating a directory of my own and later realized
> that ovl_workdir_create() already creates one.
>
> Having said that, what happens when ovl_workdir_create() fails and we
> mount overlayfs read only. In that case I think we will conclude that
> underlying fs does not support d_type and mounting will fail.
>
> Any thoughts, on how to handle this failure path better?
>
> Daniel,
>
> Yesterday Eric Sandeen told me that I can run "xfs_info <mount-point>" to
> figure out if ftype is 0 or 1. You might want to run "xfs_info /" and 
> ensure ftype=0 in your case and overlay is not detecting it wrong.
>
> Thanks
> Vivek

Re: 45aebeaf4f67 "ovl: Ensure upper filesystem supports d_type" breaking Docker

[Daniel Axtens]

Hi Vivek,

Sorry for the delay in getting back to you.


> What't the underlying fs you are using. overlayfs requires underlying
> filesystem to support d_type and there were cases where xfs was
> built with ftype=0 and in that case xfs does not support d_type. That
> means it led to issues like whiteouts not being recognized and being
> left behind during various operations.

I'm using ext4 as the backing filesystem for Docker.

However, if I manually mount an overlay directly on the ext4 fs, it
works just fine.

Therefore, I suspect the problem arises through some magic Docker is
doing. I think it is layering overlays and may be doing that in some
weird way.

I'm hopefully going to get some more time to spend on this problem (and
on that machine) today. As soon as I can unpick what Docker is doing
I'll post more details.

Regards,
Daniel

>
> So it became clear that we need a check at mount time to make sure
> d_type is supported otherwise error out. This will require users to
> do mkfs.xfs with ftype=1 to make progress.
>
> I think new defaults for mkfs.xfs are such that ftype=1 is set. I am
> not sure which version that change was made in.
>
> Thanks
> Vivek
>
>> Reverting 45aebeaf4f67 ("ovl: Ensure upper filesystem supports d_type")
>> fixes the issue for me. I haven't investigated the root cause yet - at a
>> guess I'd say either Docker's layering system, or some weird interaction
>> with namespacing, maybe? I'll have a look when I get a spare moment.
>> 
>> For reference, I'm using docker 1.11.0-dev.
>> 
>> Regards,
>> Daniel Axtens