overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Rafał Miłecki]

Hi,

In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
rootfs mounted at / we mount rootfs_data volume at /overlay and then use
overlayfs to combine these two.

1) Before:
/dev/root on / type squashfs (ro,relatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime)

2) Mounting overlayfs
mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")

3) After:
/dev/root on / type squashfs (ro,relatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime)
overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)

It works fine as long as there aren't unexpected power cuts. I can easily break
it in a well-reproducible way by doing:
date > bar.txt && rm foo.sh
sleep 2s
[power cut]

On the next boot LEDE/OpenWrt script calls the same mount function but it
results in:
[   10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only

I tracked that error and got a following forward trace for it:
ovl_fill_super
ovl_workdir_create
ovl_cleanup
vfs_rmdir
ubifs_rmdir
check_dir_empty

In other words mounting overlayfs fails because ubifs refuses to remove
work/work as this directory is not empty. On a clean boot /overlay/work/work is
empty and everything works fine but after a power cut I can see
/overlay/work/work/#c752f2a8 file.

I don't really know anything about overlayfs and its workdir, I just expected
overlayfs to recover from such situation on its own. FWIW after a power cut and
reboot my bar.txt is an empty file and foo.sh still exists.

Does it sound like an overlayfs issue to you? Can you think of some way of
fixing this?

-- 
Rafał

Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Richard Weinberger]

Rafał,

On 06.09.2016 12:46, Rafał Miłecki wrote:
> Hi,
> 
> In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
> rootfs mounted at / we mount rootfs_data volume at /overlay and then use
> overlayfs to combine these two.
> 
> 1) Before:
> /dev/root on / type squashfs (ro,relatime)
> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
> 
> 2) Mounting overlayfs
> mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
> 
> 3) After:
> /dev/root on / type squashfs (ro,relatime)
> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
> overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
> 
> It works fine as long as there aren't unexpected power cuts. I can easily break
> it in a well-reproducible way by doing:
> date > bar.txt && rm foo.sh
> sleep 2s
> [power cut]
> 
> On the next boot LEDE/OpenWrt script calls the same mount function but it
> results in:
> [   10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
> 
> I tracked that error and got a following forward trace for it:
> ovl_fill_super
> ovl_workdir_create
> ovl_cleanup
> vfs_rmdir
> ubifs_rmdir
> check_dir_empty
> 
> In other words mounting overlayfs fails because ubifs refuses to remove
> work/work as this directory is not empty. On a clean boot /overlay/work/work is
> empty and everything works fine but after a power cut I can see
> /overlay/work/work/#c752f2a8 file.
> 
> I don't really know anything about overlayfs and its workdir, I just expected
> overlayfs to recover from such situation on its own. FWIW after a power cut and
> reboot my bar.txt is an empty file and foo.sh still exists.
> 
> Does it sound like an overlayfs issue to you? Can you think of some way of
> fixing this?

overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
workdir yourself before mounting.
Miklos?

Thanks,
//richard

Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Miklos Szeredi]

On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
> Rafał,
>
> On 06.09.2016 12:46, Rafał Miłecki wrote:
>> Hi,
>>
>> In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
>> rootfs mounted at / we mount rootfs_data volume at /overlay and then use
>> overlayfs to combine these two.
>>
>> 1) Before:
>> /dev/root on / type squashfs (ro,relatime)
>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>>
>> 2) Mounting overlayfs
>> mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
>>
>> 3) After:
>> /dev/root on / type squashfs (ro,relatime)
>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>> overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
>>
>> It works fine as long as there aren't unexpected power cuts. I can easily break
>> it in a well-reproducible way by doing:
>> date > bar.txt && rm foo.sh
>> sleep 2s
>> [power cut]
>>
>> On the next boot LEDE/OpenWrt script calls the same mount function but it
>> results in:
>> [   10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
>>
>> I tracked that error and got a following forward trace for it:
>> ovl_fill_super
>> ovl_workdir_create
>> ovl_cleanup
>> vfs_rmdir
>> ubifs_rmdir
>> check_dir_empty
>>
>> In other words mounting overlayfs fails because ubifs refuses to remove
>> work/work as this directory is not empty. On a clean boot /overlay/work/work is
>> empty and everything works fine but after a power cut I can see
>> /overlay/work/work/#c752f2a8 file.
>>
>> I don't really know anything about overlayfs and its workdir, I just expected
>> overlayfs to recover from such situation on its own. FWIW after a power cut and
>> reboot my bar.txt is an empty file and foo.sh still exists.
>>
>> Does it sound like an overlayfs issue to you? Can you think of some way of
>> fixing this?
>
> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
> workdir yourself before mounting.
> Miklos?

Fixed in 4.8-rc5 by

eea2fb4851e9 ("ovl: proper cleanup of workdir")

I think it's better if kernel does the cleanup, but if it's easier to
upgrade userspace, then that's a valid workaround.

Thanks,
Miklos

Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Rafał Miłecki]

On 6 September 2016 at 14:50, Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
>> Rafał,
>>
>> On 06.09.2016 12:46, Rafał Miłecki wrote:
>>> Hi,
>>>
>>> In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
>>> rootfs mounted at / we mount rootfs_data volume at /overlay and then use
>>> overlayfs to combine these two.
>>>
>>> 1) Before:
>>> /dev/root on / type squashfs (ro,relatime)
>>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>>>
>>> 2) Mounting overlayfs
>>> mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
>>>
>>> 3) After:
>>> /dev/root on / type squashfs (ro,relatime)
>>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>>> overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
>>>
>>> It works fine as long as there aren't unexpected power cuts. I can easily break
>>> it in a well-reproducible way by doing:
>>> date > bar.txt && rm foo.sh
>>> sleep 2s
>>> [power cut]
>>>
>>> On the next boot LEDE/OpenWrt script calls the same mount function but it
>>> results in:
>>> [   10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
>>>
>>> I tracked that error and got a following forward trace for it:
>>> ovl_fill_super
>>> ovl_workdir_create
>>> ovl_cleanup
>>> vfs_rmdir
>>> ubifs_rmdir
>>> check_dir_empty
>>>
>>> In other words mounting overlayfs fails because ubifs refuses to remove
>>> work/work as this directory is not empty. On a clean boot /overlay/work/work is
>>> empty and everything works fine but after a power cut I can see
>>> /overlay/work/work/#c752f2a8 file.
>>>
>>> I don't really know anything about overlayfs and its workdir, I just expected
>>> overlayfs to recover from such situation on its own. FWIW after a power cut and
>>> reboot my bar.txt is an empty file and foo.sh still exists.
>>>
>>> Does it sound like an overlayfs issue to you? Can you think of some way of
>>> fixing this?
>>
>> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
>> workdir yourself before mounting.
>> Miklos?
>
> Fixed in 4.8-rc5 by
>
> eea2fb4851e9 ("ovl: proper cleanup of workdir")
>
> I think it's better if kernel does the cleanup, but if it's easier to
> upgrade userspace, then that's a valid workaround.

Oh, this should solve my problem! It seems it didn't git stable
kernels yet, but I can backport it on my own for now.

Thanks!

P.S.
Interesting part:
> While this has never been reported, it is possible to get a non-empty
> "work" dir from a previous mount of overlayfs in case of crash in the
> middle of an operation using the work directory.
Funny to hit never-reported problem right after someone (Miklos) fixed it ;)

-- 
Rafał

Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Rafał Miłecki]

On 6 September 2016 at 14:56, Rafał Miłecki <zajec5@gmail.com> wrote:
> On 6 September 2016 at 14:50, Miklos Szeredi <miklos@szeredi.hu> wrote:
>> On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
>>> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
>>> workdir yourself before mounting.
>>> Miklos?
>>
>> Fixed in 4.8-rc5 by
>>
>> eea2fb4851e9 ("ovl: proper cleanup of workdir")
>>
>> I think it's better if kernel does the cleanup, but if it's easier to
>> upgrade userspace, then that's a valid workaround.
>
> Oh, this should solve my problem! It seems it didn't git stable
> kernels yet, but I can backport it on my own for now.

I tried backporting it to 4.4, but it appears it won't be that
trivial. Your patch ("ovl: proper cleanup of workdir") uses e.g.
inode_lock_nested which is available in 4.5+. Commit that added this
function: 5955102c9984f ("wrappers for ->i_mutex access") wasn't
marked for stable, so I guess your patch won't go into 4.4 at all.

I don't expect any magic solution for this, just saying, leaving a
note for reference.

-- 
Rafał

Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"

[Miklos Szeredi]

On Tue, Sep 6, 2016 at 5:53 PM, Rafał Miłecki <zajec5@gmail.com> wrote:
> On 6 September 2016 at 14:56, Rafał Miłecki <zajec5@gmail.com> wrote:
>> On 6 September 2016 at 14:50, Miklos Szeredi <miklos@szeredi.hu> wrote:
>>> On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
>>>> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
>>>> workdir yourself before mounting.
>>>> Miklos?
>>>
>>> Fixed in 4.8-rc5 by
>>>
>>> eea2fb4851e9 ("ovl: proper cleanup of workdir")
>>>
>>> I think it's better if kernel does the cleanup, but if it's easier to
>>> upgrade userspace, then that's a valid workaround.
>>
>> Oh, this should solve my problem! It seems it didn't git stable
>> kernels yet, but I can backport it on my own for now.
>
> I tried backporting it to 4.4, but it appears it won't be that
> trivial. Your patch ("ovl: proper cleanup of workdir") uses e.g.
> inode_lock_nested which is available in 4.5+. Commit that added this
> function: 5955102c9984f ("wrappers for ->i_mutex access") wasn't
> marked for stable, so I guess your patch won't go into 4.4 at all.
>
> I don't expect any magic solution for this, just saying, leaving a
> note for reference.

Conversion is trivial, in fact:

#define inode_lock(inode) mutex_lock(&(inode)->i_mutex)
#define inode_unlock(inode) mutex_unlock(&(inode)->i_mutex)
#define inode_trylock(inode) mutex_trylock(&(inode)->i_mutex)
#define inode_lock_nested(inode, subclass)
mutex_lock_nested(&(inode)->i_mutex, subclass)

And Greg promised to put these defines into -stable for easier
backport experience.

Thanks,
Miklos