copy up questions

copy up questions

[Rock Lee]

Hi:
    I am reading the code of overlayfs and have some questions about copy-up.
If a file which doesn't exist on upper directory is modified, this file will
be copied to work directory and then be renamed to upper directory.

1. It seems the whole file(in lower directory) will be copied, that will be
    a big cost in time and space. Is there any way to just copy the
modifed part?
    Like "cp --reflink", keep the modified part in upper directory?

2. What's the purpose of renaming a file from work directory to upper directory?
    Because rename is atomic?

3. When you talk about "a persistent file handle", what's the exact
form of "file handle",
    is it "struct file * file"?

My questions may be silly, I would appreciate it if you could give me
a bit of hint.

Re: copy up questions

[Amir Goldstein]

On Sat, Apr 8, 2017 at 3:45 PM, Rock Lee <rockdotlee@gmail.com> wrote:
>>> Hi:
>>>     I am reading the code of overlayfs and have some questions about copy-up.
>>> If a file which doesn't exist on upper directory is modified, this file will
>>> be copied to work directory and then be renamed to upper directory.
>>>
>>> 1. It seems the whole file(in lower directory) will be copied, that will be
>>>     a big cost in time and space. Is there any way to just copy the
>>> modifed part?
>>>     Like "cp --reflink", keep the modified part in upper directory?

>> Already doing that since v4.10 see try to clone up in copy up data.
>
> Yes, this improvement is impressive. I was reading v4.9, because my
> raspberry pi is  still running this kernel. I am readling v4.10, but
> it seems only XFS and BTRFS support this clone operation.  I am a
> little surprised the UBIFS doesn't support clone, because like BTRFS，
> UBIFS uses b-tree to organize its data.

Well, that's a question for UBIFS folks [cc: Richard] , but I can tell
you this much
b-tree structure is a good starting point for reflink, but that
doesn't mean that
the implementation of reflink is trivial.

>>> 2. What's the purpose of renaming a file from work directory to upper directory?
>>>     Because rename is atomic?

>> Yes. Make new inode visible and persistent only after all data and metadata
>> has been copied.

>>> 3. When you talk about "a persistent file handle", what's the exact
>>> form of "file handle",
>>>     is it "struct file * file"?

>>
>> Where do you see reference to persistent file handle?
>
> I got this in the subject -- overlayfs NFS export.
>

The the form is "struct file_handle", see:
http://man7.org/linux/man-pages/man2/open_by_handle_at.2.html

> One more question, when a file is copied up to upper directory, its
> inode number is changed(of course, it's a new file in upper), but this
> inode number change may be the nightmare for some applications.

Which applications? I know about issues with yum and tar in containers,
but judging by the way the community has dealt with this issues, I would't
say that the term "nightmare" applies.
I am not at all trying to convince that a fix is not needed, but if you know
about reasons that makes this specific behavior a show stopper for
some use cases, please do share this information.

> Is there any way to keep the inode number unchanged?

That's next on the TODO list:
https://github.com/amir73il/overlayfs/wiki/Overlayfs-TODO

Most recent ideas thrown around on this a few days ago:
http://www.spinics.net/lists/linux-unionfs/msg01745.html

The thing is that issues "constant ino" "consistent ro/rw fd" and
"persistent file handle" largely overlap, but are not equivalent.

I am currently focusing on a solution that will solve all 3 issue,
but only for access via nfsd.

Amir.

Re: copy up questions

[Richard Weinberger]

Amir, Rock Lee,

Am 09.04.2017 um 10:43 schrieb Amir Goldstein:
 > Well, that's a question for UBIFS folks [cc: Richard] , but I can tell
> you this much
> b-tree structure is a good starting point for reflink, but that
> doesn't mean that
> the implementation of reflink is trivial.

UBIFS is older than reflinks and UBIFS has no machinery for this feature.
Mostly because nobody really wanted this feature. In combination with overlayfs
it makes now sense to support that.
I agree with Amir that B-Trees are a good start but UBIFS will lacks some
bits to support that, for example reverse block maps.

Doable, sure. On a rainy weekend, no. :-)

Thanks,
//richard