Overwriting copy functionality in filesystem

Overwriting copy functionality in filesystem

[Bharath Vedartham]

Hey everyone,

I was wondering how we can overwrite the copy functionality while
writing our own filesystem in linux. 
VFS does not offer any sort of API for copy. I think it calls create and
write when we execute the copy the file/dir.
I am interested in overwriting the way copy happens in my
filesystem(which I am writing for fun :p). 
Any ideas?

Thanks

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Valdis Klētnieks]

On Sat, 23 Mar 2019 22:29:45 +0530, Bharath Vedartham said:

> I was wondering how we can overwrite the copy functionality while
> writing our own filesystem in linux. 
> VFS does not offer any sort of API for copy. I think it calls create and
> write when we execute the copy the file/dir.

Which you can verify using strace.  Which you should already be familiar with
if you have the experience needed to write a usable filesystem.

> I am interested in overwriting the way copy happens in my
> filesystem(which I am writing for fun :p). 

And what, exactly, do you want copy to do differently on the API level,
and on the file system level?


_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Bernd Petrovitsch]

Hi all!

On 23/03/2019 17:59, Bharath Vedartham wrote:
[..]
> I was wondering how we can overwrite the copy functionality while
> writing our own filesystem in linux. 
> VFS does not offer any sort of API for copy. I think it calls create and

Yes.

> write when we execute the copy the file/dir.
> I am interested in overwriting the way copy happens in my
> filesystem(which I am writing for fun :p). 
> Any ideas?

strace a "cp" and see which system calls are used.
Look into the source code and see how it's implemented.

There is actually more than one method but the usual (while simplest,
most versatile and probably fastest) method used is: open source and
destination files, read from the source, write into the destination
until EOF.

MfG,
	Bernd
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
                     LUGA : http://www.luga.at

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Greg KH]

On Sat, Mar 23, 2019 at 10:29:45PM +0530, Bharath Vedartham wrote:
> Hey everyone,
> 
> I was wondering how we can overwrite the copy functionality while
> writing our own filesystem in linux. 
> VFS does not offer any sort of API for copy. I think it calls create and
> write when we execute the copy the file/dir.
> I am interested in overwriting the way copy happens in my
> filesystem(which I am writing for fun :p). 

There is no "copy" syscall, sorry.


_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Bharath Vedartham]

On Sat, Mar 23, 2019 at 03:01:56PM -0400, Valdis Klētnieks wrote:
> On Sat, 23 Mar 2019 22:29:45 +0530, Bharath Vedartham said:
> 
> > I was wondering how we can overwrite the copy functionality while
> > writing our own filesystem in linux. 
> > VFS does not offer any sort of API for copy. I think it calls create and
> > write when we execute the copy the file/dir.
> 
> Which you can verify using strace.  Which you should already be familiar with
> if you have the experience needed to write a usable filesystem.
>
Yes I did that. I have observed that it is a mixture of
create,read,write system calls.
> > I am interested in overwriting the way copy happens in my
> > filesystem(which I am writing for fun :p). 
> 
> And what, exactly, do you want copy to do differently on the API level,
> and on the file system level?
> 
I was interested in implementing copy-on-write for my filesystem(for fun
:P). When I do a "cp" operation, I do not want to create a seperate
inode for the new file. I only want to create a inode when I make a
change to the file. 

There is no vfs api for cp. I would need to make creat syscall aware of the
fact that it is being executed by "cp". My immediate idea was to check
if a file with the same data exists in the filesystem but that would be
way too much overhead.


_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Bharath Vedartham]

On Sat, Mar 23, 2019 at 08:03:20PM +0100, Bernd Petrovitsch wrote:
> Hi all!
> 
> On 23/03/2019 17:59, Bharath Vedartham wrote:
> [..]
> > I was wondering how we can overwrite the copy functionality while
> > writing our own filesystem in linux. 
> > VFS does not offer any sort of API for copy. I think it calls create and
> 
> Yes.
> 
> > write when we execute the copy the file/dir.
> > I am interested in overwriting the way copy happens in my
> > filesystem(which I am writing for fun :p). 
> > Any ideas?
> 
> strace a "cp" and see which system calls are used.
> Look into the source code and see how it's implemented.
> 
> There is actually more than one method but the usual (while simplest,
> most versatile and probably fastest) method used is: open source and
> destination files, read from the source, write into the destination
> until EOF.
>
I have observed that it is a mixture of system calls. I am interested in
implementing copy-on-write in my filesystem(for fun). I feel that, it
would require my creat vfs api to be aware of the fact that "cp" is
calling it. I want to keep things as generic as possible. I ll proceed
by reading the source of union filesystems (overlayfs) to see how they
implement copy-on-write.
> MfG,
> 	Bernd
> -- 
> Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
>                      LUGA : http://www.luga.at

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Valdis Klētnieks]

On Sun, 24 Mar 2019 18:48:08 +0530, Bharath Vedartham said:

> I was interested in implementing copy-on-write for my filesystem(for fun
> :P). When I do a "cp" operation, I do not want to create a seperate
> inode for the new file. I only want to create a inode when I make a
> change to the file.

Actually, /bin/cp isn't where copy-on-write gets you benefits. Where it really
shines is when you have a versioning filesystem that keeps track of the last
N versions of a file with minimum overhead. So if you have a 100 megabyte
file, open it, write 5 blocks of data, and close it, you now can read back
either the original or new versions of the file, and you're only using 100M plus
5 blocks plus a tiny bit of metadata.

> There is no vfs api for cp. I would need to make creat syscall aware of the
> fact that it is being executed by "cp". My immediate idea was to check
> if a file with the same data exists in the filesystem but that would be
> way too much overhead.

Have you looked at other filesystems that already support copy-on-write?

Hint:  How do file systems that support point-in-time snapshots work?

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Bharath Vedartham]

On Sun, Mar 24, 2019 at 10:06:36AM -0400, Valdis Klētnieks wrote:
> On Sun, 24 Mar 2019 18:48:08 +0530, Bharath Vedartham said:
> 
> > I was interested in implementing copy-on-write for my filesystem(for fun
> > :P). When I do a "cp" operation, I do not want to create a seperate
> > inode for the new file. I only want to create a inode when I make a
> > change to the file.
> 
> Actually, /bin/cp isn't where copy-on-write gets you benefits. Where it really
> shines is when you have a versioning filesystem that keeps track of the last
> N versions of a file with minimum overhead. So if you have a 100 megabyte
> file, open it, write 5 blocks of data, and close it, you now can read back
> either the original or new versions of the file, and you're only using 100M plus
> 5 blocks plus a tiny bit of metadata.
> 
That is a very interesting use case.  Any examples of filesystems like
that in the kernel? 
I was thinking of a use case where we are copying a huge file (say 100
GB), if we do copy-on-write we can speed up /bin/cp for such files i
feel. Any comments on this?
> > There is no vfs api for cp. I would need to make creat syscall aware of the
> > fact that it is being executed by "cp". My immediate idea was to check
> > if a file with the same data exists in the filesystem but that would be
> > way too much overhead.
> 
> Have you looked at other filesystems that already support copy-on-write?
>
I have been looking at overlayfs. But overlayfs uses a lower read-only
layer and an upper layer where changes are reflected. I am more
interested in filesystems without any layering which support
copy-on-write. Are there other filesystems in the kernel which support
copy-on-write?
> Hint:  How do file systems that support point-in-time snapshots work?


_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Overwriting copy functionality in filesystem

[Valdis Klētnieks]

On Fri, 29 Mar 2019 00:00:17 +0530, Bharath Vedartham said:

> I was thinking of a use case where we are copying a huge file (say 100
> GB), if we do copy-on-write we can speed up /bin/cp for such files i
> feel. Any comments on this?

Hmm.. wait a minute.  What definition of "copy on write" are you using?

Hint - if you're copying an *entire* 100GB file, the *fastest* way is to simply
make a second hard link to the file. If you're determined to make an entire
second copy, you're going to be reading 100GB and writing 100GB, and the
exact details aren't going to matter all that much.

Now, where you can get clever is if you create your 100GB file, and then
somebody only changes 8K of the file.  There's no need to copy all 100GB into a
new file if you are able to record "oh, and this 8K got changed". You only need
to write the 8K of changes, and some metadata.

(Similar tricks are used for shared libraries and pre-zero'ed storage.  Everybody
gets a reference to the same copy of the page(s) in memory - until somebody
scribbles on a page.

So say you have a 30MB shared object in memory, with 5 users.  That's 5 references
to the same data.  Now one user writes to it.  The system catches that write (usually
via a page fault), copies just the one page to a new page, and then lets the write to the new
page complete.  Now we have 5 users that all have references to the same (30M-4K)
of data, 4 users that have a reference to the old copy of that 4K, and one user that
has a reference to the modified copy of that 4K.

https://en.wikipedia.org/wiki/Copy-on-write

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies