linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [RFC 0/7] first draft of ZUFS - the Kernel part
@ 2018-03-13 17:14 Boaz Harrosh
  2018-03-13 17:15 ` [RFC 1/7] mm: Add new vma flag VM_LOCAL_CPU Boaz Harrosh
                   ` (6 more replies)
  0 siblings, 7 replies; 39+ messages in thread
From: Boaz Harrosh @ 2018-03-13 17:14 UTC (permalink / raw)
  To: linux-fsdevel
  Cc: Ric Wheeler, Miklos Szeredi, Steve French, Steven Whitehouse,
	Jefff moyer, Sage Weil, Jan Kara, Amir Goldstein, Andy Rudof,
	Anna Schumaker, Amit Golander, Sagi Manole, Shachar Sharon

Hello all

I would please like to present the ZUFS file system and the Kernel code part in this
patchset.

The Kernel code presented here can be found at:
	https://github.com/NetApp/zufs-zuf

And for a very crud User-mode Server:
	https://github.com/NetApp/zufs-zus

ZUFS - stands for Zero-copy User-mode FS
- It is geared towards true zero copy end to end of both data and meta data.
- It is geared towards very *low latency*, very high CPU locality, lock-less parallelism.
- Synchronous operations
- Numa awareness

Short description:
  ZUFS is a from scratch implementation of a filesystem-in-user-space, which tries to address the above
goals. from the get go it is aimed for pmem based FSs. But can easily support other type of FSs that
can utilize x10 latency and parallelism improvements. The novelty of this project is that the interface
is designed with a modern multi-core NUMA machine in mind down to the ABI, so to reach these goals.

Not only FSs need apply, also any kind of user-mode Server can set up a pseudo filesystem and communicate
with application via virtual files. These can then benefit from zero copy low-latency communication
directly to/from application buffers. Or Application mmap direct Server resources. As long as it looks like
a file system to the Kernel.

Current status is that we have couple trivial filesystem implementations and together with the Kernel module
the UM-Server and the FSs User-mode pluggin can actually pass a good bunch of xfstests quick run. (Still working on Stability)

Just to get some points across as I said this project is all about performance and low latency.
Here below are a POC results I have run:

	In Kernel pmem-FS			ZUFS			FUSE	
Threads	Op/s	Lat (us)		Op/s	Lat [us]	Op/s	Lat [us]
1	388361	2.271589		200799	4.6		71820	13.5
2	635115	2.604376		314321	5.9		148083	13.1
4	1260307	2.626361		565574	6.6		212133	18.3
8	2744963	2.485292		1113138	6.6		209799	37.6
12	2126945	5.020506		1598451	6.8		201689	58.7
18	4350995	3.386433		1648689	7.8		174823	101.8
24	4211180	4.784997		1702285	8		149413	159
36	3057166	9.291997		1783346	13.4		148276	240.7
48	3148972	10.382461		1741873	17.4		145296	327.3

I have used an average server machine in our lab with two NUMA nodes and total of 40 cores (Can't remember all the details).
Running fio with 4k random writes. The IO is then just memcpy_nt() to a pmem simulated DRAM. The fio was run with more and
more threads (see threads column)

We can see that we are still > x2 slower than in-Kernel FS. But I believe I can shave off another 1 us by optimizing the
app-to-server thread switch by utilizing perhaps the "Binder" scheduler object or devising another way to not be going
through the scheduler (and its locks) when switching VMs

BE-CAREFUL: This is a big code dump. And very much an RFC. Not yet very stable, not yet cleaned up.
I have sliced the FS to 4 very big patches. Please talk to me if I should split it up to many more
patches.

[I am afraid to send such huge emails so I'm posting web links instead. What is the mailing-list
 message limit anyone knows?

 For first version I'm sending web links to github of the 4 HUGE patches.
 Please clone above trees if you want to play with this.

 Please tell me if you want to be removed from CC of these emails
]

list of patches:
[RFC 1/7] mm: Add new vma flag VM_LOCAL_CPU

   This is a small but very important patch to the mmap code to support
   above scores. See inside the before and after results.
   (And is unfinished)

[RFC 2/7] fs: Add the ZUF filesystem to the build + License
   Add the makefile and Kconfig + the licensing of the code

[RFC 3/7] zuf: Preliminary Documentation
   Very unfinished Start of a Documentation for this project, Overall
   concept. Kernel side and usermode Server side.
   Please help me in asking what you need more in here. Any questions
   I will try to add here

[RFC 4/7] zuf: zuf-rootfs && zuf-core
[RFC 5/7] zus: Devices && mounting
[RFC 6/7] zuf: Filesystem operations
[RFC 7/7] zuf: Write/Read && mmap implementation

  After these 4 HUGE patches. There is a working live system. Still bugs
  and corner cases. But it can git clone and make a Kernel which is for
  me not so bad.

Thank you
Boaz

^ permalink raw reply	[flat|nested] 39+ messages in thread

end of thread, other threads:[~2018-05-07 10:46 UTC | newest]

Thread overview: 39+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-13 17:14 [RFC 0/7] first draft of ZUFS - the Kernel part Boaz Harrosh
2018-03-13 17:15 ` [RFC 1/7] mm: Add new vma flag VM_LOCAL_CPU Boaz Harrosh
2018-03-13 18:56   ` Matthew Wilcox
2018-03-14  8:20     ` Miklos Szeredi
2018-03-14 11:17       ` Matthew Wilcox
2018-03-14 11:31         ` Miklos Szeredi
2018-03-14 11:45           ` Matthew Wilcox
2018-03-14 14:49             ` Miklos Szeredi
2018-03-14 14:57               ` Matthew Wilcox
2018-03-14 15:39                 ` Miklos Szeredi
     [not found]                   ` <CAON-v2ygEDCn90C9t-zadjsd5GRgj0ECqntQSDDtO_Zjk=KoVw@mail.gmail.com>
2018-03-14 16:48                     ` Matthew Wilcox
2018-03-14 21:41       ` Boaz Harrosh
2018-03-15  8:47         ` Miklos Szeredi
2018-03-15 15:27           ` Boaz Harrosh
2018-03-15 15:34             ` Matthew Wilcox
2018-03-15 15:58               ` Boaz Harrosh
2018-03-15 16:10             ` Miklos Szeredi
2018-03-15 16:30               ` Boaz Harrosh
2018-03-15 20:42                 ` Miklos Szeredi
2018-04-25 12:21                   ` Boaz Harrosh
2018-05-07 10:46                     ` Miklos Szeredi
2018-03-13 17:17 ` [RFC 2/7] fs: Add the ZUF filesystem to the build + License Boaz Harrosh
2018-03-13 20:16   ` Andreas Dilger
2018-03-14 17:21     ` Boaz Harrosh
2018-03-15  4:21       ` Andreas Dilger
2018-03-15 13:58         ` Boaz Harrosh
2018-03-13 17:18 ` [RFC 3/7] zuf: Preliminary Documentation Boaz Harrosh
2018-03-13 20:32   ` Randy Dunlap
2018-03-14 18:01     ` Boaz Harrosh
2018-03-14 19:16       ` Randy Dunlap
2018-03-13 17:22 ` [RFC 4/7] zuf: zuf-rootfs && zuf-core Boaz Harrosh
2018-03-13 17:36   ` Boaz Harrosh
2018-03-14 12:56     ` Nikolay Borisov
2018-03-14 18:34       ` Boaz Harrosh
2018-03-13 17:25 ` [RFC 5/7] zus: Devices && mounting Boaz Harrosh
2018-03-13 17:38   ` Boaz Harrosh
2018-03-13 17:28 ` [RFC 6/7] zuf: Filesystem operations Boaz Harrosh
2018-03-13 17:39   ` Boaz Harrosh
2018-03-13 17:32 ` [RFC 7/7] zuf: Write/Read && mmap implementation Boaz Harrosh

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).