Create ext2 filesystem from a directory

Create ext2 filesystem from a directory

[Richard W.M. Jones]

This might interest/disgust/shock/scare(?!) people on this list:

$ time ./nbdkit --filter=partition -U - linuxdisk . partition=1 --run 'qemu-img convert $nbd /var/tmp/ext2fs.img'

real   0m1.314s
user   0m0.424s
sys    0m0.889s

$ ls -lh /var/tmp/ext2fs.img 
-rw-r--r--. 1 rjones rjones 351M Feb 19 07:44 /var/tmp/ext2fs.img

Code here:

https://github.com/rwmjones/nbdkit/commit/6e7908c828e60f082d84d866070fe8406e6f2b04

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: Create ext2 filesystem from a directory

[Theodore Y. Ts'o]

On Tue, Feb 19, 2019 at 08:03:33AM +0000, Richard W.M. Jones wrote:
> 
> This might interest/disgust/shock/scare(?!) people on this list:
> 
> $ time ./nbdkit --filter=partition -U - linuxdisk . partition=1 --run 'qemu-img convert $nbd /var/tmp/ext2fs.img'
> 

For those of us who aren't really familiar with nbdkit, what does this do?

    	     	    	   	  	   		- Ted

Re: Create ext2 filesystem from a directory

[Richard W.M. Jones]

On Tue, Feb 19, 2019 at 07:44:16PM -0500, Theodore Y. Ts'o wrote:
> On Tue, Feb 19, 2019 at 08:03:33AM +0000, Richard W.M. Jones wrote:
> > 
> > This might interest/disgust/shock/scare(?!) people on this list:
> > 
> > $ time ./nbdkit --filter=partition -U - linuxdisk . partition=1 --run 'qemu-img convert $nbd /var/tmp/ext2fs.img'
> > 
> 
> For those of us who aren't really familiar with nbdkit, what does this do?

There's quite a lot packed into a small space ...

Firstly nbdkit is an NBD server which is unique in having plugins to
implement different data sources.  In this case I have written a
plugin called "linuxdisk" which creates a GPT-partitioned
ext2-formatted disk image from a local directory ("." in this case).

nbdkit can serve over IP or Unix domain sockets, including randomly
created Unix domain sockets ("-U -").

nbdkit can also run other commands and exit when those commands have
finished (we call this "captive nbdkit").  In this case we ask nbdkit
to run a qemu-img command and then exit when qemu-img finishes.

"qemu-img convert $nbd /var/tmp/ext2fs.img" connects to the private
Unix domain socket (nbdkit sets $nbd to a suitable string to make this
happen).  It then copies the data out of that NBD socket to a local
file, optionally doing a format conversion although not in this case.

Finally nbdkit has a concept of filters which can be placed on top of
plugins to select parts of the data.  Remember that the "linuxdisk"
plugin creates GPT-partitioned ext2 disk images.  However we want only
the naked filesystem.  Therefore we place the partition filter on top
to pick out and serve only the partition content, which is the naked
ext2 filesystem.

I recently did a talk about nbdkit if you want to find out more:
https://rwmj.wordpress.com/2019/02/04/video-take-your-loop-mounts-to-the-next-level-with-nbdkit/

Here's another fun thing you can do with the linuxdisk plugin:
https://rwmj.wordpress.com/2019/02/19/nbdkit-linuxdisk-plugin/

Hope that helps!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Re: Create ext2 filesystem from a directory

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Feb 19, 2019, at 12:03 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
> 
> 
> This might interest/disgust/shock/scare(?!) people on this list:
> 
> $ time ./nbdkit --filter=partition -U - linuxdisk . partition=1 --run 'qemu-img convert $nbd /var/tmp/ext2fs.img'
> 
> real   0m1.314s
> user   0m0.424s
> sys    0m0.889s
> 
> $ ls -lh /var/tmp/ext2fs.img
> -rw-r--r--. 1 rjones rjones 351M Feb 19 07:44 /var/tmp/ext2fs.img
> 
> Code here:
> 
> https://github.com/rwmjones/nbdkit/commit/6e7908c828e60f082d84d866070fe8406e6f2b04

How does this compare to "mke2fs -d <source_dir>"?

Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: Create ext2 filesystem from a directory

[Richard W.M. Jones]

On Tue, Feb 19, 2019 at 10:31:24PM -0800, Andreas Dilger wrote:
> On Feb 19, 2019, at 12:03 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
> > 
> > 
> > This might interest/disgust/shock/scare(?!) people on this list:
> > 
> > $ time ./nbdkit --filter=partition -U - linuxdisk . partition=1 --run 'qemu-img convert $nbd /var/tmp/ext2fs.img'
> > 
> > real   0m1.314s
> > user   0m0.424s
> > sys    0m0.889s
> > 
> > $ ls -lh /var/tmp/ext2fs.img
> > -rw-r--r--. 1 rjones rjones 351M Feb 19 07:44 /var/tmp/ext2fs.img
> > 
> > Code here:
> > 
> > https://github.com/rwmjones/nbdkit/commit/6e7908c828e60f082d84d866070fe8406e6f2b04
> 
> How does this compare to "mke2fs -d <source_dir>"?

Wow, good question.  I was completely unaware of this option until
now, but it'll make the implementation massively simpler.  (We still
need the linuxdisk plugin.)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/