[PATCH 0/4] kvmtool: add PRE_INIT support for ARM/ARM64

[PATCH 0/4] kvmtool: add PRE_INIT support for ARM/ARM64

[Andre Przywara]

Hi,

the new PRE_INIT feature of kvmtool tries to get rid of the static
linking requirement (for ARM, at least) kvmtool currently comes with:
To be able to use the sandbox feature, kvmtool comes with an init
replacement, which is a statically linked binary injected into the
guest. It initializes the Linux environment and launches the command
provided on the kvmtool command line.
Now this results in a rather big binary blob (around 750K) sitting
in the kvmtool binary.
A recent commit provides another implementation for x86-64:
A tiny assembly stub mounts the host filesystem via 9pfs and calls
the now dynamically linked init binary. This removes the need for
statically linking the guest_init binary and is also much smaller
(find the numbers in patch 4/4).

This series provides implementations for ARM and ARM64. Since I needed
to wrap my mind around the original x86_64 asm code anyway, I also
added an i386 implementation.

Cheers,
Andre.

Andre Przywara (4):
  pre_init: rename x86/init.S to x86/init64.S
  pre_init: x86/init64.S: use defines for syscall numbers
  pre_init: add support for i386 also
  pre_init: add ARM implementations

 Makefile                 |  5 ++++-
 arm/aarch32/init.S       | 44 ++++++++++++++++++++++++++++++++++++++++++
 arm/aarch64/init.S       | 44 ++++++++++++++++++++++++++++++++++++++++++
 x86/init32.S             | 50 ++++++++++++++++++++++++++++++++++++++++++++++++
 x86/{init.S => init64.S} | 14 +++++++++++---
 5 files changed, 153 insertions(+), 4 deletions(-)
 create mode 100644 arm/aarch32/init.S
 create mode 100644 arm/aarch64/init.S
 create mode 100644 x86/init32.S
 rename x86/{init.S => init64.S} (71%)

-- 
2.6.4

[PATCH 1/4] pre_init: rename x86/init.S to x86/init64.S

[Andre Przywara]

To make room for the 32-bit implementation of that assembly stub,
rename it.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Makefile                 | 2 +-
 x86/{init.S => init64.S} | 0
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename x86/{init.S => init64.S} (100%)

diff --git a/Makefile b/Makefile
index 19eb7e5..e26afa3 100644
--- a/Makefile
+++ b/Makefile
@@ -110,7 +110,7 @@ endif
 ifeq ($(ARCH),x86_64)
 	ARCH         := x86
 	DEFINES      += -DCONFIG_X86_64
-	ARCH_PRE_INIT = x86/init.S
+	ARCH_PRE_INIT = x86/init64.S
 endif
 
 ### Arch-specific stuff
diff --git a/x86/init.S b/x86/init64.S
similarity index 100%
rename from x86/init.S
rename to x86/init64.S
-- 
2.6.4

[PATCH 2/4] pre_init: x86/init64.S: use defines for syscall numbers

[Andre Przywara]

Instead of hardcoding the syscall numbers and providing the symbol
in the comments, just include <asm/unistd.h> and use the symbol
directly.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 x86/init64.S | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/x86/init64.S b/x86/init64.S
index 488a93f..1a7c40f 100644
--- a/x86/init64.S
+++ b/x86/init64.S
@@ -1,3 +1,11 @@
+/*
+ * #!/bin/sh
+ * mount -t 9p -o trans=virtio,version=9p2000.L hostfs /host
+ * /virt/init $*
+ */
+
+#include <asm/unistd.h>
+
 .data
 
 .m_dev:
@@ -16,7 +24,7 @@
 .globl _start
 _start:
 
-	mov $165, %rax		# __NR_mount
+	mov $__NR_mount, %rax
 	mov $.m_dev, %rdi
 	mov $.m_dir, %rsi
 	mov $.m_typ, %rdx
@@ -24,7 +32,7 @@ _start:
 	mov $.m_opt, %r8
 	syscall
 
-	mov $59, %rax		# __NR_execve
+	mov $__NR_execve, %rax
 	mov $.e_nam, %rdi
 	lea 8(%rsp), %rsi	# argv[]
 	mov %rdi, (%rsi)	# change argv[0]
@@ -33,6 +41,6 @@ _start:
 	lea (%rsi,%rcx,8), %rdx # envp[]
 	syscall
 
-	mov $60, %rax		# __NR_exit
+	mov $__NR_exit, %rax
 	mov $1, %rdi
 	syscall			# panic
-- 
2.6.4

[PATCH 3/4] pre_init: add support for i386 also

[Andre Przywara]

Currently the pre_init support is provided only for x86_64. Since
having 32-bit x86 supported as well is not far off, just add an
implementation using i386 assembly instructions and the respective
syscall ABI.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Makefile     |  1 +
 x86/init32.S | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+)
 create mode 100644 x86/init32.S

diff --git a/Makefile b/Makefile
index e26afa3..54bdd31 100644
--- a/Makefile
+++ b/Makefile
@@ -106,6 +106,7 @@ ARCH ?= $(shell uname -m | sed -e s/i.86/i386/ -e s/ppc.*/powerpc/ \
 ifeq ($(ARCH),i386)
 	ARCH         := x86
 	DEFINES      += -DCONFIG_X86_32
+	ARCH_PRE_INIT = x86/init32.S
 endif
 ifeq ($(ARCH),x86_64)
 	ARCH         := x86
diff --git a/x86/init32.S b/x86/init32.S
new file mode 100644
index 0000000..8937deb
--- /dev/null
+++ b/x86/init32.S
@@ -0,0 +1,50 @@
+/*
+ * #!/bin/sh
+ * mount -t 9p -o trans=virtio,version=9p2000.L hostfs /host
+ * /virt/init $*
+ */
+
+#include <asm/unistd.h>
+.text
+.globl _start
+_start:
+
+# Linux syscall ABI
+# i386/int 0x80:  nr = eax, args: ebx ecx edx esi edi ebp
+# x86_64/syscall: nr = rax, args: rdi rsi rdx r10 r8  r9
+
+	mov	$__NR_mount, %eax
+	mov	$.m_dev, %ebx
+	mov	$.m_dir, %ecx
+	mov	$.m_typ, %edx
+	mov	$1, %esi		# MS_RDONLY
+	mov	$.m_opt, %edi
+	int	$0x80
+
+	mov	$__NR_execve, %eax
+	mov	$.e_nam, %ebx		# filename: "/virt/init"
+	lea	4(%esp), %ecx		# pass through argv
+	mov	%ebx, (%ecx)		# overwrite argv[0] with "/virt/init"
+	mov	(%esp), %edx		# read argc
+	inc	%edx
+	lea	(%ecx,%edx,4), %edx	# envp = argv + argc + 1
+	int	$0x80
+
+	mov	$__NR_exit, %eax
+	mov	$1, %ebx
+	int	$0x80			# panic
+
+loop:	hlt
+	jmp	loop
+
+.m_dev:
+.string "hostfs"
+.m_dir:
+.string "/host"
+.m_typ:
+.string "9p"
+.m_opt:
+.string "trans=virtio,version=9p2000.L"
+
+.e_nam:
+.string "/virt/init"
-- 
2.6.4

[PATCH 4/4] pre_init: add ARM implementations

[Andre Przywara]

The pre_init stub consists of two syscalls mouting the host's FS
via 9pfs and then calling the actual init binary, which can now
use normal dynamic linking.
Based on the x86 code provide an ARM and ARM64 implementation of
that. Beside removing the need for static linkage it reduces the
size of the kvmtool binary by quite a lot (numbers for aarch64):

-rwxr-xr-x 1 root root     9952 Nov 16 14:37 guest/init
-rwxr-xr-x 1 root root      512 Nov 16 14:37 guest/pre_init
-rwxr-xr-x 2 root root  1284704 Nov 16 14:37 lkvm
vs. the old version:
-rwxr-xr-x 1 root root   776024 Nov 16 14:38 guest/init
-rwxr-xr-x 2 root root  2050112 Nov 16 14:38 lkvm

Tested on Midway and Juno.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Makefile           |  2 ++
 arm/aarch32/init.S | 44 ++++++++++++++++++++++++++++++++++++++++++++
 arm/aarch64/init.S | 44 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 90 insertions(+)
 create mode 100644 arm/aarch32/init.S
 create mode 100644 arm/aarch64/init.S

diff --git a/Makefile b/Makefile
index 54bdd31..57b0bdd 100644
--- a/Makefile
+++ b/Makefile
@@ -161,6 +161,7 @@ ifeq ($(ARCH), arm)
 	OBJS		+= $(OBJS_ARM_COMMON)
 	OBJS		+= arm/aarch32/arm-cpu.o
 	OBJS		+= arm/aarch32/kvm-cpu.o
+	ARCH_PRE_INIT	=  arm/aarch32/init.S
 	ARCH_INCLUDE	:= $(HDRS_ARM_COMMON)
 	ARCH_INCLUDE	+= -Iarm/aarch32/include
 	CFLAGS		+= -march=armv7-a
@@ -174,6 +175,7 @@ ifeq ($(ARCH), arm64)
 	OBJS		+= $(OBJS_ARM_COMMON)
 	OBJS		+= arm/aarch64/arm-cpu.o
 	OBJS		+= arm/aarch64/kvm-cpu.o
+	ARCH_PRE_INIT	=  arm/aarch64/init.S
 	ARCH_INCLUDE	:= $(HDRS_ARM_COMMON)
 	ARCH_INCLUDE	+= -Iarm/aarch64/include
 
diff --git a/arm/aarch32/init.S b/arm/aarch32/init.S
new file mode 100644
index 0000000..c023195
--- /dev/null
+++ b/arm/aarch32/init.S
@@ -0,0 +1,44 @@
+/*
+ * #!/bin/sh
+ * mount -t 9p -o trans=virtio,version=9p2000.L hostfs /host
+ * /virt/init $*
+ */
+
+#include <asm/unistd.h>
+
+.text
+.globl _start
+_start:
+
+	mov	r7, #__NR_mount
+	ldr	r0, =.m_dev
+	ldr	r1, =.m_dir
+	ldr	r2, =.m_typ
+	mov	r3, #1			// MS_RDONLY
+	ldr	r4, =.m_opt
+	svc	#0
+
+	mov	r7, #__NR_execve
+	ldr	r0, =.e_nam		// 1st arg: filename
+	add	r1, sp, #4		// 2nd arg: argv[0]
+	str	r0, [r1]		// change argv[0]
+	ldr	r2, [sp]
+	add	r2, r2, #1		// r2 = argc + 1
+	add	r2, r1, r2, lsl #2	// 3rd arg: envp = argv[0] + argc + 1
+	svc	#0
+
+	mov	r7, #__NR_exit
+	mov	r0, #1
+	svc	#0			// panic
+
+.m_dev:
+.string "hostfs"
+.m_dir:
+.string "/host"
+.m_typ:
+.string "9p"
+.m_opt:
+.string "trans=virtio,version=9p2000.L"
+
+.e_nam:
+.string "/virt/init"
diff --git a/arm/aarch64/init.S b/arm/aarch64/init.S
new file mode 100644
index 0000000..567bf97
--- /dev/null
+++ b/arm/aarch64/init.S
@@ -0,0 +1,44 @@
+/*
+ * #!/bin/sh
+ * mount -t 9p -o trans=virtio,version=9p2000.L hostfs /host
+ * /virt/init $*
+ */
+
+#include <asm/unistd.h>
+
+.text
+.globl _start
+_start:
+
+	mov	x8, #__NR_mount
+	ldr	x0, =.m_dev
+	ldr	x1, =.m_dir
+	ldr	x2, =.m_typ
+	mov	x3, #1			// MS_RDONLY
+	ldr	x4, =.m_opt
+	svc	#0
+
+	mov	x8, #__NR_execve
+	ldr	x0, =.e_nam		// 1st arg: filename
+	add	x1, sp, #8		// 2nd arg: argv (passed through)
+	str	x0, [x1]		// change argv[0] to contain filename
+	ldr	x2, [sp]
+	add	x2, x2, #1		// x2 = argc + 1
+	add	x2, x1, x2, lsl #3	// 3rd arg: envp = argv[0] + argc + 1
+	svc	#0
+
+	mov	x8, #__NR_exit
+	mov	x0, #1
+	svc	#0			// panic
+
+.m_dev:
+.string "hostfs"
+.m_dir:
+.string "/host"
+.m_typ:
+.string "9p"
+.m_opt:
+.string "trans=virtio,version=9p2000.L"
+
+.e_nam:
+.string "/virt/init"
-- 
2.6.4

Re: [PATCH 3/4] pre_init: add support for i386 also

[Will Deacon]

On Wed, Feb 24, 2016 at 03:33:07PM +0000, Andre Przywara wrote:
> Currently the pre_init support is provided only for x86_64. Since
> having 32-bit x86 supported as well is not far off, just add an
> implementation using i386 assembly instructions and the respective
> syscall ABI.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  Makefile     |  1 +
>  x86/init32.S | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 51 insertions(+)
>  create mode 100644 x86/init32.S

Did you manage to test this?

Will

Re: [PATCH 4/4] pre_init: add ARM implementations

[Will Deacon]

On Wed, Feb 24, 2016 at 03:33:08PM +0000, Andre Przywara wrote:
> The pre_init stub consists of two syscalls mouting the host's FS
> via 9pfs and then calling the actual init binary, which can now
> use normal dynamic linking.
> Based on the x86 code provide an ARM and ARM64 implementation of
> that. Beside removing the need for static linkage it reduces the
> size of the kvmtool binary by quite a lot (numbers for aarch64):
> 
> -rwxr-xr-x 1 root root     9952 Nov 16 14:37 guest/init
> -rwxr-xr-x 1 root root      512 Nov 16 14:37 guest/pre_init
> -rwxr-xr-x 2 root root  1284704 Nov 16 14:37 lkvm
> vs. the old version:
> -rwxr-xr-x 1 root root   776024 Nov 16 14:38 guest/init
> -rwxr-xr-x 2 root root  2050112 Nov 16 14:38 lkvm
> 
> Tested on Midway and Juno.

Hmm, I'm not super keen on switching behaviour like this on arm, where
it's not uncommon to build a static lkvm and transfer it to a remote
target and expect init to work.

Perhaps we could only do this when building a dynamic executable?

Will

Re: [PATCH 4/4] pre_init: add ARM implementations

[André Przywara]

Hi,

On 02/03/16 03:00, Will Deacon wrote:
> On Wed, Feb 24, 2016 at 03:33:08PM +0000, Andre Przywara wrote:
>> The pre_init stub consists of two syscalls mouting the host's FS
>> via 9pfs and then calling the actual init binary, which can now
>> use normal dynamic linking.
>> Based on the x86 code provide an ARM and ARM64 implementation of
>> that. Beside removing the need for static linkage it reduces the
>> size of the kvmtool binary by quite a lot (numbers for aarch64):
>>
>> -rwxr-xr-x 1 root root     9952 Nov 16 14:37 guest/init
>> -rwxr-xr-x 1 root root      512 Nov 16 14:37 guest/pre_init
>> -rwxr-xr-x 2 root root  1284704 Nov 16 14:37 lkvm
>> vs. the old version:
>> -rwxr-xr-x 1 root root   776024 Nov 16 14:38 guest/init
>> -rwxr-xr-x 2 root root  2050112 Nov 16 14:38 lkvm
>>
>> Tested on Midway and Juno.
> 
> Hmm, I'm not super keen on switching behaviour like this on arm, where
> it's not uncommon to build a static lkvm and transfer it to a remote
> target and expect init to work.

So are you concerned about a fully static root file system on the host,
which does not provide libc.so and/or ld-linux.so at all? Is that really
a use case? I had the impression that people use a statically linked
kvmtool to avoid dependencies like to libfdt.so.
In this case I am wondering if we should provide some switch to build a
static lkvm with a static init if people are concerned, or we should
ship a guest/init binary statically linked against musl libc, for
instance: this is only 29K compared to the above multi-100 KB gcc version.
Or is there some trick to build small static binaries linked against glibc?
Actually by just looking at init.c: Should we code the whole of it in
assembly? Apart from printf it only consists of syscalls.

> Perhaps we could only do this when building a dynamic executable?

This is of course an option as well.

Cheers,
Andre

Re: [PATCH 3/4] pre_init: add support for i386 also

[Andre Przywara]

Hi Will,

On 02/03/16 02:57, Will Deacon wrote:
> On Wed, Feb 24, 2016 at 03:33:07PM +0000, Andre Przywara wrote:
>> Currently the pre_init support is provided only for x86_64. Since
>> having 32-bit x86 supported as well is not far off, just add an
>> implementation using i386 assembly instructions and the respective
>> syscall ABI.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  Makefile     |  1 +
>>  x86/init32.S | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 51 insertions(+)
>>  create mode 100644 x86/init32.S
> 
> Did you manage to test this?

Yes, I think I tested it back when I originally wrote this, but I just
redid the testing using a 32-bit Ubuntu-Core i386 chroot (and deleting
the .lkvm/default directory before testing to make sure you don't get an
older version of init).
It works fine and apart from the size of the binary I can't tell a
difference between booting with or without the patch.

Cheers,
Andre.

Re: [PATCH 4/4] pre_init: add ARM implementations

[Will Deacon]

On Wed, Mar 02, 2016 at 11:38:34PM +0000, André Przywara wrote:
> On 02/03/16 03:00, Will Deacon wrote:
> > On Wed, Feb 24, 2016 at 03:33:08PM +0000, Andre Przywara wrote:
> >> The pre_init stub consists of two syscalls mouting the host's FS
> >> via 9pfs and then calling the actual init binary, which can now
> >> use normal dynamic linking.
> >> Based on the x86 code provide an ARM and ARM64 implementation of
> >> that. Beside removing the need for static linkage it reduces the
> >> size of the kvmtool binary by quite a lot (numbers for aarch64):
> >>
> >> -rwxr-xr-x 1 root root     9952 Nov 16 14:37 guest/init
> >> -rwxr-xr-x 1 root root      512 Nov 16 14:37 guest/pre_init
> >> -rwxr-xr-x 2 root root  1284704 Nov 16 14:37 lkvm
> >> vs. the old version:
> >> -rwxr-xr-x 1 root root   776024 Nov 16 14:38 guest/init
> >> -rwxr-xr-x 2 root root  2050112 Nov 16 14:38 lkvm
> >>
> >> Tested on Midway and Juno.
> > 
> > Hmm, I'm not super keen on switching behaviour like this on arm, where
> > it's not uncommon to build a static lkvm and transfer it to a remote
> > target and expect init to work.
> 
> So are you concerned about a fully static root file system on the host,
> which does not provide libc.so and/or ld-linux.so at all? Is that really
> a use case? I had the impression that people use a statically linked
> kvmtool to avoid dependencies like to libfdt.so.

Right. There are certainly environments where kvmtool is used that don't
have a dynanmic linker, and I'd really like lkvm-static to work there.

> > Perhaps we could only do this when building a dynamic executable?
> 
> This is of course an option as well.

I think it's the right thing to do.

Will