[RFC PATCH 0/2] support xen HVM direct kernel boot

[RFC PATCH 0/2] support xen HVM direct kernel boot

[Chunyan Liu]

Following previous discussion:
https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html

I spent lot of time to understand xen xc_hvm_build, hvmloader code,
and qemu pc_memory_init, xen_ram_init code, and wrote experimental
code to test. Now it's basically working based on seabios.

Talking about qemu linuxboot.bin/multiboot.bin first, they are
implemented to intercept boot process, originally by override int19,
later changed to provide BEV entry which will be called by seabios
if option rom is selected for booting. linuxboot.bin/multiboot.bin
will load kernel/initrd and jump to execute kernel directly.

In qemu load_linux() code, it first read out kernel/initrd file and
store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
will get the ADDRs and load kernel/initrd data), then add
linuxboot.bin/multiboot.bin to option rom with bootindex=0.

In seabios side, it could load and execute ROMs by itself, and according
to boot order, it will boot from the BEV entry on the option rom. That
will then be taken over by linuxboot.bin/multiboot.bin.

So, in theory, qemu load_linux() code almost does the work. I just tried
to reuse it for xen.

xen side patch: pass kernel/initrd/append parameters to qemu-dm
qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
    Different from pc_memory_init which does lots of ram alloc work
    and rom/bios loading work, for xen, we only need to init a basic
    fw_cfg device used by load_linux() to store ADDRs and
    linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
    after that, do real add option rom work to add
    linuxboot.bin/multiboot.bin to system option rom. Other things
    would be done by seabios smoothly.

In my testing, it's working based on seabios.
Rombios case is not working. But now that xen prefers qemu upstream,
seabios will be default, do we need to put effort to make rombios work?
And stubdom case, since I didn't make my guest started even without direct
kernel boot, I don't know if it works. And I still could not well
understand about the differences to stubdom and non stubdom this work
may cover. Any problems please point out.

Personally I'm not quite familiar with the details in this part of code,
so there might be things not considered, or there might be better method.
I'm very willing to hear suggestions to improve the work. Any comment or
feedback would be very appreciated! Thanks.
 
Chunyan Liu (2):
  xen: pass kernel initrd to qemu
  qemu: support xen hvm direct kernel boot

[RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Chunyan Liu]

[support xen HVM direct kernel boot]
xen side patch: support 'kernel', 'ramdisk', 'root', 'extra'
in HVM config file, parse config file, pass -kernel, -initrd
and -append parameters to qemu.

[config example]
kernel="/mnt/vmlinuz-3.0.13-0.27-default"
ramdisk="/mnt/initrd-3.0.13-0.27-default"
root="/dev/hda2"
extra="console=tty0 console=ttyS0"
disk=[ 'file:/mnt/images/bjz_04_sles11_sp2/disk0.raw,hda,w', ]

Signed-off-by: Chunyan Liu <cyliu@suse.com>
---
 tools/libxl/libxl_dm.c      | 20 ++++++++++++++++++++
 tools/libxl/libxl_types.idl |  3 +++
 tools/libxl/xl_cmdimpl.c    | 33 +++++++++++++++++++++++++++------
 3 files changed, 50 insertions(+), 6 deletions(-)

diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 51ab2bf..133bb56 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -196,6 +196,16 @@ static char ** libxl__build_device_model_args_old(libxl__gc *gc,
         int nr_set_cpus = 0;
         char *s;
 
+        if (b_info->u.hvm.kernel) {
+            flexarray_vappend(dm_args, "-kernel", b_info->u.hvm.kernel, NULL);
+        }
+        if (b_info->u.hvm.ramdisk) {
+            flexarray_vappend(dm_args, "-initrd", b_info->u.hvm.ramdisk, NULL);
+        }
+        if (b_info->u.hvm.cmdline) {
+            flexarray_vappend(dm_args, "-append", b_info->u.hvm.cmdline, NULL);
+        }
+
         if (b_info->u.hvm.serial) {
             flexarray_vappend(dm_args, "-serial", b_info->u.hvm.serial, NULL);
         }
@@ -479,6 +489,16 @@ static char ** libxl__build_device_model_args_new(libxl__gc *gc,
     if (b_info->type == LIBXL_DOMAIN_TYPE_HVM) {
         int ioemu_nics = 0;
 
+        if (b_info->u.hvm.kernel) {
+            flexarray_vappend(dm_args, "-kernel", b_info->u.hvm.kernel, NULL);
+        }
+        if (b_info->u.hvm.ramdisk) {
+            flexarray_vappend(dm_args, "-initrd", b_info->u.hvm.ramdisk, NULL);
+        }
+        if (b_info->u.hvm.cmdline) {
+            flexarray_vappend(dm_args, "-append", b_info->u.hvm.cmdline, NULL);
+        }
+
         if (b_info->u.hvm.serial) {
             flexarray_vappend(dm_args, "-serial", b_info->u.hvm.serial, NULL);
         }
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 52f1aa9..b8b973b 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -336,6 +336,9 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("event_channels",   uint32),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
+                                       ("kernel", string),
+                                       ("cmdline", string),
+                                       ("ramdisk", string),
                                        ("bios",             libxl_bios_type),
                                        ("pae",              libxl_defbool),
                                        ("apic",             libxl_defbool),
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 5195914..efd2474 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -743,6 +743,8 @@ static void parse_config_data(const char *config_source,
     int pci_permissive = 0;
     int pci_seize = 0;
     int i, e;
+    char *cmdline = NULL;
+    const char *root = NULL, *extra = "";
 
     libxl_domain_create_info *c_info = &d_config->c_info;
     libxl_domain_build_info *b_info = &d_config->b_info;
@@ -1007,9 +1009,31 @@ static void parse_config_data(const char *config_source,
 
     switch(b_info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
-        if (!xlu_cfg_get_string (config, "kernel", &buf, 0))
-            fprintf(stderr, "WARNING: ignoring \"kernel\" directive for HVM guest. "
-                    "Use \"firmware_override\" instead if you really want a non-default firmware\n");
+        if (!xlu_cfg_get_string (config, "kernel", &buf, 0)) {
+            if (strstr(buf, "hvmloader"))
+                fprintf(stderr, "WARNING: ignoring \"kernel\" directive for HVM guest. "
+                        "Use \"firmware_override\" instead if you really want a non-default firmware\n");
+            else
+                b_info->u.hvm.kernel = strdup(buf);
+        }
+
+        xlu_cfg_get_string (config, "root", &root, 0);
+        xlu_cfg_get_string (config, "extra", &extra, 0);
+
+        if (root) {
+            if (asprintf(&cmdline, "root=%s %s", root, extra) == -1)
+                cmdline = NULL;
+        } else {
+            cmdline = strdup(extra);
+        }
+
+        if ((root || extra) && !cmdline) {
+            fprintf(stderr, "Failed to allocate memory for cmdline\n");
+            exit(1);
+        }
+
+        b_info->u.hvm.cmdline = cmdline;
+        xlu_cfg_replace_string (config, "ramdisk", &b_info->u.hvm.ramdisk, 0);
 
         xlu_cfg_replace_string (config, "firmware_override",
                                 &b_info->u.hvm.firmware, 0);
@@ -1061,9 +1085,6 @@ static void parse_config_data(const char *config_source,
         break;
     case LIBXL_DOMAIN_TYPE_PV:
     {
-        char *cmdline = NULL;
-        const char *root = NULL, *extra = "";
-
         xlu_cfg_replace_string (config, "kernel", &b_info->u.pv.kernel, 0);
 
         xlu_cfg_get_string (config, "root", &root, 0);
-- 
1.8.4.5

[RFC PATCH 2/2] qemu: support xen hvm direct kernel boot

[Chunyan Liu]

[support xen HVM direct kernel boot]
qemu side patch: if -kernel exists, calls xen_load_linux(), which
will read kernel/initrd and add a linuxboot.bin or multiboot.bin
option rom. The linuxboot.bin/multiboot.bin will load kernel/initrd
and jump to execute kernel directly. It's working when xen uses
seabios.

Signed-off-by: Chunyan Liu <cyliu@suse.com>
---
 hw/i386/pc.c         | 29 +++++++++++++++++++++++++++++
 hw/i386/pc_piix.c    |  7 +++++++
 include/hw/i386/pc.h |  5 +++++
 3 files changed, 41 insertions(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index e6369d5..dcd5d48 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1187,6 +1187,35 @@ void pc_acpi_init(const char *default_dsdt)
     }
 }
 
+FWCfgState *xen_load_linux(const char *kernel_filename,
+                           const char *kernel_cmdline,
+                           const char *initrd_filename,
+                           ram_addr_t below_4g_mem_size,
+                           PcGuestInfo *guest_info)
+{
+    int i;
+    FWCfgState *fw_cfg;
+
+    assert(kernel_filename != NULL);
+
+    fw_cfg = fw_cfg_init(BIOS_CFG_IOPORT, BIOS_CFG_IOPORT + 1, 0, 0);
+    rom_set_fw(fw_cfg);
+
+    load_linux(fw_cfg, kernel_filename, initrd_filename, kernel_cmdline, below_4g_mem_size);
+    for (i = 0; i < nb_option_roms; i++) {
+        /* For xen, we only want to add the linuxboot.bin/multiboot.bin option rom.
+         * But in option_rom, there is still kvmvapic.bin. We don't want to add it.
+         */
+        if (strcmp(option_rom[i].name, "linuxboot.bin") &&
+            strcmp(option_rom[i].name, "multiboot.bin")) {
+            continue;
+        }
+        rom_add_option(option_rom[i].name, option_rom[i].bootindex);
+    }
+    guest_info->fw_cfg = fw_cfg;
+    return fw_cfg;
+}
+
 FWCfgState *pc_memory_init(MemoryRegion *system_memory,
                            const char *kernel_filename,
                            const char *kernel_cmdline,
diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index eaf3e61..14d4164 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -157,6 +157,13 @@ static void pc_init1(QEMUMachineInitArgs *args,
                        args->initrd_filename,
                        below_4g_mem_size, above_4g_mem_size,
                        rom_memory, &ram_memory, guest_info);
+    } else if (args->kernel_filename != NULL) {
+        /* For xen HVM direct kernel boot, load linux here */
+        fw_cfg = xen_load_linux(args->kernel_filename,
+                                args->kernel_cmdline,
+                                args->initrd_filename,
+                                below_4g_mem_size,
+                                guest_info);
     }
 
     gsi_state = g_malloc0(sizeof(*gsi_state));
diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h
index 32a7687..e472184 100644
--- a/include/hw/i386/pc.h
+++ b/include/hw/i386/pc.h
@@ -134,6 +134,11 @@ PcGuestInfo *pc_guest_info_init(ram_addr_t below_4g_mem_size,
 void pc_pci_as_mapping_init(Object *owner, MemoryRegion *system_memory,
                             MemoryRegion *pci_address_space);
 
+FWCfgState *xen_load_linux(const char *kernel_filename,
+                           const char *kernel_cmdline,
+                           const char *initrd_filename,
+                           ram_addr_t below_4g_mem_size,
+                           PcGuestInfo *guest_info);
 FWCfgState *pc_memory_init(MemoryRegion *system_memory,
                            const char *kernel_filename,
                            const char *kernel_cmdline,
-- 
1.8.4.5

Re: [RFC PATCH 2/2] qemu: support xen hvm direct kernel boot

[Paolo Bonzini]

Il 29/05/2014 05:23, Chunyan Liu ha scritto:
> [support xen HVM direct kernel boot]
> qemu side patch: if -kernel exists, calls xen_load_linux(), which
> will read kernel/initrd and add a linuxboot.bin or multiboot.bin
> option rom. The linuxboot.bin/multiboot.bin will load kernel/initrd
> and jump to execute kernel directly. It's working when xen uses
> seabios.
> 
> Signed-off-by: Chunyan Liu <cyliu@suse.com>
> ---
>  hw/i386/pc.c         | 29 +++++++++++++++++++++++++++++
>  hw/i386/pc_piix.c    |  7 +++++++
>  include/hw/i386/pc.h |  5 +++++
>  3 files changed, 41 insertions(+)
> 
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index e6369d5..dcd5d48 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -1187,6 +1187,35 @@ void pc_acpi_init(const char *default_dsdt)
>      }
>  }
>  
> +FWCfgState *xen_load_linux(const char *kernel_filename,
> +                           const char *kernel_cmdline,
> +                           const char *initrd_filename,
> +                           ram_addr_t below_4g_mem_size,
> +                           PcGuestInfo *guest_info)
> +{
> +    int i;
> +    FWCfgState *fw_cfg;
> +
> +    assert(kernel_filename != NULL);
> +
> +    fw_cfg = fw_cfg_init(BIOS_CFG_IOPORT, BIOS_CFG_IOPORT + 1, 0, 0);
> +    rom_set_fw(fw_cfg);
> +
> +    load_linux(fw_cfg, kernel_filename, initrd_filename, kernel_cmdline, below_4g_mem_size);
> +    for (i = 0; i < nb_option_roms; i++) {
> +        /* For xen, we only want to add the linuxboot.bin/multiboot.bin option rom.
> +         * But in option_rom, there is still kvmvapic.bin. We don't want to add it.
> +         */
> +        if (strcmp(option_rom[i].name, "linuxboot.bin") &&
> +            strcmp(option_rom[i].name, "multiboot.bin")) {
> +            continue;
> +        }
> +        rom_add_option(option_rom[i].name, option_rom[i].bootindex);
> +    }

Instead of this hack, you can use:

diff --git a/hw/i386/xen/xen_apic.c b/hw/i386/xen/xen_apic.c
index 63bb7f7..f5acd6a 100644
--- a/hw/i386/xen/xen_apic.c
+++ b/hw/i386/xen/xen_apic.c
@@ -40,6 +40,7 @@ static void xen_apic_realize(DeviceState *dev, Error **errp)
 {
     APICCommonState *s = APIC_COMMON(dev);
 
+    s->vapic_control = 0;
     memory_region_init_io(&s->io_memory, OBJECT(s), &xen_apic_io_ops, s,
                           "xen-apic-msi", APIC_SPACE_SIZE);
 

> +    guest_info->fw_cfg = fw_cfg;
> +    return fw_cfg;
> +}
> +
>  FWCfgState *pc_memory_init(MemoryRegion *system_memory,
>                             const char *kernel_filename,
>                             const char *kernel_cmdline,
> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
> index eaf3e61..14d4164 100644
> --- a/hw/i386/pc_piix.c
> +++ b/hw/i386/pc_piix.c
> @@ -157,6 +157,13 @@ static void pc_init1(QEMUMachineInitArgs *args,
>                         args->initrd_filename,
>                         below_4g_mem_size, above_4g_mem_size,
>                         rom_memory, &ram_memory, guest_info);
> +    } else if (args->kernel_filename != NULL) {
> +        /* For xen HVM direct kernel boot, load linux here */
> +        fw_cfg = xen_load_linux(args->kernel_filename,
> +                                args->kernel_cmdline,
> +                                args->initrd_filename,
> +                                below_4g_mem_size,
> +                                guest_info);
>      }
>  
>      gsi_state = g_malloc0(sizeof(*gsi_state));
> diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h
> index 32a7687..e472184 100644
> --- a/include/hw/i386/pc.h
> +++ b/include/hw/i386/pc.h
> @@ -134,6 +134,11 @@ PcGuestInfo *pc_guest_info_init(ram_addr_t below_4g_mem_size,
>  void pc_pci_as_mapping_init(Object *owner, MemoryRegion *system_memory,
>                              MemoryRegion *pci_address_space);
>  
> +FWCfgState *xen_load_linux(const char *kernel_filename,
> +                           const char *kernel_cmdline,
> +                           const char *initrd_filename,
> +                           ram_addr_t below_4g_mem_size,
> +                           PcGuestInfo *guest_info);
>  FWCfgState *pc_memory_init(MemoryRegion *system_memory,
>                             const char *kernel_filename,
>                             const char *kernel_cmdline,
> 

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Konrad Rzeszutek Wilk]

On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:
> Following previous discussion:
> https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html
> 
> I spent lot of time to understand xen xc_hvm_build, hvmloader code,
> and qemu pc_memory_init, xen_ram_init code, and wrote experimental
> code to test. Now it's basically working based on seabios.

Excellent!
> 
> Talking about qemu linuxboot.bin/multiboot.bin first, they are
> implemented to intercept boot process, originally by override int19,
> later changed to provide BEV entry which will be called by seabios
> if option rom is selected for booting. linuxboot.bin/multiboot.bin
> will load kernel/initrd and jump to execute kernel directly.
> 
> In qemu load_linux() code, it first read out kernel/initrd file and
> store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
> will get the ADDRs and load kernel/initrd data), then add
> linuxboot.bin/multiboot.bin to option rom with bootindex=0.
> 
> In seabios side, it could load and execute ROMs by itself, and according
> to boot order, it will boot from the BEV entry on the option rom. That
> will then be taken over by linuxboot.bin/multiboot.bin.
> 
> So, in theory, qemu load_linux() code almost does the work. I just tried
> to reuse it for xen.
> 
> xen side patch: pass kernel/initrd/append parameters to qemu-dm
> qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
>     Different from pc_memory_init which does lots of ram alloc work
>     and rom/bios loading work, for xen, we only need to init a basic
>     fw_cfg device used by load_linux() to store ADDRs and
>     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
>     after that, do real add option rom work to add
>     linuxboot.bin/multiboot.bin to system option rom. Other things
>     would be done by seabios smoothly.
> 
> In my testing, it's working based on seabios.
> Rombios case is not working. But now that xen prefers qemu upstream,
> seabios will be default, do we need to put effort to make rombios work?

No. But the code (libxl) should detect whether you are using the
traditional QEMU - which would be using ROM BIOS. If it detects that
it shouldn't allow you to pass the arguments.

You probably will also need to an docs/ change to document these
new parameters.
> And stubdom case, since I didn't make my guest started even without direct
> kernel boot, I don't know if it works. And I still could not well
> understand about the differences to stubdom and non stubdom this work
> may cover. Any problems please point out.

Hmm, you would have to somehow 'transfer' the images from one domain
(the control domain) to the stubdomain which would do the execution.

At least the multiboot.bin image.
> 
> Personally I'm not quite familiar with the details in this part of code,
> so there might be things not considered, or there might be better method.
> I'm very willing to hear suggestions to improve the work. Any comment or
> feedback would be very appreciated! Thanks.
>  
> Chunyan Liu (2):
>   xen: pass kernel initrd to qemu
>   qemu: support xen hvm direct kernel boot
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Stefano Stabellini]

On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote:
> On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:
> > Following previous discussion:
> > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html
> > 
> > I spent lot of time to understand xen xc_hvm_build, hvmloader code,
> > and qemu pc_memory_init, xen_ram_init code, and wrote experimental
> > code to test. Now it's basically working based on seabios.
> 
> Excellent!
> > 
> > Talking about qemu linuxboot.bin/multiboot.bin first, they are
> > implemented to intercept boot process, originally by override int19,
> > later changed to provide BEV entry which will be called by seabios
> > if option rom is selected for booting. linuxboot.bin/multiboot.bin
> > will load kernel/initrd and jump to execute kernel directly.
> > 
> > In qemu load_linux() code, it first read out kernel/initrd file and
> > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
> > will get the ADDRs and load kernel/initrd data), then add
> > linuxboot.bin/multiboot.bin to option rom with bootindex=0.
> > 
> > In seabios side, it could load and execute ROMs by itself, and according
> > to boot order, it will boot from the BEV entry on the option rom. That
> > will then be taken over by linuxboot.bin/multiboot.bin.
> > 
> > So, in theory, qemu load_linux() code almost does the work. I just tried
> > to reuse it for xen.
> > 
> > xen side patch: pass kernel/initrd/append parameters to qemu-dm
> > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
> >     Different from pc_memory_init which does lots of ram alloc work
> >     and rom/bios loading work, for xen, we only need to init a basic
> >     fw_cfg device used by load_linux() to store ADDRs and
> >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
> >     after that, do real add option rom work to add
> >     linuxboot.bin/multiboot.bin to system option rom. Other things
> >     would be done by seabios smoothly.
> > 
> > In my testing, it's working based on seabios.
> > Rombios case is not working. But now that xen prefers qemu upstream,
> > seabios will be default, do we need to put effort to make rombios work?
> 
> No. But the code (libxl) should detect whether you are using the
> traditional QEMU - which would be using ROM BIOS. If it detects that
> it shouldn't allow you to pass the arguments.
> 
> You probably will also need to an docs/ change to document these
> new parameters.
> > And stubdom case, since I didn't make my guest started even without direct
> > kernel boot, I don't know if it works. And I still could not well
> > understand about the differences to stubdom and non stubdom this work
> > may cover. Any problems please point out.
> 
> Hmm, you would have to somehow 'transfer' the images from one domain
> (the control domain) to the stubdomain which would do the execution.
> 
> At least the multiboot.bin image.

Yeah, unfortunately the approach of implemeting direct kernel boot for
HVM guests via QEMU wouldn't work for stubdomains.

If we want this to work with stubdoms, it would be best to reimplement
kernel and initrd loading in libxl/libxc.

I have to admit that given the extreme simplicity of this patch series,
I wouldn't be against merging it, even if it cannot be made to work with
stubdoms.  However like in the rombios case, libxl would need to
understand that this feature doesn't work with stubdoms and print an
appropriate message.


> > Personally I'm not quite familiar with the details in this part of code,
> > so there might be things not considered, or there might be better method.
> > I'm very willing to hear suggestions to improve the work. Any comment or
> > feedback would be very appreciated! Thanks.
> >  
> > Chunyan Liu (2):
> >   xen: pass kernel initrd to qemu
> >   qemu: support xen hvm direct kernel boot
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Stefano Stabellini]

On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote:
> On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:
> > Following previous discussion:
> > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html
> > 
> > I spent lot of time to understand xen xc_hvm_build, hvmloader code,
> > and qemu pc_memory_init, xen_ram_init code, and wrote experimental
> > code to test. Now it's basically working based on seabios.
> 
> Excellent!
> > 
> > Talking about qemu linuxboot.bin/multiboot.bin first, they are
> > implemented to intercept boot process, originally by override int19,
> > later changed to provide BEV entry which will be called by seabios
> > if option rom is selected for booting. linuxboot.bin/multiboot.bin
> > will load kernel/initrd and jump to execute kernel directly.
> > 
> > In qemu load_linux() code, it first read out kernel/initrd file and
> > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
> > will get the ADDRs and load kernel/initrd data), then add
> > linuxboot.bin/multiboot.bin to option rom with bootindex=0.
> > 
> > In seabios side, it could load and execute ROMs by itself, and according
> > to boot order, it will boot from the BEV entry on the option rom. That
> > will then be taken over by linuxboot.bin/multiboot.bin.
> > 
> > So, in theory, qemu load_linux() code almost does the work. I just tried
> > to reuse it for xen.
> > 
> > xen side patch: pass kernel/initrd/append parameters to qemu-dm
> > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
> >     Different from pc_memory_init which does lots of ram alloc work
> >     and rom/bios loading work, for xen, we only need to init a basic
> >     fw_cfg device used by load_linux() to store ADDRs and
> >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
> >     after that, do real add option rom work to add
> >     linuxboot.bin/multiboot.bin to system option rom. Other things
> >     would be done by seabios smoothly.
> > 
> > In my testing, it's working based on seabios.
> > Rombios case is not working. But now that xen prefers qemu upstream,
> > seabios will be default, do we need to put effort to make rombios work?
> 
> No. But the code (libxl) should detect whether you are using the
> traditional QEMU - which would be using ROM BIOS. If it detects that
> it shouldn't allow you to pass the arguments.
> 
> You probably will also need to an docs/ change to document these
> new parameters.
> > And stubdom case, since I didn't make my guest started even without direct
> > kernel boot, I don't know if it works. And I still could not well
> > understand about the differences to stubdom and non stubdom this work
> > may cover. Any problems please point out.
> 
> Hmm, you would have to somehow 'transfer' the images from one domain
> (the control domain) to the stubdomain which would do the execution.
> 
> At least the multiboot.bin image.

Yeah, unfortunately the approach of implemeting direct kernel boot for
HVM guests via QEMU wouldn't work for stubdomains.

If we want this to work with stubdoms, it would be best to reimplement
kernel and initrd loading in libxl/libxc.

I have to admit that given the extreme simplicity of this patch series,
I wouldn't be against merging it, even if it cannot be made to work with
stubdoms.  However like in the rombios case, libxl would need to
understand that this feature doesn't work with stubdoms and print an
appropriate message.


> > Personally I'm not quite familiar with the details in this part of code,
> > so there might be things not considered, or there might be better method.
> > I'm very willing to hear suggestions to improve the work. Any comment or
> > feedback would be very appreciated! Thanks.
> >  
> > Chunyan Liu (2):
> >   xen: pass kernel initrd to qemu
> >   qemu: support xen hvm direct kernel boot
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Fri, 2014-05-30 at 12:04 -0400, Konrad Rzeszutek Wilk wrote:
> On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:
> > Following previous discussion:
> > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html
> > 
> > I spent lot of time to understand xen xc_hvm_build, hvmloader code,
> > and qemu pc_memory_init, xen_ram_init code, and wrote experimental
> > code to test. Now it's basically working based on seabios.
> 
> Excellent!
> > 
> > Talking about qemu linuxboot.bin/multiboot.bin first, they are
> > implemented to intercept boot process, originally by override int19,
> > later changed to provide BEV entry which will be called by seabios
> > if option rom is selected for booting. linuxboot.bin/multiboot.bin
> > will load kernel/initrd and jump to execute kernel directly.
> > 
> > In qemu load_linux() code, it first read out kernel/initrd file and
> > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
> > will get the ADDRs and load kernel/initrd data), then add
> > linuxboot.bin/multiboot.bin to option rom with bootindex=0.
> > 
> > In seabios side, it could load and execute ROMs by itself, and according
> > to boot order, it will boot from the BEV entry on the option rom. That
> > will then be taken over by linuxboot.bin/multiboot.bin.
> > 
> > So, in theory, qemu load_linux() code almost does the work. I just tried
> > to reuse it for xen.
> > 
> > xen side patch: pass kernel/initrd/append parameters to qemu-dm
> > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
> >     Different from pc_memory_init which does lots of ram alloc work
> >     and rom/bios loading work, for xen, we only need to init a basic
> >     fw_cfg device used by load_linux() to store ADDRs and
> >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
> >     after that, do real add option rom work to add
> >     linuxboot.bin/multiboot.bin to system option rom. Other things
> >     would be done by seabios smoothly.
> > 
> > In my testing, it's working based on seabios.
> > Rombios case is not working. But now that xen prefers qemu upstream,
> > seabios will be default, do we need to put effort to make rombios work?
> 
> No. But the code (libxl) should detect whether you are using the
> traditional QEMU - which would be using ROM BIOS. If it detects that
> it shouldn't allow you to pass the arguments.
> 
> You probably will also need to an docs/ change to document these
> new parameters.

Yes, at the very least the manpages...

> > And stubdom case, since I didn't make my guest started even without direct
> > kernel boot, I don't know if it works. And I still could not well
> > understand about the differences to stubdom and non stubdom this work
> > may cover. Any problems please point out.
> 
> Hmm, you would have to somehow 'transfer' the images from one domain
> (the control domain) to the stubdomain which would do the execution.

Yes, which most likely involves preseeding the guest RAM with the files
at a discoverable location. Icky.

Ian.

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Fri, 2014-05-30 at 12:04 -0400, Konrad Rzeszutek Wilk wrote:
> On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:
> > Following previous discussion:
> > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html
> > 
> > I spent lot of time to understand xen xc_hvm_build, hvmloader code,
> > and qemu pc_memory_init, xen_ram_init code, and wrote experimental
> > code to test. Now it's basically working based on seabios.
> 
> Excellent!
> > 
> > Talking about qemu linuxboot.bin/multiboot.bin first, they are
> > implemented to intercept boot process, originally by override int19,
> > later changed to provide BEV entry which will be called by seabios
> > if option rom is selected for booting. linuxboot.bin/multiboot.bin
> > will load kernel/initrd and jump to execute kernel directly.
> > 
> > In qemu load_linux() code, it first read out kernel/initrd file and
> > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
> > will get the ADDRs and load kernel/initrd data), then add
> > linuxboot.bin/multiboot.bin to option rom with bootindex=0.
> > 
> > In seabios side, it could load and execute ROMs by itself, and according
> > to boot order, it will boot from the BEV entry on the option rom. That
> > will then be taken over by linuxboot.bin/multiboot.bin.
> > 
> > So, in theory, qemu load_linux() code almost does the work. I just tried
> > to reuse it for xen.
> > 
> > xen side patch: pass kernel/initrd/append parameters to qemu-dm
> > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
> >     Different from pc_memory_init which does lots of ram alloc work
> >     and rom/bios loading work, for xen, we only need to init a basic
> >     fw_cfg device used by load_linux() to store ADDRs and
> >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
> >     after that, do real add option rom work to add
> >     linuxboot.bin/multiboot.bin to system option rom. Other things
> >     would be done by seabios smoothly.
> > 
> > In my testing, it's working based on seabios.
> > Rombios case is not working. But now that xen prefers qemu upstream,
> > seabios will be default, do we need to put effort to make rombios work?
> 
> No. But the code (libxl) should detect whether you are using the
> traditional QEMU - which would be using ROM BIOS. If it detects that
> it shouldn't allow you to pass the arguments.
> 
> You probably will also need to an docs/ change to document these
> new parameters.

Yes, at the very least the manpages...

> > And stubdom case, since I didn't make my guest started even without direct
> > kernel boot, I don't know if it works. And I still could not well
> > understand about the differences to stubdom and non stubdom this work
> > may cover. Any problems please point out.
> 
> Hmm, you would have to somehow 'transfer' the images from one domain
> (the control domain) to the stubdomain which would do the execution.

Yes, which most likely involves preseeding the guest RAM with the files
at a discoverable location. Icky.

Ian.

Re: [Qemu-devel] [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Ian Campbell]

On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote:
> [support xen HVM direct kernel boot]

What is this?

> xen side patch: support 'kernel', 'ramdisk', 'root', 'extra'
> in HVM config file, parse config file, pass -kernel, -initrd
> and -append parameters to qemu.

Is this a completely new feature or is this adding parity with a xend
feature?

> [config example]
> kernel="/mnt/vmlinuz-3.0.13-0.27-default"
> ramdisk="/mnt/initrd-3.0.13-0.27-default"
> root="/dev/hda2"
> extra="console=tty0 console=ttyS0"
> disk=[ 'file:/mnt/images/bjz_04_sles11_sp2/disk0.raw,hda,w', ]
> 
> Signed-off-by: Chunyan Liu <cyliu@suse.com>
> ---
>  tools/libxl/libxl_dm.c      | 20 ++++++++++++++++++++
>  tools/libxl/libxl_types.idl |  3 +++
>  tools/libxl/xl_cmdimpl.c    | 33 +++++++++++++++++++++++++++------
>  3 files changed, 50 insertions(+), 6 deletions(-)
> 
> diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
> index 51ab2bf..133bb56 100644
> --- a/tools/libxl/libxl_dm.c
> +++ b/tools/libxl/libxl_dm.c
> @@ -196,6 +196,16 @@ static char ** libxl__build_device_model_args_old(libxl__gc *gc,

Does this work with old qemu+rombios?

I question whether we should be adding this sort of new feature here
anyway.

>          int nr_set_cpus = 0;
>          char *s;
>  
> +        if (b_info->u.hvm.kernel) {
> +            flexarray_vappend(dm_args, "-kernel", b_info->u.hvm.kernel, NULL);
> +        }

libxl style would be to omit the {} for a single line if.

> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index 52f1aa9..b8b973b 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -336,6 +336,9 @@ libxl_domain_build_info = Struct("domain_build_info",[
>      ("event_channels",   uint32),
>      ("u", KeyedUnion(None, libxl_domain_type, "type",
>                  [("hvm", Struct(None, [("firmware",         string),
> +                                       ("kernel", string),
> +                                       ("cmdline", string),
> +                                       ("ramdisk", string),

Alignment.

>                                         ("bios",             libxl_bios_type),
>                                         ("pae",              libxl_defbool),
>                                         ("apic",             libxl_defbool),
> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
> index 5195914..efd2474 100644
> --- a/tools/libxl/xl_cmdimpl.c
> +++ b/tools/libxl/xl_cmdimpl.c

> +        xlu_cfg_get_string (config, "root", &root, 0);
> +        xlu_cfg_get_string (config, "extra", &extra, 0);
> +
> +        if (root) {
> +            if (asprintf(&cmdline, "root=%s %s", root, extra) == -1)
> +                cmdline = NULL;
> +        } else {
> +            cmdline = strdup(extra);
> +        }
> +
> +        if ((root || extra) && !cmdline) {
> +            fprintf(stderr, "Failed to allocate memory for cmdline\n");
> +            exit(1);
> +        }

This all seems to be duplicating pv code, please refactor into a common
helper or something.

> +
> +        b_info->u.hvm.cmdline = cmdline;
> +        xlu_cfg_replace_string (config, "ramdisk", &b_info->u.hvm.ramdisk, 0);
>  
>          xlu_cfg_replace_string (config, "firmware_override",
>                                  &b_info->u.hvm.firmware, 0);
> @@ -1061,9 +1085,6 @@ static void parse_config_data(const char *config_source,
>          break;
>      case LIBXL_DOMAIN_TYPE_PV:
>      {
> -        char *cmdline = NULL;
> -        const char *root = NULL, *extra = "";
> -
>          xlu_cfg_replace_string (config, "kernel", &b_info->u.pv.kernel, 0);
>  
>          xlu_cfg_get_string (config, "root", &root, 0);

Re: [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Ian Campbell]

On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote:
> [support xen HVM direct kernel boot]

What is this?

> xen side patch: support 'kernel', 'ramdisk', 'root', 'extra'
> in HVM config file, parse config file, pass -kernel, -initrd
> and -append parameters to qemu.

Is this a completely new feature or is this adding parity with a xend
feature?

> [config example]
> kernel="/mnt/vmlinuz-3.0.13-0.27-default"
> ramdisk="/mnt/initrd-3.0.13-0.27-default"
> root="/dev/hda2"
> extra="console=tty0 console=ttyS0"
> disk=[ 'file:/mnt/images/bjz_04_sles11_sp2/disk0.raw,hda,w', ]
> 
> Signed-off-by: Chunyan Liu <cyliu@suse.com>
> ---
>  tools/libxl/libxl_dm.c      | 20 ++++++++++++++++++++
>  tools/libxl/libxl_types.idl |  3 +++
>  tools/libxl/xl_cmdimpl.c    | 33 +++++++++++++++++++++++++++------
>  3 files changed, 50 insertions(+), 6 deletions(-)
> 
> diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
> index 51ab2bf..133bb56 100644
> --- a/tools/libxl/libxl_dm.c
> +++ b/tools/libxl/libxl_dm.c
> @@ -196,6 +196,16 @@ static char ** libxl__build_device_model_args_old(libxl__gc *gc,

Does this work with old qemu+rombios?

I question whether we should be adding this sort of new feature here
anyway.

>          int nr_set_cpus = 0;
>          char *s;
>  
> +        if (b_info->u.hvm.kernel) {
> +            flexarray_vappend(dm_args, "-kernel", b_info->u.hvm.kernel, NULL);
> +        }

libxl style would be to omit the {} for a single line if.

> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index 52f1aa9..b8b973b 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -336,6 +336,9 @@ libxl_domain_build_info = Struct("domain_build_info",[
>      ("event_channels",   uint32),
>      ("u", KeyedUnion(None, libxl_domain_type, "type",
>                  [("hvm", Struct(None, [("firmware",         string),
> +                                       ("kernel", string),
> +                                       ("cmdline", string),
> +                                       ("ramdisk", string),

Alignment.

>                                         ("bios",             libxl_bios_type),
>                                         ("pae",              libxl_defbool),
>                                         ("apic",             libxl_defbool),
> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
> index 5195914..efd2474 100644
> --- a/tools/libxl/xl_cmdimpl.c
> +++ b/tools/libxl/xl_cmdimpl.c

> +        xlu_cfg_get_string (config, "root", &root, 0);
> +        xlu_cfg_get_string (config, "extra", &extra, 0);
> +
> +        if (root) {
> +            if (asprintf(&cmdline, "root=%s %s", root, extra) == -1)
> +                cmdline = NULL;
> +        } else {
> +            cmdline = strdup(extra);
> +        }
> +
> +        if ((root || extra) && !cmdline) {
> +            fprintf(stderr, "Failed to allocate memory for cmdline\n");
> +            exit(1);
> +        }

This all seems to be duplicating pv code, please refactor into a common
helper or something.

> +
> +        b_info->u.hvm.cmdline = cmdline;
> +        xlu_cfg_replace_string (config, "ramdisk", &b_info->u.hvm.ramdisk, 0);
>  
>          xlu_cfg_replace_string (config, "firmware_override",
>                                  &b_info->u.hvm.firmware, 0);
> @@ -1061,9 +1085,6 @@ static void parse_config_data(const char *config_source,
>          break;
>      case LIBXL_DOMAIN_TYPE_PV:
>      {
> -        char *cmdline = NULL;
> -        const char *root = NULL, *extra = "";
> -
>          xlu_cfg_replace_string (config, "kernel", &b_info->u.pv.kernel, 0);
>  
>          xlu_cfg_get_string (config, "root", &root, 0);

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Konrad Rzeszutek Wilk]

On Mon, Jun 02, 2014 at 04:19:31PM +0100, Ian Campbell wrote:
> On Fri, 2014-05-30 at 12:04 -0400, Konrad Rzeszutek Wilk wrote:
> > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:
> > > Following previous discussion:
> > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html
> > > 
> > > I spent lot of time to understand xen xc_hvm_build, hvmloader code,
> > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental
> > > code to test. Now it's basically working based on seabios.
> > 
> > Excellent!
> > > 
> > > Talking about qemu linuxboot.bin/multiboot.bin first, they are
> > > implemented to intercept boot process, originally by override int19,
> > > later changed to provide BEV entry which will be called by seabios
> > > if option rom is selected for booting. linuxboot.bin/multiboot.bin
> > > will load kernel/initrd and jump to execute kernel directly.
> > > 
> > > In qemu load_linux() code, it first read out kernel/initrd file and
> > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
> > > will get the ADDRs and load kernel/initrd data), then add
> > > linuxboot.bin/multiboot.bin to option rom with bootindex=0.
> > > 
> > > In seabios side, it could load and execute ROMs by itself, and according
> > > to boot order, it will boot from the BEV entry on the option rom. That
> > > will then be taken over by linuxboot.bin/multiboot.bin.
> > > 
> > > So, in theory, qemu load_linux() code almost does the work. I just tried
> > > to reuse it for xen.
> > > 
> > > xen side patch: pass kernel/initrd/append parameters to qemu-dm
> > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
> > >     Different from pc_memory_init which does lots of ram alloc work
> > >     and rom/bios loading work, for xen, we only need to init a basic
> > >     fw_cfg device used by load_linux() to store ADDRs and
> > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
> > >     after that, do real add option rom work to add
> > >     linuxboot.bin/multiboot.bin to system option rom. Other things
> > >     would be done by seabios smoothly.
> > > 
> > > In my testing, it's working based on seabios.
> > > Rombios case is not working. But now that xen prefers qemu upstream,
> > > seabios will be default, do we need to put effort to make rombios work?
> > 
> > No. But the code (libxl) should detect whether you are using the
> > traditional QEMU - which would be using ROM BIOS. If it detects that
> > it shouldn't allow you to pass the arguments.
> > 
> > You probably will also need to an docs/ change to document these
> > new parameters.
> 
> Yes, at the very least the manpages...
> 
> > > And stubdom case, since I didn't make my guest started even without direct
> > > kernel boot, I don't know if it works. And I still could not well
> > > understand about the differences to stubdom and non stubdom this work
> > > may cover. Any problems please point out.
> > 
> > Hmm, you would have to somehow 'transfer' the images from one domain
> > (the control domain) to the stubdomain which would do the execution.
> 
> Yes, which most likely involves preseeding the guest RAM with the files
> at a discoverable location. Icky.

Or some form of FTP over XenBus :-)
> 
> Ian.
> 

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Konrad Rzeszutek Wilk]

On Mon, Jun 02, 2014 at 04:19:31PM +0100, Ian Campbell wrote:
> On Fri, 2014-05-30 at 12:04 -0400, Konrad Rzeszutek Wilk wrote:
> > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:
> > > Following previous discussion:
> > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html
> > > 
> > > I spent lot of time to understand xen xc_hvm_build, hvmloader code,
> > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental
> > > code to test. Now it's basically working based on seabios.
> > 
> > Excellent!
> > > 
> > > Talking about qemu linuxboot.bin/multiboot.bin first, they are
> > > implemented to intercept boot process, originally by override int19,
> > > later changed to provide BEV entry which will be called by seabios
> > > if option rom is selected for booting. linuxboot.bin/multiboot.bin
> > > will load kernel/initrd and jump to execute kernel directly.
> > > 
> > > In qemu load_linux() code, it first read out kernel/initrd file and
> > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin
> > > will get the ADDRs and load kernel/initrd data), then add
> > > linuxboot.bin/multiboot.bin to option rom with bootindex=0.
> > > 
> > > In seabios side, it could load and execute ROMs by itself, and according
> > > to boot order, it will boot from the BEV entry on the option rom. That
> > > will then be taken over by linuxboot.bin/multiboot.bin.
> > > 
> > > So, in theory, qemu load_linux() code almost does the work. I just tried
> > > to reuse it for xen.
> > > 
> > > xen side patch: pass kernel/initrd/append parameters to qemu-dm
> > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.
> > >     Different from pc_memory_init which does lots of ram alloc work
> > >     and rom/bios loading work, for xen, we only need to init a basic
> > >     fw_cfg device used by load_linux() to store ADDRs and
> > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),
> > >     after that, do real add option rom work to add
> > >     linuxboot.bin/multiboot.bin to system option rom. Other things
> > >     would be done by seabios smoothly.
> > > 
> > > In my testing, it's working based on seabios.
> > > Rombios case is not working. But now that xen prefers qemu upstream,
> > > seabios will be default, do we need to put effort to make rombios work?
> > 
> > No. But the code (libxl) should detect whether you are using the
> > traditional QEMU - which would be using ROM BIOS. If it detects that
> > it shouldn't allow you to pass the arguments.
> > 
> > You probably will also need to an docs/ change to document these
> > new parameters.
> 
> Yes, at the very least the manpages...
> 
> > > And stubdom case, since I didn't make my guest started even without direct
> > > kernel boot, I don't know if it works. And I still could not well
> > > understand about the differences to stubdom and non stubdom this work
> > > may cover. Any problems please point out.
> > 
> > Hmm, you would have to somehow 'transfer' the images from one domain
> > (the control domain) to the stubdomain which would do the execution.
> 
> Yes, which most likely involves preseeding the guest RAM with the files
> at a discoverable location. Icky.

Or some form of FTP over XenBus :-)
> 
> Ian.
> 

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/3/2014 at 12:58 AM, in message
<20140602165836.GD5127@phenom.dumpdata.com>, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote: 
> On Mon, Jun 02, 2014 at 04:19:31PM +0100, Ian Campbell wrote: 
> > On Fri, 2014-05-30 at 12:04 -0400, Konrad Rzeszutek Wilk wrote: 
> > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > > Following previous discussion: 
> > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > > >  
> > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > > code to test. Now it's basically working based on seabios. 
> > >  
> > > Excellent! 
> > > >  
> > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > > implemented to intercept boot process, originally by override int19, 
> > > > later changed to provide BEV entry which will be called by seabios 
> > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > > will load kernel/initrd and jump to execute kernel directly. 
> > > >  
> > > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > > will get the ADDRs and load kernel/initrd data), then add 
> > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > > >  
> > > > In seabios side, it could load and execute ROMs by itself, and according 
> > > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > > >  
> > > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > > to reuse it for xen. 
> > > >  
> > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > > >     Different from pc_memory_init which does lots of ram alloc work 
> > > >     and rom/bios loading work, for xen, we only need to init a basic 
> > > >     fw_cfg device used by load_linux() to store ADDRs and 
> > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > > >     after that, do real add option rom work to add 
> > > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > > >     would be done by seabios smoothly. 
> > > >  
> > > > In my testing, it's working based on seabios. 
> > > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > > seabios will be default, do we need to put effort to make rombios work? 
> > >  
> > > No. But the code (libxl) should detect whether you are using the 
> > > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > > it shouldn't allow you to pass the arguments. 
> > >  
> > > You probably will also need to an docs/ change to document these 
> > > new parameters. 
> >  
> > Yes, at the very least the manpages... 
> >  
> > > > And stubdom case, since I didn't make my guest started even without  
> direct 
> > > > kernel boot, I don't know if it works. And I still could not well 
> > > > understand about the differences to stubdom and non stubdom this work 
> > > > may cover. Any problems please point out. 
> > >  
> > > Hmm, you would have to somehow 'transfer' the images from one domain 
> > > (the control domain) to the stubdomain which would do the execution. 
> >  
> > Yes, which most likely involves preseeding the guest RAM with the files 
> > at a discoverable location. Icky. 
>  
> Or some form of FTP over XenBus :-) 

Thanks. Understand. All files including 'kernel', 'initrd' and 'linuxboot.bin' could not be
accessed directly in stubdom-dm. Have to find a way to let qemu discover them.
I'll look at and see how to properly handle it. 

Now I wonder how does it handle device? (A backend file or partition is also a place
on host.) Stubdom uses qemu-xen-traditional, maybe there is some special handling?
Will have a look.

- Chunyan
> >  
> > Ian. 
> >  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/3/2014 at 12:58 AM, in message
<20140602165836.GD5127@phenom.dumpdata.com>, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote: 
> On Mon, Jun 02, 2014 at 04:19:31PM +0100, Ian Campbell wrote: 
> > On Fri, 2014-05-30 at 12:04 -0400, Konrad Rzeszutek Wilk wrote: 
> > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > > Following previous discussion: 
> > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > > >  
> > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > > code to test. Now it's basically working based on seabios. 
> > >  
> > > Excellent! 
> > > >  
> > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > > implemented to intercept boot process, originally by override int19, 
> > > > later changed to provide BEV entry which will be called by seabios 
> > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > > will load kernel/initrd and jump to execute kernel directly. 
> > > >  
> > > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > > will get the ADDRs and load kernel/initrd data), then add 
> > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > > >  
> > > > In seabios side, it could load and execute ROMs by itself, and according 
> > > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > > >  
> > > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > > to reuse it for xen. 
> > > >  
> > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > > >     Different from pc_memory_init which does lots of ram alloc work 
> > > >     and rom/bios loading work, for xen, we only need to init a basic 
> > > >     fw_cfg device used by load_linux() to store ADDRs and 
> > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > > >     after that, do real add option rom work to add 
> > > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > > >     would be done by seabios smoothly. 
> > > >  
> > > > In my testing, it's working based on seabios. 
> > > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > > seabios will be default, do we need to put effort to make rombios work? 
> > >  
> > > No. But the code (libxl) should detect whether you are using the 
> > > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > > it shouldn't allow you to pass the arguments. 
> > >  
> > > You probably will also need to an docs/ change to document these 
> > > new parameters. 
> >  
> > Yes, at the very least the manpages... 
> >  
> > > > And stubdom case, since I didn't make my guest started even without  
> direct 
> > > > kernel boot, I don't know if it works. And I still could not well 
> > > > understand about the differences to stubdom and non stubdom this work 
> > > > may cover. Any problems please point out. 
> > >  
> > > Hmm, you would have to somehow 'transfer' the images from one domain 
> > > (the control domain) to the stubdomain which would do the execution. 
> >  
> > Yes, which most likely involves preseeding the guest RAM with the files 
> > at a discoverable location. Icky. 
>  
> Or some form of FTP over XenBus :-) 

Thanks. Understand. All files including 'kernel', 'initrd' and 'linuxboot.bin' could not be
accessed directly in stubdom-dm. Have to find a way to let qemu discover them.
I'll look at and see how to properly handle it. 

Now I wonder how does it handle device? (A backend file or partition is also a place
on host.) Stubdom uses qemu-xen-traditional, maybe there is some special handling?
Will have a look.

- Chunyan
> >  
> > Ian. 
> >  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Chun Yan Liu]

>>> On 6/2/2014 at 11:24 PM, in message
<1401722652.30097.9.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote: 
> > [support xen HVM direct kernel boot] 
>  
> What is this? 

Just to indicate this is part of work to support xen HVM direct kernel boot.
To make it work, there is another patch to qemu.

>  
> > xen side patch: support 'kernel', 'ramdisk', 'root', 'extra' 
> > in HVM config file, parse config file, pass -kernel, -initrd 
> > and -append parameters to qemu. 
>  
> Is this a completely new feature or is this adding parity with a xend 
> feature? 

I think it's new. Xend doesn't support HVM direct kernel boot.
But considering stubdom-dm, such interface might not work. I'll rework on
that.

>  
> > [config example] 
> > kernel="/mnt/vmlinuz-3.0.13-0.27-default" 
> > ramdisk="/mnt/initrd-3.0.13-0.27-default" 
> > root="/dev/hda2" 
> > extra="console=tty0 console=ttyS0" 
> > disk=[ 'file:/mnt/images/bjz_04_sles11_sp2/disk0.raw,hda,w', ] 
> >  
> > Signed-off-by: Chunyan Liu <cyliu@suse.com> 
> > --- 
> >  tools/libxl/libxl_dm.c      | 20 ++++++++++++++++++++ 
> >  tools/libxl/libxl_types.idl |  3 +++ 
> >  tools/libxl/xl_cmdimpl.c    | 33 +++++++++++++++++++++++++++------ 
> >  3 files changed, 50 insertions(+), 6 deletions(-) 
> >  
> > diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c 
> > index 51ab2bf..133bb56 100644 
> > --- a/tools/libxl/libxl_dm.c 
> > +++ b/tools/libxl/libxl_dm.c 
> > @@ -196,6 +196,16 @@ static char **  
> libxl__build_device_model_args_old(libxl__gc *gc, 
>  
> Does this work with old qemu+rombios? 

Oh, no. I'll remove this piece.

>  
> I question whether we should be adding this sort of new feature here 
> anyway. 
>  
> >          int nr_set_cpus = 0; 
> >          char *s; 
> >   
> > +        if (b_info->u.hvm.kernel) { 
> > +            flexarray_vappend(dm_args, "-kernel", b_info->u.hvm.kernel,  
> NULL); 
> > +        } 
>  
> libxl style would be to omit the {} for a single line if. 
>  
> > diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl 
> > index 52f1aa9..b8b973b 100644 
> > --- a/tools/libxl/libxl_types.idl 
> > +++ b/tools/libxl/libxl_types.idl 
> > @@ -336,6 +336,9 @@ libxl_domain_build_info = Struct("domain_build_info",[ 
> >      ("event_channels",   uint32), 
> >      ("u", KeyedUnion(None, libxl_domain_type, "type", 
> >                  [("hvm", Struct(None, [("firmware",         string), 
> > +                                       ("kernel", string), 
> > +                                       ("cmdline", string), 
> > +                                       ("ramdisk", string), 
>  
> Alignment. 
>  
> >                                         ("bios",              
> libxl_bios_type), 
> >                                         ("pae",               
> libxl_defbool), 
> >                                         ("apic",              
> libxl_defbool), 
> > diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c 
> > index 5195914..efd2474 100644 
> > --- a/tools/libxl/xl_cmdimpl.c 
> > +++ b/tools/libxl/xl_cmdimpl.c 
>  
> > +        xlu_cfg_get_string (config, "root", &root, 0); 
> > +        xlu_cfg_get_string (config, "extra", &extra, 0); 
> > + 
> > +        if (root) { 
> > +            if (asprintf(&cmdline, "root=%s %s", root, extra) == -1) 
> > +                cmdline = NULL; 
> > +        } else { 
> > +            cmdline = strdup(extra); 
> > +        } 
> > + 
> > +        if ((root || extra) && !cmdline) { 
> > +            fprintf(stderr, "Failed to allocate memory for cmdline\n"); 
> > +            exit(1); 
> > +        } 
>  
> This all seems to be duplicating pv code, please refactor into a common 
> helper or something. 
>  
> > + 
> > +        b_info->u.hvm.cmdline = cmdline; 
> > +        xlu_cfg_replace_string (config, "ramdisk", &b_info->u.hvm.ramdisk,  
> 0); 
> >   
> >          xlu_cfg_replace_string (config, "firmware_override", 
> >                                  &b_info->u.hvm.firmware, 0); 
> > @@ -1061,9 +1085,6 @@ static void parse_config_data(const char  
> *config_source, 
> >          break; 
> >      case LIBXL_DOMAIN_TYPE_PV: 
> >      { 
> > -        char *cmdline = NULL; 
> > -        const char *root = NULL, *extra = ""; 
> > - 
> >          xlu_cfg_replace_string (config, "kernel", &b_info->u.pv.kernel, 0); 
> >   
> >          xlu_cfg_get_string (config, "root", &root, 0); 
>  
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Xen-devel] [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Chun Yan Liu]

>>> On 6/2/2014 at 11:24 PM, in message
<1401722652.30097.9.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote: 
> > [support xen HVM direct kernel boot] 
>  
> What is this? 

Just to indicate this is part of work to support xen HVM direct kernel boot.
To make it work, there is another patch to qemu.

>  
> > xen side patch: support 'kernel', 'ramdisk', 'root', 'extra' 
> > in HVM config file, parse config file, pass -kernel, -initrd 
> > and -append parameters to qemu. 
>  
> Is this a completely new feature or is this adding parity with a xend 
> feature? 

I think it's new. Xend doesn't support HVM direct kernel boot.
But considering stubdom-dm, such interface might not work. I'll rework on
that.

>  
> > [config example] 
> > kernel="/mnt/vmlinuz-3.0.13-0.27-default" 
> > ramdisk="/mnt/initrd-3.0.13-0.27-default" 
> > root="/dev/hda2" 
> > extra="console=tty0 console=ttyS0" 
> > disk=[ 'file:/mnt/images/bjz_04_sles11_sp2/disk0.raw,hda,w', ] 
> >  
> > Signed-off-by: Chunyan Liu <cyliu@suse.com> 
> > --- 
> >  tools/libxl/libxl_dm.c      | 20 ++++++++++++++++++++ 
> >  tools/libxl/libxl_types.idl |  3 +++ 
> >  tools/libxl/xl_cmdimpl.c    | 33 +++++++++++++++++++++++++++------ 
> >  3 files changed, 50 insertions(+), 6 deletions(-) 
> >  
> > diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c 
> > index 51ab2bf..133bb56 100644 
> > --- a/tools/libxl/libxl_dm.c 
> > +++ b/tools/libxl/libxl_dm.c 
> > @@ -196,6 +196,16 @@ static char **  
> libxl__build_device_model_args_old(libxl__gc *gc, 
>  
> Does this work with old qemu+rombios? 

Oh, no. I'll remove this piece.

>  
> I question whether we should be adding this sort of new feature here 
> anyway. 
>  
> >          int nr_set_cpus = 0; 
> >          char *s; 
> >   
> > +        if (b_info->u.hvm.kernel) { 
> > +            flexarray_vappend(dm_args, "-kernel", b_info->u.hvm.kernel,  
> NULL); 
> > +        } 
>  
> libxl style would be to omit the {} for a single line if. 
>  
> > diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl 
> > index 52f1aa9..b8b973b 100644 
> > --- a/tools/libxl/libxl_types.idl 
> > +++ b/tools/libxl/libxl_types.idl 
> > @@ -336,6 +336,9 @@ libxl_domain_build_info = Struct("domain_build_info",[ 
> >      ("event_channels",   uint32), 
> >      ("u", KeyedUnion(None, libxl_domain_type, "type", 
> >                  [("hvm", Struct(None, [("firmware",         string), 
> > +                                       ("kernel", string), 
> > +                                       ("cmdline", string), 
> > +                                       ("ramdisk", string), 
>  
> Alignment. 
>  
> >                                         ("bios",              
> libxl_bios_type), 
> >                                         ("pae",               
> libxl_defbool), 
> >                                         ("apic",              
> libxl_defbool), 
> > diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c 
> > index 5195914..efd2474 100644 
> > --- a/tools/libxl/xl_cmdimpl.c 
> > +++ b/tools/libxl/xl_cmdimpl.c 
>  
> > +        xlu_cfg_get_string (config, "root", &root, 0); 
> > +        xlu_cfg_get_string (config, "extra", &extra, 0); 
> > + 
> > +        if (root) { 
> > +            if (asprintf(&cmdline, "root=%s %s", root, extra) == -1) 
> > +                cmdline = NULL; 
> > +        } else { 
> > +            cmdline = strdup(extra); 
> > +        } 
> > + 
> > +        if ((root || extra) && !cmdline) { 
> > +            fprintf(stderr, "Failed to allocate memory for cmdline\n"); 
> > +            exit(1); 
> > +        } 
>  
> This all seems to be duplicating pv code, please refactor into a common 
> helper or something. 
>  
> > + 
> > +        b_info->u.hvm.cmdline = cmdline; 
> > +        xlu_cfg_replace_string (config, "ramdisk", &b_info->u.hvm.ramdisk,  
> 0); 
> >   
> >          xlu_cfg_replace_string (config, "firmware_override", 
> >                                  &b_info->u.hvm.firmware, 0); 
> > @@ -1061,9 +1085,6 @@ static void parse_config_data(const char  
> *config_source, 
> >          break; 
> >      case LIBXL_DOMAIN_TYPE_PV: 
> >      { 
> > -        char *cmdline = NULL; 
> > -        const char *root = NULL, *extra = ""; 
> > - 
> >          xlu_cfg_replace_string (config, "kernel", &b_info->u.pv.kernel, 0); 
> >   
> >          xlu_cfg_get_string (config, "root", &root, 0); 
>  
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 2/2] qemu: support xen hvm direct kernel boot

[Chun Yan Liu]

>>> On 5/29/2014 at 04:23 PM, in message <5386EE82.8040208@redhat.com>, Paolo
Bonzini <pbonzini@redhat.com> wrote: 
> Il 29/05/2014 05:23, Chunyan Liu ha scritto: 
> > [support xen HVM direct kernel boot] 
> > qemu side patch: if -kernel exists, calls xen_load_linux(), which 
> > will read kernel/initrd and add a linuxboot.bin or multiboot.bin 
> > option rom. The linuxboot.bin/multiboot.bin will load kernel/initrd 
> > and jump to execute kernel directly. It's working when xen uses 
> > seabios. 
> >  
> > Signed-off-by: Chunyan Liu <cyliu@suse.com> 
> > --- 
> >  hw/i386/pc.c         | 29 +++++++++++++++++++++++++++++ 
> >  hw/i386/pc_piix.c    |  7 +++++++ 
> >  include/hw/i386/pc.h |  5 +++++ 
> >  3 files changed, 41 insertions(+) 
> >  
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c 
> > index e6369d5..dcd5d48 100644 
> > --- a/hw/i386/pc.c 
> > +++ b/hw/i386/pc.c 
> > @@ -1187,6 +1187,35 @@ void pc_acpi_init(const char *default_dsdt) 
> >      } 
> >  } 
> >   
> > +FWCfgState *xen_load_linux(const char *kernel_filename, 
> > +                           const char *kernel_cmdline, 
> > +                           const char *initrd_filename, 
> > +                           ram_addr_t below_4g_mem_size, 
> > +                           PcGuestInfo *guest_info) 
> > +{ 
> > +    int i; 
> > +    FWCfgState *fw_cfg; 
> > + 
> > +    assert(kernel_filename != NULL); 
> > + 
> > +    fw_cfg = fw_cfg_init(BIOS_CFG_IOPORT, BIOS_CFG_IOPORT + 1, 0, 0); 
> > +    rom_set_fw(fw_cfg); 
> > + 
> > +    load_linux(fw_cfg, kernel_filename, initrd_filename, kernel_cmdline,  
> below_4g_mem_size); 
> > +    for (i = 0; i < nb_option_roms; i++) { 
> > +        /* For xen, we only want to add the linuxboot.bin/multiboot.bin  
> option rom. 
> > +         * But in option_rom, there is still kvmvapic.bin. We don't want  
> to add it. 
> > +         */ 
> > +        if (strcmp(option_rom[i].name, "linuxboot.bin") && 
> > +            strcmp(option_rom[i].name, "multiboot.bin")) { 
> > +            continue; 
> > +        } 
> > +        rom_add_option(option_rom[i].name, option_rom[i].bootindex); 
> > +    } 
>  
> Instead of this hack, you can use: 
>  
> diff --git a/hw/i386/xen/xen_apic.c b/hw/i386/xen/xen_apic.c 
> index 63bb7f7..f5acd6a 100644 
> --- a/hw/i386/xen/xen_apic.c 
> +++ b/hw/i386/xen/xen_apic.c 
> @@ -40,6 +40,7 @@ static void xen_apic_realize(DeviceState *dev, Error  
> **errp) 
>  { 
>      APICCommonState *s = APIC_COMMON(dev); 
>   
> +    s->vapic_control = 0; 

Thanks. It works.

>      memory_region_init_io(&s->io_memory, OBJECT(s), &xen_apic_io_ops, s, 
>                            "xen-apic-msi", APIC_SPACE_SIZE); 
>   
>  
> > +    guest_info->fw_cfg = fw_cfg; 
> > +    return fw_cfg; 
> > +} 
> > + 
> >  FWCfgState *pc_memory_init(MemoryRegion *system_memory, 
> >                             const char *kernel_filename, 
> >                             const char *kernel_cmdline, 
> > diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c 
> > index eaf3e61..14d4164 100644 
> > --- a/hw/i386/pc_piix.c 
> > +++ b/hw/i386/pc_piix.c 
> > @@ -157,6 +157,13 @@ static void pc_init1(QEMUMachineInitArgs *args, 
> >                         args->initrd_filename, 
> >                         below_4g_mem_size, above_4g_mem_size, 
> >                         rom_memory, &ram_memory, guest_info); 
> > +    } else if (args->kernel_filename != NULL) { 
> > +        /* For xen HVM direct kernel boot, load linux here */ 
> > +        fw_cfg = xen_load_linux(args->kernel_filename, 
> > +                                args->kernel_cmdline, 
> > +                                args->initrd_filename, 
> > +                                below_4g_mem_size, 
> > +                                guest_info); 
> >      } 
> >   
> >      gsi_state = g_malloc0(sizeof(*gsi_state)); 
> > diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h 
> > index 32a7687..e472184 100644 
> > --- a/include/hw/i386/pc.h 
> > +++ b/include/hw/i386/pc.h 
> > @@ -134,6 +134,11 @@ PcGuestInfo *pc_guest_info_init(ram_addr_t  
> below_4g_mem_size, 
> >  void pc_pci_as_mapping_init(Object *owner, MemoryRegion *system_memory, 
> >                              MemoryRegion *pci_address_space); 
> >   
> > +FWCfgState *xen_load_linux(const char *kernel_filename, 
> > +                           const char *kernel_cmdline, 
> > +                           const char *initrd_filename, 
> > +                           ram_addr_t below_4g_mem_size, 
> > +                           PcGuestInfo *guest_info); 
> >  FWCfgState *pc_memory_init(MemoryRegion *system_memory, 
> >                             const char *kernel_filename, 
> >                             const char *kernel_cmdline, 
> >  
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Xen-devel] [RFC PATCH 2/2] qemu: support xen hvm direct kernel boot

[Chun Yan Liu]

>>> On 5/29/2014 at 04:23 PM, in message <5386EE82.8040208@redhat.com>, Paolo
Bonzini <pbonzini@redhat.com> wrote: 
> Il 29/05/2014 05:23, Chunyan Liu ha scritto: 
> > [support xen HVM direct kernel boot] 
> > qemu side patch: if -kernel exists, calls xen_load_linux(), which 
> > will read kernel/initrd and add a linuxboot.bin or multiboot.bin 
> > option rom. The linuxboot.bin/multiboot.bin will load kernel/initrd 
> > and jump to execute kernel directly. It's working when xen uses 
> > seabios. 
> >  
> > Signed-off-by: Chunyan Liu <cyliu@suse.com> 
> > --- 
> >  hw/i386/pc.c         | 29 +++++++++++++++++++++++++++++ 
> >  hw/i386/pc_piix.c    |  7 +++++++ 
> >  include/hw/i386/pc.h |  5 +++++ 
> >  3 files changed, 41 insertions(+) 
> >  
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c 
> > index e6369d5..dcd5d48 100644 
> > --- a/hw/i386/pc.c 
> > +++ b/hw/i386/pc.c 
> > @@ -1187,6 +1187,35 @@ void pc_acpi_init(const char *default_dsdt) 
> >      } 
> >  } 
> >   
> > +FWCfgState *xen_load_linux(const char *kernel_filename, 
> > +                           const char *kernel_cmdline, 
> > +                           const char *initrd_filename, 
> > +                           ram_addr_t below_4g_mem_size, 
> > +                           PcGuestInfo *guest_info) 
> > +{ 
> > +    int i; 
> > +    FWCfgState *fw_cfg; 
> > + 
> > +    assert(kernel_filename != NULL); 
> > + 
> > +    fw_cfg = fw_cfg_init(BIOS_CFG_IOPORT, BIOS_CFG_IOPORT + 1, 0, 0); 
> > +    rom_set_fw(fw_cfg); 
> > + 
> > +    load_linux(fw_cfg, kernel_filename, initrd_filename, kernel_cmdline,  
> below_4g_mem_size); 
> > +    for (i = 0; i < nb_option_roms; i++) { 
> > +        /* For xen, we only want to add the linuxboot.bin/multiboot.bin  
> option rom. 
> > +         * But in option_rom, there is still kvmvapic.bin. We don't want  
> to add it. 
> > +         */ 
> > +        if (strcmp(option_rom[i].name, "linuxboot.bin") && 
> > +            strcmp(option_rom[i].name, "multiboot.bin")) { 
> > +            continue; 
> > +        } 
> > +        rom_add_option(option_rom[i].name, option_rom[i].bootindex); 
> > +    } 
>  
> Instead of this hack, you can use: 
>  
> diff --git a/hw/i386/xen/xen_apic.c b/hw/i386/xen/xen_apic.c 
> index 63bb7f7..f5acd6a 100644 
> --- a/hw/i386/xen/xen_apic.c 
> +++ b/hw/i386/xen/xen_apic.c 
> @@ -40,6 +40,7 @@ static void xen_apic_realize(DeviceState *dev, Error  
> **errp) 
>  { 
>      APICCommonState *s = APIC_COMMON(dev); 
>   
> +    s->vapic_control = 0; 

Thanks. It works.

>      memory_region_init_io(&s->io_memory, OBJECT(s), &xen_apic_io_ops, s, 
>                            "xen-apic-msi", APIC_SPACE_SIZE); 
>   
>  
> > +    guest_info->fw_cfg = fw_cfg; 
> > +    return fw_cfg; 
> > +} 
> > + 
> >  FWCfgState *pc_memory_init(MemoryRegion *system_memory, 
> >                             const char *kernel_filename, 
> >                             const char *kernel_cmdline, 
> > diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c 
> > index eaf3e61..14d4164 100644 
> > --- a/hw/i386/pc_piix.c 
> > +++ b/hw/i386/pc_piix.c 
> > @@ -157,6 +157,13 @@ static void pc_init1(QEMUMachineInitArgs *args, 
> >                         args->initrd_filename, 
> >                         below_4g_mem_size, above_4g_mem_size, 
> >                         rom_memory, &ram_memory, guest_info); 
> > +    } else if (args->kernel_filename != NULL) { 
> > +        /* For xen HVM direct kernel boot, load linux here */ 
> > +        fw_cfg = xen_load_linux(args->kernel_filename, 
> > +                                args->kernel_cmdline, 
> > +                                args->initrd_filename, 
> > +                                below_4g_mem_size, 
> > +                                guest_info); 
> >      } 
> >   
> >      gsi_state = g_malloc0(sizeof(*gsi_state)); 
> > diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h 
> > index 32a7687..e472184 100644 
> > --- a/include/hw/i386/pc.h 
> > +++ b/include/hw/i386/pc.h 
> > @@ -134,6 +134,11 @@ PcGuestInfo *pc_guest_info_init(ram_addr_t  
> below_4g_mem_size, 
> >  void pc_pci_as_mapping_init(Object *owner, MemoryRegion *system_memory, 
> >                              MemoryRegion *pci_address_space); 
> >   
> > +FWCfgState *xen_load_linux(const char *kernel_filename, 
> > +                           const char *kernel_cmdline, 
> > +                           const char *initrd_filename, 
> > +                           ram_addr_t below_4g_mem_size, 
> > +                           PcGuestInfo *guest_info); 
> >  FWCfgState *pc_memory_init(MemoryRegion *system_memory, 
> >                             const char *kernel_filename, 
> >                             const char *kernel_cmdline, 
> >  
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Mon, 2014-06-02 at 12:58 -0400, Konrad Rzeszutek Wilk wrote:

> > Yes, which most likely involves preseeding the guest RAM with the files
> > at a discoverable location. Icky.
> 
> Or some form of FTP over XenBus :-)

I'll file that one in the round filing cabinet next to my desk ;-)

Ian

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Mon, 2014-06-02 at 12:58 -0400, Konrad Rzeszutek Wilk wrote:

> > Yes, which most likely involves preseeding the guest RAM with the files
> > at a discoverable location. Icky.
> 
> Or some form of FTP over XenBus :-)

I'll file that one in the round filing cabinet next to my desk ;-)

Ian

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Mon, 2014-06-02 at 21:35 -0600, Chun Yan Liu wrote:
> Now I wonder how does it handle device? (A backend file or partition is also a place
> on host.) Stubdom uses qemu-xen-traditional, maybe there is some special handling?
> Will have a look.

The stubdom VM has access to PV Xen devices via its own {blk,net}front
etc. When emulating a device it issues the resulting operations via
those devices.

Ian.

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Mon, 2014-06-02 at 21:35 -0600, Chun Yan Liu wrote:
> Now I wonder how does it handle device? (A backend file or partition is also a place
> on host.) Stubdom uses qemu-xen-traditional, maybe there is some special handling?
> Will have a look.

The stubdom VM has access to PV Xen devices via its own {blk,net}front
etc. When emulating a device it issues the resulting operations via
those devices.

Ian.

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Ian Campbell]

On Mon, 2014-06-02 at 22:49 -0600, Chun Yan Liu wrote:
> 
> >>> On 6/2/2014 at 11:24 PM, in message
> <1401722652.30097.9.camel@kazak.uk.xensource.com>, Ian Campbell
> <Ian.Campbell@citrix.com> wrote: 
> > On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote: 
> > > [support xen HVM direct kernel boot] 
> >  
> > What is this? 
> 
> Just to indicate this is part of work to support xen HVM direct kernel boot.
> To make it work, there is another patch to qemu.

Putting it there means that it will end up in the eventual commit
message unless the committed takes some special steps. I'd recommend
either adding it to the subject (i.e. "[RFC PATCH XEN 1/2]", "[RFC PATCH
QEMU 2/2]") or after the --- which follows your Signed-off-by.

> 
> >  
> > > xen side patch: support 'kernel', 'ramdisk', 'root', 'extra' 
> > > in HVM config file, parse config file, pass -kernel, -initrd 
> > > and -append parameters to qemu. 
> >  
> > Is this a completely new feature or is this adding parity with a xend 
> > feature? 
> 
> I think it's new. Xend doesn't support HVM direct kernel boot.

OK, then you certainly don't want the old qemu support (which you were
going to remove anyway)

> > > @@ -196,6 +196,16 @@ static char **  
> > libxl__build_device_model_args_old(libxl__gc *gc, 
> >  
> > Does this work with old qemu+rombios? 
> 
> Oh, no. I'll remove this piece.

Thanks.

Ian.

Re: [Xen-devel] [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Ian Campbell]

On Mon, 2014-06-02 at 22:49 -0600, Chun Yan Liu wrote:
> 
> >>> On 6/2/2014 at 11:24 PM, in message
> <1401722652.30097.9.camel@kazak.uk.xensource.com>, Ian Campbell
> <Ian.Campbell@citrix.com> wrote: 
> > On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote: 
> > > [support xen HVM direct kernel boot] 
> >  
> > What is this? 
> 
> Just to indicate this is part of work to support xen HVM direct kernel boot.
> To make it work, there is another patch to qemu.

Putting it there means that it will end up in the eventual commit
message unless the committed takes some special steps. I'd recommend
either adding it to the subject (i.e. "[RFC PATCH XEN 1/2]", "[RFC PATCH
QEMU 2/2]") or after the --- which follows your Signed-off-by.

> 
> >  
> > > xen side patch: support 'kernel', 'ramdisk', 'root', 'extra' 
> > > in HVM config file, parse config file, pass -kernel, -initrd 
> > > and -append parameters to qemu. 
> >  
> > Is this a completely new feature or is this adding parity with a xend 
> > feature? 
> 
> I think it's new. Xend doesn't support HVM direct kernel boot.

OK, then you certainly don't want the old qemu support (which you were
going to remove anyway)

> > > @@ -196,6 +196,16 @@ static char **  
> > libxl__build_device_model_args_old(libxl__gc *gc, 
> >  
> > Does this work with old qemu+rombios? 
> 
> Oh, no. I'll remove this piece.

Thanks.

Ian.

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/2/2014 at 11:14 PM, in message
<alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano
Stabellini <stefano.stabellini@eu.citrix.com> wrote: 
> On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote: 
> > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > Following previous discussion: 
> > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > >  
> > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > code to test. Now it's basically working based on seabios. 
> >  
> > Excellent! 
> > >  
> > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > implemented to intercept boot process, originally by override int19, 
> > > later changed to provide BEV entry which will be called by seabios 
> > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > will load kernel/initrd and jump to execute kernel directly. 
> > >  
> > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > will get the ADDRs and load kernel/initrd data), then add 
> > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > >  
> > > In seabios side, it could load and execute ROMs by itself, and according 
> > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > >  
> > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > to reuse it for xen. 
> > >  
> > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > >     Different from pc_memory_init which does lots of ram alloc work 
> > >     and rom/bios loading work, for xen, we only need to init a basic 
> > >     fw_cfg device used by load_linux() to store ADDRs and 
> > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > >     after that, do real add option rom work to add 
> > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > >     would be done by seabios smoothly. 
> > >  
> > > In my testing, it's working based on seabios. 
> > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > seabios will be default, do we need to put effort to make rombios work? 
> >  
> > No. But the code (libxl) should detect whether you are using the 
> > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > it shouldn't allow you to pass the arguments. 
> >  
> > You probably will also need to an docs/ change to document these 
> > new parameters. 
> > > And stubdom case, since I didn't make my guest started even without direct 
> > > kernel boot, I don't know if it works. And I still could not well 
> > > understand about the differences to stubdom and non stubdom this work 
> > > may cover. Any problems please point out. 
> >  
> > Hmm, you would have to somehow 'transfer' the images from one domain 
> > (the control domain) to the stubdomain which would do the execution. 
> >  
> > At least the multiboot.bin image. 
>  
> Yeah, unfortunately the approach of implemeting direct kernel boot for 
> HVM guests via QEMU wouldn't work for stubdomains. 
>  
> If we want this to work with stubdoms, it would be best to reimplement 
> kernel and initrd loading in libxl/libxc. 

Thanks. Rouphly thinking, the implementation might be totally different.
For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in
qemu find a way to get data and continue with existing code logic (qemu
side changes may be not very clean). And stubdom-dm uses
qemu-xen-traditional, that means rombios by default, that method doesn't
work. Maybe need to change hvmloader? I'll have a look.

> I have to admit that given the extreme simplicity of this patch series, 
> I wouldn't be against merging it, even if it cannot be made to work with 
> stubdoms.  However like in the rombios case, libxl would need to 
> understand that this feature doesn't work with stubdoms and print an 
> appropriate message. 

I'll update printing messages. But now, as you prefer, should I update
existing patch series to let it be in good shape for merge, or should I
abandon it and reimplement to try to make it work with stubdom?

- Chunyan



>  
> > > Personally I'm not quite familiar with the details in this part of code, 
> > > so there might be things not considered, or there might be better method. 
> > > I'm very willing to hear suggestions to improve the work. Any comment or 
> > > feedback would be very appreciated! Thanks. 
> > >   
> > > Chunyan Liu (2): 
> > >   xen: pass kernel initrd to qemu 
> > >   qemu: support xen hvm direct kernel boot 
> > >  
> > > _______________________________________________ 
> > > Xen-devel mailing list 
> > > Xen-devel@lists.xen.org 
> > > http://lists.xen.org/xen-devel 
> >  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/2/2014 at 11:14 PM, in message
<alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano
Stabellini <stefano.stabellini@eu.citrix.com> wrote: 
> On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote: 
> > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > Following previous discussion: 
> > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > >  
> > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > code to test. Now it's basically working based on seabios. 
> >  
> > Excellent! 
> > >  
> > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > implemented to intercept boot process, originally by override int19, 
> > > later changed to provide BEV entry which will be called by seabios 
> > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > will load kernel/initrd and jump to execute kernel directly. 
> > >  
> > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > will get the ADDRs and load kernel/initrd data), then add 
> > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > >  
> > > In seabios side, it could load and execute ROMs by itself, and according 
> > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > >  
> > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > to reuse it for xen. 
> > >  
> > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > >     Different from pc_memory_init which does lots of ram alloc work 
> > >     and rom/bios loading work, for xen, we only need to init a basic 
> > >     fw_cfg device used by load_linux() to store ADDRs and 
> > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > >     after that, do real add option rom work to add 
> > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > >     would be done by seabios smoothly. 
> > >  
> > > In my testing, it's working based on seabios. 
> > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > seabios will be default, do we need to put effort to make rombios work? 
> >  
> > No. But the code (libxl) should detect whether you are using the 
> > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > it shouldn't allow you to pass the arguments. 
> >  
> > You probably will also need to an docs/ change to document these 
> > new parameters. 
> > > And stubdom case, since I didn't make my guest started even without direct 
> > > kernel boot, I don't know if it works. And I still could not well 
> > > understand about the differences to stubdom and non stubdom this work 
> > > may cover. Any problems please point out. 
> >  
> > Hmm, you would have to somehow 'transfer' the images from one domain 
> > (the control domain) to the stubdomain which would do the execution. 
> >  
> > At least the multiboot.bin image. 
>  
> Yeah, unfortunately the approach of implemeting direct kernel boot for 
> HVM guests via QEMU wouldn't work for stubdomains. 
>  
> If we want this to work with stubdoms, it would be best to reimplement 
> kernel and initrd loading in libxl/libxc. 

Thanks. Rouphly thinking, the implementation might be totally different.
For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in
qemu find a way to get data and continue with existing code logic (qemu
side changes may be not very clean). And stubdom-dm uses
qemu-xen-traditional, that means rombios by default, that method doesn't
work. Maybe need to change hvmloader? I'll have a look.

> I have to admit that given the extreme simplicity of this patch series, 
> I wouldn't be against merging it, even if it cannot be made to work with 
> stubdoms.  However like in the rombios case, libxl would need to 
> understand that this feature doesn't work with stubdoms and print an 
> appropriate message. 

I'll update printing messages. But now, as you prefer, should I update
existing patch series to let it be in good shape for merge, or should I
abandon it and reimplement to try to make it work with stubdom?

- Chunyan



>  
> > > Personally I'm not quite familiar with the details in this part of code, 
> > > so there might be things not considered, or there might be better method. 
> > > I'm very willing to hear suggestions to improve the work. Any comment or 
> > > feedback would be very appreciated! Thanks. 
> > >   
> > > Chunyan Liu (2): 
> > >   xen: pass kernel initrd to qemu 
> > >   qemu: support xen hvm direct kernel boot 
> > >  
> > > _______________________________________________ 
> > > Xen-devel mailing list 
> > > Xen-devel@lists.xen.org 
> > > http://lists.xen.org/xen-devel 
> >  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Chun Yan Liu]

>>> On 6/3/2014 at 05:06 PM, in message
<1401786392.8841.44.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Mon, 2014-06-02 at 22:49 -0600, Chun Yan Liu wrote: 
> >  
> > >>> On 6/2/2014 at 11:24 PM, in message 
> > <1401722652.30097.9.camel@kazak.uk.xensource.com>, Ian Campbell 
> > <Ian.Campbell@citrix.com> wrote:  
> > > On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote:  
> > > > [support xen HVM direct kernel boot]  
> > >   
> > > What is this?  
> >  
> > Just to indicate this is part of work to support xen HVM direct kernel  
> boot. 
> > To make it work, there is another patch to qemu. 
>  
> Putting it there means that it will end up in the eventual commit 
> message unless the committed takes some special steps. I'd recommend 
> either adding it to the subject (i.e. "[RFC PATCH XEN 1/2]", "[RFC PATCH 
> QEMU 2/2]") or after the --- which follows your Signed-off-by. 

Oh, yes :) Thanks for reminding.

>  
> >  
> > >   
> > > > xen side patch: support 'kernel', 'ramdisk', 'root', 'extra'  
> > > > in HVM config file, parse config file, pass -kernel, -initrd  
> > > > and -append parameters to qemu.  
> > >   
> > > Is this a completely new feature or is this adding parity with a xend  
> > > feature?  
> >  
> > I think it's new. Xend doesn't support HVM direct kernel boot. 
>  
> OK, then you certainly don't want the old qemu support (which you were 
> going to remove anyway) 
>  
> > > > @@ -196,6 +196,16 @@ static char **   
> > > libxl__build_device_model_args_old(libxl__gc *gc,  
> > >   
> > > Does this work with old qemu+rombios?  
> >  
> > Oh, no. I'll remove this piece. 
>  
> Thanks. 
>  
> Ian. 
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Xen-devel] [RFC PATCH 1/2] xen: pass kernel initrd to qemu

[Chun Yan Liu]

>>> On 6/3/2014 at 05:06 PM, in message
<1401786392.8841.44.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Mon, 2014-06-02 at 22:49 -0600, Chun Yan Liu wrote: 
> >  
> > >>> On 6/2/2014 at 11:24 PM, in message 
> > <1401722652.30097.9.camel@kazak.uk.xensource.com>, Ian Campbell 
> > <Ian.Campbell@citrix.com> wrote:  
> > > On Thu, 2014-05-29 at 11:23 +0800, Chunyan Liu wrote:  
> > > > [support xen HVM direct kernel boot]  
> > >   
> > > What is this?  
> >  
> > Just to indicate this is part of work to support xen HVM direct kernel  
> boot. 
> > To make it work, there is another patch to qemu. 
>  
> Putting it there means that it will end up in the eventual commit 
> message unless the committed takes some special steps. I'd recommend 
> either adding it to the subject (i.e. "[RFC PATCH XEN 1/2]", "[RFC PATCH 
> QEMU 2/2]") or after the --- which follows your Signed-off-by. 

Oh, yes :) Thanks for reminding.

>  
> >  
> > >   
> > > > xen side patch: support 'kernel', 'ramdisk', 'root', 'extra'  
> > > > in HVM config file, parse config file, pass -kernel, -initrd  
> > > > and -append parameters to qemu.  
> > >   
> > > Is this a completely new feature or is this adding parity with a xend  
> > > feature?  
> >  
> > I think it's new. Xend doesn't support HVM direct kernel boot. 
>  
> OK, then you certainly don't want the old qemu support (which you were 
> going to remove anyway) 
>  
> > > > @@ -196,6 +196,16 @@ static char **   
> > > libxl__build_device_model_args_old(libxl__gc *gc,  
> > >   
> > > Does this work with old qemu+rombios?  
> >  
> > Oh, no. I'll remove this piece. 
>  
> Thanks. 
>  
> Ian. 
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/3/2014 at 05:04 PM, in message
<1401786258.8841.42.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Mon, 2014-06-02 at 21:35 -0600, Chun Yan Liu wrote: 
> > Now I wonder how does it handle device? (A backend file or partition is  
> also a place 
> > on host.) Stubdom uses qemu-xen-traditional, maybe there is some special  
> handling? 
> > Will have a look. 
>  
> The stubdom VM has access to PV Xen devices via its own {blk,net}front 
> etc. When emulating a device it issues the resulting operations via 
> those devices. 

See. Thanks!

>  
> Ian. 
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/3/2014 at 05:04 PM, in message
<1401786258.8841.42.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Mon, 2014-06-02 at 21:35 -0600, Chun Yan Liu wrote: 
> > Now I wonder how does it handle device? (A backend file or partition is  
> also a place 
> > on host.) Stubdom uses qemu-xen-traditional, maybe there is some special  
> handling? 
> > Will have a look. 
>  
> The stubdom VM has access to PV Xen devices via its own {blk,net}front 
> etc. When emulating a device it issues the resulting operations via 
> those devices. 

See. Thanks!

>  
> Ian. 
>  
>  
> _______________________________________________ 
> Xen-devel mailing list 
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel 
>  
>  

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Tue, 2014-06-03 at 03:16 -0600, Chun Yan Liu wrote:
> 
> >>> On 6/2/2014 at 11:14 PM, in message
> <alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano
> Stabellini <stefano.stabellini@eu.citrix.com> wrote: 
> > On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote: 
> > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > > Following previous discussion: 
> > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > > >  
> > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > > code to test. Now it's basically working based on seabios. 
> > >  
> > > Excellent! 
> > > >  
> > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > > implemented to intercept boot process, originally by override int19, 
> > > > later changed to provide BEV entry which will be called by seabios 
> > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > > will load kernel/initrd and jump to execute kernel directly. 
> > > >  
> > > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > > will get the ADDRs and load kernel/initrd data), then add 
> > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > > >  
> > > > In seabios side, it could load and execute ROMs by itself, and according 
> > > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > > >  
> > > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > > to reuse it for xen. 
> > > >  
> > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > > >     Different from pc_memory_init which does lots of ram alloc work 
> > > >     and rom/bios loading work, for xen, we only need to init a basic 
> > > >     fw_cfg device used by load_linux() to store ADDRs and 
> > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > > >     after that, do real add option rom work to add 
> > > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > > >     would be done by seabios smoothly. 
> > > >  
> > > > In my testing, it's working based on seabios. 
> > > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > > seabios will be default, do we need to put effort to make rombios work? 
> > >  
> > > No. But the code (libxl) should detect whether you are using the 
> > > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > > it shouldn't allow you to pass the arguments. 
> > >  
> > > You probably will also need to an docs/ change to document these 
> > > new parameters. 
> > > > And stubdom case, since I didn't make my guest started even without direct 
> > > > kernel boot, I don't know if it works. And I still could not well 
> > > > understand about the differences to stubdom and non stubdom this work 
> > > > may cover. Any problems please point out. 
> > >  
> > > Hmm, you would have to somehow 'transfer' the images from one domain 
> > > (the control domain) to the stubdomain which would do the execution. 
> > >  
> > > At least the multiboot.bin image. 
> >  
> > Yeah, unfortunately the approach of implemeting direct kernel boot for 
> > HVM guests via QEMU wouldn't work for stubdomains. 
> >  
> > If we want this to work with stubdoms, it would be best to reimplement 
> > kernel and initrd loading in libxl/libxc. 
> 
> Thanks. Rouphly thinking, the implementation might be totally different.
> For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in
> qemu find a way to get data and continue with existing code logic (qemu
> side changes may be not very clean).

I suppose the native qemu code path is roughly
      * qemu:
              * parse command line
              * load file from disk into appropriate place in guest RAM
              * insert the BIOS extension
              * go

For Xen it would perhaps be something like:
      * libxc:
              * load file from disk into appropriate place in guest RAM
              * communicate this to qemu somehow (xenstore, command
                line, in memory
              * data structure at known location, carrier pigeons)
      * qemu:
              * discover images in RAM according to scheme decided above
              * (perhaps) move those images to the appropriate place in
                guest ram
              * insert the BIOS extension
              * go

So at least the second half of the qemu side could be common/normal.
Perhaps the first half on the qemu side could be abstracted away as some
sort of magic "file like" thing backed by bytes already present in guest
RAM? (it should be pretty clear by now that I don't really know much
about the qemu internals side of this...!)

>  And stubdom-dm uses
> qemu-xen-traditional, that means rombios by default, that method doesn't
> work.

Sorry, I should have said: support for direct kernel from a stub qemu is
dependent on someone implementing stubdm with qemu upstream first. There
is no need at all to implement this for the existing qemu-trad stub dm,
in fact I'm actively against doing so (no new features for qemu-trad).

Ian Jackson is working on making qemu upstream work as a stubdom.

I think I'd be happy if we simply had a plausible plan for how we would
support direct kernel boot with upstream qemu in a stubdm, rather than
blocking these changes on that or expecting you to do that porting work.
Ian -- does that sound OK?

>  Maybe need to change hvmloader? I'll have a look.
> 
> > I have to admit that given the extreme simplicity of this patch series, 
> > I wouldn't be against merging it, even if it cannot be made to work with 
> > stubdoms.  However like in the rombios case, libxl would need to 
> > understand that this feature doesn't work with stubdoms and print an 
> > appropriate message. 
> 
> I'll update printing messages. But now, as you prefer, should I update
> existing patch series to let it be in good shape for merge, or should I
> abandon it and reimplement to try to make it work with stubdom?

You can update this series, I think.

Ian.

Re: [Qemu-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Tue, 2014-06-03 at 03:16 -0600, Chun Yan Liu wrote:
> 
> >>> On 6/2/2014 at 11:14 PM, in message
> <alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano
> Stabellini <stefano.stabellini@eu.citrix.com> wrote: 
> > On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote: 
> > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > > Following previous discussion: 
> > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > > >  
> > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > > code to test. Now it's basically working based on seabios. 
> > >  
> > > Excellent! 
> > > >  
> > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > > implemented to intercept boot process, originally by override int19, 
> > > > later changed to provide BEV entry which will be called by seabios 
> > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > > will load kernel/initrd and jump to execute kernel directly. 
> > > >  
> > > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > > will get the ADDRs and load kernel/initrd data), then add 
> > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > > >  
> > > > In seabios side, it could load and execute ROMs by itself, and according 
> > > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > > >  
> > > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > > to reuse it for xen. 
> > > >  
> > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > > >     Different from pc_memory_init which does lots of ram alloc work 
> > > >     and rom/bios loading work, for xen, we only need to init a basic 
> > > >     fw_cfg device used by load_linux() to store ADDRs and 
> > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > > >     after that, do real add option rom work to add 
> > > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > > >     would be done by seabios smoothly. 
> > > >  
> > > > In my testing, it's working based on seabios. 
> > > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > > seabios will be default, do we need to put effort to make rombios work? 
> > >  
> > > No. But the code (libxl) should detect whether you are using the 
> > > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > > it shouldn't allow you to pass the arguments. 
> > >  
> > > You probably will also need to an docs/ change to document these 
> > > new parameters. 
> > > > And stubdom case, since I didn't make my guest started even without direct 
> > > > kernel boot, I don't know if it works. And I still could not well 
> > > > understand about the differences to stubdom and non stubdom this work 
> > > > may cover. Any problems please point out. 
> > >  
> > > Hmm, you would have to somehow 'transfer' the images from one domain 
> > > (the control domain) to the stubdomain which would do the execution. 
> > >  
> > > At least the multiboot.bin image. 
> >  
> > Yeah, unfortunately the approach of implemeting direct kernel boot for 
> > HVM guests via QEMU wouldn't work for stubdomains. 
> >  
> > If we want this to work with stubdoms, it would be best to reimplement 
> > kernel and initrd loading in libxl/libxc. 
> 
> Thanks. Rouphly thinking, the implementation might be totally different.
> For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in
> qemu find a way to get data and continue with existing code logic (qemu
> side changes may be not very clean).

I suppose the native qemu code path is roughly
      * qemu:
              * parse command line
              * load file from disk into appropriate place in guest RAM
              * insert the BIOS extension
              * go

For Xen it would perhaps be something like:
      * libxc:
              * load file from disk into appropriate place in guest RAM
              * communicate this to qemu somehow (xenstore, command
                line, in memory
              * data structure at known location, carrier pigeons)
      * qemu:
              * discover images in RAM according to scheme decided above
              * (perhaps) move those images to the appropriate place in
                guest ram
              * insert the BIOS extension
              * go

So at least the second half of the qemu side could be common/normal.
Perhaps the first half on the qemu side could be abstracted away as some
sort of magic "file like" thing backed by bytes already present in guest
RAM? (it should be pretty clear by now that I don't really know much
about the qemu internals side of this...!)

>  And stubdom-dm uses
> qemu-xen-traditional, that means rombios by default, that method doesn't
> work.

Sorry, I should have said: support for direct kernel from a stub qemu is
dependent on someone implementing stubdm with qemu upstream first. There
is no need at all to implement this for the existing qemu-trad stub dm,
in fact I'm actively against doing so (no new features for qemu-trad).

Ian Jackson is working on making qemu upstream work as a stubdom.

I think I'd be happy if we simply had a plausible plan for how we would
support direct kernel boot with upstream qemu in a stubdm, rather than
blocking these changes on that or expecting you to do that porting work.
Ian -- does that sound OK?

>  Maybe need to change hvmloader? I'll have a look.
> 
> > I have to admit that given the extreme simplicity of this patch series, 
> > I wouldn't be against merging it, even if it cannot be made to work with 
> > stubdoms.  However like in the rombios case, libxl would need to 
> > understand that this feature doesn't work with stubdoms and print an 
> > appropriate message. 
> 
> I'll update printing messages. But now, as you prefer, should I update
> existing patch series to let it be in good shape for merge, or should I
> abandon it and reimplement to try to make it work with stubdom?

You can update this series, I think.

Ian.

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/3/2014 at 05:31 PM, in message
<1401787863.8841.55.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Tue, 2014-06-03 at 03:16 -0600, Chun Yan Liu wrote: 
> >  
> > >>> On 6/2/2014 at 11:14 PM, in message 
> > <alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano 
> > Stabellini <stefano.stabellini@eu.citrix.com> wrote:  
> > > On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote:  
> > > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:  
> > > > > Following previous discussion:  
> > > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html  
> > > > >   
> > > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code,  
> > > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental  
> > > > > code to test. Now it's basically working based on seabios.  
> > > >   
> > > > Excellent!  
> > > > >   
> > > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are  
> > > > > implemented to intercept boot process, originally by override int19,  
> > > > > later changed to provide BEV entry which will be called by seabios  
> > > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin  
> > > > > will load kernel/initrd and jump to execute kernel directly.  
> > > > >   
> > > > > In qemu load_linux() code, it first read out kernel/initrd file and  
> > > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin  
> > > > > will get the ADDRs and load kernel/initrd data), then add  
> > > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0.  
> > > > >   
> > > > > In seabios side, it could load and execute ROMs by itself, and according  
>  
> > > > > to boot order, it will boot from the BEV entry on the option rom. That  
> > > > > will then be taken over by linuxboot.bin/multiboot.bin.  
> > > > >   
> > > > > So, in theory, qemu load_linux() code almost does the work. I just tried  
>  
> > > > > to reuse it for xen.  
> > > > >   
> > > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm  
> > > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.  
> > > > >     Different from pc_memory_init which does lots of ram alloc work  
> > > > >     and rom/bios loading work, for xen, we only need to init a basic  
> > > > >     fw_cfg device used by load_linux() to store ADDRs and  
> > > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),  
> > > > >     after that, do real add option rom work to add  
> > > > >     linuxboot.bin/multiboot.bin to system option rom. Other things  
> > > > >     would be done by seabios smoothly.  
> > > > >   
> > > > > In my testing, it's working based on seabios.  
> > > > > Rombios case is not working. But now that xen prefers qemu upstream,  
> > > > > seabios will be default, do we need to put effort to make rombios work?  
> > > >   
> > > > No. But the code (libxl) should detect whether you are using the  
> > > > traditional QEMU - which would be using ROM BIOS. If it detects that  
> > > > it shouldn't allow you to pass the arguments.  
> > > >   
> > > > You probably will also need to an docs/ change to document these  
> > > > new parameters.  
> > > > > And stubdom case, since I didn't make my guest started even without  
> direct  
> > > > > kernel boot, I don't know if it works. And I still could not well  
> > > > > understand about the differences to stubdom and non stubdom this work  
> > > > > may cover. Any problems please point out.  
> > > >   
> > > > Hmm, you would have to somehow 'transfer' the images from one domain  
> > > > (the control domain) to the stubdomain which would do the execution.  
> > > >   
> > > > At least the multiboot.bin image.  
> > >   
> > > Yeah, unfortunately the approach of implemeting direct kernel boot for  
> > > HVM guests via QEMU wouldn't work for stubdomains.  
> > >   
> > > If we want this to work with stubdoms, it would be best to reimplement  
> > > kernel and initrd loading in libxl/libxc.  
> >  
> > Thanks. Rouphly thinking, the implementation might be totally different. 
> > For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in 
> > qemu find a way to get data and continue with existing code logic (qemu 
> > side changes may be not very clean). 
>  
> I suppose the native qemu code path is roughly 
>       * qemu: 
>               * parse command line 
>               * load file from disk into appropriate place in guest RAM 
>               * insert the BIOS extension 
>               * go 
>  
> For Xen it would perhaps be something like: 
>       * libxc: 
>               * load file from disk into appropriate place in guest RAM 
>               * communicate this to qemu somehow (xenstore, command 
>                 line, in memory 
>               * data structure at known location, carrier pigeons) 
>       * qemu: 
>               * discover images in RAM according to scheme decided above 
>               * (perhaps) move those images to the appropriate place in 
>                 guest ram 
>               * insert the BIOS extension 
>               * go 

Thanks very much! That's what I want to do.
The main concern is the linuxboot.bin/multiboot.bin image. We should
also load linuxboot.bin/multiboot.bin in libxc first. In native qemu, it inserts
option rom through rom_add_option("linuxboot.bin", 0). Now since
linuxboot.bin has been loaded to RAM, we could not use existing
rom_add_option or other existing APIs, possibly need to write own
rom_add function.

Another concern is, linuxboot.bin/multiboot.bin are binaries generated
by qemu, so there should be a fixed palce for xen to find
linuxboot.bin/multiboot.bin. If stubdom supports qemu upstream, it should
export linuxboot.bin/multiboot.bin to somewhere after xen installation.

- Chunyan

>  
> So at least the second half of the qemu side could be common/normal. 
> Perhaps the first half on the qemu side could be abstracted away as some 
> sort of magic "file like" thing backed by bytes already present in guest 
> RAM? (it should be pretty clear by now that I don't really know much 
> about the qemu internals side of this...!) 
>  
> >  And stubdom-dm uses 
> > qemu-xen-traditional, that means rombios by default, that method doesn't 
> > work. 
>  
> Sorry, I should have said: support for direct kernel from a stub qemu is 
> dependent on someone implementing stubdm with qemu upstream first. There 
> is no need at all to implement this for the existing qemu-trad stub dm, 
> in fact I'm actively against doing so (no new features for qemu-trad). 
>  
> Ian Jackson is working on making qemu upstream work as a stubdom. 
>  
> I think I'd be happy if we simply had a plausible plan for how we would 
> support direct kernel boot with upstream qemu in a stubdm, rather than 
> blocking these changes on that or expecting you to do that porting work. 
> Ian -- does that sound OK? 
>  
> >  Maybe need to change hvmloader? I'll have a look. 
> >  
> > > I have to admit that given the extreme simplicity of this patch series,  
> > > I wouldn't be against merging it, even if it cannot be made to work with  
> > > stubdoms.  However like in the rombios case, libxl would need to  
> > > understand that this feature doesn't work with stubdoms and print an  
> > > appropriate message.  
> >  
> > I'll update printing messages. But now, as you prefer, should I update 
> > existing patch series to let it be in good shape for merge, or should I 
> > abandon it and reimplement to try to make it work with stubdom? 
>  
> You can update this series, I think. 
>  
> Ian. 
>  
>  
>  

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Chun Yan Liu]

>>> On 6/3/2014 at 05:31 PM, in message
<1401787863.8841.55.camel@kazak.uk.xensource.com>, Ian Campbell
<Ian.Campbell@citrix.com> wrote: 
> On Tue, 2014-06-03 at 03:16 -0600, Chun Yan Liu wrote: 
> >  
> > >>> On 6/2/2014 at 11:14 PM, in message 
> > <alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano 
> > Stabellini <stefano.stabellini@eu.citrix.com> wrote:  
> > > On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote:  
> > > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote:  
> > > > > Following previous discussion:  
> > > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html  
> > > > >   
> > > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code,  
> > > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental  
> > > > > code to test. Now it's basically working based on seabios.  
> > > >   
> > > > Excellent!  
> > > > >   
> > > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are  
> > > > > implemented to intercept boot process, originally by override int19,  
> > > > > later changed to provide BEV entry which will be called by seabios  
> > > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin  
> > > > > will load kernel/initrd and jump to execute kernel directly.  
> > > > >   
> > > > > In qemu load_linux() code, it first read out kernel/initrd file and  
> > > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin  
> > > > > will get the ADDRs and load kernel/initrd data), then add  
> > > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0.  
> > > > >   
> > > > > In seabios side, it could load and execute ROMs by itself, and according  
>  
> > > > > to boot order, it will boot from the BEV entry on the option rom. That  
> > > > > will then be taken over by linuxboot.bin/multiboot.bin.  
> > > > >   
> > > > > So, in theory, qemu load_linux() code almost does the work. I just tried  
>  
> > > > > to reuse it for xen.  
> > > > >   
> > > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm  
> > > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot.  
> > > > >     Different from pc_memory_init which does lots of ram alloc work  
> > > > >     and rom/bios loading work, for xen, we only need to init a basic  
> > > > >     fw_cfg device used by load_linux() to store ADDRs and  
> > > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(),  
> > > > >     after that, do real add option rom work to add  
> > > > >     linuxboot.bin/multiboot.bin to system option rom. Other things  
> > > > >     would be done by seabios smoothly.  
> > > > >   
> > > > > In my testing, it's working based on seabios.  
> > > > > Rombios case is not working. But now that xen prefers qemu upstream,  
> > > > > seabios will be default, do we need to put effort to make rombios work?  
> > > >   
> > > > No. But the code (libxl) should detect whether you are using the  
> > > > traditional QEMU - which would be using ROM BIOS. If it detects that  
> > > > it shouldn't allow you to pass the arguments.  
> > > >   
> > > > You probably will also need to an docs/ change to document these  
> > > > new parameters.  
> > > > > And stubdom case, since I didn't make my guest started even without  
> direct  
> > > > > kernel boot, I don't know if it works. And I still could not well  
> > > > > understand about the differences to stubdom and non stubdom this work  
> > > > > may cover. Any problems please point out.  
> > > >   
> > > > Hmm, you would have to somehow 'transfer' the images from one domain  
> > > > (the control domain) to the stubdomain which would do the execution.  
> > > >   
> > > > At least the multiboot.bin image.  
> > >   
> > > Yeah, unfortunately the approach of implemeting direct kernel boot for  
> > > HVM guests via QEMU wouldn't work for stubdomains.  
> > >   
> > > If we want this to work with stubdoms, it would be best to reimplement  
> > > kernel and initrd loading in libxl/libxc.  
> >  
> > Thanks. Rouphly thinking, the implementation might be totally different. 
> > For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in 
> > qemu find a way to get data and continue with existing code logic (qemu 
> > side changes may be not very clean). 
>  
> I suppose the native qemu code path is roughly 
>       * qemu: 
>               * parse command line 
>               * load file from disk into appropriate place in guest RAM 
>               * insert the BIOS extension 
>               * go 
>  
> For Xen it would perhaps be something like: 
>       * libxc: 
>               * load file from disk into appropriate place in guest RAM 
>               * communicate this to qemu somehow (xenstore, command 
>                 line, in memory 
>               * data structure at known location, carrier pigeons) 
>       * qemu: 
>               * discover images in RAM according to scheme decided above 
>               * (perhaps) move those images to the appropriate place in 
>                 guest ram 
>               * insert the BIOS extension 
>               * go 

Thanks very much! That's what I want to do.
The main concern is the linuxboot.bin/multiboot.bin image. We should
also load linuxboot.bin/multiboot.bin in libxc first. In native qemu, it inserts
option rom through rom_add_option("linuxboot.bin", 0). Now since
linuxboot.bin has been loaded to RAM, we could not use existing
rom_add_option or other existing APIs, possibly need to write own
rom_add function.

Another concern is, linuxboot.bin/multiboot.bin are binaries generated
by qemu, so there should be a fixed palce for xen to find
linuxboot.bin/multiboot.bin. If stubdom supports qemu upstream, it should
export linuxboot.bin/multiboot.bin to somewhere after xen installation.

- Chunyan

>  
> So at least the second half of the qemu side could be common/normal. 
> Perhaps the first half on the qemu side could be abstracted away as some 
> sort of magic "file like" thing backed by bytes already present in guest 
> RAM? (it should be pretty clear by now that I don't really know much 
> about the qemu internals side of this...!) 
>  
> >  And stubdom-dm uses 
> > qemu-xen-traditional, that means rombios by default, that method doesn't 
> > work. 
>  
> Sorry, I should have said: support for direct kernel from a stub qemu is 
> dependent on someone implementing stubdm with qemu upstream first. There 
> is no need at all to implement this for the existing qemu-trad stub dm, 
> in fact I'm actively against doing so (no new features for qemu-trad). 
>  
> Ian Jackson is working on making qemu upstream work as a stubdom. 
>  
> I think I'd be happy if we simply had a plausible plan for how we would 
> support direct kernel boot with upstream qemu in a stubdm, rather than 
> blocking these changes on that or expecting you to do that porting work. 
> Ian -- does that sound OK? 
>  
> >  Maybe need to change hvmloader? I'll have a look. 
> >  
> > > I have to admit that given the extreme simplicity of this patch series,  
> > > I wouldn't be against merging it, even if it cannot be made to work with  
> > > stubdoms.  However like in the rombios case, libxl would need to  
> > > understand that this feature doesn't work with stubdoms and print an  
> > > appropriate message.  
> >  
> > I'll update printing messages. But now, as you prefer, should I update 
> > existing patch series to let it be in good shape for merge, or should I 
> > abandon it and reimplement to try to make it work with stubdom? 
>  
> You can update this series, I think. 
>  
> Ian. 
>  
>  
>  

Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Stefano Stabellini]

On Tue, 3 Jun 2014, Ian Campbell wrote:
> On Tue, 2014-06-03 at 03:16 -0600, Chun Yan Liu wrote:
> > 
> > >>> On 6/2/2014 at 11:14 PM, in message
> > <alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano
> > Stabellini <stefano.stabellini@eu.citrix.com> wrote: 
> > > On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote: 
> > > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > > > Following previous discussion: 
> > > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > > > >  
> > > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > > > code to test. Now it's basically working based on seabios. 
> > > >  
> > > > Excellent! 
> > > > >  
> > > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > > > implemented to intercept boot process, originally by override int19, 
> > > > > later changed to provide BEV entry which will be called by seabios 
> > > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > > > will load kernel/initrd and jump to execute kernel directly. 
> > > > >  
> > > > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > > > will get the ADDRs and load kernel/initrd data), then add 
> > > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > > > >  
> > > > > In seabios side, it could load and execute ROMs by itself, and according 
> > > > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > > > >  
> > > > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > > > to reuse it for xen. 
> > > > >  
> > > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > > > >     Different from pc_memory_init which does lots of ram alloc work 
> > > > >     and rom/bios loading work, for xen, we only need to init a basic 
> > > > >     fw_cfg device used by load_linux() to store ADDRs and 
> > > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > > > >     after that, do real add option rom work to add 
> > > > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > > > >     would be done by seabios smoothly. 
> > > > >  
> > > > > In my testing, it's working based on seabios. 
> > > > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > > > seabios will be default, do we need to put effort to make rombios work? 
> > > >  
> > > > No. But the code (libxl) should detect whether you are using the 
> > > > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > > > it shouldn't allow you to pass the arguments. 
> > > >  
> > > > You probably will also need to an docs/ change to document these 
> > > > new parameters. 
> > > > > And stubdom case, since I didn't make my guest started even without direct 
> > > > > kernel boot, I don't know if it works. And I still could not well 
> > > > > understand about the differences to stubdom and non stubdom this work 
> > > > > may cover. Any problems please point out. 
> > > >  
> > > > Hmm, you would have to somehow 'transfer' the images from one domain 
> > > > (the control domain) to the stubdomain which would do the execution. 
> > > >  
> > > > At least the multiboot.bin image. 
> > >  
> > > Yeah, unfortunately the approach of implemeting direct kernel boot for 
> > > HVM guests via QEMU wouldn't work for stubdomains. 
> > >  
> > > If we want this to work with stubdoms, it would be best to reimplement 
> > > kernel and initrd loading in libxl/libxc. 
> > 
> > Thanks. Rouphly thinking, the implementation might be totally different.
> > For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in
> > qemu find a way to get data and continue with existing code logic (qemu
> > side changes may be not very clean).
> 
> I suppose the native qemu code path is roughly
>       * qemu:
>               * parse command line
>               * load file from disk into appropriate place in guest RAM
>               * insert the BIOS extension
>               * go
> 
> For Xen it would perhaps be something like:
>       * libxc:
>               * load file from disk into appropriate place in guest RAM
>               * communicate this to qemu somehow (xenstore, command
>                 line, in memory
>               * data structure at known location, carrier pigeons)
>       * qemu:
>               * discover images in RAM according to scheme decided above
>               * (perhaps) move those images to the appropriate place in
>                 guest ram
>               * insert the BIOS extension
>               * go
> 
> So at least the second half of the qemu side could be common/normal.
> Perhaps the first half on the qemu side could be abstracted away as some
> sort of magic "file like" thing backed by bytes already present in guest
> RAM? (it should be pretty clear by now that I don't really know much
> about the qemu internals side of this...!)
> 
> >  And stubdom-dm uses
> > qemu-xen-traditional, that means rombios by default, that method doesn't
> > work.
> 
> Sorry, I should have said: support for direct kernel from a stub qemu is
> dependent on someone implementing stubdm with qemu upstream first. There
> is no need at all to implement this for the existing qemu-trad stub dm,
> in fact I'm actively against doing so (no new features for qemu-trad).
> 
> Ian Jackson is working on making qemu upstream work as a stubdom.
> 
> I think I'd be happy if we simply had a plausible plan for how we would
> support direct kernel boot with upstream qemu in a stubdm, rather than
> blocking these changes on that or expecting you to do that porting work.
> Ian -- does that sound OK?

I am not Ian but I agree.

Re: [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Stefano Stabellini]

On Tue, 3 Jun 2014, Ian Campbell wrote:
> On Tue, 2014-06-03 at 03:16 -0600, Chun Yan Liu wrote:
> > 
> > >>> On 6/2/2014 at 11:14 PM, in message
> > <alpine.DEB.2.02.1406021610070.4779@kaball.uk.xensource.com>, Stefano
> > Stabellini <stefano.stabellini@eu.citrix.com> wrote: 
> > > On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote: 
> > > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > > > Following previous discussion: 
> > > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > > > >  
> > > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > > > code to test. Now it's basically working based on seabios. 
> > > >  
> > > > Excellent! 
> > > > >  
> > > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > > > implemented to intercept boot process, originally by override int19, 
> > > > > later changed to provide BEV entry which will be called by seabios 
> > > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > > > will load kernel/initrd and jump to execute kernel directly. 
> > > > >  
> > > > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > > > will get the ADDRs and load kernel/initrd data), then add 
> > > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > > > >  
> > > > > In seabios side, it could load and execute ROMs by itself, and according 
> > > > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > > > >  
> > > > > So, in theory, qemu load_linux() code almost does the work. I just tried 
> > > > > to reuse it for xen. 
> > > > >  
> > > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > > > >     Different from pc_memory_init which does lots of ram alloc work 
> > > > >     and rom/bios loading work, for xen, we only need to init a basic 
> > > > >     fw_cfg device used by load_linux() to store ADDRs and 
> > > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > > > >     after that, do real add option rom work to add 
> > > > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > > > >     would be done by seabios smoothly. 
> > > > >  
> > > > > In my testing, it's working based on seabios. 
> > > > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > > > seabios will be default, do we need to put effort to make rombios work? 
> > > >  
> > > > No. But the code (libxl) should detect whether you are using the 
> > > > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > > > it shouldn't allow you to pass the arguments. 
> > > >  
> > > > You probably will also need to an docs/ change to document these 
> > > > new parameters. 
> > > > > And stubdom case, since I didn't make my guest started even without direct 
> > > > > kernel boot, I don't know if it works. And I still could not well 
> > > > > understand about the differences to stubdom and non stubdom this work 
> > > > > may cover. Any problems please point out. 
> > > >  
> > > > Hmm, you would have to somehow 'transfer' the images from one domain 
> > > > (the control domain) to the stubdomain which would do the execution. 
> > > >  
> > > > At least the multiboot.bin image. 
> > >  
> > > Yeah, unfortunately the approach of implemeting direct kernel boot for 
> > > HVM guests via QEMU wouldn't work for stubdomains. 
> > >  
> > > If we want this to work with stubdoms, it would be best to reimplement 
> > > kernel and initrd loading in libxl/libxc. 
> > 
> > Thanks. Rouphly thinking, the implementation might be totally different.
> > For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in
> > qemu find a way to get data and continue with existing code logic (qemu
> > side changes may be not very clean).
> 
> I suppose the native qemu code path is roughly
>       * qemu:
>               * parse command line
>               * load file from disk into appropriate place in guest RAM
>               * insert the BIOS extension
>               * go
> 
> For Xen it would perhaps be something like:
>       * libxc:
>               * load file from disk into appropriate place in guest RAM
>               * communicate this to qemu somehow (xenstore, command
>                 line, in memory
>               * data structure at known location, carrier pigeons)
>       * qemu:
>               * discover images in RAM according to scheme decided above
>               * (perhaps) move those images to the appropriate place in
>                 guest ram
>               * insert the BIOS extension
>               * go
> 
> So at least the second half of the qemu side could be common/normal.
> Perhaps the first half on the qemu side could be abstracted away as some
> sort of magic "file like" thing backed by bytes already present in guest
> RAM? (it should be pretty clear by now that I don't really know much
> about the qemu internals side of this...!)
> 
> >  And stubdom-dm uses
> > qemu-xen-traditional, that means rombios by default, that method doesn't
> > work.
> 
> Sorry, I should have said: support for direct kernel from a stub qemu is
> dependent on someone implementing stubdm with qemu upstream first. There
> is no need at all to implement this for the existing qemu-trad stub dm,
> in fact I'm actively against doing so (no new features for qemu-trad).
> 
> Ian Jackson is working on making qemu upstream work as a stubdom.
> 
> I think I'd be happy if we simply had a plausible plan for how we would
> support direct kernel boot with upstream qemu in a stubdm, rather than
> blocking these changes on that or expecting you to do that porting work.
> Ian -- does that sound OK?

I am not Ian but I agree.