Re: [Xen-devel] Uniform commands for booting xen

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-11-13 at 00:48 -0700, Jan Beulich wrote:
> > > > On 12.11.15 at 18:09, <ian.campbell@citrix.com> wrote:
> > On Thu, 2015-11-12 at 08:44 -0700, Jan Beulich wrote:
> > > > > > On 12.11.15 at 14:41, <phcoder@gmail.com> wrote:
> > > > Hello, all. I'd like to have set of commands that would boot xen on
> > > > all
> > > > platforms. I thought of following set:
> > > > 
> > > > xen_hypervisor FILE XEN_OPTIONS
> > > > xen_kernel FILE KERNEL_OPTIONS
> > > > xen_initrd INITRD INITRD INITRD
> > > > all initrds are concatenated.
> > > > xen_xsm ???
> > > 
> > > xen_ucode (and we might add more going forward). I don't see
> > > why the multiboot mechanism (kernel plus any number of modules)
> > > can't be used, without adding any Xen-specific directives.
> > 
> > You likely aren't aware that on ARM Xen doesn't boot via multiboot, but
> > via
> > a protocol which involves passing modules in an fdt[0].
> > 
> > I had originally hoped that this would use the same command names in
> > the
> > grub cfg, such that things would just work, however the grub
> > maintainers
> > didn't like that (and I appreciate why).
> > 
> > Hence on grub/ARM we already have xen_{hypervisor,kernel,initrd,...}.
> > 
> > The question then is what grub-mkconfig (or more precisely
> > /etc/grub.d/20_linux_xen) ought to emit so that things just work on all
> > architectures.
> > 
> > The author of the grub/ARM/Xen patches initially made it generate the
> > xen_*
> > namas for arm and the multiboot names for x86, here is Vladimir's
> > feedback
> > on that: http://lists.gnu.org/archive/html/grub-devel/2015-10/msg00133.
> > html 
> > 
> > Which I think gets us to approximately today and Vladimir's question.
> 
> Now that makes the situation really ugly (and supports my
> reservations regarding grub2 as a uniform solution for everything).
> How do you express modules other than kernel+initrd in that
> scheme, without grub needing to be aware of any new addition we
> may find necessary going forward?

When I initially designed[0] my intention was that grub.cfg would be
unchanged between x86 and arm and therefore that the usual "multiboot" and
"module" commands would be used, but would be backed by the FDT protocol
not actual multiboot (my logic was that multiboot1 would never be added to
ARM and multiboot2 uses different command names so there was no clash, grub
upstream had good reasons for objecting to that though).

However at that time Xen/ARM didn't do as Xen/x86 does and assume something
from the ordering of the modules (first==dom0 kernel, second==dom0 initrd,
mechanisms to scan for other types, etc). Which led to the stuff at [1]
which put the onus for this inference into the bootloader, in a rather
complex way.

I since got convinced that this was madness and implemented in Xen/ARM
similar logic to x86 (i.e. inference based on the module ordering) so that
the bootloader could just present the modules in the order the cfg has them
and Xen/ARM would DTRT for the same set of common cases as Xen/x86 would.

At this point I should have updated [0] to much simplify things, since now
a single module command would have been much simpler. But it looks like I
neglected to do so and so the complex module type inference described in
[1] morphed during review (i.e. objections to the complicated type handling
in [1]) into the current xen_{kernel,initrd,foo} stuff we have now. Sorry
for a) not updating that wiki page in a timely manner and b) not noticing
this discrepancy was occurring during review.

I think at this point for the ARM stuff we could now ditch all these
xen_{kernel,initrd,xsm} etc in favour of a single xen_module command which
doesn't automatically attempt to infer or specify the type, and just lets
Xen figure it out.

AFAICT on the grub side this would mean exposing grub_cmd_xen_module
directly as a command and dropping the uses of set_module_type and the
other aliases.

Aside: Ideally I'd like to see a simplified variant of the --type stuff
added, which just takes a raw fdt compat value to use for flexibility in
the future, but that's not critical and orthogonal to this discussion.

> I think any architecture following a well defined, cross-arch
> protocol (like multiboot) should not require any special xen_*
> directives. If ARM64 needs Xen to be treated specially, special
> directives are maybe warranted for this particular case, but I don't
> see why all architectures supporting Xen should then automatically
> have to use those too. But yes, it's not my call decide this...

So the above would IMHO make arm64 a) more rational and b) closer to how
multiboot/x86 behaves, but I don't really think it addresses your core
concern.

In [2] Vladimir proposed three options to avoid assumption about the
machine which runs grub-mkconfig not being the one which is going to
execute grub.cfg:
    - Check arch on boot time
    - Check that new xen commands are supported (define a new feature)

Both of which are alternatives to the proposal for uniform commands made
here. Both have the same downside which is that the grub.cfg then becomes
   if <some condition>; then
	xen_hypervisor AAAA
	xen_module BBB
	xen_module CCC
   else
	multiboot AAAA
	module BBB
	module CCC
    fi

Which is added complexity in grub.cfg and in 20_linux_xen (only the latter of which is really "user serviceable" though).

Ian.

[0] http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Multiboot
[1] http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Multiboot#Command:_module
[2] http://lists.gnu.org/archive/html/grub-devel/2015-10/msg00133.html