Re: About creating machines on the command line

[Luc Michel <luc.michel@greensocs.com>]

On 2/4/21 9:29 PM, John Snow wrote:
> On 2/3/21 12:09 PM, graf--- via wrote:
>>
>>
>> On 03.02.21 17:55, Philippe Mathieu-Daudé wrote:
>>>
>>> On 1/11/21 3:50 PM, Luc Michel wrote:
>>>> Hi,
>>>>
>>>> We would like to work on improving QEMU to be able to create custom
>>>> machines from the command line. The goal here is to get feedback from
>>>> the community and shape the future developments.
>>>
>>> Cc'ing John who is working on command line, and some developers from
>>> the "inter-VM device emulation interface" call.
>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg723252.html
>>>
> 
> I have a bunch of extremely naive questions about what it would take to 
> generalize this kind of interface; my familiarity is largely with x86 
> and so I actually just have a series of very basic questions about the 
> scope of this work.
> 
>>>>
>>>> The use case mainly comes from people working with tools to customize
>>>> their designs, such as SiFive Core Designer
>>>> (https://scs.sifive.com/core-designer). This kind of tools may allow
>>>> creation or customization of a whole SoC, from the number of cores, to
>>>> the memory and IRQ mapping of peripherals etc.
>>>>
>>>> The ultimate goal would be to be able to create any kind of machine on
>>>> the command line. However we are aware that this is a substantial 
>>>> amount
>>>> of changes in QEMU.
>>
>> Is the command line really the right abstraction level here? Wouldn't 
>> it make more sense to have a QOM / <scripting language> bridge that 
>> allows you to create and connect QOM objects using for example Python?
>>
>> You could then have machine descriptions in a script, which could be 
>> generated by the SoC customization tools.
>>
>> In combination with plugin support for platform devices, that should 
>> allow pretty much any customization you would need to happen, no?
>>
>>
>> Alex
>>
> 
> Right, I am just reading this proposal as "We want an interface language 
> that allows us to construct more arbitrary configurations."

Yes exactly.

> 
> In this case, something QMP/QOM-based would be the right place to do it 
> and not the CLI as it exists today.

Yes, we realized this through the discussions. We are now changing our 
plan to focus on QMP.

> Small details in the grand scheme of 
> things the kind of internal modeling cleanup this idea seems to imply.
> 
>>>>
>>>> In its current state, QEMU allows for very limited customization of
>>>> existing machines on the command line. We identified the following
>>>> limitations (feel free to add to the list):
>>>>
>>>>    - Most devices are not user creatable. Moreover, sysbus devices must
>>>> be explicitly allowed by a machine to be creatable through `-device`,
>>>>
>>>>    - Memory regions cannot be created on the command line,
>>>>
>>>>    - Device MMIO regions cannot be mapped on a bus from the command 
>>>> line,
>>>>
>>>>    - GPIOs and clocks cannot be wired from the command line,
>>>>
>>>>    - CPUs are not sysbus devices (and not user-creatable). They need
>>>> special care when creating them regarding system reset. Not being on a
>>>> bus means that they must be reset manually on system reset. This is 
>>>> done
>>>> in machines by registering a QEMU reset handler.
>>>>
>>>>    - Machine specific boot code is usually hard-coded into the machine
>>>> itself.  Some architectures (e.g. ARM) do factorize bootloader related
>>>> code, but there is no standard way of doing that in QEMU.
>>>>
>>>> We don't want to address all those limitations at once. We plan to 
>>>> start
>>>> with the following scenario:
>>>>
> 
> It does sound like a staggering level of work.
> 
>>>>    - Start with a base machine that would handle CPU creation and
>>>> bootloader stuff. Note that the "none" machine is probably not
>>>> sufficient in its current shape. It does allow only one CPU and
>>>> obviously does not handle the boot process.
>>>>
> 
> I suppose this implies a type of machine that doesn't have migration 
> compatibility. e.g. your "blank" might be different from someone else's 
> "blank".
> 
> Or at least, not migration compatibility in its current form. Some 
> thought might need to be put into how to negotiate migration 
> compatibility between two instances of QEMU; it sounds like a hard problem.
> 
> (Jokingly, I'd want to call a machine like this "Apple Pie", but that 
> would probably be too confusing for both Apple and Raspberry Pi 
> enthusiasts.)

Migration is definitely an issue. We didn't think too much about it yet 
because the use-case is more firmware/OS development oriented (i.e. I 
fix, compile and re-run my code in QEMU to test/debug). Migration is not 
a must-have for this use-case.

> 
>>>>    - Allow for this machine every sysbus devices we want to be user
>>>> command-line creatable (and mark them user_creatable if needed)
>>>>
> 
> Is it even possible to enumerate, for every sysbus device in QEMU, what 
> the interface requirements are?
> 
> e.g. I assume most need a memory region, or maybe an IO region, or maybe 
> an IRQ resource of some kind, or ...
> 
> doing this kind of dynamic bringup requires a lot of configuration data 
> that is at-present just hardcoded absolutely everywhere.

As of today, as I understand it, what a sysbus device exposes is part of 
its "specification" (i.e. in best cases, it is documented in a comment 
somewhere in its header file). Apart from that there is no way of 
knowing that the memory region number 0 is e.g for the ARM GIC, the 
distributor interface, while the number 1 is the CPU interface.

An introspection mechanism would be very helpful here. A simple addition 
would be to be able to name those regions at sysbus device init time, 
the same way we can name GPIOs and clocks. Having a proper name is 
already a good piece of "self-documentation" for the device, and it 
could be made observable by the user (as properties are today in QMP).

> 
>>>>    - Add command line options to create memory regions (probably ram 
>>>> ones
>>>> at first)
>>>>
> 
> Naive question: what does adding a memory region mean to you in this 
> context?
> 
> For whom (which device, spec, standard, instruction set, etc?) are these 
> regions tangible abstractions?

Good question. What we want here is basically being able to create the 
memories of our platform (rom, ram). Those can be viewed as devices 
really, and this is how in our current prototype we made them (a simple 
sysbus device that embeds the memory region and exposes it through SBD 
API). I think this is the good level of abstraction to choose because it 
aligns nicely with the rest. Such a simple device would make sense in 
vanilla IMO. Creating the memories of the platform would then resort to 
creating a device (with parameters to set the size, set if it is 
read-only, ...) and connecting it to a bus.

> 
>>>>    - Add command line options to map a memory region (including sysbus
>>>> device MMIO regions) onto another (memory_region_add_subregion)
>>>>
> 
> More naive questions:
> 
> The assumption here being that sysbus devices can be modeled using their 
> own private memory addressing namespace; and we configure them by 
> mapping that space onto the machine's physical address space.
> 
> Instead of, say, making the memory address of that device a property of 
> the device.
> 
> Yes?

Yes memory regions can be created by sysbus devices and exposed at the 
sysbus device API level. Then the parent device instantiating this 
device (or the machine itself) can use the SBD API to fetch and connect 
them to other memory regions. This is then up to the parent to choose 
where and how to map the child region(s).

> 
>>>>    - Add command line options to connect GPIOs and clocks.
>>>>
> 
> Similar to elsewhere, I wonder how much work it is to generalize this 
> kind of pluggability so that all existing clocks and GPIO devices can be 
> modeled in the same way.
> 
> (I say this meaning: Wow, I have no idea. I know a lot of the ones we 
> have are hard-coded. What does a configuration interface for this look 
> like?)

By hard-coded, you mean the connections? Because otherwise the GPIO and 
Clock APIs are well defined already. Exposing this to QMP does not seem 
to be an issue. I could be wrong though.

> 
>>>> This would hopefully allow for simple machines creation. We would then
>>>> be able to use either the command line or the `-readconfig` option to
>>>> create the machine.
>>>>
>>>> Note that we are not planning to use QMP/HMP for now. From our
>>>> understanding, a `device_add` request is always considered as hot-plug,
>>>> which is not what we want here.
>>>>
> 
> We're working our way towards being able to use QMP to define machines 
> prior to execution start in order to have a QMP-only interaction with 
> QEMU possible.
> 
> You likely want to invest in that kind of a strategy where you define 
> structures that describe the machines you want to build.

OK. It is clear to us that QMP is the way to go now. The plan is for us 
to work and come up with a first RFC series.

Thanks a lot!

-- 
Luc

> 
>>>> Please tell us what do you think about this plan. Any feedback is
>>>> appreciated.  Then we can discuss the details of how to do this 
>>>> properly.
>>>> Thanks!
>>>>
>>>> -- 
>>>> Luc
>>>>
>>
>>
>>
>> Amazon Development Center Germany GmbH
>> Krausenstr. 38
>> 10117 Berlin
>> Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
>> Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
>> Sitz: Berlin
>> Ust-ID: DE 289 237 879
>>
>>
>