Re: [PATCH v3 02/24] modules: collect module meta-data

Re: [PATCH v3 02/24] modules: collect module meta-data

[Paolo Bonzini]

On 21/06/21 14:52, Gerd Hoffmann wrote:
> ninja: error: 'libui-curses.a.p/meson-generated_.._config-host.h.o', needed by 'ui-curses.modinfo.test', missing and no known rule to make it
> 
> Hmm, not sure where this comes from.  meson doesn't try to link
> config-host.h.o into libui-curses.a, so why does extract_all_objects()
> return it?
> 
> Test patch (incremental to this series) below.

Bug in Meson, fix at https://github.com/mesonbuild/meson/pull/8900.  You 
can just ignore missing files.

Paolo

Re: [PATCH v3 02/24] modules: collect module meta-data

[Gerd Hoffmann]

On Tue, Jun 22, 2021 at 06:03:45PM +0200, Paolo Bonzini wrote:
> On 21/06/21 14:52, Gerd Hoffmann wrote:
> > ninja: error: 'libui-curses.a.p/meson-generated_.._config-host.h.o', needed by 'ui-curses.modinfo.test', missing and no known rule to make it
> > 
> > Hmm, not sure where this comes from.  meson doesn't try to link
> > config-host.h.o into libui-curses.a, so why does extract_all_objects()
> > return it?
> > 
> > Test patch (incremental to this series) below.
> 
> Bug in Meson, fix at https://github.com/mesonbuild/meson/pull/8900.  You can
> just ignore missing files.

Well, it's ninja throwing the error not the modinfo script, the script
doesn't even run ...

take care,
  Gerd

Re: [PATCH v3 02/24] modules: collect module meta-data

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

Oh well. Let's add a to-do marker.

Paolo

Il mer 23 giu 2021, 09:36 Gerd Hoffmann <kraxel@redhat.com> ha scritto:

> On Tue, Jun 22, 2021 at 06:03:45PM +0200, Paolo Bonzini wrote:
> > On 21/06/21 14:52, Gerd Hoffmann wrote:
> > > ninja: error: 'libui-curses.a.p/meson-generated_.._config-host.h.o',
> needed by 'ui-curses.modinfo.test', missing and no known rule to make it
> > >
> > > Hmm, not sure where this comes from.  meson doesn't try to link
> > > config-host.h.o into libui-curses.a, so why does extract_all_objects()
> > > return it?
> > >
> > > Test patch (incremental to this series) below.
> >
> > Bug in Meson, fix at https://github.com/mesonbuild/meson/pull/8900.
> You can
> > just ignore missing files.
>
> Well, it's ninja throwing the error not the modinfo script, the script
> doesn't even run ...
>
> take care,
>   Gerd
>
>

[-- Attachment #2: Type: text/html, Size: 1403 bytes --]

[PATCH v3 00/24] modules: add meta-data database

[Gerd Hoffmann]

This patch series adds support for module meta-data.  Today this is
either hard-coded in qemu (see qemu_load_module_for_opts) or handled
with manually maintained lists in util/module (see module_deps[] and
qom_modules[]).  This series replaced that scheme with annotation
macros, so the meta-data can go into the module source code and -- for
example -- the module_obj() annotations can go next to the TypeInfo
struct for the object class.

Patches 1-3 put the infrastructure in place:  Add the annotation macros,
add a script to collect the meta-data, add a script to compile the
meta-data into C source code which we can then add to qemu.

Patches 4-12 add annotations macros to the modules we have.

Patches 13-15 put the modinfo database into use and remove the
module_deps[] and qom_modules[] lists.

Patch 16 adds two tracepoints for easier trouble-shooting.

Patches 17-19 add support for target-specific modules.

Patches 20-14 start building accelerators modular.  So far it is
only qtest (all archs) and a small fraction of tcg (x86 only).

take care,
  Gerd

Gerd Hoffmann (24):
  modules: add modinfo macros
  modules: collect module meta-data
  modules: generate modinfo.c
  modules: add qxl module annotations
  modules: add virtio-gpu module annotations
  modules: add chardev module annotations
  modules: add audio module annotations
  modules: add usb-redir module annotations
  modules: add ccid module annotations
  modules: add ui module annotations
  modules: add s390x module annotations
  modules: add block module annotations
  modules: use modinfo for dependencies
  modules: use modinfo for qom load
  modules: use modinfo for qemu opts load
  modules: add tracepoints
  modules: check arch and block load on mismatch
  modules: check arch on qom lookup
  modules: target-specific module build infrastructure
  accel: autoload modules
  accel: add qtest module annotations
  accel: build qtest modular
  accel: add tcg module annotations
  accel: build tcg modular

 scripts/modinfo-collect.py      |  67 +++++++++++
 scripts/modinfo-generate.py     |  84 ++++++++++++++
 include/qemu/module.h           |  43 +++++++
 accel/accel-common.c            |   2 +-
 accel/accel-softmmu.c           |   2 +-
 accel/qtest/qtest.c             |   2 +
 accel/tcg/tcg-accel-ops.c       |   1 +
 accel/tcg/tcg-all.c             |   1 +
 audio/spiceaudio.c              |   2 +
 block/iscsi-opts.c              |   1 +
 chardev/baum.c                  |   1 +
 chardev/spice.c                 |   4 +
 hw/display/qxl.c                |   4 +
 hw/display/vhost-user-gpu-pci.c |   1 +
 hw/display/vhost-user-gpu.c     |   1 +
 hw/display/vhost-user-vga.c     |   1 +
 hw/display/virtio-gpu-base.c    |   1 +
 hw/display/virtio-gpu-gl.c      |   3 +
 hw/display/virtio-gpu-pci-gl.c  |   3 +
 hw/display/virtio-gpu-pci.c     |   2 +
 hw/display/virtio-gpu.c         |   1 +
 hw/display/virtio-vga-gl.c      |   3 +
 hw/display/virtio-vga.c         |   2 +
 hw/s390x/virtio-ccw-gpu.c       |   3 +
 hw/usb/ccid-card-emulated.c     |   1 +
 hw/usb/ccid-card-passthru.c     |   1 +
 hw/usb/redirect.c               |   1 +
 softmmu/vl.c                    |  24 ++--
 stubs/module-opts.c             |   4 -
 ui/egl-headless.c               |   4 +
 ui/gtk.c                        |   4 +
 ui/sdl2.c                       |   4 +
 ui/spice-app.c                  |   3 +
 ui/spice-core.c                 |   5 +
 util/module.c                   | 200 ++++++++++++++++++--------------
 accel/qtest/meson.build         |   8 +-
 accel/tcg/meson.build           |   5 +-
 meson.build                     |  76 ++++++++++++
 util/trace-events               |   4 +
 39 files changed, 462 insertions(+), 117 deletions(-)
 create mode 100755 scripts/modinfo-collect.py
 create mode 100755 scripts/modinfo-generate.py

-- 
2.31.1

[PATCH v3 02/24] modules: collect module meta-data

[Gerd Hoffmann]

Add script to collect the module meta-data from the source code,
store the results in *.modinfo files.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 scripts/modinfo-collect.py | 67 ++++++++++++++++++++++++++++++++++++++
 meson.build                | 11 +++++++
 2 files changed, 78 insertions(+)
 create mode 100755 scripts/modinfo-collect.py

diff --git a/scripts/modinfo-collect.py b/scripts/modinfo-collect.py
new file mode 100755
index 000000000000..4acb188c3e89
--- /dev/null
+++ b/scripts/modinfo-collect.py
@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import os
+import sys
+import json
+import shlex
+import subprocess
+
+def find_command(src, target, compile_commands):
+    for command in compile_commands:
+        if command['file'] != src:
+            continue
+        if target != '' and command['command'].find(target) == -1:
+            continue
+        return command['command']
+    return 'false'
+
+def process_command(src, command):
+    skip = False
+    arg = False
+    out = []
+    for item in shlex.split(command):
+        if arg:
+            out.append(x)
+            arg = False
+            continue
+        if skip:
+            skip = False
+            continue
+        if item == '-MF' or item == '-MQ' or item == '-o':
+            skip = True
+            continue
+        if item == '-c':
+            skip = True
+            continue
+        out.append(item)
+    out.append('-DQEMU_MODINFO')
+    out.append('-E')
+    out.append(src)
+    return out
+
+def main(args):
+    target = ''
+    if args[0] == '--target':
+        args.pop(0)
+        target = args.pop(0)
+        print("MODINFO_DEBUG target %s" % target)
+        arch = target[:-8] # cut '-softmmu'
+        print("MODINFO_START arch \"%s\" MODINFO_END" % arch)
+    with open('compile_commands.json') as f:
+        compile_commands = json.load(f)
+    for src in args:
+        print("MODINFO_DEBUG src %s" % src)
+        command = find_command(src, target, compile_commands)
+        cmdline = process_command(src, command)
+        print("MODINFO_DEBUG cmd", cmdline)
+        result = subprocess.run(cmdline, stdout = subprocess.PIPE,
+                                universal_newlines = True)
+        if result.returncode != 0:
+            sys.exit(result.returncode)
+        for line in result.stdout.split('\n'):
+            if line.find('MODINFO') != -1:
+                print(line)
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
diff --git a/meson.build b/meson.build
index a2311eda6ec5..bb99619257d5 100644
--- a/meson.build
+++ b/meson.build
@@ -2021,6 +2021,9 @@ subdir('tests/qtest/fuzz')
 # Library dependencies #
 ########################
 
+modinfo_collect = find_program('scripts/modinfo-collect.py')
+modinfo_files = []
+
 block_mods = []
 softmmu_mods = []
 foreach d, list : modules
@@ -2034,6 +2037,14 @@ foreach d, list : modules
       else
         softmmu_mods += sl
       endif
+      if module_ss.sources() != []
+        modinfo_files += custom_target(d + '-' + m + '.modinfo',
+                                       output: d + '-' + m + '.modinfo',
+                                       input: module_ss.sources(),
+                                       capture: true,
+                                       build_by_default: true, # to be removed when added to a target
+                                       command: [modinfo_collect, '@INPUT@'])
+      endif
     else
       if d == 'block'
         block_ss.add_all(module_ss)
-- 
2.31.1

Re: [PATCH v3 02/24] modules: collect module meta-data

[Paolo Bonzini]

On 18/06/21 06:53, Gerd Hoffmann wrote:
> +def find_command(src, target, compile_commands):
> +    for command in compile_commands:
> +        if command['file'] != src:
> +            continue
> +        if target != '' and command['command'].find(target) == -1:
> +            continue


Did you look into using extract_objects for this instead of looking for 
the target (which works, but yuck :))?

Paolo

Re: [PATCH v3 02/24] modules: collect module meta-data

[Gerd Hoffmann]

On Fri, Jun 18, 2021 at 06:09:55PM +0200, Paolo Bonzini wrote:
> On 18/06/21 06:53, Gerd Hoffmann wrote:
> > +def find_command(src, target, compile_commands):
> > +    for command in compile_commands:
> > +        if command['file'] != src:
> > +            continue
> > +        if target != '' and command['command'].find(target) == -1:
> > +            continue
> 
> 
> Did you look into using extract_objects for this instead of looking for the
> target (which works, but yuck :))?

ninja: error: 'libui-curses.a.p/meson-generated_.._config-host.h.o', needed by 'ui-curses.modinfo.test', missing and no known rule to make it

Hmm, not sure where this comes from.  meson doesn't try to link
config-host.h.o into libui-curses.a, so why does extract_all_objects()
return it?

Test patch (incremental to this series) below.

take care,
  Gerd

From 5453683429d7b08b959e2cd63ee00fdccfb0c7b7 Mon Sep 17 00:00:00 2001
From: Gerd Hoffmann <kraxel@redhat.com>
Date: Mon, 21 Jun 2021 14:45:14 +0200
Subject: [PATCH] [wip] extract_all_objects experiments

---
 meson.build             | 7 +++++++
 scripts/modinfo-test.sh | 8 ++++++++
 2 files changed, 15 insertions(+)
 create mode 100755 scripts/modinfo-test.sh

diff --git a/meson.build b/meson.build
index 03bacca7cddb..8e7ccccf176c 100644
--- a/meson.build
+++ b/meson.build
@@ -2042,6 +2042,7 @@ target_modules += { 'accel' : { 'qtest': qtest_module_ss,
 
 modinfo_collect = find_program('scripts/modinfo-collect.py')
 modinfo_generate = find_program('scripts/modinfo-generate.py')
+modinfo_test = find_program('scripts/modinfo-test.sh')
 modinfo_files = []
 
 block_mods = []
@@ -2063,6 +2064,12 @@ foreach d, list : modules
                                        input: module_ss.sources(),
                                        capture: true,
                                        command: [modinfo_collect, '@INPUT@'])
+        custom_target(d + '-' + m + '.modinfo.test',
+                      output: d + '-' + m + '.modinfo.test',
+                      input: sl.extract_all_objects(recursive: true),
+                      capture: true,
+                      build_by_default: true, # to be removed when added to a target
+                      command: [modinfo_test, '@INPUT@'])
       endif
     else
       if d == 'block'
diff --git a/scripts/modinfo-test.sh b/scripts/modinfo-test.sh
new file mode 100755
index 000000000000..979c9cc9aeef
--- /dev/null
+++ b/scripts/modinfo-test.sh
@@ -0,0 +1,8 @@
+#!/bin/sh
+if test "$1" = "--target"; then
+    echo "# target $2"
+    shift;shift
+fi
+for item in "$@"; do
+    echo "# input $item"
+done
-- 
2.31.1