On Thu, Nov 07, 2019 at 10:53:27AM -0500, Jag Raman wrote:
> 
> 
> On 11/7/2019 9:39 AM, Daniel P. Berrangé wrote:
> > On Thu, Nov 07, 2019 at 03:02:20PM +0100, Stefan Hajnoczi wrote:
> > > On Thu, Oct 24, 2019 at 05:09:30AM -0400, Jagannathan Raman wrote:
> > > > From: Elena Ufimtseva <elena.ufimtseva@oracle.com>
> > > > 
> > > > Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
> > > > Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
> > > > Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> > > > ---
> > > >   docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++
> > > >   1 file changed, 86 insertions(+)
> > > >   create mode 100644 docs/qemu-multiprocess.txt
> > > > 
> > > > diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt
> > > > new file mode 100644
> > > > index 0000000..c29f4df
> > > > --- /dev/null
> > > > +++ b/docs/qemu-multiprocess.txt
> > > > @@ -0,0 +1,86 @@
> > > > +Multi-process QEMU
> > > > +==================
> > > > +
> > > > +This document describes how to configure and use multi-process qemu.
> > > > +For the design document refer to docs/devel/qemu-multiprocess.
> > > > +
> > > > +1) Configuration
> > > > +----------------
> > > > +
> > > > +To enable support for multi-process add --enable-mpqemu
> > > > +to the list of options for the "configure" script.
> > > > +
> > > > +
> > > > +2) Usage
> > > > +--------
> > > > +
> > > > +To start qemu with devices intended to run in a separate emulation
> > > > +process without libvirtd support, the following should be used on QEMU
> > > > +command line. As of now, we only support the emulation of lsi53c895a
> > > > +in a separate process
> > > > +
> > > > +* Since parts of the RAM are shared between QEMU & remote process, a
> > > > +  memory-backend-file is required to facilitate this, as follows:
> > > > +
> > > > +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on
> > > > +
> > > > +* The devices to be emulated in the separate process are defined as
> > > > +  before with addition of "rid" suboption that serves as a remote group
> > > > +  identificator.
> > > > +
> > > > +  -device <device options>,rid="remote process id"
> > > > +
> > > > +  For exmaple, for non multi-process qemu:
> > > 
> > > s/exmaple/example/
> > > 
> > > > +    -device lsi53c895a,id=scsi0 device
> > > > +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0
> > > > +    -drive id=drive0,file=data-disk.img
> > > > +
> > > > +  and for multi-process qemu and no libvirt
> > > > +  support (i.e. QEMU forks child processes):
> > > > +    -device lsi53c895a,id=scsi0,rid=0
> > > > +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid="0"
> > > > +
> > > > +* The command-line options for the remote process is added to the "command"
> > > 
> > > s/is added/are added/
> > > 
> > > > +  suboption of the newly added "-remote" option.
> > > > +
> > > > +   -remote [socket],rid=,command="..."
> > > > +
> > > > +  The drives to be emulated by the remote process are specified as part of
> > > > +  this command sub-option. The device to be used to connect to the monitor
> > > > +  is also specified as part of this suboption.
> > > > +
> > > > +  For example, the following option adds a drive and monitor to the remote
> > > > +  process:
> > > > +  -remote rid=0,command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> > > > +
> > > > +  Note: There's an issue with this "command" subtion which we are in the
> > > 
> > > s/subtion/sub-option/
> > > 
> > > > +  process of fixing. To work around this issue, it requires additional
> > > > +  "comma" characters as illustrated above, and in the example below.
> > > > +
> > > > +* Example QEMU command-line to launch lsi53c895a in a remote process
> > > > +
> > > > +  #/bin/sh
> > > > +  qemu-system-x86_64 \
> > > > +  -name "OL7.4" \
> > > > +  -machine q35,accel=kvm \
> > > > +  -smp sockets=1,cores=1,threads=1 \
> > > > +  -cpu host \
> > > > +  -m 2048 \
> > > > +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \
> > > > +  -numa node,memdev=mem \
> > > > +  -device virtio-scsi-pci,id=virtio_scsi_pci0 \
> > > > +  -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \
> > > > +  -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \
> > > > +  -boot d \
> > > > +  -monitor stdio \
> > > > +  -vnc :0 \
> > > > +  -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \
> > > > +  -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\
> > > > +  -remote rid=8,command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> > > > +
> > > > +  We could connect to the monitor using the following command:
> > > > +  socat /home/qmp-sock stdio
> > > > +
> > > > +  After hotplugging disks to the remote process, please execute the
> > > > +  following command in the guest to refresh the list of storage devices:
> > > > +  rescan_scsi_bus.sh -a
> > > 
> > > This documentation suggests that QEMU spawns the remote processes.  How
> > > do this work with unprivileged QEMU?  Is there an additional step where
> > > QEMU drops privileges after having spawned remote processes?
> > 
> > This syntax is for the simple case without privilege separation.
> > If differing privilege levels are needed, then whatever spawns QEMU
> > should spawn the remote helper process ahead of time, and then just
> > pass the UNIX socket path to the -remote arg, instead of using
> > the 'command' parameter.
> > 
> > Regards,
> > Daniel
> 
> Thank You, Stefan, Michael & Daniel, for your comments. I had a chance
> to sit down with my teammates to understand the feedback you gave at the
> KVM Forum. Thank you for that, as well.
> 
> We currently support two ways of launching the remote process - one is
> self-launch through QEMU, as outlined in this patch series. The other
> approach is using an Orchestrator like libvirt (we haven't had the
> chance to submit those patches for review yet).
> 
> In the case where libvirt is involved, it would assume the
> responsibility of spawning the remote process first and pass in the info
> required to connect to the remote process via command-line arguments to
> QEMU. This support in QEMU is available in the current series. We
> haven't sent the libvirt side of patches out for review yet. It would be
> easier to upstream libvirt once the QEMU side of things is firmed up.
> 
> In the case of self-launch, our understanding is that QEMU has the
> privilege to fork() the remote process until the "-sandbox" argument is
> processed. However, if an Orchestrator prohibits QEMU from spawning
> other processes from the get-go, then the Orchestrator would assume the
> responsibility of spawning the remote process as well - like Daniel just
> pointed out.
> 
> In both cases, we intend to apply the security policies required to
> confine the remote process externally - probably through SELinux. We
> haven't had the chance to upstream the SELinux policies yet, but we
> previously sent a sample of the policies for your comments. Like Michael
> pointed out earlier, the SELinux policies are per binary.

Sounds good, please document -remote socket= as an alternative to
-remote command= so it's clear that both approaches are supported.

Stefan