[Xen-devel] HVM Driver Domain

[Xen-devel] HVM Driver Domain

[tosher 1]

Hi all,

I was doing some experiments on the Xen network Driver Domain using Ubuntu 18.04.  I was able to see the driver domain works fine when I run it in PV mode. However, I wasn't able to make the driver domain work when I run it in HVM mode. I get the following error when I want my DomU to use HVM driver domain for network backend.

libxl: error: libxl_nic.c:652:libxl__device_nic_set_devids: Domain 25:Unable to set nic defaults for nic 0

Other than this, I didn't get any log messages from dmesg, xl dmesg commands, and files from /var/log/xen/ directory regarding this failure. Therefore, I was wondering if it is even possible to create an HVM Driver Domain. Please let me know what you think.

Thanks,
Mehrab

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 1205 bytes --]

On Mon, Jan 20, 2020 at 06:07:57PM +0000, tosher 1 wrote:
> Hi all,
> 
> I was doing some experiments on the Xen network Driver Domain using Ubuntu 18.04.  I was able to see the driver domain works fine when I run it in PV mode. However, I wasn't able to make the driver domain work when I run it in HVM mode. I get the following error when I want my DomU to use HVM driver domain for network backend.
> 
> libxl: error: libxl_nic.c:652:libxl__device_nic_set_devids: Domain 25:Unable to set nic defaults for nic 0
> 
> Other than this, I didn't get any log messages from dmesg, xl dmesg commands, and files from /var/log/xen/ directory regarding this failure. Therefore, I was wondering if it is even possible to create an HVM Driver Domain. Please let me know what you think.

It's definitely possible, as we use it extensively in Qubes, including
for network device backend.
Can you give more details about your setup? Which Xen version, what your
domU configuration looks like (especially vif setting)?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

Hi Marek,

Thanks for your response. The server machine I am using for this setup is an x86_64 Intel Xeon. For the Dom0, I am using Ubuntu 18.04 running on kernel version 5.0.0-37-generic. My Xen version is 4.9.2. 

For the HVM driver domain, I am using Ubuntu 18.04 running on kernel version 5.0.0-23-generic. I am doing a NIC PCI passthrough to this domain. The Xen config file for this domain looks like the following.

builder = "hvm"
name = "ubuntu-doment-hvm"
memory = "2048"
pci = [ '01:00.0,permissive=1' ]
vcpus = 1
disk = ['phy:/dev/vg/ubuntu-hvm,hda,w']
vnc = 1
boot="c"

I have installed xen-tools of version 4.7 in this driver domain so that the vif-scirpts work. The network configuration here looks like the following where ens5f0 is the interface name for the NIC I did passthrough.

auto lo
iface lo inet loopback

iface ens5f0 inet manual

auto xenbr1
iface xenbr1 inet static
        bridge_ports ens5f0
        address 192.168.1.3
        netmask 255.255.255.0
        gateway 192.168.1.1

The Xen config file content for the DomU is as the following.

name = "ubuntu_on_ubuntu"
bootloader = "/usr/lib/xen-4.9/bin/pygrub"
memory = 1024
vcpus = 1
vif = [ 'backend=ubuntu-domnet-hvm,bridge=xenbr1' ]
disk = [ '/dev/vg/lv_vm_ubuntu_guest,raw,xvda,rw' ]

When I try to launch this DomU, I get the following error.

libxl: error: libxl_nic.c:652:libxl__device_nic_set_devids: Domain 31:Unable to set nic defaults for nic 0.

Are these configurations basically very different for what you do for Qubes? Please let me know your thoughts.

Thanks,
Mehrab

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 2188 bytes --]

On Wed, Jan 22, 2020 at 04:56:15PM +0000, tosher 1 wrote:
> Hi Marek,
> 
> Thanks for your response. The server machine I am using for this setup is an x86_64 Intel Xeon. For the Dom0, I am using Ubuntu 18.04 running on kernel version 5.0.0-37-generic. My Xen version is 4.9.2. 
> 
> For the HVM driver domain, I am using Ubuntu 18.04 running on kernel version 5.0.0-23-generic. I am doing a NIC PCI passthrough to this domain. The Xen config file for this domain looks like the following.
> 
> builder = "hvm"
> name = "ubuntu-doment-hvm"
> memory = "2048"
> pci = [ '01:00.0,permissive=1' ]
> vcpus = 1
> disk = ['phy:/dev/vg/ubuntu-hvm,hda,w']
> vnc = 1
> boot="c"
> 
> I have installed xen-tools of version 4.7 in this driver domain so that the vif-scirpts work. The network configuration here looks like the following where ens5f0 is the interface name for the NIC I did passthrough.
> 
> auto lo
> iface lo inet loopback
> 
> iface ens5f0 inet manual
> 
> auto xenbr1
> iface xenbr1 inet static
>         bridge_ports ens5f0
>         address 192.168.1.3
>         netmask 255.255.255.0
>         gateway 192.168.1.1

Probably irrelevant at this stage, but do you have "xendriverdomain"
service running in the backend?

> The Xen config file content for the DomU is as the following.
> 
> name = "ubuntu_on_ubuntu"
> bootloader = "/usr/lib/xen-4.9/bin/pygrub"
> memory = 1024
> vcpus = 1
> vif = [ 'backend=ubuntu-domnet-hvm,bridge=xenbr1' ]
> disk = [ '/dev/vg/lv_vm_ubuntu_guest,raw,xvda,rw' ]
> 
> When I try to launch this DomU, I get the following error.
> 
> libxl: error: libxl_nic.c:652:libxl__device_nic_set_devids: Domain 31:Unable to set nic defaults for nic 0.

I don't see what is wrong here. Are you sure the backend domain is
running?

> Are these configurations basically very different for what you do for Qubes? Please let me know your thoughts.

Looks similar, although we do that through libvirt.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

> I don't see what is wrong here. Are you sure the backend domain is running?
If you mean the HVM network driver domain then, Yes, I am running the backend domain.

> Probably irrelevant at this stage, but do you have "xendriverdomain" service running in the backend?
I do not have this service running. However, my PV network driver domain works fine, though this service is not running.

What version of Xen are you using that have xendriverdomain service?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 944 bytes --]

On Wed, Jan 22, 2020 at 07:50:13PM +0000, tosher 1 wrote:
> 
> > I don't see what is wrong here. Are you sure the backend domain is running?
> If you mean the HVM network driver domain then, Yes, I am running the backend domain.
> 
> > Probably irrelevant at this stage, but do you have "xendriverdomain" service running in the backend?
> I do not have this service running. However, my PV network driver domain works fine, though this service is not running.
> 
> What version of Xen are you using that have xendriverdomain service?

I know it works with Xen 4.8, 4.12, and 4.13, but I wouldn't expect
any issues in a version between them.

I'd try increasing verbosity (xl -vvv create ...) and looking if
something earlier didn't gone wrong.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

I wasn't able to make the HVM driver domain work even with the latest Xen version which is 4.14. I see the 'xendriverdomain' executable in the /etc/init.d/ directory, but it doesn't seem to be running in the background. 

On the other hand, I see the official "Qubes OS Architecture" document (https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf) defines the driver domain as the following.

"A driver domain is an unprivileged PV-domain that has been securely granted access to certain PCI device (e.g. the network card or disk controller) using Intel VT-d." - Page 12

Moreover, section 6.1 reads "The network domain is granted direct access to the networking hardware, e.g. the WiFi or ethernet card. Besides, it is a regular unprivileged PV domain."

Maybe you guys later moved to the HVM driver domain from PV. Would you please share the Xen config you use for the network driver domain?

Thanks,
Mehrab


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Marek Marczykowski-Górecki]

[-- Attachment #1.1.1: Type: text/plain, Size: 1510 bytes --]

On Thu, Jan 23, 2020 at 10:36:34PM +0000, tosher 1 wrote:
> 
> 
> I wasn't able to make the HVM driver domain work even with the latest Xen version which is 4.14. I see the 'xendriverdomain' executable in the /etc/init.d/ directory, but it doesn't seem to be running in the background. 
> 
> On the other hand, I see the official "Qubes OS Architecture" document (https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf) defines the driver domain as the following.
> 
> "A driver domain is an unprivileged PV-domain that has been securely granted access to certain PCI device (e.g. the network card or disk controller) using Intel VT-d." - Page 12
> 
> Moreover, section 6.1 reads "The network domain is granted direct access to the networking hardware, e.g. the WiFi or ethernet card. Besides, it is a regular unprivileged PV domain."
> 
> Maybe you guys later moved to the HVM driver domain from PV. Would you please share the Xen config you use for the network driver domain?

Yes, that PDF is quite outdated, we use HVM now.

As for the configs, as said before, we use libvirt, with some extra
patches, so the config won't be directly useful for you. I'm attaching
both libvirt XML for the driver domain, and also converted to XL (using
virsh domxml-to-native), may be inaccurate.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.1.2: sys-net.xl --]
[-- Type: text/plain, Size: 997 bytes --]

name = "sys-net"
uuid = "f8716f60-6be1-43b5-9bcb-b2c8c0910e5b"
maxmem = 400
memory = 400
vcpus = 2
pae = 1
acpi = 1
apic = 1
viridian = 1
rtc_timeoffset = 0
localtime = 1
on_poweroff = "destroy"
on_reboot = "destroy"
on_crash = "destroy"
sdl = 0
vnc = 1
vncunused = 0
vncdisplay = -5900
pci = [ "0000:00:1f.6", "0000:04:00.0" ]
parallel = "none"
serial = "none"
builder = "hvm"
cmdline = "root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 xen_scrub_pages=0 nopat iommu=soft swiotlb=8192"
boot = "dc"
nestedhvm = 0
disk = [ "format=raw,vdev=xvda,access=rw,backendtype=phy,target=/dev/qubes_dom0/vm-sys-net-root-snap", "format=raw,vdev=xvdb,access=rw,backendtype=phy,target=/dev/qubes_dom0/vm-sys-net-private-snap", "format=raw,vdev=xvdc,access=rw,backendtype=phy,target=/dev/qubes_dom0/vm-sys-net-volatile", "format=raw,vdev=xvdd,access=ro,backendtype=phy,target=/var/lib/qubes/vm-kernels/4.19.86-1/modules.img" ]
usb = 1
usbdevice = "tablet"


[-- Attachment #1.1.3: sys-net.xml --]
[-- Type: text/xml, Size: 2705 bytes --]

<domain type='xen' id='1'>
  <name>sys-net</name>
  <uuid>f8716f60-6be1-43b5-9bcb-b2c8c0910e5b</uuid>
  <memory unit='KiB'>409600</memory>
  <currentMemory unit='KiB'>409600</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='xenfv'>hvm</type>
    <loader type='rom'>hvmloader</loader>
    <cmdline>root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 xen_scrub_pages=0 nopat iommu=soft swiotlb=8192</cmdline>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <viridian/>
    <xen>
      <e820_host state='on'/>
    </xen>
  </features>
  <cpu mode='host-passthrough'>
    <feature policy='disable' name='vmx'/>
    <feature policy='disable' name='svm'/>
    <feature policy='disable' name='smap'/>
  </cpu>
  <clock offset='variable' adjustment='0' basis='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator type='stubdom-linux'/>
    <disk type='block' device='disk'>
      <driver name='phy' type='raw'/>
      <source dev='/dev/qubes_dom0/vm-sys-net-root-snap'/>
      <backingStore/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='phy' type='raw'/>
      <source dev='/dev/qubes_dom0/vm-sys-net-private-snap'/>
      <backingStore/>
      <target dev='xvdb' bus='xen'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='phy' type='raw'/>
      <source dev='/dev/qubes_dom0/vm-sys-net-volatile'/>
      <backingStore/>
      <target dev='xvdc' bus='xen'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='phy' type='raw'/>
      <source dev='/var/lib/qubes/vm-kernels/4.19.86-1/modules.img'/>
      <backingStore/>
      <target dev='xvdd' bus='xen'/>
      <readonly/>
    </disk>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='xen' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='qubes'/>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='xen'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1f' function='0x6'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='xen'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
  </devices>
</domain>


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Roger Pau Monné]

On Thu, Jan 23, 2020 at 10:36:34PM +0000, tosher 1 wrote:
> 
> 
> I wasn't able to make the HVM driver domain work even with the latest Xen version which is 4.14. I see the 'xendriverdomain' executable in the /etc/init.d/ directory, but it doesn't seem to be running in the background. 

The background process will be 'xl devd', not xendriverdomain, that's
just the init script name.

You can also start xl devd manually, as that will give you verbose
output of whats going on. In the driver domain:

# killall xl
# xl -vvv devd -F

That should give you detailed output of what's going on in the driver
domain, can you paste the output you get from the driver domain when
you try to start the failed guest?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Jason Andryuk]

On Wed, Jan 22, 2020 at 2:03 PM Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:
>
> On Wed, Jan 22, 2020 at 04:56:15PM +0000, tosher 1 wrote:
> > Hi Marek,
> >
> > Thanks for your response. The server machine I am using for this setup is an x86_64 Intel Xeon. For the Dom0, I am using Ubuntu 18.04 running on kernel version 5.0.0-37-generic. My Xen version is 4.9.2.
> >
> > For the HVM driver domain, I am using Ubuntu 18.04 running on kernel version 5.0.0-23-generic. I am doing a NIC PCI passthrough to this domain. The Xen config file for this domain looks like the following.
> >
> > builder = "hvm"
> > name = "ubuntu-doment-hvm"

This name...

> > memory = "2048"
> > pci = [ '01:00.0,permissive=1' ]
> > vcpus = 1
> > disk = ['phy:/dev/vg/ubuntu-hvm,hda,w']
> > vnc = 1
> > boot="c"

<snip>

> > name = "ubuntu_on_ubuntu"
> > bootloader = "/usr/lib/xen-4.9/bin/pygrub"
> > memory = 1024
> > vcpus = 1
> > vif = [ 'backend=ubuntu-domnet-hvm,bridge=xenbr1' ]

...and this name don't match.

Regards,
Jason

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

> > builder = "hvm"
> > name = "ubuntu-doment-hvm"

> This name...

> > vif = [ 'backend=ubuntu-domnet-hvm,bridge=xenbr1' ]

> ...and this name don't match.


Jason,

Thanks for pointing this out. I feel very stupid. However, the problem is not solved yet, but I was able to get to the next step with devd suggested by Roger. I will keep you in CC when I reply Roger with a detailed log.

Regards,
Mehrab


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

[-- Attachment #1.1: Type: text/plain, Size: 1694 bytes --]

 Marek,
Thanks for the config files.
Regards,Mehrab

    On Thursday, January 23, 2020, 6:11:12 PM EST, Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> wrote:  
 
 On Thu, Jan 23, 2020 at 10:36:34PM +0000, tosher 1 wrote:
> 
> 
> I wasn't able to make the HVM driver domain work even with the latest Xen version which is 4.14. I see the 'xendriverdomain' executable in the /etc/init.d/ directory, but it doesn't seem to be running in the background. 
> 
> On the other hand, I see the official "Qubes OS Architecture" document (https://www.qubes-os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf) defines the driver domain as the following.
> 
> "A driver domain is an unprivileged PV-domain that has been securely granted access to certain PCI device (e.g. the network card or disk controller) using Intel VT-d." - Page 12
> 
> Moreover, section 6.1 reads "The network domain is granted direct access to the networking hardware, e.g. the WiFi or ethernet card. Besides, it is a regular unprivileged PV domain."
> 
> Maybe you guys later moved to the HVM driver domain from PV. Would you please share the Xen config you use for the network driver domain?

Yes, that PDF is quite outdated, we use HVM now.

As for the configs, as said before, we use libvirt, with some extra
patches, so the config won't be directly useful for you. I'm attaching
both libvirt XML for the driver domain, and also converted to XL (using
virsh domxml-to-native), may be inaccurate.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?  

[-- Attachment #1.2: Type: text/html, Size: 3124 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

Rojer,

> You can also start xl devd manually, as that will give you verbose
> output of whats going on. In the driver domain:

> # killall xl
> # xl -vvv devd -F

> That should give you detailed output of what's going on in the driver
> domain, can you paste the output you get from the driver domain when
> you try to start the failed guest?

I ran both commands in the driver domain. Before starting the domU, I get the following verbose.

# sudo xl -vvv devd -F
libxl: debug: libxl_device.c:1733:libxl_device_events_handler: Domain 0:ao 0x556e3e940ef0: create: how=(nil) callback=(nil) poller=0x556e3e940c10
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x7ffca33549d8 wpath=/local/domain/0/backend token=3/0: register slotnum=3
libxl: debug: libxl_device.c:1790:libxl_device_events_handler: Domain 0:ao 0x556e3e940ef0: inprogress: poller=0x556e3e940c10, flags=i
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x7ffca33549d8 wpath=/local/domain/0/backend token=3/0: event epath=/local/domain/0/backend
libxl: debug: libxl_event.c:2227:libxl__nested_ao_create: ao 0x556e3e940600: nested ao, parent 0x556e3e940ef0
libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x556e3e940600: destroy

I know this is not exactly what you asked for. Unfortunately, I don't see any other verbose when try to start DomU. The error messages I get from the failed DomU launch is as the following, where driver domain id is 7.

libxl: error: libxl_device.c:1075:device_backend_callback: Domain 8:unable to add device with path /local/domain/7/backend/vif/8/0
libxl: error: libxl_create.c:1458:domcreate_attach_devices: Domain 8:unable to add nic devices
libxl: error: libxl_device.c:1075:device_backend_callback: Domain 8:unable to remove device with path /local/domain/7/backend/vif/8/0
libxl: error: libxl_domain.c:1075:devices_destroy_cb: Domain 8:libxl__devices_destroy failed
libxl: error: libxl_domain.c:1003:libxl__destroy_domid: Domain 8:Non-existant domain
libxl: error: libxl_domain.c:962:domain_destroy_callback: Domain 8:Unable to destroy guest
libxl: error: libxl_domain.c:889:domain_destroy_cb: Domain 8:Destruction of domain failed


On the other hand, if I run devd in Dom0, I get a lot of verbose when I try to launch DomU, dependent on Driver Domain for networking. I am not sure if I should paste it here. Please let me know what you think.

Thanks,
Mehrab


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Roger Pau Monné]

On Mon, Jan 27, 2020 at 08:43:21PM +0000, tosher 1 wrote:
> Rojer,
> 
> > You can also start xl devd manually, as that will give you verbose
> > output of whats going on. In the driver domain:
> 
> > # killall xl
> > # xl -vvv devd -F
> 
> > That should give you detailed output of what's going on in the driver
> > domain, can you paste the output you get from the driver domain when
> > you try to start the failed guest?
> 
> I ran both commands in the driver domain. Before starting the domU, I get the following verbose.
> 
> # sudo xl -vvv devd -F
> libxl: debug: libxl_device.c:1733:libxl_device_events_handler: Domain 0:ao 0x556e3e940ef0: create: how=(nil) callback=(nil) poller=0x556e3e940c10
> libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x7ffca33549d8 wpath=/local/domain/0/backend token=3/0: register slotnum=3
> libxl: debug: libxl_device.c:1790:libxl_device_events_handler: Domain 0:ao 0x556e3e940ef0: inprogress: poller=0x556e3e940c10, flags=i
> libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x7ffca33549d8 wpath=/local/domain/0/backend token=3/0: event epath=/local/domain/0/backend
> libxl: debug: libxl_event.c:2227:libxl__nested_ao_create: ao 0x556e3e940600: nested ao, parent 0x556e3e940ef0
> libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x556e3e940600: destroy

Are you sure this command is run on the driver domain?

The trace above shows 'xl devd' trying to get a watch on
/local/domain/0/backend which is the xenstore backend path of dom0.

BTW, are you creating the driver domain with 'driver_domain=1' in the
xl config file?

Which version of the tools are you using in the driver domain?

I'm quite sure 4.7 is too old, and 'xl devd' won't work properly in
that version.

> 
> I know this is not exactly what you asked for. Unfortunately, I don't see any other verbose when try to start DomU. The error messages I get from the failed DomU launch is as the following, where driver domain id is 7.
> 
> libxl: error: libxl_device.c:1075:device_backend_callback: Domain 8:unable to add device with path /local/domain/7/backend/vif/8/0
> libxl: error: libxl_create.c:1458:domcreate_attach_devices: Domain 8:unable to add nic devices
> libxl: error: libxl_device.c:1075:device_backend_callback: Domain 8:unable to remove device with path /local/domain/7/backend/vif/8/0
> libxl: error: libxl_domain.c:1075:devices_destroy_cb: Domain 8:libxl__devices_destroy failed
> libxl: error: libxl_domain.c:1003:libxl__destroy_domid: Domain 8:Non-existant domain
> libxl: error: libxl_domain.c:962:domain_destroy_callback: Domain 8:Unable to destroy guest
> libxl: error: libxl_domain.c:889:domain_destroy_cb: Domain 8:Destruction of domain failed

I'm afraid this log is not very helpful, would be better if you could
paste the full logs, as there might be hints of what's going on in
other parts of the file.

> 
> On the other hand, if I run devd in Dom0, I get a lot of verbose when I try to launch DomU, dependent on Driver Domain for networking. I am not sure if I should paste it here. Please let me know what you think.

No, 'xl devd' in dom0 is not useful, as it will interact badly with
the toolstack also running from dom0.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

 > BTW, are you creating the driver domain with 'driver_domain=1' in the xl config file?

No, I wasn't aware of the 'driver_domain' configuration option before, and this is what I was missing. With this configuration option, I was able to make the HVM driver domain work. However, the PV driver domain worked fine without this option configured.

> Are you sure this command is run on the driver domain?

Since I had installed xen-utils in the driver domain for the bridge to work, it installed Xen hypervisor in the driver domain. As a result, my driver domain became another Dom0. Realizing that I ran regular Ubuntu in the driver domain. This was another key point to make the driver domain work.

Thanks for all your help, which made it possible for me to test the HVM driver domain.

One last thing, backend interfaces are not being added to the bridge automatically.  Do you think it is because regular Ubuntu doesn't have Xen vif scripts? If yes, what is the proper thing to do in this case?

Please let me know.

Thanks,
Mehrab


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[Roger Pau Monné]

On Wed, Jan 29, 2020 at 10:43:07PM +0000, tosher 1 wrote:
>  > BTW, are you creating the driver domain with 'driver_domain=1' in the xl config file?
> 
> No, I wasn't aware of the 'driver_domain' configuration option before, and this is what I was missing. With this configuration option, I was able to make the HVM driver domain work. However, the PV driver domain worked fine without this option configured.
> 
> > Are you sure this command is run on the driver domain?
> 
> Since I had installed xen-utils in the driver domain for the bridge to work, it installed Xen hypervisor in the driver domain. As a result, my driver domain became another Dom0. Realizing that I ran regular Ubuntu in the driver domain. This was another key point to make the driver domain work.

Remove the Xen entry from grub in order to prevent it booting in dom0
mode, or remove the package that installs the Xen kernel if your
distro has separate packages for the Xen kernel and the tools.

It's pointless to boot a driver domain as a dom0, it's just going to
slow things down.

> Thanks for all your help, which made it possible for me to test the HVM driver domain.
> 
> One last thing, backend interfaces are not being added to the bridge automatically.  Do you think it is because regular Ubuntu doesn't have Xen vif scripts? If yes, what is the proper thing to do in this case?

'xl devd' should add the backend interfaces (vifX.Y) to the bridge if
properly configured, as it should be calling the hotplug scripts to do
that.

Can you paste the output you get from 'xl -vvv devd -F' when that
happens?

Are you sure the bridge setup in the guest config file exists in the
driver domain?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] HVM Driver Domain

[tosher 1]

> 'xl devd' should add the backend interfaces (vifX.Y) to the bridge if properly configured, as it should be calling the hotplug scripts to do that.

Yes, running ' xl devd' in the driver domain before launching the DomU, solved the bridge issue. Thanks a lot.

So, for the people who end up reading this thread, the following is a lesson I have learned. 

We need 'xl devd' running for the bridge to work, and to get the 'xl' program in the driver domain, depending on the distro being used, we may need to install xen-utils or other packages.  However, along with the xl, these packages may end up installing Xen hypervisor and update the grub accordingly. As a result, there is a chance that, in the next boot, the OS will boot into the hypervisor mode. In this case, the driver domain won't work. Therefore, be careful not to run the diver domain in Dom0 mode. Change the default boot to the regular Linux or delete the Xen image from /boot, and update the grub. After Booting the regular Linux, make sure the bridge is set up correctly, and  'xl devd' is running to have the network driver domain working.

Regards,
Mehrab



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel