[BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Alex Braunegg]

Hi all,

I have been testing the Xen 4.6.0 packages - however I have found the
following issue with attempting to reboot a Windows VM:

--------------------------------------------------------------
[root@mynas-s5000xvn services]# /usr/sbin/xl reboot Windows_2008_R2   
Rebooting domain 5
PV control interface not available: external graceful reboot not possible.
Use "-F" to fallback to ACPI reset event.
reboot failed (rc=-10)

[root@mynas-s5000xvn services]# /usr/sbin/xl -F reboot Windows_2008_R2
/usr/sbin/xl: invalid option -- 'F'
unknown global option

[root@mynas-s5000xvn services]# /usr/sbin/xl reboot -F Windows_2008_R2
Rebooting domain 5
PV control interface not available: sending ACPI reset button event.
libxl: error: libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed:
Function not implemented
reboot failed (rc=-3)

[root@mynas-s5000xvn services]# /usr/sbin/xl reboot Windows_2008_R2 -F
Rebooting domain 5
PV control interface not available: external graceful reboot not possible.
Use "-F" to fallback to ACPI reset event.
reboot failed (rc=-10)

[root@mynas-s5000xvn services]#
--------------------------------------------------------------

The VM in question is a clean install of Windows 2008 R2. The same issue
occurs when attempting to use the 'shutdown' command.

If the PV Windows Drivers are installed - then this is not an issue - the
reboot / shutdown works flawlessly.

The use case here is that not all users will install the PV Windows drivers
- especially since the easy install & signed drivers (msi package) from
'ejbdigital' are no longer available


Best regards,

Alex

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Ian Campbell]

On Thu, 2016-02-04 at 08:50 +1100, Alex Braunegg wrote:
> [root@mynas-s5000xvn services]# /usr/sbin/xl reboot -F Windows_2008_R2
> Rebooting domain 5
> PV control interface not available: sending ACPI reset button event.
> libxl: error: libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed:
> Function not implemented
> reboot failed (rc=-3)

This is the one which should have worked, please can you run as:
    /usr/sbin/xl -vvv reboot -F Windows_2008_R2
and report the full, verbose, logs.

> The VM in question is a clean install of Windows 2008 R2. The same issue
> occurs when attempting to use the 'shutdown' command.

Please can you provide your guest configuration file.

Ian.

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Alex Braunegg]

Hi Ian,

Below is the output requested:

Guest Configuration:

-------------------------------------------------------------

builder='hvm'
memory = 2048
shadow_memory = 8
uuid = '27f4787c-89b2-46ab-a797-96ea6e84c511'
name = 'Windows_2008_R2'
vif = [ 'bridge=br0, mac=00:16:3e:96:49:10' ]
disk = [ '/dev/zvol/storage0/xen/Windows_2008_R2/disk_sda,,hda','/storage0/data-shares/iso/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso,,hdc,cdrom' ]
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot='dc'
sdl=0
vnc=1
vncconsole=1
vnclisten='0.0.0.0'
stdvga=1
serial='pty'
usbdevice='tablet'
vncpasswd='bpC7fKtsDy'
vncdisplay=2
localtime=1
audio='1'
soundhw='ac97'

-------------------------------------------------------------

Reboot Log:
===========

[root@mynas-s5000xvn /]# /usr/sbin/xl -vvv reboot -F Windows_2008_R2
Rebooting domain 9
PV control interface not available: sending ACPI reset button event.
libxl: error: libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented
reboot failed (rc=-3)
xc: debug: hypercall buffer: total allocations:18 total releases:18
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:12 misses:2 toobig:4
[root@mynas-s5000xvn /]#


Shutdown Log:
=============

[root@mynas-s5000xvn /]# /usr/sbin/xl -vvv shutdown -F Windows_2008_R2      
Shutting down domain 9
PV control interface not available: sending ACPI power button event.
xc: debug: hypercall buffer: total allocations:18 total releases:18
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:12 misses:2 toobig:4
[root@mynas-s5000xvn /]#


During this test, the shutdown of the guest system occurred - so potentially the shutdown command is working.



Best regards,

Alex



-----Original Message-----
From: Ian Campbell [mailto:ian.campbell@citrix.com] 
Sent: Thursday, 4 February 2016 9:58 PM
To: Alex Braunegg; xen-devel@lists.xen.org
Subject: Re: [Xen-devel] [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

On Thu, 2016-02-04 at 08:50 +1100, Alex Braunegg wrote:
> [root@mynas-s5000xvn services]# /usr/sbin/xl reboot -F Windows_2008_R2
> Rebooting domain 5
> PV control interface not available: sending ACPI reset button event.
> libxl: error: libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed:
> Function not implemented
> reboot failed (rc=-3)

This is the one which should have worked, please can you run as:
    /usr/sbin/xl -vvv reboot -F Windows_2008_R2
and report the full, verbose, logs.

> The VM in question is a clean install of Windows 2008 R2. The same issue
> occurs when attempting to use the 'shutdown' command.

Please can you provide your guest configuration file.

Ian.

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Ian Campbell]

On Fri, 2016-02-05 at 10:08 +1100, Alex Braunegg wrote:

I'm copying the other tools maintainers as well as the x86 guys.

> Hi Ian,
> 
> Below is the output requested:
> 
> Guest Configuration:
> 
> -------------------------------------------------------------
> 
> builder='hvm'
> memory = 2048
> shadow_memory = 8
> uuid = '27f4787c-89b2-46ab-a797-96ea6e84c511'
> name = 'Windows_2008_R2'
> vif = [ 'bridge=br0, mac=00:16:3e:96:49:10' ]
> disk = [
> '/dev/zvol/storage0/xen/Windows_2008_R2/disk_sda,,hda','/storage0/data-
> shares/iso/en_windows_server_2008_r2_standard_enterprise_datacenter_and_w
> eb_x64_dvd_x15-59754.iso,,hdc,cdrom' ]
> # boot on floppy (a), hard disk (c) or CD-ROM (d)
> # default: hard disk, cd-rom, floppy
> boot='dc'
> sdl=0
> vnc=1
> vncconsole=1
> vnclisten='0.0.0.0'
> stdvga=1
> serial='pty'
> usbdevice='tablet'
> vncpasswd='bpC7fKtsDy'

You might want to change this now. ;-)


> vncdisplay=2
> localtime=1
> audio='1'
> soundhw='ac97'
> 
> -------------------------------------------------------------
> 
> Reboot Log:
> ===========
> 
> [root@mynas-s5000xvn /]# /usr/sbin/xl -vvv reboot -F Windows_2008_R2
> Rebooting domain 9
> PV control interface not available: sending ACPI reset button event.
> libxl: error: libxl.c:5947:libxl_send_trigger: Send trigger 'reset'
> failed: Function not implemented
> reboot failed (rc=-3)

-3 here is libxl's ERROR_FAIL, which is a bit non-informative (and not to
be confused, as I just did, with an ESRCH error from the hypervisor in this
context).

However the "Function not implemented" tells us, I think, that the
hypervisor has returned ENOSYS.

The xl reboot command results in
    libxl_send_trigger(ctx, dom, LIBXL_TRIGGER_RESET, 0)

which becomes a XEN_DOMCTL_sendtrigger hypercall with the
XEN_DOMCTL_SENDTRIGGER_RESET option.

However the hypervisor doesn't do anything with this event, and just
returns ENOSYS.

The only triggers which Xen supports injecting are NMI, ACPI POWER and ACPI
SLEEP. I don't know if this is due to ACPI only having those as a concept,
or that Xen just doesn't implement ACPI RESET for some reason.

In any case it seems like a bug in the toolstack to me that it offers the
tantalising -F option to xl reboot when there appears to be no possibility
of it working. In particular the man page says:

    If PV drivers are not present but you have configured the guest OS
    to behave appropriately you may be able to use the I<-F> option
    trigger a reset button press.

Which AFAICT is all lies, I can only apologise for this :-/

> xc: debug: hypercall buffer: total allocations:18 total releases:18
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
> xc: debug: hypercall buffer: cache current size:2
> xc: debug: hypercall buffer: cache hits:12 misses:2 toobig:4
> [root@mynas-s5000xvn /]#
> 
> 
> Shutdown Log:
> =============
> 
> [root@mynas-s5000xvn /]# /usr/sbin/xl -vvv shutdown -F
> Windows_2008_R2      
> Shutting down domain 9
> PV control interface not available: sending ACPI power button event.
> xc: debug: hypercall buffer: total allocations:18 total releases:18
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
> xc: debug: hypercall buffer: cache current size:2
> xc: debug: hypercall buffer: cache hits:12 misses:2 toobig:4
> [root@mynas-s5000xvn /]#
> 
> 
> During this test, the shutdown of the guest system occurred - so
> potentially the shutdown command is working.

Yes, it seems that it is and should.

Irrespective of all the stuff above about reboot I think that without PV
drivers the only thing Xen can do to the guest is send it an ACPI power
button event, which it is up to the guest to interpret as reboot, power
off, hibernate, etc, I presume this can be configured somewhere in the
depths of the Windows control centre somewhere.

I'd strongly recommend installing some PV drivers into the guest, not only
will you gain reboot control etc but the I/O performance will be much
better.

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Alex Braunegg]

Many thanks for the detailed information and analysis - so potentially chasing something that cannot occur to a guest if the Windows PV Drivers are not installed.

> You might want to change this now. ;-)

No issue there - the guest was destroyed after grabbing the debug details. The full .cfg might be useful for someone else down the track.

> I'd strongly recommend installing some PV drivers into the guest, 

So this leads to another question. Is there a mechanism that exists today that allows via xl or some other Xen tool to 'probe' the guest as to if the PV drivers are loaded?

If not - it appears that:

	xc_hvm_param_get(ctx->xch, domid, HVM_PARAM_CALLBACK_IRQ, &pvdriver);

would get a return value that would equate to if the PV drivers are loaded / PV is able to be communicated with.

In list_vm(void), something such as (as part of the cmd line 'xl vm-list':

	uint64_t pvdriver = 0;
	int pv_available;
	pv_available = xc_hvm_param_get(ctx, info[i].domid, HVM_PARAM_CALLBACK_IRQ, &pvdriver);

to extend:	

	printf(LIBXL_UUID_FMT "  %d    %-30s\t%d\n", LIBXL_UUID_BYTES(info[i].uuid),info[i].domid, domname, pv_available);

Including ' xenctrl.h' in xl_cmdimpl.c would also be needed so that the function 'xc_hvm_param_get' could be used.

I am yet to test / try this but any suggestions is greatly appreciated.

Best regards,

Alex

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Ian Campbell]

On Wed, 2016-02-10 at 11:51 +1100, Alex Braunegg wrote:
> Many thanks for the detailed information and analysis - so potentially
> chasing something that cannot occur to a guest if the Windows PV Drivers
> are not installed.
> 
> > You might want to change this now. ;-)
> 
> No issue there - the guest was destroyed after grabbing the debug
> details. The full .cfg might be useful for someone else down the track.
> 
> > I'd strongly recommend installing some PV drivers into the guest, 
> 
> So this leads to another question. Is there a mechanism that exists today
> that allows via xl or some other Xen tool to 'probe' the guest as to if
> the PV drivers are loaded?

There is a function internal to libxl (libxl__domain_pvcontrol) which
answers this question (by looking at the callback IRQ).

I suppose it would be reasonable to expose that formally in the libxl
public API in some form, although it is also the case that any function
which uses PV control mechanisms performs (or at least should perform) this
check and return ERROR_NOPARAVIRT if pv control is not avaiable, so
application code can try PV and fallback to other methods (as xl does).

[...]
> Including ' xenctrl.h' in xl_cmdimpl.c would also be needed so that the
> function 'xc_hvm_param_get' could be used.

libxc (xenctrl.h) is not a stable interface and it is a design goal of
libxl that toolstacks should only need to use libxl interfaces for their
toolstack oriented operations. So this functionality would need to be
reflected in the stable libxl API for use by xl (and libvirt, and other
toolstacks people might write)

Ian.

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Alex Braunegg]

Hi Ian,

> There is a function internal to libxl (libxl__domain_pvcontrol) which
> answers this question (by looking at the callback IRQ).

I looked at this function yesterday and it appears that it calls:

	ret = libxl__domain_pvcontrol_available(gc, domid);

which in turn calls:

	ret = xc_hvm_param_get(ctx->xch, domid, HVM_PARAM_CALLBACK_IRQ, &pvdriver);

which is defined in xenctrl.h, this is why I thought it may have been easier to call "xc_hvm_param_get" directly inside the 'xl vm-list' function with the applicable header applied.

> I suppose it would be reasonable to expose that formally in the libxl
> public API in some form, although it is also the case that any function
> which uses PV control mechanisms performs (or at least should perform) this
> check and return ERROR_NOPARAVIRT if pv control is not avaiable, so
> application code can try PV and fallback to other methods (as xl does).

> libxc (xenctrl.h) is not a stable interface and it is a design goal of
> libxl that toolstacks should only need to use libxl interfaces for their
> toolstack oriented operations. So this functionality would need to be
> reflected in the stable libxl API for use by xl (and libvirt, and other
> toolstacks people might write)

So potentially before trying to make use of libxl__domain_pvcontrol for the 'xl vm-list' command to provide feedback if PV control is available, I will need to wait for the functionality to be exposed via an applicable libxl header.

Best regards,

Alex

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Ian Campbell]

On Thu, 2016-02-11 at 09:30 +1100, Alex Braunegg wrote:
> So potentially before trying to make use of libxl__domain_pvcontrol for
> the 'xl vm-list' command to provide feedback if PV control is available,
> I will need to wait for the functionality to be exposed via an applicable
> libxl header.

Yes, exposing an internal function in the public API generally involves:

 * renaming into the libxl_foo rather than libxl__foo namespace and moving
   the prototype to libxl.h. Perhaps giving it a more "public" API friendly
   name and checking that the API is sane enough to be maintained in an API
   compatible manner going forward (this one seems simple enough to be
   sane).
 * Adjusting the function to take a libxl_ctx *ctx instead of a libxl__gc
   *gc, which involves adding GC_INIT and GC_FREE in the function itself.
   This will require ensuring the function has a single exit path which
   calls GC_FREE (see also tools/libxl/CODING_STYLE)
 * Adding a LIBXL_HAVE... to libxl.h to indicate to 3rd party applications
   that this functionality is available (nothing in xen.git should be
   conditional on this #define though)
 * Updating xl to use this new functionality

Patches welcome.

Ian.

Re: [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

[Alex Braunegg]

Hi Ian,

Any update in looking at this issue?

Best regards,

Alex

-----Original Message-----
From: Alex Braunegg [mailto:alex.braunegg@gmail.com] 
Sent: Friday, 5 February 2016 10:09 AM
To: 'Ian Campbell'; 'xen-devel@lists.xen.org'
Subject: RE: [Xen-devel] [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

Hi Ian,

Below is the output requested:

Guest Configuration:

-------------------------------------------------------------

builder='hvm'
memory = 2048
shadow_memory = 8
uuid = '27f4787c-89b2-46ab-a797-96ea6e84c511'
name = 'Windows_2008_R2'
vif = [ 'bridge=br0, mac=00:16:3e:96:49:10' ]
disk = [ '/dev/zvol/storage0/xen/Windows_2008_R2/disk_sda,,hda','/storage0/data-shares/iso/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso,,hdc,cdrom' ]
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot='dc'
sdl=0
vnc=1
vncconsole=1
vnclisten='0.0.0.0'
stdvga=1
serial='pty'
usbdevice='tablet'
vncpasswd='bpC7fKtsDy'
vncdisplay=2
localtime=1
audio='1'
soundhw='ac97'

-------------------------------------------------------------

Reboot Log:
===========

[root@mynas-s5000xvn /]# /usr/sbin/xl -vvv reboot -F Windows_2008_R2
Rebooting domain 9
PV control interface not available: sending ACPI reset button event.
libxl: error: libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented
reboot failed (rc=-3)
xc: debug: hypercall buffer: total allocations:18 total releases:18
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:12 misses:2 toobig:4
[root@mynas-s5000xvn /]#


Shutdown Log:
=============

[root@mynas-s5000xvn /]# /usr/sbin/xl -vvv shutdown -F Windows_2008_R2      
Shutting down domain 9
PV control interface not available: sending ACPI power button event.
xc: debug: hypercall buffer: total allocations:18 total releases:18
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:12 misses:2 toobig:4
[root@mynas-s5000xvn /]#


During this test, the shutdown of the guest system occurred - so potentially the shutdown command is working.



Best regards,

Alex



-----Original Message-----
From: Ian Campbell [mailto:ian.campbell@citrix.com] 
Sent: Thursday, 4 February 2016 9:58 PM
To: Alex Braunegg; xen-devel@lists.xen.org
Subject: Re: [Xen-devel] [BUG} libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed: Function not implemented

On Thu, 2016-02-04 at 08:50 +1100, Alex Braunegg wrote:
> [root@mynas-s5000xvn services]# /usr/sbin/xl reboot -F Windows_2008_R2
> Rebooting domain 5
> PV control interface not available: sending ACPI reset button event.
> libxl: error: libxl.c:5947:libxl_send_trigger: Send trigger 'reset' failed:
> Function not implemented
> reboot failed (rc=-3)

This is the one which should have worked, please can you run as:
    /usr/sbin/xl -vvv reboot -F Windows_2008_R2
and report the full, verbose, logs.

> The VM in question is a clean install of Windows 2008 R2. The same issue
> occurs when attempting to use the 'shutdown' command.

Please can you provide your guest configuration file.

Ian.