Re: libxl: locally attaching disk

Re: libxl: locally attaching disk

[Thanos Makatos]

FYI modifying the "if (got == ds->wanted)" to "if (got >= ds->wanted)" in devstate_watch_callback (libxl_event.c) resolves the problem. Is this a correct solution or will it break stuff? (There are some checks missing).

> -----Original Message-----
> From: Thanos Makatos
> Sent: 14 May 2013 14:20
> To: 'xen-devel@lists.xen.org'
> Subject: libxl: locally attaching disk
> 
> I want to implement booting from VHD images using pygrub for blktap3.
> I'm trying to use the existing code for qemu: in
> libxl__device_disk_local_initiate_attach I tell libxl to do exactly the
> same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the switch
> statement (similar for libxl__device_disk_local_initiate_detach).
> 
> I run into the following problem: blkfront goes directly from state 1
> to state 3, the back-end follows by jumping to state 4, and finally the
> front-end goes to state 4 and everything works fine (this is what is
> done for domU guests using blktap3 without pygrub). However, it seems
> that libxl expects the backend to step through each state (specifically
> it times out waiting for the back-end to go to state 2 but the back-end
> has already gone to state 4). If I correctly understand the protocol
> specification in xen/include/public/io/blkif.h, libxl shouldn't be

Re: libxl: locally attaching disk

[Roger Pau Monné]

On 14/05/13 15:42, Thanos Makatos wrote:
> FYI modifying the "if (got == ds->wanted)" to "if (got >= ds->wanted)" in devstate_watch_callback (libxl_event.c) resolves the problem. Is this a correct solution or will it break stuff? (There are some checks missing).

I don't think this is the right solution, see my reply to the other thread.

Re: libxl: locally attaching disk

[Thanos Makatos]

> local_device_attach_cb (in libxl.c) already does wait for the device to
> reach state 4 (connected), in libxl__wait_device_connection you should
> just skip the wait for state 2.

Thanks Roger, that worked fine.

Re: libxl: locally attaching disk

[Roger Pau Monné]

On 14/05/13 18:17, Thanos Makatos wrote:
>>> I run into the following problem: blkfront goes directly from state 1
>> to state 3, the back-end follows by jumping to state 4, and finally the
>> front-end goes to state 4 and everything works fine (this is what is
>> done for domU guests using blktap3 without pygrub). However, it seems
>> that libxl expects the backend to step through each state (specifically
>> it times out waiting for the back-end to go to state 2 but the back-end
>> has already gone to state 4). If I correctly understand the protocol
>> specification in xen/include/public/io/blkif.h, libxl shouldn't be
>> doing that. Here's the output of libxl:
>>
>> Hello,
>>
>> In libxl we wait for the backend to reach state 2 because for blkback
>> backends we have to execute hotplug scripts. If you take a look at
>> libxl__wait_device_connection in libx_device.c you will see that if the
>> backend is Qemu, we skip the waiting and instead jump to device_hotplug
>> directly. Since blktap backend don't use hotplug scripts, and hence
>> don't wait in state 2 for device hotplug execution you should implement
>> something similar for blktap backends.
> 
> So if I get this straight, we shouldn't execute device_hotplug for blktap3 because we don't use hotplug scripts. Also, in the same function, we do need to execute libxl__ev_devstate_wait but setting the XenbusState to XenbusStateConnected, because lixbl must wait for blkfront to connect to tapdisk in order access the block device. Is this correct?

local_device_attach_cb (in libxl.c) already does wait for the device to
reach state 4 (connected), in libxl__wait_device_connection you should
just skip the wait for state 2.

Re: libxl: locally attaching disk

[Thanos Makatos]

> > I run into the following problem: blkfront goes directly from state 1
> to state 3, the back-end follows by jumping to state 4, and finally the
> front-end goes to state 4 and everything works fine (this is what is
> done for domU guests using blktap3 without pygrub). However, it seems
> that libxl expects the backend to step through each state (specifically
> it times out waiting for the back-end to go to state 2 but the back-end
> has already gone to state 4). If I correctly understand the protocol
> specification in xen/include/public/io/blkif.h, libxl shouldn't be
> doing that. Here's the output of libxl:
> 
> Hello,
> 
> In libxl we wait for the backend to reach state 2 because for blkback
> backends we have to execute hotplug scripts. If you take a look at
> libxl__wait_device_connection in libx_device.c you will see that if the
> backend is Qemu, we skip the waiting and instead jump to device_hotplug
> directly. Since blktap backend don't use hotplug scripts, and hence
> don't wait in state 2 for device hotplug execution you should implement
> something similar for blktap backends.

So if I get this straight, we shouldn't execute device_hotplug for blktap3 because we don't use hotplug scripts. Also, in the same function, we do need to execute libxl__ev_devstate_wait but setting the XenbusState to XenbusStateConnected, because lixbl must wait for blkfront to connect to tapdisk in order access the block device. Is this correct?

Re: libxl: locally attaching disk

[Roger Pau Monné]

On 14/05/13 15:20, Thanos Makatos wrote:
> I want to implement booting from VHD images using pygrub for blktap3. I'm trying to use the existing code for qemu: in libxl__device_disk_local_initiate_attach I tell libxl to do exactly the same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the switch statement (similar for libxl__device_disk_local_initiate_detach).
> 
> I run into the following problem: blkfront goes directly from state 1 to state 3, the back-end follows by jumping to state 4, and finally the front-end goes to state 4 and everything works fine (this is what is done for domU guests using blktap3 without pygrub). However, it seems that libxl expects the backend to step through each state (specifically it times out waiting for the back-end to go to state 2 but the back-end has already gone to state 4). If I correctly understand the protocol specification in xen/include/public/io/blkif.h, libxl shouldn't be doing that. Here's the output of libxl:

Hello,

In libxl we wait for the backend to reach state 2 because for blkback
backends we have to execute hotplug scripts. If you take a look at
libxl__wait_device_connection in libx_device.c you will see that if the
backend is Qemu, we skip the waiting and instead jump to device_hotplug
directly. Since blktap backend don't use hotplug scripts, and hence
don't wait in state 2 for device hotplug execution you should implement
something similar for blktap backends.

Re: libxl: locally attaching disk

[Ian Campbell]

On Tue, 2013-05-14 at 14:20 +0100, Thanos Makatos wrote:
> I want to implement booting from VHD images using pygrub for blktap3.
> I'm trying to use the existing code for qemu: in
> libxl__device_disk_local_initiate_attach I tell libxl to do exactly
> the same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the
> switch statement (similar for
> libxl__device_disk_local_initiate_detach).
> 
> I run into the following problem: blkfront goes directly from state 1
> to state 3, the back-end follows by jumping to state 4, and finally
> the front-end goes to state 4 and everything works fine (this is what
> is done for domU guests using blktap3 without pygrub). However, it
> seems that libxl expects the backend to step through each state
> (specifically it times out waiting for the back-end to go to state 2
> but the back-end has already gone to state 4). If I correctly
> understand the protocol specification in
> xen/include/public/io/blkif.h, libxl shouldn't be doing that.

Indeed, not just because the backend might skip a state but because even
if it did go through state 2 it might have gone to state 4 by the time
libxl processes the watch and reads the state.

Roger, do you have any ideas how to fix this? Usually xenstore
interactions wait for state >= N not just == N to handle this case.

Ian.

libxl: locally attaching disk

[Thanos Makatos]

I want to implement booting from VHD images using pygrub for blktap3. I'm trying to use the existing code for qemu: in libxl__device_disk_local_initiate_attach I tell libxl to do exactly the same thing it does for the LIBXL_DISK_BACKEND_QDISK case in the switch statement (similar for libxl__device_disk_local_initiate_detach).

I run into the following problem: blkfront goes directly from state 1 to state 3, the back-end follows by jumping to state 4, and finally the front-end goes to state 4 and everything works fine (this is what is done for domU guests using blktap3 without pygrub). However, it seems that libxl expects the backend to step through each state (specifically it times out waiting for the back-end to go to state 2 but the back-end has already gone to state 4). If I correctly understand the protocol specification in xen/include/public/io/blkif.h, libxl shouldn't be doing that. Here's the output of libxl:

libxl_create.c:1244:do_domain_create: ao 0xe79160: create: how=(nil) callback=(nil) poller=0xe77090
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3
libxl_create.c:691:initiate_domain_create: running bootloader
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap3
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3
libxl_event.c:559:libxl__ev_xswatch_register: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: register slotnum=3
libxl_create.c:1257:do_domain_create: ao 0xe79160: inprogress: poller=0xe77090, flags=i
libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 1
libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 4
libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 4
libxl_event.c:503:watchfd_callback: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 4
libxl_event.c:661:devstate_timeout: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2  timed out
libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0xe78130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: deregister slotnum=3
libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0xe78130: deregister unregistered
libxl_device.c:882:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd3/0/51712
libxl.c:2687:local_device_attach_cb: unable to add vbd with id 51712: Resource temporarily unavailable
libxl_bootloader.c:405:bootloader_disk_attached_cb: failed to attach local disk for bootloader execution

If I modify the back-end to go to the same state as the front-end one but making sure that this is done by writing all the intermediate state transitions in XenStore, libxl again complains because it cannot see the back-end state passing from state 2. Here's the output of libxl:

libxl_create.c:1244:do_domain_create: ao 0x1394160: create: how=(nil) callback=(nil) poller=0x1392090
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3
libxl_create.c:691:initiate_domain_create: running bootloader
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap3
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3
libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: register slotnum=3
libxl_create.c:1257:do_domain_create: ao 0x1394160: inprogress: poller=0x1392090, flags=i
libxl_event.c:503:watchfd_callback: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 1
libxl_event.c:503:watchfd_callback: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 3
libxl_event.c:503:watchfd_callback: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 3
libxl_event.c:661:devstate_timeout: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2  timed out
libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x1393130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: deregister slotnum=3
libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x1393130: deregister unregistered
libxl_device.c:882:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd3/0/51712
libxl.c:2687:local_device_attach_cb: unable to add vbd with id 51712: Resource temporarily unavailable
libxl_bootloader.c:405:bootloader_disk_attached_cb: failed to attach local disk for bootloader execution

If I modify the back-end to only step into the next state from its current one in trying to reach the front-end's state (e.g. if the back-end state is 1 and the front-end state is 3, the back-end will go to state 2), the back-end gets stuck in state 2, which does make libxl happy but the Xenbus protocol isn't advancing because there isn't any other event in XenStore such that the back-end can continue increasing its state and utterly match the front-end's one. Here's the output of libxl:

libxl_create.c:1244:do_domain_create: ao 0x194e160: create: how=(nil) callback=(nil) poller=0x194c090
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3
libxl_create.c:691:initiate_domain_create: running bootloader
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap3
libxl_device.c:246:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap3
libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: register slotnum=3
libxl_create.c:1257:do_domain_create: ao 0x194e160: inprogress: poller=0x194c090, flags=i
libxl_event.c:503:watchfd_callback: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 still waiting state 1
libxl_event.c:503:watchfd_callback: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: event epath=/local/domain/0/backend/vbd3/0/51712/state
libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd3/0/51712/state wanted state 2 ok
libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x194d130 wpath=/local/domain/0/backend/vbd3/0/51712/state token=3/0: deregister slotnum=3
libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x194d130: deregister unregistered
libxl.c:2692:local_device_attach_cb: locally attaching qdisk /dev/xvda
libxl_device.c:1108:libxl__wait_for_backend: Backend /local/domain/0/backend/vbd3/0/51712 not ready
libxl_bootloader.c:405:bootloader_disk_attached_cb: failed to attach local disk for bootloader execution

Any help appreciated.

--
Thanos Makatos