block-iscsi with Xen 4.5 / 4.6

block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

Hi all,

I'm wading through the somewhat confusing world of documentation 
regarding storing DomU disk images on an iSCSI target.

I'm getting an error when using pygrub of:
OSError: [Errno 2] No such file or directory: 
'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'

After much hunting, I came across this post:
http://lists.xen.org/archives/html/xen-devel/2013-04/msg02796.html

As such, I'm wondering if it is still required to *NOT* use pygrub for 
booting iSCSI DomUs?

If so, what are the alternatives? Using pv-grub / pv-grub2? Something 
else?

As I'm running EL7.2, I figure if I have to use a pv-grub based 
solution, it would have to be pv-grub2?

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

On Fri, Apr 15, 2016 at 04:59:11PM +1000, Steven Haigh wrote:
> Hi all,
> 
> I'm wading through the somewhat confusing world of documentation regarding
> storing DomU disk images on an iSCSI target.
> 
> I'm getting an error when using pygrub of:
> OSError: [Errno 2] No such file or directory: 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'

Hello,

It should work. Can you please paste your guest configuration file and the 
output of the create command with "-vvv"?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

On 2016-04-15 17:23, Roger Pau Monné wrote:
> On Fri, Apr 15, 2016 at 04:59:11PM +1000, Steven Haigh wrote:
>> Hi all,
>> 
>> I'm wading through the somewhat confusing world of documentation 
>> regarding
>> storing DomU disk images on an iSCSI target.
>> 
>> I'm getting an error when using pygrub of:
>> OSError: [Errno 2] No such file or directory: 
>> 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
> 
> Hello,
> 
> It should work. Can you please paste your guest configuration file and 
> the
> output of the create command with "-vvv"?

DomU config file:
bootloader      = "pygrub"
name            = "test1.vm"
memory          = 2048
vcpus           = 2
cpus            = "1-7"
vif             = ['bridge=br-151, vifname=vm.test1']
disk            = 
['script=block-iscsi,vdev=xvda,target=iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250']
boot            = "c"

# xl create /etc/xen/test1.vm -d -c
Parsing config from /etc/xen/test1.vm
{
     "domid": null,
     "config": {
         "c_info": {
             "type": "pv",
             "name": "test1.vm",
             "uuid": "a7134f81-4616-4cf6-99db-3d2bc90b2d58",
             "run_hotplug_scripts": "True"
         },
         "b_info": {
             "max_vcpus": 2,
             "avail_vcpus": [
                 0,
                 1
             ],
             "vcpu_hard_affinity": [
                 [
                     1,
                     2,
                     3,
                     4,
                     5,
                     6,
                     7
                 ],
                 [
                     1,
                     2,
                     3,
                     4,
                     5,
                     6,
                     7
                 ]
             ],
             "numa_placement": "False",
             "max_memkb": 2097152,
             "target_memkb": 2097152,
             "shadow_memkb": 18432,
             "sched_params": {

             },
             "claim_mode": "True",
             "type.pv": {
                 "bootloader": "pygrub"
             }
         },
         "disks": [
             {
                 "pdev_path": 
"iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250",
                 "vdev": "xvda",
                 "format": "raw",
                 "script": "block-iscsi",
                 "readwrite": 1
             }
         ],
         "nics": [
             {
                 "devid": 0,
                 "bridge": "br-151",
                 "ifname": "vm.test1"
             }
         ],
         "on_reboot": "restart"
     }
}
libxl: error: libxl_bootloader.c:630:bootloader_finished: bootloader 
failed - consult logfile /var/log/xen/bootloader.11.log
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader 
[-1] exited with error status 1
libxl: error: libxl_create.c:1121:domcreate_rebuild_done: cannot 
(re-)build domain: -3
libxl: info: libxl.c:1698:devices_destroy_cb: forked pid 2982 for 
destroy of domain 11
libxl: error: libxl_dom.c:36:libxl__domain_type: unable to get domain 
type for domid=11
xl: unable to exec console client: No such file or directory
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: console 
child [2981] exited with error status 1

# cat /var/log/xen/bootloader.11.log
Traceback (most recent call last):
   File "/usr/lib/xen/bin/pygrub", line 894, in <module>
     part_offs = get_partition_offsets(file)
   File "/usr/lib/xen/bin/pygrub", line 114, in get_partition_offsets
     image_type = identify_disk_image(file)
   File "/usr/lib/xen/bin/pygrub", line 57, in identify_disk_image
     fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory: 
'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

On Fri, Apr 15, 2016 at 05:28:12PM +1000, Steven Haigh wrote:
> On 2016-04-15 17:23, Roger Pau Monné wrote:
> > On Fri, Apr 15, 2016 at 04:59:11PM +1000, Steven Haigh wrote:
> > > Hi all,
> > > 
> > > I'm wading through the somewhat confusing world of documentation
> > > regarding
> > > storing DomU disk images on an iSCSI target.
> > > 
> > > I'm getting an error when using pygrub of:
> > > OSError: [Errno 2] No such file or directory: 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
> > 
> > Hello,
> > 
> > It should work. Can you please paste your guest configuration file and
> > the
> > output of the create command with "-vvv"?
> 
> DomU config file:
> bootloader      = "pygrub"
> name            = "test1.vm"
> memory          = 2048
> vcpus           = 2
> cpus            = "1-7"
> vif             = ['bridge=br-151, vifname=vm.test1']
> disk            = ['script=block-iscsi,vdev=xvda,target=iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250']
> boot            = "c"
> 
> # xl create /etc/xen/test1.vm -d -c

Please post the output of xl -vvv create /etc/xen/test1.vm.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

On 2016-04-15 17:46, Roger Pau Monné wrote:
> On Fri, Apr 15, 2016 at 05:28:12PM +1000, Steven Haigh wrote:
>> On 2016-04-15 17:23, Roger Pau Monné wrote:
>> > On Fri, Apr 15, 2016 at 04:59:11PM +1000, Steven Haigh wrote:
>> > > Hi all,
>> > >
>> > > I'm wading through the somewhat confusing world of documentation
>> > > regarding
>> > > storing DomU disk images on an iSCSI target.
>> > >
>> > > I'm getting an error when using pygrub of:
>> > > OSError: [Errno 2] No such file or directory: 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
>> >
>> > Hello,
>> >
>> > It should work. Can you please paste your guest configuration file and
>> > the
>> > output of the create command with "-vvv"?
>> 
>> DomU config file:
>> bootloader      = "pygrub"
>> name            = "test1.vm"
>> memory          = 2048
>> vcpus           = 2
>> cpus            = "1-7"
>> vif             = ['bridge=br-151, vifname=vm.test1']
>> disk            = 
>> ['script=block-iscsi,vdev=xvda,target=iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250']
>> boot            = "c"
>> 
>> # xl create /etc/xen/test1.vm -d -c
> 
> Please post the output of xl -vvv create /etc/xen/test1.vm.

Whoops - apologies:

# xl -vvv create /etc/xen/test1.vm
Parsing config from /etc/xen/test1.vm
libxl: debug: libxl_create.c:1507:do_domain_create: ao 0x20b7260: 
create: how=(nil) callback=(nil) poller=0x20b6b30
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk 
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, uses 
script=... assuming phy backend
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk 
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:907:initiate_domain_create: running 
bootloader
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk 
vdev=(null) spec.backend=phy
libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), 
uses script=... assuming phy backend
libxl: debug: libxl.c:3064:libxl__device_disk_local_initiate_attach: 
locally attaching PHY disk 
iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250
libxl: debug: libxl_bootloader.c:411:bootloader_disk_attached_cb: Config 
bootloader value: pygrub
libxl: debug: libxl_bootloader.c:427:bootloader_disk_attached_cb: 
Checking for bootloader in libexec path: /usr/lib/xen/bin/pygrub
libxl: debug: libxl_create.c:1523:do_domain_create: ao 0x20b7260: 
inprogress: poller=0x20b6b30, flags=i
libxl: debug: libxl_event.c:581:libxl__ev_xswatch_register: watch 
w=0x20b7a30 wpath=/local/domain/12 token=3/0: register slotnum=3
libxl: debug: libxl_event.c:1950:libxl__ao_progress_report: ao 
0x20b7260: progress report: ignored
libxl: debug: libxl_bootloader.c:537:bootloader_gotptys: executing 
bootloader: /usr/lib/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:541:bootloader_gotptys:   bootloader 
arg: /usr/lib/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:541:bootloader_gotptys:   bootloader 
arg: --output=/var/run/xen/bootloader.12.out
libxl: debug: libxl_bootloader.c:541:bootloader_gotptys:   bootloader 
arg: --output-format=simple0
libxl: debug: libxl_bootloader.c:541:bootloader_gotptys:   bootloader 
arg: --output-directory=/var/run/xen/bootloader.12.d
libxl: debug: libxl_bootloader.c:541:bootloader_gotptys:   bootloader 
arg: 
iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250
libxl: debug: libxl_event.c:518:watchfd_callback: watch w=0x20b7a30 
wpath=/local/domain/12 token=3/0: event epath=/local/domain/12
libxl: error: libxl_bootloader.c:630:bootloader_finished: bootloader 
failed - consult logfile /var/log/xen/bootloader.12.log
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader 
[-1] exited with error status 1
libxl: debug: libxl_event.c:619:libxl__ev_xswatch_deregister: watch 
w=0x20b7a30 wpath=/local/domain/12 token=3/0: deregister slotnum=3
libxl: error: libxl_create.c:1121:domcreate_rebuild_done: cannot 
(re-)build domain: -3
libxl: info: libxl.c:1698:devices_destroy_cb: forked pid 3003 for 
destroy of domain 12
libxl: debug: libxl_event.c:1774:libxl__ao_complete: ao 0x20b7260: 
complete, rc=-3
libxl: debug: libxl_event.c:1746:libxl__ao__destroy: ao 0x20b7260: 
destroy
xc: debug: hypercall buffer: total allocations:41 total releases:41
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:30 misses:2 toobig:9

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

On Fri, Apr 15, 2016 at 05:48:24PM +1000, Steven Haigh wrote:
> On 2016-04-15 17:46, Roger Pau Monné wrote:
> > On Fri, Apr 15, 2016 at 05:28:12PM +1000, Steven Haigh wrote:
> > > On 2016-04-15 17:23, Roger Pau Monné wrote:
> > > > On Fri, Apr 15, 2016 at 04:59:11PM +1000, Steven Haigh wrote:
> > > > > Hi all,
> > > > >
> > > > > I'm wading through the somewhat confusing world of documentation
> > > > > regarding
> > > > > storing DomU disk images on an iSCSI target.
> > > > >
> > > > > I'm getting an error when using pygrub of:
> > > > > OSError: [Errno 2] No such file or directory: 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
> > > >
> > > > Hello,
> > > >
> > > > It should work. Can you please paste your guest configuration file and
> > > > the
> > > > output of the create command with "-vvv"?
> > > 
> > > DomU config file:
> > > bootloader      = "pygrub"
> > > name            = "test1.vm"
> > > memory          = 2048
> > > vcpus           = 2
> > > cpus            = "1-7"
> > > vif             = ['bridge=br-151, vifname=vm.test1']
> > > disk            = ['script=block-iscsi,vdev=xvda,target=iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250']
> > > boot            = "c"
> > > 
> > > # xl create /etc/xen/test1.vm -d -c
> > 
> > Please post the output of xl -vvv create /etc/xen/test1.vm.
> 
> Whoops - apologies:
> 
> # xl -vvv create /etc/xen/test1.vm
> Parsing config from /etc/xen/test1.vm
> libxl: debug: libxl_create.c:1507:do_domain_create: ao 0x20b7260: create:
> how=(nil) callback=(nil) poller=0x20b6b30
> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> vdev=xvda spec.backend=unknown
> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, uses
> script=... assuming phy backend
> libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
> vdev=xvda, using backend phy
> libxl: debug: libxl_create.c:907:initiate_domain_create: running bootloader
> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> vdev=(null) spec.backend=phy
> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), uses
> script=... assuming phy backend
> libxl: debug: libxl.c:3064:libxl__device_disk_local_initiate_attach: locally
> attaching PHY disk iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250

Now I remember, this was fixed not long ago, you will need to apply 
b1882a424ae098d722b19086b16e64b9aeccc7ca to your source tree/package in 
order to get pygrub working with hotplug scripts [0].

I guess you are using Xen 4.5, because this commit is already present in Xen 
4.6, and it should fix your issue.

Roger.

[0] http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=b1882a424ae098d722b19086b16e64b9aeccc7ca

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

On 2016-04-15 18:03, Roger Pau Monné wrote:
> On Fri, Apr 15, 2016 at 05:48:24PM +1000, Steven Haigh wrote:
>> On 2016-04-15 17:46, Roger Pau Monné wrote:
>> > On Fri, Apr 15, 2016 at 05:28:12PM +1000, Steven Haigh wrote:
>> > > On 2016-04-15 17:23, Roger Pau Monné wrote:
>> > > > On Fri, Apr 15, 2016 at 04:59:11PM +1000, Steven Haigh wrote:
>> > > > > Hi all,
>> > > > >
>> > > > > I'm wading through the somewhat confusing world of documentation
>> > > > > regarding
>> > > > > storing DomU disk images on an iSCSI target.
>> > > > >
>> > > > > I'm getting an error when using pygrub of:
>> > > > > OSError: [Errno 2] No such file or directory: 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
>> > > >
>> > > > Hello,
>> > > >
>> > > > It should work. Can you please paste your guest configuration file and
>> > > > the
>> > > > output of the create command with "-vvv"?
>> > >
>> > > DomU config file:
>> > > bootloader      = "pygrub"
>> > > name            = "test1.vm"
>> > > memory          = 2048
>> > > vcpus           = 2
>> > > cpus            = "1-7"
>> > > vif             = ['bridge=br-151, vifname=vm.test1']
>> > > disk            = ['script=block-iscsi,vdev=xvda,target=iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250']
>> > > boot            = "c"
>> > >
>> > > # xl create /etc/xen/test1.vm -d -c
>> >
>> > Please post the output of xl -vvv create /etc/xen/test1.vm.
>> 
>> Whoops - apologies:
>> 
>> # xl -vvv create /etc/xen/test1.vm
>> Parsing config from /etc/xen/test1.vm
>> libxl: debug: libxl_create.c:1507:do_domain_create: ao 0x20b7260: 
>> create:
>> how=(nil) callback=(nil) poller=0x20b6b30
>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>> vdev=xvda spec.backend=unknown
>> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, 
>> uses
>> script=... assuming phy backend
>> libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
>> vdev=xvda, using backend phy
>> libxl: debug: libxl_create.c:907:initiate_domain_create: running 
>> bootloader
>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>> vdev=(null) spec.backend=phy
>> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), 
>> uses
>> script=... assuming phy backend
>> libxl: debug: libxl.c:3064:libxl__device_disk_local_initiate_attach: 
>> locally
>> attaching PHY disk 
>> iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250
> 
> Now I remember, this was fixed not long ago, you will need to apply
> b1882a424ae098d722b19086b16e64b9aeccc7ca to your source tree/package in
> order to get pygrub working with hotplug scripts [0].
> 
> I guess you are using Xen 4.5, because this commit is already present 
> in Xen
> 4.6, and it should fix your issue.
> 
> [0]
> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=b1882a424ae098d722b19086b16e64b9aeccc7ca

Ahhh - thanks for the pointer. As this is a dev system, its probably 
easier for me to upgrade it to Xen 4.6 - however I'll take that commit 
and look at adding it to my Xen 4.5 packages for public consumption.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

On 2016-04-15 18:11, Steven Haigh wrote:
> On 2016-04-15 18:03, Roger Pau Monné wrote:
>> On Fri, Apr 15, 2016 at 05:48:24PM +1000, Steven Haigh wrote:
>>> On 2016-04-15 17:46, Roger Pau Monné wrote:
>>> > On Fri, Apr 15, 2016 at 05:28:12PM +1000, Steven Haigh wrote:
>>> > > On 2016-04-15 17:23, Roger Pau Monné wrote:
>>> > > > On Fri, Apr 15, 2016 at 04:59:11PM +1000, Steven Haigh wrote:
>>> > > > > Hi all,
>>> > > > >
>>> > > > > I'm wading through the somewhat confusing world of documentation
>>> > > > > regarding
>>> > > > > storing DomU disk images on an iSCSI target.
>>> > > > >
>>> > > > > I'm getting an error when using pygrub of:
>>> > > > > OSError: [Errno 2] No such file or directory: 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
>>> > > >
>>> > > > Hello,
>>> > > >
>>> > > > It should work. Can you please paste your guest configuration file and
>>> > > > the
>>> > > > output of the create command with "-vvv"?
>>> > >
>>> > > DomU config file:
>>> > > bootloader      = "pygrub"
>>> > > name            = "test1.vm"
>>> > > memory          = 2048
>>> > > vcpus           = 2
>>> > > cpus            = "1-7"
>>> > > vif             = ['bridge=br-151, vifname=vm.test1']
>>> > > disk            = ['script=block-iscsi,vdev=xvda,target=iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250']
>>> > > boot            = "c"
>>> > >
>>> > > # xl create /etc/xen/test1.vm -d -c
>>> >
>>> > Please post the output of xl -vvv create /etc/xen/test1.vm.
>>> 
>>> Whoops - apologies:
>>> 
>>> # xl -vvv create /etc/xen/test1.vm
>>> Parsing config from /etc/xen/test1.vm
>>> libxl: debug: libxl_create.c:1507:do_domain_create: ao 0x20b7260: 
>>> create:
>>> how=(nil) callback=(nil) poller=0x20b6b30
>>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>>> vdev=xvda spec.backend=unknown
>>> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, 
>>> uses
>>> script=... assuming phy backend
>>> libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
>>> vdev=xvda, using backend phy
>>> libxl: debug: libxl_create.c:907:initiate_domain_create: running 
>>> bootloader
>>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>>> vdev=(null) spec.backend=phy
>>> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), 
>>> uses
>>> script=... assuming phy backend
>>> libxl: debug: libxl.c:3064:libxl__device_disk_local_initiate_attach: 
>>> locally
>>> attaching PHY disk 
>>> iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250
>> 
>> Now I remember, this was fixed not long ago, you will need to apply
>> b1882a424ae098d722b19086b16e64b9aeccc7ca to your source tree/package 
>> in
>> order to get pygrub working with hotplug scripts [0].
>> 
>> I guess you are using Xen 4.5, because this commit is already present 
>> in Xen
>> 4.6, and it should fix your issue.
>> 
>> [0]
>> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=b1882a424ae098d722b19086b16e64b9aeccc7ca
> 
> Ahhh - thanks for the pointer. As this is a dev system, its probably
> easier for me to upgrade it to Xen 4.6 - however I'll take that commit
> and look at adding it to my Xen 4.5 packages for public consumption.

I might have spoken too soon here... I updated this system to 4.6.1 and 
created the DomU again - still seems to fail - although it does actually 
call the block-iscsi script this time:

# xl -vvv create /etc/xen/test1.vm
Parsing config from /etc/xen/test1.vm
libxl: debug: libxl_create.c:1560:do_domain_create: ao 0x24ad330: 
create: how=(nil) callback=(nil) poller=0x24b7070
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk 
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, uses 
script=... assuming phy backend
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk 
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:945:initiate_domain_create: running 
bootloader
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk 
vdev=(null) spec.backend=phy
libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), 
uses script=... assuming phy backend
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk 
vdev=xvde spec.backend=phy
libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvde, uses 
script=... assuming phy backend
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: 
register slotnum=3
libxl: debug: libxl_create.c:1583:do_domain_create: ao 0x24ad330: 
inprogress: poller=0x24b7070, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00 
wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event 
epath=/local/domain/0/backend/vbd/0/51776/state
libxl: debug: libxl_event.c:884:devstate_callback: backend 
/local/domain/0/backend/vbd/0/51776/state wanted state 2 still waiting 
state 1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00 
wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event 
epath=/local/domain/0/backend/vbd/0/51776/state
libxl: debug: libxl_event.c:880:devstate_callback: backend 
/local/domain/0/backend/vbd/0/51776/state wanted state 2 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: 
deregister slotnum=3
libxl: debug: libxl_device.c:937:device_backend_callback: calling 
device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x24ada00: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and 
environment ready
libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug 
script: /etc/xen/scripts/block-iscsi add
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to 
execute: /etc/xen/scripts/block-iscsi add
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
/etc/xen/scripts/block-iscsi add [2126] exited with error status 1
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x24adb00: deregister unregistered
libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: 
Device already opened
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x24adb00: deregister unregistered
libxl: error: libxl.c:3127:local_device_attach_cb: unable to add vbd 
with id 51776: Interrupted system call
libxl: error: libxl_bootloader.c:408:bootloader_disk_attached_cb: failed 
to attach local disk for bootloader execution
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x24add18: deregister unregistered
libxl: error: libxl_bootloader.c:279:bootloader_local_detached_cb: 
unable to detach locally attached disk
libxl: error: libxl_create.c:1142:domcreate_rebuild_done: cannot 
(re-)build domain: -3
libxl: debug: libxl.c:1719:devices_destroy_cb: forked pid 2154 for 
destroy of domain 7
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x24ad330: 
complete, rc=-3
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x24ad330: 
destroy
libxl: debug: libxl.c:1458:libxl_domain_destroy: ao 0x24acf80: create: 
how=(nil) callback=(nil) poller=0x24b7070
libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 7
libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy 
guest with domid 7
libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 7 
failed
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x24acf80: 
complete, rc=-21
libxl: debug: libxl.c:1467:libxl_domain_destroy: ao 0x24acf80: 
inprogress: poller=0x24b7070, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x24acf80: 
destroy
xc: debug: hypercall buffer: total allocations:49 total releases:49
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:38 misses:2 toobig:9

So - thinking about the parameters passed to block-iscsi...

I haven't verified that this parameter set is correct - however I can 
use the string pasted into the config file with iscsiadm and --login to 
connect - and I also see:
# iscsiadm -m discovery -t sendtargets -p 192.168.133.250
192.168.133.250:3260,-1 
iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5

As such, I *think* that my config is correct in this case - but I'm not 
supremely confident there.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

On Fri, Apr 15, 2016 at 06:20:56PM +1000, Steven Haigh wrote:
[...]
> I might have spoken too soon here... I updated this system to 4.6.1 and
> created the DomU again - still seems to fail - although it does actually
> call the block-iscsi script this time:
> 
> # xl -vvv create /etc/xen/test1.vm
> Parsing config from /etc/xen/test1.vm
> libxl: debug: libxl_create.c:1560:do_domain_create: ao 0x24ad330: create:
> how=(nil) callback=(nil) poller=0x24b7070
> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> vdev=xvda spec.backend=unknown
> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, uses
> script=... assuming phy backend
> libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
> vdev=xvda, using backend phy
> libxl: debug: libxl_create.c:945:initiate_domain_create: running bootloader
> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> vdev=(null) spec.backend=phy
> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), uses
> script=... assuming phy backend
> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> vdev=xvde spec.backend=phy
> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvde, uses
> script=... assuming phy backend
> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
> register slotnum=3
> libxl: debug: libxl_create.c:1583:do_domain_create: ao 0x24ad330:
> inprogress: poller=0x24b7070, flags=i
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
> epath=/local/domain/0/backend/vbd/0/51776/state
> libxl: debug: libxl_event.c:884:devstate_callback: backend
> /local/domain/0/backend/vbd/0/51776/state wanted state 2 still waiting state
> 1
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
> epath=/local/domain/0/backend/vbd/0/51776/state
> libxl: debug: libxl_event.c:880:devstate_callback: backend
> /local/domain/0/backend/vbd/0/51776/state wanted state 2 ok
> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
> deregister slotnum=3
> libxl: debug: libxl_device.c:937:device_backend_callback: calling
> device_backend_cleanup
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x24ada00: deregister unregistered
> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment
> ready
> libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script:
> /etc/xen/scripts/block-iscsi add
> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
> execute: /etc/xen/scripts/block-iscsi add
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
> /etc/xen/scripts/block-iscsi add [2126] exited with error status 1
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x24adb00: deregister unregistered
> libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script:
> Device already opened

The message indicates that you have this device already opened in this 
system, this is detected by the following check, that you can also run from 
a shell:

# iscsiadm -m session 2>&1 | grep -q "$iqn" && fatal "Device already opened"

You will have to perform a logout in order for the hotplug script to 
correctly attach it.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

[-- Attachment #1.1.1: Type: text/plain, Size: 4598 bytes --]

On 15/04/2016 7:01 PM, Roger Pau Monné wrote:
> On Fri, Apr 15, 2016 at 06:20:56PM +1000, Steven Haigh wrote:
> [...]
>> I might have spoken too soon here... I updated this system to 4.6.1 and
>> created the DomU again - still seems to fail - although it does actually
>> call the block-iscsi script this time:
>>
>> # xl -vvv create /etc/xen/test1.vm
>> Parsing config from /etc/xen/test1.vm
>> libxl: debug: libxl_create.c:1560:do_domain_create: ao 0x24ad330: create:
>> how=(nil) callback=(nil) poller=0x24b7070
>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>> vdev=xvda spec.backend=unknown
>> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, uses
>> script=... assuming phy backend
>> libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
>> vdev=xvda, using backend phy
>> libxl: debug: libxl_create.c:945:initiate_domain_create: running bootloader
>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>> vdev=(null) spec.backend=phy
>> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), uses
>> script=... assuming phy backend
>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>> vdev=xvde spec.backend=phy
>> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvde, uses
>> script=... assuming phy backend
>> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
>> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
>> register slotnum=3
>> libxl: debug: libxl_create.c:1583:do_domain_create: ao 0x24ad330:
>> inprogress: poller=0x24b7070, flags=i
>> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
>> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
>> epath=/local/domain/0/backend/vbd/0/51776/state
>> libxl: debug: libxl_event.c:884:devstate_callback: backend
>> /local/domain/0/backend/vbd/0/51776/state wanted state 2 still waiting state
>> 1
>> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
>> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
>> epath=/local/domain/0/backend/vbd/0/51776/state
>> libxl: debug: libxl_event.c:880:devstate_callback: backend
>> /local/domain/0/backend/vbd/0/51776/state wanted state 2 ok
>> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
>> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
>> deregister slotnum=3
>> libxl: debug: libxl_device.c:937:device_backend_callback: calling
>> device_backend_cleanup
>> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
>> w=0x24ada00: deregister unregistered
>> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment
>> ready
>> libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script:
>> /etc/xen/scripts/block-iscsi add
>> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
>> execute: /etc/xen/scripts/block-iscsi add
>> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
>> /etc/xen/scripts/block-iscsi add [2126] exited with error status 1
>> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
>> w=0x24adb00: deregister unregistered
>> libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script:
>> Device already opened
> 
> The message indicates that you have this device already opened in this 
> system, this is detected by the following check, that you can also run from 
> a shell:
> 
> # iscsiadm -m session 2>&1 | grep -q "$iqn" && fatal "Device already opened"
> 
> You will have to perform a logout in order for the hotplug script to 
> correctly attach it.

How right you are :)

# iscsiadm -m session
tcp: [1] 192.168.133.250:3260,1
iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5 (non-flash)

# iscsiadm -m node --targetname
iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5 -p
192.168.133.250 --logout
Logging out of session [sid: 1, target:
iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5, portal:
192.168.133.250,3260]
Logout of [sid: 1, target:
iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5, portal:
192.168.133.250,3260] successful.

# iscsiadm -m session
iscsiadm: No active sessions.

The DomU then started successfully. Thanks for your help.

I'll try the previously mentioned patch on 4.5 and see how I go with
that next week.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[George Dunlap]

On Fri, Apr 15, 2016 at 7:59 AM, Steven Haigh <netwiz@crc.id.au> wrote:
> Hi all,
>
> I'm wading through the somewhat confusing world of documentation regarding
> storing DomU disk images on an iSCSI target.
>
> I'm getting an error when using pygrub of:
> OSError: [Errno 2] No such file or directory:
> 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
>
> After much hunting, I came across this post:
> http://lists.xen.org/archives/html/xen-devel/2013-04/msg02796.html
>
> As such, I'm wondering if it is still required to *NOT* use pygrub for
> booting iSCSI DomUs?
>
> If so, what are the alternatives? Using pv-grub / pv-grub2? Something else?
>
> As I'm running EL7.2, I figure if I have to use a pv-grub based solution, it
> would have to be pv-grub2?

I see you've got a fix already.  But even so, if using pv-grub2 is a
possibility for you, it might be worth pursuing anyway:
- it will be much more secure than pygrub
- it will probably more reliable, since it's actually a native grub
binary ported to Xen, while pygrub is just a python script that
attempts to duplicate some of grub's functionality.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

[-- Attachment #1.1.1: Type: text/plain, Size: 1585 bytes --]

On 16/04/2016 12:30 AM, George Dunlap wrote:
> On Fri, Apr 15, 2016 at 7:59 AM, Steven Haigh <netwiz@crc.id.au> wrote:
>> Hi all,
>>
>> I'm wading through the somewhat confusing world of documentation regarding
>> storing DomU disk images on an iSCSI target.
>>
>> I'm getting an error when using pygrub of:
>> OSError: [Errno 2] No such file or directory:
>> 'iqn=iqn.1986-03.com.sun:02:ff2d12c0-b709-4ec0-999d-976506c666f5,portal=192.168.133.250'
>>
>> After much hunting, I came across this post:
>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg02796.html
>>
>> As such, I'm wondering if it is still required to *NOT* use pygrub for
>> booting iSCSI DomUs?
>>
>> If so, what are the alternatives? Using pv-grub / pv-grub2? Something else?
>>
>> As I'm running EL7.2, I figure if I have to use a pv-grub based solution, it
>> would have to be pv-grub2?
> 
> I see you've got a fix already.  But even so, if using pv-grub2 is a
> possibility for you, it might be worth pursuing anyway:
> - it will be much more secure than pygrub
> - it will probably more reliable, since it's actually a native grub
> binary ported to Xen, while pygrub is just a python script that
> attempts to duplicate some of grub's functionality.

Hi George,

I kind of agree - its on my todo list, but I haven't managed to see much
about including it in a .spec build yet.

If you've done any work on this so far, I'd be happy to discuss off-list.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

Hello,

I'm re-adding xen-devel in case someone else also wants to provide feedback.

On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
> Hi Roger,
> 
> I've been getting some good progress with iSCSI thanks to your insights.
> 
> I'm now trying to add support for locking via Persistent Reservations to
> ensure that only one Dom0 can attach / use a single iSCSI target at once.

This might be problematic with migrations. IIRC there's a point during the 
migration where both the sending and the receiving side have the disk open 
at the same time. However Xen always makes sure that only one guest is 
actually accessing the disk, either the one on the receiving side (if 
everything has gone OK) or the one on the senders side (if migration has 
failed).

> In a nutshell, my thoughts are to use the following to 'lock' a device:
> 	## Create a hex key for the lock from the systems IP.
> 	key=$(gethostip -x $(uname -n))
> 	sg_persist -d ${dev} -o -G -S ${key}
> 	sg_persist -d ${dev} -o -R -K ${key} -T 6
> 
> This registers the device, and sets an Exclusive Access (-T 6) flag on
> the iSCSI device which means nothing else will be able to open the
> device until the lock is removed.
> 
> To unlock the device, on remove, we should do something like:
> 	key=$(gethostip -x $(uname -n))
>         sg_persist -d ${dev} -o -L -K ${key} -T 6
>         sg_persist -d ${dev} -o -G -K ${key} -S 0
> 
> This releases the device for other things to use.
> 
> I've tried putting these in block-iscsi - by using a lock_device and
> unlock_device function and calling it after find_device in both attach()
> and remove().
> 
> My problems:
> 1) -e is set on the script - and maybe elsewhere - so any time something
> returns non-zero, you can't clean up. For example, if you can't get a
> lock, you should make sure all locks are removed from the host in
> question and then detach the iSCSI target.

You can avoid this by adding something like:

sg_persist ... || true

Of course you can replace the "true" command with something else, like a 
fatal message or some cleanup code. You can also place the command inside of 
a conditional if you know it might fail:

if ! sg_persist ...; then
	fatal ...
fi

It is important for us to use the '-e' in order to make sure all the failure 
points are correctly handled, without the '-e' some command might fail and 
the script wouldn't realize.

> 2) I can't find an easy way to clean up by doing an iscsiadm --logout if
> the locking fails.

I'm not really following here, maybe because I don't know that much about 
iSCSI. Can you just put whatever code is needed in order to unlock before 
doing the logout? Or that's not how it works?

> I'm wondering if there is a reason that the script is currently in the
> stucture that it is - or if it just evolved like this? It may be a good
> candidate for a complete re-write :\

TBH, I thought this was one of the most clean and well structured block 
scripts that Xen has ;).

Roger.

> On 15/04/2016 7:01 PM, Roger Pau Monné wrote:
> > On Fri, Apr 15, 2016 at 06:20:56PM +1000, Steven Haigh wrote:
> > [...]
> >> I might have spoken too soon here... I updated this system to 4.6.1 and
> >> created the DomU again - still seems to fail - although it does actually
> >> call the block-iscsi script this time:
> >>
> >> # xl -vvv create /etc/xen/test1.vm
> >> Parsing config from /etc/xen/test1.vm
> >> libxl: debug: libxl_create.c:1560:do_domain_create: ao 0x24ad330: create:
> >> how=(nil) callback=(nil) poller=0x24b7070
> >> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> >> vdev=xvda spec.backend=unknown
> >> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, uses
> >> script=... assuming phy backend
> >> libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
> >> vdev=xvda, using backend phy
> >> libxl: debug: libxl_create.c:945:initiate_domain_create: running bootloader
> >> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> >> vdev=(null) spec.backend=phy
> >> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), uses
> >> script=... assuming phy backend
> >> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> >> vdev=xvde spec.backend=phy
> >> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvde, uses
> >> script=... assuming phy backend
> >> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
> >> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
> >> register slotnum=3
> >> libxl: debug: libxl_create.c:1583:do_domain_create: ao 0x24ad330:
> >> inprogress: poller=0x24b7070, flags=i
> >> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
> >> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
> >> epath=/local/domain/0/backend/vbd/0/51776/state
> >> libxl: debug: libxl_event.c:884:devstate_callback: backend
> >> /local/domain/0/backend/vbd/0/51776/state wanted state 2 still waiting state
> >> 1
> >> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
> >> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
> >> epath=/local/domain/0/backend/vbd/0/51776/state
> >> libxl: debug: libxl_event.c:880:devstate_callback: backend
> >> /local/domain/0/backend/vbd/0/51776/state wanted state 2 ok
> >> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
> >> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
> >> deregister slotnum=3
> >> libxl: debug: libxl_device.c:937:device_backend_callback: calling
> >> device_backend_cleanup
> >> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> >> w=0x24ada00: deregister unregistered
> >> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment
> >> ready
> >> libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script:
> >> /etc/xen/scripts/block-iscsi add
> >> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
> >> execute: /etc/xen/scripts/block-iscsi add
> >> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
> >> /etc/xen/scripts/block-iscsi add [2126] exited with error status 1
> >> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> >> w=0x24adb00: deregister unregistered
> >> libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script:
> >> Device already opened
> > 
> > The message indicates that you have this device already opened in this 
> > system, this is detected by the following check, that you can also run from 
> > a shell:
> > 
> > # iscsiadm -m session 2>&1 | grep -q "$iqn" && fatal "Device already opened"
> > 
> > You will have to perform a logout in order for the hotplug script to 
> > correctly attach it.
> > 
> > Roger.
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> > 
> 
> -- 
> Steven Haigh
> 
> Email: netwiz@crc.id.au
> Web: https://www.crc.id.au
> Phone: (03) 9001 6090 - 0412 935 897
> 




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[George Dunlap]

On Wed, May 4, 2016 at 8:34 AM, Roger Pau Monné <roger.pau@citrix.com> wrote:
> Hello,
>
> I'm re-adding xen-devel in case someone else also wants to provide feedback.
>
> On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
>> Hi Roger,
>>
>> I've been getting some good progress with iSCSI thanks to your insights.
>>
>> I'm now trying to add support for locking via Persistent Reservations to
>> ensure that only one Dom0 can attach / use a single iSCSI target at once.
>
> This might be problematic with migrations. IIRC there's a point during the
> migration where both the sending and the receiving side have the disk open
> at the same time. However Xen always makes sure that only one guest is
> actually accessing the disk, either the one on the receiving side (if
> everything has gone OK) or the one on the senders side (if migration has
> failed).
>
>> In a nutshell, my thoughts are to use the following to 'lock' a device:
>>       ## Create a hex key for the lock from the systems IP.
>>       key=$(gethostip -x $(uname -n))
>>       sg_persist -d ${dev} -o -G -S ${key}
>>       sg_persist -d ${dev} -o -R -K ${key} -T 6
>>
>> This registers the device, and sets an Exclusive Access (-T 6) flag on
>> the iSCSI device which means nothing else will be able to open the
>> device until the lock is removed.
>>
>> To unlock the device, on remove, we should do something like:
>>       key=$(gethostip -x $(uname -n))
>>         sg_persist -d ${dev} -o -L -K ${key} -T 6
>>         sg_persist -d ${dev} -o -G -K ${key} -S 0
>>
>> This releases the device for other things to use.
>>
>> I've tried putting these in block-iscsi - by using a lock_device and
>> unlock_device function and calling it after find_device in both attach()
>> and remove().
>>
>> My problems:
>> 1) -e is set on the script - and maybe elsewhere - so any time something
>> returns non-zero, you can't clean up. For example, if you can't get a
>> lock, you should make sure all locks are removed from the host in
>> question and then detach the iSCSI target.
>
> You can avoid this by adding something like:
>
> sg_persist ... || true
>
> Of course you can replace the "true" command with something else, like a
> fatal message or some cleanup code. You can also place the command inside of
> a conditional if you know it might fail:
>
> if ! sg_persist ...; then
>         fatal ...
> fi
>
> It is important for us to use the '-e' in order to make sure all the failure
> points are correctly handled, without the '-e' some command might fail and
> the script wouldn't realize.

I realize I'm a bit in the minority here, but I've always thought this
was rather a strange habit of bash scripts.  In every other language,
you check the error codes of things that can fail and you handle them
appropriately.  If you're just hacking something together, then "set
-e" is probably OK, but wouldn't it make more sense in an
infrastructure script like this to actually go through and handle all
the errors?   Worst case you could just if ! [command] ; then exit 1 ;
fi.

Regarding the "maybe elsewhere" -- AFAIK the block-scsi script itself
is run directly from libxl, so nothing "above" it should be setting
-e; and it only includes block-common, which does not seem to be
setting it.

That said, in theory as Roger said, if you actually are checking the
error code of all your commands (which you should be if you need to do
clean-up on failure), then 'set -e' shouldn't actually be causing an
exit.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

[-- Attachment #1.1.1: Type: text/plain, Size: 4784 bytes --]

On 4/05/2016 5:34 PM, Roger Pau Monné wrote:
> Hello,
> 
> I'm re-adding xen-devel in case someone else also wants to provide feedback.
> 
> On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
>> Hi Roger,
>>
>> I've been getting some good progress with iSCSI thanks to your insights.
>>
>> I'm now trying to add support for locking via Persistent Reservations to
>> ensure that only one Dom0 can attach / use a single iSCSI target at once.
> 
> This might be problematic with migrations. IIRC there's a point during the 
> migration where both the sending and the receiving side have the disk open 
> at the same time. However Xen always makes sure that only one guest is 
> actually accessing the disk, either the one on the receiving side (if 
> everything has gone OK) or the one on the senders side (if migration has 
> failed).

True - however I'd like to eventually attempt to commit changes to the
project and allow locking to be done as an option - just like iqn /
portal / multipath.

In my specific use case, its to stop someone accidentally starting the
same VM on multiple Dom0's at the same time - which from what I've seen
causes disk corruption and all kinds of issues. It leads to people not
having a good time.

The iSCSI system has a limit to the max connections - however it seems
that only applies *per host* meaning max connections = 1 will allow one
connection per Dom0.

>> In a nutshell, my thoughts are to use the following to 'lock' a device:
>> 	## Create a hex key for the lock from the systems IP.
>> 	key=$(gethostip -x $(uname -n))
>> 	sg_persist -d ${dev} -o -G -S ${key}
>> 	sg_persist -d ${dev} -o -R -K ${key} -T 6
>>
>> This registers the device, and sets an Exclusive Access (-T 6) flag on
>> the iSCSI device which means nothing else will be able to open the
>> device until the lock is removed.
>>
>> To unlock the device, on remove, we should do something like:
>> 	key=$(gethostip -x $(uname -n))
>>         sg_persist -d ${dev} -o -L -K ${key} -T 6
>>         sg_persist -d ${dev} -o -G -K ${key} -S 0
>>
>> This releases the device for other things to use.
>>
>> I've tried putting these in block-iscsi - by using a lock_device and
>> unlock_device function and calling it after find_device in both attach()
>> and remove().
>>
>> My problems:
>> 1) -e is set on the script - and maybe elsewhere - so any time something
>> returns non-zero, you can't clean up. For example, if you can't get a
>> lock, you should make sure all locks are removed from the host in
>> question and then detach the iSCSI target.
> 
> You can avoid this by adding something like:
> 
> sg_persist ... || true
> 
> Of course you can replace the "true" command with something else, like a 
> fatal message or some cleanup code. You can also place the command inside of 
> a conditional if you know it might fail:
> 
> if ! sg_persist ...; then
> 	fatal ...
> fi
> 
> It is important for us to use the '-e' in order to make sure all the failure 
> points are correctly handled, without the '-e' some command might fail and 
> the script wouldn't realize.

I honestly think this is pretty nasty. While it may not be true of all
scripts, the block-iscsi script can only really fail in a couple of
places - yet we have this set of procedures called:

parse_target -> check_tools -> prepare -> add -> attach -> find_device
-> write_dev.

At least check_tools, prepare, add, attach, find_device could all be
rolled into a single function - as the majority of the rest is 1-4 lines
of code.

There are situations where you may want to evaluate the result of
sg_persist beyond a simple "worked or failed" - and that seems to be the
idea of fatal "The reason that I died is X".

>> 2) I can't find an easy way to clean up by doing an iscsiadm --logout if
>> the locking fails.
> 
> I'm not really following here, maybe because I don't know that much about 
> iSCSI. Can you just put whatever code is needed in order to unlock before 
> doing the logout? Or that's not how it works?

Yes, but if one of the two unlocks fails, the script terminates. It
makes different error checking *VERY* difficult. If I remove the -e from
line #1, the script still acts as if -e is still set - so something else
is enforcing that.

>> I'm wondering if there is a reason that the script is currently in the
>> stucture that it is - or if it just evolved like this? It may be a good
>> candidate for a complete re-write :\
> 
> TBH, I thought this was one of the most clean and well structured block 
> scripts that Xen has ;).

Please don't scare me ;)

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

[-- Attachment #1.1.1: Type: text/plain, Size: 4718 bytes --]

On 4/05/2016 6:25 PM, George Dunlap wrote:
> On Wed, May 4, 2016 at 8:34 AM, Roger Pau Monné <roger.pau@citrix.com> wrote:
>> Hello,
>>
>> I'm re-adding xen-devel in case someone else also wants to provide feedback.
>>
>> On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
>>> Hi Roger,
>>>
>>> I've been getting some good progress with iSCSI thanks to your insights.
>>>
>>> I'm now trying to add support for locking via Persistent Reservations to
>>> ensure that only one Dom0 can attach / use a single iSCSI target at once.
>>
>> This might be problematic with migrations. IIRC there's a point during the
>> migration where both the sending and the receiving side have the disk open
>> at the same time. However Xen always makes sure that only one guest is
>> actually accessing the disk, either the one on the receiving side (if
>> everything has gone OK) or the one on the senders side (if migration has
>> failed).
>>
>>> In a nutshell, my thoughts are to use the following to 'lock' a device:
>>>       ## Create a hex key for the lock from the systems IP.
>>>       key=$(gethostip -x $(uname -n))
>>>       sg_persist -d ${dev} -o -G -S ${key}
>>>       sg_persist -d ${dev} -o -R -K ${key} -T 6
>>>
>>> This registers the device, and sets an Exclusive Access (-T 6) flag on
>>> the iSCSI device which means nothing else will be able to open the
>>> device until the lock is removed.
>>>
>>> To unlock the device, on remove, we should do something like:
>>>       key=$(gethostip -x $(uname -n))
>>>         sg_persist -d ${dev} -o -L -K ${key} -T 6
>>>         sg_persist -d ${dev} -o -G -K ${key} -S 0
>>>
>>> This releases the device for other things to use.
>>>
>>> I've tried putting these in block-iscsi - by using a lock_device and
>>> unlock_device function and calling it after find_device in both attach()
>>> and remove().
>>>
>>> My problems:
>>> 1) -e is set on the script - and maybe elsewhere - so any time something
>>> returns non-zero, you can't clean up. For example, if you can't get a
>>> lock, you should make sure all locks are removed from the host in
>>> question and then detach the iSCSI target.
>>
>> You can avoid this by adding something like:
>>
>> sg_persist ... || true
>>
>> Of course you can replace the "true" command with something else, like a
>> fatal message or some cleanup code. You can also place the command inside of
>> a conditional if you know it might fail:
>>
>> if ! sg_persist ...; then
>>         fatal ...
>> fi
>>
>> It is important for us to use the '-e' in order to make sure all the failure
>> points are correctly handled, without the '-e' some command might fail and
>> the script wouldn't realize.
> 
> I realize I'm a bit in the minority here, but I've always thought this
> was rather a strange habit of bash scripts.  In every other language,
> you check the error codes of things that can fail and you handle them
> appropriately.  If you're just hacking something together, then "set
> -e" is probably OK, but wouldn't it make more sense in an
> infrastructure script like this to actually go through and handle all
> the errors?   Worst case you could just if ! [command] ; then exit 1 ;
> fi.

Then I'm in the minority too ;)

> Regarding the "maybe elsewhere" -- AFAIK the block-scsi script itself
> is run directly from libxl, so nothing "above" it should be setting
> -e; and it only includes block-common, which does not seem to be
> setting it.

Right now, even removing -e from line 1 causes something like this to
exit the script:

if [ $? != 0 ]; then

So, if I run the following, the script will always fail:

key=$(gethostip -x $(uname -n))
sg_persist -d ${dev} -o -G -S ${key} > /dev/null
if [ $? != 0 ]; then
	iscsiadm -m node -T ${iqn} --logout > /dev/null
	fatal "Could not obtain lock on $iqn"
fi

man 3 sg3_utils (http://linux.die.net/man/8/sg3_utils) shows a possible
list of exit codes - which it would be useful to consult at least *some*
of them with useful errors - such as:

Exit status 3:
the DEVICE reports that it is not ready for the operation requested. The
device may be in the process of becoming ready (e.g. spinning up but not
at speed) so the utility may work after a wait.

Also possibly a case for locking not being supported.

> That said, in theory as Roger said, if you actually are checking the
> error code of all your commands (which you should be if you need to do
> clean-up on failure), then 'set -e' shouldn't actually be causing an
> exit.
> 
>  -George
> 

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

On Wed, May 04, 2016 at 06:48:13PM +1000, Steven Haigh wrote:
> On 4/05/2016 6:25 PM, George Dunlap wrote:
> > On Wed, May 4, 2016 at 8:34 AM, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>> 1) -e is set on the script - and maybe elsewhere - so any time something
> >>> returns non-zero, you can't clean up. For example, if you can't get a
> >>> lock, you should make sure all locks are removed from the host in
> >>> question and then detach the iSCSI target.
> >>
> >> You can avoid this by adding something like:
> >>
> >> sg_persist ... || true
> >>
> >> Of course you can replace the "true" command with something else, like a
> >> fatal message or some cleanup code. You can also place the command inside of
> >> a conditional if you know it might fail:
> >>
> >> if ! sg_persist ...; then
> >>         fatal ...
> >> fi
> >>
> >> It is important for us to use the '-e' in order to make sure all the failure
> >> points are correctly handled, without the '-e' some command might fail and
> >> the script wouldn't realize.
> > 
> > I realize I'm a bit in the minority here, but I've always thought this
> > was rather a strange habit of bash scripts.  In every other language,
> > you check the error codes of things that can fail and you handle them
> > appropriately.  If you're just hacking something together, then "set
> > -e" is probably OK, but wouldn't it make more sense in an
> > infrastructure script like this to actually go through and handle all
> > the errors?   Worst case you could just if ! [command] ; then exit 1 ;
> > fi.
> 
> Then I'm in the minority too ;)
> 
> > Regarding the "maybe elsewhere" -- AFAIK the block-scsi script itself
> > is run directly from libxl, so nothing "above" it should be setting
> > -e; and it only includes block-common, which does not seem to be
> > setting it.
> 
> Right now, even removing -e from line 1 causes something like this to
> exit the script:
> 
> if [ $? != 0 ]; then
> 
> So, if I run the following, the script will always fail:
> 
> key=$(gethostip -x $(uname -n))
> sg_persist -d ${dev} -o -G -S ${key} > /dev/null
> if [ $? != 0 ]; then
> 	iscsiadm -m node -T ${iqn} --logout > /dev/null
> 	fatal "Could not obtain lock on $iqn"
> fi

I cannot explain this behaviour, removing the -e from the shebang should 
remove the 'exit on error' behaviour. IMHO, I would rather prefer that you 
keep the '-e' at the top and use:

set +e
... code which might fail ...
set -e

> man 3 sg3_utils (http://linux.die.net/man/8/sg3_utils) shows a possible
> list of exit codes - which it would be useful to consult at least *some*
> of them with useful errors - such as:
> 
> Exit status 3:
> the DEVICE reports that it is not ready for the operation requested. The
> device may be in the process of becoming ready (e.g. spinning up but not
> at speed) so the utility may work after a wait.
> 
> Also possibly a case for locking not being supported.

It seems like you want to use a case statement then in order to handle the 
error codes.
 
Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

On Wed, May 04, 2016 at 06:41:26PM +1000, Steven Haigh wrote:
> On 4/05/2016 5:34 PM, Roger Pau Monné wrote:
> > On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
> > It is important for us to use the '-e' in order to make sure all the failure 
> > points are correctly handled, without the '-e' some command might fail and 
> > the script wouldn't realize.
> 
> I honestly think this is pretty nasty. While it may not be true of all
> scripts, the block-iscsi script can only really fail in a couple of
> places - yet we have this set of procedures called:
> 
> parse_target -> check_tools -> prepare -> add -> attach -> find_device
> -> write_dev.
> 
> At least check_tools, prepare, add, attach, find_device could all be
> rolled into a single function - as the majority of the rest is 1-4 lines
> of code.

No, check_tools is used by both the attach and the detach path, so it cannot 
be rolled into a single function together with the other ones, and the same 
applies to mostly all other functions (find_device is also shared between 
the add and remove functions).

IMHO, I think the current code is fine because each function has a small 
logical task to accomplish, so it's easy to make sure each function does 
what it's supposed to do, nothing more and nothing less. Batching everything 
into one big function would make this harder.

That doesn't mean that I'm not open to improving it, so if you think it 
would be better/easier using some other logical organization patches are 
welcome :).

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

[-- Attachment #1.1.1: Type: text/plain, Size: 4391 bytes --]

On 4/05/2016 7:37 PM, Roger Pau Monné wrote:
> On Wed, May 04, 2016 at 06:41:26PM +1000, Steven Haigh wrote:
>> On 4/05/2016 5:34 PM, Roger Pau Monné wrote:
>>> On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
>>> It is important for us to use the '-e' in order to make sure all the failure 
>>> points are correctly handled, without the '-e' some command might fail and 
>>> the script wouldn't realize.
>>
>> I honestly think this is pretty nasty. While it may not be true of all
>> scripts, the block-iscsi script can only really fail in a couple of
>> places - yet we have this set of procedures called:
>>
>> parse_target -> check_tools -> prepare -> add -> attach -> find_device
>> -> write_dev.
>>
>> At least check_tools, prepare, add, attach, find_device could all be
>> rolled into a single function - as the majority of the rest is 1-4 lines
>> of code.
> 
> No, check_tools is used by both the attach and the detach path, so it cannot 
> be rolled into a single function together with the other ones, and the same 
> applies to mostly all other functions (find_device is also shared between 
> the add and remove functions).
> 
> IMHO, I think the current code is fine because each function has a small 
> logical task to accomplish, so it's easy to make sure each function does 
> what it's supposed to do, nothing more and nothing less. Batching everything 
> into one big function would make this harder.
> 
> That doesn't mean that I'm not open to improving it, so if you think it 
> would be better/easier using some other logical organization patches are 
> welcome :).

Right now, my changes are here:
	http://paste.fedoraproject.org/362462/62356799/

It works perfectly well if you're the ONLY device connecting to the
specified iSCSI target, but falls apart when something else has the lock
and doesn't clean up after itself.

From using set -x in there, I can see this when it runs:
++ dirname /etc/xen/scripts/block-iscsi-lock
+ dir=/etc/xen/scripts
+ . /etc/xen/scripts/block-common.sh
+++ dirname /etc/xen/scripts/block-iscsi-lock
++ dir=/etc/xen/scripts
++ . /etc/xen/scripts/xen-hotplug-common.sh
++++ dirname /etc/xen/scripts/block-iscsi-lock
+++ dir=/etc/xen/scripts
+++ . /etc/xen/scripts/hotplugpath.sh
++++ sbindir=/usr/sbin
++++ bindir=/usr/bin
++++ LIBEXEC=/usr/lib/xen
++++ LIBEXEC_BIN=/usr/lib/xen/bin
++++ libdir=/usr/lib64
++++ SHAREDIR=/usr/share
++++ XENFIRMWAREDIR=/usr/lib/xen/boot
++++ XEN_CONFIG_DIR=/etc/xen
++++ XEN_SCRIPT_DIR=/etc/xen/scripts
++++ XEN_LOCK_DIR=/var/lock
++++ XEN_RUN_DIR=/var/run/xen
++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging
++++ XEN_DUMP_DIR=/var/lib/xen/dump
+++ . /etc/xen/scripts/logging.sh
+++ . /etc/xen/scripts/xen-script-common.sh
++++ set -e
+++ . /etc/xen/scripts/locking.sh
++++ LOCK_BASEDIR=/var/run/xen-hotplug
+++ exec
Entered lock_device
  SUN       COMSTAR           1.0
  Peripheral device type: disk
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
/etc/xen/scripts/block-iscsi-lock add [22016] exited with error status 99
libxl: error: libxl.c:3127:local_device_attach_cb: unable to add vbd
with id 268446976: No such file or directory
libxl: error: libxl_bootloader.c:408:bootloader_disk_attached_cb: failed
to attach local disk for bootloader execution
libxl: error: libxl_bootloader.c:279:bootloader_local_detached_cb:
unable to detach locally attached disk
libxl: error: libxl_create.c:1142:domcreate_rebuild_done: cannot
(re-)build domain: -3
libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 57
libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy
guest with domid 57
libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 57
failed

On a side note, it looks like the -e is not very uniformly used:
block:#!/bin/bash
block-enbd:#!/bin/bash
block-iscsi:#!/bin/bash -e
block-iscsi-lock:#!/bin/bash -x
block-nbd:#!/bin/bash
block-tap:#!/bin/bash -e
external-device-migrate:#!/bin/bash
vif2:#!/bin/bash
vif-bridge:#!/bin/bash
vif-nat:#!/bin/bash
vif-openvswitch:#!/bin/bash
vif-route:#!/bin/bash
vif-setup:#!/bin/bash

but probably gets set everywhere via xen-script-common.sh - hence things
dying easily.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné]

On Wed, May 04, 2016 at 08:18:18PM +1000, Steven Haigh wrote:
> On 4/05/2016 7:37 PM, Roger Pau Monné wrote:
> > On Wed, May 04, 2016 at 06:41:26PM +1000, Steven Haigh wrote:
> >> On 4/05/2016 5:34 PM, Roger Pau Monné wrote:
> >>> On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
> >>> It is important for us to use the '-e' in order to make sure all the failure 
> >>> points are correctly handled, without the '-e' some command might fail and 
> >>> the script wouldn't realize.
> >>
> >> I honestly think this is pretty nasty. While it may not be true of all
> >> scripts, the block-iscsi script can only really fail in a couple of
> >> places - yet we have this set of procedures called:
> >>
> >> parse_target -> check_tools -> prepare -> add -> attach -> find_device
> >> -> write_dev.
> >>
> >> At least check_tools, prepare, add, attach, find_device could all be
> >> rolled into a single function - as the majority of the rest is 1-4 lines
> >> of code.
> > 
> > No, check_tools is used by both the attach and the detach path, so it cannot 
> > be rolled into a single function together with the other ones, and the same 
> > applies to mostly all other functions (find_device is also shared between 
> > the add and remove functions).
> > 
> > IMHO, I think the current code is fine because each function has a small 
> > logical task to accomplish, so it's easy to make sure each function does 
> > what it's supposed to do, nothing more and nothing less. Batching everything 
> > into one big function would make this harder.
> > 
> > That doesn't mean that I'm not open to improving it, so if you think it 
> > would be better/easier using some other logical organization patches are 
> > welcome :).
> 
> Right now, my changes are here:
> 	http://paste.fedoraproject.org/362462/62356799/

It seems like you can achieve what you are trying to do by using:

if ! sg_persist -d ${dev} -o -G -S ${key}; then
	cleanup code (possibly just calling the remove function)
fi

This should work fine with '-e' set.

Having that said, you should post the code as a patch with a proper SoB [0]. 
Also, look at how the "multipath" argument is passed, IMHO you should do the 
same and add a "locked" parameter that you can also pass in inside of the 
"target" field.

Roger.

[0] http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[George Dunlap]

On Wed, May 4, 2016 at 11:18 AM, Steven Haigh <netwiz@crc.id.au> wrote:
> On 4/05/2016 7:37 PM, Roger Pau Monné wrote:
>> On Wed, May 04, 2016 at 06:41:26PM +1000, Steven Haigh wrote:
>>> On 4/05/2016 5:34 PM, Roger Pau Monné wrote:
>>>> On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
>>>> It is important for us to use the '-e' in order to make sure all the failure
>>>> points are correctly handled, without the '-e' some command might fail and
>>>> the script wouldn't realize.
>>>
>>> I honestly think this is pretty nasty. While it may not be true of all
>>> scripts, the block-iscsi script can only really fail in a couple of
>>> places - yet we have this set of procedures called:
>>>
>>> parse_target -> check_tools -> prepare -> add -> attach -> find_device
>>> -> write_dev.
>>>
>>> At least check_tools, prepare, add, attach, find_device could all be
>>> rolled into a single function - as the majority of the rest is 1-4 lines
>>> of code.
>>
>> No, check_tools is used by both the attach and the detach path, so it cannot
>> be rolled into a single function together with the other ones, and the same
>> applies to mostly all other functions (find_device is also shared between
>> the add and remove functions).
>>
>> IMHO, I think the current code is fine because each function has a small
>> logical task to accomplish, so it's easy to make sure each function does
>> what it's supposed to do, nothing more and nothing less. Batching everything
>> into one big function would make this harder.
>>
>> That doesn't mean that I'm not open to improving it, so if you think it
>> would be better/easier using some other logical organization patches are
>> welcome :).
>
> Right now, my changes are here:
>         http://paste.fedoraproject.org/362462/62356799/
>
> It works perfectly well if you're the ONLY device connecting to the
> specified iSCSI target, but falls apart when something else has the lock
> and doesn't clean up after itself.
>
> From using set -x in there, I can see this when it runs:
> ++ dirname /etc/xen/scripts/block-iscsi-lock
> + dir=/etc/xen/scripts
> + . /etc/xen/scripts/block-common.sh
> +++ dirname /etc/xen/scripts/block-iscsi-lock
> ++ dir=/etc/xen/scripts
> ++ . /etc/xen/scripts/xen-hotplug-common.sh
> ++++ dirname /etc/xen/scripts/block-iscsi-lock
> +++ dir=/etc/xen/scripts
> +++ . /etc/xen/scripts/hotplugpath.sh
> ++++ sbindir=/usr/sbin
> ++++ bindir=/usr/bin
> ++++ LIBEXEC=/usr/lib/xen
> ++++ LIBEXEC_BIN=/usr/lib/xen/bin
> ++++ libdir=/usr/lib64
> ++++ SHAREDIR=/usr/share
> ++++ XENFIRMWAREDIR=/usr/lib/xen/boot
> ++++ XEN_CONFIG_DIR=/etc/xen
> ++++ XEN_SCRIPT_DIR=/etc/xen/scripts
> ++++ XEN_LOCK_DIR=/var/lock
> ++++ XEN_RUN_DIR=/var/run/xen
> ++++ XEN_PAGING_DIR=/var/lib/xen/xenpaging
> ++++ XEN_DUMP_DIR=/var/lib/xen/dump
> +++ . /etc/xen/scripts/logging.sh
> +++ . /etc/xen/scripts/xen-script-common.sh
> ++++ set -e

Ah, there's the culprit. :-)

> +++ . /etc/xen/scripts/locking.sh
> ++++ LOCK_BASEDIR=/var/run/xen-hotplug
> +++ exec
> Entered lock_device
>   SUN       COMSTAR           1.0
>   Peripheral device type: disk
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
> /etc/xen/scripts/block-iscsi-lock add [22016] exited with error status 99
> libxl: error: libxl.c:3127:local_device_attach_cb: unable to add vbd
> with id 268446976: No such file or directory
> libxl: error: libxl_bootloader.c:408:bootloader_disk_attached_cb: failed
> to attach local disk for bootloader execution
> libxl: error: libxl_bootloader.c:279:bootloader_local_detached_cb:
> unable to detach locally attached disk
> libxl: error: libxl_create.c:1142:domcreate_rebuild_done: cannot
> (re-)build domain: -3
> libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 57
> libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy
> guest with domid 57
> libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 57
> failed
>
> On a side note, it looks like the -e is not very uniformly used:
> block:#!/bin/bash
> block-enbd:#!/bin/bash
> block-iscsi:#!/bin/bash -e
> block-iscsi-lock:#!/bin/bash -x
> block-nbd:#!/bin/bash
> block-tap:#!/bin/bash -e
> external-device-migrate:#!/bin/bash
> vif2:#!/bin/bash
> vif-bridge:#!/bin/bash
> vif-nat:#!/bin/bash
> vif-openvswitch:#!/bin/bash
> vif-route:#!/bin/bash
> vif-setup:#!/bin/bash
>
> but probably gets set everywhere via xen-script-common.sh - hence things
> dying easily.

Yeah, set -e should probably certainly be set only in the top-level
script, not something included 3 levels down.

But as Roger said, the reason you're having problems is that you're
not calling sg_persist in a way that allows bash to know that failure
is being handled.  Putting the stg_persist in an if statement, rather
than just running it and checking the error code afterwards, should
mitigate your problem with -e.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: block-iscsi with Xen 4.5 / 4.6

[Steven Haigh]

On 2016-05-04 20:50, Roger Pau Monné wrote:
> On Wed, May 04, 2016 at 08:18:18PM +1000, Steven Haigh wrote:
>> On 4/05/2016 7:37 PM, Roger Pau Monné wrote:
>> > On Wed, May 04, 2016 at 06:41:26PM +1000, Steven Haigh wrote:
>> >> On 4/05/2016 5:34 PM, Roger Pau Monné wrote:
>> >>> On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
>> >>> It is important for us to use the '-e' in order to make sure all the failure
>> >>> points are correctly handled, without the '-e' some command might fail and
>> >>> the script wouldn't realize.
>> >>
>> >> I honestly think this is pretty nasty. While it may not be true of all
>> >> scripts, the block-iscsi script can only really fail in a couple of
>> >> places - yet we have this set of procedures called:
>> >>
>> >> parse_target -> check_tools -> prepare -> add -> attach -> find_device
>> >> -> write_dev.
>> >>
>> >> At least check_tools, prepare, add, attach, find_device could all be
>> >> rolled into a single function - as the majority of the rest is 1-4 lines
>> >> of code.
>> >
>> > No, check_tools is used by both the attach and the detach path, so it cannot
>> > be rolled into a single function together with the other ones, and the same
>> > applies to mostly all other functions (find_device is also shared between
>> > the add and remove functions).
>> >
>> > IMHO, I think the current code is fine because each function has a small
>> > logical task to accomplish, so it's easy to make sure each function does
>> > what it's supposed to do, nothing more and nothing less. Batching everything
>> > into one big function would make this harder.
>> >
>> > That doesn't mean that I'm not open to improving it, so if you think it
>> > would be better/easier using some other logical organization patches are
>> > welcome :).
>> 
>> Right now, my changes are here:
>> 	http://paste.fedoraproject.org/362462/62356799/
> 
> It seems like you can achieve what you are trying to do by using:
> 
> if ! sg_persist -d ${dev} -o -G -S ${key}; then
> 	cleanup code (possibly just calling the remove function)
> fi
> 
> This should work fine with '-e' set.

I'll look at this - I've been steered away from this style of code my 
entire life, so its a little unfamiliar to me.

> Having that said, you should post the code as a patch with a proper SoB 
> [0].
> Also, look at how the "multipath" argument is passed, IMHO you should 
> do the
> same and add a "locked" parameter that you can also pass in inside of 
> the
> "target" field.

I agree. The reason for not sending things through as a patch right now 
is:
a) It doesn't work; and
b) It's currently in a horrible state.

I do intend to share this for inclusion once I get the problems worked 
out. I do agree it should be structured like the multipath option here.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel