On 31.03.2017 18:03, Ciprian Barbu wrote:
> Hello,
> 
> Similar to the other thread about possible regression with rbd, there might be a regression with nbd.
> This time we are launching an instance from an image (not volume) and try to live migrate it:
> 
> nova live-migration <test_instance>
> 
> The nova-compute service complains with:
> 
> 2017-03-31 15:32:56.179 7806 INFO nova.virt.libvirt.driver [req-15d79cbe-5956-4738-92df-3624e6b993ee d795de59fb9a4ea38776a11d20ae8469 cee03e74881f4ccba3b83345fb652b2c - - -] [instance: 6a04508f-5d79-4582-8e2c-4cc368753f6c] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
> 2017-03-31 15:32:58.029 7806 WARNING stevedore.named [req-73bc0113-5555-4dd8-8903-d3540cc61b47 b9fbceeadd2d4d1bab9c90ae104db1f7 7e7db99b32c6467184701e9a0c2f1de7 - - -] Could not load instance_network_info
> 2017-03-31 15:32:59.038 7806 ERROR nova.virt.libvirt.driver [req-15d79cbe-5956-4738-92df-3624e6b993ee d795de59fb9a4ea38776a11d20ae8469 cee03e74881f4ccba3b83345fb652b2c - - -] [instance: 6a04508f-5d79-4582-8e2c-4cc368753f6c] Live Migration failure: internal error: unable to execute QEMU command 'nbd-server-add': Conflicts with use by drive-virtio-disk0 as 'root', which does not allow 'write' on #block143
> 2017-03-31 15:32:59.190 7806 ERROR nova.virt.libvirt.driver [req-15d79cbe-5956-4738-92df-3624e6b993ee d795de59fb9a4ea38776a11d20ae8469 cee03e74881f4ccba3b83345fb652b2c - - -] [instance: 6a04508f-5d79-4582-8e2c-4cc368753f6c] Migration operation has aborted
> 
> I will try and bisect it myself, but I thought I would paste this here first, just so you know there is this issue too.

Well, I already know the commit in question. It's
8a7ce4f9338c475df1afc12502af704e4300a3e0 ("nbd/server: Use real
permissions for NBD exports").

Whether this is a bug depends on the standpoint. I would very much
consider it a bug fix because as of this commit you can no longer create
a writable NBD server on a block device that is in use by a guest device
without the guest device being aware of this.

The problem is that the functionality to "make" the guest device "aware"
of it was introduced only a couple of commits before, and it's called
"share-rw".

So this doesn't work:

$ x86_64-softmmu/qemu-system-x86_64 \
    -blockdev node-name=image,driver=qcow2,\
file.driver=file,file.filename=foo.qcow2 \
    -device virtio-blk,drive=image \
    -qmp stdio
{"QMP": {"version": {"qemu": {"micro": 92, "minor": 8, "major": 2},
"package": " (v2.8.0-2038-g6604c893d0)"}, "capabilities": []}}
{'execute':'qmp_capabilities'}
{"return": {}}
{'execute':'nbd-server-start','arguments':{'addr':{'type':'inet','data':{'host':'localhost','port':'10809'}}}}
{"return": {}}
{'execute':'nbd-server-add','arguments':{'device':'image','writable':true}}
{"error": {"class": "GenericError", "desc": "Conflicts with use by
/machine/peripheral-anon/device[0]/virtio-backend as 'root', which does
not allow 'write' on image"}

But this works:

$ x86_64-softmmu/qemu-system-x86_64 \
    -blockdev node-name=image,driver=qcow2,\
file.driver=file,file.filename=foo.qcow2 \
    -device virtio-blk,drive=image,share-rw=on \
    -qmp stdio
{"QMP": {"version": {"qemu": {"micro": 92, "minor": 8, "major": 2},
"package": " (v2.8.0-2038-g6604c893d0)"}, "capabilities": []}}
{'execute':'qmp_capabilities'}
{"return": {}}
{'execute':'nbd-server-start','arguments':{'addr':{'type':'inet','data':{'host':'localhost','port':'10809'}}}}
{"return": {}}
{'execute':'nbd-server-add','arguments':{'device':'image','writable':true}}
{"return": {}}

(The difference is the share-rw=on in the -device parameter.)


So in theory all that's necessary is to set share-rw=on for the device
in the management layer. But I'm not sure whether that's practical.

As for just allowing the NBD server write access to the device... To me
that appears pretty difficult from an implementation perspective. We
assert that nobody can write without having requested write access and
we make sure that nobody can request write access without it being
allowed. Making an exception for NBD seems very difficult and would
probably mean we'd have to drop the assertion for write accesses altogether.

Max