xl block-attach vs block-detach

xl block-attach vs block-detach

[Jan Beulich]

So it seems that "xl block-attach" allows a variety of ways to specify the
devices ID (xvdN, dNpN, decimal or hex number) - why is it that the
inverse operation ("xl block-detach") can't deal with anything by a decimal
number?

Further, why is it that with no blktap module loaded I'm getting an
incomplete attach when using the (deprecated) file:/ format for
specifying the backing file? It reports that it would be using qdisk,
and blkfront also sees the device appearing, but all I'm seeing in the
kernel log is the single message from blkfront's probe function. (With
no blktap in pv-ops, I wonder how file backed disks work there.)
When trying to detach such a broken device I'm getting
"unrecognized disk backend type: 0", and the remove fails.

Jan

Re: xl block-attach vs block-detach

[Ian Campbell]

On Thu, 2012-03-01 at 16:33 +0000, Jan Beulich wrote:
> So it seems that "xl block-attach" allows a variety of ways to specify the
> devices ID (xvdN, dNpN, decimal or hex number) - why is it that the
> inverse operation ("xl block-detach") can't deal with anything by a decimal
> number?

Just an omission?

> Further, why is it that with no blktap module loaded I'm getting an
> incomplete attach when using the (deprecated) file:/ format for
> specifying the backing file? It reports that it would be using qdisk,
> and blkfront also sees the device appearing, but all I'm seeing in the
> kernel log is the single message from blkfront's probe function.

What do you mean by "incomplete"? What else would you expect to see?

What do the xenstore entries for the device look like?

> (With no blktap in pv-ops, I wonder how file backed disks work there.)

file backed disks without blktap use the qdisk backend supplied by qemu.

> When trying to detach such a broken device I'm getting
> "unrecognized disk backend type: 0", and the remove fails.

What version of xen is this?

libxl__device_disk_from_xs_be tries to read the backend type from
xenstore, the be "type" node. Is that present for you?

libxl_string_to_backend looks a bit suspect to me...

Ian.

Re: xl block-attach vs block-detach

[Jan Beulich]

>>> On 01.03.12 at 17:45, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2012-03-01 at 16:33 +0000, Jan Beulich wrote:
>> So it seems that "xl block-attach" allows a variety of ways to specify the
>> devices ID (xvdN, dNpN, decimal or hex number) - why is it that the
>> inverse operation ("xl block-detach") can't deal with anything by a decimal
>> number?
> 
> Just an omission?

Maybe. How many else are there throughout the xl stack then? (I'm
asking because any time I do something more advanced than "xl dmesg"
or "xl debug-key ..." with xl, I'm running into endless problems.

>> Further, why is it that with no blktap module loaded I'm getting an
>> incomplete attach when using the (deprecated) file:/ format for
>> specifying the backing file? It reports that it would be using qdisk,
>> and blkfront also sees the device appearing, but all I'm seeing in the
>> kernel log is the single message from blkfront's probe function.
> 
> What do you mean by "incomplete"? What else would you expect to see?

Either a fully failed operation (including an error message indicating so
even without wading through the -vvv output), or a fully working one
(after all, the -vvv messages suggest that a usable backend was
selected).

> What do the xenstore entries for the device look like?

local = ""
 domain = ""
  0 = ""
   name = "Domain-0"
   device = ""
    vbd = ""
     51712 = ""
      backend = "/local/domain/0/backend/qdisk/0/51712"
      backend-id = "0"
      state = "1"
      virtual-device = "51712"
      device-type = "disk"
     51728 = ""
      backend = "/local/domain/0/backend/qdisk/0/51728"
      backend-id = "0"
      state = "3"
      virtual-device = "51728"
      device-type = "disk"
      ring-ref = "9"
      event-channel = "60"
      protocol = "x86_64-abi"
     51760 = ""
      backend = "/local/domain/0/backend/qdisk/0/51760"
      backend-id = "0"
      state = "3"
      virtual-device = "51760"
      device-type = "disk"
      ring-ref = "10"
      event-channel = "61"
      protocol = "x86_64-abi"
   backend = ""
    qdisk = ""
     0 = ""
      51712 = ""
       frontend = "/local/domain/0/device/vbd/51712"
       params = "aio:/srv/SuSE/SLES-11-SP1-MINI-ISO-x86_64-GMC3-CD.iso"
       frontend-id = "0"
       online = "1"
       removable = "0"
       bootable = "1"
       state = "1"
       dev = "xvda"
       type = "qdisk"
       mode = "r"
       device-type = "disk"
      51728 = ""
       frontend = "/local/domain/0/device/vbd/51728"
       params = "aio:/srv/SuSE/SLES-11-SP1-MINI-ISO-x86_64-GMC3-CD.iso"
       frontend-id = "0"
       online = "1"
       removable = "0"
       bootable = "1"
       state = "1"
       dev = "xvdb"
       type = "qdisk"
       mode = "r"
       device-type = "disk"
      51760 = ""
       frontend = "/local/domain/0/device/vbd/51760"
       params = "aio:/srv/SuSE/SLES-11-SP1-MINI-ISO-x86_64-GMC3-CD.iso"
       frontend-id = "0"
       online = "1"
       removable = "0"
       bootable = "1"
       state = "1"
       dev = "d3p0"
       type = "qdisk"
       mode = "r"
       device-type = "disk"

>> (With no blktap in pv-ops, I wonder how file backed disks work there.)
> 
> file backed disks without blktap use the qdisk backend supplied by qemu.

Which apparently doesn't work (for me).

>> When trying to detach such a broken device I'm getting
>> "unrecognized disk backend type: 0", and the remove fails.
> 
> What version of xen is this?

-unstable c/s 24691:3432abcf9380.

I should probably add that I'm running the tools from the build tree,
but with xend this never caused any problems after I had added the
necessary paths to various environment variables in a wrapper script.
I'd expect xl to be similarly tolerable of such an environment,
otherwise it's not a drop-in replacement.

> libxl__device_disk_from_xs_be tries to read the backend type from
> xenstore, the be "type" node. Is that present for you?

See above.

Jan

Re: xl block-attach vs block-detach

[Ian Jackson]

Jan Beulich writes ("[Xen-devel] xl block-attach vs block-detach"):
> So it seems that "xl block-attach" allows a variety of ways to specify the
> devices ID (xvdN, dNpN, decimal or hex number) - why is it that the
> inverse operation ("xl block-detach") can't deal with anything by a decimal
> number?

Probably because I didn't notice that xl block-detach was using a
different parser when I replaced the block-attach one with a call to
the same parser as is used for disks specified in the config file.

Would you like to fix it ? :-)

> Further, why is it that with no blktap module loaded I'm getting an
> incomplete attach when using the (deprecated) file:/ format for
> specifying the backing file? It reports that it would be using qdisk,
> and blkfront also sees the device appearing, but all I'm seeing in the
> kernel log is the single message from blkfront's probe function. (With
> no blktap in pv-ops, I wonder how file backed disks work there.)
> When trying to detach such a broken device I'm getting
> "unrecognized disk backend type: 0", and the remove fails.

That might well be a bug.  In addition to Ian's questions, what do you
get if you turn on the debug by passing xl lots of -v flags (before
the block-attach) ?  Can you attach the disk by naming it in the
config file ?

thanks,
Ian.

Re: xl block-attach vs block-detach

[Jan Beulich]

>>> On 01.03.12 at 17:45, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2012-03-01 at 16:33 +0000, Jan Beulich wrote:
>> Further, why is it that with no blktap module loaded I'm getting an
>> incomplete attach when using the (deprecated) file:/ format for
>> specifying the backing file? It reports that it would be using qdisk,
>> and blkfront also sees the device appearing, but all I'm seeing in the
>> kernel log is the single message from blkfront's probe function.
>> (With no blktap in pv-ops, I wonder how file backed disks work there.)
> 
> file backed disks without blktap use the qdisk backend supplied by qemu.

What is the rationale for using blkback over blktap for file backed
disks anyway? If using blktap, why not directly do so? And if using
blkback, why not (as in xend) via loop devices?

Jan

Re: xl block-attach vs block-detach

[Jan Beulich]

>>> On 01.03.12 at 18:30, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>> Further, why is it that with no blktap module loaded I'm getting an
>> incomplete attach when using the (deprecated) file:/ format for
>> specifying the backing file? It reports that it would be using qdisk,
>> and blkfront also sees the device appearing, but all I'm seeing in the
>> kernel log is the single message from blkfront's probe function. (With
>> no blktap in pv-ops, I wonder how file backed disks work there.)
>> When trying to detach such a broken device I'm getting
>> "unrecognized disk backend type: 0", and the remove fails.
> 
> That might well be a bug.  In addition to Ian's questions, what do you
> get if you turn on the debug by passing xl lots of -v flags (before
> the block-attach) ?

+ xl -vvvvv block-attach 0 file:/srv/SuSE/SLES-11-SP1-MINI-ISO-x86_64-GMC3-CD.iso 0xca00 r
libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=unknown
libxl: debug: libxl_device.c:137:disk_try_backend: Disk vdev=0xca00, backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:144:disk_try_backend: Disk vdev=0xca00, backend tap unsuitable because blktap not available
libxl: debug: libxl_device.c:219:libxl__device_disk_set_backend: Disk vdev=0xca00, using backend qdisk
libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=qdisk
xc: debug: hypercall buffer: total allocations:2 total releases:2
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0
+ xl -vvvvv block-attach 0 file:/srv/SuSE/SLES-11-SP1-MINI-ISO-x86_64-GMC3-CD.iso 0xca00 r
libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=unknown
libxl: debug: libxl_device.c:137:disk_try_backend: Disk vdev=0xca00, backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:144:disk_try_backend: Disk vdev=0xca00, backend tap unsuitable because blktap not available
libxl: debug: libxl_device.c:219:libxl__device_disk_set_backend: Disk vdev=0xca00, using backend qdisk
libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=qdisk
xc: debug: hypercall buffer: total allocations:2 total releases:2
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0
+ xl -vvvvv block-detach 0 51712
libxl: error: libxl.c:1223:libxl__device_from_disk: unrecognized disk backend type: 0

libxl_device_disk_remove failed.
xc: debug: hypercall buffer: total allocations:2 total releases:2
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0
+ xl -vvvvv block-detach 0 51712
libxl: error: libxl.c:1223:libxl__device_from_disk: unrecognized disk backend type: 0

libxl_device_disk_remove failed.
xc: debug: hypercall buffer: total allocations:2 total releases:2
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0

> Can you attach the disk by naming it in the config file ?

Didn't try, for the purpose at hand I want the disk attached to Dom0.

Jan

Re: xl block-attach vs block-detach

[Ian Campbell]

On Fri, 2012-03-02 at 07:53 +0000, Jan Beulich wrote:
> >>> On 01.03.12 at 18:30, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> >> Further, why is it that with no blktap module loaded I'm getting an
> >> incomplete attach when using the (deprecated) file:/ format for
> >> specifying the backing file? It reports that it would be using qdisk,
> >> and blkfront also sees the device appearing, but all I'm seeing in the
> >> kernel log is the single message from blkfront's probe function. (With
> >> no blktap in pv-ops, I wonder how file backed disks work there.)
> >> When trying to detach such a broken device I'm getting
> >> "unrecognized disk backend type: 0", and the remove fails.
> > 
> > That might well be a bug.  In addition to Ian's questions, what do you
> > get if you turn on the debug by passing xl lots of -v flags (before
> > the block-attach) ?
> 
> + xl -vvvvv block-attach 0 file:/srv/SuSE/SLES-11-SP1-MINI-ISO-x86_64-GMC3-CD.iso 0xca00 r
> libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=unknown
> libxl: debug: libxl_device.c:137:disk_try_backend: Disk vdev=0xca00, backend phy unsuitable as phys path not a block device
> libxl: debug: libxl_device.c:144:disk_try_backend: Disk vdev=0xca00, backend tap unsuitable because blktap not available
> libxl: debug: libxl_device.c:219:libxl__device_disk_set_backend: Disk vdev=0xca00, using backend qdisk
> libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=qdisk
> xc: debug: hypercall buffer: total allocations:2 total releases:2
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
> xc: debug: hypercall buffer: cache current size:2
> xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0
> + xl -vvvvv block-attach 0 file:/srv/SuSE/SLES-11-SP1-MINI-ISO-x86_64-GMC3-CD.iso 0xca00 r
> libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=unknown
> libxl: debug: libxl_device.c:137:disk_try_backend: Disk vdev=0xca00, backend phy unsuitable as phys path not a block device
> libxl: debug: libxl_device.c:144:disk_try_backend: Disk vdev=0xca00, backend tap unsuitable because blktap not available
> libxl: debug: libxl_device.c:219:libxl__device_disk_set_backend: Disk vdev=0xca00, using backend qdisk
> libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk vdev=0xca00 spec.backend=qdisk
> xc: debug: hypercall buffer: total allocations:2 total releases:2
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
> xc: debug: hypercall buffer: cache current size:2
> xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0
> + xl -vvvvv block-detach 0 51712
> libxl: error: libxl.c:1223:libxl__device_from_disk: unrecognized disk backend type: 0
> 
> libxl_device_disk_remove failed.
> xc: debug: hypercall buffer: total allocations:2 total releases:2
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
> xc: debug: hypercall buffer: cache current size:2
> xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0
> + xl -vvvvv block-detach 0 51712
> libxl: error: libxl.c:1223:libxl__device_from_disk: unrecognized disk backend type: 0
> 
> libxl_device_disk_remove failed.
> xc: debug: hypercall buffer: total allocations:2 total releases:2
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
> xc: debug: hypercall buffer: cache current size:2
> xc: debug: hypercall buffer: cache hits:0 misses:2 toobig:0
> 
> > Can you attach the disk by naming it in the config file ?
> 
> Didn't try, for the purpose at hand I want the disk attached to Dom0.

AH, I bet that is it -- it is very unlikely that dom0 has a qemu which
would process the qdisk backend stuff.

Hrm, in fact I wonder if block-attach handles starting a qemu at all if
one isn't already running. I also wonder how well qemu handles hotplug
of disks if it is running.

I think you may have opened a can of works here. Hopefully someone will
correct me but I expect there is work to be done here...

Ian.

Re: xl block-attach vs block-detach

[Ian Campbell]

On Fri, 2012-03-02 at 07:40 +0000, Jan Beulich wrote:
> >>> On 01.03.12 at 17:45, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Thu, 2012-03-01 at 16:33 +0000, Jan Beulich wrote:
> >> Further, why is it that with no blktap module loaded I'm getting an
> >> incomplete attach when using the (deprecated) file:/ format for
> >> specifying the backing file? It reports that it would be using qdisk,
> >> and blkfront also sees the device appearing, but all I'm seeing in the
> >> kernel log is the single message from blkfront's probe function.
> >> (With no blktap in pv-ops, I wonder how file backed disks work there.)
> > 
> > file backed disks without blktap use the qdisk backend supplied by qemu.
> 
> What is the rationale for using blkback over blktap for file backed
> disks anyway? If using blktap, why not directly do so?

"directly" how? blktap always requires a blkback to export it to the
guest. The old blktap1 incorporated a disk backend but this lead to
duplicated code, complex set up and teardown interlocking and
interesting interactions between the "backdev" (which exposed the blktap
device to dom0 for toolstack and qemu use) and the pv disk backend.
blktap2 was intended to fix those by layering blkback on blktap2, which
now just provides a block device in dom0 (blktap2 is almost independent
of Xen). 

(I originally misread the above and wrote the following, so it's not
actually relevant to your questions, but they are perhaps interest facts
so I've left them in:...

There is no blktap in upstream kernels and the kernel module is
considered non-upstreamable.

If blktap is available then libxl will use it in preference to qdisk.

There are some folks working on a purely userspace blktap ("blktap3" ==
blktap2 with a userspace disk backend bolted on). I hope this will be
ready in time for 4.2 but I'm not sure.

...)

> And if using blkback, why not (as in xend) via loop devices?

Support for block device script= in xl/libxl is on the 4.2 blocker list,
this feature would re-enable the loop+blkback case -- I think this would
be a better option than blktap* or qdisk once it becomes available.

Ian.

Re: xl block-attach vs block-detach

[Jan Beulich]

>>> On 02.03.12 at 09:05, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2012-03-02 at 07:40 +0000, Jan Beulich wrote:
>> And if using blkback, why not (as in xend) via loop devices?
> 
> Support for block device script= in xl/libxl is on the 4.2 blocker list,

Ah, that's good to know. (However, you mentioning script= makes me
assume that one would have to specify this among the block-attach
options, which again wouldn't be a drop-in replacement of how xm
worked in this regard.)

> this feature would re-enable the loop+blkback case -- I think this would
> be a better option than blktap* or qdisk once it becomes available.

Jan

Re: xl block-attach vs block-detach

[Ian Campbell]

On Fri, 2012-03-02 at 08:14 +0000, Jan Beulich wrote:
> >>> On 02.03.12 at 09:05, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2012-03-02 at 07:40 +0000, Jan Beulich wrote:
> >> And if using blkback, why not (as in xend) via loop devices?
> > 
> > Support for block device script= in xl/libxl is on the 4.2 blocker list,
> 
> Ah, that's good to know. (However, you mentioning script= makes me
> assume that one would have to specify this among the block-attach
> options, which again wouldn't be a drop-in replacement of how xm
> worked in this regard.)

We could make a decision about the default script to run based on the
type of the device and the selected backend.

> > this feature would re-enable the loop+blkback case -- I think this would
> > be a better option than blktap* or qdisk once it becomes available.
> 
> Jan
> 

Re: xl block-attach vs block-detach

[Stefano Stabellini]

On Fri, 2 Mar 2012, Ian Campbell wrote:
> > And if using blkback, why not (as in xend) via loop devices?
> 
> Support for block device script= in xl/libxl is on the 4.2 blocker list,
> this feature would re-enable the loop+blkback case -- I think this would
> be a better option than blktap* or qdisk once it becomes available.

Of course we need to make sure that blkback with loop devices performs
well before doing that, and it was certainly not the case in my last tests.

Re: xl block-attach vs block-detach

[Jan Beulich]

>>> On 02.03.12 at 14:49, Stefano Stabellini <stefano.stabellini@eu.citrix.com>
wrote:
> On Fri, 2 Mar 2012, Ian Campbell wrote:
>> > And if using blkback, why not (as in xend) via loop devices?
>> 
>> Support for block device script= in xl/libxl is on the 4.2 blocker list,
>> this feature would re-enable the loop+blkback case -- I think this would
>> be a better option than blktap* or qdisk once it becomes available.
> 
> Of course we need to make sure that blkback with loop devices performs
> well before doing that, and it was certainly not the case in my last tests.

No - no policy should be involved here: If file:/ is specified, one should
get blkback alone (no tap, qdisk, or what not. If another protocol was
specified, that one should be used. Only if guessing is needed, some
sort of heuristic (into which performance considerations may play) will
(naturally) be required.

Jan

Re: xl block-attach vs block-detach

[Jan Beulich]

>>> On 02.03.12 at 15:35, Stefano Stabellini <stefano.stabellini@eu.citrix.com>
wrote:
> On Fri, 2 Mar 2012, Jan Beulich wrote:
>> >>> On 02.03.12 at 14:49, Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>> wrote:
>> > On Fri, 2 Mar 2012, Ian Campbell wrote:
>> >> > And if using blkback, why not (as in xend) via loop devices?
>> >> 
>> >> Support for block device script= in xl/libxl is on the 4.2 blocker list,
>> >> this feature would re-enable the loop+blkback case -- I think this would
>> >> be a better option than blktap* or qdisk once it becomes available.
>> > 
>> > Of course we need to make sure that blkback with loop devices performs
>> > well before doing that, and it was certainly not the case in my last tests.
>> 
>> No - no policy should be involved here: If file:/ is specified, one should
>> get blkback alone (no tap, qdisk, or what not. If another protocol was
>> specified, that one should be used. Only if guessing is needed, some
>> sort of heuristic (into which performance considerations may play) will
>> (naturally) be required.
> 
> Let's suppose we find out that blkback+loop is slower than qdisk and
> that block-attach/detach work well with qdisk by the time of the next
> release.
> 
> What would be the rationale behind using blkback+loop for "file:"?
> Backward compatibility?

Yes.

> Do you think it might break something for users if we change the backend
> from xend to xl?

This cannot be excluded, particularly because (just like me here)
users tend to do things you didn't expect them to when you write
the code.

> On the other hand do you think that using qdisk with the new disk syntax
> introduced with xl is reasonable because users are not supposed to make
> any assumptions there?

Perhaps yes.

Jan

Re: xl block-attach vs block-detach

[Stefano Stabellini]

On Fri, 2 Mar 2012, Jan Beulich wrote:
> >>> On 02.03.12 at 14:49, Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> wrote:
> > On Fri, 2 Mar 2012, Ian Campbell wrote:
> >> > And if using blkback, why not (as in xend) via loop devices?
> >> 
> >> Support for block device script= in xl/libxl is on the 4.2 blocker list,
> >> this feature would re-enable the loop+blkback case -- I think this would
> >> be a better option than blktap* or qdisk once it becomes available.
> > 
> > Of course we need to make sure that blkback with loop devices performs
> > well before doing that, and it was certainly not the case in my last tests.
> 
> No - no policy should be involved here: If file:/ is specified, one should
> get blkback alone (no tap, qdisk, or what not. If another protocol was
> specified, that one should be used. Only if guessing is needed, some
> sort of heuristic (into which performance considerations may play) will
> (naturally) be required.

Let's suppose we find out that blkback+loop is slower than qdisk and
that block-attach/detach work well with qdisk by the time of the next
release.

What would be the rationale behind using blkback+loop for "file:"?
Backward compatibility?
Do you think it might break something for users if we change the backend
from xend to xl?

On the other hand do you think that using qdisk with the new disk syntax
introduced with xl is reasonable because users are not supposed to make
any assumptions there?

Re: xl block-attach vs block-detach

[Stefano Stabellini]

On Fri, 2 Mar 2012, Jan Beulich wrote:
> > What would be the rationale behind using blkback+loop for "file:"?
> > Backward compatibility?
> 
> Yes.
> 
> > Do you think it might break something for users if we change the backend
> > from xend to xl?
> 
> This cannot be excluded, particularly because (just like me here)
> users tend to do things you didn't expect them to when you write
> the code.

I see your point but actually that is quite an obvious bug, not a very
subtle one that only happens in strange user configs.

Re: xl block-attach vs block-detach

[Stefano Stabellini]

On Fri, 2 Mar 2012, Stefano Stabellini wrote:
> On Fri, 2 Mar 2012, Jan Beulich wrote:
> > > What would be the rationale behind using blkback+loop for "file:"?
> > > Backward compatibility?
> > 
> > Yes.
> > 
> > > Do you think it might break something for users if we change the backend
> > > from xend to xl?
> > 
> > This cannot be excluded, particularly because (just like me here)
> > users tend to do things you didn't expect them to when you write
> > the code.
> 
> I see your point but actually that is quite an obvious bug, not a very
> subtle one that only happens in strange user configs.
> 

Scratch that: I have just tried on Linux 3.3 and the performance of
blkback with loopback is very good. We should use it whenever we can.

Re: xl block-attach vs block-detach

[Joseph Glanville]

Hi,

As I understand it prefering tapdisk over loop+blkback has never been
for performance reasons historically. (tapdisk2:aio does however
exhibit very good performance)
The primary reason that tapdisk was always recommended over file: is
that the Linux file cache does very interesting things to your data
and sync is returned to the blkback backend much sooner than the data
actually resides safely on disk (which can sit in the linux disk cache
for a sizeable amount of time if they machine has alot of ram).

Unfortunately changing the default behavior to tapdisk probably isn't
viable at this time for a number of reasons - not least of which is
the fact it is yet to be included in mainline.
However it would definitely be preferable in the long term - atleast
from the perspective of data integrity and principle of least
surprises.

Just my 2c.

Joseph.

On 3 March 2012 04:37, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Fri, 2 Mar 2012, Stefano Stabellini wrote:
>> On Fri, 2 Mar 2012, Jan Beulich wrote:
>> > > What would be the rationale behind using blkback+loop for "file:"?
>> > > Backward compatibility?
>> >
>> > Yes.
>> >
>> > > Do you think it might break something for users if we change the backend
>> > > from xend to xl?
>> >
>> > This cannot be excluded, particularly because (just like me here)
>> > users tend to do things you didn't expect them to when you write
>> > the code.
>>
>> I see your point but actually that is quite an obvious bug, not a very
>> subtle one that only happens in strange user configs.
>>
>
> Scratch that: I have just tried on Linux 3.3 and the performance of
> blkback with loopback is very good. We should use it whenever we can.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel



-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846

Re: xl block-attach vs block-detach

[Ian Campbell]

Please don't top post, it destroys the flow of the conversation.

On Fri, 2012-03-02 at 22:54 +0000, Joseph Glanville wrote:
> Hi,
> 
> As I understand it prefering tapdisk over loop+blkback has never been
> for performance reasons historically. (tapdisk2:aio does however
> exhibit very good performance)
> The primary reason that tapdisk was always recommended over file: is
> that the Linux file cache does very interesting things to your data
> and sync is returned to the blkback backend much sooner than the data
> actually resides safely on disk (which can sit in the linux disk cache
> for a sizeable amount of time if they machine has alot of ram).

Are you suggesting that the loop device doesn't support O_DIRECT and
will leave stuff dirty in the page cache even when direct access is
used? That is worth knowing!

> Unfortunately changing the default behavior to tapdisk

What exactly needs changing?

>  probably isn't
> viable at this time for a number of reasons - not least of which is
> the fact it is yet to be included in mainline.

tapdisk is not going to be included in mainline. The kernel side is
deemed to be non-upstreamble.

Someone is working on a fully userspace version of bkltap which we hope
will be ready soon.

Ian.

> However it would definitely be preferable in the long term - atleast
> from the perspective of data integrity and principle of least
> surprises.
> 
> Just my 2c.
> 
> Joseph.
> 
> On 3 March 2012 04:37, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Fri, 2 Mar 2012, Stefano Stabellini wrote:
> >> On Fri, 2 Mar 2012, Jan Beulich wrote:
> >> > > What would be the rationale behind using blkback+loop for "file:"?
> >> > > Backward compatibility?
> >> >
> >> > Yes.
> >> >
> >> > > Do you think it might break something for users if we change the backend
> >> > > from xend to xl?
> >> >
> >> > This cannot be excluded, particularly because (just like me here)
> >> > users tend to do things you didn't expect them to when you write
> >> > the code.
> >>
> >> I see your point but actually that is quite an obvious bug, not a very
> >> subtle one that only happens in strange user configs.
> >>
> >
> > Scratch that: I have just tried on Linux 3.3 and the performance of
> > blkback with loopback is very good. We should use it whenever we can.
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> 
> 

Re: xl block-attach vs block-detach

[Joseph Glanville]

On 3 March 2012 16:25, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> Please don't top post, it destroys the flow of the conversation.

My apologies, GMail is a very irritating client.

>
> On Fri, 2012-03-02 at 22:54 +0000, Joseph Glanville wrote:
>> Hi,
>>
>> As I understand it prefering tapdisk over loop+blkback has never been
>> for performance reasons historically. (tapdisk2:aio does however
>> exhibit very good performance)
>> The primary reason that tapdisk was always recommended over file: is
>> that the Linux file cache does very interesting things to your data
>> and sync is returned to the blkback backend much sooner than the data
>> actually resides safely on disk (which can sit in the linux disk cache
>> for a sizeable amount of time if they machine has alot of ram).
>
> Are you suggesting that the loop device doesn't support O_DIRECT and
> will leave stuff dirty in the page cache even when direct access is
> used? That is worth knowing!

As far as I am aware this is the case. It doesn't support O_DIRECT in
any capacity.
There was some patches submitted to the list a very long time ago ( I
could probably find them if I tried) but they were knocked back by
upstream.
It is possible this could have changed so I will take a glance at the
source of loop.c tonight to see if that is actually the case.

>
>> Unfortunately changing the default behavior to tapdisk
>
> What exactly needs changing?

I was suggesting that tapdisk would be a much safer default in terms
of data integrity for the above reason that the loop driver doesn't
support O_DIRECT.

>
>>  probably isn't
>> viable at this time for a number of reasons - not least of which is
>> the fact it is yet to be included in mainline.
>
> tapdisk is not going to be included in mainline. The kernel side is
> deemed to be non-upstreamble.
>
> Someone is working on a fully userspace version of bkltap which we hope
> will be ready soon.

That is unfortunate, would you mind pointing me in the direction of
the pure userspace version?
Would the performance be considerably worse than the current implementation?

>
> Ian.
>
>> However it would definitely be preferable in the long term - atleast
>> from the perspective of data integrity and principle of least
>> surprises.
>>
>> Just my 2c.
>>
>> Joseph.
>>
>> On 3 March 2012 04:37, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Fri, 2 Mar 2012, Stefano Stabellini wrote:
>> >> On Fri, 2 Mar 2012, Jan Beulich wrote:
>> >> > > What would be the rationale behind using blkback+loop for "file:"?
>> >> > > Backward compatibility?
>> >> >
>> >> > Yes.
>> >> >
>> >> > > Do you think it might break something for users if we change the backend
>> >> > > from xend to xl?
>> >> >
>> >> > This cannot be excluded, particularly because (just like me here)
>> >> > users tend to do things you didn't expect them to when you write
>> >> > the code.
>> >>
>> >> I see your point but actually that is quite an obvious bug, not a very
>> >> subtle one that only happens in strange user configs.
>> >>
>> >
>> > Scratch that: I have just tried on Linux 3.3 and the performance of
>> > blkback with loopback is very good. We should use it whenever we can.
>> >
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xen.org
>> > http://lists.xen.org/xen-devel
>>
>>
>>
>
>

All of that being said it is still relatively OK to leave blkback+loop
as the default as the majority of users I would assume use actual
block devices (though I could be horribly wrong here).
All documentation and best-practices on the wiki outline this in detail.

Joseph.

-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846

Re: xl block-attach vs block-detach

[Joseph Glanville]

On 3 March 2012 16:25, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> Please don't top post, it destroys the flow of the conversation.
>
> On Fri, 2012-03-02 at 22:54 +0000, Joseph Glanville wrote:
>> Hi,
>>
>> As I understand it prefering tapdisk over loop+blkback has never been
>> for performance reasons historically. (tapdisk2:aio does however
>> exhibit very good performance)
>> The primary reason that tapdisk was always recommended over file: is
>> that the Linux file cache does very interesting things to your data
>> and sync is returned to the blkback backend much sooner than the data
>> actually resides safely on disk (which can sit in the linux disk cache
>> for a sizeable amount of time if they machine has alot of ram).
>
> Are you suggesting that the loop device doesn't support O_DIRECT and
> will leave stuff dirty in the page cache even when direct access is
> used? That is worth knowing!
>
>> Unfortunately changing the default behavior to tapdisk
>
> What exactly needs changing?

Doh! I finally get what you meant by this comment. Tapdisk -is- the
current default for xl.
xm uses loop and thus my mistake, sigh must learn to think before typing.

>
>>  probably isn't
>> viable at this time for a number of reasons - not least of which is
>> the fact it is yet to be included in mainline.
>
> tapdisk is not going to be included in mainline. The kernel side is
> deemed to be non-upstreamble.
>
> Someone is working on a fully userspace version of bkltap which we hope
> will be ready soon.
>
> Ian.
>
>> However it would definitely be preferable in the long term - atleast
>> from the perspective of data integrity and principle of least
>> surprises.
>>
>> Just my 2c.
>>
>> Joseph.
>>
>> On 3 March 2012 04:37, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Fri, 2 Mar 2012, Stefano Stabellini wrote:
>> >> On Fri, 2 Mar 2012, Jan Beulich wrote:
>> >> > > What would be the rationale behind using blkback+loop for "file:"?
>> >> > > Backward compatibility?
>> >> >
>> >> > Yes.
>> >> >
>> >> > > Do you think it might break something for users if we change the backend
>> >> > > from xend to xl?
>> >> >
>> >> > This cannot be excluded, particularly because (just like me here)
>> >> > users tend to do things you didn't expect them to when you write
>> >> > the code.
>> >>
>> >> I see your point but actually that is quite an obvious bug, not a very
>> >> subtle one that only happens in strange user configs.
>> >>
>> >
>> > Scratch that: I have just tried on Linux 3.3 and the performance of
>> > blkback with loopback is very good. We should use it whenever we can.
>> >
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xen.org
>> > http://lists.xen.org/xen-devel
>>
>>
>>
>
>

I did have a trawl through the linux kernel and linux-util repos last
night however and it appears that none of the direct-io/O_DIRECT
patches were ever merged.

Joseph.


-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846

Re: xl block-attach vs block-detach

[Joseph Glanville]

On 4 March 2012 14:01, Joseph Glanville <joseph.glanville@orionvm.com.au> wrote:
> On 3 March 2012 16:25, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> Please don't top post, it destroys the flow of the conversation.
>>
>> On Fri, 2012-03-02 at 22:54 +0000, Joseph Glanville wrote:
>>> Hi,
>>>
>>> As I understand it prefering tapdisk over loop+blkback has never been
>>> for performance reasons historically. (tapdisk2:aio does however
>>> exhibit very good performance)
>>> The primary reason that tapdisk was always recommended over file: is
>>> that the Linux file cache does very interesting things to your data
>>> and sync is returned to the blkback backend much sooner than the data
>>> actually resides safely on disk (which can sit in the linux disk cache
>>> for a sizeable amount of time if they machine has alot of ram).
>>
>> Are you suggesting that the loop device doesn't support O_DIRECT and
>> will leave stuff dirty in the page cache even when direct access is
>> used? That is worth knowing!
>>
>>> Unfortunately changing the default behavior to tapdisk
>>
>> What exactly needs changing?
>
> Doh! I finally get what you meant by this comment. Tapdisk -is- the
> current default for xl.
> xm uses loop and thus my mistake, sigh must learn to think before typing.
>
>>
>>>  probably isn't
>>> viable at this time for a number of reasons - not least of which is
>>> the fact it is yet to be included in mainline.
>>
>> tapdisk is not going to be included in mainline. The kernel side is
>> deemed to be non-upstreamble.
>>
>> Someone is working on a fully userspace version of bkltap which we hope
>> will be ready soon.
>>
>> Ian.
>>
>>> However it would definitely be preferable in the long term - atleast
>>> from the perspective of data integrity and principle of least
>>> surprises.
>>>
>>> Just my 2c.
>>>
>>> Joseph.
>>>
>>> On 3 March 2012 04:37, Stefano Stabellini
>>> <stefano.stabellini@eu.citrix.com> wrote:
>>> > On Fri, 2 Mar 2012, Stefano Stabellini wrote:
>>> >> On Fri, 2 Mar 2012, Jan Beulich wrote:
>>> >> > > What would be the rationale behind using blkback+loop for "file:"?
>>> >> > > Backward compatibility?
>>> >> >
>>> >> > Yes.
>>> >> >
>>> >> > > Do you think it might break something for users if we change the backend
>>> >> > > from xend to xl?
>>> >> >
>>> >> > This cannot be excluded, particularly because (just like me here)
>>> >> > users tend to do things you didn't expect them to when you write
>>> >> > the code.
>>> >>
>>> >> I see your point but actually that is quite an obvious bug, not a very
>>> >> subtle one that only happens in strange user configs.
>>> >>
>>> >
>>> > Scratch that: I have just tried on Linux 3.3 and the performance of
>>> > blkback with loopback is very good. We should use it whenever we can.
>>> >
>>> > _______________________________________________
>>> > Xen-devel mailing list
>>> > Xen-devel@lists.xen.org
>>> > http://lists.xen.org/xen-devel
>>>
>>>
>>>
>>
>>
>
> I did have a trawl through the linux kernel and linux-util repos last
> night however and it appears that none of the direct-io/O_DIRECT
> patches were ever merged.
>
> Joseph.
>
>
> --
> Founder | Director | VP Research
> Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
> 99 52 | Mobile: 0428 754 846

For interested parties this is the original patch series I was reffering to:
http://www.spinics.net/lists/linux-fsdevel/msg27514.html


There also seems to be a revival of this effort as of about a week ago:
https://lkml.org/lkml/2012/2/28/251

Kind regards,
Joseph.

-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846

Re: xl block-attach vs block-detach

[Ian Campbell]

On Sat, 2012-03-03 at 22:46 -0500, Joseph Glanville wrote:
> On 4 March 2012 14:01, Joseph Glanville <joseph.glanville@orionvm.com.au> wrote:
> > On 3 March 2012 16:25, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > I did have a trawl through the linux kernel and linux-util repos last
> > night however and it appears that none of the direct-io/O_DIRECT
> > patches were ever merged.

> For interested parties this is the original patch series I was reffering to:
> http://www.spinics.net/lists/linux-fsdevel/msg27514.html
> 
> 
> There also seems to be a revival of this effort as of about a week ago:
> https://lkml.org/lkml/2012/2/28/251

Thanks for all the useful info and pointers. We should keep an eye on
this series.

Ian.