Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Jeff Garzik]

Ian Pratt wrote:
>>This is another thing that has always put me off.  The 
>>virtual block device driver has the ability to masquerade as 
>>other types of block devices.  It actually claims to be an 
>>IDE or SCSI device allocating the appropriate major/minor numbers.
>>
>>This seems to be pretty evil and creating interesting failure 
>>conditions for users who load IDE or SCSI modules.  I've seen 
>>it trip up a number of people in the past.  I think we should 
>>only ever use the major number that was actually allocated to us.
> 
> 
> We certainly should be pushing everyone toward using the 'xdX' etc
> devices that are allocated to us. However, the installers of certain
> older distros and other user space tools won't except anything other
> than hdX/sdX, so its useful from a compatibility POV even if it never
> goes into mainline, which I agree it probably shouldn't. 

Yes, this is true.  Red Hat installer guys grumbled at me when I wrote 
the 'sx8' block driver:  since it wasn't hda/sda, they had to write 
special code for it, as they apparently must do for any new block driver 
"class".  SuSE and other distros are probably similar, since each block 
driver provides its own special behaviors and feature exports.

I should have spoken up a long time ago about this, but anyway:

An IBM hypervisor on ppc64 communicates uses SCSI RPC messages.  I think 
this would be quite nice for Xen, because SCSI (a) is a message-based 
model, and (b) implementing block using SCSI has a very high Just 
Works(tm) value which cannot be ignored.  And perhaps (c) SCSI target 
code already exists, so implementing the server side doesn't require 
starting from scratch, but rather simply connecting the Legos.

	Jeff

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Alan Cox]

On Mer, 2006-03-22 at 18:09 -0500, Jeff Garzik wrote:
> An IBM hypervisor on ppc64 communicates uses SCSI RPC messages.  I think 
> this would be quite nice for Xen, because SCSI (a) is a message-based 
> model, and (b) implementing block using SCSI has a very high Just 
> Works(tm) value which cannot be ignored.  And perhaps (c) SCSI target 
> code already exists, so implementing the server side doesn't require 
> starting from scratch, but rather simply connecting the Legos.

A pure SCSI abstraction doesn't allow for shared head scheduling which
you will need to scale Xen sanely on typical PC boxes. SCSI emulations
are also always full of bits people got wrong, often critical bits like
tagged queues and error sequences - things that break your journalled
file system.

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Jeff Garzik]

Alan Cox wrote:
> On Mer, 2006-03-22 at 18:09 -0500, Jeff Garzik wrote:
>> An IBM hypervisor on ppc64 communicates uses SCSI RPC messages.  I think 
>> this would be quite nice for Xen, because SCSI (a) is a message-based 
>> model, and (b) implementing block using SCSI has a very high Just 
>> Works(tm) value which cannot be ignored.  And perhaps (c) SCSI target 
>> code already exists, so implementing the server side doesn't require 
>> starting from scratch, but rather simply connecting the Legos.
> 
> A pure SCSI abstraction doesn't allow for shared head scheduling which
> you will need to scale Xen sanely on typical PC boxes.

Not true at all.  If you can do it with a block device, you can do it 
with a SCSI block device.

In fact, SCSI should make a few things easier, because the notion of 
host+bus topology is already present, and notion of messaging is already 
present, so you don't have to recreate that in a Xen block device 
infrastructure.


> SCSI emulations
> are also always full of bits people got wrong, often critical bits like
> tagged queues and error sequences - things that break your journalled
> file system.

This I'll grant you.

	Jeff

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Jeff Garzik]

Jeff Garzik wrote:
> In fact, SCSI should make a few things easier, because the notion of 
> host+bus topology is already present, and notion of messaging is already 
> present, so you don't have to recreate that in a Xen block device 
> infrastructure.

Another benefit of SCSI:  when an IBM hypervisor in the Linux kernel 
switched to SCSI, that allowed them to replace several drivers (virt 
disk, virt cdrom, virt floppy?) with a single virt-SCSI driver.

	Jeff

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Arjan van de Ven]

On Fri, 2006-03-24 at 08:37 -0500, Jeff Garzik wrote:
> Jeff Garzik wrote:
> > In fact, SCSI should make a few things easier, because the notion of 
> > host+bus topology is already present, and notion of messaging is already 
> > present, so you don't have to recreate that in a Xen block device 
> > infrastructure.
> 
> Another benefit of SCSI:  when an IBM hypervisor in the Linux kernel 
> switched to SCSI, that allowed them to replace several drivers (virt 
> disk, virt cdrom, virt floppy?) with a single virt-SCSI driver.

but there's a generic one for that: iSCSI
so in theory you only need to provide a network driver then ;)

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Jeff Garzik]

Arjan van de Ven wrote:
> On Fri, 2006-03-24 at 08:37 -0500, Jeff Garzik wrote:
>> Jeff Garzik wrote:
>>> In fact, SCSI should make a few things easier, because the notion of 
>>> host+bus topology is already present, and notion of messaging is already 
>>> present, so you don't have to recreate that in a Xen block device 
>>> infrastructure.
>> Another benefit of SCSI:  when an IBM hypervisor in the Linux kernel 
>> switched to SCSI, that allowed them to replace several drivers (virt 
>> disk, virt cdrom, virt floppy?) with a single virt-SCSI driver.

> but there's a generic one for that: iSCSI
> so in theory you only need to provide a network driver then ;)

Talk about lots of overhead :)

OTOH, I bet that T10 is acting at high speed, right this second, to form 
a committee, and multiple sub-committees, to standardize SCSI 
transported over XenBus.  SXP anyone?  :)

	Jeff

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Dave C Boutcher]

Jeff Garzik wrote:
>Arjan van de Ven wrote:
>> On Fri, 2006-03-24 at 08:37 -0500, Jeff Garzik wrote:
>>> Jeff Garzik wrote:
>>>> In fact, SCSI should make a few things easier, because the notion of 
>>>> host+bus topology is already present, and notion of messaging is already 
>>>> present, so you don't have to recreate that in a Xen block device 
>>>> infrastructure.
>>> Another benefit of SCSI:  when an IBM hypervisor in the Linux kernel 
>>> switched to SCSI, that allowed them to replace several drivers (virt 
>>> disk, virt cdrom, virt floppy?) with a single virt-SCSI driver.
>
>> but there's a generic one for that: iSCSI
>> so in theory you only need to provide a network driver then ;)
>
>Talk about lots of overhead :)
>
>OTOH, I bet that T10 is acting at high speed, right this second, to form 
>a committee, and multiple sub-committees, to standardize SCSI 
>transported over XenBus.  SXP anyone?  :)

Actually SRP (which T10 has now stopped working on) fits the bill very
nicely.

I have to say that moving the IBM virtual drivers from a random
collection of unique drivers (viodisk, viotape, viocd) to a single
virtual SCSI HBA made life much easier.

There is a group (actually, at least two groups) working on SCSI
target infrastructures...once that is in place, I would expect we
could start hacking a Xen virtual HBA.

We looked at iSCSI as a transport (instead of SRP) but we felt that 
the added complexity made it unlikely that the average human could
successfully boot their virtual machine

Dave B

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Alan Cox]

On Gwe, 2006-03-24 at 07:38 -0500, Jeff Garzik wrote:
> > A pure SCSI abstraction doesn't allow for shared head scheduling which
> > you will need to scale Xen sanely on typical PC boxes.
> 
> Not true at all.  If you can do it with a block device, you can do it 
> with a SCSI block device.

I don't believe this is true. The complexity of expressing sequences of
command ordering between virtual machines acting in a co-operative but
secure manner isn't as far as I can see expressable sanely in SCSI TCQ
> 
> In fact, SCSI should make a few things easier, because the notion of 
> host+bus topology is already present, and notion of messaging is already 
> present, so you don't have to recreate that in a Xen block device 
> infrastructure.

Those are the easy bits. 

> > are also always full of bits people got wrong, often critical bits like
> > tagged queues and error sequences - things that break your journalled
> > file system.
> 
> This I'll grant you.

And every one you get wrong is a corruptor....

Alan

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Mike Christie]

Dave C Boutcher wrote:
> Jeff Garzik wrote:
>> Arjan van de Ven wrote:
>>> On Fri, 2006-03-24 at 08:37 -0500, Jeff Garzik wrote:
>>>> Jeff Garzik wrote:
>>>>> In fact, SCSI should make a few things easier, because the notion of 
>>>>> host+bus topology is already present, and notion of messaging is already 
>>>>> present, so you don't have to recreate that in a Xen block device 
>>>>> infrastructure.
>>>> Another benefit of SCSI:  when an IBM hypervisor in the Linux kernel 
>>>> switched to SCSI, that allowed them to replace several drivers (virt 
>>>> disk, virt cdrom, virt floppy?) with a single virt-SCSI driver.
>>> but there's a generic one for that: iSCSI
>>> so in theory you only need to provide a network driver then ;)
>> Talk about lots of overhead :)
>>
>> OTOH, I bet that T10 is acting at high speed, right this second, to form 
>> a committee, and multiple sub-committees, to standardize SCSI 
>> transported over XenBus.  SXP anyone?  :)
> 
> Actually SRP (which T10 has now stopped working on) fits the bill very
> nicely.
> 

Does the IBM vscsi code/SPEC follow the SRP SPEC or is it slightly 
modified? We also have a SRP initiator in kernel now too. It is just not 
in the drivers/scsi dir.

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Dave C Boutcher]

Mike Christie wrote:
> Does the IBM vscsi code/SPEC follow the SRP SPEC or is it slightly 
> modified? We also have a SRP initiator in kernel now too. It is just not 
> in the drivers/scsi dir.

The goal was to follow the SRP spec 100%.  We added one other optional
command set (different protocol identifier than SRP) to exchange some
information like "who is at the other end", but the intent was that
the SRP part was right from the spec.

I think, since we implemented this in three operating systems (Linux,
AIX, and OS/400) using the T10 spec as the reference that we are probably
pretty close.

And yeah, I'm aware that there is another SRP implementation in the
kernel...Merging would be good...

Dave B

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[FUJITA Tomonori]

From: boutcher@cs.umn.edu (Dave C Boutcher)
Subject: Re: [RFC PATCH 35/35] Add Xen virtual block device driver.
Date: Fri, 24 Mar 2006 13:19:56 -0600

> 
> Mike Christie wrote:
> > Does the IBM vscsi code/SPEC follow the SRP SPEC or is it slightly 
> > modified? We also have a SRP initiator in kernel now too. It is just not 
> > in the drivers/scsi dir.
> 
> The goal was to follow the SRP spec 100%.  We added one other optional
> command set (different protocol identifier than SRP) to exchange some
> information like "who is at the other end", but the intent was that
> the SRP part was right from the spec.
> 
> I think, since we implemented this in three operating systems (Linux,
> AIX, and OS/400) using the T10 spec as the reference that we are probably
> pretty close.

About the target side, the lun structure is very different the spec
(tgt implements this as a user-space library).


> And yeah, I'm aware that there is another SRP implementation in the
> kernel...Merging would be good...

Do you have any plans for this?

I've been thinking about writing something like scsi_transport_srp,
which can help the initiator and target drivers. I like to enable tgt
to support RDMA-capable adapters.

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Roland Dreier]

    Dave> And yeah, I'm aware that there is another SRP implementation
    Dave> in the kernel...Merging would be good...

Changing the ibmvscsi driver to use the include/scsi/srp.h header file
at least is on my list of things to do.  Probably a 2.6.18 type of thing.

 - R.

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Rusty Russell]

On Fri, 2006-03-24 at 15:55 +0000, Alan Cox wrote:
> On Gwe, 2006-03-24 at 07:38 -0500, Jeff Garzik wrote:
> > > A pure SCSI abstraction doesn't allow for shared head scheduling which
> > > you will need to scale Xen sanely on typical PC boxes.
> > 
> > Not true at all.  If you can do it with a block device, you can do it 
> > with a SCSI block device.
> 
> I don't believe this is true. The complexity of expressing sequences of
> command ordering between virtual machines acting in a co-operative but
> secure manner isn't as far as I can see expressable sanely in SCSI TCQ

I thought usb_scsi taught us that SCSI was overkill for a block
abstraction?  I have a much simpler Xen block-device implementation
which seems to perform OK, and is a lot less LOC than the in-tree one,
so I don't think the "SCSI would be better than what's there" (while
possibly true) is valid.

Cheers!
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: [RFC PATCH 35/35] Add Xen virtual block device driver.

[Peter Chubb]

>>>>> "Jeff" == Jeff Garzik <jeff@garzik.org> writes:

Jeff> Ian Pratt wrote:
>> 
>> We certainly should be pushing everyone toward using the 'xdX' etc
>> devices that are allocated to us. However, the installers of
>> certain older distros and other user space tools won't except
>> anything other than hdX/sdX, so its useful from a compatibility POV
>> even if it never goes into mainline, which I agree it probably
>> shouldn't.

Jeff> Yes, this is true.  Red Hat installer guys grumbled at me when I
Jeff> wrote the 'sx8' block driver: since it wasn't hda/sda, they had
Jeff> to write special code for it, as they apparently must do for any
Jeff> new block driver "class".  SuSE and other distros are probably
Jeff> similar, since each block driver provides its own special
Jeff> behaviors and feature exports.

Jeff> I should have spoken up a long time ago about this, but anyway:

Jeff> An IBM hypervisor on ppc64 communicates uses SCSI RPC messages.
Jeff> I think this would be quite nice for Xen, because SCSI (a) is a
Jeff> message-based model, and (b) implementing block using SCSI has a
Jeff> very high Just Works(tm) value which cannot be ignored.  And
Jeff> perhaps (c) SCSI target code already exists, so implementing the
Jeff> server side doesn't require starting from scratch, but rather
Jeff> simply connecting the Legos.

The IA64 virtualisation work (Xen and Linux-on-Linux) uses the SKI
simulator virtual scsi device --- which looks just like any other scsi
disk, but uses hypervisor calls to do read/write/open/close calls like
a user-mode process.  For performance, it needs to be extended a bit
to do asynchronous I/O and interrupt on completion.  As a halfway
house, the ski simscsi driver would be fairly easy to port, I think.

-- 
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au           ERTOS within National ICT Australia