[PATCH v3 3/4] docs: add pvscsi.txt

[Olaf Hering <olaf@aepfle.de>]

Signed-off-by: Olaf Hering <olaf@aepfle.de>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Tim Deegan <tim@xen.org>
---
 docs/misc/pvscsi.txt | 188 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 188 insertions(+)

diff --git a/docs/misc/pvscsi.txt b/docs/misc/pvscsi.txt
new file mode 100644
index 0000000..988a12b
--- /dev/null
+++ b/docs/misc/pvscsi.txt
@@ -0,0 +1,188 @@
+. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
+PVSCSI
+
+== Overview ==
+
+PVSCSI allows to assing a "physical" SCSI device from dom0 to a domU. The device
+is not limited to be a native SCSI device. Everything visible as SCSI device in
+dom0 can be used. Currently PVSCSI is only available in Linux dom0 and domU.
+
+== TODO ===
+
+How to do live migration?
+ - pdev will likely be evaluated again on the target host if it came from
+   domU.cfg. But what about pdev from 'xl scsi-attach pdev vdev'? Its required
+   to adjust h:c:t:l on the target host.
+
+How to handle FIXME in libxl_retrieve_domain_configuration?
+ - "MERGE(vscsi, vscsis, COMPARE_DEVID, {});", when does this code run?
+ 
+Implement libvirt integration.
+ - Is the API used by xl usable for libvirt?
+ - How does libvirt assign a raw SCSI device?
+   Jim said:
+    > vscsi=[ '/dev/sda,0:1:2:3', '4:3:2:1,1:1:1:1,feature-host' ]
+    In libvirt-speak, this is SCSI host device assignment
+    http://libvirt.org/formatdomain.html#elementsHostDev
+    Here's the SCSI example
+      <devices>
+       <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'>
+        <source>
+         <adapter name='scsi_host0'/>
+         <address type='scsi' bus='0' target='0' unit='0'/>
+        </source>
+        <readonly/>
+        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
+       </hostdev>
+      ...
+      </devices>
+    I suppose 'feature-host' maps to attributes such as 'sgio' and 'rawio'.
+    The source element describes the SCSI device as seen from the host.
+
+
+== History ==
+
+Support for PVSCSI was added to the classic xenlinux kernel and to xend in
+xen-3.3 in 2008. Support for PVSCSI was added to the upstream pvops kernel 3.18
+in 2014. Backends exists just for Linux. A frontend exists for Linux, and maybe
+for FreeBSD according to google.
+
+
+== Config Format ==
+
+The domU.cfg syntax is "vscsi=[ 'pdev,vdev[,option]', 'pdev,vdev[,option]' ]".
+'pdev' describes the physical device, preferable in a persistant format.
+'vdev' is the domU device in vHOST:CHANNEL:TARGET:LUN notation, all integers.
+'option' lists additional flags which a backend may recognize.
+
+
+== Configuring xenlinux backend ==
+
+"pdev" is the dom0 device in either /dev/scsidev or pHOST:CHANNEL:TARGET:LUN
+notation.  Currently only the option value "feature-host" is recognized. SCSI
+command emulation in backend driver is bypassed when "feature-host" is
+specified.
+
+Its recommended to use persistant names "/dev/disk/by-*/*" to refer to a "pdev".
+The toolstack will translate this internally to "h:c:t:l" notation, which is how
+the backend driver will access the device. Using the "h:c:t:l" notation for
+"pdev" in domU.cfg is discouraged because this value will change across reboots,
+depending on the detection order in the OS.
+
+
+== Configuring pvops backend ==
+
+The backend driver in the pvops kernel is part of the Linux-IO Target framework
+(LIO).  As such the SCSI devices have to be configured first with the tools
+provided by this framework, such as targetcli. The "pdev" in domU.cfg has to
+refer to a config item in that framework instead of the raw device.
+
+The LIO uses a concept of backstore and frontend drivers. LIO backstore drivers
+provide access to the actual storage, LIO frontend drivers provide an interface
+which is accessed with SCSI commands. Within the LIO the frontend is connected
+to a certain backstore.
+
+For Xen its recommended to use the PSCSI backstore. Other backstores may work as
+well, but they are not tested. The xen-scsiback module is the LIO frontend, and
+at the same time the backend for the domU. In general the config flow is:
+- the LIO backstore refers to physical storage in a persistant way
+- the given storage gets a free, unique name within the LIO backstore
+- the LIO frontend refers to that free, unique name in the LIO backstore
+- the connection to the LIO backstore gets a naa.WWN:LUN in the LIO frontend
+- the "pdev" in domU.cfg refers to that naa.WWN:LUN in the LIO frontend
+
+Example for targetcli and a xen-scsiback aware python-rtslib:
+
+ # modprobe -v xen-scsiback
+ # DEV=/dev/disk/by-id/ata-TSSTcorp_DVD+_-RW_TS-H653G_R4576GFZ638411
+ # NAME="my_dvd"
+ # NAA=$(printf "naa.%s\n" `od -N 8 -t x8 -A n < /dev/urandom`)
+ # targetcli "/backstores/pscsi create name=$NAME dev=${DEV}"
+ # targetcli "/xen-pvscsi create wwn=$NAA"
+ # targetcli "/xen-pvscsi/$NAA/tpg1/luns create /backstores/pscsi/$NAME"
+ # echo "vscsi=[ '${NAA}:0,0:0:0:0' ]"
+
+The last line is the domU.cfg snippet.
+
+
+== Xenstore Format ==
+
+The backend driver uses a single_host:many_devices notation to manage domU
+devices. Everything is stored in /local/domain/0/backend/vscsi/. The xenstore
+layout looks like this:
+
+    <domid>/<vhost>/feature-host = "0"
+    <domid>/<vhost>/frontend = "/local/domain/<domid>/device/vscsi/0"
+    <domid>/<vhost>/frontend-id = "<domid>"
+    <domid>/<vhost>/online = "1"
+    <domid>/<vhost>/state = "4"
+    <domid>/<vhost>/vscsi-devs/dev-0/p-dev = "8:0:2:1" or "naa.wwn:lun"
+    <domid>/<vhost>/vscsi-devs/dev-0/state = "4"
+    <domid>/<vhost>/vscsi-devs/dev-0/v-dev = "0:0:0:0"
+    <domid>/<vhost>/vscsi-devs/dev-1/p-dev = "8:0:2:2"
+    <domid>/<vhost>/vscsi-devs/dev-1/state = "4"
+    <domid>/<vhost>/vscsi-devs/dev-1/v-dev = "0:0:1:0"
+
+The frontend driver maintains its state in /local/domain/<domid>/device/vscsi/.
+
+    <vhost>/backend = "/local/domain/0/backend/vscsi/<domid>/<vhost>"
+    <vhost>/backend-id = "0"
+    <vhost>/event-channel = "20"
+    <vhost>/ring-ref = "43"
+    <vhost>/state = "4"
+    <vhost>/vscsi-devs/dev-0/state = "4"
+    <vhost>/vscsi-devs/dev-1/state = "4"
+
+In addition to the entries for backend and frontend these flags are stored for
+the toolstack:
+
+    <domid>/<vhost>/vscsi-devs/dev-1/p-devname = "/dev/$device"
+
+
+== Backend - Frontend Protocol ==
+
+To create a vhost along with a device:
+    <domid>/<vhost>/feature-host = "0"
+    <domid>/<vhost>/frontend = "/local/domain/<domid>/device/vscsi/0"
+    <domid>/<vhost>/frontend-id = "<domid>"
+    <domid>/<vhost>/online = "1"
+    <domid>/<vhost>/state = "1"
+    <domid>/<vhost>/vscsi-devs/dev-0/p-dev = "8:0:2:1"
+    <domid>/<vhost>/vscsi-devs/dev-0/state = "1"
+    <domid>/<vhost>/vscsi-devs/dev-0/v-dev = "0:0:0:0"
+Wait for <domid>/<vhost>/state + <domid>/<vhost>/vscsi-devs/dev-0/state become 4
+
+To add another device to a vhost:
+    <domid>/<vhost>/state = "7"
+    <domid>/<vhost>/vscsi-devs/dev-1/p-dev = "8:0:2:2"
+    <domid>/<vhost>/vscsi-devs/dev-1/state = "1"
+    <domid>/<vhost>/vscsi-devs/dev-1/v-dev = "0:0:1:0"
+Wait for <domid>/<vhost>/state + <domid>/<vhost>/vscsi-devs/dev-1/state become 4
+
+To remove a device from a vhost:
+    <domid>/<vhost>/state = "7"
+    <domid>/<vhost>/vscsi-devs/dev-1/state = "5"
+Wait for <domid>/<vhost>/state to become 4
+Wait for <domid>/<vhost>/vscsi-devs/dev-1/state become 6
+Remove <domid>/<vhost>/vscsi-devs/dev-1/{state,p-dev,v-dev,p-devname}
+Remove <domid>/<vhost>/vscsi-devs/dev-1/
+
+
+== Interface in xl ==
+
+xl scsi-attach domid <PhysDevice> <VirtDevice>[,<option>]
+xl scsi-detach domid <VirtDevice>
+xl scsi-list domid [domid, ...]
+
+
+== Interface in libxl ==
+
+To be defined so that both xl and libvirt pass in the same data.
+
+
+== Interface in libvirt ==
+
+Has yet to be found.
+
+. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
+# vim: tw=80 et ts=4