[PATCH v5 00/17] Introduce Xen support on ARM (based on 3.6-rc5)

[PATCH v5 00/17] Introduce Xen support on ARM (based on 3.6-rc5)

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 7062 bytes --]

Hi all,
this patch series implements Xen support for ARMv7 with virtualization
extensions.  It allows a Linux guest to boot as dom0 and
as domU on Xen on ARM. PV console, disk and network frontends and
backends are all working correctly.

It has been tested on a Versatile Express Cortex A15 emulator, using the
latest Xen ARM developement branch
(git://xenbits.xen.org/people/ianc/xen-unstable.git arm-for-4.3) plus
the "ARM hypercall ABI: 64 bit ready" patch series
(http://marc.info/?l=xen-devel&m=134426267205408), and a simple ad-hoc
tool to build guest domains (marc.info/?l=xen-devel&m=134089788016546).

The patch marked with [HACK] has been dropped from this series, however
you can find it here:
http://marc.info/?l=linux-kernel&m=134513277823527&w=2.

I am also attaching to this email the dts'es that I am currently using
for dom0 and domU: vexpress-v2p-ca15-tc1.dts (that includes
vexpress-v2m-rs1-rtsm.dtsi) is the dts used for dom0 and it is passed to
Linux by Xen, while vexpress-virt.dts is the dts used for other domUs
and it is appended in binary form to the guest kernel image. I am not
sure where they are supposed to live yet, so I am just attaching them
here so that people can actually try out this series if they want to.

Comments are very welcome!



The patch series has been rebased on Konrad's stable/for-linus-3.7
branch.

Arnd, Russell, what do you think about this series? If you are OK with
it, to whom should I submit it?



Changes in v5:
- rebase on Konrad's stable/for-linus-3.7;
- add a comment about the size of the grant table memory region in the
  device tree documentation;
- add a comment about the required presence of a GIC node in the device
  tree documentation;
- specify that the described properties are part of a top-level
  "hypervisor" node in the device tree documentation;
- specify #address-cells and #size-cells for the device tree example;
- make XEN_DOM0 depend on XEN;
- avoid "select XEN_DOM0" in XEN.


Changes in v4:
- rebase on 3.6-rc5;
- devicetree: "xen,xen" should be last as it is less specific;
- devicetree: use 2 address-cells and 2 size-cells in the reg property;
- do not xs_reset_watches on dom0;
- compile drivers/xen/pcpu.c only on x86;
- use "+=" instead of ":=" for dom0- targets;
- add a patch to update the MAINTAINERS file.


Changes in v3:
- move patches that have been picked up by Konrad at the end of the
  series;
- improve comments;
- add a doc to describe the Xen Device Tree format;
- do not use xen_ulong_t for multicalls and apic_physbase;
- add a patch at the end of the series to use the new __HVC macro;
- add missing pvclock-abi.h include to ia64 header files;
- do not use an anonymous union in struct xen_add_to_physmap.


Changes in v2:
- fix up many comments and commit messages;
- remove the early_printk patches: rely on the emulated serial for now;
- remove the xen_guest_init patch: without any PV early_printk, we don't
  need any early call to xen_guest_init, we can rely on core_initcall
  alone;
- define an HYPERCALL macro for 5 arguments hypercall wrappers, even if
  at the moment is unused;
- use ldm instead of pop in the hypercall wrappers;
- return -ENOSYS rather than -1 from the unimplemented grant_table
  functions;
- remove the pvclock ifdef in the Xen headers;
- remove include linux/types.h from xen/interface/xen.h;
- replace pr_info with pr_debug in xen_guest_init;
- add a new patch to introduce xen_ulong_t and use it top replace all
  the occurences of unsigned long in the public Xen interface;
- explicitely size all the pointers to 64 bit on ARM, so that the
  hypercall ABI is "64 bit ready";
- clean up xenbus_init;
- make pci.o depend on CONFIG_PCI and acpi.o depend on CONFIG_ACPI;
- mark Xen guest support on ARM as EXPERIMENTAL;
- introduce GRANT_TABLE_PHYSADDR;
- remove unneeded initialization of boot_max_nr_grant_frames;
- add a new patch to clear IRQ_NOAUTOEN and IRQ_NOREQUEST in events.c;
- return -EINVAL from xen_remap_domain_mfn_range if
  auto_translated_physmap;
- retain binary compatibility in xen_add_to_physmap: use a union to
  introduce foreign_domid.



Shortlog and diffstat:

Stefano Stabellini (17):
      arm: initial Xen support
      xen/arm: hypercalls
      xen/arm: page.h definitions
      xen/arm: sync_bitops
      xen/arm: empty implementation of grant_table arch specific functions
      docs: Xen ARM DT bindings
      xen/arm: Xen detection and shared_info page mapping
      xen/arm: Introduce xen_ulong_t for unsigned long
      xen: do not compile manage, balloon, pci, acpi, pcpu and cpu_hotplug on ARM
      xen/arm: introduce CONFIG_XEN on ARM
      xen/arm: get privilege status
      xen/arm: initialize grant_table on ARM
      xen/arm: receive Xen events on ARM
      xen/arm: implement alloc/free_xenballooned_pages with alloc_pages/kfree
      xen/arm: compile blkfront and blkback
      xen/arm: compile netback
      MAINTAINERS: add myself as Xen ARM maintainer

 Documentation/devicetree/bindings/arm/xen.txt |   25 ++++
 MAINTAINERS                                   |    7 +
 arch/arm/Kconfig                              |   10 ++
 arch/arm/Makefile                             |    1 +
 arch/arm/include/asm/hypervisor.h             |    6 +
 arch/arm/include/asm/sync_bitops.h            |   27 ++++
 arch/arm/include/asm/xen/events.h             |   18 +++
 arch/arm/include/asm/xen/hypercall.h          |   69 ++++++++++
 arch/arm/include/asm/xen/hypervisor.h         |   19 +++
 arch/arm/include/asm/xen/interface.h          |   73 +++++++++++
 arch/arm/include/asm/xen/page.h               |   82 ++++++++++++
 arch/arm/xen/Makefile                         |    1 +
 arch/arm/xen/enlighten.c                      |  168 +++++++++++++++++++++++++
 arch/arm/xen/grant-table.c                    |   53 ++++++++
 arch/arm/xen/hypercall.S                      |  106 ++++++++++++++++
 arch/ia64/include/asm/xen/interface.h         |    1 +
 arch/x86/include/asm/xen/interface.h          |    1 +
 arch/x86/xen/enlighten.c                      |    1 +
 arch/x86/xen/irq.c                            |    1 +
 arch/x86/xen/xen-ops.h                        |    1 -
 drivers/block/xen-blkback/blkback.c           |    1 +
 drivers/net/xen-netback/netback.c             |    1 +
 drivers/net/xen-netfront.c                    |    1 +
 drivers/xen/Makefile                          |   13 ++-
 drivers/xen/events.c                          |   17 ++-
 include/xen/events.h                          |    2 +
 include/xen/interface/features.h              |    3 +
 include/xen/interface/io/protocols.h          |    3 +
 include/xen/interface/memory.h                |   12 +-
 include/xen/interface/physdev.h               |    2 +-
 include/xen/interface/version.h               |    2 +-
 include/xen/xen.h                             |    2 +-
 32 files changed, 712 insertions(+), 17 deletions(-)


A branch based on 3.6-rc5 and Konrad's stable/for-linus-3.7 is available
here:

git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git 3.6-rc5-arm-5


Cheers,

Stefano

[-- Attachment #2: vexpress-v2m-rs1.dtsi --]
[-- Type: text/plain, Size: 4613 bytes --]

/*
 * ARM Ltd. Versatile Express
 *
 * Motherboard Express uATX
 * V2M-P1
 *
 * HBI-0190D
 *
 * RS1 memory map ("ARM Cortex-A Series memory map" in the board's
 * Technical Reference Manual)
 *
 * WARNING! The hardware described in this file is independent from the
 * original variant (vexpress-v2m.dtsi), but there is a strong
 * correspondence between the two configurations.
 *
 * TAKE CARE WHEN MAINTAINING THIS FILE TO PROPAGATE ANY RELEVANT
 * CHANGES TO vexpress-v2m.dtsi!
 */

/ {
	aliases {
		arm,v2m_timer = &v2m_timer01;
	};

	motherboard {
		compatible = "simple-bus";
		arm,v2m-memory-map = "rs1";
		#address-cells = <2>; /* SMB chipselect number and offset */
		#size-cells = <1>;
		#interrupt-cells = <1>;

		flash@0,00000000 {
			compatible = "arm,vexpress-flash", "cfi-flash";
			reg = <0 0x00000000 0x04000000>,
			      <4 0x00000000 0x04000000>;
			bank-width = <4>;
		};

		psram@1,00000000 {
			compatible = "arm,vexpress-psram", "mtd-ram";
			reg = <1 0x00000000 0x02000000>;
			bank-width = <4>;
		};

		vram@2,00000000 {
			compatible = "arm,vexpress-vram";
			reg = <2 0x00000000 0x00800000>;
		};

		ethernet@2,02000000 {
			compatible = "smsc,lan91c111";
			reg = <2 0x02000000 0x10000>;
			interrupts = <15>;
		};

		usb@2,03000000 {
			compatible = "nxp,usb-isp1761";
			reg = <2 0x03000000 0x20000>;
			interrupts = <16>;
			port1-otg;
		};

		iofpga@3,00000000 {
			compatible = "arm,amba-bus", "simple-bus";
			#address-cells = <1>;
			#size-cells = <1>;
			ranges = <0 3 0 0x200000>;

			sysreg@010000 {
				compatible = "arm,vexpress-sysreg";
				reg = <0x010000 0x1000>;
			};

			sysctl@020000 {
				compatible = "arm,sp810", "arm,primecell";
				reg = <0x020000 0x1000>;
			};

			/* PCI-E I2C bus */
			v2m_i2c_pcie: i2c@030000 {
				compatible = "arm,versatile-i2c";
				reg = <0x030000 0x1000>;

				#address-cells = <1>;
				#size-cells = <0>;

				pcie-switch@60 {
					compatible = "idt,89hpes32h8";
					reg = <0x60>;
				};
			};

			aaci@040000 {
				compatible = "arm,pl041", "arm,primecell";
				reg = <0x040000 0x1000>;
				interrupts = <11>;
			};

			mmci@050000 {
				compatible = "arm,pl180", "arm,primecell";
				reg = <0x050000 0x1000>;
				interrupts = <9 10>;
			};

			kmi@060000 {
				compatible = "arm,pl050", "arm,primecell";
				reg = <0x060000 0x1000>;
				interrupts = <12>;
			};

			kmi@070000 {
				compatible = "arm,pl050", "arm,primecell";
				reg = <0x070000 0x1000>;
				interrupts = <13>;
			};

			v2m_serial0: uart@090000 {
				compatible = "arm,pl011", "arm,primecell";
				reg = <0x090000 0x1000>;
				interrupts = <5>;
			};

			v2m_serial1: uart@0a0000 {
				compatible = "arm,pl011", "arm,primecell";
				reg = <0x0a0000 0x1000>;
				interrupts = <6>;
			};

			v2m_serial2: uart@0b0000 {
				compatible = "arm,pl011", "arm,primecell";
				reg = <0x0b0000 0x1000>;
				interrupts = <7>;
			};

			v2m_serial3: uart@0c0000 {
				compatible = "arm,pl011", "arm,primecell";
				reg = <0x0c0000 0x1000>;
				interrupts = <8>;
			};

			wdt@0f0000 {
				compatible = "arm,sp805", "arm,primecell";
				reg = <0x0f0000 0x1000>;
				interrupts = <0>;
			};

			v2m_timer01: timer@110000 {
				compatible = "arm,sp804", "arm,primecell";
				reg = <0x110000 0x1000>;
				interrupts = <2>;
			};

			v2m_timer23: timer@120000 {
				compatible = "arm,sp804", "arm,primecell";
				reg = <0x120000 0x1000>;
				interrupts = <3>;
			};

			/* DVI I2C bus */
			v2m_i2c_dvi: i2c@160000 {
				compatible = "arm,versatile-i2c";
				reg = <0x160000 0x1000>;

				#address-cells = <1>;
				#size-cells = <0>;

				dvi-transmitter@39 {
					compatible = "sil,sii9022-tpi", "sil,sii9022";
					reg = <0x39>;
				};

				dvi-transmitter@60 {
					compatible = "sil,sii9022-cpi", "sil,sii9022";
					reg = <0x60>;
				};
			};

			rtc@170000 {
				compatible = "arm,pl031", "arm,primecell";
				reg = <0x170000 0x1000>;
				interrupts = <4>;
			};

			compact-flash@1a0000 {
				compatible = "arm,vexpress-cf", "ata-generic";
				reg = <0x1a0000 0x100
				       0x1a0100 0xf00>;
				reg-shift = <2>;
			};

			clcd@1f0000 {
				compatible = "arm,pl111", "arm,primecell";
				reg = <0x1f0000 0x1000>;
				interrupts = <14>;
			};
		};

		v2m_fixed_3v3: fixedregulator@0 {
			compatible = "regulator-fixed";
			regulator-name = "3V3";
			regulator-min-microvolt = <3300000>;
			regulator-max-microvolt = <3300000>;
			regulator-always-on;
		};
	};
};

[-- Attachment #3: vexpress-v2p-ca15-tc1.dts --]
[-- Type: text/plain, Size: 3200 bytes --]

/*
 * ARM Ltd. Versatile Express
 *
 * CoreTile Express A15x2 (version with Test Chip 1)
 * Cortex-A15 MPCore (V2P-CA15)
 *
 * HBI-0237A
 */

/dts-v1/;

/ {
	model = "V2P-CA15";
	arm,hbi = <0x237>;
	compatible = "arm,vexpress,v2p-ca15,tc1", "arm,vexpress,v2p-ca15", "arm,vexpress";
	interrupt-parent = <&gic>;
	#address-cells = <2>;
	#size-cells = <2>;

	chosen {
                 bootargs = "dom0_mem=128M";
                 xen,dom0-bootargs = "earlyprintk console=ttyAMA1 root=/dev/mmcblk0 debug rw";
	};

	aliases {
		serial0 = &v2m_serial0;
		serial1 = &v2m_serial1;
		serial2 = &v2m_serial2;
		serial3 = &v2m_serial3;
		i2c0 = &v2m_i2c_dvi;
		i2c1 = &v2m_i2c_pcie;
	};

	cpus {
		#address-cells = <1>;
		#size-cells = <0>;

		cpu@0 {
			device_type = "cpu";
			compatible = "arm,cortex-a15";
			reg = <0>;
		};
	};

	memory@80000000 {
		device_type = "memory";
		reg = <0 0x80000000 0 0x80000000>;
	};

	gic: interrupt-controller@2c001000 {
		compatible = "arm,cortex-a15-gic", "arm,cortex-a9-gic";
		#interrupt-cells = <3>;
		#address-cells = <0>;
		interrupt-controller;
		reg = <0 0x2c001000 0 0x1000>,
		      <0 0x2c002000 0 0x1000>,
		      <0 0x2c004000 0 0x2000>,
		      <0 0x2c006000 0 0x2000>;
		interrupts = <1 9 0xf04>;
	};

	timer {
		compatible = "arm,armv7-timer";
		interrupts = <1 13 0xf08>,
			     <1 14 0xf08>,
			     <1 11 0xf08>,
			     <1 10 0xf08>;
	};

	pmu {
		compatible = "arm,cortex-a15-pmu", "arm,cortex-a9-pmu";
		interrupts = <0 68 4>,
			     <0 69 4>;
	};

	hypervisor {
		compatible = "xen,xen-4.2", "xen,xen";
		reg = <0 0xb0000000 0 0x20000>;
		interrupts = <1 15 0xf08>;
	};

	motherboard {
		ranges = <0 0 0 0x08000000 0x04000000>,
			 <1 0 0 0x14000000 0x04000000>,
			 <2 0 0 0x18000000 0x04000000>,
			 <3 0 0 0x1c000000 0x04000000>,
			 <4 0 0 0x0c000000 0x04000000>,
			 <5 0 0 0x10000000 0x04000000>;

		interrupt-map-mask = <0 0 63>;
		interrupt-map = <0 0  0 &gic 0  0 4>,
				<0 0  1 &gic 0  1 4>,
				<0 0  2 &gic 0  2 4>,
				<0 0  3 &gic 0  3 4>,
				<0 0  4 &gic 0  4 4>,
				<0 0  5 &gic 0  5 4>,
				<0 0  6 &gic 0  6 4>,
				<0 0  7 &gic 0  7 4>,
				<0 0  8 &gic 0  8 4>,
				<0 0  9 &gic 0  9 4>,
				<0 0 10 &gic 0 10 4>,
				<0 0 11 &gic 0 11 4>,
				<0 0 12 &gic 0 12 4>,
				<0 0 13 &gic 0 13 4>,
				<0 0 14 &gic 0 14 4>,
				<0 0 15 &gic 0 15 4>,
				<0 0 16 &gic 0 16 4>,
				<0 0 17 &gic 0 17 4>,
				<0 0 18 &gic 0 18 4>,
				<0 0 19 &gic 0 19 4>,
				<0 0 20 &gic 0 20 4>,
				<0 0 21 &gic 0 21 4>,
				<0 0 22 &gic 0 22 4>,
				<0 0 23 &gic 0 23 4>,
				<0 0 24 &gic 0 24 4>,
				<0 0 25 &gic 0 25 4>,
				<0 0 26 &gic 0 26 4>,
				<0 0 27 &gic 0 27 4>,
				<0 0 28 &gic 0 28 4>,
				<0 0 29 &gic 0 29 4>,
				<0 0 30 &gic 0 30 4>,
				<0 0 31 &gic 0 31 4>,
				<0 0 32 &gic 0 32 4>,
				<0 0 33 &gic 0 33 4>,
				<0 0 34 &gic 0 34 4>,
				<0 0 35 &gic 0 35 4>,
				<0 0 36 &gic 0 36 4>,
				<0 0 37 &gic 0 37 4>,
				<0 0 38 &gic 0 38 4>,
				<0 0 39 &gic 0 39 4>,
				<0 0 40 &gic 0 40 4>,
				<0 0 41 &gic 0 41 4>,
				<0 0 42 &gic 0 42 4>;
	};
};

/include/ "vexpress-v2m-rs1.dtsi"

[-- Attachment #4: vexpress-virt.dts --]
[-- Type: text/plain, Size: 2665 bytes --]

/*
 * ARM Ltd. Versatile Express
 *
 * ARM Envelope Model v7A (single CPU).
 */

/dts-v1/;

/include/ "skeleton.dtsi"

/ {
	model = "V2P-AEMv7A";
	compatible = "arm,vexpress,v2p-aem,v7a", "arm,vexpress,v2p-aem", "arm,vexpress";
	interrupt-parent = <&gic>;

        chosen {
                bootargs = "earlyprintk debug loglevel=9 console=hvc0 root=/dev/xvda init=/sbin/init";
        };

	cpus {
		#address-cells = <1>;
		#size-cells = <0>;

		cpu@0 {
			device_type = "cpu";
			compatible = "arm,cortex-a15";
			reg = <0>;
		};
	};

	memory {
		device_type = "memory";
		reg = <0x80000000 0x08000000>;
	};

	gic: interrupt-controller@2c001000 {
		compatible = "arm,cortex-a9-gic";
		#interrupt-cells = <3>;
		#address-cells = <0>;
		interrupt-controller;
		reg = <0x2c001000 0x1000>,
		      <0x2c002000 0x100>;
	};

	timer {
		compatible = "arm,armv7-timer";
		interrupts = <1 13 0xf08>,
			     <1 14 0xf08>,
			     <1 11 0xf08>,
			     <1 10 0xf08>;
	};

	hypervisor {
		compatible = "xen,xen-4.2", "xen,xen";
		reg = <0xb0000000 0x20000>;
		interrupts = <1 15 0xf08>;
	};

	motherboard {
		arm,v2m-memory-map = "rs1";
		ranges = <0 0 0x08000000 0x04000000>,
			 <1 0 0x14000000 0x04000000>,
			 <2 0 0x18000000 0x04000000>,
			 <3 0 0x1c000000 0x04000000>,
			 <4 0 0x0c000000 0x04000000>,
			 <5 0 0x10000000 0x04000000>;

		interrupt-map-mask = <0 0 63>;
		interrupt-map = <0 0  0 &gic 0  0 4>,
				<0 0  1 &gic 0  1 4>,
				<0 0  2 &gic 0  2 4>,
				<0 0  3 &gic 0  3 4>,
				<0 0  4 &gic 0  4 4>,
				<0 0  5 &gic 0  5 4>,
				<0 0  6 &gic 0  6 4>,
				<0 0  7 &gic 0  7 4>,
				<0 0  8 &gic 0  8 4>,
				<0 0  9 &gic 0  9 4>,
				<0 0 10 &gic 0 10 4>,
				<0 0 11 &gic 0 11 4>,
				<0 0 12 &gic 0 12 4>,
				<0 0 13 &gic 0 13 4>,
				<0 0 14 &gic 0 14 4>,
				<0 0 15 &gic 0 15 4>,
				<0 0 16 &gic 0 16 4>,
				<0 0 17 &gic 0 17 4>,
				<0 0 18 &gic 0 18 4>,
				<0 0 19 &gic 0 19 4>,
				<0 0 20 &gic 0 20 4>,
				<0 0 21 &gic 0 21 4>,
				<0 0 22 &gic 0 22 4>,
				<0 0 23 &gic 0 23 4>,
				<0 0 24 &gic 0 24 4>,
				<0 0 25 &gic 0 25 4>,
				<0 0 26 &gic 0 26 4>,
				<0 0 27 &gic 0 27 4>,
				<0 0 28 &gic 0 28 4>,
				<0 0 29 &gic 0 29 4>,
				<0 0 30 &gic 0 30 4>,
				<0 0 31 &gic 0 31 4>,
				<0 0 32 &gic 0 32 4>,
				<0 0 33 &gic 0 33 4>,
				<0 0 34 &gic 0 34 4>,
				<0 0 35 &gic 0 35 4>,
				<0 0 36 &gic 0 36 4>,
				<0 0 37 &gic 0 37 4>,
				<0 0 38 &gic 0 38 4>,
				<0 0 39 &gic 0 39 4>,
				<0 0 40 &gic 0 40 4>,
				<0 0 41 &gic 0 41 4>,
				<0 0 42 &gic 0 42 4>;
	};
};

/* /include/ "vexpress-v2m-rs1-rtsm.dtsi" */

[PATCH v5 01/17] arm: initial Xen support

[Stefano Stabellini]

- Basic hypervisor.h and interface.h definitions.
- Skeleton enlighten.c, set xen_start_info to an empty struct.
- Make xen_initial_domain dependent on the SIF_PRIVILIGED_BIT.

The new code only compiles when CONFIG_XEN is set, that is going to be
added to arch/arm/Kconfig in patch #11 "xen/arm: introduce CONFIG_XEN on
ARM".

Changes in v3:

- improve comments.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/Makefile                     |    1 +
 arch/arm/include/asm/hypervisor.h     |    6 +++
 arch/arm/include/asm/xen/hypervisor.h |   19 +++++++++
 arch/arm/include/asm/xen/interface.h  |   69 +++++++++++++++++++++++++++++++++
 arch/arm/xen/Makefile                 |    1 +
 arch/arm/xen/enlighten.c              |   35 +++++++++++++++++
 include/xen/xen.h                     |    2 +-
 7 files changed, 132 insertions(+), 1 deletions(-)
 create mode 100644 arch/arm/include/asm/hypervisor.h
 create mode 100644 arch/arm/include/asm/xen/hypervisor.h
 create mode 100644 arch/arm/include/asm/xen/interface.h
 create mode 100644 arch/arm/xen/Makefile
 create mode 100644 arch/arm/xen/enlighten.c

diff --git a/arch/arm/Makefile b/arch/arm/Makefile
index 30eae87..f42968a 100644
--- a/arch/arm/Makefile
+++ b/arch/arm/Makefile
@@ -251,6 +251,7 @@ endif
 core-$(CONFIG_FPE_NWFPE)	+= arch/arm/nwfpe/
 core-$(CONFIG_FPE_FASTFPE)	+= $(FASTFPE_OBJ)
 core-$(CONFIG_VFP)		+= arch/arm/vfp/
+core-$(CONFIG_XEN)		+= arch/arm/xen/
 
 # If we have a machine-specific directory, then include it in the build.
 core-y				+= arch/arm/kernel/ arch/arm/mm/ arch/arm/common/
diff --git a/arch/arm/include/asm/hypervisor.h b/arch/arm/include/asm/hypervisor.h
new file mode 100644
index 0000000..b90d9e5
--- /dev/null
+++ b/arch/arm/include/asm/hypervisor.h
@@ -0,0 +1,6 @@
+#ifndef _ASM_ARM_HYPERVISOR_H
+#define _ASM_ARM_HYPERVISOR_H
+
+#include <asm/xen/hypervisor.h>
+
+#endif
diff --git a/arch/arm/include/asm/xen/hypervisor.h b/arch/arm/include/asm/xen/hypervisor.h
new file mode 100644
index 0000000..d7ab99a
--- /dev/null
+++ b/arch/arm/include/asm/xen/hypervisor.h
@@ -0,0 +1,19 @@
+#ifndef _ASM_ARM_XEN_HYPERVISOR_H
+#define _ASM_ARM_XEN_HYPERVISOR_H
+
+extern struct shared_info *HYPERVISOR_shared_info;
+extern struct start_info *xen_start_info;
+
+/* Lazy mode for batching updates / context switch */
+enum paravirt_lazy_mode {
+	PARAVIRT_LAZY_NONE,
+	PARAVIRT_LAZY_MMU,
+	PARAVIRT_LAZY_CPU,
+};
+
+static inline enum paravirt_lazy_mode paravirt_get_lazy_mode(void)
+{
+	return PARAVIRT_LAZY_NONE;
+}
+
+#endif /* _ASM_ARM_XEN_HYPERVISOR_H */
diff --git a/arch/arm/include/asm/xen/interface.h b/arch/arm/include/asm/xen/interface.h
new file mode 100644
index 0000000..74c72b5
--- /dev/null
+++ b/arch/arm/include/asm/xen/interface.h
@@ -0,0 +1,69 @@
+/******************************************************************************
+ * Guest OS interface to ARM Xen.
+ *
+ * Stefano Stabellini <stefano.stabellini@eu.citrix.com>, Citrix, 2012
+ */
+
+#ifndef _ASM_ARM_XEN_INTERFACE_H
+#define _ASM_ARM_XEN_INTERFACE_H
+
+#include <linux/types.h>
+
+#define __DEFINE_GUEST_HANDLE(name, type) \
+	typedef type * __guest_handle_ ## name
+
+#define DEFINE_GUEST_HANDLE_STRUCT(name) \
+	__DEFINE_GUEST_HANDLE(name, struct name)
+#define DEFINE_GUEST_HANDLE(name) __DEFINE_GUEST_HANDLE(name, name)
+#define GUEST_HANDLE(name)        __guest_handle_ ## name
+
+#define set_xen_guest_handle(hnd, val)			\
+	do {						\
+		if (sizeof(hnd) == 8)			\
+			*(uint64_t *)&(hnd) = 0;	\
+		(hnd) = val;				\
+	} while (0)
+
+#ifndef __ASSEMBLY__
+/* Explicitly size integers that represent pfns in the interface with
+ * Xen so that we can have one ABI that works for 32 and 64 bit guests. */
+typedef uint64_t xen_pfn_t;
+/* Guest handles for primitive C types. */
+__DEFINE_GUEST_HANDLE(uchar, unsigned char);
+__DEFINE_GUEST_HANDLE(uint,  unsigned int);
+__DEFINE_GUEST_HANDLE(ulong, unsigned long);
+DEFINE_GUEST_HANDLE(char);
+DEFINE_GUEST_HANDLE(int);
+DEFINE_GUEST_HANDLE(long);
+DEFINE_GUEST_HANDLE(void);
+DEFINE_GUEST_HANDLE(uint64_t);
+DEFINE_GUEST_HANDLE(uint32_t);
+DEFINE_GUEST_HANDLE(xen_pfn_t);
+
+/* Maximum number of virtual CPUs in multi-processor guests. */
+#define MAX_VIRT_CPUS 1
+
+struct arch_vcpu_info { };
+struct arch_shared_info { };
+
+/* TODO: Move pvclock definitions some place arch independent */
+struct pvclock_vcpu_time_info {
+	u32   version;
+	u32   pad0;
+	u64   tsc_timestamp;
+	u64   system_time;
+	u32   tsc_to_system_mul;
+	s8    tsc_shift;
+	u8    flags;
+	u8    pad[2];
+} __attribute__((__packed__)); /* 32 bytes */
+
+/* It is OK to have a 12 bytes struct with no padding because it is packed */
+struct pvclock_wall_clock {
+	u32   version;
+	u32   sec;
+	u32   nsec;
+} __attribute__((__packed__));
+#endif
+
+#endif /* _ASM_ARM_XEN_INTERFACE_H */
diff --git a/arch/arm/xen/Makefile b/arch/arm/xen/Makefile
new file mode 100644
index 0000000..0bad594
--- /dev/null
+++ b/arch/arm/xen/Makefile
@@ -0,0 +1 @@
+obj-y		:= enlighten.o
diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
new file mode 100644
index 0000000..c535540
--- /dev/null
+++ b/arch/arm/xen/enlighten.c
@@ -0,0 +1,35 @@
+#include <xen/xen.h>
+#include <xen/interface/xen.h>
+#include <xen/interface/memory.h>
+#include <xen/platform_pci.h>
+#include <asm/xen/hypervisor.h>
+#include <asm/xen/hypercall.h>
+#include <linux/module.h>
+
+struct start_info _xen_start_info;
+struct start_info *xen_start_info = &_xen_start_info;
+EXPORT_SYMBOL_GPL(xen_start_info);
+
+enum xen_domain_type xen_domain_type = XEN_NATIVE;
+EXPORT_SYMBOL_GPL(xen_domain_type);
+
+struct shared_info xen_dummy_shared_info;
+struct shared_info *HYPERVISOR_shared_info = (void *)&xen_dummy_shared_info;
+
+DEFINE_PER_CPU(struct vcpu_info *, xen_vcpu);
+
+/* TODO: to be removed */
+__read_mostly int xen_have_vector_callback;
+EXPORT_SYMBOL_GPL(xen_have_vector_callback);
+
+int xen_platform_pci_unplug = XEN_UNPLUG_ALL;
+EXPORT_SYMBOL_GPL(xen_platform_pci_unplug);
+
+int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
+			       unsigned long addr,
+			       unsigned long mfn, int nr,
+			       pgprot_t prot, unsigned domid)
+{
+	return -ENOSYS;
+}
+EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range);
diff --git a/include/xen/xen.h b/include/xen/xen.h
index a164024..2c0d3a5 100644
--- a/include/xen/xen.h
+++ b/include/xen/xen.h
@@ -23,7 +23,7 @@ extern enum xen_domain_type xen_domain_type;
 #include <xen/interface/xen.h>
 #include <asm/xen/hypervisor.h>
 
-#define xen_initial_domain()	(xen_pv_domain() && \
+#define xen_initial_domain()	(xen_domain() && \
 				 xen_start_info->flags & SIF_INITDOMAIN)
 #else  /* !CONFIG_XEN_DOM0 */
 #define xen_initial_domain()	(0)
-- 
1.7.2.5

[PATCH v5 02/17] xen/arm: hypercalls

[Stefano Stabellini]

Use r12 to pass the hypercall number to the hypervisor.

We need a register to pass the hypercall number because we might not
know it at compile time and HVC only takes an immediate argument.

Among the available registers r12 seems to be the best choice because it
is defined as "intra-procedure call scratch register".

Use the ISS to pass an hypervisor specific tag.


Changes in v2:
- define an HYPERCALL macro for 5 arguments hypercall wrappers, even if
at the moment is unused;
- use ldm instead of pop;
- fix up comments.


Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/include/asm/xen/hypercall.h |   50 ++++++++++++++++
 arch/arm/xen/Makefile                |    2 +-
 arch/arm/xen/hypercall.S             |  106 ++++++++++++++++++++++++++++++++++
 3 files changed, 157 insertions(+), 1 deletions(-)
 create mode 100644 arch/arm/include/asm/xen/hypercall.h
 create mode 100644 arch/arm/xen/hypercall.S

diff --git a/arch/arm/include/asm/xen/hypercall.h b/arch/arm/include/asm/xen/hypercall.h
new file mode 100644
index 0000000..4ac0624
--- /dev/null
+++ b/arch/arm/include/asm/xen/hypercall.h
@@ -0,0 +1,50 @@
+/******************************************************************************
+ * hypercall.h
+ *
+ * Linux-specific hypervisor handling.
+ *
+ * Stefano Stabellini <stefano.stabellini@eu.citrix.com>, Citrix, 2012
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation; or, when distributed
+ * separately from the Linux kernel or incorporated into other
+ * software packages, subject to the following license:
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this source file (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use, copy, modify,
+ * merge, publish, distribute, sublicense, and/or sell copies of the Software,
+ * and to permit persons to whom the Software is furnished to do so, subject to
+ * the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef _ASM_ARM_XEN_HYPERCALL_H
+#define _ASM_ARM_XEN_HYPERCALL_H
+
+#include <xen/interface/xen.h>
+
+long privcmd_call(unsigned call, unsigned long a1,
+		unsigned long a2, unsigned long a3,
+		unsigned long a4, unsigned long a5);
+int HYPERVISOR_xen_version(int cmd, void *arg);
+int HYPERVISOR_console_io(int cmd, int count, char *str);
+int HYPERVISOR_grant_table_op(unsigned int cmd, void *uop, unsigned int count);
+int HYPERVISOR_sched_op(int cmd, void *arg);
+int HYPERVISOR_event_channel_op(int cmd, void *arg);
+unsigned long HYPERVISOR_hvm_op(int op, void *arg);
+int HYPERVISOR_memory_op(unsigned int cmd, void *arg);
+int HYPERVISOR_physdev_op(int cmd, void *arg);
+
+#endif /* _ASM_ARM_XEN_HYPERCALL_H */
diff --git a/arch/arm/xen/Makefile b/arch/arm/xen/Makefile
index 0bad594..b9d6acc 100644
--- a/arch/arm/xen/Makefile
+++ b/arch/arm/xen/Makefile
@@ -1 +1 @@
-obj-y		:= enlighten.o
+obj-y		:= enlighten.o hypercall.o
diff --git a/arch/arm/xen/hypercall.S b/arch/arm/xen/hypercall.S
new file mode 100644
index 0000000..074f5ed
--- /dev/null
+++ b/arch/arm/xen/hypercall.S
@@ -0,0 +1,106 @@
+/******************************************************************************
+ * hypercall.S
+ *
+ * Xen hypercall wrappers
+ *
+ * Stefano Stabellini <stefano.stabellini@eu.citrix.com>, Citrix, 2012
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation; or, when distributed
+ * separately from the Linux kernel or incorporated into other
+ * software packages, subject to the following license:
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this source file (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use, copy, modify,
+ * merge, publish, distribute, sublicense, and/or sell copies of the Software,
+ * and to permit persons to whom the Software is furnished to do so, subject to
+ * the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+/*
+ * The Xen hypercall calling convention is very similar to the ARM
+ * procedure calling convention: the first paramter is passed in r0, the
+ * second in r1, the third in r2 and the fourth in r3. Considering that
+ * Xen hypercalls have 5 arguments at most, the fifth paramter is passed
+ * in r4, differently from the procedure calling convention of using the
+ * stack for that case.
+ *
+ * The hypercall number is passed in r12.
+ *
+ * The return value is in r0.
+ *
+ * The hvc ISS is required to be 0xEA1, that is the Xen specific ARM
+ * hypercall tag.
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+#include <xen/interface/xen.h>
+
+
+/* HVC 0xEA1 */
+#ifdef CONFIG_THUMB2_KERNEL
+#define xen_hvc .word 0xf7e08ea1
+#else
+#define xen_hvc .word 0xe140ea71
+#endif
+
+#define HYPERCALL_SIMPLE(hypercall)		\
+ENTRY(HYPERVISOR_##hypercall)			\
+	mov r12, #__HYPERVISOR_##hypercall;	\
+	xen_hvc;							\
+	mov pc, lr;							\
+ENDPROC(HYPERVISOR_##hypercall)
+
+#define HYPERCALL0 HYPERCALL_SIMPLE
+#define HYPERCALL1 HYPERCALL_SIMPLE
+#define HYPERCALL2 HYPERCALL_SIMPLE
+#define HYPERCALL3 HYPERCALL_SIMPLE
+#define HYPERCALL4 HYPERCALL_SIMPLE
+
+#define HYPERCALL5(hypercall)			\
+ENTRY(HYPERVISOR_##hypercall)			\
+	stmdb sp!, {r4}						\
+	ldr r4, [sp, #4]					\
+	mov r12, #__HYPERVISOR_##hypercall;	\
+	xen_hvc								\
+	ldm sp!, {r4}						\
+	mov pc, lr							\
+ENDPROC(HYPERVISOR_##hypercall)
+
+                .text
+
+HYPERCALL2(xen_version);
+HYPERCALL3(console_io);
+HYPERCALL3(grant_table_op);
+HYPERCALL2(sched_op);
+HYPERCALL2(event_channel_op);
+HYPERCALL2(hvm_op);
+HYPERCALL2(memory_op);
+HYPERCALL2(physdev_op);
+
+ENTRY(privcmd_call)
+	stmdb sp!, {r4}
+	mov r12, r0
+	mov r0, r1
+	mov r1, r2
+	mov r2, r3
+	ldr r3, [sp, #8]
+	ldr r4, [sp, #4]
+	xen_hvc
+	ldm sp!, {r4}
+	mov pc, lr
+ENDPROC(privcmd_call);
-- 
1.7.2.5

[PATCH v5 03/17] xen/arm: page.h definitions

[Stefano Stabellini]

ARM Xen guests always use paging in hardware, like PV on HVM guests in
the X86 world.

Changes in v3:

- improve comments.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/include/asm/xen/page.h |   82 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 82 insertions(+), 0 deletions(-)
 create mode 100644 arch/arm/include/asm/xen/page.h

diff --git a/arch/arm/include/asm/xen/page.h b/arch/arm/include/asm/xen/page.h
new file mode 100644
index 0000000..1742023
--- /dev/null
+++ b/arch/arm/include/asm/xen/page.h
@@ -0,0 +1,82 @@
+#ifndef _ASM_ARM_XEN_PAGE_H
+#define _ASM_ARM_XEN_PAGE_H
+
+#include <asm/page.h>
+#include <asm/pgtable.h>
+
+#include <linux/pfn.h>
+#include <linux/types.h>
+
+#include <xen/interface/grant_table.h>
+
+#define pfn_to_mfn(pfn)			(pfn)
+#define phys_to_machine_mapping_valid	(1)
+#define mfn_to_pfn(mfn)			(mfn)
+#define mfn_to_virt(m)			(__va(mfn_to_pfn(m) << PAGE_SHIFT))
+
+#define pte_mfn	    pte_pfn
+#define mfn_pte	    pfn_pte
+
+/* Xen machine address */
+typedef struct xmaddr {
+	phys_addr_t maddr;
+} xmaddr_t;
+
+/* Xen pseudo-physical address */
+typedef struct xpaddr {
+	phys_addr_t paddr;
+} xpaddr_t;
+
+#define XMADDR(x)	((xmaddr_t) { .maddr = (x) })
+#define XPADDR(x)	((xpaddr_t) { .paddr = (x) })
+
+static inline xmaddr_t phys_to_machine(xpaddr_t phys)
+{
+	unsigned offset = phys.paddr & ~PAGE_MASK;
+	return XMADDR(PFN_PHYS(pfn_to_mfn(PFN_DOWN(phys.paddr))) | offset);
+}
+
+static inline xpaddr_t machine_to_phys(xmaddr_t machine)
+{
+	unsigned offset = machine.maddr & ~PAGE_MASK;
+	return XPADDR(PFN_PHYS(mfn_to_pfn(PFN_DOWN(machine.maddr))) | offset);
+}
+/* VIRT <-> MACHINE conversion */
+#define virt_to_machine(v)	(phys_to_machine(XPADDR(__pa(v))))
+#define virt_to_pfn(v)          (PFN_DOWN(__pa(v)))
+#define virt_to_mfn(v)		(pfn_to_mfn(virt_to_pfn(v)))
+#define mfn_to_virt(m)		(__va(mfn_to_pfn(m) << PAGE_SHIFT))
+
+static inline xmaddr_t arbitrary_virt_to_machine(void *vaddr)
+{
+	/* TODO: assuming it is mapped in the kernel 1:1 */
+	return virt_to_machine(vaddr);
+}
+
+/* TODO: this shouldn't be here but it is because the frontend drivers
+ * are using it (its rolled in headers) even though we won't hit the code path.
+ * So for right now just punt with this.
+ */
+static inline pte_t *lookup_address(unsigned long address, unsigned int *level)
+{
+	BUG();
+	return NULL;
+}
+
+static inline int m2p_add_override(unsigned long mfn, struct page *page,
+		struct gnttab_map_grant_ref *kmap_op)
+{
+	return 0;
+}
+
+static inline int m2p_remove_override(struct page *page, bool clear_pte)
+{
+	return 0;
+}
+
+static inline bool set_phys_to_machine(unsigned long pfn, unsigned long mfn)
+{
+	BUG();
+	return false;
+}
+#endif /* _ASM_ARM_XEN_PAGE_H */
-- 
1.7.2.5

[PATCH v5 04/17] xen/arm: sync_bitops

[Stefano Stabellini]

sync_bitops functions are equivalent to the SMP implementation of the
original functions, independently from CONFIG_SMP being defined.

We need them because _set_bit etc are not SMP safe if !CONFIG_SMP. But
under Xen you might be communicating with a completely external entity
who might be on another CPU (e.g. two uniprocessor guests communicating
via event channels and grant tables). So we need a variant of the bit
ops which are SMP safe even on a UP kernel.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/include/asm/sync_bitops.h |   27 +++++++++++++++++++++++++++
 1 files changed, 27 insertions(+), 0 deletions(-)
 create mode 100644 arch/arm/include/asm/sync_bitops.h

diff --git a/arch/arm/include/asm/sync_bitops.h b/arch/arm/include/asm/sync_bitops.h
new file mode 100644
index 0000000..63479ee
--- /dev/null
+++ b/arch/arm/include/asm/sync_bitops.h
@@ -0,0 +1,27 @@
+#ifndef __ASM_SYNC_BITOPS_H__
+#define __ASM_SYNC_BITOPS_H__
+
+#include <asm/bitops.h>
+#include <asm/system.h>
+
+/* sync_bitops functions are equivalent to the SMP implementation of the
+ * original functions, independently from CONFIG_SMP being defined.
+ *
+ * We need them because _set_bit etc are not SMP safe if !CONFIG_SMP. But
+ * under Xen you might be communicating with a completely external entity
+ * who might be on another CPU (e.g. two uniprocessor guests communicating
+ * via event channels and grant tables). So we need a variant of the bit
+ * ops which are SMP safe even on a UP kernel.
+ */
+
+#define sync_set_bit(nr, p)		_set_bit(nr, p)
+#define sync_clear_bit(nr, p)		_clear_bit(nr, p)
+#define sync_change_bit(nr, p)		_change_bit(nr, p)
+#define sync_test_and_set_bit(nr, p)	_test_and_set_bit(nr, p)
+#define sync_test_and_clear_bit(nr, p)	_test_and_clear_bit(nr, p)
+#define sync_test_and_change_bit(nr, p)	_test_and_change_bit(nr, p)
+#define sync_test_bit(nr, addr)		test_bit(nr, addr)
+#define sync_cmpxchg			cmpxchg
+
+
+#endif
-- 
1.7.2.5

[PATCH v5 05/17] xen/arm: empty implementation of grant_table arch specific functions

[Stefano Stabellini]

Changes in v2:

- return -ENOSYS rather than -1.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/xen/Makefile      |    2 +-
 arch/arm/xen/grant-table.c |   53 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 1 deletions(-)
 create mode 100644 arch/arm/xen/grant-table.c

diff --git a/arch/arm/xen/Makefile b/arch/arm/xen/Makefile
index b9d6acc..4384103 100644
--- a/arch/arm/xen/Makefile
+++ b/arch/arm/xen/Makefile
@@ -1 +1 @@
-obj-y		:= enlighten.o hypercall.o
+obj-y		:= enlighten.o hypercall.o grant-table.o
diff --git a/arch/arm/xen/grant-table.c b/arch/arm/xen/grant-table.c
new file mode 100644
index 0000000..dbd1330
--- /dev/null
+++ b/arch/arm/xen/grant-table.c
@@ -0,0 +1,53 @@
+/******************************************************************************
+ * grant_table.c
+ * ARM specific part
+ *
+ * Granting foreign access to our memory reservation.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation; or, when distributed
+ * separately from the Linux kernel or incorporated into other
+ * software packages, subject to the following license:
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this source file (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use, copy, modify,
+ * merge, publish, distribute, sublicense, and/or sell copies of the Software,
+ * and to permit persons to whom the Software is furnished to do so, subject to
+ * the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#include <xen/interface/xen.h>
+#include <xen/page.h>
+#include <xen/grant_table.h>
+
+int arch_gnttab_map_shared(unsigned long *frames, unsigned long nr_gframes,
+			   unsigned long max_nr_gframes,
+			   void **__shared)
+{
+	return -ENOSYS;
+}
+
+void arch_gnttab_unmap(void *shared, unsigned long nr_gframes)
+{
+	return;
+}
+
+int arch_gnttab_map_status(uint64_t *frames, unsigned long nr_gframes,
+			   unsigned long max_nr_gframes,
+			   grant_status_t **__shared)
+{
+	return -ENOSYS;
+}
-- 
1.7.2.5

[PATCH v5 06/17] docs: Xen ARM DT bindings

[Stefano Stabellini]

Add a doc to describe the Xen ARM device tree bindings


Changes in v5:

- add a comment about the size of the grant table memory region;
- add a comment about the required presence of a GIC node;
- specify that the described properties are part of a top-level
"hypervisor" node;
- specify #address-cells and #size-cells for the example.


Changes in v4:

- "xen,xen" should be last as it is less specific;
- update reg property using 2 address-cells and 2 size-cells.


Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
CC: devicetree-discuss@lists.ozlabs.org
CC: David Vrabel <david.vrabel@citrix.com>
CC: Rob Herring <robherring2@gmail.com>
CC: Dave Martin <dave.martin@linaro.org>
---
 Documentation/devicetree/bindings/arm/xen.txt |   25 +++++++++++++++++++++++++
 1 files changed, 25 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/arm/xen.txt

diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt
new file mode 100644
index 0000000..0f7b9c2
--- /dev/null
+++ b/Documentation/devicetree/bindings/arm/xen.txt
@@ -0,0 +1,25 @@
+* Xen hypervisor device tree bindings
+
+Xen ARM virtual platforms shall have a top-level "hypervisor" node with
+the following properties:
+
+- compatible:
+	compatible = "xen,xen-<version>", "xen,xen";
+  where <version> is the version of the Xen ABI of the platform.
+
+- reg: specifies the base physical address and size of a region in
+  memory where the grant table should be mapped to, using an
+  HYPERVISOR_memory_op hypercall. The memory region is large enough to map
+  the whole grant table (it is larger or equal to gnttab_max_grant_frames()).
+
+- interrupts: the interrupt used by Xen to inject event notifications.
+  A GIC node is also required.
+
+
+Example (assuming #address-cells = <2> and #size-cells = <2>):
+
+hypervisor {
+	compatible = "xen,xen-4.3", "xen,xen";
+	reg = <0 0xb0000000 0 0x20000>;
+	interrupts = <1 15 0xf08>;
+};
-- 
1.7.2.5

[PATCH v5 07/17] xen/arm: Xen detection and shared_info page mapping

[Stefano Stabellini]

Check for a node in the device tree compatible with "xen,xen", if it is
present set xen_domain_type to XEN_HVM_DOMAIN and continue
initialization.

Map the real shared info page using XENMEM_add_to_physmap with
XENMAPSPACE_shared_info.

Changes in v4:

- simpler parsing of Xen version in the compatible DT node.

Changes in v3:

- use the "xen,xen" notation rather than "arm,xen";
- add an additional check on the presence of the Xen version.

Changes in v2:

- replace pr_info with pr_debug.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/arm/xen/enlighten.c |   61 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 61 insertions(+), 0 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index c535540..6a0217d 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -5,6 +5,9 @@
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
 #include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_irq.h>
+#include <linux/of_address.h>
 
 struct start_info _xen_start_info;
 struct start_info *xen_start_info = &_xen_start_info;
@@ -33,3 +36,61 @@ int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
 	return -ENOSYS;
 }
 EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range);
+
+/*
+ * see Documentation/devicetree/bindings/arm/xen.txt for the
+ * documentation of the Xen Device Tree format.
+ */
+static int __init xen_guest_init(void)
+{
+	struct xen_add_to_physmap xatp;
+	static struct shared_info *shared_info_page = 0;
+	struct device_node *node;
+	int len;
+	const char *s = NULL;
+	const char *version = NULL;
+	const char *xen_prefix = "xen,xen-";
+
+	node = of_find_compatible_node(NULL, NULL, "xen,xen");
+	if (!node) {
+		pr_debug("No Xen support\n");
+		return 0;
+	}
+	s = of_get_property(node, "compatible", &len);
+	if (strlen(xen_prefix) + 3  < len &&
+			!strncmp(xen_prefix, s, strlen(xen_prefix)))
+		version = s + strlen(xen_prefix);
+	if (version == NULL) {
+		pr_debug("Xen version not found\n");
+		return 0;
+	}
+	xen_domain_type = XEN_HVM_DOMAIN;
+
+	if (!shared_info_page)
+		shared_info_page = (struct shared_info *)
+			get_zeroed_page(GFP_KERNEL);
+	if (!shared_info_page) {
+		pr_err("not enough memory\n");
+		return -ENOMEM;
+	}
+	xatp.domid = DOMID_SELF;
+	xatp.idx = 0;
+	xatp.space = XENMAPSPACE_shared_info;
+	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
+		BUG();
+
+	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+
+	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
+	 * page, we use it in the event channel upcall and in some pvclock
+	 * related functions. We don't need the vcpu_info placement
+	 * optimizations because we don't use any pv_mmu or pv_irq op on
+	 * HVM.
+	 * The shared info contains exactly 1 CPU (the boot CPU). The guest
+	 * is required to use VCPUOP_register_vcpu_info to place vcpu info
+	 * for secondary CPUs as they are brought up. */
+	per_cpu(xen_vcpu, 0) = &HYPERVISOR_shared_info->vcpu_info[0];
+	return 0;
+}
+core_initcall(xen_guest_init);
-- 
1.7.2.5

[PATCH v5 08/17] xen/arm: Introduce xen_ulong_t for unsigned long

[Stefano Stabellini]

All the original Xen headers have xen_ulong_t as unsigned long type, however
when they have been imported in Linux, xen_ulong_t has been replaced with
unsigned long. That might work for x86 and ia64 but it does not for arm.
Bring back xen_ulong_t and let each architecture define xen_ulong_t as they
see fit.

Also explicitly size pointers (__DEFINE_GUEST_HANDLE) to 64 bit.


Changes in v3:

- remove the incorrect changes to multicall_entry;
- remove the change to apic_physbase.


Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/include/asm/xen/interface.h  |    8 ++++++--
 arch/ia64/include/asm/xen/interface.h |    1 +
 arch/x86/include/asm/xen/interface.h  |    1 +
 include/xen/interface/memory.h        |   12 ++++++------
 include/xen/interface/physdev.h       |    2 +-
 include/xen/interface/version.h       |    2 +-
 6 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/arch/arm/include/asm/xen/interface.h b/arch/arm/include/asm/xen/interface.h
index 74c72b5..ae05e56 100644
--- a/arch/arm/include/asm/xen/interface.h
+++ b/arch/arm/include/asm/xen/interface.h
@@ -9,8 +9,11 @@
 
 #include <linux/types.h>
 
+#define uint64_aligned_t uint64_t __attribute__((aligned(8)))
+
 #define __DEFINE_GUEST_HANDLE(name, type) \
-	typedef type * __guest_handle_ ## name
+	typedef struct { union { type *p; uint64_aligned_t q; }; }  \
+        __guest_handle_ ## name
 
 #define DEFINE_GUEST_HANDLE_STRUCT(name) \
 	__DEFINE_GUEST_HANDLE(name, struct name)
@@ -21,13 +24,14 @@
 	do {						\
 		if (sizeof(hnd) == 8)			\
 			*(uint64_t *)&(hnd) = 0;	\
-		(hnd) = val;				\
+		(hnd).p = val;				\
 	} while (0)
 
 #ifndef __ASSEMBLY__
 /* Explicitly size integers that represent pfns in the interface with
  * Xen so that we can have one ABI that works for 32 and 64 bit guests. */
 typedef uint64_t xen_pfn_t;
+typedef uint64_t xen_ulong_t;
 /* Guest handles for primitive C types. */
 __DEFINE_GUEST_HANDLE(uchar, unsigned char);
 __DEFINE_GUEST_HANDLE(uint,  unsigned int);
diff --git a/arch/ia64/include/asm/xen/interface.h b/arch/ia64/include/asm/xen/interface.h
index 3d52a5b..e88c5de 100644
--- a/arch/ia64/include/asm/xen/interface.h
+++ b/arch/ia64/include/asm/xen/interface.h
@@ -71,6 +71,7 @@
  * with Xen so that we could have one ABI that works for 32 and 64 bit
  * guests. */
 typedef unsigned long xen_pfn_t;
+typedef unsigned long xen_ulong_t;
 /* Guest handles for primitive C types. */
 __DEFINE_GUEST_HANDLE(uchar, unsigned char);
 __DEFINE_GUEST_HANDLE(uint, unsigned int);
diff --git a/arch/x86/include/asm/xen/interface.h b/arch/x86/include/asm/xen/interface.h
index 555f94d..28fc621 100644
--- a/arch/x86/include/asm/xen/interface.h
+++ b/arch/x86/include/asm/xen/interface.h
@@ -51,6 +51,7 @@
  * with Xen so that on ARM we can have one ABI that works for 32 and 64
  * bit guests. */
 typedef unsigned long xen_pfn_t;
+typedef unsigned long xen_ulong_t;
 /* Guest handles for primitive C types. */
 __DEFINE_GUEST_HANDLE(uchar, unsigned char);
 __DEFINE_GUEST_HANDLE(uint,  unsigned int);
diff --git a/include/xen/interface/memory.h b/include/xen/interface/memory.h
index d8e33a9..b66d04c 100644
--- a/include/xen/interface/memory.h
+++ b/include/xen/interface/memory.h
@@ -34,7 +34,7 @@ struct xen_memory_reservation {
     GUEST_HANDLE(xen_pfn_t) extent_start;
 
     /* Number of extents, and size/alignment of each (2^extent_order pages). */
-    unsigned long  nr_extents;
+    xen_ulong_t  nr_extents;
     unsigned int   extent_order;
 
     /*
@@ -92,7 +92,7 @@ struct xen_memory_exchange {
      *     command will be non-zero.
      *  5. THIS FIELD MUST BE INITIALISED TO ZERO BY THE CALLER!
      */
-    unsigned long nr_exchanged;
+    xen_ulong_t nr_exchanged;
 };
 
 DEFINE_GUEST_HANDLE_STRUCT(xen_memory_exchange);
@@ -148,8 +148,8 @@ DEFINE_GUEST_HANDLE_STRUCT(xen_machphys_mfn_list);
  */
 #define XENMEM_machphys_mapping     12
 struct xen_machphys_mapping {
-    unsigned long v_start, v_end; /* Start and end virtual addresses.   */
-    unsigned long max_mfn;        /* Maximum MFN that can be looked up. */
+    xen_ulong_t v_start, v_end; /* Start and end virtual addresses.   */
+    xen_ulong_t max_mfn;        /* Maximum MFN that can be looked up. */
 };
 DEFINE_GUEST_HANDLE_STRUCT(xen_machphys_mapping_t);
 
@@ -172,7 +172,7 @@ struct xen_add_to_physmap {
     unsigned int space;
 
     /* Index into source mapping space. */
-    unsigned long idx;
+    xen_ulong_t idx;
 
     /* GPFN where the source mapping page should appear. */
     xen_pfn_t gpfn;
@@ -189,7 +189,7 @@ struct xen_translate_gpfn_list {
     domid_t domid;
 
     /* Length of list. */
-    unsigned long nr_gpfns;
+    xen_ulong_t nr_gpfns;
 
     /* List of GPFNs to translate. */
     GUEST_HANDLE(ulong) gpfn_list;
diff --git a/include/xen/interface/physdev.h b/include/xen/interface/physdev.h
index 9ce788d..f616514 100644
--- a/include/xen/interface/physdev.h
+++ b/include/xen/interface/physdev.h
@@ -56,7 +56,7 @@ struct physdev_eoi {
 #define PHYSDEVOP_pirq_eoi_gmfn_v2       28
 struct physdev_pirq_eoi_gmfn {
     /* IN */
-    unsigned long gmfn;
+    xen_ulong_t gmfn;
 };
 
 /*
diff --git a/include/xen/interface/version.h b/include/xen/interface/version.h
index dd58cf5..3030c81 100644
--- a/include/xen/interface/version.h
+++ b/include/xen/interface/version.h
@@ -45,7 +45,7 @@ struct xen_changeset_info {
 
 #define XENVER_platform_parameters 5
 struct xen_platform_parameters {
-    unsigned long virt_start;
+    xen_ulong_t virt_start;
 };
 
 #define XENVER_get_features 6
-- 
1.7.2.5

[PATCH v5 09/17] xen: do not compile manage, balloon, pci, acpi, pcpu and cpu_hotplug on ARM

[Stefano Stabellini]

Changes in v4:
- compile pcpu only on x86;
- use "+=" instead of ":=" for dom0- targets.

Changes in v2:

- make pci.o depend on CONFIG_PCI and acpi.o depend on CONFIG_ACPI.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/Makefile |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile
index d80bea5..cd28aae 100644
--- a/drivers/xen/Makefile
+++ b/drivers/xen/Makefile
@@ -1,11 +1,18 @@
-obj-y	+= grant-table.o features.o events.o manage.o balloon.o
+ifneq ($(CONFIG_ARM),y)
+obj-y	+= manage.o balloon.o
+obj-$(CONFIG_HOTPLUG_CPU)		+= cpu_hotplug.o
+endif
+obj-y	+= grant-table.o features.o events.o
 obj-y	+= xenbus/
 
 nostackp := $(call cc-option, -fno-stack-protector)
 CFLAGS_features.o			:= $(nostackp)
 
+obj-$(CONFIG_XEN_DOM0)			+= $(dom0-y)
+dom0-$(CONFIG_PCI) += pci.o
+dom0-$(CONFIG_ACPI) += acpi.o
+dom0-$(CONFIG_X86) += pcpu.o
 obj-$(CONFIG_BLOCK)			+= biomerge.o
-obj-$(CONFIG_HOTPLUG_CPU)		+= cpu_hotplug.o
 obj-$(CONFIG_XEN_XENCOMM)		+= xencomm.o
 obj-$(CONFIG_XEN_BALLOON)		+= xen-balloon.o
 obj-$(CONFIG_XEN_SELFBALLOONING)	+= xen-selfballoon.o
@@ -17,8 +24,6 @@ obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
 obj-$(CONFIG_XEN_PVHVM)			+= platform-pci.o
 obj-$(CONFIG_XEN_TMEM)			+= tmem.o
 obj-$(CONFIG_SWIOTLB_XEN)		+= swiotlb-xen.o
-obj-$(CONFIG_XEN_DOM0)			+= pcpu.o
-obj-$(CONFIG_XEN_DOM0)			+= pci.o acpi.o
 obj-$(CONFIG_XEN_MCE_LOG)		+= mcelog.o
 obj-$(CONFIG_XEN_PCIDEV_BACKEND)	+= xen-pciback/
 obj-$(CONFIG_XEN_PRIVCMD)		+= xen-privcmd.o
-- 
1.7.2.5

[PATCH v5 10/17] xen/arm: introduce CONFIG_XEN on ARM

[Stefano Stabellini]

Changes in v5:

- make XEN_DOM0 depend on XEN;
- avoid "select XEN_DOM0" in XEN.


Changes in v2:

- mark Xen guest support on ARM as EXPERIMENTAL.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
CC: Sergei Shtylyov <sshtylyov@mvista.com>
---
 arch/arm/Kconfig |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 2f88d8d..3361466 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1897,6 +1897,16 @@ config DEPRECATED_PARAM_STRUCT
 	  This was deprecated in 2001 and announced to live on for 5 years.
 	  Some old boot loaders still use this way.
 
+config XEN_DOM0
+	def_bool y
+	depends on XEN
+
+config XEN
+	bool "Xen guest support on ARM (EXPERIMENTAL)"
+	depends on EXPERIMENTAL && ARM && OF
+	help
+	  Say Y if you want to run Linux in a Virtual Machine on Xen on ARM.
+
 endmenu
 
 menu "Boot options"
-- 
1.7.2.5

[PATCH v5 11/17] xen/arm: get privilege status

[Stefano Stabellini]

Use Xen features to figure out if we are privileged.

XENFEAT_dom0 was introduced by 23735 in xen-unstable.hg.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/xen/enlighten.c         |    7 +++++++
 include/xen/interface/features.h |    3 +++
 2 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index 6a0217d..c2a47a7 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -1,6 +1,7 @@
 #include <xen/xen.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/memory.h>
+#include <xen/features.h>
 #include <xen/platform_pci.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
@@ -66,6 +67,12 @@ static int __init xen_guest_init(void)
 	}
 	xen_domain_type = XEN_HVM_DOMAIN;
 
+	xen_setup_features();
+	if (xen_feature(XENFEAT_dom0))
+		xen_start_info->flags |= SIF_INITDOMAIN|SIF_PRIVILEGED;
+	else
+		xen_start_info->flags &= ~(SIF_INITDOMAIN|SIF_PRIVILEGED);
+
 	if (!shared_info_page)
 		shared_info_page = (struct shared_info *)
 			get_zeroed_page(GFP_KERNEL);
diff --git a/include/xen/interface/features.h b/include/xen/interface/features.h
index b6ca39a..131a6cc 100644
--- a/include/xen/interface/features.h
+++ b/include/xen/interface/features.h
@@ -50,6 +50,9 @@
 /* x86: pirq can be used by HVM guests */
 #define XENFEAT_hvm_pirqs           10
 
+/* operation as Dom0 is supported */
+#define XENFEAT_dom0                      11
+
 #define XENFEAT_NR_SUBMAPS 1
 
 #endif /* __XEN_PUBLIC_FEATURES_H__ */
-- 
1.7.2.5

[PATCH v5 12/17] xen/arm: initialize grant_table on ARM

[Stefano Stabellini]

Initialize the grant table mapping at the address specified at index 0
in the DT under the /xen node.
After the grant table is initialized, call xenbus_probe (if not dom0).

Changes in v2:

- introduce GRANT_TABLE_PHYSADDR;
- remove unneeded initialization of boot_max_nr_grant_frames.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/arm/xen/enlighten.c |   14 ++++++++++++++
 1 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index c2a47a7..036a4d8 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -1,8 +1,12 @@
 #include <xen/xen.h>
+#include <xen/grant_table.h>
+#include <xen/hvm.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/memory.h>
+#include <xen/interface/hvm/params.h>
 #include <xen/features.h>
 #include <xen/platform_pci.h>
+#include <xen/xenbus.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
 #include <linux/module.h>
@@ -42,6 +46,7 @@ EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_range);
  * see Documentation/devicetree/bindings/arm/xen.txt for the
  * documentation of the Xen Device Tree format.
  */
+#define GRANT_TABLE_PHYSADDR 0
 static int __init xen_guest_init(void)
 {
 	struct xen_add_to_physmap xatp;
@@ -51,6 +56,7 @@ static int __init xen_guest_init(void)
 	const char *s = NULL;
 	const char *version = NULL;
 	const char *xen_prefix = "xen,xen-";
+	struct resource res;
 
 	node = of_find_compatible_node(NULL, NULL, "xen,xen");
 	if (!node) {
@@ -65,6 +71,9 @@ static int __init xen_guest_init(void)
 		pr_debug("Xen version not found\n");
 		return 0;
 	}
+	if (of_address_to_resource(node, GRANT_TABLE_PHYSADDR, &res))
+		return 0;
+	xen_hvm_resume_frames = res.start >> PAGE_SHIFT;
 	xen_domain_type = XEN_HVM_DOMAIN;
 
 	xen_setup_features();
@@ -98,6 +107,11 @@ static int __init xen_guest_init(void)
 	 * is required to use VCPUOP_register_vcpu_info to place vcpu info
 	 * for secondary CPUs as they are brought up. */
 	per_cpu(xen_vcpu, 0) = &HYPERVISOR_shared_info->vcpu_info[0];
+
+	gnttab_init();
+	if (!xen_initial_domain())
+		xenbus_probe(NULL);
+
 	return 0;
 }
 core_initcall(xen_guest_init);
-- 
1.7.2.5

[PATCH v5 13/17] xen/arm: receive Xen events on ARM

[Stefano Stabellini]

Compile events.c on ARM.
Parse, map and enable the IRQ to get event notifications from the device
tree (node "/xen").

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/include/asm/xen/events.h |   18 ++++++++++++++++++
 arch/arm/xen/enlighten.c          |   33 +++++++++++++++++++++++++++++++++
 arch/x86/xen/enlighten.c          |    1 +
 arch/x86/xen/irq.c                |    1 +
 arch/x86/xen/xen-ops.h            |    1 -
 drivers/xen/events.c              |   17 ++++++++++++++---
 include/xen/events.h              |    2 ++
 7 files changed, 69 insertions(+), 4 deletions(-)
 create mode 100644 arch/arm/include/asm/xen/events.h

diff --git a/arch/arm/include/asm/xen/events.h b/arch/arm/include/asm/xen/events.h
new file mode 100644
index 0000000..94b4e90
--- /dev/null
+++ b/arch/arm/include/asm/xen/events.h
@@ -0,0 +1,18 @@
+#ifndef _ASM_ARM_XEN_EVENTS_H
+#define _ASM_ARM_XEN_EVENTS_H
+
+#include <asm/ptrace.h>
+
+enum ipi_vector {
+	XEN_PLACEHOLDER_VECTOR,
+
+	/* Xen IPIs go here */
+	XEN_NR_IPIS,
+};
+
+static inline int xen_irqs_disabled(struct pt_regs *regs)
+{
+	return raw_irqs_disabled_flags(regs->ARM_cpsr);
+}
+
+#endif /* _ASM_ARM_XEN_EVENTS_H */
diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index 036a4d8..bad67ad 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -1,4 +1,5 @@
 #include <xen/xen.h>
+#include <xen/events.h>
 #include <xen/grant_table.h>
 #include <xen/hvm.h>
 #include <xen/interface/xen.h>
@@ -9,6 +10,8 @@
 #include <xen/xenbus.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
+#include <linux/interrupt.h>
+#include <linux/irqreturn.h>
 #include <linux/module.h>
 #include <linux/of.h>
 #include <linux/of_irq.h>
@@ -33,6 +36,8 @@ EXPORT_SYMBOL_GPL(xen_have_vector_callback);
 int xen_platform_pci_unplug = XEN_UNPLUG_ALL;
 EXPORT_SYMBOL_GPL(xen_platform_pci_unplug);
 
+static __read_mostly int xen_events_irq = -1;
+
 int xen_remap_domain_mfn_range(struct vm_area_struct *vma,
 			       unsigned long addr,
 			       unsigned long mfn, int nr,
@@ -74,6 +79,9 @@ static int __init xen_guest_init(void)
 	if (of_address_to_resource(node, GRANT_TABLE_PHYSADDR, &res))
 		return 0;
 	xen_hvm_resume_frames = res.start >> PAGE_SHIFT;
+	xen_events_irq = irq_of_parse_and_map(node, 0);
+	pr_info("Xen %s support found, events_irq=%d gnttab_frame_pfn=%lx\n",
+			version, xen_events_irq, xen_hvm_resume_frames);
 	xen_domain_type = XEN_HVM_DOMAIN;
 
 	xen_setup_features();
@@ -115,3 +123,28 @@ static int __init xen_guest_init(void)
 	return 0;
 }
 core_initcall(xen_guest_init);
+
+static irqreturn_t xen_arm_callback(int irq, void *arg)
+{
+	xen_hvm_evtchn_do_upcall();
+	return IRQ_HANDLED;
+}
+
+static int __init xen_init_events(void)
+{
+	if (!xen_domain() || xen_events_irq < 0)
+		return -ENODEV;
+
+	xen_init_IRQ();
+
+	if (request_percpu_irq(xen_events_irq, xen_arm_callback,
+			"events", xen_vcpu)) {
+		pr_err("Error requesting IRQ %d\n", xen_events_irq);
+		return -EINVAL;
+	}
+
+	enable_percpu_irq(xen_events_irq, 0);
+
+	return 0;
+}
+postcore_initcall(xen_init_events);
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 47b3acd..689a4c9 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -33,6 +33,7 @@
 #include <linux/memblock.h>
 
 #include <xen/xen.h>
+#include <xen/events.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/version.h>
 #include <xen/interface/physdev.h>
diff --git a/arch/x86/xen/irq.c b/arch/x86/xen/irq.c
index 1573376..01a4dc0 100644
--- a/arch/x86/xen/irq.c
+++ b/arch/x86/xen/irq.c
@@ -5,6 +5,7 @@
 #include <xen/interface/xen.h>
 #include <xen/interface/sched.h>
 #include <xen/interface/vcpu.h>
+#include <xen/events.h>
 
 #include <asm/xen/hypercall.h>
 #include <asm/xen/hypervisor.h>
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index bb5a810..a95b417 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -35,7 +35,6 @@ void xen_set_pat(u64);
 
 char * __init xen_memory_setup(void);
 void __init xen_arch_setup(void);
-void __init xen_init_IRQ(void);
 void xen_enable_sysenter(void);
 void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index c60d162..8672211 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -31,14 +31,16 @@
 #include <linux/irqnr.h>
 #include <linux/pci.h>
 
+#ifdef CONFIG_X86
 #include <asm/desc.h>
 #include <asm/ptrace.h>
 #include <asm/irq.h>
 #include <asm/idle.h>
 #include <asm/io_apic.h>
-#include <asm/sync_bitops.h>
 #include <asm/xen/page.h>
 #include <asm/xen/pci.h>
+#endif
+#include <asm/sync_bitops.h>
 #include <asm/xen/hypercall.h>
 #include <asm/xen/hypervisor.h>
 
@@ -50,6 +52,9 @@
 #include <xen/interface/event_channel.h>
 #include <xen/interface/hvm/hvm_op.h>
 #include <xen/interface/hvm/params.h>
+#include <xen/interface/physdev.h>
+#include <xen/interface/sched.h>
+#include <asm/hw_irq.h>
 
 /*
  * This lock protects updates to the following mapping and reference-count
@@ -1386,7 +1391,9 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
+#ifdef CONFIG_X86
 	exit_idle();
+#endif
 	irq_enter();
 
 	__xen_evtchn_do_upcall();
@@ -1795,9 +1802,9 @@ void xen_callback_vector(void)
 void xen_callback_vector(void) {}
 #endif
 
-void __init xen_init_IRQ(void)
+void xen_init_IRQ(void)
 {
-	int i, rc;
+	int i;
 
 	evtchn_to_irq = kcalloc(NR_EVENT_CHANNELS, sizeof(*evtchn_to_irq),
 				    GFP_KERNEL);
@@ -1813,6 +1820,7 @@ void __init xen_init_IRQ(void)
 
 	pirq_needs_eoi = pirq_needs_eoi_flag;
 
+#ifdef CONFIG_X86
 	if (xen_hvm_domain()) {
 		xen_callback_vector();
 		native_init_IRQ();
@@ -1820,6 +1828,7 @@ void __init xen_init_IRQ(void)
 		 * __acpi_register_gsi can point at the right function */
 		pci_xen_hvm_init();
 	} else {
+		int rc;
 		struct physdev_pirq_eoi_gmfn eoi_gmfn;
 
 		irq_ctx_init(smp_processor_id());
@@ -1835,4 +1844,6 @@ void __init xen_init_IRQ(void)
 		} else
 			pirq_needs_eoi = pirq_check_eoi_map;
 	}
+#endif
 }
+EXPORT_SYMBOL_GPL(xen_init_IRQ);
diff --git a/include/xen/events.h b/include/xen/events.h
index 04399b2..c6bfe01 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -109,4 +109,6 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* initialize Xen IRQ subsystem */
+void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
-- 
1.7.2.5

[PATCH v5 14/17] xen/arm: implement alloc/free_xenballooned_pages with alloc_pages/kfree

[Stefano Stabellini]

Only until we get the balloon driver to work.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/xen/enlighten.c |   18 ++++++++++++++++++
 1 files changed, 18 insertions(+), 0 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index bad67ad..59bcb96 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -148,3 +148,21 @@ static int __init xen_init_events(void)
 	return 0;
 }
 postcore_initcall(xen_init_events);
+
+/* XXX: only until balloon is properly working */
+int alloc_xenballooned_pages(int nr_pages, struct page **pages, bool highmem)
+{
+	*pages = alloc_pages(highmem ? GFP_HIGHUSER : GFP_KERNEL,
+			get_order(nr_pages));
+	if (*pages == NULL)
+		return -ENOMEM;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(alloc_xenballooned_pages);
+
+void free_xenballooned_pages(int nr_pages, struct page **pages)
+{
+	kfree(*pages);
+	*pages = NULL;
+}
+EXPORT_SYMBOL_GPL(free_xenballooned_pages);
-- 
1.7.2.5

[PATCH v5 15/17] xen/arm: compile blkfront and blkback

[Stefano Stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/block/xen-blkback/blkback.c  |    1 +
 include/xen/interface/io/protocols.h |    3 +++
 2 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
index 73f196c..63dd5b9 100644
--- a/drivers/block/xen-blkback/blkback.c
+++ b/drivers/block/xen-blkback/blkback.c
@@ -42,6 +42,7 @@
 
 #include <xen/events.h>
 #include <xen/page.h>
+#include <xen/xen.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
 #include "common.h"
diff --git a/include/xen/interface/io/protocols.h b/include/xen/interface/io/protocols.h
index 01fc8ae..0eafaf2 100644
--- a/include/xen/interface/io/protocols.h
+++ b/include/xen/interface/io/protocols.h
@@ -5,6 +5,7 @@
 #define XEN_IO_PROTO_ABI_X86_64     "x86_64-abi"
 #define XEN_IO_PROTO_ABI_IA64       "ia64-abi"
 #define XEN_IO_PROTO_ABI_POWERPC64  "powerpc64-abi"
+#define XEN_IO_PROTO_ABI_ARM        "arm-abi"
 
 #if defined(__i386__)
 # define XEN_IO_PROTO_ABI_NATIVE XEN_IO_PROTO_ABI_X86_32
@@ -14,6 +15,8 @@
 # define XEN_IO_PROTO_ABI_NATIVE XEN_IO_PROTO_ABI_IA64
 #elif defined(__powerpc64__)
 # define XEN_IO_PROTO_ABI_NATIVE XEN_IO_PROTO_ABI_POWERPC64
+#elif defined(__arm__)
+# define XEN_IO_PROTO_ABI_NATIVE XEN_IO_PROTO_ABI_ARM
 #else
 # error arch fixup needed here
 #endif
-- 
1.7.2.5

[PATCH v5 16/17] xen/arm: compile netback

[Stefano Stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/arm/include/asm/xen/hypercall.h |   19 +++++++++++++++++++
 drivers/net/xen-netback/netback.c    |    1 +
 drivers/net/xen-netfront.c           |    1 +
 3 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/arch/arm/include/asm/xen/hypercall.h b/arch/arm/include/asm/xen/hypercall.h
index 4ac0624..8a82325 100644
--- a/arch/arm/include/asm/xen/hypercall.h
+++ b/arch/arm/include/asm/xen/hypercall.h
@@ -47,4 +47,23 @@ unsigned long HYPERVISOR_hvm_op(int op, void *arg);
 int HYPERVISOR_memory_op(unsigned int cmd, void *arg);
 int HYPERVISOR_physdev_op(int cmd, void *arg);
 
+static inline void
+MULTI_update_va_mapping(struct multicall_entry *mcl, unsigned long va,
+			unsigned int new_val, unsigned long flags)
+{
+	BUG();
+}
+
+static inline void
+MULTI_mmu_update(struct multicall_entry *mcl, struct mmu_update *req,
+		 int count, int *success_count, domid_t domid)
+{
+	BUG();
+}
+
+static inline int
+HYPERVISOR_multicall(void *call_list, int nr_calls)
+{
+	BUG();
+}
 #endif /* _ASM_ARM_XEN_HYPERCALL_H */
diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 682633b..1c0a302 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -40,6 +40,7 @@
 
 #include <net/tcp.h>
 
+#include <xen/xen.h>
 #include <xen/events.h>
 #include <xen/interface/memory.h>
 
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 650f79a..24d968d 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -43,6 +43,7 @@
 #include <linux/slab.h>
 #include <net/ip.h>
 
+#include <asm/xen/page.h>
 #include <xen/xen.h>
 #include <xen/xenbus.h>
 #include <xen/events.h>
-- 
1.7.2.5

[PATCH v5 17/17] MAINTAINERS: add myself as Xen ARM maintainer

[Stefano Stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Arnd Bergmann <arnd@arndb.de>
---
 MAINTAINERS |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index fdc0119..3d38291 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7752,6 +7752,13 @@ F:	drivers/xen/
 F:	arch/x86/include/asm/xen/
 F:	include/xen/
 
+XEN HYPERVISOR ARM
+M:	Stefano Stabellini <stefano.stabellini@eu.citrix.com>
+L:	xen-devel@lists.xensource.com (moderated for non-subscribers)
+S:	Supported
+F:	arch/arm/xen/
+F:	arch/arm/include/asm/xen/
+
 XEN NETWORK BACKEND DRIVER
 M:	Ian Campbell <ian.campbell@citrix.com>
 L:	xen-devel@lists.xensource.com (moderated for non-subscribers)
-- 
1.7.2.5

Re: [PATCH v5 06/17] docs: Xen ARM DT bindings

[Rob Herring]

On 09/17/2012 12:38 PM, Stefano Stabellini wrote:
> Add a doc to describe the Xen ARM device tree bindings
> 
> 
> Changes in v5:
> 
> - add a comment about the size of the grant table memory region;
> - add a comment about the required presence of a GIC node;
> - specify that the described properties are part of a top-level
> "hypervisor" node;
> - specify #address-cells and #size-cells for the example.
> 
> 
> Changes in v4:
> 
> - "xen,xen" should be last as it is less specific;
> - update reg property using 2 address-cells and 2 size-cells.
> 
> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> CC: devicetree-discuss@lists.ozlabs.org
> CC: David Vrabel <david.vrabel@citrix.com>
> CC: Rob Herring <robherring2@gmail.com>
> CC: Dave Martin <dave.martin@linaro.org>

This is so minimal now and it doesn't seem to have any overlap with kvm, so:

Acked-by: Rob Herring <rob.herring@calxeda.com>

> ---
>  Documentation/devicetree/bindings/arm/xen.txt |   25 +++++++++++++++++++++++++
>  1 files changed, 25 insertions(+), 0 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/arm/xen.txt
> 
> diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt
> new file mode 100644
> index 0000000..0f7b9c2
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/arm/xen.txt
> @@ -0,0 +1,25 @@
> +* Xen hypervisor device tree bindings
> +
> +Xen ARM virtual platforms shall have a top-level "hypervisor" node with
> +the following properties:
> +
> +- compatible:
> +	compatible = "xen,xen-<version>", "xen,xen";
> +  where <version> is the version of the Xen ABI of the platform.
> +
> +- reg: specifies the base physical address and size of a region in
> +  memory where the grant table should be mapped to, using an
> +  HYPERVISOR_memory_op hypercall. The memory region is large enough to map
> +  the whole grant table (it is larger or equal to gnttab_max_grant_frames()).
> +
> +- interrupts: the interrupt used by Xen to inject event notifications.
> +  A GIC node is also required.
> +
> +
> +Example (assuming #address-cells = <2> and #size-cells = <2>):
> +
> +hypervisor {
> +	compatible = "xen,xen-4.3", "xen,xen";
> +	reg = <0 0xb0000000 0 0x20000>;
> +	interrupts = <1 15 0xf08>;
> +};
> 

Re: [PATCH v5 00/17] Introduce Xen support on ARM (based on 3.6-rc5)

[Arnd Bergmann]

On Monday 17 September 2012, Stefano Stabellini wrote:

> I am also attaching to this email the dts'es that I am currently using
> for dom0 and domU: vexpress-v2p-ca15-tc1.dts (that includes
> vexpress-v2m-rs1-rtsm.dtsi) is the dts used for dom0 and it is passed to
> Linux by Xen, while vexpress-virt.dts is the dts used for other domUs
> and it is appended in binary form to the guest kernel image. I am not
> sure where they are supposed to live yet, so I am just attaching them
> here so that people can actually try out this series if they want to.

You can submit the dts files separately so we can include them in
the arm-soc tree. Pawel Moll is handling vexpress related changes these
days, so it would be reasonable if he included them in a pull request.
 
> The patch series has been rebased on Konrad's stable/for-linus-3.7
> branch.
> 
> Arnd, Russell, what do you think about this series? If you are OK with
> it, to whom should I submit it?

I have no objections to your patches from this series. IMHO they should
be submitted through the Xen tree, but it would be good to have an Ack
from Russell.

	Arnd

Re: [PATCH v5 00/17] Introduce Xen support on ARM (based on 3.6-rc5)

[Konrad Rzeszutek Wilk]

On Tue, Sep 18, 2012 at 12:34:29PM +0000, Arnd Bergmann wrote:
> On Monday 17 September 2012, Stefano Stabellini wrote:
> 
> > I am also attaching to this email the dts'es that I am currently using
> > for dom0 and domU: vexpress-v2p-ca15-tc1.dts (that includes
> > vexpress-v2m-rs1-rtsm.dtsi) is the dts used for dom0 and it is passed to
> > Linux by Xen, while vexpress-virt.dts is the dts used for other domUs
> > and it is appended in binary form to the guest kernel image. I am not
> > sure where they are supposed to live yet, so I am just attaching them
> > here so that people can actually try out this series if they want to.
> 
> You can submit the dts files separately so we can include them in
> the arm-soc tree. Pawel Moll is handling vexpress related changes these
> days, so it would be reasonable if he included them in a pull request.
>  
> > The patch series has been rebased on Konrad's stable/for-linus-3.7
> > branch.
> > 
> > Arnd, Russell, what do you think about this series? If you are OK with
> > it, to whom should I submit it?
> 
> I have no objections to your patches from this series. IMHO they should
> be submitted through the Xen tree, but it would be good to have an Ack
> from Russell.

OK, I can do that.

How do we get Russell to look at these patches? Bribes? Ship him some
beer/wine to be delievered to his office?

Re: [PATCH v5 00/17] Introduce Xen support on ARM (based on 3.6-rc5)

[Stefano Stabellini]

On Tue, 18 Sep 2012, Arnd Bergmann wrote:
> On Monday 17 September 2012, Stefano Stabellini wrote:
> 
> > I am also attaching to this email the dts'es that I am currently using
> > for dom0 and domU: vexpress-v2p-ca15-tc1.dts (that includes
> > vexpress-v2m-rs1-rtsm.dtsi) is the dts used for dom0 and it is passed to
> > Linux by Xen, while vexpress-virt.dts is the dts used for other domUs
> > and it is appended in binary form to the guest kernel image. I am not
> > sure where they are supposed to live yet, so I am just attaching them
> > here so that people can actually try out this series if they want to.
> 
> You can submit the dts files separately so we can include them in
> the arm-soc tree. Pawel Moll is handling vexpress related changes these
> days, so it would be reasonable if he included them in a pull request.

OK, I'll submit them as a patch series separately.


> > The patch series has been rebased on Konrad's stable/for-linus-3.7
> > branch.
> > 
> > Arnd, Russell, what do you think about this series? If you are OK with
> > it, to whom should I submit it?
> 
> I have no objections to your patches from this series. IMHO they should
> be submitted through the Xen tree,
    
Konrad, are you happy to carry this series in your tree?


>but it would be good to have an Ack from Russell.

Indeed

Re: [PATCH v5 00/17] Introduce Xen support on ARM (based on 3.6-rc5)

[Konrad Rzeszutek Wilk]

On Tue, Sep 18, 2012 at 02:57:05PM +0100, Stefano Stabellini wrote:
> On Tue, 18 Sep 2012, Arnd Bergmann wrote:
> > On Monday 17 September 2012, Stefano Stabellini wrote:
> > 
> > > I am also attaching to this email the dts'es that I am currently using
> > > for dom0 and domU: vexpress-v2p-ca15-tc1.dts (that includes
> > > vexpress-v2m-rs1-rtsm.dtsi) is the dts used for dom0 and it is passed to
> > > Linux by Xen, while vexpress-virt.dts is the dts used for other domUs
> > > and it is appended in binary form to the guest kernel image. I am not
> > > sure where they are supposed to live yet, so I am just attaching them
> > > here so that people can actually try out this series if they want to.
> > 
> > You can submit the dts files separately so we can include them in
> > the arm-soc tree. Pawel Moll is handling vexpress related changes these
> > days, so it would be reasonable if he included them in a pull request.
> 
> OK, I'll submit them as a patch series separately.
> 
> 
> > > The patch series has been rebased on Konrad's stable/for-linus-3.7
> > > branch.
> > > 
> > > Arnd, Russell, what do you think about this series? If you are OK with
> > > it, to whom should I submit it?
> > 
> > I have no objections to your patches from this series. IMHO they should
> > be submitted through the Xen tree,
>     
> Konrad, are you happy to carry this series in your tree?

Yes.
> 
> 
> >but it would be good to have an Ack from Russell.
> 
> Indeed